User:Ted

From Wiki at Neela Nurseries
Revision as of 18:44, 18 December 2017 by Ted (talk | contribs) (→‎example shell script - back up several databases)
Jump to: navigation, search

-- 2017-07-21 शुक्रवार --
User Ted's wiki page at Neela Nurseries


^ OVERVIEW

This wiki document a starting point of Ted's notes on Linux use and configuration, Open Source Software and web development. Recent summer 2017 efforts of Ted's are focused on several high-level programming language and web development frameworks, which can be studied and used separately but are often glued together to achieve meaningful tasks and end-user tools and interfaces. This personal page of Ted's notes on Neela Nurseries wiki not yet well organized, but here as a quick stash point for holding useful references, and a starting point for more complete and formal documentation.



- 2017-10-05 - Ted noting here that would be good to have a technical glossary to make easily accessible all the technical terms which arise in daily work and exploration of technologies . . .


- 2017-10-19 - TO-DO list: 

  *  [✓] complete LFS 8.1 exercise
  *  [ ] configure xserver to run stand-along without window manager (seems to be working on 1604 spare LTS host . . .)
  *  [ ] set up two mysql servers on one host
  *  [ ] write "best photo contribution" practices page for ASI
  *  [ ] code "re-use or reference text block" module for MediaWiki  <-- this needs further explanation - TMH 
  *  [ ] in local Neela Nurseries PHP code base, amend hybrid building of nav menus to include highlighting parent page of given viewed page
  *  [ ] in OpenCart 2.x find way to customize top-of-page information and links
  *  [ ] in OpenCart 2.x find way to format product lists as compact lists, like Gmail inbox or Windows 'list view' of files



^ Web Site Building Blocks - Summer 2017 Work

 
The following tools and web and programming frameworks are all part of Ted's volunteer efforts with ASI web site, and in-progress study of how to configure and customize shopping carts, article and document management pages, and easy-to-read easy-to-adjust web page layouts using CSS and third party, open source frameworks:


  • Linux package selection for constrained systems


^ MYSQL database server and client

An important MYSQL database access recovery step is to reset the database server's root password in cases where that is lost. The following statement run in a mysqld_safe --skip-grant-tables session works, while some of the more standard and MYSQL 5.7 documented password change statements fail due to a bug in MYSQL . . . 


  mysql> update mysql.user set authentication_string=password('MyNewPass') where user='root';


  • PHP set up

^ Apache2 web server configuration

  • SSL Self-signed certificate creation and config
  • Sub-domain creation under Ubuntu 16.04 LTS and similar Linux releases
  • Mediawiki work . . .
  • OpenCart 2.x install and config, 'Wish List' and 'To Do List' . . .


    ^ Cascading Style Sheets (CSS)


    ^ 2017-11-27 - Web Page Fonts

    ^ 2017-11-28 - Javascript tutorials, examples and frameworks


    These are the first topics on which Ted wants to gather together notes. Hard to remember all details regarding pitfalls encountered, solutions found, and ideas for improving the configuration and use experience of these softwares . . . - TMH



    ^ Linux Packages (Separate Article)

    Linux open source software packages now a separate article on Neela wiki, see Linux software package notes.



    ^ Bash Shell Scripting

    Shell scripting and use of built-in shell commands is a really practical knowledge to employ, when working in Unix-like environments. One common task is to find all the instances of a given file or program. The locate command can perform this kind of search, but it's results don't show whether the file instances differ. To check at the rough level of file size, we can use a one-line shell script technique involving shell piping, to "long list" the results of the locate command, like this . . . 


      $ for file in `locate tavrasm | grep 'asm$'`; do ls -l ${file}; done


    Hmm strange, the above command calls `grep` with a pattern that ends in the shell end-of-line anchoring character $, and appears to filter for results of `locate` which end in 'asm'. But on 2017-09-12 needed to add a "one or more wildcard" character pattern to limit search results for instances of `locate` results ending in 'qemu': 


    $ locate qemu | grep '.*qemu$'


    Why the apparent difference in command invocation? Need to test . . . - TMH


    ^ edit point - shell variable quoting

    - 2017-10-19 THU -

    Running into issues when need to expand shell variable in single quotes pair . . .

    Stack Exchange, Unix forum post 178411
    TLPD Advanced Bash Scripting Guide, chapter 4.2 example 4-3 double quotes preserve white space


    WIKI WISH LIST - While adding to wiki, Ted noting that would be useful to have relative-depth wiki section markers. The standard wiki section markers '== ==', '=== ===' and similar have fixed depth. That is, '== ==' is always a top-level or first-level section in a wiki document, '=== ===' is always a sub-subsection of wiki documents, and '==== ====' a sub-sub-section. When section gets re-factored into its own article, would be handy to have those section markers rise up to levels of top-most and successive section markers. - TMH



    ^ example shell script - back up several databases

     
    Here is a simple shell script to call mysqldump utility and back up several databases. Couple of things Ted wants to add to this script include script variables to hold back-up filename prefix and infix patterns, and an option to compress the MYSQL dump files . . .

    Figure x - shell script to back-up multiple MYSQL databases 

    
#!/bin/bash


DATABASE_LIST="information_schema mysql drupal_8p0 phpmyadmin wiki_database"

user="root"
pass_phrase_for_mysql="mysql_user_password"
options_extra="--skip-lock-tables"

response="n"

mode_interactive="n"



echo "shell script starting,"



for database in ${DATABASE_LIST};
do echo "backing up alta-ubuntu database $database . . .";

#    command="mysqldump --databases $database -u$user -p$pass_phrase_for_mysql $options_extra >> au-database-back-up--${database}.sql"
    command="mysqldump --databases $database -u$user -p$pass_phrase_for_mysql $options_extra"
    redirect="au-database-back-up--${database}.sql"

    if [ $mode_interactive = 'y' ]; then

        echo "build command '$command' and database back-up filename'$redirect',"
        echo "full command will be '$command' > '$redirect',"
        echo "trying running this command and redirect? [y/N/q]  yes, no, 'q' to quit"

        read response
        if [ $response = 'Y' -o $response = 'y' ]; then
            ${command} > $redirect
        elif [ $response = 'N' -o $response = 'n' ]; then
            echo "skipping present command . . ."
        elif [ $response = 'Q' -o $response = 'q' ]; then
            echo "stopping script '$0' and exiting."
            break
        fi
        echo

    else

        echo "$0:  backing up database '$database' to file '$redirect' . . ."
        ${command} > $redirect
        ls -l $redirect

    fi

done


echo "done."

exit 0

    show kernel version script

    Figure x - simple bash script to show kernel version in root of kernel sources tree 

    #!/bin/bash

## - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
## Started 2017-12-15 FRI - script to parse and show Linux kernel
##  version from top-level makefile in set of kernel sources.  Patterns
##  to `grep` chosen based on kernel version identifiers in first
##  three lines of typical kernel top-level makefile.  Example:
##
##  $ head -n 6 Makefile
##  VERSION = 4
##  PATCHLEVEL = 9
##  SUBLEVEL = 66
##  EXTRAVERSION =
##  NAME = Roaring Lionus
##
##
## - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


if [ ]; then
    head -n 5 Makefile | grep ^VERSION | cut -d " " -f 3
    head -n 5 Makefile | grep ^PATCH | cut -d " " -f 3
    head -n 5 Makefile | grep ^SUBLEVEL | cut -d " " -f 3
fi

MAJOR=$(head -n 5 Makefile | grep ^VERSION | cut -d " " -f 3)
MINOR1=$(head -n 5 Makefile | grep ^PATCH | cut -d " " -f 3)
MINOR2=$(head -n 5 Makefile | grep ^SUBLEVEL | cut -d " " -f 3)

KERNEL_REVISION="${MAJOR}.${MINOR1}.${MINOR2}"

echo
echo "Kernel version in present kernel sources tree, per makefile, is ${KERNEL_REVISION}"
echo


exit 0



    ^ Python Scripting Language



    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    ^ Aboriginal Linux - project now defunct, new project is 'makeroot'

    Ok, looks like Robert Landley's Aboriginal Linux project has been superceded by mkroot. Ladnley's newer makeroot project sources are available at GitHub:



    Having trouble carrying out the basic build of Aboriginal Linux with armv6l as target . . . 

    
   .
   .
   .

Confirmed e2fsprogs-1.42.13.tar.gz
Confirmed zlib-1.2.7.tar.bz2
Confirmed squashfs-4.2.tar.gz
=== Got all source.

real    0m2.240s
user    0m1.188s
sys     0m0.176s
=== toybox (host host-tools)
Snapshot 'toybox'...
scripts/genconfig.sh
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
                -DPROJECT_NAME=\"ToyBox\"
kconfig/conf -D /dev/null Config.in > /dev/null
scripts/make.sh
Generate headers from toys/*/*.c...
generated/newtoys.h Library probe.......
Make generated/config.h from .config.
generated/flags.h generated/globals.h generated/help.h
Compile toybox................................................................................................................................................................generated/obj/nsenter.o: In function `unshare_main':
nsenter.c:(.text.unshare_main+0x152): undefined reference to `setns'
collect2: error: ld returned 1 exit status
make: *** [toybox] Error 1

Exiting due to errors (host host-tools toybox)


    Robert Landley also has a presence on www.patreon.com/landley, where he talks about the over-arching goals in his programming and software systems level work.



    ^ Linux From Scratch (Separate Article)

    Linux From Scratch Book 8.1 - Build and Notes



    ^ QEMU Emulator

    Overview

    Notes on QEMU, an emulator which can help with building and configuring Linux systems to run on embedded computers and development boards. Ted noting 2017-09-08 that to compile today's latest QEMU source, a 2.10.x release, needed to install Debian stretch packages pkg-config, libglib2.0-dev, dhautoreconf. Here are some links to downloading QEMU project sources, a manual for using QEMU, and an article about emulating a RaspberryPi system by using QEMU:


    References:

    QEMU to emulate Arm on Linux . . .

    How to avoid "raw image format" warning . . .


    So got QEMU 2.10.0 sources and dependencies installed, ran `configure` and `make` steps, and let the build process run for an hour or two. But in the end could not find any `qemu` executable. Why is the emulator itself apparently missing? Building latest QEMU under Linux . . .


    So I'm following the above article author's typical build steps, and run into this message part-way down those steps, at the step of calling `make`: 

    user@host-6:~/Downloads/qemu/qemu-2.10.0/bin/debug/native$ make
Makefile:21: *** This is an out of tree build but your source tree (/home/veris/Downloads/qemu/qemu-2.10.0) seems to have been used for an in-tree build. You can fix this by running "make distclean && rm -rf *-linux-user *-softmmu" in your source tree.  Stop.
user@host-6:~/Downloads/qemu/qemu-2.10.0/bin/debug/native$


    Ok interesting, I know that QEMU is a complicated project with a complex build process, but clearly I don't know what I'm doing. Following the above 'distclean' command given in the error message from QEMU's makefile, I'm now able to begin an "out of tree" build as described in the "Building QEMU" article at qemu.org just above. Tired already however of QEMU project builds taking more than an hour to complete. Here is one of many articles describing how to invoke `make` to build only limited parts of QEMU project, for example just the pieces needed to emulate ARM type systems:



    Ted to test this type of build soon . . .

    On a different note, QEMU build has completed and the stock RaspberryPi Debian "Stretch" image is coming up. Needed to install a VNC client to interact with the emulated RaspberryPi operating and software, stored in the downloaded .img file from Rpi's web site. Here's a link to a forum post which clued Ted into Remmina VNC client:



    - 2017-11-21 Tuesday -

    The version of qemu which installed from Debian's package list didn't seem to work. It's been some weeks now since trying that instance, but qemu 2.10.0 works to bring up a command line image built for RaspberryPi. Here are the versions of the two qemu instances:

    Figure x - locating qemu instances, determining their versions 

    
user@localhost:~$ which qemu
/usr/bin/qemu

user@localhost:~$ /usr/bin/qemu --version
QEMU emulator version 1.1.2 (Debian 1.1.2+dfsg-6+deb7u23), Copyright (c) 2003-2008 Fabrice Bellard

user@localhost:~$ ls /opt/qemu
qemu-system-arm

user@localhost:~$ ls -l /opt/qemu/qemu-system-arm
lrwxrwxrwx 1 root root 83 Sep 12 16:37 /opt/qemu/qemu-system-arm -> /home/user/Downloads/qemu/qemu-2.10.0/bin/debug/native/arm-softmmu/qemu-system-arm

user@localhost:~$ /opt/qemu/qemu-system-arm --version
QEMU emulator version 2.10.0
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

user@localhost:~


    - 2017-11-22 Wednesday -

    Yesterday could not bring up raspbian image in qemu, same image which booted and appeared, and permitted shell based login and shell use a month ago on the local host. Looking at a new on-line reference for running raspbian images in qemu: 


      *  https://ownyourbits.com/2017/02/06/raspbian-on-qemu-with-network-access/


    This above article looks really interesting, appears to go to a second installment of the article where the developers there are building some kind of cloud-based or cloud involved web, database, mail and PHP servers. But on my end having trouble with a kernel panic this morning when invoking qemu-system-arm with a large set of options . . . ok, just located a different invocation of qemu-system-arm version 2.10.0, noting here in following code figure:


    Figure x - emulate-rpi shell script 

    
#!/bin/bash
# Starts raspberry pi image in configuration mode


##----------------------------------------------------------------------
## - SECTION - script variables
##----------------------------------------------------------------------

EMULATOR=/home/veris/Downloads/qemu/qemu-2.10.0/bin/debug/native/arm-softmmu/qemu-system-arm



##----------------------------------------------------------------------
## - SECTION - original emulator invocation, everything hard-coded
##----------------------------------------------------------------------

# qemu-system-arm -kernel ./qemu-rpi-kernel/kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw init=/bin/bash" -hda rpi.img

# qemu-system-arm -kernel ./qemu-rpi-kernel/kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" -hda rpi.img



##----------------------------------------------------------------------
## - SECTION - invocation, emulator as variable:
##----------------------------------------------------------------------

# ${EMULATOR} -kernel ./qemu-rpi-kernel/kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw init=/bin/bash" -hda rpi.img

# Start RaspberryPi in fully functional mode:
${EMULATOR} -kernel ./qemu-rpi-kernel/kernel-qemu -cpu arm1176 -m 256 -M versatilepb -no-reboot -serial stdio -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" -hda rpi.img






## - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
## 2017-09-08 FRI - this script copied from http://embedonix.com/articles/linux/emulating-raspberry-pi-on-linux/
## - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

exit 0


    The final token rpi.img points to localhost:/home/user/Downloads/raspberrypi/2017-08-16-raspbian-stretch-lite.img which we downloaded from https://www.raspberrypi.org/downloads/raspbian/. The working qemu invocation doesn't open qemu's native window, but puts to standard out that is the same shell in which qemu invoked, a couple of lines of output from the booting kernel image. Then using remmina we connect via "VNC server running on ::1:5900".

    In summary shell script in latest figure successfully invokes qemu-system-arm 2.10.0 built from sources about August 2017, and boots up Raspbian Stretch lite image and kernel named kernel-qemu-4.4.34-jessie. Next figure is screenshot showing successful login, and last login message . . . though emulation was and is very slow on host system!


    Figure x - emulated raspbian stretch lite screenshot, Remmina window

    media:z.png


    A shorter script to start raspberrypi kernel plus image, running in QEMU: 

    #!/bin/bash

qemu-system-arm -kernel ./kernel-qemu-4.4.34-jessie -cpu arm1176 -m 256 -M versatilepb -serial stdio -append "root=/dev/sda2 rootfstype=ext4 rw" -hda ./2017-09-07-raspbian-stretch-lite.img -redir tcp:5022::22 -no-reboot

exit 0


    - 2017-12-12 WED -

    An excerpt from qemu.org/download#source, on how to download, configure and compile latest QEMU sources: 

    "
Build instructions

To download and build QEMU 2.11.0-rc5:

wget https://download.qemu.org/qemu-2.11.0-rc5.tar.xz
tar xvJf qemu-2.11.0-rc5.tar.xz
cd qemu-2.11.0-rc5
./configure
make

To download and build QEMU from git:

git clone git://git.qemu.org/qemu.git
cd qemu
git submodule init
git submodule update --recursive
./configure
make

The latest development happens on the master branch. The stable trees are located in branches named stable-X.YY branch, where X.YY is the release version.
"



    ^ * * * * * Embedded Linux Notes (Separate Article) * * * * *

    On this wiki Ted's embedded Linux notes now factored into their own article, Embedded Linux Notes . . .


    ^ FreeRTOS Notes (Separate Article)

    FreeRTOS notes

    Ted to move this link to nn wiki article named 'FreeRTOS':


    NOTE 2017-12-10: this section 'embedded operating systems' to become a shorter list of major significant embedded OS's. As of 2017 winter Ted studying couple distributions of embedded Linux and FreeRTOS.



    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This section marks a change of topics, from specific technical topics to more world, human and creative area topics.



    ^ Other Things To Explore


    Here are some technical and science topics to explore, perhaps on a rainy day . . .


    3D Graphics, mathematics behind and programming:


    Creative Commons and other Open Source licensed materials:


    Dublin Core:


    ^ edit point

    - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  


    Javascript and JSON:


    PC Video, Bochs VBE Extensions:


    Perl programming related:


    Unix and Linux building blocks:

    • nss, Name Service Switch
    • "Burrows-Wheeler block sorting text compression algorithm", noted 2017-09-29 FRI from LFS 8.1 chapter 6.21 - TMH


    Webvanta:


    Virtual server image creation:


    Github Users Sharing Their Work:


    ^ edit point

    - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  

    Cryptography Scientists and Articles:




    ^ Community, Culture, the World

    ^ Human languages


    - 2017-12-15 FRI -



    ^ References


    Reproducible builds, byte-wise reproducible software:


    Server Side services . . .


    Couple of MediaWiki and publishing issues to look into, which came up during wiki configuration:

    Some Hindi language and UTF-8 encoding references, to be factored to a dedicated wiki page later:

    Linux programs and utilities:


    Computer hardware and firmware:


    QEMU

    QEMU]



    - - - top of page - - -


    • Retrieved from "https://wiki.neelanurseries.com/index.php?title=User:Ted&oldid=638"