User:Ted

From Wiki at Neela Nurseries
Revision as of 19:40, 25 September 2017 by Ted (talk | contribs) (→‎Python Scripting Language)
Jump to: navigation, search

-- 2017-07-21 शुक्रवार --
Ted's page on Neela Nurseries Wiki


OVERVIEW

This wiki document a starting point of Ted's notes on Linux use and configuration, Open Source Software and web development. Recent summer 2017 efforts of Ted's are focused on several high-level programming language and web development frameworks, which can be studied and used separately but are often glued together to achieve meaningful tasks and end-user tools and interfaces. This personal page of Ted's notes on Neela Nurseries wiki not yet well organized, but here as a quick stash point for holding useful references, and a starting point for more complete and formal documentation.



Web Site Building Blocks - Summer 2017 Work

 
The following tools and web and programming frameworks are all part of Ted's volunteer efforts with ASI web site, and in-progress study of how to configure and customize shopping carts, article and document management pages, and easy-to-read easy-to-adjust web page layouts using CSS and third party, open source frameworks:



These are the first topics on which Ted wants to gather together notes. Hard to remember all details regarding pitfalls encountered, solutions found, and ideas for improving the configuration and use experience of these softwares . . . - TMH




^ Linux Packages

- 2017-07-31 सोमवार -


This section a starting point for notes on Linux packages, namely Debian and Ubuntu packages. Focus here is on userland software, and which packages are needed to set up various tools sets in a working Unix / Linux environment . . .

  • libreoffice-base
  • postfix . . . see section below
  • gpm
  • ctags
  • vim-doc
  • vim-scripts
  • xfonts-cyrillic
  • devhelp
  • distcc . . . distributed compiling of C and C++ programs across multiple machines


^ Postfix


^ Linux packages for Raspberry Pi system


^ Logitech wireless devices and unifying receiver


^ sudo

Ted noting sudo manual page URL here, in order to revisit and read about sudo -e as well as sudo invoked with the -l and -v options. Also noting there is an environment variable sometimes set, variable named $SUDO_USER.



^ Bash Shell Scripting

Shell scripting and use of built-in shell commands is a really practical knowledge to employ, when working in Unix-like environments. One common task is to find all the instances of a given file or program. The locate command can perform this kind of search, but it's results don't show whether the file instances differ. To check at the rough level of file size, we can use a one-line shell script technique involving shell piping, to "long list" the results of the locate command, like this . . . 


  $ for file in `locate tavrasm | grep 'asm$'`; do ls -l ${file}; done


Hmm strange, the above command calls `grep` with a pattern that ends in the shell end-of-line anchoring character $, and appears to filter for results of `locate` which end in 'asm'. But on 2017-09-12 needed to add a "one or more wildcard" character pattern to limit search results for instances of `locate` results ending in 'qemu': 


$ locate qemu | grep '.*qemu$'


Why the apparent difference in command invocation? Need to test . . . - TMH




^ Aboriginal Linux

Ok, looks like Robert Landley's Aboriginal Linux project has been superceded by mkroot. Ladnley's newer makeroot project sources are available at GitHub:



Having trouble carrying out the basic build of Aboriginal Linux with armv6l as target . . . 


   .
   .
   .

Confirmed e2fsprogs-1.42.13.tar.gz
Confirmed zlib-1.2.7.tar.bz2
Confirmed squashfs-4.2.tar.gz
=== Got all source.

real    0m2.240s
user    0m1.188s
sys     0m0.176s
=== toybox (host host-tools)
Snapshot 'toybox'...
scripts/genconfig.sh
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
                -DPROJECT_NAME=\"ToyBox\"
kconfig/conf -D /dev/null Config.in > /dev/null
scripts/make.sh
Generate headers from toys/*/*.c...
generated/newtoys.h Library probe.......
Make generated/config.h from .config.
generated/flags.h generated/globals.h generated/help.h
Compile toybox................................................................................................................................................................generated/obj/nsenter.o: In function `unshare_main':
nsenter.c:(.text.unshare_main+0x152): undefined reference to `setns'
collect2: error: ld returned 1 exit status
make: *** [toybox] Error 1

Exiting due to errors (host host-tools toybox)


Robert Landley also has a presence on www.patreon.com/landley, where he talks about the over-arching goals in his programming and software systems level work.



^ Linux From Scratch (LFS) project


Here are notes by Ted, beginning work to build a 'Linux From Scratch' system crossed-compiled for the ARM architecture, in a way suitable to run with complete function on RaspberryPi development boards. At same time Ted looking to build minimal system and tool chain, offering environment in which to compile and to cross-compile C and C++ programs. Notes here begin with some useful, initially found links to LFS home page of Gerard Beekman et al:



Linux From Scratch project observes Linux Standard Base (LSB) requirements. These requirements include,


Cross-compiling Linux system from scratch . . .


Linux from Scratch for RaspberryPi target system . . .



First mount and build steps

- 2017-90-18 MON -

Working through LFS stable release book or instructions, at end of chapter two page titled chapter 2.7 LFS book calls for first mount of newly created system-to-be partition. Ted using ext4 type journaled file system. Somewhere in or about chapter five, further mounts of other file systems of the new system are needed. Here 2017-09-25 Monday Ted noting history of mounts, which were completed prior to software compilations in chapter 6: 



ted@localhost:~$ echo $LFS
/mnt/lfs

ted@localhost:~$ history | grep mount
   .
   .
   .
  556  sudo mount -v -t ext4 /dev/sdb5 /mnt/lfs
  673  history | grep mount
  674  sudo mount -v -t ext4 /dev/sdb5 /mnt/lfs
  779  mount
  939  mount
  976  sudo mount -v --bind /dev $LFS/dev
  977  sudo mount -vt devpts devpts $LFS/dev/pts -o gid=5,mode=620
  978  mount -vt proc proc $LFS/proc
  979  sudo mount -vt proc proc $LFS/proc
  980  sudo mount -vt sysfs sysfs $LFS/sys
  981  sudo mount -vt tmpfs tmpfs $LFS/run
  986  mount
  988  sudo umount /mnt/lfs
 1001  history | grep mount

ted@localhost:~$


Ted noting specific mount steps as root partition at 7.7GB too small to permit chapter 6 builds. Ted learning parted disk partitioning utility in order to resize /dev/sdb5 root partition of 2017 September LFS project build.

After building first package binutils-2.29 the directory /tools holds directories and files in the following command line snippet of figure x. Bin-utils 2p29 took about 5 mins 30 seconds to build:


Figure x - looking at results of LFS first build step, building bin-utils 2.29: 


lfs@localhost:~$ ls /tools
bin  i686-lfs-linux-gnu  share

ted@localhost:~$ ls /tools/bin
i686-lfs-linux-gnu-addr2line  i686-lfs-linux-gnu-elfedit  i686-lfs-linux-gnu-nm       i686-lfs-linux-gnu-readelf
i686-lfs-linux-gnu-ar         i686-lfs-linux-gnu-gprof    i686-lfs-linux-gnu-objcopy  i686-lfs-linux-gnu-size
i686-lfs-linux-gnu-as         i686-lfs-linux-gnu-ld       i686-lfs-linux-gnu-objdump  i686-lfs-linux-gnu-strings
i686-lfs-linux-gnu-c++filt    i686-lfs-linux-gnu-ld.bfd   i686-lfs-linux-gnu-ranlib   i686-lfs-linux-gnu-strip

lfs@localhost:~$ ls /tools/share
info  man

lfs@localhost:~$ ls /tools/i686-lfs-linux-gnu/
bin  lib

lfs@localhost:~$ ls /tools/i686-lfs-linux-gnu/lib
ldscripts

lfs@localhost:~$ ls /tools/i686-lfs-linux-gnu/lib/ldscripts/
elf32_x86_64.x    elf32_x86_64.xdw  elf32_x86_64.xsw  elf_i386.xc   elf_i386.xr   elf_i386.xw    elf_iamcu.xdc  elf_iamcu.xsc  i386linux.xbn
elf32_x86_64.xbn  elf32_x86_64.xn   elf32_x86_64.xu   elf_i386.xd   elf_i386.xs   elf_iamcu.x    elf_iamcu.xdw  elf_iamcu.xsw  i386linux.xn
elf32_x86_64.xc   elf32_x86_64.xr   elf32_x86_64.xw   elf_i386.xdc  elf_i386.xsc  elf_iamcu.xbn  elf_iamcu.xn   elf_iamcu.xu   i386linux.xr
elf32_x86_64.xd   elf32_x86_64.xs   elf_i386.x        elf_i386.xdw  elf_i386.xsw  elf_iamcu.xc   elf_iamcu.xr   elf_iamcu.xw   i386linux.xu
elf32_x86_64.xdc  elf32_x86_64.xsc  elf_i386.xbn      elf_i386.xn   elf_i386.xu   elf_iamcu.xd   elf_iamcu.xs   i386linux.x

lfs@localhost:~$ ls /tools/i686-lfs-linux-gnu/bin
ar  as  ld  ld.bfd  nm  objcopy  objdump  ranlib  readelf  strip

lfs@localhost:~$ /tools/i686-lfs-linux-gnu/bin/ar --version
GNU ar (GNU Binutils) 2.29
Copyright (C) 2017 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

lfs@localhost:~$


Chapter 5.5, building gcc first pass, took about sixty three (63) minutes: 


   .
   .
   .
make[3]: Leaving directory `/mnt/lfs/sources/gcc-7.2.0/build/i686-lfs-linux-gnu/libgcc'
make[2]: Leaving directory `/mnt/lfs/sources/gcc-7.2.0/build/i686-lfs-linux-gnu/libgcc'
make[1]: Leaving directory `/mnt/lfs/sources/gcc-7.2.0/build'

real    63m2.166s
user    53m41.453s
sys     3m11.888s
lfs@localhost:/mnt/lfs/sources/gcc-7.2.0/build$


Time to build glibc: 


   .
   .
   .
-lgcc  `i686-lfs-linux-gnu-gcc  --print-file-name=crtend.o` /mnt/lfs/sources/glibc-2.26/build/csu/crtn.o
make[2]: Leaving directory `/mnt/lfs/sources/glibc-2.26/elf'
make[1]: Leaving directory `/mnt/lfs/sources/glibc-2.26'

real    41m19.981s
user    32m31.794s
sys     5m49.582s

lfs@localhost:/mnt/lfs/sources/glibc-2.26/build$


^ LFS 8.1 chapter 6.5 - create new system directories

Some captured terminal work, following steps in LFS 8.1 book, chapter 6.5:

Figure x - creating some of first directories in new LFS system, following chroot . . . 


I have no name!:/# ls
dev  lost+found  proc  run  sources  sys  tools

I have no name!:/# mkdir -pv /{bin,boot,etc/{opt,sysconfig},home,lib/firmware,mnt,opt}
mkdir: created directory '/bin'
mkdir: created directory '/boot'
mkdir: created directory '/etc'
mkdir: created directory '/etc/opt'
mkdir: created directory '/etc/sysconfig'
mkdir: created directory '/home'
mkdir: created directory '/lib'
mkdir: created directory '/lib/firmware'
mkdir: created directory '/mnt'
mkdir: created directory '/opt'

I have no name!:/# mkdir -pv /{media/{floppy,cdrom},sbin,srv,var}
mkdir: created directory '/media'
mkdir: created directory '/media/floppy'
mkdir: created directory '/media/cdrom'
mkdir: created directory '/sbin'
mkdir: created directory '/srv'
mkdir: created directory '/var'

I have no name!:/# install -dv -m 0750 /root
install: creating directory '/root'

I have no name!:/# install -dv -m 1777 /tmp /var/tmp
install: creating directory '/tmp'
install: creating directory '/var/tmp'

I have no name!:/# mkdir -pv /usr/{,local}{bin,include,lib,sbin,src}
mkdir: created directory '/usr'
mkdir: created directory '/usr/bin'
mkdir: created directory '/usr/include'
mkdir: created directory '/usr/lib'
mkdir: created directory '/usr/sbin'
mkdir: created directory '/usr/src'
mkdir: created directory '/usr/localbin'
mkdir: created directory '/usr/localinclude'
mkdir: created directory '/usr/locallib'
mkdir: created directory '/usr/localsbin'
mkdir: created directory '/usr/localsrc'

I have no name!:/#


LFS 8.1 book explains that the "I have no name!" results from there not yet being a /etc/passwd file in the new LFS system. Continuing the terminal capture to illustrate bash shell's pattern expansion from curly braced text: 



I have no name!:/# mkdir -pv /usr/{,local/}share/{color,dict,doc,info,locale,man}
mkdir: created directory '/usr/share'
mkdir: created directory '/usr/share/color'
mkdir: created directory '/usr/share/dict'
mkdir: created directory '/usr/share/doc'
mkdir: created directory '/usr/share/info'
mkdir: created directory '/usr/share/locale'
mkdir: created directory '/usr/share/man'
mkdir: created directory '/usr/local'
mkdir: created directory '/usr/local/share'
mkdir: created directory '/usr/local/share/color'
mkdir: created directory '/usr/local/share/dict'
mkdir: created directory '/usr/local/share/doc'
mkdir: created directory '/usr/local/share/info'
mkdir: created directory '/usr/local/share/locale'
mkdir: created directory '/usr/local/share/man'

I have no name!:/# mkdir -v  /usr/{,local/}share/{misc,terminfo,zoneinfo}
mkdir: created directory '/usr/share/misc'
mkdir: created directory '/usr/share/terminfo'
mkdir: created directory '/usr/share/zoneinfo'
mkdir: created directory '/usr/local/share/misc'
mkdir: created directory '/usr/local/share/terminfo'
mkdir: created directory '/usr/local/share/zoneinfo'

I have no name!:/# mkdir -v  /usr/libexec
mkdir: created directory '/usr/libexec'

I have no name!:/# mkdir -pv /usr/{,local/}share/man/man{1..8}
mkdir: created directory '/usr/share/man/man1'
mkdir: created directory '/usr/share/man/man2'
mkdir: created directory '/usr/share/man/man3'
mkdir: created directory '/usr/share/man/man4'
mkdir: created directory '/usr/share/man/man5'
mkdir: created directory '/usr/share/man/man6'
mkdir: created directory '/usr/share/man/man7'
mkdir: created directory '/usr/share/man/man8'
mkdir: created directory '/usr/local/share/man/man1'
mkdir: created directory '/usr/local/share/man/man2'
mkdir: created directory '/usr/local/share/man/man3'
mkdir: created directory '/usr/local/share/man/man4'
mkdir: created directory '/usr/local/share/man/man5'
mkdir: created directory '/usr/local/share/man/man6'
mkdir: created directory '/usr/local/share/man/man7'
mkdir: created directory '/usr/local/share/man/man8'
I have no name!:/#

I have no name!:/# mkdir -v /var/{log,mail,spool}
mkdir: created directory '/var/log'
mkdir: created directory '/var/mail'
mkdir: created directory '/var/spool'

I have no name!:/# ln -sv /run /var/run
'/var/run' -> '/run'

I have no name!:/# ln -sv /run/lock /var/lock
'/var/lock' -> '/run/lock'

I have no name!:/# mkdir -pv /var/{opt,cache,lib/{color,misc,locate},local}
mkdir: created directory '/var/opt'
mkdir: created directory '/var/cache'
mkdir: created directory '/var/lib'
mkdir: created directory '/var/lib/color'
mkdir: created directory '/var/lib/misc'
mkdir: created directory '/var/lib/locate'
mkdir: created directory '/var/local'

I have no name!:/#


^ LFS 8.1 chapter 6.6 - create essential files and syminks


The purpose of the following symbolic links are explained in LFS 8.1 book chapter 6.6: 



I have no name!:/# # LFS book 8.1, chapter 6.6 . . .

I have no name!:/# ln -sv /tools/bin/{bash,cat,dd,echo,ln,pwd,rm,stty} /bin
'/bin/bash' -> '/tools/bin/bash'
'/bin/cat' -> '/tools/bin/cat'
'/bin/dd' -> '/tools/bin/dd'
'/bin/echo' -> '/tools/bin/echo'
'/bin/ln' -> '/tools/bin/ln'
'/bin/pwd' -> '/tools/bin/pwd'
'/bin/rm' -> '/tools/bin/rm'
'/bin/stty' -> '/tools/bin/stty'

I have no name!:/# ln -sv /tools/bin/{install,perl} /usr/bin
'/usr/bin/install' -> '/tools/bin/install'
'/usr/bin/perl' -> '/tools/bin/perl'

I have no name!:/# ln -sv /tools/lib/libgcc_s.so{,.1} /usr/lib
'/usr/lib/libgcc_s.so' -> '/tools/lib/libgcc_s.so'
'/usr/lib/libgcc_s.so.1' -> '/tools/lib/libgcc_s.so.1'

I have no name!:/# ln -sv /tools/lib/libstdc++.{a,so{,.6}} /usr/lib
'/usr/lib/libstdc++.a' -> '/tools/lib/libstdc++.a'
'/usr/lib/libstdc++.so' -> '/tools/lib/libstdc++.so'
'/usr/lib/libstdc++.so.6' -> '/tools/lib/libstdc++.so.6'

I have no name!:/# sed 's/tools/usr/' /tools/lib/libstdc++.la > /usr/lib/libstdc++.la

I have no name!:/# ln -sv bash /bin/sh
'/bin/sh' -> 'bash'

I have no name!:/#


^ uses of patch during LFS 8.1 chapter 5


Uses of patch during LFS 8.1 chapter 5 system build steps: 


lfs@localhost:/mnt/lfs$ history | grep patch
   31  ls *patch
   36  vi glibc-2.26-fhs-1.patch
   45  patch -Np1 -i ../glibc-2.26-fhs-1.patch
  194  history | grep patch
  195  patch -Np1 -i ../bash-4.4-upstream_fixes-1.patch
  218  history | grep patch
  219  patch -Np1 -i ../bzip2-1.0.6-install_docs-1.patch
  228  patch -Np1 -i ../coreutils-8.27-i18n-1.patch 
  364  unxz patch-2.7.5.tar.xz 
  365  tar xvf patch-2.7.5.tar 
  367  cd patch-2.7.5
  373  mv patch-2.7.5 z--set-aside/
  501  history | grep patch

lfs@localhost:/mnt/lfs$



^     Aside:  LFS 8.1 chapter 6.9 - glibc-2.26 programs and libraries

Really interesting section of the LFS 8.1 book here, chapter 6.9, which describes some of the utilities which come with GNU C library project. Quoting this book these utilities include "catchsegv, gencat, getconf, getent, iconv, iconvconfig, ldconfig, ldd, lddlibc4, locale, localedef, makedb, mtrace, nscd, pldd, sln, sotruss, sprof, tzselect, xtrace, zdump, and zic".



Here is a snippet of last lines of `make check` on glibc-2.26 package: 



scripts/merge-test-results.sh -t /sources/glibc-2.26/build/ subdir-tests.sum \
  . argp assert catgets conform crypt csu ctype debug dirent dlfcn elf gmon gnulib grp gshadow hesiod iconv iconvdata inet intl io libidn libio locale localedata login malloc manual math mathvec misc nis nptl nptl_db nscd nss po posix pwd resolv resource rt setjmp shadow signal socket stdio-common stdlib streams string sunrpc support sysvipc termios time timezone wcsmbs wctype \
  > /sources/glibc-2.26/build/tests.sum
XPASS: elf/tst-protected1a
XPASS: elf/tst-protected1b
UNSUPPORTED: io/tst-open-tmpfile
UNSUPPORTED: nptl/test-cond-printers
UNSUPPORTED: nptl/test-condattr-printers
UNSUPPORTED: nptl/test-mutex-printers
UNSUPPORTED: nptl/test-mutexattr-printers
UNSUPPORTED: nptl/test-rwlock-printers
UNSUPPORTED: nptl/test-rwlockattr-printers
FAIL: posix/tst-getaddrinfo4
FAIL: posix/tst-getaddrinfo5
UNSUPPORTED: stdlib/tst-getrandom
Summary of test results:
      2 FAIL
   4108 PASS
      8 UNSUPPORTED
     29 XFAIL
      2 XPASS
make[1]: *** [Makefile:304: tests] Error 1
make[1]: Leaving directory '/sources/glibc-2.26'
make: *** [Makefile:9: check] Error 2

real    116m41.448s
user    72m16.311s
sys     21m0.223s



^ Chapter 6.10 - adjusting toolchain, respecting new glibc

Chapter 6.10 - 


root:/sources/glibc-2.26/build# mv -v /tools/bin/{ld,ld-old}
'/tools/bin/ld' -> '/tools/bin/ld-old'

root:/sources/glibc-2.26/build# mv -v /tools/$(uname -m)-pc-linux-gnu/bin/{ld,ld-old}
'/tools/i686-pc-linux-gnu/bin/ld' -> '/tools/i686-pc-linux-gnu/bin/ld-old'

root:/sources/glibc-2.26/build# mv -v /tools/bin/{ld-new,ld}
'/tools/bin/ld-new' -> '/tools/bin/ld'

root:/sources/glibc-2.26/build# ln -sv /tools/bin/ld /tools/$(uname -m)-pc-linux-gnu/bin/ld
'/tools/i686-pc-linux-gnu/bin/ld' -> '/tools/bin/ld'

root:/sources/glibc-2.26/build#



^ 6.16 - binutils-2.29 requires 4.2 GB disk space!


Ok we've run into a too small partition for our root LFS file system. 7.6GB was not sufficient, though in an early chapter (chapter 2 or 3 of the LFS 8.1 book) Ted recalls there being word that a partition between five (5) and ten (10) GB would typically be sufficient to compile and install all Linux From Scratch programs. Ted now looking to re-partitioning software options under Linux . . .



^ next LFS chapter . . .




^ QEMU Emulator

Notes on QEMU, an emulator which can help with building and configuring Linux systems to run on embedded computers and development boards. Ted noting 2017-09-08 that to compile today's latest QEMU source, a 2.10.x release, needed to install Debian stretch packages pkg-config, libglib2.0-dev, dhautoreconf. Here are some links to downloading QEMU project sources, a manual for using QEMU, and an article about emulating a RaspberryPi system by using QEMU:



So got QEMU 2.10.0 sources and dependencies installed, ran `configure` and `make` steps, and let the build process run for an hour or two. But in the end could not find any `qemu` executable. Why is the emulator itself apparently missing? Building latest QEMU under Linux . . .


So I'm following the above article author's typical build steps, and run into this message part-way down those steps, at the step of calling `make`: 

user@host-6:~/Downloads/qemu/qemu-2.10.0/bin/debug/native$ make
Makefile:21: *** This is an out of tree build but your source tree (/home/veris/Downloads/qemu/qemu-2.10.0) seems to have been used for an in-tree build. You can fix this by running "make distclean && rm -rf *-linux-user *-softmmu" in your source tree.  Stop.
user@host-6:~/Downloads/qemu/qemu-2.10.0/bin/debug/native$


Ok interesting, I know that QEMU is a complicated project with a complex build process, but clearly I don't know what I'm doing. Following the above 'distclean' command given in the error message from QEMU's makefile, I'm now able to begin an "out of tree" build as described in the "Building QEMU" article at qemu.org just above. Tired already however of QEMU project builds taking more than an hour to complete. Here is one of many articles describing how to invoke `make` to build only limited parts of QEMU project, for example just the pieces needed to emulate ARM type systems:



Ted to test this type of build soon . . .

On a different note, QEMU build has completed and the stock RaspberryPi Debian "Stretch" image is coming up. Needed to install a VNC client to interact with the emulated RaspberryPi operating and software, stored in the downloaded .img file from Rpi's web site. Here's a link to a forum post which clued Ted into Remmina VNC client:




^ Python Scripting Language



^ Other Things To Explore

Dublin Core . . .

3D Graphics, mathematics behind and programming:

PC Video, Bochs VBE Extensions:

Webvanta:



^ References


Reproducible builds, byte-wise reproducible software:

Apache2 configuration:

Couple of MediaWiki and publishing issues to look into, which came up during wiki configuration:

Some Hindi language and UTF-8 encoding references, to be factored to a dedicated wiki page later:

Linux programs and utilities:





Retrieved from "https://wiki.neelanurseries.com/index.php?title=User:Ted&oldid=193"