Difference between revisions of "Shell"

From Wiki at Neela Nurseries
Jump to navigation Jump to search
m (Amended script to show set of files changes across a series of commits)
m (Bash scriptlets : comment the script)
 
(23 intermediate revisions by the same user not shown)
Line 4: Line 4:
  
 
This local page for Bash shell scripts and notes on shell scripting.
 
This local page for Bash shell scripts and notes on shell scripting.
 +
 +
== [[#top|^]] Shell Resources ==
 +
 +
*  https://tldp.org/guides.html . . . look for Mendel Cooper bash guide here.
 +
 +
*  https://www.shellcheck.net/ . . . run shell scripts online here for error checking purposes.
  
 
== [[#top|^]] Bash Built-in Variables ==
 
== [[#top|^]] Bash Built-in Variables ==
Line 32: Line 38:
  
 
<!-- odne komentar -->
 
<!-- odne komentar -->
−
== [[#top|^]] Rename Badly Named Files ==
+
 
 +
== [[#top|^]] Getopt and Getopts ==
 +
 
 +
The shell page section to cover `getopt` and `getopts`, two libraries or facilities available to Bash shell scripts to process command line arguments.
 +
 
 +
An introduction to `getopt` at Labex dot io:
 +
 
 +
*  https://labex.io/tutorials/shell-bash-getopt-391993
 +
 
 +
<!-- odne komentar -->
 +
 
 +
== [[#top|^]] Variables ==
 +
 
 +
An interesting use of the shell with variable assignment and multiple variable assignments:
 +
 
 +
* https://www.baeldung.com/linux/bash-multiple-variable-assignment
 +
 
 +
Bash 'declare' built-in or keyword, makes intended variable type explicit, if not truly lasting during script run time:
 +
 
 +
*  https://unix.stackexchange.com/questions/510220/what-is-declare-in-bash
 +
 
 +
Comparing bash 'set' and 'declare' built-ins produces almost identical output, at least sometimes:
 +
 
 +
<pre>
 +
ted@localhost1:~$ declare >> 1.txt
 +
ted@localhost1:~$ set >> 2.txt
 +
ted@localhost1:~$ ls -l 1.txt 2.txt
 +
-rw-rw-r-- 1 ted ted 88439 Jan 15 14:01 1.txt
 +
-rw-rw-r-- 1 ted ted 88434 Jan 15 14:01 2.txt
 +
ted@localhost1:~$ diff 1.txt 2.txt
 +
126c126
 +
< _=--color=auto
 +
---
 +
> _=declare
 +
ted@localhost1:~$
 +
</pre>
 +
 
 +
=== [[#top|^]] shell arrays ===
 +
 
 +
Shell arrays can be indexed or associated (pairs of keys and values).  A good introduction to these things is here:
 +
 
 +
*  https://www.gnu.org/software/bash/manual/html_node/Arrays.html
 +
 
 +
== [[#top|^]] Tests ==
 +
 
 +
*  https://unix.stackexchange.com/questions/717758/grouping-of-bash-conditionals
 +
 
 +
<!--
 +
if [[ $exec_rc -eq 1 && (-z $keymap_actm || -n $modmap_actm) ]]; then . . .
 +
-->
 +
 
 +
== [[#top|^]] Examples ==
 +
 
 +
Some example scripts
 +
 
 +
=== [[#top|^]] rename badly named files ===
  
 
How to produce a usable filename which contains <ESC> characters, and to rename that badly named file:
 
How to produce a usable filename which contains <ESC> characters, and to rename that badly named file:
Line 44: Line 105:
 
<!-- odne komentar -->
 
<!-- odne komentar -->
 
<span id="nn_anchor__shell__helper_script_git_diff-tree"></span>
 
<span id="nn_anchor__shell__helper_script_git_diff-tree"></span>
−
== [[#top|^]] Git Diff-tree Helper Script ==
+
=== [[#top|^]] Git diff-tree helper script ===
  
 
A helper script for calling `git diff-tree`, which can be used to determine at which project commit one or more files have changed:
 
A helper script for calling `git diff-tree`, which can be used to determine at which project commit one or more files have changed:
Line 140: Line 201:
 
</pre>
 
</pre>
  
−
<!--
+
=== [[#top|^]] shell logging helper ===
 +
 
 +
Bash and related shell programming definitely have their limitations, with numeric and mathematical handling, name spaces and more.  Despite these shortfalls we still want to debug shell programs.  Noted here is an idea to collect some logging shell functions in a script which can be sourced by other shell scripts.
 +
 
 +
Code excerpt, not complete:
 +
 
 +
<pre>
 +
#!/bin/bash
 +
 
 +
# @brief Bash script fragment, collection of functions to aid with shell script
 +
#  debugging.
 +
#
 +
# @note Script adopts the notion of log levels as implemented in Zephyr RTOS.
 +
#  From most critical to most verbose these levels are:
 +
#
 +
#  Error, Warning, Info, Debug
 +
#
 +
#  ERR
 +
#  WRN
 +
#  INF
 +
#  DBG
 +
 
 +
function set_script_log_level()
 +
{
 +
 
 +
}
 +
 
 +
# @brief This function expects:
 +
#
 +
# $1 . . . message to print
 +
# $2 . . . log level used to filter printing
 +
 
 +
function diag()
 +
{
 +
 
 +
}
 +
</pre>
  
 +
<!-- ************************************************************************************************************ --> <!--
 
#!/bin/bash
 
#!/bin/bash
  
 +
# TODO [ ] Add sanity checking and modestly vigorous search for system
 +
#  git and file utilities:
 
CMD_GIT=/usr/bin/git
 
CMD_GIT=/usr/bin/git
 +
CMD_TREE=/usr/bin/tree
  
 
# Changed file filter, may differ on various work stations:
 
# Changed file filter, may differ on various work stations:
Line 154: Line 255:
 
     echo
 
     echo
 
     echo "  $ $0 <count> [-u|-d]"
 
     echo "  $ $0 <count> [-u|-d]"
−
     echo "  $ $0 <count> [--show-dirs --make-dirs] <dest_dit> # <- both long options required here"
+
     echo "  $ $0 <count> [--show-dirs --make-dirs] <dest_dir> # <- both long options required here"
 
     echo "  $ $0 <count> -c <dest_dir>"
 
     echo "  $ $0 <count> -c <dest_dir>"
 
     echo
 
     echo
 
     echo "count . . . count of commits to review from tip of git branch"
 
     echo "count . . . count of commits to review from tip of git branch"
 +
    echo
 
     echo "-u    . . . optionally summarize file changes to list of unique files changed"
 
     echo "-u    . . . optionally summarize file changes to list of unique files changed"
 +
    echo
 +
    echo "-d    . . . output essentially like -u.  To be updated."
 +
    echo
 +
    echo "Examples:"
 +
    echo
 +
    echo "  $ helper-git-diff-tree-extended.sh 5"
 +
    echo
 +
    echo "...to report per commit a list of files changed and some git details"
 +
    echo "including git's internal file hashes.  This helpful in highlighting"
 +
    echo "files deleted and newly created files, whose commit hashes before and"
 +
    echo "after the given commit respectively end as all zeroes (file deletion)"
 +
    echo "and begin as all zeroes (file creation)."
 +
    echo
 +
    echo "As a side note, file renaming in git involves a file deleation"
 +
    echo "operation followed immediately by a file creation operation."
 +
    echo
 +
    echo "  $ helper-git-diff-tree-extended.sh 5 -u"
 +
    echo
 +
    echo "...to show a list of relative paths ending in names of files changed"
 +
    echo "anywhere in the commit series of the latest 'count' commits of the"
 +
    echo "local repo at its current commit."
 +
    echo
 
}
 
}
  
Line 164: Line 288:
 
{
 
{
 
     echo "function f1 starting"
 
     echo "function f1 starting"
−
#    $CMD_GIT log --oneline | head | cut -d " " -f 1
 
−
#    echo
 
−
 
 
     hashes=`$CMD_GIT log --oneline | head -n ${1} | cut -d " " -f 1`
 
     hashes=`$CMD_GIT log --oneline | head -n ${1} | cut -d " " -f 1`
 
     echo "In $PWD found git commit hashes:"
 
     echo "In $PWD found git commit hashes:"
 
     echo " " $hashes
 
     echo " " $hashes
  
−
#echo "Files changed between commit pairs youngest pairings to oldest:
 
−
#for hash in $hashes; do echo "Files changed in git commit:"; git diff-tree -r $hash; echo; done
 
 
     for hash in $hashes
 
     for hash in $hashes
 
         do echo "Files changed in git commit:"
 
         do echo "Files changed in git commit:"
Line 187: Line 306:
 
}
 
}
  
 +
# @brief This function reports a list of files changed across a range of
 +
#  commits, prepending relative path from current working dir to file.
 +
#
 
# This function expects:
 
# This function expects:
 
# $1 . . . count of commits to check, starting from branch tip
 
# $1 . . . count of commits to check, starting from branch tip
Line 198: Line 320:
 
}
 
}
  
 +
# TODO [ ] trim terminating dir separators, e.g. foward slash characters
 +
#  from passed scratch directory path in which to create detected
 +
#  paths holding changed files.
 +
 +
# @brief This function recreates in a scratch directory each relative path to
 +
#  a changed file, those files being unique changed files across series of
 +
#  commits.
 +
#
 
# This function expects:
 
# This function expects:
 
# $1 . . . count of commits to check, starting from branch tip
 
# $1 . . . count of commits to check, starting from branch tip
Line 205: Line 335:
 
{
 
{
 
     i=1
 
     i=1
−
#    for hash in $hashes; do echo "("$i")" $hash; (( i++ )); done
+
    commit_count=$1
−
 
+
    scratch_dir=$2
−
#    awk -F "/" ' { print NF-1 } ' # . . . provides count of dirs
 
−
#    dirs=`sed 's/\//\ /g'`        # . . . provides dirnames and filename per path
 
−
 
 
 
     echo "f4 called with $# arguments,"
 
     echo "f4 called with $# arguments,"
−
     echo "caller wants to recreate dir structures in '${2}'"
+
    echo "caller checking $commit_count commits,"
 +
     echo "caller wants relpaths created in '$scratch_dir' . . ."
  
−
     files=`shows_changed_files_per_commit ${1} | \
+
     files=`shows_changed_files_per_commit $commit_count | \
 
       grep ${MATCH_CHANGED_FILE_LINE} | cut -f 2 | sort --unique`
 
       grep ${MATCH_CHANGED_FILE_LINE} | cut -f 2 | sort --unique`
  
 
# TODO [ ] find a way to remove just the filename, then call `mkdir -pv one_or_more_dirs`
 
# TODO [ ] find a way to remove just the filename, then call `mkdir -pv one_or_more_dirs`
−
    # echo ${files}
 
 
     for file in ${files}; do
 
     for file in ${files}; do
 
         dir_count=`echo $file | awk -F "/" ' { print NF-1 } '`
 
         dir_count=`echo $file | awk -F "/" ' { print NF-1 } '`
 
         if [ $dir_count -gt 0 ]; then
 
         if [ $dir_count -gt 0 ]; then
−
            # dirs=`sed 's/\//\ /g'`
 
 
             dirs=`echo ${file} | rev | sed 's/\//\ /' | rev | cut -d " " -f 1`
 
             dirs=`echo ${file} | rev | sed 's/\//\ /' | rev | cut -d " " -f 1`
 +
            echo "($i)"
 
             echo "(d) ${file} has dirs ${dirs}"
 
             echo "(d) ${file} has dirs ${dirs}"
−
             echo "(c)  creating ${2}/${dirs} . . ."
+
             echo "(c)  creating $scratch_dir/${dirs} . . ."
−
             mkdir -pv ${2}/${dirs}
+
             mkdir -pv $scratch_dir/${dirs}
 
             (( i++ ))
 
             (( i++ ))
 
         fi
 
         fi
Line 231: Line 358:
 
}
 
}
  
 +
# @brief This function recreates each file changed across a series of commits,
 +
#  in a recreated dir path matching its relative path from current working dir.
 +
#
 +
# @note The copied files in recreated relative paths can be displayed by the
 +
#  `tree` command to highlight file changes across commit series more
 +
#  graphically.
 +
#
 
# This function expects:
 
# This function expects:
 
# $1 . . . count of commits to check, starting from branch tip
 
# $1 . . . count of commits to check, starting from branch tip
Line 237: Line 371:
 
function copy_relpaths_and_file()
 
function copy_relpaths_and_file()
 
{
 
{
−
     echo "f5 called with $# arguments,"
+
     i=1
−
    echo "caller wants to copy relative paths and changed files to '${2}'"
 
 
     commit_count=$1
 
     commit_count=$1
 
     scratch_dir=$2
 
     scratch_dir=$2
 +
    echo "f5 called with $# arguments,"
 +
    echo "caller wants to copy relative paths and changed files to '$scratch_dir'"
  
 
     files=`shows_changed_files_per_commit $1 | \
 
     files=`shows_changed_files_per_commit $1 | \
Line 248: Line 383:
 
         dir_count=`echo $file | awk -F "/" ' { print NF-1 } '`
 
         dir_count=`echo $file | awk -F "/" ' { print NF-1 } '`
 
         if [ $dir_count -gt 0 ]; then
 
         if [ $dir_count -gt 0 ]; then
−
            # dirs=`sed 's/\//\ /g'`
 
 
             dirs=`echo $file | rev | sed 's/\//\ /' | rev | cut -d " " -f 1`
 
             dirs=`echo $file | rev | sed 's/\//\ /' | rev | cut -d " " -f 1`
 
             echo "(d) $file has dirs $dirs"
 
             echo "(d) $file has dirs $dirs"
 
             echo "(c)  creating $scratch_dir/$dirs . . ."
 
             echo "(c)  creating $scratch_dir/$dirs . . ."
 
             mkdir -pv $scratch_dir/$dirs
 
             mkdir -pv $scratch_dir/$dirs
−
 
 
             cp -pv $file $scratch_dir/$dirs
 
             cp -pv $file $scratch_dir/$dirs
−
 
 
             (( i++ ))
 
             (( i++ ))
 
         else
 
         else
−
             cp -pv $file $scratch_dir//$file
+
             cp -pv $file $scratch_dir/$file
 
         fi
 
         fi
 
     done
 
     done
 +
 +
    echo
 +
    echo "Summary of files changed in commit series, and their final"
 +
    echo "relative locations:"
 +
    echo
 +
    $CMD_TREE $scratch_dir
 
}
 
}
  
−
#
+
# ----------------------------------------------------------------------
 
# - SECTION - entry point akin to int main
 
# - SECTION - entry point akin to int main
−
#
+
# ----------------------------------------------------------------------
  
 
echo "2024-01-31 git diff-tree helper script in progress . . ."
 
echo "2024-01-31 git diff-tree helper script in progress . . ."
Line 287: Line 425:
 
fi
 
fi
  
 +
exit $?
 +
-->
 +
<!-- ************************************************************************************************************ -->
 +
 +
=== [[#top|^]] aliases fragment ===
 +
 +
<pre>
 +
function set_custom_shell_aliases()
 +
{
 +
## - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 +
## Note:  from recent experimentation and reading, shell alias setting
 +
##  does not lend itself to being sourced in a script other than
 +
##  .bashrc, at least in the case of bash shell environment.  For this
 +
##  reason restoring custom shell aliases to dot-bashrc-amendments.sh
 +
##  file - TMH
 +
## - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 +
 +
    echo "setting custom shell aliases . . ."
 +
 +
    alias gw='mv ${HOME}/.ssh/config ${HOME}/.ssh/archive; ln -s ${HOME}/.ssh/config--work ${HOME}/.ssh/config; echo "switching ssh auth to github work account"'
 +
    alias gp='mv ${HOME}/.ssh/config ${HOME}/.ssh/archive; ln -s ${HOME}/.ssh/config--personal ${HOME}/.ssh/config; echo "switching ssh auth to github personal account"'
 +
# for safety:
 +
    alias rm='rm -i'
 +
 +
    alias xscr='xscreensaver-command -lock'
 +
 +
# 2024-01-04 THU
 +
    alias vu='ps -u ted x | grep " vi"'
 +
 +
# 2024-12-16 MON - list FTDI like devices
 +
    alias lsf='ls -l /dev/ttyUSB*; ls -l /dev/ttyACM*'
 +
 +
# 2025-02-05 WED - git log --all --graph --oneline
 +
    alias gla='git log --all --graph --oneline'
 +
 +
# 2025-10-26 SUN
 +
    alias xscr='xscreensaver-command -lock'
 +
    alias lgrep='grep -nr --exclude-dir=build $@'
 +
 +
    echo ". . . done."
 +
}
 +
</pre>
 +
<!-- odne komentar - [[#top|^]] -->
 +
 +
=== [[#top|^]] to source versus to call ===
 +
 +
A script fragment which acts as an entry point of a script and treats the absence of all arguments as sign that the script is being sourced, rather than called:
 +
 +
<pre>
 +
SCRIPT_NAME=`echo $0 | sed -e 's,^.*/,,'`
 +
 +
SCRIPT_NAME_WHEN_SOURCED="check-path-existence.sh"
 +
 +
EXPECTED_ARG_COUNT=3
 +
 +
if [ $# -eq $EXPECTED_ARG_COUNT ]; then
 +
    SCRIPT_STATUS=0
 +
elif [ $# -eq 0 ]; then
 +
    echo "[INF] $SCRIPT_NAME_WHEN_SOURCED received no arguments, assuming we "
 +
    echo "      are being sourced and not called."
 +
    echo "[INF] (this means we won't run most of this script's code.)"
 +
    SCRIPT_STATUS=300
 +
elif [[ $# -le $EXPECTED_ARG_COUNT && $# -ne 0 ]]; then
 +
    echo "[ERR] $SCRIPT_NAME requires $EXPECTED_ARG_COUNT arguments, got $#."
 +
    SCRIPT_STATUS=400
 +
    help
 +
elif [[ $# -gt $EXPECTED_ARG_COUNT && $# -ne 0 ]]; then
 +
    echo "[WRN] $SCRIPT_NAME expects $EXPECTED_ARG_COUNT arguments, got $#."
 +
    echo "[WRN] Extra arguments are ignored."
 +
fi
 +
</pre>
 +
 +
== [[#top|^]] Bash scriptlets ==
  
 +
The following scripts are especially short, sometimes just one command.  They could be implemented as shell aliases but are presented here as independent scripts.
 +
 +
<i>
 +
Script:  show most recently updated git branches
 +
</i>
 +
<pre>
 +
#!/bin/bash
 +
# How to use:  name this script something short, for example `gbl`, and put it in the shell's search path.
 +
# Where to use:  in a local git repository `gbl` shows the ten most recently modified git branchs.
 +
git branch --sort=committerdate | tail
 
exit $?
 
exit $?
−
-->
+
</pre>
 +
 
 +
== [[#top|^]] Bash helper functions ==
 +
 
 +
Common helper functions in bash scripts:
 +
<pre>
 +
function help()
 +
{
 +
    # echo script usage
 +
}
 +
 
 +
function exit_early()
 +
{
 +
    # echo script name, echo status of last command
 +
}
 +
 
 +
function set_local_vars()
 +
{
 +
    # set variables particular to given script, but note they're global
 +
    # (good to rename this function.)
 +
}
 +
</pre>
 +
 
 +
Some shell scripts to provide handy localhost commands, for embedded work on a Linux host:
 +
 
 +
#!/bin/bash
 +
 
 +
grep -nr --exclude-dir=.git --exclude-dir=build --exclude-dir=docker_cache --exclude-dir=twister-out* $1 | grep $1
  
−
<!-- odne komentar -->
+
# exit $?
  
 
== [[#top|^]] To Research ==
 
== [[#top|^]] To Research ==
Line 302: Line 550:
  
 
`zephyr/include/zephyr/toolchain/xcc_missing_defs.h`
 
`zephyr/include/zephyr/toolchain/xcc_missing_defs.h`
 +
 +
Shell command substitution backticks versus braces . . .
 +
 +
*  https://stackoverflow.com/questions/22709371/backticks-vs-braces-in-bash
  
 
<!-- odne komentar -->
 
<!-- odne komentar -->
  
 
<!--
 
<!--
 +
https://stackoverflow.com/questions/22709371/backticks-vs-braces-in-bash
 +
 +
# later on in the man page:
 +
When the old-style backquote form of substitution is used, backslash retains its literal meaning except when followed by $, `, or \. The first backquote not preceded by a backslash terminates the command substitution. When using the $(command) form, all characters between the parentheses make up the command; none are treated specially.
 +
 +
-->
 +
 +
<!--
 +
2026-01-18 domingo
 +
-->
 +
 +
<!--
 +
*************************************************************************************************************************
  
 
#!/bin/bash
 
#!/bin/bash
  
−
export AWS_PROFILE=prod
+
function script_help()
−
echo "starting, exporting AWS_PROFILE=$AWS_PROFILE . . ."
+
{
 +
    echo "Help for append-tests.sh:"
 +
    echo
 +
    echo "-h --help              . . . this help message"
 +
    echo "-c --create-test-files . . . create small files for append tests"
 +
    echo "-f --file-list        . . . specify file from which to read list"
 +
    echo "                              files to process"
 +
    echo
 +
}
 +
 
 +
function create_test_files()
 +
{
 +
    for file in a.txt b.txt c.txt; do
 +
        cat <<'EOL' > $file
 +
aaaa
 +
bbbb
 +
cccc
 +
EOL
 +
    done
 +
}
 +
 
 +
function debug_message()
 +
{
 +
    if [[ "$2" != "--quiet" ]]; then
 +
        echo $1
 +
    fi
 +
}
 +
 
 +
# @note This function updates the variable $file_size.
 +
# TODO [ ] Amend function to return a filesize and let caller update any
 +
#  variables in need.
 +
 
 +
function parse_file_size()
 +
{
 +
    file_size=`ls -l $1 | cut -d " " -f 5`
 +
    debug_message "(f5) parse_file_size : $1 has size $file_size"
 +
}
 +
 
 +
# ----------------------------------------------------------------------
 +
# - remainder of block handling -
 +
# ----------------------------------------------------------------------
 +
 
 +
function write_partial_block()
 +
{
 +
    debug_message "(f4) write_partial_block : handling byte count of $fsize,"
 +
    debug_message "(f4) write_partial_block : \$idx holds $idx,"
 +
 
 +
    for i in `seq $idx`
 +
    do
 +
        if [ $MODE = "umode" ]; then
 +
            echo -n $CHAR_1 >> $DATA_FILE_1
 +
        else
 +
            echo -n '*' >> $DATA_FILE_1
 +
        fi
 +
    done
 +
}
 +
 
 +
# ----------------------------------------------------------------------
 +
# - block handling -
 +
# ----------------------------------------------------------------------
 +
 
 +
function handle_block()
 +
{
 +
    byte_count=$(seq $BS)
 +
    debug_message "(f3) handle_block : at byte index $idx, writing $BS bytes . . ." --quiet
 +
 
 +
    for i in $byte_count
 +
    do
 +
        if [ $MODE = "umode" ]; then
 +
            echo -n $CHAR_1 >> $DATA_FILE_1
 +
        else
 +
            echo -n '*' >> $DATA_FILE_1
 +
        fi
 +
    done
 +
}
 +
 
 +
function iterate_blocks()
 +
{
 +
    debug_message "(f2) iterate_blocks : at byte index $idx,"
 +
 
 +
    while [[ $idx -ge $BS ]]
 +
    do
 +
        handle_block
 +
        (( idx -= $BS ))
 +
        debug_message "(f2) iterate_blocks : remaining bytes index (count) now $idx" --quiet
 +
    done
 +
 
 +
    if [[ $idx -ge 0 && $idx -lt $BS ]]; then
 +
        debug_message "(f2) iterate_blocks : byte index now holds partial block size $idx"
 +
        write_partial_block $idx
 +
    fi
 +
}
 +
 
 +
function write_pattern()
 +
{
 +
# Write first byte:
 +
    if [ $MODE = "umode" ]; then
 +
        echo -n $CHAR_1 > $DATA_FILE_1
 +
    else
 +
        echo -n '*' > $DATA_FILE_1
 +
    fi
 +
 
 +
    # When a file is greater than our selected block size we interate over
 +
    # blocks of the file first:
 +
    if [ $file_size -gt $BS ]
 +
    then
 +
        echo "(f1) write_pattern : file larger than $BS bytes."
 +
        iterate_blocks
 +
    fi
 +
}
  
−
# aws s3api put-object --bucket edge-fota-images --key img-EPN687-MMMM.bin --body /home/ted/projects-sandbox/workspace-for-nexus/nexus-gateway-firmware/build/img-MMMM.bin
+
# ----------------------------------------------------------------------
 +
# - SECTION - akin to int main
 +
# ----------------------------------------------------------------------
  
−
fname1=op-and-location--image-MMMM--pr-316.json
+
DATA_FILE_1=1.dat
−
fname2=op-and-location--image-SSSS--pr-316.json
+
DATA_FILE_2=2.dat
−
fname3=img-EPN687-MMMM-PR-310.bin
+
# block size
−
fname4=img-EPN687-SSSS-PR-310.bin
+
BS=1024
  
−
filenames="$fname1 $fname2 $fname3 $fname4"
+
# Prefer ASCII values 42 and 85 . . .
 +
CHAR_1="U"
 +
CHAR_2="A"
 +
MODE="umode"
  
−
i=1
+
echo "Append script starting,"
  
−
for filename in $filenames; do
+
if [ $# -gt 0 ]
−
     echo "($i) to S3 uploading $filename . . ."
+
then
−
     (( i++ ))
+
    echo "in first argument detected filename: $1"
−
     aws s3api put-object --bucket edge-fota-images \
+
else
−
     --key $filename \
+
     echo "$0 expects a filename to work with!"
−
     --body $filename
+
    echo "exiting early . . ."
 +
     echo
 +
    exit -1
 +
fi
 +
 
 +
OPTS=$(getopt -o cf:h --long help,create-test-files,file-list: -n append.sh -- "$@")
 +
 
 +
if [ $? -ne 0 ]; then
 +
    echo "Failed to parse options" >&2
 +
    exit 1
 +
fi
 +
 
 +
# Reset the positional parameters to the parsed options
 +
eval set -- "$OPTS"
 +
 
 +
## Process the options
 +
while true; do
 +
  case "$1" in
 +
    -h | --help)
 +
      script_help
 +
      EXIT_AFTER_OPTION=1
 +
      shift
 +
      ;;
 +
     -c | --create-test-files)
 +
      create_test_files
 +
      shift
 +
      ;;
 +
     --)
 +
      # debug_message "got additional arguments '$@'"
 +
      shift
 +
      # debug_message "clearing options and args separator have '$@'"
 +
      break
 +
      ;;
 +
     *)
 +
      echo "$0 Failed to parse options, exiting early!"
 +
      exit 1
 +
      ;;
 +
  esac
 
done
 
done
  
−
echo "done."
+
DATA_FILE_1=$1
 +
 
 +
# DEBUG BLOCK BLOCK
 +
file_size=`ls -l $DATA_FILE_1 | cut -d " " -f 5`
 +
debug_message "(0) main : file $DATA_FILE_1 has size $file_size"
 +
parse_file_size $DATA_FILE_1
 +
# DEBUG BLOCK END
 +
 
 +
# TODO [ ] account for files of size one byte.
 +
 
 +
# - STEP - check for file size greater than one byte:
 +
b=`expr $file_size \> 1`
 +
if [ $b ]
 +
then
 +
    fsize=$file_size
 +
    (( fsize -= 1 ))
 +
else
 +
    echo "file has size of zero or one."
 +
    return 0
 +
fi
 +
 
 +
idx=$file_size
 +
write_pattern
 +
 
 +
debug_message ""
 +
 
 +
debug_message "main : switching to asterisk mode,"
 +
MODE="asterisk"
 +
idx=$file_size
 +
debug_message "main : index idx assigned value $idx"
 +
 
 +
# DEBUG BLOCK BEGIN
 +
file_size=`ls -l $DATA_FILE_1 | cut -d " " -f 5`
 +
debug_message "(0) main : file $DATA_FILE_1 has size $file_size"
 +
parse_file_size $DATA_FILE_1
 +
# DEBUG BLOCK END
 +
 
 +
write_pattern
 +
 
 +
debug_message ""
 +
 
 +
echo "Test of file append done, returning $?"
  
 
exit $?
 
exit $?
−
 
+
***************************************************************************************************************************
 
-->
 
-->

Latest revision as of 16:28, 23 January 2026

Shell Scripting

^ OVERVIEW

This local page for Bash shell scripts and notes on shell scripting.

^ Shell Resources

^ Bash Built-in Variables

Some links to useful articles discussing `bash` built-in variables:

Bash's built-in variables with short names, as listed in Kernigan and Ritchie's "Unix Programming" book and at the first linked article above: 

$0
    The first element passed to the shell is the command name.
$n
    The nth argument passed on the command line. If n ≥ 10, then the syntax must be ${n}.
$*
    All the arguments on the command line. The values are separated by the first character in the shell variable IFS: (${1} … ${n}). See also: the IFS entry in Other Shell Variables.
$@
    All the arguments on the command line. The values are individually quoted: ("${1}" … "${n}").
$#
    The number of command-line arguments.
$?
    The exit value of the last executed command.
$_
    The last argument of the previous command.
$!
    The process ID of the most recent background process.


^ Getopt and Getopts

The shell page section to cover `getopt` and `getopts`, two libraries or facilities available to Bash shell scripts to process command line arguments.

An introduction to `getopt` at Labex dot io:


^ Variables

An interesting use of the shell with variable assignment and multiple variable assignments:

Bash 'declare' built-in or keyword, makes intended variable type explicit, if not truly lasting during script run time:

Comparing bash 'set' and 'declare' built-ins produces almost identical output, at least sometimes: 

ted@localhost1:~$ declare >> 1.txt
ted@localhost1:~$ set >> 2.txt
ted@localhost1:~$ ls -l 1.txt 2.txt
-rw-rw-r-- 1 ted ted 88439 Jan 15 14:01 1.txt
-rw-rw-r-- 1 ted ted 88434 Jan 15 14:01 2.txt
ted@localhost1:~$ diff 1.txt 2.txt
126c126
< _=--color=auto
---
> _=declare
ted@localhost1:~$

^ shell arrays

Shell arrays can be indexed or associated (pairs of keys and values). A good introduction to these things is here:

^ Tests


^ Examples

Some example scripts

^ rename badly named files

How to produce a usable filename which contains <ESC> characters, and to rename that badly named file: 

 $ ls -i
 9704871 $'\033\033'   9703029  CMakeLists.txt  10899590  dts          9708845  samples
 $ ls -q `find . -inum 9704871`
 './'$'\033\033'
 $ mv './'$'\033\033' betterfilename

^ Git diff-tree helper script

A helper script for calling `git diff-tree`, which can be used to determine at which project commit one or more files have changed: 

#!/bin/bash

CMD_GIT=/usr/bin/git

$CMD_GIT log --oneline | head | cut -d " " -f 1

echo "2024-01-31 git diff-tree helper script in progress . . ."
echo "got $# arguments, arg one single quoted is '$1'"

hashes=`$CMD_GIT log --oneline | head -n ${1} | cut -d " " -f 1`
echo "In $PWD found git commit hashes:"
echo " " $hashes

#i=1
#for hash in $hashes; do echo "("$i")" $hash; (( i++ )); done

#echo "Files changed between commit pairs youngest pairings to oldest:
#for hash in $hashes; do echo "Files changed in git commit:"; git diff-tree -r $hash; echo; done
for hash in $hashes
    do echo "Files changed in git commit:"
    git log -1 --oneline $hash
    git diff-tree -r $hash
    echo
done

echo "done"

exit 0

Script to test compilation of series of git commits: 

#!/bin/bash

CMD_GIT=/usr/bin/git

VETTED_COMMITS_FILE="branch-commit-vetting-report.txt"
current_branch="NOT_SET"
hashes="NOT_SET"

function usage()
{
    echo "How to use vet-branch.sh:":
    echo
    echo "  $ vet-branches.sh n"
    echo
    echo "Argument 'n' is a number of commits to test in current git branch."
    echo
}

#
# script starting point, akin to int main
#

echo "$0: starting"
if [ $# -lt 1 ] 
then
    echo "$0 called with too few arguments!"
    usage
    exit -1  
fi

current_branch=`$CMD_GIT branch --show-current`
hashes=`$CMD_GIT log --oneline | head -n $1 | cut -d " " -f 1`

echo "Vetting $1 commits of branch ${current_branch}."
echo "Asked to test commits: $hashes"
echo

date >> $VETTED_COMMITS_FILE
echo "Testing compilation of $1 commits start at tip of '$current_branch':" >> $VETTED_COMMITS_FILE

for hash in $hashes
    do  
    $CMD_GIT checkout --quiet $hash
    echo
    echo "At commit $hash, vetting script in progress"
    echo "---------------------------------------------"
    ./scripts/docker-helper.sh build_both_debug
    echo "$hash build result: $?" >> $VETTED_COMMITS_FILE
done

echo
echo "Restoring git checkout to starting branch:"
$CMD_GIT checkout $current_branch
echo "$0: done."

exit #?

^ shell logging helper

Bash and related shell programming definitely have their limitations, with numeric and mathematical handling, name spaces and more. Despite these shortfalls we still want to debug shell programs. Noted here is an idea to collect some logging shell functions in a script which can be sourced by other shell scripts.

Code excerpt, not complete: 

#!/bin/bash

# @brief Bash script fragment, collection of functions to aid with shell script
#  debugging.
#
# @note Script adopts the notion of log levels as implemented in Zephyr RTOS.
#   From most critical to most verbose these levels are:
#
#   Error, Warning, Info, Debug
#
#  ERR
#  WRN
#  INF
#  DBG

function set_script_log_level()
{

}

# @brief This function expects:
#
# $1 . . . message to print
# $2 . . . log level used to filter printing

function diag()
{

}


^ aliases fragment

function set_custom_shell_aliases()
{
## - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
## Note:  from recent experimentation and reading, shell alias setting
##  does not lend itself to being sourced in a script other than
##  .bashrc, at least in the case of bash shell environment.  For this
##  reason restoring custom shell aliases to dot-bashrc-amendments.sh
##  file - TMH
## - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    echo "setting custom shell aliases . . ."

    alias gw='mv ${HOME}/.ssh/config ${HOME}/.ssh/archive; ln -s ${HOME}/.ssh/config--work ${HOME}/.ssh/config; echo "switching ssh auth to github work account"'
    alias gp='mv ${HOME}/.ssh/config ${HOME}/.ssh/archive; ln -s ${HOME}/.ssh/config--personal ${HOME}/.ssh/config; echo "switching ssh auth to github personal account"'
# for safety:
    alias rm='rm -i'

    alias xscr='xscreensaver-command -lock'
 
# 2024-01-04 THU
    alias vu='ps -u ted x | grep " vi"'

# 2024-12-16 MON - list FTDI like devices
    alias lsf='ls -l /dev/ttyUSB*; ls -l /dev/ttyACM*'

# 2025-02-05 WED - git log --all --graph --oneline
    alias gla='git log --all --graph --oneline'

# 2025-10-26 SUN
    alias xscr='xscreensaver-command -lock'
    alias lgrep='grep -nr --exclude-dir=build $@'

    echo ". . . done."
}

^ to source versus to call

A script fragment which acts as an entry point of a script and treats the absence of all arguments as sign that the script is being sourced, rather than called: 

SCRIPT_NAME=`echo $0 | sed -e 's,^.*/,,'`

SCRIPT_NAME_WHEN_SOURCED="check-path-existence.sh"

EXPECTED_ARG_COUNT=3

if [ $# -eq $EXPECTED_ARG_COUNT ]; then
    SCRIPT_STATUS=0
elif [ $# -eq 0 ]; then
    echo "[INF] $SCRIPT_NAME_WHEN_SOURCED received no arguments, assuming we "
    echo "       are being sourced and not called."
    echo "[INF] (this means we won't run most of this script's code.)"
    SCRIPT_STATUS=300
elif [[ $# -le $EXPECTED_ARG_COUNT && $# -ne 0 ]]; then
    echo "[ERR] $SCRIPT_NAME requires $EXPECTED_ARG_COUNT arguments, got $#."
    SCRIPT_STATUS=400
    help
elif [[ $# -gt $EXPECTED_ARG_COUNT && $# -ne 0 ]]; then
    echo "[WRN] $SCRIPT_NAME expects $EXPECTED_ARG_COUNT arguments, got $#."
    echo "[WRN] Extra arguments are ignored."
fi

^ Bash scriptlets

The following scripts are especially short, sometimes just one command. They could be implemented as shell aliases but are presented here as independent scripts.

Script: show most recently updated git branches 

#!/bin/bash
# How to use:  name this script something short, for example `gbl`, and put it in the shell's search path.
# Where to use:  in a local git repository `gbl` shows the ten most recently modified git branchs.
git branch --sort=committerdate | tail
exit $?

^ Bash helper functions

Common helper functions in bash scripts: 

function help()
{
    # echo script usage
}

function exit_early()
{
    # echo script name, echo status of last command
}

function set_local_vars()
{
    # set variables particular to given script, but note they're global
    # (good to rename this function.)
}

Some shell scripts to provide handy localhost commands, for embedded work on a Linux host:

  1. !/bin/bash

grep -nr --exclude-dir=.git --exclude-dir=build --exclude-dir=docker_cache --exclude-dir=twister-out* $1 | grep $1

  1. exit $?

^ To Research

[ ] Look up `git ls-files` and its options.

[ ] Review `xargs` called with the dash zero option.

Interesting header file from Zephyr RTOS 3.4.0:

`zephyr/include/zephyr/toolchain/xcc_missing_defs.h`

Shell command substitution backticks versus braces . . .



Retrieved from "https://wiki.neelanurseries.com/index.php?title=Shell&oldid=8771"