Difference between revisions of "Shell"

From Wiki at Neela Nurseries
Jump to navigation Jump to search
m (Update extended git-tree helper script.)
(Shell command substitution backticks versus braces)
 
(4 intermediate revisions by the same user not shown)
Line 32: Line 32:
  
 
<!-- odne komentar -->
 
<!-- odne komentar -->
 +
 +
== [[#top|^]] Getopt and Getopts ==
 +
 +
The shell page section to cover `getopt` and `getopts`, two libraries or facilities available to Bash shell scripts to process command line arguments.
 +
 +
An introduction to `getopt` at Labex dot io:
 +
 +
*  https://labex.io/tutorials/shell-bash-getopt-391993
 +
 +
<!-- odne komentar -->
 +
 
== [[#top|^]] Rename Badly Named Files ==
 
== [[#top|^]] Rename Badly Named Files ==
  
Line 140: Line 151:
 
</pre>
 
</pre>
  
−
<!--
+
== [[#top|^]] Shell Logging Helper ==
−
********************************************************************************************************
+
 
−
-->
+
Bash and related shell programming definitely have their limitations, with numeric and mathematical handling, name spaces and more.  Despite these shortfalls we still want to debug shell programs.  Noted here is an idea to collect some logging shell functions in a script which can be sourced by other shell scripts.
 +
 
 +
Code excerpt, not complete:
 +
 
 +
<pre>
 +
#!/bin/bash
 +
 
 +
# @brief Bash script fragment, collection of functions to aid with shell script
 +
#  debugging.
 +
#
 +
# @note Script adopts the notion of log levels as implemented in Zephyr RTOS.
 +
#  From most critical to most verbose these levels are:
 +
#
 +
#  Error, Warning, Info, Debug
 +
#
 +
#  ERR
 +
#  WRN
 +
#  INF
 +
#  DBG
 +
 
 +
function set_script_log_level()
 +
{
  
−
<!--
+
}
  
−
-->
+
# @brief This function expects:
 +
#
 +
# $1 . . . message to print
 +
# $2 . . . log level used to filter printing
  
−
<!--
+
function diag()
−
************************************************************************************************************
+
{
−
-->
 
  
−
<!--
+
}
 +
</pre>
 +
<!-- ************************************************************************************************************ --> <!--
 
#!/bin/bash
 
#!/bin/bash
  
 +
# TODO [ ] Add sanity checking and modestly vigorous search for system
 +
#  git and file utilities:
 
CMD_GIT=/usr/bin/git
 
CMD_GIT=/usr/bin/git
 +
CMD_TREE=/usr/bin/tree
  
 
# Changed file filter, may differ on various work stations:
 
# Changed file filter, may differ on various work stations:
Line 165: Line 204:
 
     echo
 
     echo
 
     echo "  $ $0 <count> [-u|-d]"
 
     echo "  $ $0 <count> [-u|-d]"
−
     echo "  $ $0 <count> [--show-dirs --make-dirs] <dest_dit> # <- both long options required here"
+
     echo "  $ $0 <count> [--show-dirs --make-dirs] <dest_dir> # <- both long options required here"
 
     echo "  $ $0 <count> -c <dest_dir>"
 
     echo "  $ $0 <count> -c <dest_dir>"
 
     echo
 
     echo
 
     echo "count . . . count of commits to review from tip of git branch"
 
     echo "count . . . count of commits to review from tip of git branch"
 +
    echo
 
     echo "-u    . . . optionally summarize file changes to list of unique files changed"
 
     echo "-u    . . . optionally summarize file changes to list of unique files changed"
 +
    echo
 +
    echo "-d    . . . output essentially like -u.  To be updated."
 +
    echo
 +
    echo "Examples:"
 +
    echo
 +
    echo "  $ helper-git-diff-tree-extended.sh 5"
 +
    echo
 +
    echo "...to report per commit a list of files changed and some git details"
 +
    echo "including git's internal file hashes.  This helpful in highlighting"
 +
    echo "files deleted and newly created files, whose commit hashes before and"
 +
    echo "after the given commit respectively end as all zeroes (file deletion)"
 +
    echo "and begin as all zeroes (file creation)."
 +
    echo
 +
    echo "As a side note, file renaming in git involves a file deleation"
 +
    echo "operation followed immediately by a file creation operation."
 +
    echo
 +
    echo "  $ helper-git-diff-tree-extended.sh 5 -u"
 +
    echo
 +
    echo "...to show a list of relative paths ending in names of files changed"
 +
    echo "anywhere in the commit series of the latest 'count' commits of the"
 +
    echo "local repo at its current commit."
 +
    echo
 
}
 
}
  
Line 193: Line 255:
 
}
 
}
  
 +
# @brief This function reports a list of files changed across a range of
 +
#  commits, prepending relative path from current working dir to file.
 +
#
 
# This function expects:
 
# This function expects:
 
# $1 . . . count of commits to check, starting from branch tip
 
# $1 . . . count of commits to check, starting from branch tip
Line 204: Line 269:
 
}
 
}
  
 +
# TODO [ ] trim terminating dir separators, e.g. foward slash characters
 +
#  from passed scratch directory path in which to create detected
 +
#  paths holding changed files.
 +
 +
# @brief This function recreates in a scratch directory each relative path to
 +
#  a changed file, those files being unique changed files across series of
 +
#  commits.
 +
#
 
# This function expects:
 
# This function expects:
 
# $1 . . . count of commits to check, starting from branch tip
 
# $1 . . . count of commits to check, starting from branch tip
Line 234: Line 307:
 
}
 
}
  
 +
# @brief This function recreates each file changed across a series of commits,
 +
#  in a recreated dir path matching its relative path from current working dir.
 +
#
 +
# @note The copied files in recreated relative paths can be displayed by the
 +
#  `tree` command to highlight file changes across commit series more
 +
#  graphically.
 +
#
 
# This function expects:
 
# This function expects:
 
# $1 . . . count of commits to check, starting from branch tip
 
# $1 . . . count of commits to check, starting from branch tip
Line 240: Line 320:
 
function copy_relpaths_and_file()
 
function copy_relpaths_and_file()
 
{
 
{
−
     echo "f5 called with $# arguments,"
+
     i=1
−
    echo "caller wants to copy relative paths and changed files to '${2}'"
 
 
     commit_count=$1
 
     commit_count=$1
 
     scratch_dir=$2
 
     scratch_dir=$2
 +
    echo "f5 called with $# arguments,"
 +
    echo "caller wants to copy relative paths and changed files to '$scratch_dir'"
  
 
     files=`shows_changed_files_per_commit $1 | \
 
     files=`shows_changed_files_per_commit $1 | \
Line 251: Line 332:
 
         dir_count=`echo $file | awk -F "/" ' { print NF-1 } '`
 
         dir_count=`echo $file | awk -F "/" ' { print NF-1 } '`
 
         if [ $dir_count -gt 0 ]; then
 
         if [ $dir_count -gt 0 ]; then
−
            # dirs=`sed 's/\//\ /g'`
 
 
             dirs=`echo $file | rev | sed 's/\//\ /' | rev | cut -d " " -f 1`
 
             dirs=`echo $file | rev | sed 's/\//\ /' | rev | cut -d " " -f 1`
 
             echo "(d) $file has dirs $dirs"
 
             echo "(d) $file has dirs $dirs"
 
             echo "(c)  creating $scratch_dir/$dirs . . ."
 
             echo "(c)  creating $scratch_dir/$dirs . . ."
 
             mkdir -pv $scratch_dir/$dirs
 
             mkdir -pv $scratch_dir/$dirs
−
 
 
             cp -pv $file $scratch_dir/$dirs
 
             cp -pv $file $scratch_dir/$dirs
−
 
 
             (( i++ ))
 
             (( i++ ))
 
         else
 
         else
−
             cp -pv $file $scratch_dir//$file
+
             cp -pv $file $scratch_dir/$file
 
         fi
 
         fi
 
     done
 
     done
 +
 +
    echo
 +
    echo "Summary of files changed in commit series, and their final"
 +
    echo "relative locations:"
 +
    echo
 +
    $CMD_TREE $scratch_dir
 
}
 
}
  
−
#
+
# ----------------------------------------------------------------------
 
# - SECTION - entry point akin to int main
 
# - SECTION - entry point akin to int main
−
#
+
# ----------------------------------------------------------------------
  
 
echo "2024-01-31 git diff-tree helper script in progress . . ."
 
echo "2024-01-31 git diff-tree helper script in progress . . ."
Line 282: Line 366:
 
elif [ ${2} == "-d" ] ; then
 
elif [ ${2} == "-d" ] ; then
 
     show_dirs_per_changed_file ${1}
 
     show_dirs_per_changed_file ${1}
−
elif [[ ${3} == "--show-dirs"  &&  ${4} == "--make-dirs" ]] ; then
+
elif [[ ${2} == "--show-dirs"  &&  ${3} == "--make-dirs" ]] ; then
−
     recreate_directory_paths ${1} ${2}
+
     recreate_directory_paths ${1} ${4}
 
elif [ ${2} == "-c" ] ; then
 
elif [ ${2} == "-c" ] ; then
 
     copy_relpaths_and_file ${1} ${3}
 
     copy_relpaths_and_file ${1} ${3}
Line 292: Line 376:
 
exit $?
 
exit $?
 
-->
 
-->
−
 
+
<!-- ************************************************************************************************************ -->
 
<!-- odne komentar -->
 
<!-- odne komentar -->
−
 
 
== [[#top|^]] To Research ==
 
== [[#top|^]] To Research ==
  
Line 304: Line 387:
  
 
`zephyr/include/zephyr/toolchain/xcc_missing_defs.h`
 
`zephyr/include/zephyr/toolchain/xcc_missing_defs.h`
 +
 +
Shell command substitution backticks versus braces . . .
 +
 +
*  https://stackoverflow.com/questions/22709371/backticks-vs-braces-in-bash
  
 
<!-- odne komentar -->
 
<!-- odne komentar -->
  
 
<!--
 
<!--
 +
https://stackoverflow.com/questions/22709371/backticks-vs-braces-in-bash
  
−
#!/bin/bash
+
# later on in the man page:
−
 
+
When the old-style backquote form of substitution is used, backslash retains its literal meaning except when followed by $, `, or \. The first backquote not preceded by a backslash terminates the command substitution. When using the $(command) form, all characters between the parentheses make up the command; none are treated specially.
−
export AWS_PROFILE=prod
 
−
echo "starting, exporting AWS_PROFILE=$AWS_PROFILE . . ."
 
−
 
 
−
# aws s3api put-object --bucket edge-fota-images --key img-EPN687-MMMM.bin --body /home/ted/projects-sandbox/workspace-for-nexus/nexus-gateway-firmware/build/img-MMMM.bin
 
−
 
 
−
fname1=op-and-location--image-MMMM--pr-316.json
 
−
fname2=op-and-location--image-SSSS--pr-316.json
 
−
fname3=img-EPN687-MMMM-PR-310.bin
 
−
fname4=img-EPN687-SSSS-PR-310.bin
 
−
 
 
−
filenames="$fname1 $fname2 $fname3 $fname4"
 
−
 
 
−
i=1
 
−
 
 
−
for filename in $filenames; do
 
−
    echo "($i) to S3 uploading $filename . . ."
 
−
    (( i++ ))
 
−
    aws s3api put-object --bucket edge-fota-images \
 
−
    --key $filename \
 
−
    --body $filename
 
−
done
 
−
 
 
−
echo "done."
 
−
 
 
−
exit $?
 
  
 
-->
 
-->

Latest revision as of 19:05, 3 October 2025

Shell Scripting

^ OVERVIEW

This local page for Bash shell scripts and notes on shell scripting.

^ Bash Built-in Variables

Some links to useful articles discussing `bash` built-in variables:

Bash's built-in variables with short names, as listed in Kernigan and Ritchie's "Unix Programming" book and at the first linked article above: 

$0
    The first element passed to the shell is the command name.
$n
    The nth argument passed on the command line. If n ≥ 10, then the syntax must be ${n}.
$*
    All the arguments on the command line. The values are separated by the first character in the shell variable IFS: (${1} … ${n}). See also: the IFS entry in Other Shell Variables.
$@
    All the arguments on the command line. The values are individually quoted: ("${1}" … "${n}").
$#
    The number of command-line arguments.
$?
    The exit value of the last executed command.
$_
    The last argument of the previous command.
$!
    The process ID of the most recent background process.


^ Getopt and Getopts

The shell page section to cover `getopt` and `getopts`, two libraries or facilities available to Bash shell scripts to process command line arguments.

An introduction to `getopt` at Labex dot io:


^ Rename Badly Named Files

How to produce a usable filename which contains <ESC> characters, and to rename that badly named file: 

 $ ls -i
 9704871 $'\033\033'   9703029  CMakeLists.txt  10899590  dts          9708845  samples
 $ ls -q `find . -inum 9704871`
 './'$'\033\033'
 $ mv './'$'\033\033' betterfilename

^ Git Diff-tree Helper Script

A helper script for calling `git diff-tree`, which can be used to determine at which project commit one or more files have changed: 

#!/bin/bash

CMD_GIT=/usr/bin/git

$CMD_GIT log --oneline | head | cut -d " " -f 1

echo "2024-01-31 git diff-tree helper script in progress . . ."
echo "got $# arguments, arg one single quoted is '$1'"

hashes=`$CMD_GIT log --oneline | head -n ${1} | cut -d " " -f 1`
echo "In $PWD found git commit hashes:"
echo " " $hashes

#i=1
#for hash in $hashes; do echo "("$i")" $hash; (( i++ )); done

#echo "Files changed between commit pairs youngest pairings to oldest:
#for hash in $hashes; do echo "Files changed in git commit:"; git diff-tree -r $hash; echo; done
for hash in $hashes
    do echo "Files changed in git commit:"
    git log -1 --oneline $hash
    git diff-tree -r $hash
    echo
done

echo "done"

exit 0

Script to test compilation of series of git commits: 

#!/bin/bash

CMD_GIT=/usr/bin/git

VETTED_COMMITS_FILE="branch-commit-vetting-report.txt"
current_branch="NOT_SET"
hashes="NOT_SET"

function usage()
{
    echo "How to use vet-branch.sh:":
    echo
    echo "  $ vet-branches.sh n"
    echo
    echo "Argument 'n' is a number of commits to test in current git branch."
    echo
}

#
# script starting point, akin to int main
#

echo "$0: starting"
if [ $# -lt 1 ] 
then
    echo "$0 called with too few arguments!"
    usage
    exit -1  
fi

current_branch=`$CMD_GIT branch --show-current`
hashes=`$CMD_GIT log --oneline | head -n $1 | cut -d " " -f 1`

echo "Vetting $1 commits of branch ${current_branch}."
echo "Asked to test commits: $hashes"
echo

date >> $VETTED_COMMITS_FILE
echo "Testing compilation of $1 commits start at tip of '$current_branch':" >> $VETTED_COMMITS_FILE

for hash in $hashes
    do  
    $CMD_GIT checkout --quiet $hash
    echo
    echo "At commit $hash, vetting script in progress"
    echo "---------------------------------------------"
    ./scripts/docker-helper.sh build_both_debug
    echo "$hash build result: $?" >> $VETTED_COMMITS_FILE
done

echo
echo "Restoring git checkout to starting branch:"
$CMD_GIT checkout $current_branch
echo "$0: done."

exit #?

^ Shell Logging Helper

Bash and related shell programming definitely have their limitations, with numeric and mathematical handling, name spaces and more. Despite these shortfalls we still want to debug shell programs. Noted here is an idea to collect some logging shell functions in a script which can be sourced by other shell scripts.

Code excerpt, not complete: 

#!/bin/bash

# @brief Bash script fragment, collection of functions to aid with shell script
#  debugging.
#
# @note Script adopts the notion of log levels as implemented in Zephyr RTOS.
#   From most critical to most verbose these levels are:
#
#   Error, Warning, Info, Debug
#
#  ERR
#  WRN
#  INF
#  DBG

function set_script_log_level()
{

}

# @brief This function expects:
#
# $1 . . . message to print
# $2 . . . log level used to filter printing

function diag()
{

}

^ To Research

[ ] Look up `git ls-files` and its options.

[ ] Review `xargs` called with the dash zero option.

Interesting header file from Zephyr RTOS 3.4.0:

`zephyr/include/zephyr/toolchain/xcc_missing_defs.h`

Shell command substitution backticks versus braces . . .


Retrieved from "https://wiki.neelanurseries.com/index.php?title=Shell&oldid=8531"