Difference between revisions of "Shell"

From Wiki at Neela Nurseries
Jump to navigation Jump to search
m (→‎^ Git Diff-tree Helper Script: - Add TODO comments for script improvements.)
(Shell command substitution backticks versus braces)
 
(14 intermediate revisions by the same user not shown)
Line 32: Line 32:
  
 
<!-- odne komentar -->
 
<!-- odne komentar -->
 +
 +
== [[#top|^]] Getopt and Getopts ==
 +
 +
The shell page section to cover `getopt` and `getopts`, two libraries or facilities available to Bash shell scripts to process command line arguments.
 +
 +
An introduction to `getopt` at Labex dot io:
 +
 +
*  https://labex.io/tutorials/shell-bash-getopt-391993
 +
 +
<!-- odne komentar -->
 +
 
== [[#top|^]] Rename Badly Named Files ==
 
== [[#top|^]] Rename Badly Named Files ==
  
Line 49: Line 60:
  
 
<pre>
 
<pre>
−
 
 
#!/bin/bash
 
#!/bin/bash
−
 
−
# TODO: [ ] add first line of commit message above each set of file changes
 
−
# TODO: [ ] add command line option to accept count of commits to compare each neighbor to next commit
 
  
 
CMD_GIT=/usr/bin/git
 
CMD_GIT=/usr/bin/git
Line 60: Line 67:
  
 
echo "2024-01-31 git diff-tree helper script in progress . . ."
 
echo "2024-01-31 git diff-tree helper script in progress . . ."
 +
echo "got $# arguments, arg one single quoted is '$1'"
  
−
hashes=`$CMD_GIT log --oneline | head | cut -d " " -f 1`
+
hashes=`$CMD_GIT log --oneline | head -n ${1} | cut -d " " -f 1`
 
echo "In $PWD found git commit hashes:"
 
echo "In $PWD found git commit hashes:"
 
echo " " $hashes
 
echo " " $hashes
Line 69: Line 77:
  
 
#echo "Files changed between commit pairs youngest pairings to oldest:
 
#echo "Files changed between commit pairs youngest pairings to oldest:
−
for hash in $hashes; do echo "Files changed in git commit:"; git diff-tree -r $hash; echo; done
+
#for hash in $hashes; do echo "Files changed in git commit:"; git diff-tree -r $hash; echo; done
 +
for hash in $hashes
 +
    do echo "Files changed in git commit:"
 +
    git log -1 --oneline $hash
 +
    git diff-tree -r $hash
 +
    echo
 +
done
  
 
echo "done"
 
echo "done"
Line 75: Line 89:
 
exit 0
 
exit 0
 
</pre>
 
</pre>
 +
 +
Script to test compilation of series of git commits:
 +
 +
<pre>
 +
#!/bin/bash
 +
 +
CMD_GIT=/usr/bin/git
 +
 +
VETTED_COMMITS_FILE="branch-commit-vetting-report.txt"
 +
current_branch="NOT_SET"
 +
hashes="NOT_SET"
 +
 +
function usage()
 +
{
 +
    echo "How to use vet-branch.sh:":
 +
    echo
 +
    echo "  $ vet-branches.sh n"
 +
    echo
 +
    echo "Argument 'n' is a number of commits to test in current git branch."
 +
    echo
 +
}
 +
 +
#
 +
# script starting point, akin to int main
 +
#
 +
 +
echo "$0: starting"
 +
if [ $# -lt 1 ]
 +
then
 +
    echo "$0 called with too few arguments!"
 +
    usage
 +
    exit -1 
 +
fi
 +
 +
current_branch=`$CMD_GIT branch --show-current`
 +
hashes=`$CMD_GIT log --oneline | head -n $1 | cut -d " " -f 1`
 +
 +
echo "Vetting $1 commits of branch ${current_branch}."
 +
echo "Asked to test commits: $hashes"
 +
echo
 +
 +
date >> $VETTED_COMMITS_FILE
 +
echo "Testing compilation of $1 commits start at tip of '$current_branch':" >> $VETTED_COMMITS_FILE
 +
 +
for hash in $hashes
 +
    do 
 +
    $CMD_GIT checkout --quiet $hash
 +
    echo
 +
    echo "At commit $hash, vetting script in progress"
 +
    echo "---------------------------------------------"
 +
    ./scripts/docker-helper.sh build_both_debug
 +
    echo "$hash build result: $?" >> $VETTED_COMMITS_FILE
 +
done
 +
 +
echo
 +
echo "Restoring git checkout to starting branch:"
 +
$CMD_GIT checkout $current_branch
 +
echo "$0: done."
 +
 +
exit #?
 +
</pre>
 +
 +
== [[#top|^]] Shell Logging Helper ==
 +
 +
Bash and related shell programming definitely have their limitations, with numeric and mathematical handling, name spaces and more.  Despite these shortfalls we still want to debug shell programs.  Noted here is an idea to collect some logging shell functions in a script which can be sourced by other shell scripts.
 +
 +
Code excerpt, not complete:
 +
 +
<pre>
 +
#!/bin/bash
 +
 +
# @brief Bash script fragment, collection of functions to aid with shell script
 +
#  debugging.
 +
#
 +
# @note Script adopts the notion of log levels as implemented in Zephyr RTOS.
 +
#  From most critical to most verbose these levels are:
 +
#
 +
#  Error, Warning, Info, Debug
 +
#
 +
#  ERR
 +
#  WRN
 +
#  INF
 +
#  DBG
 +
 +
function set_script_log_level()
 +
{
 +
 +
}
 +
 +
# @brief This function expects:
 +
#
 +
# $1 . . . message to print
 +
# $2 . . . log level used to filter printing
 +
 +
function diag()
 +
{
 +
 +
}
 +
</pre>
 +
<!-- ************************************************************************************************************ --> <!--
 +
#!/bin/bash
 +
 +
# TODO [ ] Add sanity checking and modestly vigorous search for system
 +
#  git and file utilities:
 +
CMD_GIT=/usr/bin/git
 +
CMD_TREE=/usr/bin/tree
 +
 +
# Changed file filter, may differ on various work stations:
 +
MATCH_CHANGED_FILE_LINE=":100644"
 +
 +
function usage()
 +
{
 +
    echo "call $0 with:"
 +
    echo
 +
    echo "  $ $0 <count> [-u|-d]"
 +
    echo "  $ $0 <count> [--show-dirs --make-dirs] <dest_dir> # <- both long options required here"
 +
    echo "  $ $0 <count> -c <dest_dir>"
 +
    echo
 +
    echo "count . . . count of commits to review from tip of git branch"
 +
    echo
 +
    echo "-u    . . . optionally summarize file changes to list of unique files changed"
 +
    echo
 +
    echo "-d    . . . output essentially like -u.  To be updated."
 +
    echo
 +
    echo "Examples:"
 +
    echo
 +
    echo "  $ helper-git-diff-tree-extended.sh 5"
 +
    echo
 +
    echo "...to report per commit a list of files changed and some git details"
 +
    echo "including git's internal file hashes.  This helpful in highlighting"
 +
    echo "files deleted and newly created files, whose commit hashes before and"
 +
    echo "after the given commit respectively end as all zeroes (file deletion)"
 +
    echo "and begin as all zeroes (file creation)."
 +
    echo
 +
    echo "As a side note, file renaming in git involves a file deleation"
 +
    echo "operation followed immediately by a file creation operation."
 +
    echo
 +
    echo "  $ helper-git-diff-tree-extended.sh 5 -u"
 +
    echo
 +
    echo "...to show a list of relative paths ending in names of files changed"
 +
    echo "anywhere in the commit series of the latest 'count' commits of the"
 +
    echo "local repo at its current commit."
 +
    echo
 +
}
 +
 +
function shows_changed_files_per_commit()
 +
{
 +
    echo "function f1 starting"
 +
    hashes=`$CMD_GIT log --oneline | head -n ${1} | cut -d " " -f 1`
 +
    echo "In $PWD found git commit hashes:"
 +
    echo " " $hashes
 +
 +
    for hash in $hashes
 +
        do echo "Files changed in git commit:"
 +
        git log -1 --oneline $hash
 +
        git diff-tree -r $hash
 +
        echo
 +
    done
 +
}
 +
 +
function show_unique_changed_files_across_all_commits()
 +
{
 +
    shows_changed_files_per_commit ${1} | \
 +
    grep ${MATCH_CHANGED_FILE_LINE} | cut -f 2 | sort --unique
 +
}
 +
 +
# @brief This function reports a list of files changed across a range of
 +
#  commits, prepending relative path from current working dir to file.
 +
#
 +
# This function expects:
 +
# $1 . . . count of commits to check, starting from branch tip
 +
 +
function show_dirs_per_changed_file()
 +
{
 +
    echo "f3 called with $# arguments,"
 +
 +
    shows_changed_files_per_commit ${1} | \
 +
    grep ${MATCH_CHANGED_FILE_LINE} | cut -f 2 | sort --unique
 +
}
 +
 +
# TODO [ ] trim terminating dir separators, e.g. foward slash characters
 +
#  from passed scratch directory path in which to create detected
 +
#  paths holding changed files.
 +
 +
# @brief This function recreates in a scratch directory each relative path to
 +
#  a changed file, those files being unique changed files across series of
 +
#  commits.
 +
#
 +
# This function expects:
 +
# $1 . . . count of commits to check, starting from branch tip
 +
# $2 . . . path relative or absolute in which to create copies of paths to each changed file
 +
 +
function recreate_directory_paths()
 +
{
 +
    i=1
 +
    commit_count=$1
 +
    scratch_dir=$2
 +
    echo "f4 called with $# arguments,"
 +
    echo "caller checking $commit_count commits,"
 +
    echo "caller wants relpaths created in '$scratch_dir' . . ."
 +
 +
    files=`shows_changed_files_per_commit $commit_count | \
 +
      grep ${MATCH_CHANGED_FILE_LINE} | cut -f 2 | sort --unique`
 +
 +
# TODO [ ] find a way to remove just the filename, then call `mkdir -pv one_or_more_dirs`
 +
    for file in ${files}; do
 +
        dir_count=`echo $file | awk -F "/" ' { print NF-1 } '`
 +
        if [ $dir_count -gt 0 ]; then
 +
            dirs=`echo ${file} | rev | sed 's/\//\ /' | rev | cut -d " " -f 1`
 +
            echo "($i)"
 +
            echo "(d) ${file} has dirs ${dirs}"
 +
            echo "(c)  creating $scratch_dir/${dirs} . . ."
 +
            mkdir -pv $scratch_dir/${dirs}
 +
            (( i++ ))
 +
        fi
 +
    done
 +
}
 +
 +
# @brief This function recreates each file changed across a series of commits,
 +
#  in a recreated dir path matching its relative path from current working dir.
 +
#
 +
# @note The copied files in recreated relative paths can be displayed by the
 +
#  `tree` command to highlight file changes across commit series more
 +
#  graphically.
 +
#
 +
# This function expects:
 +
# $1 . . . count of commits to check, starting from branch tip
 +
# $2 . . . path relative or absolute in which to create copies of paths to each changed file
 +
 +
function copy_relpaths_and_file()
 +
{
 +
    i=1
 +
    commit_count=$1
 +
    scratch_dir=$2
 +
    echo "f5 called with $# arguments,"
 +
    echo "caller wants to copy relative paths and changed files to '$scratch_dir'"
 +
 +
    files=`shows_changed_files_per_commit $1 | \
 +
      grep ${MATCH_CHANGED_FILE_LINE} | cut -f 2 | sort --unique`
 +
 +
    for file in $files; do
 +
        dir_count=`echo $file | awk -F "/" ' { print NF-1 } '`
 +
        if [ $dir_count -gt 0 ]; then
 +
            dirs=`echo $file | rev | sed 's/\//\ /' | rev | cut -d " " -f 1`
 +
            echo "(d) $file has dirs $dirs"
 +
            echo "(c)  creating $scratch_dir/$dirs . . ."
 +
            mkdir -pv $scratch_dir/$dirs
 +
            cp -pv $file $scratch_dir/$dirs
 +
            (( i++ ))
 +
        else
 +
            cp -pv $file $scratch_dir/$file
 +
        fi
 +
    done
 +
 +
    echo
 +
    echo "Summary of files changed in commit series, and their final"
 +
    echo "relative locations:"
 +
    echo
 +
    $CMD_TREE $scratch_dir
 +
}
 +
 +
# ----------------------------------------------------------------------
 +
# - SECTION - entry point akin to int main
 +
# ----------------------------------------------------------------------
 +
 +
echo "2024-01-31 git diff-tree helper script in progress . . ."
 +
echo "got $# arguments, arg one single quoted is '$1'"
 +
echo
 +
 +
if [ $# -lt 1 ] ; then
 +
    usage
 +
elif [ $# -eq 1 ] ; then
 +
    shows_changed_files_per_commit ${1}
 +
elif [ ${2} == "-u" ] ; then
 +
    show_unique_changed_files_across_all_commits ${1}
 +
elif [ ${2} == "-d" ] ; then
 +
    show_dirs_per_changed_file ${1}
 +
elif [[ ${2} == "--show-dirs"  &&  ${3} == "--make-dirs" ]] ; then
 +
    recreate_directory_paths ${1} ${4}
 +
elif [ ${2} == "-c" ] ; then
 +
    copy_relpaths_and_file ${1} ${3}
 +
else
 +
    usage
 +
fi
 +
 +
exit $?
 +
-->
 +
<!-- ************************************************************************************************************ -->
 +
<!-- odne komentar -->
 +
== [[#top|^]] To Research ==
 +
 +
[ ] Look up `git ls-files` and its options.
 +
 +
[ ] Review `xargs` called with the dash zero option.
 +
 +
Interesting header file from Zephyr RTOS 3.4.0:
 +
 +
`zephyr/include/zephyr/toolchain/xcc_missing_defs.h`
 +
 +
Shell command substitution backticks versus braces . . .
 +
 +
*  https://stackoverflow.com/questions/22709371/backticks-vs-braces-in-bash
  
 
<!-- odne komentar -->
 
<!-- odne komentar -->
 +
 +
<!--
 +
https://stackoverflow.com/questions/22709371/backticks-vs-braces-in-bash
 +
 +
# later on in the man page:
 +
When the old-style backquote form of substitution is used, backslash retains its literal meaning except when followed by $, `, or \. The first backquote not preceded by a backslash terminates the command substitution. When using the $(command) form, all characters between the parentheses make up the command; none are treated specially.
 +
 +
-->

Latest revision as of 19:05, 3 October 2025

Shell Scripting

^ OVERVIEW

This local page for Bash shell scripts and notes on shell scripting.

^ Bash Built-in Variables

Some links to useful articles discussing `bash` built-in variables:

Bash's built-in variables with short names, as listed in Kernigan and Ritchie's "Unix Programming" book and at the first linked article above: 

$0
    The first element passed to the shell is the command name.
$n
    The nth argument passed on the command line. If n ≥ 10, then the syntax must be ${n}.
$*
    All the arguments on the command line. The values are separated by the first character in the shell variable IFS: (${1} … ${n}). See also: the IFS entry in Other Shell Variables.
$@
    All the arguments on the command line. The values are individually quoted: ("${1}" … "${n}").
$#
    The number of command-line arguments.
$?
    The exit value of the last executed command.
$_
    The last argument of the previous command.
$!
    The process ID of the most recent background process.


^ Getopt and Getopts

The shell page section to cover `getopt` and `getopts`, two libraries or facilities available to Bash shell scripts to process command line arguments.

An introduction to `getopt` at Labex dot io:


^ Rename Badly Named Files

How to produce a usable filename which contains <ESC> characters, and to rename that badly named file: 

 $ ls -i
 9704871 $'\033\033'   9703029  CMakeLists.txt  10899590  dts          9708845  samples
 $ ls -q `find . -inum 9704871`
 './'$'\033\033'
 $ mv './'$'\033\033' betterfilename

^ Git Diff-tree Helper Script

A helper script for calling `git diff-tree`, which can be used to determine at which project commit one or more files have changed: 

#!/bin/bash

CMD_GIT=/usr/bin/git

$CMD_GIT log --oneline | head | cut -d " " -f 1

echo "2024-01-31 git diff-tree helper script in progress . . ."
echo "got $# arguments, arg one single quoted is '$1'"

hashes=`$CMD_GIT log --oneline | head -n ${1} | cut -d " " -f 1`
echo "In $PWD found git commit hashes:"
echo " " $hashes

#i=1
#for hash in $hashes; do echo "("$i")" $hash; (( i++ )); done

#echo "Files changed between commit pairs youngest pairings to oldest:
#for hash in $hashes; do echo "Files changed in git commit:"; git diff-tree -r $hash; echo; done
for hash in $hashes
    do echo "Files changed in git commit:"
    git log -1 --oneline $hash
    git diff-tree -r $hash
    echo
done

echo "done"

exit 0

Script to test compilation of series of git commits: 

#!/bin/bash

CMD_GIT=/usr/bin/git

VETTED_COMMITS_FILE="branch-commit-vetting-report.txt"
current_branch="NOT_SET"
hashes="NOT_SET"

function usage()
{
    echo "How to use vet-branch.sh:":
    echo
    echo "  $ vet-branches.sh n"
    echo
    echo "Argument 'n' is a number of commits to test in current git branch."
    echo
}

#
# script starting point, akin to int main
#

echo "$0: starting"
if [ $# -lt 1 ] 
then
    echo "$0 called with too few arguments!"
    usage
    exit -1  
fi

current_branch=`$CMD_GIT branch --show-current`
hashes=`$CMD_GIT log --oneline | head -n $1 | cut -d " " -f 1`

echo "Vetting $1 commits of branch ${current_branch}."
echo "Asked to test commits: $hashes"
echo

date >> $VETTED_COMMITS_FILE
echo "Testing compilation of $1 commits start at tip of '$current_branch':" >> $VETTED_COMMITS_FILE

for hash in $hashes
    do  
    $CMD_GIT checkout --quiet $hash
    echo
    echo "At commit $hash, vetting script in progress"
    echo "---------------------------------------------"
    ./scripts/docker-helper.sh build_both_debug
    echo "$hash build result: $?" >> $VETTED_COMMITS_FILE
done

echo
echo "Restoring git checkout to starting branch:"
$CMD_GIT checkout $current_branch
echo "$0: done."

exit #?

^ Shell Logging Helper

Bash and related shell programming definitely have their limitations, with numeric and mathematical handling, name spaces and more. Despite these shortfalls we still want to debug shell programs. Noted here is an idea to collect some logging shell functions in a script which can be sourced by other shell scripts.

Code excerpt, not complete: 

#!/bin/bash

# @brief Bash script fragment, collection of functions to aid with shell script
#  debugging.
#
# @note Script adopts the notion of log levels as implemented in Zephyr RTOS.
#   From most critical to most verbose these levels are:
#
#   Error, Warning, Info, Debug
#
#  ERR
#  WRN
#  INF
#  DBG

function set_script_log_level()
{

}

# @brief This function expects:
#
# $1 . . . message to print
# $2 . . . log level used to filter printing

function diag()
{

}

^ To Research

[ ] Look up `git ls-files` and its options.

[ ] Review `xargs` called with the dash zero option.

Interesting header file from Zephyr RTOS 3.4.0:

`zephyr/include/zephyr/toolchain/xcc_missing_defs.h`

Shell command substitution backticks versus braces . . .


Retrieved from "https://wiki.neelanurseries.com/index.php?title=Shell&oldid=8531"