Enhanced BASH Scripts

• Enhanced BASH Scripts
  • Roadmap
  • Local Dev Environment - Requirements
  • TDD - Test Driven Development, run tests on file change
  • Usage
    • Colors
    • Script Dependencies
    • Logger
    • Arguments Parsing
    • Common(s) Functions And Inputs
    • UI: Selector
    • UI: Ask for Password
  • Semver - Semantic Versioning
  • Self-Update
  • Profile BASH script execution
  • Colors support in my terminal
  • References

Roadmap

• High-level scripts should be in own bin OR scripts
• Git helpers
• GitLabs helper scripts (work with branches, forks, submodules)
• Slack notifications helper scripts
• Telemetry module (report metrics to CI or DataDog)
• Globals module (declarative way of defining script dependencies to global environment variables)
• Logs monitoring documentation (different streams/files/tty for different information: info, debug, telemetry, dependencies)
• Copyright headers composing/parsing (extract from the file, update, insert)

Local Dev Environment - Requirements

• DirEnv - https://github.com/direnv/direnv
• ShellFormat - https://github.com/mvdan/sh
• ShellCheck - https://github.com/koalaman/shellcheck
• KCov - https://github.com/SimonKagstrom/kcov
• ShellSpec - https://github.com/shellspec/shellspec

Note: alternative Unit Test Frameworks, Bats - https://github.com/bats-core/bats-core

brew install direnv
brew install shellcheck
brew install shfmt
brew install shellspec
brew install kcov

TDD - Test Driven Development, run tests on file change

# run all unit tests on file change
watchman-make -p 'spec/*_spec.sh' '.scripts/*.sh' --run "shellspec"

# run failed only unit tests on file change
watchman-make -p 'spec/*_spec.sh' '.scripts/*.sh' --run "shellspec --quick"

Usage

Installation into your project:

# subtree approach (TODO: fix me!)
git subtree --squash -P vendor/bats-all add https://github.com/hyperupcall/bats-all HEAD

refs:

• https://gist.github.com/SKempin/b7857a6ff6bddb05717cc17a44091202
• https://github.com/epcim/git-cross
• https://github.com/ingydotnet/git-subrepo
• https://gist.github.com/icheko/9ff2a0a90ef2b676a5fc8d76f69db1d3 article

Colors

source ".scripts/_colors.sh"

echo -e "${cl_red}Hello World${cl_reset}"

Script Dependencies

source ".scripts/_dependencies.sh"

dependency bash "5.*.*" "brew install bash"
dependency direnv "2.*.*" "curl -sfL https://direnv.net/install.sh | bash"
dependency shellspec "0.28.*" "brew install shellspec"
optional kcov "42" "brew install kcov"
dependency shellcheck "0.9.*" "curl -sS https://webi.sh/shellcheck | sh"
dependency shfmt "3.*.*" "curl -sS https://webi.sh/shfmt | sh"
dependency watchman "2023.07.*.*" "brew install watchman"

Logger

Requirements:

• zero dependencies, pure BASH
• prefix for all logger messages
• work in pipe mode (forward logs to the named pipe)
  • write logs to pipe; single line or multiple lines in '|' pipe mode
  • read logs from the named pipe and output to the console (or file). Required PID and log tag.
  • run logs to file/stream in background process mode
• support prefix for each log message
• listen to DEBUG environment variable for enabling/disabling logs
  • enable/disable log by tag name or tag name prefix (support wildcards)
• execute command with logging the command and it parameters first (ref: https://bpkg.sh/pkg/echo-eval)

source ".scripts/_logger.sh"
logger common "$@" # declare echo:Common and printf:Common functions, tag: common
logger debug "$@" # declare echo:Debug and printf:Debug functions, tag: debug

echo:Common "Hello World" # output "Hello World" only if tag common is enabled

export DEBUG=*          # enable logger output for all tags
export DEBUG=common     # enable logger output for common tag only
export DEBUG=*,-common  # enable logger output for all tags except common

# advanced functions
config:logger:Common "$@" # re-configure logger enable/disable for common tag

# echo in pipe mode
find . -type d -max-depth 1 | echo:Common

# echo in output redirect
find . -type d -max-depth 1 >echo:Common

Arguments Parsing

Requirements:

• zero dependencies, pure BASH
• support short and long arguments
• support default values
• support required arguments
• support aliases for arguments
• support destination variables for argument
• compose help documentation from arguments definition

# pattern: "{argument},-{short},--{alias}={output_variable}:{default_initialize_value}:{reserved_args_quantity}"
# example: "-h,--help=args_help:true:0", on --help or -h set $args_help variable to true, expect no arguments;
# example: "$1,--id=args_id::1", expect first unnamed argument to be assigned to $args_id variable; can be also provided as --id=123
export ARGS_DEFINITION="-h,--help -v,--version=:1.0.0 --debug=DEBUG:*"

# will automatically parse script arguments with definition from $ARGS_DEFINITION global variable
source ".scripts/_arguments.sh"

# check variables that are extracted
echo "Is --help: $help"
echo "Is --version: $version"
echo "Is --debug: $DEBUG"

# advanced run. parse provided arguments with definition from $ARGS_DEFINITION global variable
parse:arguments "$@"

Common(s) Functions And Inputs

source ".scripts/_commons.sh"

# Extract parameter from global env variable OR from secret file (file content)
env:variable:or:secret:file "new_value" \
  "GITLAB_CI_INTEGRATION_TEST" \
  ".secrets/gitlab_ci_integration_test" \
  "{user friendly message}"

echo "Extracted: ${new_value}"

# validate/confirm input parameter by user input
# string
# Yes/No

UI: Selector

source ".scripts/_commons.sh"

# Select value from short list of choices
declare -A -g connections && connections=(["d"]="production" ["s"]="cors-proxy:staging" ["p"]="cors-proxy:local")
echo -n "Select connection type: " && tput civis # hide cursor
selected=$(input:selector "connections") && echo "${cl_blue}${selected}${cl_reset}"

UI: Ask for Password

source ".scripts/_commons.sh"

# Usage:
echo -n "Enter password: "
password=$(input:readpwd) && echo "" && echo "Password: $password"

Semver - Semantic Versioning

Requirements:

• parse version code, according to semver specification
• compare version code
• verify version constraints
• compose version code from array of segments

source ".scripts/_semver.sh"

# verify that version is passing the constraints expression
semver:constraints "1.0.0-alpha" ">1.0.0-beta || <1.0.0" && echo "$? - OK!" || echo "$? - FAIL!" # expected OK

# more specific cases
semver:constraints:simple "1.0.0-beta.10 != 1.0.0-beta.2" && echo "OK!" || echo "$? - FAIL!"

# parse and recompose version code
semver:parse "2.0.0-rc.1+build.123" "V" \
  && for i in "${!V[@]}"; do echo "$i: ${V[$i]}"; done \
  && semver:recompose "V"
  
# test version code
echo "1" | grep -E "${SEMVER_LINE}" --color=always --ignore-case || echo "OK!"

Self-Update

Requirements:

• detect a new version of the script
• download multiple versions into folder and do a symbolic link to a specific version
• download from GIT repo (git clone)
  • keep MASTER as default, extract version tags as sub-folders
• download from GIT repo release URL (tar/zip archive)
  • extract archive to a version sub-folder
• rollback to previous version (or specified one)
  • rollback to latest backup file (if exists)
• partial update of the scripts, different versions of scripts from different version sub-folders
  • developer can bind file to a specific version by calling function self-update:version:bind
• verify SHA1 hash of the scripts
  • compute file SHA1 hash and store it in *.sha1 file
• understand version expressions
  • latest - latest stable version
  • * or next - any highest version tag (INCLUDING: alpha, beta, rc etc)
  • branch:{any_branch} or tag:{any_tag} - any branch name (also works for TAGs)
  • >, <, >=, <=, ~, !=, || - comparison syntax
  • 1.0.0 or =1.0.0 - exact version
  • ~1.0.0 - version in range >= 1.0.x, patch releases allowed
  • ^1.0.0 - version in range >= 1.x.x, minor & patch releases allowed
  • >1.0.0 <=1.5.0 - version in range > 1.0.0 && <= 1.5.0
  • >1.0.0 <1.1.0 || >1.5.0 - version in range (> 1.0.0 < 1.1.0) || (> 1.5.0)

refs:

• https://classic.yarnpkg.com/lang/en/docs/dependency-versions/
• https://github.com/fsaintjacques/semver-tool
• https://github.com/Masterminds/semver
• https://stackoverflow.com/questions/356100/how-to-wait-in-bash-for-several-subprocesses-to-finish-and-return-exit-code-0

source ".scripts/_self-update.sh"

# check for version update in range >= 1.0.x, stable versions
# try to update itself from https://github.com/OleksandrKucherenko/e-bash.git repository
self-update "~1.0.0"                          # patch releases allowed
self-update "^1.0.0"                          # minor releases allowed
self-update "> 1.0.0 <= 1.5.0"                # stay in range

# update specific file to latest version tag
self-update "latest" ".scripts/_colors.sh"    # latest stable
self-update "*" ".scripts/_colors.sh"         # any highest version tag

# update specific file to MASTER version (can be used any branch name)
self-update "branch:master" ".scripts/_colors.sh"   
self-update "tag:v1.0.0" ".scripts/_colors.sh"

# bind file to a specific version
self-update:version:bind "v1.0.0" ".scripts/_colors.sh"

# TBD

# INTEGRATION EXAMPLE

# do self-update on script exit
trap "self-update '^1.0.0'" EXIT

# OR:
function __exit() {
  # TODO: add more cleanup logic here
  self-update '^1.0.0'
}
trap "__exit" EXIT

Troubleshooting

# rollback with use of backup file(s)
source ".scripts/_self-update.sh" && self-update:rollback:backup "${full_path_to_file}"

# rollback to specific version
source ".scripts/_self-update.sh" && self-update:rollback:version "v1.0.0" "${full_path_to_file}"

Profile BASH script execution

# print timestamp for each line of executed script
PS4='+ $(gdate "+%s.%N ($LINENO) ")' bash -x bin/version-up.sh

# save trace to file
PS4='+ $(echo -n "$EPOCHREALTIME [$LINENO]: ")' bash -x bin/version-up.sh 2>trace.log

# process output to more user-friendly format: `execution_time | line_number | line_content`
PS4='+ $(echo -n "$EPOCHREALTIME [$LINENO]: ")' bash -x bin/version-up.sh 2>trace.log 1>/dev/null && cat trace.log | bin/profiler/tracing.sh

# profile script execution and print summary
bin/profiler/profile.sh bin/version-up.sh

• ref1: https://itecnote.com/tecnote/r-performance-profiling-tools-for-shell-scripts/
• ref2: https://www.thegeekstuff.com/2008/09/bash-shell-take-control-of-ps1-ps2-ps3-ps4-and-prompt_command/

Colors support in my terminal

# print all colors for easier selection
demos/demo.colors.sh

References

• PV - https://manpages.ubuntu.com/manpages/focal/man1/pv.1.html
• https://catern.com/posts/pipes.html
• https://stackoverflow.com/questions/238073/how-to-add-a-progress-bar-to-a-shell-script
• bash-core, trap enhancement
• bash-bastion BASH helpers
• https://github.com/dylanaraps/writing-a-tui-in-bash