From 1e9611e732dae94782ed02886785a81f06ad2410 Mon Sep 17 00:00:00 2001 From: Michael Klement Date: Fri, 17 Oct 2014 17:16:12 -0400 Subject: [PATCH] Support for cross-shell testing added, via option `-s ` and env. variable `TEST_SHELL`. * For **tests that _source_ shell scripts**: **option `-s `** now tells urchin to invoke test scripts with the specified shell (only shebang-less and `#!/bin/sh` tests scripts). * For **tests that _invoke_ schell scripts**: instruct users to write their tests to always **invoke via environment variable `TEST_SHELL` (e.g., `$TEST_SHELL ../foo`)**, and invoke urchin with that variable defined as needed, e.g., `TEST_SHELL=ksh urchin ./tests`; urchin defaults `TEST_SHELL` to `/bin/sh`. See updated `readme.md` for details. --- readme.md | 51 +++++++++++++++++ .../The -s option should be documented. | 3 + .../$TEST_SHELL should be bash. | 7 +++ ...ing a command with $TEST_SHELL should work | 4 ++ ...sh test script should be invoked directly. | 5 ++ ...hould be invoked with the specified shell. | 10 ++++ ...hould be invoked with the specified shell. | 10 ++++ ... should be set to the specified shell too. | 9 +++ ...iable TEST_SHELL should be passed through. | 9 +++ ... TEST_SHELL should default to sh if empty. | 8 +++ ... TEST_SHELL should default to sh if unset. | 9 +++ ...uld invoke tests with the specified shell. | 8 +++ urchin | 56 +++++++++++++++---- 13 files changed, 177 insertions(+), 12 deletions(-) create mode 100755 tests/Command-line help contents/The -s option should be documented. create mode 100755 tests/Cross-shell test support/.test-TEST_SHELL-passed-through/$TEST_SHELL should be bash. create mode 100755 tests/Cross-shell test support/.test-TEST_SHELL-undefined_or_empty/Invoking a command with $TEST_SHELL should work create mode 100755 tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a non-sh test script should be invoked directly. create mode 100755 tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a sh test script should be invoked with the specified shell. create mode 100755 tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a shebang-less test script should be invoked with the specified shell. create mode 100755 tests/Cross-shell test support/.test-run-by-specified-shell/With -s, the TEST_SHELL environment variable should be set to the specified shell too. create mode 100755 tests/Cross-shell test support/Environment variable TEST_SHELL should be passed through. create mode 100755 tests/Cross-shell test support/Environment variable TEST_SHELL should default to sh if empty. create mode 100755 tests/Cross-shell test support/Environment variable TEST_SHELL should default to sh if unset. create mode 100755 tests/Cross-shell test support/The -s option should invoke tests with the specified shell. diff --git a/readme.md b/readme.md index ac17a37..24e43af 100644 --- a/readme.md +++ b/readme.md @@ -42,6 +42,8 @@ Now you can run it. urchin +Run `urchin -h` to get command-line help. + ## Writing tests Make a root directory for your tests. Inside it, put executable files that exit `0` on success and something else on fail. Non-executable files and hidden @@ -84,6 +86,55 @@ Files are only run if they are executable, and files beginning with `.` are ignored. Thus, fixtures and libraries can be included sloppily within the test directory tree. The test passes if the file exits 0; otherwise, it fails. +### Writing cross-shell compatibility tests for testing shell code + +While you could write your test scripts to explicitly invoke the functionality +to test with various shells, urchin facilitates a more flexible approach. + +The specific approach depends on your test scenario: + +* (a) Your test scripts _invoke_ scripts containing portable shell code. +* (b) Your scripts _source_ scripts containing portable shell code. + +#### (a) Cross-shell tests with test scripts that _invoke_ shell scripts + +Write your test scripts to invoke the shell scripts to test via the shell +specified in environment variable `TEST_SHELL` rather than directly; +e.g.: `$TEST_SHELL ../foo bar` (rather than just `../foo bar`) + +Then, on invocation of urchin, prepend a definition of environment variable `TEST_SHELL` +specifying the shell to test with, e.g.: `TEST_SHELL=zsh urchin ./tests`. +To test with multiple shells in sequence, use something like: + + for shell in sh bash ksh zsh; do + TEST_SHELL=$shell urchin ./tests + done + +If `TEST_SHELL` has no value, urchin defines it as `/bin/sh`, so the test +scripts can rely on `$TEST_SHELL` always containing a value. + +#### (b) Cross-shell tests with test scripts that _source_ shell scripts + +If you _source_ shell code in your test scripts, it is the test scripts +themselves that must be run with the shell specified. + +To that end, urchin supports the `-s ` option, which instructs +urchin to invoke the test scripts with the specified shell; e.g., `-s bash` + +Note that only test scripts that either have no shebang line at all or +have shebang line '#!/bin/sh' are invoked with the specified shell. +This allows non-shell test scripts or test scripts for _specific, hard-coded_ +shells to coexist with those whose invocation should be controlled by `-s`. + +To test with multiple shells in sequence, use something like: + + for shell in sh bash ksh zsh; do + urchin -s $shell ./tests + done + +Urchin will also define environment variable `TEST_SHELL` to contain the +the shell specified via `-s`. + ## Alternatives to Urchin Alternatives to Urchin are discussed in [this blog post](https://blog.scraperwiki.com/2012/12/how-to-test-shell-scripts/). diff --git a/tests/Command-line help contents/The -s option should be documented. b/tests/Command-line help contents/The -s option should be documented. new file mode 100755 index 0000000..7b631f8 --- /dev/null +++ b/tests/Command-line help contents/The -s option should be documented. @@ -0,0 +1,3 @@ +#!/bin/sh + +../../urchin -h | grep -- -s diff --git a/tests/Cross-shell test support/.test-TEST_SHELL-passed-through/$TEST_SHELL should be bash. b/tests/Cross-shell test support/.test-TEST_SHELL-passed-through/$TEST_SHELL should be bash. new file mode 100755 index 0000000..07999fd --- /dev/null +++ b/tests/Cross-shell test support/.test-TEST_SHELL-passed-through/$TEST_SHELL should be bash. @@ -0,0 +1,7 @@ +#!/bin/sh + +# Assuming that urchin was invoked with `TEST_SHELL=bash urchin ...`, $TEST_SHELL should contain 'bash'. + +echo "\$TEST_SHELL: $TEST_SHELL" + +[ "$TEST_SHELL" = 'bash' ] diff --git a/tests/Cross-shell test support/.test-TEST_SHELL-undefined_or_empty/Invoking a command with $TEST_SHELL should work b/tests/Cross-shell test support/.test-TEST_SHELL-undefined_or_empty/Invoking a command with $TEST_SHELL should work new file mode 100755 index 0000000..6af3010 --- /dev/null +++ b/tests/Cross-shell test support/.test-TEST_SHELL-undefined_or_empty/Invoking a command with $TEST_SHELL should work @@ -0,0 +1,4 @@ +#!/bin/sh + +# Invoke a simple test command with $TEST_SHELL as the executable. +[ "$($TEST_SHELL -c 'echo $0')" = "$TEST_SHELL" ] diff --git a/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a non-sh test script should be invoked directly. b/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a non-sh test script should be invoked directly. new file mode 100755 index 0000000..2dd5303 --- /dev/null +++ b/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a non-sh test script should be invoked directly. @@ -0,0 +1,5 @@ +#!/usr/bin/awk -f + +# This script will only succeed if it is indeed processed by awk. + +BEGIN { print "ok" } diff --git a/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a sh test script should be invoked with the specified shell. b/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a sh test script should be invoked with the specified shell. new file mode 100755 index 0000000..2fb6fbb --- /dev/null +++ b/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a sh test script should be invoked with the specified shell. @@ -0,0 +1,10 @@ +#!/bin/sh + +# Assuming that urchin was invoked with `-s bash`, this script should be being run with bash. + +this_shell=$(ps -o comm= -p $$ && :) + +echo "Running shell: $this_shell" + +[ "$this_shell" = 'bash' ] + diff --git a/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a shebang-less test script should be invoked with the specified shell. b/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a shebang-less test script should be invoked with the specified shell. new file mode 100755 index 0000000..ffc2ec4 --- /dev/null +++ b/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a shebang-less test script should be invoked with the specified shell. @@ -0,0 +1,10 @@ +# By design, this file has no shebang line. + +# Assuming that urchin was invoked with `-s bash`, this script should be being run with bash. + +this_shell=$(ps -o comm= -p $$ && :) + +echo "Running shell: $this_shell" + +[ "$this_shell" = 'bash' ] + diff --git a/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, the TEST_SHELL environment variable should be set to the specified shell too. b/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, the TEST_SHELL environment variable should be set to the specified shell too. new file mode 100755 index 0000000..910b7fd --- /dev/null +++ b/tests/Cross-shell test support/.test-run-by-specified-shell/With -s, the TEST_SHELL environment variable should be set to the specified shell too. @@ -0,0 +1,9 @@ +#!/bin/sh + +# Assuming that urchin was invoked with `-s bash`, $TEST_SHELL should contain 'bash'. + +echo "Running shell: $(ps -o comm= -p $$ && :)" + +echo "\$TEST_SHELL: $TEST_SHELL" + +[ "$TEST_SHELL" = 'bash' ] diff --git a/tests/Cross-shell test support/Environment variable TEST_SHELL should be passed through. b/tests/Cross-shell test support/Environment variable TEST_SHELL should be passed through. new file mode 100755 index 0000000..4f2d555 --- /dev/null +++ b/tests/Cross-shell test support/Environment variable TEST_SHELL should be passed through. @@ -0,0 +1,9 @@ +#!/bin/sh + +# Tests support for either passing through or defining a default value for environment variable TEST_SHELL. +# (for test scripts that want to invoke shell scripts with a specified shell). + +which bash 2>/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 1; } + +# Test if $TEST_SHELL, when placed in urchin's environment, is passed through to the test scripts. +TEST_SHELL=bash ../../urchin ./.test-TEST_SHELL-passed-through diff --git a/tests/Cross-shell test support/Environment variable TEST_SHELL should default to sh if empty. b/tests/Cross-shell test support/Environment variable TEST_SHELL should default to sh if empty. new file mode 100755 index 0000000..472f0b4 --- /dev/null +++ b/tests/Cross-shell test support/Environment variable TEST_SHELL should default to sh if empty. @@ -0,0 +1,8 @@ +#!/bin/sh + +# Tests support for either passing through or defining a default value for environment variable TEST_SHELL. +# (for test scripts that want to invoke shell scripts with a specified shell). + +# Test if $TEST_SHELL - if *defined, but empty* - is exported with value '/bin/sh' by urchin +# and thus has that value inside the scripts. +TEST_SHELL= ../../urchin ./.test-TEST_SHELL-undefined_or_empty diff --git a/tests/Cross-shell test support/Environment variable TEST_SHELL should default to sh if unset. b/tests/Cross-shell test support/Environment variable TEST_SHELL should default to sh if unset. new file mode 100755 index 0000000..89affdd --- /dev/null +++ b/tests/Cross-shell test support/Environment variable TEST_SHELL should default to sh if unset. @@ -0,0 +1,9 @@ +#!/bin/sh + +# Tests support for either passing through or defining a default value for environment variable TEST_SHELL. +# (for test scripts that want to invoke shell scripts with a specified shell). + +# Test if $TEST_SHELL - if *undefined* - is exported with value '/bin/sh' by urchin +# and thus has that value inside test scripts. +unset -v TEST_SHELL +../../urchin ./.test-TEST_SHELL-undefined_or_empty diff --git a/tests/Cross-shell test support/The -s option should invoke tests with the specified shell. b/tests/Cross-shell test support/The -s option should invoke tests with the specified shell. new file mode 100755 index 0000000..62e1264 --- /dev/null +++ b/tests/Cross-shell test support/The -s option should invoke tests with the specified shell. @@ -0,0 +1,8 @@ +#!/bin/sh + +# Tests the `-s option, which invokes shebang-less and sh-shebang-line test scripts with the specified shell (for testing *sourced* shell code). + +which bash >/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 2; } +which /usr/bin/awk >/dev/null || { echo "Cannot test -s option: /usr/bin/awk not found." >&2; exit 2; } + +../../urchin -s bash ./.test-run-by-specified-shell diff --git a/urchin b/urchin index 1455772..fd197a1 100755 --- a/urchin +++ b/urchin @@ -15,6 +15,7 @@ indent() { recurse() { potential_test="$1" indent_level="$2" + shell_for_sh_tests="$3" [ "$potential_test" = 'setup_dir' ] && return [ "$potential_test" = 'teardown_dir' ] && return @@ -35,7 +36,7 @@ recurse() { [ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file" # $2 instead of $indent_level so it doesn't clash - recurse "${test}" $(( $2 + 1 )) + recurse "${test}" $(( $2 + 1 )) "$shell_for_sh_tests" [ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file" done @@ -48,9 +49,15 @@ recurse() { [ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file" # Run the test - ./"$potential_test" > "$stdout_file" 2>&1 + if [ -n "$shell_for_sh_tests" ] && has_sh_or_no_shebang_line ./"$potential_test" + then + TEST_SHELL="$TEST_SHELL" "$shell_for_sh_tests" ./"$potential_test" > "$stdout_file" 2>&1 + else + TEST_SHELL="$TEST_SHELL" ./"$potential_test" > "$stdout_file" 2>&1 + fi exit_code="$?" + [ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file" indent $indent_level @@ -73,15 +80,23 @@ recurse() { [ $indent_level -eq 0 ] && rm "$stdout_file" } -USAGE="usage: $0 " +has_sh_or_no_shebang_line() { + head -n 1 "$1" | grep -vqE '^#!' && return 0 # no shebang line at all + head -n 1 "$1" | grep -qE '^#![[:blank:]]*/bin/sh($|[[:blank:]])' && return 0 # shebang line is '#!/bin/sh' or legal variations thereof + return 1 +} + +USAGE="usage: $0 [] " urchin_help() { echo echo "$USAGE" echo - echo '-f Force urchin to run on directories whose name does not contain' - echo ' the word "test".' - echo '-h This help' + echo '-s Invoke test scripts that either have no shebang line or' + echo ' shebang line "#!/bin/sh" with the specified shell.' + echo '-f Force running even if the test directory'\''s name does not' + echo ' contain the word "test".' + echo '-h This help.' # echo # echo '--xsd Output xUnit XML schema for an integration server.' echo @@ -105,7 +120,20 @@ urchin_go() { echo Running tests at $(date +%Y-%m-%dT%H:%M:%S) | tee "$logfile" start=$(date +%s) - recurse "$1" 0 # test folder, indentation level + # Determine the environment variable to define for test scripts + # that reflects the specified or implied shell to use for shell-code tests. + # - Set it to the shell specified via -s, if any. + # - Otherwise, use its present value, if non-empty. + # - Otherwise, default to '/bin/sh'. + if [ -n "$2" ] + then + TEST_SHELL="$2" + elif [ -z "$TEST_SHELL" ] + then + TEST_SHELL='/bin/sh' + fi + + recurse "$1" 0 "$2" # test folder -- indentation level -- [shell to invoke test scripts with] finish=$(date +%s) elapsed=$(($finish - $start)) @@ -130,16 +158,20 @@ urchin_molly_guard() { exit 1 } +shell_for_sh_tests= force=false while [ $# -gt 0 ] do case "$1" in -f) force=true;; + -s) + shift + shell_for_sh_tests=$1 + which "$shell_for_sh_tests" >/dev/null || { echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2; urchin_help >&2; exit 2; } + ;; -h|--help) urchin_help exit 0;; -# --xsd) action=testsuite;; -# --) shift; break;; - -*) urchin_help 1>&2 + -*) urchin_help >&2 exit 1;; *) break;; esac @@ -151,7 +183,7 @@ if [ "$#" != '1' ] || [ ! -d "$1" ] then [ -d "$1" ] || echo "Not a directory: '$1'" >&2 echo "$USAGE" >&2 - exit 1 + exit 2 fi # Constants @@ -161,7 +193,7 @@ stdout_file=$(fullpath "$1")/.urchin_stdout # Run or present the Molly guard. if basename "$(fullpath "$1")" | grep -Fi 'test' > /dev/null || $force then - urchin_go "$1" + urchin_go "$1" "$shell_for_sh_tests" else urchin_molly_guard fi