Support for cross-shell testing added, via option `-s <shell>` and env. variable `TEST_SHELL`.

* For **tests that _source_ shell scripts**: **option `-s <shell>`** now tells urchin to invoke test scripts with the specified shell (only shebang-less and `#!/bin/sh` tests scripts).
* For **tests that _invoke_ schell scripts**: instruct users to write their tests to always **invoke via environment variable `TEST_SHELL` (e.g., `$TEST_SHELL ../foo`)**, and invoke urchin with that variable defined as needed, e.g., `TEST_SHELL=ksh urchin ./tests`; urchin defaults `TEST_SHELL` to `/bin/sh`.

See updated `readme.md` for details.
This commit is contained in:
Michael Klement 2014-10-17 17:16:12 -04:00
parent deb77cb5de
commit 1e9611e732
13 changed files with 177 additions and 12 deletions

View File

@ -42,6 +42,8 @@ Now you can run it.
urchin <test directory> urchin <test directory>
Run `urchin -h` to get command-line help.
## Writing tests ## Writing tests
Make a root directory for your tests. Inside it, put executable files that Make a root directory for your tests. Inside it, put executable files that
exit `0` on success and something else on fail. Non-executable files and hidden exit `0` on success and something else on fail. Non-executable files and hidden
@ -84,6 +86,55 @@ Files are only run if they are executable, and files beginning with `.` are
ignored. Thus, fixtures and libraries can be included sloppily within the test ignored. Thus, fixtures and libraries can be included sloppily within the test
directory tree. The test passes if the file exits 0; otherwise, it fails. directory tree. The test passes if the file exits 0; otherwise, it fails.
### Writing cross-shell compatibility tests for testing shell code
While you could write your test scripts to explicitly invoke the functionality
to test with various shells, urchin facilitates a more flexible approach.
The specific approach depends on your test scenario:
* (a) Your test scripts _invoke_ scripts containing portable shell code.
* (b) Your scripts _source_ scripts containing portable shell code.
#### (a) Cross-shell tests with test scripts that _invoke_ shell scripts
Write your test scripts to invoke the shell scripts to test via the shell
specified in environment variable `TEST_SHELL` rather than directly;
e.g.: `$TEST_SHELL ../foo bar` (rather than just `../foo bar`)
Then, on invocation of urchin, prepend a definition of environment variable `TEST_SHELL`
specifying the shell to test with, e.g.: `TEST_SHELL=zsh urchin ./tests`.
To test with multiple shells in sequence, use something like:
for shell in sh bash ksh zsh; do
TEST_SHELL=$shell urchin ./tests
done
If `TEST_SHELL` has no value, urchin defines it as `/bin/sh`, so the test
scripts can rely on `$TEST_SHELL` always containing a value.
#### (b) Cross-shell tests with test scripts that _source_ shell scripts
If you _source_ shell code in your test scripts, it is the test scripts
themselves that must be run with the shell specified.
To that end, urchin supports the `-s <shell>` option, which instructs
urchin to invoke the test scripts with the specified shell; e.g., `-s bash`
Note that only test scripts that either have no shebang line at all or
have shebang line '#!/bin/sh' are invoked with the specified shell.
This allows non-shell test scripts or test scripts for _specific, hard-coded_
shells to coexist with those whose invocation should be controlled by `-s`.
To test with multiple shells in sequence, use something like:
for shell in sh bash ksh zsh; do
urchin -s $shell ./tests
done
Urchin will also define environment variable `TEST_SHELL` to contain the
the shell specified via `-s`.
## Alternatives to Urchin ## Alternatives to Urchin
Alternatives to Urchin are discussed in Alternatives to Urchin are discussed in
[this blog post](https://blog.scraperwiki.com/2012/12/how-to-test-shell-scripts/). [this blog post](https://blog.scraperwiki.com/2012/12/how-to-test-shell-scripts/).

View File

@ -0,0 +1,3 @@
#!/bin/sh
../../urchin -h | grep -- -s

View File

@ -0,0 +1,7 @@
#!/bin/sh
# Assuming that urchin was invoked with `TEST_SHELL=bash urchin ...`, $TEST_SHELL should contain 'bash'.
echo "\$TEST_SHELL: $TEST_SHELL"
[ "$TEST_SHELL" = 'bash' ]

View File

@ -0,0 +1,4 @@
#!/bin/sh
# Invoke a simple test command with $TEST_SHELL as the executable.
[ "$($TEST_SHELL -c 'echo $0')" = "$TEST_SHELL" ]

View File

@ -0,0 +1,5 @@
#!/usr/bin/awk -f
# This script will only succeed if it is indeed processed by awk.
BEGIN { print "ok" }

View File

@ -0,0 +1,10 @@
#!/bin/sh
# Assuming that urchin was invoked with `-s bash`, this script should be being run with bash.
this_shell=$(ps -o comm= -p $$ && :)
echo "Running shell: $this_shell"
[ "$this_shell" = 'bash' ]

View File

@ -0,0 +1,10 @@
# By design, this file has no shebang line.
# Assuming that urchin was invoked with `-s bash`, this script should be being run with bash.
this_shell=$(ps -o comm= -p $$ && :)
echo "Running shell: $this_shell"
[ "$this_shell" = 'bash' ]

View File

@ -0,0 +1,9 @@
#!/bin/sh
# Assuming that urchin was invoked with `-s bash`, $TEST_SHELL should contain 'bash'.
echo "Running shell: $(ps -o comm= -p $$ && :)"
echo "\$TEST_SHELL: $TEST_SHELL"
[ "$TEST_SHELL" = 'bash' ]

View File

@ -0,0 +1,9 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
which bash 2>/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 1; }
# Test if $TEST_SHELL, when placed in urchin's environment, is passed through to the test scripts.
TEST_SHELL=bash ../../urchin ./.test-TEST_SHELL-passed-through

View File

@ -0,0 +1,8 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
# Test if $TEST_SHELL - if *defined, but empty* - is exported with value '/bin/sh' by urchin
# and thus has that value inside the scripts.
TEST_SHELL= ../../urchin ./.test-TEST_SHELL-undefined_or_empty

View File

@ -0,0 +1,9 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
# Test if $TEST_SHELL - if *undefined* - is exported with value '/bin/sh' by urchin
# and thus has that value inside test scripts.
unset -v TEST_SHELL
../../urchin ./.test-TEST_SHELL-undefined_or_empty

View File

@ -0,0 +1,8 @@
#!/bin/sh
# Tests the `-s <shell> option, which invokes shebang-less and sh-shebang-line test scripts with the specified shell (for testing *sourced* shell code).
which bash >/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 2; }
which /usr/bin/awk >/dev/null || { echo "Cannot test -s option: /usr/bin/awk not found." >&2; exit 2; }
../../urchin -s bash ./.test-run-by-specified-shell

56
urchin
View File

@ -15,6 +15,7 @@ indent() {
recurse() { recurse() {
potential_test="$1" potential_test="$1"
indent_level="$2" indent_level="$2"
shell_for_sh_tests="$3"
[ "$potential_test" = 'setup_dir' ] && return [ "$potential_test" = 'setup_dir' ] && return
[ "$potential_test" = 'teardown_dir' ] && return [ "$potential_test" = 'teardown_dir' ] && return
@ -35,7 +36,7 @@ recurse() {
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file" [ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# $2 instead of $indent_level so it doesn't clash # $2 instead of $indent_level so it doesn't clash
recurse "${test}" $(( $2 + 1 )) recurse "${test}" $(( $2 + 1 )) "$shell_for_sh_tests"
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file" [ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
done done
@ -48,9 +49,15 @@ recurse() {
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file" [ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# Run the test # Run the test
./"$potential_test" > "$stdout_file" 2>&1 if [ -n "$shell_for_sh_tests" ] && has_sh_or_no_shebang_line ./"$potential_test"
then
TEST_SHELL="$TEST_SHELL" "$shell_for_sh_tests" ./"$potential_test" > "$stdout_file" 2>&1
else
TEST_SHELL="$TEST_SHELL" ./"$potential_test" > "$stdout_file" 2>&1
fi
exit_code="$?" exit_code="$?"
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file" [ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
indent $indent_level indent $indent_level
@ -73,15 +80,23 @@ recurse() {
[ $indent_level -eq 0 ] && rm "$stdout_file" [ $indent_level -eq 0 ] && rm "$stdout_file"
} }
USAGE="usage: $0 <test directory>" has_sh_or_no_shebang_line() {
head -n 1 "$1" | grep -vqE '^#!' && return 0 # no shebang line at all
head -n 1 "$1" | grep -qE '^#![[:blank:]]*/bin/sh($|[[:blank:]])' && return 0 # shebang line is '#!/bin/sh' or legal variations thereof
return 1
}
USAGE="usage: $0 [<options>] <test directory>"
urchin_help() { urchin_help() {
echo echo
echo "$USAGE" echo "$USAGE"
echo echo
echo '-f Force urchin to run on directories whose name does not contain' echo '-s <shell> Invoke test scripts that either have no shebang line or'
echo ' the word "test".' echo ' shebang line "#!/bin/sh" with the specified shell.'
echo '-h This help' echo '-f Force running even if the test directory'\''s name does not'
echo ' contain the word "test".'
echo '-h This help.'
# echo # echo
# echo '--xsd Output xUnit XML schema for an integration server.' # echo '--xsd Output xUnit XML schema for an integration server.'
echo echo
@ -105,7 +120,20 @@ urchin_go() {
echo Running tests at $(date +%Y-%m-%dT%H:%M:%S) | tee "$logfile" echo Running tests at $(date +%Y-%m-%dT%H:%M:%S) | tee "$logfile"
start=$(date +%s) start=$(date +%s)
recurse "$1" 0 # test folder, indentation level # Determine the environment variable to define for test scripts
# that reflects the specified or implied shell to use for shell-code tests.
# - Set it to the shell specified via -s, if any.
# - Otherwise, use its present value, if non-empty.
# - Otherwise, default to '/bin/sh'.
if [ -n "$2" ]
then
TEST_SHELL="$2"
elif [ -z "$TEST_SHELL" ]
then
TEST_SHELL='/bin/sh'
fi
recurse "$1" 0 "$2" # test folder -- indentation level -- [shell to invoke test scripts with]
finish=$(date +%s) finish=$(date +%s)
elapsed=$(($finish - $start)) elapsed=$(($finish - $start))
@ -130,16 +158,20 @@ urchin_molly_guard() {
exit 1 exit 1
} }
shell_for_sh_tests=
force=false force=false
while [ $# -gt 0 ] while [ $# -gt 0 ]
do do
case "$1" in case "$1" in
-f) force=true;; -f) force=true;;
-s)
shift
shell_for_sh_tests=$1
which "$shell_for_sh_tests" >/dev/null || { echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2; urchin_help >&2; exit 2; }
;;
-h|--help) urchin_help -h|--help) urchin_help
exit 0;; exit 0;;
# --xsd) action=testsuite;; -*) urchin_help >&2
# --) shift; break;;
-*) urchin_help 1>&2
exit 1;; exit 1;;
*) break;; *) break;;
esac esac
@ -151,7 +183,7 @@ if [ "$#" != '1' ] || [ ! -d "$1" ]
then then
[ -d "$1" ] || echo "Not a directory: '$1'" >&2 [ -d "$1" ] || echo "Not a directory: '$1'" >&2
echo "$USAGE" >&2 echo "$USAGE" >&2
exit 1 exit 2
fi fi
# Constants # Constants
@ -161,7 +193,7 @@ stdout_file=$(fullpath "$1")/.urchin_stdout
# Run or present the Molly guard. # Run or present the Molly guard.
if basename "$(fullpath "$1")" | grep -Fi 'test' > /dev/null || $force if basename "$(fullpath "$1")" | grep -Fi 'test' > /dev/null || $force
then then
urchin_go "$1" urchin_go "$1" "$shell_for_sh_tests"
else else
urchin_molly_guard urchin_molly_guard
fi fi