53 Commits
v0.0.2 ... tap

Author SHA1 Message Date
Thomas Levine
5e3c918675 add + indent symbol so directories line up 2016-01-28 15:12:34 +00:00
Thomas Levine
f5f98c8c2f fix fixture to not expect stderr 2016-01-28 15:11:27 +00:00
Thomas Levine
ce175cc3df test stdout stderr for TAP 2016-01-28 14:55:11 +00:00
Thomas Levine
cabaaa7ba4 TAP test 2016-01-28 14:51:43 +00:00
Thomas Levine
a663085069 remove awk dependency in test suite
I'm on NixOS, so awk isn't in /usr/bin. I can search for it with
/usr/bin/env, but then I can't use -f in the shebang line.
2016-01-27 10:51:04 +00:00
Thomas Levine
6bb606a3cc test plan 2016-01-27 10:39:07 +00:00
Thomas Levine
1c93e9a5c2 logfile 2016-01-27 10:31:51 +00:00
Thomas Levine
508e695dc3 error on fail 2016-01-27 10:25:21 +00:00
Thomas Levine
a51d96631f print skip count at end 2016-01-27 10:24:28 +00:00
Thomas Levine
1cd9991587 print stdout for tap and not-tap 2016-01-27 10:17:33 +00:00
Thomas Levine
0f1c2848b4 tap indentation comments for directories 2016-01-27 10:14:22 +00:00
Thomas Levine
9d10e12633 assorted tap stuff 2016-01-27 09:48:30 +00:00
Thomas Levine
40979f6e18 more tap 2016-01-27 01:13:55 +00:00
Thomas Levine
33e158e8f7 more tap 2016-01-27 01:08:04 +00:00
Thomas Levine
7ecacad132 oops 2016-01-27 01:05:43 +00:00
Thomas Levine
c8df46014d test test successes better 2016-01-27 01:05:08 +00:00
Thomas Levine
de2da89169 convert indents to comments 2016-01-27 00:50:55 +00:00
Thomas Levine
a6d6730e74 start writing non-tap cases 2016-01-27 00:32:51 +00:00
Thomas Levine
d01e993041 call urchin -s in cross-shell tests 2016-01-25 14:00:47 +00:00
Thomas Levine
b5c6464eab remove "urchin -x" test 2016-01-25 13:57:14 +00:00
Thomas Levine
329fc27929 remove "urchin -x"
unnecessary now that shall exists
2016-01-25 13:35:54 +00:00
Thomas Levine
48e5090091 start writing alternatives 2015-12-09 07:46:16 -05:00
Thomas Levine
5abf088eb8 add new ideas 2015-10-19 12:52:24 -04:00
Thomas Levine
c84851cb5c document the ordering of tests 2015-08-21 09:03:15 -04:00
Thomas Levine
510476f924 mention shall in the documentation 2015-07-05 17:39:38 +00:00
Thomas Levine
f01869fb97 packaging 2014-12-08 02:08:00 +00:00
Michael Klement
0385dcd86e Fixed blog link in read-me and commented out experimental -x option for now. 2014-12-05 18:50:19 -05:00
Michael Klement
2847b020b4 Make sure that CDPATH isn't set, as it causes cd to behave unpredictably - notably, it can produce output, which breaks fullpath().
Also: Improved CLI help, updated URLs in read-me, cleaned up package.json:

I've tried to clarify the intent of `-x` in the CLI help, but I haven't touched the read-me in that respect.

I don't see any benefit to `-x`:

* Just using `#/bin/sh` as the shebang line in combination with `-s <shell>` gives you the same functionality,
* When it comes to invoking scripts from _within_ test scripts, nothing can do the work for you: you consciously have to mark the invocation with _something_ to indicate that it should be controlled from the outside; it won't get any easier than `$TEST_SHELL ...`
* Finally, using a shebang line such as `#!/usr/bin/env urchin -x` is problematic for two reasons:
  * Some platforms can handle only *1* argument in a shebang line.
  * In a _package-local_ installation, `#!/usr/bin/env` may not find the Urchin executable.

I'm also not sure how the following (from `readme.md`) fits in the picture:

> It might make sense if you do this.

    export TEST_SHELL=zsh && urchin -x
    export TEST_SHELL=bash && urchin -x

(As an aside: To achieve the same thing, you don't need `export`; `TEST_SHELL=zsh urchin -x`  and `TEST_SHELL=bash urchin -x`  is the better choice.)
How does this relate to use in a  _shebang line_?

`urchin_help()` now uses a here-doc: easier to maintain, and should work in all Bourne-like shells.

`readmeFilename` removed from `package.json`:

> "The readmeFilename does not need to ever be in your actual package.json file" - npm/npm#3573
2014-12-03 09:54:15 -05:00
Michael Klement
fae24e926a Improved CLI help, updated URLs in read-me 2014-11-06 22:21:05 -05:00
Thomas Levine
2ab070c353 fix the -x documentation 2014-11-06 00:52:34 +00:00
Thomas Levine
292663dd73 contributors 2014-11-06 00:51:14 +00:00
Thomas Levine
90abc2640d update the -h help text 2014-11-05 17:49:53 +00:00
Thomas Levine
19ed3b02e8 HISTORY 2014-11-05 17:49:08 +00:00
Thomas Levine
b0429315ec urchin -sh to urchin -x 2014-11-05 17:47:36 +00:00
Thomas Levine
5e06ffb1c1 version 2014-11-05 17:47:01 +00:00
Thomas Levine
bf8c404784 fix -sh 2014-11-05 17:46:19 +00:00
Thomas Levine
bb41d5a156 zsh support 2014-11-05 17:38:22 +00:00
Thomas Levine
ed09351df6 fix a test 2014-11-05 17:28:54 +00:00
Thomas Levine
1e2d4cc8a3 remove gitHead 2014-11-05 17:27:26 +00:00
Thomas Levine
f8d921b5e5 document urchin -sh 2014-11-05 17:26:01 +00:00
Thomas Levine
1f115df222 urchin -sh 2014-11-05 17:18:31 +00:00
Thomas Levine
eafed6ac59 put history back 2014-11-05 16:54:36 +00:00
Thomas Levine
8273661440 tlevine maintainer 2014-11-05 16:53:17 +00:00
Thomas Levine
f57e99be6f we're not maintaining the history, and i don't care 2014-11-05 16:52:13 +00:00
Michael Klement
afc02582fb Fixed typo in read-me. 2014-10-18 10:41:22 -04:00
Michael Klement
2f97421164 Hint added to read-me re standalone invocation of tests that use TEST_SHELL; revision of wording in read-me; argument checking no longer treats a missing dir. argument as an explicitly specified argument. 2014-10-18 10:34:30 -04:00
Michael Klement
1e9611e732 Support for cross-shell testing added, via option -s <shell> and env. variable TEST_SHELL.
* For **tests that _source_ shell scripts**: **option `-s <shell>`** now tells urchin to invoke test scripts with the specified shell (only shebang-less and `#!/bin/sh` tests scripts).
* For **tests that _invoke_ schell scripts**: instruct users to write their tests to always **invoke via environment variable `TEST_SHELL` (e.g., `$TEST_SHELL ../foo`)**, and invoke urchin with that variable defined as needed, e.g., `TEST_SHELL=ksh urchin ./tests`; urchin defaults `TEST_SHELL` to `/bin/sh`.

See updated `readme.md` for details.
2014-10-17 17:16:12 -04:00
Michael Klement
deb77cb5de Various bug fixes and cleanup.
* bug fixes:
  * fixed directory-existence argument check
  * fixed how `.urchin_stdout` files are cleaned up
  * fixed instances of error/warning messages outputting to stdout instead of stderr
* documentation, help and error-message improvements:
  * added specific error message if the (non-option) argument is not a directory
  * improved wording of CLI help
  * readme.md: replaced obsolete URL http://www.urchin.sh with https://github.com/scraperwiki/urchin
  * readme.md: made the fact clearer that `./cross-shell-tests` only tests urchin's _own_ cross-shell compatibility
  * HISTORY: fixed typo
* formatting and logging improvements:
  * added timestamp to the beginning of log files (`.urchin.log`)
  * captured output from failed tests is now printed in *red* to draw attention
  * test summary now prints the number of failed tests in the appropriate color for instant feedback (green, if none failed; red, otherwise)
* cleanup
  * removed unused test files, simplified some tests
2014-10-16 16:37:51 -04:00
Tom Mortimer-Jones
f12636e029 Merge pull request #6 from scraperwiki/exit-code
Set the exit status to reflect pass/fail.
2014-03-17 14:04:22 +00:00
David Jones
0587a615f3 Document the non zero exit code case. 2014-03-17 14:00:30 +00:00
David Jones
b95b010c0d Set the exit status to reflect pass/fail.
For Travis compatibility, we set the exit code to non-zero when
any tests fail.

Fixes #5.
2014-03-17 13:43:13 +00:00
David Jones
98c23a2ab8 Merge pull request #4 from tlevine/link-blog-post
Link to the blog post
2013-10-14 01:20:38 -07:00
Thomas Levine
fc711672d6 alternatives 2013-10-13 16:48:42 +00:00
36 changed files with 442 additions and 106 deletions

1
.gitignore vendored
View File

@@ -1 +0,0 @@
.urchin_stdout

33
HISTORY
View File

@@ -1,18 +1,31 @@
History
======
HISTORY
-------
Version 0.0.5
---------------------
* urchin now unsets `CDPATH`.
* The documentation for `urchin -x` was removed because it was confusing.
Not done
------
We made the colors brighter.
Version 0.0.4
---------------------
* Switch urchin -x to urchin -sh and fix some problems with it
* Documentation
We allowed for multiple tests to be written in a file
Version 0.0.3
---------------------
General tidying
We added flags for verbosity and for Molly guards
Run with different shells in three ways
Done
-----
* urchin -s
* $TEST_SHELL variable with $TEST_SHELL
* $TEST_SHELL variable with urchin -sh
We adjusted the input parameters so it is harder to accidentally run all executible files in your home directory.
Set NULL_GLOB so zsh doesn't print a warning.
Before version 0.0.3
----------------------
We adjusted the input parameters so it is harder to accidentally run all executable files in you
r home directory.
We added directory-based indents.

2
alternatives Normal file
View File

@@ -0,0 +1,2 @@
Totally different syntax and similar features, plus TAP output
https://github.com/sstephenson/bats

View File

@@ -6,7 +6,7 @@ for shell in dash bash ksh zsh; do
if which $shell > /dev/null 2> /dev/null; then
echo
echo Running urchin tests in $shell
$shell urchin tests | tail -n 2
$shell urchin -s $shell tests | tail -n 3
else
echo
echo Skipping $shell because it is not in the PATH

View File

@@ -1,26 +1,32 @@
{
"name": "urchin",
"version": "0.0.2",
"version": "0.0.5",
"description": "Test framework for shell",
"main": "urchin",
"directories": {
"test": "tests"
},
"scripts": {
"test": "./urchin tests"
"test": "export PATH=\"$PWD:$PATH\" && urchin tests"
},
"bin": "./urchin",
"repository": {
"type": "git",
"url": "git://github.com/scraperwiki/urchin.git"
"url": "git://github.com/tlevine/urchin.git"
},
"keywords": [
"shell",
"test",
"testing"
],
"author": "ScraperWiki Limited <feedback@scraperwiki.com>",
"license": "BSD",
"readmeFilename": "readme.md",
"gitHead": "015052dcfd7152a0df037bef1bc5452e0c956695"
"author": "Thomas Levine <_@thomaslevine.com>",
"contributors": [
{"name": "Thomas Levine", "email": "_@thomaslevine.com"},
{"name": "David Jones", "email": "drj@pobox.com"},
{"name": "Francis Irving", "email": "francis@flourish.org"},
{"name": "Zarino Zappia", "email": "mail@zarino.co.uk"},
{"name": "Tom Mortimer-Jones", "email": "tom@morty.co.uk"},
{"name": "Michael Klement", "email": "mklement0@gmail.com"}
],
"license": "BSD"
}

103
readme.md
View File

@@ -12,16 +12,16 @@ other Unix platforms.
Urchin's tests are written in Urchin, so you can run them to see what Urchin
is like. Clone the repository
git clone git://github.com/scraperwiki/urchin.git
git clone git://github.com/tlevine/urchin.git
Run the tests
cd urchin
./urchin tests
The above command will run the tests in your systems default
The above command will run the tests in your system's default
shell, /bin/sh (on recent Ubuntu this is dash, but it could be
ksh or bash on other systems); to test cross-shell compatibility,
ksh or bash on other systems); to test urchin's cross-shell compatibility,
run this:
cd urchin
@@ -31,7 +31,7 @@ run this:
Download Urchin like so (as root) (or use npm, below):
cd /usr/local/bin
wget https://raw.github.com/scraperwiki/urchin/master/urchin
wget https://raw.github.com/tlevine/urchin/master/urchin
chmod +x urchin
Can be installed with npm too:
@@ -42,6 +42,8 @@ Now you can run it.
urchin <test directory>
Run `urchin -h` to get command-line help.
## Writing tests
Make a root directory for your tests. Inside it, put executable files that
exit `0` on success and something else on fail. Non-executable files and hidden
@@ -83,3 +85,96 @@ within the particular directory, and the `teardown` file is run right after.
Files are only run if they are executable, and files beginning with `.` are
ignored. Thus, fixtures and libraries can be included sloppily within the test
directory tree. The test passes if the file exits 0; otherwise, it fails.
In case you care about the order in which your tests execute, consider that
urchin looks for files within a directory in the following manner.
for file in *; do
do_something_with_test_file $file
done
Tests within a directory are executed in whatever order `*` returns.
### Writing cross-shell compatibility tests for testing shell code
While you could write your test scripts to explicitly invoke the functionality
to test with various shells, Urchin facilitates a more flexible approach.
The specific approach depends on your test scenario:
* (a) Your test scripts _invoke_ scripts containing portable shell code.
* (b) Your scripts _source_ scripts containing portable shell code.
#### (a) Cross-shell tests with test scripts that _invoke_ shell scripts
First, consider using [shall](https://github.com/mklement0/shall).
#!/usr/bin/env shall
echo This is a test file.
Alternatively, you can use urchin's built-in recognition of the
`TEST_SHELL` environment variable.
In your test scripts, invoke the shell scripts to test via the shell
specified in environment variable `TEST_SHELL` rather than directly;
e.g.: `$TEST_SHELL ../foo bar` (rather than just `../foo bar`).
Note that if you alsow want your test scripts to work when run directly,
outside of Urchin, be sure to target scripts that happen to be in the
current directory with prefix `./`; e.g., `$TEST_SHELL ./baz`
(rather than `$TEST_SHELL baz`).
Then, on invocation of Urchin, prepend a definition of environment variable
`TEST_SHELL` specifying the shell to test with, e.g.: `TEST_SHELL=zsh urchin ./tests`.
To test with multiple shells in sequence, use something like:
for shell in sh bash ksh zsh; do
TEST_SHELL=$shell urchin ./tests
done
If `TEST_SHELL` has no value, Urchin defines it as `/bin/sh`, so the test
scripts can rely on `$TEST_SHELL` always containing a value.
#### (b) Cross-shell tests with test scripts that _source_ shell scripts
If you _source_ shell code in your test scripts, it is the test scripts
themselves that must be run with the shell specified.
To that end, Urchin supports the `-s <shell>` option, which instructs
Urchin to invoke the test scripts with the specified shell; e.g., `-s bash`.
(In addition, Urchin sets environment variable `TEST_SHELL` to the specified
shell.)
Note that only test scripts that either have no shebang line at all or
have shebang line `#!/bin/sh` are invoked with the specified shell.
This allows non-shell test scripts or test scripts for specific
shells to coexist with those whose invocation should be controlled by `-s`.
To test with multiple shells in sequence, use something like:
for shell in sh bash ksh zsh; do
urchin -s $shell ./tests
done
<!--
#### (c) Cross shell tests with `urchin -x` (experimental)
If you run urchin with the `-x` flag, it will be as if you ran
`$TEST_SHELL`. Unless `$TEST_SHELL` isn't set, in which case it'll
be as if you ran `/bin/sh`. Putting this in she shebang line might
eventually work out to be a cleaner way of doing cross-shell testing.
#!/usr/bin/env urchin -x
test a = a
It might make sense if you do this.
export TEST_SHELL=zsh && urchin -x
export TEST_SHELL=bash && urchin -x
-->
## Alternatives to Urchin
Alternatives to Urchin are discussed in
[this blog post](https://blog.scraperwiki.com/2012/12/how-to-test-shell-scripts/).
## Ideas for new features
* Support [Nagios plugins](https://nagios-plugins.org/doc/guidelines.html)
* Stop running if a test fails so one can use Urchin as a
[setup framework](https://github.com/tlevine/urchin/issues/16).

1
tests/.gitignore vendored
View File

@@ -1 +1,2 @@
.urchin.log
.urchin_stdout

1
tests/.print-arg-3 Normal file
View File

@@ -0,0 +1 @@
echo $3

View File

@@ -0,0 +1,6 @@
#!/bin/sh
cd ..
export CDPATH=$PWD
./urchin -f 'tests/urchin exit code' >/dev/null

View File

@@ -0,0 +1,3 @@
#!/bin/sh
../../urchin -h | grep -- -s

View File

@@ -1,3 +0,0 @@
#!/bin/sh
../../urchin -h|grep -i xsd

View File

@@ -0,0 +1,7 @@
#!/bin/sh
# Assuming that urchin was invoked with `TEST_SHELL=bash urchin ...`, $TEST_SHELL should contain 'bash'.
echo "\$TEST_SHELL: $TEST_SHELL"
[ "$TEST_SHELL" = 'bash' ]

View File

@@ -0,0 +1,4 @@
#!/bin/sh
# Invoke a simple test command with $TEST_SHELL as the executable.
[ "$($TEST_SHELL -c 'echo $0')" = "$TEST_SHELL" ]

View File

@@ -0,0 +1,3 @@
#!/usr/bin/env true
true will processed the contents of this script, but that
means that nothing will happen and the script will exit 0

View File

@@ -0,0 +1,10 @@
#!/bin/sh
# Assuming that urchin was invoked with `-s bash`, this script should be being run with bash.
this_shell=$(ps -o comm= -p $$ && :)
echo "Running shell: $this_shell"
[ "$this_shell" = 'bash' ]

View File

@@ -0,0 +1,10 @@
# By design, this file has no shebang line.
# Assuming that urchin was invoked with `-s bash`, this script should be being run with bash.
this_shell=$(ps -o comm= -p $$ && :)
echo "Running shell: $this_shell"
[ "$this_shell" = 'bash' ]

View File

@@ -0,0 +1,9 @@
#!/bin/sh
# Assuming that urchin was invoked with `-s bash`, $TEST_SHELL should contain 'bash'.
echo "Running shell: $(ps -o comm= -p $$ && :)"
echo "\$TEST_SHELL: $TEST_SHELL"
[ "$TEST_SHELL" = 'bash' ]

View File

@@ -0,0 +1,9 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
which bash 2>/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 1; }
# Test if $TEST_SHELL, when placed in urchin's environment, is passed through to the test scripts.
TEST_SHELL=bash ../../urchin ./.test-TEST_SHELL-passed-through

View File

@@ -0,0 +1,8 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
# Test if $TEST_SHELL - if *defined, but empty* - is exported with value '/bin/sh' by urchin
# and thus has that value inside the scripts.
TEST_SHELL= ../../urchin ./.test-TEST_SHELL-undefined_or_empty

View File

@@ -0,0 +1,9 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
# Test if $TEST_SHELL - if *undefined* - is exported with value '/bin/sh' by urchin
# and thus has that value inside test scripts.
unset -v TEST_SHELL
../../urchin ./.test-TEST_SHELL-undefined_or_empty

View File

@@ -0,0 +1,7 @@
#!/bin/sh
# Tests the `-s <shell> option, which invokes shebang-less and sh-shebang-line test scripts with the specified shell (for testing *sourced* shell code).
which bash >/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 2; }
../../urchin -s bash ./.test-run-by-specified-shell

View File

@@ -0,0 +1,10 @@
# Begin - .testsuite/
not ok 1 - a
# ------------ Begin output ------------
# This is stdout from a.
# ------------ End output ------------
ok 2 - b
ok 3 - # SKIP c
# End - .testsuite/
# Took 0 seconds.
1..3

4
tests/TAP/.testsuite/a Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/sh
echo This is stderr from a. > /dev/stderr
echo This is stdout from a. > /dev/stdout
false

4
tests/TAP/.testsuite/b Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/sh
echo This is stderr from b. > /dev/stderr
echo This is stdout from b. > /dev/stdout
true

1
tests/TAP/.testsuite/c Normal file
View File

@@ -0,0 +1 @@
This should not be run.

View File

@@ -0,0 +1,4 @@
tmp=$(mktemp)
../../urchin -t .testsuite/ | sed 1d > $tmp
diff $tmp .expected-output

View File

@@ -1,3 +0,0 @@
#!/bin/sh
exit 1

View File

@@ -1,3 +0,0 @@
#!/bin/sh
exit

View File

@@ -1,3 +0,0 @@
#!/bin/sh
../../urchin .fixture | grep 'http://www.w3.org/2001/XMLSchema'

View File

@@ -1,3 +0,0 @@
#!/bin/sh
export variable_set_in_setup='value'

View File

@@ -0,0 +1,13 @@
#!/bin/sh
# Borrow this test which has a failure in it (3 in fact), and
# run it using urchin:
../../urchin "../Counts should be kept of successes and failures./.test" > /dev/null
# Now invert the result, we want _this_ test to fail if the exit
# code was 0.
case $? in
0) exit 8;;
*) exit 0;;
esac

View File

@@ -0,0 +1 @@
../../urchin .test > /dev/null

245
urchin
View File

@@ -1,5 +1,9 @@
#!/bin/sh
# Make sure that CDPATH isn't set, as it causes `cd` to behave unpredictably - notably, it can produce output,
# which breaks fullpath().
unset CDPATH
fullpath() {
(
cd -- "$1"
@@ -15,75 +19,146 @@ indent() {
recurse() {
potential_test="$1"
indent_level="$2"
shell_for_sh_tests="$3"
[ "$potential_test" = 'setup_dir' ] && return
[ "$potential_test" = 'teardown_dir' ] && return
[ "$potential_test" = 'setup' ] && return
[ "$potential_test" = 'teardown' ] && return
echo > "$stdout_file"
[ $indent_level -eq 0 ] && : > "$stdout_file"
if [ -d "$potential_test" ]
then
(
if $tap_format; then
indent $indent_level | sed 's/ /#/g'
echo "# Begin - ${potential_test}"
else
indent $indent_level
echo " ${potential_test}"
echo "+ ${potential_test}"
fi
(
cd -- "$potential_test"
[ -f setup_dir ] && [ -x setup_dir ] && ./setup_dir >> "$stdout_file"
if [ -n "$ZSH_VERSION" ]; then
# avoid "no matches found: *" error when directories are empty
setopt NULL_GLOB
fi
for test in *
do
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# $2 instead of $indent_level so it doesn't clash
recurse "${test}" $(( $2 + 1 ))
recurse "${test}" $(( $2 + 1 )) "$shell_for_sh_tests"
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
done
[ -f teardown_dir ] && [ -x teardown_dir ] && ./teardown_dir >> "$stdout_file"
echo
)
elif [ -x "$potential_test" ]
then
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# Run the test
./"$potential_test" > "$stdout_file" 2>&1
exit_code="$?"
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
indent $indent_level
if [ "$exit_code" = '0' ]
then
# On success, print a '✓'
printf '\033[32m✓ \033[0m'
printf '%s\n' "${potential_test}"
printf '%s\n' "${potential_test} passed" >> "$logfile"
if $tap_format; then
indent $indent_level | sed 's/ /#/g'
echo "# End - ${potential_test}"
else
# On fail, print a red '✗'
printf '\033[31m✗ \033[0m'
printf '%s\n' "${potential_test}"
printf '%s\n' "${potential_test} failed" >> "$logfile"
cat "$stdout_file"
echo
fi
else
if [ -x "$potential_test" ]
then
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# Run the test
if [ -n "$shell_for_sh_tests" ] && has_sh_or_no_shebang_line ./"$potential_test"
then
TEST_SHELL="$TEST_SHELL" "$shell_for_sh_tests" ./"$potential_test" > "$stdout_file" 2>&1
else
TEST_SHELL="$TEST_SHELL" ./"$potential_test" > "$stdout_file" 2>&1
fi
exit_code="$?"
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
if [ $exit_code -eq 0 ]; then
result=success
else
result=fail
fi
else
result=skip
fi
echo "${result}" >> "$logfile"
if $tap_format; then
n=$(grep -ce '^\(success\|fail\|skip\)' "$logfile")
if [ "$result" == fail ]; then
not='not '
else
not=''
fi
if [ "$result" == skip ]; then
skip='# SKIP '
else
skip=''
fi
echo "${not}ok $n - ${skip}${potential_test}"
if [ "$result" == fail ]; then
echo '# ------------ Begin output ------------'
sed 's/^/# /' "$stdout_file"
echo '# ------------ End output ------------'
fi
else
indent $indent_level
case "$result" in
success)
# On success, print a green '✓'
printf '\033[32m✓ \033[0m'
printf '%s\n' "${potential_test}"
;;
fail)
# On fail, print a red '✗'
printf '\033[31m✗ \033[0m'
printf '%s\n' "${potential_test}"
printf '\033[31m' # Print output captured from failed test in red.
cat "$stdout_file"
printf '\033[0m'
;;
skip)
printf ' %s\n' "${potential_test}"
;;
esac
fi
rm "$stdout_file"
fi
[ $indent_level -eq 0 ] && rm "$stdout_file"
}
USAGE="usage: $0 <test directory>"
has_sh_or_no_shebang_line() {
head -n 1 "$1" | grep -vqE '^#!' && return 0 # no shebang line at all
head -n 1 "$1" | grep -qE '^#![[:blank:]]*/bin/sh($|[[:blank:]])' && return 0 # shebang line is '#!/bin/sh' or legal variations thereof
return 1
}
USAGE="usage: $0 [<options>] <test directory>"
urchin_help() {
echo
echo "$USAGE"
echo
echo '-f Force urchin to run on directories not named "test".'
echo '-h This help'
echo
echo '--xsd Output xUnit XML schema for an integration server.'
echo
echo 'Go to http://www.urchin.sh for documentation on writing tests.'
echo
cat <<EOF
$USAGE
-s <shell> Invoke test scripts that either have no shebang line at all or
have shebang line "#!/bin/sh" with the specified shell.
-f Force running even if the test directory's name does not
contain the word "test".
-t Format output in Test Anything Protocol (TAP)
-h This help.
Go to https://github.com/tlevine/urchin for documentation on writing tests.
EOF
}
plural () {
@@ -99,61 +174,99 @@ plural () {
}
urchin_go() {
rm -f "$logfile"
if "$tap_format"; then
printf \#\
fi
echo Running tests at $(date +%Y-%m-%dT%H:%M:%S)
start=$(date +%s)
echo > "$logfile"
recurse "$1" 0
# Determine the environment variable to define for test scripts
# that reflects the specified or implied shell to use for shell-code tests.
# - Set it to the shell specified via -s, if any.
# - Otherwise, use its present value, if non-empty.
# - Otherwise, default to '/bin/sh'.
if [ -n "$2" ]
then
TEST_SHELL="$2"
elif [ -z "$TEST_SHELL" ]
then
TEST_SHELL='/bin/sh'
fi
recurse "$1" 0 "$2" # test folder -- indentation level -- [shell to invoke test scripts with]
finish=$(date +%s)
elapsed=$(($finish - $start))
echo Done, took $elapsed $(plural second $elapsed)
set -- $(grep -e 'passed$' "$logfile"|wc -l) $(grep -e 'failed$' "$logfile"|wc -l)
printf '%s\n' "$1 $(plural test "$1") passed."
printf '%s\n' "$2 $(plural test "$2") failed."
passed=$(grep -c '^success' "$logfile")
failed=$(grep -c '^fail' "$logfile")
skipped=$(grep -c '^skip' "$logfile")
if $tap_format; then
echo "# Took $elapsed $(plural second $elapsed)."
echo 1..$(($passed + $failed + $skipped))
else
echo "Done, took $elapsed $(plural second $elapsed)."
printf '%s\n' "$passed $(plural test "$passed") passed."
printf '%s\n' "$skipped $(plural test "$skipped") skipped."
[ $failed -gt 0 ] && printf '\033[31m' || printf '\033[32m' # If tests failed, print the message in red, otherwise in green.
printf '%s\n' "$failed $(plural test "$failed") failed."
printf '\033[m'
fi
rm -f "$logfile"
test -z "$failed" || test "$failed" -eq '0'
}
urchin_molly_guard() {
echo
echo 'The directory on which you are running urchin is not'
echo 'called "test", so I am not running in case that'
echo 'was an accident. Use the -f flag if you really want'
echo 'to run urchin on that directory.'
echo
{
echo
echo 'The name of the directory on which you are running urchin'
echo 'does not contain the word "test", so I am not running,'
echo 'in case that was an accident. Use the -f flag if you really'
echo 'want to run urchin on that directory.'
echo
} >&2
exit 1
}
FORCE=false
shell_for_sh_tests=
force=false
tap_format=false
while [ $# -gt 0 ]
do
case "$1" in
-f) FORCE=true;;
-f) force=true;;
-s)
shift
shell_for_sh_tests=$1
which "$shell_for_sh_tests" >/dev/null || { echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2; urchin_help >&2; exit 2; }
;;
-t) tap_format=true;;
-h|--help) urchin_help
exit 0;;
# --xsd) action=testsuite;;
# --) shift; break;;
-*) urchin_help 1>&2
-*) urchin_help >&2
exit 1;;
*) break;;
esac
shift
done
# Verify argument for main stuff
if [ "$#" != '1' ] || [ ! -d "$1" ]
then
[ -n "$1" ] && [ ! -d "$1" ] && echo "Not a directory: '$1'" >&2
echo "$USAGE" >&2
exit 2
fi
# Constants
logfile=$(fullpath "$1")/.urchin.log
stdout_file=$(fullpath "$1")/.urchin_stdout
# Verify argument for main stuff
if [ "$#" != '1' ] && [ ! -d "$1" ]
then
echo "$USAGE"
exit 1
fi
# Run or present the Molly guard.
if basename "$(fullpath "$1")" | grep test > /dev/null || $FORCE
if basename "$(fullpath "$1")" | grep -Fi 'test' > /dev/null || $force
then
urchin_go "$1"
urchin_go "$1" "$shell_for_sh_tests"
else
urchin_molly_guard
fi