73 Commits

Author SHA1 Message Date
Thomas Levine
7126872a8b put mksh in the cross-shell-tests 2016-01-29 19:45:39 +00:00
Thomas Levine
96ab43c233 explain why urchin is called urchin 2016-01-29 17:28:26 +00:00
Thomas Levine
2be0151213 document the --help flag 2016-01-29 17:20:18 +00:00
Thomas Levine
655daf4b74 show all counts in cross-shell-tests runner
It previously just showed the last three lines. It needs to show the
last four lines now because urchin now displays skipped tests.
2016-01-29 17:19:15 +00:00
Thomas Levine
31023bce21 update HISTORY 2016-01-29 17:17:35 +00:00
Thomas Levine
d96f3707e9 implement -v 2016-01-29 17:17:31 +00:00
Thomas Levine
3f05794d7e add -v to usage 2016-01-29 17:16:01 +00:00
Thomas Levine
d7c8549206 urchin -v test 2016-01-29 17:14:34 +00:00
Thomas Levine
bc74b20c5c add authors 2016-01-28 15:33:43 +00:00
Thomas Levine
8bdd00c3e9 explain the branching 2016-01-28 15:30:59 +00:00
Thomas Levine
42dd0947e8 more changes 2016-01-28 15:28:36 +00:00
Thomas Levine
da6d89aa56 add to change log 2016-01-28 15:27:32 +00:00
Thomas Levine
3ea035d188 rename test to be more clear 2016-01-28 15:24:24 +00:00
Thomas Levine
8e669c40a6 add + indent symbol so directories line up 2016-01-28 15:23:14 +00:00
Thomas Levine
66c010b694 fix fixture to not expect stderr 2016-01-28 15:23:14 +00:00
Thomas Levine
7339e84d7c test stdout stderr for TAP 2016-01-28 15:23:14 +00:00
Thomas Levine
e364d40f3a TAP test 2016-01-28 15:23:14 +00:00
Thomas Levine
13e748d684 remove awk dependency in test suite
I'm on NixOS, so awk isn't in /usr/bin. I can search for it with
/usr/bin/env, but then I can't use -f in the shebang line.
2016-01-28 15:23:14 +00:00
Thomas Levine
bb42d27917 test plan 2016-01-28 15:23:14 +00:00
Thomas Levine
c1ec06c6fe logfile 2016-01-28 15:23:14 +00:00
Thomas Levine
5382946639 error on fail 2016-01-28 15:23:14 +00:00
Thomas Levine
144013c71c print skip count at end 2016-01-28 15:23:13 +00:00
Thomas Levine
009fd7ea05 print stdout for tap and not-tap 2016-01-28 15:23:13 +00:00
Thomas Levine
c3901fe175 tap indentation comments for directories 2016-01-28 15:23:13 +00:00
Thomas Levine
538c8437a7 assorted tap stuff 2016-01-28 15:23:13 +00:00
Thomas Levine
cd124e09ee more tap 2016-01-28 15:23:13 +00:00
Thomas Levine
d963f10be1 more tap 2016-01-28 15:23:13 +00:00
Thomas Levine
0ca55d155b oops 2016-01-28 15:23:13 +00:00
Thomas Levine
8d55550248 test test successes better 2016-01-28 15:23:13 +00:00
Thomas Levine
0f86778d79 convert indents to comments 2016-01-28 15:23:13 +00:00
Thomas Levine
92cb735bb7 start writing non-tap cases 2016-01-28 15:23:13 +00:00
Thomas Levine
c817355e94 not working 2016-01-28 15:23:13 +00:00
Thomas Levine
d4bedbbfcf first pass implementation of exit on fail 2016-01-28 15:23:13 +00:00
Thomas Levine
f6b57772f2 document unimplemented -e flag 2016-01-28 15:23:13 +00:00
Thomas Levine
72fa30e787 Test die on fail. 2016-01-28 15:23:13 +00:00
Thomas Levine
eda4a6e42c call urchin -s in cross-shell tests 2016-01-28 15:23:12 +00:00
Thomas Levine
df80586d79 remove "urchin -x" test 2016-01-28 15:23:12 +00:00
Thomas Levine
d525c1793b remove "urchin -x"
unnecessary now that shall exists
2016-01-28 15:23:12 +00:00
Thomas Levine
56a4e5d8af introduction paragraph 2016-01-25 14:06:33 +00:00
Thomas Levine
d81b7efdc3 asciibetical 2016-01-25 13:56:33 +00:00
Thomas Levine
7f5455739d update readme
* move shall reference to the right place
* further explain cross-shell testing methods
2016-01-25 13:49:36 +00:00
Thomas Levine
48e5090091 start writing alternatives 2015-12-09 07:46:16 -05:00
Thomas Levine
5abf088eb8 add new ideas 2015-10-19 12:52:24 -04:00
Thomas Levine
c84851cb5c document the ordering of tests 2015-08-21 09:03:15 -04:00
Thomas Levine
510476f924 mention shall in the documentation 2015-07-05 17:39:38 +00:00
Thomas Levine
f01869fb97 packaging 2014-12-08 02:08:00 +00:00
Michael Klement
0385dcd86e Fixed blog link in read-me and commented out experimental -x option for now. 2014-12-05 18:50:19 -05:00
Michael Klement
2847b020b4 Make sure that CDPATH isn't set, as it causes cd to behave unpredictably - notably, it can produce output, which breaks fullpath().
Also: Improved CLI help, updated URLs in read-me, cleaned up package.json:

I've tried to clarify the intent of `-x` in the CLI help, but I haven't touched the read-me in that respect.

I don't see any benefit to `-x`:

* Just using `#/bin/sh` as the shebang line in combination with `-s <shell>` gives you the same functionality,
* When it comes to invoking scripts from _within_ test scripts, nothing can do the work for you: you consciously have to mark the invocation with _something_ to indicate that it should be controlled from the outside; it won't get any easier than `$TEST_SHELL ...`
* Finally, using a shebang line such as `#!/usr/bin/env urchin -x` is problematic for two reasons:
  * Some platforms can handle only *1* argument in a shebang line.
  * In a _package-local_ installation, `#!/usr/bin/env` may not find the Urchin executable.

I'm also not sure how the following (from `readme.md`) fits in the picture:

> It might make sense if you do this.

    export TEST_SHELL=zsh && urchin -x
    export TEST_SHELL=bash && urchin -x

(As an aside: To achieve the same thing, you don't need `export`; `TEST_SHELL=zsh urchin -x`  and `TEST_SHELL=bash urchin -x`  is the better choice.)
How does this relate to use in a  _shebang line_?

`urchin_help()` now uses a here-doc: easier to maintain, and should work in all Bourne-like shells.

`readmeFilename` removed from `package.json`:

> "The readmeFilename does not need to ever be in your actual package.json file" - npm/npm#3573
2014-12-03 09:54:15 -05:00
Michael Klement
fae24e926a Improved CLI help, updated URLs in read-me 2014-11-06 22:21:05 -05:00
Thomas Levine
2ab070c353 fix the -x documentation 2014-11-06 00:52:34 +00:00
Thomas Levine
292663dd73 contributors 2014-11-06 00:51:14 +00:00
Thomas Levine
90abc2640d update the -h help text 2014-11-05 17:49:53 +00:00
Thomas Levine
19ed3b02e8 HISTORY 2014-11-05 17:49:08 +00:00
Thomas Levine
b0429315ec urchin -sh to urchin -x 2014-11-05 17:47:36 +00:00
Thomas Levine
5e06ffb1c1 version 2014-11-05 17:47:01 +00:00
Thomas Levine
bf8c404784 fix -sh 2014-11-05 17:46:19 +00:00
Thomas Levine
bb41d5a156 zsh support 2014-11-05 17:38:22 +00:00
Thomas Levine
ed09351df6 fix a test 2014-11-05 17:28:54 +00:00
Thomas Levine
1e2d4cc8a3 remove gitHead 2014-11-05 17:27:26 +00:00
Thomas Levine
f8d921b5e5 document urchin -sh 2014-11-05 17:26:01 +00:00
Thomas Levine
1f115df222 urchin -sh 2014-11-05 17:18:31 +00:00
Thomas Levine
eafed6ac59 put history back 2014-11-05 16:54:36 +00:00
Thomas Levine
8273661440 tlevine maintainer 2014-11-05 16:53:17 +00:00
Thomas Levine
f57e99be6f we're not maintaining the history, and i don't care 2014-11-05 16:52:13 +00:00
Michael Klement
afc02582fb Fixed typo in read-me. 2014-10-18 10:41:22 -04:00
Michael Klement
2f97421164 Hint added to read-me re standalone invocation of tests that use TEST_SHELL; revision of wording in read-me; argument checking no longer treats a missing dir. argument as an explicitly specified argument. 2014-10-18 10:34:30 -04:00
Michael Klement
1e9611e732 Support for cross-shell testing added, via option -s <shell> and env. variable TEST_SHELL.
* For **tests that _source_ shell scripts**: **option `-s <shell>`** now tells urchin to invoke test scripts with the specified shell (only shebang-less and `#!/bin/sh` tests scripts).
* For **tests that _invoke_ schell scripts**: instruct users to write their tests to always **invoke via environment variable `TEST_SHELL` (e.g., `$TEST_SHELL ../foo`)**, and invoke urchin with that variable defined as needed, e.g., `TEST_SHELL=ksh urchin ./tests`; urchin defaults `TEST_SHELL` to `/bin/sh`.

See updated `readme.md` for details.
2014-10-17 17:16:12 -04:00
Michael Klement
deb77cb5de Various bug fixes and cleanup.
* bug fixes:
  * fixed directory-existence argument check
  * fixed how `.urchin_stdout` files are cleaned up
  * fixed instances of error/warning messages outputting to stdout instead of stderr
* documentation, help and error-message improvements:
  * added specific error message if the (non-option) argument is not a directory
  * improved wording of CLI help
  * readme.md: replaced obsolete URL http://www.urchin.sh with https://github.com/scraperwiki/urchin
  * readme.md: made the fact clearer that `./cross-shell-tests` only tests urchin's _own_ cross-shell compatibility
  * HISTORY: fixed typo
* formatting and logging improvements:
  * added timestamp to the beginning of log files (`.urchin.log`)
  * captured output from failed tests is now printed in *red* to draw attention
  * test summary now prints the number of failed tests in the appropriate color for instant feedback (green, if none failed; red, otherwise)
* cleanup
  * removed unused test files, simplified some tests
2014-10-16 16:37:51 -04:00
Tom Mortimer-Jones
f12636e029 Merge pull request #6 from scraperwiki/exit-code
Set the exit status to reflect pass/fail.
2014-03-17 14:04:22 +00:00
David Jones
0587a615f3 Document the non zero exit code case. 2014-03-17 14:00:30 +00:00
David Jones
b95b010c0d Set the exit status to reflect pass/fail.
For Travis compatibility, we set the exit code to non-zero when
any tests fail.

Fixes #5.
2014-03-17 13:43:13 +00:00
David Jones
98c23a2ab8 Merge pull request #4 from tlevine/link-blog-post
Link to the blog post
2013-10-14 01:20:38 -07:00
Thomas Levine
fc711672d6 alternatives 2013-10-13 16:48:42 +00:00
43 changed files with 509 additions and 113 deletions

1
.gitignore vendored
View File

@@ -1 +0,0 @@
.urchin_stdout

9
AUTHORS Normal file
View File

@@ -0,0 +1,9 @@
Authors
-------
David Jones
Michael Klement
Thomas Levine
Maintainer
-------
Thomas Levine <_@thomaslevine.com>

51
HISTORY
View File

@@ -1,18 +1,49 @@
History
======
HISTORY
=======
Version 0.0.6
---------------------
Not done
------
We made the colors brighter.
* Produce TAP output with the -t flag.
* Add a + sign in front of directories in the normal output so that they
line up with non-directories.
* Display skipped tests in the normal output and in the TAP output.
* Correct some things in the documentation.
* Rearrange things in the documentation to be more clear.
* Pass the -e flag to exit urchin if any single test fails.
* Remove the undocumented, experimental -x flag now that shall exists.
* Display version number with the -v flag.
* Document why Urchin is called "Urchin"
We allowed for multiple tests to be written in a file
These changes are made somewhat separately in the branches "exit-on-fail",
"remove-urchin-x", "tap", and "update-readme". They are rebased into one
branch, "tlevine-2016-02", for merging into "master".
We added flags for verbosity and for Molly guards
Version 0.0.5
---------------------
* urchin now unsets `CDPATH`.
* The documentation for `urchin -x` was removed because it was confusing.
Done
-----
Version 0.0.4
---------------------
* Switch urchin -x to urchin -sh and fix some problems with it
* Documentation
We adjusted the input parameters so it is harder to accidentally run all executible files in your home directory.
Version 0.0.3
---------------------
General tidying
Run with different shells in three ways
* urchin -s
* $TEST_SHELL variable with $TEST_SHELL
* $TEST_SHELL variable with urchin -sh
Set NULL_GLOB so zsh doesn't print a warning.
Before version 0.0.3
----------------------
We adjusted the input parameters so it is harder to accidentally run all executable files in you
r home directory.
We added directory-based indents.

2
alternatives Normal file
View File

@@ -0,0 +1,2 @@
Totally different syntax and similar features, plus TAP output
https://github.com/sstephenson/bats

View File

@@ -2,11 +2,11 @@
# Run urchin in a bunch of different shells,
# including a shell that isn't quite POSIX-compatible (zsh)
for shell in dash bash ksh zsh; do
for shell in dash bash mksh ksh zsh; do
if which $shell > /dev/null 2> /dev/null; then
echo
echo Running urchin tests in $shell
$shell urchin tests | tail -n 2
$shell urchin -s $shell tests | tail -n 4
else
echo
echo Skipping $shell because it is not in the PATH

View File

@@ -1,26 +1,32 @@
{
"name": "urchin",
"version": "0.0.2",
"version": "0.0.5",
"description": "Test framework for shell",
"main": "urchin",
"directories": {
"test": "tests"
},
"scripts": {
"test": "./urchin tests"
"test": "export PATH=\"$PWD:$PATH\" && urchin tests"
},
"bin": "./urchin",
"repository": {
"type": "git",
"url": "git://github.com/scraperwiki/urchin.git"
"url": "git://github.com/tlevine/urchin.git"
},
"keywords": [
"shell",
"test",
"testing"
],
"author": "ScraperWiki Limited <feedback@scraperwiki.com>",
"license": "BSD",
"readmeFilename": "readme.md",
"gitHead": "015052dcfd7152a0df037bef1bc5452e0c956695"
"author": "Thomas Levine <_@thomaslevine.com>",
"contributors": [
{"name": "Thomas Levine", "email": "_@thomaslevine.com"},
{"name": "David Jones", "email": "drj@pobox.com"},
{"name": "Francis Irving", "email": "francis@flourish.org"},
{"name": "Zarino Zappia", "email": "mail@zarino.co.uk"},
{"name": "Tom Mortimer-Jones", "email": "tom@morty.co.uk"},
{"name": "Michael Klement", "email": "mklement0@gmail.com"}
],
"license": "BSD"
}

112
readme.md
View File

@@ -4,37 +4,42 @@
/ /_/ / / / /__/ / / / / / / /
\__,_/_/ \___/_/ /_/_/_/ /_/
Urchin is a test framework for shell. It is implemented in
portable /bin/sh and should work on GNU/Linux, Mac OS X, and
other Unix platforms.
Urchin is a file-based test harness, normally used for testing shell programs.
It is written in portable shell and should thus work on GNU/Linux, BSD
(including Mac OS X), and other Unix-like platforms.
Urchin is called "Urchin" because
[sea urchins](https://en.wikipedia.org/wiki/Sea_urchin)
have shells called "tests".
## Try it out
Urchin's tests are written in Urchin, so you can run them to see what Urchin
is like. Clone the repository
git clone git://github.com/scraperwiki/urchin.git
git clone git://github.com/tlevine/urchin.git
Run the tests
cd urchin
./urchin tests
The above command will run the tests in your systems default
The above command will run the tests in your system's default
shell, /bin/sh (on recent Ubuntu this is dash, but it could be
ksh or bash on other systems); to test cross-shell compatibility,
ksh or bash on other systems); to test urchin's cross-shell compatibility,
run this:
cd urchin
./cross-shell-tests
## Globally
Download Urchin like so (as root) (or use npm, below):
## Install
Urchin is contained in a single file, so you can install it by copying it to a
directory in your `PATH`. For example, you can run the following as root.
cd /usr/local/bin
wget https://raw.github.com/scraperwiki/urchin/master/urchin
wget https://raw.github.com/tlevine/urchin/master/urchin
chmod +x urchin
Can be installed with npm too:
Urchin can be installed with npm too.
npm install -g urchin
@@ -42,6 +47,8 @@ Now you can run it.
urchin <test directory>
Run `urchin -h` to get command-line help.
## Writing tests
Make a root directory for your tests. Inside it, put executable files that
exit `0` on success and something else on fail. Non-executable files and hidden
@@ -83,3 +90,88 @@ within the particular directory, and the `teardown` file is run right after.
Files are only run if they are executable, and files beginning with `.` are
ignored. Thus, fixtures and libraries can be included sloppily within the test
directory tree. The test passes if the file exits 0; otherwise, it fails.
Tests files and subdirectories are run in ASCIIbetical order within each
directory; that is,
urchin looks for files within a directory in the following manner.
for file in *; do
do_something_with_test_file $file
done
### Writing cross-shell compatibility tests for testing shell code
While you could write your test scripts to explicitly invoke the functionality
to test with various shells, Urchin facilitates a more flexible approach.
The specific approach depends on your test scenario:
* (a) Your test scripts _invoke_ scripts containing portable shell code.
* (b) Your scripts _source_ scripts containing portable shell code.
#### (a) Cross-shell tests with test scripts that _invoke_ shell scripts
Urchin sets the `TEST_SHELL` environment variable so that you may change the
shell with which your tests call other shell programs. To run your test
scripts in multiple shells you must call `$TEST_SHELL` in your tests and then
run urchin with the appropriate option.
In your test scripts, invoke the shell scripts to test via the shell
specified in environment variable `TEST_SHELL` rather than directly;
e.g.: `$TEST_SHELL ../foo bar` (rather than just `../foo bar`).
On invocation of Urchin, prepend a definition of environment variable
`TEST_SHELL` specifying the shell to test with, e.g.,
TEST_SHELL=zsh urchin ./tests
To test with multiple shells in sequence, use something like:
for shell in sh bash ksh zsh; do
TEST_SHELL=$shell urchin ./tests
done
If `TEST_SHELL` has no value, Urchin defines it as `/bin/sh`, so the test
scripts can rely on `$TEST_SHELL` always containing a value when Urchin runs
them.
That said, we still recommand that you account for the possibility that
`$TEST_SHELL` does not contain a value so that you may run your test scripts
without Urchin. Supporting this case is very simple; when you invoke scripts
that happen to be in the current directory, be sure to use the prefix `./`,
e.g., `$TEST_SHELL ./baz` rather than `$TEST_SHELL baz`.
#### (b) Cross-shell tests with test scripts that _source_ shell scripts
If you _source_ shell code in your test scripts, it is the test scripts
themselves that must be run with the shell specified.
Urchin supports the `-s <shell>` option, which instructs
Urchin to invoke the test scripts with the specified shell; e.g., `-s bash`.
(In addition, Urchin sets environment variable `TEST_SHELL` to the specified
shell.)
Note that only test scripts that either have no shebang line at all or
have shebang line `#!/bin/sh` are invoked with the specified shell.
This allows non-shell test scripts or test scripts for specific
shells to coexist with those whose invocation should be controlled by `-s`.
To test with multiple shells in sequence, use something like:
for shell in sh bash ksh zsh; do
urchin -s $shell ./tests
done
Also consider using [shall](https://github.com/mklement0/shall).
It does something similar, but the interface may be more intuitive.
#!/usr/bin/env shall
echo This is a test file.
## Alternatives to Urchin
Alternatives to Urchin are discussed in
[this blog post](https://blog.scraperwiki.com/2012/12/how-to-test-shell-scripts/).
## Ideas for new features
* Support [Nagios plugins](https://nagios-plugins.org/doc/guidelines.html)
* Stop running if a test fails so one can use Urchin as a
[setup framework](https://github.com/tlevine/urchin/issues/16).

View File

@@ -0,0 +1 @@
false

View File

@@ -0,0 +1 @@
false

View File

@@ -0,0 +1 @@
false

1
tests/.gitignore vendored
View File

@@ -1 +1,2 @@
.urchin.log
.urchin_stdout

1
tests/.print-arg-3 Normal file
View File

@@ -0,0 +1 @@
echo $3

View File

@@ -0,0 +1,6 @@
#!/bin/sh
cd ..
export CDPATH=$PWD
./urchin -f 'tests/urchin exit code' >/dev/null

View File

@@ -0,0 +1,3 @@
#!/bin/sh
../../urchin -h | grep -- -s

View File

@@ -1,3 +0,0 @@
#!/bin/sh
../../urchin -h|grep -i xsd

View File

@@ -0,0 +1,7 @@
#!/bin/sh
# Assuming that urchin was invoked with `TEST_SHELL=bash urchin ...`, $TEST_SHELL should contain 'bash'.
echo "\$TEST_SHELL: $TEST_SHELL"
[ "$TEST_SHELL" = 'bash' ]

View File

@@ -0,0 +1,4 @@
#!/bin/sh
# Invoke a simple test command with $TEST_SHELL as the executable.
[ "$($TEST_SHELL -c 'echo $0')" = "$TEST_SHELL" ]

View File

@@ -0,0 +1,3 @@
#!/usr/bin/env true
true will processed the contents of this script, but that
means that nothing will happen and the script will exit 0

View File

@@ -0,0 +1,10 @@
#!/bin/sh
# Assuming that urchin was invoked with `-s bash`, this script should be being run with bash.
this_shell=$(ps -o comm= -p $$ && :)
echo "Running shell: $this_shell"
[ "$this_shell" = 'bash' ]

View File

@@ -0,0 +1,10 @@
# By design, this file has no shebang line.
# Assuming that urchin was invoked with `-s bash`, this script should be being run with bash.
this_shell=$(ps -o comm= -p $$ && :)
echo "Running shell: $this_shell"
[ "$this_shell" = 'bash' ]

View File

@@ -0,0 +1,9 @@
#!/bin/sh
# Assuming that urchin was invoked with `-s bash`, $TEST_SHELL should contain 'bash'.
echo "Running shell: $(ps -o comm= -p $$ && :)"
echo "\$TEST_SHELL: $TEST_SHELL"
[ "$TEST_SHELL" = 'bash' ]

View File

@@ -0,0 +1,9 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
which bash 2>/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 1; }
# Test if $TEST_SHELL, when placed in urchin's environment, is passed through to the test scripts.
TEST_SHELL=bash ../../urchin ./.test-TEST_SHELL-passed-through

View File

@@ -0,0 +1,8 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
# Test if $TEST_SHELL - if *defined, but empty* - is exported with value '/bin/sh' by urchin
# and thus has that value inside the scripts.
TEST_SHELL= ../../urchin ./.test-TEST_SHELL-undefined_or_empty

View File

@@ -0,0 +1,9 @@
#!/bin/sh
# Tests support for either passing through or defining a default value for environment variable TEST_SHELL.
# (for test scripts that want to invoke shell scripts with a specified shell).
# Test if $TEST_SHELL - if *undefined* - is exported with value '/bin/sh' by urchin
# and thus has that value inside test scripts.
unset -v TEST_SHELL
../../urchin ./.test-TEST_SHELL-undefined_or_empty

View File

@@ -0,0 +1,7 @@
#!/bin/sh
# Tests the `-s <shell> option, which invokes shebang-less and sh-shebang-line test scripts with the specified shell (for testing *sourced* shell code).
which bash >/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 2; }
../../urchin -s bash ./.test-run-by-specified-shell

View File

@@ -0,0 +1,11 @@
tmp=$(mktemp)
../urchin -e -f ./.die-on-fail > $tmp
result=$?
grep '1 should run.' $tmp
grep '2 should run.' $tmp
grep -v '3 should not run.' $tmp
grep -v '4 should not run.' $tmp
rm $tmp
exit $result

2
tests/Print version on -v. Executable file
View File

@@ -0,0 +1,2 @@
#!/bin/sh
../urchin -v | grep '[0-9.]\{3,\}'

View File

@@ -0,0 +1,10 @@
# Begin - .testsuite/
not ok 1 - a
# ------------ Begin output ------------
# This is stdout from a.
# ------------ End output ------------
ok 2 - b
ok 3 - # SKIP c
# End - .testsuite/
# Took 0 seconds.
1..3

4
tests/TAP/.testsuite/a Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/sh
echo This is stderr from a. > /dev/stderr
echo This is stdout from a. > /dev/stdout
false

4
tests/TAP/.testsuite/b Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/sh
echo This is stderr from b. > /dev/stderr
echo This is stdout from b. > /dev/stdout
true

1
tests/TAP/.testsuite/c Normal file
View File

@@ -0,0 +1 @@
This should not be run.

View File

@@ -0,0 +1,4 @@
tmp=$(mktemp)
../../urchin -t .testsuite/ | sed 1d > $tmp
diff $tmp .expected-output

View File

@@ -1,3 +0,0 @@
#!/bin/sh
exit 1

View File

@@ -1,3 +0,0 @@
#!/bin/sh
exit

View File

@@ -1,3 +0,0 @@
#!/bin/sh
../../urchin .fixture | grep 'http://www.w3.org/2001/XMLSchema'

View File

@@ -1,3 +0,0 @@
#!/bin/sh
export variable_set_in_setup='value'

View File

View File

@@ -0,0 +1,13 @@
#!/bin/sh
# Borrow this test which has a failure in it (3 in fact), and
# run it using urchin:
../../urchin "../Counts should be kept of successes and failures./.test" > /dev/null
# Now invert the result, we want _this_ test to fail if the exit
# code was 0.
case $? in
0) exit 8;;
*) exit 0;;
esac

View File

@@ -0,0 +1 @@
../../urchin .test > /dev/null

265
urchin
View File

@@ -1,5 +1,12 @@
#!/bin/sh
# Make sure that CDPATH isn't set, as it causes `cd` to behave unpredictably - notably, it can produce output,
# which breaks fullpath().
unset CDPATH
# Urchin version number
VERSION=0.0.6
fullpath() {
(
cd -- "$1"
@@ -15,75 +22,159 @@ indent() {
recurse() {
potential_test="$1"
indent_level="$2"
shell_for_sh_tests="$3"
[ "$potential_test" = 'setup_dir' ] && return
[ "$potential_test" = 'teardown_dir' ] && return
[ "$potential_test" = 'setup' ] && return
[ "$potential_test" = 'teardown' ] && return
echo > "$stdout_file"
[ $indent_level -eq 0 ] && : > "$stdout_file"
if [ -d "$potential_test" ]
then
(
if $tap_format; then
indent $indent_level | sed 's/ /#/g'
echo "# Begin - ${potential_test}"
else
indent $indent_level
echo " ${potential_test}"
echo "+ ${potential_test}"
fi
(
cd -- "$potential_test"
[ -f setup_dir ] && [ -x setup_dir ] && ./setup_dir >> "$stdout_file"
if [ -n "$ZSH_VERSION" ]; then
# avoid "no matches found: *" error when directories are empty
setopt NULL_GLOB
fi
for test in *
do
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# $2 instead of $indent_level so it doesn't clash
recurse "${test}" $(( $2 + 1 ))
recurse "${test}" $(( $2 + 1 )) "$shell_for_sh_tests"
exit_code=$?
if $exit_on_fail && test $exit_code -ne 0; then
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
[ -f teardown_dir ] && [ -x teardown_dir ] && ./teardown_dir >> "$stdout_file"
return 1
fi
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
done
[ -f teardown_dir ] && [ -x teardown_dir ] && ./teardown_dir >> "$stdout_file"
echo
)
elif [ -x "$potential_test" ]
then
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# Run the test
./"$potential_test" > "$stdout_file" 2>&1
exit_code="$?"
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
indent $indent_level
if [ "$exit_code" = '0' ]
then
# On success, print a '✓'
printf '\033[32m✓ \033[0m'
printf '%s\n' "${potential_test}"
printf '%s\n' "${potential_test} passed" >> "$logfile"
if $tap_format; then
indent $indent_level | sed 's/ /#/g'
echo "# End - ${potential_test}"
else
# On fail, print a red '✗'
printf '\033[31m✗ \033[0m'
printf '%s\n' "${potential_test}"
printf '%s\n' "${potential_test} failed" >> "$logfile"
cat "$stdout_file"
echo
fi
else
if [ -x "$potential_test" ]
then
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# Run the test
if [ -n "$shell_for_sh_tests" ] && has_sh_or_no_shebang_line ./"$potential_test"
then
TEST_SHELL="$TEST_SHELL" "$shell_for_sh_tests" ./"$potential_test" > "$stdout_file" 2>&1
else
TEST_SHELL="$TEST_SHELL" ./"$potential_test" > "$stdout_file" 2>&1
fi
exit_code="$?"
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
if [ $exit_code -eq 0 ]; then
result=success
else
result=fail
fi
else
result=skip
fi
echo "${result}" >> "$logfile"
if $tap_format; then
n=$(grep -ce '^\(success\|fail\|skip\)' "$logfile")
if [ "$result" == fail ]; then
not='not '
else
not=''
fi
if [ "$result" == skip ]; then
skip='# SKIP '
else
skip=''
fi
echo "${not}ok $n - ${skip}${potential_test}"
if [ "$result" == fail ]; then
echo '# ------------ Begin output ------------'
sed 's/^/# /' "$stdout_file"
echo '# ------------ End output ------------'
fi
else
indent $indent_level
case "$result" in
success)
# On success, print a green '✓'
printf '\033[32m✓ \033[0m'
printf '%s\n' "${potential_test}"
;;
fail)
# On fail, print a red '✗'
printf '\033[31m✗ \033[0m'
printf '%s\n' "${potential_test}"
printf '\033[31m' # Print output captured from failed test in red.
cat "$stdout_file"
printf '\033[0m'
;;
skip)
printf ' %s\n' "${potential_test}"
;;
esac
fi
if $exit_on_fail && test 0 -ne $exit_code; then
return 1
fi
rm "$stdout_file"
fi
[ $indent_level -eq 0 ] && rm "$stdout_file"
}
USAGE="usage: $0 <test directory>"
has_sh_or_no_shebang_line() {
head -n 1 "$1" | grep -vqE '^#!' && return 0 # no shebang line at all
head -n 1 "$1" | grep -qE '^#![[:blank:]]*/bin/sh($|[[:blank:]])' && return 0 # shebang line is '#!/bin/sh' or legal variations thereof
return 1
}
USAGE="usage: $0 [<options>] <test directory>"
urchin_help() {
echo
echo "$USAGE"
echo
echo '-f Force urchin to run on directories not named "test".'
echo '-h This help'
echo
echo '--xsd Output xUnit XML schema for an integration server.'
echo
echo 'Go to http://www.urchin.sh for documentation on writing tests.'
echo
cat <<EOF
$USAGE
-s <shell> Invoke test scripts that either have no shebang line at all or
have shebang line "#!/bin/sh" with the specified shell.
-e Stop running if any single test fails. This is helpful if you want
to use Urchin to run things other than tests, such as a set of
configuration scripts.
-f Force running even if the test directory's name does not
contain the word "test".
-t Format output in Test Anything Protocol (TAP)
-h, --help This help.
-v Display the version number.
Go to https://github.com/tlevine/urchin for documentation on writing tests.
EOF
}
plural () {
@@ -99,61 +190,103 @@ plural () {
}
urchin_go() {
rm -f "$logfile"
if "$tap_format"; then
printf \#\
fi
echo Running tests at $(date +%Y-%m-%dT%H:%M:%S)
start=$(date +%s)
echo > "$logfile"
recurse "$1" 0
# Determine the environment variable to define for test scripts
# that reflects the specified or implied shell to use for shell-code tests.
# - Set it to the shell specified via -s, if any.
# - Otherwise, use its present value, if non-empty.
# - Otherwise, default to '/bin/sh'.
if [ -n "$2" ]
then
TEST_SHELL="$2"
elif [ -z "$TEST_SHELL" ]
then
TEST_SHELL='/bin/sh'
fi
recurse "$1" 0 "$2" # test folder -- indentation level -- [shell to invoke test scripts with]
finish=$(date +%s)
elapsed=$(($finish - $start))
echo Done, took $elapsed $(plural second $elapsed)
set -- $(grep -e 'passed$' "$logfile"|wc -l) $(grep -e 'failed$' "$logfile"|wc -l)
printf '%s\n' "$1 $(plural test "$1") passed."
printf '%s\n' "$2 $(plural test "$2") failed."
passed=$(grep -c '^success' "$logfile")
failed=$(grep -c '^fail' "$logfile")
skipped=$(grep -c '^skip' "$logfile")
if $tap_format; then
echo "# Took $elapsed $(plural second $elapsed)."
echo 1..$(($passed + $failed + $skipped))
else
echo "Done, took $elapsed $(plural second $elapsed)."
printf '%s\n' "$passed $(plural test "$passed") passed."
printf '%s\n' "$skipped $(plural test "$skipped") skipped."
[ $failed -gt 0 ] && printf '\033[31m' || printf '\033[32m' # If tests failed, print the message in red, otherwise in green.
printf '%s\n' "$failed $(plural test "$failed") failed."
printf '\033[m'
fi
rm -f "$logfile"
test -z "$failed" || test "$failed" -eq '0'
}
urchin_molly_guard() {
echo
echo 'The directory on which you are running urchin is not'
echo 'called "test", so I am not running in case that'
echo 'was an accident. Use the -f flag if you really want'
echo 'to run urchin on that directory.'
echo
{
echo
echo 'The name of the directory on which you are running urchin'
echo 'does not contain the word "test", so I am not running,'
echo 'in case that was an accident. Use the -f flag if you really'
echo 'want to run urchin on that directory.'
echo
} >&2
exit 1
}
FORCE=false
shell_for_sh_tests=
force=false
exit_on_fail=false
tap_format=false
while [ $# -gt 0 ]
do
case "$1" in
-f) FORCE=true;;
-e) exit_on_fail=true;;
-f) force=true;;
-s)
shift
shell_for_sh_tests=$1
which "$shell_for_sh_tests" >/dev/null || { echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2; urchin_help >&2; exit 2; }
;;
-t) tap_format=true;;
-h|--help) urchin_help
exit 0;;
# --xsd) action=testsuite;;
# --) shift; break;;
-*) urchin_help 1>&2
-v) echo "$VERSION"
exit;;
-*) urchin_help >&2
exit 1;;
*) break;;
esac
shift
done
# Verify argument for main stuff
if [ "$#" != '1' ] || [ ! -d "$1" ]
then
[ -n "$1" ] && [ ! -d "$1" ] && echo "Not a directory: '$1'" >&2
echo "$USAGE" >&2
exit 2
fi
# Constants
logfile=$(fullpath "$1")/.urchin.log
stdout_file=$(fullpath "$1")/.urchin_stdout
# Verify argument for main stuff
if [ "$#" != '1' ] && [ ! -d "$1" ]
then
echo "$USAGE"
exit 1
fi
# Run or present the Molly guard.
if basename "$(fullpath "$1")" | grep test > /dev/null || $FORCE
if basename "$(fullpath "$1")" | grep -Fi 'test' > /dev/null || $force
then
urchin_go "$1"
urchin_go "$1" "$shell_for_sh_tests"
else
urchin_molly_guard
fi