urchin/readme.md

204 lines
6.3 KiB
Markdown
Raw Normal View History

2013-06-26 03:22:54 -04:00
__ _
__ ____________/ /_ (_)___
/ / / / ___/ ___/ __ \/ / __ \
/ /_/ / / / /__/ / / / / / / /
\__,_/_/ \___/_/ /_/_/_/ /_/
2012-10-08 10:01:29 -04:00
Urchin is a portable shell program that runs a directory of Unix-style
programs and produces pretty output. It is normally used for testing
shell programs, where each test case corresponds to a single file in
the directory that Urchin runs.
2012-10-08 10:01:29 -04:00
2016-01-29 12:28:26 -05:00
Urchin is called "Urchin" because
[sea urchins](https://en.wikipedia.org/wiki/Sea_urchin)
have shells called "tests".
2012-10-11 14:57:10 -04:00
## Try it out
Urchin's tests are written in Urchin, so you can run them to see what Urchin
is like. Clone the repository
git clone git://github.com/tlevine/urchin.git
2012-10-11 14:57:10 -04:00
Run the tests
2013-07-02 04:43:59 -04:00
cd urchin
./urchin tests
2012-10-11 14:57:10 -04:00
2016-03-06 06:45:24 -05:00
## Dependencies
Urchin depends on the following programs.
* sh
* echo
* printf
* mktemp
* readlink
* basename
* dirname
* sed
* grep
* cut
* true
* false
* which
* timeout
* sort
2016-03-31 14:55:28 -04:00
Vanilla installations of modern BSD and GNU systems usually include all
of these programs.
2016-03-06 06:53:37 -05:00
2016-01-25 08:56:33 -05:00
## Install
Urchin is contained in a single file, so you can install it by copying it to a
directory in your `PATH`. For example, you can run the following as root.
2012-10-08 10:01:29 -04:00
2013-06-21 12:14:44 -04:00
cd /usr/local/bin
2016-04-03 21:22:05 -04:00
wget https://raw.githubusercontent.com/tlevine/urchin/v0.1.0-rc3/urchin
2013-06-21 12:14:44 -04:00
chmod +x urchin
2016-01-25 08:56:33 -05:00
Urchin can be installed with npm too.
2013-06-21 12:14:44 -04:00
npm install -g urchin
2012-10-08 10:01:29 -04:00
Now you can run it.
2012-10-10 15:51:06 -04:00
urchin <test directory>
2012-10-08 10:01:29 -04:00
Run `urchin -h` to get command-line help.
2012-10-08 10:01:29 -04:00
## Writing tests
2012-10-08 10:13:43 -04:00
Make a root directory for your tests. Inside it, put executable files that
2012-10-08 10:16:49 -04:00
exit `0` on success and something else on fail. Non-executable files and hidden
files (dotfiles) are ignored, so you can store fixtures right next to your
tests. Run urchin from inside the tests directory.
2012-10-08 10:01:29 -04:00
2013-06-19 04:20:13 -04:00
Urchin only cares about the exit status, so you can actually write your tests
2012-10-10 12:53:03 -04:00
in any language, not just shell.
2012-10-08 10:01:29 -04:00
## More about writing tests
2012-10-04 07:22:44 -04:00
Tests are organized recursively in directories, where the names of the files
and directories have special meanings.
tests/
setup
2012-10-10 15:47:21 -04:00
setup_dir
2012-10-04 07:22:44 -04:00
bar/
setup
test_that_something_works
2012-10-08 09:54:59 -04:00
teardown
2012-10-04 07:22:44 -04:00
baz/
jack-in-the-box/
setup
test_that_something_works
teardown
cat-in-the-box/
2012-10-08 09:54:59 -04:00
fixtures/
thingy.pdf
test_thingy
teardown
2012-10-04 07:22:44 -04:00
Directories are processed in a depth-first order. When a particular directory
2016-02-29 00:07:15 -05:00
is processed, `setup_dir` is sourced before everything else in the directory,
including subdirectories. `teardown_dir` is sourced after everything else in
the directory.
2012-10-04 07:22:44 -04:00
2016-02-29 00:07:15 -05:00
A directory's `setup` file, if it exists, is sourced right before each test
file within the particular directory is run, and the `teardown` file is
sourced right after.
2012-10-04 07:22:44 -04:00
2012-10-08 10:16:49 -04:00
Files are only run if they are executable, and files beginning with `.` are
2012-10-08 10:24:32 -04:00
ignored. Thus, fixtures and libraries can be included sloppily within the test
directory tree. The test passes if the file exits 0; otherwise, it fails.
2013-10-13 12:48:42 -04:00
2016-03-31 14:55:28 -04:00
urchin looks for files within a directory in the following manner,
2015-08-21 09:03:12 -04:00
for file in *; do
do_something_with_test_file $file
done
2016-03-31 16:42:22 -04:00
so files are run in whatever order `*` produces. The order is
configured in your environment, at least in
[GNU systems](https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021).
2016-03-31 14:55:28 -04:00
Other systems may ignore the locales configured in the environment and
always produce ASCIIbetical order.
Results are always printed in ASCIIbetical order, regardless of what
order the tests ran in.
2016-03-29 12:50:18 -04:00
2016-03-31 16:42:22 -04:00
Below you can see how the locale can affect the order.
$ printf '!c\n@a\n~b\n' | LC_COLLATE=C sort
!c
@a
~b
$ printf '!c\n@a\n~b\n' | LC_COLLATE=en_US.UTF-8 sort
@a
~b
!c
$ printf '!c\n@a\n~b\n' | sort -d
@a
~b
!c
### Writing cross-shell compatibility tests for testing shell code
While you could write your test scripts to explicitly invoke the functionality
to test with various shells, Urchin facilitates a more flexible approach.
The specific approach depends on your test scenario:
* (a) Your test scripts _invoke_ scripts containing portable shell code.
* (b) Your scripts _source_ scripts containing portable shell code.
#### (a) Cross-shell tests with test scripts that _invoke_ shell scripts
Urchin sets the `TEST_SHELL` environment variable so that you may change the
shell with which your tests call other shell programs. To run your test
scripts in multiple shells you must call `$TEST_SHELL` in your tests and then
run urchin with the appropriate option.
In your test scripts, invoke the shell scripts to test via the shell
specified in environment variable `TEST_SHELL` rather than directly;
e.g.: `$TEST_SHELL ../foo bar` (rather than just `../foo bar`).
2016-02-29 00:07:15 -05:00
Urchin runs tests in multiple different shells by default; Urchin has a
list of default shells, and the following command will run your tests in
all of those shells that Urchin detects.
2016-02-29 00:07:15 -05:00
./urchin ./tests
2016-02-29 00:07:15 -05:00
You can override the default list of shells with the `-s` flag.
2016-02-29 00:07:15 -05:00
urchin -s sh -s ksh ./tests
You can also
If `TEST_SHELL` has no value, Urchin defines it as `/bin/sh`, so the test
scripts can rely on `$TEST_SHELL` always containing a value when Urchin runs
them.
That said, we still recommand that you account for the possibility that
`$TEST_SHELL` does not contain a value so that you may run your test scripts
without Urchin. Supporting this case is very simple; when you invoke scripts
that happen to be in the current directory, be sure to use the prefix `./`,
e.g., `$TEST_SHELL ./baz` rather than `$TEST_SHELL baz`.
#### (b) Cross-shell tests with test scripts that _source_ shell scripts
If you _source_ shell code in your test scripts, it is the test scripts
themselves that must be run with the shell specified.
Urchin supports the `-s <shell>` option, which instructs
Urchin to invoke the test scripts with the specified shell; e.g., `-s bash`.
2014-10-18 10:41:22 -04:00
(In addition, Urchin sets environment variable `TEST_SHELL` to the specified
shell.)
Note that only test scripts that either have no shebang line at all or
have shebang line `#!/bin/sh` are invoked with the specified shell.
2016-03-29 12:51:38 -04:00
This allows non-shell test scripts or test scripts for other languages
or for specific shells to coexist with those whose invocation should be
controlled by `-s`.
2016-03-03 06:18:09 -05:00
## References
On shell programming
* http://blackskyresearch.net/shelltables.txt
* http://blackskyresearch.net/try.sh.txt