21 Commits
todo ... tap

Author SHA1 Message Date
Thomas Levine
5e3c918675 add + indent symbol so directories line up 2016-01-28 15:12:34 +00:00
Thomas Levine
f5f98c8c2f fix fixture to not expect stderr 2016-01-28 15:11:27 +00:00
Thomas Levine
ce175cc3df test stdout stderr for TAP 2016-01-28 14:55:11 +00:00
Thomas Levine
cabaaa7ba4 TAP test 2016-01-28 14:51:43 +00:00
Thomas Levine
a663085069 remove awk dependency in test suite
I'm on NixOS, so awk isn't in /usr/bin. I can search for it with
/usr/bin/env, but then I can't use -f in the shebang line.
2016-01-27 10:51:04 +00:00
Thomas Levine
6bb606a3cc test plan 2016-01-27 10:39:07 +00:00
Thomas Levine
1c93e9a5c2 logfile 2016-01-27 10:31:51 +00:00
Thomas Levine
508e695dc3 error on fail 2016-01-27 10:25:21 +00:00
Thomas Levine
a51d96631f print skip count at end 2016-01-27 10:24:28 +00:00
Thomas Levine
1cd9991587 print stdout for tap and not-tap 2016-01-27 10:17:33 +00:00
Thomas Levine
0f1c2848b4 tap indentation comments for directories 2016-01-27 10:14:22 +00:00
Thomas Levine
9d10e12633 assorted tap stuff 2016-01-27 09:48:30 +00:00
Thomas Levine
40979f6e18 more tap 2016-01-27 01:13:55 +00:00
Thomas Levine
33e158e8f7 more tap 2016-01-27 01:08:04 +00:00
Thomas Levine
7ecacad132 oops 2016-01-27 01:05:43 +00:00
Thomas Levine
c8df46014d test test successes better 2016-01-27 01:05:08 +00:00
Thomas Levine
de2da89169 convert indents to comments 2016-01-27 00:50:55 +00:00
Thomas Levine
a6d6730e74 start writing non-tap cases 2016-01-27 00:32:51 +00:00
Thomas Levine
d01e993041 call urchin -s in cross-shell tests 2016-01-25 14:00:47 +00:00
Thomas Levine
b5c6464eab remove "urchin -x" test 2016-01-25 13:57:14 +00:00
Thomas Levine
329fc27929 remove "urchin -x"
unnecessary now that shall exists
2016-01-25 13:35:54 +00:00
14 changed files with 43 additions and 214 deletions

View File

@@ -1,9 +0,0 @@
Authors
-------
David Jones
Michael Klement
Thomas Levine
Maintainer
-------
Thomas Levine <_@thomaslevine.com>

20
HISTORY
View File

@@ -1,23 +1,5 @@
HISTORY
=======
Version 0.0.6
---------------------
* Produce TAP output with the -t flag.
* Add a + sign in front of directories in the normal output so that they
line up with non-directories.
* Display skipped tests in the normal output and in the TAP output.
* Correct some things in the documentation.
* Rearrange things in the documentation to be more clear.
* Pass the -e flag to exit urchin if any single test fails.
* Remove the undocumented, experimental -x flag now that shall exists.
* Display version number with the -v flag.
* Document why Urchin is called "Urchin"
These changes are made somewhat separately in the branches "exit-on-fail",
"remove-urchin-x", "tap", and "update-readme". They are rebased into one
branch, "tlevine-2016-02", for merging into "master".
-------
Version 0.0.5
---------------------

111
TODO
View File

@@ -1,111 +0,0 @@
Things I want
=============
Molly guard
-------------
The Molly-guard should be more accepting so that people don't have to use it
all the time and thus get used to using it. For example, you shouldn't need to
pass -f in this case.
https://github.com/creationix/nvm/issues/357
Test speed
-------------
Make tests run faster.
https://github.com/bike-barn/hermit/issues/62
First, easier thing is probably to run tests in parallel.
Second, also easier thing is to tell people to save things to RAM rather than
disk whenever they can.
Third, harder thing is to put the test suite in RAM automatically. Maybe the
whole test directory, which includes fixtures, gets copied to a tmpfs if one
exists.
Hmm or maybe there's a compromise: Tell people to mount /tmp as a tmpfs so
that temp files are fast. Maybe allow people to set some other directory as
the temporary file place, in case they want a different tmpfs location.
Options
-------------
I want long options. For example, there's presently -f and -e.
I want to make them -f|--force and -e|--exit.
Environment variables
-------------
Do something to make it easier to debug environment variables, because that is
often confusing.
https://github.com/creationix/nvm/issues/719
https://github.com/creationix/nvm/issues/589
Documenting that people should run "env" when their tests fail might be good
enough.
Licensing and copyright
------------------------
* Reference all owners and years in the Copyright file
* Consider copyleft licenses
* Add license notices to other files if necessary
Packaging
------------
Package for package managers.
* I want NixOS, of course.
* Debian is probably the big one.
Other interesting package managers
* Update the npm package
* Homebrew (for Mac)
Windows
----------
Try running Urchin in Windows somehow. Interpreters include
* CygWin (https://www.cygwin.com/)
* MSYS (http://mingw.org/wiki/msys)
* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
* Git for Windows (https://git-scm.com/download/win)
* win-bash (http://win-bash.sourceforge.net/)
Consider copyleft licenses
----------
ScraperWiki owns the original version of Urchin (Thomas Levine did the early
work as part of his work for ScraperWiki.) and originally licensed it under an
MIT-style license. Other people made changes after this original ScraperWiki
version. As of January 2016, they are just Thomas Levine (when he wasn't
working for ScraperWiki) and Michael Klement.
The original license was MIT just because that's what ScraperWiki put on
everything. Should we change the license?
The MIT-style license grants pretty much all rights. It says that you need
to attribute when you redistribute source code, but you don't
necessarily have to redistribute source code.
A copyleft license adds the restriction that modified versions of the
code need to be licensed under the same license. GNU licenses in
particular require that source code be released if non-source versions are
released, and the different GNU licenses differ in what how the
non-source version is defined. (The original, GPL, discusses compiled
binaries.) Copyleft doesn't mean anything specific for commercial use.
MIT-licensed code can be modified and then licensed as GPL, because MIT
license allows that, but GPL code can't be modified as MIT, because MIT
doesn't allow that. And if we get all of the authors to agree on it, we
can always add whatever crazy license we want, regardless of what we
have already.
The distinction between MIT-style and GNU-something might matter quite little
in the case of Urchin.
1. Urchin is written in an interpreted language (shell), so it might be
hard to distribute usefully without providing the source code.
2. Urchin just runs tests; it doesn't get compiled with the rest of the
code (also because it's in shell). Thus, I think a GPL license on
Urchin wouldn't infect the code being tested.
This is as far as I have gotten with contemplating license changes. For now
we're sticking with the original MIT-style license, but it's easy to change
licenses later.

View File

@@ -6,7 +6,7 @@ for shell in dash bash ksh zsh; do
if which $shell > /dev/null 2> /dev/null; then
echo
echo Running urchin tests in $shell
$shell urchin -s $shell tests | tail -n 4
$shell urchin -s $shell tests | tail -n 3
else
echo
echo Skipping $shell because it is not in the PATH

View File

@@ -4,13 +4,9 @@
/ /_/ / / / /__/ / / / / / / /
\__,_/_/ \___/_/ /_/_/_/ /_/
Urchin is a file-based test harness, normally used for testing shell programs.
It is written in portable shell and should thus work on GNU/Linux, BSD
(including Mac OS X), and other Unix-like platforms.
Urchin is called "Urchin" because
[sea urchins](https://en.wikipedia.org/wiki/Sea_urchin)
have shells called "tests".
Urchin is a test framework for shell. It is implemented in
portable /bin/sh and should work on GNU/Linux, Mac OS X, and
other Unix platforms.
## Try it out
Urchin's tests are written in Urchin, so you can run them to see what Urchin
@@ -31,15 +27,14 @@ run this:
cd urchin
./cross-shell-tests
## Install
Urchin is contained in a single file, so you can install it by copying it to a
directory in your `PATH`. For example, you can run the following as root.
## Globally
Download Urchin like so (as root) (or use npm, below):
cd /usr/local/bin
wget https://raw.github.com/tlevine/urchin/master/urchin
chmod +x urchin
Urchin can be installed with npm too.
Can be installed with npm too:
npm install -g urchin
@@ -91,14 +86,15 @@ Files are only run if they are executable, and files beginning with `.` are
ignored. Thus, fixtures and libraries can be included sloppily within the test
directory tree. The test passes if the file exits 0; otherwise, it fails.
Tests files and subdirectories are run in ASCIIbetical order within each
directory; that is,
In case you care about the order in which your tests execute, consider that
urchin looks for files within a directory in the following manner.
for file in *; do
do_something_with_test_file $file
done
Tests within a directory are executed in whatever order `*` returns.
### Writing cross-shell compatibility tests for testing shell code
While you could write your test scripts to explicitly invoke the functionality
@@ -110,20 +106,24 @@ The specific approach depends on your test scenario:
* (b) Your scripts _source_ scripts containing portable shell code.
#### (a) Cross-shell tests with test scripts that _invoke_ shell scripts
Urchin sets the `TEST_SHELL` environment variable so that you may change the
shell with which your tests call other shell programs. To run your test
scripts in multiple shells you must call `$TEST_SHELL` in your tests and then
run urchin with the appropriate option.
First, consider using [shall](https://github.com/mklement0/shall).
#!/usr/bin/env shall
echo This is a test file.
Alternatively, you can use urchin's built-in recognition of the
`TEST_SHELL` environment variable.
In your test scripts, invoke the shell scripts to test via the shell
specified in environment variable `TEST_SHELL` rather than directly;
e.g.: `$TEST_SHELL ../foo bar` (rather than just `../foo bar`).
Note that if you alsow want your test scripts to work when run directly,
outside of Urchin, be sure to target scripts that happen to be in the
current directory with prefix `./`; e.g., `$TEST_SHELL ./baz`
(rather than `$TEST_SHELL baz`).
On invocation of Urchin, prepend a definition of environment variable
`TEST_SHELL` specifying the shell to test with, e.g.,
TEST_SHELL=zsh urchin ./tests
Then, on invocation of Urchin, prepend a definition of environment variable
`TEST_SHELL` specifying the shell to test with, e.g.: `TEST_SHELL=zsh urchin ./tests`.
To test with multiple shells in sequence, use something like:
for shell in sh bash ksh zsh; do
@@ -131,20 +131,14 @@ To test with multiple shells in sequence, use something like:
done
If `TEST_SHELL` has no value, Urchin defines it as `/bin/sh`, so the test
scripts can rely on `$TEST_SHELL` always containing a value when Urchin runs
them.
That said, we still recommand that you account for the possibility that
`$TEST_SHELL` does not contain a value so that you may run your test scripts
without Urchin. Supporting this case is very simple; when you invoke scripts
that happen to be in the current directory, be sure to use the prefix `./`,
e.g., `$TEST_SHELL ./baz` rather than `$TEST_SHELL baz`.
scripts can rely on `$TEST_SHELL` always containing a value.
#### (b) Cross-shell tests with test scripts that _source_ shell scripts
If you _source_ shell code in your test scripts, it is the test scripts
themselves that must be run with the shell specified.
Urchin supports the `-s <shell>` option, which instructs
To that end, Urchin supports the `-s <shell>` option, which instructs
Urchin to invoke the test scripts with the specified shell; e.g., `-s bash`.
(In addition, Urchin sets environment variable `TEST_SHELL` to the specified
shell.)
@@ -160,12 +154,21 @@ To test with multiple shells in sequence, use something like:
urchin -s $shell ./tests
done
Also consider using [shall](https://github.com/mklement0/shall).
It does something similar, but the interface may be more intuitive.
<!--
#### (c) Cross shell tests with `urchin -x` (experimental)
If you run urchin with the `-x` flag, it will be as if you ran
`$TEST_SHELL`. Unless `$TEST_SHELL` isn't set, in which case it'll
be as if you ran `/bin/sh`. Putting this in she shebang line might
eventually work out to be a cleaner way of doing cross-shell testing.
#!/usr/bin/env shall
echo This is a test file.
#!/usr/bin/env urchin -x
test a = a
It might make sense if you do this.
export TEST_SHELL=zsh && urchin -x
export TEST_SHELL=bash && urchin -x
-->
## Alternatives to Urchin
Alternatives to Urchin are discussed in
[this blog post](https://blog.scraperwiki.com/2012/12/how-to-test-shell-scripts/).

View File

@@ -1 +0,0 @@
false

View File

@@ -1 +0,0 @@
false

View File

@@ -1 +0,0 @@
false

View File

@@ -1,11 +0,0 @@
tmp=$(mktemp)
../urchin -e -f ./.die-on-fail > $tmp
result=$?
grep '1 should run.' $tmp
grep '2 should run.' $tmp
grep -v '3 should not run.' $tmp
grep -v '4 should not run.' $tmp
rm $tmp
exit $result

View File

@@ -1,2 +0,0 @@
#!/bin/sh
../urchin -v | grep '[0-9.]\{3,\}'

26
urchin
View File

@@ -4,9 +4,6 @@
# which breaks fullpath().
unset CDPATH
# Urchin version number
VERSION=0.0.6
fullpath() {
(
cd -- "$1"
@@ -57,13 +54,6 @@ recurse() {
# $2 instead of $indent_level so it doesn't clash
recurse "${test}" $(( $2 + 1 )) "$shell_for_sh_tests"
exit_code=$?
if $exit_on_fail && test $exit_code -ne 0; then
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
[ -f teardown_dir ] && [ -x teardown_dir ] && ./teardown_dir >> "$stdout_file"
return 1
fi
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
done
@@ -78,6 +68,7 @@ recurse() {
else
if [ -x "$potential_test" ]
then
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# Run the test
@@ -89,6 +80,7 @@ recurse() {
fi
exit_code="$?"
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
if [ $exit_code -eq 0 ]; then
result=success
@@ -140,10 +132,6 @@ recurse() {
;;
esac
fi
if $exit_on_fail && test 0 -ne $exit_code; then
return 1
fi
fi
[ $indent_level -eq 0 ] && rm "$stdout_file"
}
@@ -163,14 +151,10 @@ $USAGE
-s <shell> Invoke test scripts that either have no shebang line at all or
have shebang line "#!/bin/sh" with the specified shell.
-e Stop running if any single test fails. This is helpful if you want
to use Urchin to run things other than tests, such as a set of
configuration scripts.
-f Force running even if the test directory's name does not
contain the word "test".
-t Format output in Test Anything Protocol (TAP)
-h, --help This help.
-v Display the version number.
-h This help.
Go to https://github.com/tlevine/urchin for documentation on writing tests.
@@ -247,12 +231,10 @@ urchin_molly_guard() {
shell_for_sh_tests=
force=false
exit_on_fail=false
tap_format=false
while [ $# -gt 0 ]
do
case "$1" in
-e) exit_on_fail=true;;
-f) force=true;;
-s)
shift
@@ -262,8 +244,6 @@ do
-t) tap_format=true;;
-h|--help) urchin_help
exit 0;;
-v) echo "$VERSION"
exit;;
-*) urchin_help >&2
exit 1;;
*) break;;

Binary file not shown.

Before

Width:  |  Height:  |  Size: 99 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 115 KiB