nagios todo

It previously just showed the last three lines. It needs to show the
last four lines now because urchin now displays skipped tests.

AUTHORS

@@ -0,0 +1,9 @@
+Authors
+-------
+David Jones
+Michael Klement
+Thomas Levine
+
+Maintainer
+-------
+Thomas Levine <_@thomaslevine.com>

HISTORY

@@ -1,5 +1,28 @@
 HISTORY
--------
+=======
+
+Version 0.0.6
+---------------------
+
+* Produce TAP output with the -t flag.
+* Add a + sign in front of directories in the normal output so that they
+  line up with non-directories.
+* Display skipped tests in the normal output and in the TAP output.
+* Correct some things in the documentation.
+* Rearrange things in the documentation to be more clear.
+* Pass the -e flag to exit urchin if any single test fails.
+* Remove the undocumented, experimental -x flag now that shall exists.
+* Display version number with the -v flag.
+* Document why Urchin is called "Urchin"
+
+These changes are made somewhat separately in the branches "exit-on-fail",
+"remove-urchin-x", "tap", and "update-readme". They are rebased into one
+branch, "tlevine-2016-02", for merging into "master".
+
+Version 0.0.5
+---------------------
+* urchin now unsets `CDPATH`.
+* The documentation for `urchin -x` was removed because it was confusing.
 
 Version 0.0.4
 ---------------------

TODO

@@ -0,0 +1,117 @@
+Things I want
+=============
+
+Molly guard
+-------------
+The Molly-guard should be more accepting so that people don't have to use it
+all the time and thus get used to using it. For example, you shouldn't need to
+pass -f in this case.
+https://github.com/creationix/nvm/issues/357
+
+Test speed
+-------------
+Make tests run faster.
+https://github.com/bike-barn/hermit/issues/62
+
+First, easier thing is probably to run tests in parallel.
+
+Second, also easier thing is to tell people to save things to RAM rather than
+disk whenever they can.
+
+Third, harder thing is to put the test suite in RAM automatically. Maybe the
+whole test directory, which includes fixtures, gets copied to a tmpfs if one
+exists.
+
+Hmm or maybe there's a compromise: Tell people to mount /tmp as a tmpfs so
+that temp files are fast. Maybe allow people to set some other directory as
+the temporary file place, in case they want a different tmpfs location.
+
+Options
+-------------
+I want long options. For example, there's presently -f and -e.
+I want to make them -f|--force and -e|--exit.
+
+Environment variables
+-------------
+Do something to make it easier to debug environment variables, because that is
+often confusing.
+https://github.com/creationix/nvm/issues/719
+https://github.com/creationix/nvm/issues/589
+
+Documenting that people should run "env" when their tests fail might be good
+enough.
+
+Licensing and copyright
+------------------------
+* Reference all owners and years in the Copyright file
+* Consider copyleft licenses
+* Add license notices to other files if necessary
+
+Packaging
+------------
+Package for package managers.
+
+* I want NixOS, of course.
+* Debian is probably the big one.
+
+Other interesting package managers
+
+* Update the npm package
+* Homebrew (for Mac)
+
+Windows
+----------
+Try running Urchin in Windows somehow. Interpreters include
+
+* CygWin (https://www.cygwin.com/)
+* MSYS (http://mingw.org/wiki/msys)
+* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
+* Git for Windows (https://git-scm.com/download/win)
+* win-bash (http://win-bash.sourceforge.net/)
+
+Consider copyleft licenses
+----------
+ScraperWiki owns the original version of Urchin (Thomas Levine did the early
+work as part of his work for ScraperWiki.) and originally licensed it under an
+MIT-style license. Other people made changes after this original ScraperWiki
+version. As of January 2016, they are just Thomas Levine (when he wasn't
+working for ScraperWiki) and Michael Klement.
+
+The original license was MIT just because that's what ScraperWiki put on
+everything. Should we change the license?
+
+The MIT-style license grants pretty much all rights. It says that you need
+to attribute when you redistribute source code, but you don't
+necessarily have to redistribute source code.
+
+A copyleft license adds the restriction that modified versions of the
+code need to be licensed under the same license. GNU licenses in
+particular require that source code be released if non-source versions are
+released, and the different GNU licenses differ in what how the
+non-source version is defined. (The original, GPL, discusses compiled
+binaries.) Copyleft doesn't mean anything specific for commercial use.
+
+MIT-licensed code can be modified and then licensed as GPL, because MIT
+license allows that, but GPL code can't be modified as MIT, because MIT
+doesn't allow that. And if we get all of the authors to agree on it, we
+can always add whatever crazy license we want, regardless of what we
+have already.
+
+The distinction between MIT-style and GNU-something might matter quite little
+in the case of Urchin.
+
+1. Urchin is written in an interpreted language (shell), so it might be
+   hard to distribute usefully without providing the source code.
+2. Urchin just runs tests; it doesn't get compiled with the rest of the
+   code (also because it's in shell). Thus, I think a GPL license on
+   Urchin wouldn't infect the code being tested.
+
+This is as far as I have gotten with contemplating license changes. For now
+we're sticking with the original MIT-style license, but it's easy to change
+licenses later.
+
+Nagios plugins
+-----------------
+It would be cool to run Nagios plugins with Urchin. This is already possible,
+actually, but it might be worth giving some special thought to it.
+https://nagios-plugins.org/doc/guidelines.html

alternatives

@@ -0,0 +1,2 @@
+Totally different syntax and similar features, plus TAP output
+https://github.com/sstephenson/bats

cross-shell-tests

@@ -2,11 +2,11 @@
 # Run urchin in a bunch of different shells,
 # including a shell that isn't quite POSIX-compatible (zsh)
 
-for shell in dash bash ksh zsh; do
+for shell in dash bash mksh ksh zsh; do
   if which $shell > /dev/null 2> /dev/null; then
     echo
     echo Running urchin tests in $shell
-    $shell urchin tests | tail -n 3
+    $shell urchin -s $shell tests | tail -n 4
   else
     echo
     echo Skipping $shell because it is not in the PATH

package.json

@@ -1,6 +1,6 @@
 {
   "name": "urchin",
-  "version": "0.0.4",
+  "version": "0.0.5",
   "description": "Test framework for shell",
   "main": "urchin",
   "directories": {
@@ -24,10 +24,9 @@
     {"name": "Thomas Levine", "email": "_@thomaslevine.com"},
     {"name": "David Jones", "email": "drj@pobox.com"},
     {"name": "Francis Irving", "email": "francis@flourish.org"},
-    {"name": "Zarino Zappia", "email": "mail@zarino.co.uk"}
-    {"name": "Tom Mortimer-Jones", "email": "tom@morty.co.uk"}
+    {"name": "Zarino Zappia", "email": "mail@zarino.co.uk"},
+    {"name": "Tom Mortimer-Jones", "email": "tom@morty.co.uk"},
     {"name": "Michael Klement", "email": "mklement0@gmail.com"}
   ],
-  "license": "BSD",
-  "readmeFilename": "readme.md"
+  "license": "BSD"
 }

readme.md

@@ -4,22 +4,26 @@
     / /_/ / /  / /__/ / / / / / / /
     \__,_/_/   \___/_/ /_/_/_/ /_/ 
 
-Urchin is a test framework for shell. It is implemented in
-portable /bin/sh and should work on GNU/Linux, Mac OS X, and
-other Unix platforms.
+Urchin is a file-based test harness, normally used for testing shell programs.
+It is written in portable shell and should thus work on GNU/Linux, BSD
+(including Mac OS X), and other Unix-like platforms.
+
+Urchin is called "Urchin" because
+[sea urchins](https://en.wikipedia.org/wiki/Sea_urchin)
+have shells called "tests".
 
 ## Try it out
 Urchin's tests are written in Urchin, so you can run them to see what Urchin
 is like. Clone the repository
 
-    git clone git://github.com/scraperwiki/urchin.git
+    git clone git://github.com/tlevine/urchin.git
 
 Run the tests
 
     cd urchin
     ./urchin tests
 
-The above command will run the tests in your systems default
+The above command will run the tests in your system's default
 shell, /bin/sh (on recent Ubuntu this is dash, but it could be
 ksh or bash on other systems); to test urchin's cross-shell compatibility,
 run this:
@@ -27,14 +31,15 @@ run this:
     cd urchin
     ./cross-shell-tests
 
-## Globally
-Download Urchin like so (as root) (or use npm, below):
+## Install
+Urchin is contained in a single file, so you can install it by copying it to a
+directory in your `PATH`. For example, you can run the following as root.
 
     cd /usr/local/bin
-    wget https://raw.github.com/scraperwiki/urchin/master/urchin
+    wget https://raw.github.com/tlevine/urchin/master/urchin
     chmod +x urchin
 
-Can be installed with npm too:
+Urchin can be installed with npm too.
 
     npm install -g urchin
 
@@ -86,6 +91,14 @@ Files are only run if they are executable, and files beginning with `.` are
 ignored. Thus, fixtures and libraries can be included sloppily within the test
 directory tree. The test passes if the file exits 0; otherwise, it fails.
 
+Tests files and subdirectories are run in ASCIIbetical order within each
+directory; that is,
+urchin looks for files within a directory in the following manner.
+
+    for file in *; do
+      do_something_with_test_file $file
+    done
+
 ### Writing cross-shell compatibility tests for testing shell code
 
 While you could write your test scripts to explicitly invoke the functionality
@@ -97,17 +110,20 @@ The specific approach depends on your test scenario:
 * (b) Your scripts _source_ scripts containing portable shell code.
 
 #### (a) Cross-shell tests with test scripts that _invoke_ shell scripts
+Urchin sets the `TEST_SHELL` environment variable so that you may change the
+shell with which your tests call other shell programs. To run your test
+scripts in multiple shells you must call `$TEST_SHELL` in your tests and then
+run urchin with the appropriate option.
 
 In your test scripts, invoke the shell scripts to test via the shell
 specified in environment variable `TEST_SHELL` rather than directly;
 e.g.: `$TEST_SHELL ../foo bar` (rather than just `../foo bar`).  
-Note that if you alsow want your test scripts to work when run directly,
-outside of Urchin, be sure to target scripts that happen to be in the 
-current directory with prefix `./`; e.g., `$TEST_SHELL ./baz`
-(rather than `$TEST_SHELL baz`).
 
-Then, on invocation of Urchin, prepend a definition of environment variable
-`TEST_SHELL` specifying the shell to test with, e.g.: `TEST_SHELL=zsh urchin ./tests`.  
+On invocation of Urchin, prepend a definition of environment variable
+`TEST_SHELL` specifying the shell to test with, e.g.,
+
+    TEST_SHELL=zsh urchin ./tests
+
 To test with multiple shells in sequence, use something like:
 
     for shell in sh bash ksh zsh; do
@@ -115,14 +131,20 @@ To test with multiple shells in sequence, use something like:
     done
 
 If `TEST_SHELL` has no value, Urchin defines it as `/bin/sh`, so the test
-scripts can rely on `$TEST_SHELL` always containing a value.
+scripts can rely on `$TEST_SHELL` always containing a value when Urchin runs
+them.
+
+That said, we still recommand that you account for the possibility that
+`$TEST_SHELL` does not contain a value so that you may run your test scripts
+without Urchin. Supporting this case is very simple; when you invoke scripts
+that happen to be in the current directory, be sure to use the prefix `./`,
+e.g., `$TEST_SHELL ./baz` rather than `$TEST_SHELL baz`.
 
 #### (b) Cross-shell tests with test scripts that _source_ shell scripts
-
 If you _source_ shell code in your test scripts, it is the test scripts
 themselves that must be run with the shell specified.
 
-To that end, Urchin supports the `-s <shell>` option, which instructs
+Urchin supports the `-s <shell>` option, which instructs
 Urchin to invoke the test scripts with the specified shell; e.g., `-s bash`.  
 (In addition, Urchin sets environment variable `TEST_SHELL` to the specified
 shell.)
@@ -138,19 +160,11 @@ To test with multiple shells in sequence, use something like:
       urchin -s $shell ./tests
     done
 
-#### (c) Cross shell tests with `urchin -x` (experimental)
-If you run urchin with the `-x` flag, it will be as if you ran
-`$TEST_SHELL`. Unless `$TEST_SHELL` isn't set, in which case it'll
-be as if you ran `/bin/sh`. Putting this in she shebang line might
-eventually work out to be a cleaner way of doing cross-shell testing.
+Also consider using [shall](https://github.com/mklement0/shall).
+It does something similar, but the interface may be more intuitive.
 
-    #!/usr/bin/env urchin -x
-    test a = a
-
-It might make sense if you do this.
-
-    export TEST_SHELL=zsh && urchin -x
-    export TEST_SHELL=bash && urchin -x
+    #!/usr/bin/env shall
+    echo This is a test file.
 
 ## Alternatives to Urchin
 Alternatives to Urchin are discussed in

tests/.die-on-fail/2 should run.

@@ -0,0 +1 @@
+false

tests/.die-on-fail/3 should not run.

@@ -0,0 +1 @@
+false

tests/.die-on-fail/4 should not run.

@@ -0,0 +1 @@
+false

tests/A nonempty CDPATH should not break urchin.

@@ -0,0 +1,6 @@
+#!/bin/sh
+
+cd ..
+export CDPATH=$PWD
+./urchin -f 'tests/urchin exit code' >/dev/null
+

tests/Cross-shell test support/.test-run-by-specified-shell/With -s, a non-sh test script should be invoked directly.

@@ -1,5 +1,3 @@
-#!/usr/bin/awk -f
-
-# This script will only succeed if it is indeed processed by awk.
-
-BEGIN { print "ok" }
+#!/usr/bin/env true
+true will processed the contents of this script, but that
+means that nothing will happen and the script will exit 0 

tests/Cross-shell test support/The -s option should invoke tests with the specified shell.

@@ -3,6 +3,5 @@
 # Tests the `-s <shell> option, which invokes shebang-less and sh-shebang-line test scripts with the specified shell (for testing *sourced* shell code).
 
 which bash >/dev/null || { echo "Cannot test -s option: bash cannot be located." >&2; exit 2; }
-which /usr/bin/awk >/dev/null || { echo "Cannot test -s option: /usr/bin/awk not found." >&2; exit 2; }
 
 ../../urchin -s bash ./.test-run-by-specified-shell

tests/Exit on fail if -e is passed.

@@ -0,0 +1,11 @@
+tmp=$(mktemp)
+../urchin -e -f ./.die-on-fail > $tmp
+result=$?
+
+grep '1 should run.' $tmp
+grep '2 should run.' $tmp
+grep -v '3 should not run.' $tmp
+grep -v '4 should not run.' $tmp
+
+rm $tmp
+exit $result

tests/Print version on -v.

@@ -0,0 +1,2 @@
+#!/bin/sh
+../urchin -v | grep '[0-9.]\{3,\}'

tests/TAP/.expected-output

@@ -0,0 +1,10 @@
+# Begin - .testsuite/
+not ok 1 - a
+# ------------ Begin output ------------
+# This is stdout from a.
+# ------------ End output ------------
+ok 2 - b
+ok 3 - # SKIP c
+# End - .testsuite/
+# Took 0 seconds.
+1..3

tests/TAP/.testsuite/a

@@ -0,0 +1,4 @@
+#!/bin/sh
+echo This is stderr from a. > /dev/stderr
+echo This is stdout from a. > /dev/stdout
+false

tests/TAP/.testsuite/b

@@ -0,0 +1,4 @@
+#!/bin/sh
+echo This is stderr from b. > /dev/stderr
+echo This is stdout from b. > /dev/stdout
+true

tests/TAP/.testsuite/c

@@ -0,0 +1 @@
+This should not be run.

tests/TAP/Running with -t should produce TAP output.

@@ -0,0 +1,4 @@
+tmp=$(mktemp)
+
+../../urchin -t .testsuite/ | sed 1d > $tmp
+diff $tmp .expected-output

tests/urchin -x should start the $TEST_SHELL.

@@ -1,3 +0,0 @@
-#!/bin/sh
-
-test c = $(../urchin -x .print-arg-3 a 'b b b b' c d e)

urchin

@@ -1,5 +1,12 @@
 #!/bin/sh
 
+# Make sure that CDPATH isn't set, as it causes `cd` to behave unpredictably - notably, it can produce output,
+# which breaks fullpath().
+unset CDPATH
+
+# Urchin version number
+VERSION=0.0.6
+
 fullpath() {
   (
     cd -- "$1"
@@ -9,7 +16,9 @@ fullpath() {
 
 indent() {
   level="$1"
+  if test "$level" -gt 0; then
     printf "%$((2 * ${level}))s"
+  fi
 }
 
 recurse() {
@@ -26,9 +35,16 @@ recurse() {
 
   if [ -d "$potential_test" ]
     then
-    (
+
+    if $tap_format; then
+      indent $indent_level | sed 's/  /#/g'
+      echo "# Begin - ${potential_test}"
+    else
       indent $indent_level
-      echo "  ${potential_test}"
+      echo "+ ${potential_test}"
+    fi
+
+    (
       cd -- "$potential_test"
       [ -f setup_dir ] && [ -x setup_dir ] && ./setup_dir >> "$stdout_file"
 
@@ -43,15 +59,27 @@ recurse() {
 
         # $2 instead of $indent_level so it doesn't clash
         recurse "${test}" $(( $2 + 1 )) "$shell_for_sh_tests"
+        exit_code=$?
+
+        if $exit_on_fail && test $exit_code -ne 0; then
+          [ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
+          [ -f teardown_dir ] && [ -x teardown_dir ] && ./teardown_dir >> "$stdout_file"
+          return 1
+        fi
 
         [ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
       done
       [ -f teardown_dir ] && [ -x teardown_dir ] && ./teardown_dir >> "$stdout_file"
-      echo
     )
-  elif [ -x "$potential_test" ]
+    if $tap_format; then
+      indent $indent_level | sed 's/  /#/g'
+      echo "# End - ${potential_test}"
+    else
+      echo
+    fi
+  else
+    if [ -x "$potential_test" ]
       then
-
       [ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
 
       # Run the test
@@ -63,24 +91,60 @@ recurse() {
       fi
       exit_code="$?"
 
-
       [ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
+      if [ $exit_code -eq 0 ]; then
+        result=success
+      else
+        result=fail
+      fi
+    else
+      result=skip
+    fi
 
+    echo "${result}" >> "$logfile"
+    if $tap_format; then
+      n=$(grep -ce '^\(success\|fail\|skip\)' "$logfile")
+
+      if [ "$result" == fail ]; then
+        not='not '
+      else
+        not=''
+      fi
+      if [ "$result" == skip ]; then
+        skip='# SKIP '
+      else
+        skip=''
+      fi
+      echo "${not}ok $n - ${skip}${potential_test}"
+      if [ "$result" == fail ]; then
+        echo '# ------------ Begin output ------------'
+        sed 's/^/# /' "$stdout_file"
+        echo '# ------------ End output ------------'
+      fi
+    else
       indent $indent_level
-    if [ $exit_code -eq 0 ]
-      then
+      case "$result" in
+        success)
           # On success, print a green '✓'
           printf '\033[32m✓ \033[0m'
           printf '%s\n' "${potential_test}"
-      printf '%s\n' "${potential_test} passed" >> "$logfile"
-    else
+        ;;
+        fail)
           # On fail, print a red '✗'
           printf '\033[31m✗ \033[0m'
           printf '%s\n' "${potential_test}"
-      printf '%s\n' "${potential_test} failed" >> "$logfile"
           printf '\033[31m' # Print output captured from failed test in red.
           cat "$stdout_file"
           printf '\033[0m'
+        ;;
+        skip)
+          printf '  %s\n' "${potential_test}"
+        ;;
+      esac
+    fi
+
+    if $exit_on_fail && test 0 -ne $exit_code; then
+      return 1
     fi
   fi
   [ $indent_level -eq 0 ] && rm "$stdout_file"
@@ -95,19 +159,24 @@ has_sh_or_no_shebang_line() {
 USAGE="usage: $0 [<options>] <test directory>"
 
 urchin_help() {
-  echo
-  echo "$USAGE"
-  echo
-  echo '-s <shell>  Invoke test scripts that either have no shebang line or'
-  echo '            shebang line "#!/bin/sh" with the specified shell.'
-  echo '-f          Force running even if the test directory'\''s name does not'
-  echo '            contain the word "test".'
-  echo '-x          Run "$TEST_SHELL", falling back on /bin/sh. This might be'
-  echo '            useful in the shebang line (experimental).'
-  echo '-h          This help.'
-  echo
-  echo 'Go to https://github.com/tlevine/urchin for documentation on writing tests.'
-  echo
+  cat <<EOF
+
+$USAGE
+
+-s <shell>  Invoke test scripts that either have no shebang line at all or
+            have shebang line "#!/bin/sh" with the specified shell.
+-e          Stop running if any single test fails. This is helpful if you want
+            to use Urchin to run things other than tests, such as a set of
+            configuration scripts.
+-f          Force running even if the test directory's name does not
+            contain the word "test".
+-t          Format output in Test Anything Protocol (TAP)
+-h, --help  This help.
+-v          Display the version number.
+
+Go to https://github.com/tlevine/urchin for documentation on writing tests.
+
+EOF
 }
 
 plural () {
@@ -123,7 +192,11 @@ plural () {
 }
 
 urchin_go() {
-  echo Running tests at $(date +%Y-%m-%dT%H:%M:%S) | tee "$logfile"
+  rm -f "$logfile"
+  if "$tap_format"; then
+    printf \#\ 
+  fi
+  echo Running tests at $(date +%Y-%m-%dT%H:%M:%S)
   start=$(date +%s)
 
   # Determine the environment variable to define for test scripts
@@ -143,13 +216,23 @@ urchin_go() {
 
   finish=$(date +%s)
   elapsed=$(($finish - $start))
+
+  passed=$(grep -c '^success' "$logfile")
+  failed=$(grep -c '^fail' "$logfile")
+  skipped=$(grep -c '^skip' "$logfile")
+  if $tap_format; then
+    echo "# Took $elapsed $(plural second $elapsed)."
+    echo 1..$(($passed + $failed + $skipped))
+  else
     echo "Done, took $elapsed $(plural second $elapsed)."
-  set -- $(grep -e 'passed$' "$logfile"|wc -l) $(grep -e 'failed$' "$logfile"|wc -l)
-  printf '%s\n' "$1 $(plural test "$1") passed."
-  [ $2 -gt 0 ] && printf '\033[31m' || printf '\033[32m' # If tests failed, print the message in red, otherwise in green.
-  printf '%s\n' "$2 $(plural test "$2") failed."
+    printf '%s\n' "$passed $(plural test "$passed") passed."
+    printf '%s\n' "$skipped $(plural test "$skipped") skipped."
+    [ $failed -gt 0 ] && printf '\033[31m' || printf '\033[32m' # If tests failed, print the message in red, otherwise in green.
+    printf '%s\n' "$failed $(plural test "$failed") failed."
     printf '\033[m'
-  return "$2"
+  fi
+  rm -f "$logfile"
+  test -z "$failed" || test "$failed" -eq '0'
 }
 
 urchin_molly_guard() {
@@ -166,22 +249,23 @@ urchin_molly_guard() {
 
 shell_for_sh_tests=
 force=false
+exit_on_fail=false
+tap_format=false
 while [ $# -gt 0 ]
 do
     case "$1" in
+        -e) exit_on_fail=true;;
         -f) force=true;;
         -s)
           shift
           shell_for_sh_tests=$1
           which "$shell_for_sh_tests" >/dev/null || { echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2; urchin_help >&2; exit 2; }
           ;;
-        -x) # `urchin -sh` is equivalent to "$TEST_SHELL"
-          shift
-          urchinsh=${TEST_SHELL:-/bin/sh}
-          "$urchinsh" "$@"
-          exit $?;;
+        -t) tap_format=true;;
         -h|--help) urchin_help
           exit 0;;
+        -v) echo "$VERSION"
+            exit;;
         -*) urchin_help >&2
             exit 1;;
         *)  break;;