bump version

HISTORY

@@ -12,18 +12,18 @@ test suite, as we need to know how far up to apply the setup
 and teardown files.
 
 The Urchin root directory is determined by moving higher in the directory
-tree in search of a file named `.urchin`.
+tree in search of a file named `.urchin_root`.
 The closest directory that contains such a file is considered the root.
 In the following filesystem, for example, `/a/b/c` would be the root.
 
     mkdir -p /a/b/c/d
     touch /a/b/c/d/e
     chmod +x /a/b/c/d/e
-    touch /a/b/c/.urchin
+    touch /a/b/c/.urchin_root
     urchin /a/b/c/d
 
 There are two situations in which we would stop looking without having
-found a `.urchin` file.
+found a `.urchin_root` file.
 
 1. The system root, `/`, because we can't go any higher
 2. A directory that starts with a dot, because an urchin call on a higher
@@ -97,6 +97,13 @@ turned on.
 
 Parallel processing and shell cycling are both enabled by default.
 
+You may want make only some directories run in series, you can create
+".urchin_dir" files in those directories.
+If .urchin_dir contains the phrase "series", run that directory in series
+rather than in parallel.
+This is helpful when directories actually need to run in series
+and also when running all your tests in parallel crashes your computer.
+
 ### Options
 Long options are now available for all command line flags.
 For example, the `-s` flag is now available as `--shell` as well.
@@ -229,7 +236,7 @@ other Urchin call), and the test suite is recursively descended. Setup and
 teardown files are sourced, and everything but the specified test file is
 otherwise ignored.
 
-If you don't explicitly specify the Urchin root with a .urchin file, we
+If you don't explicitly specify the Urchin root with a .urchin_root file, we
 consider the test suite root directory to be the parent of the file that
 you ran Urchin on.
 

SORTING

@@ -1,5 +1,67 @@
 On the criteria for ordering
 ==============================
+
+I was confused by the documentation for sort's "-d" flag. This confusion
+relates to GNU coreutil's locale-specific sort. [^]
+
+Below I discuss sort order differences between different implementations
+of sort and of sh "*" for my particular environments.
+
+Sorting with sort
+------------
+Consider the following two sort commands.
+
+    printf '@ b\n- d\n? a\n~ c\n! e\n' | sort
+    printf '@ b\n- d\n? a\n~ c\n! e\n' | sort -d
+
+With BusyBox v1.23.2 on NixOS 15.09, the first of these commands returns
+ASCIIbetical order,
+
+    ! e
+    - d
+    ? a
+    @ b
+    ~ c
+
+and the second returns dictionary order.
+
+    ? a
+    @ b
+    ~ c
+    - d
+    ! e
+
+With GNU coreutils version 8.24 on NixOS, both commands return
+dictionary order. The same is true for GNU coreutils version 8.23 on
+Debian Wheezy.
+
+    ? a
+    @ b
+    ~ c
+    - d
+    ! e
+
+IEEE Std 1003.1, 2013 Edition [^^] specifies that the "-d" flag should
+enable dictionary order. All of these versions of sort have clear
+documentation about the order that should be returned when the "-d" flag
+is set, (See --help, man, or info.) and the implementations match the
+documentation as far as I can tell.
+
+I have found no explicit documentation from any relevant source as to
+what the default sort order should be. On the other hand, they all
+suggest that "-d" produces an order different from the default order.
+
+In GNU coreutils 8.24, for example, "-d" is a direction to "consider
+only blanks and alphanumeric characters". It lacks any mention that the
+"-d" flag has no effect or that it is the default. Furthermore, on my
+first reading, I took it to mean that the default is to consider all
+characters and that "-d" limits the considered characters to blanks and
+alphanumeric characters.
+
+
+Sorting in *
+-------------
+I think this is related to the order returned by "*" in sh.
 The following sh code creates several files in a directory and then
 calls "*", listing them in order.
 
@@ -8,7 +70,8 @@ calls "*", listing them in order.
     done
     for file in *; do echo "$file"; done
 
-On one computer, running FreeBSD, the order is apparently ASCIIbetical.
+On one computer, running FreeBSD, the order is apparently
+ASCIIbetical.
 
     ! e
     - d
@@ -16,7 +79,7 @@ On one computer, running FreeBSD, the order is apparently ASCIIbetical.
     @ b
     ~ c
 
-On another computer, running NixOS, the following commands print results
+On two GNU systems, running NixOS and Debian, respectively, output is
 in dictionary order. I'm not exactly sure what dictionary order is, but
 it is something like sorting on the alphabetical characters before
 sorting on the rest of the line.
@@ -27,16 +90,9 @@ sorting on the rest of the line.
     - d
     ! e
 
-While I don't really know what dictionary order is, I was able to determine
+(I don't really know what dictionary order is, I was able to determine
 that the above results are in dictionary order because of my investigation of
-incompatible implementations of sort. Consider the following two sort
-commands.
+incompatible implementations of sort.)
 
-    printf '@ b\n- d\n? a\n~ c\n! e\n' | sort
-    printf '@ b\n- d\n? a\n~ c\n! e\n' | sort -d
-
-With BSD sort, the first of these commands print ASCIIbetical order and
-the second prints dictionary order. With GNU sort, both print dictionary
-order.
-
-How annoying.
+[^] https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021
+[^^] http://pubs.opengroup.org/onlinepubs/9699919799/

TODO

@@ -26,11 +26,14 @@ Windows
 Try running Urchin in Windows somehow. Interpreters include
 
 * CygWin (https://www.cygwin.com/)
+  * https://cygwin.com/setup-x86.exe
 * MSYS (http://mingw.org/wiki/msys)
 * GNU on Windows (https://github.com/bmatzelle/gow/wiki)
 * Git for Windows (https://git-scm.com/download/win)
+  * https://github.com/git-for-windows/git/releases/download/v2.7.2.windows.1/Git-2.7.2-32-bit.exe
 * win-bash (http://win-bash.sourceforge.net/)
 
+
 shall
 ----------
 Add shall to my NYC*BUG talk.
@@ -46,15 +49,17 @@ List some shell linters somewhere.
 * checkbashisms
 
 
-Set parallel and series
-----------
-I want to be able to make only some directories run in series.
-
-* Rename .urchin to .urchin_root.
-* Look for a .urchin_dir file.
-* If .urchin_dir contains "series", run that directory in series rather
-  than in parallel.
-
 Rename to something other than "test"?
 ----------
 Maybe wait until I have a use for this.
+
+More sort alternatives
+-----------
+awk
+  https://stackoverflow.com/questions/20250937/sorting-lines-in-a-file-alphabetically-using-awk-and-or-sed
+bash
+  https://stackoverflow.com/questions/7442417/how-to-sort-an-array-in-bash
+
+Alternatives
+--------------
+JSON.sh test suite

readme.md

@@ -43,18 +43,8 @@ Urchin depends on the following programs.
 * timeout
 * sort
 
-All of the above programs are usually included on base BSD installations.
-On GNU systems it should be sufficient to install the busybox package.
-
-Urchin uses sort to format its output. GNU sort (as of GNU coreutils version
-8.24) lacks the ability to sort in lexicographic order, and this feature is
-necessary for the output to look right. If your version of sort lacks this
-feature, Urchin will try to use one of the following tools for sorting.
-
-If no acceptable sorting program is available, Urchin will print a warning
-and use the incomplete sort that is installed on your system. This is not a
-big deal; if your test files all start with alphanumeric letters, the output
-should look fine.
+Vanilla installations of modern BSD and GNU systems usually include all
+of these programs.
 
 ## Install
 Urchin is contained in a single file, so you can install it by copying it to a
@@ -118,14 +108,36 @@ Files are only run if they are executable, and files beginning with `.` are
 ignored. Thus, fixtures and libraries can be included sloppily within the test
 directory tree. The test passes if the file exits 0; otherwise, it fails.
 
-Tests files and subdirectories are run in ASCIIbetical order within each
-directory; that is,
-urchin looks for files within a directory in the following manner.
+urchin looks for files within a directory in the following manner,
 
     for file in *; do
       do_something_with_test_file $file
     done
 
+so files are run in whatever order `*` produces. The order is
+configured in your environment, at least in
+[GNU systems](https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021).
+Other systems may ignore the locales configured in the environment and
+always produce ASCIIbetical order.
+
+Results are always printed in ASCIIbetical order, regardless of what
+order the tests ran in.
+
+Below you can see how the locale can affect the order.
+
+    $ printf '!c\n@a\n~b\n' | LC_COLLATE=C sort
+    !c
+    @a
+    ~b
+    $ printf '!c\n@a\n~b\n' | LC_COLLATE=en_US.UTF-8 sort
+    @a
+    ~b
+    !c
+    $ printf '!c\n@a\n~b\n' | sort -d
+    @a
+    ~b
+    !c
+
 ### Writing cross-shell compatibility tests for testing shell code
 
 While you could write your test scripts to explicitly invoke the functionality
@@ -179,14 +191,9 @@ shell.)
 
 Note that only test scripts that either have no shebang line at all or
 have shebang line `#!/bin/sh` are invoked with the specified shell.
-This allows non-shell test scripts or test scripts for specific
-shells to coexist with those whose invocation should be controlled by `-s`.
-
-To test with multiple shells in sequence, use something like:
-
-    for shell in sh bash ksh zsh; do
-      urchin -s $shell ./tests
-    done
+This allows non-shell test scripts or test scripts for other languages
+or for specific shells to coexist with those whose invocation should be
+controlled by `-s`.
 
 ## References
 

tests/.urchin_dir

@@ -0,0 +1 @@
+series

tests/Cross-shell testing/.test-run-by-specified-shell/With -s, a sh test should be invoked with sh.

@@ -1,6 +0,0 @@
-#!/bin/sh
-
-# This script should run with /bin/sh
-# regardless of whether -s or -n is passed.
-
-ps -o pid,comm,args | grep $$ | grep /bin/sh

tests/Cross-shell testing/.test-run-by-specified-shell/With -s, a sh test should be invoked with the special shell.

@@ -0,0 +1,6 @@
+#!/bin/sh
+
+# Assuming that urchin was invoked with `-s bash`,
+# this script should be being run with bash.
+
+ps -o pid,comm,args | grep $$ | grep .special-shell

tests/File discovery/Really ignore other files. (2)

@@ -1 +1 @@
-! $TEST_SHELL ../../urchin ../Flags/Urchin\ format|grep -- --color
+! $TEST_SHELL ../../urchin ../Flags/Urchin\ format|grep -- --pretty

tests/Flags/-e,--exit-on-fail

@@ -1,7 +1,7 @@
 #!/bin/sh
 set -e
 
-! $TEST_SHELL ../../urchin --run-in-series --exit-on-fail \
+! $TEST_SHELL ../../urchin -vv --run-in-series --exit-on-fail \
   ./.test_-e,--exit-on-fail > $tmp
 
 grep '1 should run.' $tmp

tests/Flags/.urchin-output-expectation

@@ -1,11 +1,11 @@
 
 ./
 > a
-✗ sh (1 second)
-  # This is stdout from a.
+F sh (1 second)
+  | This is stdout from a.
 ./
 > b
-✓ sh (1 second)
+. sh (1 second)
 ./
 > c
   (File is not executable.)

tests/Flags/.urchin-output-expectation-color

@@ -2,7 +2,7 @@
 ./
 > a
 ✗ sh (1 second)
-  # This is stdout from a.
+  | This is stdout from a.
 ./
 > b
 ✓ sh (1 second)

tests/Flags/Test Anything Protocol format

@@ -1,3 +1,3 @@
-$TEST_SHELL ../../urchin -s sh -t .testsuite/ |
+$TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
   sed -e 1d -e /second/d > $tmp
 diff $tmp .tap-output-expectation

tests/Flags/Urchin format with --color

@@ -1,3 +1,3 @@
-$TEST_SHELL ../../urchin --color -vv --shell sh .testsuite/ |
+$TEST_SHELL ../../urchin --pretty -vv --shell sh .testsuite/ |
   sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
 diff $tmp .urchin-output-expectation-color

tests/Internals/has_shebang_line/.run

@@ -0,0 +1,3 @@
+echo "$1" > $tmp
+TESTING_URCHIN_INTERNALS=true . ../../../urchin
+has_shebang_line $tmp

tests/Internals/has_shebang_line/bash

@@ -0,0 +1 @@
+./.run '#!/bin/bash'

tests/Internals/has_shebang_line/empty

@@ -0,0 +1 @@
+! ./.run ''

tests/Internals/has_shebang_line/empty-line

@@ -0,0 +1,5 @@
+! ./.run '
+
+
+
+'

tests/Internals/has_shebang_line/env

@@ -0,0 +1 @@
+./.run '#!/usr/bin/env true'

tests/Internals/has_shebang_line/setup

@@ -0,0 +1 @@
+export tmp=$(mktemp)

tests/Internals/has_shebang_line/sh

@@ -0,0 +1 @@
+! ./.run '#!/bin/sh'

tests/Internals/has_shebang_line/teardown

@@ -0,0 +1 @@
+rm -R "$tmp"

tests/Internals/sort_python

@@ -1,4 +0,0 @@
-set -e
-TESTING_URCHIN_INTERNALS=true . ../../urchin
-unsorted='@ b\n- d\n? a\n~ c\n! e\n'
-test $(printf "${unsorted}" | sort_python | cut -d\  -f2|tr -d '\n') = edabc

tests/Setup and Teardown/.test/setup count a

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+[ $(grep -c 'setup has run' $log) -eq '1' ]

tests/Setup and Teardown/.test/setup count b

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+[ $(grep -c 'setup has run' $log) -eq '2' ]

tests/Setup and Teardown/.test/setup has run twice a

@@ -1,3 +0,0 @@
-#!/bin/sh
-
-[ $(grep -c 'setup has run' $log) -gt '2' ]

tests/Setup and Teardown/.test/setup has run twice b

@@ -1,3 +0,0 @@
-#!/bin/sh
-
-[ $(grep -c 'setup has run' $log) -gt '2' ]

urchin

@@ -70,7 +70,7 @@ validate_test_arg() {
   root="$(urchin_root "${1}")"
   if ! {
     basename "$(fullpath "${root}")" |
-    grep -i 'test' > /dev/null || "${force}"
+    grep -qi 'test' || "${force}"
   }; then
     echo 'The root directory of the tests that you are running urchin on
   does not contain the word "test", so I am not running,
@@ -80,13 +80,6 @@ validate_test_arg() {
   fi
 }
 
-sort_python() {
-  python -c 'import sys
-for line in sorted(sys.stdin.readlines()):
-    sys.stdout.write(line)
-'
-}
-
 # All temporary files go here
 urchin_tmp=$(mktemp -d)
 > "${urchin_tmp}/log"
@@ -124,7 +117,7 @@ urchin_root() {
 
   abscurrent="$(fullpath "${1}")"
   if test "${abscurrent}" = / ||
-    basename "${abscurrent}" | grep '^\.' > /dev/null; then
+    basename "${abscurrent}" | grep -q '^\.' ; then
     # Stop traversing upwards at / and at hidden directories.
     if test -d "${orig}"; then
       echo "${orig}"
@@ -136,7 +129,7 @@ urchin_root() {
     return 1
   elif test -f "${current}"; then
     urchin_root "$(dirname -- "${current}")" "${orig}"
-  elif test -f "${current}"/.urchin; then
+  elif test -f "${current}"/.urchin_root; then
     remove_trailing_slash "${current}"
   else
     urchin_root "${current}"/.. "${orig}"
@@ -144,7 +137,7 @@ urchin_root() {
 }
 
 # Urchin version number
-VERSION=0.1.0-rc1
+VERSION=0.1.0-rc2
 
 indent() {
   level="${1}"
@@ -165,8 +158,8 @@ recurse() {
     fi
   done
 
-  if echo "${requested_path}" | grep "^${potential_test}" > /dev/null ||
-     echo "${potential_test}" | grep "^${requested_path}" > /dev/null; then
+  if echo "${requested_path}" | grep -q "^${potential_test}" ||
+     echo "${potential_test}" | grep -q "^${requested_path}" ; then
     if test "$(dirname "${potential_test}")" = \
             "$(dirname "${requested_path}")" &&
        test "${potential_test}" != "${requested_path}"; then
@@ -182,6 +175,12 @@ recurse() {
     if [ -d "${potential_test}" ]; then
     (
       cd -- "${potential_test}"
+      if test -f .urchin_dir && grep -q series .urchin_dir ; then
+        run_in_series_dir=true
+      else
+        run_in_series_dir=false
+      fi
+
       if test -f setup_dir; then . ./setup_dir; fi
 
       for test in *; do
@@ -190,19 +189,10 @@ recurse() {
           break
         fi
 
-        (
-          if test -f setup; then . ./setup; fi
-          if recurse "${requested_path}" "${test}" "${cycle_shell}" \
-            "${TEST_SHELL}"; then
-            exit_code=0
-          else
-            exit_code="${?}"
-          fi
-          if test -f teardown; then . ./teardown; fi
-          exit "${exit_code}"
-        ) &
+        recurse "${requested_path}" "${test}" "${cycle_shell}" \
+          "${TEST_SHELL}" &
 
-        if "${run_in_series}"; then
+        if "${run_in_series}" || "${run_in_series_dir}"; then
           if wait "${!}"; then exit_code=0; else exit_code="${?}"; fi
           if "${exit_on_not_ok}" && test "${exit_code}" -ne 0; then
             if test -f teardown_dir; then . ./teardown_dir; fi
@@ -247,7 +237,7 @@ recurse() {
           set -e
           finish=$(date +%s)
 
-          if test -f teardown; then . ./setup; fi
+          if test -f teardown; then . ./teardown; fi
 
           if [ "${exit_code}" -eq 0 ]; then
             result=ok
@@ -305,7 +295,7 @@ report_outcome() {
   # Use a temporary file rather than a pipe because a pipe starts a sub-shell
   # and thus makes the above variables local.
   sorted_log_file=$(mktemp)
-  cat "${log_file}" | "${sort}" > "${sorted_log_file}"
+  cat "${log_file}" | LC_ALL=POSIX sort > "${sorted_log_file}"
 
   while read line; do
     abspath=$(echo "${line}" | cut -f1)
@@ -356,15 +346,15 @@ report_outcome() {
           printf "$(dirname -- "${path}")/\n> $(basename -- "${path}")\n"
         fi
       }
+
       case "${result}" in
         ok)
           if "${print_ok}"; then
             header
-            # On success, print a green '✓'
             if "${print_in_color}"; then
-              printf '\033[32m✓ \033[0m'
+              printf "\033[32m${success_mark} \033[0m"
             else
-              printf '✓ '
+              printf "${success_mark} "
             fi
             echo "${the_shell} ("${file_elapsed}" $(plural second $file_elapsed))"
           fi
@@ -372,11 +362,10 @@ report_outcome() {
         not_ok)
           if "${print_not_ok}"; then
             header
-            # On not_ok, print a red '✗'
             if "${print_in_color}"; then
-              printf '\033[31m✗ \033[0m'
+              printf "\033[31m${fail_mark} \033[0m"
             else
-              printf '✗ '
+              printf "${fail_mark} "
             fi
             echo "${the_shell} ("${file_elapsed}" $(plural second $file_elapsed))"
           fi
@@ -394,7 +383,7 @@ report_outcome() {
       esac
       if { test "${result}" = not_ok && "${print_not_ok_stdout}"; } ||
          { test "${result}" = ok && "${print_ok_stdout}"; }; then
-        sed 's/^/  # /' "$(stdout_file "${abspath}" "${the_shell}")"
+        sed 's/^/  | /' "$(stdout_file "${abspath}" "${the_shell}")"
       fi
     fi
 
@@ -424,10 +413,10 @@ report_outcome() {
 }
 
 has_shebang_line() {
-  head -n 1 "${1}" | grep -qE '^#!'
+  head -n 1 "${1}" | grep -v '^#!/bin/sh$' | grep -q '^#!'
 }
 
-USAGE="usage: $0 [<options>] <test directory>"
+USAGE="usage: $0 [options]... [test file or directory]..."
 
 urchin_help() {
   cat <<EOF
@@ -442,8 +431,8 @@ particular test file once per shell.
 On each run,
 
 1. The TEST_SHELL environment variable is set to the particular shell.
-2. If the test file lacks a shebang line, the test script is also
-   executed in that shell.
+2. If the test file lacks a shebang line or has a shebang line of
+   "#!/bin/sh", the test script is also executed in that shell.
 
 The following flags affect how this multiple-shell testing is handled.
 
@@ -472,7 +461,7 @@ These options affect how results are formatted. Options -q, and -v
 have no effect when combined with --tap. -vv, -vvv, and -vvvv do have
 effect when combined with --tap.
 
--c, --color              Print results in color.
+-p, --pretty             Print results in color and with fancy symbols.
 -t, --tap                Format output in Test Anything Protocol (TAP)
 
 -q, --quiet              Print nothing to stdout;
@@ -546,7 +535,7 @@ main() {
             }
 
             validate_strings "${shell_for_sh_tests}" 'Shell paths'
-            if echo "${shell_for_sh_tests}" | grep \  > /dev/null; then
+            if echo "${shell_for_sh_tests}" | grep -q \ ; then
               echo "Warning: It is best if shell paths contain no spaces so that
   you don't need to quote the TEST_SHELL variable." > /dev/stderr
             fi
@@ -556,7 +545,7 @@ main() {
           -n|--disable-cycling) cycle_shell=false;; 
           -t|--tap) tap_format=true;;
           -T|--timeout) shift; urchin_timeout="${1}" ;;
-          -c|--color) print_in_color=true;;
+          -p|--pretty) print_in_color=true;;
 
           -q|--quiet)    print_not_ok=false
                          print_margins=false;; 
@@ -581,6 +570,14 @@ main() {
       shift
   done
 
+  if $print_in_color; then
+    success_mark=✓
+    fail_mark=✗
+  else
+    success_mark=.
+    fail_mark=F
+  fi
+
   # -------------------- VALIDATE INPUT -------------------- #
   if ! "${cycle_shell}" && test -f "${shell_list}"; then
     echo "The -n/--disable-cycling and -s/--shell options clash with each other." >&2
@@ -600,22 +597,6 @@ main() {
     fi
   fi
 
-  # Warn about strange sort commands
-  weird_string='@ b\n- d\n? a\n~ c\n! e\n'
-  sort_result="$(printf "${weird_string}" | sort | cut -d\  -f2 | tr -d '\n')"
-  if test "${sort_result}" = edacb; then
-    sort=sort
-  else
-    if which python > /dev/null; then
-      sort=sort_python
-    else
-      echo 'Your version of sort sorts in dictionary order (-d) by default.
-    Depending on how you name your tests, your Urchin output may look strange.
-    If this is a problem, install BusyBox or BSD coreutils.' >&2
-      sort=sort
-    fi
-  fi
-
   if test -n "${urchin_timeout}"; then
     # Choose the timeout command
     if timeout -t 0 true; then
@@ -623,7 +604,7 @@ main() {
     elif timeout 0 true; then
       TIMEOUT="timeout ${urchin_timeout}"
     else
-      echo I couldn\'t figure out to use your version of timeout >&2
+      echo I couldn\'t figure out how to use your version of timeout >&2
       urchin_exit 1
     fi
   fi