Syntax highlight in readme.md

HISTORY

@@ -1,10 +1,8 @@
 HISTORY
 =======
 
-Version 0.1.0
+Version 0.2.0 (unstable)
 ---------------------
-This release includes breaking changes.
-
 ### Cross-OS testing
 I have started testing Urchin across multiple operating systems.
 This gives access to more shells, as some shels are easier to install on
@@ -13,6 +11,10 @@ certain operating systems.
 With this cross-OS test suite, I have extended support to more shells.
 A later version of Urchin could include a remote testing feature.
 
+Version 0.1.0 (stable)
+---------------------
+This release includes breaking changes.
+
 ### Test root directory
 We introduce a concept of the root directory of a test suite.
 Such a concept is important in case you want to run subsets of your

TODO

@@ -72,6 +72,114 @@ Setup for other environments includes the following.
 * `touch .zshrc`
 * Copy urchin and tests
 
+
+Fixtures
+------------
+I want to change the way that fixtures are done.
+
+Instead of using setup, teardown, &c., use ordinary programs from within
+your tests. For example.
+
+    # tests/.fixtures/tmp-dir
+    tmp=$(mktemp -d)
+    cd $tmp
+    @$
+    code=$?
+    cd /
+    rm -Rf $tmp
+    exit $code
+
+    # tests/blah
+    ../.fixtures/tmp-dir 'blah blah blah'
+
+It's best if I can wrap a bunch of commands in braces or paratheses
+rather than just one command. Is there a nice way to do that?
+
+Once I have this new way, I guess I might as well keep the old way.
+I think the setup, teardown thing can be easier if you only have simple
+fixtures. And since I'm going to keep it, I'm going to add another one.
+
+* setup_dir runs once for the present directory.
+* setup_children runs once for each child.
+* setup_file runs once for each file descendent.
+
+The present `setup` is renamed to `setup_children`, and the new
+`setup_file` runs on each file (not directory) that is a child,
+grandchild, great-grandchild, and so on.
+
+Dependency checking
+----------------------
+You might want to skip tests based on dependencies. Currently you can
+conditionally skip tests one at a time by exiting with code 3. I want to
+be able to skip an entire directory.
+
+So we add a new magic file called `dep`. If it exists, it is run before
+everything else in the directory.
+
+* If it exits with code 0, tests continue as if dep did not exist.
+* If it exits with code 3, all tests in the directory are marked as
+  skipped.
+* If it exits with code 1, all tests in the directory are marked as
+  failed. To make the implementation easier, I'll probably treat the
+  directory as a single test in this case.
+
+A note on magic files
+-------------------------
+It is nice to have access to things like setup and dep (magic files)
+once in a while, but you need to be doing rather substantial testing
+before they make your test suite simpler; the documentation should
+strongly recommend writing your tests without magic files and then
+refactoring and only then considering moving things to magic files.
+
+Remote testing
+----------------
+In order to test Urchin across multiple operating systems, I have
+already added tests in Urchin's test suite that run Urchin tests in
+remote servers. I would like to move this to Urchin itself so that
+Urchin can test other things on remote servers.
+
+Urchin's output presently looks like this.
+
+    Cycling with the following shells: sh bash dash mksh zsh 
+    Running tests at 2016-04-07T12:33:49
+
+    Flags/
+    > --timeout output
+    . bash (0 seconds)
+    . dash (0 seconds)
+    . mksh (0 seconds)
+    . sh (0 seconds)
+    . zsh (0 seconds)
+
+    Done, took 1 second.
+    5 tests passed.
+    0 tests skipped.
+    0 tests failed.
+
+After the change, the output should look like this.
+
+    Cycling with the following shells: sh dash mksh
+    Running tests at 2016-04-07T12:33:49
+
+    Flags/
+    > --timeout output
+    . dash on localhost (0 seconds)
+    . dash on localhost:8080 (0 seconds)
+    . dash on tlevine@hpux.polarhome.com (0 seconds)
+    . mksh on localhost (0 seconds)
+    . mksh on tlevine@hpux.polarhome.com (0 seconds)
+    . sh on localhost (0 seconds)
+    . sh on localhost:8080 (0 seconds)
+    . sh on tlevine@hpux.polarhome.com (0 seconds)
+
+    Done, took 1 second.
+    8 tests passed.
+    0 tests skipped.
+    0 tests failed.
+
+This is just how the output should look; the tests run in whatever order
+makes sense.
+
 Bugs
 -------
 
@@ -114,3 +222,73 @@ date
     tlevine@hpux64$ ./urchin tests/ -n -vv
     date: bad format character - s
 
+So I need a portable seconds-from epoch
+
+I also need to handle when no arguments are passed to urchin.
+
+Exit code is wrong for which on HP-UX
+
+## `$(...)`
+Solaris doesn't support `$(...)`; you need `\`...\`` instead.
+
+    tlevine@solaris$ ./urchin --run-in-series tests/Errors/
+    ./urchin: syntax error at line 84: `tmp=$' unexpected
+
+I use this a lot.
+
+    $ grep -c '\$(' urchin 
+    52
+
+Darn
+
+
+
+
+Update tests to support
+
+* md5
+* rsync
+* mktemp
+* epoch
+* Report cycling by default
+* New format for reporting cycling
+
+
+
+
+Support systems without rsync
+
+
+BSD mktemp
+
+    | NetBSD 6.1.3 
+    | Welcome to NetBSD ...member of polarhome.com realm
+    | 
+    | Usage: mktemp [-dqu] [-p <tmpdir>] {-t prefix | template ...}
+    | mkdir: : No such file or directory
+    | ./urchin: cannot create /log: permission denied
+
+
+NetBSD
+
+    md5: unknown option -- q
+    usage: cksum [-n] [-a algorithm [-ptx] [-s string]] [-o 1|2]
+                 [file ... | -c [-w] [sumfile]]
+
+
+
+
+Things I can use to make things better
+------------------------
+${x##*blah}
+$IFS and set --
+Redirection, especiall <<-
+Maybe fifo
+for x in "$@"
+until
+readonly
+getopts
+
+Variable assignments specified with special built-in utilities remain in
+effect after the built-in completes; this shall not be the case with a
+regular built-in or other utility.

cross-os-tests/.run

@@ -8,6 +8,7 @@ fi
 urchin_dir=.urchin-cross-shell-test
 
 rsync --archive -e "ssh ${flags}" $RSYNC_FLAGS \
-  ../urchin ../tests "${hostname}":"${urchin_dir}"
+  ../urchin ../tests "${hostname}":"${urchin_dir}" ||
+  scp -r ${flags} ../urchin ../tests "${hostname}":"${urchin_dir}"
 ssh "${hostname}" ${flags} \
   "cd ${urchin_dir} && ./urchin --run-in-series tests"

docs/execution-flow

@@ -0,0 +1,109 @@
+Here I discuss Urchin's general execution flow and how it is handled
+specifically when tests are run on remote environments.
+
+
+Steps of an Urchin run
+----------------------
+When Urchin runs a directory of files, it goes through the following
+steps.
+
+1. Head
+2. Test
+3. Foot
+4. Reporting
+
+Urchin stores files in a temporary directory, creating a new directory
+on each invocation. The directory contains these things.
+
+* head (file)
+* test (file)
+* foot (file)
+* stdout (directory)
+
+When run on remotes, the temporary directory corresponding to the local
+master process additionally has these files.
+
+* remote-test
+
+Messages from the head, test, and foot steps go in the corresponding
+files. In the head and foot phases, messages are just simple prints.
+Messages from the test phase always correspond to a particular test
+file, and they are written to the test file in a delimiter-separated
+format.
+
+Stdout and stderr from test runs are written to files in the stdout
+directory, one file per test file per shell that the file is run in.
+
+The reporting phase
+----------------------
+In most cases Urchin begins printing to the screen only during the
+reporting phase. The only case where anything is printed beforehand is
+when Urchin is run with -vvvv; that sets "+x", so the commands are
+printed as they run, though all other output is still suppressed.
+
+Test results are reported in the reporting phase. Four output formats
+are available.
+
+1. Urchin's human-readable format (default)
+2. Test Anything Protocol
+3. Delimiter-separated values (used internally)
+4. Remote Urchin worker output
+
+Most of the output is generated based on the delimiter-separated values
+in the test log file. The first two formats also include stdout and
+stderr from the tests, depending on verbosity level flags; when it needs
+these, Urchin reads them from appropriate files in the temporary
+directory.
+
+I could discuss the further details of each format elsewhere.
+
+Remotes
+----------------------
+When Urchin runs tests on a remote, it copies tests to the remote and
+then calls Urchin on the remote with "--format=remote". This specifies
+the following.
+
+* The temporary directory should be kept, rather than deleted, after
+  Urchin runs.
+* The path of the temporary directory should be printed as output.
+* No other output should be printed to stdout.
+
+After the remote Urchin finishes running, the local urchin downloads
+the remote Urchin's test log file from the temporary directory.
+It modifies the file to include the remote's name and then concatenates
+the result to the "remote-test" file in the local temporary directory.
+For example, the file from the remote might look like this,
+
+    :sh:Counting tests/.test/faila:0:not_ok
+
+and the result might look like this.
+
+    nsa:sh:Counting tests/.test/faila:0:not_ok
+
+This gets processed in the reporting step like usual, according to
+whatever format is specified. Instead of printing just "sh" as the
+environment in which the particular test was run, the report will print
+"sh on nsa".
+
+When it needs the stdout files, it prints them over ssh.
+
+New flags
+----------
+In making this remotes feature, I wound up adding some others.
+
+-r, --remote      SSH host to use as a remote
+-F, --format      Output format, one of "urchin", "tap", "dsv", "remote"
+
+Urchin runs only locally by default. If you pass at least one --remote
+flag, Urchin runs tests only on the specified remotes; it can't run both
+locally and remotely in the same run. If you want to do that, you could
+wait until I add that feature, or you can add "localhost" as a remote.
+
+Settings that I'm thinking about
+
+* Port for rsync/ssh
+* SSH protocol version
+* --rsync-path
+
+Can those all be set in ssh_config? Probably not --rsync-path, but
+I guess I could just fix it on the remote.

environments/nixos-container/configuration.nix

@@ -0,0 +1,39 @@
+# A NixOS container to protect against accidental fork bombs
+#
+# Put this in /var/lib/containers/test/etc/nixos/configuration.nix
+# See https://nixos.org/wiki/NixOS:Containers
+{ config, lib, pkgs, ... }:
+   
+with lib;
+ 
+{ boot.isContainer = true;
+  networking.hostName = mkDefault "urchin";
+  networking.useDHCP = false;
+
+  environment.systemPackages = with pkgs; [
+    # Urchin
+    bash dash mksh zsh
+    busybox
+
+    # Other
+    vim git rsync tmux
+  ];
+  security.pam.loginLimits = [
+    # Prevent accidental fork bombs.
+    { domain = "*"; item = "nproc"; type = "hard"; value = "200"; }
+  ];
+  services.openssh = {
+    enable = true;
+    passwordAuthentication = false;
+  };
+  users.extraUsers.user = {
+    name = "tlevine";
+    uid = 1000;
+    isNormalUser = true;
+    home = "/home/tlevine";
+    extraGroups = [ "users" "wheel" ];
+    openssh.authorizedKeys.keys = [
+      "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGvQyzr42/96acUTUedaeM2ee+DMt9bkxeurdeXji9sNE10MjjAUFtxPmSI8/BUZW2/a9ByblfaJEI+H+kFVPjVr+QGKXZluxcFMj2BLbH53fi9xLgoQRjb2aAXutb2Bp74/E8R1K+CuFfRRGQ5Spdnv44SLt04D6JbBLcLIcWTpQ4v5RaYr2U27jfiF9z0m+/opxvowEy2gnqlEXFxFk8jZHT4K0uLWm2ENjT6OpyOx8hWcKeAN2vRVRex3pJfSzswn0LpuCrM1rUZ4DRE+FABi8N21Q3MBaMRkwnZPwaZwKzv06q8bu23jYTqK5BrUPtOXeeVuroQXMc12H/6/Nh laptop"
+    ];
+  };
+}

environments/nixos-container/run.sh

@@ -0,0 +1,36 @@
+#!/bin/sh
+set -e
+
+# Create the container.
+if ! nixos-container list | grep ^urchin$ > /dev/null; then
+  sudo nixos-container create urchin
+fi
+
+# Configure the container.
+sudo cp configuration.nix \
+  /var/lib/containers/urchin/etc/nixos/configuration.nix
+sudo nixos-container update urchin
+sudo nixos-container start urchin
+
+# Create the git repository.
+host="tlevine@$(nixos-container show-ip urchin)"
+ssh "${host}" 'if mkdir urchin 2> /dev/null; then
+    cd urchin
+    git init
+    git config --add receive.denyCurrentBranch ignore
+  fi
+'
+
+# Push to the git repository
+git push "${host}":urchin
+
+# Print information
+echo "Log in:
+
+  ssh ${host}
+
+Add git remote
+
+  git remote add ${host} container
+
+"

readme.md

@@ -21,8 +21,10 @@ is like. Clone the repository
 
 Run the tests
 
-    cd urchin
-    ./urchin tests
+```sh
+cd urchin
+./urchin tests
+```
 
 ## Dependencies
 Urchin depends on the following programs.
@@ -50,9 +52,11 @@ of these programs.
 Urchin is contained in a single file, so you can install it by copying it to a
 directory in your `PATH`. For example, you can run the following as root.
 
-    cd /usr/local/bin
-    wget https://raw.githubusercontent.com/tlevine/urchin/v0.1.0-rc3/urchin
-    chmod +x urchin
+```sh
+cd /usr/local/bin
+wget https://raw.githubusercontent.com/tlevine/urchin/v0.1.0-rc3/urchin
+chmod +x urchin
+```
 
 Urchin can be installed with npm too.
 
@@ -110,9 +114,11 @@ directory tree. The test passes if the file exits 0; otherwise, it fails.
 
 urchin looks for files within a directory in the following manner,
 
-    for file in *; do
-      do_something_with_test_file $file
-    done
+```sh
+for file in *; do
+  do_something_with_test_file $file
+done
+```
 
 so files are run in whatever order `*` produces. The order is
 configured in your environment, at least in
@@ -125,18 +131,20 @@ order the tests ran in.
 
 Below you can see how the locale can affect the order.
 
-    $ printf '!c\n@a\n~b\n' | LC_COLLATE=C sort
-    !c
-    @a
-    ~b
-    $ printf '!c\n@a\n~b\n' | LC_COLLATE=en_US.UTF-8 sort
-    @a
-    ~b
-    !c
-    $ printf '!c\n@a\n~b\n' | sort -d
-    @a
-    ~b
-    !c
+```sh
+$ printf '!c\n@a\n~b\n' | LC_COLLATE=C sort
+!c
+@a
+~b
+$ printf '!c\n@a\n~b\n' | LC_COLLATE=en_US.UTF-8 sort
+@a
+~b
+!c
+$ printf '!c\n@a\n~b\n' | sort -d
+@a
+~b
+!c
+```
 
 ### Writing cross-shell compatibility tests for testing shell code
 

tests/Flags/Test Anything Protocol format

@@ -1,3 +1,3 @@
 $TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
-  sed -e 1d -e /second/d > $tmp
+  sed -e 1,2\ d -e /second/d > $tmp
 diff $tmp .tap-output-expectation

tests/Flags/Test Anything Protocol format 2

@@ -0,0 +1,5 @@
+lines=$(
+  $TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
+    tee $tmp | grep -v '^#' | wc -l)
+cat $tmp
+test $lines -eq 4

tests/Flags/Urchin format

@@ -1,3 +1,3 @@
 $TEST_SHELL ../../urchin -vv -s sh .testsuite/ |
-  sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
+  sed -e 1,2\ d -e 's/. seconds\?/1 second/' > $tmp
 diff $tmp .urchin-output-expectation

tests/Flags/Urchin format with --color

@@ -1,3 +1,3 @@
 $TEST_SHELL ../../urchin --pretty -vv --shell sh .testsuite/ |
-  sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
+  sed -e 1,2\ d -e 's/. seconds\?/1 second/' > $tmp
 diff $tmp .urchin-output-expectation-color

tests/Internals/has_shebang_line/.run

@@ -1,3 +1,3 @@
 echo "$1" > $tmp
-TESTING_URCHIN_INTERNALS=true . ../../../urchin
+NO_MAIN= . ../../../urchin
 has_shebang_line $tmp