GitHub-only note about repository address change

because i sometimes run them on weak computers

HISTORY

@@ -1,7 +1,17 @@
 HISTORY
 =======
 
-Version 0.1.0-rc1
+Version 0.2.0 (unstable)
+---------------------
+### Cross-OS testing
+I have started testing Urchin across multiple operating systems.
+This gives access to more shells, as some shels are easier to install on
+certain operating systems.
+
+With this cross-OS test suite, I have extended support to more shells.
+A later version of Urchin could include a remote testing feature.
+
+Version 0.1.0 (stable)
 ---------------------
 This release includes breaking changes.
 
@@ -72,7 +82,7 @@ might look for the dependency and then skip if it does not see the dependency.
 It might look like this.
 
     #!/bin/sh
-    if which inkscape; then
+    if ! which inkscape; then
       exit 3 # status code 3 for skip
     fi
     inkscape blah blah ...
@@ -171,10 +181,6 @@ set as an environment variable, and the latter was set with the -s flag..
 Urchin now uses the -s flag for both of these settings, and it mostly ignores
 the exported TEST_SHELL variable.
 
-Urchin also inspects the shebang line differently. Previously, Urchin would
-vary the shells with which a test is run if the shebang line either was absent
-or was #!/bin/sh. Now it varies the shell only if the shebang line is absent.
-
 If you pass -n/--disable-cycling, Urchin will invoke tests ordinarily and will
 only set the TEST_SHELL variable if it does not exist. If the TEST_SHELL
 variable is absent, it will be set to /bin/sh.

Makefile

@@ -0,0 +1,8 @@
+.PHONY: test install
+
+test:
+	./urchin tests
+	./urchin -s sh -v ./cross-os-tests
+
+install:
+	cp ./urchin /usr/bin

TODO

@@ -63,3 +63,232 @@ bash
 Alternatives
 --------------
 JSON.sh test suite
+
+Running in multiple environments
+-----------------------------------
+Setup for other environments includes the following.
+
+* Installing packages
+* `touch .zshrc`
+* Copy urchin and tests
+
+
+Fixtures
+------------
+I want to change the way that fixtures are done.
+
+Instead of using setup, teardown, &c., use ordinary programs from within
+your tests. For example.
+
+    # tests/.fixtures/tmp-dir
+    tmp=$(mktemp -d)
+    cd $tmp
+    @$
+    code=$?
+    cd /
+    rm -Rf $tmp
+    exit $code
+
+    # tests/blah
+    ../.fixtures/tmp-dir 'blah blah blah'
+
+It's best if I can wrap a bunch of commands in braces or paratheses
+rather than just one command. Is there a nice way to do that?
+
+Once I have this new way, I guess I might as well keep the old way.
+I think the setup, teardown thing can be easier if you only have simple
+fixtures. And since I'm going to keep it, I'm going to add another one.
+
+* setup_dir runs once for the present directory.
+* setup_children runs once for each child.
+* setup_file runs once for each file descendent.
+
+The present `setup` is renamed to `setup_children`, and the new
+`setup_file` runs on each file (not directory) that is a child,
+grandchild, great-grandchild, and so on.
+
+Dependency checking
+----------------------
+You might want to skip tests based on dependencies. Currently you can
+conditionally skip tests one at a time by exiting with code 3. I want to
+be able to skip an entire directory.
+
+So we add a new magic file called `dep`. If it exists, it is run before
+everything else in the directory.
+
+* If it exits with code 0, tests continue as if dep did not exist.
+* If it exits with code 3, all tests in the directory are marked as
+  skipped.
+* If it exits with code 1, all tests in the directory are marked as
+  failed. To make the implementation easier, I'll probably treat the
+  directory as a single test in this case.
+
+A note on magic files
+-------------------------
+It is nice to have access to things like setup and dep (magic files)
+once in a while, but you need to be doing rather substantial testing
+before they make your test suite simpler; the documentation should
+strongly recommend writing your tests without magic files and then
+refactoring and only then considering moving things to magic files.
+
+Remote testing
+----------------
+In order to test Urchin across multiple operating systems, I have
+already added tests in Urchin's test suite that run Urchin tests in
+remote servers. I would like to move this to Urchin itself so that
+Urchin can test other things on remote servers.
+
+Urchin's output presently looks like this.
+
+    Cycling with the following shells: sh bash dash mksh zsh 
+    Running tests at 2016-04-07T12:33:49
+
+    Flags/
+    > --timeout output
+    . bash (0 seconds)
+    . dash (0 seconds)
+    . mksh (0 seconds)
+    . sh (0 seconds)
+    . zsh (0 seconds)
+
+    Done, took 1 second.
+    5 tests passed.
+    0 tests skipped.
+    0 tests failed.
+
+After the change, the output should look like this.
+
+    Cycling with the following shells: sh dash mksh
+    Running tests at 2016-04-07T12:33:49
+
+    Flags/
+    > --timeout output
+    . dash on localhost (0 seconds)
+    . dash on localhost:8080 (0 seconds)
+    . dash on tlevine@hpux.polarhome.com (0 seconds)
+    . mksh on localhost (0 seconds)
+    . mksh on tlevine@hpux.polarhome.com (0 seconds)
+    . sh on localhost (0 seconds)
+    . sh on localhost:8080 (0 seconds)
+    . sh on tlevine@hpux.polarhome.com (0 seconds)
+
+    Done, took 1 second.
+    8 tests passed.
+    0 tests skipped.
+    0 tests failed.
+
+This is just how the output should look; the tests run in whatever order
+makes sense.
+
+Bugs
+-------
+
+Both md5sum and md5 should be supported.
+
+Trouble logging in to hpux, irix, miros, netbsd, tru64, qnx, ....
+
+    $ rsync -e 'ssh -p 785' urchin tlevine@hpux.polarhome.com:.blah
+    HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
+    Welcome to HPUX/PA... member of polarhome.com realm
+
+    bash: rsync: command not found
+    rsync: connection unexpectedly closed (0 bytes received so far) [sender]
+    rsync error: remote command not found (code 127) at io.c(226)
+    [sender=3.1.1]
+
+
+OpenIndiana grep does not support -q
+
+I get `/urchin: syntax error at line 84: \`}' unexpected` on
+unixware and solaris.
+
+mktemp
+
+    > tlevine@hpux.polarhome.com -p 785
+    F sh (8 seconds)
+      | 
+      | HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
+      | Welcome to HPUX/PA... member of polarhome.com realm
+      | 
+      | 
+      | HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
+      | Welcome to HPUX/PA... member of polarhome.com realm
+      | 
+      | mktemp: option requires an argument -- d
+      | ./urchin[96]: /tmp/tlevinea21441/log: Cannot create the specified file.
+
+date
+
+    tlevine@hpux64$ ./urchin tests/ -n -vv
+    date: bad format character - s
+
+So I need a portable seconds-from epoch
+
+I also need to handle when no arguments are passed to urchin.
+
+Exit code is wrong for which on HP-UX
+
+## `$(...)`
+Solaris doesn't support `$(...)`; you need `\`...\`` instead.
+
+    tlevine@solaris$ ./urchin --run-in-series tests/Errors/
+    ./urchin: syntax error at line 84: `tmp=$' unexpected
+
+I use this a lot.
+
+    $ grep -c '\$(' urchin 
+    52
+
+Darn
+
+
+
+
+Update tests to support
+
+* md5
+* rsync
+* mktemp
+* epoch
+* Report cycling by default
+* New format for reporting cycling
+
+
+
+
+Support systems without rsync
+
+
+BSD mktemp
+
+    | NetBSD 6.1.3 
+    | Welcome to NetBSD ...member of polarhome.com realm
+    | 
+    | Usage: mktemp [-dqu] [-p <tmpdir>] {-t prefix | template ...}
+    | mkdir: : No such file or directory
+    | ./urchin: cannot create /log: permission denied
+
+
+NetBSD
+
+    md5: unknown option -- q
+    usage: cksum [-n] [-a algorithm [-ptx] [-s string]] [-o 1|2]
+                 [file ... | -c [-w] [sumfile]]
+
+
+
+
+Things I can use to make things better
+------------------------
+${x##*blah}
+$IFS and set --
+Redirection, especiall <<-
+Maybe fifo
+for x in "$@"
+until
+readonly
+getopts
+
+Variable assignments specified with special built-in utilities remain in
+effect after the built-in completes; this shall not be the case with a
+regular built-in or other utility.

cross-os-tests/.run

@@ -0,0 +1,14 @@
+params="$(basename "${0}")"
+
+hostname="$(echo "${params}" | cut -d\  -f1)"
+if echo "${params}" | grep -q \ ; then
+  flags="$(echo "${params}" | cut -d\  -f2-)"
+fi
+
+urchin_dir=.urchin-cross-shell-test
+
+rsync --archive -e "ssh ${flags}" $RSYNC_FLAGS \
+  ../urchin ../tests "${hostname}":"${urchin_dir}" ||
+  scp -r ${flags} ../urchin ../tests "${hostname}":"${urchin_dir}"
+ssh "${hostname}" ${flags} \
+  "cd ${urchin_dir} && ./urchin --run-in-series tests"

cross-os-tests/tlevine@centos.polarhome.com -p 925

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@dada.pink

@@ -0,0 +1,3 @@
+#!/bin/sh
+# apt-get install bash dash ksh posh pdksh mksh yash zsh
+. ./.run

cross-os-tests/tlevine@debian-ppc.polarhome.com -p 625

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@debian.polarhome.com -p 755

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@dragonfly.polarhome.com -p 915

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@freebsd.polarhome.com -p 715

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@hpux-ia64.polarhome.com -p 965

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@hpux.polarhome.com -p 785

@@ -0,0 +1,3 @@
+#!/bin/sh
+RSYNC_FLAGS='--rsync-path=/usr/local/bin/rsync'
+. ./.run

cross-os-tests/tlevine@irix.polarhome.com -p 825

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@miros.polarhome.com -p 935

@@ -0,0 +1,3 @@
+#!/bin/sh
+# SSH public key needs to be in ~/.etc/ssh/authorized_keys
+. ./.run

cross-os-tests/tlevine@netbsd.polarhome.com -p 745

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@openbsd.polarhome.com -p 735

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@openindiana.polarhome.com -p 845

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@pidora.polarhome.com -p 615

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@qnx.polarhome.com -p 815

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@raspbian.polarhome.com -p 975

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@redhat.polarhome.com -p 795

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@scosysv.polarhome.com -p 895

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@solaris-x86.polarhome.com -p 865

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@solaris.polarhome.com -p 725

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@tru64.polarhome.com -p 835

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@ubuntu.polarhome.com -p 885

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

cross-os-tests/tlevine@unixware.polarhome.com -p 905

@@ -0,0 +1,2 @@
+#!/bin/sh
+. ./.run

docs/execution-flow

@@ -0,0 +1,109 @@
+Here I discuss Urchin's general execution flow and how it is handled
+specifically when tests are run on remote environments.
+
+
+Steps of an Urchin run
+----------------------
+When Urchin runs a directory of files, it goes through the following
+steps.
+
+1. Head
+2. Test
+3. Foot
+4. Reporting
+
+Urchin stores files in a temporary directory, creating a new directory
+on each invocation. The directory contains these things.
+
+* head (file)
+* test (file)
+* foot (file)
+* stdout (directory)
+
+When run on remotes, the temporary directory corresponding to the local
+master process additionally has these files.
+
+* remote-test
+
+Messages from the head, test, and foot steps go in the corresponding
+files. In the head and foot phases, messages are just simple prints.
+Messages from the test phase always correspond to a particular test
+file, and they are written to the test file in a delimiter-separated
+format.
+
+Stdout and stderr from test runs are written to files in the stdout
+directory, one file per test file per shell that the file is run in.
+
+The reporting phase
+----------------------
+In most cases Urchin begins printing to the screen only during the
+reporting phase. The only case where anything is printed beforehand is
+when Urchin is run with -vvvv; that sets "+x", so the commands are
+printed as they run, though all other output is still suppressed.
+
+Test results are reported in the reporting phase. Four output formats
+are available.
+
+1. Urchin's human-readable format (default)
+2. Test Anything Protocol
+3. Delimiter-separated values (used internally)
+4. Remote Urchin worker output
+
+Most of the output is generated based on the delimiter-separated values
+in the test log file. The first two formats also include stdout and
+stderr from the tests, depending on verbosity level flags; when it needs
+these, Urchin reads them from appropriate files in the temporary
+directory.
+
+I could discuss the further details of each format elsewhere.
+
+Remotes
+----------------------
+When Urchin runs tests on a remote, it copies tests to the remote and
+then calls Urchin on the remote with "--format=remote". This specifies
+the following.
+
+* The temporary directory should be kept, rather than deleted, after
+  Urchin runs.
+* The path of the temporary directory should be printed as output.
+* No other output should be printed to stdout.
+
+After the remote Urchin finishes running, the local urchin downloads
+the remote Urchin's test log file from the temporary directory.
+It modifies the file to include the remote's name and then concatenates
+the result to the "remote-test" file in the local temporary directory.
+For example, the file from the remote might look like this,
+
+    :sh:Counting tests/.test/faila:0:not_ok
+
+and the result might look like this.
+
+    nsa:sh:Counting tests/.test/faila:0:not_ok
+
+This gets processed in the reporting step like usual, according to
+whatever format is specified. Instead of printing just "sh" as the
+environment in which the particular test was run, the report will print
+"sh on nsa".
+
+When it needs the stdout files, it prints them over ssh.
+
+New flags
+----------
+In making this remotes feature, I wound up adding some others.
+
+-r, --remote      SSH host to use as a remote
+-F, --format      Output format, one of "urchin", "tap", "dsv", "remote"
+
+Urchin runs only locally by default. If you pass at least one --remote
+flag, Urchin runs tests only on the specified remotes; it can't run both
+locally and remotely in the same run. If you want to do that, you could
+wait until I add that feature, or you can add "localhost" as a remote.
+
+Settings that I'm thinking about
+
+* Port for rsync/ssh
+* SSH protocol version
+* --rsync-path
+
+Can those all be set in ssh_config? Probably not --rsync-path, but
+I guess I could just fix it on the remote.

environments/nixos-container/configuration.nix

@@ -0,0 +1,39 @@
+# A NixOS container to protect against accidental fork bombs
+#
+# Put this in /var/lib/containers/test/etc/nixos/configuration.nix
+# See https://nixos.org/wiki/NixOS:Containers
+{ config, lib, pkgs, ... }:
+   
+with lib;
+ 
+{ boot.isContainer = true;
+  networking.hostName = mkDefault "urchin";
+  networking.useDHCP = false;
+
+  environment.systemPackages = with pkgs; [
+    # Urchin
+    bash dash mksh zsh
+    busybox
+
+    # Other
+    vim git rsync tmux
+  ];
+  security.pam.loginLimits = [
+    # Prevent accidental fork bombs.
+    { domain = "*"; item = "nproc"; type = "hard"; value = "200"; }
+  ];
+  services.openssh = {
+    enable = true;
+    passwordAuthentication = false;
+  };
+  users.extraUsers.user = {
+    name = "tlevine";
+    uid = 1000;
+    isNormalUser = true;
+    home = "/home/tlevine";
+    extraGroups = [ "users" "wheel" ];
+    openssh.authorizedKeys.keys = [
+      "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGvQyzr42/96acUTUedaeM2ee+DMt9bkxeurdeXji9sNE10MjjAUFtxPmSI8/BUZW2/a9ByblfaJEI+H+kFVPjVr+QGKXZluxcFMj2BLbH53fi9xLgoQRjb2aAXutb2Bp74/E8R1K+CuFfRRGQ5Spdnv44SLt04D6JbBLcLIcWTpQ4v5RaYr2U27jfiF9z0m+/opxvowEy2gnqlEXFxFk8jZHT4K0uLWm2ENjT6OpyOx8hWcKeAN2vRVRex3pJfSzswn0LpuCrM1rUZ4DRE+FABi8N21Q3MBaMRkwnZPwaZwKzv06q8bu23jYTqK5BrUPtOXeeVuroQXMc12H/6/Nh laptop"
+    ];
+  };
+}

environments/nixos-container/run.sh

@@ -0,0 +1,36 @@
+#!/bin/sh
+set -e
+
+# Create the container.
+if ! nixos-container list | grep ^urchin$ > /dev/null; then
+  sudo nixos-container create urchin
+fi
+
+# Configure the container.
+sudo cp configuration.nix \
+  /var/lib/containers/urchin/etc/nixos/configuration.nix
+sudo nixos-container update urchin
+sudo nixos-container start urchin
+
+# Create the git repository.
+host="tlevine@$(nixos-container show-ip urchin)"
+ssh "${host}" 'if mkdir urchin 2> /dev/null; then
+    cd urchin
+    git init
+    git config --add receive.denyCurrentBranch ignore
+  fi
+'
+
+# Push to the git repository
+git push "${host}":urchin
+
+# Print information
+echo "Log in:
+
+  ssh ${host}
+
+Add git remote
+
+  git remote add ${host} container
+
+"

packages/.gitignore

@@ -0,0 +1 @@
+*.tar.gz

packages/nongnu.sh

@@ -0,0 +1,11 @@
+#!/bin/sh
+name=urchin-$(../urchin --version)
+
+tmp=$(mktemp -d)
+mkdir $tmp/$name
+cp ../urchin ../readme.md ../AUTHORS ../COPYING $tmp/$name
+cd $tmp
+tar czf $name.tar.gz $name
+cd - > /dev/null
+mv $tmp/$name.tar.gz .
+rm -R $tmp

packages/package.json

@@ -12,7 +12,7 @@
   "bin": "./urchin",
   "repository": {
     "type": "git",
-    "url": "git://github.com/tlevine/urchin.git"
+    "url": "https://git.sdf.org/tlevine/urchin"
   },
   "keywords": [
     "shell",

readme.md

@@ -1,3 +1,5 @@
+**The repository at https://github.com/tlevine/urchin will go away. New location is https://git.sdf.org/tlevine/urchin.**
+
                        __    _     
       __  ____________/ /_  (_)___ 
      / / / / ___/ ___/ __ \/ / __ \
@@ -17,7 +19,7 @@ have shells called "tests".
 Urchin's tests are written in Urchin, so you can run them to see what Urchin
 is like. Clone the repository
 
-    git clone git://github.com/tlevine/urchin.git
+    git clone https://git.sdf.org/tlevine/urchin
 
 Run the tests
 
@@ -51,7 +53,7 @@ Urchin is contained in a single file, so you can install it by copying it to a
 directory in your `PATH`. For example, you can run the following as root.
 
     cd /usr/local/bin
-    wget https://raw.githubusercontent.com/tlevine/urchin/v0.0.6/urchin
+    wget https://git.sdf.org/tlevine/urchin/raw/branch/master/urchin
     chmod +x urchin
 
 Urchin can be installed with npm too.

tests/Flags/--timeout error message

@@ -0,0 +1,2 @@
+../../urchin -T aoeu .testsuite 2>&1 | grep Bad
+../../urchin -T .testsuite 2>&1 | grep Bad

tests/Flags/--timeout fail

@@ -1,3 +1 @@
-set -e
-$TEST_SHELL ../../urchin --shell sh .slow-tests
 ! $TEST_SHELL ../../urchin --shell sh --timeout 0.3 .slow-tests

tests/Flags/--timeout output

@@ -0,0 +1,4 @@
+$TEST_SHELL ../../urchin --shell sh --timeout 0.3 .slow-tests 2>&1 |
+  grep -v -- --timeout |
+  grep timeout
+test $? = 1

tests/Flags/--timeout success

@@ -0,0 +1 @@
+$TEST_SHELL ../../urchin --shell sh .slow-tests --timeout 1000

tests/Flags/Test Anything Protocol format

@@ -1,3 +1,3 @@
 $TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
-  sed -e 1d -e /second/d > $tmp
+  sed -e 1,2\ d -e /second/d > $tmp
 diff $tmp .tap-output-expectation

tests/Flags/Test Anything Protocol format 2

@@ -0,0 +1,5 @@
+lines=$(
+  $TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
+    tee $tmp | grep -v '^#' | wc -l)
+cat $tmp
+test $lines -eq 4

tests/Flags/Urchin format

@@ -1,3 +1,3 @@
 $TEST_SHELL ../../urchin -vv -s sh .testsuite/ |
-  sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
+  sed -e 1,2\ d -e 's/. seconds\?/1 second/' > $tmp
 diff $tmp .urchin-output-expectation

tests/Flags/Urchin format with --color

@@ -1,3 +1,3 @@
 $TEST_SHELL ../../urchin --pretty -vv --shell sh .testsuite/ |
-  sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
+  sed -e 1,2\ d -e 's/. seconds\?/1 second/' > $tmp
 diff $tmp .urchin-output-expectation-color

tests/Internals/has_shebang_line/.run

@@ -1,3 +1,3 @@
 echo "$1" > $tmp
-TESTING_URCHIN_INTERNALS=true . ../../../urchin
+NO_MAIN= . ../../../urchin
 has_shebang_line $tmp