urchin/HISTORY

HISTORY
=======

Version 0.1.0-rc1
---------------------
This release includes breaking changes.

### Test root directory
We introduce a concept of the root directory of a test suite.
Such a concept is important in case you want to run subsets of your
test suite, as we need to know how far up to apply the setup
and teardown files.

The Urchin root directory is determined by moving higher in the directory
tree in search of a file named `.urchin`.
The closest directory that contains such a file is considered the root.
In the following filesystem, for example, `/a/b/c` would be the root.

    mkdir -p /a/b/c/d
    touch /a/b/c/d/e
    chmod +x /a/b/c/d/e
    touch /a/b/c/.urchin
    urchin /a/b/c/d

There are two situations in which we would stop looking without having
found a `.urchin` file.

1. The system root, `/`, because we can't go any higher
2. A directory that starts with a dot, because an urchin call on a higher
   directory would ignore such a directory

In either of these cases, Urchin uses the user-specified directory as
the root; this is how Urchin `0.0.*` worked.

### Molly guard
The Molly-guard works differently because it now considers the test suite
root directory. The point of the Molly-guard originally was to protect
you from things like this.

    urchin /

Urchin would run fine if called on a directory named something like "test",

    urchin test

and it would fail on directories named something else, like `/`.
Unfortunately, it would also fail on directories like this.

    urchin test/database

It now now looks instead at the basename of the test suite root directory and
otherwise ignores the entered directory. Urchin runs without error if the basename contains the phrase "test".

As before, you can override the Molly guard with `-f`.

    urchin -f build-scripts

### Consolidation of temporary files in /tmp
All of Urchin's temporary files are now stored in /tmp. Urchin previously
created `.urchin.log` files alongside the tests, which led to such
inconveniences as accidentally commiting them to version control repositories.

This also means that Urchin will keep all of its temporary files in RAM
if you mount a tmpfs on /tmp. On large test suites you may find the tmpfs
to be slightly faster than slower storage media like solid-state drives.

### Skipping of tests
Previously, tests were run if they were executable and were otherwise marked
as skipped. Now, an executable script can indicate that it is skipped by
exiting with code 3.  For example, if a test requires some dependancy, it
might look for the dependency and then skip if it does not see the dependency.
It might look like this.

    #!/bin/sh
    if which inkscape; then
      exit 3 # status code 3 for skip
    fi
    inkscape blah blah ...

I chose status code 3 sort of arbitrarily at first, but it turns out that it
would the appropriate status code if these tests were Nagios plugins, as the
concept of skipping a test is similar to the Nagios concept of unknown service
status (https://nagios-plugins.org/doc/guidelines.html#AEN78).

### Parallel test execution
Tests now run in parallel when possible.

Parallel processes come about in two situations when parallel execution is
turned on.

1. All files and immediate subdirectories of one particular directory
   are run in parallel. This happens recursively; during the execution
   of each particular subdirectory, that subdirectory's children are
   also run in parallel.
2. When cycling of shells is enabled, execution of a particular file in
   different shells are run parellel.

Parallel processing and shell cycling are both enabled by default.

### Options
Long options are now available for all command line flags.
For example, the `-s` flag is now available as `--shell` as well.
See the help for the full list.

    urchin -h

### Source setup and teardown
setup, teardown, setup_dir, and teardown_dir are now sourced instead of
executed; they are referenced a bit like this.

    (
      . ./setup
      ./$thetestfile
      . ./teardown
    )

My intent is that you should be able to export variables in the setup files.
I think it would be fine to invoke the teardown files instead of sourcing them,
but I chose to source them anyway for consistency.

The disadvantage of this, and the reason I have been reluctant to do it,
is that these files now become much harder to debug, so I recommend keeping
your setup and teardown files very simple. I recommend either of the following
strategies if your setup file gets complicated.

1. Rename it to something starting with a dot, and explicitly source it
   in your test file.
2. Export a path in your setup file, rewrite your setup file as a shell
   program, and put the rewritten file in your path.

### Run on a file
Previously you could run urchin only on a directory (and, in turn, all files
in that directory). Now you can run Urchin on a single file.

This occurred to me when I wanted to run

    urchin test/fast/Unit\ tests/nvm_ls_current

on the nvm tests. I wound up running this instead.

    urchin test/fast/Unit\ tests/ | grep nvm_ls_current

But now I don't have to; the first of these commands will work.

When you run urchin on a file, the test suite root is determined (as with any
other Urchin call), and the test suite is recursively descended. Setup and
teardown files are sourced, and everything but the specified test file is
otherwise ignored.

If you don't explicitly specify the Urchin root with a .urchin file, we
consider the test suite root directory to be the parent of the file that
you ran Urchin on.

### Verbose output

### Timing
Urchin now reports the time, in seconds, that each test took and also the
total time that it took to run the whole test suite.

Urchin also allows you to set timeouts, in seconds, with the --timeout flag.
If you set a timeout flag and a test file takes longer to run, that run will
be killed, and the test will thus fail. The standard error message from the
timeout program will show up in the test output.

Both of these timers use the real time (not the CPU time for example),
so the times are not very precise and may be much larger than you expect.

Version 0.0.6
---------------------

* Produce TAP output with the -t flag.
* Add a + sign in front of directories in the normal output so that they
  line up with non-directories.
* Display skipped tests in the normal output and in the TAP output.
* Correct some things in the documentation.
* Rearrange things in the documentation to be more clear.
* Pass the -e flag to exit urchin if any single test fails.
* Remove the undocumented, experimental -x flag now that shall exists.
* Display version number with the -v flag.
* Document why Urchin is called "Urchin"
* Update TODO
* Support mksh (Change a printf command.)
* Make long lines shorter.

These changes are made somewhat separately in the branches "exit-on-fail",
"remove-urchin-x", "tap", and "update-readme". They are rebased into one
branch, "tlevine-2016-02", for merging into "master".

Version 0.0.5
---------------------
* urchin now unsets `CDPATH`.
* The documentation for `urchin -x` was removed because it was confusing.

Version 0.0.4
---------------------
* Switch urchin -x to urchin -sh and fix some problems with it
* Documentation

Version 0.0.3
---------------------
General tidying

Run with different shells in three ways

* urchin -s
* $TEST_SHELL variable with $TEST_SHELL
* $TEST_SHELL variable with urchin -sh

Set NULL_GLOB so zsh doesn't print a warning.

Before version 0.0.3
----------------------
We adjusted the input parameters so it is harder to accidentally run all executable files in you
r home directory.

We added directory-based indents.