urchin/HISTORY

HISTORY
=======

Version 0.2.0 (unstable)
---------------------
### Cross-OS testing
I have started testing Urchin across multiple operating systems.
This gives access to more shells, as some shels are easier to install on
certain operating systems.

With this cross-OS test suite, I have extended support to more shells.
A later version of Urchin could include a remote testing feature.

Version 0.1.0 (stable)
---------------------
This release includes breaking changes.

### Test root directory
We introduce a concept of the root directory of a test suite.
Such a concept is important in case you want to run subsets of your
test suite, as we need to know how far up to apply the setup
and teardown files.

The Urchin root directory is determined by moving higher in the directory
tree in search of a file named `.urchin_root`.
The closest directory that contains such a file is considered the root.
In the following filesystem, for example, `/a/b/c` would be the root.

    mkdir -p /a/b/c/d
    touch /a/b/c/d/e
    chmod +x /a/b/c/d/e
    touch /a/b/c/.urchin_root
    urchin /a/b/c/d

There are two situations in which we would stop looking without having
found a `.urchin_root` file.

1. The system root, `/`, because we can't go any higher
2. A directory that starts with a dot, because an urchin call on a higher
   directory would ignore such a directory

In either of these cases, Urchin uses the user-specified directory as
the root; this is how Urchin `0.0.*` worked.

### Molly guard
The Molly-guard works differently because it now considers the test suite
root directory. The point of the Molly-guard originally was to protect
you from things like this.

    urchin /

Urchin would run fine if called on a directory named something like "test",

    urchin test

and it would fail on directories named something else, like `/`.
Unfortunately, it would also fail on directories like this.

    urchin test/database

It now now looks instead at the basename of the test suite root directory and
otherwise ignores the entered directory. Urchin runs without error if the basename contains the phrase "test".

As before, you can override the Molly guard with `-f`.

    urchin -f build-scripts

### Consolidation of temporary files in /tmp
All of Urchin's temporary files are now stored in /tmp. Urchin previously
created `.urchin.log` files alongside the tests, which led to such
inconveniences as accidentally commiting them to version control repositories.

This also means that Urchin will keep all of its temporary files in RAM
if you mount a tmpfs on /tmp. On large test suites you may find the tmpfs
to be slightly faster than slower storage media like solid-state drives.

### Skipping of tests
Previously, tests were run if they were executable and were otherwise marked
as skipped. Now, an executable script can indicate that it is skipped by
exiting with code 3.  For example, if a test requires some dependancy, it
might look for the dependency and then skip if it does not see the dependency.
It might look like this.

    #!/bin/sh
    if ! which inkscape; then
      exit 3 # status code 3 for skip
    fi
    inkscape blah blah ...

I chose status code 3 sort of arbitrarily at first, but it turns out that it
would the appropriate status code if these tests were Nagios plugins, as the
concept of skipping a test is similar to the Nagios concept of unknown service
status (https://nagios-plugins.org/doc/guidelines.html#AEN78).

### Parallel test execution
Tests now run in parallel when possible.

Parallel processes come about in two situations when parallel execution is
turned on.

1. All files and immediate subdirectories of one particular directory
   are run in parallel. This happens recursively; during the execution
   of each particular subdirectory, that subdirectory's children are
   also run in parallel.
2. When cycling of shells is enabled, execution of a particular file in
   different shells are run parellel.

Parallel processing and shell cycling are both enabled by default.

You may want make only some directories run in series, you can create
".urchin_dir" files in those directories.
If .urchin_dir contains the phrase "series", run that directory in series
rather than in parallel.
This is helpful when directories actually need to run in series
and also when running all your tests in parallel crashes your computer.

### Options
Long options are now available for all command line flags.
For example, the `-s` flag is now available as `--shell` as well.
See the help for the full list.

    urchin -h

### Copyrights
Some people had contributed to Urchin but had not been added to the copyright
notice. I have updated the copyright notice to include everyone whom I believe
to have contributed patches.

### License
I, Thomas Levine, have switched the previous BSD-style license for the Afferro
Gnu Public License (AGPL) after determining that the added restrictions in the
AGPL shouldn't have any practical legal consequences for people who want to
use Urchin. I did not get approval from the other authors as I believe the
licenses to be compatible.  Here are the considerations that I considered.

#### History
ScraperWiki owns the original version of Urchin (Thomas Levine did the early
work as part of his work for ScraperWiki.) and originally licensed it under a
BSD-style license with the advertising clauses removed. (This makes it a
"2-clause BSD license", similar to the FreeBSD license.) We had the previous
license just because that's what ScraperWiki put on everything. 

Other people made changes after this original ScraperWiki version. As of
January 2016, they are just Thomas Levine (when he wasn't working for
ScraperWiki) and Michael Klement.

The 2-clause BSD license grants pretty much all rights. It says that you need
to attribute when you redistribute source code, but you don't necessarily have
to redistribute source code.

#### License compatibility
A copyleft license adds the restriction that modified versions of the code
need to be licensed under the same license. GNU licenses in particular require
that source code be released if non-source versions are released, and the
different GNU licenses differ in what how the non-source version is defined.
(The original, GPL, discusses compiled binaries, for example.) Copyleft
doesn't mean anything specific for commercial use.

Code licensed under the 2-clause BSD license can be modified and then licensed
as AGPL, because the 2-clause BSD license license allows that, but AGPL code
can't be modified as 2-clause BSD, because AGPL doesn't allow that.

Of course, if we get all of the authors to agree on it, we can always add
whatever crazy license we want, regardless of what we have already.

#### Practical differences
The distinction between the permissive 2-clause BSD license and the AGPL seem
to matter quite little in the case of Urchin.

1. Urchin is written in an interpreted language (shell), so it would be
   hard to distribute usefully without providing the source code.
2. Urchin usually just runs tests; it doesn't get compiled with the rest of
   the code (also because it's in shell). Thus, I think a GPL license on Urchin
   wouldn't infect the code being tested.

### Specification of the shell to run tests in
Urchin previously had separate methods for setting the TEST_SHELL environment
variable and for setting the shell that would run the tests; the former was
set as an environment variable, and the latter was set with the -s flag..

Urchin now uses the -s flag for both of these settings, and it mostly ignores
the exported TEST_SHELL variable.

If you pass -n/--disable-cycling, Urchin will invoke tests ordinarily and will
only set the TEST_SHELL variable if it does not exist. If the TEST_SHELL
variable is absent, it will be set to /bin/sh.

Here is how you should write your tests for cross-shell testing, depending on
their structure.

* If you want a test file to run in the same shell every time and to have
  access to the TEST_SHELL variable, usually for invoking the program that
  you are testing, then set the file's shebang line.
* If you want a test file to be run in a different shell every time, do not
  set the shebang line. TEST_SHELL variable will be set to correspond with the
  shell that is presently invoking the test file, though you probably won't
  need this variable.
* If you want a test file to have access to a TEST_SHELL variable that you
  set yourself, pass -n/--disable-cycling to urchin. Urchin will ignore the
  shebang lines in this case.

### Source setup and teardown
setup, teardown, setup_dir, and teardown_dir are now sourced instead of
executed; they are referenced a bit like this.

    (
      . ./setup
      ./$thetestfile
      . ./teardown
    )

My intent is that you should be able to export variables in the setup files.
I think it would be fine to invoke the teardown files instead of sourcing them,
but I chose to source them anyway for consistency.

The disadvantage of this, and the reason I have been reluctant to do it,
is that these files now become much harder to debug, so I recommend keeping
your setup and teardown files very simple. I recommend either of the following
strategies if your setup file gets complicated.

1. Rename it to something starting with a dot, and explicitly source it
   in your test file.
2. Export a path in your setup file, rewrite your setup file as a shell
   program, and put the rewritten file in your path.

### Run on a file
Previously you could run urchin only on a directory (and, in turn, all files
in that directory). Now you can run Urchin on a single file.

This occurred to me when I wanted to run

    urchin test/fast/Unit\ tests/nvm_ls_current

on the nvm tests. I wound up running this instead.

    urchin test/fast/Unit\ tests/ | grep nvm_ls_current

But now I don't have to; the first of these commands will work.

When you run urchin on a file, the test suite root is determined (as with any
other Urchin call), and the test suite is recursively descended. Setup and
teardown files are sourced, and everything but the specified test file is
otherwise ignored.

If you don't explicitly specify the Urchin root with a .urchin_root file, we
consider the test suite root directory to be the parent of the file that
you ran Urchin on.

### Verbose output

### Timing
Urchin now reports the time, in seconds, that each test took and also the
total time that it took to run the whole test suite.

Urchin also allows you to set timeouts, in seconds, with the --timeout flag.
If you set a timeout flag and a test file takes longer to run, that run will
be killed, and the test will thus fail. The standard error message from the
timeout program will show up in the test output.

Both of these timers use the real time (not the CPU time for example),
so the times are not very precise and may be much larger than you expect.

Version 0.0.6
---------------------

* Produce TAP output with the -t flag.
* Add a + sign in front of directories in the normal output so that they
  line up with non-directories.
* Display skipped tests in the normal output and in the TAP output.
* Correct some things in the documentation.
* Rearrange things in the documentation to be more clear.
* Pass the -e flag to exit urchin if any single test fails.
* Remove the undocumented, experimental -x flag now that shall exists.
* Display version number with the -v flag.
* Document why Urchin is called "Urchin"
* Update TODO
* Support mksh (Change a printf command.)
* Make long lines shorter.

These changes are made somewhat separately in the branches "exit-on-fail",
"remove-urchin-x", "tap", and "update-readme". They are rebased into one
branch, "tlevine-2016-02", for merging into "master".

Version 0.0.5
---------------------
* urchin now unsets `CDPATH`.
* The documentation for `urchin -x` was removed because it was confusing.

Version 0.0.4
---------------------
* Switch urchin -x to urchin -sh and fix some problems with it
* Documentation

Version 0.0.3
---------------------
General tidying

Run with different shells in three ways

* urchin -s
* $TEST_SHELL variable with $TEST_SHELL
* $TEST_SHELL variable with urchin -sh

Set NULL_GLOB so zsh doesn't print a warning.

Before version 0.0.3
----------------------
We adjusted the input parameters so it is harder to accidentally run all executable files in you
r home directory.

We added directory-based indents.