HISTORY ======= Version 0.1.0-rc1 --------------------- This release includes breaking changes. ### Test root directory We introduce a concept of the root directory of a test suite. Such a concept is important in case you want to run subsets of your test suite, as we need to know how far up to apply the setup and teardown files. The Urchin root directory is determined by moving higher in the directory tree in search of a file named `.urchin`. The closest directory that contains such a file is considered the root. In the following filesystem, for example, `/a/b/c` would be the root. mkdir -p /a/b/c/d touch /a/b/c/d/e chmod +x /a/b/c/d/e touch /a/b/c/.urchin urchin /a/b/c/d There are two situations in which we would stop looking without having found a `.urchin` file. 1. The system root, `/`, because we can't go any higher 2. A directory that starts with a dot, because an urchin call on a higher directory would ignore such a directory In either of these cases, Urchin uses the user-specified directory as the root; this is how Urchin `0.0.*` worked. ### Molly guard The Molly-guard works differently because it now considers the test suite root directory. The point of the Molly-guard originally was to protect you from things like this. urchin / Urchin would run fine if called on a directory named something like "test", urchin test and it would fail on directories named something else, like `/`. Unfortunately, it would also fail on directories like this. urchin test/database It now now looks instead at the basename of the test suite root directory and otherwise ignores the entered directory. Urchin runs without error if the basename contains the phrase "test". As before, you can override the Molly guard with `-f`. urchin -f build-scripts ### Consolidation of temporary files in /tmp All of Urchin's temporary files are now stored in /tmp. Urchin previously created `.urchin.log` files alongside the tests, which led to such inconveniences as accidentally commiting them to version control repositories. This also means that Urchin will keep all of its temporary files in RAM if you mount a tmpfs on /tmp. On large test suites you may find the tmpfs to be slightly faster than slower storage media like solid-state drives. ### Skipping of tests Previously, tests were run if they were executable and were otherwise marked as skipped. Now, an executable script can indicate that it is skipped by exiting with code 3. For example, if a test requires some dependancy, it might look for the dependency and then skip if it does not see the dependency. It might look like this. #!/bin/sh if which inkscape; then exit 3 # status code 3 for skip fi inkscape blah blah ... I chose status code 3 sort of arbitrarily at first, but it turns out that it would the appropriate status code if these tests were Nagios plugins, as the concept of skipping a test is similar to the Nagios concept of unknown service status (https://nagios-plugins.org/doc/guidelines.html#AEN78). ### Parallel test execution Tests now run in parallel when possible. Parallel processes come about in two situations when parallel execution is turned on. 1. All files and immediate subdirectories of one particular directory are run in parallel. This happens recursively; during the execution of each particular subdirectory, that subdirectory's children are also run in parallel. 2. When cycling of shells is enabled, execution of a particular file in different shells are run parellel. Parallel processing and shell cycling are both enabled by default. ### Options Long options are now available for all command line flags. For example, the `-s` flag is now available as `--shell` as well. See the help for the full list. urchin -h ### Source setup and teardown setup, teardown, setup_dir, and teardown_dir are now sourced instead of executed; they are referenced a bit like this. ( . ./setup ./$thetestfile . ./teardown ) My intent is that you should be able to export variables in the setup files. I think it would be fine to invoke the teardown files instead of sourcing them, but I chose to source them anyway for consistency. The disadvantage of this, and the reason I have been reluctant to do it, is that these files now become much harder to debug, so I recommend keeping your setup and teardown files very simple. I recommend either of the following strategies if your setup file gets complicated. 1. Rename it to something starting with a dot, and explicitly source it in your test file. 2. Export a path in your setup file, rewrite your setup file as a shell program, and put the rewritten file in your path. ### Run on a file Previously you could run urchin only on a directory (and, in turn, all files in that directory). Now you can run Urchin on a single file. This occurred to me when I wanted to run urchin test/fast/Unit\ tests/nvm_ls_current on the nvm tests. I wound up running this instead. urchin test/fast/Unit\ tests/ | grep nvm_ls_current But now I don't have to; the first of these commands will work. When you run urchin on a file, the test suite root is determined (as with any other Urchin call), and the test suite is recursively descended. Setup and teardown files are sourced, and everything but the specified test file is otherwise ignored. If you don't explicitly specify the Urchin root with a .urchin file, we consider the test suite root directory to be the parent of the file that you ran Urchin on. ### Verbose output ### Timing Urchin now reports the time, in seconds, that each test took and also the total time that it took to run the whole test suite. Urchin also allows you to set timeouts, in seconds, with the --timeout flag. If you set a timeout flag and a test file takes longer to run, that run will be killed, and the test will thus fail. The standard error message from the timeout program will show up in the test output. Both of these timers use the real time (not the CPU time for example), so the times are not very precise and may be much larger than you expect. Version 0.0.6 --------------------- * Produce TAP output with the -t flag. * Add a + sign in front of directories in the normal output so that they line up with non-directories. * Display skipped tests in the normal output and in the TAP output. * Correct some things in the documentation. * Rearrange things in the documentation to be more clear. * Pass the -e flag to exit urchin if any single test fails. * Remove the undocumented, experimental -x flag now that shall exists. * Display version number with the -v flag. * Document why Urchin is called "Urchin" * Update TODO * Support mksh (Change a printf command.) * Make long lines shorter. These changes are made somewhat separately in the branches "exit-on-fail", "remove-urchin-x", "tap", and "update-readme". They are rebased into one branch, "tlevine-2016-02", for merging into "master". Version 0.0.5 --------------------- * urchin now unsets `CDPATH`. * The documentation for `urchin -x` was removed because it was confusing. Version 0.0.4 --------------------- * Switch urchin -x to urchin -sh and fix some problems with it * Documentation Version 0.0.3 --------------------- General tidying Run with different shells in three ways * urchin -s * $TEST_SHELL variable with $TEST_SHELL * $TEST_SHELL variable with urchin -sh Set NULL_GLOB so zsh doesn't print a warning. Before version 0.0.3 ---------------------- We adjusted the input parameters so it is harder to accidentally run all executable files in you r home directory. We added directory-based indents.