HISTORY ======= Version 0.2.0 (unstable) --------------------- ### Cross-OS testing I have started testing Urchin across multiple operating systems. This gives access to more shells, as some shels are easier to install on certain operating systems. With this cross-OS test suite, I have extended support to more shells. A later version of Urchin could include a remote testing feature. Version 0.1.0 (stable) --------------------- This release includes breaking changes. ### Test root directory We introduce a concept of the root directory of a test suite. Such a concept is important in case you want to run subsets of your test suite, as we need to know how far up to apply the setup and teardown files. The Urchin root directory is determined by moving higher in the directory tree in search of a file named `.urchin_root`. The closest directory that contains such a file is considered the root. In the following filesystem, for example, `/a/b/c` would be the root. mkdir -p /a/b/c/d touch /a/b/c/d/e chmod +x /a/b/c/d/e touch /a/b/c/.urchin_root urchin /a/b/c/d There are two situations in which we would stop looking without having found a `.urchin_root` file. 1. The system root, `/`, because we can't go any higher 2. A directory that starts with a dot, because an urchin call on a higher directory would ignore such a directory In either of these cases, Urchin uses the user-specified directory as the root; this is how Urchin `0.0.*` worked. ### Molly guard The Molly-guard works differently because it now considers the test suite root directory. The point of the Molly-guard originally was to protect you from things like this. urchin / Urchin would run fine if called on a directory named something like "test", urchin test and it would fail on directories named something else, like `/`. Unfortunately, it would also fail on directories like this. urchin test/database It now now looks instead at the basename of the test suite root directory and otherwise ignores the entered directory. Urchin runs without error if the basename contains the phrase "test". As before, you can override the Molly guard with `-f`. urchin -f build-scripts ### Consolidation of temporary files in /tmp All of Urchin's temporary files are now stored in /tmp. Urchin previously created `.urchin.log` files alongside the tests, which led to such inconveniences as accidentally commiting them to version control repositories. This also means that Urchin will keep all of its temporary files in RAM if you mount a tmpfs on /tmp. On large test suites you may find the tmpfs to be slightly faster than slower storage media like solid-state drives. ### Skipping of tests Previously, tests were run if they were executable and were otherwise marked as skipped. Now, an executable script can indicate that it is skipped by exiting with code 3. For example, if a test requires some dependancy, it might look for the dependency and then skip if it does not see the dependency. It might look like this. #!/bin/sh if ! which inkscape; then exit 3 # status code 3 for skip fi inkscape blah blah ... I chose status code 3 sort of arbitrarily at first, but it turns out that it would the appropriate status code if these tests were Nagios plugins, as the concept of skipping a test is similar to the Nagios concept of unknown service status (https://nagios-plugins.org/doc/guidelines.html#AEN78). ### Parallel test execution Tests now run in parallel when possible. Parallel processes come about in two situations when parallel execution is turned on. 1. All files and immediate subdirectories of one particular directory are run in parallel. This happens recursively; during the execution of each particular subdirectory, that subdirectory's children are also run in parallel. 2. When cycling of shells is enabled, execution of a particular file in different shells are run parellel. Parallel processing and shell cycling are both enabled by default. You may want make only some directories run in series, you can create ".urchin_dir" files in those directories. If .urchin_dir contains the phrase "series", run that directory in series rather than in parallel. This is helpful when directories actually need to run in series and also when running all your tests in parallel crashes your computer. ### Options Long options are now available for all command line flags. For example, the `-s` flag is now available as `--shell` as well. See the help for the full list. urchin -h ### Copyrights Some people had contributed to Urchin but had not been added to the copyright notice. I have updated the copyright notice to include everyone whom I believe to have contributed patches. ### License I, Thomas Levine, have switched the previous BSD-style license for the Afferro Gnu Public License (AGPL) after determining that the added restrictions in the AGPL shouldn't have any practical legal consequences for people who want to use Urchin. I did not get approval from the other authors as I believe the licenses to be compatible. Here are the considerations that I considered. #### History ScraperWiki owns the original version of Urchin (Thomas Levine did the early work as part of his work for ScraperWiki.) and originally licensed it under a BSD-style license with the advertising clauses removed. (This makes it a "2-clause BSD license", similar to the FreeBSD license.) We had the previous license just because that's what ScraperWiki put on everything. Other people made changes after this original ScraperWiki version. As of January 2016, they are just Thomas Levine (when he wasn't working for ScraperWiki) and Michael Klement. The 2-clause BSD license grants pretty much all rights. It says that you need to attribute when you redistribute source code, but you don't necessarily have to redistribute source code. #### License compatibility A copyleft license adds the restriction that modified versions of the code need to be licensed under the same license. GNU licenses in particular require that source code be released if non-source versions are released, and the different GNU licenses differ in what how the non-source version is defined. (The original, GPL, discusses compiled binaries, for example.) Copyleft doesn't mean anything specific for commercial use. Code licensed under the 2-clause BSD license can be modified and then licensed as AGPL, because the 2-clause BSD license license allows that, but AGPL code can't be modified as 2-clause BSD, because AGPL doesn't allow that. Of course, if we get all of the authors to agree on it, we can always add whatever crazy license we want, regardless of what we have already. #### Practical differences The distinction between the permissive 2-clause BSD license and the AGPL seem to matter quite little in the case of Urchin. 1. Urchin is written in an interpreted language (shell), so it would be hard to distribute usefully without providing the source code. 2. Urchin usually just runs tests; it doesn't get compiled with the rest of the code (also because it's in shell). Thus, I think a GPL license on Urchin wouldn't infect the code being tested. ### Specification of the shell to run tests in Urchin previously had separate methods for setting the TEST_SHELL environment variable and for setting the shell that would run the tests; the former was set as an environment variable, and the latter was set with the -s flag.. Urchin now uses the -s flag for both of these settings, and it mostly ignores the exported TEST_SHELL variable. If you pass -n/--disable-cycling, Urchin will invoke tests ordinarily and will only set the TEST_SHELL variable if it does not exist. If the TEST_SHELL variable is absent, it will be set to /bin/sh. Here is how you should write your tests for cross-shell testing, depending on their structure. * If you want a test file to run in the same shell every time and to have access to the TEST_SHELL variable, usually for invoking the program that you are testing, then set the file's shebang line. * If you want a test file to be run in a different shell every time, do not set the shebang line. TEST_SHELL variable will be set to correspond with the shell that is presently invoking the test file, though you probably won't need this variable. * If you want a test file to have access to a TEST_SHELL variable that you set yourself, pass -n/--disable-cycling to urchin. Urchin will ignore the shebang lines in this case. ### Source setup and teardown setup, teardown, setup_dir, and teardown_dir are now sourced instead of executed; they are referenced a bit like this. ( . ./setup ./$thetestfile . ./teardown ) My intent is that you should be able to export variables in the setup files. I think it would be fine to invoke the teardown files instead of sourcing them, but I chose to source them anyway for consistency. The disadvantage of this, and the reason I have been reluctant to do it, is that these files now become much harder to debug, so I recommend keeping your setup and teardown files very simple. I recommend either of the following strategies if your setup file gets complicated. 1. Rename it to something starting with a dot, and explicitly source it in your test file. 2. Export a path in your setup file, rewrite your setup file as a shell program, and put the rewritten file in your path. ### Run on a file Previously you could run urchin only on a directory (and, in turn, all files in that directory). Now you can run Urchin on a single file. This occurred to me when I wanted to run urchin test/fast/Unit\ tests/nvm_ls_current on the nvm tests. I wound up running this instead. urchin test/fast/Unit\ tests/ | grep nvm_ls_current But now I don't have to; the first of these commands will work. When you run urchin on a file, the test suite root is determined (as with any other Urchin call), and the test suite is recursively descended. Setup and teardown files are sourced, and everything but the specified test file is otherwise ignored. If you don't explicitly specify the Urchin root with a .urchin_root file, we consider the test suite root directory to be the parent of the file that you ran Urchin on. ### Verbose output ### Timing Urchin now reports the time, in seconds, that each test took and also the total time that it took to run the whole test suite. Urchin also allows you to set timeouts, in seconds, with the --timeout flag. If you set a timeout flag and a test file takes longer to run, that run will be killed, and the test will thus fail. The standard error message from the timeout program will show up in the test output. Both of these timers use the real time (not the CPU time for example), so the times are not very precise and may be much larger than you expect. Version 0.0.6 --------------------- * Produce TAP output with the -t flag. * Add a + sign in front of directories in the normal output so that they line up with non-directories. * Display skipped tests in the normal output and in the TAP output. * Correct some things in the documentation. * Rearrange things in the documentation to be more clear. * Pass the -e flag to exit urchin if any single test fails. * Remove the undocumented, experimental -x flag now that shall exists. * Display version number with the -v flag. * Document why Urchin is called "Urchin" * Update TODO * Support mksh (Change a printf command.) * Make long lines shorter. These changes are made somewhat separately in the branches "exit-on-fail", "remove-urchin-x", "tap", and "update-readme". They are rebased into one branch, "tlevine-2016-02", for merging into "master". Version 0.0.5 --------------------- * urchin now unsets `CDPATH`. * The documentation for `urchin -x` was removed because it was confusing. Version 0.0.4 --------------------- * Switch urchin -x to urchin -sh and fix some problems with it * Documentation Version 0.0.3 --------------------- General tidying Run with different shells in three ways * urchin -s * $TEST_SHELL variable with $TEST_SHELL * $TEST_SHELL variable with urchin -sh Set NULL_GLOB so zsh doesn't print a warning. Before version 0.0.3 ---------------------- We adjusted the input parameters so it is harder to accidentally run all executable files in you r home directory. We added directory-based indents.