urchin/TODO

Things I want
=============

Test speed
-------------
Make tests run faster.
https://github.com/bike-barn/hermit/issues/62

First, easier thing is probably to run tests in parallel.

Second, also easier thing is to tell people to save things to RAM rather than
disk whenever they can.

Third, harder thing is to put the test suite in RAM automatically. Maybe the
whole test directory, which includes fixtures, gets copied to a tmpfs if one
exists.

Hmm or maybe there's a compromise: Tell people to mount /tmp as a tmpfs so
that temp files are fast. Maybe allow people to set some other directory as
the temporary file place, in case they want a different tmpfs location.

In order to run things in parallel, we have to change how we do the
stdout_file. I think it's easiest to create separate files for each test and
to save them in testroot/.urchin/stdout/$filename. The test root would be
defined as the closest ancestor containing a .urchin directory.

Options
-------------
I want long options. For example, there's presently -f and -e.
I want to make them -f|--force and -e|--exit.

Environment variables
-------------
Do something to make it easier to debug environment variables, because that is
often confusing.
https://github.com/creationix/nvm/issues/719
https://github.com/creationix/nvm/issues/589

Documenting that people should run "env" when their tests fail might be good
enough.

Licensing and copyright
------------------------
* Reference all owners and years in the Copyright file
* Consider copyleft licenses
* Add license notices to other files if necessary

Packaging
------------
Package for package managers.

* I want NixOS, of course.
* Debian is probably the big one.

Other interesting package managers

* Update the npm package
* Homebrew (for Mac)

Windows
----------
Try running Urchin in Windows somehow. Interpreters include

* CygWin (https://www.cygwin.com/)
* MSYS (http://mingw.org/wiki/msys)
* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
* Git for Windows (https://git-scm.com/download/win)
* win-bash (http://win-bash.sourceforge.net/)

Consider copyleft licenses
----------
ScraperWiki owns the original version of Urchin (Thomas Levine did the early
work as part of his work for ScraperWiki.) and originally licensed it under an
MIT-style license. Other people made changes after this original ScraperWiki
version. As of January 2016, they are just Thomas Levine (when he wasn't
working for ScraperWiki) and Michael Klement.

The original license was MIT just because that's what ScraperWiki put on
everything. Should we change the license?

The MIT-style license grants pretty much all rights. It says that you need
to attribute when you redistribute source code, but you don't
necessarily have to redistribute source code.

A copyleft license adds the restriction that modified versions of the
code need to be licensed under the same license. GNU licenses in
particular require that source code be released if non-source versions are
released, and the different GNU licenses differ in what how the
non-source version is defined. (The original, GPL, discusses compiled
binaries.) Copyleft doesn't mean anything specific for commercial use.

MIT-licensed code can be modified and then licensed as GPL, because MIT
license allows that, but GPL code can't be modified as MIT, because MIT
doesn't allow that. And if we get all of the authors to agree on it, we
can always add whatever crazy license we want, regardless of what we
have already.

The distinction between MIT-style and GNU-something might matter quite little
in the case of Urchin.

1. Urchin is written in an interpreted language (shell), so it might be
   hard to distribute usefully without providing the source code.
2. Urchin just runs tests; it doesn't get compiled with the rest of the
   code (also because it's in shell). Thus, I think a GPL license on
   Urchin wouldn't infect the code being tested.

This is as far as I have gotten with contemplating license changes. For now
we're sticking with the original MIT-style license, but it's easy to change
licenses later.

Nagios plugins
-----------------
It would be cool to run Nagios plugins with Urchin. This is already possible,
actually, but it might be worth giving some special thought to it.
https://nagios-plugins.org/doc/guidelines.html

Source setup and teardown
--------------------
If setup and teardown are sourced instead of executed, maybe we can more
cleanly create and teardown temporary files.

    (
      . ./setup
      ./$thetestfile
      . ./teardown
    )

On the other hand, this could just be sourced explicitly in the test file,
without the special setup and teardown feature.

Run on a file
----------------
Presently you can run urchin only on a directory.
It would be neat if you could run it on a file as well.

This occurred to me when I wanted to run

    urchin test/fast/Unit\ tests/nvm_ls_current

on the nvm tests. I wound up running this instead.

    urchin test/fast/Unit\ tests/ | grep nvm_ls_current

The Molly guard would be assessed, and the corresponding setup, setup_dir,
teardown, and teardown_dir files would be run in the appropriate order.

In order to know how far up the tree to evaluate the setup, &c. files,
I think it would make sense to require that a ".urchin" file be placed in the
root of the tests. Urchin would keep going up until it sees this file, and it
would evaluate the appropriate setup, &c. files from there down to the
particular test file of interest. We would also use this for testing
directtories more correctly.

Running automated tasks
-------------------------
Urchin might be appropriate for if you have lots of tasks that you want to run
periodically; add an urchin call to your crontab, and call all of your other
tasks with urchin. Here are some features that might make urchin better for
this sort of thing.
                                                      
* Time how long each test/job takes
* Optionally kill tests/jobs after a specific timeout threshold
* Send output of different tests/jobs to different files for each file
  descriptor (STDOUT, STDERR)

sorting
-------
How do sort properly?

  printf '- a\n-- a\n--- a\n- b\n-- b\n--- b\n'|sor