165 lines
6.0 KiB
Plaintext
165 lines
6.0 KiB
Plaintext
Things I want
|
|
=============
|
|
|
|
Test speed
|
|
-------------
|
|
Make tests run faster.
|
|
https://github.com/bike-barn/hermit/issues/62
|
|
|
|
First, easier thing is probably to run tests in parallel.
|
|
|
|
Second, also easier thing is to tell people to save things to RAM rather than
|
|
disk whenever they can.
|
|
|
|
Third, harder thing is to put the test suite in RAM automatically. Maybe the
|
|
whole test directory, which includes fixtures, gets copied to a tmpfs if one
|
|
exists.
|
|
|
|
Hmm or maybe there's a compromise: Tell people to mount /tmp as a tmpfs so
|
|
that temp files are fast. Maybe allow people to set some other directory as
|
|
the temporary file place, in case they want a different tmpfs location.
|
|
|
|
In order to run things in parallel, we have to change how we do the
|
|
stdout_file. I think it's easiest to create separate files for each test and
|
|
to save them in testroot/.urchin/stdout/$filename. The test root would be
|
|
defined as the closest ancestor containing a .urchin directory.
|
|
|
|
Options
|
|
-------------
|
|
I want long options. For example, there's presently -f and -e.
|
|
I want to make them -f|--force and -e|--exit.
|
|
|
|
Environment variables
|
|
-------------
|
|
Do something to make it easier to debug environment variables, because that is
|
|
often confusing.
|
|
https://github.com/creationix/nvm/issues/719
|
|
https://github.com/creationix/nvm/issues/589
|
|
|
|
Documenting that people should run "env" when their tests fail might be good
|
|
enough.
|
|
|
|
Licensing and copyright
|
|
------------------------
|
|
* Reference all owners and years in the Copyright file
|
|
* Consider copyleft licenses
|
|
* Add license notices to other files if necessary
|
|
|
|
Packaging
|
|
------------
|
|
Package for package managers.
|
|
|
|
* I want NixOS, of course.
|
|
* Debian is probably the big one.
|
|
|
|
Other interesting package managers
|
|
|
|
* Update the npm package
|
|
* Homebrew (for Mac)
|
|
|
|
Windows
|
|
----------
|
|
Try running Urchin in Windows somehow. Interpreters include
|
|
|
|
* CygWin (https://www.cygwin.com/)
|
|
* MSYS (http://mingw.org/wiki/msys)
|
|
* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
|
|
* Git for Windows (https://git-scm.com/download/win)
|
|
* win-bash (http://win-bash.sourceforge.net/)
|
|
|
|
Consider copyleft licenses
|
|
----------
|
|
ScraperWiki owns the original version of Urchin (Thomas Levine did the early
|
|
work as part of his work for ScraperWiki.) and originally licensed it under an
|
|
MIT-style license. Other people made changes after this original ScraperWiki
|
|
version. As of January 2016, they are just Thomas Levine (when he wasn't
|
|
working for ScraperWiki) and Michael Klement.
|
|
|
|
The original license was MIT just because that's what ScraperWiki put on
|
|
everything. Should we change the license?
|
|
|
|
The MIT-style license grants pretty much all rights. It says that you need
|
|
to attribute when you redistribute source code, but you don't
|
|
necessarily have to redistribute source code.
|
|
|
|
A copyleft license adds the restriction that modified versions of the
|
|
code need to be licensed under the same license. GNU licenses in
|
|
particular require that source code be released if non-source versions are
|
|
released, and the different GNU licenses differ in what how the
|
|
non-source version is defined. (The original, GPL, discusses compiled
|
|
binaries.) Copyleft doesn't mean anything specific for commercial use.
|
|
|
|
MIT-licensed code can be modified and then licensed as GPL, because MIT
|
|
license allows that, but GPL code can't be modified as MIT, because MIT
|
|
doesn't allow that. And if we get all of the authors to agree on it, we
|
|
can always add whatever crazy license we want, regardless of what we
|
|
have already.
|
|
|
|
The distinction between MIT-style and GNU-something might matter quite little
|
|
in the case of Urchin.
|
|
|
|
1. Urchin is written in an interpreted language (shell), so it might be
|
|
hard to distribute usefully without providing the source code.
|
|
2. Urchin just runs tests; it doesn't get compiled with the rest of the
|
|
code (also because it's in shell). Thus, I think a GPL license on
|
|
Urchin wouldn't infect the code being tested.
|
|
|
|
This is as far as I have gotten with contemplating license changes. For now
|
|
we're sticking with the original MIT-style license, but it's easy to change
|
|
licenses later.
|
|
|
|
Nagios plugins
|
|
-----------------
|
|
It would be cool to run Nagios plugins with Urchin. This is already possible,
|
|
actually, but it might be worth giving some special thought to it.
|
|
https://nagios-plugins.org/doc/guidelines.html
|
|
|
|
Source setup and teardown
|
|
--------------------
|
|
If setup and teardown are sourced instead of executed, maybe we can more
|
|
cleanly create and teardown temporary files.
|
|
|
|
(
|
|
. ./setup
|
|
./$thetestfile
|
|
. ./teardown
|
|
)
|
|
|
|
On the other hand, this could just be sourced explicitly in the test file,
|
|
without the special setup and teardown feature.
|
|
|
|
Run on a file
|
|
----------------
|
|
Presently you can run urchin only on a directory.
|
|
It would be neat if you could run it on a file as well.
|
|
|
|
This occurred to me when I wanted to run
|
|
|
|
urchin test/fast/Unit\ tests/nvm_ls_current
|
|
|
|
on the nvm tests. I wound up running this instead.
|
|
|
|
urchin test/fast/Unit\ tests/ | grep nvm_ls_current
|
|
|
|
The Molly guard would be assessed, and the corresponding setup, setup_dir,
|
|
teardown, and teardown_dir files would be run in the appropriate order.
|
|
|
|
In order to know how far up the tree to evaluate the setup, &c. files,
|
|
I think it would make sense to require that a ".urchin" file be placed in the
|
|
root of the tests. Urchin would keep going up until it sees this file, and it
|
|
would evaluate the appropriate setup, &c. files from there down to the
|
|
particular test file of interest. We would also use this for testing
|
|
directtories more correctly.
|
|
|
|
Running automated tasks
|
|
-------------------------
|
|
Urchin might be appropriate for if you have lots of tasks that you want to run
|
|
periodically; add an urchin call to your crontab, and call all of your other
|
|
tasks with urchin. Here are some features that might make urchin better for
|
|
this sort of thing.
|
|
|
|
* Time how long each test/job takes
|
|
* Optionally kill tests/jobs after a specific timeout threshold
|
|
* Send output of different tests/jobs to different files for each file
|
|
descriptor (STDOUT, STDERR)
|