295 lines
7.3 KiB

Things I want
Wider testing
Test in other environments
* Specify a few different ones with Nix.
* Some sort of BSD
* Windows
Package for package managers.
* I want NixOS, of course.
* Debian is probably the big one.
Other interesting package managers
* Update the npm package
* Homebrew (for Mac)
Try running Urchin in Windows somehow. Interpreters include
* CygWin (https://www.cygwin.com/)
* https://cygwin.com/setup-x86.exe
* MSYS (http://mingw.org/wiki/msys)
* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
* Git for Windows (https://git-scm.com/download/win)
* https://github.com/git-for-windows/git/releases/download/v2.7.2.windows.1/Git-2.7.2-32-bit.exe
* win-bash (http://win-bash.sourceforge.net/)
Add shall to my NYC*BUG talk.
#!/usr/bin/env shall
echo This runs in several shells.
List some shell linters somewhere.
* ShellCheck
* checkbashisms
Rename to something other than "test"?
Maybe wait until I have a use for this.
More sort alternatives
JSON.sh test suite
Running in multiple environments
Setup for other environments includes the following.
* Installing packages
* `touch .zshrc`
* Copy urchin and tests
I want to change the way that fixtures are done.
Instead of using setup, teardown, &c., use ordinary programs from within
your tests. For example.
# tests/.fixtures/tmp-dir
tmp=$(mktemp -d)
cd $tmp
cd /
rm -Rf $tmp
exit $code
# tests/blah
../.fixtures/tmp-dir 'blah blah blah'
It's best if I can wrap a bunch of commands in braces or paratheses
rather than just one command. Is there a nice way to do that?
Once I have this new way, I guess I might as well keep the old way.
I think the setup, teardown thing can be easier if you only have simple
fixtures. And since I'm going to keep it, I'm going to add another one.
* setup_dir runs once for the present directory.
* setup_children runs once for each child.
* setup_file runs once for each file descendent.
The present `setup` is renamed to `setup_children`, and the new
`setup_file` runs on each file (not directory) that is a child,
grandchild, great-grandchild, and so on.
Dependency checking
You might want to skip tests based on dependencies. Currently you can
conditionally skip tests one at a time by exiting with code 3. I want to
be able to skip an entire directory.
So we add a new magic file called `dep`. If it exists, it is run before
everything else in the directory.
* If it exits with code 0, tests continue as if dep did not exist.
* If it exits with code 3, all tests in the directory are marked as
* If it exits with code 1, all tests in the directory are marked as
failed. To make the implementation easier, I'll probably treat the
directory as a single test in this case.
A note on magic files
It is nice to have access to things like setup and dep (magic files)
once in a while, but you need to be doing rather substantial testing
before they make your test suite simpler; the documentation should
strongly recommend writing your tests without magic files and then
refactoring and only then considering moving things to magic files.
Remote testing
In order to test Urchin across multiple operating systems, I have
already added tests in Urchin's test suite that run Urchin tests in
remote servers. I would like to move this to Urchin itself so that
Urchin can test other things on remote servers.
Urchin's output presently looks like this.
Cycling with the following shells: sh bash dash mksh zsh
Running tests at 2016-04-07T12:33:49
> --timeout output
. bash (0 seconds)
. dash (0 seconds)
. mksh (0 seconds)
. sh (0 seconds)
. zsh (0 seconds)
Done, took 1 second.
5 tests passed.
0 tests skipped.
0 tests failed.
After the change, the output should look like this.
Cycling with the following shells: sh dash mksh
Running tests at 2016-04-07T12:33:49
> --timeout output
. dash on localhost (0 seconds)
. dash on localhost:8080 (0 seconds)
. dash on tlevine@hpux.polarhome.com (0 seconds)
. mksh on localhost (0 seconds)
. mksh on tlevine@hpux.polarhome.com (0 seconds)
. sh on localhost (0 seconds)
. sh on localhost:8080 (0 seconds)
. sh on tlevine@hpux.polarhome.com (0 seconds)
Done, took 1 second.
8 tests passed.
0 tests skipped.
0 tests failed.
This is just how the output should look; the tests run in whatever order
makes sense.
Both md5sum and md5 should be supported.
Trouble logging in to hpux, irix, miros, netbsd, tru64, qnx, ....
$ rsync -e 'ssh -p 785' urchin tlevine@hpux.polarhome.com:.blah
HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
Welcome to HPUX/PA... member of polarhome.com realm
bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: remote command not found (code 127) at io.c(226)
OpenIndiana grep does not support -q
I get `/urchin: syntax error at line 84: \`}' unexpected` on
unixware and solaris.
> tlevine@hpux.polarhome.com -p 785
F sh (8 seconds)
| HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
| Welcome to HPUX/PA... member of polarhome.com realm
| HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
| Welcome to HPUX/PA... member of polarhome.com realm
| mktemp: option requires an argument -- d
| ./urchin[96]: /tmp/tlevinea21441/log: Cannot create the specified file.
tlevine@hpux64$ ./urchin tests/ -n -vv
date: bad format character - s
So I need a portable seconds-from epoch
I also need to handle when no arguments are passed to urchin.
Exit code is wrong for which on HP-UX
## `$(...)`
Solaris doesn't support `$(...)`; you need `\`...\`` instead.
tlevine@solaris$ ./urchin --run-in-series tests/Errors/
./urchin: syntax error at line 84: `tmp=$' unexpected
I use this a lot.
$ grep -c '\$(' urchin
Update tests to support
* md5
* rsync
* mktemp
* epoch
* Report cycling by default
* New format for reporting cycling
Support systems without rsync
BSD mktemp
| NetBSD 6.1.3
| Welcome to NetBSD ...member of polarhome.com realm
| Usage: mktemp [-dqu] [-p <tmpdir>] {-t prefix | template ...}
| mkdir: : No such file or directory
| ./urchin: cannot create /log: permission denied
md5: unknown option -- q
usage: cksum [-n] [-a algorithm [-ptx] [-s string]] [-o 1|2]
[file ... | -c [-w] [sumfile]]
Things I can use to make things better
$IFS and set --
Redirection, especiall <<-
Maybe fifo
for x in "$@"
Variable assignments specified with special built-in utilities remain in
effect after the built-in completes; this shall not be the case with a
regular built-in or other utility.