Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
9d1b16ed57 |
33
HISTORY
33
HISTORY
@ -1,17 +1,7 @@
|
||||
HISTORY
|
||||
=======
|
||||
|
||||
Version 0.2.0 (unstable)
|
||||
---------------------
|
||||
### Cross-OS testing
|
||||
I have started testing Urchin across multiple operating systems.
|
||||
This gives access to more shells, as some shels are easier to install on
|
||||
certain operating systems.
|
||||
|
||||
With this cross-OS test suite, I have extended support to more shells.
|
||||
A later version of Urchin could include a remote testing feature.
|
||||
|
||||
Version 0.1.0 (stable)
|
||||
Version 0.1.0-rc1
|
||||
---------------------
|
||||
This release includes breaking changes.
|
||||
|
||||
@ -22,18 +12,18 @@ test suite, as we need to know how far up to apply the setup
|
||||
and teardown files.
|
||||
|
||||
The Urchin root directory is determined by moving higher in the directory
|
||||
tree in search of a file named `.urchin_root`.
|
||||
tree in search of a file named `.urchin`.
|
||||
The closest directory that contains such a file is considered the root.
|
||||
In the following filesystem, for example, `/a/b/c` would be the root.
|
||||
|
||||
mkdir -p /a/b/c/d
|
||||
touch /a/b/c/d/e
|
||||
chmod +x /a/b/c/d/e
|
||||
touch /a/b/c/.urchin_root
|
||||
touch /a/b/c/.urchin
|
||||
urchin /a/b/c/d
|
||||
|
||||
There are two situations in which we would stop looking without having
|
||||
found a `.urchin_root` file.
|
||||
found a `.urchin` file.
|
||||
|
||||
1. The system root, `/`, because we can't go any higher
|
||||
2. A directory that starts with a dot, because an urchin call on a higher
|
||||
@ -82,7 +72,7 @@ might look for the dependency and then skip if it does not see the dependency.
|
||||
It might look like this.
|
||||
|
||||
#!/bin/sh
|
||||
if ! which inkscape; then
|
||||
if which inkscape; then
|
||||
exit 3 # status code 3 for skip
|
||||
fi
|
||||
inkscape blah blah ...
|
||||
@ -107,13 +97,6 @@ turned on.
|
||||
|
||||
Parallel processing and shell cycling are both enabled by default.
|
||||
|
||||
You may want make only some directories run in series, you can create
|
||||
".urchin_dir" files in those directories.
|
||||
If .urchin_dir contains the phrase "series", run that directory in series
|
||||
rather than in parallel.
|
||||
This is helpful when directories actually need to run in series
|
||||
and also when running all your tests in parallel crashes your computer.
|
||||
|
||||
### Options
|
||||
Long options are now available for all command line flags.
|
||||
For example, the `-s` flag is now available as `--shell` as well.
|
||||
@ -181,6 +164,10 @@ set as an environment variable, and the latter was set with the -s flag..
|
||||
Urchin now uses the -s flag for both of these settings, and it mostly ignores
|
||||
the exported TEST_SHELL variable.
|
||||
|
||||
Urchin also inspects the shebang line differently. Previously, Urchin would
|
||||
vary the shells with which a test is run if the shebang line either was absent
|
||||
or was #!/bin/sh. Now it varies the shell only if the shebang line is absent.
|
||||
|
||||
If you pass -n/--disable-cycling, Urchin will invoke tests ordinarily and will
|
||||
only set the TEST_SHELL variable if it does not exist. If the TEST_SHELL
|
||||
variable is absent, it will be set to /bin/sh.
|
||||
@ -242,7 +229,7 @@ other Urchin call), and the test suite is recursively descended. Setup and
|
||||
teardown files are sourced, and everything but the specified test file is
|
||||
otherwise ignored.
|
||||
|
||||
If you don't explicitly specify the Urchin root with a .urchin_root file, we
|
||||
If you don't explicitly specify the Urchin root with a .urchin file, we
|
||||
consider the test suite root directory to be the parent of the file that
|
||||
you ran Urchin on.
|
||||
|
||||
|
||||
8
Makefile
8
Makefile
@ -1,8 +0,0 @@
|
||||
.PHONY: test install
|
||||
|
||||
test:
|
||||
./urchin tests
|
||||
./urchin -s sh -v ./cross-os-tests
|
||||
|
||||
install:
|
||||
cp ./urchin /usr/bin
|
||||
42
SORTING
Normal file
42
SORTING
Normal file
@ -0,0 +1,42 @@
|
||||
On the criteria for ordering
|
||||
==============================
|
||||
The following sh code creates several files in a directory and then
|
||||
calls "*", listing them in order.
|
||||
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | while read line; do
|
||||
touch -- "${line}"
|
||||
done
|
||||
for file in *; do echo "$file"; done
|
||||
|
||||
On one computer, running FreeBSD, the order is apparently ASCIIbetical.
|
||||
|
||||
! e
|
||||
- d
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
|
||||
On another computer, running NixOS, the following commands print results
|
||||
in dictionary order. I'm not exactly sure what dictionary order is, but
|
||||
it is something like sorting on the alphabetical characters before
|
||||
sorting on the rest of the line.
|
||||
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
- d
|
||||
! e
|
||||
|
||||
While I don't really know what dictionary order is, I was able to determine
|
||||
that the above results are in dictionary order because of my investigation of
|
||||
incompatible implementations of sort. Consider the following two sort
|
||||
commands.
|
||||
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort -d
|
||||
|
||||
With BSD sort, the first of these commands print ASCIIbetical order and
|
||||
the second prints dictionary order. With GNU sort, both print dictionary
|
||||
order.
|
||||
|
||||
How annoying.
|
||||
252
TODO
252
TODO
@ -26,14 +26,11 @@ Windows
|
||||
Try running Urchin in Windows somehow. Interpreters include
|
||||
|
||||
* CygWin (https://www.cygwin.com/)
|
||||
* https://cygwin.com/setup-x86.exe
|
||||
* MSYS (http://mingw.org/wiki/msys)
|
||||
* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
|
||||
* Git for Windows (https://git-scm.com/download/win)
|
||||
* https://github.com/git-for-windows/git/releases/download/v2.7.2.windows.1/Git-2.7.2-32-bit.exe
|
||||
* win-bash (http://win-bash.sourceforge.net/)
|
||||
|
||||
|
||||
shall
|
||||
----------
|
||||
Add shall to my NYC*BUG talk.
|
||||
@ -49,246 +46,15 @@ List some shell linters somewhere.
|
||||
* checkbashisms
|
||||
|
||||
|
||||
Set parallel and series
|
||||
----------
|
||||
I want to be able to make only some directories run in series.
|
||||
|
||||
* Rename .urchin to .urchin_root.
|
||||
* Look for a .urchin_dir file.
|
||||
* If .urchin_dir contains "series", run that directory in series rather
|
||||
than in parallel.
|
||||
|
||||
Rename to something other than "test"?
|
||||
----------
|
||||
Maybe wait until I have a use for this.
|
||||
|
||||
More sort alternatives
|
||||
-----------
|
||||
awk
|
||||
https://stackoverflow.com/questions/20250937/sorting-lines-in-a-file-alphabetically-using-awk-and-or-sed
|
||||
bash
|
||||
https://stackoverflow.com/questions/7442417/how-to-sort-an-array-in-bash
|
||||
|
||||
Alternatives
|
||||
--------------
|
||||
JSON.sh test suite
|
||||
|
||||
Running in multiple environments
|
||||
-----------------------------------
|
||||
Setup for other environments includes the following.
|
||||
|
||||
* Installing packages
|
||||
* `touch .zshrc`
|
||||
* Copy urchin and tests
|
||||
|
||||
|
||||
Fixtures
|
||||
------------
|
||||
I want to change the way that fixtures are done.
|
||||
|
||||
Instead of using setup, teardown, &c., use ordinary programs from within
|
||||
your tests. For example.
|
||||
|
||||
# tests/.fixtures/tmp-dir
|
||||
tmp=$(mktemp -d)
|
||||
cd $tmp
|
||||
@$
|
||||
code=$?
|
||||
cd /
|
||||
rm -Rf $tmp
|
||||
exit $code
|
||||
|
||||
# tests/blah
|
||||
../.fixtures/tmp-dir 'blah blah blah'
|
||||
|
||||
It's best if I can wrap a bunch of commands in braces or paratheses
|
||||
rather than just one command. Is there a nice way to do that?
|
||||
|
||||
Once I have this new way, I guess I might as well keep the old way.
|
||||
I think the setup, teardown thing can be easier if you only have simple
|
||||
fixtures. And since I'm going to keep it, I'm going to add another one.
|
||||
|
||||
* setup_dir runs once for the present directory.
|
||||
* setup_children runs once for each child.
|
||||
* setup_file runs once for each file descendent.
|
||||
|
||||
The present `setup` is renamed to `setup_children`, and the new
|
||||
`setup_file` runs on each file (not directory) that is a child,
|
||||
grandchild, great-grandchild, and so on.
|
||||
|
||||
Dependency checking
|
||||
----------------------
|
||||
You might want to skip tests based on dependencies. Currently you can
|
||||
conditionally skip tests one at a time by exiting with code 3. I want to
|
||||
be able to skip an entire directory.
|
||||
|
||||
So we add a new magic file called `dep`. If it exists, it is run before
|
||||
everything else in the directory.
|
||||
|
||||
* If it exits with code 0, tests continue as if dep did not exist.
|
||||
* If it exits with code 3, all tests in the directory are marked as
|
||||
skipped.
|
||||
* If it exits with code 1, all tests in the directory are marked as
|
||||
failed. To make the implementation easier, I'll probably treat the
|
||||
directory as a single test in this case.
|
||||
|
||||
A note on magic files
|
||||
-------------------------
|
||||
It is nice to have access to things like setup and dep (magic files)
|
||||
once in a while, but you need to be doing rather substantial testing
|
||||
before they make your test suite simpler; the documentation should
|
||||
strongly recommend writing your tests without magic files and then
|
||||
refactoring and only then considering moving things to magic files.
|
||||
|
||||
Remote testing
|
||||
----------------
|
||||
In order to test Urchin across multiple operating systems, I have
|
||||
already added tests in Urchin's test suite that run Urchin tests in
|
||||
remote servers. I would like to move this to Urchin itself so that
|
||||
Urchin can test other things on remote servers.
|
||||
|
||||
Urchin's output presently looks like this.
|
||||
|
||||
Cycling with the following shells: sh bash dash mksh zsh
|
||||
Running tests at 2016-04-07T12:33:49
|
||||
|
||||
Flags/
|
||||
> --timeout output
|
||||
. bash (0 seconds)
|
||||
. dash (0 seconds)
|
||||
. mksh (0 seconds)
|
||||
. sh (0 seconds)
|
||||
. zsh (0 seconds)
|
||||
|
||||
Done, took 1 second.
|
||||
5 tests passed.
|
||||
0 tests skipped.
|
||||
0 tests failed.
|
||||
|
||||
After the change, the output should look like this.
|
||||
|
||||
Cycling with the following shells: sh dash mksh
|
||||
Running tests at 2016-04-07T12:33:49
|
||||
|
||||
Flags/
|
||||
> --timeout output
|
||||
. dash on localhost (0 seconds)
|
||||
. dash on localhost:8080 (0 seconds)
|
||||
. dash on tlevine@hpux.polarhome.com (0 seconds)
|
||||
. mksh on localhost (0 seconds)
|
||||
. mksh on tlevine@hpux.polarhome.com (0 seconds)
|
||||
. sh on localhost (0 seconds)
|
||||
. sh on localhost:8080 (0 seconds)
|
||||
. sh on tlevine@hpux.polarhome.com (0 seconds)
|
||||
|
||||
Done, took 1 second.
|
||||
8 tests passed.
|
||||
0 tests skipped.
|
||||
0 tests failed.
|
||||
|
||||
This is just how the output should look; the tests run in whatever order
|
||||
makes sense.
|
||||
|
||||
Bugs
|
||||
-------
|
||||
|
||||
Both md5sum and md5 should be supported.
|
||||
|
||||
Trouble logging in to hpux, irix, miros, netbsd, tru64, qnx, ....
|
||||
|
||||
$ rsync -e 'ssh -p 785' urchin tlevine@hpux.polarhome.com:.blah
|
||||
HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
|
||||
Welcome to HPUX/PA... member of polarhome.com realm
|
||||
|
||||
bash: rsync: command not found
|
||||
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
|
||||
rsync error: remote command not found (code 127) at io.c(226)
|
||||
[sender=3.1.1]
|
||||
|
||||
|
||||
OpenIndiana grep does not support -q
|
||||
|
||||
I get `/urchin: syntax error at line 84: \`}' unexpected` on
|
||||
unixware and solaris.
|
||||
|
||||
mktemp
|
||||
|
||||
> tlevine@hpux.polarhome.com -p 785
|
||||
F sh (8 seconds)
|
||||
|
|
||||
| HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
|
||||
| Welcome to HPUX/PA... member of polarhome.com realm
|
||||
|
|
||||
|
|
||||
| HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
|
||||
| Welcome to HPUX/PA... member of polarhome.com realm
|
||||
|
|
||||
| mktemp: option requires an argument -- d
|
||||
| ./urchin[96]: /tmp/tlevinea21441/log: Cannot create the specified file.
|
||||
|
||||
date
|
||||
|
||||
tlevine@hpux64$ ./urchin tests/ -n -vv
|
||||
date: bad format character - s
|
||||
|
||||
So I need a portable seconds-from epoch
|
||||
|
||||
I also need to handle when no arguments are passed to urchin.
|
||||
|
||||
Exit code is wrong for which on HP-UX
|
||||
|
||||
## `$(...)`
|
||||
Solaris doesn't support `$(...)`; you need `\`...\`` instead.
|
||||
|
||||
tlevine@solaris$ ./urchin --run-in-series tests/Errors/
|
||||
./urchin: syntax error at line 84: `tmp=$' unexpected
|
||||
|
||||
I use this a lot.
|
||||
|
||||
$ grep -c '\$(' urchin
|
||||
52
|
||||
|
||||
Darn
|
||||
|
||||
|
||||
|
||||
|
||||
Update tests to support
|
||||
|
||||
* md5
|
||||
* rsync
|
||||
* mktemp
|
||||
* epoch
|
||||
* Report cycling by default
|
||||
* New format for reporting cycling
|
||||
|
||||
|
||||
|
||||
|
||||
Support systems without rsync
|
||||
|
||||
|
||||
BSD mktemp
|
||||
|
||||
| NetBSD 6.1.3
|
||||
| Welcome to NetBSD ...member of polarhome.com realm
|
||||
|
|
||||
| Usage: mktemp [-dqu] [-p <tmpdir>] {-t prefix | template ...}
|
||||
| mkdir: : No such file or directory
|
||||
| ./urchin: cannot create /log: permission denied
|
||||
|
||||
|
||||
NetBSD
|
||||
|
||||
md5: unknown option -- q
|
||||
usage: cksum [-n] [-a algorithm [-ptx] [-s string]] [-o 1|2]
|
||||
[file ... | -c [-w] [sumfile]]
|
||||
|
||||
|
||||
|
||||
|
||||
Things I can use to make things better
|
||||
------------------------
|
||||
${x##*blah}
|
||||
$IFS and set --
|
||||
Redirection, especiall <<-
|
||||
Maybe fifo
|
||||
for x in "$@"
|
||||
until
|
||||
readonly
|
||||
getopts
|
||||
|
||||
Variable assignments specified with special built-in utilities remain in
|
||||
effect after the built-in completes; this shall not be the case with a
|
||||
regular built-in or other utility.
|
||||
|
||||
@ -1,14 +0,0 @@
|
||||
params="$(basename "${0}")"
|
||||
|
||||
hostname="$(echo "${params}" | cut -d\ -f1)"
|
||||
if echo "${params}" | grep -q \ ; then
|
||||
flags="$(echo "${params}" | cut -d\ -f2-)"
|
||||
fi
|
||||
|
||||
urchin_dir=.urchin-cross-shell-test
|
||||
|
||||
rsync --archive -e "ssh ${flags}" $RSYNC_FLAGS \
|
||||
../urchin ../tests "${hostname}":"${urchin_dir}" ||
|
||||
scp -r ${flags} ../urchin ../tests "${hostname}":"${urchin_dir}"
|
||||
ssh "${hostname}" ${flags} \
|
||||
"cd ${urchin_dir} && ./urchin --run-in-series tests"
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
# apt-get install bash dash ksh posh pdksh mksh yash zsh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
RSYNC_FLAGS='--rsync-path=/usr/local/bin/rsync'
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
# SSH public key needs to be in ~/.etc/ssh/authorized_keys
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
98
docs/SORTING
98
docs/SORTING
@ -1,98 +0,0 @@
|
||||
On the criteria for ordering
|
||||
==============================
|
||||
|
||||
I was confused by the documentation for sort's "-d" flag. This confusion
|
||||
relates to GNU coreutil's locale-specific sort. [^]
|
||||
|
||||
Below I discuss sort order differences between different implementations
|
||||
of sort and of sh "*" for my particular environments.
|
||||
|
||||
Sorting with sort
|
||||
------------
|
||||
Consider the following two sort commands.
|
||||
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort -d
|
||||
|
||||
With BusyBox v1.23.2 on NixOS 15.09, the first of these commands returns
|
||||
ASCIIbetical order,
|
||||
|
||||
! e
|
||||
- d
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
|
||||
and the second returns dictionary order.
|
||||
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
- d
|
||||
! e
|
||||
|
||||
With GNU coreutils version 8.24 on NixOS, both commands return
|
||||
dictionary order. The same is true for GNU coreutils version 8.23 on
|
||||
Debian Wheezy.
|
||||
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
- d
|
||||
! e
|
||||
|
||||
IEEE Std 1003.1, 2013 Edition [^^] specifies that the "-d" flag should
|
||||
enable dictionary order. All of these versions of sort have clear
|
||||
documentation about the order that should be returned when the "-d" flag
|
||||
is set, (See --help, man, or info.) and the implementations match the
|
||||
documentation as far as I can tell.
|
||||
|
||||
I have found no explicit documentation from any relevant source as to
|
||||
what the default sort order should be. On the other hand, they all
|
||||
suggest that "-d" produces an order different from the default order.
|
||||
|
||||
In GNU coreutils 8.24, for example, "-d" is a direction to "consider
|
||||
only blanks and alphanumeric characters". It lacks any mention that the
|
||||
"-d" flag has no effect or that it is the default. Furthermore, on my
|
||||
first reading, I took it to mean that the default is to consider all
|
||||
characters and that "-d" limits the considered characters to blanks and
|
||||
alphanumeric characters.
|
||||
|
||||
|
||||
Sorting in *
|
||||
-------------
|
||||
I think this is related to the order returned by "*" in sh.
|
||||
The following sh code creates several files in a directory and then
|
||||
calls "*", listing them in order.
|
||||
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | while read line; do
|
||||
touch -- "${line}"
|
||||
done
|
||||
for file in *; do echo "$file"; done
|
||||
|
||||
On one computer, running FreeBSD, the order is apparently
|
||||
ASCIIbetical.
|
||||
|
||||
! e
|
||||
- d
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
|
||||
On two GNU systems, running NixOS and Debian, respectively, output is
|
||||
in dictionary order. I'm not exactly sure what dictionary order is, but
|
||||
it is something like sorting on the alphabetical characters before
|
||||
sorting on the rest of the line.
|
||||
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
- d
|
||||
! e
|
||||
|
||||
(I don't really know what dictionary order is, I was able to determine
|
||||
that the above results are in dictionary order because of my investigation of
|
||||
incompatible implementations of sort.)
|
||||
|
||||
[^] https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021
|
||||
[^^] http://pubs.opengroup.org/onlinepubs/9699919799/
|
||||
@ -1,109 +0,0 @@
|
||||
Here I discuss Urchin's general execution flow and how it is handled
|
||||
specifically when tests are run on remote environments.
|
||||
|
||||
|
||||
Steps of an Urchin run
|
||||
----------------------
|
||||
When Urchin runs a directory of files, it goes through the following
|
||||
steps.
|
||||
|
||||
1. Head
|
||||
2. Test
|
||||
3. Foot
|
||||
4. Reporting
|
||||
|
||||
Urchin stores files in a temporary directory, creating a new directory
|
||||
on each invocation. The directory contains these things.
|
||||
|
||||
* head (file)
|
||||
* test (file)
|
||||
* foot (file)
|
||||
* stdout (directory)
|
||||
|
||||
When run on remotes, the temporary directory corresponding to the local
|
||||
master process additionally has these files.
|
||||
|
||||
* remote-test
|
||||
|
||||
Messages from the head, test, and foot steps go in the corresponding
|
||||
files. In the head and foot phases, messages are just simple prints.
|
||||
Messages from the test phase always correspond to a particular test
|
||||
file, and they are written to the test file in a delimiter-separated
|
||||
format.
|
||||
|
||||
Stdout and stderr from test runs are written to files in the stdout
|
||||
directory, one file per test file per shell that the file is run in.
|
||||
|
||||
The reporting phase
|
||||
----------------------
|
||||
In most cases Urchin begins printing to the screen only during the
|
||||
reporting phase. The only case where anything is printed beforehand is
|
||||
when Urchin is run with -vvvv; that sets "+x", so the commands are
|
||||
printed as they run, though all other output is still suppressed.
|
||||
|
||||
Test results are reported in the reporting phase. Four output formats
|
||||
are available.
|
||||
|
||||
1. Urchin's human-readable format (default)
|
||||
2. Test Anything Protocol
|
||||
3. Delimiter-separated values (used internally)
|
||||
4. Remote Urchin worker output
|
||||
|
||||
Most of the output is generated based on the delimiter-separated values
|
||||
in the test log file. The first two formats also include stdout and
|
||||
stderr from the tests, depending on verbosity level flags; when it needs
|
||||
these, Urchin reads them from appropriate files in the temporary
|
||||
directory.
|
||||
|
||||
I could discuss the further details of each format elsewhere.
|
||||
|
||||
Remotes
|
||||
----------------------
|
||||
When Urchin runs tests on a remote, it copies tests to the remote and
|
||||
then calls Urchin on the remote with "--format=remote". This specifies
|
||||
the following.
|
||||
|
||||
* The temporary directory should be kept, rather than deleted, after
|
||||
Urchin runs.
|
||||
* The path of the temporary directory should be printed as output.
|
||||
* No other output should be printed to stdout.
|
||||
|
||||
After the remote Urchin finishes running, the local urchin downloads
|
||||
the remote Urchin's test log file from the temporary directory.
|
||||
It modifies the file to include the remote's name and then concatenates
|
||||
the result to the "remote-test" file in the local temporary directory.
|
||||
For example, the file from the remote might look like this,
|
||||
|
||||
:sh:Counting tests/.test/faila:0:not_ok
|
||||
|
||||
and the result might look like this.
|
||||
|
||||
nsa:sh:Counting tests/.test/faila:0:not_ok
|
||||
|
||||
This gets processed in the reporting step like usual, according to
|
||||
whatever format is specified. Instead of printing just "sh" as the
|
||||
environment in which the particular test was run, the report will print
|
||||
"sh on nsa".
|
||||
|
||||
When it needs the stdout files, it prints them over ssh.
|
||||
|
||||
New flags
|
||||
----------
|
||||
In making this remotes feature, I wound up adding some others.
|
||||
|
||||
-r, --remote SSH host to use as a remote
|
||||
-F, --format Output format, one of "urchin", "tap", "dsv", "remote"
|
||||
|
||||
Urchin runs only locally by default. If you pass at least one --remote
|
||||
flag, Urchin runs tests only on the specified remotes; it can't run both
|
||||
locally and remotely in the same run. If you want to do that, you could
|
||||
wait until I add that feature, or you can add "localhost" as a remote.
|
||||
|
||||
Settings that I'm thinking about
|
||||
|
||||
* Port for rsync/ssh
|
||||
* SSH protocol version
|
||||
* --rsync-path
|
||||
|
||||
Can those all be set in ssh_config? Probably not --rsync-path, but
|
||||
I guess I could just fix it on the remote.
|
||||
@ -1,39 +0,0 @@
|
||||
# A NixOS container to protect against accidental fork bombs
|
||||
#
|
||||
# Put this in /var/lib/containers/test/etc/nixos/configuration.nix
|
||||
# See https://nixos.org/wiki/NixOS:Containers
|
||||
{ config, lib, pkgs, ... }:
|
||||
|
||||
with lib;
|
||||
|
||||
{ boot.isContainer = true;
|
||||
networking.hostName = mkDefault "urchin";
|
||||
networking.useDHCP = false;
|
||||
|
||||
environment.systemPackages = with pkgs; [
|
||||
# Urchin
|
||||
bash dash mksh zsh
|
||||
busybox
|
||||
|
||||
# Other
|
||||
vim git rsync tmux
|
||||
];
|
||||
security.pam.loginLimits = [
|
||||
# Prevent accidental fork bombs.
|
||||
{ domain = "*"; item = "nproc"; type = "hard"; value = "200"; }
|
||||
];
|
||||
services.openssh = {
|
||||
enable = true;
|
||||
passwordAuthentication = false;
|
||||
};
|
||||
users.extraUsers.user = {
|
||||
name = "tlevine";
|
||||
uid = 1000;
|
||||
isNormalUser = true;
|
||||
home = "/home/tlevine";
|
||||
extraGroups = [ "users" "wheel" ];
|
||||
openssh.authorizedKeys.keys = [
|
||||
"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGvQyzr42/96acUTUedaeM2ee+DMt9bkxeurdeXji9sNE10MjjAUFtxPmSI8/BUZW2/a9ByblfaJEI+H+kFVPjVr+QGKXZluxcFMj2BLbH53fi9xLgoQRjb2aAXutb2Bp74/E8R1K+CuFfRRGQ5Spdnv44SLt04D6JbBLcLIcWTpQ4v5RaYr2U27jfiF9z0m+/opxvowEy2gnqlEXFxFk8jZHT4K0uLWm2ENjT6OpyOx8hWcKeAN2vRVRex3pJfSzswn0LpuCrM1rUZ4DRE+FABi8N21Q3MBaMRkwnZPwaZwKzv06q8bu23jYTqK5BrUPtOXeeVuroQXMc12H/6/Nh laptop"
|
||||
];
|
||||
};
|
||||
}
|
||||
@ -1,36 +0,0 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
# Create the container.
|
||||
if ! nixos-container list | grep ^urchin$ > /dev/null; then
|
||||
sudo nixos-container create urchin
|
||||
fi
|
||||
|
||||
# Configure the container.
|
||||
sudo cp configuration.nix \
|
||||
/var/lib/containers/urchin/etc/nixos/configuration.nix
|
||||
sudo nixos-container update urchin
|
||||
sudo nixos-container start urchin
|
||||
|
||||
# Create the git repository.
|
||||
host="tlevine@$(nixos-container show-ip urchin)"
|
||||
ssh "${host}" 'if mkdir urchin 2> /dev/null; then
|
||||
cd urchin
|
||||
git init
|
||||
git config --add receive.denyCurrentBranch ignore
|
||||
fi
|
||||
'
|
||||
|
||||
# Push to the git repository
|
||||
git push "${host}":urchin
|
||||
|
||||
# Print information
|
||||
echo "Log in:
|
||||
|
||||
ssh ${host}
|
||||
|
||||
Add git remote
|
||||
|
||||
git remote add ${host} container
|
||||
|
||||
"
|
||||
@ -12,7 +12,7 @@
|
||||
"bin": "./urchin",
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://git.sdf.org/tlevine/urchin"
|
||||
"url": "git://github.com/tlevine/urchin.git"
|
||||
},
|
||||
"keywords": [
|
||||
"shell",
|
||||
1
packages/.gitignore
vendored
1
packages/.gitignore
vendored
@ -1 +0,0 @@
|
||||
*.tar.gz
|
||||
@ -1,11 +0,0 @@
|
||||
#!/bin/sh
|
||||
name=urchin-$(../urchin --version)
|
||||
|
||||
tmp=$(mktemp -d)
|
||||
mkdir $tmp/$name
|
||||
cp ../urchin ../readme.md ../AUTHORS ../COPYING $tmp/$name
|
||||
cd $tmp
|
||||
tar czf $name.tar.gz $name
|
||||
cd - > /dev/null
|
||||
mv $tmp/$name.tar.gz .
|
||||
rm -R $tmp
|
||||
59
readme.md
59
readme.md
@ -1,5 +1,3 @@
|
||||
**The repository at https://github.com/tlevine/urchin will go away. New location is https://git.sdf.org/tlevine/urchin.**
|
||||
|
||||
__ _
|
||||
__ ____________/ /_ (_)___
|
||||
/ / / / ___/ ___/ __ \/ / __ \
|
||||
@ -19,7 +17,7 @@ have shells called "tests".
|
||||
Urchin's tests are written in Urchin, so you can run them to see what Urchin
|
||||
is like. Clone the repository
|
||||
|
||||
git clone https://git.sdf.org/tlevine/urchin
|
||||
git clone git://github.com/tlevine/urchin.git
|
||||
|
||||
Run the tests
|
||||
|
||||
@ -45,15 +43,25 @@ Urchin depends on the following programs.
|
||||
* timeout
|
||||
* sort
|
||||
|
||||
Vanilla installations of modern BSD and GNU systems usually include all
|
||||
of these programs.
|
||||
All of the above programs are usually included on base BSD installations.
|
||||
On GNU systems it should be sufficient to install the busybox package.
|
||||
|
||||
Urchin uses sort to format its output. GNU sort (as of GNU coreutils version
|
||||
8.24) lacks the ability to sort in lexicographic order, and this feature is
|
||||
necessary for the output to look right. If your version of sort lacks this
|
||||
feature, Urchin will try to use one of the following tools for sorting.
|
||||
|
||||
If no acceptable sorting program is available, Urchin will print a warning
|
||||
and use the incomplete sort that is installed on your system. This is not a
|
||||
big deal; if your test files all start with alphanumeric letters, the output
|
||||
should look fine.
|
||||
|
||||
## Install
|
||||
Urchin is contained in a single file, so you can install it by copying it to a
|
||||
directory in your `PATH`. For example, you can run the following as root.
|
||||
|
||||
cd /usr/local/bin
|
||||
wget https://git.sdf.org/tlevine/urchin/raw/branch/master/urchin
|
||||
wget https://raw.githubusercontent.com/tlevine/urchin/v0.0.6/urchin
|
||||
chmod +x urchin
|
||||
|
||||
Urchin can be installed with npm too.
|
||||
@ -110,36 +118,14 @@ Files are only run if they are executable, and files beginning with `.` are
|
||||
ignored. Thus, fixtures and libraries can be included sloppily within the test
|
||||
directory tree. The test passes if the file exits 0; otherwise, it fails.
|
||||
|
||||
urchin looks for files within a directory in the following manner,
|
||||
Tests files and subdirectories are run in ASCIIbetical order within each
|
||||
directory; that is,
|
||||
urchin looks for files within a directory in the following manner.
|
||||
|
||||
for file in *; do
|
||||
do_something_with_test_file $file
|
||||
done
|
||||
|
||||
so files are run in whatever order `*` produces. The order is
|
||||
configured in your environment, at least in
|
||||
[GNU systems](https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021).
|
||||
Other systems may ignore the locales configured in the environment and
|
||||
always produce ASCIIbetical order.
|
||||
|
||||
Results are always printed in ASCIIbetical order, regardless of what
|
||||
order the tests ran in.
|
||||
|
||||
Below you can see how the locale can affect the order.
|
||||
|
||||
$ printf '!c\n@a\n~b\n' | LC_COLLATE=C sort
|
||||
!c
|
||||
@a
|
||||
~b
|
||||
$ printf '!c\n@a\n~b\n' | LC_COLLATE=en_US.UTF-8 sort
|
||||
@a
|
||||
~b
|
||||
!c
|
||||
$ printf '!c\n@a\n~b\n' | sort -d
|
||||
@a
|
||||
~b
|
||||
!c
|
||||
|
||||
### Writing cross-shell compatibility tests for testing shell code
|
||||
|
||||
While you could write your test scripts to explicitly invoke the functionality
|
||||
@ -193,9 +179,14 @@ shell.)
|
||||
|
||||
Note that only test scripts that either have no shebang line at all or
|
||||
have shebang line `#!/bin/sh` are invoked with the specified shell.
|
||||
This allows non-shell test scripts or test scripts for other languages
|
||||
or for specific shells to coexist with those whose invocation should be
|
||||
controlled by `-s`.
|
||||
This allows non-shell test scripts or test scripts for specific
|
||||
shells to coexist with those whose invocation should be controlled by `-s`.
|
||||
|
||||
To test with multiple shells in sequence, use something like:
|
||||
|
||||
for shell in sh bash ksh zsh; do
|
||||
urchin -s $shell ./tests
|
||||
done
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@ -1 +0,0 @@
|
||||
series
|
||||
@ -0,0 +1,6 @@
|
||||
#!/bin/sh
|
||||
|
||||
# This script should run with /bin/sh
|
||||
# regardless of whether -s or -n is passed.
|
||||
|
||||
ps -o pid,comm,args | grep $$ | grep /bin/sh
|
||||
@ -1,6 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
# Assuming that urchin was invoked with `-s bash`,
|
||||
# this script should be being run with bash.
|
||||
|
||||
ps -o pid,comm,args | grep $$ | grep .special-shell
|
||||
@ -1 +1 @@
|
||||
! $TEST_SHELL ../../urchin ../Flags/Urchin\ format|grep -- --pretty
|
||||
! $TEST_SHELL ../../urchin ../Flags/Urchin\ format|grep -- --color
|
||||
|
||||
@ -1 +1,3 @@
|
||||
set -e
|
||||
$TEST_SHELL ../../urchin --shell sh .slow-tests
|
||||
! $TEST_SHELL ../../urchin --shell sh --timeout 0.3 .slow-tests
|
||||
@ -1,2 +0,0 @@
|
||||
../../urchin -T aoeu .testsuite 2>&1 | grep Bad
|
||||
../../urchin -T .testsuite 2>&1 | grep Bad
|
||||
@ -1,4 +0,0 @@
|
||||
$TEST_SHELL ../../urchin --shell sh --timeout 0.3 .slow-tests 2>&1 |
|
||||
grep -v -- --timeout |
|
||||
grep timeout
|
||||
test $? = 1
|
||||
@ -1 +0,0 @@
|
||||
$TEST_SHELL ../../urchin --shell sh .slow-tests --timeout 1000
|
||||
@ -1,7 +1,7 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
! $TEST_SHELL ../../urchin -vv --run-in-series --exit-on-fail \
|
||||
! $TEST_SHELL ../../urchin --run-in-series --exit-on-fail \
|
||||
./.test_-e,--exit-on-fail > $tmp
|
||||
|
||||
grep '1 should run.' $tmp
|
||||
|
||||
@ -1,11 +1,11 @@
|
||||
|
||||
./
|
||||
> a
|
||||
F sh (1 second)
|
||||
| This is stdout from a.
|
||||
✗ sh (1 second)
|
||||
# This is stdout from a.
|
||||
./
|
||||
> b
|
||||
. sh (1 second)
|
||||
✓ sh (1 second)
|
||||
./
|
||||
> c
|
||||
(File is not executable.)
|
||||
|
||||
@ -2,7 +2,7 @@
|
||||
./
|
||||
> a
|
||||
[31m✗ [0msh (1 second)
|
||||
| This is stdout from a.
|
||||
# This is stdout from a.
|
||||
./
|
||||
> b
|
||||
[32m✓ [0msh (1 second)
|
||||
|
||||
@ -1,3 +1,3 @@
|
||||
$TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
|
||||
sed -e 1,2\ d -e /second/d > $tmp
|
||||
$TEST_SHELL ../../urchin -s sh -t .testsuite/ |
|
||||
sed -e 1d -e /second/d > $tmp
|
||||
diff $tmp .tap-output-expectation
|
||||
|
||||
@ -1,5 +0,0 @@
|
||||
lines=$(
|
||||
$TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
|
||||
tee $tmp | grep -v '^#' | wc -l)
|
||||
cat $tmp
|
||||
test $lines -eq 4
|
||||
@ -1,3 +1,3 @@
|
||||
$TEST_SHELL ../../urchin -vv -s sh .testsuite/ |
|
||||
sed -e 1,2\ d -e 's/. seconds\?/1 second/' > $tmp
|
||||
sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
|
||||
diff $tmp .urchin-output-expectation
|
||||
|
||||
@ -1,3 +1,3 @@
|
||||
$TEST_SHELL ../../urchin --pretty -vv --shell sh .testsuite/ |
|
||||
sed -e 1,2\ d -e 's/. seconds\?/1 second/' > $tmp
|
||||
$TEST_SHELL ../../urchin --color -vv --shell sh .testsuite/ |
|
||||
sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
|
||||
diff $tmp .urchin-output-expectation-color
|
||||
|
||||
@ -1,3 +0,0 @@
|
||||
echo "$1" > $tmp
|
||||
NO_MAIN= . ../../../urchin
|
||||
has_shebang_line $tmp
|
||||
@ -1 +0,0 @@
|
||||
./.run '#!/bin/bash'
|
||||
@ -1 +0,0 @@
|
||||
! ./.run ''
|
||||
@ -1,5 +0,0 @@
|
||||
! ./.run '
|
||||
|
||||
|
||||
|
||||
'
|
||||
@ -1 +0,0 @@
|
||||
./.run '#!/usr/bin/env true'
|
||||
@ -1 +0,0 @@
|
||||
export tmp=$(mktemp)
|
||||
@ -1 +0,0 @@
|
||||
! ./.run '#!/bin/sh'
|
||||
@ -1 +0,0 @@
|
||||
rm -R "$tmp"
|
||||
4
tests/Internals/sort_python
Executable file
4
tests/Internals/sort_python
Executable file
@ -0,0 +1,4 @@
|
||||
set -e
|
||||
TESTING_URCHIN_INTERNALS=true . ../../urchin
|
||||
unsorted='@ b\n- d\n? a\n~ c\n! e\n'
|
||||
test $(printf "${unsorted}" | sort_python | cut -d\ -f2|tr -d '\n') = edabc
|
||||
@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
[ $(grep -c 'setup has run' $log) -eq '1' ]
|
||||
@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
[ $(grep -c 'setup has run' $log) -eq '2' ]
|
||||
3
tests/Setup and Teardown/.test/setup has run twice a
Executable file
3
tests/Setup and Teardown/.test/setup has run twice a
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
[ $(grep -c 'setup has run' $log) -gt '2' ]
|
||||
3
tests/Setup and Teardown/.test/setup has run twice b
Executable file
3
tests/Setup and Teardown/.test/setup has run twice b
Executable file
@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
[ $(grep -c 'setup has run' $log) -gt '2' ]
|
||||
BIN
urchin_test_on_msys64_1.png
Normal file
BIN
urchin_test_on_msys64_1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
BIN
urchin_test_on_msys64_2.png
Normal file
BIN
urchin_test_on_msys64_2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 115 KiB |
Loading…
x
Reference in New Issue
Block a user