Compare commits
1 Commits
remote-tes
...
max-forks
| Author | SHA1 | Date | |
|---|---|---|---|
|
9d1b16ed57 |
31
HISTORY
31
HISTORY
@@ -1,18 +1,10 @@
|
||||
HISTORY
|
||||
=======
|
||||
|
||||
Version 0.1.0
|
||||
Version 0.1.0-rc1
|
||||
---------------------
|
||||
This release includes breaking changes.
|
||||
|
||||
### Cross-OS testing
|
||||
I have started testing Urchin across multiple operating systems.
|
||||
This gives access to more shells, as some shels are easier to install on
|
||||
certain operating systems.
|
||||
|
||||
With this cross-OS test suite, I have extended support to more shells.
|
||||
A later version of Urchin could include a remote testing feature.
|
||||
|
||||
### Test root directory
|
||||
We introduce a concept of the root directory of a test suite.
|
||||
Such a concept is important in case you want to run subsets of your
|
||||
@@ -20,18 +12,18 @@ test suite, as we need to know how far up to apply the setup
|
||||
and teardown files.
|
||||
|
||||
The Urchin root directory is determined by moving higher in the directory
|
||||
tree in search of a file named `.urchin_root`.
|
||||
tree in search of a file named `.urchin`.
|
||||
The closest directory that contains such a file is considered the root.
|
||||
In the following filesystem, for example, `/a/b/c` would be the root.
|
||||
|
||||
mkdir -p /a/b/c/d
|
||||
touch /a/b/c/d/e
|
||||
chmod +x /a/b/c/d/e
|
||||
touch /a/b/c/.urchin_root
|
||||
touch /a/b/c/.urchin
|
||||
urchin /a/b/c/d
|
||||
|
||||
There are two situations in which we would stop looking without having
|
||||
found a `.urchin_root` file.
|
||||
found a `.urchin` file.
|
||||
|
||||
1. The system root, `/`, because we can't go any higher
|
||||
2. A directory that starts with a dot, because an urchin call on a higher
|
||||
@@ -80,7 +72,7 @@ might look for the dependency and then skip if it does not see the dependency.
|
||||
It might look like this.
|
||||
|
||||
#!/bin/sh
|
||||
if ! which inkscape; then
|
||||
if which inkscape; then
|
||||
exit 3 # status code 3 for skip
|
||||
fi
|
||||
inkscape blah blah ...
|
||||
@@ -105,13 +97,6 @@ turned on.
|
||||
|
||||
Parallel processing and shell cycling are both enabled by default.
|
||||
|
||||
You may want make only some directories run in series, you can create
|
||||
".urchin_dir" files in those directories.
|
||||
If .urchin_dir contains the phrase "series", run that directory in series
|
||||
rather than in parallel.
|
||||
This is helpful when directories actually need to run in series
|
||||
and also when running all your tests in parallel crashes your computer.
|
||||
|
||||
### Options
|
||||
Long options are now available for all command line flags.
|
||||
For example, the `-s` flag is now available as `--shell` as well.
|
||||
@@ -179,6 +164,10 @@ set as an environment variable, and the latter was set with the -s flag..
|
||||
Urchin now uses the -s flag for both of these settings, and it mostly ignores
|
||||
the exported TEST_SHELL variable.
|
||||
|
||||
Urchin also inspects the shebang line differently. Previously, Urchin would
|
||||
vary the shells with which a test is run if the shebang line either was absent
|
||||
or was #!/bin/sh. Now it varies the shell only if the shebang line is absent.
|
||||
|
||||
If you pass -n/--disable-cycling, Urchin will invoke tests ordinarily and will
|
||||
only set the TEST_SHELL variable if it does not exist. If the TEST_SHELL
|
||||
variable is absent, it will be set to /bin/sh.
|
||||
@@ -240,7 +229,7 @@ other Urchin call), and the test suite is recursively descended. Setup and
|
||||
teardown files are sourced, and everything but the specified test file is
|
||||
otherwise ignored.
|
||||
|
||||
If you don't explicitly specify the Urchin root with a .urchin_root file, we
|
||||
If you don't explicitly specify the Urchin root with a .urchin file, we
|
||||
consider the test suite root directory to be the parent of the file that
|
||||
you ran Urchin on.
|
||||
|
||||
|
||||
8
Makefile
8
Makefile
@@ -1,8 +0,0 @@
|
||||
.PHONY: test install
|
||||
|
||||
test:
|
||||
./urchin tests
|
||||
./urchin -s sh -v ./cross-os-tests
|
||||
|
||||
install:
|
||||
cp ./urchin /usr/bin
|
||||
82
SORTING
82
SORTING
@@ -1,67 +1,5 @@
|
||||
On the criteria for ordering
|
||||
==============================
|
||||
|
||||
I was confused by the documentation for sort's "-d" flag. This confusion
|
||||
relates to GNU coreutil's locale-specific sort. [^]
|
||||
|
||||
Below I discuss sort order differences between different implementations
|
||||
of sort and of sh "*" for my particular environments.
|
||||
|
||||
Sorting with sort
|
||||
------------
|
||||
Consider the following two sort commands.
|
||||
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort -d
|
||||
|
||||
With BusyBox v1.23.2 on NixOS 15.09, the first of these commands returns
|
||||
ASCIIbetical order,
|
||||
|
||||
! e
|
||||
- d
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
|
||||
and the second returns dictionary order.
|
||||
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
- d
|
||||
! e
|
||||
|
||||
With GNU coreutils version 8.24 on NixOS, both commands return
|
||||
dictionary order. The same is true for GNU coreutils version 8.23 on
|
||||
Debian Wheezy.
|
||||
|
||||
? a
|
||||
@ b
|
||||
~ c
|
||||
- d
|
||||
! e
|
||||
|
||||
IEEE Std 1003.1, 2013 Edition [^^] specifies that the "-d" flag should
|
||||
enable dictionary order. All of these versions of sort have clear
|
||||
documentation about the order that should be returned when the "-d" flag
|
||||
is set, (See --help, man, or info.) and the implementations match the
|
||||
documentation as far as I can tell.
|
||||
|
||||
I have found no explicit documentation from any relevant source as to
|
||||
what the default sort order should be. On the other hand, they all
|
||||
suggest that "-d" produces an order different from the default order.
|
||||
|
||||
In GNU coreutils 8.24, for example, "-d" is a direction to "consider
|
||||
only blanks and alphanumeric characters". It lacks any mention that the
|
||||
"-d" flag has no effect or that it is the default. Furthermore, on my
|
||||
first reading, I took it to mean that the default is to consider all
|
||||
characters and that "-d" limits the considered characters to blanks and
|
||||
alphanumeric characters.
|
||||
|
||||
|
||||
Sorting in *
|
||||
-------------
|
||||
I think this is related to the order returned by "*" in sh.
|
||||
The following sh code creates several files in a directory and then
|
||||
calls "*", listing them in order.
|
||||
|
||||
@@ -70,8 +8,7 @@ calls "*", listing them in order.
|
||||
done
|
||||
for file in *; do echo "$file"; done
|
||||
|
||||
On one computer, running FreeBSD, the order is apparently
|
||||
ASCIIbetical.
|
||||
On one computer, running FreeBSD, the order is apparently ASCIIbetical.
|
||||
|
||||
! e
|
||||
- d
|
||||
@@ -79,7 +16,7 @@ ASCIIbetical.
|
||||
@ b
|
||||
~ c
|
||||
|
||||
On two GNU systems, running NixOS and Debian, respectively, output is
|
||||
On another computer, running NixOS, the following commands print results
|
||||
in dictionary order. I'm not exactly sure what dictionary order is, but
|
||||
it is something like sorting on the alphabetical characters before
|
||||
sorting on the rest of the line.
|
||||
@@ -90,9 +27,16 @@ sorting on the rest of the line.
|
||||
- d
|
||||
! e
|
||||
|
||||
(I don't really know what dictionary order is, I was able to determine
|
||||
While I don't really know what dictionary order is, I was able to determine
|
||||
that the above results are in dictionary order because of my investigation of
|
||||
incompatible implementations of sort.)
|
||||
incompatible implementations of sort. Consider the following two sort
|
||||
commands.
|
||||
|
||||
[^] https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021
|
||||
[^^] http://pubs.opengroup.org/onlinepubs/9699919799/
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort
|
||||
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort -d
|
||||
|
||||
With BSD sort, the first of these commands print ASCIIbetical order and
|
||||
the second prints dictionary order. With GNU sort, both print dictionary
|
||||
order.
|
||||
|
||||
How annoying.
|
||||
|
||||
238
TODO
238
TODO
@@ -26,14 +26,11 @@ Windows
|
||||
Try running Urchin in Windows somehow. Interpreters include
|
||||
|
||||
* CygWin (https://www.cygwin.com/)
|
||||
* https://cygwin.com/setup-x86.exe
|
||||
* MSYS (http://mingw.org/wiki/msys)
|
||||
* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
|
||||
* Git for Windows (https://git-scm.com/download/win)
|
||||
* https://github.com/git-for-windows/git/releases/download/v2.7.2.windows.1/Git-2.7.2-32-bit.exe
|
||||
* win-bash (http://win-bash.sourceforge.net/)
|
||||
|
||||
|
||||
shall
|
||||
----------
|
||||
Add shall to my NYC*BUG talk.
|
||||
@@ -49,232 +46,15 @@ List some shell linters somewhere.
|
||||
* checkbashisms
|
||||
|
||||
|
||||
Set parallel and series
|
||||
----------
|
||||
I want to be able to make only some directories run in series.
|
||||
|
||||
* Rename .urchin to .urchin_root.
|
||||
* Look for a .urchin_dir file.
|
||||
* If .urchin_dir contains "series", run that directory in series rather
|
||||
than in parallel.
|
||||
|
||||
Rename to something other than "test"?
|
||||
----------
|
||||
Maybe wait until I have a use for this.
|
||||
|
||||
More sort alternatives
|
||||
-----------
|
||||
awk
|
||||
https://stackoverflow.com/questions/20250937/sorting-lines-in-a-file-alphabetically-using-awk-and-or-sed
|
||||
bash
|
||||
https://stackoverflow.com/questions/7442417/how-to-sort-an-array-in-bash
|
||||
|
||||
Alternatives
|
||||
--------------
|
||||
JSON.sh test suite
|
||||
|
||||
Running in multiple environments
|
||||
-----------------------------------
|
||||
Setup for other environments includes the following.
|
||||
|
||||
* Installing packages
|
||||
* `touch .zshrc`
|
||||
* Copy urchin and tests
|
||||
|
||||
|
||||
Fixtures
|
||||
------------
|
||||
I want to change the way that fixtures are done.
|
||||
|
||||
Instead of using setup, teardown, &c., use ordinary programs from within
|
||||
your tests. For example.
|
||||
|
||||
# tests/.fixtures/tmp-dir
|
||||
tmp=$(mktemp -d)
|
||||
cd $tmp
|
||||
@$
|
||||
code=$?
|
||||
cd /
|
||||
rm -Rf $tmp
|
||||
exit $code
|
||||
|
||||
# tests/blah
|
||||
../.fixtures/tmp-dir 'blah blah blah'
|
||||
|
||||
It's best if I can wrap a bunch of commands in braces or paratheses
|
||||
rather than just one command. Is there a nice way to do that?
|
||||
|
||||
Once I have this new way, I guess I might as well keep the old way.
|
||||
I think the setup, teardown thing can be easier if you only have simple
|
||||
fixtures. And since I'm going to keep it, I'm going to add another one.
|
||||
|
||||
* setup_dir runs once for the present directory.
|
||||
* setup_children runs once for each child.
|
||||
* setup_file runs once for each file descendent.
|
||||
|
||||
The present `setup` is renamed to `setup_children`, and the new
|
||||
`setup_file` runs on each file (not directory) that is a child,
|
||||
grandchild, great-grandchild, and so on.
|
||||
|
||||
Dependency checking
|
||||
----------------------
|
||||
You might want to skip tests based on dependencies. Currently you can
|
||||
conditionally skip tests one at a time by exiting with code 3. I want to
|
||||
be able to skip an entire directory.
|
||||
|
||||
So we add a new magic file called `dep`. If it exists, it is run before
|
||||
everything else in the directory.
|
||||
|
||||
* If it exits with code 0, tests continue as if dep did not exist.
|
||||
* If it exits with code 3, all tests in the directory are marked as
|
||||
skipped.
|
||||
* If it exits with code 1, all tests in the directory are marked as
|
||||
failed. To make the implementation easier, I'll probably treat the
|
||||
directory as a single test in this case.
|
||||
|
||||
A note on magic files
|
||||
-------------------------
|
||||
It is nice to have access to things like setup and dep (magic files)
|
||||
once in a while, but you need to be doing rather substantial testing
|
||||
before they make your test suite simpler; the documentation should
|
||||
strongly recommend writing your tests without magic files and then
|
||||
refactoring and only then considering moving things to magic files.
|
||||
|
||||
Remote testing
|
||||
----------------
|
||||
In order to test Urchin across multiple operating systems, I have
|
||||
already added tests in Urchin's test suite that run Urchin tests in
|
||||
remote servers. I would like to move this to Urchin itself so that
|
||||
Urchin can test other things on remote servers.
|
||||
|
||||
Urchin's output presently looks like this.
|
||||
|
||||
Cycling with the following shells: sh bash dash mksh zsh
|
||||
Running tests at 2016-04-07T12:33:49
|
||||
|
||||
Flags/
|
||||
> --timeout output
|
||||
. bash (0 seconds)
|
||||
. dash (0 seconds)
|
||||
. mksh (0 seconds)
|
||||
. sh (0 seconds)
|
||||
. zsh (0 seconds)
|
||||
|
||||
Done, took 1 second.
|
||||
5 tests passed.
|
||||
0 tests skipped.
|
||||
0 tests failed.
|
||||
|
||||
After the change, the output should look like this.
|
||||
|
||||
Cycling with the following shells: sh dash mksh
|
||||
Running tests at 2016-04-07T12:33:49
|
||||
|
||||
Flags/
|
||||
> --timeout output
|
||||
. dash on localhost (0 seconds)
|
||||
. dash on localhost:8080 (0 seconds)
|
||||
. dash on tlevine@hpux.polarhome.com (0 seconds)
|
||||
. mksh on localhost (0 seconds)
|
||||
. mksh on tlevine@hpux.polarhome.com (0 seconds)
|
||||
. sh on localhost (0 seconds)
|
||||
. sh on localhost:8080 (0 seconds)
|
||||
. sh on tlevine@hpux.polarhome.com (0 seconds)
|
||||
|
||||
Done, took 1 second.
|
||||
8 tests passed.
|
||||
0 tests skipped.
|
||||
0 tests failed.
|
||||
|
||||
This is just how the output should look; the tests run in whatever order
|
||||
makes sense.
|
||||
|
||||
Bugs
|
||||
-------
|
||||
|
||||
Both md5sum and md5 should be supported.
|
||||
|
||||
Trouble logging in to hpux, irix, miros, netbsd, tru64, qnx, ....
|
||||
|
||||
$ rsync -e 'ssh -p 785' urchin tlevine@hpux.polarhome.com:.blah
|
||||
HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
|
||||
Welcome to HPUX/PA... member of polarhome.com realm
|
||||
|
||||
bash: rsync: command not found
|
||||
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
|
||||
rsync error: remote command not found (code 127) at io.c(226)
|
||||
[sender=3.1.1]
|
||||
|
||||
|
||||
OpenIndiana grep does not support -q
|
||||
|
||||
I get `/urchin: syntax error at line 84: \`}' unexpected` on
|
||||
unixware and solaris.
|
||||
|
||||
mktemp
|
||||
|
||||
> tlevine@hpux.polarhome.com -p 785
|
||||
F sh (8 seconds)
|
||||
|
|
||||
| HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
|
||||
| Welcome to HPUX/PA... member of polarhome.com realm
|
||||
|
|
||||
|
|
||||
| HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
|
||||
| Welcome to HPUX/PA... member of polarhome.com realm
|
||||
|
|
||||
| mktemp: option requires an argument -- d
|
||||
| ./urchin[96]: /tmp/tlevinea21441/log: Cannot create the specified file.
|
||||
|
||||
date
|
||||
|
||||
tlevine@hpux64$ ./urchin tests/ -n -vv
|
||||
date: bad format character - s
|
||||
|
||||
So I need a portable seconds-from epoch
|
||||
|
||||
I also need to handle when no arguments are passed to urchin.
|
||||
|
||||
Exit code is wrong for which on HP-UX
|
||||
|
||||
## `$(...)`
|
||||
Solaris doesn't support `$(...)`; you need `\`...\`` instead.
|
||||
|
||||
tlevine@solaris$ ./urchin --run-in-series tests/Errors/
|
||||
./urchin: syntax error at line 84: `tmp=$' unexpected
|
||||
|
||||
I use this a lot.
|
||||
|
||||
$ grep -c '\$(' urchin
|
||||
52
|
||||
|
||||
Darn
|
||||
|
||||
|
||||
|
||||
|
||||
Update tests to support
|
||||
|
||||
* md5
|
||||
* rsync
|
||||
* mktemp
|
||||
* epoch
|
||||
* Report cycling by default
|
||||
* New format for reporting cycling
|
||||
|
||||
|
||||
|
||||
|
||||
Support systems without rsync
|
||||
|
||||
|
||||
BSD mktemp
|
||||
|
||||
| NetBSD 6.1.3
|
||||
| Welcome to NetBSD ...member of polarhome.com realm
|
||||
|
|
||||
| Usage: mktemp [-dqu] [-p <tmpdir>] {-t prefix | template ...}
|
||||
| mkdir: : No such file or directory
|
||||
| ./urchin: cannot create /log: permission denied
|
||||
|
||||
|
||||
NetBSD
|
||||
|
||||
md5: unknown option -- q
|
||||
usage: cksum [-n] [-a algorithm [-ptx] [-s string]] [-o 1|2]
|
||||
[file ... | -c [-w] [sumfile]]
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,14 +0,0 @@
|
||||
params="$(basename "${0}")"
|
||||
|
||||
hostname="$(echo "${params}" | cut -d\ -f1)"
|
||||
if echo "${params}" | grep -q \ ; then
|
||||
flags="$(echo "${params}" | cut -d\ -f2-)"
|
||||
fi
|
||||
|
||||
urchin_dir=.urchin-cross-shell-test
|
||||
|
||||
rsync --archive -e "ssh ${flags}" $RSYNC_FLAGS \
|
||||
../urchin ../tests "${hostname}":"${urchin_dir}" ||
|
||||
scp -r ${flags} ../urchin ../tests "${hostname}":"${urchin_dir}"
|
||||
ssh "${hostname}" ${flags} \
|
||||
"cd ${urchin_dir} && ./urchin --run-in-series tests"
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
# apt-get install bash dash ksh posh pdksh mksh yash zsh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
RSYNC_FLAGS='--rsync-path=/usr/local/bin/rsync'
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
# SSH public key needs to be in ~/.etc/ssh/authorized_keys
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
@@ -1,2 +0,0 @@
|
||||
#!/bin/sh
|
||||
. ./.run
|
||||
1
packages/.gitignore
vendored
1
packages/.gitignore
vendored
@@ -1 +0,0 @@
|
||||
*.tar.gz
|
||||
@@ -1,11 +0,0 @@
|
||||
#!/bin/sh
|
||||
name=urchin-$(../urchin --version)
|
||||
|
||||
tmp=$(mktemp -d)
|
||||
mkdir $tmp/$name
|
||||
cp ../urchin ../readme.md ../AUTHORS ../COPYING $tmp/$name
|
||||
cd $tmp
|
||||
tar czf $name.tar.gz $name
|
||||
cd - > /dev/null
|
||||
mv $tmp/$name.tar.gz .
|
||||
rm -R $tmp
|
||||
55
readme.md
55
readme.md
@@ -43,15 +43,25 @@ Urchin depends on the following programs.
|
||||
* timeout
|
||||
* sort
|
||||
|
||||
Vanilla installations of modern BSD and GNU systems usually include all
|
||||
of these programs.
|
||||
All of the above programs are usually included on base BSD installations.
|
||||
On GNU systems it should be sufficient to install the busybox package.
|
||||
|
||||
Urchin uses sort to format its output. GNU sort (as of GNU coreutils version
|
||||
8.24) lacks the ability to sort in lexicographic order, and this feature is
|
||||
necessary for the output to look right. If your version of sort lacks this
|
||||
feature, Urchin will try to use one of the following tools for sorting.
|
||||
|
||||
If no acceptable sorting program is available, Urchin will print a warning
|
||||
and use the incomplete sort that is installed on your system. This is not a
|
||||
big deal; if your test files all start with alphanumeric letters, the output
|
||||
should look fine.
|
||||
|
||||
## Install
|
||||
Urchin is contained in a single file, so you can install it by copying it to a
|
||||
directory in your `PATH`. For example, you can run the following as root.
|
||||
|
||||
cd /usr/local/bin
|
||||
wget https://raw.githubusercontent.com/tlevine/urchin/v0.1.0-rc3/urchin
|
||||
wget https://raw.githubusercontent.com/tlevine/urchin/v0.0.6/urchin
|
||||
chmod +x urchin
|
||||
|
||||
Urchin can be installed with npm too.
|
||||
@@ -108,36 +118,14 @@ Files are only run if they are executable, and files beginning with `.` are
|
||||
ignored. Thus, fixtures and libraries can be included sloppily within the test
|
||||
directory tree. The test passes if the file exits 0; otherwise, it fails.
|
||||
|
||||
urchin looks for files within a directory in the following manner,
|
||||
Tests files and subdirectories are run in ASCIIbetical order within each
|
||||
directory; that is,
|
||||
urchin looks for files within a directory in the following manner.
|
||||
|
||||
for file in *; do
|
||||
do_something_with_test_file $file
|
||||
done
|
||||
|
||||
so files are run in whatever order `*` produces. The order is
|
||||
configured in your environment, at least in
|
||||
[GNU systems](https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021).
|
||||
Other systems may ignore the locales configured in the environment and
|
||||
always produce ASCIIbetical order.
|
||||
|
||||
Results are always printed in ASCIIbetical order, regardless of what
|
||||
order the tests ran in.
|
||||
|
||||
Below you can see how the locale can affect the order.
|
||||
|
||||
$ printf '!c\n@a\n~b\n' | LC_COLLATE=C sort
|
||||
!c
|
||||
@a
|
||||
~b
|
||||
$ printf '!c\n@a\n~b\n' | LC_COLLATE=en_US.UTF-8 sort
|
||||
@a
|
||||
~b
|
||||
!c
|
||||
$ printf '!c\n@a\n~b\n' | sort -d
|
||||
@a
|
||||
~b
|
||||
!c
|
||||
|
||||
### Writing cross-shell compatibility tests for testing shell code
|
||||
|
||||
While you could write your test scripts to explicitly invoke the functionality
|
||||
@@ -191,9 +179,14 @@ shell.)
|
||||
|
||||
Note that only test scripts that either have no shebang line at all or
|
||||
have shebang line `#!/bin/sh` are invoked with the specified shell.
|
||||
This allows non-shell test scripts or test scripts for other languages
|
||||
or for specific shells to coexist with those whose invocation should be
|
||||
controlled by `-s`.
|
||||
This allows non-shell test scripts or test scripts for specific
|
||||
shells to coexist with those whose invocation should be controlled by `-s`.
|
||||
|
||||
To test with multiple shells in sequence, use something like:
|
||||
|
||||
for shell in sh bash ksh zsh; do
|
||||
urchin -s $shell ./tests
|
||||
done
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
series
|
||||
@@ -0,0 +1,6 @@
|
||||
#!/bin/sh
|
||||
|
||||
# This script should run with /bin/sh
|
||||
# regardless of whether -s or -n is passed.
|
||||
|
||||
ps -o pid,comm,args | grep $$ | grep /bin/sh
|
||||
@@ -1,6 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
# Assuming that urchin was invoked with `-s bash`,
|
||||
# this script should be being run with bash.
|
||||
|
||||
ps -o pid,comm,args | grep $$ | grep .special-shell
|
||||
@@ -1 +1 @@
|
||||
! $TEST_SHELL ../../urchin ../Flags/Urchin\ format|grep -- --pretty
|
||||
! $TEST_SHELL ../../urchin ../Flags/Urchin\ format|grep -- --color
|
||||
|
||||
@@ -1 +1,3 @@
|
||||
set -e
|
||||
$TEST_SHELL ../../urchin --shell sh .slow-tests
|
||||
! $TEST_SHELL ../../urchin --shell sh --timeout 0.3 .slow-tests
|
||||
@@ -1,2 +0,0 @@
|
||||
../../urchin -T aoeu .testsuite 2>&1 | grep Bad
|
||||
../../urchin -T .testsuite 2>&1 | grep Bad
|
||||
@@ -1,4 +0,0 @@
|
||||
$TEST_SHELL ../../urchin --shell sh --timeout 0.3 .slow-tests 2>&1 |
|
||||
grep -v -- --timeout |
|
||||
grep timeout
|
||||
test $? = 1
|
||||
@@ -1 +0,0 @@
|
||||
$TEST_SHELL ../../urchin --shell sh .slow-tests --timeout 1000
|
||||
@@ -1,7 +1,7 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
|
||||
! $TEST_SHELL ../../urchin -vv --run-in-series --exit-on-fail \
|
||||
! $TEST_SHELL ../../urchin --run-in-series --exit-on-fail \
|
||||
./.test_-e,--exit-on-fail > $tmp
|
||||
|
||||
grep '1 should run.' $tmp
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
|
||||
./
|
||||
> a
|
||||
F sh (1 second)
|
||||
| This is stdout from a.
|
||||
✗ sh (1 second)
|
||||
# This is stdout from a.
|
||||
./
|
||||
> b
|
||||
. sh (1 second)
|
||||
✓ sh (1 second)
|
||||
./
|
||||
> c
|
||||
(File is not executable.)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
./
|
||||
> a
|
||||
[31m✗ [0msh (1 second)
|
||||
| This is stdout from a.
|
||||
# This is stdout from a.
|
||||
./
|
||||
> b
|
||||
[32m✓ [0msh (1 second)
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
$TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
|
||||
sed -e 1,2\ d -e /second/d > $tmp
|
||||
$TEST_SHELL ../../urchin -s sh -t .testsuite/ |
|
||||
sed -e 1d -e /second/d > $tmp
|
||||
diff $tmp .tap-output-expectation
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
lines=$(
|
||||
$TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
|
||||
tee $tmp | grep -v '^#' | wc -l)
|
||||
cat $tmp
|
||||
test $lines -eq 4
|
||||
@@ -1,3 +1,3 @@
|
||||
$TEST_SHELL ../../urchin -vv -s sh .testsuite/ |
|
||||
sed -e 1,2\ d -e 's/. seconds\?/1 second/' > $tmp
|
||||
sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
|
||||
diff $tmp .urchin-output-expectation
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
$TEST_SHELL ../../urchin --pretty -vv --shell sh .testsuite/ |
|
||||
sed -e 1,2\ d -e 's/. seconds\?/1 second/' > $tmp
|
||||
$TEST_SHELL ../../urchin --color -vv --shell sh .testsuite/ |
|
||||
sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
|
||||
diff $tmp .urchin-output-expectation-color
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
echo "$1" > $tmp
|
||||
TESTING_URCHIN_INTERNALS=true . ../../../urchin
|
||||
has_shebang_line $tmp
|
||||
@@ -1 +0,0 @@
|
||||
./.run '#!/bin/bash'
|
||||
@@ -1 +0,0 @@
|
||||
! ./.run ''
|
||||
@@ -1,5 +0,0 @@
|
||||
! ./.run '
|
||||
|
||||
|
||||
|
||||
'
|
||||
@@ -1 +0,0 @@
|
||||
./.run '#!/usr/bin/env true'
|
||||
@@ -1 +0,0 @@
|
||||
export tmp=$(mktemp)
|
||||
@@ -1 +0,0 @@
|
||||
! ./.run '#!/bin/sh'
|
||||
@@ -1 +0,0 @@
|
||||
rm -R "$tmp"
|
||||
4
tests/Internals/sort_python
Executable file
4
tests/Internals/sort_python
Executable file
@@ -0,0 +1,4 @@
|
||||
set -e
|
||||
TESTING_URCHIN_INTERNALS=true . ../../urchin
|
||||
unsorted='@ b\n- d\n? a\n~ c\n! e\n'
|
||||
test $(printf "${unsorted}" | sort_python | cut -d\ -f2|tr -d '\n') = edabc
|
||||
@@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
[ $(grep -c 'setup has run' $log) -eq '1' ]
|
||||
@@ -1,3 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
[ $(grep -c 'setup has run' $log) -eq '2' ]
|
||||
3
tests/Setup and Teardown/.test/setup has run twice a
Executable file
3
tests/Setup and Teardown/.test/setup has run twice a
Executable file
@@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
[ $(grep -c 'setup has run' $log) -gt '2' ]
|
||||
3
tests/Setup and Teardown/.test/setup has run twice b
Executable file
3
tests/Setup and Teardown/.test/setup has run twice b
Executable file
@@ -0,0 +1,3 @@
|
||||
#!/bin/sh
|
||||
|
||||
[ $(grep -c 'setup has run' $log) -gt '2' ]
|
||||
343
urchin
343
urchin
@@ -51,52 +51,13 @@ set -e
|
||||
# Kill subprocesses on interrupt.
|
||||
trap "kill -$$; exit" HUP INT TERM
|
||||
|
||||
DEFAULT_SHELLS='
|
||||
sh
|
||||
bash
|
||||
dash
|
||||
ksh
|
||||
posh
|
||||
pdksh
|
||||
mksh
|
||||
yash
|
||||
zsh
|
||||
'
|
||||
DEFAULT_SHELLS='sh bash dash mksh zsh'
|
||||
if [ -n "${ZSH_VERSION}" ]; then
|
||||
# avoid "no matches found: *" error when directories are empty
|
||||
setopt NULL_GLOB
|
||||
emulate sh
|
||||
fi
|
||||
|
||||
epoch_date() {
|
||||
date +%s
|
||||
}
|
||||
epoch_pax() {
|
||||
# Based on http://stackoverflow.com/a/7262588/407226
|
||||
tmp="$(mktemp_file)"
|
||||
echo "ibase=8;$({ pax -wx cpio "${tmp}"; echo; } | cut -c 48-59)" | bc
|
||||
rm "${tmp}"
|
||||
}
|
||||
|
||||
mktemp_dir() {
|
||||
# Support HP-UX mktemp that has wrong exit codes and
|
||||
# can't make directories.
|
||||
tmp=$(mktemp /tmp/urchin.XXXXXXXX)
|
||||
if test -f "${tmp}"; then
|
||||
rm "${tmp}"
|
||||
fi
|
||||
mkdir "${tmp}"
|
||||
echo "${tmp}"
|
||||
}
|
||||
mktemp_file() {
|
||||
tmp=$(mktemp /tmp/urchin.XXXXXXXX)
|
||||
if ! test -f "${tmp}"; then
|
||||
> "${tmp}"
|
||||
fi
|
||||
echo "${tmp}"
|
||||
}
|
||||
|
||||
|
||||
validate_test_arg() {
|
||||
# Must be a file or directory
|
||||
if [ ! -e "${1}" ]; then
|
||||
@@ -109,7 +70,7 @@ validate_test_arg() {
|
||||
root="$(urchin_root "${1}")"
|
||||
if ! {
|
||||
basename "$(fullpath "${root}")" |
|
||||
grep -qi 'test' || "${force}"
|
||||
grep -i 'test' > /dev/null || "${force}"
|
||||
}; then
|
||||
echo 'The root directory of the tests that you are running urchin on
|
||||
does not contain the word "test", so I am not running,
|
||||
@@ -119,54 +80,28 @@ validate_test_arg() {
|
||||
fi
|
||||
}
|
||||
|
||||
# All temporary files go here
|
||||
urchin_tmp=$(mktemp_dir)
|
||||
> "${urchin_tmp}/log"
|
||||
sort_python() {
|
||||
python -c 'import sys
|
||||
for line in sorted(sys.stdin.readlines()):
|
||||
sys.stdout.write(line)
|
||||
'
|
||||
}
|
||||
|
||||
# All temporary files go here
|
||||
urchin_tmp=$(mktemp -d)
|
||||
> "${urchin_tmp}/log"
|
||||
urchin_exit() {
|
||||
rm -Rf "${urchin_tmp}"
|
||||
exit "$@"
|
||||
}
|
||||
|
||||
if which md5 1> /dev/null 2> /dev/null; then
|
||||
urchin_md5=md5
|
||||
elif which md5sum 1> /dev/null 2> /dev/null; then
|
||||
urchin_md5=md5sum
|
||||
else
|
||||
echo Could not find MD5 hash command >&2
|
||||
urchin_exit 1
|
||||
fi
|
||||
|
||||
if epoch_date 2>&1 > /dev/null; then
|
||||
epoch=epoch_date
|
||||
elif epoch_pax 2>&1 > /dev/null; then
|
||||
epoch=epoch_pax
|
||||
else
|
||||
echo I could not find a seconds counter. >&2
|
||||
urchin_exit 1
|
||||
fi
|
||||
|
||||
stdout_file() {
|
||||
the_test="${1}"
|
||||
the_shell="${2}"
|
||||
host="${3}"
|
||||
|
||||
if test -n "${host}"; then
|
||||
# This assumes the tests ran on a remote and have been copied.
|
||||
x="${urchin_tmp}/remote/${host}/stdout${the_test}"
|
||||
else
|
||||
# This can be run during the tests.
|
||||
x="${urchin_tmp}/stdout$(fullpath "$the_test")"
|
||||
mkdir -p "${x}"
|
||||
fi
|
||||
|
||||
case "${urchin_md5}" in
|
||||
md5sum) y=$(echo "${the_shell}" | md5sum | cut -d\ -f1) ;;
|
||||
md5) y=$(echo "${the_shell}" | md5 | sed 's/.* //') ;;
|
||||
*) echo md5 command is not configured >&2; urchin_exit 1;;
|
||||
esac
|
||||
|
||||
echo "${x}/${y}"
|
||||
x="${urchin_tmp}/stdout$(fullpath "$the_test")"
|
||||
mkdir -p "${x}"
|
||||
echo "${x}/$(echo "${the_shell}" | md5sum | cut -d\ -f1)"
|
||||
}
|
||||
|
||||
# Expand relative paths
|
||||
@@ -189,7 +124,7 @@ urchin_root() {
|
||||
|
||||
abscurrent="$(fullpath "${1}")"
|
||||
if test "${abscurrent}" = / ||
|
||||
basename "${abscurrent}" | grep -q '^\.' ; then
|
||||
basename "${abscurrent}" | grep '^\.' > /dev/null; then
|
||||
# Stop traversing upwards at / and at hidden directories.
|
||||
if test -d "${orig}"; then
|
||||
echo "${orig}"
|
||||
@@ -201,7 +136,7 @@ urchin_root() {
|
||||
return 1
|
||||
elif test -f "${current}"; then
|
||||
urchin_root "$(dirname -- "${current}")" "${orig}"
|
||||
elif test -f "${current}"/.urchin_root; then
|
||||
elif test -f "${current}"/.urchin; then
|
||||
remove_trailing_slash "${current}"
|
||||
else
|
||||
urchin_root "${current}"/.. "${orig}"
|
||||
@@ -209,7 +144,7 @@ urchin_root() {
|
||||
}
|
||||
|
||||
# Urchin version number
|
||||
VERSION=0.1.1-unstable
|
||||
VERSION=0.1.0-rc1
|
||||
|
||||
indent() {
|
||||
level="${1}"
|
||||
@@ -230,8 +165,8 @@ recurse() {
|
||||
fi
|
||||
done
|
||||
|
||||
if echo "${requested_path}" | grep -q "^${potential_test}" ||
|
||||
echo "${potential_test}" | grep -q "^${requested_path}" ; then
|
||||
if echo "${requested_path}" | grep "^${potential_test}" > /dev/null ||
|
||||
echo "${potential_test}" | grep "^${requested_path}" > /dev/null; then
|
||||
if test "$(dirname "${potential_test}")" = \
|
||||
"$(dirname "${requested_path}")" &&
|
||||
test "${potential_test}" != "${requested_path}"; then
|
||||
@@ -247,15 +182,7 @@ recurse() {
|
||||
if [ -d "${potential_test}" ]; then
|
||||
(
|
||||
cd -- "${potential_test}"
|
||||
if test -f .urchin_dir && grep -q series .urchin_dir ; then
|
||||
run_in_series_dir=true
|
||||
else
|
||||
run_in_series_dir=false
|
||||
fi
|
||||
|
||||
if test -f setup_dir; then
|
||||
. ./setup_dir
|
||||
fi
|
||||
if test -f setup_dir; then . ./setup_dir; fi
|
||||
|
||||
for test in *; do
|
||||
if test "${test}" = '*' && ! test -e "${test}"; then
|
||||
@@ -263,23 +190,28 @@ recurse() {
|
||||
break
|
||||
fi
|
||||
|
||||
recurse "${requested_path}" "${test}" "${cycle_shell}" \
|
||||
"${TEST_SHELL}" &
|
||||
(
|
||||
if test -f setup; then . ./setup; fi
|
||||
if recurse "${requested_path}" "${test}" "${cycle_shell}" \
|
||||
"${TEST_SHELL}"; then
|
||||
exit_code=0
|
||||
else
|
||||
exit_code="${?}"
|
||||
fi
|
||||
if test -f teardown; then . ./teardown; fi
|
||||
exit "${exit_code}"
|
||||
) &
|
||||
|
||||
if "${run_in_series}" || "${run_in_series_dir}"; then
|
||||
if "${run_in_series}"; then
|
||||
if wait "${!}"; then exit_code=0; else exit_code="${?}"; fi
|
||||
if "${exit_on_not_ok}" && test "${exit_code}" -ne 0; then
|
||||
if test -f teardown_dir; then
|
||||
. ./teardown_dir
|
||||
fi
|
||||
if test -f teardown_dir; then . ./teardown_dir; fi
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
wait
|
||||
if test -f teardown_dir; then
|
||||
. ./teardown_dir
|
||||
fi
|
||||
if test -f teardown_dir; then . ./teardown_dir; fi
|
||||
)
|
||||
elif [ -f "${potential_test}" ]; then
|
||||
cd -- "$(dirname -- "${potential_test}")"
|
||||
@@ -288,12 +220,10 @@ recurse() {
|
||||
# that reflects the specified or implied shell to use for shell-code tests.
|
||||
while read the_test_shell; do
|
||||
(
|
||||
if test -f setup; then
|
||||
. ./setup
|
||||
fi
|
||||
if test -f setup; then . ./setup; fi
|
||||
|
||||
# Run the test
|
||||
start=$("${epoch}")
|
||||
start=$(date +%s)
|
||||
set +e
|
||||
{
|
||||
if "${cycle_shell}"; then
|
||||
@@ -304,7 +234,7 @@ recurse() {
|
||||
"${the_test_shell}" "${potential_test}"
|
||||
fi
|
||||
else
|
||||
# Shell cycling is disabled with -n; use the present value of
|
||||
# Shell cycling is disabled with -d; use the present value of
|
||||
# TEST_SHELL or default to /bin/sh
|
||||
if [ -n "${TEST_SHELL}" ]; then
|
||||
$TIMEOUT "${potential_test}"
|
||||
@@ -315,11 +245,9 @@ recurse() {
|
||||
} > "$(stdout_file "${potential_test}" "${the_test_shell}")" 2>&1
|
||||
exit_code="${?}"
|
||||
set -e
|
||||
finish=$("${epoch}")
|
||||
finish=$(date +%s)
|
||||
|
||||
if test -f teardown; then
|
||||
. ./teardown
|
||||
fi
|
||||
if test -f teardown; then . ./setup; fi
|
||||
|
||||
if [ "${exit_code}" -eq 0 ]; then
|
||||
result=ok
|
||||
@@ -338,16 +266,14 @@ recurse() {
|
||||
if "${run_in_series}"; then
|
||||
if wait "${!}"; then exit_code=0; else exit_code="${?}"; fi
|
||||
if "${exit_on_not_ok}" && test "${exit_code}" -ne 0; then
|
||||
if test -f teardown_dir; then
|
||||
. ./teardown_dir
|
||||
fi
|
||||
if test -f teardown_dir; then . ./teardown_dir; fi
|
||||
return 1
|
||||
fi
|
||||
fi
|
||||
done < "${shell_list}"
|
||||
wait
|
||||
else
|
||||
echo "${potential_test}: Neither file nor directory!?" >&2
|
||||
echo "${potential_test}: Neither file nor directory!?" > /dev/stderr
|
||||
fi
|
||||
else
|
||||
# Shell is ''
|
||||
@@ -362,22 +288,15 @@ report_outcome() {
|
||||
start="${4}"
|
||||
finish="${5}"
|
||||
|
||||
host="${6}"
|
||||
if test -n "${host}"; then
|
||||
onhost=" on ${host}"
|
||||
fi
|
||||
|
||||
escaped_root="$(fullpath "${root}" | sed 's/\//\\\//g')"
|
||||
elapsed=$(($finish - $start))
|
||||
|
||||
if "${print_margins}" || "${tap_format}"; then
|
||||
if $tap_format; then printf \#\ ; fi
|
||||
echo Ran tests at $(date +%Y-%m-%dT%H:%M:%S) with the following shells:
|
||||
if $tap_format; then printf \#\ ; fi
|
||||
cat "${shell_list}" | tr '\n' \
|
||||
echo
|
||||
if "${tap_format}"; then
|
||||
printf \#\
|
||||
fi
|
||||
if "${print_margins}" || "${tap_format}"; then
|
||||
echo Running tests at $(date +%Y-%m-%dT%H:%M:%S)
|
||||
fi
|
||||
|
||||
|
||||
for number in n oks skips not_oks; do
|
||||
eval "${number}=0"
|
||||
@@ -385,8 +304,8 @@ report_outcome() {
|
||||
|
||||
# Use a temporary file rather than a pipe because a pipe starts a sub-shell
|
||||
# and thus makes the above variables local.
|
||||
sorted_log_file=$(mktemp_file)
|
||||
cat "${log_file}" | LC_COLLATE=C sort > "${sorted_log_file}"
|
||||
sorted_log_file=$(mktemp)
|
||||
cat "${log_file}" | "${sort}" > "${sorted_log_file}"
|
||||
|
||||
while read line; do
|
||||
abspath=$(echo "${line}" | cut -f1)
|
||||
@@ -418,13 +337,13 @@ report_outcome() {
|
||||
fi
|
||||
|
||||
if test -z "${the_shell}"; then
|
||||
the_shell='File is not executable'
|
||||
the_shell='File is not executable.'
|
||||
fi
|
||||
echo "${not}ok $n - ${path} (${the_shell}${onhost}) ${skip}"
|
||||
echo "${not}ok $n - ${path} (${the_shell}) ${skip}"
|
||||
if { test "${result}" = not_ok && "${print_not_ok_stdout}"; } ||
|
||||
{ test "${result}" = ok && "${print_ok_stdout}"; }; then
|
||||
echo '# ------------ Begin output ------------'
|
||||
sed 's/^/# /' "$(stdout_file "${abspath}" "${the_shell}" "${host}")"
|
||||
sed 's/^/# /' "$(stdout_file "${abspath}" "${the_shell}")"
|
||||
echo '# ------------ End output ------------'
|
||||
fi
|
||||
echo "# Previous test took ${file_elapsed} seconds."
|
||||
@@ -437,44 +356,45 @@ report_outcome() {
|
||||
printf "$(dirname -- "${path}")/\n> $(basename -- "${path}")\n"
|
||||
fi
|
||||
}
|
||||
|
||||
case "${result}" in
|
||||
ok)
|
||||
if "${print_ok}"; then
|
||||
header
|
||||
# On success, print a green '✓'
|
||||
if "${print_in_color}"; then
|
||||
printf "\033[32m${success_mark} \033[0m"
|
||||
printf '\033[32m✓ \033[0m'
|
||||
else
|
||||
printf "${success_mark} "
|
||||
printf '✓ '
|
||||
fi
|
||||
echo "${the_shell}${onhost} (${file_elapsed} $(plural second $file_elapsed))"
|
||||
echo "${the_shell} ("${file_elapsed}" $(plural second $file_elapsed))"
|
||||
fi
|
||||
;;
|
||||
not_ok)
|
||||
if "${print_not_ok}"; then
|
||||
header
|
||||
# On not_ok, print a red '✗'
|
||||
if "${print_in_color}"; then
|
||||
printf "\033[31m${fail_mark} \033[0m"
|
||||
printf '\033[31m✗ \033[0m'
|
||||
else
|
||||
printf "${fail_mark} "
|
||||
printf '✗ '
|
||||
fi
|
||||
echo "${the_shell}${onhost} (${file_elapsed} $(plural second $file_elapsed))"
|
||||
echo "${the_shell} ("${file_elapsed}" $(plural second $file_elapsed))"
|
||||
fi
|
||||
;;
|
||||
skip)
|
||||
if "${print_ok}"; then
|
||||
header
|
||||
if test -z "${the_shell}"; then
|
||||
echo " (File is not executable${onhost}.)"
|
||||
echo ' (File is not executable.)'
|
||||
else
|
||||
echo " ${the_shell}${onhost} (${file_elapsed} $(plural second $file_elapsed))"
|
||||
echo " ${the_shell} ("${file_elapsed}" $(plural second $file_elapsed))"
|
||||
fi
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
if { test "${result}" = not_ok && "${print_not_ok_stdout}"; } ||
|
||||
{ test "${result}" = ok && "${print_ok_stdout}"; }; then
|
||||
sed 's/^/ | /' "$(stdout_file "${abspath}" "${the_shell}" "${host}")"
|
||||
sed 's/^/ # /' "$(stdout_file "${abspath}" "${the_shell}")"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -504,10 +424,10 @@ report_outcome() {
|
||||
}
|
||||
|
||||
has_shebang_line() {
|
||||
head -n 1 "${1}" | grep -v '^#!/bin/sh$' | grep -q '^#!'
|
||||
head -n 1 "${1}" | grep -qE '^#!'
|
||||
}
|
||||
|
||||
USAGE="usage: $0 [options]... [test file or directory]..."
|
||||
USAGE="usage: $0 [<options>] <test directory>"
|
||||
|
||||
urchin_help() {
|
||||
cat <<EOF
|
||||
@@ -522,15 +442,14 @@ particular test file once per shell.
|
||||
On each run,
|
||||
|
||||
1. The TEST_SHELL environment variable is set to the particular shell.
|
||||
2. If the test file lacks a shebang line or has a shebang line of
|
||||
"#!/bin/sh", the test script is also executed in that shell.
|
||||
2. If the test file lacks a shebang line, the test script is also
|
||||
executed in that shell.
|
||||
|
||||
The following flags affect how this multiple-shell testing is handled.
|
||||
|
||||
-s, --shell <shell> Tell Urchin to use a different list of shells.
|
||||
(You can pass this flag multiple times.)
|
||||
|
||||
-n, --disable-cycling Disable the cycling of shells; Urchin will
|
||||
-d, --disable-cycling Disable the cycling of shells; Urchin will
|
||||
execute test files ordinarily, implicitly using
|
||||
sh for files that lack shebang lines. It will
|
||||
set the TEST_SHELL variable to "/bin/sh" if and
|
||||
@@ -540,9 +459,10 @@ The following flags affect how Urchin processes tests.
|
||||
|
||||
-b, --run-in-series Run tests in series. The default is to run tests
|
||||
in parallel where possible.
|
||||
-n, --max-forks Maximum number of parallel tests (Default is 50.)
|
||||
-e, --exit-on-fail Stop running if any single test fails.
|
||||
This can be useful if you are running something
|
||||
other than test files with Urchin.
|
||||
This is useful if you are running something
|
||||
configuration files with Urchin.
|
||||
-T, --timeout <seconds> Kill a test if it runs for longer than the
|
||||
specified duration. The default is no timeout.
|
||||
-f, --force Force running even if the test directory's name
|
||||
@@ -552,11 +472,9 @@ These options affect how results are formatted. Options -q, and -v
|
||||
have no effect when combined with --tap. -vv, -vvv, and -vvvv do have
|
||||
effect when combined with --tap.
|
||||
|
||||
-p, --pretty Print results in color and with fancy symbols.
|
||||
-c, --color Print results in color.
|
||||
-t, --tap Format output in Test Anything Protocol (TAP)
|
||||
|
||||
And these options affect how much is printed.
|
||||
|
||||
-q, --quiet Print nothing to stdout;
|
||||
the only output is the exit code.
|
||||
(default verbosity) Print names of failed tests and counts
|
||||
@@ -564,21 +482,14 @@ And these options affect how much is printed.
|
||||
-v Print stdout from failing tests.
|
||||
-vv Print names of passed tests.
|
||||
-vvv, --verbose Print stdout from all tests.
|
||||
-vvvv, --debug Run with set -x.
|
||||
-vvvv, --debug Print debugging messages (XXX not implemented)
|
||||
|
||||
The remaining flags provide information about urchin.
|
||||
|
||||
-h, --help Display this help.
|
||||
--version Display the version number.
|
||||
|
||||
Urchin recognizes certain environment variables.
|
||||
|
||||
TEST_SHELL This is sometimes over-ridden; see -s.
|
||||
RUN_IN_SERIES Set this to true to have the same effect as
|
||||
-b/--run-in-series. This is helpful if you are
|
||||
calling urchin inside an urchin test suite.
|
||||
|
||||
Go to https://thomaslevine.com/!/urchin/ for documentation on writing tests.
|
||||
Go to https://github.com/tlevine/urchin for documentation on writing tests.
|
||||
|
||||
EOF
|
||||
}
|
||||
@@ -604,13 +515,10 @@ validate_strings() {
|
||||
}
|
||||
|
||||
main() {
|
||||
argv="$@"
|
||||
|
||||
max_forks=50
|
||||
cycle_shell=true
|
||||
shell_list="${urchin_tmp}"/shell_list
|
||||
test_arg_list="${urchin_tmp}"/test_list
|
||||
> "${test_arg_list}"
|
||||
remotes_list="${urchin_tmp}"/remotes
|
||||
run_in_series=false
|
||||
force=false
|
||||
exit_on_not_ok=false
|
||||
@@ -626,6 +534,8 @@ main() {
|
||||
do
|
||||
case "${1}" in
|
||||
-b|--run-in-series) run_in_series=true;;
|
||||
-n|--max-forks) shift
|
||||
max_forks="${1}";;
|
||||
-e|--exit-on-fail) exit_on_not_ok=true;;
|
||||
-f|--force) force=true;;
|
||||
-s|--shell)
|
||||
@@ -639,26 +549,17 @@ main() {
|
||||
}
|
||||
|
||||
validate_strings "${shell_for_sh_tests}" 'Shell paths'
|
||||
if echo "${shell_for_sh_tests}" | grep -q \ ; then
|
||||
if echo "${shell_for_sh_tests}" | grep \ > /dev/null; then
|
||||
echo "Warning: It is best if shell paths contain no spaces so that
|
||||
you don't need to quote the TEST_SHELL variable." >&2
|
||||
you don't need to quote the TEST_SHELL variable." > /dev/stderr
|
||||
fi
|
||||
|
||||
echo "${shell_for_sh_tests}" >> "${shell_list}"
|
||||
;;
|
||||
-n|--disable-cycling) cycle_shell=false;;
|
||||
-d|--disable-cycling) cycle_shell=false;;
|
||||
-t|--tap) tap_format=true;;
|
||||
-T|--timeout)
|
||||
shift
|
||||
urchin_timeout="${1}"
|
||||
if ! {
|
||||
echo "${urchin_timeout}" |
|
||||
grep '[0-9][0-9.]*\(s\|m\|h\|d\|\)'
|
||||
}; then
|
||||
echo Bad timeout argument: "${urchin_timeout}" >&2
|
||||
urchin_exit 1
|
||||
fi ;;
|
||||
-p|--pretty) print_in_color=true;;
|
||||
-T|--timeout) shift; urchin_timeout="${1}" ;;
|
||||
-c|--color) print_in_color=true;;
|
||||
|
||||
-q|--quiet) print_not_ok=false
|
||||
print_margins=false;;
|
||||
@@ -668,10 +569,7 @@ main() {
|
||||
-vvv|--verbose)print_not_ok_stdout=true
|
||||
print_ok=true;
|
||||
print_ok_stdout=true;;
|
||||
-vvvv|--debug) print_not_ok_stdout=true
|
||||
print_ok=true;
|
||||
print_ok_stdout=true
|
||||
set -x;;
|
||||
-vvvv|--debug) echo 'Not implemented' > /dev/stderr && exit 1;;
|
||||
|
||||
-h|--help) urchin_help
|
||||
urchin_exit 0;;
|
||||
@@ -685,17 +583,6 @@ main() {
|
||||
esac
|
||||
shift
|
||||
done
|
||||
if "${RUN_IN_SERIES}" 2> /dev/null; then
|
||||
run_in_series=true
|
||||
fi
|
||||
|
||||
if $print_in_color; then
|
||||
success_mark=✓
|
||||
fail_mark=✗
|
||||
else
|
||||
success_mark=.
|
||||
fail_mark=F
|
||||
fi
|
||||
|
||||
# -------------------- VALIDATE INPUT -------------------- #
|
||||
if ! "${cycle_shell}" && test -f "${shell_list}"; then
|
||||
@@ -707,7 +594,7 @@ main() {
|
||||
if ! test -f "${shell_list}"; then
|
||||
if $cycle_shell; then
|
||||
for shell in $DEFAULT_SHELLS; do
|
||||
if which $shell 1> /dev/null 2> /dev/null; then
|
||||
if which $shell > /dev/null; then
|
||||
echo $shell >> "$shell_list"
|
||||
fi
|
||||
done
|
||||
@@ -716,14 +603,30 @@ main() {
|
||||
fi
|
||||
fi
|
||||
|
||||
# Warn about strange sort commands
|
||||
weird_string='@ b\n- d\n? a\n~ c\n! e\n'
|
||||
sort_result="$(printf "${weird_string}" | sort | cut -d\ -f2 | tr -d '\n')"
|
||||
if test "${sort_result}" = edacb; then
|
||||
sort=sort
|
||||
else
|
||||
if which python > /dev/null; then
|
||||
sort=sort_python
|
||||
else
|
||||
echo 'Your version of sort sorts in dictionary order (-d) by default.
|
||||
Depending on how you name your tests, your Urchin output may look strange.
|
||||
If this is a problem, install BusyBox or BSD coreutils.' >&2
|
||||
sort=sort
|
||||
fi
|
||||
fi
|
||||
|
||||
if test -n "${urchin_timeout}"; then
|
||||
# Choose the timeout command
|
||||
if timeout -t 0 true 2> /dev/null; then
|
||||
if timeout -t 0 true; then
|
||||
TIMEOUT="timeout -t ${urchin_timeout}"
|
||||
elif timeout 0 true 2> /dev/null; then
|
||||
elif timeout 0 true; then
|
||||
TIMEOUT="timeout ${urchin_timeout}"
|
||||
else
|
||||
echo I couldn\'t figure out how to use your version of timeout >&2
|
||||
echo I couldn\'t figure out to use your version of timeout >&2
|
||||
urchin_exit 1
|
||||
fi
|
||||
fi
|
||||
@@ -734,7 +637,7 @@ main() {
|
||||
fi
|
||||
|
||||
# -------------------- REALLY RUN -------------------- #
|
||||
start=$("${epoch}")
|
||||
start=$(date +%s)
|
||||
|
||||
# 1 test file or folder to run
|
||||
# 2 urchin root
|
||||
@@ -744,42 +647,16 @@ main() {
|
||||
recurse "$(fullpath "${seed}")" "${root}" "${cycle_shell}" \
|
||||
"${TEST_SHELL}" || break
|
||||
done < "${test_arg_list}"
|
||||
finish=$("${epoch}")
|
||||
finish=$(date +%s)
|
||||
|
||||
if test $(cat "${urchin_tmp}"/log | wc -l) -eq 0; then
|
||||
echo 'No tests found' >&2
|
||||
test $(cat "${urchin_tmp}"/log | wc -l) -gt 0 || {
|
||||
echo 'No tests found' > /dev/stderr
|
||||
urchin_exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if test -n "${RUNNING_ON_REMOTE}"; then
|
||||
echo "${urchin_tmp}"
|
||||
elif test -f "${remotes_list}"; then
|
||||
while read remote; do
|
||||
remote_main $remote "${argv}"
|
||||
done < "${remotes_list}"
|
||||
else
|
||||
report_outcome "${root}" "${tap_format}" "${urchin_tmp}"/log \
|
||||
"${start}" "${finish}"
|
||||
urchin_exit "${?}"
|
||||
fi
|
||||
}
|
||||
|
||||
remote_main() {
|
||||
hostname="${1}"; shift
|
||||
flags="${1}"; shift
|
||||
urchin_dir=.urchin-cross-shell-test
|
||||
|
||||
rsync --archive -e "ssh ${flags}"
|
||||
../urchin ../tests "${hostname}":"${urchin_dir}" ||
|
||||
scp -r ${flags} ../urchin ../tests "${hostname}":"${urchin_dir}"
|
||||
|
||||
remote_tmp="$(ssh "${hostname}" ${flags} "cd ${urchin_dir} && ./urchin $@")"
|
||||
|
||||
remotedir="${hostname}":"${remote_tmp}"
|
||||
localdir="${urchin_tmp}/remote/${host}"
|
||||
rsync --archive -e "ssh ${flags}" "${remotedir}" "${localdir}" ||
|
||||
scp -r ${flags} "${remotedir}"
|
||||
|
||||
report_outcome "${root}" "${tap_format}" "${urchin_tmp}"/log "${start}" \
|
||||
"${finish}"
|
||||
urchin_exit "${?}"
|
||||
}
|
||||
|
||||
test -n "${TESTING_URCHIN_INTERNALS}" || main "$@"
|
||||
|
||||
BIN
urchin_test_on_msys64_1.png
Normal file
BIN
urchin_test_on_msys64_1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
BIN
urchin_test_on_msys64_2.png
Normal file
BIN
urchin_test_on_msys64_2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 115 KiB |
Reference in New Issue
Block a user