1 Commits

Author SHA1 Message Date
Thomas Levine
9d1b16ed57 start with max forks 2016-03-06 14:33:25 +00:00
69 changed files with 181 additions and 478 deletions

31
HISTORY
View File

@@ -1,18 +1,10 @@
HISTORY
=======
Version 0.1.0
Version 0.1.0-rc1
---------------------
This release includes breaking changes.
### Cross-OS testing
I have started testing Urchin across multiple operating systems.
This gives access to more shells, as some shels are easier to install on
certain operating systems.
With this cross-OS test suite, I have extended support to more shells.
A later version of Urchin could include a remote testing feature.
### Test root directory
We introduce a concept of the root directory of a test suite.
Such a concept is important in case you want to run subsets of your
@@ -20,18 +12,18 @@ test suite, as we need to know how far up to apply the setup
and teardown files.
The Urchin root directory is determined by moving higher in the directory
tree in search of a file named `.urchin_root`.
tree in search of a file named `.urchin`.
The closest directory that contains such a file is considered the root.
In the following filesystem, for example, `/a/b/c` would be the root.
mkdir -p /a/b/c/d
touch /a/b/c/d/e
chmod +x /a/b/c/d/e
touch /a/b/c/.urchin_root
touch /a/b/c/.urchin
urchin /a/b/c/d
There are two situations in which we would stop looking without having
found a `.urchin_root` file.
found a `.urchin` file.
1. The system root, `/`, because we can't go any higher
2. A directory that starts with a dot, because an urchin call on a higher
@@ -80,7 +72,7 @@ might look for the dependency and then skip if it does not see the dependency.
It might look like this.
#!/bin/sh
if ! which inkscape; then
if which inkscape; then
exit 3 # status code 3 for skip
fi
inkscape blah blah ...
@@ -105,13 +97,6 @@ turned on.
Parallel processing and shell cycling are both enabled by default.
You may want make only some directories run in series, you can create
".urchin_dir" files in those directories.
If .urchin_dir contains the phrase "series", run that directory in series
rather than in parallel.
This is helpful when directories actually need to run in series
and also when running all your tests in parallel crashes your computer.
### Options
Long options are now available for all command line flags.
For example, the `-s` flag is now available as `--shell` as well.
@@ -179,6 +164,10 @@ set as an environment variable, and the latter was set with the -s flag..
Urchin now uses the -s flag for both of these settings, and it mostly ignores
the exported TEST_SHELL variable.
Urchin also inspects the shebang line differently. Previously, Urchin would
vary the shells with which a test is run if the shebang line either was absent
or was #!/bin/sh. Now it varies the shell only if the shebang line is absent.
If you pass -n/--disable-cycling, Urchin will invoke tests ordinarily and will
only set the TEST_SHELL variable if it does not exist. If the TEST_SHELL
variable is absent, it will be set to /bin/sh.
@@ -240,7 +229,7 @@ other Urchin call), and the test suite is recursively descended. Setup and
teardown files are sourced, and everything but the specified test file is
otherwise ignored.
If you don't explicitly specify the Urchin root with a .urchin_root file, we
If you don't explicitly specify the Urchin root with a .urchin file, we
consider the test suite root directory to be the parent of the file that
you ran Urchin on.

View File

@@ -1,8 +0,0 @@
.PHONY: test install
test:
./urchin tests
./urchin -s sh -v ./cross-os-tests
install:
cp ./urchin /usr/bin

82
SORTING
View File

@@ -1,67 +1,5 @@
On the criteria for ordering
==============================
I was confused by the documentation for sort's "-d" flag. This confusion
relates to GNU coreutil's locale-specific sort. [^]
Below I discuss sort order differences between different implementations
of sort and of sh "*" for my particular environments.
Sorting with sort
------------
Consider the following two sort commands.
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort -d
With BusyBox v1.23.2 on NixOS 15.09, the first of these commands returns
ASCIIbetical order,
! e
- d
? a
@ b
~ c
and the second returns dictionary order.
? a
@ b
~ c
- d
! e
With GNU coreutils version 8.24 on NixOS, both commands return
dictionary order. The same is true for GNU coreutils version 8.23 on
Debian Wheezy.
? a
@ b
~ c
- d
! e
IEEE Std 1003.1, 2013 Edition [^^] specifies that the "-d" flag should
enable dictionary order. All of these versions of sort have clear
documentation about the order that should be returned when the "-d" flag
is set, (See --help, man, or info.) and the implementations match the
documentation as far as I can tell.
I have found no explicit documentation from any relevant source as to
what the default sort order should be. On the other hand, they all
suggest that "-d" produces an order different from the default order.
In GNU coreutils 8.24, for example, "-d" is a direction to "consider
only blanks and alphanumeric characters". It lacks any mention that the
"-d" flag has no effect or that it is the default. Furthermore, on my
first reading, I took it to mean that the default is to consider all
characters and that "-d" limits the considered characters to blanks and
alphanumeric characters.
Sorting in *
-------------
I think this is related to the order returned by "*" in sh.
The following sh code creates several files in a directory and then
calls "*", listing them in order.
@@ -70,8 +8,7 @@ calls "*", listing them in order.
done
for file in *; do echo "$file"; done
On one computer, running FreeBSD, the order is apparently
ASCIIbetical.
On one computer, running FreeBSD, the order is apparently ASCIIbetical.
! e
- d
@@ -79,7 +16,7 @@ ASCIIbetical.
@ b
~ c
On two GNU systems, running NixOS and Debian, respectively, output is
On another computer, running NixOS, the following commands print results
in dictionary order. I'm not exactly sure what dictionary order is, but
it is something like sorting on the alphabetical characters before
sorting on the rest of the line.
@@ -90,9 +27,16 @@ sorting on the rest of the line.
- d
! e
(I don't really know what dictionary order is, I was able to determine
While I don't really know what dictionary order is, I was able to determine
that the above results are in dictionary order because of my investigation of
incompatible implementations of sort.)
incompatible implementations of sort. Consider the following two sort
commands.
[^] https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021
[^^] http://pubs.opengroup.org/onlinepubs/9699919799/
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort
printf '@ b\n- d\n? a\n~ c\n! e\n' | sort -d
With BSD sort, the first of these commands print ASCIIbetical order and
the second prints dictionary order. With GNU sort, both print dictionary
order.
How annoying.

74
TODO
View File

@@ -26,14 +26,11 @@ Windows
Try running Urchin in Windows somehow. Interpreters include
* CygWin (https://www.cygwin.com/)
* https://cygwin.com/setup-x86.exe
* MSYS (http://mingw.org/wiki/msys)
* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
* Git for Windows (https://git-scm.com/download/win)
* https://github.com/git-for-windows/git/releases/download/v2.7.2.windows.1/Git-2.7.2-32-bit.exe
* win-bash (http://win-bash.sourceforge.net/)
shall
----------
Add shall to my NYC*BUG talk.
@@ -49,68 +46,15 @@ List some shell linters somewhere.
* checkbashisms
Set parallel and series
----------
I want to be able to make only some directories run in series.
* Rename .urchin to .urchin_root.
* Look for a .urchin_dir file.
* If .urchin_dir contains "series", run that directory in series rather
than in parallel.
Rename to something other than "test"?
----------
Maybe wait until I have a use for this.
More sort alternatives
-----------
awk
https://stackoverflow.com/questions/20250937/sorting-lines-in-a-file-alphabetically-using-awk-and-or-sed
bash
https://stackoverflow.com/questions/7442417/how-to-sort-an-array-in-bash
Alternatives
--------------
JSON.sh test suite
Running in multiple environments
-----------------------------------
Setup for other environments includes the following.
* Installing packages
* `touch .zshrc`
* Copy urchin and tests
Bugs
-------
Both md5sum and md5 should be supported.
Trouble logging in to hpux, irix, miros, netbsd, tru64, qnx, ....
$ rsync -e 'ssh -p 785' urchin tlevine@hpux.polarhome.com:.blah
HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
Welcome to HPUX/PA... member of polarhome.com realm
bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: remote command not found (code 127) at io.c(226)
[sender=3.1.1]
OpenIndiana grep does not support -q
I get `/urchin: syntax error at line 84: \`}' unexpected` on
unixware and solaris.
mktemp
> tlevine@hpux.polarhome.com -p 785
F sh (8 seconds)
|
| HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
| Welcome to HPUX/PA... member of polarhome.com realm
|
|
| HP-UX hpux.polarhome.com B.11.11 U 9000/785 (ta)
| Welcome to HPUX/PA... member of polarhome.com realm
|
| mktemp: option requires an argument -- d
| ./urchin[96]: /tmp/tlevinea21441/log: Cannot create the specified file.
date
tlevine@hpux64$ ./urchin tests/ -n -vv
date: bad format character - s

View File

@@ -1,13 +0,0 @@
params="$(basename "${0}")"
hostname="$(echo "${params}" | cut -d\ -f1)"
if echo "${params}" | grep -q \ ; then
flags="$(echo "${params}" | cut -d\ -f2-)"
fi
urchin_dir=.urchin-cross-shell-test
rsync --archive -e "ssh ${flags}" $RSYNC_FLAGS \
../urchin ../tests "${hostname}":"${urchin_dir}"
ssh "${hostname}" ${flags} \
"cd ${urchin_dir} && ./urchin --run-in-series tests"

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,3 +0,0 @@
#!/bin/sh
# apt-get install bash dash ksh posh pdksh mksh yash zsh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,3 +0,0 @@
#!/bin/sh
RSYNC_FLAGS='--rsync-path=/usr/local/bin/rsync'
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,3 +0,0 @@
#!/bin/sh
# SSH public key needs to be in ~/.etc/ssh/authorized_keys
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

View File

@@ -1,2 +0,0 @@
#!/bin/sh
. ./.run

1
packages/.gitignore vendored
View File

@@ -1 +0,0 @@
*.tar.gz

View File

@@ -1,11 +0,0 @@
#!/bin/sh
name=urchin-$(../urchin --version)
tmp=$(mktemp -d)
mkdir $tmp/$name
cp ../urchin ../readme.md ../AUTHORS ../COPYING $tmp/$name
cd $tmp
tar czf $name.tar.gz $name
cd - > /dev/null
mv $tmp/$name.tar.gz .
rm -R $tmp

View File

@@ -43,15 +43,25 @@ Urchin depends on the following programs.
* timeout
* sort
Vanilla installations of modern BSD and GNU systems usually include all
of these programs.
All of the above programs are usually included on base BSD installations.
On GNU systems it should be sufficient to install the busybox package.
Urchin uses sort to format its output. GNU sort (as of GNU coreutils version
8.24) lacks the ability to sort in lexicographic order, and this feature is
necessary for the output to look right. If your version of sort lacks this
feature, Urchin will try to use one of the following tools for sorting.
If no acceptable sorting program is available, Urchin will print a warning
and use the incomplete sort that is installed on your system. This is not a
big deal; if your test files all start with alphanumeric letters, the output
should look fine.
## Install
Urchin is contained in a single file, so you can install it by copying it to a
directory in your `PATH`. For example, you can run the following as root.
cd /usr/local/bin
wget https://raw.githubusercontent.com/tlevine/urchin/v0.1.0-rc3/urchin
wget https://raw.githubusercontent.com/tlevine/urchin/v0.0.6/urchin
chmod +x urchin
Urchin can be installed with npm too.
@@ -108,36 +118,14 @@ Files are only run if they are executable, and files beginning with `.` are
ignored. Thus, fixtures and libraries can be included sloppily within the test
directory tree. The test passes if the file exits 0; otherwise, it fails.
urchin looks for files within a directory in the following manner,
Tests files and subdirectories are run in ASCIIbetical order within each
directory; that is,
urchin looks for files within a directory in the following manner.
for file in *; do
do_something_with_test_file $file
done
so files are run in whatever order `*` produces. The order is
configured in your environment, at least in
[GNU systems](https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#Sort-does-not-sort-in-normal-order_0021).
Other systems may ignore the locales configured in the environment and
always produce ASCIIbetical order.
Results are always printed in ASCIIbetical order, regardless of what
order the tests ran in.
Below you can see how the locale can affect the order.
$ printf '!c\n@a\n~b\n' | LC_COLLATE=C sort
!c
@a
~b
$ printf '!c\n@a\n~b\n' | LC_COLLATE=en_US.UTF-8 sort
@a
~b
!c
$ printf '!c\n@a\n~b\n' | sort -d
@a
~b
!c
### Writing cross-shell compatibility tests for testing shell code
While you could write your test scripts to explicitly invoke the functionality
@@ -191,9 +179,14 @@ shell.)
Note that only test scripts that either have no shebang line at all or
have shebang line `#!/bin/sh` are invoked with the specified shell.
This allows non-shell test scripts or test scripts for other languages
or for specific shells to coexist with those whose invocation should be
controlled by `-s`.
This allows non-shell test scripts or test scripts for specific
shells to coexist with those whose invocation should be controlled by `-s`.
To test with multiple shells in sequence, use something like:
for shell in sh bash ksh zsh; do
urchin -s $shell ./tests
done
## References

View File

@@ -1 +0,0 @@
series

View File

@@ -0,0 +1,6 @@
#!/bin/sh
# This script should run with /bin/sh
# regardless of whether -s or -n is passed.
ps -o pid,comm,args | grep $$ | grep /bin/sh

View File

@@ -1,6 +0,0 @@
#!/bin/sh
# Assuming that urchin was invoked with `-s bash`,
# this script should be being run with bash.
ps -o pid,comm,args | grep $$ | grep .special-shell

View File

@@ -1 +1 @@
! $TEST_SHELL ../../urchin ../Flags/Urchin\ format|grep -- --pretty
! $TEST_SHELL ../../urchin ../Flags/Urchin\ format|grep -- --color

View File

@@ -1 +1,3 @@
set -e
$TEST_SHELL ../../urchin --shell sh .slow-tests
! $TEST_SHELL ../../urchin --shell sh --timeout 0.3 .slow-tests

View File

@@ -1,2 +0,0 @@
../../urchin -T aoeu .testsuite 2>&1 | grep Bad
../../urchin -T .testsuite 2>&1 | grep Bad

View File

@@ -1,4 +0,0 @@
$TEST_SHELL ../../urchin --shell sh --timeout 0.3 .slow-tests 2>&1 |
grep -v -- --timeout |
grep timeout
test $? = 1

View File

@@ -1 +0,0 @@
$TEST_SHELL ../../urchin --shell sh .slow-tests --timeout 1000

View File

@@ -1,7 +1,7 @@
#!/bin/sh
set -e
! $TEST_SHELL ../../urchin -vv --run-in-series --exit-on-fail \
! $TEST_SHELL ../../urchin --run-in-series --exit-on-fail \
./.test_-e,--exit-on-fail > $tmp
grep '1 should run.' $tmp

View File

@@ -1,11 +1,11 @@
./
> a
F sh (1 second)
| This is stdout from a.
sh (1 second)
# This is stdout from a.
./
> b
. sh (1 second)
sh (1 second)
./
> c
(File is not executable.)

View File

@@ -2,7 +2,7 @@
./
> a
✗ sh (1 second)
| This is stdout from a.
# This is stdout from a.
./
> b
✓ sh (1 second)

View File

@@ -1,3 +1,3 @@
$TEST_SHELL ../../urchin -v -s sh -t .testsuite/ |
$TEST_SHELL ../../urchin -s sh -t .testsuite/ |
sed -e 1d -e /second/d > $tmp
diff $tmp .tap-output-expectation

View File

@@ -1,3 +1,3 @@
$TEST_SHELL ../../urchin --pretty -vv --shell sh .testsuite/ |
$TEST_SHELL ../../urchin --color -vv --shell sh .testsuite/ |
sed -e 1d -e 's/. seconds\?/1 second/' > $tmp
diff $tmp .urchin-output-expectation-color

View File

@@ -1,3 +0,0 @@
echo "$1" > $tmp
TESTING_URCHIN_INTERNALS=true . ../../../urchin
has_shebang_line $tmp

View File

@@ -1 +0,0 @@
./.run '#!/bin/bash'

View File

@@ -1 +0,0 @@
! ./.run ''

View File

@@ -1,5 +0,0 @@
! ./.run '
'

View File

@@ -1 +0,0 @@
./.run '#!/usr/bin/env true'

View File

@@ -1 +0,0 @@
export tmp=$(mktemp)

View File

@@ -1 +0,0 @@
! ./.run '#!/bin/sh'

View File

@@ -1 +0,0 @@
rm -R "$tmp"

4
tests/Internals/sort_python Executable file
View File

@@ -0,0 +1,4 @@
set -e
TESTING_URCHIN_INTERNALS=true . ../../urchin
unsorted='@ b\n- d\n? a\n~ c\n! e\n'
test $(printf "${unsorted}" | sort_python | cut -d\ -f2|tr -d '\n') = edabc

View File

@@ -1,3 +0,0 @@
#!/bin/sh
[ $(grep -c 'setup has run' $log) -eq '1' ]

View File

@@ -1,3 +0,0 @@
#!/bin/sh
[ $(grep -c 'setup has run' $log) -eq '2' ]

View File

@@ -0,0 +1,3 @@
#!/bin/sh
[ $(grep -c 'setup has run' $log) -gt '2' ]

View File

@@ -0,0 +1,3 @@
#!/bin/sh
[ $(grep -c 'setup has run' $log) -gt '2' ]

267
urchin
View File

@@ -51,17 +51,7 @@ set -e
# Kill subprocesses on interrupt.
trap "kill -$$; exit" HUP INT TERM
DEFAULT_SHELLS='
sh
bash
dash
ksh
posh
pdksh
mksh
yash
zsh
'
DEFAULT_SHELLS='sh bash dash mksh zsh'
if [ -n "${ZSH_VERSION}" ]; then
# avoid "no matches found: *" error when directories are empty
setopt NULL_GLOB
@@ -80,7 +70,7 @@ validate_test_arg() {
root="$(urchin_root "${1}")"
if ! {
basename "$(fullpath "${root}")" |
grep -qi 'test' || "${force}"
grep -i 'test' > /dev/null || "${force}"
}; then
echo 'The root directory of the tests that you are running urchin on
does not contain the word "test", so I am not running,
@@ -90,45 +80,28 @@ validate_test_arg() {
fi
}
sort_python() {
python -c 'import sys
for line in sorted(sys.stdin.readlines()):
sys.stdout.write(line)
'
}
# All temporary files go here
urchin_tmp=$(mktemp)
# Support HP-UX mktemp that has wrong exit codes and
# can't make directories.
if test -f "${urchin_tmp}"; then
rm "${urchin_tmp}"
fi
mkdir "${urchin_tmp}"
urchin_tmp=$(mktemp -d)
> "${urchin_tmp}/log"
urchin_exit() {
rm -Rf "${urchin_tmp}"
exit "$@"
}
if which md5 1> /dev/null 2> /dev/null; then
urchin_md5=md5
elif which md5sum 1> /dev/null 2> /dev/null; then
urchin_md5=md5sum
else
echo Could not find MD5 hash command >&2
urchin_exit 1
fi
stdout_file() {
the_test="${1}"
the_shell="${2}"
x="${urchin_tmp}/stdout$(fullpath "$the_test")"
mkdir -p "${x}"
case "${urchin_md5}" in
md5sum) y=$(echo "${the_shell}" | md5sum | cut -d\ -f1) ;;
md5) y=$(echo "${the_shell}" | md5 -q) ;;
*) echo md5 command is not configured >&2; urchin_exit 1;;
esac
echo "${x}/${y}"
echo "${x}/$(echo "${the_shell}" | md5sum | cut -d\ -f1)"
}
# Expand relative paths
@@ -151,7 +124,7 @@ urchin_root() {
abscurrent="$(fullpath "${1}")"
if test "${abscurrent}" = / ||
basename "${abscurrent}" | grep -q '^\.' ; then
basename "${abscurrent}" | grep '^\.' > /dev/null; then
# Stop traversing upwards at / and at hidden directories.
if test -d "${orig}"; then
echo "${orig}"
@@ -163,7 +136,7 @@ urchin_root() {
return 1
elif test -f "${current}"; then
urchin_root "$(dirname -- "${current}")" "${orig}"
elif test -f "${current}"/.urchin_root; then
elif test -f "${current}"/.urchin; then
remove_trailing_slash "${current}"
else
urchin_root "${current}"/.. "${orig}"
@@ -171,7 +144,7 @@ urchin_root() {
}
# Urchin version number
VERSION=0.1.1-unstable
VERSION=0.1.0-rc1
indent() {
level="${1}"
@@ -186,18 +159,14 @@ recurse() {
cycle_shell="${3}"
TEST_SHELL="${4}"
if $print_debug; then
echo Entered directory "${PWD}"
fi
for ignore in setup_dir teardown_dir setup teardown; do
if test "$(basename "${potential_test}")" = "${ignore}"; then
return
fi
done
if echo "${requested_path}" | grep -q "^${potential_test}" ||
echo "${potential_test}" | grep -q "^${requested_path}" ; then
if echo "${requested_path}" | grep "^${potential_test}" > /dev/null ||
echo "${potential_test}" | grep "^${requested_path}" > /dev/null; then
if test "$(dirname "${potential_test}")" = \
"$(dirname "${requested_path}")" &&
test "${potential_test}" != "${requested_path}"; then
@@ -213,18 +182,7 @@ recurse() {
if [ -d "${potential_test}" ]; then
(
cd -- "${potential_test}"
if test -f .urchin_dir && grep -q series .urchin_dir ; then
run_in_series_dir=true
else
run_in_series_dir=false
fi
if test -f setup_dir; then
if $print_debug; then
echo Sourcing "${PWD}/setup_dir"
fi
. ./setup_dir
fi
if test -f setup_dir; then . ./setup_dir; fi
for test in *; do
if test "${test}" = '*' && ! test -e "${test}"; then
@@ -232,47 +190,37 @@ recurse() {
break
fi
recurse "${requested_path}" "${test}" "${cycle_shell}" \
"${TEST_SHELL}" &
(
if test -f setup; then . ./setup; fi
if recurse "${requested_path}" "${test}" "${cycle_shell}" \
"${TEST_SHELL}"; then
exit_code=0
else
exit_code="${?}"
fi
if test -f teardown; then . ./teardown; fi
exit "${exit_code}"
) &
if "${run_in_series}" || "${run_in_series_dir}"; then
if "${run_in_series}"; then
if wait "${!}"; then exit_code=0; else exit_code="${?}"; fi
if "${exit_on_not_ok}" && test "${exit_code}" -ne 0; then
if test -f teardown_dir; then
if $print_debug; then
echo Sourcing "${PWD}/teardown_dir"
fi
. ./teardown_dir
fi
if test -f teardown_dir; then . ./teardown_dir; fi
return 1
fi
fi
done
wait
if test -f teardown_dir; then
if $print_debug; then
echo Sourcing "${PWD}/teardown_dir"
fi
. ./teardown_dir
fi
if test -f teardown_dir; then . ./teardown_dir; fi
)
elif [ -f "${potential_test}" ]; then
cd -- "$(dirname -- "${potential_test}")"
if $print_debug; then
echo Running "${potential_test}"
fi
# Determine the environment variable to define for test scripts
# that reflects the specified or implied shell to use for shell-code tests.
while read the_test_shell; do
(
if test -f setup; then
if $print_debug; then
echo Sourcing "${PWD}/setup"
fi
. ./setup
fi
if test -f setup; then . ./setup; fi
# Run the test
start=$(date +%s)
@@ -286,7 +234,7 @@ recurse() {
"${the_test_shell}" "${potential_test}"
fi
else
# Shell cycling is disabled with -n; use the present value of
# Shell cycling is disabled with -d; use the present value of
# TEST_SHELL or default to /bin/sh
if [ -n "${TEST_SHELL}" ]; then
$TIMEOUT "${potential_test}"
@@ -299,12 +247,7 @@ recurse() {
set -e
finish=$(date +%s)
if test -f teardown; then
if $print_debug; then
echo Sourcing "${PWD}/teardown"
fi
. ./teardown
fi
if test -f teardown; then . ./setup; fi
if [ "${exit_code}" -eq 0 ]; then
result=ok
@@ -323,19 +266,14 @@ recurse() {
if "${run_in_series}"; then
if wait "${!}"; then exit_code=0; else exit_code="${?}"; fi
if "${exit_on_not_ok}" && test "${exit_code}" -ne 0; then
if test -f teardown_dir; then
if $print_debug; then
echo Sourcing "${PWD}/teardown_dir"
fi
. ./teardown_dir
fi
if test -f teardown_dir; then . ./teardown_dir; fi
return 1
fi
fi
done < "${shell_list}"
wait
else
echo "${potential_test}: Neither file nor directory!?" >&2
echo "${potential_test}: Neither file nor directory!?" > /dev/stderr
fi
else
# Shell is ''
@@ -347,9 +285,11 @@ report_outcome() {
root="${1}"
tap_format="${2}"
log_file="${3}"
elapsed="${4}"
start="${4}"
finish="${5}"
escaped_root="$(fullpath "${root}" | sed 's/\//\\\//g')"
elapsed=$(($finish - $start))
if "${tap_format}"; then
printf \#\
@@ -365,7 +305,7 @@ report_outcome() {
# Use a temporary file rather than a pipe because a pipe starts a sub-shell
# and thus makes the above variables local.
sorted_log_file=$(mktemp)
cat "${log_file}" | LC_COLLATE=C sort > "${sorted_log_file}"
cat "${log_file}" | "${sort}" > "${sorted_log_file}"
while read line; do
abspath=$(echo "${line}" | cut -f1)
@@ -416,15 +356,15 @@ report_outcome() {
printf "$(dirname -- "${path}")/\n> $(basename -- "${path}")\n"
fi
}
case "${result}" in
ok)
if "${print_ok}"; then
header
# On success, print a green '✓'
if "${print_in_color}"; then
printf "\033[32m${success_mark} \033[0m"
printf '\033[32m \033[0m'
else
printf "${success_mark} "
printf '✓ '
fi
echo "${the_shell} ("${file_elapsed}" $(plural second $file_elapsed))"
fi
@@ -432,10 +372,11 @@ report_outcome() {
not_ok)
if "${print_not_ok}"; then
header
# On not_ok, print a red '✗'
if "${print_in_color}"; then
printf "\033[31m${fail_mark} \033[0m"
printf '\033[31m \033[0m'
else
printf "${fail_mark} "
printf '✗ '
fi
echo "${the_shell} ("${file_elapsed}" $(plural second $file_elapsed))"
fi
@@ -453,7 +394,7 @@ report_outcome() {
esac
if { test "${result}" = not_ok && "${print_not_ok_stdout}"; } ||
{ test "${result}" = ok && "${print_ok_stdout}"; }; then
sed 's/^/ | /' "$(stdout_file "${abspath}" "${the_shell}")"
sed 's/^/ # /' "$(stdout_file "${abspath}" "${the_shell}")"
fi
fi
@@ -483,10 +424,10 @@ report_outcome() {
}
has_shebang_line() {
head -n 1 "${1}" | grep -v '^#!/bin/sh$' | grep -q '^#!'
head -n 1 "${1}" | grep -qE '^#!'
}
USAGE="usage: $0 [options]... [test file or directory]..."
USAGE="usage: $0 [<options>] <test directory>"
urchin_help() {
cat <<EOF
@@ -501,15 +442,14 @@ particular test file once per shell.
On each run,
1. The TEST_SHELL environment variable is set to the particular shell.
2. If the test file lacks a shebang line or has a shebang line of
"#!/bin/sh", the test script is also executed in that shell.
2. If the test file lacks a shebang line, the test script is also
executed in that shell.
The following flags affect how this multiple-shell testing is handled.
-s, --shell <shell> Tell Urchin to use a different list of shells.
(You can pass this flag multiple times.)
-n, --disable-cycling Disable the cycling of shells; Urchin will
-d, --disable-cycling Disable the cycling of shells; Urchin will
execute test files ordinarily, implicitly using
sh for files that lack shebang lines. It will
set the TEST_SHELL variable to "/bin/sh" if and
@@ -519,9 +459,10 @@ The following flags affect how Urchin processes tests.
-b, --run-in-series Run tests in series. The default is to run tests
in parallel where possible.
-n, --max-forks Maximum number of parallel tests (Default is 50.)
-e, --exit-on-fail Stop running if any single test fails.
This can be useful if you are running something
other than test files with Urchin.
This is useful if you are running something
configuration files with Urchin.
-T, --timeout <seconds> Kill a test if it runs for longer than the
specified duration. The default is no timeout.
-f, --force Force running even if the test directory's name
@@ -531,11 +472,9 @@ These options affect how results are formatted. Options -q, and -v
have no effect when combined with --tap. -vv, -vvv, and -vvvv do have
effect when combined with --tap.
-p, --pretty Print results in color and with fancy symbols.
-c, --color Print results in color.
-t, --tap Format output in Test Anything Protocol (TAP)
And these options affect how much is printed.
-q, --quiet Print nothing to stdout;
the only output is the exit code.
(default verbosity) Print names of failed tests and counts
@@ -543,14 +482,14 @@ And these options affect how much is printed.
-v Print stdout from failing tests.
-vv Print names of passed tests.
-vvv, --verbose Print stdout from all tests.
-vvvv, --debug Print debugging messages.
-vvvv, --debug Print debugging messages (XXX not implemented)
The remaining flags provide information about urchin.
-h, --help Display this help.
--version Display the version number.
Go to https://thomaslevine.com/!/urchin/ for documentation on writing tests.
Go to https://github.com/tlevine/urchin for documentation on writing tests.
EOF
}
@@ -576,6 +515,7 @@ validate_strings() {
}
main() {
max_forks=50
cycle_shell=true
shell_list="${urchin_tmp}"/shell_list
test_arg_list="${urchin_tmp}"/test_list
@@ -590,11 +530,12 @@ main() {
print_not_ok=true
print_ok_stdout=false
print_not_ok_stdout=false
print_debug=false
while [ "${#}" -gt 0 ]
do
case "${1}" in
-b|--run-in-series) run_in_series=true;;
-n|--max-forks) shift
max_forks="${1}";;
-e|--exit-on-fail) exit_on_not_ok=true;;
-f|--force) force=true;;
-s|--shell)
@@ -608,26 +549,17 @@ main() {
}
validate_strings "${shell_for_sh_tests}" 'Shell paths'
if echo "${shell_for_sh_tests}" | grep -q \ ; then
if echo "${shell_for_sh_tests}" | grep \ > /dev/null; then
echo "Warning: It is best if shell paths contain no spaces so that
you don't need to quote the TEST_SHELL variable." >&2
you don't need to quote the TEST_SHELL variable." > /dev/stderr
fi
echo "${shell_for_sh_tests}" >> "${shell_list}"
;;
-n|--disable-cycling) cycle_shell=false;;
-d|--disable-cycling) cycle_shell=false;;
-t|--tap) tap_format=true;;
-T|--timeout)
shift
urchin_timeout="${1}"
if ! {
echo "${urchin_timeout}" |
grep '[0-9][0-9.]*\(s\|m\|h\|d\|\)'
}; then
echo Bad timeout argument: "${urchin_timeout}" >&2
urchin_exit 1
fi ;;
-p|--pretty) print_in_color=true;;
-T|--timeout) shift; urchin_timeout="${1}" ;;
-c|--color) print_in_color=true;;
-q|--quiet) print_not_ok=false
print_margins=false;;
@@ -637,10 +569,7 @@ main() {
-vvv|--verbose)print_not_ok_stdout=true
print_ok=true;
print_ok_stdout=true;;
-vvvv|--debug) print_not_ok_stdout=true
print_ok=true;
print_ok_stdout=true
print_debug=true;;
-vvvv|--debug) echo 'Not implemented' > /dev/stderr && exit 1;;
-h|--help) urchin_help
urchin_exit 0;;
@@ -655,14 +584,6 @@ main() {
shift
done
if $print_in_color; then
success_mark=✓
fail_mark=✗
else
success_mark=.
fail_mark=F
fi
# -------------------- VALIDATE INPUT -------------------- #
if ! "${cycle_shell}" && test -f "${shell_list}"; then
echo "The -n/--disable-cycling and -s/--shell options clash with each other." >&2
@@ -673,7 +594,7 @@ main() {
if ! test -f "${shell_list}"; then
if $cycle_shell; then
for shell in $DEFAULT_SHELLS; do
if which $shell 1> /dev/null 2> /dev/null; then
if which $shell > /dev/null; then
echo $shell >> "$shell_list"
fi
done
@@ -682,19 +603,30 @@ main() {
fi
fi
if $print_debug; then
echo Cycling with the following shells:
cat "${shell_list}"
# Warn about strange sort commands
weird_string='@ b\n- d\n? a\n~ c\n! e\n'
sort_result="$(printf "${weird_string}" | sort | cut -d\ -f2 | tr -d '\n')"
if test "${sort_result}" = edacb; then
sort=sort
else
if which python > /dev/null; then
sort=sort_python
else
echo 'Your version of sort sorts in dictionary order (-d) by default.
Depending on how you name your tests, your Urchin output may look strange.
If this is a problem, install BusyBox or BSD coreutils.' >&2
sort=sort
fi
fi
if test -n "${urchin_timeout}"; then
# Choose the timeout command
if timeout -t 0 true 2> /dev/null; then
if timeout -t 0 true; then
TIMEOUT="timeout -t ${urchin_timeout}"
elif timeout 0 true 2> /dev/null; then
elif timeout 0 true; then
TIMEOUT="timeout ${urchin_timeout}"
else
echo I couldn\'t figure out how to use your version of timeout >&2
echo I couldn\'t figure out to use your version of timeout >&2
urchin_exit 1
fi
fi
@@ -705,23 +637,22 @@ main() {
fi
# -------------------- REALLY RUN -------------------- #
elapsed=$({
time {
# 1 test file or folder to run
# 2 urchin root
# 3 Should we cycle shells?
# 4 TEST_SHELL
while read seed; do
recurse "$(fullpath "${seed}")" "${root}" "${cycle_shell}" \
"${TEST_SHELL}" || break
done < "${test_arg_list}"
}
} 3>&1 1>&2 2>&3 3>& | grep ^real | cut -f 2)
start=$(date +%s)
if test $(cat "${urchin_tmp}"/log | wc -l) -eq 0; then
echo 'No tests found' >&2
# 1 test file or folder to run
# 2 urchin root
# 3 Should we cycle shells?
# 4 TEST_SHELL
while read seed; do
recurse "$(fullpath "${seed}")" "${root}" "${cycle_shell}" \
"${TEST_SHELL}" || break
done < "${test_arg_list}"
finish=$(date +%s)
test $(cat "${urchin_tmp}"/log | wc -l) -gt 0 || {
echo 'No tests found' > /dev/stderr
urchin_exit 1
fi
}
report_outcome "${root}" "${tap_format}" "${urchin_tmp}"/log "${start}" \
"${finish}"

BIN
urchin_test_on_msys64_1.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 99 KiB

BIN
urchin_test_on_msys64_2.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 115 KiB