50 Commits

Author SHA1 Message Date
Thomas Levine
40950a0b52 add -- after readlink 2016-02-27 19:10:28 +00:00
Thomas Levine
dc53523deb fix / detection 2016-02-27 18:41:57 +00:00
Thomas Levine
a797fc2c8b typo 2016-02-27 18:31:41 +00:00
Thomas Levine
e07b340906 Automatic commit with j 2016-02-27 18:26:14 +00:00
Thomas Levine
a4a0aabbef cleaner comment 2016-02-27 18:13:12 +00:00
Thomas Levine
f4d3ffa0e4 Automatic commit with j 2016-02-27 18:12:33 +00:00
Thomas Levine
28832f22a1 check hidden files better 2016-02-27 18:09:40 +00:00
Thomas Levine
f5ef61633e better check 2016-02-27 18:06:19 +00:00
Thomas Levine
2836d4b9a9 woo 2016-02-27 17:53:15 +00:00
Thomas Levine
972fe6bc35 aoeu 2016-02-27 17:51:14 +00:00
Thomas Levine
633d6d32c9 better than main 2016-02-27 17:44:35 +00:00
Thomas Levine
b9d72aef72 remove debug statements 2016-02-27 17:41:37 +00:00
Thomas Levine
92cec52c97 main function 2016-02-27 17:33:46 +00:00
Thomas Levine
b9a067c68e make the .urhin directory 2016-02-27 17:14:32 +00:00
Thomas Levine
e9d6b73dbd urchin_root 2016-02-27 17:13:55 +00:00
Thomas Levine
b8bd097f5c Automatic commit with j 2016-02-27 16:59:47 +00:00
Thomas Levine
f83df14868 set +e 2016-02-27 16:56:44 +00:00
Thomas Levine
fdc7129921 urchin root 2016-02-27 16:53:27 +00:00
Thomas Levine
e5ba45ae96 nicer log file location 2016-02-27 16:05:48 +00:00
Thomas Levine
31e0b9fcb7 quote 2016-02-27 16:01:18 +00:00
Thomas Levine
3efcf0aa33 blah 2016-02-27 15:59:17 +00:00
Thomas Levine
97faea610c error message when no root is set 2016-02-27 14:50:04 +00:00
Thomas Levine
f9ddefcf54 fix test 2016-02-27 14:45:57 +00:00
Thomas Levine
54b9e5887d more urchin_root 2016-02-27 14:45:37 +00:00
Thomas Levine
c9af70b947 simpler urchin root 2016-02-27 14:41:37 +00:00
Thomas Levine
83af249dcd test urchin_root 2016-02-27 14:39:32 +00:00
Thomas Levine
160222c0eb test suite to test .urchin 2016-02-27 14:14:15 +00:00
Thomas Levine
4cd9d41a1e Automatic commit with j 2016-02-26 20:38:31 +00:00
Thomas Levine
e14d8edc05 .urchin in root idea 2016-02-26 18:30:33 +00:00
Thomas Levine
15acd676f9 nagios explanation 2016-02-26 17:43:05 +00:00
Thomas Levine
0c73e5eb44 move skip tests to history 2016-02-26 17:39:18 +00:00
Thomas Levine
bb103757e4 implement exit code 3 thing 2016-02-26 17:37:39 +00:00
Thomas Levine
dce0adc768 test skip on exit code 3 2016-02-26 17:37:12 +00:00
Thomas Levine
80ebf28da9 test skipping idea 2016-02-26 16:39:53 +00:00
Thomas Levine
3f0e925975 move molly-guard change from todo to history 2016-02-26 16:26:22 +00:00
Thomas Levine
43674f12e2 more accepting molly-guard 2016-02-26 16:23:41 +00:00
Thomas Levine
9bb1366549 simpler truth test 2016-02-26 16:19:32 +00:00
Thomas Levine
643a1bf576 test a more accepting molly-guard 2016-02-26 16:18:49 +00:00
Thomas Levine
08ea7bb88f specify version in readme 2016-02-26 16:14:54 +00:00
Thomas Levine
e5ada4899a task runner ideas 2016-02-26 15:57:21 +00:00
Thomas Levine
c818033faa list alternatives 2016-02-14 16:38:02 +00:00
Thomas Levine
88572b6c90 to support single file runs 2016-02-14 07:44:07 +00:00
Thomas Levine
3e5e7334e8 sourcing idea 2016-02-11 20:18:35 +00:00
Thomas Levine
87bb9f4660 link to COPYING file on web 2016-02-08 16:09:00 +00:00
Thomas Levine
caabcd25ff update HISTORY 2016-02-08 16:06:22 +00:00
Thomas Levine
8f82cf5aeb make long lines shorter 2016-02-08 16:05:56 +00:00
Thomas Levine
385ec3fe81 add a copying notice to the urchin excutable 2016-02-08 16:02:03 +00:00
Thomas Levine
d4eb99ffb0 rename LICENSE -> COPYING 2016-02-08 15:59:56 +00:00
Thomas Levine
ed939ff39a update history 2016-02-08 15:59:47 +00:00
Thomas Levine
d24a71d0d7 copyright notice
add Thomas Levine and Michael Klement
2016-02-08 15:59:16 +00:00
25 changed files with 276 additions and 74 deletions

View File

@@ -1,3 +1,5 @@
Copyright (c) 2013, 2014, 2015, 2016 Thomas Levine
Copyright (c) 2014, Michael Klement
Copyright (c) 2012, ScraperWiki Limited
All rights reserved.

26
HISTORY
View File

@@ -1,6 +1,29 @@
HISTORY
=======
Version 0.0.7
---------------------
The Molly-guard is now more accepting. For example, you no longer need to
pass -f in this case: https://github.com/creationix/nvm/issues/357
Previously, tests were run if they were executable and were otherwise marked
as skipped. Now, an executable script can indicate that it is skipped by
exiting with code 3. For example, if a test requires some dependancy, it
might look for the dependency and then skip if it does not see the dependency.
It might look like this.
#!/bin/sh
if which inkscape; then
exit 3 # status code 3 for skip
fi
inkscape blah blah ...
I chose status code 3 sort of arbitrarily at first, but it turns out that it
would the appropriate status code if these tests were Nagios plugins, as the
concept of skipping a test is similar to the Nagios concept of unknown service
status (https://nagios-plugins.org/doc/guidelines.html#AEN78).
Version 0.0.6
---------------------
@@ -14,6 +37,9 @@ Version 0.0.6
* Remove the undocumented, experimental -x flag now that shall exists.
* Display version number with the -v flag.
* Document why Urchin is called "Urchin"
* Update TODO
* Support mksh (Change a printf command.)
* Make long lines shorter.
These changes are made somewhat separately in the branches "exit-on-fail",
"remove-urchin-x", "tap", and "update-readme". They are rebased into one

61
TODO
View File

@@ -1,13 +1,6 @@
Things I want
=============
Molly guard
-------------
The Molly-guard should be more accepting so that people don't have to use it
all the time and thus get used to using it. For example, you shouldn't need to
pass -f in this case.
https://github.com/creationix/nvm/issues/357
Test speed
-------------
Make tests run faster.
@@ -26,6 +19,11 @@ Hmm or maybe there's a compromise: Tell people to mount /tmp as a tmpfs so
that temp files are fast. Maybe allow people to set some other directory as
the temporary file place, in case they want a different tmpfs location.
In order to run things in parallel, we have to change how we do the
stdout_file. I think it's easiest to create separate files for each test and
to save them in testroot/.urchin/stdout/$filename. The test root would be
defined as the closest ancestor containing a .urchin directory.
Options
-------------
I want long options. For example, there's presently -f and -e.
@@ -115,3 +113,52 @@ Nagios plugins
It would be cool to run Nagios plugins with Urchin. This is already possible,
actually, but it might be worth giving some special thought to it.
https://nagios-plugins.org/doc/guidelines.html
Source setup and teardown
--------------------
If setup and teardown are sourced instead of executed, maybe we can more
cleanly create and teardown temporary files.
(
. ./setup
./$thetestfile
. ./teardown
)
On the other hand, this could just be sourced explicitly in the test file,
without the special setup and teardown feature.
Run on a file
----------------
Presently you can run urchin only on a directory.
It would be neat if you could run it on a file as well.
This occurred to me when I wanted to run
urchin test/fast/Unit\ tests/nvm_ls_current
on the nvm tests. I wound up running this instead.
urchin test/fast/Unit\ tests/ | grep nvm_ls_current
The Molly guard would be assessed, and the corresponding setup, setup_dir,
teardown, and teardown_dir files would be run in the appropriate order.
In order to know how far up the tree to evaluate the setup, &c. files,
I think it would make sense to require that a ".urchin" file be placed in the
root of the tests. Urchin would keep going up until it sees this file, and it
would evaluate the appropriate setup, &c. files from there down to the
particular test file of interest. We would also use this for testing
directtories more correctly.
Running automated tasks
-------------------------
Urchin might be appropriate for if you have lots of tasks that you want to run
periodically; add an urchin call to your crontab, and call all of your other
tasks with urchin. Here are some features that might make urchin better for
this sort of thing.
* Time how long each test/job takes
* Optionally kill tests/jobs after a specific timeout threshold
* Send output of different tests/jobs to different files for each file
descriptor (STDOUT, STDERR)

View File

@@ -1,2 +1,9 @@
Totally different syntax and similar features, plus TAP output
https://github.com/sstephenson/bats
Relatively similar interface
https://github.com/mlafeldt/sharness
Lists of alternatives
https://thomaslevine.com/!/shell-testing/
https://github.com/mlafeldt/sharness#alternatives

View File

@@ -36,7 +36,7 @@ Urchin is contained in a single file, so you can install it by copying it to a
directory in your `PATH`. For example, you can run the following as root.
cd /usr/local/bin
wget https://raw.github.com/tlevine/urchin/master/urchin
wget https://raw.githubusercontent.com/tlevine/urchin/v0.0.6/urchin
chmod +x urchin
Urchin can be installed with npm too.

2
tests/.urchin/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
*
!.gitignore

View File

@@ -0,0 +1,6 @@
#!/bin/sh
observed=$(../../urchin "$1" .testsuite/a/b)
expected=.testsuite/a/b/../..
test "$observed" = "$expected"

View File

@@ -0,0 +1,2 @@
*
!.gitignore

View File

@@ -0,0 +1 @@
#!/usr/bin/env true

View File

@@ -0,0 +1,2 @@
#!/bin/sh
../../urchin --root .testsuite/a/b/c/testcase

View File

@@ -0,0 +1,2 @@
#!/bin/sh
./.meta-assess --root

View File

@@ -0,0 +1,2 @@
#!/bin/sh
./.meta-assess -r

View File

@@ -0,0 +1,2 @@
#!/bin/sh
../../urchin --root /bin 2>&1 | grep "'/bin/.urchin'"

View File

@@ -0,0 +1,2 @@
#!/bin/sh
! ../../urchin --root .testsuite/a/not-a-file

View File

@@ -1,3 +1,5 @@
#!/bin/sh
! ../../urchin ./.chainsaw
tmp=$(mktemp -d)/blah
echo '#!/usr/bin/env true' > $tmp
! ../../urchin $tmp

4
tests/get_stdout_file/t Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/sh
I_AM_SOURCING_URCHIN_IN_A_TEST=true . ../../urchin
test $(get_stdout_file t) = aoeu

1
tests/skip/.test/fail Executable file
View File

@@ -0,0 +1 @@
exit 1

1
tests/skip/.test/skip Executable file
View File

@@ -0,0 +1 @@
exit 3

1
tests/skip/.test/succeed Executable file
View File

@@ -0,0 +1 @@
exit 0

View File

@@ -0,0 +1 @@
test $(../../urchin -t .test/ | grep -c SKIP) -eq 1

208
urchin
View File

@@ -1,17 +1,57 @@
#!/bin/sh
# Make sure that CDPATH isn't set, as it causes `cd` to behave unpredictably - notably, it can produce output,
# which breaks fullpath().
# This file is part of urchin. It is subject to the license terms in the
# COPYING file found in the top-level directory of this distribution or at
# https://raw.githubusercontent.com/tlevine/urchin/master/COPYING
# No part of urchin, including this file, may be copied, modified, propagated,
# or distributed except according to the terms contained in the COPYING file.
set +e
# Make sure that CDPATH isn't set, as it causes `cd` to behave unpredictably -
# notably, it can produce output, which breaks fullpath().
unset CDPATH
# Urchin version number
VERSION=0.0.6
fullpath() {
(
cd -- "$1"
pwd
)
urchin_root() {
# Call recursively but remember the original argument.
current="$(remove_trailing_slash "$1")"
if test -n "$2"; then
orig="$2"
else
orig="$1"
fi
if test "$(readlink -f -- "$1")" = /; then
# Stop traversing upwards at /
if test -d "$orig"; then
origdir="$orig"
else
origdir=$(dirname "$orig")
fi
echo "You need to create the .urchin directory in the root of your tests,
maybe like this:
mkdir '$(readlink -f -- "$(remove_trailing_slash "$origdir")")/.urchin'
" >&2
return 1
elif ! test -e "$current"; then
echo "$current: No such file or directory">&2
return 1
elif test -f "$current"; then
urchin_root "$(dirname "$current")" "$orig"
elif test -d "$current"/.urchin; then
remove_trailing_slash "$current"
elif test "$current" != . && test "$current" != .. &&
echo "$current" | grep '^\.' && > /dev/null; then
# Stop traversing upwards at hidden directories.
urchin_root / "$orig"
else
urchin_root "$current"/.. "$orig"
fi
}
indent() {
@@ -21,16 +61,40 @@ indent() {
fi
}
remove_trailing_slash() {
echo "$1" | sed s/\\/$//
}
escape_slashes() {
echo "$1" | sed s+/+\\\\/+g
}
get_stdout_file() {
root="$(readlink -f -- "$(urchin_root "$1")")"
test_file="$(readlink -f -- "$1")"
eroot="$(escape_slashes "$root")"
base="$(echo "$test_file" | sed "s/^$eroot/$eroot\/.urchin/")"
if test -d "$1"; then
echo "$base"/.index
else
echo "$base"
fi
}
recurse() {
potential_test="$1"
indent_level="$2"
shell_for_sh_tests="$3"
[ "$potential_test" = '.urchin' ] && return
[ "$potential_test" = 'setup_dir' ] && return
[ "$potential_test" = 'teardown_dir' ] && return
[ "$potential_test" = 'setup' ] && return
[ "$potential_test" = 'teardown' ] && return
stdout_file="$(get_stdout_file "$potential_test")"
mkdir -p "$(dirname "$stdout_file")"
[ $indent_level -eq 0 ] && : > "$stdout_file"
if [ -d "$potential_test" ]
@@ -58,8 +122,10 @@ recurse() {
[ -f setup ] && [ -x setup ] && ./setup >> "$stdout_file"
# $2 instead of $indent_level so it doesn't clash
recurse "${test}" $(( $2 + 1 )) "$shell_for_sh_tests"
set +e
recurse "${test}" "$(( $2 + 1 ))" "$shell_for_sh_tests"
exit_code=$?
set -e
if $exit_on_fail && test $exit_code -ne 0; then
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
@@ -94,6 +160,8 @@ recurse() {
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
if [ $exit_code -eq 0 ]; then
result=success
elif [ $exit_code -eq 3 ]; then
result=skip
else
result=fail
fi
@@ -133,7 +201,10 @@ recurse() {
# On fail, print a red '✗'
printf '\033[31m✗ \033[0m'
printf '%s\n' "${potential_test}"
printf '\033[31m' # Print output captured from failed test in red.
# Print output captured from failed test in red.
printf '\033[31m'
cat "$stdout_file"
printf '\033[0m'
;;
@@ -147,12 +218,16 @@ recurse() {
return 1
fi
fi
[ $indent_level -eq 0 ] && rm "$stdout_file"
}
has_sh_or_no_shebang_line() {
head -n 1 "$1" | grep -vqE '^#!' && return 0 # no shebang line at all
head -n 1 "$1" | grep -qE '^#![[:blank:]]*/bin/sh($|[[:blank:]])' && return 0 # shebang line is '#!/bin/sh' or legal variations thereof
# no shebang line at all
head -n 1 "$1" | grep -vqE '^#!' && return 0
# shebang line is '#!/bin/sh' or legal variations thereof
head -n 1 "$1" | grep -qE '^#![[:blank:]]*/bin/sh($|[[:blank:]])' &&
return 0
return 1
}
@@ -172,6 +247,7 @@ $USAGE
contain the word "test".
-t Format output in Test Anything Protocol (TAP)
-h, --help This help.
-r, --root Print the Urchin root for a particular file or directory.
-v Display the version number.
Go to https://github.com/tlevine/urchin for documentation on writing tests.
@@ -192,12 +268,11 @@ plural () {
}
urchin_go() {
rm -f "$logfile"
if "$tap_format"; then
printf \#\
fi
echo Running tests at $(date +%Y-%m-%dT%H:%M:%S)
start=$(date +%s)
start="$(date +%s)"
# Determine the environment variable to define for test scripts
# that reflects the specified or implied shell to use for shell-code tests.
@@ -212,7 +287,10 @@ urchin_go() {
TEST_SHELL='/bin/sh'
fi
recurse "$1" 0 "$2" # test folder -- indentation level -- [shell to invoke test scripts with]
# 1 test folder
# 2 indentation level
# 3 shell to invoke test scripts with
recurse "$1" 0 "$2"
finish=$(date +%s)
elapsed=$(($finish - $start))
@@ -227,11 +305,14 @@ urchin_go() {
echo "Done, took $elapsed $(plural second $elapsed)."
printf '%s\n' "$passed $(plural test "$passed") passed."
printf '%s\n' "$skipped $(plural test "$skipped") skipped."
[ $failed -gt 0 ] && printf '\033[31m' || printf '\033[32m' # If tests failed, print the message in red, otherwise in green.
# If tests failed, print the message in red, otherwise in green.
[ $failed -gt 0 ] && printf '\033[31m' || printf '\033[32m'
printf '%s\n' "$failed $(plural test "$failed") failed."
printf '\033[m'
fi
rm -f "$logfile"
# rm -f "$logfile"
test -z "$failed" || test "$failed" -eq '0'
}
@@ -247,48 +328,57 @@ urchin_molly_guard() {
exit 1
}
shell_for_sh_tests=
force=false
exit_on_fail=false
tap_format=false
while [ $# -gt 0 ]
do
case "$1" in
-e) exit_on_fail=true;;
-f) force=true;;
-s)
shift
shell_for_sh_tests=$1
which "$shell_for_sh_tests" >/dev/null || { echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2; urchin_help >&2; exit 2; }
;;
-t) tap_format=true;;
-h|--help) urchin_help
exit 0;;
-v) echo "$VERSION"
if test -z "$I_AM_SOURCING_URCHIN_IN_A_TEST"; then
shell_for_sh_tests=
force=false
exit_on_fail=false
tap_format=false
while [ $# -gt 0 ]
do
case "$1" in
-e) exit_on_fail=true;;
-f) force=true;;
-r|--root)
shift
urchin_root "$1"
exit;;
-*) urchin_help >&2
exit 1;;
*) break;;
esac
shift
done
-s)
shift
shell_for_sh_tests=$1
which "$shell_for_sh_tests" > /dev/null || {
echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2
urchin_help >&2
exit 11
} ;;
-t) tap_format=true;;
-h|--help) urchin_help
exit 0;;
-v) echo "$VERSION"
exit;;
-*) urchin_help >&2
exit 1;;
*) break;;
esac
shift
done
# Verify argument for main stuff
if [ "$#" != '1' ] || [ ! -d "$1" ]
then
[ -n "$1" ] && [ ! -d "$1" ] && echo "Not a directory: '$1'" >&2
echo "$USAGE" >&2
exit 2
fi
# Constants
logfile=$(fullpath "$1")/.urchin.log
stdout_file=$(fullpath "$1")/.urchin_stdout
# Run or present the Molly guard.
if basename "$(fullpath "$1")" | grep -Fi 'test' > /dev/null || $force
then
urchin_go "$1" "$shell_for_sh_tests"
else
urchin_molly_guard
# Verify argument for main stuff
if [ "$#" != '1' ] || [ ! -d "$1" ]
then
[ -n "$1" ] && [ ! -d "$1" ] && echo "Not a directory: '$1'" >&2
echo "$USAGE" >&2
exit 11
fi
# Run or present the Molly guard.
if ! urchin_root "$1" > /dev/null; then
exit 1
elif basename "$(readlink -f -- "$(urchin_root "$1")")" |
grep -Fi 'test' > /dev/null || $force; then
logfile="$(readlink -f -- "$(urchin_root "$1")/.urchin/.log")"
printf '' > "$logfile"
urchin_go "$1" "$shell_for_sh_tests"
else
urchin_molly_guard
fi
fi