1 Commits

Author SHA1 Message Date
Thomas Levine
7126872a8b put mksh in the cross-shell-tests 2016-01-29 19:45:39 +00:00
26 changed files with 44 additions and 260 deletions

41
HISTORY
View File

@@ -1,44 +1,6 @@
HISTORY
=======
Version 0.0.7
---------------------
### Molly-guard
The Molly-guard is now more accepting. For example, you no longer need to
pass -f in this case: https://github.com/creationix/nvm/issues/357
### Skipping tests
Previously, tests were run if they were executable and were otherwise marked
as skipped. Now, an executable script can indicate that it is skipped by
exiting with code 3. For example, if a test requires some dependancy, it
might look for the dependency and then skip if it does not see the dependency.
It might look like this.
#!/bin/sh
if which inkscape; then
exit 3 # status code 3 for skip
fi
inkscape blah blah ...
I chose status code 3 sort of arbitrarily at first, but it turns out that it
would the appropriate status code if these tests were Nagios plugins, as the
concept of skipping a test is similar to the Nagios concept of unknown service
status (https://nagios-plugins.org/doc/guidelines.html#AEN78).
### Run on a file
It is now possible to run urchin on a single file.
This occurred to me when I wanted to run
urchin test/fast/Unit\ tests/nvm_ls_current
on the nvm tests. I wound up running this instead.
urchin test/fast/Unit\ tests/ | grep nvm_ls_current
The Molly guard is assessed, and the corresponding setup, setup_dir,
teardown, and teardown_dir files are run in the appropriate order.
Version 0.0.6
---------------------
@@ -52,9 +14,6 @@ Version 0.0.6
* Remove the undocumented, experimental -x flag now that shall exists.
* Display version number with the -v flag.
* Document why Urchin is called "Urchin"
* Update TODO
* Support mksh (Change a printf command.)
* Make long lines shorter.
These changes are made somewhat separately in the branches "exit-on-fail",
"remove-urchin-x", "tap", and "update-readme". They are rebased into one

View File

@@ -1,5 +1,3 @@
Copyright (c) 2013, 2014, 2015, 2016 Thomas Levine
Copyright (c) 2014, Michael Klement
Copyright (c) 2012, ScraperWiki Limited
All rights reserved.

136
TODO
View File

@@ -1,136 +0,0 @@
Things I want
=============
Test speed
-------------
Make tests run faster.
https://github.com/bike-barn/hermit/issues/62
First, easier thing is probably to run tests in parallel.
Second, also easier thing is to tell people to save things to RAM rather than
disk whenever they can.
Third, harder thing is to put the test suite in RAM automatically. Maybe the
whole test directory, which includes fixtures, gets copied to a tmpfs if one
exists.
Hmm or maybe there's a compromise: Tell people to mount /tmp as a tmpfs so
that temp files are fast. Maybe allow people to set some other directory as
the temporary file place, in case they want a different tmpfs location.
Options
-------------
I want long options. For example, there's presently -f and -e.
I want to make them -f|--force and -e|--exit.
Environment variables
-------------
Do something to make it easier to debug environment variables, because that is
often confusing.
https://github.com/creationix/nvm/issues/719
https://github.com/creationix/nvm/issues/589
Documenting that people should run "env" when their tests fail might be good
enough.
Licensing and copyright
------------------------
* Reference all owners and years in the Copyright file
* Consider copyleft licenses
* Add license notices to other files if necessary
Packaging
------------
Package for package managers.
* I want NixOS, of course.
* Debian is probably the big one.
Other interesting package managers
* Update the npm package
* Homebrew (for Mac)
Windows
----------
Try running Urchin in Windows somehow. Interpreters include
* CygWin (https://www.cygwin.com/)
* MSYS (http://mingw.org/wiki/msys)
* GNU on Windows (https://github.com/bmatzelle/gow/wiki)
* Git for Windows (https://git-scm.com/download/win)
* win-bash (http://win-bash.sourceforge.net/)
Consider copyleft licenses
----------
ScraperWiki owns the original version of Urchin (Thomas Levine did the early
work as part of his work for ScraperWiki.) and originally licensed it under an
MIT-style license. Other people made changes after this original ScraperWiki
version. As of January 2016, they are just Thomas Levine (when he wasn't
working for ScraperWiki) and Michael Klement.
The original license was MIT just because that's what ScraperWiki put on
everything. Should we change the license?
The MIT-style license grants pretty much all rights. It says that you need
to attribute when you redistribute source code, but you don't
necessarily have to redistribute source code.
A copyleft license adds the restriction that modified versions of the
code need to be licensed under the same license. GNU licenses in
particular require that source code be released if non-source versions are
released, and the different GNU licenses differ in what how the
non-source version is defined. (The original, GPL, discusses compiled
binaries.) Copyleft doesn't mean anything specific for commercial use.
MIT-licensed code can be modified and then licensed as GPL, because MIT
license allows that, but GPL code can't be modified as MIT, because MIT
doesn't allow that. And if we get all of the authors to agree on it, we
can always add whatever crazy license we want, regardless of what we
have already.
The distinction between MIT-style and GNU-something might matter quite little
in the case of Urchin.
1. Urchin is written in an interpreted language (shell), so it might be
hard to distribute usefully without providing the source code.
2. Urchin just runs tests; it doesn't get compiled with the rest of the
code (also because it's in shell). Thus, I think a GPL license on
Urchin wouldn't infect the code being tested.
This is as far as I have gotten with contemplating license changes. For now
we're sticking with the original MIT-style license, but it's easy to change
licenses later.
Nagios plugins
-----------------
It would be cool to run Nagios plugins with Urchin. This is already possible,
actually, but it might be worth giving some special thought to it.
https://nagios-plugins.org/doc/guidelines.html
Source setup and teardown
--------------------
If setup and teardown are sourced instead of executed, maybe we can more
cleanly create and teardown temporary files.
(
. ./setup
./$thetestfile
. ./teardown
)
On the other hand, this could just be sourced explicitly in the test file,
without the special setup and teardown feature.
Running automated tasks
-------------------------
Urchin might be appropriate for if you have lots of tasks that you want to run
periodically; add an urchin call to your crontab, and call all of your other
tasks with urchin. Here are some features that might make urchin better for
this sort of thing.
* Time how long each test/job takes
* Optionally kill tests/jobs after a specific timeout threshold
* Send output of different tests/jobs to different files for each file
descriptor (STDOUT, STDERR)

View File

@@ -1,9 +1,2 @@
Totally different syntax and similar features, plus TAP output
https://github.com/sstephenson/bats
Relatively similar interface
https://github.com/mlafeldt/sharness
Lists of alternatives
https://thomaslevine.com/!/shell-testing/
https://github.com/mlafeldt/sharness#alternatives

View File

@@ -36,7 +36,7 @@ Urchin is contained in a single file, so you can install it by copying it to a
directory in your `PATH`. For example, you can run the following as root.
cd /usr/local/bin
wget https://raw.githubusercontent.com/tlevine/urchin/v0.0.6/urchin
wget https://raw.github.com/tlevine/urchin/master/urchin
chmod +x urchin
Urchin can be installed with npm too.
@@ -169,3 +169,9 @@ It does something similar, but the interface may be more intuitive.
## Alternatives to Urchin
Alternatives to Urchin are discussed in
[this blog post](https://blog.scraperwiki.com/2012/12/how-to-test-shell-scripts/).
## Ideas for new features
* Support [Nagios plugins](https://nagios-plugins.org/doc/guidelines.html)
* Stop running if a test fails so one can use Urchin as a
[setup framework](https://github.com/tlevine/urchin/issues/16).

View File

@@ -1 +0,0 @@
echo setup

View File

@@ -1 +0,0 @@
echo setup_dir

View File

@@ -1 +0,0 @@
echo teardown

View File

@@ -1 +0,0 @@
echo teardown_dir

View File

@@ -1,2 +0,0 @@
echo thetest
exit 1

View File

@@ -1,3 +0,0 @@
#!/bin/sh
../../urchin .test
# This will exit 0 if it worked.

View File

@@ -1,2 +0,0 @@
#!/bin/sh
! ../../urchin not-a-file

View File

@@ -1,4 +0,0 @@
tmp=$(mktemp)
echo "Using temp file $tmp"
../../urchin .wrappers > $tmp
diff $tmp .wrapper-expectation

View File

@@ -1,5 +1,3 @@
#!/bin/sh
tmp=$(mktemp -d)/blah
echo '#!/usr/bin/env true' > $tmp
! ../../urchin $tmp
! ../../urchin ./.chainsaw

View File

@@ -1 +0,0 @@
exit 1

View File

@@ -1 +0,0 @@
exit 3

View File

@@ -1 +0,0 @@
exit 0

View File

@@ -1 +0,0 @@
test $(../../urchin -t .test/ | grep -c SKIP) -eq 1

76
urchin
View File

@@ -1,26 +1,22 @@
#!/bin/sh
# This file is part of urchin. It is subject to the license terms in the
# COPYING file found in the top-level directory of this distribution or at
# https://raw.githubusercontent.com/tlevine/urchin/master/COPYING
# No part of urchin, including this file, may be copied, modified, propagated,
# or distributed except according to the terms contained in the COPYING file.
# Make sure that CDPATH isn't set, as it causes `cd` to behave unpredictably -
# notably, it can produce output, which breaks fullpath().
# Make sure that CDPATH isn't set, as it causes `cd` to behave unpredictably - notably, it can produce output,
# which breaks fullpath().
unset CDPATH
# Urchin version number
VERSION=0.0.6
fullpath() {
(
cd -- "$1"
pwd
)
}
indent() {
level="$1"
if test "$level" -gt 0; then
printf "%$((2 * ${level}))s"
fi
printf "%$((2 * ${level}))s"
}
recurse() {
@@ -96,8 +92,6 @@ recurse() {
[ -f teardown ] && [ -x teardown ] && ./teardown >> "$stdout_file"
if [ $exit_code -eq 0 ]; then
result=success
elif [ $exit_code -eq 3 ]; then
result=skip
else
result=fail
fi
@@ -137,10 +131,7 @@ recurse() {
# On fail, print a red '✗'
printf '\033[31m✗ \033[0m'
printf '%s\n' "${potential_test}"
# Print output captured from failed test in red.
printf '\033[31m'
printf '\033[31m' # Print output captured from failed test in red.
cat "$stdout_file"
printf '\033[0m'
;;
@@ -158,13 +149,8 @@ recurse() {
}
has_sh_or_no_shebang_line() {
# no shebang line at all
head -n 1 "$1" | grep -vqE '^#!' && return 0
# shebang line is '#!/bin/sh' or legal variations thereof
head -n 1 "$1" | grep -qE '^#![[:blank:]]*/bin/sh($|[[:blank:]])' &&
return 0
head -n 1 "$1" | grep -vqE '^#!' && return 0 # no shebang line at all
head -n 1 "$1" | grep -qE '^#![[:blank:]]*/bin/sh($|[[:blank:]])' && return 0 # shebang line is '#!/bin/sh' or legal variations thereof
return 1
}
@@ -239,10 +225,7 @@ urchin_go() {
echo "Done, took $elapsed $(plural second $elapsed)."
printf '%s\n' "$passed $(plural test "$passed") passed."
printf '%s\n' "$skipped $(plural test "$skipped") skipped."
# If tests failed, print the message in red, otherwise in green.
[ $failed -gt 0 ] && printf '\033[31m' || printf '\033[32m'
[ $failed -gt 0 ] && printf '\033[31m' || printf '\033[32m' # If tests failed, print the message in red, otherwise in green.
printf '%s\n' "$failed $(plural test "$failed") failed."
printf '\033[m'
fi
@@ -274,11 +257,8 @@ do
-s)
shift
shell_for_sh_tests=$1
which "$shell_for_sh_tests" > /dev/null || {
echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2
urchin_help >&2
exit 11
} ;;
which "$shell_for_sh_tests" >/dev/null || { echo "Cannot find specified shell: '$shell_for_sh_tests'" >&2; urchin_help >&2; exit 2; }
;;
-t) tap_format=true;;
-h|--help) urchin_help
exit 0;;
@@ -291,17 +271,21 @@ do
shift
done
# Verify argument for main stuff
if [ "$#" != '1' ] || [ ! -d "$1" ]
then
[ -n "$1" ] && [ ! -d "$1" ] && echo "Not a directory: '$1'" >&2
echo "$USAGE" >&2
exit 2
fi
# Constants
logfile=$(fullpath "$1")/.urchin.log
stdout_file=$(fullpath "$1")/.urchin_stdout
# Run or present the Molly guard.
thetest=$(readlink -f "$1")
if echo "$thetest" | grep -Fi 'test' > /dev/null || $force; then
if test -d "$1"; then
d="$thetest"
else
d=$(echo "$thetest" | sed 's/\/[^\/]\{1,\}$//')
cd "$d"
fi
logfile="$d/.urchin.log"
stdout_file="$d/.urchin_stdout"
if basename "$(fullpath "$1")" | grep -Fi 'test' > /dev/null || $force
then
urchin_go "$1" "$shell_for_sh_tests"
else
urchin_molly_guard

Binary file not shown.

Before

Width:  |  Height:  |  Size: 99 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 115 KiB