When certain changes guarantee failure for old syntax tests,
opt for faster failure by reducing the number of screendumps
made for each file "page" to be no greater than the assigned
value of a VIM_SYNTAX_TEST_WAIT_TIME environment variable.
(This variable will be ignored and more screendumps may be
made when Make is GNU Make and a parent Makefile is used.)
Barring regressions, and assuming that v9.1.1163~1 succeeds
in providing a correct synchronisation mechanism outside of
"VerifyScreenDump()", and assuming that "readfile()" always
obtains the latest contents written by "term_dumpwrite()" in
"VerifyScreenDump()"; making a single screendump of a file
"page" and following it with a single reading of the written
screendump file should be enough to decide whether to pass
or fail a syntax test.
In addition, re-enable self testing after v9.1.1183~2.
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Doug Kearns <dougkearns@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
* Accommodate the calling of "EraseLineAndReturnCarriage()"
to not interfere with the "skipped" and "failed" reports.
* Create the "failed" directory, if unavailable, without
relying on "VerifyScreenDump()" to do it so that reporting
with "Xtestscript#s:AssertCursorForwardProgress()" can be
uniformly attempted.
* Make an only list copy of the "Xtestscript" contents and
share it with every syntax test.
* Narrow the scope of the "filetype" and "failed_root" local
variables.
closes: #16789
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
As of patch v9.1.1176~1, there are no longer makeshift Make
targets to accommodate language names that can match phoney
targets. For example, "clean_" was previously generated for
Clean because otherwise it clashed with the "clean" target.
Additionally, enable test filtering for makeshift targets.
Reference:
https://wiki.clean.cs.ru.nl/Cleancloses: #16810
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
The current lightweight synchronisation with ":redraw" needs further
reinforcement in the light of v9.1.1110. And, with v9.1.0820, make
another synchronisation point _before_ the first (or only) screenful is
dumped.
Also add a script to regenerate all screendumps.
closes: #16632
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tiny vim fails because of line-continuation, although
the test script should be skipped for non-eval builds of Vim.
Solution: :set cpo&vim
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: tests: cannot run single syntax tests
Solution: Support running a subset of syntax tests
(Aliaksei Budavei)
Two methods of assembling a subset of test filenames for
selection are provided:
* Filename and filetype Make targets will be generated, and
multiple such targets can be passed before the mandated
trailing "test" target, e.g. "make html markdown test".
* Filenames and their parts can be specified as a regular
expression that is assigned to a "VIM_SYNTAX_TEST_FILTER"
environment variable, and used with the test Make target,
e.g. "VIM_SYNTAX_TEST_FILTER=html\\\|markdown make test".
(This variable will be ignored and the whole suite will be
run when Make is GNU Make and a parent Makefile is used.)
Methods can be used alone or together, with the Make targets
having the higher precedence. Neither method will influence
the order of test execution.
closes: #15670
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
With the submitted "viewdumps.vim" script, a few manual
steps in typical workflows (see below) can be automated.
The updated "README.txt" contains additional information.
============================================================
Reviewing LOCAL failed syntax tests can be arranged as
follows:
1) Run tests and generate screendumps:
------------------------------------------------------------
cd /path/to/fork/runtime/syntax
make clean test
------------------------------------------------------------
2) Examine the screendumps from the "failed" directory:
------------------------------------------------------------
../../src/vim --clean -S testdir/viewdumps.vim
------------------------------------------------------------
============================================================
Reviewing UPLOADED failed syntax tests can be arranged as
follows (it can be further locally scripted):
1) Fetch an artifact with failed screendumps from
"github.com/vim/vim/actions/runs/A_ID/artifacts/B_ID".
2) Extract the archived files:
------------------------------------------------------------
unzip /tmp/failed-tests.zip -d /tmp
------------------------------------------------------------
3) Set up the "dumps" directory. Create a symlink to
"/path/to/fork/dirs/dumps" in the extracted directories so
that term_dumpdiff() can be used. (The lookup algorithm
resolves "dumps" for every loaded filename. So, with
"/tmp/runtime/syntax/testdir/failed/*.dump" files passed
as script arguments, the algorithm will make the files in
"/tmp/runtime/syntax/testdir/dumps" queried.)
------------------------------------------------------------
cd /path/to/fork
ln -s $(pwd)/runtime/syntax/testdir/dumps \
/tmp/runtime/syntax/testdir/dumps
------------------------------------------------------------
4) Examine the extracted screendumps:
------------------------------------------------------------
./src/vim --clean -S runtime/syntax/testdir/viewdumps.vim \
/tmp/runtime/syntax/testdir/failed/*.dump
------------------------------------------------------------
5) Clean up:
------------------------------------------------------------
unlink /tmp/runtime/syntax/testdir/dumps
rm -rf /tmp/runtime
------------------------------------------------------------
============================================================
Reviewing SUBMITTED FOR PULL REQUEST syntax tests can be
arranged as follows (it can be further locally scripted):
1) List the fetched changeset and write the changed "dumps"
filenames to "/tmp/filelist":
------------------------------------------------------------
cd /path/to/fork
git switch prs/1234
git diff-index --relative=runtime/syntax/testdir/dumps/ \
--name-only prs/1234~1 > /tmp/filelist
------------------------------------------------------------
2) Reconcile relative filepaths, and copy next-to-be-updated
"dumps" files in the "failed" directory (note the missing
new screendumps, if any):
------------------------------------------------------------
git switch master
cd runtime/syntax/testdir/dumps
cp -t ../failed $(cat /tmp/filelist)
------------------------------------------------------------
3) Remember about the introduced INVERTED relation between
"dumps" and "failed", i.e. the files to be committed are in
"dumps" already and their previous versions are in "failed";
therefore, copy the missing new screendumps from "dumps" to
"failed" (otherwise these won't be shown):
------------------------------------------------------------
git switch prs/1234
cp -t ../failed foo_10.dump foo_11.dump foo_12.dump
------------------------------------------------------------
4) Examine the screendumps from the "failed" directory (new
screendumps will be shown with no difference between their
versions):
------------------------------------------------------------
cd ..
../../../src/vim --clean -S viewdumps.vim
------------------------------------------------------------
closes: #15476
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
- Remove the code handling their generation.
- Remove the code written for the shell filetype to clear up
the buffer screen before such a file can be generated.
- Remove all but "yaml.yaml_99.dump" of such generated files
(since there is no published "input/yaml.yaml.yaml", let
its maintainer regenerate screen dump files and remove
redundant ones).
The new algorithm turns a "*_99.dump" file obsolete because
all lines written in such a file will already be present in
lower-numbered "*.dump" files.
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Track the cursor forward progress through a syntax test file
so that any degenerate input can be checked and reported.
closes: #15150
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Write a batch of test files with made-up syntax to serve for
additional linewise checks to be manually performed whenever
the algorithm for screen dump file generation is modified.
Define a shell variable VIM_SYNTAX_SELF_TESTING to run these
tests:
cd runtime/syntax/
VIM_SYNTAX_SELF_TESTING=1 make clean test
related: #15150
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
The current contents of sh_01_99.dump and sh_09_99.dump
begin with ":redraw" in the ruler line whereas both the
comment and the need for the redraw command execution imply
blanks " ".
related: #15150
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
The current implementation falls short for syntax test files
on two accounts:
1. With folded lines -- some lines before folded lines are
unnecessarily repeated in generated dump files because
closed folded lines are always treated as opened for the
cursor to move _in_ instead of to move _over_ them.
2. With wrapped lines (longer than 75 columns) -- some lines
are omitted in generated dump files because calculations
for the cursor progress and its movement commands only
refer to file lines and not their layout within a 20x75
buffer (less &cmdheight).
As an alternative, we abandon deterministic (and inaccurate
at times) calculations for the cursor progress and, instead,
advance the cursor by as much as before for a single dump
file, but now rely on marking the last visible line and
additional movement to position lines at desired offsets,
carefully preserving compatibility for the &scrolloff and
&ruler values inherited from defaults.vim. The parent Vim
process will keep track of progress through a syntax test
file made by its child process ("terminal") by reading the
rightmost end of the ruler line from the terminal buffer,
looking for " All " or " Bot " for its cue to finish dump
file generation.
With these changes applied, the lossless line length limit
will be raised from 75 to 1425 (for a 19x75 view) columns.
Also, prefer "lastline" to "truncate" for &display; hiding
the content of any last _long_ line in a view goes against
the purpose of syntax file testing -- all lines should be
recorded.
related: #15150fixes: #14245
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Also:
- Include a syntax test for Markdown that takes advantage of
a sourceable setup configuration.
- Update Filelist
closes: #14215
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Not all programming languages support comments; without such
support, the TEST_SETUP functionality that offers embeddable
Vim Ex commands may no longer be applicable.
We can achieve similar functionality by storing Ex commands
in a Vim file with its basename matching the basename of the
test file, and having this Vim file sourced, and then having
the test file loaded and tested. When such a Vim file would
be used for a language that has comments and whose matching
test file has embedded TEST_SETUP lines, we will accommodate
it by letting the TEST_SETUP lines augment and/or overrule
sourced configuration.
Further details can be found in the discussion thread at
https://github.com/vim/vim/discussions/14117.
related: #14215
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Currently, the very same syntax file for which a test can
be written is the only place where global variables can be
defined so that the file parts guarded with such variables
can be read during screen dump generation. This approach
would lead to littering the syntax file with test-related
queries.
Instead, we could borrow the idea of comment-based mechanism
for test setup from the indent test runner. With it, the
first 20 lines of each test file would be ALWAYS scanned in
search of the TEST_SETUP markers and, when found, the part
between the end of the marker and the end of the line would
be treated as a Vim Ex command.
Note that with these changes, runtime/defaults.vim is no
longer sourced for screen dump generation; however, some of
its functionality is reintroduced.
Further details can be found in the discussion thread at
https://github.com/vim/vim/discussions/14117.
related: #14215
Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
The whole testdir/runtest.vim is a bit of a mess. So this is an attempt
to improving the current state. First of all, let's move most of the
code into a single RunTest() function so that it's easier to run this
manually.
Then, let's add the possibility to customize filetype specific tests by
applying filetype specific options. We will make use of it in the next
commit, where the provided java file uses tabs, which cause line
wrapping which will then cause the scrolling to miss some lines when
redrawing (see #14105)
related: #14105
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: Syntax test fails when run with non C locale
Solution: Run syntax tests with C locale, clean up Xtestscript file,
strip environment variables from GetVimCommand()
(h-east)
closes: #14007
Co-authored-by: h-east <h.east.727@gmail.com>
Signed-off-by: h-east <h.east.727@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
Problem: runtime tests fail with tiny vim
Solution: check for tiny vim, run runtime tests in CI
even for tiny version
closes: #13169closes: #13170
Signed-off-by: Christian Brabandt <cb@256bit.org>
Co-authored-by: Dominique Pellé <dominique.pelle@tomtom.com>
Problem: Syntax test fails when detected shell type differs.
Solution: Avoid using "/bin/sh", it depends on the system. Add a check that
the shell type detection is correct.
Problem: Insufficient testing for syntax plugins.
Solution: Add shell file examples. (Charles Campbell) Create a messages
file for easier debugging and reporting the test results.
