0
0
mirror of https://github.com/vim/vim.git synced 2025-07-26 11:04:33 -04:00
Aliaksei Budavei 0fde6aebdd
CI: Manage multibyte characters in syntax tests
As reported in #16559, bytes of a multibyte character may
be written as separate U+FFFD characters in a ":terminal"
window on a busy machine.  The testing facilities currently
offer an optional filtering step to be carried out between
reading and comparing the contents of two screendump files
for each such file.  This filtering has been resorted to
(#14767 and #16560) in an attempt to unconditionally replace
known non-Latin-1 characters with an arbitrary substitute
ASCII character and avoid this rendering mishap leading to
syntax tests failures.  However, it has been overlooked at
the time that metadata description (in shorthand) to follow
spurious U+FFFD characters may be *distinct* and make the
remainder of such a line, ASCII characters and whatnot, also
unequal between compared screendump files.

While it is straightforward to adapt current filter files to
ignore the line characters after the leftmost U+FFFD,

> It is challenging and error-prone to keep up to date filter
> files because moving around examples in source files will
> likely make redundant some previously required filter files
> and, at the same time, it may require creating new filter
> files for the same source file; substituting one multibyte
> character for another multibyte character will also demand
> a coordinated change for filter files.

Besides, unconditionally dropping arbitrary parts of a line
is rather too blunt an instrument.  An alternative approach
is to not use the supported filtering for this purpose; let
a syntax test pass or fail initially; then *if* the same
failure is imminent, drop the leftmost U+FFFD and the rest
of the previously seen line (repeating it for all previously
seen unequal lines) before another round of file contents
comparing.  The obvious disadvantage with this filtering,
unconditional and otherwise, is that if there are consistent
failures for _other reasons_ and the unequal parts happen to
be after U+FFFDs, then spurious test passing can happen when
stars align for _a particular test runner_.

Hence syntax test authors should strive to write as little
significant text after multibyte characters as syntactically
permissible, write multibyte characters closer to EOL in
general, and make sure that their checked-in and published
"*.dump" files do not have any U+FFFDs.

It is also practical to refrain from attempting screendump
generation if U+FFFDs can already be discovered, and instead
try re-running from scratch the syntax test in hand, while
accepting other recently generated screendumps without going
through with new rounds of verification.

Reference:
https://github.com/vim/vim/pull/16470#issuecomment-2599848525

closes: #17704

Signed-off-by: Aliaksei Budavei <0x000c70@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>
2025-07-25 20:08:52 +02:00
..
2017-03-05 17:04:09 +01:00
2023-08-23 21:20:00 +02:00
2023-10-16 09:53:37 +02:00
2022-08-08 15:42:38 +01:00
2021-09-09 21:55:11 +02:00
2021-09-09 21:55:11 +02:00
2021-09-09 21:55:11 +02:00
2020-12-10 21:11:27 +01:00
2017-03-05 17:04:09 +01:00
2017-12-17 17:17:07 +01:00
2021-07-28 20:10:16 +02:00
2017-03-05 17:04:09 +01:00
2021-11-07 20:27:04 +00:00
2011-12-30 13:11:27 +01:00
2020-05-07 18:56:00 +02:00
2023-10-16 09:53:37 +02:00
2020-11-05 20:41:49 +01:00
2021-09-09 21:55:11 +02:00
2020-10-11 13:57:40 +02:00
2018-01-28 22:47:25 +01:00
2022-07-29 21:36:21 +01:00
2021-11-16 19:18:26 +00:00
2023-02-02 13:59:48 +00:00
2016-09-25 22:11:48 +02:00
2016-10-11 08:57:33 +02:00
2022-07-01 18:45:04 +01:00
2017-03-05 17:04:09 +01:00
2017-11-02 22:58:42 +01:00
2022-07-25 15:42:07 +01:00
2020-03-19 20:37:11 +01:00
2015-10-13 23:21:27 +02:00
2022-11-09 21:21:04 +00:00
2020-11-29 14:36:24 +01:00
2020-11-29 14:36:24 +01:00
2017-03-05 17:04:09 +01:00
2010-01-06 20:54:52 +01:00
2016-09-22 22:33:02 +02:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2017-11-30 20:27:52 +01:00
2022-09-27 17:30:34 +01:00
2023-10-16 09:53:37 +02:00
2022-03-30 10:51:39 +01:00
2018-01-28 22:47:25 +01:00
2017-03-05 17:04:09 +01:00
2019-03-24 15:09:13 +01:00
2022-06-17 15:42:40 +01:00
2008-06-24 21:16:56 +00:00
2018-02-09 22:00:53 +01:00
2017-03-05 17:04:09 +01:00
2025-03-15 09:50:41 +01:00
2021-09-09 21:55:11 +02:00
2013-03-07 13:20:54 +01:00
2024-03-04 16:19:48 +01:00
2021-09-09 21:55:11 +02:00
2019-12-19 22:12:03 +01:00
2017-03-05 17:04:09 +01:00
2016-02-04 20:57:07 +01:00
2019-10-26 19:53:45 +02:00
2017-03-05 17:04:09 +01:00
2023-02-02 13:59:48 +00:00
2022-09-27 17:30:34 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2018-02-27 14:53:30 +01:00
2022-06-17 15:42:40 +01:00
2021-12-05 21:54:04 +00:00
2021-03-31 20:07:33 +02:00
2022-06-26 12:21:15 +01:00
2010-01-06 20:54:52 +01:00
2021-12-24 13:18:38 +00:00
2023-05-14 18:50:25 +01:00
2020-10-26 21:12:46 +01:00
2017-03-05 17:04:09 +01:00
2020-06-07 21:07:18 +02:00
2022-03-30 10:51:39 +01:00
2017-03-05 17:04:09 +01:00
2022-07-06 13:18:11 +01:00
2023-02-20 20:44:55 +00:00
2021-09-09 21:55:11 +02:00
2017-09-19 22:06:03 +02:00
2021-05-02 17:19:11 +02:00
2017-03-05 17:04:09 +01:00
2023-08-23 21:20:00 +02:00
2022-09-04 17:45:43 +01:00
2022-09-18 13:04:22 +01:00
2022-08-30 20:26:34 +01:00
2021-08-29 21:55:35 +02:00
2021-01-02 16:39:53 +01:00
2023-12-28 23:17:54 +01:00
2022-09-10 13:13:14 +01:00
2019-12-05 22:47:25 +01:00
2022-09-10 13:13:14 +01:00
2017-11-02 22:58:42 +01:00
2022-01-08 21:51:59 +00:00
2021-11-16 19:18:26 +00:00
2021-09-21 20:09:51 +02:00
2021-03-31 20:07:33 +02:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2021-06-13 19:02:49 +02:00
2022-09-27 17:30:34 +01:00
2019-12-05 22:47:25 +01:00
2021-09-09 21:55:11 +02:00
2024-10-16 22:11:17 +02:00
2022-12-31 15:30:45 +00:00
2022-09-27 17:30:34 +01:00
2019-09-27 19:34:08 +02:00
2017-03-05 17:04:09 +01:00
2020-10-11 13:57:40 +02:00
2007-05-05 17:54:07 +00:00
2022-06-17 15:42:40 +01:00
2021-09-09 21:55:11 +02:00
2017-03-05 17:04:09 +01:00
2022-06-17 15:42:40 +01:00
2019-11-20 22:31:13 +01:00
2022-06-17 15:42:40 +01:00
2023-08-13 10:25:41 +02:00
2019-11-17 15:55:14 +01:00
2020-03-19 20:37:11 +01:00
2019-11-17 15:55:14 +01:00
2011-06-19 05:09:16 +02:00
2021-07-07 20:26:08 +02:00
2021-08-14 21:25:52 +02:00
2023-04-22 22:40:14 +01:00
2017-03-05 17:04:09 +01:00
2020-10-11 13:57:40 +02:00
2017-03-05 17:04:09 +01:00
2021-02-27 16:38:07 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2022-03-19 15:18:53 +00:00
2023-08-23 21:20:00 +02:00
2017-03-05 17:04:09 +01:00
2023-04-22 22:40:14 +01:00
2017-03-05 17:04:09 +01:00
2020-10-26 21:12:46 +01:00
2023-08-23 21:20:00 +02:00
2023-05-14 18:50:25 +01:00
2021-04-17 16:31:09 +02:00
2021-04-17 16:31:09 +02:00
2022-06-12 22:15:57 +01:00
2024-01-25 19:52:47 +01:00
2017-03-05 17:04:09 +01:00
2013-08-03 18:35:36 +02:00
2013-02-13 16:15:55 +01:00
2017-03-05 17:04:09 +01:00
2022-10-16 13:11:12 +01:00
2022-06-14 17:41:28 +01:00
2019-06-10 22:47:40 +02:00
2023-06-10 21:40:39 +01:00
2021-02-13 18:14:48 +01:00
2022-08-15 18:51:32 +01:00
2010-05-14 23:24:24 +02:00
2017-12-24 13:22:00 +01:00
2020-10-26 21:12:46 +01:00
2017-03-05 17:04:09 +01:00
2022-11-09 21:21:04 +00:00
2020-10-26 21:12:46 +01:00
2022-08-15 18:51:32 +01:00
2015-01-25 13:02:57 +01:00
2012-02-04 21:59:01 +01:00
2022-06-17 15:42:40 +01:00
2019-08-31 16:20:32 +02:00
2024-07-21 09:27:27 +02:00
2017-08-27 16:52:01 +02:00
2019-10-16 22:35:02 +02:00
2017-03-05 17:04:09 +01:00
2023-04-22 22:40:14 +01:00
2018-04-06 11:14:11 +02:00
2023-10-16 09:53:37 +02:00
2022-11-18 21:49:57 +00:00
2021-09-09 21:55:11 +02:00
2022-11-23 23:46:31 +00:00
2022-04-18 15:36:40 +01:00
2022-10-28 20:47:54 +01:00
2023-10-16 09:53:37 +02:00
2020-05-12 22:49:12 +02:00
2021-07-28 20:10:16 +02:00
2017-03-05 17:04:09 +01:00
2020-11-21 13:16:30 +01:00
2008-06-24 20:39:31 +00:00
2022-06-17 15:42:40 +01:00
2018-07-21 05:56:22 +02:00
2021-09-21 20:09:51 +02:00
2008-06-24 22:58:06 +00:00
2018-01-28 22:47:25 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2022-06-17 15:42:40 +01:00
2023-02-20 20:44:55 +00:00
2021-03-13 13:28:13 +01:00
2023-10-16 09:53:37 +02:00
2007-05-05 17:54:07 +00:00
2021-01-11 19:40:15 +01:00
2020-11-05 20:41:49 +01:00
2017-03-05 17:04:09 +01:00
2021-04-07 21:07:20 +02:00
2021-04-07 21:07:20 +02:00
2021-04-17 16:31:09 +02:00
2022-11-18 21:49:57 +00:00
2022-01-29 22:20:48 +00:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2021-04-21 18:09:37 +02:00
2018-11-04 14:45:49 +01:00
2021-04-17 16:31:09 +02:00
2022-01-23 12:07:04 +00:00
2018-07-29 15:07:52 +02:00
2021-08-14 21:25:52 +02:00
2022-12-06 18:20:10 +00:00
2020-03-14 08:19:51 +01:00
2017-03-05 17:04:09 +01:00
2013-04-05 22:26:15 +02:00
2021-04-21 18:09:37 +02:00
2020-10-26 21:12:46 +01:00
2021-11-16 19:18:26 +00:00
2020-10-26 21:12:46 +01:00
2018-08-28 22:58:02 +02:00
2020-04-10 22:10:56 +02:00
2020-10-26 21:12:46 +01:00
2017-08-27 16:52:01 +02:00
2022-03-19 15:18:53 +00:00
2017-05-27 16:45:17 +02:00
2015-09-25 20:34:21 +02:00
2019-12-05 22:47:25 +01:00
2022-10-16 13:11:12 +01:00
2022-06-17 15:42:40 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2021-01-02 16:39:53 +01:00
2023-08-23 21:20:00 +02:00
2017-03-05 17:04:09 +01:00
2020-12-23 13:56:35 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2023-08-23 21:20:00 +02:00
2022-04-08 17:45:08 +01:00
2018-01-28 22:47:25 +01:00
2023-10-16 09:53:37 +02:00
2023-08-23 21:20:00 +02:00
2018-06-24 20:42:01 +02:00
2023-08-23 21:20:00 +02:00
2022-09-18 13:04:22 +01:00
2021-09-09 21:55:11 +02:00
2022-01-29 22:20:48 +00:00
2010-05-14 23:24:24 +02:00
2020-12-23 13:56:35 +01:00
2017-03-05 17:04:09 +01:00
2015-10-13 23:21:27 +02:00
2022-06-14 17:41:28 +01:00
2022-06-17 15:42:40 +01:00
2023-08-23 21:20:00 +02:00
2021-11-16 19:18:26 +00:00
2021-10-16 15:23:36 +01:00
2018-09-02 21:07:30 +02:00
2017-03-05 17:04:09 +01:00
2021-12-16 14:41:10 +00:00
2023-08-23 21:20:00 +02:00
2021-01-25 21:14:57 +01:00
2020-10-26 21:12:46 +01:00
2024-09-04 22:15:40 +02:00
2021-10-16 15:23:36 +01:00
2023-02-02 13:59:48 +00:00
2017-03-05 17:04:09 +01:00
2022-06-17 15:42:40 +01:00
2023-08-23 21:20:00 +02:00
2023-10-16 09:53:37 +02:00
2022-06-14 17:41:28 +01:00
2022-06-14 17:41:28 +01:00
2022-06-14 17:41:28 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2017-03-05 17:04:09 +01:00
2016-09-06 22:12:34 +02:00
2023-08-09 16:50:52 +02:00
2017-03-05 17:04:09 +01:00
2015-03-05 21:21:19 +01:00
2013-06-24 22:22:18 +02:00
2013-11-03 21:14:31 +01:00
2018-05-17 17:04:55 +02:00
2023-06-10 21:40:39 +01:00
2013-11-03 21:14:31 +01:00
2015-03-05 21:21:19 +01:00
2019-07-31 21:07:14 +02:00
2021-12-05 21:54:04 +00:00
2022-09-18 13:04:22 +01:00
2020-04-10 22:10:56 +02:00
2008-06-24 21:16:56 +00:00
2019-08-08 22:15:18 +02:00
2022-12-06 18:20:10 +00:00
2022-06-09 20:45:54 +01:00
2004-07-19 20:55:54 +00:00
2008-06-24 21:56:24 +00:00
2017-03-05 17:04:09 +01:00
2020-10-26 21:12:46 +01:00
2021-12-24 13:18:38 +00:00
2017-03-05 17:04:09 +01:00
2017-02-17 22:47:16 +01:00
2023-06-10 21:40:39 +01:00
2011-06-19 05:09:16 +02:00
2022-06-14 17:41:28 +01:00
2010-01-06 20:54:52 +01:00
2020-01-21 22:33:58 +01:00
2022-12-06 18:20:10 +00:00

This directory contains Vim scripts for syntax highlighting.

These scripts are not for a language, but are used by Vim itself:

syntax.vim	Used for the ":syntax on" command.  Uses synload.vim.

manual.vim	Used for the ":syntax manual" command.  Uses synload.vim.

synload.vim	Contains autocommands to load a language file when a certain
		file name (extension) is used.  And sets up the Syntax menu
		for the GUI.

nosyntax.vim	Used for the ":syntax off" command.  Undo the loading of
		synload.vim.

The "shared" directory contains generated files and what is used by more than
one syntax.


A few special files:

2html.vim	Converts any highlighted file to HTML (GUI only).
colortest.vim	Check for color names and actual color on screen.
hitest.vim	View the current highlight settings.
whitespace.vim  View Tabs and Spaces.


If you want to write a syntax file, read the docs at ":help usr_44.txt".

If you make a new syntax file which would be useful for others, please send it
to the vim-dev mailing list <vim-dev@vim.org>.  Include instructions for
detecting the file type for this language, by file name extension or by
checking a few lines in the file. And please write the file in a portable way,
see ":help 44.12".

If you have remarks about an existing file, send them to the maintainer of
that file.  Only when you get no response send a message to the vim-dev
mailing list: <vim-dev@vim.org>.

If you are the maintainer of a syntax file and make improvements, send the new
version to the vim-dev mailing list: <vim-dev@vim.org>

For further info see ":help syntax" in Vim.