I am not hooking these to "make test", for two reasons:
1. utf8_step_forward is inside #ifdef CONFIG_UTF8 and I don't see
how to make tests conditional on such options.
2. test/libtest.sh was copied from Git, which is under GPLv2-only.
Adding more dependencies on it could make ELinks more difficult
to relicense under GPLv2-or-later.
This check used to be in src/elinks.h. Move it to configure.in so
that (1) the result can be logged and (2) ELinks won't even link with
TRE if wchar_t prevents its use.
Also, rename HAVE_TRE_REGEX_H to CONFIG_TRE, to reflect that it is not
always defined if the header exists.
C99 6.7.4p3 and 6.7.4p6 set some constraints on what can be done in
inline functions and how they can be declared. In particular, any
function declared inline must also be defined in the same translation
unit. To comply with that, remove inline specifiers from function
declarations in header files when the functions are not also defined
in those header files.
Sun Studio 11 on Solaris 9 is stricter than C99 and does not allow
references to static identifiers in extern inline functions. Make the
configure script detect this and define NONSTATIC_INLINE accordingly
in config.h. Then use that in the definitions of all non-static
inline functions.
Document the restrictions and this scheme in doc/hacking.txt.
When the file is being read, Expat provides the strings to ELinks in
UTF-8, so ELinks can put them in struct bookmark without conversions.
Make sure gettext returns any placeholder strings in UTF-8, too.
Replace '\r' with ' ' in bookmark titles and URLs.
When the file is being written, put encoding="UTF-8" in the XML
declaration, and then write out the strings from struct bookmark
without character set conversions. Do replace some characters
with entity references though, by calling add_html_to_string().
This simplifies the callers a little and may help implement
simultaneous support for different charsets on different terminals
of the same type (bug 1064).
In get_entity_string and point_intersect, do not initialise arrays with
static storage duration to zero; the C standard states that such objects
are automatically initialised to zero.
Add #include directives to fix these errors:
[CC] src/intl/gettext/l10nflist.o
cc1: warnings being treated as errors
.../src/intl/gettext/l10nflist.c: In function ‘_nl_normalize_codeset’:
.../src/intl/gettext/l10nflist.c:352: error: implicit declaration of function ‘c_tolower’
[CC] src/dom/css/scanner.o
cc1: warnings being treated as errors
In file included from .../src/dom/scanner.h:4,
from .../src/dom/css/scanner.h:4,
from .../src/dom/css/scanner.c:12:
.../src/dom/string.h: In function ‘dom_string_casecmp’:
.../src/dom/string.h:34: error: implicit declaration of function ‘c_strncasecmp’
Bug 932 is about ELinks letting control characters 0x80...0x9F through
to the terminal. It did not occur with ISO 8859-1, 8859-2, 8859-15,
or 8859-16, because the ELinks mappings for those charsets did not
include those bytes. However, the www.unicode.org versions imported
in the previous commit do include the problematic bytes.
To avoid a possible regression before the ELinks 0.12.0 release,
comment those control-character mappings out again. This workaround
should be reverted after bug 932 has been fixed properly.
Add copyright and licence notices, and a NEWS entry.
The data in the new versions is not entirely the same as what ELinks
used to have:
- Unicode/8859_1.cp: Adds control characters.
- Unicode/8859_2.cp: Adds control characters.
- Unicode/8859_4.cp: Adds some control characters that ELinks assumed
there already.
- Unicode/8859_7.cp: Adds three characters.
- Unicode/8859_15.cp: Adds control characters.
- Unicode/8859_16.cp: Adds control characters and swaps 0xA5 with 0xAB.
- Unicode/koi8_r.cp: Changes 0x95 and adds some control characters
that ELinks assumed there already.
- Unicode/macroman.cp: Changes 0xC6 and removes some control characters
that ELinks assumes there anyway.
Instead, convert the element pointers inside the comparison functions.
The last argument of qsort() is supposed to be of type
int (*)(const void *, const void *). Previously, comp_links() was
defined to take struct link * instead of const void *, and the type
mismatch was silenced by casting the function pointer to void *.
This was in principle not portable because:
(1) The different pointer types may have different representations.
In a word-oriented machine, the const void * might include a byte
selector while the struct link * might not.
(2) Casting a function pointer to a data pointer can lose bits in some
memory models. Apparently this does not occur in POSIX-conforming
systems though, as dlsym() would fail if it did.
This commit also fixes hits_cmp() and compare_dir_entries(), which
had similar problems. However, I'm leaving alias_compare() in
src/intl/gettext/localealias.c unchanged for now, so as not to diverge
from the GNU version.
I also checked the bsearch() calls but they were all okay, apart from
one that used the alias_compare() mentioned above.
Don't look for gettext message catalogs in ../po/ unless ELinks is being
run as src/elinks, ./src/elinks, or .../src/elinks.
Discovered by Arnaud Giersch, this alternate fix (than what is in debian
package 0.11.1-1.4) closes debian bug #417789 and redhat bug #235411.
Also reported in: CVE-2007-2027.
Restricting it to only work with --enable-debug was also considered,
however, it is an important feature for translaters so this less
paranoid fix was chosen.
This ensures that no other string can have the same address. It
probably never was a problem though, because the strings to which it
can be compared either are allocated from the heap or are in
strings[][] which already has unshared storage.
Convert each byte of them to UCS_REPLACEMENT_CHARACTER. This may not
be the optimal solution but at least it ought to be safe. Also raise
an internal error if the value read from utf8char_len_tab[] is out of
range.
Note that ELinks is still using the RFC 2279 definition of UTF-8 and
thus allows characters up to 0x7FFFFFFF, even though RFC 3629 has
changed the maximum to 0x10FFFF.
UCS_ORPHAN_CELL is currently defined as U+0020 SPACE, which was
already used before this macro, so the behaviour does not change,
but the code seems clearer now.
I searched for ' ' and 32 and 0x20 and \x20, and replaced with
UCS_ORPHAN_CELL wherever UCS_NO_CHAR was involved. However,
some BFU widgets first draw spaces and then overwrite with text;
those will require a more complex fix if UCS_ORPHAN_CELL is ever
changed to some other character.
/usr/bin/ld: warning multiple definitions of symbol _locale_charset
lib.o definition of _locale_charset in section (__TEXT,__text)
/usr/lib/libiconv.dylib(localcharset.o) definition of _locale_charset
This reverts the following commits:
- 86ed79deaf
Use wcwidth if available and applicable.
- 304f5fa1ea
comment fix (__STDC_ISO_10646__, not __STDC_ISO_10646)
- part of 71eebf1cc7
Compensate for glibc not defining wcwidth() when _XOPEN_SOURCE is not set
And adds a lengthy comment about LC_CTYPE problems.
Previously, each mapping between a codepage byte and a Unicode
character was stored as a struct table_entry, which listed both the
byte and the character. This representation may be optimal for sparse
mappings, but codepages map almost every possible byte to a character,
so it is more efficient to just have an array that lists the Unicode
character corresponding to each byte from 0x80 to 0xFF. The bytes are
not stored but rather implied by the array index. The tcvn5712 and
viscii codepages have a total of four mappings that do not fit in the
arrays, so we still use struct table_entry for those.
This change also makes cp2u() operate in O(1) time and may speed up
other functions as well.
The "sed | while read" concoction in Unicode/gen-cp looks rather
unhealthy. It would probably be faster and more readable if rewritten
in Perl, but IMO that goes for the previous version as well, so I
suppose whoever wrote it had a reason not to use Perl here.
Before:
text data bss dec hex filename
38948 28528 3311 70787 11483 src/intl/charsets.o
500096 85568 82112 667776 a3080 src/elinks
After:
text data bss dec hex filename
31558 28528 3311 63397 f7a5 src/intl/charsets.o
492878 85568 82112 660558 a144e src/elinks
So the text section shrank by 7390 bytes.
Measured on i686-pc-linux-gnu with: --disable-xbel --disable-nls
--disable-cookies --disable-formhist --disable-globhist
--disable-mailcap --disable-mimetypes --disable-smb --disable-mouse
--disable-sysmouse --disable-leds --disable-marks --disable-css
--enable-small --enable-utf-8 --without-gpm --without-bzlib
--without-idn --without-spidermonkey --without-lua --without-gnutls
--without-openssl CFLAGS="-Os -ggdb -Wall"
Before:
text data bss dec hex filename
25726 62992 3343 92061 1679d src/intl/charsets.o
653856 120020 82144 856020 d0fd4 src/elinks
After:
text data bss dec hex filename
60190 28528 3311 92029 1677d src/intl/charsets.o
688320 85556 82112 855988 d0fb4 src/elinks
So 34464 bytes were moved from the data section to the text section
and should be more likely to get shared between ELinks processes.
Measured on i686-pc-linux-gnu with: --disable-xbel --disable-nls
--disable-cookies --disable-formhist --disable-globhist
--disable-mailcap --disable-mimetypes --disable-smb --disable-mouse
--disable-sysmouse --disable-leds --disable-marks --disable-css
--enable-small --enable-utf-8 --without-gpm --without-bzlib
--without-idn --without-spidermonkey --without-lua --without-gnutls
--without-openssl CFLAGS="-O2 -ggdb -Wall"
The configure script no longer recognizes "CONFIG_UTF_8=yes" lines
in custom features.conf files. They will have to be changed to
"CONFIG_UTF8=yes". This incompatibility was deemed acceptable
because no released version of ELinks supports CONFIG_UTF_8.
The --enable-utf-8 option was not renamed.
Suggested by Miciah on #elinks.
What was renamed:
add_utf_8 => add_utf8
cp2utf_8 => cp2utf8
encode_utf_8 => encode_utf8
get_translation_table_to_utf_8 => get_translation_table_to_utf8
goto invalid_utf_8_start_byte => goto invalid_utf8_start_byte
goto utf_8 => goto utf8
goto utf_8_select => goto utf8_select
terminal_interlink.utf_8 => terminal_interlink.utf8
utf_8_to_unicode => utf8_to_unicode
What was not renamed:
terminal._template_.utf_8_io option, TERM_OPT_UTF_8_IO
Compatibility with existing elinks.conf files would require an alias.
--enable-utf-8
Because the name of the charset is UTF-8, --enable-utf-8 looks better
than --enable-utf8.
CONFIG_UTF_8
Will be renamed in a later commit.
Unicode/utf_8.cp, table_utf_8, aliases_utf_8
Will be renamed in a later commit.
Previously, utf8_step_forward() and utf8_step_backward() left *count
unchanged if some parameter was invalid. Now, they properly store 0.
This flaw had no effect in practice, because all current callers pass
count=NULL, and invalid parameters shouldn't be used anyway.
In the previous version, invalid UTF-8 from a terminal caused
UCS_NO_CHAR (0xFFFFFFFD) to be stored in a term_event_key_T, resulting
in -3 which was then incidentally treated as an unassigned special key.
Now, invalid UTF-8 is instead mapped to UCS_REPLACEMENT_CHARACTER
and treated as a character. The fact that handle_interlink_event
calls term_send_ucs when it receives invalid UTF-8 makes it pretty
clear that this is how it was intended.
src/viewer/text/link.c (not changed in this commit) already referred
to UCS_REPLACEMENT_CHARACTER in a comment even though it was not
previously defined.
It cannot be restricted just to characters that have passed
check_kbd_label_key(), because hotkeys in strings received from
gettext must also be processed with it, and there we don't have
a struct term_event for check_kbd_label_key().
This requires compiling cp2u() in even without CONFIG_UTF_8.
I also added an is_kbd_character macro to make try_document_key
more resilient to changes in the definition of term_event_key_T.
to encode it using utf_8_to_unicode. If every unicode_val_T value
could be a result of that function then one must add out param
to the utf_8_to_unicode signaling 'true' UCS_NO_CHAR.
