than 2GiB in the download dialog. Also let regetting big files.
Previously the Content-Length variable in http.c was int, what is
not enough. Now it is long long.
There are two identical tp_show_header() functions: one in
src/session/download.c and one in src/protocol/bittorrent/dialogs.c.
Neither is declared in any header, but the latter was not static.
Documentation strings of most options used to contain a "\n" at the
end of each source line. When the option manager displayed these
strings, it treated each "\n" as a hard newline. On 80x24 terminals
however, the option description window has only 60 columes available
for the text (with the default setup.h), and the hard newlines were
further apart, so the option manager wrapped the text a second time,
resulting in rather ugly output where long lones are interleaved with
short ones. This could also cause the text to take up too much
vertical space and not fit in the window.
Replace most of those hard newlines with spaces so that the option
manager (or perhaps BFU) will take care of the wrapping. At the same
time, rewrap the strings in source code so that the source lines are
at most 79 columns wide.
In some options though, there is a list of possible values and their
meanings. In those lists, if the description of one value does not
fit in one line, then continuation lines should be indented. The
option manager and BFU are not currently able to do that. So, keep
the hard newlines in those lists, but rewrap them to 60 columns so
that they are less likely to require further wrapping at runtime.
This makes the option manager display it much better on an 80x24
terminal.
Alternatively, the "\n" newline characters within paragraphs could
have been removed entirely. ELinks would then have line-wrapped the
text to the appropriate width in the info window of the option
manager, but unfortunately not in --config-help.
AFAIK, all bugs in it have been fixed. Some bugs may still be lurking
but they are more likely to get caught if compression is enabled.
I also replaced COMP_NOTE with static text because xgettext does not
support macros in the argument of N_.
(cherry picked from commit 3a9b5d091d)
This simplifies the callers a little and may help implement
simultaneous support for different charsets on different terminals
of the same type (bug 1064).
Except if they have external handlers.
When ELinks receives an event from a terminal, move that terminal to
the beginning of the global "terminals" list, so that the terminals
are always sorted according to the time of the most recent use. Note,
this affects the numbering of bookmark folders in session snapshots.
Add get_default_terminal(), which returns the most recently used
terminal that is still open. Use that in various places that
previously used terminals.prev or terminals.next. Four functions
fetch the size of the terminal for User-Agent headers, and
get_default_terminal() is not really right, but neither was the
original code; add TODO comments in those functions.
When the user chooses "Background and Notify", associate the download
with the terminal where the dialog box is. So any later messages will
then appear in that terminal, if it is still open. However, don't
change the terminal if the download has an external handler.
When a download gets some data, don't immediately check the associated
terminal. Instead, wait for the download to end. Then, if the
terminal of the download has been closed, use get_default_terminal()
instead. If there is no default terminal either, just skip any
message boxes.
make_bittorrent_peer_connection() used to construct a struct uri on
the stack. This was hacky but worked nicely because the struct uri
was not really accessed after make_connection() returned. However,
since commit a83ff1f565, the struct uri
is also needed when the connection is being closed. Valgrind shows:
Invalid read of size 2
at 0x8100764: get_blacklist_entry (blacklist.c:33)
by 0x8100985: del_blacklist_entry (blacklist.c:64)
by 0x80DA579: complete_connect_socket (socket.c:448)
by 0x80DA84A: connected (socket.c:513)
by 0x80D0DDF: select_loop (select.c:297)
by 0x80D00C6: main (main.c:353)
Address 0xBEC3BFAE is just below the stack ptr. To suppress, use: --workaround-gcc296-bugs=yes
To fix this, allocate the struct uri on the heap instead, by
constructing a string and giving that to get_uri(). This string
cannot use the "bittorrent" URI scheme because parse_uri() does not
recognize the host and port fields in that. (The "bittorrent" scheme
has protocol_backend.free_syntax = 1 in order to support strings like
"bittorrent:http://beta.legaltorrents.com/get/159-noisome-beasts".)
Instead, define a new "bittorrent-peer" URI scheme for this purpose.
If the user attempts to use this URI scheme, its handler aborts the
connection with an error; but when make_bittorrent_peer_connection()
uses a bittorrent-peer URI, the handler is not called.
This change also lets get_uri() set the ipv6 flag if peer_info->ip is
an IPv6 address literal.
Reported by Witold Filipczyk.
Separate the formatting of unparsed lines from ftp_process_dirlist()
to a new function ftp_add_unparsed_line(). Check for all possible
out-of-memory errors. Encode HTML metacharacters as entity references
and document how charsets are handled FTP directory listings.
Add a NEWS entry.
fsp_open_session() has a bug where it does not set errno if getaddrinfo fails.
Before the bug 1013 fix, this caused an assertion failure.
After the bug 1013 fix, this caused a "Success" error message.
Now it instead causes "FSP server not found".
Replace almost all uses of enum connection_state with struct
connection_status. This removes the assumption that errno values used
by the system are between 0 and 100000. The GNU Hurd uses values like
ENOENT = 0x40000002 and EMIG_SERVER_DIED = -308.
This commit is derived from my attachments 450 and 467 to bug 1013.
It seems GnuTLS is not as good at negotiating a supported protocol as
OpenSSL is. ELinks tries to work around that by retrying with a
different protocol if the SSL library reports an error. However,
ELinks must not automatically retry POST requests where some data may
have already reached the server; POST is not a safe method in HTTP.
So instead, collect the name of the TLS-incapable server in a blacklist
when ELinks e.g. loads an HTML form from it; the actual POST can then
immediately use the protocol that worked.
It's a bit ugly that src/network/socket.c now uses
protocol/http/blacklist.h. It might be better to move the blacklist
files out of the http directory, and perhaps merge them with the
BitTorrent blacklisting code.
Conflicts:
NEWS
configure.in
The following files also conflicted, but they had not been manually
edited in the elinks-0.12 branch after the previous merge, so I just
kept the 0.13.GIT versions:
doc/man/man1/elinks.1.in
doc/man/man5/elinks.conf.5
doc/man/man5/elinkskeys.5
po/fr.po
po/pl.po
In uri.post, each file name begins and ends with FILE_CHAR.
Previously, file names were not encoded, and names containing
FILE_CHAR could not be used. Because FILE_CHAR is a control
character, the user cannot directly type it in a file input field,
so ELinks asserted that the field did not contain FILE_CHAR.
However, it is possible to get FILE_CHAR in a file input field
with file name completion (ACT_EDIT_AUTO_COMPLETE), causing the
assertion to fail. Now, ELinks encodes FILE_CHAR as "%02", so it
is no longer ambiguous and the assertion is not needed.
gcc-4.3 -O2 was complaining that http_got_header may use uninitialized
version.major and version.minor. That indeed happened with HTTP/0.9
servers, and the PRE_HTTP_1_1(version) check then had an undefined
result, so http->close could remain 0 even though it should have
become 1; fortunately, it was then set to 1 anyway, because there was
no Content-Length header. The undefined version was also saved in
http->recv_version, but it appears nothing ever reads that. So in the
end, the bug did not cause any symptoms at runtime, but the warning
broke the build on gcc-4.3 if ELinks was configured with --enable-debug.
I am reverting all /dev/fd recognition because of bug 917.
This reverts commit c283f8cfd9,
except src/protocol/file/file.c still needs #include "osdep/osdep.h"
for STRING_DIR_SEP.
I am reverting all copiousoutput support because of bug 917.
This reverts commit 4dc4ea47f2.
Conflicts:
src/network/connection.h: After the original commit, the declaration
of copiousoutput_data had been changed to use the LIST_OF macro.
Also, connection.cgi had been added next to the connection.popen
member added by the original commit.
src/session/download.c: After the original commit, the definition of
copiousoutput_data had been changed to use the INIT_LIST_OF macro.
If the user opens the same file again after it is in the cache, then
ELinks does not always open a new connection, so download->conn can be
NULL in init_type_query(), and download->conn->cgi would crash.
Don't read that, then; instead add a new flag cache_entry.cgi, which
http_got_header() sets or clears as soon as possible after the cache
entry has been created.
(cherry picked from commit 81f8ee1fa2)
CGI scripts are distinguishable from normal files. I hope that this
fixes the bug 991. This commit also reverts the previous revert.
(cherry picked from commit 7ceba1e461)
The comment said "it is not possible to call kill_timer from a timer
handler." Sure, such calls used to crash occasionally, but that was
bug 868 and has already been fixed.
Previously, each progress timer function registered with
start_update_progress() was directly used as the timer function of
progress.timer, so it was responsible of erasing the expired timer ID
from that member. Failing to do this could result in heap corruption.
The progress timer functions normally fulfilled the requirement by
calling update_progress(), but one such function upload_stat_timer()
had to erase the timer ID on its own too.
Now instead, there is a wrapper function progress_timeout(), which
progress.c sets as the timer function of progress.timer. This wrapper
erases the expired timer ID from progress.timer and then calls the
progress timer function registered with start_update_progress(). So
the progress timer function is no longer responsible of erasing the
timer ID and there's no risk that it could fail to do that in some
error situation.
This commit introduces a new risk though. Previously, if the struct
progress was freed while the timer was running, the (progress) timer
function would still be called, and it would be able to detect that
the progress pointer is NULL and recover from this situation. Now,
the timer function progress_timeout() has a pointer to the struct
progress and will dereference that pointer without being able to check
whether the structure has been freed. Fortunately, done_progress()
asserts that the timer is not running, so this should not occur.
Posting a 91762123-byte file to test/cgi/big_file.cgi. The CPU
percentages are from "top" set up to update every 10 seconds and
checked near the end of the transfer, so they are less accurate
than the upload rate, which averages over the whole transfer.
buffer=4096: average 1.7 MiB/s, elinks 62% CPU, python 35% CPU.
buffer=8192: average 2.5 MiB/s, elinks 49% CPU, python 42% CPU.
buffer=16384: average 3.1 MiB/s, elinks 40% CPU, python 55% CPU.
buffer=32768: average 3.8 MiB/s, elinks 33% CPU, python 61% CPU.
buffer=65536: average 4.1 MiB/s, elinks 26% CPU, python 70% CPU.
buffer=131072: average 4.2 MiB/s, elinks 28% CPU, python 67% CPU.
buffer=262144: average 4.4 MiB/s, elinks 26% CPU, python 69% CPU.
I'm choosing 32768 as POST_BUFFER_SIZE because the advantages of
larger buffers don't seem very high and keeping this under 65536
may help anyone trying to port ELinks to DOS.
I'm using the same value for HTTP too, just to keep things consistent
until there is a reason to diverge.
Without this patch, ELinks showed garbage at
<http://www.dwheeler.com/oss_fs_why.html> when bzip2 decompression was
enabled. safe_read() in bzip2_read() did not see all of the body
bytes that ELinks had received from the server. After bzip2_read()
received EAGAIN from safe_read() and returned 0, something skipped
1460 bytes.
decompress_data() apparently assumed that read_encoded() returning 0
meant the end of the file, and returned even though len still was
nonzero, i.e. it had not yet written to the pipe all the data that
the caller (read_chunked_http_data() or read_normal_http_data()) had
provided. The caller did not know this, and discarded the data.
(cherry picked from commit 7e5e05ca60)
Without this patch, ELinks showed garbage at
<http://www.dwheeler.com/oss_fs_why.html> when bzip2 decompression was
enabled. safe_read() in bzip2_read() did not see all of the body
bytes that ELinks had received from the server. After bzip2_read()
received EAGAIN from safe_read() and returned 0, something skipped
1460 bytes.
decompress_data() apparently assumed that read_encoded() returning 0
meant the end of the file, and returned even though len still was
nonzero, i.e. it had not yet written to the pipe all the data that
the caller (read_chunked_http_data() or read_normal_http_data()) had
provided. The caller did not know this, and discarded the data.
Move connection.post_fd to http_post.post_fd.
Make connection.done point to the new done_http_connection(),
which calls the new done_http_post(), which closes post_fd.
So done_connection() no longer needs to do that.
Now that done_http_post() exists, a later commit can add dynamically
allocated data in struct http_post and ensure that it will be freed.
As the comment near the end of this function says, conn->info is
already non-NULL if a HTTPS proxy is being used, and the code in fact
correctly frees the previous info. So there is no need to assert its
nonexistence. I added that bug on 2008-05-22, in commit 291a913d1e.
If ELinks is being linked with SSL library, use its random number
generator.
Otherwise, try /dev/urandom and /dev/prandom. If they do not work,
fall back to rand(), calling srand() only once. This fallback is
mostly interesting for the Hurd and Microsoft Windows.
BitTorrent piece selection and dom/test/html-mangle.c still use rand()
(but not srand()) directly. Those would not benefit from being
unpredictable, I think.
To reduce code duplication, src/protocol/file/cgi.c no longer parses
connection->uri->post on its own but rather calls the new function
http_read_post_data(), provided by src/protocol/http/http.c. The same
code is now also used for POST requests that do not include files.
Conflicts:
NEWS (bug 939 was listed twice)
doc/man/man5/elinks.conf.5 (regenerated)
po/fr.po (only in comments and such)
po/pl.po (only in comments and such)
src/protocol/fsp/fsp.c (the relevant changes were already here)
*fresult pointed to nowhere. On FreeBSD *fresult == NULL
and directories weren't displayed.
Check also if safe_write writes all data.
(cherry picked from commit 06bcc48487)
If the user opens the same file again after it is in the cache, then
ELinks does not always open a new connection, so download->conn can be
NULL in init_type_query(), and download->conn->cgi would crash.
Don't read that, then; instead add a new flag cache_entry.cgi, which
http_got_header() sets or clears as soon as possible after the cache
entry has been created.
CGI scripts are distinguishable from normal files. I hope that this
fixes the bug 991. This commit also reverts the previous revert.
(cherry picked from commit 7ceba1e461)
libsmbclient's stdout and stderr interferred with ELinks's stdout
and stdin. That caused an assertion failure. Now the ELinks uses
different streams for processing of the smb protocol.
This reverts commit 7ceba1e461,
which is causing an assertion to fail if I open the same PDF
twice in a row, even if I cancel the dialog box when ELinks
first asks which program to run:
INTERNAL ERROR at /home/Kalle/src/elinks-0.12/src/session/download.c:980: assertion download && download->conn failed!
Forcing core dump! Man the Lifeboats! Women and children first!
But please DO NOT report this as a segfault!!! It is an internal error, not a
normal segfault, there is a huge difference in these for us the developers.
Also, noting the EXACT error you got above is crucial for hunting the problem
down. Thanks, and please get in touch with us.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1216698688 (LWP 17877)]
0xb7a02d76 in raise () from /lib/libc.so.6
(gdb) backtrace 6
at /home/Kalle/src/elinks-0.12/src/util/error.c:179
fmt=0x816984c "assertion download && download->conn failed!")
at /home/Kalle/src/elinks-0.12/src/util/error.c:122
cached=0x8253ca8) at /home/Kalle/src/elinks-0.12/src/session/download.c:980
cached=0x8253ca8, frame=0)
at /home/Kalle/src/elinks-0.12/src/session/download.c:1339
at /home/Kalle/src/elinks-0.12/src/session/task.c:493
(More stack frames follow...)
There is a fix available but I don't trust it yet.
This syncs some changes (ie. -> e.g. etc.) from elinks-0.12 or beyond.
I noticed them while updating the web pages, and apologize that I will
not spent the time to attribute it to the individual commits.
(cherry picked from commit 2bfc7b3724,
omitting generated files)
AFAIK, all bugs in it have been fixed. Some bugs may still be lurking
but they are more likely to get caught if compression is enabled.
I also replaced COMP_NOTE with static text because xgettext does not
support macros in the argument of N_.
The bug was reported by Paul B. Mahol on elinks-users. The example is
from the FTP site he provided:
ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-ia64/
Message-ID: <3a142e750802262008l6fd55be5v44207bc4479dd3fc@mail.gmail.com>
(cherry picked from commit c069403b75)
... so all the tests with responses stretching multiple lines are
actually tested in their entirety.
(cherry picked from commit aa9a847c00,
resolving a conflict due to the use of get_test_opt)
On AMD64 apparently, off_t is long but ELinks detected SIZEOF_OFF_T == 8
and defined OFF_T_FORMAT as "lld", which expects long long and so causes
GCC to warn about a mismatching format specifier. Because --enable-debug
adds -Werror to $CFLAGS, this warning breaks the build. When both
SIZEOF_LONG and SIZEOF_LONG_LONG are 8, ELinks cannot know which type
it should use.
To fix this, do not attempt to find a format specifier for off_t itself.
Instead cast all printed off_t values to a new typedef off_print_T that
is large enough, and replace OFF_T_FORMAT with OFF_PRINT_FORMAT which
is suitable for off_print_T altough not necessarily for off_t. ELinks
already had a similar scheme with time_print_T and TIME_PRINT_FORMAT.
Previously, struct string was used here. However,
bittorrent_fetch_callback does not initialize response.magic,
and parse_bittorrent_tracker_response changes response->source
to point to data that must not be freed. So the util/string.h
functions are not actually safe to use on these objects.
For this reason, it is safer to use a separate type.
The previous check (integer > (off_t) integer * 10) did not detect all
overflows. Examples with 32-bit off_t:
integer = 0x1C71C71D (0x100000000/9 rounded up);
integer * 10 = 0x11C71C722, wraps to 0x1C71C722 which is > integer.
integer = 0x73333333;
integer * 10 = 0x47FFFFFFE, wraps to 0x7FFFFFFE which is > integer.
Examples with 64-bit off_t:
integer = 0x1C71C71C71C71C72 (0x10000000000000000/9 rounded up);
integer * 10 = 0x11C71C71C71C71C74, wraps to 0x1C71C71C71C71C74
which is > integer.
integer = 0x7333333333333333;
integer * 10 = 0x47FFFFFFFFFFFFFFE, wraps to 0x7FFFFFFFFFFFFFFE
which is > integer.
It is unclear to me what effect an undetected overflow would actually
have from the user's viewpoint, so I'm not adding a NEWS entry.
(cherry picked from commit a25fd18e56)
The compression support in ELinks has always been buggy, with some large pages
failing to decompress and containing garbage at the end instead. However,
with the recent attempts to fix the compression support, it has been actually
made *so* buggy that not only these cases seem to occur more often, but in
some cases, the page is just silently chopped and no content visible; in other
cases, "Resource temporarily unavailable" is displayed. Etc.
The compression support got now to the point where it is so awfully unstable
that it is actively harmful to have it enabled by default. I've been burnt by
it several times already and once made a very serious error because of page
being chopped silently.
This change avoids linker warnings when building with Debian tcc
0.9.23-4 + patch from Debian bug 418360:
[LD] src/protocol/bittorrent/lib.o
bittorrent.o: 'BITTORRENT_NULL_ID' defined twice
common.o: 'BITTORRENT_NULL_ID' defined twice
connection.o: 'BITTORRENT_NULL_ID' defined twice
dialogs.o: 'BITTORRENT_NULL_ID' defined twice
peerconnect.o: 'BITTORRENT_NULL_ID' defined twice
peerwire.o: 'BITTORRENT_NULL_ID' defined twice
piececache.o: 'BITTORRENT_NULL_ID' defined twice
tracker.o: 'BITTORRENT_NULL_ID' defined twice
Add a boolean protocol flag which says whether "//" in the path
part of an URI can be safely substituted with "/". Be conservative
and enable it only for file://, ftp:// and nntp[s]://. Other
can be turned on later, if needed.
Generalizes the fix from 58b3b1e752.
This reverts commit 4f0aaa166e
and insert check for the "//" -> "/" change only to occur for
file:// URIs. This fixes the recent reports on broken handling
of relative file URIs starting with "..".
<http://www.wikipedia.org/w/wiki.phtml?search=sue%20lawley>
incorrectly redirects to
<http://en.wikipedia.org/w/wiki.phtml?search=sue%2520lawley>
which searches for "sue%20lawley" rather than "sue lawley".
By using en.wikipedia.org directly, we avoid the server bug.
Prompted by an elinks-users post on 2007-07-27.
I asked on #wikimedia-tech, and www.wikipedia.org does always
redirect to en.wikipedia.org; it does not guess any other
language based on headers or IP addresses or such. Also, the
redirection exists only for compatibility, and skipping it
avoids a few roundtrips to the server. So this change is good
even if the server is eventually fixed.
There were conflicts in src/document/css/ because 0.12.GIT switched
to LIST_OF(struct css_selector) and 0.13.GIT switched to struct
css_selector_set. Resolved by using LIST_OF(struct css_selector)
inside struct css_selector_set.
This patch changes normalize_uri() to no replace "//" with "/" in URIs. This
fixed this bug but will also lead to possibility that duplicate entries can
exist in ELinks' cache. ELinks might be able to detect in another way by
hashing the content or something.
[ From attachment 310 of bug 744. --KON ]
This change avoids linker warnings when building with Debian tcc
0.9.23-4 + patch from Debian bug 418360:
[LD] src/protocol/bittorrent/lib.o
bittorrent.o: 'BITTORRENT_NULL_ID' defined twice
common.o: 'BITTORRENT_NULL_ID' defined twice
connection.o: 'BITTORRENT_NULL_ID' defined twice
dialogs.o: 'BITTORRENT_NULL_ID' defined twice
peerconnect.o: 'BITTORRENT_NULL_ID' defined twice
peerwire.o: 'BITTORRENT_NULL_ID' defined twice
piececache.o: 'BITTORRENT_NULL_ID' defined twice
tracker.o: 'BITTORRENT_NULL_ID' defined twice
And reorder the characters in the string given to strcspn(), to match
their expected order in the URI. This is also how strcspn() is called
elsewhere in uri.c.
Use it for the actual I/O only. Previously, defining CONFIG_UTF8 and
enabling UTF-8 used to force many strings to the UTF-8 charset
regardless of the terminal charset option. Now, those strings always
follow the terminal charset. This fixes bug 914 which was caused
because _() returned strings in the terminal charset and functions
then assumed they were in UTF-8. This reduction in the effects of
UTF-8 I/O may also simplify future testing.
Give them a corresponding Content-Type header. This must go in
cached->head because cached->content_type is supposed to be just
type/subtype. It will also be deduced from cached->head, so don't set
it separately.
string_concat reads the args with va_arg(ap, const unsigned char *),
and the NULL macro may have the wrong type (e.g. int).
Many places pass string literals of type char * to string_concat.
This is in principle also a violation, but I'm ignoring it for now
because if it becomes a problem with some C implementation, then so
will the use of unsigned char * with printf "%s", which is so
widespread in ELinks that I'm not going to try fixing it now.
straconcat reads the args with va_arg(ap, const unsigned char *),
and the NULL macro may have the wrong type (e.g. int).
Many places pass string literals of type char * to straconcat. This
is in principle also a violation, but I'm ignoring it for now because
if it becomes a problem with some C implementation, then so will the
use of unsigned char * with printf "%s", which is so widespread in
ELinks that I'm not going to try fixing it now.
Don't cast function pointers; calling functions via pointers of
incorrect types is not guaranteed to work. Instead, define the
functions with the desired types, and make them cast the incoming
parameters. Or define wrapper functions if the return types don't
match.
really_exit_prog wasn't being used outside src/dialogs/menu.c,
and I had to change its parameter type, so it's now static.
Revert commit 11e6aa8d97.
It is not useful to complicate the code to free the memory,
because the process is soon going to exit, and it has inherited
a lot more memory allocations from its parent.
The libsmbclient 3.0.10 compatibility changes are good though.
I'll re-apply them soon enough.
FSP v2 uses 32-bit file offsets in its UDP packets and cannot get past
4 GiB. ELinks should however try not to wrap the numbers to negative
at 2 GiB, if it is built with large file support.
