c03c63ba1a
Larbin is a web crawler (also called (web) robot, spider, scooter...). It is intended to fetch a large number of web pages to fill the database of a search engine. With a network fast enough, Larbin should be able to fetch more than 100 millions pages on a standard PC. from Giovanni Bechis <g.bechis@snb.it> with tweaks by me and ajacoutot@ ok ajacoutot@
5 lines
270 B
Plaintext
5 lines
270 B
Plaintext
Larbin is a web crawler (also called (web) robot, spider, scooter...).
|
|
It is intended to fetch a large number of web pages to fill the database
|
|
of a search engine. With a network fast enough, Larbin should be able to
|
|
fetch more than 100 millions pages on a standard PC.
|