1eeb4dd082
This module implements a configurable web traversal engine, for a robot
or other web agent. Given an initial web page (URL), the Robot will get
the contents of that page, and extract all links on the page, adding
them to a list of URLs to visit.
Features of the Robot module include:
* Follows the Robot Exclusion Protocol.
* Supports the META element proposed extensions to the Protocol.
* Implements many of the Guidelines for Robot Writers.
* Configurable.
* Builds on standard Perl 5 modules for WWW, HTTP, HTML, etc.
5 lines
250 B
Plaintext
5 lines
250 B
Plaintext
MD5 (WWW-Robot-0.024.tar.gz) = 893df367ff4eabc477a62a2a5cbbfda3
|
|
RMD160 (WWW-Robot-0.024.tar.gz) = 2c4360a488a47ce10e9a0c380aa8302c9bddd216
|
|
SHA1 (WWW-Robot-0.024.tar.gz) = 2837684c6905cce3570dfa7bdad33608422ea86c
|
|
SIZE (WWW-Robot-0.024.tar.gz) = 17465
|