1eeb4dd082
This module implements a configurable web traversal engine, for a robot
or other web agent. Given an initial web page (URL), the Robot will get
the contents of that page, and extract all links on the page, adding
them to a list of URLs to visit.
Features of the Robot module include:
* Follows the Robot Exclusion Protocol.
* Supports the META element proposed extensions to the Protocol.
* Implements many of the Guidelines for Robot Writers.
* Configurable.
* Builds on standard Perl 5 modules for WWW, HTTP, HTML, etc.
4 lines
120 B
Plaintext
Executable File
4 lines
120 B
Plaintext
Executable File
@comment $OpenBSD: PLIST,v 1.1.1.1 2006/12/17 18:09:41 jasper Exp $
|
|
${P5SITE}/WWW/Robot.pm
|
|
@man man/man3p/WWW::Robot.3p
|