openbsd-ports/net/crawl/Makefile

# $OpenBSD: Makefile,v 1.17 2004/04/07 22:51:03 brad Exp $

COMMENT=		"small and efficient HTTP crawler"

DISTNAME=		crawl-0.4
CATEGORIES=		net

HOMEPAGE=		http://www.monkey.org/~provos/crawl/

# BSD
PERMIT_PACKAGE_CDROM=	Yes
PERMIT_PACKAGE_FTP=	Yes 
PERMIT_DISTFILES_CDROM=	Yes
PERMIT_DISTFILES_FTP=	Yes

MASTER_SITES=		http://www.monkey.org/~provos/

LIB_DEPENDS=		lib/db/db.3:db->3,<4:databases/db/v3

CONFIGURE_STYLE=	gnu
CONFIGURE_ARGS=		--with-db="${LOCALBASE}"
CONFIGURE_ENV=		CFLAGS="${CFLAGS} -I/usr/local/include/db"

NO_REGRESS=		Yes

.include <bsd.port.mk>
upgrade to crawl 0.4 -- From: Robert Nagy <thuglife at bsd dot hu> 2004-04-07 18:51:03 -04:00			`# $OpenBSD: Makefile,v 1.17 2004/04/07 22:51:03 brad Exp $`
crawl-0.1b import - provos@ ok The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites. The main reason for writing crawl was the lack of simple open source web crawlers. Crawl is only a few thousand lines of code and fairly easy to debug and customize. Features + Saves encountered JPEG images + Image selection based on regular expressions and size contraints + Resume previous crawl after graceful termination + Persistent database of visited URLs + Very small and efficient code + Supports robots.txt 2001-09-09 17:57:12 -04:00
			`COMMENT= "small and efficient HTTP crawler"`

upgrade to crawl 0.4 -- From: Robert Nagy <thuglife at bsd dot hu> 2004-04-07 18:51:03 -04:00			`DISTNAME= crawl-0.4`
crawl-0.1b import - provos@ ok The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites. The main reason for writing crawl was the lack of simple open source web crawlers. Crawl is only a few thousand lines of code and fairly easy to debug and customize. Features + Saves encountered JPEG images + Image selection based on regular expressions and size contraints + Resume previous crawl after graceful termination + Persistent database of visited URLs + Very small and efficient code + Supports robots.txt 2001-09-09 17:57:12 -04:00			`CATEGORIES= net`

			`HOMEPAGE= http://www.monkey.org/~provos/crawl/`

drop maintainership on some stuff i don't use anymore, lop off WWW: ${HOMEPAGE} while touching 'em 2003-09-25 02:25:13 -04:00			`# BSD`
crawl-0.1b import - provos@ ok The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites. The main reason for writing crawl was the lack of simple open source web crawlers. Crawl is only a few thousand lines of code and fairly easy to debug and customize. Features + Saves encountered JPEG images + Image selection based on regular expressions and size contraints + Resume previous crawl after graceful termination + Persistent database of visited URLs + Very small and efficient code + Supports robots.txt 2001-09-09 17:57:12 -04:00			`PERMIT_PACKAGE_CDROM= Yes`
			`PERMIT_PACKAGE_FTP= Yes`
			`PERMIT_DISTFILES_CDROM= Yes`
			`PERMIT_DISTFILES_FTP= Yes`

			`MASTER_SITES= http://www.monkey.org/~provos/`

fix db dependencies to ensure db/v3 is installed ensure db/v3 is used also fixes build on NO_SHARED_ARCHS with conceptual help from brad@ 2004-01-10 03:33:11 -05:00			`LIB_DEPENDS= lib/db/db.3:db->3,<4:databases/db/v3`
crawl-0.1b import - provos@ ok The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites. The main reason for writing crawl was the lack of simple open source web crawlers. Crawl is only a few thousand lines of code and fairly easy to debug and customize. Features + Saves encountered JPEG images + Image selection based on regular expressions and size contraints + Resume previous crawl after graceful termination + Persistent database of visited URLs + Very small and efficient code + Supports robots.txt 2001-09-09 17:57:12 -04:00
- remove unnecessary patches - use DESTDIRNAME 2002-06-23 14:23:03 -04:00			`CONFIGURE_STYLE= gnu`
			`CONFIGURE_ARGS= --with-db="${LOCALBASE}"`
use new databases/db layout db update and these modifications by Aleksander Piotrowski <aleksander dot piotrowski at nic dot com dot pl> 2003-12-08 12:42:34 -05:00			`CONFIGURE_ENV= CFLAGS="${CFLAGS} -I/usr/local/include/db"`
crawl-0.1b import - provos@ ok The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites. The main reason for writing crawl was the lack of simple open source web crawlers. Crawl is only a few thousand lines of code and fairly easy to debug and customize. Features + Saves encountered JPEG images + Image selection based on regular expressions and size contraints + Resume previous crawl after graceful termination + Persistent database of visited URLs + Very small and efficient code + Supports robots.txt 2001-09-09 17:57:12 -04:00
No regression tests available. 2002-10-27 20:38:44 -05:00			`NO_REGRESS= Yes`

crawl-0.1b import - provos@ ok The crawl utility starts a depth-first traversal of the web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. Crawl keeps a persistent database that allows multiple crawls without revisiting sites. The main reason for writing crawl was the lack of simple open source web crawlers. Crawl is only a few thousand lines of code and fairly easy to debug and customize. Features + Saves encountered JPEG images + Image selection based on regular expressions and size contraints + Resume previous crawl after graceful termination + Persistent database of visited URLs + Very small and efficient code + Supports robots.txt 2001-09-09 17:57:12 -04:00			`.include <bsd.port.mk>`