Movatterモバイル変換

Webbot - the Libwww Robot

Basics |Get it! | Features | Options |Release Notes |How to

The Basics

Author(s)

Henrik Frystyk Nielsen, John Punin, Bob Racko

Status

An example application forlibwww. The robot is theprimary tool used for theHTTPperformance measurements.

Plans

New releases follow the releases oflibwww with noexception.

Platforms

It runs on thesame platforms aslibwww

Getting the Source Code

The webbot comes as an integral part of thelibwww distribution package.

Forums

Discussions on libwww and its example applications take place on<www-lib@w3.org> (archives). See thedocumentation for how to subscribe.

Features

• Really fastlibwww HTTP/1.1 engine - usesHTTP/1.1 pipelining whichspeeds up performance a lot
• Supportsrobot.txtfiles androbotMETA tags
• Smart use ofGET andHEAD - only downloads thestrictly necessary
• Check ofinlined images as well ashyperlinks
• Support forregular expressionsallows detailed Web traversal constraints
• Support forSQL based loggingwhich allows for an open-ended set of thongs you can do with the logs
• Support fortraditional file basedlogging using Common Log File formats and Referer log file formats
• Awealth of logging facilitiesincluding:
  • Missing or empty ALT tags in inlined images
  • Broken links
  • Document titles based on theTITLE HTML tag and theTitle HTTP header field
  • Link relationships between documents based on theLINK HTML tag
  • Modification dates based on theLast-Modified HTTP header field
  • Distributions of content-types and charsets encountered in the traversal

How to Run It

Be careful - this is a robot and hence can be used to traverse many links -it should be used with care and is not designed to be let loose on theInternet at large. Its primary design goal was to be able to testHTTP/1.1 pipeliningfeatures.

The robot has a large set ofcommand lineoptions that can be used in a large set of different combinations. You cantry andsee this simple script in order to see anexample of how it can be run.