Search RFCs

RFC Editor

File formats:

Also available:XML file for editing

 

Status:

PROPOSED STANDARD

Authors:

M. Koster
G. Illyes
H. Zeller
L. Sassman

Stream:

IETF

Source:

NON WORKING GROUP

Cite this RFC:TXT  | XML  |  BibTeX

DOI:  https://doi.org/10.17487/RFC9309

Discuss this RFC: Send questions or comments to the mailing listiesg@ietf.org

Other actions:View Errata  | Submit Errata  | Find IPR Disclosures from the IETF  | View History of RFC 9309

Abstract

This document specifies and extends the "Robots Exclusion Protocol"method originally defined by Martijn Koster in 1994 for serviceowners to control how content served by their services may beaccessed, if at all, by automatic clients known as crawlers.Specifically, it adds definition language for the protocol,instructions for handling errors, and instructions for caching.

For the definition ofStatus,seeRFC 2026.

For the definition ofStream, seeRFC 8729.