2.1.Defining the Important Milestones

The IETF Datatracker records a list of events for each document processed by IETFworking groups. This has a high granularity, and also a high variability. Most documentsstart life as an individual draft, are adopted by a working group, undergo aWorking Group Last Call, are submitted to the IESG, undergo an IETF Last Calland an IESG review, get eventually approved by the IESG, and are processedfor publication by the RFC Editor, but there are exceptions. Some documentsare first submitted to one working group and then moved to another. Some documentsare published through the Independent Stream, and are submitted to theIndependent Submissions Editor instead of the IESG.¶

In order to simplify tabulation, we break the period from the submission of the firstdraft to the publication of the RFC into three big components:¶

• The working group processing time, from the first draft to the start of the IETFlast call;¶
• The IETF processing time, which lasts from the beginning of the IETF last call tothe approval by the IESG, including the reviews byvarious directorates;¶
• The RFC production, from approval by the IESG to publication, includingthe AUTH48 reviews.¶

For submissions to the Independent Stream, we don't have a working group.We consider instead the progression of the individual draft until theadoption by the Independent Submissions Editor (ISE) as the equivalent of the "Working Group" period, and the delay from adoption by the ISE until submission to the RFC Editoras the equivalent of the IETF processing time.¶

We measure the starting point of the process using the date of submissionof the first draft listed on that RFC page in the IETF Datatracker. In mostcases, this first draft is an individual draft that then resubmitted as aworking group draft, or maybe resubmitted with a new name as the draft wassearching for a home in an IETF working group, or before deciding forsubmission on the Independent Stream.¶

The IETF Datatracker entries for RFCs and drafts do notalways list working group events like Working Group Last Call. The only intermediate event that we listbetween the first draft and the submission to the IESG is the working groupadoption, for which we use the date of submission of version 00 of thedraft eventually published as RFC. We also use that date (of submission of version 00) for draftssubmitted to the Independent Stream.¶

2.2.Selecting a Random Sample of RFCs

Basic production mechanisms could be evaluated by processing data fromthe IETF Datatracker, but subjective data requires manual assessment of results,which can be time-consuming. Since our resources are limited, we will onlyperform this analysis for a small sample of RFCs, selected at randomfrom the list of RFCs approved in 2018. Specifically, we will pick20 RFC numbers at random between:¶

• RFC 8307, published in January 2018, and¶
• RFC 8511, published December 2018.¶

The list of 20 selected RFCs is: RFC 8411, RFC 8456, RFC 8446, RFC 8355, RFC 8441, RFC 8324, RFC 8377, RFC 8498, RFC 8479, RFC 8453, RFC 8429, RFC 8312, RFC 8492 , RFC 8378, RFC 8361, RFC 8472, RFC 8471, RFC 8466, RFC 8362, and RFC 8468.¶

When evaluating delays and impact, we will compare the year 2018 to 2008 and1998, 10 and 20 years ago. To drive this comparison, we pick 20 RFCs at randomamong those published in 2008, and another 20 among those published in 1998.¶

The list of the 20 randomly selected RFCs from 2008 is: RFC 5227, RFC 5174, RFC 5172, RFC 5354,RFC 5195, RFC 5236, RFC 5348, RFC 5281, RFC 5186, RFC 5326, RFC 5277, RFC 5373, RFC 5404,RFC 5329, RFC 5283, RFC 5358, RFC 5142, RFC 5271, RFC 5349, and RFC 5301.¶

The list of the 20 randomly selected RFCs from 1998 is: RFC 2431, RFC 2381, RFC 2387, RFC 2348,RFC 2391, RFC 2267, RFC 2312, RFC 2448, RFC 2374, RFC 2398, RFC 2283, RFC 2382, RFC 2289,RFC 2282, RFC 2404, RFC 2449, RFC 2317, RFC 2394, RFC 2297, and RFC 2323.¶

3.1.RFC 8411

"IANA Registration for the Cryptographic Algorithm Object Identifier Range"[RFC8411]:¶

Status (Length):

Informational (5 pages)¶

Overview:

4 individual drafts¶

First draft:

2017-05-08¶

Last Call start:

2017-10-09¶

IESG eval. start:

2017-12-28¶

IESG approved:

2018-02-26 (draft 03)¶

AUTH48 start:

2018-04-20¶

AUTH48 complete:

2018-07-17¶

Published:

2018-08-06¶

IANA action:

create table¶

This RFC was published from the individual draft, which was not resubmittedas a working group draft.¶

The draft underwent minor copy editing before publication.¶

Some but not all of the long delay in AUTH48 is due to clustering with[RFC8410].MISSREF state concluded on 2018-05-09 and the document re-entered AUTH48 atonce. AUTH48 lasted over two months after that. (For state definitions, see<https://www.rfc-editor.org/about/queue/#state_def>.)¶

The time after AUTH48 and before publication (3 weeks) partlyoverlaps with travel for IETF 102 and is partly due to coordinating thecluster.¶

3.2.RFC 8456

"Benchmarking Methodology for Software-Defined Networking (SDN) Controller Performance"[RFC8456]:¶

Status (Length):

Informational (64 pages)¶

Overview:

2 individual drafts; 9 WG drafts¶

First draft:

2015-03-23¶

WG adoption:

2015-10-18¶

Last Call start:

2018-01-19¶

IESG eval. start:

2018-02-27¶

IESG approved:

2018-05-25¶

AUTH48 start:

2018-08-31¶

AUTH48 complete:

2018-10-16¶

Published:

2018-10-30¶

The draft underwent extensive copy editing, covering use of articles, syntax, and word choice. The changes are enough to cause pagination differences. The "diff" tool marks prettymuch every page as changed. Some diagrams see change in protocol elements like message names.¶

According to the author, the experience of producing this document mirrors a typical one in theBenchmarking Methodologies Working Group (BMWG). There were multiple authors in multiple timezones, which slowed down the AUTH48 process somewhat, although the AUTH48 delay of 46 days is onlya bit longer than the average draft.¶

The RFC was part of cluster with[RFC8455].¶

BMWG publishes Informational RFCs centered around benchmarking,and the methodologies in RFC 8456 have been implemented in benchmarking products.¶

3.3.RFC 8446

"The Transport Layer Security (TLS) Protocol Version 1.3"[RFC8446], as the titleindicates, defines the new version of the TLS protocol. From the IETF Datatracker, we extractthe following:¶

Status (Length):

Proposed Standard (160 pages)¶

Overview:

29 WG drafts¶

First draft:

2014-04-17¶

Last Call start:

2018-02-15¶

IESG eval. start:

2018-03-02¶

IESG approved:

2018-03-21 (draft 28)¶

AUTH48 start:

2018-06-14¶

AUTH48 complete:

2018-08-10¶

Published:

2018-08-10¶

This draft started as a WG effort.¶

The RFC was a major effort in the IETF. Working group participants developed and testedseveral implementations. Researchers analyzed the specifications and performed formal verifications. Deployment tests outlined issues that caused extra workwhen the specification was almost ready. This complexity largely explains thetime spent in the working group.¶

Comparing the final draft to the published version, we find relatively light copyediting. It includes explaining acronyms on first use, clarifying some definitions standardizing punctuation and capitalization, and spelling out some numbers in text.This generally fall in the category of "style", although some of the clarificationsgo into message definitions. However, that simple analysis does not explain whythe AUTH48 phase took almost two months.¶

This document's AUTH48 process was part of the "GitHub experiment", which tried touse GitHub pull requests to track the AUTH48 changes and review comments. TheRFC Production Center (RPC) staff had to learn using GitHub for that process, and this required more workthan the usual RFC. The author and AD thoroughly reviewed each proposed edit, accepting some and rejecting some. The concern there was thatany change in a complex specification might affect a protocol that was extensivelyreviewed in the working group, but of course these reviews added time to theAUTH48 delays.¶

There are 21 implementations listedin the Wiki of the TLS 1.3 project[TLS13IMP]. It has been deployed on major browsers, andis already used in a large fraction of TLS connections.¶

3.4.RFC 8355

"Resiliency Use Cases in Source Packet Routing in Networking (SPRING) Networks"[RFC8355] is an Informational RFC.It originated from an informational use-case draft; it was mostly used for the BOF creating the WG, and then todrive initial work and evolutions from the WG.¶

Status (Length):

Informational (13 pages)¶

Overview:

2 individual drafts; 13 WG drafts¶

First draft:

2014-01-31¶

WG adoption:

2014-05-13¶

Last Call start:

2017-04-20¶

IESG eval. start:

2017-05-04 (draft 09)¶

IESG approved:

2017-12-19 (draft 12)¶

AUTH48 start:

2018-03-12¶

AUTH48 complete:

2018-03-27¶

Published:

2018-03-28¶

Minor set of copy edits, mostly for style.¶

No implementation of the RFC itself, but the technology behind it (such asSegment Routing Architecture[RFC8402] and TI-LFA[TI-LFA]) is widely implementedand deployment is ongoing.¶

According to participants in the discussion, the process of adoption of the source packet routingstandards was very contentious. The establishment of consensus at both the working group leveland the IETF level was difficult and time-consuming.¶

3.5.RFC 8441

"Bootstrapping WebSockets with HTTP/2"[RFC8441]¶

Status (Length):

Proposed Standard (8 pages)¶

Overview:

3 individual drafts; 8 WG drafts; Updates RFC 6455¶

First draft:

2017-10-15¶

WG adoption:

2017-12-19¶

Last Call start:

2018-05-07 (draft 05)¶

IESG eval. start:

2018-05-29 (draft 06)¶

IESG approved:

2018-06-18 (draft 07)¶

AUTH48 start:

2018-08-13¶

AUTH48 complete:

2018-09-15¶

Published:

2018-09-18¶

IANA action:

table entries¶

This RFC defines the support of WebSockets in HTTP/2, which is differentfrom the mechanism defined for HTTP/1.1 in[RFC6455]. The process wasrelatively straightforward, involving the usual type of discussions, someon details and some on important points.¶

Comparing the final draft and published RFC shows a minor set of copy edits,mostly for style. However, the author recalls a painful process. The RFCincludes many charts and graphs that were very difficult to formatcorrectly in the author's production process that involved conversionsfrom markdown to XML, and then from XML to text. The author had toget substantial help from the RFC Editor.¶

There are several implementations, including Firefox and Chrome,making RFC 8441 a very successful specification.¶

3.6.RFC 8324

"DNS Privacy, Authorization, Special Uses, Encoding, Characters, Matching, and Root Structure:Time for Another Look?"[RFC8324]. This is an opinion piece on DNS development,published on the Independent Stream.¶

Status (Length):

Informational (29 pages)¶

Overview:

5 individual drafts; Independent Stream¶

First draft:

2017-06-02¶

ISE review start:

2017-07-10 (draft 03)¶

IETF conflict review start:

2017-10-29¶

Approved:

2017-12-18 (draft 04)¶

AUTH48 start:

2018-01-29 (draft 05)¶

AUTH48 complete:

2018-02-26¶

Published:

2018-02-27¶

This RFC took only 9 months from first draft to publication, which is the shortest inthe 2018 sample set. In part, this is because the text was privately circulatedand reviewed by the ISE's selected experts before the first draft was published.The nature of the document isanother reason for the short delay. It is an opinion piece and does not requirethe same type of consensus building and reviews as a protocol specification.¶

Comparing the final draft and the published version shows only minor copy edits, mostlyfor style. According to the author, this is because he knows how to write in RFCstyle with the result that his documents often need a minimum of editing. He alsomakes sure that the document on which theRFC Production Center starts working already has changes discussedand approved during Last Call and IESG review incorporated,rather than expecting the Production Center to operate off ofnotes about changes to be made.¶

3.7.RFC 8377

"Transparent Interconnection of Lots of Links (TRILL): Multi-Topology"[RFC8377]¶

Status (Length):

Proposed Standard (20 pages)¶

Overview:

3 individual drafts; 7 WG drafts; Updates RFCs 6325 and 7177¶

First draft:

2013-09-03¶

WG adoption:

2015-09-01¶

Last Call start:

2018-02-19 (draft 05)¶

IESG eval. start:

2018-03-06 (draft 05)¶

IESG approved:

2018-03-12 (draft 06)¶

AUTH48 start:

2018-04-20 (draft 06)¶

AUTH48 complete:

2018-07-31¶

Published:

2018-07-31¶

IANA action:

table entries¶

Minor set of copy edits, mostly for style, also clarity.¶

3.8.RFC 8498

"A P-Served-User Header Field Parameter for an Originating Call Diversion (CDIV)Session Case in the Session Initiation Protocol (SIP)"[RFC8498].¶

Status (Length):

Informational (15 pages)¶

Overview:

5 individual drafts; 9 WG drafts¶

First draft:

2016-03-21¶

WG adoption:

2017-05-15¶

Last Call start:

2018-10-12 (draft 05)¶

IESG eval. start:

2018-11-28 (draft 07)¶

IESG approved:

2018-12-11 (draft 08)¶

AUTH48 start:

2019-01-28¶

AUTH48 complete:

2019-02-13¶

Published:

2019-02-14¶

IANA action:

table rows added.¶

Copy edits for style, but also clarification of ambiguous sentences.¶

3.9.RFC 8479

"Storing Validation Parameters in PKCS#8"[RFC8479]¶

Status (Length):

Informational (8 pages)¶

Overview:

5 individual drafts; Independent Stream¶

First draft:

2017-08-08¶

ISE review start:

2018-12-10 (draft 00)¶

IETF conflict review start:

2018-03-29¶

Approved:

2018-08-20 (draft 03)¶

AUTH48 start:

2018-09-20 (draft 04)¶

AUTH48 complete:

2018-09-25¶

Published:

2018-09-26¶

The goal of the draft was to document what thegnutls implementation was using for storing provably generated RSA keys.This is a short RFC that was published relatively quickly, althoughdiscussion between the author, the Independent Submissions Editor, and theIESG lasted several months. In the initial conflict review, the IESG askedthe ISE to not publish this document before IETF working groups hadan opportunity to pick up the work. The author met that requirement bya presentation to the SECDISPATCH WG during IETF 102. Since no WG wasinterested in picking up the work, the document progressed on theIndependent Stream.¶

Very minor set of copy edits, moving some references from normative to informative.¶

The author is not aware of other implementations than gnutls relying on this RFC.¶

3.10.RFC 8453

"Framework for Abstraction and Control of TE Networks (ACTN)"[RFC8453]¶

Status (Length):

Informational (42 pages)¶

Overview:

3 individual drafts; 16 WG drafts¶

First draft:

2015-06-15¶

WG adoption:

2016-07-15¶

Out of WG:

2018-01-26 (draft 11)¶

Expert review requested:

2018-02-13¶

Last Call start:

2018-04-16 (draft 13)¶

IESG eval. start:

2018-05-16 (draft 14)¶

IESG approved:

2018-06-01 (draft 15)¶

AUTH48 start:

2018-08-13¶

AUTH48 complete:

2018-08-20¶

Published:

2018-08-23¶

IANA action:

table rows added.¶

Minor copy editing.¶

3.11.RFC 8429

"Deprecate Triple-DES (3DES) and RC4 in Kerberos"[RFC8429]¶

Status (Length):

BCP (10 pages)¶

Overview:

6 WG drafts¶

First draft:

2017-05-01¶

Last Call start:

2017-07-16 (draft 03)¶

IESG eval. start:

2017-08-18 (draft 04)¶

IESG approved:

2018-05-25 (draft 05)¶

AUTH48 start:

2018-07-24¶

AUTH48 complete:

2018-10-31¶

Published:

2018-10-31¶

IANA action:

table rows added.¶

This draft started as a working group effort.¶

This RFC recommends deprecating two encryption algorithms that are now consideredobsolete and possibly broken. The document was sent back to the WG after the first Last Call,edited, and then there was a second Last Call. The delay from first draft to Working GroupLast Call was relatively short, but the number may be misleading. The initial draft was areplacement of a similar draft in the KITTEN Working Group, which stagnated for some timebefore the CURDLE Working Group took up the work. The deprecation of RC4 was somewhat contentious, but the WG had already debated thisprior to the production of this draft, and the draft was not delayed by this debate.¶

Most of the 280 days between IETF LC and IESG approval werebecause the IESG had to talk about whether this document should obsolete RFC 4757 ormove it to Historic status, and no one was really actively pushing thatdiscussion for a while.¶

The 99 days in AUTH48 are mostly because one of the authors was a sitting AD, and thoseduties ended up taking precedence over reviewing this document.¶

Minor copy editing, for style.¶

The implementation of the draft would be the actual removal of support for 3DES and RC4in major implementations. This is happening, but very slowly.¶

3.12.RFC 8312

"CUBIC for Fast Long-Distance Networks"[RFC8312]¶

Status (Length):

Informational (18 pages)¶

Overview:

2 individual drafts; 8 WG drafts¶

First draft:

2014-09-01¶

WG adoption:

2015-06-08¶

Last Call start:

2017-09-18 (draft 06)¶

IESG eval. start:

2017-10-04¶

IESG approved:

2017-11-14 (draft 07)¶

AUTH48 start:

2018-01-08¶

AUTH48 complete:

2018-02-07¶

Published:

2018-02-07¶

IANA action:

table rows added.¶

Minor copy editing, for style.¶

The TCP congestion control algorithm Cubic was first defined in 2005, was implementedin Linux soon after, and was implemented in major OSes after that. After some debatesfrom 2015 to 2015, the TCPM Working Group adopted the draft, with a goal ofdocumenting Cubic in the RFC Series. According to the authors, this was nota high-priority effort, as Cubic was already implemented in multiple OSesand documented in research papers. At some point, only one of the authorswas actively working on the draft. This may explain why another two years was spentprogressing the draft after adoption by the WG.¶

The RFC publication may or may not have triggered further implementations. Onthe other hand, several OSes picked up bug fixes from the draft and the RFC.¶

3.13.RFC 8492

"Secure Password Ciphersuites for Transport Layer Security (TLS)"[RFC8492]¶

Status (Length):

Informational (40 pages)¶

Overview:

10 individual drafts; 8 WG drafts; Independent Stream¶

First draft:

2012-09-07¶

Targeted to ISE:

2016-08-05¶

ISE review start:

2017-05-10 (draft 01)¶

IETF conflict review start:

2017-09-04¶

Approved:

2017-10-29 (draft 02)¶

AUTH48 start:

2018-10-19 (draft 05)¶

AUTH48 complete:

2019-02-19¶

Published:

2019-02-21¶

IANA action:

table rows added.¶

This RFC has a complex history. The first individual draft was submitted to theTLS Working Group on September 7, 2012. It progressed there, and was adoptedby the WG after 3 revisions. There were then 8 revisions in the TLS WG,until the WG decided to not progress it. The draft was parked in 2013 bythe WG chairs after failing to get consensus in WG Last Call. The AD finallypulled the plug in 2016, and the draft was then resubmitted to the ISE.¶

At that point, the author was busy and was treating this RFC with a low priority because, in his words, it would not be a "real RFC".There were problems with the draft that only came up late. In particular,it had to wait for a change in registry policy that only came about withthe publication of TLS 1.3, which caused the draft to be publishedafter RFC 8446, and also required adding references to TLS 1.3.The author also got a very late comment while in AUTH48 that caused some rewriting. Finally, there was some IANA issue with the extensionregistry where a similar extension was added by someone else. The draftwas changed to just use it.¶

Changes in AUTH48 include adding a reference to TLS 1.3, copy editing for style,some added requirements, added paragraphs, and changes in algorithms specification.¶

3.14.RFC 8378

"Signal-Free Locator/ID Separation Protocol (LISP) Multicast"[RFC8378] isan Experimental RFC, defining how to implement Multicast in the LISParchitecture.¶

Status (Length):

Experimental (21 pages)¶

Overview:

5 individual drafts; 10 WG drafts¶

First draft:

2014-02-28¶

WG adoption:

2015-12-21¶

Last Call start:

2018-02-13 (draft 07)¶

IESG eval. start:

2018-02-28 (draft 08)¶

IESG approved:

2018-03-12 (draft 09)¶

AUTH48 start:

2018-04-23¶

AUTH48 complete:

2018-05-02¶

Published:

2018-05-02¶

Preparing the RFC took more than 4 years. According to the authors, they werenot aggressively pushing it and just let the working group process decide to paceit. They also did implementations during that time.¶

Minor copy editing, for style.¶

The RFC was implemented by lispers.net and Cisco,and it was used in doing IPv6 multicast over IPv4 unicast/multicast at the Olympicsin PyeungChang. The plan is to work on a Proposed Standard once theexperiment concludes.¶

3.15.RFC 8361

"Transparent Interconnection of Lots of Links (TRILL):Centralized Replication for Active-Active Broadcast,Unknown Unicast, and Multicast (BUM) Traffic"[RFC8361]¶

Status (Length):

Proposed Standard (17 pages)¶

Overview:

3 individual drafts; 14 WG drafts¶

First draft:

2013-11-12¶

WG adoption:

2014-12-16¶

Last Call start:

2017-11-28 (draft 10)¶

IESG eval. start:

2017-12-18 (draft 11)¶

IESG approved:

2018-01-29 (draft 13)¶

AUTH48 start:

2018-03-09¶

AUTH48 complete:

2018-04-09¶

Published:

2018-04-12¶

According to the authors, the long delays in producing this RFC weredue to a slow uptake of the technology in the industry.¶

Minor copy editing, for style.¶

There was at least one partial implementation.¶

3.16.RFC 8472

"Transport Layer Security (TLS) Extension for Token Binding Protocol Negotiation"[RFC8472]¶

Status (Length):

Proposed Standard (8 pages)¶

Overview:

1 individual draft; 15 WG drafts¶

First draft:

2015-05-29¶

WG adoption:

2015-09-11¶

Last Call start:

2017-11-13 (draft 10)¶

IESG eval. start:

2018-03-19¶

IESG approved:

2018-07-20 (draft 14)¶

AUTH48 start:

2018-09-17¶

AUTH48 complete:

2018-09-25¶

Published:

2018-10-08¶

This is a pretty simple document, but it took over 3 years from individual draft to RFC. According tothe authors,the biggest setbacks occurred at the start: it took a while to find a home for this draft.It was presented in the TLS WG (because it's a TLS extension) and UTA WG (because it has to do withapplications using TLS). Then the ADs determined that a new WG was needed, so the authors had to workthrough the WG creation process, including running a BOF.¶

Minor copy editing, for style, with the addition of a reference to TLS 1.3.¶

Perhaps partially due to the delays, some of the implementers lost interest in supporting this RFC.¶

3.17.RFC 8471

"The Token Binding Protocol Version 1.0"[RFC8471]¶

Status (Length):

Proposed Standard (18 pages)¶

Overview:

1 individual draft; 19 WG drafts¶

First draft:

2014-10-13¶

WG adoption:

2015-03-15¶

Last Call start:

2017-11-13 (draft 16)¶

IESG eval. start:

2018-03-19¶

IESG approved:

2018-07-20 (draft 19)¶

AUTH48 start:

2018-09-17¶

AUTH48 complete:

2018-09-25¶

Published:

2018-10-08¶

This document presents a Token Binding Protocol for TLS. We can notice a period of 5 months before adoption of the draft by the WG. That explains in part the overall time of almost 4 years from first draft to publication.¶

Minor copy editing, for style.¶

The web references indicate adoption in multiple development projects.¶

3.18.RFC 8466

"A YANG Data Model for Layer 2 Virtual Private Network (L2VPN) Service Delivery"[RFC8466]¶

Status (Length):

Proposed Standard (158 pages)¶

Overview:

5 individual drafts; 11 WG drafts¶

First draft:

2016-09-01¶

WG adoption:

2017-02-26¶

Last Call start:

2018-02-21 (draft 07)¶

IESG eval. start:

2018-03-14 (draft 08)¶

IESG approved:

2018-06-25 (draft 10)¶

AUTH48 start:

2018-09-17¶

AUTH48 complete:

2018-10-09¶

Published:

2018-10-12¶

Copy editing for style and clarity, with also corrections to the YANG model.¶

3.19.RFC 8362

"OSPFv3 Link State Advertisement (LSA) Extensibility"[RFC8362] is a major extension to theOSPF protocol. It makes OSPFv3 fully extensible.¶

Status (Length):

Proposed Standard (33 pages)¶

Overview:

4 individual drafts; 24 WG drafts¶

First draft:

2013-02-17¶

WG adoption:

2013-10-15¶

Last Call start:

2017-12-19 (draft 19)¶

IESG eval. start:

2018-01-18 (draft 20)¶

IESG approved:

2018-01-29 (draft 23)¶

AUTH48 start:

2018-03-19¶

AUTH48 complete:

2018-03-30¶

Published:

2018-04-03¶

The specification was first submitted as an individual draft in the IPv6 WG, then moved to the OSPF WG.The long delay of producing this RFC is due to the complexity of the problem,and the need to wait for implementations. It is a very important change to OSPFthat makes OSPFv3 fully extensible. Since it was a non-backward compatible change,the developers started out with some very complex migration scenarios but ended upwith either legacy or extended OSPFv3 LSAs within an OSPFv3 routing domain. The initial attemptsto have a hybrid mode of operation with both legacy and extended LSAs also delayed implementationdue to the complexity.¶

Copy editing for style and clarity.¶

This specification either was or will be implemented by all the router vendors.¶

3.20.RFC 8468

"IPv4, IPv6, and IPv4-IPv6 Coexistence: Updates for the IPPerformance Metrics (IPPM) Framework"[RFC8468].¶

Status (Length):

Informational (15 pages)¶

Overview:

3 individual drafts; 7 WG drafts¶

First draft:

2015-08-06¶

WG adoption:

2016-07-04¶

Last Call start:

2018-04-11 (draft 04)¶

IESG eval. start:

2018-05-24 (draft 05)¶

IESG approved:

2018-07-10 (draft 06)¶

AUTH48 start:

2018-09-13¶

AUTH48 complete:

2018-11-05¶

Published:

2018-11-14¶

RFC 8468 was somehow special in thatthere was not a technical reason or interest that triggered it, butrather a formal requirement.While writing RFC 7312, the IP PerformanceMetrics (IPPM) Working Group realized that RFC 2330, the IP PerformanceMetrics Framework supported IPv4 onlyand explicitly excluded support for IPv6. Nevertheless, people usedthe metrics that were defined on top of RFC 2330 (and, therefore, IPv4only) for IPv6, too. Although the IPPM WG agreed that the work was needed, theinterest of IPPM attendees in progressing (and reading/reviewing) theIPv6 draft was limited. Resolving the IPv6 technical part wasstraightforward, but subsequently some people asked for a broader scope(topics like header compression, 6LoWPAN, etc.), and it took some time tofigure out and later on convince people that these topics are out of scope.The group also had to resolve contentious topics, for example, how tomeasure the processing of IPv6 extension headers, which is sometimes nonstandard.¶

The time in AUTH48 state for this document was longer than average. According to the authors,the main reasons include:¶

• Workload and travel caused by busy work periods of all coauthors¶
• Time zone difference between coauthors and editor (at least US, Europe, and India, not considering travel)¶
• RFC Production Center proposed and committed some unacceptable modifications that needed to be reverted¶
• Lengthy discussions on a new document title (required high effort andtook a long time, in particular reaching consensus between coauthorsand editor was time-consuming and involved the AD)¶
• RFC Production Center correctly identified some nits (obsoleted personal websites ofcoauthors) and coauthors attempting to fix them.¶

The differences between the final draft and the published RFC show copy editing for styleand clarity, but do not account for the back and forth between authors and editorsmentioned by the authors.¶

4.1.Delays from First Draft to RFC

We look at the distribution of delays between the submission of the firstdraft and the publication of the RFC, using the three milestones definedinSection 2.1: processing time in the working group, IETF processing time,and RFC production time. The following tableshows the number of days in each phase for the 20 RFCs in the sample:¶

The average delay from first draft to publication is about 3 years and 3 months, but thisvaries widely. Excluding the RFCs from the Independent Stream, the averagedelay from start to finish is 3 years and 4 months, of which on average2 years and 9 months are spent getting consensus in the working group,and 3 to 4 months each for IETF consensus and for RFC production.¶

The longest delay is found for[RFC8492], 6.5 years from start to finish.This is however a very special case -- a draft that was prepared forthe TLS Working Group and failed to reach consensus. After that, it wasresubmitted to the ISE, and incurred atypical production delays.¶

On average, we see that 80% of the delay is incurred in WG processing,10% in IETF review, and 10% for edition and publication.¶

For IETF Stream RFCs, it appears that the delays for Informational documentsare slightly shorter than those for protocol specifications, maybe six monthsshorter on average. However, there are lots of differences betweenindividual documents. The delays range from less than a year to more than 5 years forprotocol specifications, and from a year and 3 months to a bit more than 4 years forInformational documents.¶

We can compare the delays in the 2018 samples to those observed 10 years ago and 20 yearsbefore:¶

We can compare the median delay, and the delays observed by the fastest andslowest quartiles in the three years:¶

The IETF takes three to four times more to produce an RFC in 2018than it did in 1998, but about the same time as it did in 2008.We can get a rough estimate of how this translates in terms of"level of attention" per RFC by comparing the number of participantsin the IETF meetings of 2018, 2008, and 1998[IETFCOUNT] to the number of RFCspublished these years[RFCYEAR].¶

The last column in the table provides the ratio of average numberof participants to the number of RFCs published. If the IETF were a centralizedorganization, and if all participants and documents were equivalent, thisratio would be the number of participants dedicated to produce an RFCon a given year. This is of course a completely abstract figure becausenone of the hypotheses above are true, but it still gives a vagueindication of the "level of attention" applied to documents. We seethat this ratio has increased from 2008 to 2018, as the number ofparticipants was about the same for these two years but the number ofpublished RFCs decreased. However, this ratio was much higher in 1998.The IETF had many more participants, and there were probablymany more eyes available to review any given draft. If we applied theratios of 1998, the IETF would be producing 119 documents in 2018instead of 208.¶

4.2.Working Group Processing Time

The largest part of the delays is spent in the working groups, beforethe draft is submitted to the IESG for IETF review. As mentioned inSection 2.1, the only intermediate milestone that we can extractfrom the IETF Datatracker is the date at which the document wasadopted by the working group, or targeted for independent submission.The breakdown of the delays for the documents in our sample is:¶

The time before working group adoption averages to a bit more than 9 months,compared to 1 year and almost 10 months for processing time after adoption.We see that RFC 8492 stands out, with long delays spent attempting publication througha working group before submission to the Independent Submissions Editor. If we remove RFC8492 from the list, the average time until adoption drops to just over 7 months,and becomes just 25% of the total processing time in the WG.¶

There are a fewdocuments that started immediately as working group efforts, or were immediately targetedfor publication in the Independent Stream. Those documents tend to see short processing times,with the exception of RFC 8446 on which the TLS Working Group spent a long time working.¶

4.3.Preparation and Publication Delays

The preparation and publication delays include three components:¶

• the delay from submission to the RFC Editor to beginning of AUTH48, during which the document is prepared (referred to as "RFC edit" below);¶
• the AUTH48 delay, during which authors review and eventually approve thechanges proposed by the editors (referred to as "AUTH48" below);¶
• the publication delay, from final agreement by authors and editors toactual publication (referred to as "RFC Pub" below).¶

The breakdown of the publication delays for each RFC is shown in thefollowing table.¶

On average, the total delay appears to be about four months, but theaverage is skewed by the extreme values encountered for[RFC8492]. If weexclude that RFC from the computations, the average delay drops to a just a bitmore than 3 months: about 2 months for the preparation, a bit more than onemonth for the AUTH48 phase, and 5 days for the publishing.¶

Of course, these delays vary from RFC to RFC. To try explain the causes of thedelay, we compute the correlation factor between the observed delays and severalplausible explanation factors:¶

• the number of pages in the document,¶
• the amount of copy editing, as discussed inSection 4.4,¶
• whether or not IANA actions were required,¶
• the number of authors,¶
• the number of draft revisions,¶
• the working group delay.¶

We find the following values:¶

We see some plausible explanations for the production delay. It will be somewhat longer forlonger documents or for documents that require a lot of copy editing (seeSection 4.4).Somewhat surprisingly, it also tends to increase with the number of authors. It doesnot appear significantly correlated with the presence or absence of IANA action.¶

The analysis of RFC 8324 inSection 3.6 explains its shortediting delays by the experience of the author. This makes sense: if a documentneeds less editing, the editing delays would be shorter. This is partiallyconfirmed by the relation between the amount of copy editing and thepublication delay.¶

We see fewer plausible explanations for the AUTH48 delays. These delaysvary much more than the preparationdelay, with a standard deviation of 20 days for AUTH48 versus 10 days forthe preparation delay. In theory, AUTH48 is just a finalverification: the authors receive the document prepared by the RFC production center,and just have to give their approval, or maybe request a last minutecorrection. The name indicates that this is expected to last just two days, butin average it lasts more than a month.¶

We often hypothesize that thenumber of authors influences the AUTH48 delay, or that authors who have spenta long time working on the document in the working group somehow get demotivatedand spend even longer to answer questions during AUTH48. This may happensometimes, but our statistics don't show that - if anything, the numericalresults point in the opposite direction.¶

After asking the authors of the RFCs in the sample why the AUTH48 phase tooka long time, we got three explanations:¶

1. Some RFCs have multiple authors in multiple time zones. This slows down the coordination required for approving changes.¶
2. Some authors found some of the proposed changes unnecessary or undesirable, and asked that they be reversed. This required long exchanges between authors and editors.¶
3. Some authors were not giving high priority to AUTH48 responses.¶

As mentioned above, we were not able to verify these hypotheses by looking atthe data. The author's experience with this document suggests another potentialdelay for the Independent Stream RFC: processing delay by the IndependentSubmissions Editor, discussed inSection 4.5.¶

4.4.Copy Editing

We can assess the amount of copy editing applied to each published RFC bycomparing the text of the draft approved for publication and the text of theRFC. We do expect differences in the "boilerplate" and in the IANA section,but we will also see differences due to copy editing. Assessing the amountof copy editing is subjective, and we do it using a scale of 1 to 4:¶

1:

Minor editing¶

2:

Editing for style, such as capitalization, hyphens, "that" versus "which", and expanding all acronyms at least once.¶

3:

Editing for clarity in addition to style, such as rewriting ambiguous sentences and clarifying use of internal references. For YANG models, that may include model corrections suggested by the verifier.¶

4:

Extensive editing.¶

The following table shows that about half of the RFCs requiredediting for style, and the other half at least some editing for clarity.¶

This method of assessment does not take into accountthe number of changes proposed by the editors and eventually rejectedby the authors, since these changes are not present in either thefinal draft or the published RFC. It might be possible to getan evaluation of these "phantom changes" from the RFC Production Center.¶

4.5.Independent Stream

Out of 20 randomly selected RFCs, 3 were published through the Independent Stream.One is an independent opinion, another a description of a non-IETF protocolformat, and the third was[RFC8492], which is a special case. Apart fromthis special case, the publication delays were significantly shorter for the Independent Stream than for the IETF Stream.¶

The authors of these 3 RFCs are regular IETF contributors. Thisobservation motivated a secondary analysis of all the RFCspublished in the Independent Stream in 2018. There are 14 such RFCs:8507, 8494, 8493, 8492, 8483, 8479, 8433, 8409, 8374, 8369, 8367, 8351,8328, and 8324. (RFCs 8367 and 8369 werepublished on 1 April 2018.) The majority ofthe documents were published by regular IETF participants, buttwo of them were not. One describes "The BagIt File Packaging Format (V1.0)"[RFC8493], and the other the "Yeti DNS Testbed"[RFC8483]. Theydocument a data format and a system developed outside the IETF and illustratethe outreach function of the Independent Stream. In both cases, theauthors include one experienced IETF participant, who presumably helpedoutsiders navigate the publication process.¶

The present document experienced some publication delays due to the Independent Submissions Editor.The ISE is a bottleneck and is a volunteer resource. Although the ISE as a lone personoperating as a volunteer is still roughly adequate resource for thejob, the delivery will necessarily be best effort with delays causedby spikes in ISE load, work commitments, and other life events. Thesedelays may not be fundamentally critical to RFC delivery, but theyare capable of introducing a significant percentage delay into whatmight otherwise be a smooth process.¶

5.1.Citation Numbers

As measured on October 6, 2019, the citation counts for the RFC inour sample set were:¶

The results indicate that[RFC8446] is by far the most cited of the 20RFC in our sample. This is not surprising, since TLS is a key Internet Protocol.The TLS 1.3 protocol was also the subject of extensive studies by researchers,and thus was mentioned in a number of published papers. Surprisingly, the Semantic Scholar mentions a number of citations that predatethe publication date. These are probably citations of the various draftversions of the protocol.¶

The next most cited RFC in the sample is[RFC8312] which describes theCubic congestion control algorithm for TCP. That protocol was also thetarget of a large number of academic publications. The other RFCs in thesample only have a small number of citations.¶

There is probably a small bias when measuring citations at a fixed date.An RFC published in January 2018 would have more time to accrue citations thanone published in December. That may be true to some extent, as the second mostcited RFC in the set was published in January. However, the effect has to belimited. The most cited RFC was published in August, and the second most citedwas published in 2019. (That RFC got an RFC number in 2018, but publicationwas slowed by long AUTH48 delays.)¶

5.2.Comparison to 1998 and 2008

In order to get a baseline, we can look at the number of references for theRFCs published in 2008 and 1998. However, we need to take time into account.Documents published a long time ago are expected to have accrued more references.We try to address this by looking at three counts for each document: theoverall number of references over the document's lifetime, the number ofreferences obtained in the year following publication, and the number ofreferences observed since 2018:¶

We can compare the median number of citations and the numbers of citationsfor the least and most popular quartiles in the three years:¶

The total numbers show new documents with fewer citations than the older ones.This can be explained to some degree by the passage of time. If werestrict the analysis to the number of citations accrued in the year ofpublishing and the year after that, we still see about the same distributionfor the three samples.¶

We also see that the number of references to RFCs fades over time. Only themost popular of the RFC produced in 1998 are still cited in 2019.¶

5.3.Citations versus Deployments

The following table showsside by side the number of citations as measured inSection 5.1 andthe estimation of deployment as indicated inSection 3.¶

From looking at these results, it is fairly obvious that citation countscannot be used as proxies for the "value" of an RFC. In our sample, the twoRFCs that have high citation counts were both widely deployed, and can certainly be described as successful, but we also see many RFCs that saw significant deploymentwithout garnering a high level of citations.¶

Citation counts are driven by academic interest,but are only loosely correlated with actual deployment. We saw that[RFC8446]was widely cited in part because the standardization process involved manyresearchers, and that the high citation count of[RFC8312] islargely due to the academic interest in evaluating congestion control protocols.If we look at previous years, the most cited RFC in the 2008 sample is[RFC5326], anexperimental RFC defining security extensions to anexperimental delay tolerant transport protocol. This protocol does notcarry a significant proportion of Internet traffic, but has been the objectof a fair number of academic studies.¶

The citation process tends to privilege the first expression of a concept.We see that with the most cited RFC in the 1998 set is[RFC2267], an informationalRFC defining Network Ingress Filtering that was obsoleted in May2000 by[RFC2827]. It is still cited frequently in 2018 and2019, regardless of its formal status inthe RFC Series. We see the same effect at work with[RFC8441], whichgarners very few citations although it updates[RFC6455] that hasa large number of citations. The same goes for[RFC8468], which issparsely cited while the[RFC2330] is widely cited. Just counting citationswill not indicate whether developers still use an old specification orhave adopted the revised RFC.¶

5.4.Citations versus Web References

Web references might be another indicator of the popularity of an RFC.In order to evaluate these references, we list here the number of resultsreturned by searches on Google and Bing, looking for the search term "RFCnnnn"(e.g., "RFC8411"), and copying the number of results returned by thesearch engines. The table below presents the results of these searches,performed on April 4, 2020.¶

The result counts from Bing are sometimes surprising. Why would RFC 8441 gather59,500 web references? Looking at the results in detail, we find a mix of data.Some of them are logs of development projects implementing Web Sockets, whichis exactly what we are looking for, but others appear spurious. For example,a shop selling rugby jerseys is listed because its phone number ends with "8441".Other pages were listed because street numbers or product numbers matched theRFC number.The same type of collision may explain the large reference counts on Bing forRFCs 8377, 8498, 8479, 8453, 8429, 8378, and 8471. The result counts on Bingdo not appear to provide a good metric.¶

On Google, all RFCs garner at least a 250 references, largely because the wholeRFC catalog is replicated on a large number of web servers. Deviations from thatbaseline are largely correlated with the number of citations in the SemanticScholar, with a couple of exception: RFC 8441 and RFC 8471 garner morereferences than the low citation counts would predict. Looking at theresults, we find many references in development databases explaininghow these protocols are implemented in various code bases and open sourceprojects. This means that counting Google results would give some indicationabout an RFC's popularity, complementing the citation counts.¶

There are some practical problems in using the counts of Googleresults. Google searches are personalized, the results dependon the source of the queries, and the counts may vary as well. Thesearch results depend on the search algorithm, and there is no guaranteethat counts will not change when the algorithm changes. On the otherhand, the results do indicate that some of the RFCs in our sampleare being used by developers or in deployments.¶