Kernel documentation with Sphinx, part 1: how we got here

Benefits for LWN subscribers

The primary benefit fromsubscribing to LWN is helping to keep us publishing, but, beyond that, subscribers get immediate access to all site content and access to a number of extra site features. Please sign up today!

The last time LWN looked atformatted kernel documentationin January, it seemed like the merging ofAsciiDoc support for thekernel's structured source-code documentation ("kernel-doc") comments, wasimminent. As Jonathan Corbet, in the capacity of the kernel documentationmaintainer, wrote: "A good-enough solution that exists nowshould not be held up overly long in the hopes that vague ideas forsomething else might turn into real, working code." Sometimes,however, the threat that something not quite perfect might be mergedis enough to motivate people to turn those vague ideas into somethingreal.

In the end,Sphinx andreStructuredText areemerging as the future of Linux kernel documentation, with far moreambitious goals than the original AsciiDoc support patches ever had. Withthe bulk of the infrastructure work now merged to thedocs-nextbranch headed for v4.8, it's a good time to reflect on how this came to happen and give an overviewof the promising future of kernel documentation.

Background

The patches to support lightweight markup (initially usingMarkdown, laterAsciiDoc) in kernel-doc comments were borne out of a desire to write betterdocumentation for the graphics subsystem. One of the goals was to enhancethe in-source graphics subsystem internals documentationfor two main reasons. First, if the documentation is nextto the code it describes, the documentation has a better chance of beingupdated along with the code. Second, if the documentation can be written inplain text rather than DocBook XML, it's more likely to be written in the firstplace.

However, plain text proves to be just a little too plain when youventure beyond documenting functions and types, or if you want togenerate pretty HTML or PDF documents out of it. Adding support forlightweight markup in the kernel-doc comments was the natural thing todo. However, bolting this to the existing DocBook toolchain turned outto be problematic.

As part of the documentation build process, thescripts/kernel-doc script extracts the structured comments andemits them in DocBook format. Thekernel-doc script supports somestructure but fairly little formatting. To fit into this scheme, thelightweight markup support patches causedkernel-doc to invoke an external conversion tool(initiallypandoc, laterasciidoc) on each documentationcomment block to convert them from lightweight markup to DocBook. This waspainfully slow.

Doing the conversion inkernel-doc kept the DocBook pipelineside of things mostly intact and oblivious to any markup, but it addedanother point of failure in the already long and fragile path from commentsto HTML or PDF. Problems with markup and mismatches at each point of conversionmade debugging challenging. The tools involved were not designed towork together and often disagreed about when and how markup should beapplied.

It was clear that this was not the best solution, but at the time itworked and there was nothing else around.

AsciiDoc all-in, muddying the waters

Inspired by Jonathan's article and frustrated by the long documentationbuild times (we were testing the patches in the Intel graphics integrationtree), I had the idea to makekernel-doc output AsciiDocdirectly instead of DocBook. Converting the few structural features in thecomments to AsciiDoc and just passing through the rest was trivial;kernel-docalready supported several output formats with reasonable abstractions. Likemany ideas, this was the obvious thing to do—in retrospect. Suddenly, thisopened the door to writing all of the high-level documents underDocumentation/DocBook in AsciiDoc, embedding the documentationcomments at that level, and getting rid of the DocBook template filesaltogether. This has massive benefits, and Jonathan soon followed up with aproof-of-conceptthat did just that.

There was a little bit of excited buzz around this, with folksexploring, experimenting, and actually trying things out with documentconversion. A number of conversations between interested developers atlinux.conf.au seemed to further confirm that this was the path forward.But, just when it felt like people were settling on switching todoing everything in AsciiDoc, Jonathanmuddied thewaters by taking a hard look at Sphinx as an alternative toAsciiDoc.

Sphinx vs. AsciiDoc

Sphinx is a documentation generator that uses reStructuredText as itsmarkup language, extending and usingDocutils for parsing. BothSphinx and Docutils werecreated in Python to document Python, but documenting C and C++ is alsosupported. Sphinx supports several output formats directly, such as HTML,LaTeX, and ePub, and supports PDF output via either LaTeX or the externalrst2pdf tool.

TheAsciiDoc format, on the other hand, is semantically equivalent to DocBookXML, with the DocBook constructs expressed in terms of lightweightmarkup. AsciiDoc is easier for humans to read and write than XML, but sinceit is designed to translate to DocBook, it fits nicely in front of anexisting DocBook toolchain. The original Python AsciiDoc tool has beenaround for a long time, but has been superseded by a Ruby reimplementationcalledAsciidoctor in recentyears. As far as the AsciiDoc markup goes, Asciidoctor was designed tobe a drop-in replacement, but any extensions are implementation-specificdue to the change in implementation language. Both tools support HTML andDocBook output natively; other output formats are generated from DocBook.

When comparing the markup formats for the purposes of kerneldocumentation, only the table support, which is much needed for the mediasubsystem documentation in particular, was clearly identified as beingsuperior in AsciiDoc. Otherwise, the markup comparison was ratherdispassionate; it really boiled down to the tools themselves and, to someextent, which languages the tools were written in. Indeed, the markups andtools were not independent choices. All the lightweight markups have theirpros and cons.

Superficially, the implementation language of the tools shouldn't playany role in the decision. But it seemed that neither tool would workas-is, or at least we wouldn't be able to get their full potentialwithout extending the tools ourselves. In the kernel tree, there are notools written in Ruby, but there are plenty of tools written inPython. It was fairly easy to lean towards Sphinx in this regard.

If you are looking for flexibility, one great advantage of AsciiDoc isthat it's so closely tied to DocBook. By switching to AsciiDoc, thekernel documentation could reuse the existing DocBook toolchain. Thedownside is that AsciiDoc would add another step in front of the alreadyfragile DocBook toolchain.Dan Allen of Asciidoctor said: "One of thekey goals of the Asciidoctor project is to be able to directly produce awide variety of outputs from the same source (without DocBook)."However, this support isn't quite there yet.

The Asciidoctor project has a promising future. But Sphinx is stable,available now, and fits the needs of the kernel.GrantLikely summed it up this way: "Honestly, in the end I thinkwe could make either tool do what is needed of it. However, my impressionafter trying to do a document that needs to have nice publishable outputwith both tools is that Sphinx is easier to work with, simpler to extend,better supported."In the end, Jonathan's verdict was to go with Sphinx. The patches havebeen merged, and the first Sphinx-based documentation will appear in the4.8 kernel.

Thesecond and final part of this serieswill look into how the kernel's new Sphinx-based toolchain works and how to write documentation using it.