7.8 C
New York
Friday, November 25, 2022

Literate programming in Go – InfoWorld

InfoWorld |
During a recent company hackathon, my team’s charter was to improve the documentation for the Steampipe plugin SDK. Like other components of the Steampipe system, the plugin SDK is written in Go and published to pkg.go.dev. The version that existed when we started is here. As is typical, the documentation was an autogenerated catalog of functions and types. To explain how to use those functions and types, we provided guidance on the steampipe.io site.
Our hackathon challenge was to weave such guidance into the generated documentation. If you’re writing a Steampipe plugin, your list of tasks looks like this:
These tasks require that you use functions and types, but while comments attached to those functions and types can enhance the generated documentation, they’re too granular for the high-level exposition we aimed for. Searching for inspiration, Steampipe lead developer Kai Daguerre noticed that our top-level page lacked the overview section he saw in the documentation for pgx, a Go driver for Postgres.
That overview comes from https://github.com/jackc/pgx/blob/master/doc.go, which is one long comment (that uses Go comment syntax) followed by a package declaration.
So we added a doc.go at the top level of our repo to produce this overview.
We used the name doc.go because that seems to be conventional, but it could have been called foo.go. What’s salient is that it’s a valid Go file that belongs to a package. The package contains no code, only documentation. We wrote headers to create the sections of the overview, and in each section we described a plugin writer’s task using narrative, inline code examples, internal links to functions and types, and external links to examples elsewhere.
The ability to link within the code’s namespace was a revelation. To describe the task called Add hydrate functions, for example, we wrote this:
The section defined by the Add hydrate functions header is a link target: Add hydrate functions. And the bracketed items render as links to a type, plugin.HydrateFunc, and to a property, plugin.Column.Hydrate. This was starting to feel like wiki writing!
What we still lacked, though, was the ability to create new wiki pages where we could explain higher-level concepts. The overview was one place to do that, but we wanted to reserve that for narration of the plugin writer’s journey. Where could we discuss a concept like dynamic tables, an advanced feature that enables plugins like CSV which have no fixed schema and must define columns on the fly?
Kai realized that not only could we create new documentation-only packages for such topics, we could also import them so that their names were available for the same kind of shorthand linking we could do with, e.g., [plugin.HydrateFunc]. In the top-level doc.go he did this:
And in /docs/dynamic_tables/doc.go he did this:
Now we could write a sentence in a comment like this:
And the bracketed term autolinks just like any other name in the code’s namespace.
If that seems like more trouble than it’s worth, you can skip the import gymnastics and just use an external link:
Either way, the authoring experience now feels more fully wiki-like. You can create new pages as needed, and weave them into the hypertextual documentation that lives within the code base.
It was, admittedly, a bit of a struggle to use the Go system in the ways described here. The comment syntax is Markdown-like but frustratingly not Markdown; it depends on many implicit formatting conventions. If you go this route you’ll need a local previewing tool, and it’s not super-obvious that the one you want is not godoc but rather pkgsite. Had we discovered the command go install golang.org/x/pkgsite/cmd/pkgsite@latest we would have saved ourselves a lot of grief.
It isn’t! In his eponymous paper Knuth wrote:
To support this practice he invented a system called web whose components, tangle and weave, enabled the author of a program to tell its story in a language that mixed code (originally, Pascal) and documentation (originally, TeX) in a narrative-first way. Our modern ways of mixing code and documentation, and generating docs from embedded comments, only superficially resemble Knuth’s practice, as Mark Jason Dominus pointed out in his essay POD is not Literate Programming (2000).
And yet, now that our hackathon exercise has given me a taste of what code-as-wiki can be, it does feel like a step toward the storytelling approach that Knuth advocates. Recall that he named his system web before there was the web we now inhabit, never mind wikis! I can’t claim that we’re now literate programmers in the Knuthian sense, and I’ve always wondered if only a Knuthian mind could implement that original vision. But this way of improving the narrative quality of autogenerated docs does feel like a step in the right direction.
Jon Udell was once “blogger in chief” for InfoWorld and is now the community lead for steampipe.io.
Copyright © 2022 IDG Communications, Inc.
Copyright © 2022 IDG Communications, Inc.


Related Articles


Please enter your comment!
Please enter your name here

Latest Articles