The mind boggles. Now, here's Es: A Shell with Higher-Order Functions, written in 1993, and it has not spread like wildfire. It was even mentioned over at LtU some years ago.

On a related topic, Chung-chieh Shan has written about higher-order shell programs.

The CLiki domain is back up now. I was wondering whether we should leave it unavailable, as that would perhaps accelerate the work on an alternative. But who knows how long that would take… A quick grep over my files showed that it is cheaper if I just pay the emergency renewal fee than to find and replace all links to it.

Now we have one year to figure out a permanent solution so that the domain does not expire accidentally for the fourth year in a row.

UPDATE 2009-04-05: Just to be clear

Cliff Click (of HotSpot and other fame) writes about an email conversation between him, David Moon and Dan Weinreb.

No kidding:

I spent some time on updating the manual of CL-DOT. Initially, I had extracted it with DOCUMENTATION-TEMPLATE (plus some CL-Markdown integration) from CL-DOT's docstrings. A couple of things bothered me from the beginning:

• It was rather incomplete. That's what one gets from docstrings alone.
• No pictures of Dot-rendered graphs included either.
• It was HTML-only, and while the generated HTML conveyed the essentials, it was a little on the ugly side, and not easily stylable with CSS.
• When the code changed I had to adapt the documentation writing the same type of HTML, but this time without tool support (DT only extracts a starting point).

After looking around a bit what other Lisp projects use, I came to the conclusion that generating documentation from docstrings currently does not give me much advantage. I don't like to see docstrings in SLIME which are littered with random markup junk, nor do I particularly like having to build the software to generate the documentation.

So, why not doing it the other way around, i.e., generate docstrings from the documentation (basically by filtering out most markup)? Seems easy enough.

Then comes the question which documentation format to use? With something XML-based I have enough tools available to frob it any way I want to, but I'd rather not write XML myself. I have used markdown-like languages like AsciiDoc and Emacs Muse in other contexts, but they all seem to have some annoying limitations.

Eventually, I settled on Texinfo as the lesser evil. It can produce HTML, PDF, and Info documention, it seemed to support cross-references, generation of tables of contents and indices (which is quite useful for a reference manual), and can even deal with images. There is also a chance to get Hyperdoc support working. The algorithm to generate anchor names for cross-reference is documented.

Writing the markup is bearable, and in particular markup for function entries is straight-forward enough that for the moment I can do without an extractor for function signatures:

@anchor{print-graph}
@defun print-graph graph @keyword{&optional} stream
Prints a dot-format representation of @var{graph} (an instance of
@code{graph}, @pxref{graph, Classes}) to @var{stream}.
@end defun

Then the Struggling Began…

Converting the documentation to Texinfo format was actually done quite fast, however, the tweaking took some time. Some things are harder to achieve with Texinfo than I had, perhaps naïvely, expected:

• Anchors cannot be attached to some places where I want them (funny effects in various output formats).
• The Texinfo macro system is incredibly weak. The built-in @def… macros are not a good fit for CL. Your own @def… macros? No way.
• The cross-reference macros are rendered extremely verbose in TeX and also in Info output. References inside code or example sections need special-casing.
• No support for floating figures in TeX!?
• The HTML generated with makeinfo seems to be designed that it looks suitable in vintage 1995 browsers, and at the same time is maximally hard to style nicely with CSS.
• Menus get in the way. All the time. I learned the hard way not to use four-argument @node commands, and instead use the one-argument version and let makeinfo handle the rest.

Would I use Texinfo again? Not sure. I guess I'll stick with it for the time being, until something better comes along. I have accumulated enough boilerplate to reuse in another project, at least. That should alleviate some of the pain.

What about the Generation of Docstrings?

…is left as an exercise for the reader. After wrestling down Texinfo I ran out of steam, so makeinfo ‑‑plain and a little cut-and-pasting works for the moment and the workflow is not worse than the old way of generating documentation from docstrings.