Did you ever want to recreate the source form of a package, to see what state it is in currently? With defpackage-form, you can!

Then again, I seem to fiddle too much with packages lately. In particular, currently I am experimenting with a new work flow of Lisp package management.

Package Forms during Development

When writing new Lisp code, I start with just the essentials of a new package in my current Emacs buffer:

(defpackage #:foo
  (:use #:cl))

(This is basically what I get with C-x C-r P RET RET RET.)

During development, I find it distracting to remind myself of keeping the package form up-to-date. When I decide to import new symbols or shadow others I manipulate the package object directly, instead of updating the above DEFPACKAGE form and reevaluating it: (import 'bar:baz), (shadow 'qux), (use-package 'cl-fred), etc.. All this can be automated with a few editor key bindings, and the effects are visible in my working environment immediately.

At this point, I seldomly bother with export lists, because I might change names around later on, and also I am working mostly in the package.

At the end of the hacking session, I can just evaluate (defpackage-form 'foo) to get the current state of package FOO conveniently as a DEFPACKAGE form, which I can then use to replace the initial stub. Eventually, if the code grows big enough to split it up, I move the package form to a separate file packages.lisp.

At least in the first iteration, the export list is likely still incomplete, so I can choose to include all symbols whose home package is the package I am working on, nicely sorted by name. Then I remove those symbols which are not meant for exporting. Like that, it is easy to bulk export many symbols at once. As a common convention, symbols starting with % are meant to be internal. They can be omitted from the export list automatically.

(defun pprint-defpackage (stream defpackage-form)
  (format stream "~:<~W~^ ~3I~:_~W~^~1I~@{~:@_~:<~W~^ ~:I~@_~@{~W~^ ~_~}~:>~}~:>"
          defpackage-form))

(set-pprint-dispatch '(cons (member defpackage))
                     'pprint-defpackage)

Incremental Updates

Incremental updates of package forms are a little trickier, though. The generated form could, for example, be compared to the original with diff-sexp. I have not automated that stage much yet. If there are only few additions, it is probably easier to just add them manually.

Otherwise, I could imagine an Emacs before-save-hook which checks whether the export list of a package matches the DEFPACKAGE form in packages.lisp (or wherever it is stored—this could be figured out with source locations), and some more automation and integration with Emacs.

Very unfortunately, some loss of information occurs on the round trip from package form to object and back:

• Reader conditionals inside DEFPACKAGE forms cannot be recreated easily. Neither can comments, for what it's worth.
• There appears to be no way to figure out which package a symbol was actually imported from. For example, if we export CL:NIL from package FOO and then import FOO:NIL into some other package BAR, it will appear the same as if CL:NIL was imported directly into BAR.
• The default :USE list (i.e., when none is specified) is implementation dependent, which naturally can cause some implementation dependent packages to show up in this list.

Dozsa et al., "How Lisp Systems Look Different"

In this paper we propose a suite of new visualizations that reveal the special traits of the Lisp language and thus help in understanding complex Lisp systems. To validate our approach we apply them on several large Lisp case studies, and summarize our experience in terms of a series of recurring visual patterns that we have detected.

One of the case studies is SBCL.

It amused me for while that I found myself writing something similar to the following piece of code:

(defun %apply (function arg &rest args)
  (apply #'apply function arg args))

(defun %funcall (function &rest args)
  (%apply function args))

I will leave it to you, dear reader, to figure out what the double APPLY is good for. Surprisingly, there is even some reason behind this madness, namely the ability to mangle the function argument before calling.

There seems to be an interesting attraction between Erlang and Lisp and several times it has been tried to marry them, in different ways. Bill Clementson wrote about it already in his article Concurrent/Parallel Programming - The Next Generation. Here is an updated list:

• ETOS, an Erlang to (Gambit) Scheme compiler, dating from 1997, was probably the first attempt to combine Erlang and Lisp. One of the things that came out of ETOS is Marc Feeley's paper A Case for the Unified Heap Approach to Erlang Memory Management.
• Erlisp was another attempt, this time in Common Lisp. It's a (partial) reimplementation of the Erlang concurrency model. I don't think anybody is working actively on it at the moment.
• CL-MUPROC is a Common Lisp library which strives to offer some of the multiprocessing abstractions found in Erlang, very much in the same way as Erlisp. It implements some of Erlang's fault-tolerance mechanisms, but not yet distributed operations.
• Distel, a distributed Emacs Lisp with Erlang-style processes and message passing, and the Erlang distribution protocol, originally developed by Luke Gorrie of LtU, SLIME, OLPC, and other fame. Some time ago, Bill Clementson wrote a nice article about Distel.
• Termite, another Erlang-like distributed programming system, this time written in Gambit-C Scheme.
• Lisp Flavoured Erlang (LFE) is a Lisp syntax front-end to the Erlang compiler by Robert Virding (one of the inventors of Erlang). This is an interesting one, because it comes out of the Erlang community for a change, and instead of porting Erlang ideas to Lisp, it goes the other route: porting Lisp ideas to Erlang.
• Erlang-in-Lisp, a Google LispNYC Summer of Code 2008 project, again very much in the spirit of Erlisp, it seems. The project is in its early stages, let's see how far this goes.
• Erlang Server, Common Lisp Client by Berlin Brown appears to be the latest attempt of integrating Erlang and Lisp. They are connected via sockets and exchange messages in a custom protocol. This has the advantage that either side can easily be replaced by a different implementation.
• erlang-scheme is a port of Distel from ELisp to Scheme.

And this list does not even include projects like (again defunct?) Kali Scheme, which are clearly related.

Personally, I think that the reimplementation approach will have a tough stance against integration approaches like Distel. They lock out either one or the other of the two language eco-systems: libraries, development tools, etc., and recreating this is a lot of work (but don't let that stop you!) With Distel, I can choose to program parts of the application in ELisp or Erlang, whatever is a better fit.

Distel implements Erlang's on-the-wire protocol, which is nice because there is no need to mess around with a Foreign Function Interface. Alternatively, one could bind to the Erlang C libraries. As far as rapid prototyping is concerned, this should be the fastest and most straight-forward approach. I wonder why everybody is doing it the hard way (reimplementation)?

UPDATE 2008-07-06: Forgot ETOS

UPDATE 2008-07-06: Erlang Interoperability

UPDATE 2008-07-10: PLT Scheme meets Erlang

Recently, I made good use of Michael Parker's CLAWK, a Lisp-embedded variant of AWK.

So far, I had used the real AWK, along with perl and other parts of the Unix toolbox to analyze data from experiments, and munge it into HTML and LaTeX tables. However, this time I expected the experiments to be carried out in a run–tweak–rerun fashion with several iterations, and I did not want to reparse several hundred megabytes each time, just for a change in table layout, or for adding another analysis. (Also, I had not much time for lispy things recently, so this was a good way to sneak some Lisp back into my current C++ hell...)

Enter CLAWK. The following function parses my benchmark data into Lisp objects:

(defvar *foo-table* (make-hash-table :test 'equal))

(defawk parse-foo-benchmark (&aux model)
  (#/^memtime/
   (when model
     (emit-model model *foo-table*))
   (let ((path (parse-namestring $6)))
     (setf model (make-instance 'foo-model-record :filename path))))
  (#/^Instantiator: Explored/
   (setf (get-states model) $#3
         (get-transitions model) $#6
         (get-bfs-levels model) $#9
         (completed? model) t))
  ((string= $6 "elapsed")
   (setf (get-generation-time model) $#5))
  ((string= $13 "RSS")
   (setf (get-generation-memory model) (parse-integer $15 :junk-allowed t)))
  (END
   (when model
     (emit-model model *foo-table*))
   *foo-table*))

Parsing the bulk of my data with the above function takes about 80 seconds, probably slower than it would be with AWK. However, once parsed I have all the data available at my fingertips inside the Lisp image. I can prod it with the SLIME inspector, identify outliers, and play around with it in the REPL until I am satisfied. In addition, I can selectively rerun parts of the experiments, and parse just the newly produced output to update the in-memory representation, again in no time flat. Beats the Unix everything is a byte stream way every day of the week.

Rendering the results as HTML is a breeze with CL-WHO. The same holds for reordering columns, marking interesting entries programmatically, cross-referencing with other entries (or earlier versions of the data) and refining the analyses as much as I wish, with instant feedback.

Unfortunately, CLAWK has acquired some bitrot since its release in 2002 (?). I needed to tweak it slightly to make it compile (only tested with SBCL). When I tried to contact Michael Parker, his email bounced, so I decided to put up a patched version locally until the changes are folded back into his distribution. In addition, CLAWK is now ASDF-installable (along with its dependency, REGEX), courtesy of Redshank's ASDF Defsystem skeleton.

If time permits, I will fix up the code some more to get rid of the warnings, and perhaps allow CL-PPCRE as alternative regular expression engine. However, patches from the open-source fairies are very welcome, too.