Recently, I made good use of Michael Parker's CLAWK, a Lisp-embedded variant of AWK.

So far, I had used the real AWK, along with perl and other parts of the Unix toolbox to analyze data from experiments, and munge it into HTML and LaTeX tables. However, this time I expected the experiments to be carried out in a run–tweak–rerun fashion with several iterations, and I did not want to reparse several hundred megabytes each time, just for a change in table layout, or for adding another analysis. (Also, I had not much time for lispy things recently, so this was a good way to sneak some Lisp back into my current C++ hell...)

Enter CLAWK. The following function parses my benchmark data into Lisp objects:

(defvar *foo-table* (make-hash-table :test 'equal))

(defawk parse-foo-benchmark (&aux model)
  (#/^memtime/
   (when model
     (emit-model model *foo-table*))
   (let ((path (parse-namestring $6)))
     (setf model (make-instance 'foo-model-record :filename path))))
  (#/^Instantiator: Explored/
   (setf (get-states model) $#3
         (get-transitions model) $#6
         (get-bfs-levels model) $#9
         (completed? model) t))
  ((string= $6 "elapsed")
   (setf (get-generation-time model) $#5))
  ((string= $13 "RSS")
   (setf (get-generation-memory model) (parse-integer $15 :junk-allowed t)))
  (END
   (when model
     (emit-model model *foo-table*))
   *foo-table*))

Parsing the bulk of my data with the above function takes about 80 seconds, probably slower than it would be with AWK. However, once parsed I have all the data available at my fingertips inside the Lisp image. I can prod it with the SLIME inspector, identify outliers, and play around with it in the REPL until I am satisfied. In addition, I can selectively rerun parts of the experiments, and parse just the newly produced output to update the in-memory representation, again in no time flat. Beats the Unix everything is a byte stream way every day of the week.

Rendering the results as HTML is a breeze with CL-WHO. The same holds for reordering columns, marking interesting entries programmatically, cross-referencing with other entries (or earlier versions of the data) and refining the analyses as much as I wish, with instant feedback.

Unfortunately, CLAWK has acquired some bitrot since its release in 2002 (?). I needed to tweak it slightly to make it compile (only tested with SBCL). When I tried to contact Michael Parker, his email bounced, so I decided to put up a patched version locally until the changes are folded back into his distribution. In addition, CLAWK is now ASDF-installable (along with its dependency, REGEX), courtesy of Redshank's ASDF Defsystem skeleton.

If time permits, I will fix up the code some more to get rid of the warnings, and perhaps allow CL-PPCRE as alternative regular expression engine. However, patches from the open-source fairies are very welcome, too.

In best billc style... ;^)
I have mentioned some of my Emacs hacks for editing Common Lisp code here, here, and here (the latter of which Troels Henriksen ported to Climacs). So, I finally dusted off some more code-wrangling Emacs macros and bundled them up in the Redshank minor mode. Obligatory screencast:

The provided functionality is quite embryonic still, but I expect it to continue to grow.

For actual code motion and manipulation of Lisp code, I heavily rely on the excellent Paredit mode, with a minimal dependency on SLIME (the dependency might grow in the future, as more sophisticated refactoring likely requires cooperation from a running Lisp.)

For laughs comparison, I'd like to see the same done for unparenthesized code...
Okay, enough smugness for today. Now read the best Common Lisp Tutorial, and start hacking away!

First Reactions after Prerelease to the Raving Masses a Priviledged Audience

antifuchs | michaelw: o_O this is awesome

     jmbr | michaelw: I've just downloaded redshank. It's quite cool!

 michaelw | pkhuong: eh, okay ECONTEXT :) defclass-skeleton now
            inserts parenthesis balanced
  pkhuong | michaelw: ah good, good! I might have to try it out then
 michaelw | (bonus: customizable accessor style)
  pkhuong | so I can have foo-of instead of get-foo too? That
          | was my other objection (:

     Xach | I am eagerly awaiting the Planet Lisp link!

p.s.: In the right window of the screencast, key strokes were recorded with mwe-log-commands.

p.p.s: The screencasting itself was a major pain. In this case, I am willing to put part of the blame on my inexperience, for the rest I blame the tools I used. A somewhat more detailed account of this may appear under the Rants section.

Somebody in comp.lang.lisp asked about a DEFCLASS skeleton, after I showed a DEFPACKAGE skeleton earlier. So, here it is:

(define-skeleton mwe:cl-defclass-skeleton
  "Inserts a Common Lisp DEFCLASS skeleton."
  "Class: "
  "(defclass " str " (" ((skeleton-read "Superclass: ") str " ") & -1 ")"
  \n "(" ((skeleton-read "Slot: ")
          "(" str " :accessor get-" str " :initarg :" str ")" \n) & '(join-line)
  ")"
  ;; \n "(:default-initargs " - ")" ;; add to your liking...
  ")\n" \n
  _)

(define-skeleton mwe:cl-defclass-slot-skeleton
  "Inserts a Common Lisp DEFCLASS slot skeleton."
  "Slot: "
  ((skeleton-read "Slot: ")
   "(" str " :accessor get-" str " :initarg :" str ")" \n) & '(join-line)
  _)

I have not felt a need for it so far, mostly because it turns out that my DEFCLASS forms rarely are that regular, and tend to grow iteratively, so I have to go back and change them anyway. Using skeletons then feels to me like interrupting the flow, for lack of a better explanation...

Also, I would rather have somebody step forward and publish their msf-abbrev abbreviations for Common Lisp. Thanks.

UPDATE 2008-01-08: Old News...

Some time ago, Xach mentioned his Elisp snippet which inserts DEFPACKAGE forms into the current buffer.

I finally beefed up my version a little, so that it uses Emacs' buffer filename as default package name. For quick dabbling, visiting a new file and typing RET RET RET is enough. Otherwise, packages to be USEd can be autocompleted from the lists of loaded packages, thanks to SLIME.

Here goes:

(define-skeleton mwe:cl-defpackage-skeleton
  "Inserts a Common Lisp DEFPACKAGE skeleton."
  (skeleton-read "Package: " (if v1
                                 (file-name-sans-extension
                                  (file-name-nondirectory
                                   (buffer-file-name)))))
  (if (setq v1 (bobp)) ";;; -*- Mode:Lisp; Syntax:ANSI-Common-Lisp;")
  & (if buffer-file-coding-system
        (concat " Coding:"
                (symbol-name 
                 (coding-system-get buffer-file-coding-system 
                                    'mime-charset))))
  & " -*-"
  & \n
  & \n "(defpackage #:" str
  \n "(:nicknames" ("Nickname: " " #:" str) & ")" | '(kill-whole-line -1)
  \n "(:use #:CL" ((slime-read-package-name "USEd package: ") " #:" str) ")"
  ")" \n
  \n
  (if v1 "(in-package #:") & str & ")" & \n &
  \n
  _)

Works with skeleton mode from GNU Emacs (version 22.0.92). First-time users of autoinsert might want to use the following additional setup:

(eval-after-load 'autoinsert
  '(progn
     (push '(lisp-mode . mwe:cl-defpackage-skeleton) auto-insert-alist)
     (auto-insert-mode +1)))

(require 'autoinsert)

It is also possible to just M-x mwe:cl-defpackage-skeleton. If done somewhere else than at the beginning of a buffer, the comment and IN-PACKAGE form is not generated (useful for a packages.lisp file containing multiple package forms.)

UPDATE 2008-01-08: Old News...

Andreas Fuchs wrote a visualization tool for string search. I could not resist to add the Knuth-Morris-Pratt algorithm, given that it was just a handful code for the algorithm itself, the visualization then comes for free, thanks to Andreas' preparatory work.

That reminded me, however, that I have a KMP implementation lying around. The KMP algorithm is easy to implement, and works well with streams:

KMP> (with-input-from-string (s "12345AaAaB67890")
       (let ((ts (make-array 10 :element-type 'character
                             :adjustable t :fill-pointer 0)))
         (do-delimited-stream
             ((token expected (read-char s)
                     (multiple-value-call #'values ts (read-line s nil)))
              "AAB"           ; restart-vector generated automatically
              :predicate #'char-equal)
           (vector-push-extend token ts)
           (format *trace-output* "~&Expected ~A, got ~A" expected token))))
Expected A, got 1
Expected A, got 2
Expected A, got 3
Expected A, got 4
Expected A, got 5
Expected A, got A
Expected A, got a
Expected B, got A
Expected B, got a
Expected B, got B
"12345AaAaB"
"67890"
T
KMP> 

KMP works in a way that it does not have to backtrack on the stream after a partial match, unlike a naïve algorithm. Neither does it store parts of the stream.

I used it to read from a stream until the given delimiter is reached, and save everything upto and including the delimiter for further processing. Since it does not seize control of the stream I can, for example, process escape sequences or skip parts of the stream, before I hand stream elements to DO-DELIMITED-STREAM.