I Published My Common Lisp Docstring Search Engine

My Common Lisp documentation search engine has been published to http://lisp-search.acceleration.net. In a previous post I wrote about using the montezuma full-text search engine to build an index of documentation available from within my common lisp runtime. I ended up going the extra mile on this one and indexing all of the documentation available for all of the packages in quicklisp (as well as readme files and other packages that sbcl had already loaded). The result is a 90M search index (4M tar.gz) that can be used search through all of the doc strings of all of the easily loadable packages.

The user interface is a bit clunky, searches don’t always return the most relevant results first, but it is live, fast, and seems already useful. Perhaps with some help from the internet, this search engine can reach its full potential. I named the software package that does this manifest-search-web, because it was inspired by gigamonkey’s manifest project. I still have not come up with a reasonable name for the published search engine (lisp-search seems a touch blasé and under-descriptive).

Hopefully, I will never again spend time writing a library only to find the already written, open source alternative after I publish mine. Also, perhaps this will inspire better doc-strings, now that doc-strings might be what leads to someone finding your project.

Other things todo:

  • Integrate manifest-search with slime
  • Have the documentation index be distributable in quicklisp (not sure how to do that efficiently)
  • Find a way to unify CLIKI, l1sp.org, lisp-search and other lisp documentation resources into a more cohesive single website / search
  • Improve the query language to ensure that it behaves according to user expectations as opposed to lucene expectations

As always, please report bugs and make suggestions for improvements. Cheers and happy lisping.

Manifest Search: A Common Lisp Documentation Search Engine

A common complaint from a co-worker is not being able to find relevant library functionality. We have libraries that do some tasks well, but if you haven’t used it before, how are you to know that it is there. More over, how do you find what you are looking for from all of the available utility libraries currently loaded.

After seeing Peter Seibel’s Manifest screencast. I was struck by the idea that you could index all the doc strings to provide a powerful search tool. I dont know about powerful yet, but this idea has turned into at least a search tool: Manifest-Search. This is the product of one days hacking and so should not be construed as the end-all-be-all common lisp search tool, however, it is at least a step in that direction.

I would like to eventually get this integrated more fully with both quicklisp and manifest, but that is all in the future. I think it would be amazing to search for functionality I need, and get documentation for a library I have not yet installed, but is distributed by quicklisp.

Access update – corrected SETF expanders and more examples

In the first released version of access I defined the setf versions as (defun (setf accesses) (new o &rest keys)…). In order to make this work out for plists and alists (where adding a key can result in a new HEAD element), I was forced to return the updated object rather than the “new” value that setf usually returns. I was unhappy with this oddity at the time but didn’t know directly how to fix it (obviously some macrology was in order to capture the “place” being modified).

Today I looked into the docs for define-set-expander and saw how to transform my code into “correct” setf’s. To do this i transformed my previous setf functions into set-access and set-accesses which return (values new-value possibly-new-object). I then define my setf expanders in terms of calling those functions and setting the place passed in to possibly-new-object. It took a little while to figure out and I’m still not entirely sure I wrote the optimal common lisp for this. However I was able to elide the outer setf from these expressions in the tests (setf pl (setf (access pl ‘one) ‘new-val)) and now the new (setf (access pl ‘one) ‘new-val) returns ‘new-val as would be expected.

There were some requests for more, better examples of where access might be useful:

  • My html components have a plist representing direct html attributes. I update these with (setf (accesses ctl ‘attributes ‘name) “myFormName”) and its correlary (accesses ctl #’attributes :name). Note that both forms work even though one uses a local symbol and one a keyword (they are compared by symbol-name so that I can think about it less). Also I am ok referring to the attributes function by name or function object (both will result in calling the attributes function on ctl).
  • Another example from the web domain: I often store a reference to a database object on the control that is responsible for displaying it. Thus getting the database primary key off of the data for a control can be (accesses client-form ‘data ‘adwolf-db:accountid). This allows me (where useful) to ignore the difference between an unsaved, new object and an object that hasn’t been created yet (for things like putting the id in the url, the difference is irrelevant).
  • While not currently implemented this way, my group-by library which groups items into nested alists or hashtable could potentially use access to handle the different implementations
  • Printing my database objects in debug / log messages, I want to output some columns (but only if the database object has those). This way I can define one printer for all my db objects with a minimum of fuss
    (defmethod print-object ((o clsql:mssql-db-object) (s stream))
    "Print the database object, and a couple of the most common identity slots."
    (print-unreadable-object (o s :type t :identity t)
    (iter (for c in '(id accountid serviceid transactionid title amount name))
    (for v = (access o c))
    (when v (format s "~A:~A " c v)))

In general I find access useful whenever I need to operate on some set of keys that may or may not exist in a dictionary-like object and I don’t care to receive any errors related to missing keys.


  • A keys / values interface to ease arbitrary dictionary iteration would be a worthy addition (alexandria seems to have all the relevant functions implemented, so it would mostly be a dispatch to those)
  • When a dictionary doesnt exist, there should be someway of telling it how to create that dictionary (currently you will get a plist).
  • Extensibility to allow support for other dict like structures.

New Common Lisp project: Access

Access is a common lisp library I just culled out of our immense utility mud ball and refactored into a library all its own. Access makes getting and setting values in common data structures support a single unified api. As such you could access a specific key from an alist stored in a hashtable stored in the slot of an object as (accesses o ‘k1 ‘k2 ‘k3). It also supports setting values (setf (accesses o ‘k1 ‘k2 ‘k3) “new-val”). Obviously there are some limitations to this approach, but for me, with my coding conventions, I don’t tend to run into them (see the README for details).

Access has removed some of my need for forms like (awhen a (awhen (fn1 it) (fn2 it))) with (access a ‘fn1 ‘fn2). To me, it allows me to more accurately express what I am trying to do while ignoring the vagaries of shifting implementation details. It also eases setting values in nested objects because it handles propagating the value up the chain rather than me having to do that myself (ie adding a new key-value pair to a the front of an alist stored in an object, automatically saves the new resulting alist in the object). I don’t expect that this is tasteful coding, but it is easier and allows me to not get mired down trying to decide if I want it to be an alist, plist, hashtable, or object because the cost to change it later is essentially zero.

Performance is rarely in issue in the apps that I tend to write. However, if it were, I would not use access as it does significant type and dispatch analysis that could be avoided by using the specific access functions of the data structure I am using.

A dot syntax familiar to those who use javascript/python/ruby type languages is available as well. This transforms calls like foo.bar.bast into (accesses foo ‘bar ‘bast). I don’t use this syntax as I tend to prefer the lisp function-call syntax, but it seems to be an oft requested / discussed feature, and I had fun writing the code.

Comparing Recursive-Regex and CL-YACC

In my previous blog post, I discussed how recursive-regex has been
maturing but that it still wasn’t and was not intended to compete with
actual parser toolkits. I wanted to quantify this assumption and
present a head-to-head analysis of using recursive-regex vs using
I chose cl-yacc because

I already had a working css3-selector parser implemented
. My
existing cl-yacc implementation is based on
the published
CSS3-Selectors Grammar
(but the modified some to get it to work).

CL-YACC Parser

I found implementing the parser in cl-yacc to be fairly tedious,
time consuming and error prone, even for such a small language as
css-selectors. It doesn’t help that I tried to do it using the
published css3 grammar and lex files which are somewhat awkward (eg:
open parens are in the lexer and close parens are in the grammar).

I wrote a not very well documented (f)lex file reader to read in the
existing CSS3 flex
and build a lexer compatible with cl-yacc. After getting a
valid lexer, I started working with the published grammar to convert
it to a format cl-yacc would approve of. Along the way I was finding
the syntax for cl-yacc to be pretty cumbersome, so I used some
read time execution to turn forms like the first below, into forms like
the second (macro expanded) one below which are valid cl-yacc
productions. (This is not a great idea or great code, but it did simplify
the parser def for me.)

   #.(rule (comb-sel :|,| spaces or-sel)
       (list :or comb-sel or-sel))
   #.(rule (comb-sel) comb-sel))

 (comb-sel :|,| spaces or-sel
  #'(lambda (comb-sel |,| spaces or-sel)
      (declare (ignorable comb-sel |,| spaces or-sel))
      (list :or comb-sel or-sel)))
  #'(lambda (comb-sel) (declare (ignorable comb-sel)) comb-sel)))

The difficulties I had in implementation were mostly concerning where
white space could appear, in which productions, at which levels in the
grammar. It seemed like it was quite a task to get an unambiguous
grammar and have all my extra unimportant white space everywhere still
be parsable. After I had finally managed to rid my grammar of
reduce/reduce conflicts, I was able to parse my language and it seemed
like a fairly peppy parser.

The only problem I really have with this implementation is that it
seems like it is totally illegible after the fact. Even knowing about
parsers and having written this one, I don’t feel comfortable
modifying the language. It seems like it would be difficult to get
working again. Thankfully, I don’t anticipate having to do much
language rewriting in this case.

CL-YACC parser definition (lexer not shown)

(yacc:define-parser *css3-selector-parser*
  (:start-symbol selector)
  (:terminals (:|,| :|*| :|)| :|(| :|>| :|+| :|~| :|:| :|[| :|]| :|=| :|-|
  (:precedence ((:left :|)| :s :|,| :|+| :|~| )) )
  (selector #.(rule (or-sel) or-sel))

   #.(rule (comb-sel :|,| spaces or-sel)
       (list :or comb-sel or-sel))
   #.(rule (comb-sel) comb-sel))
   #.(rule (and-sel combinator comb-sel)
       (list combinator and-sel comb-sel))
	 ;; need to handle trailing spaces here
	 ;; to avoid s/r
	 (and-sel spaces) and-sel))

   (:s (constantly :child))
   (spaces :|>| spaces (constantly :immediate-child))
   (spaces :|~| spaces (constantly :preceded-by))
   (spaces :|+| spaces (constantly :immediatly-preceded-by)))
   #.(rule (and-sel simple-selector)
       (list :and and-sel simple-selector))
   #.(rule (simple-selector) simple-selector))
   #.(rule (:HASH) `(:hash ,(but-first hash)))
   #.(rule (:CLASS) `(:class ,(but-first class)))
   #.(rule (:IDENT) `(:element ,ident))
   (:|*| (constantly :everything))
   (attrib #'identity)
   (pseudo #'identity))

   #.(rule (:|[| spaces :IDENT spaces :|]|)
       `(:attribute ,ident))
   #.(rule (:|[| spaces :IDENT spaces attrib-value-def spaces :|]|)
       `(:attribute ,ident ,attrib-value-def))

   #.(rule (attrib-match-type attrib-value)
       (list attrib-match-type attrib-value)))

   #.(rule (:|=|) :equals)
   #.(rule (:includes) :includes)
   #.(rule (:dashmatch) :dashmatch)
   #.(rule (:begins-with) :begins-with)
   #.(rule (:ends-with) :ends-with)
   #.(rule (:substring) :substring))

   #.(rule (:ident) ident)
   #.(rule (:string) (but-quotes string)))
   #.(rule (:|:| :IDENT) (list :pseudo ident))
   #.(rule (:|:| :FUNCTION spaces selector :|)|)
       (list :pseudo (but-last function) selector))
   #.(rule (:|:| :NTH-FUNCTION spaces nth-expr spaces :|)| )
       `(:nth-pseudo ,(but-last nth-function)

   #.(rule (:ident)
       (cond ((string-equal ident "even") (list 2 0))
	     ((string-equal ident "odd") (list 2 1))
	     (T (error "invalid nth subexpression"))))
   #.(rule (nth-sign :integer)
       (list 0 (if (string-equal nth-sign "-")
		   (* -1 (parse-integer integer))
		   (parse-integer integer))))
   #.(rule (nth-sign :integer :ident)
       (let (extra-num)
	   ((string-equal "n" ident) T)
	   ;; this is because our lexer will recogince n-1 as a valid ident
	   ;; but n+1 will hit the rule below
	   ((alexandria:starts-with-subseq "n" ident)
	    (setf extra-num (parse-integer (subseq ident 1))))
	   (T (error "invalid nth subexpression in (what is ~A)" ident)))
	 (list (or (if (string-equal nth-sign "-")
		       (* -1 (parse-integer integer))
		       (parse-integer integer))
	       (or extra-num 0))))
   #.(rule (nth-sign :integer :ident nth-sign :integer)
       (when (and integer-1 (null nth-sign-1))
	 (error "invalid nth subexpression 2n+1 style requires a sign before the second number"))
       (list (or (if (string-equal nth-sign-0 "-")
		     (* -1 (parse-integer integer-0))
		     (parse-integer integer-0))
	     (or (if (string-equal nth-sign-1 "-")
		     (* -1 (parse-integer integer-1))
		     (parse-integer integer-1))
    #.(rule (:|+|) :|+|)
    #.(rule (:|-|) :|-|)
    #.(rule () ()))
    (:S )
    ( )))

Recursive-Regex Parser

After having written recursive-regex, I wanted a way to beat some of
the bugs out as well as have a good example of what I could use this
tool for, and with what performance characteristics. To accomplish
this task, I converted the CL-YACC grammar and lex file into a single
parser definition and

wrote some code to turn that file into the recursive dispatch functions

REX: a recursive expression file format based (loosely) on lex

I had a really shoddy lexish reader for the existing lex based lexer.
I converted this to a similar tool (without the c blocks) for defining
named recursive expressions. These files have the .rex extension.
They consists of options, inline-definitions, and named productions.

The definitions for css-selectors are in css3.rex.

Once I had my recursive expression definition, getting the parser was
a fairly easy task. Along the way I added some code to minimize the
parse tree results by promoting children, when a parent only had a
single child that matched the same length as the parent. I also
improved the parse tracing, so that I could observer and debug what
the expression was doing while it was matching. With the tree minimized
I also had to revise many of my tests.

css3.rex: css3-selectors parser definition

%option case-insensitive

h => [0-9a-f]
nmstart => [a-z]
nmchar => [a-z0-9-]
nl => \n|\r\n|\r|\f
string1 => \"([\t !#$%&(-~]|\\{nl}|\')*\"
string2 => \'([\t !#$%&(-~]|\\{nl}|\")*\'
id => [-]?{nmstart}{nmchar}*
name => {nmchar}+
int => [0-9]+
num => [0-9]+|[0-9]*\.[0-9]+
string => {string1}|{string2}
url => ([!#$%&*-~])*
w => [ \t\r\n\f]*
s => [\s\f]+


comment => \/\*[^*]*\*+([^/][^*]*\*+)*\/
cdo => <!--
cdc => -->
class => \.(?<ident>)
hash => #(?<ident>{name})
ident => {id}
element => {id}
important_sym => !{w}important

selector => (?<or>)
or => (?<comb-sel>)(\s*,\s*(?<or>))?

comb-sel => (?<and>)((?<combinator>)(?<comb-sel>))?
combinator => (?<child>\s+)|(?<immediate-child>>)|(?<preceded-by>~)|(?<immediatly-preceded-by>\+)
and => (?<sim-sel>)(?<and>)?
sim-sel => (?<hash>)|(?<class>)|(?<element>)|\*|(?<attrib>)|(?<pseudo>)
attrib => \[\s*(?<ident>)\s*(?<attrib-val>)?\]
attrib-val => [\^\$\*\|\~]?=((?<ident>)|(?<string>))
pseudo => :(?<ident>)(?<parens>\s*(?<selector>)|(?<nth-expr>)\s*)?
nth-expr => even|odd|[\+-]?{int}(n([\+-]{int})?)?

Performance Numbers

These are the performance numbers for the two parsers each parsing 6
short inputs 1000 times. Also included is (a version) of that
output. (Recursive-expressions return CLOS object trees that have in
the results below been converted to a list representation for easy of
viewing.) As you can see the recursive-expressions version is ten
times slower and uses twenty times the memory.

(defun run-some-tests ()
(list ".foo" "#bar" ":bast" "div[onclick]"
":nth-last-child( 2n+1 ), foo.bar>bast:blech( .foo )" )

CSS> (run-some-tests)

;;; Recursive-Regex Results
Evaluation took:
7.548 seconds of real time
7.480000 seconds of total run time (6.850000 user, 0.630000 system)
[ Run times consist of 1.410 seconds GC time, and 6.070 seconds non-GC time. ]
99.10% CPU
18,821,417,333 processor cycles
1,167,530,480 bytes consed

((:CLASS ".foo" (:IDENT "foo")) (:HASH "#bar" (:IDENT "bar"))
(:PSEUDO ":bast" (:IDENT "bast"))
(:AND "div[onclick]" (:ELEMENT "div")
(:ATTRIB "[onclick]" (:IDENT "onclick")))
(:AND "div[onclick=foo]" (:ELEMENT "div")
(:ATTRIB "[onclick=foo]" (:IDENT "onclick")
(:ATTRIB-VAL "=foo" (:IDENT "foo"))))
(:OR ":nth-last-child( 2n+1 ), foo.bar>bast:blech( .foo )"
(:PSEUDO ":nth-last-child( 2n+1 )" (:IDENT "nth-last-child")
(:MATCHED-PARENS "( 2n+1 )" (:BODY "2n+1 " (:NTH-EXPR "2n+1"))))
(:COMB-SEL "foo.bar>bast:blech( .foo )"
(:AND "foo.bar" (:ELEMENT "foo") (:CLASS ".bar" (:IDENT "bar")))
(:AND "bast:blech( .foo )" (:ELEMENT "bast")
(:PSEUDO ":blech( .foo )" (:IDENT "blech")
(:MATCHED-PARENS "( .foo )"
(:BODY " .foo" (:CLASS ".foo" (:IDENT "foo")))))))))

;;; CL-YACC Results

Evaluation took:
0.790 seconds of real time
0.790000 seconds of total run time (0.770000 user, 0.020000 system)
[ Run times consist of 0.080 seconds GC time, and 0.710 seconds non-GC time. ]
100.00% CPU
1,968,753,765 processor cycles
64,695,792 bytes consed

((:CLASS "foo") (:HASH "bar") (:PSEUDO "bast")
(:AND (:ELEMENT "div") (:ATTRIBUTE "onclick"))
(:AND (:ELEMENT "div") (:ATTRIBUTE "onclick" (:EQUALS "foo")))
(:OR (:NTH-PSEUDO "nth-last-child" 2 1)
(:AND (:ELEMENT "bast") (:PSEUDO "blech" (:CLASS "foo"))))))


In conclusion, I am certainly not going to replace my working, fast,
memory efficient cl-yacc parser with my recursive-expressions parser.
However, if I wanted to have a working, legible (maybe) parser
definition, that will match as I intuitively expect, I might use
recursive-expressions. Because I am so used to using regex’s for
matching, if performance was not an issue, I would probably always
prefer the recursive expressions version. I could also see the
recursive expressions solution being a nice prototyping tool to help
develop the cl-yacc parser.

Obviously some of these opinions are going to be biased because
I wrote one of these libraries and not the other


  • Pretty quick and very memory efficient parser
  • Easy to customize parser output (each production has a lambda body to build whatever output is necessary)
  • Theoretically well grounded
  • Its hard to make unambiguous grammars
  • Not exceedingly helpful with its suggestions for how to fix your ambiguities

Recursive Regex

  • Relatively easy to get working
  • lexer and parser are the same tool, built on top of cl-ppcre, which
    you presumably already know, and has a good test environment (regex-coach, repl)
  • Parses ambiguous grammars
  • Has reasonable parser tracing built in, so debugging can be somewhat easier
  • Not very concerned with parse time or memory consumption
  • Parses ambiguous grammars
  • Bad pathological cases (exponential search)
  • Currently no architecture for modifying the parse tree other than an after
    the fact rewrite

Recursive-Regex Update

I think I have enough of the bugs worked out and enough tests now to actually recommend that others use the recursive-regex library up at my github.

There is a brief overview of the project in the introductory blog post. Also, in response to one of the comments on that blog post, there is an example s-exp parser as part of the test suite.

While this started as a toy, to scratch an intellectual itch, I think that this project is potentially a nice mid point between full blown parser frame work and regular expressions. Grammars are hard to get right though, so if you are writing your own language you might want to investigate something from the cliki parser generators page (eg: cl-yacc).

Introducing Recursive-Regex

Recursive-Regex is the end result of a weekend of playing with the code I published on Thursday about adding named dispatch functions to CL-PPCRE regular expressions. I kept at it and I think that this approach might have some promise for building up a library of reusable regexp/matcher chunks. I also found that this made it somewhat easier to obtain results from the regular expression search because I get back a full parse tree rather than the bindings typically supplied by CL-PPCRE.

I have it somewhat documented, loadable and testable, with all my current tests passing. There is even a recursive regex csv parser defined in the default dispatch table (mostly as a simple, but practical proof of concept).

Comma-List: [\t ]*(?:(?<body>[^,]*)[\t ]*,)*[\t ]*(?<body>[^,]*)[\t ]*
CSV-Row: (?<comma-list>((?<double-quotes>)|[^\n,]*))(?:\n|$)
CSV-File: (?<csv-row>)*

Double quotes and body both go to custom dispatcher functions. Body defines where the body regex should be matched and what to use if no body is supplied.

I don’t really have long term plans for this project, but it scratched an intellectual itch I was experiencing. Perhaps it will be useful for someone down the road.

Even More Fun With CL-PPCRE Filter Functions

A while ago I posted about my adventures playing with CL-PPCRE filter functions. In the previous blog post I destructively modify a cl-ppcre parse tree to add a filter function that can handle matching matched pairs of parentheses (a typical example of what regular expressions are NOT capable of). In this post I formalize that example into something that could be more broadly applied with less understanding of the underlying mechanics.

To begin with I define a function create-scanner-with-filters that will handle creating these special scanners for me. My idea is to provide a table of functions that should be called when we see certain strings inside of the regular expression. Because there are already named groups (see *allow-named-registers*) that can have parameters and that CL-PPCRE is already parsing for me, I decided to tie into the named registers to handle my function dispatching. This has the added niceness that whatever your filter matches is going to be stored in a register.

An over view of this process is: parse the regex, replace any named-register nodes’ (that have a function in the table) third element (usually a regex whose match will be stored in a register) with our specialized filter function, compile the new scanner and return that to the end user. I also decided that the regex that is the body of the named group should be available to the filter and in most cases should probably be used as part of the filter function.

If I continue to play with this, I might eventually release it as a library, but for now its stands well on its own.

Without further ado:

(declaim (optimize (debug 3)))

;; TODO: group binds in body expressions
;; TODO: propogate current scanner options to body scanners

(defun make-matched-pair-matcher (open-char close-char)
  "Will create a regex filter that can match arbitrary pairs of matched characters
   such as (start (other () some) end)"
  (lambda (body-regex)
    (setf body-regex (if (eql body-regex :void)
                          `(:SEQUENCE :START-ANCHOR ,body-regex :END-ANCHOR))))
    (lambda (pos)
      ;;(format T "TEST3 ~A ~A ~%" cl-ppcre::*reg-starts* cl-ppcre::*reg-ends*)
        (with fail = nil)
        (with start = pos)
        (with cnt = 0)
        (for c = (char cl-ppcre::*string* pos))
        (if (first-iteration-p)
            (unless (eql c open-char) (return fail))
            ;; went past the string without matching
            (when (>= pos (length cl-ppcre::*string*))
              (return fail)))
          ((eql c open-char) (incf cnt))
          ((eql c close-char)
           (decf cnt)
           (when (zerop cnt) ;; found our last matching char
             (if (or (null body-regex)
                     (cl-ppcre:scan body-regex cl-ppcre::*string*
                                    :start (+ 1 start)
                                    :end pos))
                 (return (+ 1 pos))
                 (return fail)))))
        (incf pos)))))

(defun default-dispatch-table ()
  "Creates a default dispatch table with a parens dispatcher that can match
   pairs of parentheses"
  `(("parens" . ,(make-matched-pair-matcher #\( #\) ))))

(defun create-scanner-with-filters
    (regex &optional (function-table (default-dispatch-table)) )
  "Allows named registers to refer to functions that should be in
   the place of the named register"
  (let* ((cl-ppcre:*allow-named-registers* T)
         (p-tree (cl-ppcre:parse-string regex)))
    (labels ((dispatcher? (name)
               "Return the name of the dispatcher from the table if
               (cdr (assoc name function-table :test #'string-equal)))
             (mutate-tree (tree)
               "Changes the scanner parse tree to include any filter
                functions specified in the table"
               (typecase tree
                 (null nil)
                 (atom tree)
                  (aif (and (eql :named-register (first tree))
                            (dispatcher? (second tree)))
                       `(:named-register (second tree)
                         (:filter ,(funcall it (third tree))))
                       (iter (for item in tree)
                         (collect (mutate-tree item))))))))
      ;; mutate the regex to contain our matcher functions
      ;; then compile it
      (cl-ppcre:create-scanner (mutate-tree p-tree)))))

(defparameter *example-function-phrase*
  "some times I like to \"function (calling all coppers (), another param (), test)\" just to see what happens")

(defun run-examples ()
  "Just runs some examples expected results:

   ((\"function (calling all coppers (), another param (), test)\"
     #(\"(calling all coppers (), another param (), test)\"))
    (\"function (calling all coppers (), another param (), test)\"
     #(\"(calling all coppers (), another param (), test)\"))

  (flet ((doit (regex)
             (create-scanner-with-filters regex)
   (doit #?r"function\s*(?<parens>)")
   (doit #?r"function\s*(?<parens>([^,]+,)*[^,]+)")
   (doit #?r"function\s*(?<parens>not-matching-at-all)"))))

PS. I don’t claim this is actually worth anything, only that I had fun doing it.

Data-Table and CL-CSV


I have quite a few database driven web applications that make heavy use of tabular imports and exports (from their primary database, other databases, and exterior data sources (eg: CSVs). This data structure provides column, row, and cell access to getting and setting values, as well as providing functionality to create composite data-tables by retrieving and combining subsections of existing data-tables. This library also aims to ease type coercion from strings to common-lisp types.


I had many scattered, not well tested, not easily runnable pieces of CSV code. I was unhappy with this situation, then decided to refactor all of this into a single project. I wrote tests for it and had a library so I thought I might release it. This project started as extensions and bugfixes on arnesi’s CSV.

I then looked around and saw there are other CSV libraries out there that probably mostly accomplished what I had set out to do. However, I already had my code that was tested, had an easier license (BSD), and provided a framework to interact with my other libraries and systems, so I figured why not just release it anyway.

The only interesting code in this library (to me) is that I managed to make the read/write-csv functions accept a string, pathname, or stream as the first argument and I managed to make sure that streams get closed if these functions created them (file streams for example), but not if the stream was passed in. Nothing great, but I had fun writing it.

Other niceties I would like to continue to build out in this library is its integration with other related libs (like CLSQL). I have code to handle exporting database queries as CSVs as well as code to handle importing CSVs into databases both serially and in bulk. I also use data-tables to have a lisp representation of the just parsed data-table and to coerce that table of string values into relevant common-lisp types.