CLSQL & Webapps

A commonly experienced error when using CLSQL in a web environment is database connections conflicting with each other from simultaneous web requests. These problems arise because, by default, clsql standard-db-objects keep a reference to the connection they were queried / created from and reuse this database connection (rather than a new one you may have provided with clsql-sys:with-database). This means that two separate threads could try to use the same database connection (provided through clsql-sys:with-database or by having objects queried from the same connection accessed in multiple threads / http requests).

We solved this problem by introducing a clsql-sys::choose-database-for-instance method (available in clsql master branch from http://git.b9.com/clsql.git. (This branch will eventually be released as CLSQL6) Then in our web applications we define the following class and method override. Usually I then pass this name to clsql-orm or as a direct superclass to any of my web def-view-classes. After this, I just use with-database to establish dynamic connection bindings and everything pretty much works out (as these dynamic bindings are not shared across threads).

(defclass clsql::web-db-obj (clsql-sys:standard-db-object)
    nil
    (:metaclass clsql-sys::standard-db-class))

(defmethod clsql-sys::choose-database-for-instance
    ((object clsql::web-db-obj) &optional database)
  (or database clsql-sys:*default-database*))

(clsql-sys:def-view-class table-1 (clsql::web-db-obj)
    (...))

(clsql-orm:gen-view-classes
 :package :net.company.my.db
 :nicknames :my-db
 :export-symbols t
 :classes '(users employees salaries)
 :inherits-from '(clsql::web-db-obj))

Note: CLSQL-Fluid seems to be trying to accomplish much the same goals.

Collectors – A common lisp collection library

Collectors is a common lisp library to help accumulate values, that I just pushed to my github account.

Sometimes you just want to collect a list of things. Actually I need to do this all the time. Usually I end up iterating over something in which case Iterate‘s collecting/appending/unioning clauses serve me well. Sometimes though, that just is not a good fit, or I need to accumulate in places iterate deems unacceptable. In these cases it is nice to have specific collector macros. These setup an environment where a function is available that when called with an argument collects it, or when called without arguments returns the results of the collection.

These macros started as a piece of arnesi, but have been modified and added to. It is also nice to be able to include a library for a specific functionality set rather than a bag of semi-related, useful things.

CL-Inflector

CL-Inflector is a branch of a port of ruby/ActiveRecord’s inflector class to make it easier singularize and pluralize english words. The original author didn’t seem much interested in it any more, so hoping to give it a better life, I added asdf files and a test suite and fleshed out some of the special cases. I also use it in clsql-orm to make singular class names from plural table names if that is your kind of thing.

Group-By Refactor

As always, as soon as I release a library, I can see all the mistakes I was happy leaving in until other people could see it. In Group-By I found all sorts of inconsistencies in my approach, and so to make this tiny library better I rewrote the important bits. The main problem was that this started as an alist grouping mechanism. But alists became untenable at depths greater than 1 or 2, or if linear lookup was unacceptably slow. For more efficiency I had looked at grouping into hash table; for a usable interface I looked at grouping into CLOS tree-nodes. Then I combined all three approaches into a monstrosity. The problem with this approach was that it conflated wanting a nice/usable interface (which CLOS can provide), with the efficiency issues of looking up children via a hash table or list. As such I had this strange mirroring of awful to use datastructure backends, barely wrapped in a nicer CLOS interface.

No more, now the structure of multiple groupings is a CLOS tree of grouped-list objects, while the children are stored in a single hashtable or list on each tree node (with methods defined so you should never have to worry about the implementation other than to adjust performance). This greatly simplified my ability to think about what this library was doing, and cleaned up what I considered to be some fairly glaring ugliness. Overall i think this refactoring was a victory.

It would be nice to switch implementations from list to hashtable when we noticed the number of children increasing past a certain threshold, but I have left that for a later date.

Group-By: A Common Lisp library to help group data into trees

A recurring problem I have experienced while programming, is the need to convert a flat list of data into a more complex tree data structure. This is especially common when dealing with results from relational databases (where all data is intrinsically flat, and queries return tables of data). To solve this problem I wrote a small library named group-by (in honor of the sql operator that performs much the same task).

The easiest example:

(group-by '((a 1 2) (a 3 4) (b 5 6)))
=> ((A (1 2) (3 4)) (B (5 6)))

A more concrete example is from trac, the ticketing system we use. Trac tickets contain fields for author, project, milestone, and summary (among others). When displaying this data, my project manager wants to be able to see what everybody is working on (a tree view organized by author, project, and milestone), as well as being able to see what is being worked on in a project and by whom (a tree view organized by project and milestone). To accomplish this I pull a flat list of ticket objects from the database (using a clsql-orm generated class). I then create a tree from this data table by calling make-grouped-list. I can then perform a standard recursive tree walk to render this with the desired organization directly.

Example Call:
(make-grouped-list tickets
:keys (list #'author #'project #'milestone)
:tests (list #'equal #'equal #'equal))

Example Rendering:

Group-by supports grouping into alists, hashtables, and CLOS tree-nodes. To hide the difference between these implementations, I created a grouped-list CLOS object that manages all of the grouping and presents a unified interface to each of these implementation strategies. I support each of these implementations because which to use is strongly dependent on the workload you anticipate performing with the tree. Simply grouping once then recursively rendering the tree, is often more efficient as an alist, than a heavier weight data structure. Conversely, hashtables tend to perform better for lots of accesses into the grouping structure.

To see more, runnable examples, please checkout the project page and the examples file.

CL-Creditcard & CL-Authorize-net: Processing credit card payments with common lisp

I just pushed cl-creditcard to my github account. CL-Creditcard( and sub-library cl-authorize-net) is a library that we use to process payments with Authorize.Net. We have a large internal application that tracks and manages all our business and customer logic including billing and invoicing. (Invoices are generated using cl-typesetting). This application charges credit card payments through this cl-authorize-net.

It has been stable and charging cards for years and I just got around to releasing it. Soon it will also support ACH (echeck) transactions and we will be moving to lisp-unit from lift.

As with all payment processing, test very well before putting into production :)

CLSQL-ORM: Turn you existing database schema into Common Lisp CLSQL-View-Classes

CLSQL-ORM is a common lisp library I just pushed to my git hub account. Its primary goal is to generate CLSQL-view-classes based on an existing database. It uses the “information_schema” views to introspect relevant data from the database, then builds and evals the view-class definitions. This project is a significant branch of clsql-pg-introspect, attempting to remove its “pg” aspects in favor of the standard “information_schema”. It might have changed some semantics/conventions along the way, (I’m not sure as I didn’t use the original project much, and that was long ago).

I wanted to generate my lisp objects from the database for a couple reasons. One, I am fairly comfortable with SQL databases and am used to specifying them in whatever variant of sql the database engine supports. Two, I am most often presented with an extant working database that I want to interact with (such as a wordpress install), where the schema of the database can change, and I just want my common lisp to match whatever the database says, rather than trying to keep both up to date with each other manually. Obviously this project encodes many of my own, personal thoughts and tastes about databases, which may not be the same as your thoughts and tastes. This project is perhaps best though of as a jumping off point for creating your own personal common lisp ORM, though it should be usable as is, if your tastes and mine coincide.

Symbol-Munger: A common lisp library to help convert symbols between their representations in various systems

I just posted a small common lisp library to my github account named Symbol-Munger. Symbol-Munger provides functionality to ease conversion between the same symbol in different environments.

For example, when generating common lisp classes from a database schema, I want to change column names into lisp slot / accessor names, and then later when I am displaying that common lisp slot on the screen I want to display its slot-name as an english column header. (IE: my_db_col > my-db-col > My Db Col)

I have had this code laying about for years and use it everywhere frequently. Earlier today while cleaning other code, I ran across this TODO in vana-inflector, and realized I already had code that performed this function. This library contains one function that does the bulk of the work (normalize-capitalization-and-spacing) and many functions that set default arguments to that one (lisp->english, english->camel-case, english->keyword. etc). Its only dependency is iterate.

CSS-Selectors: quicker compiler with lambda trees

CSS-Selectors and TALCL are both, at heart, translators from some language (css-selectors and xml respectively) into compiled common lisp functions. We translate these expressions by translating the source language into a tree of common-lisp, inserting that common-lisp into a lambda form, then calling compile on that lambda. Because this is a central part of the utility they provide, slow compilation speed can be somewhat annoying. This led to me implementing compiler macros and other caching schemes so that compiling could be handled at compile time (when possible/constantp), where the slow compilation would be less problematic.

A while back I read an article by Xach about compiling queries without calling either “eval” or “compile” by utilizing lambda building functions. He wrote that this technique yielded much quicker compilation and no real loss of performance. To investigate, I wrote a second compiler for CSS-Selectors utilizing this compilation technique. I then wrote tests to compare this new compiler with the old.

Below are the results comparing the two compilers. My experiment confirmed Xachs findings from 2007. The lambda tree version of the compiler is MUCH quicker (~1000x in SBCL) with comparable execution speed. This hasn’t sped up the (cl-yacc produced) parser, so I think that much of the caching and compiler macro code is still valid and still saving user time, just less crucially so now. The only downside I saw was that debugging was a touch harder because there was no human readable version of the fully translated expression (which I could get from the old translator).


15:37:03 BEGIN Running Test TEST-COMPILER1-SPEED | form translation and compile
Evaluation took:
7.266 seconds of real time
7.140000 seconds of total run time (5.840000 user, 1.300000 system)
[ Run times consist of 1.780 seconds GC time, and 5.360 seconds non-GC time. ]
98.27% CPU
606 forms interpreted
6,262 lambdas converted
18,119,140,305 processor cycles
683,883,136 bytes consed

15:37:11 BEGIN Running Test TEST-COMPILER2-SPEED | lambda trees
Evaluation took:
0.008 seconds of real time
0.000000 seconds of total run time (0.000000 user, 0.000000 system)
0.00% CPU
19,072,358 processor cycles
1,572,384 bytes consed

15:37:02 BEGIN Running Test TEST-COMPILER1-EXECUTION-SPEED
Evaluation took:
0.907 seconds of real time
0.890000 seconds of total run time (0.880000 user, 0.010000 system)
[ Run times consist of 0.080 seconds GC time, and 0.810 seconds non-GC time. ]
98.13% CPU
2,261,249,550 processor cycles
39,185,920 bytes consed

15:37:11 BEGIN Running Test TEST-COMPILER2-EXECUTION-SPEED
Evaluation took:
0.881 seconds of real time
0.850000 seconds of total run time (0.840000 user, 0.010000 system)
[ Run times consist of 0.040 seconds GC time, and 0.810 seconds non-GC time. ]
96.48% CPU
2,196,945,765 processor cycles
39,155,152 bytes consed

Source Code: