Symbol-Munger: A common lisp library to help convert symbols between their representations in various systems

Posted on by
4

I just posted a small common lisp library to my github account named Symbol-Munger. Symbol-Munger provides functionality to ease conversion between the same symbol in different environments.

For example, when generating common lisp classes from a database schema, I want to change column names into lisp slot / accessor names, and then later when I am displaying that common lisp slot on the screen I want to display its slot-name as an english column header. (IE: my_db_col > my-db-col > My Db Col)

I have had this code laying about for years and use it everywhere frequently. Earlier today while cleaning other code, I ran across this TODO in vana-inflector, and realized I already had code that performed this function. This library contains one function that does the bulk of the work (normalize-capitalization-and-spacing) and many functions that set default arguments to that one (lisp->english, english->camel-case, english->keyword. etc). Its only dependency is iterate.

CSS-Selectors: quicker compiler with lambda trees

Posted on by
1

CSS-Selectors and TALCL are both, at heart, translators from some language (css-selectors and xml respectively) into compiled common lisp functions. We translate these expressions by translating the source language into a tree of common-lisp, inserting that common-lisp into a lambda form, then calling compile on that lambda. Because this is a central part of the utility they provide, slow compilation speed can be somewhat annoying. This led to me implementing compiler macros and other caching schemes so that compiling could be handled at compile time (when possible/constantp), where the slow compilation would be less problematic.

A while back I read an article by Xach about compiling queries without calling either “eval” or “compile” by utilizing lambda building functions. He wrote that this technique yielded much quicker compilation and no real loss of performance. To investigate, I wrote a second compiler for CSS-Selectors utilizing this compilation technique. I then wrote tests to compare this new compiler with the old.

Below are the results comparing the two compilers. My experiment confirmed Xachs findings from 2007. The lambda tree version of the compiler is MUCH quicker (~1000x in SBCL) with comparable execution speed. This hasn’t sped up the (cl-yacc produced) parser, so I think that much of the caching and compiler macro code is still valid and still saving user time, just less crucially so now. The only downside I saw was that debugging was a touch harder because there was no human readable version of the fully translated expression (which I could get from the old translator).


15:37:03 BEGIN Running Test TEST-COMPILER1-SPEED | form translation and compile
Evaluation took:
7.266 seconds of real time
7.140000 seconds of total run time (5.840000 user, 1.300000 system)
[ Run times consist of 1.780 seconds GC time, and 5.360 seconds non-GC time. ]
98.27% CPU
606 forms interpreted
6,262 lambdas converted
18,119,140,305 processor cycles
683,883,136 bytes consed

15:37:11 BEGIN Running Test TEST-COMPILER2-SPEED | lambda trees
Evaluation took:
0.008 seconds of real time
0.000000 seconds of total run time (0.000000 user, 0.000000 system)
0.00% CPU
19,072,358 processor cycles
1,572,384 bytes consed

15:37:02 BEGIN Running Test TEST-COMPILER1-EXECUTION-SPEED
Evaluation took:
0.907 seconds of real time
0.890000 seconds of total run time (0.880000 user, 0.010000 system)
[ Run times consist of 0.080 seconds GC time, and 0.810 seconds non-GC time. ]
98.13% CPU
2,261,249,550 processor cycles
39,185,920 bytes consed

15:37:11 BEGIN Running Test TEST-COMPILER2-EXECUTION-SPEED
Evaluation took:
0.881 seconds of real time
0.850000 seconds of total run time (0.840000 user, 0.010000 system)
[ Run times consist of 0.040 seconds GC time, and 0.810 seconds non-GC time. ]
96.48% CPU
2,196,945,765 processor cycles
39,155,152 bytes consed

Source Code:

CL-Mediawiki Moved to GitHub

Posted on by
Reply

I have moved CL-Mediawiki to github.  Why, you might ask?

  • Better authentication / user scheme (and other people manage it)
  • Pull / merge requests and request viewer (essentially I think github has a better story for distributing patches than repo.or.cz’s mob branch)
  • Everything about the project in one place, no longer need both trac an repo.or.cz just one git hub location
  • Prettier git-web
  • I am more comfortable with the github interface
  • It has a name and url I can remember without google’s help

I am still in the process of changing all the links and I will keep the repo.or.cz links up (at least for a bit) so that people pointing to it don’t get lost.

Reliable Common Lisp Builds with Quicklisp

Posted on by
Reply

Earlier this week I released a couple common lisp libraries.  Thus for the rest of the week I have tried to make them usable by anyone who is not me.  The situation is complicated by a few facts about our development environment.  Our shared lisp library directory has been accreted over more than 5 years, and some of these libraries having been patched (some in darcs and some in git and most not patched upstream).  We also make heavy use of precompiled cores to speed up start up time.  All of this leads to an environment that is very hard to replicate.  Quicklisp to the rescue! Quicklisp makes it easy to have a lisp environment with the same set of libraries available as everybody else, which is a tremendous win when compared to our difficult-to-replicate system.

To fix the bugs other people would see in the software that builds and works fine for me, I wrote an sbcl script to perform a clean build and fetch all unknown dependencies from quicklisp.  (This script is not common lisp; there is sb-xxx all over the place.)  I learned quite a bit about all of the related systems, so the end result seems somewhat trivial, but perhaps it will provide decent examples. Quicklisp is easy to install, and makes deploying usable common lisp libraries MUCH easier.

By running the following script I will load and test the buildnode system using only libraries pulled from quicklisp.
~$ sbcl --script ~/run-builds.lisp --test-system buildnode --quicklisp-only

... Style warnings and other build detritus...
** TEST RESULTS: Buildnode **
-----------
ATTRIB-MANIP: 9 assertions passed, 0 failed.
CLASS-MANIP: 7 assertions passed, 0 failed.
TEST-ADD-CHILREN: 1 assertions passed, 0 failed.
TEST-BASIC-HTML-DOC: 1 assertions passed, 0 failed.
TEST-FLATTEN-&-ITER-DOM-CHILDREN: 1 assertions passed, 0 failed.
TEST-INSERT-CHILREN: 3 assertions passed, 0 failed.
TEST-ITER-CHILDREN: 6 assertions passed, 0 failed.
TEST-ITER-NODES: 6 assertions passed, 0 failed.
TEST-ITER-PARENTS: 4 assertions passed, 0 failed.
TEST-TEXT-OF-DOM: 2 assertions passed, 0 failed.
TOTAL: 40 assertions passed, 0 failed, 0 execution errors.
------ END TEST RESULTS ------
... MORE style warnings and other build detritus...

I hope to incorporate this script into an automated lisp builder soon.

Announcing Buildnode, CSS-Selectors & TALCL

Posted on by
1

I have just pushed three Common Lisp libraries to my github account.  We have been using Buildnode and TALCL internally for quite some time and CSS-Selectors was something of a learning project I just wrote.

Buildnode

Buildnode is a library to make working with CXML DOM documents and nodes easier.  It smoothes some of the DOM interfaces to be a bit nicer and more in line with common lisp conventions.  We use buildnode primarily to generate the output of webpages hooked up to our extensively modified UCW lisp web server.  We also use it to generate excel spreadsheet XML and google earth KML.  It facilitates writing small generation functions that can be built up and combined in any way.  We also use it to generate “tag” packages which are a package of functions that build the XML tree for us (see the example).

Primary Features

  • Iterate drivers for the dom (in-dom-children, in-dom-parents, and in-dom)
  • DOM manipulation functions such as set-attribute, add-children etc that return the node they are manipulating to ease stringing many calls together and then appending the result to the dom
  • TAG Packages that make a library of functions for interacting with a specific XML dialect

TALCL

TALCL is a branch of ARNESI YACLML/UCW Template Attribute Language, which in turn was branch of Template Attribute Language originally developed in python for Zope.  We think that this version of TAL is much improved over the one originally shipped with UCW by being simpler to use, has better integration with the lisp environment, and simpler evaluation rules.  TALCL is divorced entirely from the UCW/ARNESI/YACLML stack and should be a usable choice for any templating need (though certainly specializing in XML templating).  We use this library for HTML templates that our designers can work with as well as processing both plain text and HTML email templates in our internal billing software. Examples can be found in the repository.

CSS-Selectors

CSS-Selectors is a library that implements a parser for css-selectors level 3 and provides a compiler that can compile node-matcher functions from these selectors.  It also provides a query function (much like jQuery), that can be used to retrieve a list of matching nodes from the dom.  I use this for selecting dom nodes to manipulate from the output of an (unreleased) form controls library and for manipulating and pulling information from DOM documents.  If static, compilation of CSS-selectors into node-matcher functions occurs at compile time.

 

Aqua Data Studio OSS License Program rocks!

Posted on by
1

I have been using Aqua Data Studio 7 since its release (and had used a couple versions before that I believe). I really enjoy using ADS, but have been wanting a new version for quite a while. Today while browsing their site I saw that a version 9 beta is available and also that they have a FREE license available for OSS developers. (They also offer student licensing). This is seriously quality software that makes my life so much easier. If you work with a variety of database systems, you can’t do much better than Aqua Data Studio to talk to all of them with one program.

Thanks Aqua Fold for my free licence! (I qualified for my trac plugin work)

Copying Tables in Postgresql

Posted on by
Reply

If you want to copy a table, all of its data and all of its constraints, indexes, defaults, etc., you have a few options in postgresql

Create table has a couple forms that are helpful:

This form will create a table like yours with all your data, but without any sequences, indexes, etc on it.  As such it is very quick for copying data and nothing else.
CREATE TABLE new_blah AS SELECT * FROM blah;

This form will create a table without any of the data but with all appropriate schema objects. Be careful, this will cause serial columns on the table you are creating to point toward sequence values of the table you are copying.
CREATE TABLE new_blah ( LIKE blah INCLUDING DEFAULTS INCLUDING CONSTRAINTS INCLUDING INDEXES);
ALTER TABLE new_blah ALTER id DROP DEFAULT;
CREATE sequence new_blah_seq;
-- do what you actually need here, 1 might be a good first id for your situation
SELECT setval('new_blah_seq', (SELECT max(id) FROM blah), true);
ALTER TABLE new_blah ALTER ID SET DEFAULT nextval('new_blah_seq');

Data can then be inserted with the following, but this could take a long time on a large table because all constraints and indexes are checked/built one at a time for each row of data.

INSERT INTO new_blah (SELECT * FROM blah);

The next option is to have all your constraints and indexes be declared externally so that they can be re-added later. This allows the CREATE TABLE AS syntax to do its quick inserts and then you can reinstate all schema objects around this table after the data has been copied. This seems to be the absolutely fastest way to copy a table, but requires you to manage your schema objects outside of postgresql or to pg_dump the schema, do some tricky find and replace on names, and then re-run each schema create/alter command.

The final option is to pg_dump the tables you wish to copy, rename the originals and then restore the copy that you made. This takes longer because it must write the data to disk twice (once into the stream/file you dump to then once again when you run the restore and it gives the results

Sending HTML Email with attachments in Common Lisp

Posted on by
2

Ryepup gave us a great couple of simple examples for CL-SMTP.  Unfortunately when I tried to combine them to send an HTML email that had an attachment, I did not get the expected results.  Instead, we insert an inappropriate ContentType header above the multipart then the message body has content type text/plain.  I saw Leslie Polzer’s comment about a BKNR branch of cl-smtp that handled mime-types better.  So I downloaded it, fired it up, bound the content-type variable and it almost produced the correct email.  Unfortunately the result email was missing a newline after the header so my client skipped displaying this first email part (assuming it to be part of the header).  I submitted a patch, that was accepted, that adds the newline after the multipart message header.

[Paste Example]

Munich, Eagles Nest, and Salzberg

Posted on by
Reply

<Sorry for no updates till now; it has been hard to get free internet at our hotels.  I will try to improve the formatting of this article in the future.  For now I just want to try to get as much down as possible.>

Plane Flight

Airport security was far more friendly and I left Orlando TSA a nice comment on their comment card.  I had been so nervous about air travel security that to be treated with such kindness and respect by all employees was a nice treat.  Our trip started with a 3 hour delay on our first flight which caused us to miss our DC to Munich connection.  Instead half our party was rerouted on a DC to to Frankfurt flight and the rest were rescheduled for the next day.  After much cajoling with United staff they were convinced that we should all be put on the 10 pm flight to Munich with a connection to Frankfurt.  The rest of the flights were relatively uneventful, save for my and Shannon’s luggage being left in Frankfurt. (It arrived at our hotel later in the day.)

Munich

Finally we were happily at our hotel with very little sleep.  After an all too brief nap we headed out for dinner and some of Munich’s famous Bier.  We wandered (via underground) to Marienplatz and saw the Rathaus/Glockenspiel at night (which was gorgeous).  I have been in constant awe of all the beautiful architecture we have seen.  It seems that here they actually care about the way things look.  We found the Hackerhaus Restaurant, we John and I had a sampler platter of traditional German foods.  I had sausage roast pork, boiled beef, and a beef & beef-liver meatball.  All the food (even the liver ball) was delicious.

After dinner we took in the Munich saturday nightlife.  The first pub we stopped at had cask beer which was something I hope to have frequently here.  We drank with some very nice Germans and toasted our arrival in their country (Prost!).  They convinced us that litres of beer are superior to half litres.

Sunday we took in the Glockenspiel in the morning and then had a traditional Munich Bratwurst for lunch.  It was white and much softer than the brats I am used to, but still delicious.  By the end of the meal I had been about sauerkrauted out.  The rest of the afternoon was spent at the Munich Technology Museum (biggest in the world) and their English Gardens.

The technology museum was about the coolest museum that I have ever been to.  It was essentially a museum that contained the history of every type of technology that humans have endeavored at.  We covered about a fifth of the museum in about 3-4 hours, focusing on airflight, ships, computers, the Altima Cave, and musical instruments. I also walked through areas of glass, ceramics, astronomy, measurement devices, and other areas I cannot remember right now.  Some highlights for me were: A Cray One super computer, an IBM System 360, a Curta hand calculator (including a clear version that aloud some examination of the inner workings), and boats of all variety (including U-1 the first/prototype german u-boat).

After everyone was museumed out, we decided to head through the English Gardens back to our hotel.  We stopped at the beer garden in the center for a brief rest and imbibement (Ein Dunkle Beir – One Dark Beer).  The garden was beautiful and offered a great view of the skyline.  After our beer and coffee, we decided to head back to the hotel for a rest before heading out to dinner at Hofbreauhaus.  While it was a tad more commercialized than what I remember when I was 10, it was still a good time (though the food was much less good than the night before, but what can you expect from a buffet).  It was here I heard my favorite anti-American joke thus far: “What do you call a person who knows two languages? Bilingual.  What do you call a person who knows only one? American.”  This was told by a German entertaining Italians claiming to know 5 languages (and was extremely nice during the entire meal).  

<tipsier now after returning from Nuremberg pubbing>

After dinner we took our entire party around the corner to introduced the rest of the group to some cask beer from the same bar as the previous night.  This led to drunken conversation with my father-in-law about how to improve businesses, particularly Acceleration.  His suggestion was to institute a system of employee incentives and rewards for new business.  After finishing my 2nd and a half liter the rents returned to the hotel.  We stayed out drunkenly chatting with a swede we met on the street.  We visited the hard rock Munich (where they played the crappiest American music they could find), in deference to my sister/law’s desire to drink some vodka instead of beer.  While everybody else went to bed I decided to stay up and drink some more with my brother/law Brian and his wife Sarah.  This devolved into a heated discussion over whether it is better to believe in the message of Jesus or the divinity of the man (obviously with no clear answer), until we were chased off by the hotel front desk for being too loud.

Eagles Nest

A very hungover morning and a nap in the car later, we arrived at Eagle’s Nest.  Gorgeous, nothing more can improve that sentiment.  The alps are so very pretty and I got to climb some mountains in my Birks.  (A nice german man asked if my shoes had been stolen).  The thing that seemed so strange to me is that in spending something like 150 million dollars they didn’t manage to make any bedrooms.  Its not like this place is close to anything (at least a half hour ride now to anything that resembles a town).  Why would you buld one of the most gorgeous retreats in the world placement wise and not bother to allow anyone to stay there over night? (Seriously there are like 3 rooms that are not the kitchen.)  Oh well, I guess that they were not thinking things through so clearly at this time.  

Salzburg / Mozart Dinner

After our visit to Eagles Nest we headed to Salzburg for a trip to the Mozart Dinner Concert.  If you get the chance this is definitely worth it.  Old Salzburg is more beautiful yet than even Munich, on the trip we saw engravings eulogizing Schubert and statues dedicated to Mozart.  We wandered through old Salzburg to get to our destination, St. Peters Cloister.  We ate and enjoyed immensely the concert in the Baroque Hall of St. Peter’s, which focused on Mozart’s better known instrumental pieces and excerpts of his operas.  The singing and playing was heavenly, especially in this ancient room.  The performance was by a string sextet including two violins, a viola, cello, double bass, harpsichord, and two vocalists.  It could not have been more wonderful.  After this wonderful dinner, we went back to our Mariott hotel in the newer part of Salzburg.

Salzburg Shopping

The next morning, we went back into the old city, to view it during the day and do a bit of shopping for all the nice people back at home.  We saw a huge amount of gorgeous architecture and visited a beautiful old baroque church that had been in continuous existence since the 1200 hundreds.  We visited a market that had fresh and cooked food where I was able to pick up a hotdog.  It was almost American, but with a tougher skin, and instead of sitting in a bun, it was jammed down a hole in the middle of a baguette.  We came back for our departure via a park where we purchased a couple paintings.  On the way we grabbed a quick lunch at a cafe where I was (finally) able to satiate my need for a cheese burger (with ham instead of bacon).  Culture Note: there is nothing that the Germans will not put pig on; I like that in a people.

~ Fin ~

I will write more when I get a chance, thanks to those at home, and sorry if this is less legible than it currently seems.  I will make another post linking to our pictures ASAP