Coffee|Code : Dan Scott - Libraries

Making the Evergreen catalogue mobile-friendly via responsive CSS

dan@coffeecode.net (Dan Scott) — Sun, 21 Apr 2013 22:48:40 -0400

Back in November the Evergreen community was discussing the desire for a mobile catalogue, and expressed a strong opinion that the right way forward would be to teach the current catalogue to be mobile-friendly by applying principles of responsive design. In fact, I stated:

Almost all of this can be achieved via CSS, possibly with some changes to the underlying HTML (e.g. tables to divs or whatever so that "Place Hold" appears under the bib info instead of way over to the right).

I have this bad habit of talking more than doing. So when I saw the Beanstalk mobile catalogue resurrected again at the Evergreen 2013 lightning talks, it bugged me that I still hadn't put any effort into a proof of concept of what was possible with CSS media queries. Thus, today, on the last day of my holidays, I spent a few hours trying things out on our development server and came up with this rough branch to work towards making the exact same HTML that we serve up for desktops provide an experience similar to that of the Beanstalk generation of catalogues for mobile, just via CSS.

As you can see from the commits, I made one change to the HTML to define a viewport, and added one set of CSS rules wrapped in a media query; in essence:

...
<head>
...
<meta content="initial-scale=1.0,width=device-width" name="viewport">
<style>
@media only screen and (max-width: 600px) {
    #header {
        padding: 0px;
        margin: 0px;
    }
    .facet_sidebar {
        display: none;
    }
    ...
}
</style>
<head>
...

Results and trade-offs

Here are a few example URLs from our test server (which is slow, and might get wiped any day, so test them quickly if you have a mobile device around!):

Search results - sacrificed facets, per-item actions, and the language picker
Record details - sacrificed per-item actions, flattened the item table vertically

In general, I removed a lot of the frippery from the header, while trying to retain the most valuable pieces. However, some bits are broken: Another Search doesn't actually let you do another search because the search bar is totally hidden. Other bits haven't been touched (Advanced search is still overwhelming, and My Account, while functional, could be much prettier.

What I've done so far is oriented towards our 2.3-ish lightly customized Laurentian skin (we force full details in search results, for example) but the principles should be applicable to an out-of-the-box Evergreen catalogue. In working through some of the challenges, I've determined that I was pretty much on target back in November; with a few HTML tweaks that would improve the layout for desktops as well, we could keep the per-item actions and facets around, but just move them to a different location.

Less talk, more action

So who's with me? What we have to gain is a single set of HTML to support for TPAC, and a single set of CSS, all available from the same URL, rather than trying to maintain overlays and monkeying about with mobile-vs-desktop URLs and the like. Feel free to dig in and start pushing branches with improvements over my rough attempts and let's make this thing happen for Evergreen 2.5.

With thanks to Firefox...

I would be remiss if I did not mention the marvellous Responsive Design View introduced in Firefox 15, along with the Style Editor; together, these tools (built into Firefox) made my developing and testing work so much easier.

If you want to live on the cutting edge of Firefox, you want Aurora - go and get it

Structured data: making metadata matter for machines

dan@coffeecode.net (Dan Scott) — Fri, 12 Apr 2013 15:11:47 -0400

Update 2013-04-18: Now with video of the presentation, thanks to the awesome #egcon2013 volunteers!

I've been attending the Evergreen 2013 Conference in beautiful Vancouver. This morning, I was honoured to be able to give a presentation on some of the work I've been doing on implementing linked data via schema.org in Evergreen. I think I did a good job of explaining the potential value of linked data and arguing for improving Evergreen's schema.org publishing ninja skills.

My slides, with a reasonable number of useful speaker notes to provide context, are available in LibreOffice format.[1]

In addition, the amazing organizers of the conference also streamed most[2] of the talk and the recording will be available on the conference web site in a week or two.

Footnotes

I felt pretty dirty not using HTML5 + RDFa Lite to actually mark the whole thing up; there was some question close to the time of the conference as to whether anything but PPT or perhaps PDF would be an acceptable format... a concern that was subsequently removed, but a little too late to be worthwhile changing course.
The room was standing-room only (well, sitting-on-the-floor-room only), and one of the organizers accidentally sat on and unplugged the Ethernet cable, so something like ten minutes were lost. Heh!

Introducing SQL to Evergreen administrators, round two

dan@coffeecode.net (Dan Scott) — Fri, 15 Feb 2013 21:32:08 -0500

Three years ago I was asked to create and deliver a two-day course introducing SQL to Evergreen users. Things went well and I was able to share the resulting materials with the Evergreen and PostgreSQL community. Perhaps one of my happiest moments at the Evergreen conference last year was when one of the participants in that course, told me that many of his fellow participants were still successfully writing SQL queries and getting work done. Huzzah!

Time goes by and another group, OHIONET, was running into difficulties getting started with PostgreSQL and Evergreen. They asked me if I would be willing to give the same sort of training I had given a few years back. "Sure", I said, thinking it would be a great opportunity to polish the materials and add some updates to cover new features in PostgreSQL and Evergreen. We also opted to skip the travel and do an entirely virtual training session via Google Hangouts, which worked out rather nicely (but that's a different story).

As it turned out, I probably ended up putting about four days worth of effort (crammed into lots of late nights, weekends, and vacation days) into overhauling the instruction materials. But the results were worth it, in my opinion; I'm rather proud of the content, and while I believe it stands up on its own, the guidance that I was able to provide during the live instruction sessions was well-received by the participants.

Thus, I am pleased to be able to offer to the broader community the latest version of the Introduction to SQL for Evergreen Administrators, under a Creative Commons Attribution-ShareAlike 3.0 (Unported) license.

Reference documentation--30 pages introducing SQL with examples drawn from the Evergreen schema: (HTML) (PDF) (ePub) (AsciiDoc)
Presentation: (LibreOffice Impress) (PDF)
Solutions to exercises: (Day 1) (Day 2)

So, a huge thanks to OHIONET for giving me the impetus to overhaul this material, and for giving me a chance to introduce them to the wonders of SQL with PostgreSQL, and to the inner workings of the Evergreen schema. It was a blast! And thanks for agreeing to let me share these materials with the broader community.

Introducing version control & git in 1.5 hours to undergraduates

dan@coffeecode.net (Dan Scott) — Sun, 20 Jan 2013 08:22:51 -0500

Our university offers a Computer Science degree, but the formal curriculum does not cover version control (or a number of other common tools and practices in software development). Students that have worked for me in part-time jobs or summer positions have said things like:

if it wasn't for that one summer I worked at the library for/with you using the version control software (which was my first real time ever using it) I'd have been really confused in my first job after graduating

That kind of statement reaffirms my belief that a Computer Science degree should include some form of introduction to software development norms--you could perhaps call it software development literacy. While I've heard the "sink or swim" argument when it comes to having a clue about what you are expected to do when you land an actual programming job (as in, "students that don't learn this on their own probably aren't going to learn it from us"), I'm sure that some minimal level of exposure to these norms and specific tools from future peers will help more students swim. As I am particularly interested in development within free software projects, I've been in close contact with the Computer Science Club to try to start providing this outside of the formal curriculum.

So, as part of this mission, a few days ago I gave a presentation to the Club on "version control", with a particular emphasis on using git. The turnout was low, alas; not a sexy subject, perhaps, or just the start of term, or perhaps I need to improve my marketing efforts. In any case, I believe it went well--heads were nodding and I had the complete attention of the participants for 1.5 hours while I built up a conceptual framework for understanding version control as practiced in git.

I'm happy to make my presentation materials available under a CC BY-SA 3.0 license; maybe you can use them and improve on them!

Triumph of the tiny brain: Dan vs. Drupal / Panels

dan@coffeecode.net (Dan Scott) — Thu, 18 Oct 2012 17:48:31 -0400

A while ago I inherited responsibility for a Drupal 6 instance and a rather out-of-date server. (You know it's not good when your production operating system is so old that it is no longer getting security updates).

I'm not a Drupal person. I dabbled with Drupal years and years ago when I was heavily into PHP, but it never stuck with me. Every time I poked around at the database schema, with serialized objects stuck inside columns, I found something else that I wanted to work on instead. Thus, inheriting a Drupal instance wasn't something I had been looking forward to. As this production server was running a number of different services that were in use by our library, I went through a number of trial runs to ensure that the base packages wouldn't introduce regressions or outages. Fast-forward past a reasonably successful early-morning upgrade from Debian Lenny to Squeeze and I was able to start looking at addressing the Drupal instance that was also approximately 18 months out of date.

Initially, after I worked out the how-to of Drupal upgrades (in short: upgrade just Drupal core, then upgrade the modules), I thought all was well. I even got over the hump of realizing that our instance had had all of the modules dumped into Drupal's core directory, rather than sites/all/modules, and (even more impressively) got over the problem that the core bluemarine them had been hacked directly rather than having been separated out into a new custom theme. After working through those learning pains, I realized that somewhere in all of the Drupal and module upgrades, that something got "more secure" and started truncating IMG links to files with spaces in them at the first space. So "foo%20bar.jpg" was becoming "foo.jpg" and we were getting 404s everywhere.

Did I mention that I didn't notice this until I upgraded our production instance? Oh yes, I went through iteration after iteration of upgrades on the test server, and dutifully fixed up the problems that I found in the subset of content that I was testing against. I discovered and fixed problems like the production server content linking directly to the test server (slight copy-and-paste errors on the part of the content creators, I suppose). But I didn't notice all of the 404s, because who uploads images with spaces in their filename?

Turns out, everyone else in my library does that. Of course! And from what I was able to piece together via Google and browsing drupal.org, there was supposed to be some sanitization of the incoming filenames so that spaces would be normalized, etc. But either that wasn't introduced until well after our content had been created, or my predecessor had lightly hacked one of the modules, or Drupal itself, and hadn't bothered to use a source code repository to track those customizations. So, realizing that I needed to make some bulk changes, I went at it with a two-step plan:

Create symbolic links for both the truncated filename and the spaces-normalized-to-underscores filenames. Creating symlinks for the truncated filenames would fix the 404s immediately, at the cost of some clash in the intended targets; there were plenty of Foo illustration.JPG and Foo info.JPG pairs of files, but like the Highlander, there can be only on Foo.JPG.
Munge the database entries so that all of those now apparently insecure %20-containing filenames would become underscores.

If you're a Drupal user or a Drupal with Panels module user, you might know that the database schema suffers from some fairly horrible tricks being played on it. In this case, the Panels module creates a panels_pane table with a configuration TEXT column. Based on the name alone, it might seem odd that column is used to store the HTML content of the corresponding panel. Even odder to me is that this is not just a TEXT column, it's a column that expects a very particular structure - something like:

a:5:{s:11:"admin_title";s:5:"RACER";s:5:"title";s:0:"";s:4:"body";s:639:"<p><img width="225" height [...]}

Ah, nothing like storing an object within a single database column. Of particular interest was the result that I had when I tested updating the column value with a basic "replace(configuration, '%20', '_')" - the panel showed only n/a, presumably because the size (defined by the s properties in the object) for the "body" text property was no longer a match. That would be an instance of http://drupal.org/node/926448 - so okay, clearly I had to change tactics and update the entire object.

I tried quickly finding the Drupal way to do this: clearly there's an API and there must be some simple way to retrieve an object, change it's values, and update it so that the serialized object gets stored in the database and Drupal is happy. However, I couldn't find a simple tutorial, and trying #drupal on Freenode was unfortunately fruitless as well (although some people did try to suggest running REPLACE() at the database level, that was nice but they didn't recognize that that would actually damage things significantly).

So... out came the Perl, and here's what I hacked together:

#!/bin/perl
use strict;
use warnings;

foreach (<DATA>) {
    chomp();
    my $i = 0;
    my $body = 0;
    my @fixed;
    my @row = split /\t/;
    my $pid = $row[1];
    my $configuration = $row[0];
    my @chunks = split /";s:/, $configuration;
    foreach my $chunk (@chunks) {
        if (!$i++) {
            push @fixed, $chunk;
            next;
        }
        if ($chunk =~ m/"body/) {
            $body = 1;
            push @fixed, $chunk;
            next;
        }
        if ($body) {
            my ($length, $content) = $chunk =~ m/^(\d+):"(.+)$/;
			for (my $j = 0; $j < 50; $j++) {
            $content =~ s{(/pictures/[^\./]+?)%20}{$1_}g;
			}
            $content =~ s{%20}{+}g;
            $length = length($content);
            $chunk = "$length:\"$content";
            $body = 0;
        }
        push @fixed, $chunk;
    }
    print 'UPDATE panels_pane SET configuration = $ashes$' . 
        join('";s:', @fixed) . '$ashes$' . " WHERE pid = $pid;\n";
}
__DATA__

Against the trusty database (I ? PostgreSQL!), I ran COPY (SELECT configuration, pid FROM panels_pane WHERE configuration ~ '%20') TO 'conf_pids.out';, then slapped the Perl code on top and generated a load of UPDATE statements. It's far from my best Perl code, but it worked and once I gave up on doing things the Drupal way I was able to put it together in a handful of minutes. I now have a functional Drupal 6 instance again, updated such that there are no known security vulnerabilities with either core or the modules we're using, and there are no broken image links.

And now I need to begin working towards either grokking Drupal, or finding a content management system that my tiny brain can comprehend, because I don't want to have to go through these kinds of contortions again with future upgrades... Suggestions welcome!