Evergreen

Laurentian University is part of the Ontario Council of University Libraries (OCUL), and a user of the centrally hosted Ontario Scholars Portal SFX link resolver, so one of the things we needed when we migrated to Evergreen was a target parser for our link resolver. This is the target associated with Search the library catalogue that is the last resort when the resolver fails to turn up any full-text resources for a given OpenURL - so hopefully it won't need to be invoked too often, as we have a very rich set of full-text electronic resources at Laurentian University.

The code

Here is a quick implementation of a target parser that generates search URLs based on ISSN, ISBN, book title, or journal title. Pretty impoverished from an OpenURL perspective, but it maintains the same level of functionality from our previous system. In TargetParser/Evergreen/Conifer.pm I created a target parser called Evergreen::Conifer that implements a subset of the Parsers::TargetParser API for SFX as follows:

package Parsers::TargetParser::Evergreen::Conifer;
use Parsers::TargetParser;
use base qw(Parsers::TargetParser);
use strict;

sub getHolding {
  my ($this,$genRequestObj) = @_;
  
  my $objectType = $genRequestObj->{'objectType'};
  my $ISBN = $genRequestObj->{'ISBN'};
  my $eISBN = $genRequestObj->{'eISBN'};
  my $ISSN = $genRequestObj->{'ISSN'};
  my $eISSN = $genRequestObj->{'eISSN'};

  my $CODEN = $genRequestObj->{'CODEN'};
  my $bookTitle = $genRequestObj->{'bookTitle'};
  my $journalTitle = $genRequestObj->{'journalTitle'};

  # Canonical search results URL for simple searches:
  # http://laurentian.concat.ca/opac/en-CA/skin/lul/xml/rresult.xml?rt=keyword&tp=keyword&t=0895-2779&l=105&d=2&f=&av=

  my $svc = $this->{svc};
  my $egHost = $svc->parse_param('eg_host');
  my $egLocale = $svc->parse_param('eg_locale');
  my $egSkin = $svc->parse_param('eg_skin');
  my $egOrgUnit = $svc->parse_param('eg_org_unit');
  my $egDepth = $svc->parse_param('eg_depth');

  my $path = "http://${egHost}/opac/${egLocale}/skin/${egSkin}/xml/rresult.xml?l=${egOrgUnit}&d=${egDepth}";

  my $searchString = '&rt=keyword&tp=keyword&t=';

  if (defined($ISSN)) {
    if ($ISSN =~ m/x/i) {
      # Current indexer doesn't deal well with ISSNs containing an X, so break it up
      $ISSN =~ s/^(\d{4})-?(\d+)x/$1 -$2 x/i;
      $searchString .= $ISSN;
    } else {
      $searchString .= "\"$ISSN\"";      # format 9999-9999 for MARC
    }
  } 
  elsif (defined($ISBN)) {
    # Evergreen doesn't force ISBNs to be stripped of hyphens, so take whatever
    $searchString .= "\"$ISBN\"";
  }
  elsif (defined($journalTitle)) {
    # Restrict searches to title index, with bibliographic level = s
    $searchString .= "ti:${journalTitle}&bl=s";
  }
  elsif (defined($bookTitle)) {
    # Restrict searches to title index, with bibliographic level = m
    $searchString .= "ti:${bookTitle}&bl=m";
  }

  return ($path . $searchString);

}

1;

And here's the help that I added to the corresponding Conifer.hlp file:

General Information
Target - LOCAL_CATALOGUE_EVERGREEN_CONIFER
Service - getHolding
Parser - Evergreen::Conifer

Information needed in the Target Service:
In the PARSE_PARAM field, replace the following information:
eg_host = $$LOCAL_CATALOGUE_SERVER
eg_locale = Locale (en-US, en-CA, fr-CA, etc)
eg_skin = algoma, default, lul, nohin, uwin
eg_org_unit = 103, 1, etc
eg_depth = 0, 1, 2, 3, etc

Findings and wishlists

While it's quite easy to set up Evergreen as a searchable resource, thanks to its straightforward URL syntax, one of the things that leaps out at me is that Evergreen, by default, has no identifier index for limiting searches by ISBN / ISSN / LCCN / OCLCnum. Ideally, we would disable full-text indexing on this index so that we can more accurately search for ISSNs that include an x. Right now we have to split ISSNs with an "x" into constituent parts and generate searches on those parts, which results in false hits from across the database. This would also be useful for limiting Z39.50 searches.

I would also like to teach Evergreen about ISBN-10/ISBN-13 equivalence, to broaden the search while maintaining precision. And I would like to automatically normalize ISSN and ISBN formats so that I don't have to worry about whether a cataloguer entered hyphens or not - and the same for incoming search terms.

Finally, to support services like xISBN that search for multiple formats and editions of a given work by generating a shotgun blast of ISBNs for all known representations, I would love to teach Evergreen how to accept a list of identifiers as search input.

Don't ask me when these things will happen, though; if it requires work from me, it will probably be 2010 before any of it happens.

I was fortunate to be invited to give a talk (OpenOffice.org Impress / PDF ) on Evergreen's progress on the globalization front at the first ever Evergreen International Conference. My friend Tigran Zargaryan from the Fundamental Science Library of the National Academy of Sciences of the Republic of Armenia gave a talk at almost the same time about his library's progress in adopting Evergreen. Tigran himself was responsible for the translation of the Evergreen catalogue and staff client into Armenian, and he confided that he also expected to make significant progress towards a Russian translation during the lengthy layovers at airports that are part of his normal travel routine.

So, my goal was to provide an overview of the progress we have made in taking Evergreen from its American English roots and enabling it to support not just translated interfaces, but properly localized content display - and to provide some pointers towards where we need to go next. We have been making progress towards a more formalized translation process, so keep an eye out for a call for translations in the next week or two when the Evergreen 1.6 release candidate is made available for testing. We currently sport Armenian, Canadian English, Canadian French, and Czech translations, and welcome both new translations and revisions to our current translations.

To make it easier for translators to collaborate, we need to take our Pootle translation server from a beta service running on my poor little VPS to a real server. We have some technical challenges to overcome - providing translation support for the Template::Toolkit framework, for example. And we have some basic grunt work to do to replace the hard-coded display of numbers, currencies, dates, and times with localized variations throughout our code.

I was pleasantly surprised by the number of people attending the session; I hadn't expected such an interest in the topic, despite it nominally being an international conference. My only regret was that I rushed off the stage without taking questions in the mistaken belief that I had used up all of my time and was eating into my successor's presentation timeslot; as it turned out, there was a built-in 15 minute buffer that I had overlooked. Ah well. Thanks to everyone who came out, and for everyone else who wasn't able to make it to the session, I hope you'll find the slides a good introduction to the state of globalization in Evergreen. And if you have the skills to contribute, please consider pitching into the globalization enablement effort!

Yes, all of a sudden and rather quietly, Evergreen has serials support.

A few weeks ago, I finished hooking up a rudimentary serials holdings display based on David Fiander's MFHD parsing code to our production instance of Evergreen. We loaded our MFHD records from our legacy system into Evergreen and that gave us enough breathing room to keep working on the problem. By rudimentary I mean:

• limited to displaying one MFHD record per bibliographic record (a problem for journals for which you have separate sets of holdings in microfiche, print, etc)
• serials holdings were displayed for a given bibliographic record no matter what library scope you were searching in (more of a problem in theory than in practice as we currently have one copy of a given bibliographic record per library... that will change over time...)
• no way to edit the MFHD records, which is a problem as the issues we have received since migrating to Evergreen three weeks ago are starting to pile up
• limited to English labels in the interface

Here's the rudimentary serials holdings display:

The operative phrase is was rudimentary. In the past two weeks, things have come a long way in Evergreen. The primary result of my afternoon of work at the Evergreen International Hackfest, with lots of help from Mike Rylander and Bill Erickson in navigating the impressive new Dojo toolkit-based Evergreen JavaScript widgets and services in the upcoming Evergreen 1.6 release, was to add an Edit button to the holdings display that shows up when the record is viewed in the staff client. When pressed, the Edit button invokes a MARC editor so that you can copy an 86[345] field and fill in the pertinent information; or collapse holdings in the 86[678] fields, etc. It seems like a minor victory, but it was a real result from the hackfest, and that cannot be discounted!

Here's the MARC editor in action:

Since then, I've been on fire... or maybe on a slow burn, as I put a few hours in here and there, and am happy to say that when Evergreen 1.6 is released, serials support will feature:

• support for display unlimited MFHD records per bibliographic record
• holdings display scoped by library search context - so you'll only see holdings for the part of the library hierarchy that you're searching, rather than the whole consortium
• the Edit button for editing the raw MFHD record
• internationalization support for interface labels, based on Dojo string substitution

I have already committed these features to the Evergreen trunk, but I hope to add a few more pieces to the mix before the Evergreen 1.6 release is cut. We need to display the 852 field contents to identify the location of each set of holdings, and we need to give cataloguers the ability to edit some of the attributes (such as owning library).

Here are the slides I presented (largely screenshots of the serials display and edit button) for the hackfest results lightning talk that I gave with Jeff Godin of Traverse Area District Library. Jeff did some interesting work in his own right on generating feeds for recently added titles based on copy location during the hackfest.

I awoke around 4:48 am today. At the time, I thought it was just our baby kicking away excitedly. However, later this afternoon, I realized that it had been almost exactly a week ago, around 4:30 am on Monday, May 4th that I sent a broadcast email message to librarians and staff at 24 different libraries. The Conifer consortial library system, built on the solid foundations of the Evergreen open-source library system, had gone live - and I was exhausted after a long weekend of migrating all of that data. I was proud to see the Laurentian catalogue sporting a completely different look and new functionality - reviews! book covers! sharable book bags! format & edition grouping! - and excited by the promise of more to come.

Conifer represents the first flowering of an effort that began back in July 2007 with a hand-shake agreement between Laurentian University, McMaster University, and the University of Windsor to build a provincial, primarily academic, library system on Evergreen. The system is centrally hosted by the top-notch IT team at the University of Guelph. Things change, and along the way Algoma University and the Northern Ontario School of Medicine joined us as full partners, and McMaster University opted to continue contributing to the common development effort but withdrew from the centrally hosted system.

As noted, we went live on Monday, May 4th and we survived the first day. On Tuesday, May 5th we corrected a problem in our configuration that had caused some instability (thanks to Mike Rylander for providing the patch that set things straight). Since then, we have been slowly refining aspects of the system - setting up circulation rules, migrating records and items that had been missed over the weekend, polishing the Z39.50 server, fine-tuning the permissions scheme - but the core of the system is solid. We have a consortial system that stretches from the southern-most tip of Ontario to the north-west corner of the province (hello, Thunder Bay!), and so far connectivity seems good and the reliability of the system - which, upon launch, has probably become the second largest Evergreen implementation by number of bibliographic records - has been superb.

A few interesting statistics about Conifer... (have I mentioned how much I love that Evergreen is built on PostgreSQL because it becomes so simple to generate basic reports in plain SQL?):

Number of staff and user accounts per library in Conifer

conifer=# SELECT aou.name, count(au.id)
    FROM actor.org_unit aou
    INNER JOIN actor.usr au
    ON aou.id = au.home_ou
    GROUP BY aou.name
    ORDER BY 2 DESC;

                    name                    | count 
--------------------------------------------+-------
 Leddy Library                              | 19468
 J.N. Desmarais Library                     | 11921
 Algoma University, Wishart Library         |  2431
 University of Sudbury                      |  1100
 Hearst, Bibliothèque Maurice-Saulnier      |  1043
 Huntington College Library                 |   834
 Paul Martin Law Library                    |   592
 Northern Ontario School of Medicine (West) |   284
 HRSRH Health Sciences Library              |   261
 Northern Ontario School of Medicine (East) |   224
 Xstrata Process Support Centre Library     |   122
 NOHIN                                      |   121
 Instructional Media Centre                 |     9
 Laboratoire de didactiques, E.S.E.         |     7
 Vale Inco                                  |     4
 Mines Library, Willet Green Miller Centre  |     2
 Art Gallery of Sudbury                     |     1
 Curriculum Resource Centre                 |     1
 Sault Area Hospital                        |     1
 Centre Franco-Ontarien de Folklore         |     1
 Conifer                                    |     1
(21 rows)

Number of copies held per library in Conifer

conifer=# SELECT aou.name, count(ac.barcode)
    FROM actor.org_unit aou
    INNER JOIN asset.copy ac
    ON aou.id = ac.circ_lib
    GROUP BY aou.name
    ORDER BY 2 DESC;

                    name                    |  count
--------------------------------------------+---------
 Leddy Library                              | 1373197
 J.N. Desmarais Library                     |  614380
 Paul Martin Law Library                    |  229391
 Algoma University, Wishart Library         |  115156
 University of Sudbury                      |   42154
 Hearst, Bibliothèque Maurice-Saulnier      |   34276
 Huntington College Library                 |   12517
 Laboratoire de didactiques, E.S.E.         |   10284
 Mining and the Environment Database        |    9940
 HRSRH Health Sciences Library              |    7512
 Music Resource Centre                      |    7511
 Xstrata Process Support Centre Library     |    5477
 Centre Franco-Ontarien de Folklore         |    4365
 Northern Ontario School of Medicine (East) |    3779
 Northern Ontario School of Medicine (West) |    3301
 NOHIN                                      |    2647
 Mines Library, Willet Green Miller Centre  |    2617
 Curriculum Resource Centre                 |    2583
 Sault Area Hospital                        |    2515
 Art Gallery of Sudbury                     |    2237
 Hearst Timmins, Centre de Ressources       |    2202
 Hearst Kapuskasing, Centre de Ressources   |    2007
 Vale Inco                                  |    1106
 Instructional Media Centre                 |    1095
(24 rows)

What about acquisitions, serials, and reserves?

One of the reasons we had a hard migration date of early May was because it matches nicely with the fiscal year-end for those institutions who were running a traditional acquisitions system on their legacy ILS. We normally shut down all purchases for a period of weeks while we roll over the encumbrances into the next fiscal year and set up our budgets. This year, we're migrating all of the old financial data twice: first, and foremost, into the most sophisticated set of spreadsheets you'll ever see attached to a library system (as pulled together by the inestimable Art Rhyno); and second, into the Evergreen acquisitions system that will launch with Evergreen 1.6. The first migration of a given set of data is always the hardest part, so once we have the fund / order / provider data in spreadsheets, the migration into Evergreen proper will be trivial. This will give us the summer to use both systems side-by-side and refine what we need from Evergreen.

We have migrated all of our serials data from the legacy system, I just haven't enabled the display of that data in our live system. A prototype was running on my laptop for a few days until I accidentally blew it away - ah well, anything worthwhile doing is better the second time around anyway. This, too, will be part of the Evergreen 1.6 release, and will feature full MFHD compliance built on the code that David Fiander has been writing on behalf of Equinox. I should note that this first cut at serials is in some ways relatively basic; while the system in Evergreen 1.6 will be fully MFHD compliant, down to the point of letting you to edit an MFHD record to "check in" a new issue by adding a new 863 field, it won't associate barcodes with individual issues. Most of the database schema exists to support that, but there's still a large amount of code to be written on top of the schema and we need Something That Works Right Now I'm confident that that's coming not too far down the road, though.

Finally, what would an academic library be without reserves? Art Rhyno (again!) has been working with Graham Fawcett for the past six months on Syrup - a really impressive melding of the world of electronic reserves and traditional physical library system reserves that uses SIP and Z39.50 to talk to Evergreen. Syrup is just about at a full boil now, so in a few more weeks we should have it deployed so that we can savour its sweetness through the relatively slow summer months before ensuring that the taste is just right for all of our incoming students and faculty in the fall.

This Easter weekend I had the opportunity to play with someone's iPod Touch. Of course, the only thing I tried was the Evergreen 1.4 catalogue interface. Lo and behold, it came up just fine on Safari in all of its heavily dynamic JavaScript and less-than-XHTML-compliant glory - even sporting several Dojo widgets. Nice. So we don't have to worry about writing an iPhone-specific application to access Evergreen; users of such devices can just use the normal dynamic catalogue with full functionality.

Evergreen doesn't fare quite as well with Microsoft's rather decrepit PocketExplorer browser on my HTC Touch smartphone (it's a Windows Mobile monstrosity, sigh), but it does work well with the Opera Mobile 9.5 beta browser. I eagerly anticipate the first good release of Fennec for Windows Mobile (coming soon!), as I'm confident that's going to improve my mobile Web browsing experience even further.

I predict that in another year or two the idea of building mobile-specific Web portals to complement your full-function Web site will be pretty passé. I already get really irritated when Web sites think they're being helpful by automatically redirecting my smartphone to an extremely limited interface; in most cases, the full site runs fine. Give me the option, sure, but don't force me down that path. As hardware costs continue to drop, and 3G networks expand, and more people upgrade to more capable mobile devices, one full-function Web site will be all we need--as long as that site is written in (X)HTML and CSS and JavaScript.

Those sites that decide to push core functionality into Flash or SilverLight, on the other hand, can go straight to hell, thankyouverymuch. I'm looking at you, PTOnTheNet. This is a site to which Lynn has been a paying customer for years. It recently announced that it was revising the Web site, which is all well and good. What's not so good is that they adopted SilverLight: not just for pretty effects here and there, but as a core technology. Problem: Lynn has been using Linux at home since I introduced her to it somewhere around eight years ago, and last year bought one of the early models of the Linux-based Asus EEE netbook. Not only did the site redesign destroy the personal training programs she had set up for her clients over the years (breaking site redesign rule #1: Thou shalt not destroy your clients' data), but it also renders her netbook useless for that site.

Even with the Moonlight plugin installed, it looks like the cretinous site developers are using detection scripts to prevent the plugin from even trying to render the content. With Linux-based netbooks on the rise--and with netbooks being the right form factor and price for personal trainers who want to throw them into their backpacks and not weep too bitterly if their netbook suffers the misfortune of being knocked around or sweated to death--this seems very much like a technology choice that was not based on the needs of the customers. Worst of all, they deliberately chose to exclude Linux, when a (X)HTML, CSS, and JavaScript platform would have supported almost any modern platform: not just Linux netbooks, but other mobile devices like the iPhone and smartphones that are so well-suited to the personal trainer. So, at least one customer is going to be walking away, and if there's a competing Web site out there that caters to a broader clientele, I bet there will be far more customers moving in that direction.

Almost exactly a year ago, Jason Etheridge (the primary developer of the Evergreen staff client) and I managed to get our hands on a developer edition of the Nokia N810 Internet tablet device. It's a nifty little handheld computer that packs 128 MB of memory, a touch screen, and a beautiful 800x480 screen, and I've had my hands on it from almost the beginning. The primary rationale of the Nokia developer program was to encourage developers to put together useful applications for their platform, of course... and as the months ticked by and I did nothing of interest, my guilt slowly grew.

Well, today I feel a little bit better. Here's what happened: when I was attending FSOSS 2008 at Seneca College, I ran into Madhava Enros. Madhava and I had worked together on some help UI designs back when we were both DB2 employees; since then, he had joined the Mozilla Foundation and was working on Fennec, the mobile version of Firefox targeting the N810 device (to begin with, at least). The first alpha of Fennec had been released to coincide with FSOSS 2008, so I gave it a shot a few days later. Madhava's team made some great innovative decisions for Fennec's UI, but what really caught my eye was that they had packaged a port of XULRunner-1.9 to the N810.

See, the Evergreen staff client is built on XUL, the same XML/JavaScript/CSS foundation as Firefox and Thunderbird and Fennec - and to run XUL, you need XULRunner. At the time, though, the Evergreen staff client needed the 1.8 version of XULRunner; it simply wouldn't work with 1.9. So, I stuffed the N810 back into its case and forgot about it for a few more months while I focused on other things like the never-ending effort to improve Evergreen's internationalization support.

Over the last few weeks, though, Jason has been steadily enhancing the staff client in Evergreen trunk - and the comment for one of his recent commits was we're kicking xulrunner 1.8 to the curb with trunk. I had a spare hour or two on my hands today, so I copied a staff client build from Conifer's Evergreen trunk test box to the N810, kicked off the XULRunner command, and waited... expecting failure. Instead, I found that the staff client worked almost exactly as it does on my laptop - the major difference being that some of the default function key mappings on the staff client conflict with the mappings of special buttons on the N810 (like the full screen toggle gets mapped to F6 - Record In-House Use on the staff client). Otherwise, the client did a great job of adjusting to the available screen width, and even Dojo-based interfaces like the Vandelay MARC batch importer/exporter and the pop-up calendar worked. Very cool!

So, if I can find a barcode scanner with a mini-USB attachment, I could have a nice little inventory tool on my hands. Or a mobile circulation station. All because the Evergreen developers made the decision years ago to build on XUL as a cross-platform framework... this should be sweet confirmation that they made a good choice. XUL continues to be ported to more platforms, and anyone using the Evergreen staff client benefits from the optimizations and bug fixes that go into XULRunner. Nice. When we cut a release from Evergreen trunk that supports XULRunner 1.9, I'll do my best to package up a version of the staff client for the N810, and some of my guilt will be assuaged. Yes!

Updated 2009-03-02 10:35 am: Correcting Madhava's name; I shouldn't write past midnight without proof-reading! Sorry Madhava.

Updated 2009-02-25 00:29 EST: Corrected setuptools installation step.

Updated 2009-02-08 23:39 EST: Trimmed width of some of the <pre> code sections for better formatting. Created bzr repository for unicorn2evergreen scripts at http://bzr.coffeecode.net/unicorn2evergreen

I did this once a long time ago for the Robertson Library at the University of Prince Edward Island. For our own migration to Evergreen, I have to load a representative sample of records from our Unicorn system onto one of our test servers. This has been a good refresher of the process... and a reminder to myself to post the other part of the Unicorn to Evergreen migration scripts in a publicly available location. Okay, they're posted to this bzr repository called unicorn2evergreen

1. Export bibliographic records from Unicorn using Unicorn's catalog key (basic sequential accession number) as the unique identifier (I plopped the catalog key into the 935a field/subfield combo). I use the catalog key because the "flexkey" is not guaranteed to be unique within a single Unicorn instance - and because the catalog key makes it easy for us to match call numbers and copies.
2. For each item, export call number / barcode / owning library / current location / home location / item type using the catalog key as the identifier.
3. Set up the organization unit hierarchy on your Evergreen system. You can dump it from an existing Evergreen system into a file named "orgunits.dump" like so:

   pg_dump -U evergreen --data-only --table actor.org_unit_type \
       --table actor.org_unit > orgunits.sql
   

   Then drop all of the existing org_units and org_unit_types and load your custom data in a psql session:

   BEGIN;
   SET CONSTRAINTS ALL DEFERRED;
   DELETE FROM actor.org_unit;
   DELETE FROM actor.org_unit_type;
   \i orgunits.sql
   COMMIT

4. Import bibliographic records using the standard marc2bre.pl / direct_ingest.pl / pg_loader.pl process. Point the --idfield / --idsubfield and --tcnfield / --tcnsubfield options for marc2bre.pl at 935a (yes, this sucks for title control numbers, but as noted above they are not guaranteed to be unique in Unicorn and we need uniqueness in Evergreen). We need the bibliographic record entry ID field to be the catalog key to set up subsequent call number/barcode matches.
5. Enable the subsequent addition of new bibliographic records by setting the sequence object values to avoid conflicting ID / TCN values by issuing the following SQL statements:

   SELECT setval('biblio.autogen_tcn_value_seq',
        (select max(id) from biblio.record_entry) + 100);
   SELECT setval('biblio.record_entry_id_seq',
        (select max(id) from biblio.record_entry) + 100);

6. Process holdings records.
   1. Call numbers might have MARC8 encoded characters, so process'em and convert to UTF8. Theoretically "yaz-iconv -f MARC-8 -t UTF-8 < holdings.lst > holdings_utf8.lst" should do it, but instead it eats linefeeds and creates an unusable field. Ugh. We use a little Python script instead that requires pymarc, which in turn requires a version of setuptools (0.6c5) newer than Debian Etch's packaged version (0.6c3). So:

      wget http://pypi.python.org/packages/2.4/s/setuptools/setuptools-0.6c9-py2.4.egg
      sudo sh setuptools-0.6c9-py2.4.egg
      sudo easy_install pymarc

   2. Now actually generate the 'holdings_utf8.lst' file.

      cat holdings.lst | python marc8_to_utf8.py

   3. Adjust parse_unicorn.py to match up the holdings fields (added flexkey to the start). Then parse the holdings_utf8.lst to generate an SQL file (holdings_eg.sql) that we can load into the import staging table.

      python parse_unicorn.py

      Note that the holdings data for the item with barcode 30007007751786 didn't process cleanly and won't load. Weird - possibly a corrupt character in the item data? Augh, no - there are flexkeys and callnumbers that contain '|' characters (16 occurrences for "|z", 37 for "|b"), which is of course also what we are using as our delimiters. ARGH. I deleted it for now with:

      grep -v '|z' holdings_utf8.lst > holdings_clean.lst
      grep -v '|z' holdings_clean.lst > holdings_clean.lst2
      mv holdings_clean.lst2 holdings_clean.lst

      Adjust parse_unicorn.py to match the new input name and generate a new holdings_eg.sql.
7. Create the import staging table:

   psql -f Open-ILS/src/extras/import/import_staging_table.sql

8. Load the items into the import staging table:

   psql -f holdings_eg_clean.sql

   We discover that some more of our data sucks - for example, one item ("Research in autism spectrum disorders", HIRC PER-WEB) has a create date of '0' which is not a valid date format because the barcode is "1750-9467|21". For now, grep it out as above and reload.
9. Investigate possibilities of collapsing unnecessary duplicate item types:

   SELECT item_type, COUNT(item_type)
   FROM staging_items
   GROUP BY item_type
   ORDER BY item_type;
   
    item_type  | item_count 
   ------------+------------
    ATLAS      |        162
    AUDIO      |        792
    AUD_VISUAL |       1790
    AV         |         69
    AV-EQUIP   |        182
    BOOK       |        996
    BOOKS      |     581592
    BOOK_ART   |          1
    BOOK_RARE  |       4949
    BOOK_SHWK  |          5
    BOOK_WEB   |      49163
    COMPUTER   |         33
   ...
   (40 rows)

   How about locations?

   SELECT location, COUNT(location)
   FROM staging_items
   GROUP BY location
   ORDER BY location;
   
     location  | count  
   ------------+--------
    ALGO-ACH   |     13
    ALGO-ATLAS |    148
    ALGO-AV    |   1837
   ...
   (212 rows)

   Now we can collapse categories pretty simply inside the staging table. For example, if we want to collapse all of the BOOK types into a single type of BOOK:

   UPDATE staging_items
   SET item_type = 'BOOK'
   WHERE item_type IN 
   ('BOOKS', 'BOOK_ART', 'BOOK_RARE', 'BOOK_SHWK', 'BOOK_WEB', 'REF-BOOK');

10. Update legacy library names to new Evergreen library short names (we're using OCLC codes where possible). Some will be straightforward old names to new names. Others will require a little more logic based on location + legacy library name; we're splitting the DESMARAIS collection into multiple org-units (Music Resource Centre, Hearst locations, hospital locations, etc).

    -- Laurentian Music Resource Centre
    UPDATE staging_items
    SET owning_lib = 'LUMUSIC'
    WHERE location = 'DESM-MRC';
    
    -- Hearst - Kapuskasing location
    UPDATE staging_items
    SET owning_lib = 'KAP'
    WHERE location LIKE 'HRSTK%';
    
    -- Hearst - Timmins location
    UPDATE staging_items
    SET owning_lib = 'TIMMINS'
    WHERE location LIKE 'HRSTT%';

11. Generate the copies in the system:

    psql -f generate_copies.sql

12. Make the metarecords:

    psql -f quick_metarecord_map.sql

Ah, recognize that any electronic resources (which don't have associated copies) won't appear. Check for 856 40 and change the bre source to a transcendent one mayhaps?

-- Create a new transcendant resource; 
-- this autogenerates an ID of 4 in a default, untouched system
INSERT INTO config.bib_source (quality, source, transcendant)
VALUES (10, 'Electronic resource', 't');
-- Make the electronic full text resources (856 40) transcendant
-- by setting their bib record source to the new bib_source value of 4
UPDATE biblio.record_entry 
SET source = 4 
WHERE id IN (
    SELECT DISTINCT(record) 
    FROM metabib.full_rec 
    WHERE tag = '856' AND ind1 = '4' AND ind2 = '0'
);

And no transcendence. Hmm. Oh well, worry about that later.

Update 2009-02-19: uploaded diffs from Evergreen 1.4.0.2 (EG_exposed.tar.gz) for adding details to record summary; and Bill Erickson's slides and code examples are also available for download

The slides: Evergreen exposed, part 1 (OpenOffice)

My second presentation at the OLA SuperConference 2009 was Evergreen Exposed: hacking the open library system, which promised to take attendees on a tour of the architecture and source code of the Evergreen library system. I was very fortunate to have Bill Erickson, one of the original Evergreen developers, agree to join me as a co-presenter. Given the hour-and-fifteen-minute time slot that we were allotted, we opted to take an incremental approach to introducing parts of Evergreen to the audience, starting with basic tasks and working up to more complex customisations. We also tried to focus on answering questions that had been posted to the Evergreen mailing lists to ensure that we would satisfy our target audience's interests.

Dan starts with the basics

I started the session with an introduction of how to create a different skin for the catalogue, starting with text, CSS, JavaScript, and images and extending to the translation and customization framework. We talked about how to future-proof your customizations against future upgrades and how consortia can use skins to provide not just different look-and-feel, but different functionality, for each member of the consortium. Not much more than XML entities defined by DTDs, massaged via Apache server side includes (SSI), but it's an important conceptual building block for both the catalogue and the staff client.

I then ran through the exercise of adding a new metadata export format that brought the Federal Geographic Data Committee's Content Standard for Geospatial Data Metadata (FGDC CSGDM) format to Evergreen's existing list of supported formats. On the one hand: big deal, another metadata format. Hold that thought in that one hand; we'll come back to it later.

I also walked through two other common requests on the mailing lists: how do I define a new index or tweak the behaviour of an existing index and how do I hide or show more information on the detailed record display page? I'll follow up with separate posts for each of these pieces to augment what you have before you in the slides; suffice to say that there's a lot of MODS, a little bit of JavaScript, a smidgin of XPath, a dollop of Evergreen's interface definition language (IDL), and a slice of Perl mixed together. Along the way, I peeled back the covers to show a bit of OpenSRF in operation, setting up Bill's part of the show...

Bill leads us into the promised land

Note I'll update this with a link to Bill's slides when he manages to post them!

Bill gave a quick "big picture" view of how OpenSRF operates, including a much clearer overview of Evergreen's object-relational IDL that maps objects to relational tables. He also covered the cstore OpenSRF application that offers access to the underlying database without requiring SQL but still with support for full transactions (commit/rollback) and sub-transactions (savepoints). During Bill's demonstrations of these features, he exercised srfsh in a way that was new to me - he used the introspect command with a partial method name to perform a left-anchored search for matching method names. Cool!

Oh, and he also showed that if OpenSRF would normally return a reference to an object defined in the IDL, you can ask it to flesh the object in-place with its complete set of attributes instead; and of course if any of those attributes are object references, you have the option of fleshing those as well. It's a lovely way to cut down on chattiness in your application.

From there, Bill whipped out DojoSRF, the OpenSRF-aware extensions for dojo, the JavaScript toolkit that Evergreen adopted as its core JavaScript framework in release 1.4. In 90 lines of HTML and JavaScript code, he implemented a basic but workable catalogue - and then, with a few more lines of code, he gave the audience the payoff for that FGDC CSGDM (geographic metadata) format that I had earlier hacked into Evergreen. As part of the transform separates out the geographic coordinates of the subject matter (in the case of our demo data, maps of Northern California), Bill was able, in just a few more lines of code, to easily extract the coordinates from the FGDC CSGDM representation of the bibliographic material and plot the bounding box for the coverage area on a Google Map image. Very cool.

We had about 15 to 20 people attend our session, and I was happy with that attendance given the extremely technical content and relatively niche product. If as a result we end up adding just one more developer to the Evergreen community, that would be a great outcome. And for myself, I was forced to learn much more of Evergreen - just in time for Project Conifer, I hope

Updated 2009-02-02 to add PDF formatted slides because the free and libre formats just isn't good enough for some people - heh

The slides, up front and center:

• OpenOffice Impress format (ODP)
• Portable Document Format (PDF)

Last year I gave a presentation at the OLA SuperConference 2008 on The State of Evergreen. Yesterday, John Fink and I gave an update on the state of Project Conifer, the partnership between Algoma University, Laurentian University, Northern Ontario School of Medicine, and the University of Windsor to mount a consortial instance of Evergreen for our respective academic libraries.

McMaster University (John Fink's employer) is another Project Conifer institutional partner, albeit with a slightly different relationship. They are contributing resources towards development of academic features, but working towards their own Evergreen instance on their own timeline. Their relationship in the project changed the week before our presentation, so John and I had a fun time adjusting our presentation to match the new reality

In comparison to last year, which was largely an introduction to Evergreen and the state of its various features, this session was much more focused on Project Conifer. John gave the background of the project and the importance of having an open source library system at the core of our academic libraries, particularly given the short-term challenges that most of the Project Conifer participants face with their/our current library systems. I focused on the challenges and lessons learned in managing the project, with most of the challenges being the difficulty of getting skilled resources to work on our development requirements, and most of the lessons learned being in working out cost-sharing agreements and priority-setting procedures early on in the project.

The session was well-attended, and there is clearly growing interest in Evergreen as a viable option, as well a a bit of frustration at the pace of development of some of the features that academics in particular are interested in. These are "interesting times" for academic libraries - this week an announcement has been rippling through the Ontario library community that the BiblioCentre consortial library system that has served many Ontario college libraries since 2003 is being shut down. If Evergreen's academic features were already in place, it would have been a slam-dunk to put together a business case for a centrally hosted Evergreen system to serve the same constituency. As those features are still in active development, it's not quite as easy to make that business case.

Happily, Art Rhyno and Graham Fawcett have taken support for academic reserves for managing both print and electronic materials from ground zero to a reasonable interface in just a few months. They expect to start wiring in direct Evergreen support over the next few months so that we will have a functioning reserves system that goes far beyond our current library system's capabilities ("our" being Laurentian University, in this case).

After an exciting drive from Buffalo on a very snowy Wednesday afternoon, Bill Erickson of Equinox Software Incorporated gave Project Conifer participants a demo of the current state of acquisitions on Wednesday night, and it's not too far from meeting our base requirements. Equinox has hired a second developer to contribute to acquisitions development, documentation is being concurrently produced, and one of Project Conifer's contractors is working on adding EDI support. So we're optimistic that a functioning base acquisitions system will be in place in May - although, as one of our collection development librarians has wryly noted, our budgets might not have any room for book purchases in the coming fiscal year in any case.

A highlight of the session was when I asked Susan Downs, CEO of the Innisfil Public Library, to talk about their success story. In October 2008, Innisfil announced to the library world that they had migrated to Evergreen without any vendor assistance - certainly the first known instance in Ontario, and possibly the first self-migrated and self-supported public library on Evergreen in the world. It was great to meet the people behind that project and I was glad to let Susan share some of her energy, enthusiasm, and insights with our audience.

I had some feedback from one attendee who was happy to see a presentation on an in-process project, with warts and all exposed, rather than the usual post-project stories that quickly put the rough patches behind them (or forget them entirely). I'm happy to do as good a job as I can to represent an objective look at the project - for one thing, it's my job as project manager - and I hope that in some small way I've been able to help others prepare for similar projects.

Just like my last entry, this is a preview of one part of my upcoming session at the OLA SuperConference, Evergreen Exposed: Hacking the open source library system. We know from the last entry that Evergreen internally converts MARC21 to MODS to support item display; and in fact it also includes support for exposing records as OAI, RDF, SRW, and HTML. Today, we're going to be looking at adding support for an entirely new metadata format to Evergreen.

Back in November, 2008, George Duimovich requested "I would like to hear from anyone on the process for adding an additional supported format" in the specific context of the FGDC metadata format for digital geospatial data. George did a great thing to support his request and included links to the metadata format itself, along with a pointer to an XSLT stylesheet that the inestimable Terry Reese had written and published for converting MARC21 to FGDC XML. His request has been burning at the back of my mind since then, partially because I had quickly responded with the oh-so-helpful:

Assuming that we can get over the licensing hump, it should be a relatively straightforward matter of dropping the transform into Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm and Open-ILS/src/perlmods/OpenILS/WWW/SuperCat/Feed.pm (using something like MODS32 as a template).

Simple and straightforward, right? Well... yes and no. I had just gone through the process of adding MODS 3.2 support because I needed the more granular treatment of URLs to fix an item display problem, so I was pretty comfortable with the code at the time. After a few months, that familiarity goes away and one gets to go through the discovery process again. (Oh, and about a week after the MODS 3.2 support went in and Mike Rylander went the extra mile to update all of the indexes to use MODS 3.2, MODS 3.3 was released to the world. Sigh).

Without further ado, following are the diffs required to roughly support FGDC as a SuperCat format:

dbs@dbs-laptop:~/source/Evergreen-rel_1_4$ svn diff Open-ILS/src/perlmods/
Index: Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm
===================================================================
--- Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm	(revision 11952)
+++ Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm	(working copy)
@@ -143,6 +143,18 @@
 	# and stash a transformer
 	$record_xslt{rss2}{xslt} = $_xslt->parse_stylesheet( $rss_xslt );
 
+	# parse the FGDC xslt ...
+	my $fgdc_xslt = $_parser->parse_file(
+		OpenSRF::Utils::SettingsClient
+			->new
+			->config_value( dirs => 'xsl' ).
+		"/MARC21slim2FGDC.xsl"
+	);
+	# and stash a transformer
+	$record_xslt{fgdc}{xslt} = $_xslt->parse_stylesheet( $fgdc_xslt );
+	$record_xslt{fgdc}{docs} = 'http://www.fgdc.gov/metadata/csdgm/index_html';
+	$record_xslt{fgdc}{schema_location} = 'http://www.fgdc.gov/metadata/fgdc-std-001-1998.xsd';
+
 	register_record_transforms();
 
 	return 1;

If you're still with me after that whack of code, and you're counting, that's about 12 lines of code. Okay, I'm cheating - the diff doesn't include the MARC21 to FGDC stylesheet - for one thing, I'm still waiting to see a version of the stylesheet with a license attached to it. For another, do you really want to see all that XSL? After you patch your copy of OpenILS::Application::SuperCat.pm, copy the MARC21 to FGDC stylesheet into /openils/var/xsl, and restart the Evergreen Perl services, you'll be able to take advantage of the new functionality. That's it!

What's going on in this code? This patch against Open-ILS/src/perlmods/OpenILS/Application/SuperCat.pm enables SuperCat (and therefore unAPI) support for the new format. We just add an entry to the hash of XSLT stylesheets that SuperCat knows about, and the rest is visible in URLs like:

• http://localhost/opac/extras/supercat/formats/record - list of supported record formats
• http://localhost/opac/extras/supercat/retrieve/fgdc/record/1 - display record #1 in FGDC format
• http://localhost/opac/extras/unapi?id=tag:localhost,2009:biblio-record_entry/1 - display the record formats that unAPI can return
• http://localhost/opac/extras/unapi?id=tag:localhost,2009:biblio-record_entry/1&format=fgdc - return record #1 in FGDC format via unAPI

So who cares about this? Well, George cares, and (I'm guessing wildly here), perhaps it's because his organization has tools that can import FGDC but that also want to maintain the data in their library catalogue because they love MARC. That might be sufficient reason. Another reasonable use case would be to use the FGDC transform to populate spatial data tables built on the geospatial extensions offered by PostGIS and index these for lightning-fast retrieval of maps and map data that cover a given range of coordinates.

I'm sure the same approach could be used for other specialized metadata formats. This is just one example of why I'm sold on Evergreen's capability as a platform for the future of our library.

This is a preview of one part of my upcoming session at the OLA SuperConference, Evergreen Exposed: Hacking the open source library system. In the Conifer implementation of Evergreen, at least one of the partners plans to use a decoupled discovery layer rather than the Evergreen OPAC. So we needed to answer the typical question "How do I retrieve the availability of copies for a given work at my institution?" Note that this mini-tutorial is based entirely on OpenSRF 1.0 / Evergreen 1.4; OpenSRF 0.9 will generate different JSON output, and the URL for the OpenSRF gateway will be different.

Learning from the old masters: how the Evergreen OPAC does it

The Evergreen OPAC itself relies heavily on JavaScript to dynamically flesh out item details and retrieve item status, so it's actually pretty easy to work out how to do this without even delving too deeply into OpenSRF. First, let's use the Firebug Mozilla extension to follow network requests for a given "title details" page in the OPAC search results for the title: The new world guide to beer. Open up Firebug, enable network monitoring for the OPAC site, and watch the requests flood past for the title details page. We can see that there are a number of POST requests to http://dev.gapines.org/osrf-gateway-v1:

• POST request #1 parameters
  • method = open-ils.search.biblio.record.mods_slim.retrieve
  • service = open-ils.search
  • locale = en-US
  • param = 8526
  This is how we retrieve the title / author / ISBN and other bibliographic details of interest for display; as we're talking about a decoupled discovery layer, we won't need to worry about this piece of the puzzle.
• POST request #2 parameters
  • method = open-ils.search.config.copy_status.retrieve.all
  • service = open-ils.search
  • locale = en-US
  This is how we retrieve the list of all possible copy statuses that have been configured for this Evergreen system; here's the response (truncated for legibility):

  {
     "status" : 200,
     "payload" : [
        [
           {
              "__c" : "ccs",
              "__p" : [
                 null, null, null, "f", 3, "Lost", "f"
              ]
           },
           {
              "__c" : "ccs",
              "__p" : [
                 null, null, null, "t", 0, "Available", "t"
              ]
           },
           {
              "__c" : "ccs",
              "__p" : [
                 null, null, null, "t", 1, "Checked out", "t"
              ]
           },
           {
              "__c" : "ccs",
              "__p" : [
                 null, null, null, "f", 2, "Bindery", "t"
              ]
           }
        ]
     ]
  }
  

  We're getting a response in JavaScript Object Notation (JSON) format - the nice, compact, easy-to-read data interchange format that almost every programming language under the sun can interpret and generate. Yay!
• POST request #3 parameters
  • method = open-ils.search.biblio.copy_counts.summary.retrieve
  • service = open-ils.search
  • locale = en-US
  • param = 8526
  • param = 1
  • param = 0
  This is how we retrieve the call numbers, copies, and copy status for a given title. We pass in the the TCN input parameter ("8526"), the numeric ID of the organization being searched ("1" = "every branch"), and the depth of the organization ("0" = top of the hierarchy). The response for this request is:

  {
     "status" : 200,
     "payload" : [
        [
           [
              "127",
              "663.42 JACKSON, MICHAEL",
              {
                 "0" : 1
              }
           ],
           [
              "130",
              "663.42 JACKSON, MICHAEL",
              {
                 "0" : 1
              }
           ],
           [
              "125",
              "663.42 JACKSON, MICHAEL",
              {
                 "0" : 1
              }
           ],
           [
              "34",
              "R 641.23 JACKSON, MICHAEL",
              {
                 "0" : 1
              }
           ]
        ]
     ]
  }
  

Interpreting the HTTP requests and responses

Okay, so we've found a couple of requests that are pertinent to our goal. And you might be able to guess that the fifth element of the __p entry in the copy status response is the numeric identifier for the copy status, while the sixth element is the copy status name (which, as of OpenSRF 1.0 / Evergreen 1.4, if you pass a different locale value can return a translated value).

You might even be able to guess that the response from the copy_counts.summary request returns an array of responses consisting of the organization ID, the call number, and a hash of copy status and the respective counts for each copy status. And you would be guessing correctly. But why guess, when you can get an authoritative interpretation by looking up the class hint (the __c value in the copy_status response of "ccs") in Evergreen's intermediate definition language file /openils/conf/fm_IDL.xml:

<class id="ccs" controller="open-ils.cstore" 
  oils_obj:fieldmapper="config::copy_status" oils_persist:tablename="config.copy_status">
  <fields oils_persist:primary="id" oils_persist:sequence="config.copy_status_id_seq">
    <field name="isnew" oils_obj:array_position="0" oils_persist:virtual="true" />
    <field name="ischanged" oils_obj:array_position="1" oils_persist:virtual="true" />
    <field name="isdeleted" oils_obj:array_position="2" oils_persist:virtual="true" />
    <field name="holdable" oils_obj:array_position="3" 
      oils_persist:virtual="false" reporter:datatype="bool"/>
    <field name="id" oils_obj:array_position="4" 
      oils_persist:virtual="false" reporter:selector="name" reporter:datatype="id"/>
    <field name="name" oils_obj:array_position="5" 
      oils_persist:virtual="false"  reporter:datatype="text" oils_persist:i18n="true"/>
    <field name="opac_visible" oils_obj:array_position="6" 
      oils_persist:virtual="false" reporter:datatype="bool"/>
  </fields>

So now, by taking our first steps into Evergreen's object persistence model, we can determine authoritatively that the order of values in the __p array maps to "isnew", "ischanged", "isdeleted", "holdable", "id", "name", and "opac_visible". As for the response from the copy_counts.summary call, well, these are not Evergreen objects (they don't have a __c class hint) - but you can use the OpenSRF shell "srfsh" introspect command to view the documentation for the applicable method:

bash$ srfsh
srfsh# introspect open-ils.search
... (truncated for legibility) ...
Received Data: {
  "__c":"OpenILS_Application",
  "__p":{
    "api_level":1,
    "stream":0,
    "object_hint":"OpenILS_Application_Search_Biblio",
    "package":"OpenILS::Application::Search::Biblio",
    "remote":0,
    "api_name":"open-ils.search.biblio.copy_counts.summary.retrieve",
    "signature":{
      "params":[
        
      ],
      "desc":"returns an array of these: [
         org_id,
         callnumber_label,
         ,
         ,
        ...
      ]       where statusx is a copy status name.  the statuses are sorted by id.",
      "return":{
        "desc":null,
        "type":null,
        "class":null
      }      
    },
    "server_class":"open-ils.search",
    "notes":"\treturns an array of these:\n\t\t[
       org_id,
       callnumber_label,
       ,
       ,
      ...
    ]    \n\t\twhere statusx is a copy status name.  the statuses are sorted\n\t\tby id.\n",
    "method":"copy_count_summary",
    "argc":0
  }  

The introspect output is a bit rough - it's really intended for the doxygen API help interface - but it's good enough for our purposes. If we want to dig into what's going on under the covers, we can follow the package_name value "OpenILS::Application::Search::Biblio" to read the source code for the OpenILS::Application::Search::Biblio Perl module, and look up the method "copy_count_summary" as indicated by the "method" value in the introspect output. That reveals that the input arguments are "($self, $client, $rid, $org, $depth)". Every OpenSRF method automatically receives $self and $client as the first two arguments, so $rid (record ID), $org (organization unit ID), and $depth (organization unit depth) are the variables over which we have control.

Zeroing in on the copies for a particular library or library system

If we want to retrieve the visible copies for just a single organization unit in the entire Evergreen system, we just have to adjust the values of the organization unit ID and organization unit depth parameters accordingly. If we ask for the visible copies for just org_unit ID "125" at depth "2", we narrow down our results to a single hit:

{
   "status" : 200,
   "payload" : [
      [
         [
            "125",
            "663.42 JACKSON, MICHAEL",
            {
               "0" : 1
            }
         ]
      ]
   ]
}

So, with all of that ammunition at your disposal, you can write an Evergreen copy status lookup in any decoupled discovery layer that supports HTTP POST or GET requests. Which should be pretty much any discovery layer, right?

Frequently used tools and methods for Evergreen / OpenSRF hacking

Note, the first: you can easily play with different parameter values for the HTTP POST requests using the json_xs command to pretty print the JSON response:

curl -d service=open-ils.search   -d locale=en-US \
  -d method=open-ils.search.biblio.copy_counts.summary.retrieve \
  -d param=8526 -d param=1 -d param=0 \
  http://dev.gapines.org/osrf-gateway-v1 | json_xs -t json-pretty

Note, the second: the OpenSRF gateway also supports GET requests; simply concatenate the request parameters in a single URL like this

.

As I announced on the Evergreen mailing lists last night:

This release has been a long time in the making, and we'd love to have your help in testing it and flushing out bugs. Also, if you would like to contribute a translation, this is your chance to step up! We already have Brazilian Portugese (pt_BR), Georgian (ka), and Canadian English (en_CA) translations in the works, along with a commitment to update the Canadian French (fr_CA) translation. As this is the first real round of translations for Evergreen, I fully expect that there will be some work ahead of us to smooth out the translation process - but we have to take the plunge some time. Many thanks to Tigran Zargaryan and Natural Resources Canada for their respective contributions of the Armenian (hy_AM) and Canadian French (fr_CA) translations this summer; their willingness to be early guinea pigs for the translation process helped immensely.

Update: I noticed that the speedy Warren Layton beat me to the punch on the blog announcement of the releases. Warren's been very helpful with testing and suggestions for improvements to the documentation, so I don't mind being scooped at all

Just poked at the OLA SuperConference 2009 schedule (January 28 - 31, 2009) and found four sessions listed that are all about Evergreen. Wow! Check this out:

Date	Time	Title	Description (may be abridged)	Presenters
Thursday, January 29	9:05 am	It.s Just a Little Bit of Programming Isn.t It?	Follow the progress of the Library @ Mohawk.s development of the open source ILS Evergreen. Hear the trials and tribulations and learn from the mistakes and successes that have occurred along the way . we are truly a learning organization on this project. We went live in summer 2008 . come and hear about where we.ve been, where we are and where we hope to be soon.	Robert Soulliere, Systems Librarian; Cynthia Williamson, Collection & Access Librarian, Mohawk College of Applied Arts and Technology
Thursday, January 29	3:45 pm	Project Conifer: Evergreen library system for Ontario Universities	Find out how the Evergreen open source library system, originally developed for a public library consortium, is being adapted for academic libraries by three Ontario universities. Discussion will focus on the challenges, successes and mistakes (err, .learning opportunities.) of the project.	John Fink, Digital Technologies Development Librarian, McMaster University; Dan Scott, Systems Librarian, Laurentian University
Friday, January 30	9:05 am	Evergreen exposed: hacking the open source library system	Join an Evergreen developer on a tour of the architecture and source code of the Evergreen library system [...] Get ready to get your hands dirty with Evergreen . this will be a session filled with code!	William Erickson, Vice President, Software Development & Integration, Equinox Software Inc; Dan Scott, Systems Librarian, Laurentian University
Saturday, January 31	10:40 am	Multilingual Language Issues of Open Source ILS	Discover the Chinese version of Evergreen along with various multilingual issues related MARC standards, encoding, indexing, searching, and sorting especially associated with Chinese language.	Jason Zou, Systems Librarian, Lakehead University; Guoying (Grace) Liu, Systems Librarian, Leddy Library, University of Windsor

I was responsible for the sole Evergreen presentation at OLA SuperConference 2008 - it's awesome to see a lot more people jumping in this year! I'm keenly anticipating this conference - we'll have to set up at least one Evergreen "Birds of a Feather" session.

In a few minutes I'll be giving a talk with John Fink at the Free Software Open Source Symposium at Seneca College on Evergreen: an enterprise-strength OSS solution for library ossification. I'm jazzed!

Here are the slides: (ODP format) (PDF format).

Update: 2008-10-07 As of changeset 10774, the detailed record view in Evergreen's dynamic catalog is now recognized by Zotero.

I really like Zotero. And it works really well with Evergreen's current "basic search" because it embeds unAPI links that enable Zotero to consume MODS representations of the underlying bibliographic records and generate a complete citation based on that.

However, Zotero doesn't work with Evergreen's current "dynamic search" interface - which is a problem, because it is the default search interface. Evergreen embeds a link to the unAPI server, and fills in the unAPI link via an AJAX call after the underlying XHTML has been loaded - but it seems that Zotero doesn't recognize that the DOM has been changed by the AJAX event and never discovers the unAPI link. So... I had submitted a challenge to Hackfest to fix this, because I really want to be able to use Zotero with Evergreen when Project Conifer launches.

And, as with every other Hackfest I have attended, I end up working on my own challenge. In discussing the problem with William from canadiana.org and Walter Lewis from Knowledge Ontario, I described how the dynamic interface doesn't use any templating (apart from entity substitution for localization support), that there wasn't really any way to inject content server side into the underlying XHTML, and that I really didn't want to have to dig into the guts of Zotero to enable it to parse the DOM after events had completed. William asked "so you can't even do a server side include?", which ended up breaking the problem wide open - because yes, we already use server side includes to identify which DTD to load for localization purposes.

Step 1 was to modify the detailed record display to put the unAPI link template in place, and to modify the Apache configuration to pass in hardcoded values for each of the SSI variables. A quick test and - it didn't work. Uh oh.

That led to much scratching of the head. Was Zotero getting tripped up by the masses of XHTML elements in the dynamic template that are simply hidden? Did it give up after trying to parse 100K or so of content? Were there differences in the content types being served up by Apache? The next step was to compare the content of the "basic search" output against the "dynamic search" output - and that led to one seemingly innocent difference.

The unAPI server link in the "basic search" output included an absolute link to the server, while the corresponding link in the "dynamic search" output used a relative link to point to the root of the server. I didn't think that would be a problem, but eliminating variables is always good - and when I tested with a hardcoded server link, the Zotero hint icon lit up and the mystery was solved. Between enabling the record unAPI link to appear in the static XHTML via SSI and changing the unAPI server link to use an absolute value, Zotero and Evergreen could work together in harmony.

I haven't committed the fix for this yet to the repository, as I haven't finalized the exact SSI incantations that will be needed to embed the record ID in the unAPI link. But now you know the solution, and could tackle the problem yourself if you get tired of waiting for me and feel inspired. And once the problem is fixed, I'll update the post to let you know what version of Evergreen carries the fix.

Oh, and my hackfest report slides are attached, in case anyone cares.