Coffee|Code : Dan Scott

The following summary is taken almost directly from an email I wrote to one of the would-be participants who was, sadly, prevented from making it to Yerevan due to travel complications. I meant to clean this up earlier and post it, but have not yet found the time - so I might as well just post it as is with most names obfuscated and possibly some additional editorial comments. Those who are new to installing and configuring Evergreen might find this useful; and reading through it, I remembered a few challenges I planned to tackle

Shortly after I arrived on Monday, I was able to try out the install of Evergreen 1.2.1.4 that A. and G. from the Fundamental Science Library (FSL) had completed with only two email exchanges with me. I was very happy to see that they had successfully completed the install! There was only one minor problem with the structure of the "organizational unit" hierarchy that I had to fix. After that, we confirmed that we were able to import bibliographic records from Z39.50 and attached call numbers and copies to those records. Finally, we tried searching for the records in the catalogue and were delighted to see that everything was working as we had hoped. That allowed me to sleep well on Monday, in preparation for the first day of the workshop on Tuesday.

After the introductions of the workshop participants on Tuesday, I gave the introduction to Evergreen presentation and Henri Damien Laurent of BibLibre demonstrated Koha. Both Henri Damien Laurent and I showed our respective library systems running with an Armenian interface, thanks to the translation efforts of Tigran! Then we broke into separate Koha and Evergreen groups to work together on our respective library systems. Of the attendees of the workshop, E. was the most interested in migrating his library (with 40,000 volumes) to Evergreen. A., from one of the 29 branches of the American University of Armenia (AUA), also attended most of the Evergreen session. Even though his institution is mostly interested in Koha, he wanted to be able to compare the two systems. Albert's colleague S. attended the Koha training session so they would be able to compare their experiences later. Our group also had R. from the Netherlands and A., G., and A. from FSL -- apparently Tigran is considering running Evergreen as a union catalogue, so his IT people are very interested in learning more.

Our first exercise was to model the organizational unit hierarchy using the configuration bootstrap interfaces in the /cgi-bin/config.cgi. We began by drawing the hierarchy on a whiteboard. The "Yerevan Consortium" represented the Evergreen system as a whole; we added the FSL, MSU, and AUA systems as children of the Yerevan Consortium, and then added specific branches as children of each of these systems. While we were creating this hierarchy, I showed the participants how the organization unit type defines the labels used in the catalogue as well as the respective depth in the hierarchy for each type.

We then ensured that the systems and branches in the hierarchy had the right types, and that the types were defined with valid parent-child relationships. We found a few types that were children of themselves, which causes a problem in searching. There was also some confusion about the role of types to organization units, resulting in the creation of types with labels like "FSL" rather than "Library System". After a few minutes of explanation and working through correcting the exercises, I think the participants were better able to understand the relationship between types and organization units.

After we were satisfied with the structure of the organization unit hierarchy, I ran the autogen.sh script to update the catalogue and staff client representations of the hierarchy. Well, first I demonstrated how search in the catalogue will quickly be broken if you do not run the autogen.sh script

Our next step was to register new users with the Evergreen staff client. This helped introduce the participants to the staff client, as well as giving them a quick introduction to some parts of Evergreen that still need to be localized to allow regional variations on postal code formats, telephone numbers, and forms of identification. The default Evergreen staff client still enforces American conventions, but fortunately I have had to create patches for Evergreen to support my own country's standards so I can assure you that it is relatively easy to change or remove these format checks. In the future, it would be wonderful to include a localization pack for each locale interested in using Evergreen that supports regional variations on date formats, phone number patterns, etc. The participants were pleased with the feedback mechanism in the staff client that summarized all of the remaining problems with the current patron record (missing address, invalid phone number, etc) and made it easy to switch between screens without losing any of the data they had already entered.

Once we had registered new users for each of our branches, we went to work importing new bibliographic records and attaching call numbers and copies to those records. This gave us a good opportunity to see how changing the scope of a search in Evergreen from "Everywhere" down to a specific branch changes the search results, and demonstrated how the organization type labels are displayed in the catalogue. As an aside, I should point out that in Evergreen 1.4 (due by the end of this summer), the labels are internationalized so that different labels can be displayed depending on the locale in which you are using the catalogue or staff client. Good news for those of us who work in bilingual or multilingual libraries!

Now that we had records with copies attached and patrons registered in our Evergreen instance, we were able to use the catalogue's "My Account" features to try out features like sharable bookbags, account preferences, and the account summary. Users also have the ability to specify their own user names and to log in with those instead (which means that they can simply remember their unique nickname rather than, say, a 14-digit barcode).

The first feature that the participants discovered, of course, was the strong password enforcement feature. When a patron is registered, the system automatically generates a random 4-digit password; however, this is not considered to be a safe password, so when they log in they are forced to change it to a longer password containing both numbers and letters.

At this point, we also discovered a data validation bug: in the staff client, it is possible to enter a user barcode that consists of letters and numbers. However, in the catalogue, user barcodes containing letters are considered invalid and the system will not even attempt to log that user in; it simply rejects the barcode. I plan to ask E. to report this bug to the Evergreen mailing list; it would be an excellent outcome of the workshop if participants felt comfortable reporting problems to the mailing list, and reporting this problem in particular would help improve the quality of Evergreen.

Things were going reasonably well, but we noticed that the system was running into a problem if you tried to edit a bibliographic record after you had already created or imported the record. I had rather fortunately already experienced this problem (it is a result of different behaviour regarding XML namespaces between different versions of LibXML2) and knew that it had been fixed in 1.2.2.1. So rather than trying to fix the problem with the installed version of 1.2.1.4, I decided to try upgrading our Evergreen system to the recently released 1.2.2.1 to demonstrate to the participants that the upgrade process was fast, reasonably well documented, and not nearly as complicated as the install process. This was, by the way, something Randy had urged me to do, so I blame him for the subsequent problems we experienced (hah!).

The first problem is that the change from 1.2.1.x to 1.2.2.x requires the installation of a new Perl module from CPAN (JSON::XS). This is not much of a problem in itself, as the module is very easy to install and compile; however, given our internet connection I had to wait a long time for the CPAN repository metadata to be downloaded. The participants were still able to use the system while this was happening, but we ended up hitting the coffee break still waiting for CPAN to finish. (As an aside, Irakli and I were discussing the possibility of having the eIFL-FOSS coordinators investigate setting up local mirrors of FOSS resources like CPAN to speed up access to frequently used resources).

When we returned from the coffee break, the JSON::XS install had finished but the participants were having problems searching and using the staff client. I checked the logs (using the "grep ERR /openils/var/log/*" command to start with) and saw that our database connections were dying for some reason. On a hunch, I checked the system logs ("dmesg") and discovered that the Linux "out of memory (OOM) killer" had started killing random processes to try to free up memory. It was killing the PostgreSQL processes, the Evergreen processes - anything! I was lucky, because I had been reading about the OOM on Linux after hearing about a Linux user that had run into a similar problem, and knew that the way to disable the OOM was to prevent Linux from overcommitting memory to processes in the first place. Wondering why our system had started running out of memory in the first place, I ran "free" and saw that it had been set up with no swap space; I confirmed this by running fdisk to see that there were no swap partitions. Here, however, I made a mistake. I ran "echo '2' > /proc/sys/vm/overcommit_memory" to prevent Linux from overcommitting memory to new processes and to prevent the OOM killer from killing any more random processes. But this also meant that I was immediately unable to launch any new programs - so I could not safely shut down PostgreSQL and Evergreen, and we had to turn the power off to the system.

Fortunately, the system started up cleanly again (hurray for journalled filesystems) and I was able to complete the upgrade before the rest of our hands on session for the day was finished. A few things that are missing in the current upgrade instructions:

1. You have to compile the new version of Evergreen. The easiest way to do this is to copy install.conf over from your previous version of Evergreen and run "make config" to ensure that all of the settings are still correct, then run "make" to build the new version of Evergreen.
2. Very important: Before installing the new version of Evergreen, you must prevent the database schema from being completely recreated or it will destroy any data that is already in your system. One way of doing this is, during the "make config" step, to list all of the Evergreen targets _except for_ openils_db. I am simply incapable of remembering all of those targets, so my dirty workaround is to open Open-ILS/src/Makefile in an editor and modify the "install: " make target by removing the "storage-bootstrap" make target. What we really need is an "upgrade" target for "make config" that simply installs everything except for the database schema.
3. Confirm that the new version of Evergreen has been installed by running the srfsh command "request open-ils.storage open-ils.system.version".

For tomorrow (today, by the time you receive this), A. and G. are going to create a swap file to enable the system to swap memory to disk if need be; the system has 1 GB of RAM, which is enough for a small Evergreen system but when one is compiling programs at the same time as running Evergreen swap space really is necessary. This was a very good lesson learned for all of us!

E. also interested in learning more about basic Linux administration. His institution currently runs on an entirely Windows infrastructure, so the requirement to learn Linux is a fairly high hurdle. I'm hoping that the eIFL-FOSS list will be a good resource for him to start that journey. He has also asked to go over the step-by-step instructions for installing Evergreen, so I'm considering starting that in a VMWare session so that we can run through the steps. Our major goal for tomorrow is to migrate some data from FSL's legacy system into Evergreen. Wish us luck!

Editorial comment: The combination of Armenian and Russian MARC records refused to load into the Evergreen 1.2.2.1 system, but on the flight home I confirmed that they loaded perfectly and were searchable on my Evergreen development system. As the development version will become this summer's 1.4 "internationalization" release, we are in good shape.

Editorial comment 2:On the second day, while running in circles trying to figure out why the records were refusing to load into the 1.2.2.1 system, I decided to try the #openils-evergreen IRC channel. Yerevan is 9 hours ahead of the Toronto/Atlanta time zone, so at noon Yerevan time I was hardly expecting any of the current core Evergreen developers to be online - yet, to our amazement, Mike Rylander responded. This was a pretty convincing demonstration to the attendees that the core developers really aren't far away or hard to contact at all.

As Mark Leggott mentioned in Vendor to Open Source ILS in 1 Month #1, I had the pleasure of assisting the migration of the University of Prince Edward Island library system from Unicorn to Evergreen. A little over a year ago, in discussing the business case for open source library systems, I stated that one of the problems we faced with migrations is that the license for a proprietary system often inhibits openly sharing of information about how to export data from those systems in machine-usable formats. Thus, the open source library community needs to encourage the development of "migration ninjas". Little did I know that I would soon join the guild of ninjas and become deadly and silent, and unspeakably violent(1)(2).

As a result, I have created a utility script that should be of assistance to SirsiDynix Unicorn or Symphony sites who are interested in exploring the possibilities offered by other library systems. The rather dryly named "export_unicorn.pl" script was added to the Unicorn API repository as entry # 228 today under a GPL v2 license(3). As the script uses the Unicorn/Symphony API, however, I am sadly (to the best of my knowledge) not free to simply share the script with anyone. Therefore, to gain access to the script you must be an API-certified Unicorn or Symphony customer. Still, by making an export script available to SirsiDynix customers that provides the raw data in a relatively standard output format, it should ease the effort required by the migration ninjas for open source systems to massage the data into the needed input formats, and to avoid the tetsu-bishi scattered by the proprietary systems in defence of "their" data(4)(5).

1. Barenaked Ladies, "The Ninjas". Their website is horrible Flash and JavaScript overkill but damnit Jim, they're musicians, not webmasters; the "Snacktime" album is especially recommended if you have kids.
2. Although I have to say I'm nowhere near as violent as Mike Rylander, who with his PostgreSQL-fu can carve seemingly any piece of data into the shape needed for import into Evergreen.
3. Thanks to Mark Leggott for insisting that I retain copyright over the scripts created during the UPEI migration and for allowing me to share those scripts in the appropriate avenues. It's another weapon (shuriken? ninja-to?) in the migration ninja arsenal.
4. This data does, after all, belong to the libraries who license a library system, but at least one company reportedly has a pattern of repeatedly removing interfaces that enable easy machine-readable access to library data...
5. I find myself being thankful that Unicorn does provide an API for generating machine-readable data exports; all that it cost our library was a week of my life and the associated training fees and travel expenses

I was in Armenia last week, leading a workshop on open source library systems along with Henri Damien Laurent from BibLibre. My charge was to introduce Evergreen and lead participants in two days of hands-on experience with the system; Henri took on the same task for Koha. I cannot say enough good things about our host for the workshop, the Fundamental Library of the National Academy of Sciences of Armenia headed up by Tigran Zargaryan; nor can I offer enough compliments to Randy Metcalfe on his skills in ensuring that everything ran smoothly; nor can I express how rewarding it was to meet representatives of so many different countries and how much I enjoyed their company! I look forward to helping the pilot sites succeed with their implementations.

So, for the short term, I'll simply link to the "Introduction to Evergreen" presentation that I gave at the start of the workshop in OpenOffice and PowerPoint formats (as I promised to the participants). In the next day or two I plan to post a summary of the workshop activities; some of the lessons learned; and where I think I'll focus my attention next.