Evergreen

The following summary is taken almost directly from an email I wrote to one of the would-be participants who was, sadly, prevented from making it to Yerevan due to travel complications. I meant to clean this up earlier and post it, but have not yet found the time - so I might as well just post it as is with most names obfuscated and possibly some additional editorial comments. Those who are new to installing and configuring Evergreen might find this useful; and reading through it, I remembered a few challenges I planned to tackle

Shortly after I arrived on Monday, I was able to try out the install of Evergreen 1.2.1.4 that A. and G. from the Fundamental Science Library (FSL) had completed with only two email exchanges with me. I was very happy to see that they had successfully completed the install! There was only one minor problem with the structure of the "organizational unit" hierarchy that I had to fix. After that, we confirmed that we were able to import bibliographic records from Z39.50 and attached call numbers and copies to those records. Finally, we tried searching for the records in the catalogue and were delighted to see that everything was working as we had hoped. That allowed me to sleep well on Monday, in preparation for the first day of the workshop on Tuesday.

After the introductions of the workshop participants on Tuesday, I gave the introduction to Evergreen presentation and Henri Damien Laurent of BibLibre demonstrated Koha. Both Henri Damien Laurent and I showed our respective library systems running with an Armenian interface, thanks to the translation efforts of Tigran! Then we broke into separate Koha and Evergreen groups to work together on our respective library systems. Of the attendees of the workshop, E. was the most interested in migrating his library (with 40,000 volumes) to Evergreen. A., from one of the 29 branches of the American University of Armenia (AUA), also attended most of the Evergreen session. Even though his institution is mostly interested in Koha, he wanted to be able to compare the two systems. Albert's colleague S. attended the Koha training session so they would be able to compare their experiences later. Our group also had R. from the Netherlands and A., G., and A. from FSL -- apparently Tigran is considering running Evergreen as a union catalogue, so his IT people are very interested in learning more.

Our first exercise was to model the organizational unit hierarchy using the configuration bootstrap interfaces in the /cgi-bin/config.cgi. We began by drawing the hierarchy on a whiteboard. The "Yerevan Consortium" represented the Evergreen system as a whole; we added the FSL, MSU, and AUA systems as children of the Yerevan Consortium, and then added specific branches as children of each of these systems. While we were creating this hierarchy, I showed the participants how the organization unit type defines the labels used in the catalogue as well as the respective depth in the hierarchy for each type.

We then ensured that the systems and branches in the hierarchy had the right types, and that the types were defined with valid parent-child relationships. We found a few types that were children of themselves, which causes a problem in searching. There was also some confusion about the role of types to organization units, resulting in the creation of types with labels like "FSL" rather than "Library System". After a few minutes of explanation and working through correcting the exercises, I think the participants were better able to understand the relationship between types and organization units.

After we were satisfied with the structure of the organization unit hierarchy, I ran the autogen.sh script to update the catalogue and staff client representations of the hierarchy. Well, first I demonstrated how search in the catalogue will quickly be broken if you do not run the autogen.sh script

Our next step was to register new users with the Evergreen staff client. This helped introduce the participants to the staff client, as well as giving them a quick introduction to some parts of Evergreen that still need to be localized to allow regional variations on postal code formats, telephone numbers, and forms of identification. The default Evergreen staff client still enforces American conventions, but fortunately I have had to create patches for Evergreen to support my own country's standards so I can assure you that it is relatively easy to change or remove these format checks. In the future, it would be wonderful to include a localization pack for each locale interested in using Evergreen that supports regional variations on date formats, phone number patterns, etc. The participants were pleased with the feedback mechanism in the staff client that summarized all of the remaining problems with the current patron record (missing address, invalid phone number, etc) and made it easy to switch between screens without losing any of the data they had already entered.

Once we had registered new users for each of our branches, we went to work importing new bibliographic records and attaching call numbers and copies to those records. This gave us a good opportunity to see how changing the scope of a search in Evergreen from "Everywhere" down to a specific branch changes the search results, and demonstrated how the organization type labels are displayed in the catalogue. As an aside, I should point out that in Evergreen 1.4 (due by the end of this summer), the labels are internationalized so that different labels can be displayed depending on the locale in which you are using the catalogue or staff client. Good news for those of us who work in bilingual or multilingual libraries!

Now that we had records with copies attached and patrons registered in our Evergreen instance, we were able to use the catalogue's "My Account" features to try out features like sharable bookbags, account preferences, and the account summary. Users also have the ability to specify their own user names and to log in with those instead (which means that they can simply remember their unique nickname rather than, say, a 14-digit barcode).

The first feature that the participants discovered, of course, was the strong password enforcement feature. When a patron is registered, the system automatically generates a random 4-digit password; however, this is not considered to be a safe password, so when they log in they are forced to change it to a longer password containing both numbers and letters.

At this point, we also discovered a data validation bug: in the staff client, it is possible to enter a user barcode that consists of letters and numbers. However, in the catalogue, user barcodes containing letters are considered invalid and the system will not even attempt to log that user in; it simply rejects the barcode. I plan to ask E. to report this bug to the Evergreen mailing list; it would be an excellent outcome of the workshop if participants felt comfortable reporting problems to the mailing list, and reporting this problem in particular would help improve the quality of Evergreen.

Things were going reasonably well, but we noticed that the system was running into a problem if you tried to edit a bibliographic record after you had already created or imported the record. I had rather fortunately already experienced this problem (it is a result of different behaviour regarding XML namespaces between different versions of LibXML2) and knew that it had been fixed in 1.2.2.1. So rather than trying to fix the problem with the installed version of 1.2.1.4, I decided to try upgrading our Evergreen system to the recently released 1.2.2.1 to demonstrate to the participants that the upgrade process was fast, reasonably well documented, and not nearly as complicated as the install process. This was, by the way, something Randy had urged me to do, so I blame him for the subsequent problems we experienced (hah!).

The first problem is that the change from 1.2.1.x to 1.2.2.x requires the installation of a new Perl module from CPAN (JSON::XS). This is not much of a problem in itself, as the module is very easy to install and compile; however, given our internet connection I had to wait a long time for the CPAN repository metadata to be downloaded. The participants were still able to use the system while this was happening, but we ended up hitting the coffee break still waiting for CPAN to finish. (As an aside, Irakli and I were discussing the possibility of having the eIFL-FOSS coordinators investigate setting up local mirrors of FOSS resources like CPAN to speed up access to frequently used resources).

When we returned from the coffee break, the JSON::XS install had finished but the participants were having problems searching and using the staff client. I checked the logs (using the "grep ERR /openils/var/log/*" command to start with) and saw that our database connections were dying for some reason. On a hunch, I checked the system logs ("dmesg") and discovered that the Linux "out of memory (OOM) killer" had started killing random processes to try to free up memory. It was killing the PostgreSQL processes, the Evergreen processes - anything! I was lucky, because I had been reading about the OOM on Linux after hearing about a Linux user that had run into a similar problem, and knew that the way to disable the OOM was to prevent Linux from overcommitting memory to processes in the first place. Wondering why our system had started running out of memory in the first place, I ran "free" and saw that it had been set up with no swap space; I confirmed this by running fdisk to see that there were no swap partitions. Here, however, I made a mistake. I ran "echo '2' > /proc/sys/vm/overcommit_memory" to prevent Linux from overcommitting memory to new processes and to prevent the OOM killer from killing any more random processes. But this also meant that I was immediately unable to launch any new programs - so I could not safely shut down PostgreSQL and Evergreen, and we had to turn the power off to the system.

Fortunately, the system started up cleanly again (hurray for journalled filesystems) and I was able to complete the upgrade before the rest of our hands on session for the day was finished. A few things that are missing in the current upgrade instructions:

1. You have to compile the new version of Evergreen. The easiest way to do this is to copy install.conf over from your previous version of Evergreen and run "make config" to ensure that all of the settings are still correct, then run "make" to build the new version of Evergreen.
2. Very important: Before installing the new version of Evergreen, you must prevent the database schema from being completely recreated or it will destroy any data that is already in your system. One way of doing this is, during the "make config" step, to list all of the Evergreen targets except for openils_db. I am simply incapable of remembering all of those targets, so my dirty workaround is to open Open-ILS/src/Makefile in an editor and modify the "install: " make target by removing the "storage-bootstrap" make target. What we really need is an "upgrade" target for "make config" that simply installs everything except for the database schema.
3. Confirm that the new version of Evergreen has been installed by running the srfsh command "request open-ils.storage open-ils.system.version".

For tomorrow (today, by the time you receive this), A. and G. are going to create a swap file to enable the system to swap memory to disk if need be; the system has 1 GB of RAM, which is enough for a small Evergreen system but when one is compiling programs at the same time as running Evergreen swap space really is necessary. This was a very good lesson learned for all of us!

E. also interested in learning more about basic Linux administration. His institution currently runs on an entirely Windows infrastructure, so the requirement to learn Linux is a fairly high hurdle. I'm hoping that the eIFL-FOSS list will be a good resource for him to start that journey. He has also asked to go over the step-by-step instructions for installing Evergreen, so I'm considering starting that in a VMWare session so that we can run through the steps. Our major goal for tomorrow is to migrate some data from FSL's legacy system into Evergreen. Wish us luck!

Editorial comment: The combination of Armenian and Russian MARC records refused to load into the Evergreen 1.2.2.1 system, but on the flight home I confirmed that they loaded perfectly and were searchable on my Evergreen development system. As the development version will become this summer's 1.4 "internationalization" release, we are in good shape.

Editorial comment 2:On the second day, while running in circles trying to figure out why the records were refusing to load into the 1.2.2.1 system, I decided to try the #openils-evergreen IRC channel. Yerevan is 9 hours ahead of the Toronto/Atlanta time zone, so at noon Yerevan time I was hardly expecting any of the current core Evergreen developers to be online - yet, to our amazement, Mike Rylander responded. This was a pretty convincing demonstration to the attendees that the core developers really aren't far away or hard to contact at all.

As Mark Leggott mentioned in Vendor to Open Source ILS in 1 Month #1, I had the pleasure of assisting the migration of the University of Prince Edward Island library system from Unicorn to Evergreen. A little over a year ago, in discussing the business case for open source library systems, I stated that one of the problems we faced with migrations is that the license for a proprietary system often inhibits openly sharing of information about how to export data from those systems in machine-usable formats. Thus, the open source library community needs to encourage the development of "migration ninjas". Little did I know that I would soon join the guild of ninjas and become deadly and silent, and unspeakably violent(1)(2).

As a result, I have created a utility script that should be of assistance to SirsiDynix Unicorn or Symphony sites who are interested in exploring the possibilities offered by other library systems. The rather dryly named "export_unicorn.pl" script was added to the Unicorn API repository as entry # 228 today under a GPL v2 license(3). As the script uses the Unicorn/Symphony API, however, I am sadly (to the best of my knowledge) not free to simply share the script with anyone. Therefore, to gain access to the script you must be an API-certified Unicorn or Symphony customer. Still, by making an export script available to SirsiDynix customers that provides the raw data in a relatively standard output format, it should ease the effort required by the migration ninjas for open source systems to massage the data into the needed input formats, and to avoid the tetsu-bishi scattered by the proprietary systems in defence of "their" data(4)(5).

1. Barenaked Ladies, "The Ninjas". Their website is horrible Flash and JavaScript overkill but damnit Jim, they're musicians, not webmasters; the "Snacktime" album is especially recommended if you have kids.
2. Although I have to say I'm nowhere near as violent as Mike Rylander, who with his PostgreSQL-fu can carve seemingly any piece of data into the shape needed for import into Evergreen.
3. Thanks to Mark Leggott for insisting that I retain copyright over the scripts created during the UPEI migration and for allowing me to share those scripts in the appropriate avenues. It's another weapon (shuriken? ninja-to?) in the migration ninja arsenal.
4. This data does, after all, belong to the libraries who license a library system, but at least one company reportedly has a pattern of repeatedly removing interfaces that enable easy machine-readable access to library data...
5. I find myself being thankful that Unicorn does provide an API for generating machine-readable data exports; all that it cost our library was a week of my life and the associated training fees and travel expenses

I was in Armenia last week, leading a workshop on open source library systems along with Henri Damien Laurent from BibLibre. My charge was to introduce Evergreen and lead participants in two days of hands-on experience with the system; Henri took on the same task for Koha. I cannot say enough good things about our host for the workshop, the Fundamental Library of the National Academy of Sciences of Armenia headed up by Tigran Zargaryan; nor can I offer enough compliments to Randy Metcalfe on his skills in ensuring that everything ran smoothly; nor can I express how rewarding it was to meet representatives of so many different countries and how much I enjoyed their company! I look forward to helping the pilot sites succeed with their implementations.

So, for the short term, I'll simply link to the "Introduction to Evergreen" presentation that I gave at the start of the workshop in OpenOffice and PowerPoint formats (as I promised to the participants). In the next day or two I plan to post a summary of the workshop activities; some of the lessons learned; and where I think I'll focus my attention next.

Okay, this is definitely a lame thing to be thinking about at midnight on a Saturday, but I was just playing with the shelf browser in the Evergreen representation of our 780,000 bibliographic records (okay, that is definitely the wrong thing to be doing at midnight on a Saturday). For some reason, I was wandering through the subject collection pertinent to librarians (pray for my soul), noticed a book that probably should have been discarded years ago, and thought "Gee, i don't want to deal with this right now, but wouldn't it be nice if I could just mark this Weed me and forget about it until Monday?"

Then I realized that that wouldn't be a stretch at all. In Evergreen, users have "bookbags" to which they can add items. These bookbags can be shared as RSS feeds and otherwise easily exported into other formats. If we were running Evergreen for real, I could create a "Weed me!" bookbag, add in the suspect along with a bunch of other festering tomes, and send the RSS feed to a student to perform the manual labour. Or perhaps the RSS feed gets aggregated with other weeders' feeds and a weeding list gets generated on a monthly basis for efficient labour practices. You get the idea.

Of course, you would really want to have more information than just the stock shelf browsing interface at hand when making weeding decisions. For example, you would need a tally of recorded uses displayed beside the item, with the ability to drill down for totals by year. If you participate in a consortial "last copy standing" program, you would want a quick check to see if any other institutions still hold a copy of the resource. So, an enhanced interface would be needed to provide an experience that combines the traditional weeding approach of roaming the stacks and generating reports of items matching some minimum age and minimum usage criteria.

Think about it a little further though (I'm sure you're thinking a lot faster than me at this point; you're probably having the luxury of reading this at the beginning of the day, coffee in hand, invigorated after an early morning run in the lingering late spring chill... or not), and there are points in our institutional workflows where we could naturally introduce weeding activities. How do we get to the point of having three editions of a given text on the shelf? If I have the 1995, 2003, and 2007 editions of a text, I can assure you that when I ordered the 2007 edition I had already checked our ILS to see if we had a copy of that edition already, and would have noticed the previous editions. At that point, I should have the ability to say "Oh - get rid of the 1995 edition now and once the 2007 edition is processed and on the shelf, cull the 2003 edition to boot." If I was designing an acquisitions module today, that's certainly something I would consider as a nice-to-have. Ahem.

Weeding 2.0 may not be a sexy subject. Google and Yahoo each turn up exactly four hits, none of them related to libraries, which is remarkable in this overly-hyped everything 2.0 world. But it's something we should consider in the design and tailoring of our library systems; and while it's not going to rank in my top level of priorities for Evergreen, it will work its way in there somewhere, sometime. Hopefully before the stacks in my subject areas buckle under the weight of unused, out-of-date books.

Update 2008-05-01: Fixed a typo for sysctl: -a parameter simply shows all settings; -w parameter is needed to write the setting. Duh.

Once you have decided on and acquired your test hardware for Evergreen, you need to think about tuning your PostgreSQL database server. Once you start loading bibliographic records, you might notice that after 100,000 records or so that your search response times aren't too snappy. Don't snarl at Evergreen. By default, PostgreSQL ships with very conservative settings (something like machines with 256 MB of RAM!) so if you don't tune those settings you're getting a false representation of your system's capabilities.

The "right" settings for PostgreSQL depend significantly on your hardware and deployment context, but in almost any circumstance you will want to bump up the settings from the delivered defaults. To give you an idea of what you need to consider, I thought I would share the settings that we're currently using on our Evergreen test server at Laurentian University. You might be able to use these as a starting point and adjust them accordingly once you've run some representative load tests against your configuration. And it's useful documentation for me to fall back on in a few months, when all of this has escaped my grasp

The defaults (as shipped in Debian Etch)

The defaults in Debian Etch are quite conservative. Consider that our test server has 12GB of RAM. The default only allocates 1MB of RAM to work memory (which is critical for sorting performance) and only 8MB of RAM to shared buffers. Following are the defaults set in /etc/postgresql/8.1/main/postgresql.conf:

# - Memory -

#shared_buffers = 1000                  # min 16 or max_connections*2, 8KB each
#temp_buffers = 1000                    # min 100, 8KB each
#max_prepared_transactions = 5          # can be 0 or more
# note: increasing max_prepared_transactions costs ~600 bytes of shared memory
# per transaction slot, plus lock space (see max_locks_per_transaction).
#work_mem = 1024                        # min 64, size in KB
#maintenance_work_mem = 16384           # min 1024, size in KB
#max_stack_depth = 2048                 # min 100, size in KB

# - Free Space Map -

#max_fsm_pages = 20000                  # min max_fsm_relations*16, 6 bytes each
#max_fsm_relations = 1000               # min 100, ~70 bytes each

Our test server settings

Our test server has 12 GB of RAM. Assuming that the PostgreSQL defaults were set for a system with 1 GB of RAM, we should be able to multiply the memory-based settings by at least a factor of 12. We're a little bit more aggressive than that in our settings. Note, however, that this is a single-server install of Evergreen, so we're also running memcached, ejabberd, Apache, and all of the Evergreen services as well as the database - oh, and a test instance of an institutional repository, among other apps - so we're not nearly as aggressive as we would be in a dedicated PostgreSQL server configuration. Please note that I'm making no claims that this is the optimal set of configuration values for PostgreSQL even on our own hardware!

# shared_buffers: much of our performance depends on sorting, so we'll set it 100X the default
# some tuning guides suggest cranking this up to as much 30% of your available RAM
shared_buffers = 100000 # 8K * 100000 = ~ 0.8 GB

# work_mem: how much RAM each concurrent process is allowed to claim before swapping to disk
# your workload will probably have a large number of concurrent processes
work_mem=524288 # 512 MB

# max_fsm_pages: increased because PostgreSQL demanded it
max_fsm_pages = 200000

After you change these settings, you will need to restart PostgreSQL to make the settings take effect.

Kernel tuning

In addition to PostgreSQL complaining about max_fsm_pages not being high enough, your operating system kernel defaults for SysV shared memory might not be high enough to support the amount of RAM PostgreSQL demands as a result of your modifications. In one of our test configurations, we had cranked up work_mem to 8GB; Debian complained about an insufficient SHMMAX setting, so we were able to adjust that by running the following command as root to set the kernel SHMMAX to 8GB (8*1024^2):

sysctl -w kernel.shmmax=8589934592

To make this setting sticky through reboots, you can simply modify /etc/sysctl.conf to include the following line:

# Set SHMMAX to 8GB for PostgreSQL
#kernel.shmmax=8589934592

Other measures

Debian Etch comes with PostgreSQL 8.1. The first version of PostgreSQL 8.1 was released in November 2005. That's a long time in computer years. Version 8.2, which was released less than a year later, "adds many functionality and performance improvements" (according to the release notes). If you're not getting the performance you expect from your hardware with Debian Etch, perhaps a backport of PostgreSQL 8.2 would help out.

Further resources

This is just a shallow dip into PostgreSQL tuning for Evergreen - hopefully enough to alert you to some of the factors you need to consider if you're putting Evergreen into a serious testing environment or production environment. Here are a few places to dig deeper into the art of PostgreSQL tuning:

• PostgreSQL manual, resource consumption section of server configuration: version 8.1 and version 8.2
• An annotated version of the 8.0 parameters with more explicit advice is available at
• Some good advice is buried about halfway down Christopher Browne's page under the heading "Tuning PostgreSQL", along with links to further resources
• The "Performance Whack-A-Mole" presentation at PowerPostgreSQL is a great tutorial for holistic system tuning

Occasionally on the #OpenILS-Evergreen IRC channel, a question comes up what kind of hardware a site should buy if they're getting serious about trying out Evergreen. I had exactly the same chat with Mike Rylander back in December, so I thought it might be useful to share the strategy we developed in case other organizations are interested in piggy-backing on our research. We came up with three different scenarios, depending on the funding available to the organization and how serious the organization is about testing, developing, and deploying Evergreen.

You can also look at the scenarios as stages, as the scenarios enable progressively more realistic testing. An organization can always start with a single server and add more servers over time; if you can swing a significant discount for buying in bulk, however, it might make sense to bite the bullet early.

Some pertinent facts about our requirements: we will eventually be loading around 5 million bibliographic records onto the system. We're an academic organization, so concurrent searching and circulation loads will be low relative to public libraries.

Scenario 1: A single bargain-basement testing server

In this scenario, the organization purchases a single server for the short term, and configures it to run the entire Evergreen + OpenSRF stack:

• database
• Web server
• Jabber messaging
• memcached
• OpenSRF applications

This server needs to have powerful CPUs, large amounts of RAM, and many fast (10K RPM or higher) hard drives in a striped RAID configuration (the latter because database performance typically gets knee-capped by disk access). A "higher education" quote online from a reputable big-name vendor for a rack-mounted 2U database server with 2x4-core CPU, 16GB RAM, 6x73GB RAID 5 drives comes in at approximately $7000.

This scenario is fine for development and testing with a limited number of users, but if you intend to do any sort of stress testing with this server or throw it open to the public, performance will likely grind to a halt. Note: This is close to the system that we're currently running at http://biblio-dev.laurentian.ca - 12 GB of RAM, 2 dual-core CPUs - with 800K bibliographic records and pretty snappy search performance. It's certainly nothing to sneeze at.

Scenario 2: one database server, one network server

In this scenario, you purchase a database server and a network server. We'll use the same specs from scenario 1 for the database server, and a CPU + RAM-oriented server for the network server (disk access isn't a factor for the network apps, so you just buy two small mirrored drives). The stock higher education quote for a rack-mounted 1U network server with 2x4-core CPU, 16GB RAM, 2x73GB RAID 1 drives is approximately $5250.

This scenario will support development and testing, as well as enable you perform relatively representative stress testing runs with a significant number of simultaneous users.

Scenario 3: two database servers, two or three network servers

In this scenario, you purchase two database servers so that you can test database replication, split database loads between search and reporting, and two or three network servers to test different distributions of the caching and network apps across the servers to determine the configuration that best meets your expected demands. The cost of the five servers adds up to less than $30,000 - less than a single traditional proprietary UNIX server - and would be less if you can negotiate a bulk discount.

The third scenario supports development and testing, and will give you practical experience with a configuration that would approximate your production deployment of servers. When you go live, you could move one of the database servers and all but one of the network servers over to the production cluster, and revert back to scenario one for your ongoing test and development environment.

The Conifer approach

We opted to go with the third scenario to build a serious test cluster for our consortium. However, the "scenarios as stages" approach ended up being our strategy as our original choice of Dell servers came with RAID controllers that do not work well under Debian. After returning the servers to Dell, we were forced to press one of our backup servers into service as a scenario-one style server while waiting for our new order from HP to arrive.

Project Conifer is the effort by McMaster University, University of Windsor, and Laurentian University to put together a consortial instance of Evergreen. A few weeks back, we agreed that May 2009 would be our go-live date. So the clock is ticking quite loudly in my ears.

Today I got an Evergreen test server up and running, loaded with the records from the consortium of Laurentian partners. I hit a few bumps on the road, but eventually successfully loaded about 800,000 bibliographic records and about 500,000 items. I also turned on the Syndetics enrichment data, so some items offer cover images, tables of contents, reviews, and author information. The response time is pretty snappy (it's running on a 4-core server with 12GB of RAM).

Things that made my task harder than it probably should have been:

• yaz-marcdump generated invalid XML when I converted our MARC records from MARC21 to MARC21XML format. Maybe this problem is fixed in later versions of yaz-marcdump (I was using the stable Debian Etch version, 2.1.56, which is crazy old), or I could have tried marc4j or MarcEdit instead to try for better results, but I didn't, and it cascaded into problems with...
• Dumping all of the holdings as part of the bibliographic records threw things off when some of the records had so many holdings attached (think a weekly periodical that a library circulates and therefore each issue has its own barcode) that they spilled over MARC's record length limit, resulting in multiple MARC records just to hold the holdings - which causes some problems for the basic import process. I eventually punted on trying to parse the MARC21XML for holdings and just dumped the data I needed directly from Unicorn in pipe-delimited format.
• Not tuning PostgreSQL before starting to load data into the database was just plain stupid. The defaults for PostgreSQL are incredibly conservative, and must be modified to handle large transactions and to perform. Here are the tweaks I made for our 12GB machine, starting with the Linux kernel memory settings:

  # -- in /etc/sysctl.conf --
  # Set SHMMAX to 8GB for PostgreSQL
  kernel.shmmax=8589934592
  

  # -- in /etc/postgresql/8.1/main/postgresql.conf --
  # Crank up shared_buffers and work_mem
  shared_buffers = 10000
  work_mem=8388608 # 8 GB, equal to our kernel.shmmax
  max_fsm_pages = 200000
  

• Evergreen depends on accurate fixed fields to determine the format of an item. Unfortunately, many of our electronic resources appear not to have been coded as such... so we have some data clean-up to do.

Ah well: as Jerry Pournelle used to say in his Chaos Manor column, "I do these things so that you don't have to." Hopefully it makes a smoother path for others to get to Evergreen.

On Wednesday, I was fortunate enough to join a distinguished panel of speakers and a crowded music hall at VALE's Next Generation Academic Library System Symposium at The College of New Jersey. I had been invited to present an update on the state of acquisitions support in Evergreen, as well as to provide a brief overview of Project Conifer (the collaboration between Laurentian University, McMaster University, and the University of Windsor to create a consortial implementation of Evergreen).

To summarize what I intended to be the main points of my presentation (which may or may not have come through in real life):

• Project Conifer is an existing effort to create a shared consortial implementation of Evergreen for academic institutions; we would be delighted to have others join forces with us
• If acquisitions isn't as far along as we would have hoped by now, it's because
  • We (the Project Conifer institutions) haven't contributed enough development resource to the effort thus far - although we are planning to correct this problem in the near term by hiring one or more developers to work on the requirements that we, as academic institutions, need for a successful Evergreen experience. If you're interested in a position as an Evergreen developer for Project Conifer, let's talk.
  • Creating an enterprise-grade acquisitions system demands much more effort and attention to detail than creating a simplistic acquisitions system that would be acceptable for a small library. If it took two years to build Evergreen's circulation, cataloging, reporting, and OPAC functionality from scratch, it's not unreasonable that it should take a year or more to build an acquisitions system to the same standards as the rest of Evergreen
• Evergreen acquisitions has made significant progress since December 2007, and at this pace we expect a complete set of basic functionality to be in place by the end of April. By "basic functionality" I mean that the manual acquisitions mode should be supported with a minimalist user interface. MARC order record batch loading, EDI send/receive support, and a more polished user interface will take some more time - probably September-ish 2008. You can see the in-development, regularly updated bare-bones interface at http://acq.open-ils.org/oils/acq/base/index.

I have to say that Equinox is making incredible progress considering that they're still doing the bulk of the work with the same amount of development resource that they had before Georgia PINES went live on Evergreen, and they started their own company, and they started bringing BC PINES on line, and they began receiving an onslaught of requests for visits and presentations and conference calls... imagine what we could do with Evergreen, together, if a few more sites or consortiums were able to devote human or financial resources to enhancing Evergreen.

Here are my slides in OpenOffice and PowerPoint format. If you're going to look at my slides, I highly recommend reading the presenter notes that I wrote; I've recently realized that presenter notes are as much for the benefit of a disconnected audience as they are useful preparation material for the presenter. In the absence of a full paper on the subject matter at hand, presenter notes should help flesh out the brevity forced by slideware.

A huge thanks to Ed Corrado, Anne Hoang, and Kurt Wagner for making the overall experience so enjoyable. I was honoured to be part of such a high-quality panel of speakers.

Oh, and as an aside - the entire symposium was videotaped, and the presentations and question and answer sessions will be made available from the VALE Web site. I will update this post when those become available. I wonder if Ed got this idea from code4lib... in any case, I certainly applaud the initiative.

Update: Umm, more polished acquisitions will likely be available in Sept. 2008, not 2007... thanks to Brad Lajeunesse for pointing out that time travel would be required to make that happen

Yesterday morning we (Bill Erickson, Sally Murphy aka "Murph", and I ran an Evergreen workshop (rough agenda, presentation, and links to associated resources from that page) for the code4lib 2008 preconference session. My personal goals were:

1. Walk people through a simple Evergreen install
2. Get a small set of bib records and holdings imported
3. Attract some more developers to the project by demonstrating how seductively simple it is to add a new service to Evergreen at the OpenSRF layer and then expose it in the catalogue or staff client
4. Show off some of the great features of Evergreen that haven't had nearly enough exposure (reports, "fresh meat" feeds, exporter interface)

Problems

Problem #1: I started organizing the pre-conference too late. To save time on the install section, I asked attendees to prepare by setting up a VMWare image or bootable Debian or Ubuntu partition and get a bunch of the prerequisite packages installed ahead of time. But by the time I sent my request out, the attendees only had a few days to prepare - and many of them probably hadn't worked with VMWare before, so they suddenly had another learning barrier to overcome. I wasn't too surprised when only about 25% of the room had been able to "do their homework".

Problem #2: I lost at least six hours of preparation time when, due to my own stupidity, I left my passport in a hotel in Atlanta and ended up having to drive across the border from Vancouver to Portland, Oregon. Six hours, man... that's almost a full day thrown away, which is critical when you've left things too late (see #1). Continuing on the negative side, all I could listen to during the drive was completely formulaic rock stations and political rhetoric worthy of 10-year-olds as I drove through Washington. If radio is a dying medium, I have a very good idea why...

Problem #3: We ran into bizarre projector problems that, for some reason, prevented us from being able to see our laptop screens at the same time as the projected screen. This laptop worked fine with the projector at the OLA Superconference just a few weeks ago, and Bill was afflicted by the same problem - so it really put a crimp in my ability to switch from the presentation to the live install image. My neck was wrecked from constantly twisting around to peer up at the screen while trying to do some minor mousing around.

Problem #4: I severely underestimated how long the install process would take when trying to support a whole group of people at once; you're guaranteed to have a question on almost every step. When we were preparing for the workshop, we had this idea that we would take a hard line and spend no more than one or two minutes on each step - which certainly would have saved a lot of time. But when you've made a connection with the audience, and people have made it through the first dozen steps, it suddenly becomes a lot, lot harder to simply abandon them with the promise that you'll help them later. So we ended up spending something like 2 hours on the install (including a break) rather than the 45 minutes we had been aiming for.

Problem #5: We were overly optimistic about how much we could get done in 2.5 hours. Even without the severe compounding of our time crunch by #4, in retrospect its clear we would still have been rushing through all of the other pieces. I think we knew that anyways, but we were just so excited about showing off Evergreen that we wanted to show off as much as possible.

It's not really all that bleak though. There were successes, too.

Successes

Success #1: We have at least one person who successfully made it through the install phase and who successfully imported the bib records and holdings, and several others who feel they are very close to finishing. I'm hoping that we can spend a few minutes over the course of the conference to help them reach that finish line.

Success #2: We have a real example of how to import holdings into Evergreen now. This is something that people have been asking for on the list, and I'm really happy to have been able to package up what Mike Rylander provided with a set of sample records and a sample "parse holdings" script that hopefully others will be able to adopt to their own needs.

Success #3: I had feedback from a number of people who, even though they weren't trying to go through the install, still felt it was worthwhile getting an explanation of all the pieces that OpenSRF and Evergreen depend on and how they fit together. I think it was clear that the complexity involved in installing Evergreen isn't so much OpenSRF or Evergreen themselves as it is a few finicky details involving networking - largely ejabberd and Net::Domain's insistence on specific and sometimes conflicting definitions of hostnames.

Success #4: Bill did get to quickly demonstrate how to add a new OpenSRF service ("reset my password and email it to me") and how to integrate that into the catalogue. It was rough and dirty code, but at approximately one page of Perl code and about 10 lines of JavaScript I think it was a convincing demonstration of how easy it is to extend Evergreen.

Success #5: We have laid the groundwork for an Evergreen workshop now, and having gone through the experience once we'll be able to refine the concept for future events. One idea that we've already kicked around is to split it into several tracks so that attendees can self-select what they're interested in and so that we can give enough time to each section. Say, two (or three) hours for an installfest; two hours for "exploring the dark corners of Evergreen"; and two hours on developing and extending Evergreen (OpenSRF, catalogue, staff client). Or we could have spent the entire pre-conference day on Evergreen.

Reflection

I think it might have been really cool if we had worked with LibraryFind and Zotero to set up an ongoing theme throughout the three pre-conference sessions. We could have collaborated on pre-requisites, so that the LibraryFind install could go on top of the same image as the Evergreen install, and then the newly installed Evergreen image could have been added as a LibraryFind source during the LibraryFind administration section. Then, during the Zotero session, Evergreen and LibraryFind could have been added as new sources for capturing citation information (by making Evergreen and LibraryFind generate COInS objects that Zotero understands or giving Zotero the ability to understand the various formats that Evergreen offers via unAPI).

Of course, it also would have required a heck of a lot of pre-conference planning. A suggestion I would make for next year's pre-conference organizers would be to communicate as much as possible ahead of time to set expectations and help your attendees determine what your agenda should be. We could have just thrown out the entire Evergreen install section, had people get comfortable with a pre-installed VMWare ahead of time, and focused most of the session on developing and exposing OpenSRF services, for example, if that's what our attendees wanted.

Well, despite getting less than four hours of broken sleep before my 9:00 am presentation, I think I successfully delivered an update on Evergreen:State of the Open ILS to approximately 45 people at the OLA Super Conference today. There were some great questions from the audience that kept me on my toes. Thank heavens David Fiander was there to provide colour commentary and solid advice. Overall, the talk seemed to be well received.

Perhaps the most pleasant surprise of the session was when I discovered that one of the libraries close to my old home town has been working for the last six months on migrating to Evergreen. Marvelous!

If you want the slides from my presentation, I've licensed them under a Creative Commons 2.5 By Attribution license. Presentations available below in two different formats:

• Evergreen:State of the Open ILS (OpenOffice Impress)
• Evergreen:State of the Open ILS (Microsoft PowerPoint)

My employer, Laurentian University, issued a press release today announcing that we have selected Evergreen as our future library system. I wrote more about this on the Evergreen blog, but what I didn't say was ... yay!

We still have a long road ahead of us, but knowing that we'll be migrating to a system that I can poke with a sharp stick and make it do my bidding goes a long way towards making me feel warm and fuzzy inside.

I predict that we'll see a few more announcements from universities and colleges in North America joining the Evergreen development effort / adoption process in 2008. Outside of Ontario, I know about the University of Utah' s interest and interest from a New Jersey consortium of academic institutions (see session h. "Open Library Systems and NJ: From Vision to Transformation")... are there other academics who have made public statements of interest in Evergreen that I'm missing out on?

Wow. Eileen R. Kontrovitz, a board member of CODI (Customers of Dynix, Inc.) wrote, as part of her summary of the recent CODI conference:

Many very nice things happened at the conference but the buzz, the thing everyone who was not there wants to hear about is the unannounced, invitation only meeting with Martin Taylor from Vista Equity Partners about open source. Mr. Taylor began the meeting in a very cordial way and with a charm about him that belied the basic message he proceeded to give. That message was, we know more than you do and if you don't like it you can go to some other "happy place". He actually said that on more than one occasion.

I guess Vista feels quite threatened by open source library systems, even though they currently account for only 1% of the US public library market today, if they're willing to have their top brass lay on the fear, uncertainty, and doubt campaign to their top customers behind closed doors. It also seems that Mr. Taylor's tactics have backfired, at least in this case; after praising the common SirsiDynix employees, Eileen goes on to say that Vista's attitude has almost ensured that her library (Ouachita Parish Public Library) will not move to Symphony. This, even though Eileen says:

...I happen to agree with his assessment of open source for a library ILS at this point in time and the many problems with the very nature of the beast without some kind of regulating body...

Perhaps this should be the subject of another blog post, but I believe there are actually a number ways Evergreen is regulated. First is that an open source project is not an "anything goes" project: the committers for the project act as a level of quality control for Evergreen. If code doesn't further the Evergreen mission by contributing towards stability, robustness, flexibility, security, or user-friendliness, then it's simply not going to go into the project proper. Second is that at least one company (Equinox) is staking its success on Evergreen, and others are starting to build up business around Evergreen. They're not going to sit back and rest easy; they know that they have to enhance Evergreen beyond its current core strengths if they want to build inroads into markets like academic libraries. Third is that Evergreen's open source license also acts as a regulator - the code that comprises Evergreen can never be pulled from the market, so the future of Evergreen is always in the hands of the community.

This is slightly in advance of our official press release, which is currently in translation, but I will be giving / have given a lightning talk at Access 2007 on this subject and have decided to make the following materials available:

• Report: Assessing Evergreen for an academic bilingual library
• Evergreen Business Readiness Rating: (OpenOffice) (Excel) - see OpenBRR for more information about the Open Business Readiness Rating templates)
• Presentation: Lightning talk: Going Ever Greener at Laurentian University

Yesterday, over on the Evergreen blog, Mike announced that I am now a full committer to the Subversion repository for Evergreen. (It was blog post #100 for Evergreen, by the way - two milestones in one!). The road to getting here was pretty standard fare for open-source projects: submit patches that do useful things (like simplify build processes or add i18n support); listen to feedback about those patches and incorporate those lessons leanred into the next patches; and repeat, as described in Evergreen's contribution process:

From time to time, and as individual community members become more familiar and skilled with the complete codebase of Evergreen, some individuals may be asked to join the core team. We see this as both an honor and a responsibility, as this group is charged with being the final quality control mechanism for the source code, as well as helping other less experienced community members come up to speed. It is not simply a way to get code into Subversion, but also about mentoring new contributors and helping to keep the overall vision of the project in focus, tempered by the history and evolution of the code and lessons learned from past successes and failures.

I'm not just tooting my own horn, here. I think it's important to emphasize that the Evergreen community is healthy, welcoming to newcomers, and growing. I am honoured to join the Evergreen team (as Mike says, "again"), this time as a committer - and I look forward to helping the Evergreen community continue to grow.

If you're interested, there are plenty of ways to help us - through your contribution of use cases, documentation, graphics and design, patches, translations, testing... Hmm. I'm repeating myself a bit here See you on the lists / IRC!

Karen G. Schneider has a great post on Enterprise Open Source on the ALA TechSource blog:

But the truly significant activity in LibraryLand technology hasn't been vendor-driven. It has been the maturation of what I call “enterprise open source”: products such as Evergreen and Koha that are robust, well-implemented library automation packages with strong development communities and equally strong funded-support models.

Hear hear! Karen examines the value of open source, and finds that it's not so much in that it's a lower-cost alternative (although that can be a persuasive argument), and not so much that you have the ability to modify the code (although that can also be a persuasive argument), but that it depends on the strength of the community to continue to exist and improve. And that makes it a very good match for libraries, because we seem to do "community" better than most other industries.

So let me take a different tack than Karen, and assume that if you've read this far that you're interested in supporting open source for your library, but maybe you don't have a programmer on staff. How can you help?

Well, there are many ways other than programming to contribute to an open source community. Evergreen, for example, just posted a call on its development mailing list for help in defining and prioritizing the requirements for its acquisitions and serials modules. If you have experience with these areas, and have blue-sky ideas for how you could build a better system, this is a great opportunity to step into the conversation. There's a bit of a parallel here to proprietary systems, although with a proprietary system it's called a "request for enhancement" and most of those tend to get filed in the distant future. With Evergreen's invitation for discussion, you know the developers are happy to listen to the ideas for making the best possible product. They don't have prior baggage holding them back, so they really can start from square one - and they have a huge incentive to do better than the existing options, because they want to convince you to pick Evergreen the next time you're thinking about your next ILS.

Or you can contribute your hard-won experience and knowledge to the documentation wiki. Evergreen and Koha both have wikis to which you can contribute. Interestingly, there is a parallel here to at least one proprietary vendor, which set up a wiki (behind a password-protected site) after many requests from their user group. It boggles my mind, but some of these same customers have also argued that they (the customers) should pool their efforts and write a new set of manuals for the product for which they are paying support and licensing fees. I'm sure the Evergreen and Koha projects would really appreciate your assistance in writing a good set of manuals, and they won't charge you for the privilege, either.

You can also participate simply by joining in the conversations on the mailing lists or chat rooms (#OpenILS-Evergreen on Freenode for Evergreen, #koha on Freenode for Koha). You'll take some time to get familiar with the products, no doubt, but once you've climbed over the brick walls (with the help of the others on the mailing list), you will have the opportunity to pay it back a dozen times over as others face the same walls that you faced. I see this same principle on our proprietary vendor's mailing lists. The customers do a far better job of supporting each other than the vendor to whom they're paying support and licensing fees.

And it feels good, working together to build something that belongs to an entire community. For the little bits that I've been able to do, I get a huge sense of satisfaction. It's a nice little addiction.