Coffee|Code : Dan Scott

Making Skype work in a Windows XP VirtualBox guest instance

dan@coffeecode.net (Dan Scott) — 2009-01-06T04:23:57Z

If you, like me, install Skype in a Windows XP VirtualBox guest instance running on an Ubuntu host on a ThinkPad T60 with an Intel 2300 dual-core 32-bit processor, it might throw Windows exceptions and generate error reports as reported in VirtualBox ticket #1710.

If you then go into your ThinkPad BIOS and enable the virtualization extensions (under the Config->CPU menu) and reboot your Ubuntu host, you might get the same problem again. I'll tell you right now that the problem is that you need to power off your system after changing the BIOS option, rather than deciding to just Save and Exit with an immediate reboot.

You might then try loading the KVM module with the Intel extensions to test whether your system actually does have the Intel virtualization extensions enabled - and you might get a FATAL: Error inserting kvm_intel (/lib/modules/2.6.27-9-generic/kernel/drivers/kvm/kvm-intel.ko): Operation not supported error message. Perhaps you'll end up consulting a page like the Ubuntu KVM wiki entry and think "Man, I'm doing everything right... what's going on?"; you then re-read the VirtualBox ticket #1710 and decide to try updating the BIOS on your laptop just in case it has some problem enabling the virtualization extensions.

Huzzah! After updating the BIOS, you can then load the kvm and kvm-intel modules. (Aside: this has nothing to do with the updated BIOS, and everything to do with the enforced power-off cycle that comes with the update process.) You triumphantly fire up your Windows XP VirtualBox guest instance, ready to get Skype going, and... the image won't even boot, dying with the horrible error: VirtualBox can't operate in VMX root mode. VBox status code: -4011 (VERR_VMX_IN_VMX_ROOT_MODE). What the...? You do a bit more searching of the forums, and all of the solutions (like Ubuntu bug 292588) tell you to disable KVM. By this time, you're thoroughly confused and have convinced yourself that KVM is necessary for the virtualization extension that will satisfy Skype.

Eventually, you rmmod kvm-intel and try VirtualBox again - and much to your surprise, the image loads and Skype is stable. Huzzah! (Aside: this is because VirtualBox uses its own kernel module, vboxdrv.ko, for virtualization support, and the kvm / kvm-intel kernel modules won't co-exist with that... so your little test to see if the virtualization extensions were working ended up preventing your virtualization application from working.)

I'm sure you're wondering why I was going to all of this effort to get Skype running on a virtual machine in the first place, when there's a native package for Ubuntu available that works. The punch line to all of this is that I convinced my mom to start using Skype as a backup for when her phone line goes dead (which it does, all too often), but for some really weird reason I haven't been able to find her as a contact in the Linux Skype client, even though I know her exact user ID. On a hunch, I wanted to try searching for her in a Windows Skype client. And you know what?

I was able to get exactly one hit, the right result, under the Windows Skype client. That is both baffling and frustrating. Skype is both a proprietary application and service and there's nothing I can do to fix the search problem. It seems that Skype's APIs are messed up enough that they can't even support a simple search reliably on a cross-platform basis. Hopefully that doesn't carry over to, say, being unable to maintain an accurate count of your credit balance.

Ah well. I don't think that the open source alternatives like Ekiga or QuteCom are solutions yet for my Mom's needs; we need the service to match the application. Maybe in a parallel universe Skype's services would support any SIP client, and their contact search API would be open. But in this universe, I have both learned how to properly enable virtualization extension support in VirtualBox on Ubuntu and I have finally added my Mom as a contact in a proprietary application / service. That's good enough for today.

Update a few minutes later: Hmm, looks like Tpad might be close to the service that I'm looking for... Perhaps I'll follow up later with a report.

Just in time delivery - Arik Nathan

dan@coffeecode.net (Dan Scott) — 2008-12-29T18:54:27Z

49 minutes before Christmas, Arik Nathan Kabaroff-Scott was born - much to the delight of his mother. Well, and father too, but some of us are pickier about avoiding holidays as birth dates than others... Arik weighed in at 8 pounds 11 ounces, and was 23 inches long, although his bulk belied the long slender toes and fingers that he came equipped with. Perhaps we have a future bass player in our midst?

There was much relief on the Kabaroff side of the family as it was revealed that Arik did not inherit his father's webbed toes. He definitely has his mother's ears, at least as far as attached vs. detached goes.

So far, Arik has been exceedingly kind to his parents, sleeping for two and three hours at a time, eating well, and generally being everything we could hope for. He's healthy and seems happy.

Big sister Amber has adjusted quite well to her new brother. On her first visit to see Arik in the hospital, she commented on his tiny toes and ears, and sang him a lullaby that she had been practicing for months in advance. Although her Mommy, Daddy, and new brother spent Christmas Day in the hospital, Santa conveniently waited until a few days later to arrive at home. And luckily Grandma was on hand to help Amber cope at home for a few days.

Daddy and Arik are getting along rather well, although Daddy has already been taught the hard way one of the major differences between changing a little girl's diaper and a little boy's diaper. Although Daddy's first instinct was to duck and cover, apparently the opposite order of operations is advised...

Mommy is doing quite well recovering at home and bonding with Arik. It's certainly a big adjustment from working on the thesis!

P.S. Daddy apologizes for the delayed announcement and pictures and thanks everyone for their kind wishes. It's been a little bit busy around here...

In which my words also appear elsewhere

dan@coffeecode.net (Dan Scott) — 2008-12-02T17:50:09Z

I'm excited to announce the availability of my first post as an invited contributor to the More than Bookends blog over at the revamped Academic Matters web site. My fellow contributors are Anne Fullerton and Amy Greenberg, and I'm delighted to be included with them in our appointed task of writing about the position of the librarian in the academy. My focus at Academic Matters is going to be less "hardcore geek" and more approachable to a general audience of "understanding systems librarians in the university: what we do for you". You can always get your hardcore geek fix here at Coffee|Code. Here's a snippet from my first post to give you a feel for what I'm talking about:

... Stefano Mazzochi has some interesting ideas about the impact of interruption on creativity with which I concur. I'm in demand for services both great and small, and it often feels like I'm making no significant progress in any area. This leads directly to tenure deficit disorder, a new condition of anxiety about one's potential for future employment within one's institution...

If you want to read more, come on over to the More than Bookends blog and check out what Anne has to say about open access and what Amy has to say about the everything-old-is-new-again theme of connecting with library users.

Aside: Apologies for the lack of RSS feeds for the site; the blog software that we are using still appears to be under construction. With any luck that will be rectified soon!

Presentation: LibX and Zotero

dan@coffeecode.net (Dan Scott) — 2008-11-25T22:51:59Z

Direct link to the instructional presentation on LibX and Zotero at Laurentian University (ODT) (PDF)

I had the pleasure of giving an instructional session to a class of graduate students on Monday, November 24th. The topic I had been asked to present was an extended version of the Artificially Enhanced Research session that I gave at Software Freedom Day earlier in the year. Apparently one faculty member was so impressed by seeing LibX and Zotero in action that he pushed to make this a mandatory session for all of the graduate students in his department. Cool! And a bit scary, too; it's one thing to present to people who are interested in what you have to say, but another thing when your audience is captive and have other demands on their time.

So of course it all started as poorly as possible - while plugging the VGA cable into the projector, my laptop suddenly died. Kaput. No power, and pressing the power button resulted in no response at all. Normally I wouldn't sweat too much, because I usually send the presentation in advance via email and copy it to a USB key. But this was, of course, the one time that I didn't take defensive measures. On top of that, I intended to run the presentation with lots of live demos and knew that my laptop was happily configured. Finally, I had come down with a killer head cold the day before, and the only reason I even showed up at work was to give this presentation - so my mind was already fuzzy (that's what I attribute my lack of file copying precautions to).

So, I went ahead with the session doing a live demo of everything on a machine that I had never used before with an operating system (Windows XP) that I rarely use and a keyboard layout (fr-CA) that I also rarely use... in the throes of a virus. With my wife (she's a grad student in the program) in attendance. Pressure much?

As it turned out, things went quite well. LibX and Zotero installed happily on the semi-locked down workstation and the only piece I couldn't demonstrate was Zotero's integration with Word - but I was able to vouch that it did work as one would expect and hope. In fact, doing everything from download and install to actual use live without a net was probably the best instructional session that the students could get - it made all of the steps nice and concrete, and reassured them that this wasn't something that only propeller-heads would be able to use.

Most pertinent question that I wasn't able to answer at the time: Does Zotero offer duplicate detection and elimination? The right answer is "it's been on the roadmap for a while, but there are no guarantees (and I can't sign up to add that feature, sorry)".

Oh, and the whole laptop dying thing? Apparently your laptop battery can act as a sort of circuit breaker - so if the laptop gets zapped by a jolt of static electricity, the battery can disable power to the system and protect it. After making the walk of shame to the local computer store for support, it turned out that I simply had to remove and reinsert the battery to enable the laptop to start drawing power again. A good lesson to learn, I suppose, and hopefully one that I'll remember the next time I find myself without any backups on hand in front of an expectant class

Evergreen 1.4.0.0 RC2 and OpenSRF 1.0.1 are out

dan@coffeecode.net (Dan Scott) — 2008-11-21T03:41:10Z

As I announced on the Evergreen mailing lists last night:

One month after the first release candidate of Evergreen 1.4.0.0, the Evergreen development team is pleased to announce the availability of Evergreen 1.4.0.0, release candidate 2, from http://open-ils.org/downloads.php

A partial overview of the changes since 1.4.0.0 RC1:

MARC importer / exporter enhancements
Improved support for marking long overdue items
Z39.50 client enhancements
An interface for switching locales in the staff client
Localization in every interface - although we have undoubtedly missed a few strings
Bundled Armenian and French (Canadian) translations
Performance improvements for new and changed item feeds
Various staff client, build, and source tree fixes

The complete change log between 1.4.0.0 RC1 and 1.4.0.0 RC2 can be found here: http://open-ils.org/downloads/ChangeLog-1.4.0.0rc1-1.4.0.0rc2

Please help us reach a solid 1.4.0.0 final release by testing out 1.4.0.0 RC2 with the freshly released OpenSRF 1.0.1 and reporting problems, sending patches for improvements or fixes, or sending new or updated translations to the Evergreen Development mailing list.

Coming soon for the 1.4.0.0 RC2 release:

Windows staff client
Updated install instructions at http://open-ils.org/dokuwiki/doku.php?id=server:1.4.0.0:ubuntu804:install
VMWare image

This release has been a long time in the making, and we'd love to have your help in testing it and flushing out bugs. Also, if you would like to contribute a translation, this is your chance to step up! We already have Brazilian Portugese (pt_BR), Georgian (ka), and Canadian English (en_CA) translations in the works, along with a commitment to update the Canadian French (fr_CA) translation. As this is the first real round of translations for Evergreen, I fully expect that there will be some work ahead of us to smooth out the translation process - but we have to take the plunge some time. Many thanks to Tigran Zargaryan and Natural Resources Canada for their respective contributions of the Armenian (hy_AM) and Canadian French (fr_CA) translations this summer; their willingness to be early guinea pigs for the translation process helped immensely.

Update: I noticed that the speedy Warren Layton beat me to the punch on the blog announcement of the releases. Warren's been very helpful with testing and suggestions for improvements to the documentation, so I don't mind being scooped at all

Archive of OCLC WorldCat Policy as posted 2008-11-02

dan@coffeecode.net (Dan Scott) — 2008-11-03T14:47:48Z

I noticed last night (Sunday, November 2nd, 2008) that the new and much-anticipated / feared OCLC WorldCat Policy had been posted. As far as the clarified terms went, I was willing to give them the benefit of the doubt until they were actually posted. I was first alerted to the freshly released policy by Terry Reese's initial analysis and was fascinated by the discussion in the comments.

Then I noticed this morning on the AUTOCAT mailing list ("OCLC New Policy down after one day?") that the new policy had been pulled and replaced by

We are reconsidering some aspects of the policy. More information will be available in the near future.

Very interesting. That's not even enough time for most interested parties to read the policy and provide feedback.

Luckily, I made use of Zotero last night to capture a snapshot of the freshly released policy. So here, for your pleasure, is the archived version (retrieved Sun 02 Nov 2008 10:24:15 PM EST according to Zotero - and how sad a statement is that about my life that I'm cruising the library 'nets on Sunday evenings?) of the OCLC policy that had been posted and then was immediately yanked. I'm hoping that this helps informed discussion lead towards a reconsidered policy that reflects direct input from OCLC contributors.

Full disclosure: my library is not an OCLC Member library, not from any ideological stance but because I haven't had time to contribute records yet. Kind of busy with the Evergreen thing, you know? But we pay for WorldCat and some other OCLC services and intend to become members. On a personal note, I think we would best be served by a plurality of services developed using the metadata to which all of the OCLC Member institutions contribute.

Dear Dan: why is using Flash for navigation a bad idea?

dan@coffeecode.net (Dan Scott) — 2008-11-03T02:03:02Z

I received the following email late last week, and took the time to reply to it tonight. I had originally been asked by a friend to help diagnose why his organization's site navigation wasn't working in some of his browsers. I noticed that the navigation bar was implemented in Flash, and suggested that the Flash be replaced with plain old HTML, CSS, and perhaps some JavaScript as Flash for navigation is generally a bad idea. My friend's response last week was:

Dan, I would be interested to know why the flash is a bad idea. ...Our [website] designer seems to think otherwise.

Uh-oh: getting into it with the designer via proxy can't be good. But here's what I wrote in response - feel free to set me straight if I've gone off-track:

Jakob Neilsen said it best back in 2000 - and all of his concerns still hold up: http://www.useit.com/alertbox/20001029.html

To which I would add several more reasons:

Flash is not a native part of the Web, it's a third party browser plug-in; HTML + CSS and arguably JavaScript are the core parts of the Web. Anything as basic as navigation should not have to rely on a third-party plug-in to function.
How do you expect a person using a screen reader (due to visual impairment) to navigate through the site? With HTML/CSS/JavaScript, they can tab through the links on the page; with Flash, they hit something that forces them to use the mouse.
Until very recently, Google could not navigate through Flash to crawl pages - and even now it's capabilities are subject to some limitations (http://googlewebmastercentral.blogspot.com/2008/06/improved-flash-indexing.html).
On breaking the Web experience - if you try to right-click on the Flash navigation, you don't get the option to open the link in a new tab or window, or add a bookmark to the page that link represents. And I bet the links would work for you everywhere
For the paranoid, Flash is a proprietary format and therefore requires you to trust that the content that is being served up isn't doing something nefarious while appearing innocuous. A small percentage of people use plug-ins like NoScript to block Flash content and only enable Flash for specific pages when necessary. Having to enable it for navigation is annoying.

If your designer really, really wants to keep the Flash experience, the least they could do would be to put a plain HTML navigation menu inside the <noscript>...</noscript> section rather than the "Get Flash!" thing that currently appears.

For the paranoid: deleting Flash local storage objects

dan@coffeecode.net (Dan Scott) — 2008-10-30T11:24:41Z

I'm reasonably careful about the cookies I accept from Web sites - I don't want companies to be able to track every site I visit, for example, so that they can build a nice little profile about me. It's for the protection of the companies more than anything else: someone there might die of extreme boredom following the trail of "Evergreen", "Linux Weekly News", "Python docs"...

However, I recently learned about Flash "local storage objects" (LSO), which are similar to browser cookies but capable of storing much richer information and also completely inscrutable in terms of the effectiveness of Adobe's security model. Is Flash really capable of preventing a Flash application running on microsoft.com from accessing an LSO from mail.google.com? I certainly don't know, and as Flash is a closed-source application it's hard for anyone except for the developers at Adobe to know--but I bet there are people extremely motivated to find out. (Insert obligatory "See? Closed source sucks!" comment here.)

So, in my crude attempt to prevent too much garbage accumulating due to the occasional YouTube video or NBC Saturday Night Live skit that I might watch, I've added the following rules to my cron entries to delete my entire set of LSOs every four hours:

5 ∗/4 ∗ ∗ ∗ rm -fr /home/dan/.macromedia/Flash_Player/#SharedObjects 
5 ∗/4 ∗ ∗ ∗ rm -fr /home/dan/.macromedia/Flash_Player/macromedia.com/support/flashplayer/sys/

You Windows users can probably do something similar, but I haven't bothered to track that down yet. Sorry.

An Evergreen track at the OLA SuperConference 2009?

dan@coffeecode.net (Dan Scott) — 2008-10-28T20:10:18Z

Just poked at the OLA SuperConference 2009 schedule (January 28 - 31, 2009) and found four sessions listed that are all about Evergreen. Wow! Check this out:


Date	Time	Title	Description (may be abridged)	Presenters
Thursday, January 29	9:05 am	It.s Just a Little Bit of Programming Isn.t It?	Follow the progress of the Library @ Mohawk.s development of the open source ILS Evergreen. Hear the trials and tribulations and learn from the mistakes and successes that have occurred along the way . we are truly a learning organization on this project. We went live in summer 2008 . come and hear about where we.ve been, where we are and where we hope to be soon.	Robert Soulliere, Systems Librarian; Cynthia Williamson, Collection & Access Librarian, Mohawk College of Applied Arts and Technology
Thursday, January 29	3:45 pm	Project Conifer: Evergreen library system for Ontario Universities	Find out how the Evergreen open source library system, originally developed for a public library consortium, is being adapted for academic libraries by three Ontario universities. Discussion will focus on the challenges, successes and mistakes (err, .learning opportunities.) of the project.	John Fink, Digital Technologies Development Librarian, McMaster University; Dan Scott, Systems Librarian, Laurentian University
Friday, January 30	9:05 am	Evergreen exposed: hacking the open source library system	Join an Evergreen developer on a tour of the architecture and source code of the Evergreen library system [...] Get ready to get your hands dirty with Evergreen . this will be a session filled with code!	William Erickson, Vice President, Software Development & Integration, Equinox Software Inc; Dan Scott, Systems Librarian, Laurentian University
Saturday, January 31	10:40 am	Multilingual Language Issues of Open Source ILS	Discover the Chinese version of Evergreen along with various multilingual issues related MARC standards, encoding, indexing, searching, and sorting especially associated with Chinese language.	Jason Zou, Systems Librarian, Lakehead University; Guoying (Grace) Liu, Systems Librarian, Leddy Library, University of Windsor

I was responsible for the sole Evergreen presentation at OLA SuperConference 2008 - it's awesome to see a lot more people jumping in this year! I'm keenly anticipating this conference - we'll have to set up at least one Evergreen "Birds of a Feather" session.

Evergreen: deOSSification of library software

dan@coffeecode.net (Dan Scott) — 2008-10-23T17:45:16Z

In a few minutes I'll be giving a talk with John Fink at the Free Software Open Source Symposium at Seneca College on Evergreen: an enterprise-strength OSS solution for library ossification. I'm jazzed!

Here are the slides: (ODP format) (PDF format).

Boss me around, s'il vous plait

dan@coffeecode.net (Dan Scott) — 2008-10-09T16:10:08Z

My place of work, Laurentian University, is looking for a new Director of the J.N. Desmarais Library. The call for applications closes October 30th. I think our library has done some impressive work (participating in the food security project for the Democratic Republic of Congo, building the Mining Environment Database), are doing some pretty interesting things (Project Conifer), and have the potential to do a lot more. Our outgoing director has laid the groundwork for his successor to take Laurentian University's libraries to even more exciting destinations.

On the chance that you haven't heard about this opening, and would be interested - or perhaps you know someone who might be interested and would be a good candidate - I'm going to attach the full job posting below. One heads-up: Laurentian is a bilingual (French / English) institution, and candidates would be expected to be bilingual.

Laurentian University – Director of the Library and Archives – effective July 1, 2009

Laurentian University is committed to serving the needs of Northern Ontario and its English-language, French-language and First Nations communities, and to extending its student recruitment into other areas of Canada. Laurentian’s recent major accomplishments include the establishment, with Lakehead University, of the first new medical school in more than 30 years in Canada, the introduction of 6 new PhD programs, the construction of new residences and a building custom-designed to house the B.Ed. program, establishment of a new Faculty of Management, and increasing success attracting research funding, all within a vibrant bilingual milieu that appreciates the past and is building toward the future. Please consult www.laurentian.ca for more information.

The Director of the Library and Archives will be an innovative leader who will build on the momentum of the Libraries’ information technology services, operational systems and capabilities, and provide strategic direction in accordance with the Strategic Plan approved by the Laurentian University Board of Governors.

The ideal candidate will be a respected academic who will bring dynamic vision and academic administrative experience, and who will be able to function in both English and French. The candidate will have a Masters’ of Library Science from an ALA accredited library school or equivalent institution plus at least five years experience in library administration, including planning, personnel, budget, policy development and services. The candidate will have the ability to view issues from an organization-wide perspective and to contribute effectively to collaborative projects within and outside the institution. The candidate will have knowledge of trends in higher education and information technology that affect acquisition, management, storage and dissemination of print, multi-media and electronic resources. Experience in writing grant proposals and effectively administering such grants would be a strong asset.

Laurentian University is committed to employment equity, welcomes diversity in the workplace and encourages applications from all qualified individuals including women, members of visible minorities, aboriginal persons and persons with disabilities. All qualified candidates are encouraged to apply; however, Canadians and permanent residents will be given priority.

Should you want to learn more about this unique leadership opportunity, e-mail Susan Silverton, Vice-President, Academic (Anglophone Affairs) at ssilverton@laurentian.ca or forward your CV, a letter of introduction and the names of three referees in confidence, to Linda Mainville, Assistant to the Vice-Presidents, Academic, Laurentian University, Ramsey Lake Road, Sudbury, Ontario P3E 2C6

Access 2008 hackfest report: Zotero vs Evergreen

dan@coffeecode.net (Dan Scott) — 2008-10-07T04:36:00Z

Update: 2008-10-07 As of changeset 10774, the detailed record view in Evergreen's dynamic catalog is now recognized by Zotero.

I really like Zotero. And it works really well with Evergreen's current "basic search" because it embeds unAPI links that enable Zotero to consume MODS representations of the underlying bibliographic records and generate a complete citation based on that.

However, Zotero doesn't work with Evergreen's current "dynamic search" interface - which is a problem, because it is the default search interface. Evergreen embeds a link to the unAPI server, and fills in the unAPI link via an AJAX call after the underlying XHTML has been loaded - but it seems that Zotero doesn't recognize that the DOM has been changed by the AJAX event and never discovers the unAPI link. So... I had submitted a challenge to Hackfest to fix this, because I really want to be able to use Zotero with Evergreen when Project Conifer launches.

And, as with every other Hackfest I have attended, I end up working on my own challenge. In discussing the problem with William from canadiana.org and Walter Lewis from Knowledge Ontario, I described how the dynamic interface doesn't use any templating (apart from entity substitution for localization support), that there wasn't really any way to inject content server side into the underlying XHTML, and that I really didn't want to have to dig into the guts of Zotero to enable it to parse the DOM after events had completed. William asked "so you can't even do a server side include?", which ended up breaking the problem wide open - because yes, we already use server side includes to identify which DTD to load for localization purposes.

Step 1 was to modify the detailed record display to put the unAPI link template in place, and to modify the Apache configuration to pass in hardcoded values for each of the SSI variables. A quick test and - it didn't work. Uh oh.

That led to much scratching of the head. Was Zotero getting tripped up by the masses of XHTML elements in the dynamic template that are simply hidden? Did it give up after trying to parse 100K or so of content? Were there differences in the content types being served up by Apache? The next step was to compare the content of the "basic search" output against the "dynamic search" output - and that led to one seemingly innocent difference.

The unAPI server link in the "basic search" output included an absolute link to the server, while the corresponding link in the "dynamic search" output used a relative link to point to the root of the server. I didn't think that would be a problem, but eliminating variables is always good - and when I tested with a hardcoded server link, the Zotero hint icon lit up and the mystery was solved. Between enabling the record unAPI link to appear in the static XHTML via SSI and changing the unAPI server link to use an absolute value, Zotero and Evergreen could work together in harmony.

I haven't committed the fix for this yet to the repository, as I haven't finalized the exact SSI incantations that will be needed to embed the record ID in the unAPI link. But now you know the solution, and could tackle the problem yourself if you get tired of waiting for me and feel inspired. And once the problem is fixed, I'll update the post to let you know what version of Evergreen carries the fix.

Oh, and my hackfest report slides are attached, in case anyone cares.

Access 2008 presentation: Project Conifer report

dan@coffeecode.net (Dan Scott) — 2008-10-04T23:13:09Z

On Friday, October 3rd, I had the honour of presenting the progress of Project Conifer with my colleague John Fink to my peers at Access 2008. Project Conifer is the effort to bring the Evergreen open source library system to a consortium of academic libraries in Ontario (Algoma, Laurentian, McMaster, Northern Ontario School of Medicine, and Windsor).

I'm just going to link quickly to the slides for now, as I'm a little bit brain-dead after the conference. John led off the talk with an overview of what Conifer is all about and why we were motivated to tackle such a large project - he has posted his slides via the SlideShare thingy. Editorial comment: I really enjoy John's presentation style and content. He's a hard act to follow!

And then I rambled on with an overview of the ups and downs of the project so far, the resources we have invested in the project, our progress towards our target go-live date (May 2009), and some sneak previews of the goodies that are included in the any-day-now-if-I-would-just-stop-going-to-conferences-and-apply-myself-for-a-few-days-dangit Evergreen 1.4 release. Well - they're not really sneak previews, because of course you could check the code out of the repository and built it yourself - but it's so much easier when somebody else already has it running, right?

Anyway, my slides are available in both OpenOffice.org Impress format and PDF.

Software Freedom Day 2008 - Sudbury

dan@coffeecode.net (Dan Scott) — 2008-09-21T02:24:30Z

I opted to do something out of the unusual (for me) this year when I learned about Software Freedom Day; I signed up to organize an event in Sudbury. Given everything that was already on my plate, it was pure foolishness to do so - but it was also important to me to try and pull together people in Sudbury with an interest in open source and free software. I'm hoping that this is just the first of many such events.

Did I mention that trying to organize an event in an academic environment during August and early September is madness? I was incredibly fortunate to land three excellent guest speakers on very short notice.

Keynote: Open Source and Open Learning

Dr. Rachel Ellaway, Assistant Dean of Education Informatics at the Northern Ontario School of Medicine, kicked off the event and instantly repaid all of the effort of putting together the whole event. In just one hour, she:

eloquently introduced the four freedoms of free software,
mapped the continuum of free software from public domain to closed source,
situated the academy and publicly funded institutions in the same philosophical category as open source,
posited that in education the process of learning was more important than the product,
introduced MIT's OpenCourseWare effort to make most of their course materials (syllabus, readings, lecture notes, assignments, and projects) freely available online,
discussed the problems of learning objects from the 80's and 90's that are now locked away in binary files that simply won't run on today's computers - and an effort named BoneYard that she is involved in to convince the rights holders to donate the source code (be it COBOL or BASIC or anything) behind these learning objects to a repository so that others can reuse the underlying algorithms and breathe new life into the objects,
questioned the continuing faith in "contact hours" as the gold standard for academic service when one hour spent developing a scenario for a virtual learning environment could engage students in multiple hours of active learning
wondered whether our institutions could move towards open administration and governance models

... all in an entertaining and engaging style. Sudbury and the Northern Ontario School of Medicine are lucky to have Dr. Ellaway in our midst.

Moodle course management software

We were also fortunate to have Dr. Rick Danielson, Full Professor in the School of Human Kinetics, on hand to discuss the history of course management software at Laurentian University and his own experiences with Ilias and Moodle. While on sabbatical in 1998, Dr. Danielson piloted the use of WebCT at Laurentian for course management and lead the creation of a purchasing consortium to license WebCT for more than 40 institutions across Ontario. A few years later, Dr. Danielson switched his personal WebCT server to the open source alternative Ilias coure management system; and, shortly after that, to Moodle. At the time, Ilias did not offer a module that supported online quizzes, and Dr. Danielson was also won over by Moodle's more refined interface. For Dr. Danielson, the choice between using the university's WebCT instance and Moodle running on his own server is all about control, saving money, and being lazy:

He craves control over his teaching environment and independence from central IT, and running his own Moodle server gives him that.
He watched the license fees for WebCT rise as it was acquired first by ULT and then by Blackboard, and doesn't mind that he has to spend some money on his own server because Moodle itself is freely redistributable
He was happy to work hard to learn how to install and administer Red Hat Linux, Apache, MySQL, and PHP in order to run Moodle - now he can be lazy because it simply works.

Dr. Danielson's talk was full of tangents and insider stories about the history of the university and Northern Ontario - quite entertaining and quite irreverent. He did a good job conveying the spirit of the early adopter of new technology to the audience, and in answer to the question of whether Laurentian's next course management system should be WebCT, he said Tuum est ("It's in your hands!").

Given that UQAM successfully automated the conversion of over 90% of their WebCT courses to Moodle (laziness is a virtue, remember), and given that we are facing a migration from WebCT to something else (the product name does not even appear on Blackboard's Web site), I hope that Laurentian will seriously consider Moodle. I have asked to be part of the evaluation committee for the next course management system here, as I want to ensure that we can provide much better integration between our library systems and our learning systems than we have today. I know that Evergreen will give us the flexibility we require on the library side; we need to ensure that we have a course management product with the flexibility that we need on the other side - and finally, we need the people willing to make the integration between the systems work and to demonstrate to faculty how they can easily take advantage of that integration.

ParaViewGeo - open source visualization from MIRARCO

The third guest speaker was Robert Maynard, lead developer for ParaViewGeo at E2VO, a subsidiary of MIRARCO. ParaViewGeo is a customized version of ParaView by KitWare. The BSD license used by ParaView gives E2VO the right to modify the software and redistribute binary versions of it without redistributing the source code; however, E2VO does plan to make most of their additions and customizations available, and has been pushing patches back to the upstream product.

Robert showed a number of screenshots that demonstrated how ParaViewGeo could transform basic two-dimensional diagrams into much richer three-dimensional visualizations. For example, ParaViewGeo can read AutoCAD files and do a far better job of displaying three-dimensional figures. Also, a CD of data produced for the Discover Abitibi initiative included a proprietary application with a license that timed out after one year, but ParaViewGeo was able to read and display the data in more detail than the original application was capable of.

Perhaps as interesting as the product was the insight that Robert provided on their software development process. Robert described how, due to a lack of students with software development skills in the Sudbury region, he opted to lead a three-week software development boot camp for their student developers starting from extremely basic principles ('let's open a file and write to it'), motivating them to achieve via competitive gaming techniques and tangible rewards ('whichever team successfully completes this problem first gets free lunch today'), group learning techniques (all seven team members worked in one large room; when a programmer hit a road block, Robert would demonstrate the solution on a projector so that all the team members could learn at once), and leading by example (Robert was always the first person in the room and the last person to leave).

Project Conifer: Open Source Academic Library System

I gave a presentation on Conifer, of course. Project Conifer is the effort to bring Evergreen to the Ontario academic libraries for Algoma University, Laurentian University and partners, McMaster University, Northern Ontario Health Integration Network, and the University of Windsor. I'm the project manager for Conifer and one of the core developers for Evergreen, so I can't possibly not talk about it on Software Freedom Day. As regular readers of this blog already know plenty about Conifer, I won't recap much here. The audience was quite impressed with the upgraded user experience that Evergreen offers out of the box over our current system, and sensed the potential for a system like this to offer integration with course management systems, integration with our inter-library loan system, and even the simple pleasures of direct physical borrowing between participating institutions.

I was also given the perfect excuse to purchase an iTouch with library or Conifer funds. One of the audience members asked if the dynamic interface supported the iPhone or iTouch, as they would like the ability to search for books or journals while in the stacks. I had to answer that I did not know, not being in possession of such a device, but was able to assert that Opera Mobile on my Windows Mobile cellphone worked extremely well. The audience member then tried out their own iTouch and determined that the interface was at least able to support search - but clearly I need to do a deeper investigation

Artificially enhanced research : free software and fantastic research

This was my mostly-demo session, which probably would have been better delivered earlier in the day while both audience members and myself were fresher, but somebody had to have the last slot in the session. I presented a few free software tools that I thought would be of interest to researchers:

LibX browser toolbar for Laurentian University: I showed that while the LibX toolbar is handy as a quick way of searching our catalogue, it's much more about embedding quick access to our library resources within your browser. I quickly got off on a tangent, however, as I was showing the Google Scholar "magic button" and had to explain what Google Scholar was, how it differed from regular Google, and answer the faculty member's question about whether he should send students to our subject databases page or to Google Scholar (my answer: "Well, both!").
Zotero research plug-in for Firefox: while LibX got their attention, Zotero got the audience buzzing. Just a quick demonstration of capturing a set of citations from a Wikipedia page, adding a note, taking a snapshot of the Wikipedia page, organizing research into folders, adding tags, and searching in full-text through your captured materials (optionally filtering by tag) was enough to get most of the people in the room interested. Zotero makes research fun - students will want to do research just so they can play with Zotero!
LU|ZONE|UL institutional repository : by this point, I was running out of time, so I tried to quickly link back to Dr. Ellaway's discussion of open access and funding agencies' new requirements to deposit articles and datasets in institutional repositories. However, after pointing out that Laurentian has actually had an institutional repository in production for over a year, one faculty member asked why we should care about depositing articles if we have access to the article through our electronic journal subscriptions? I'm not as good at answering this question as I probably should be; my answer was that it's a means of making that content available to researchers at those institutions that don't have a subscription to that journal; and that it's a means for our institution to have direct access to our own researchers' output without necessarily having to continue to pay annual licensing fees to access that ouput. I also pointed out some of the opportunities that alternate publishing formats like institutional repositories make possible; Guy Gaudreau's book Les hauts et les bas de la vie des mineurs de Kirkland Lake would not have been published with the striking colour charts he was able to include in the fully-digital version he contributed to LU|ZONE|UL - nor would he have had the ability to publish each chapter as it was finished and use the feedback to improve the rest of the book or to reissue corrections to that chapter.
Finally, I mentioned our library's quick lookup laptops which now run on Linux LiveCDs for stability, robustness, environmental friendliness, and budget purposes. They're an open source experience that our library members use on a daily basis without any apparent problems; they "just work".

So... to sum up, I believe the event was well worth the effort. We had approximately twenty participants with a mix of undergraduate and graduate students, faculty, librarians, and members of the public (including one fellow who drove all the way from North Bay to attend!). Now that I know what I'm doing, and now that we have a better idea of the interested parties on campus, I believe that we will be able to build a bigger event next year. I don't think I would hold it on a Saturday again, even if that's when Software Freedom Day is officially scheduled, because students and faculty really value their weekends. And I hope that we have more events - perhaps smaller, perhaps less formal, and perhaps more frequent - throughout the year.

Finally, I want to thank the team of people at the J.N. Desmarais Library that helped me to make this event possible, in no particular order: Leïla Saadaouai, Dorothy Robb, Christine Guerra, Noella Cliche, Ashley Thomson, and Joscelyne Meilleur. Your efforts were certainly appreciated by our participants today.

Heating up Evergreen search

dan@coffeecode.net (Dan Scott) — 2008-08-25T16:23:54Z

So, after loading 3.7 million records into the Project Conifer test server, we have found that search can be slow. Not really a big surprise, because I've spent very little time tuning the database beyond running a VACUUM FULL and tweaking just a few parameters. But one of the extremely useful hints that Mike Rylander gave me about PostgreSQL a long time back is that it relies primarily on file system caching to cache access to data, from the reasonable perspective that your file system already knows which files are being accessed most often. PostgreSQL's data is stored in files that map back to individual tables and indexes; unlike some other database systems that I've worked with, you don't dedicate system memory specifically to caching those database files (hello, DB2 buffers!); instead, you just trust the file system to know what's best.

That caching approach works great on a system that's in production and getting a steady stream of queries reflecting what users actually search for on a day to day basis. However, if you've just loaded a test system, then it doesn't have much opportunity to cache and the first dozen (or hundreds, or thousands!) of queries will be slow as your database goes out and loads up files from disk. Even worse, if you have a system like ours where backups have temporarily been set up as "tar czf /backups/backup.tar.gz /", then on a nightly basis your file system cache is going to be filled with all kinds of irrelevant data.

So what are we to do? Well, actually, another extremely useful hint that Mike Rylander gave me was to just run the pertinent data files through /dev/null to load up the file system cache. On the surface, it seems like a dirty hack, but it's a smart one, and we can even make it elegant. Let's walk through the process:

You need to know where your data files are. You (or your system installer) will have created a PostgreSQL cluster. In my case (on Debian Etch), I can find it at /var/lib/postgresql/main/base. Then, by running "du -hs /var/lib/postgresql/main/base" I can see that one of our databases (represented by a directory name that's just an integer - "16385") weighs in at 60GB. That's our 3.7 million record baby. If you run an "ls" command on that directory, you'll see that it's filled with hundreds of files of differing sizes, most of them with just plain integers for their names. This is where the data is stored.

You need to know the base filenames that you want to use to warm up the file system cache. For my first stab at this, I decided to warm up the cache with the full-text search indexes, as I know those are frequently used by Evergreen's search. To figure out the base filenames for these indexes, we can query PostgreSQL's catalog of its own objects:

evergreen=# SELECT relfilenode, relname, relpages
evergreen-# FROM pg_class WHERE relname LIKE '%vector%';
 relfilenode |                   relname                    | relpages 
-------------+----------------------------------------------+----------
      648864 | authority_full_rec_index_vector_idx          |    59282
      649137 | metabib_title_field_entry_index_vector_idx   |    29766
      649149 | metabib_author_field_entry_index_vector_idx  |    20125
      649161 | metabib_subject_field_entry_index_vector_idx |    23481
      649173 | metabib_keyword_field_entry_index_vector_idx |    90709
      649185 | metabib_series_field_entry_index_vector_idx  |     8682
      649210 | metabib_full_rec_index_vector_idx            |   452980
(7 rows)

relfilenode is the basename of the files that we want to load into the file system cache.

The maximum size of your file system cache cannot be more than the physical RAM installed on your system, so you'll want to tally up the size of the index data files to ensure that their total is less than the total amount of your physical RAM. Note that in the example from our system, below, I'm using "*" because database objects with lots of data will be split between multiple files with extensions like ".1" and ".2" in sequential order:
```
# cd /var/lib/postgresql/main/base/16385
# du -hs 649185* 649210* 1065608*
68M	649185
1.1G	649210
1.1G	649210.1
1.1G	649210.2
1.1G	1065608
1.1G	1065608.1
1.1G	1065608.2
842M	1065608.3
# du -hs 649207*
1.1G	649207
1.1G	649207.1
1.1G	649207.2
467M	649207.3
```
Adding all of this up, we're getting close to the 16GB of RAM installed on our database server. If we add any more data, we will want to add more RAM to the system.

Now we warm up the cache by outputting the contents of each file into /dev/null.

# cd /var/lib/postgresql/main/base/16385
# cat 648864* > /dev/null
# cat 649137* > /dev/null
# cat 649149* > /dev/null
# cat 649161* > /dev/null
# cat 649173* > /dev/null
# cat 649185* > /dev/null
# cat 649210* > /dev/null

After running through this relatively simple exercise, searches were definitely much snappier on our test system. I plan to automate the process so it runs after every one of those cache-killing backups. If there is interest, I could package it into a simple Perl script that other sites could use to assist with their testing - or to help warm up the file system cache after a large data load, for example.