Marshall Breeding published the results of his 2009 International Survey of Library Automation a few days ago. Juicy stuff, with averages, medians, and modes for the negative/positive responses on a variety of ILS and vendor-related questions, and some written comments from the respondents. One would expect the library geek blog niche to light up with reaction to the revelations contained in the data.
Except: there were 2,098 total responses to the survey, with 1,633 from the United States [1]. Responses were limited to a single response per library. The ALA estimates there are 122,356 libraries in the United States [2]. So, slightly more than 1% of the libraries in the United States are represented in this survey. Which, if it was a truly random sampling, might be enough to derive some validity from the results. But these survey responders? They're self-selected, therefore more likely to either have an axe to grind or a selection decision to defend. And the perspective of the responder on which these perceptions have been based, and in turn on which Breeding has drawn his observations, may over-represent one aspect of the system. If a circulation clerk responds, they may base their responses primarily on the speed and ease with which they can handle typical circulation tasks, while ignoring the experience of the patron using search and self-serve account interfaces; or a cataloguer may focus on the ease of copy cataloguing but gloss over the system's integration with authentication and financial systems, etc.
In my opinion, the survey's methodology makes its results more akin to gossip than science, and Breeding himself recognizes the lack of validity of his effort. After summarizing all of the findings, making some observations about open source and companies, and providing an overview of his methodology, he issues a caveat at the very end of all of this that effectively admits that the survey is worthless: "one should not read too much into the survey results."
If that's the case, then why even bother issuing this flawed, unscientific survey and publishing its results? A rigorous comparison of library systems that employs both quantitative and qualitative aspects in the assessments would be a useful contribution to the library world. This survey, however, deserves a swift trip to the round file.
References
- Perceptions 2009: An International Survey of Library Automation
- ALA Library Fact Sheet 1
![[frdm] Support SFLC](http://www.softwarefreedom.org/img/support-sflc.png "Software Freedom Law Center"){kind=small}
I do think that libraries needing to make decisions regarding a new ILS or other technology components wants information beyond what they are going to hear from the companies themselves. I hope that this survey is one of many channels of information that they might tap.
Of course I agree that libraries should do the kind of rigorous assessments of library systems that you mention as they develop their automation strategies. I spend a lot of time trying to develop that kind of data as well. Information on the perceptions that other libraries have that are already using these products can flag issues that might not otherwise be apparent.
I'm sounding a bit defensive here, but I want you to understand why I think that it is a worthwhile project.
For example: select a random set of libraries. Maybe you want to divvy it up by classic library type: academic, public, K12, special... you can base the categories on previous research, if you're smart and lazy (as a good researcher should be). Then at each sample library, you interview a set of library system users - patrons, circulation staff, technical services staff, system administrator, library administrator - to get a broad representation of views at that library about their library system. You can use a standardized set of questions, but allow follow-up questions. Transcribe the interviews and assess them qualitatively (categorize negative and positive aspects, for example).
On the quantitative side, the database world defined standardized benchmarks ages ago that enabled the comparison of databases from different vendors on a reasonably equal basis: from setup through data loading through calculating transactions / second at various loads (or data warehousing queries over various quantities of data) through cost of hardware and software. Surely we could define some sets of standard workloads for library systems: loading data, workloads such as running simultaneous searches / circulations / cataloguing / reports / notifications...
Would there be a lot of work required to follow through on something like this? Sure. But most worthwhile research requires work, and that's why I take exception to this survey being taken seriously.