Now, encouraged by positive JISC feedback, both institutions—Cambridge and Lincoln jointly—have applied for follow-up project funding under the project title CLOCK. If our bid is successful, the new project will run between December 2011–July 2012, employing a web developer based at the University of Lincoln, and distilling the work of both institutions into the development of new innovative library metadata discovery services for the scholarly community.

You can read the project proposal for CLOCK at http://lncn.eu/ijt4 – the introductory section is below.

The University of Lincoln and Cambridge University Library both delivered successful projects (Jerome and COMET) for the JISC Infrastructure for Resource Discovery Programme in 2011. This is a proposal for the continuation of and elaboration upon the work of both projects, via a programme of development work shared between the two institutions.

Throughout both projects (COMET-Jerome), parallel approaches in technology and data structure were noted and commented upon. A ‘mash day’ workshop event held in Cambridge in August aimed to explore these differences as well as areas of potential synergy. Here project members identified several points of interest to take forward.

Both projects produced outputs of interest to researchers, students, librarians, developers, and designers of bibliographic discovery environments. The CLOCK project will harness the success of these two complementary initiatives and investigate new approaches to data creation and discovery in the library domain. In particular, it will investigate, propose, and develop new, web-based bibliographic tools/APIs which will make it easier for developers, academic libraries and library end-users (esp. researchers) to find Open Bibliographic Data and incorporate that data into systems and workflows.

This project is an opportunity to [1] exploit through real-world applications the significant amount of data released openly by Cambridge University Library; [2] apply the Jerome database architecture, iterative development methodology, and API framework to a bibliographic dataset an order of magnitude greater than the University of Lincoln’s; and [3] to build and enable a new set of tools and demonstrator services which will enable the future development of public Open Bib Data web applications of practical utility to libraries and end-users.

The project will be supported by library consultant Owen Stephens, who will help to put the work into a national context, relating CLOCK to the wider movement toward Open Bib Data and the work of the JISC Discovery initiative. It will take place in an environment (Lincoln/Cambridge) where a culture of developer inquiry and experimentation is encouraged and nurtured. It is also endorsed by senior library management at both universities.

Both universities are involved in complementary development work which will  both inform and be informed by CLOCK: at Cambridge, Ed Chamberlain is guiding the development of the JISC Open Bibliography 2 project; in Lincoln, Paul Stainthorp is lead researcher on the #jiscmrd Orbital project, which is investigating the management of research data, with some areas of overlap.

CLOCK will operate as part of the wider JISC Digital Infrastructure: Information and library infrastructure: Resource discovery, and support the recent concerted effort to move toward openly licensed library discovery in UK Higher Education and beyond.

In attendance (for the record):

• Alex Bilbie (University of Lincoln)
• Ed Chamberlain (University of Cambridge)
• Nick Jackson (University of Lincoln)
• Chris Keene (University of Sussex)
• Phillip Heels (University of Lincoln)
• Tony Hirst (The Open University)
• Huw Jones (University of Cambridge)
• Chris Leach (University of Lincoln)
• Dan Sheppard (University of Cambridge)
• Paul Stainthorp (University of Lincoln)
• Owen Stephens (Owen Stephens Consulting)
• Laura Waldoch (University of Cambridge)
• Lihua Zhu (University of Cambridge)

Train mayhem aside (in the end the Lincoln contingent didn’t arrive until nearly midday), it was a really useful day and well worth doing. Particular thanks to Ed Chamberlain and his colleagues for hosting the event and for arranging the food and refreshments. Thanks also to everyone who travelled from afar for no other reason than they love a good mashup.

Typically, the ever-prolific Tony Hirst has already managed to write up not one, but two blog posts about ideas that came out of the day:

• Getting Library Catalogue Searches Out There…
• Open Data Processes: the Open Metadata Laundry (N.B. this one relates specifically to Jerome – in particular, our notion of ‘scrubbing’ dodgy MARC records by taking only the identifiers plus the bare citation-only fields, and using that minimal set to grab additional free and Open data from the web, automatically creating new full versions of records that are inherently Open. ‘Metadata laundry’, me like.)

Here are three more ideas/conversations we had in Cambridge that I thought were going somewhere interesting. Yeah, we might get around to actually doing these, sometime…

1. Using COMET data to enhance Jerome

Similar to the ‘metadata laundry’, above, and to the way Jerome already uses data from the Open Library, JournalTOCs, LibraryThing, etc., to enhance its book records with additional metadata. Jerome constructs a URL in the form http://data.lib.cam.ac.uk/isbn/_______, with the ISBN from the Jerome record dropped in at the end. COMET responds with a link to an open record in RDF and/or JSON, which Jerome gladly sucks in, adding any additional fields to its original source record. Enrichment ensues.

2. Using Jerome search to ‘skin’ COMET

I called this one ”Jerome Scholar” …we make use of the search aspects of Jerome (in particular, the speed of Sphinx, the ‘mixing desk‘ idea, the neat record presentation, to provide a really smooth way of interacting with the much more well-structured (hence “Scholar”) data that resides in COMET.

3. Using the differences between the two datasets to tell us something interesting

I have a notion that there’s something inherently useful about being able to compare two versions of a record for the ‘same’ object. If we could use Jerome+COMET to generate a web application/data feed – one that other discovery services could themselves consume, we’d have ways of ‘sparking off’ whole new avenues of discovery: from misspelled names, variant titles, different subject terms assigned by different cataloguing practices, etc. Like xISBN, but for non-standardised data(?). All right, that’s the fuzziest of the three ideas. And as the eminiently sensible Owen Stephens kept asking me, “…what’s the use case?”.

And then we went to the pub.

I’ve put ‘final’ in inverted commas in the title of this blog post (which should be sung—of course—to the tune of this song) – because while the JISC-funded Jerome project has indeed come to an end, Jerome itself is going nowhere. We’ll continue to tweak and develop it as an “un-project” (from whence it came), and—we sincerely hope—Jerome will lead in time, in whole or in part, to a real, live university library service of awesome.

Before we get started, though, thanks are due to the whole of the Jerome project team: Chris Leach, Dave Raines, Tim Simmonds, Elif Varol, Joss Winn, developers Nick Jackson and Alex Bilbie times a million, and also to people outside the University of Lincoln who have offered support and advice, including Ed Chamberlain, Owen Stephens, and our JISC programme manager, Andy McGregor.

Ssssooooo…

Just what exactly have we produced?

1. A public-facing search portal service available at: http://jerome.library.lincoln.ac.uk/
   • Featuring search, browse, and bibliographic record views.
   • Search is provided by Sphinx.
   • A ‘mixing desk‘ allows user control over advanced search parameters.
   • Each record is augmented by data from OpenLibrary (licenced under CC0) to help boost the depth and accuracy of our own catalogue. Where possible, OpenLibrary also provides our book cover images.
   • Bibliographic work pages sport COinS metadata and links to previews from Google Books.
   • Item data is harvested from the Library Management System.
   • Social tools allow sharing of works on Facebook, Twitter, etc.
2. Openly licensed bibliographic data, available at http://data.lincoln.ac.uk/documentation.html#bib, and including:
   • 170,000 library catalogue records, released under a CC0 licence
   • 3,100 repository records (CC0)
   • 92,000 e-journal records (CC0)
   • XXX,XXX (we still don’t know yet! It hasn’t finished counting…) journal tables of contents derived from JournalTOCs, available under CC-BY
   • See our licensing page for more information.
3. Attractive, documented, supported APIs for all data, with a timeline of data refresh cycles. The APIs will provide data in the following formats:
   1. RDF/XML
   2. JSON
   3. RIS
   4. The potential for MARC
4. Source code for Jerome will be made Open and publicly available (after a shakedown) on GitHub.
5. While the user interface, technical infrastructure, analytics and machine learning/personalisation aspects of Jerome have been discussed fairly heavily on the project blog, you’ll have to wait a little while for formal case studies.
6. Contributions to community events. We presented/discussed Jerome at:
   • JISC Infrastructure for Resource Discovery start-up meeting, Birmingham, March
   • M25 Consortium CPD event on mashups and open data, London, April
   • JISC/RLUK ‘Opening Data – Opening Doors’ event, Manchester, April (report)
   • CILIP UC&R Yorkshire & Humberside training event, Huddersfield, May
   • JISC/RLUK Discovery launch event, London, May (report)
   • Post-project Jerome/COMET open-data mashup day, Cambridge, August!
   • We are also submitting a magazine article to SCONUL Focus about Jerome in the context of next-generation library resource discovery (“OPAC2.0″) services, to be published later in 2011.

What ought to be done next?

1. There’s a lot more interesting work to be done around the use of activity/recommendation data and Jerome. We’re using the historical library loan data both to provide user recommendations (“People who borrowed X…“), and to inform the search and ranking algorithms of Jerome itself. However, there are lots of other measures of implicit and explicit activity (e.g. use of the social sharing tools) that could be used to provide even more accurate recommendations.
2. Jerome has concentrated on data at the bibliographic/work level. But there is potentially even more value to be had out of aggregating and querying library item data (i.e. information about a library’s physical and electronic holdings of individual copies of a bibliographic work) – e.g. using geo-lookup services to highlight the nearest available copies of a work. This is perhaps the next great untapped sphere for the open data/Discovery movement.
3. Demonstrate use of the APIs to do cool stuff! Mashing up library data with other sets of institutional data (user profiles, mapping, calendaring data) to provide a really useful ‘portal’ experience for users. Also: tapping into Jerome for reporting/administrative purposes; for example identifying and sanitising bad data!

Has Jerome’s data actually been used?

Probably not yet. We were delighted to be able to offer something up (in the form of an early, bare-bones Jerome bibliographic API) to the #discodev Developer Competition, where we still hope to see it used. Also, we are holding a post-project hack day (on 8 August 2011) with the COMET project in Cambridge to share data, code, and best practices around handling Open Data. We certainly intend to make use of the APIs internally to enhance the University of Lincoln’s own library services. If you’re interested in making use of the Jerome open data, please email me or leave a comment here.

What skills did we need?

At the University of Lincoln we have been experimenting with a new (for us) way of managing development projects: the Agile method, using shared tools (Pivotal Tracker, GitHub) to allow a distributed team of developers and interested parties to work together. On a practical level, we’ve had to come to terms with matching a schemaless database architecture with traditional formats for describing resources… Nick and Alex have learned more about library standards and cataloguing practice (*cough*MARC*cough*) than they may have wished! . There are also now plans to extend MongoDB training to more staff within the ICT Services department.

What did we learn along the way?

Three things to take away from Jerome:

1. MARC is evil. But still, perhaps, a necessary evil. Until there’s a critical mass of libraries and library applications using newer, more sane languages to describe their collections, developers will just have to bite down hard and learn to parse MARC records. Librarians, in turn, need to accept the limitations of MARC and actively engage in developing the alternative</lecture over>.
2. Don’t battle: use technology to find a way around licensing issues. Rather than spending time negotiating with third parties to release their data openly, Jerome took a different approach, which was to release openly those (sometimes minimal) bits of data which we know are free from third-party interest, then to use existing open data sources to enhance and extend those records.
3. Don’t waste time trying to handle every nuance of a record. Whilst it’s important from a catalogue standpoint, people really don’t care if it’s a main title, subtitle, spine title or any other form of title when they’re searching. Perfection is a goal, but not a restriction. Releasing 40% of data and working on the other 60% later is better than aiming for 100% and never releasing anything.

Thanks! It’s been fun…

Paul Stainthorp
July, 2011

As a result Jerome uses its own internal format for storing data which is built around what Jerome needs to know to provide the necessary search and discovery services. Since we’re storing objects in MongoDB I’ve opted for a nice clean JSON-based object model with very clear field names and structure. It throws away all kinds of ‘useful’ data from a detailed metadata standpoint purely because it’s irrelevant to discovery. The vast majority of fields we store are completely optional (since not all resources have everything we care to know about). In a nutshell, we are capturing and storing the following data:

• A resource’s unique Jerome ID.
• Information on the collection the resource belongs to, including its unique ID within that collection.
• Title, secondary title and edition.
• An array of author names and secondary author names.
• A date of publishing split into year, month and day.
• An abstract or synopsis.
• Keywords and subject categories.
• A periodical name, volume number and issue number.
• Availability dates, including start and stop of availability.
• Publisher name and location.
• ISBN or ISSN.
• A URL to access the resource, and optionally a label for that URL.
• A URL for direct access to the full text of the resource.
• Name and URL for the licence information of the resource metadata.

As part of the magic of this approach we completely disassociate the individual type of resource from its representation within Jerome. We can represent pretty much any collection of resources within the same framework with no having to muck about writing custom code to turn one format into another for representation.

As a result of this approach we can use Jerome as a true multi-channel resource discovery tool. All that’s required for anybody to add resources to Jerome and immediately make them searchable and recommendable is for a ‘collection’ to be created and for them to write a bit of code which can make the following API calls as necessary:

• Create a new resource as part of the collection, telling us as much about it as they can.
• Update an existing resource when it changes.
• Delete a resource which is no longer available.
• Optionally record a use of a resource against a user’s account to help build our recommendations dataset.

That’s it. Got a collection of awesome lecture slides you want to feed into Jerome and instantly make known as a resource? You can do that.

We’ll have your API documentation up soon.

The costs of getting Jerome to this stage are relatively easy to work out. Under the Infrastructure for Resource Discovery programme, JISC awarded us the sum of £36,585 which (institutional overheads aside) we used to pay for the following:

• Developer staff time: 825 hours over six months.
• Library and project staff time: 250 hours over six months.
• The cost of travel to a number of programme events and relevant conferences at which we presented Jerome, including this one, this one, this one, this one and this one.

As all the other aspects of Jerome—hardware, software etc.—either already existed or were free to use, that figure represents the total cost of getting Jerome to its current state.

The benefits (see also section 2.4 of the original bid) of Jerome are less easily quantified financially, but we ought to consider these operational benefits:

1. The potential for using Jerome as a ‘production’ resource discovery system by the University of Lincoln. As such it could replace our current OPAC web catalogue as the Library’s primary public tool of discovery. The Library ought also to consider Jerome as a viable alternative to the purchase of a commercial, hosted next-generation resource discovery service (which it is currently reviewing), with the potential for replacing the investment it would make in such a system with investment in developer time to maintain and extend Jerome. In addition, the Common Web Design (on which the Jerome search portal is based) is inherently mobile-friendly.

2. Related: even if the Jerome search portal is not adopted in toto, there’s real potential for using Jerome’s APIs and code (open sourced) to enhance our existing user interfaces (catalogues, student portals, etc.) by ‘hacking in’ additional useful data and services via Jerome (similar to the Talis Juice service). This could lead to cost savings: a modern OPAC would not have to be developed in isolation or tools bought in. And these enhancements are as available to other institutions and libraries as much as to Lincoln.

3. The use of Jerome as an operational tool for checking and sanitising bibliographic data. Jerome can already be used to generate lists of ‘bad’ data (e.g. invalid ISBNs in MARC records); this intelligence could be fed back into the Library to make the work of cataloguers, e-resources admin staff, etc., easier and faster (efficiency savings) and again to improve the user experience.

4. Benefits of Open Data: in releasing our bibliographic collections openly Jerome is adding to the UK’s academic resource discovery ‘ecosystem‘, with benefits to scholarly activity both in Lincoln and elsewhere. We are already working with the COMET team at Cambridge University Library on a cross-Fens spin-off miniproject(!) to share data, code, and best practices around handling Open Data. Related to this are the ‘fuzzier’ benefits of associating the University of Lincoln’s name with innovation in technology for education (which is a stated aim in the University’s draft institutional strategy).

5. Finally, there is the potential for the university to use Jerome as a platform for future development: Jerome already sits in a ‘suite’ of interconnecting innovative institutional web services (excuse the unintentional alliteration!) which include the Common Web Design presentation framework, Total ReCal space/time data, lncn.eu URL shortener and link proxy, a university-wide open data platform, and the Nucleus data storage layer. Just as each of these (notionally separate) services has facilitated the development of all the others, so it’s likely that Jerome will itself act as a catalyst for further innovation.

From the start, Jerome took an approach that can be summed up as:

• Use the tools and abilities that you have to hand. Go for low-hanging fruit and demonstrate value early on.
• Follow the path of least resistance. If it works, it’s fine. Route around problems rather than fighting against them.
• Consider the benefits of a ‘minimally invasive’ approach to library systems. Use the web to passively gather copies of your data without having to make changes to your exisiting root systems (such as LMSes).

What tools would people need to take this approach?

1. Technical ability. Jerome would have got nowhere without the coding abilities of University of Lincoln developers Nick Jackson and Alex Bilbie, and the approach they take to development (see below). It’s fair to say that their involvement has been something of a culture-shift for the Library: they have brought a fresh approach to dealing with our systems and data.

2. An agile, rapid, iterative project methodology and a suite of (often free; always web-based) software tools to support that way of working. This probably can’t be overstated: a ‘traditional’ project management methodology just wouldn’t have worked for Jerome.

3. An understanding of where data resides in your current systems. We’ve had to become uncomfortably au fait with the structure of SirsiDynix‘s internal data tables, MARCXML, OAI-PMH, RIS and all sorts of other unpleasantness. Also an awareness of the rich ecosystem of third-party library/bibliographic/other useful data that exists on the web.

4. Related: a willingness to try different approaches to getting hold of this data (the absolutely-anything-goes, mashup-heavy approach): APIs: great. Screen-scraping: yeah, if we must. SQL querying, .csv dumps, proprietary admin interfaces: all fine. Don’t be precious about finding a way in. By far the most important thing is to provide the open data service in the first place. Things can always be tidied up, rationalised, at a later date.

5. A box to run things on. This doesn’t have to be a large institutional server: we’ve successfully run Jerome on a Mac Mini.

6. Finally, the use of blogs—such as this one—and social media to engage a (self-selecting, admittedly) community of potential users and fellow-travellers.

Priorities:

• Use tools such as Pivotal Tracker and mind maps to capture requirements and turn them into a plan for development.
• Meet regularly to review the plan and push the ideas on to the next stage.
• Decide what’s ‘out of scope’ early on so you can concentrate on maximum value.
• Realise that there’s value in releasing even part of your data—for example, a list of ISSNs—which you can exploit immediately without having to worry about issues (e.g. third-party copyright in records) that might affect your complete dataset.
• Blog about what you’re doing. Little and often is the way to go.

What to avoid:

While it’s important to be aware of the bigger picture, don’t get too distracted by the way things are being done elsewhere.

A lot of the open data movement seems to be closely tied up with providing access to large quantities of Linked Data (the dreaded RDF triple store!), and initially I was worried that because we were not taking that approach we were somehow out of step. (In fact, I think that our not concentrating on Linked Data has allowed Jerome to explore the open data landscape in a different and valuable way, i.e. the provision of open bibliographic data services via API – see Paul Walk’s slides on providing data vs. providing a data service. I know which side my bread’s buttered: Open Data ≠ merely open; Open Data = open and usable.)

Similarly, while a lot of the discussion around third-party intellectual property rights in data has been phrased in terms of negotiating with those third parties to release the data openly (or taking a risk-assessment approach to releasing it anyway!), Jerome took a different approach, which has been first to release openly those (sometimes minimal) bits of data which we know are free from third-party interest, then to use existing open data sources to enhance the minimal set: what we end up with are data that may differ from the original bibliographic record, but which are inherently open. It’s not a better approach, just a different one.

Who you will need to have ‘on side’:

• Your systems librarian, or at least someone who has a copy of the manual to your library catalogue!
• A cataloguer, to excuse explain the intricacies of MARC.
• A university librarian / head of service who is willing to take risks and sanction the release of open data.
• A small-but-thriving community of developers, mashup-enthusiasts and shambrarians (see above), including people on Twitter who can act as a sounding board for ideas.
• Ideally, your project team should include as many people as possible who do not have a vested interest in the way libraries have historically got things done. Jerome has been valuable in arising from an approach that’s different from the library norm.

To start with, we’re merging a wide variety of individual tables¹ – one for each type of resource offered – into a single table which handles multiple resource types. We’ve opted to use all the fields in the RIS format as our ‘basic information’ fields, although obviously each individual resource type can extend this with their own data if necessary. This has a few benefits; first of all we can interface with our data easier than before without needing to write type-specific code which translates things back to our standardised search set. As a byproduct of this we can optimise our search algorithms even further, making it far more accurate and following generally accepted algorithms for this sort of thing. Of course, you’ll still be able to fine-tune how we search in the Mixing Deck.

To make this even easier to interface with from an admin side, we’ll be strapping some APIs (hooray!) on to this which support the addition, modification and removal of resources programmatically. What this means is that potentially anybody who has a resource collection they want to expose through Jerome can do, they just need to make sure their collection is registered to prevent people flooding it with nonsense that isn’t ‘approved’ as a resource. Things like the DIVERSE research project can now not only pull Jerome resource data into their interface, but also push into our discovery tool and harness Jerome’s recommendation tools. Which brings me neatly on to the next point.

Recommendation is something we want to get absolutely right in Jerome. The amount of information out there is simply staggering. Jerome already handles nearly 300,000 individual items and we want to expand that to way more by using data from more sources such as journal table of contents. Finding what you’re actually after in this can be like the proverbial needle in a haystack, and straight search can only find so much. To explore a subject further we need some form of recommendation and ‘similar item engine. What we’re using is an approach with a variety of angles.

At a basic level Jerome runs term extraction on any available textual content to gather a set of terms which describe the content, very similar to what you’ll know as tags. These are generated automatically from titles, synopses, abstracts and any available full text. We can then use the intersection of terms across multiple works to find and rank similar items based on how many of these terms are shared. This gives us a very simple “items like this” set of results for any item, with the advantage that it’ll work across all our collections. In other words, we can find useful journal articles based on a book, or suggest a paper in the repository which is on a similar subject to an article you’re looking for.

We then also have a second layer very similar to Amazon’s “people who bought this also bought…”, where we look over the history of users who used a specific resource to find common resources. These are then added to the mix and the rankings are tweaked accordingly, providing a human twist to the similar items by suppressing results which initially seem similar but which in actuality don’t have much in common at a content level, and pushing results which are related but which don’t have enough terms extracted for Jerome to infer this (for example books which only have a title and for which we can’t get a summary) up to where a user will find them easier.

Third of all in recommendation there’s the “people on your course also used” element, which is an attempt to make a third pass at fine-tuning the recommendation using data we have available on which course you’re studying or which department you’re in. This is very similar to the “used this also used” recommendation, but operating at a higher level. We analyse the borrowing patterns of an entire department or course to extract both titles and semantic terms which prove popular, and then boost these titles and terms in any recommendation results set. By only using this as a ‘booster’ in most cases it prevents recommendation sets from being populated with every book ever borrowed whilst at the same time providing a more relevant response.

So, that’s how we recommend items. APIs for this will abound, allowing external resource providers to register ‘uses’ of a resource with us for purposes of recommendation. We’re not done yet though, recommendation has another use!

As we have historical usage data for both individuals and courses, we can throw this into the mix for searching by using semantic terms to actively move results up or down (but never remove them) based on the tags which both the current user and similar users have actually found useful in the past. This means that (as an example) a computing student searching for the author name “J Bloggs” would have “Software Design by Joe Bloggs” boosted above “18th Century Needlework by Jessie Bloggs”, despite there being nothing else in the search term to make this distinction. As a final bit of epic coolness, Jerome will sport a “Recommended for You” section where we use all the recommendation systems at our disposal to find items which other similar users have found useful, as well as which share themes with items borrowed by the individual user.

1. Strictly speaking Mongo calls them Collections, but I’ll stick with tables for clarity

UK Discovery (http://discovery.ac.uk/) and the Developer Community Supporting Innovation (DevCSI) project based at UKOLN are running a global Developer Competition throughout July 2011 to build open source software applications / tools, using at least one of our 10 open data sources collected from libraries, museums and archives.

…and one of the 10 open data sources is the Jerome API we announced last week!

Enter simply by blogging about your application and emailing the blog post URI to joy.palmer@manchester.ac.uk by the deadline of 2359 (your local time) on Monday 1 August 2011.

Full details of the competition, the data sets and how to enter are at http://discovery.ac.uk/developers/competition/

Follow #discodev on Twitter to see what people are up to.

http://data.online.lincoln.ac.uk/documentation.html#bib

This is only the initial, bare-bones JSON-only service. A complete (and fully-documented) API will be released in stages over the next month, providing data in a range of output formats. We’re keeping all API and open institutional data documentation in the one place, on our open data site.