Archive for the ‘Behind The Scenes’ Category

If I may make a suggestion?

Posted on April 26th, 2011 by Nick Jackson

One of the cool things that Jerome wants to do as part of its looking at the library access portal (give our current build a whirl) is provide relevant, accurate suggestions of books which you may find useful. This is a whopping huge piece of statistics which is causing me to dust off my A Level statistics, whip out some algorithms and start thinking about ways of actually doing it in a reliable way. This blog post is a summary of some of my thoughts on the issue, and I’d really appreciate it if people could weigh in with any suggestions or experience before I delve in to the depths of coding recommendations.

Here goes. First of all, we want to use as many different sources as possible from which to derive similarity suggestions. At the moment we’ve got a few suggestions:

  • People who borrowed this book also borrowed… (See how Huddersfield do it). This provides direct, human-based connections between items which are weighted by both how common the combination of books are and also how unique the combination is1.
  • Catalogued Subject Headings. Using the mad cataloguing skills of our ninja cataloguers to determine which books share a subject, as well as boosting these subject headings with data from OpenLibrary. The weight of a subject heading in the overall similarity ranking is inversely proportional to the number of times it’s used. This gives stronger recommendations to books which are within a smaller field of interest. We can also use subject headings to suggest similar journals.
  • Extracted Semantic Tags. As we pull OpenLibrary summaries for books and abstracts from our repository we’ll be slamming them through OpenCalais to extract semantic information on what the item is actually about. These then go into the item’s tags and (using a similar algorithm to subject headings) are weighed to find works about the same type of things.
  • Manual Groupings. Jerome has in-built support for ‘lists’ of items, intended to provide for things such as reading lists, collections of citations for a specific paper and so-on. We can assume that items in a list together are related, giving us a potentially huge set of manually curated similarities. To prevent too much positive reinforcement, again we’ll be weighting a link in inverse proportion to its popularity.

Read the rest of this entry »

  1. Not mutually exclusive. One his how many times book A is borrowed by people who also borrowed book B, one is how often book B is only borrowed by people who borrowed book A [↩]

Tags: , , , , , ,
Posted in Behind The Scenes, Experience, Planning | (0) Comments | Click here to add a comment »

Iteration Roundup

Posted on April 21st, 2011 by Nick Jackson

Another week, another iteration down. Here’s the summary for last week:

  • JournalTOCs Licensing: This seems to be a CC-BY licence, but we’re just double checking with Heriot-Watt about if this licences the API itself, or the data that comes from it.
  • Journal entries in search now sport availability dates in a nice human readable format (eg “From 1982 to now”, “From 1996 to 6 months ago”)
  • If you add Jerome to an iOS device home screen it now has a slick new icon.
  • Item pages for catalogue items include availability of current stock in most cases. We’re aware of some journals which are catalogued as books where this isn’t the case, and we’re working on it.
  • Fixed a bug where our import script was importing empty records for books which didn’t exist. Blamed Horizon Information Portal for returning pages of empty content rather than a HTTP 404.
  • Journal search results now point to an individual Jerome item page, rather than directly to our OpenURL resolver. OpenURL now lives in the bright orange “Online” box in the top right of an item page.

In other news, we’ve spent a fair bit of time starting to boost our ‘master plan’ mind map, and have moved a lot of the development points into the iterative model so that we’ll get round to them eventually. At some point next week I’ll be trying to get them into some kind of rough order out of the icebox so we can start to forecast iterations. Don’t forget that you can follow our current progress on our tracker if you’re really interested in the inner workings.

Tags: , , , , , ,
Posted in Behind The Scenes, iterative development, Planning | (0) Comments | Click here to add a comment »

Opening Our Data

Posted on April 18th, 2011 by Nick Jackson

One of the main things we’re trying to do with Jerome is open up all of our Library data for you to do things with. By this we mean that we want to expose as much as possible of what we know using a properly open licence such as CC0. Unfortunately, we’re hitting a few snags with this. None are insurmountable but we’re having to tread carefully, particularly with issues of licensing.

The licensing of data is an interesting one, since we run into a whole bunch of questions around who actually owns the information in our catalogue. Since it’s all factual information (and you can’t copyright a fact) then surely it’s a free for all – except that EU law introduces a curve ball in the form of database right. Broadly speaking this provides specific protection for collections of records, but not the records themselves.

Jerome at this point begins to play merry hell with licensing since (as far as catalogue data) is concerned we never touch the database, we only ask for catalogue information on a specific record and then scrape it back into our own collection. Since we’re only dealing with facts about a work (and not the work itself, which clearly is copyrighted1) then theoretically we may (or may not) be in the clear to do what we want. Personally, I’m waiting for some definitive legal advice before going any further on this one.

So, assuming for a moment that all legal wrangling over licensing has been sorted out we still hit a problem of how we subsequently expose that data in a useful way. The beautiful thing about standards is that there are so many to choose from, and we have a list of ways to expose data including MARC, XML, JSON, CSV, BibTeX, RIS, RDF, COinS and more. These are then subsequently mixed up even further using various standards for metadata, although we seem to have decided that Dublin Core is the way to go.

Fortunately since we’ve built our own data abstraction layer which (loosely) follows Dublin Core this is going to be slightly easier, but still ultimately a pain. We’re also making sure that we talk to other people engaged in open library data so that we can agree on at least a few basic things, ensuring data can be effortlessly collated from multiple sources.

We’ll have data with you as soon as possible, just let us work out the kinks of licensing and formatting first.

  1. Unless it’s public domain, in which case it isn’t [↩]

Tags: , , , ,
Posted in Behind The Scenes, MARC, Planning, Resources | (0) Comments | Click here to add a comment »

Progress, progress…

Posted on April 11th, 2011 by Nick Jackson

It’s an end-of-iteration time, and that means a round-up of what’s been going on in the Jerome world. I’m going to kick off by announcing the exciting news that we’ve standardised on Pivotal Tracker as our agile workflow manager. This means that we can all see exactly what’s planned, what needs tweaking and what’s coming next as well as getting immediate updates on the state of the current iteration. Since this is a JISC project we’ve made the whole thing public so you can see how we’re getting on.

So what’s been happening recently? Here’s a quick breakdown.

  • Catalogue import now uses the full MARC record instead of a stripped-down ‘friendly’ version, giving higher quality data and metadata.
  • Individual items in the catalogue and journals collections now have their own information pages. The same for Repository items is coming soon.
  • Item pages sport COinS metadata for ease of referencing and OpenURL lookups. Give it a whirl with any COinS-compatible browser plugin, like Zotero in Firefox.
  • Item pages all have a huge set of social media baked right in, allowing easy sharing and bookmarking.
  • We’re now aware of where an e-book is available, and highlight it accordingly in search and on the item page.
  • Cover images are now available for both books and a limited (Elsevier) set of journals.
  • Catalogue item pages have links to Google Books previews where they are available
  • We understand different media types (we’re still adding some more), which now highlight things like videos in search results. Soon they’ll be adjustable in search.
  • Tweaked default search weightings to provide slightly more accurate default results.
  • Pulling data from OpenLibrary for items with valid ISBNs provides a richer experience in the “Other Resources” section.
  • Book cover images are now coming from OpenLibrary, giving a higher quality and generally wider range.
  • Search data sets are now much cleaner in terms of character encoding and special character escaping, giving much richer international/foreign character support.

What’s coming next? We’re meeting soon to decide the key points for the next iteration but in the meantime I can reveal that we’ll be busting out our list-fu (reading lists, citation lists, pick lists and more), our first user-aware tools (custom search weighting sets and history!), journal contents, richer subject data, browsing by subject, similar books and more.

Stay tuned!

Tags: , , , , , ,
Posted in Behind The Scenes, MARC | (0) Comments | Click here to add a comment »

We Love All Our Data

Posted on February 27th, 2011 by Nick Jackson

It’s been a while since our last update, so I’m going to go over the key data sets which we’ll be using to drive the Jerome project. These are the collections of data which we lump together into the unified search index, as well as those which provide the supporting metrics to drive the intelligent generation of results.

The obvious one to include is the contents of our library catalogue, currently being scraped from our Horizon LMS using the HiP1 system. This contains information on titles available within our own collections, including both physical books and ebooks as well as ‘auxiliary’ collections such as reference and dissertations. This data is supplemented (where available) from sources such as Open Library and LibraryThing to provide as rich an experience as possible.

On top of the catalogue we’re also including the contents of our institutional repository. This is a collection of papers, datasets and other useful bits and pieces of academic importance from the depths of the University. It’s also harvestable through the initially horrific-seeming but actually delightfully sensible OAI-PMH2) standard. It’s a little slower than I expected, but it allows us to cleanly extract all the data we want to regarding titles, authors, summaries and access URIs. The OAI-PMH harvester also has the handy side effect of being compatible with archiving software that the Library is proposing acquiring, so we reduce the workload required to add other sources.

Journals are up next, and this is a tricky one since many publishers (being of the old-skool “we don’t understand this ‘Internet’ thing” and “why would you want our data?”) don’t tell us anything remotely useful about their journals or their contents. Fortunately for us, help is at hand from the people at Heriot-Watt in the form of JournalTOCs, a service which provides information on journals and what’s in them based purely on ISSNs3. Since we have a gigantic list of the ISSNs of all the journals we can access it’s a fairly simple matter to loop through them and extract all the data we can.

JournalTOCs also – as the name suggests – provides us with tables of contents for some of these journals meaning that we can even provide searching down to the individual journal articles.

So they’re the four big things we’re initially launching Jerome’s integrated search with: catalogue, repository, journals, and journal contents.

  1. Horizon Information Portal [↩]
  2. Open Archives Initiative Protocol for Metadata Harvesting – A standard way of getting information about the contents of an archive collection (although not the contents themselves [↩]
  3. International Standard Serial Number – like ISBNs, but for serial publications [↩]

Tags: , , , , ,
Posted in Behind The Scenes, Resources | (0) Comments | Click here to add a comment »

« Older Entries