All posts by Dave Kennedy

Open Repositories 2009

I attended Open Repositories 2009 Conference this past week.  Overall it was a very informative conference on the open source repository platforms (Fedora, dSpace, ePrints, Zentity), current projects and developments using these platforms, and future directions of repositories.  Below are some relevant notes from the conference.

Repository Workflow

There were a few presentations that discussed how institutions were managing their repositories, in particular, repositories built with Fedora.  Two of these, eSciDoc and Hydra, had some very useful nuggets.

Hydra is a grant-funded collaboration between Hull University, University of Virginia and Stanford University to build a repository management toolkit to manage their three very different workflows, and be extensible to manage heterogeneous workflows around the Fedora community.  There are a few practices or ideas that we might want to adopt from this project, as well as some possible points of convergence with Trident.

  • The idea to treat workflow processes discretely.  Hull is using BPEL (Business Process Execution Language) to define and implement the processes.  They are using Active Endpoints (was open source, may not be any longer) which provides a really nice GUI for defining and connecting workflow processes.  Not sure if this tool is worth investigating it, but I have seen it before, and have heard good things.
  • Stanford has a good design for representing the state of multiple workflows for an item.  Items have workflow datastreams, which include a number of processes, each with an indicator of state.  They then represent these workflow processes as a checklist in management interface.
  • UVA, like us, is thinking RESTfully.  RESTful approach to workflow steps allows processes to be encapsulated nicely and reused in a variety of ways.
  • Repository API – This is a possible point of eventual convergence, Hydra will be creating a RESTful API layer on top of Fedora, similar in architecture to the one that we have developed for Trident.

eSciDoc is an eResearch environment built on top of Fedora.

  • They have a well established object life cycle.  An item’s stage in it’s life cycle determines who is allowed to do what to the item.  For instance, pending (only the creator can access and modify, collaborators may be invited, item may be deleted), submitted (QC/editorial process, creator cannot modify any longer, metadata may still be enriched),…
  • They have a very tight versioning design in their Fedora repository.  They use an atomistic approach to Fedora, with items and components as separate Fedora objects.  With this approach, they can represent multiple versions of an item in their repository.  They do this by creating a copy of the item fedora object with each version, and a copy of only the changed component with each version.  Their item fedora objects contain all of the pointers to the components.  A handle gets assigned to the published version.

Cloud Storage

Sandy Payette and Michele Kimpton gave an update on the emerging DuraCloud services.  They are currently in development, and will be tested with a few beta sites before general release.  The DuraCloud services will definitely be worth Duke looking into; however, will probably need to wait for more Akubra development before these services can be properly integrated into Fedora.  For Duke’s repository, cloud storage should be evaluated for storage of preservation masters.  Also on the topic of cloud storage, David Tarrant gave an update from ePrints, as well as a reminder, “Clouds do blow away.”

Smart storage underpinning repositories

  • ePrints has exactly what is needed.  Their storage controller allows for rule based storage configuration.  This is now in their current release.
  • Fedora is still developing Akubra.  Some of the beginnings of this code are in version 3.2, but it is not implemented.  From what I gather, if we have a use case, we need to implement it ourselves.
  • dSpace will be looking at incorporating Akubra into version 2 of dSpace
  • Reagan Moore (UNC) and Bing Zhu (UCSD) gave a very detailed discussion on iRods.  iRods has a very detailed architecture for rule-based storage.  It defines many micro-services to be performed on objects.  These micro services can be chained together.  iRods has a clean rule-based configuration for defining chains of micoservices and the conditions under which these workflow chains should be executed on an object.  iRods allows for a good separation between remote storage layer and “metadata repository.”  Bing discussed how iRods is integrated with Fedora.  From what I understood, Fedora does not directly manage iRods, rather datastreams are created in Fedora as external references to iRods, and iRods must be managed separately.


djatoka continues to impress me.  It takes the math out of jpeg2000.  Ryan Chute discussed how this can be integrated into Fedora, and the service definitions involved in doing so.  He also showed some of the image viewers that have been built using djatoka.  With djatoka, the primary use of jpeg2000 is as a presentation format.  The integration with Fedora relies on a separate jpeg2000 “caching” server for serving up jpeg2000 services, which would live outside of Fedora.  In this model, it may be that Fedora never even needs to hold a jpeg2000 file.  I need a little more understanding on how the caching server gets populated, but will be investigating this in the coming months.


UPEI has packaged an integration of Drupal and Fedora.  There is a mixed bag between what Drupal content is stored in Fedora and what content gets stored in Drupal.  As new types of content are stored in Drupal, new content models need to be created in Fedora to support them.  Presenter indicated that work still needs to be done on updates on Fedora being reflected in Drupal and vice-versa.  Without more than a presentation to base my opinions on, this seems like an extensible model, but one that also requires continued hand-tuning and management.

Complex object packaging

METS and OAI-ORE, or should it be METS vs OAI-ORE.  There is a lot more discussion and work in the last year around OAI-ORE.  It is a lot more flexible packaging model for complex objects than is METS.  And it has been the medium by which SWORD and other similar models are based on.  With flexibility though comes programmatic complexity.  Our repository model is based on a METS-centric view of digital repositories.  We did generalize item structure in such a way though that we could conceivably change the underlying structure from METS to something like ORE.  More to come on this

Cool stuff

@mire showed off some authoring tools integrated into Microsoft Office as add-ins.  I’m told these won’t be released for at least six months, but showed some real possibility and value that repositories to add to authors.  The authoring tools decomposed powerpoint presentations and word documents and stored them in the repository, and then allowed for searching of the repository (from within powerpoint and word) to include slides, images, text, etc from the repository into the working document.

Peter Sefton showed off his Fascinator.  It features click to create portals that could then be customized fairly easily.  He also talked about work he is currently doing on a “desktop sucker upper” which extracts data from a laptop to store into a repository.

Programming notes

  • eSciDoc is using the same terminology as us, in terms of items and components.  This is good, although I have not heard our terminology really used in other contexts.  Also dSpace seems to be moving away from this terminology.
  • Enhanced content modeling – this development allows for more precise description of datastreams and more precise description of relationships.  This is not incorporated into Fedora proper, although it should be because it adds a lot of value to the core.
  • There are others taking a RESTful approach to repositories, at least in representing the R in CRUD
  • Others confirmed my belief that web services (RESTful ones) should be programmer friendly as well as computer friendly.  In other words, the responses should display in web pages and give a programmer at least a rudimentary but helpful view of the data


FIZ Karlsruhe has done extensive performance testing and tuning of Fedora.  They tested with data sets up to 40 million objects.  In terms of scaling, performance was not effected by size of the repository.  They were also able to increase performance by tuning the database, as well as separating the database from the repository.  They found that I/O was the limiting factor in all cases.

Fedora 3.2 highlights – beginnings of Akubra, SWORD integration, will be switching to new development environment (maven, OSGi/Spring DM)


SWORD support, Shibboleth supported out of the box, new content model in dSpace 2.0 (based on entities and relationships)

CNI Spring Task Force Meeting – April 6-7, 2009

I attended the CNI Spring Task Force Meeting in Minneapolis, April 6-7, 2009. Below are some takeaways that I found noteworthy, especially as they relate to repositories.

Keynote Address – David Rosenthal, Chief Scientist, LOCKSS, Stanford University: David challenged some of the prevailing thought on digital preservation regarding format obsolescence.  He stated that incompatibility is not inevitable, rather that “creating incompatibility = reinventing the wheel”.  He argued that format obsolescence never happens.  He backed this up with evidence from the last few decades.  The moral of the story: If we go ahead and just collect the bits, we will be fine.  A rather freeing thought, given that the perceived complexities often make digital preservation a non-starter.

JPEG2000 is a viable alternative: Ryan Chute, from Los Alamos National Library, demonstrated the Djatoka (pronounced jay-too-kay), which is an open source JPEG2000 image server, built with the Kakadu software library.  The Djatoka server now has two client implementations (IIP implementation at the Biodiversity Heritage Library, and Open Layers at UNC).  Conceivably, JPEG2000 could be used as both a presentation format and as a preservation format (lossless compression around 2:1 and visually lossless compression around 10:1 from tiffs).  Demonstration looked very sharp, will need to pay attention to how it performs in production environments.  Discussed with Ryan the plans for integration with Fedora, and there are a few implementation paths to evaluate.

Preservation services in the clouds, Duraspace: Sandy Payette and Michele Kimpton discussed the joint venture between Fedora Commons and Dspace Foundation.  Duraspace will be a service (eventually a set of services) as well as open source software.  The initial use case will allow for a preservation based service in the cloud.  They have identified a few sites that they will be piloting these services with.  By Q1 2010, they expect to have extensions available for Fedora and Dspace to plug into these cloud services.  I asked about a scenario where we might store preservation copies in the cloud and store derivatives locally, and have Fedora and Akubra broker the data to the right store; they said this is a scenario they are planning for.

Cool Book Digitization Workflow at Northwestern: I attended a presentation by Claire Stewart and Steve DiDomenico from Northwestern on their web-based book digitization workflow, codename “crabcake”.  They are digitizing books and ingesting into Fedora.  Their Fedora implementation is similar to ours with an atomistic content model and use of METS for structural metadata.  Very clean set of workflow tools.  The most impressive part of their presentation is their GUI for manipulating the METS structure for a book digital object.  This interface is built heavily with Ext JS.  Their project is grant funded, and they will be releasing as open source in the summer.  From what I can tell, installation of their tools may require some adoption of their local practices, at the very least, their interpretation of METS.  Regarding their digitization/QC process, they have a lot of throughput, they push things into Fedora with very little human intervention and fix later, in essence getting things online with very little impediment.

Trident project report: I gave an update on the Trident project.  The presentation was well attended, and the project was well received.  There was good discussion around the metadata application profile, its possible extension to different metadata schemas, and general use cases for the Editor.  There was a general validation that our project continues to head in the right direction.