Folks, developing digital repositories is hard. \u00a0There are so many different layers of complexity built into the stack, compounded by the unique variety of end-users, or stakeholders, that we serve.<\/p>\n
Consider the breadth of this work:<\/p>\n
Starting at the bottom of the stack, you have our Preservation layer. \u00a0This is where we capture your bits, and ensure the long-term preservation of your digital assets. \u00a0But it goes well beyond just logging a single record in a database. \u00a0It involves capturing the data stream, writing that file and all associated files (metadata) to storage, replicating the data to various geographically dispersed servers, validating the ingest, logging the validation, ensuring successful recovery of replicated assets, and more.<\/p>\n
All of that comes post-ingest. \u00a0I’ll not even belabor the complexities of data modeling and ingest here, but you get the idea… it’s hairy stuff. \u00a0Receiving and massaging a highly diverse body of data into a package appropriate for homogeneous ingest is a monumental effort in normalization.<\/p>\n
Move up the stack into our Curation layer. \u00a0Currently we have a single administrative application that facilitates management and curatorial activities of our digital objects following ingest. \u00a0Roles or access controls can be managed here, in addition to various types of metadata (description about the item), etc. \u00a0There are a variety of other applications that are managed at this layer, which interact with, and store, various values that fuel display and functionality within the user interface. \u00a0This layer is quickly evolving in a way that necessitates diversification. \u00a0We have found that a single monolithic application is not a one-size-fits-all solution for our stakeholders who are in the business of data production\/curation; it is at this layer where we are getting increasingly more pressure to integrate and inter-operate with a myriad of other tools and platforms for resource\/data management. \u00a0This is tricky business as each of these tools handle data in different ways.<\/p>\n
Finally, we have the Discovery layer. \u00a0The user interface. \u00a0This is what the public sees and consumes. \u00a0It’s where access to ingested materials occurs. \u00a0It is itself an application requiring significant custom development to meet the needs of various programs and collections of materials. \u00a0It is tightly coupled with the Curation layer, and therefore highly complex and customized to meet the needs of different focal areas. \u00a0Search functionality is yet another piece of complexity that requires maintenance and customization of a central index. \u00a0Nothing is OOTB (out of the box). \u00a0Everything requires configuration and customization.<\/p>\n
And ALL of this- all of it- is inter-related. \u00a0Highly coupled and complex. \u00a0Few things reap easy wins, and often our work challenges foundational assumptions that have come well before. \u00a0It’s an exercise in balancing technical debt and moving forward without re-inventing the wheel every six months.<\/p>\n
What I have presented here is a simplistic view of our software eco-system. \u00a0It’s just a snapshot of the various puzzle pieces that\u00a0support the operation of a production repository. \u00a0In general, digital repositories are still fairly new on the scene. \u00a0No one has them figured out entirely and everyone does them a little bit differently. \u00a0There’s a strength to that which manifests in diverse platforms and a breadth of development possibilities. \u00a0There’s a weakness to it because there is no cookie-cutter approach that defines an easy path forward.<\/p>\n
So it’s an exercise in evolution. \u00a0In iteration. \u00a0In patience. \u00a0In requirements definition. \u00a0We’re not going to always get it right, and our efforts will largely take a bit of time and experimentation, but we’re constantly working to improve, to enhance, and to mature our repository platform to meet the growing and evolving needs of our University.<\/p>\n
So, here’s to many years of hard work ahead! \u00a0And many successful collaborations with our Duke community to realize our repository’s future. \u00a0We’re ready if you are!<\/p>\n","protected":false},"excerpt":{"rendered":"
Folks, developing digital repositories is hard. \u00a0There are so many different layers of complexity built into the stack, compounded by the unique variety of end-users, or stakeholders, that we serve. Consider the breadth of this work: Starting at the bottom of the stack, you have our Preservation layer. \u00a0This is where we capture your bits, … Continue reading A Tough Nut to Crack: Developing Digital Repositories<\/span>