Born digital archival material present unique challenges to representation, access, and discovery in the DDR. A hard drive arrives at the archives and we want to preserve and provide access to the files. In addition to the content of the files, it’s often important to preserve to some degree the organization of the material on the hard drive in nested directories.
One challenge to representing complex inter-object relationships in the repository is the repository’s relatively simple object model. A collection contains one or more items. An item contains one or more components. And a component has one or more data streams. There’s no accommodation in this model for complex groups and hierarchies of items. We tend to talk about this as a limitation, but it also makes it possible to provide search and discovery of a wide range of kinds and arrangements of materials in a single repository and forces us to make decisions about how to model collections in sustainable and consistent ways. But we still need to preserve and provide access to the original structure of the material.
One approach is to ingest the disk image or a zip archive of the directories and files and store the content as a single file in the repository. This approach is straightforward, but makes it impossible to search for individual files in the repository or to understand much about the content without first downloading and unarchiving it.
As a first pass at solving this problem of how to preserve and represent files in nested directories in the DDR we’ve taken a two-pronged approach. We will use a simple approach to modeling disk image and directory content in the repository. Every file is modeled in the repository as an item with a single component that contains the data stream of the file. This provides convenient discovery and access to each individual file from the collection in the DDR, but does not represent any folder hierarchies. The files are just a flat list of objects contained by a collection.
To preserve and store information about the structure of the files we add an XML METS structMap as metadata on the collection. In addition we store on each item a metadata field that stores the complete original file path of the file.
Below is a small sample of the kind of structural metadata that encodes the nested folder information on the collection. It encodes the structure and nesting, directory names (in the LABEL attribute), the order of files and directories, as well as the identifiers for each of the files/items in the collection.
<?xml version="1.0"?>
<mets xmlns="http://www.loc.gov/METS/" xmlns:xlink="http://www.w3.org/1999/xlink">
<metsHdr>
<agent ROLE="CREATOR">
<name>REPOSITORY DEFAULT</name>
</agent>
</metsHdr>
<structMap TYPE="default">
<div LABEL="2017-0040" ORDER="1" TYPE="Directory">
<div ORDER="1">
<mptr LOCTYPE="ARK" xlink:href="ark:/99999/fk42j6qc37"/>
</div>
<div LABEL="RL11405-LFF-0001_Programs" ORDER="2" TYPE="Directory">
<div ORDER="1">
<mptr LOCTYPE="ARK" xlink:href="ark:/99999/fk4j67r45s"/>
</div>
<div ORDER="2">
<mptr LOCTYPE="ARK" xlink:href="ark:/99999/fk4d50x529"/>
</div>
<div ORDER="3">
<mptr LOCTYPE="ARK" xlink:href="ark:/99999/fk4086jd3r"/>
</div>
</div>
<div LABEL="RL11405-LFF-0002_H1_Early-Records-of-Decentralization-Conference" ORDER="3" TYPE="Directory">
<div ORDER="1">
<mptr LOCTYPE="ARK" xlink:href="ark:/99999/fk4697f56f"/>
</div>
<div ORDER="2">
<mptr LOCTYPE="ARK" xlink:href="ark:/99999/fk45h7t22s"/>
</div>
</div>
</div>
</structMap>
</mets>
Combining the 1:1 (item:component) object model with structural metadata that preserves the original directory structure of the files on the file system enables us to display a user interface that reflects the original structure of the content even though the structure of the items in the repository is flat.
There’s more to it of course. We had to develop a new ingest process that could take as its starting point a file path and then crawl it and its subdirectories to ingest files and construct the necessary structural metadata.
On the UI end of things a nifty Javascript plugin called jsTree powers the interactive directory structure display on the collection page.
Because some of the collections are very large and loading a directory tree structure of 100,000 or more items would be very slow, we implemented a small web service in the application that loads the jsTree data only when someone clicks to open a directory in the interface.
The file paths are also keyword searchable from within the public interface. So if a file is contained in a directory named “kitchen/fruits/bananas/this-banana.txt” you would be able to find the file this-banana.txt by searching for “kitchen” or “fruit” or “banana.”
This new functionality to ingest, preserve, and represent files in nested folder structures in the Duke Digital Repository will be included in the September release of the Duke Digital Repository.