Category Archives: Behind the Scenes

Congratulations and farewell to Mike Adamo

This week, Digitization Specialist Mike Adamo will move on from Duke Libraries after 14 years to assume a new position as Digital Imaging Coordinator at the Libraries of Virginia Tech University. Mike has contributed so much to our Digital Collections program during his tenure, providing years of uncompromising still imaging services, stewardship in times of change for the Digital Production Center, as well as leadership of and then years of service on our Digital Collections Implementation Team. He has also been the lead digitization specialist on some of our most well known digital collections like the Hugh Mangum photographs, James Karales photographs and William Gedney collection.

In addition, Mike has been a principal figure on our Multispectral Imaging Team and has been invaluable to our development of this service for the library. He established the setup and led all MSI imaging sessions; collaborated cross-departmentally with other members on the MSI Team to vet requests and develop workflows; and worked with vendors and other MSI practitioners to develop best practices, documentation, and a preservation plan and service model for MSI services at Duke Libraries. He’s also provided maintenance for our MSI equipment, researching options for additional equipment as our program grew.

Side by side comparison of a papyri item under natural light and the same item after multispectral imaging and processing.

We are grateful to Mike for his years of dedication to the job at to the field of cultural heritage digitization as well as for the instrumental role he’s played in developing MSI Services at DUL. We offer a huge thank you to Mike for his work and wish him well in his future position!

Post contributed by Giao Luong Baker and Erin Hammeke

Features and Gaps and Bees, Oh My!

Since my last post about our integrated library system (ILS), there’s been a few changes. First, my team is now the Library Systems and Integration Support Department. We’ve also added three business analysts to our team and we have a developer coming on board this summer. We continue to work on FOLIO as a replacement for our current ILS. So what work are we doing on FOLIO?

FOLIO is a community-sourced product. There are currently more than 30 institutions, over a dozen developer organizations, and vendors such as EBSCO and IndexData involved. The members of the community come together in Special Interest Groups (SIGs). The SIGs discuss what functionality and data is needed, write the user stories, and develop workflows so the library staff will be able to do their tasks. There are ten main SIGs, an Implementation Group, and Product and Technical Councils. Here at Duke, we have staff from all over the libraries involved in the SIGs. They speak up to be sure the product will work for Duke Libraries.

Features

The institutions planning to implement FOLIO in Summer 2020 spent April ranking 468 open features. They needed to choose  whether the feature was needed at the time the institution planned to go live, or if they could wait for the feature to be added (one quarter later or one year later). Duke voted for 62% of the features be available at the time we go live with FOLIO. These features include things like  default reports, user experience enhancements, and more detailed permission settings, to name a few.

Gaps

After the feature prioritization was complete, we conducted a gap analysis. The gap analysis required our business analysts to take what they’ve learned from conducting interviews with library staff across the University and compare it to what FOLIO can currently do and what is planned. The Duke Libraries’ staff who have been active on the SIGs were extremely helpful in identifying gaps. Some feature requests that came out of the gap analysis included making sure a user has an expiration date associated with it. Another was being able to re-print notices to patrons. Others had to do with workflow, for example, making sure that when a holdings record is “empty” (no items attached), that an alert is sent so a staff person can decide to delete the empty record or not.

Bees?

So where to the bees come into all of this? Well, the logo for FOLIO includes a bee!folio: future of libraries is open. Bee icon

The release names and logos are flowers. And we’re working together in a community toward a single goal – a new Library Services Platform that is community-sourced and works for the future of libraries.

Learn more about FOLIO@Duke by visiting our site: https://sites.duke.edu/folioatduke/. We’ve posted newsletters, presentations, and videos from the FOLIO project team.

hexagon badge, image of aster flower, words folio aster release Jan 2019

hexagon badge, image of bellis flower, words folio bellis release Apr  2019

hexagon badge, image of clove flower, words folio clover release May 2019

hexagon badge, image of daisy flower, words folio daisy release Oct 2019

A simple tool with a lot of power: Project Estimates

It takes a lot to build and publish digital collections as you can see from the variety and scope of the blog posts here on Bitstreams.  We all have our internal workflows and tools we use to make our jobs easier and more efficient.  The number and scale of activities going on behind the scenes is mind-boggling and we would never be able to do as much as we do if we didn’t continually refine our workflows and create tools and systems that help manage our data and work.  Some of these tools are big, like the Duke Digital Repository (DDR), with its public, staff and backend interface used to preserve, secure, and provide access to digital resources, while others are small, like scripts built to transform ArchiveSpace output into a starter digitization guides.  In the Digital Production Center (DPC) we use a homegrown tool that not only tracks production statistics but is also used to do project projections and to help isolate problems that occur during the digitization process.  This tool is a relational database that is affectionately named the Daily Work Report and has collected over 9 years of data on nearly every project in that time.

A long time ago, in a newly minted DPC, supervisors and other Library staff often asked me, “How long will that take?”, “How many students will we need to digitize this collection?”, “What will the data foot print of this project be?”, “How fast does this scanner go?”, “How many scans did we do last year?”, “How many items is that?”.  While I used to provide general information and anecdotal evidence to answer all of these questions, along with some manual hunting down of this information, it became more and more difficult to answer these questions as the number of projects multiplied, our services grew, the number of capture devices multiplied and the types of projects grew to include preservation projects, donor requests, patron request and exhibits.  Answering these seemingly simple questions became more complicated and time consuming as the department grew.  I thought to myself, I need a simple way to track the work being done on these projects that would help me answer these recurring common questions.

We were already using a FileMaker Pro database with a GUI interface as a checkout system to assign students batches of material to scan, but it was only tracking what student worked on what material.  I decided I could build out this concept to include all of the data points needed to answer the questions above.  I decided to use Microsoft Access because it was a common tool installed on every workstation in the department, I had used it before, and classes and instructional videos abound if I wanted to do anything fancy.

Enter the Daily Work Report (DWR).  I created a number of discrete tables to hold various types of data: project names, digitization tasks, employee names and so on.  These fields are connected to a datasheet represented as a form, which allowed for dropdown lists and auto filling for rapid and consistent entry of information. 

At the end of each shift students and professionals alike fill out the DWR for each task they performed on each project and how long they worked on each task.  These range from the obvious tasks of scanning and quality control to more minute tasks of derivative creation, equipment cleaning, calibration, documentation, material transfer, file movement, file renaming, ingest prep, and ingest.

Some of these tasks may seem minor and possibly too insignificant to record but they add up.  They add up to ~30% of the time it takes to complete a project.   When projecting the time it will take to complete a project we collect Scanning and Quality Control data from a similar project, calculate the time and add 30%.

Common Digitization Tasks

Task
Hours Overall % of project
Scanning 406.5 57.9
Quality Control 1 133 19
Running Scripts 24.5 3.5
Collection Analysis 21 3
Derivative Creation 20.5 2.9
File Renaming 15.5 2.2
Material Transfer 14 2
Testing 12.5 1.8
Documentation 10 1.4
File Movement 9.75 1.4
Digitization Guide 7 1
Quality Control 2 6.75 1
Training 6 0.9
Quality Control 3 5.5 0.9
Stitching 3 0.4
Rescanning 1.5 0.2
Finalize 1.5 0.2
Troubleshooting 1.5 0.2
Conservation Consultation 1 0.1
Total 701 100

New Project Estimates

Using the Daily Work Report’s Datasheet View, the database can be filtered by project, then by the “Scanning” task to get the total number of scans and the hours worked to complete those scans.  The same can be done for the Quality Control task.  With this information the average number of scans per hour can be calculated for the project and applied to the new project estimate.

Gather information from an existing project that is most similar to the project you are creating the estimate for.  For example, if you need to develop an estimate for a collection of bound volumes that will be captured on the Zeutschel you should find a similar collection in the DWR to run your numbers.

Gather data from an existing project:

Scanning

  • Number of scans = 3,473
  • Number of hours = 78.5
  • 3,473/78.5 = 2/hr

Quality Control

  • Number of scans = 3,473
  • Number of hours = 52.75
  • 3,473/52.75 = 8/hr

Apply the per hour rates to the new project:

Estimated number of scans: 7,800

  • Scanning: 7,800 / 44.2/hr = 176.5 hrs
  • QC: 7,800 / 68.8/hr = 113.4 hrs
  • Total: 290 hrs
  • + 30%: 87 hrs
  • Grand Total: 377 hrs

Rolling Production Rate

When an update is required for an ongoing project the Daily Work Report can be used to see how much has been done and calculate how much longer it will take.  The number of images scanned in a collection can be found by filtering by project then by the “Scanning” Task.  That number can then be subtracted from the total number of scans in the project.  Then, using a similar project to the one above you can calculate the production rate for the project and estimate the number of hours it will take to complete the project.

Scanning

  • Number of scans in the project = 7,800
  • Number of scans completed = 4,951
  • Number of scans left to do = 7,800 – 4,951 = 2,849

Scanning time to completion

  • Number of scans left = 2,849
  • 2,849/42.4/hr = 2 hrs

Quality Control

  • Number of files to QC in the project = 7,800
  • Number of files completed = 3,712
  • Number of files left to do = 7,800 – 3,712 = 4,088

QC hours to completion

  • Number of scans left to scan = 4,088
  • 4,088/68.8 = 4 hrs

The amount of time left to complete the project

  • Scanning – 67.2 hrs
  • Quality Control – 59.4 hrs
  • Total = 126.2 hrs
  • + 30% = 38
  • Grand Total = 164.2 hrs

Isolate an error

Errors inevitably occur during most digitization projects.  The DWR can be used to identify how widespread the error is by using a combination of filtering, the digitization guide (which is an inventory of images captured along with other metadata about the capture process), and inspecting the images.  As an example, a set of files may be found to have no color profile.  The digitization guide can be used to identify the day the erroneous images were created and who created them. The DWR can be used to filter by the scanner operator and date to see if the error is isolated to a particular person, a particular machine or a particular day.  This information can then be used to filter by the same variables across collections to see if the error exists elsewhere.  The result of this search can facilitate retraining, recalibrating of capture devices and also identify groups of images that need to be rescanned without having to comb through an entire collection.

While I’ve only touched on the uses of the Daily Work Report, we have used this database in many different ways over the years.  It has continued to answer those recurring questions that come up year after year.  How many scans did we do last year?  How many students worked on that multiyear project?  How many patron requests did we complete last quarter?  This database has helped us do our estimates, isolate problems and provide accurate updates over the years.  For such a simple tool it sure does come in handy.

Mythical Beasts of Audio

Gear. Kit. Hardware. Rig. Equipment.

In the audio world, we take our tools seriously, sometimes to an unhealthy and obsessive degree. We give them pet names, endow them with human qualities, and imbue them with magical powers. In this context, it’s not really strange that a manufacturer of professional audio interfaces would call themselves “Mark of the Unicorn.”

Here at the Digital Production Center, we recently upgraded our audio interface to a MOTU 896 mk3 from an ancient (in tech years) Edirol UA-101. The audio interface, which converts analog signals to digital and vice-versa, is the heart of any computer-based audio system. It controls all of the routing from the analog sources (mostly cassette and open reel tape decks in our case) to the computer workstation and the audio recording/editing software. If the audio interface isn’t seamlessly performing analog to digital conversion at archival standards, we have no hope of fulfilling our mission of creating high-quality digital surrogates of library A/V materials.

Edirol UA-101
The Edirol enjoying its retirement with some other pieces of kit

While the Edirol served us well from the very beginning of the Library’s forays into audio digitization, it had recently begun to cause issues resulting in crashes, restarts, and lost work. Given that the Edirol is over 10 years old and has been discontinued, it is expected that it would eventually fail to keep up with continued OS and software updates. After re-assessing our needs and doing a bit of research, we settled on the MOTU 896 mk3 as its replacement. The 896 had the input, output, and sync options we needed along with plenty of other bells and whistles.

I’ve been using the MOTU for several weeks now, and here are some things that I’m liking about it:

  • Easy installation of drivers
  • Designed to fit into standard audio rack
  • Choice of USB or Firewire connection to PC workstation
  • Good visual feedback on audio levels, sample rate, etc. via LED meters on front panel
  • Clarity and definition of sound
MOTU 896mk3
The MOTU sitting atop the audio tower

I haven’t had a chance to explore all of the additional features of the MOTU yet, but so far it has lived up to expectations and improved our digitization workflow. However, in a production environment such as ours, each piece of equipment needs to be a workhorse that can perform its function day in and day out as we work our way through the vaults. Only time can tell if the Mark of the Unicorn will be elevated to the pantheon of gear that its whimsical name suggests!

Bringing 500 Years of Women’s Work Online

Back in 2015, Lisa Unger Baskin placed her extensive collection of more than 11,000 books, manuscripts, photographs, and artifacts in the Sallie Bingham Center for Women’s History & Culture in the David M. Rubenstein Rare Book & Manuscript Library. In late February of 2019, the Libraries opened the exhibit “Five Hundred Years of Women’s Work: The Lisa Unger Baskin Collection” presenting visitors with a first look at the diversity and depth of the collection, revealing the lives of women both famous and forgotten and recognizing their accomplishments. I was fortunate to work on the online component of the exhibit in which we aimed to offer an alternate way to interact with the materials.

Homepage of the exhibit
Homepage of the exhibit

Most of the online exhibits I have worked on have not had the benefit of a long planning timeframe, which usually means we have to be somewhat conservative in our vision for the end product. However, with this high-profile exhibit, we did have the luxury of a (relatively) generous schedule and as such we were able to put a lot more thought and care into the planning phase. The goal was to present a wide range and number of items in an intuitive and user-friendly manner. We settled on the idea of arranging items by time period (items in the collection span seven centuries!) and highlighting the creators of those items.

We also decided to use Omeka (classic!) for our content management system as we’ve done with most of our other online exhibits. Usually exhibit curators manually enter the item information for their exhibits, which can get somewhat tedious. In this case, we were dealing with more than 250 items, which seemed like a lot of work to enter one at a time. I was familiar with the CSV Import plugin for Omeka, which allows for batch uploading items and mapping metadata fields. It seemed like the perfect solution to our situation. My favorite feature of the plugin is that it also allows for quickly undoing an ingest in case you discover that you’ve made a mistake with mapping fields or the like, which made me less nervous about applying batch uploads to our production Omeka instance that already contained about 1,100 items.

Metadata used for batch upload
Metadata used for batch upload

Working with the curators, we came up with a data model that would nest well within Omeka’s default Dublin-core based approach and expanded that with a few extra non-standard fields that we attached to a new custom item type. We then assembled a small sample set of data in spreadsheet form and I worked on spinning up a local instance of Omeka to test and make sure our approach was actually going to work! After some frustrating moments with MAMP and tracking down strange paths to things like imagemagick (thank you eternally, Stack Overflow!) I was able to get things running well and was convinced the batch uploads via spreadsheet was a good approach.

Now that we had a process in place, I began work on a custom theme to use with the exhibit. I’d previously used Omeka Foundation (a grid-based starter theme using the Zurb Foundation CSS framework) and thought it seemed like a good place to start with this project. The curators had a good idea of the site structure that they wanted to use, so I jumped right in on creating some high-fidelity mockups borrowing look-and-feel cues from the beautiful print catalog that was produced for the exhibit. After a few iterations we arrived at a place where everyone was happy and I started to work on functionality. I also worked on incorporating a more recent version of the Foundation framework as the starter theme was out of date.

Print catalog for the exhibit
Print catalog for the exhibit

The core feature of the site would be the ability to browse all of the items we wanted to feature via the Explore menu, which we broke into seven sections — primarily by time period, but also by context. After looking at some other online exhibit examples that I thought were successful, we decided to use a masonry layout approach (popularized by sites like Pinterest) to display the items. Foundation includes a great masonry plugin that was very easy to implement. Another functionality issue had to do with displaying multi-page items. Out of the box, I think Omeka doesn’t do a great job displaying items that contain multiple images. I’ve found combining them into PDFs works much better, so that’s what we did in this case. I also installed the PDF Embed plugin (based on the PDF.js engine) in order to get a consistent experience across browsers and platforms.

Once we got the theme to a point that everyone was happy with it, I batch imported all of the content and proceeded with a great deal of cross-platform testing to make sure things were working as expected. We also spent some time refining the display of metadata fields and making small tweaks to the content. Overall I’m very pleased with how everything turned out. User traffic has been great so far so it’s exciting to know that so many people have been able to experience the wonderful items in the exhibit. Please check out the website and also come visit in person — on display until June 15, 2019.

Examples of 'Explore' and 'Item' pages
Examples of ‘Explore’ and ‘Item’ pages

It Takes a Village to Curate Your Data: Duke Partners with the Data Curation Network

In early 2017, Duke University Libraries launched a research data curation program designed to help researchers on campus ensure that their data are adequately prepared for both sharing and publication, and long term preservation and re-use. Why the focus on research data? Data generated by scholars in the course of their investigation are increasingly being recognized as outputs similar in importance to the scholarly publications they support. Open data sharing reinforces unfettered intellectual inquiry, fosters transparency, reproducibility and broader analysis, and permits the creation of new data sets when data from multiple sources are combined. For these reasons, a growing number of publishers and funding agencies like PLoS ONE and the National Science Foundation are requiring researchers to make openly available the data underlying the results of their research.

Data curation steps

But data sharing can only be successful if the data have been properly documented and described. And they are only useful in the long term if steps have been taken to mitigate the risks of file format obsolescence and bit rot. To address these concerns, Duke’s data curation workflow will review a researcher’s data for appropriate documentation (such as README files or codebooks), solicit and refine Dublin Core metadata about the dataset, and make sure files are named and arranged in a way that facilitates secondary use. Additionally, the curation team can make suggestions about preferred file formats for long-term re-use and conduct a brief review for personally identifiable information. Once the data package has been reviewed, the curation team can then help researchers make their data available in Duke’s own Research Data Repository, where the data can be licensed and assigned a Digital Object Identifier, ensuring persistent access.

 

“The Data Curation Network (DCN) serves as the “human layer” in the data repository stack and seamlessly connects local data sets to expert data curators via a cross-institutional shared staffing model.”

 

New to Duke’s curation workflow is the ability to rely on the domain expertise of our colleagues at a few other research institutions. While our data curators here at Duke possess a wealth of knowledge about general research data-related best practices, and are especially well-versed in the vagaries of social sciences data, they may not always have the all the information they need to sufficiently assess the state of a dataset from a researcher. As an answer to this problem, the Data Curation Network, an Alfred P. Sloan Foundation-funded endeavor, has established a cross-institutional staffing model that distributes the domain expertise of each of its partner institutions. Should a curator at one institution encounter data of a kind with which they are unfamiliar, submission to the DCN opens up the possibility for enhanced curation from a network partner with the requisite knowledge.

DCN Partner Institutions
DCN Partner Institutions

Duke joins Cornell University, Dryad Digital Repository, Johns Hopkins University, University of Illinois, University of Michigan, University of Minnesota, and Pennsylvania State University in partnering to provide curatorial expertise to the DCN. As of January of this year, the project has moved out of pilot phase into production, and is actively moving data through the network. If a Duke researcher were to submit a dataset our curation team thought would benefit from further examination by a curator with domain knowledge, we will now reach out to the potential depositor to receive clearance to submit the data to the network. We’re very excited about this opportunity to provide this enhancement to our service!

Looking forward, the DCN hopes to expand their offerings to include nation-wide training on specialized data curation and to extend the curation services the network offers beyond the partner institutions to individual end users. Duke looks forward to contributing as the project grows and evolves.

Sustainability Planning for a Better Tomorrow

In March of last year I wrote about efforts of the Resource Discovery Systems and Strategies team (RDSS, previously called the Discovery Strategy Team) to map Duke University Libraries’ discovery system environment in a visual way. As part of this project we created supporting documentation for each system that appeared in a visualization, including identifying functional and technical owners as well as links to supporting documentation. Gathering this information wasn’t as straightforward as it ideally should have been, however. When attempting to identify ownership, for example, we were often asked questions like, “what IS a functional owner, anyway?”, or told “I guess I’m the owner… I don’t know who else it would be”. And for many systems, local documentation was outdated, distributed across platforms, or simply nonexistent.

As a quick glance through the Networked Discovery Systems document will evince, we work with a LOT of different systems here at DUL, supporting a great breadth of processes and workflows. And we’ve been steadily adding to the list of systems we support every year, without necessarily articulating how we will manage the ever-growing list. This has led to situations of benign neglect, confusion as to roles and responsibilities and, in a few cases, we’ve hung onto systems for too long because we hadn’t defined a plan for responsible decommission.

So, to promote the healthier management of our Networked Discovery Systems, the RDSS team developed a set of best practices for sustainability planning. Originally we framed this document as best practices for maintenance planning, but in conversations with other groups in the Libraries, we realized that this didn’t quite capture our intention. While maintenance planning is often considered from a technical standpoint, we wanted to convey that the responsible management of our systems involves stakeholders beyond just those in ITS, to include the perspective and engagement of non-technical staff. So, we landed on the term sustainability, which we hope captures the full lifecycle of a system in our suite of tools, from implementation, through maintenance, to sunsetting, when necessary.

The best practices are fairly short, intended to be a high-level guide rather than overly prescriptive, recognizing that every system has unique needs. Each section of the framework is described, and key terms are defined. Functional and technical ownership are described, including the types of activities that may attend each role, and we acknowledge that ownership responsibilities may be jointly accomplished by groups or teams of stakeholders. We lay out the following suggested framework for developing a sustainability plan, which we define as “a living document that addresses the major components of a system’s life cycle”:

  • Governance:
    • Ownership
    • Stakeholders
    • Users
  • Maintenance:
    • System Updates
    • Training
    • Documentation
  • Review:
    • Assessments
    • Enhancements
    • Sunsetting

Interestingly, and perhaps tellingly, many of the conversations we had about the framework ended up focusing on the last part – sunsetting. How to responsibly decommission or sunset a system in a methodical, process-oriented way is something we haven’t really tackled yet, but we’re not alone in this, and the topic is one that is garnering some attention in project management circles.

So far, the best practices have been used to create a sustainability plan for one of our systems, Dukespace, and the feedback was positive. We hope that these guidelines will facilitate the work we do to sustain our system, and in so doing lead to better communication and understanding throughout the organization. And we didn’t forget to create a sustainability plan for the best practices themselves – the RDSS team has committed to reviewing and updating it at least annually!

Something Good

One of the highlights of the Association of Moving Image Archivists’ annual conference is “Archival Screening Night,” where members of the AMIA community showcase recently-discovered and newly-restored film and video footage. The event usually takes place in a historic movie theater, with skilled projectionists that are able to present the film materials in their original format, on the big screen. At the most recent AMIA conference, in Portland, Oregon, there was a wide array of impressive material presented, but one film in particular left the audience speechless, and is a wonderful example of how archivists can unearth treasures that can alter our perspective on human history, and humanity itself.

The film, “Something Good – Negro Kiss” was made in 1898. It’s silent, black & white, and is less than a minute long. But it’s groundbreaking, in that it shows the earliest known depiction of an African-American couple kissing, and stands in opposition to the racist, minstrel-show portrayals of black people so common in the early days of American filmmaking. The couple embrace, kiss, and sway back and forth in a playful, spontaneous dance that comes across as genuine and heartwarming. Although it may not have been intentional, the short film seems to be free of negative racial stereotypes. You can watch it here:

Dino Everett, an archivist at the University of Southern California’s Hugh M. Hefner Moving Image Archive, recently discovered the 35mm nitrate film within a batch of silent films once owned by a Louisiana collector. Unique perforation marks helped him to identify the film’s age, and Allyson Nadia Field, of the University of Chicago, was able to help track down the history: where it was shot (Chicago), who the filmmaker was (William Selig), and what the original title of the the film was (Something Good). The actors have been identified as Saint Suttle and Gertie Brown. The film has now been added to the Library of Congress’ National Film Registry.

The film is likely an homage to “The Kiss” (also known as the May Irwin Kiss), a film made in 1896, with a white couple kissing. It was one of the first films ever shown commercially, and is the very first kiss on film. Even though the couple was white, and the kissing is remarkably tame by today’s standards,  it created a lot of controversy at the time, because kissing in public was prohibited by law.  The Catholic church and newspaper editorials denounced “The Kiss” and called for censorship and prosecution.  Although there is no documented history yet about the public reaction to “Something Good – Negro Kiss,” one can only imagine the shock and scandal it must have caused, showing an African-American couple kissing each other, only two years later.

Digital Collections Round Up 2018

It’s that item of year where we like to reflect on all we have done in 2018, and your favorite digital collections and curation services team is no exception.  This year, Digital Collections and Curation Services have been really focusing on getting collections and data into the Digital Repository and making it accessible to the world!

As you will see from the list below we launched 320 new digital collections, managed substantial additions to 2, and migrated 8. However, these publicly accessible digital collections are just the tip of the iceberg in terms of our work in Digital Collections and Curation Services.

A cover from the Ladyslipper, Inc Retail Catalogs digital collection.

So much more digitization happens behind the scenes than is reflected in the list of new digital collections.  Many of our larger projects are years in the making. For example, we continue to digitize Gedney photographs and we hope to make them publicly accessible next year.  There is also preservation digitization happening that we cannot make publicly accessible online.  This work is essential to our preservation mission, though we cannot share the collections widely in the short term.

We strongly believe in keeping our metadata healthy, so in addition to managing new metadata, we often revisit existing metadata across our repositories in order to ensure its overall quality and functionality.

Our team is also responsible for ingesting not just digital collections, but research data and library collections as well.  We preserved 20 datasets produced by the Duke Scholarly Community in the Research Data Repository (https://research.repository.duke.edu/)  via the Research Data Curation program https://library.duke.edu/data/data-management.

A selection from the Buffalo Bill papers, digitized as part of the Section A project.

New Digital Collections 2018

Additions to Existing Digital Collections

The men’s basketball team celebrates its 1991 championship win.

Collections Migrated into the Digital Repository

Digitization Details: The Process of Digitizing a Collection

About four and a half years ago I wrote a blog post here on Bitstreams titled: “Digitization Details: Before We Push the “Scan” Button” in which I wrote about how we use color calibration, device profiling and modified viewing environments to produce “consistent results of a measurable quality” in our digital images.  About two and a half years ago, I wrote a blog post adjacent to that subject titled “The FADGI Still Image standard: It isn’t just about file specs” about the details of the FADGI standard and how its guidelines go beyond ppi and bit depth to include information about UV light, vacuum tables, translucent material, oversized material and more.  I’m surprised that I have never shared the actual process of digitizing a collection because that is what we do in the Digital Production Center.

Building digital collections is a complex endeavor that requires a cross-departmental team that analyzes project proposals, performs feasibility assessments, gathers project requirements, develops project plans, and documents workflows and guidelines in order to produce a consistent and scalable outcome in an efficient manner.  We call our cross-departmental team the Digital Collections Implementation Team (DCIT) which includes representatives from the Conservation staff, Technical Services, Digital Production, Metadata Architects and Digital Collections UI developers, among others.  By having representatives from each department participate, we are able to consider all perspectives including the sticking points, technical limitations and time constraints of each department. Over time, our understanding of each other’s workflows and sticking points has enabled us to refine our approach to efficiently hand off a project between departments.

I will not be going into the details of all the work other departments contribute to building digital collections (you can read just about any post on the blog for that). I will just dip my toe into what goes on in the Digital Production Center to digitize a collection.

Digitization

Once the specifics of a project are nailed down, the scope of the project has been finalized, the material has been organized by Technical Services, Conservation has prepared the material for digitization, the material has been transferred to the Digital Production Center and an Assessment Checklist is filled out describing the type, condition, size and number of items in a collection, we are ready to begin the digitization process.

Digitization Guide
A starter digitization guide is created using output from ArchivesSpace and the DPC adds 16-20 fields to capture technical metadata during the digitization process. The digitization guide is an itemized list representing each item in a collection and is centrally stored for ease of access. 

Setup
Cameras and monitors are calibrated with a spectrometer.  A color profile is built for each capture device along with job settings in the capture software. This will produce consistent results from each capture device and produce an accurate representation of any items captured which in turn removes subjective evaluation from the scanning process.

Training
Instructions are developed describing the scanning, quality control, and handling procedures for the project and students are trained.

Scanning
Following instructions developed for each collection, the scanner operator will use the appropriate equipment, settings and digitization guide to digitize the collection.  Benchmark tests are performed and evaluated periodically during the project. During the capture process the images are monitored for color fidelity and file naming errors. The images are saved in a structured way on the local drive and the digitization guide is updated to reflect the completion of an item.   At the end of each shift the files are moved to a production server.

Quality Control 1
The Quality Control process is different depending on the device with which an item was captured and the nature of the material.  All images are inspected for:  correct file name, skew, clipping, banding, blocking, color fidelity, uniform crop, and color profile.  The digitization guide is updated to reflect the completion of an item.

Quality Control 2
Images are cropped (leaving no background) and saved as JPEGs for online display.  During the second pass of quality control each image is inspected for: image consistency from operator to operator and image to image, skew and other anomalies.

Finalize
During this phase we compare the digitization guide against the item and file counts of the archival and derivative images on our production server.   Discrepancies such as missing files, misnamed files and missing line items in the digitization guide and are resolved.

Create Checksums and dark storage
We then create a SHA1 checksum for each image file in the collection and push the collection into a staging area for ingest into the repository.

Sometimes this process is referred to simply as “scanning”.

Not only is this process in active motion for multiple projects at the same time, the Digital Production Center also participates in remediation of legacy projects for ingest into the Duke Digital Repository, multispectral imaging, audio digitization and video digitization for, preservation, patron and staff requests… it is quite a juggling act with lots of little details but we love our work!

Time to get back to it so I can get to a comfortable stopping point before the Thanksgiving break!