Category Archives: MorphoSource

The Shortest Year

Featured image – screenshot from the Sunset Tripod2 project charter.

Realizing that my most recent post here went up more than a year ago, I pause to reflect. What even happened over these last twelve months? Pandemic and vaccine, election and insurrection, mandates and mayhem – outside of our work bubble, October 2020 to October 2021 has been a churn of unprecedented and often dark happenings. Bitstreams, however, broadcasts from inside the bubble, where we have modeled cooperation and productivity, met many milestones, and kept our collective cool, despite working nearly 100% remotely as a team, with our stakeholders, and across organizational lines.

Last October, I wrote about Sunsetting Tripod2, a homegrown platform for our digital collections and archival finding aids that was also the final service we had running on a physical server. “Firm plans,” I said we had for the work that remained. Still, in looking toward that setting sun, I worried about “all sorts of comical and embarrassing misestimations by myself on the pages of this very blog over the years.” I was optimistic, but cautiously so, that we would banish the ghosts of Django-based systems past.

Reader, I have returned to Bitstreams to tell you that we did it. Sometime in Q1 of 2021, we said so long, farewell, adieu to Tripod2. It was a good feeling, like when you get your laundry folded, or your teeth cleaned, only better.

However, we did more in the past year than just power down exhausted old servers. What follows are a few highlights from the work of the Digital Strategies and Technology division of Duke University Libraries’ software developers, and our collaborators (whom we cannot thank or praise enough) over the past twelve months. 

In November, Digital Projects Developer Sean Aery posted on Implementing ArcLight: A Reflection. The work of replacing and improving upon our implementation for the Rubenstein Library’s collection guides was one of the main components that allowed us to turn off Tripod2. We actually completed it in July of 2020, but that team earned its Q4 victory laps, including Sean’s post and a session at Blacklight Summit a few days after my own post last October.

As the new year began, the MorphoSource team rolled out version 2.0 of that platform. MorphoSource Repository Developer Jocelyn Triplett shared a A Preview of MorphoSource 2 Beta in these pages on January 20. The launch took place on February 1.

One project we had underway as I was writing last October was the integration of Globus, a transfer service for large datasets, into the Duke Research Data Repository. We completed that work in Q1 of 2021, prompting our colleague, Senior Research Data Management Consultant Sophia Lafferty-Hess, to post Share More Data in the Duke Research Data Repository! in a neighboring location that shares our charming cul-de-sac of library blogs.

The seventeen months since the murder of George Floyd have seen major changes in how we think and talk about race in the Libraries. We committed ourselves to the DUL Racial Justice Roadmap, a pathway for recognizing and attacking the pervasive influence of white supremacy in our society, in higher education, at Duke, in the field of librarianship, in our library, in the field of information technology, and in our own IT practices. During this time, members of our division have also participated broadly in DiversifyIT, a campus-wide group of IT professionals who seek to foster a culture of inclusion “by providing professional development, networking, and outreach opportunities.”

Digital Projects Developer Michael Daul shared his own point of view with great thoughtfulness in his April post, What does it mean to be an actively antiracist developer? He touched on representation in the IT industry, acknowledging bias, being aware of one’s own patterns of communication, and bringing these ideas to the systems we build and maintain. 

One of the ideas that Michael identified for software development is web accessibility; as he wrote, we can “promote the benefits of building accessible interfaces that follow the practices of universal design.” We put that idea into action a few months later, as Sean described in precise technical terms in his July post, Automated Accessibility Testing and Continuous Integration. Currently that process applies to the ArcLight platform, but when we have a chance, we’ll see if we can expand it to other services.

The question of when we’ll have that chance is a big one, as it hinges on the undertaking that now dominates our attention. Over the past year we have ramped up on the migration of our website from Drupal 7 to Drupal 9, to head off the end-of-life for 7. This project has transformed into the raging beast that our colleagues at NC State Libraries warned us it would at the Code4Lib Southeast in May of 2019

Screenshot of NC State Libraries presentation on Drupal migration
They warned us – Screenshot from “Drupal 7 to Drupal 8: Our Journey,” by Erik Olson and Meredith Wynn of NC State Libraries’ User Experience Department, presented at Code4Lib Southeast in May of 2019.

We are on a path to complete the Drupal migration in March 2022 – we have “firm plans,” you could say – and I’m certain that its various aspects will come to feature in Bitstreams in due time. For now I will mention that it spawned two sub-projects that have challenged our team over the past six months or so, both of which involve refactoring functionality previously implemented as Drupal modules into standalone Rails applications:

  1. Quicksearch, aka unified search, aka “Bento search” – see Michael’s Bento is Coming! from 2014 – is now a standalone app; it also uses the open-source tool Apache Nutch, rather than Google CSE.
  2. The staff directory app that went live in 2019, which Michael wrote about in Building a new Staff Directory, also no longer runs as a Drupal module.

Each of these implementations was necessary to prepare the way for a massive migration of theme and content that will take place over the coming months. 

Screenshot of a Jira issue related to the Decouple Staff Directory project.
Screenshot of a Jira issue related to the Decouple Staff Directory project.

When it’s done, maybe we’ll have a chance to catch our breath. Who can really say? I could not have guessed a year ago where we’d be now, and anyway, the period of the last twelve months gets my nod as the shortest year ever. Assuming we’re here, whatever “here” means in the age of remote/hybrid/flexible work arrangements, then I expect we’ll be burning down backlogs, refactoring this or that, deploying some service, and making firm plans for something grand.

A Preview of MorphoSource 2 Beta

It’s an exciting time for the MorphoSource team, as we work to launch the MorphoSource 2 Beta application next Wednesday!

The new application improves and expands upon the original MorphoSource, a repository for 3D research data, and is being built using Hyrax, an open-source digital repository application widely implemented by libraries to manage digital repositories and collections. The team has been working on the site for the last two and a half years, and is looking forward to our efforts being made available to the MorphoSource community. At launch, users will be able to access records for over 140,000 media files, contributed by 1,500 researchers from all over the world.

MorphoSource 2 Beta Homepage
MorphoSource 2 Beta Homepage

While the current site is still available for browsing at www.morphosource.org, we are migrating the repository data over to the new site in preparation for the launch, and have paused the ingest of new data sets. When the migration is complete, users will be able to access the new application at the current url. Users with an account on the old site will be able to log in to the new site using their MorphoSource 1 credentials.

In my last post in June, I described some of the features that were in development at that time. In this post, I’ll highlight a few recent additions with screenshots from the beta site: Browse, Search, and User Dashboards.

Browse

Browse pages have been added as a quick entry point for users to discover data in several different ways. Users can use these pages to immediately access media, biological specimens, cultural heritage objects, organizations, teams, or projects.

MorphoSource Browse Categories
MorphoSource 2 Beta Browse Categories

Media Types and Modalities: Users can view all media records of a specific file type, such as image, CT image series, or mesh or point cloud. There are also links to records created by different methods, such as X-Ray, Magnetic Resource Imaging, or Photogrammetry.

Physical Object Types: Links to view either all the Biological Specimens or Cultural Heritage Objects in MorphoSource

Biological Taxonomy:  Users can find specimen records through the taxonomy browse by drilling down through the taxonomic ranks. The MorphoSource taxonomy records have been imported from the GBIF Backbone Taxonomy or have been created by MorphoSource users.

Taxonomy browse page
Taxonomy Browse Page

Projects: Projects are user-created groupings of media and specimens. From the browse page, projects can be searched by title and sorted by title, description, team, creator, or number of associated media or objects.

MorphoSource 2 Project Browse
Project Browse Page

Teams: Teams are groups of MorphoSource users that share management of media and team projects. A Team may be associated with an organization. The Team browse page lets users search and sort teams in a similar way to the Projects browse page.

Organizations: Lastly, users can view all of the organizations that have biological specimens or cultural heritage objects in MorphoSource. An organization may be an institution, department, collection, facility, lab, or other group. From the Organizations browse page, users can search by name and sort by parent institution name or institution code.

Faceted Searching

In addition to the browse pages, records for Media, Biological Specimens, Cultural Heritage Objects, Organizations, Teams, and Projects can also be found through the MorphoSource search interface. Searching has been customized for the different record types to include relevant facets. The different search categories can be chosen from the dropdown next to the search box ‘Go’ button.

MorphoSource 2 Beta Media Search Results
Media Search Results

Search results for media records can be faceted by file type, modality, object type (biological specimen or cultural heritage object), organization, tag, or membership in a team or project, while search results for objects can be limited by object type, creator, organization, taxonomy, associated media types, associated media tags, and membership of associated media in a team or project. Organization and Team/Project searches similarly have their own sets of facets.

MorphoSource 2 Object Search
Biological Specimen and Cultural Heritage Object Search Results

User Dashboards

Users who register an account on the site will have access to a dashboard that enables them to manage their data downloads. The dashboard is accessed by clicking on the profile icon at the top right of the site, and will open to the user’s media cart. The media cart contains two sections – the top holds all media items that the user currently has permission to download, while the bottom has media items with a restricted status where download has not been requested or approved:

MorphoSource 2 Beta Media Cart
Default User Dashboard

Users who have been granted contributor access to the site will have a dashboard that opens to the media and objects that they have contributed:

MorphoSource 2 Beta Contributor Dashboard
Contributor Dashboard

From the menu at the left,  all users can access their previous downloads, or projects, teams, or other repository content to which they have been granted access, and manage their user profile.  In addition, contributors can also create and manage projects and teams.

We hope that the browse, search, and dashboard enhancements, along with the other features we have been working on over the last couple of years, will enable users to easily discover and manage data sets in MorphoSource. And although we are looking forward to the launch, we are also excited to continue working on the site, and will be adding even more features in the near future.

MorphoSource: Features in Development

For the last two years, developers in Software Services and Duke’s department of Evolutionary Anthropology have been working to rebuild MorphoSource, a repository for 3D research data representing physical objects, primarily biological specimens. MorphoSource 2.0 is being built using Hyrax, an open-source digital repository application widely implemented by libraries to manage digital repositories and collections. While Hyrax already provided much of the core functionality needed for management, access, and preservation of our data, the MorphoSource team has been customizing the application and adding additional features to tailor it to the needs of our users.

As a preview, here are some of the features we’re developing for the MorphoSource community:

Guided Submission Process

MorphoSource is open to subject experts, collection curators, and the public to submit their data and make it accessible to others. The MorphoSource submission process will guide users in entering metadata that provides additional description and context for the files being uploaded. Users will be able to save information about the specimen that was scanned, the equipment that was used to capture the 3d data, the data capture process, and related media in MorphoSource. The form is multi-step and nested, with different fields available or data pre-filled in depending on what the user has already entered in earlier sections of the form. The user also has opportunities to search for related organizations, devices, specimens, and media to link to their submission. Once the submission process is complete,  depositors are able to return to their data page to edit  metadata and redefine relationships to other MorphoSource records.

Morphosource data submission screenshots
A user proceeds through the MorphoSource media submission process.

In the gif above, a MorphoSource user proceeds through the submission process. If the media being uploaded (such as an image stack or 3d mesh) is a derivative of another object that is already in MorphoSource, the user can search for that media and link it to their submission, whereupon related metadata will be auto-filled for them. If it is a totally new submission, the user will proceed from filling in information about the object that was scanned (ownership, taxonomy, and descriptive details) to devices and processing steps used before attaching their files. This ability to nest, search and associate metadata from the organization level all the way down to an individual object component in a single interface is unique to MorphoSource and is a substantial addition to Hyrax’s code.

Displaying and Updating Media Records

Screenshot of a Media Record in MorphoSource
A media record in MorphoSource.

Following submission, the depositor can view their completed media record, as shown above. Data owners (and other users who have been granted edit permissions by the data owner) have the ability to freely edit their submissions at any time. The edit view (below) allows the user to move between tabs to update metadata or change links to other records in MorphoSource. These tabs  offer a fast summary of the types of associated data that are typically collected for complex objects and digitization strategies. This system is also designed to encourage best practices in project and object documentation by reminding the user of critical metadata categories.

Editing a record in MorphoSource

3D Viewer

Through a collaboration with Mnemoscene, MorphoSource has created Aleph, a web viewer for 3D models and volumes that can be used by itself or as an extension to the popular library document viewer Universal Viewer. A live example of a specimen in the viewer is below. You can rotate the object by clicking and dragging inside the frame, or switch between slices and volume view by changing the mode under tools.

The Media Cart and Restricted Downloads

A media depositor can choose from several different publication statuses for their data, allowing the data owner to retain different levels of control of both their data sets and the metadata record describing their deposit. While some may choose to publish both their metadata and data sets with an open status, allowing other users to freely view and download them, it is also possible for depositors to restrict either their records or data to other users or organizations, or require that a user is granted permission before they are permitted to download data.

When data owners choose to publish using restricted download, other users are able to view metadata and preview the 3D data in the Aleph viewer, but can’t download the data set until they are granted permission by the data owner, and are required to fill out a request including the manner they intend to use the data. Data owners can easily review and manage these download requests from their user dashboard.

Screenshot of MorphoSource media cartAbove is a view of a user’s media cart, where they collect media they intend to download. The top section has media items the user is free to download, either because the media was published with an open publication status or because the user was approved to download by the media owner. Users can download any or all of these items at one time. The bottom section of the page contains items with a restricted download publication status, and allows the user to request these items individually or as a group. Users can also track the status of their request from this page.

Screenshot showing request management

In the next image, a data owner views requests to download their media. Requests are grouped by requesting user and then the intended use of the data. Data owners can approve, deny, or clear any or all of the requests. When approving a request, the data owner specifies the amount of time that the media will be available for the requester to download. This person can also clear a request if they want more information from the requester before they approve the download. Requests that have already been decided are available for review in the Previous Requests tab.

Organizational Teams

Organizational teams are the final substantial addition to the new MorphoSource. Any organization, such as a university, museum, or department, can create an organizational team that stores metadata about the team and assigns roles to individual members. Members of the organizational team can also curate team projects, grant other users access to the team’s media, edit the organization’s metadata, and view any media created from objects in their collections. Below is the public view of one of our first sample organizational teams, the Nasher Museum.

Screenshot of an Organizational Team in MorphoSource
An organization page in MorphoSource. Image via Boyer, Silverton, and Winchester, 2020. MorphoSource: Creating a 3D web repository capable of archiving complex workflows and providing novel viewing experiences

The MorphoSource team is looking forward to unveiling the beta version of MorphoSource 2.0 later this year. In the meantime, please visit the current repository at www.morphosource.org. For further reading, check out the recent EuropeanaTech article MorphoSource: Creating a 3D web repository capable of archiving complex workflows and providing novel viewing experiences.

Looking Ahead to MorphoSource 2.0

MorphoSource logo For the past year, developers in the Library’s Software Services department have been working to rebuild Duke’s MorphoSource repository for 3D research data. The current repository, available at www.morphosource.org, provides a place for researchers and curators to make scans of biological specimens available to other researchers and to the general public.

MorphoSource, first launched in 2013, has become the most popular website for virtual fossils in the world.  The site currently contains sixty thousand data sets representing twenty thousand specimens from seven thousand different species. In 2017, led by Doug Boyer in Duke Evolutionary Anthropology, the project received a National Science Foundation grant. Under this grant, the technical infrastructure for the repository will be moved to the Library’s management, and the user interface is being rebuilt using Hyrax, an open-source digital repository application widely implemented by libraries that manage research data.  The scope of the repository is being expanded to include data for cultural heritage objects, such as museum artifacts, architecture, and archaeological sites. Most importantly, MorphoSource is being improved with better performance, a more intuitive user experience, and expanded functionality for users to view and interact with the data within the site.

Viewing and manipulating CT scans and the derived 3D model of a platypus in the MorphoSource viewer

Management of 3D data is in itself complicated.  It becomes even more so when striving towards long-term preservation of the digital representation of a unique biological specimen. In many cases, these specimens no longer exist, and the 3D data becomes the only record of their particular morphology.  It’s necessary to collect not only the actual digital files, but extensive metadata describing both the data’s creation and the specimen that was scanned to create the data. This can make the process of contributing data daunting for researchers. To improve the user experience and assist users with entering metadata about their files, MorphoSource 2.0 will guide them through the process. Users will be asked questions about their data, what it represents, when and how it was created, and if it is a derivative of data already in MorphoSource. As they progress through making their deposit, the answers they provide will direct them through linking their deposit to records already in the repository, or help them with entering new metadata about the specimen that was scanned, the facility and equipment used to scan the specimen, and any automated processes that were run to create the files.

MorphoSource page showing an alligator skull
Screenshot of a MorphoSource media page showing an alligator skull.

The new repository will also improve the experience for users exploring metadata about contributed resources and viewing the accompanying 3D files. All of the data describing technical information, acquisition and processing information, ownership and permissions, and related files will be gathered in one page, and give users the option to expand or collapse different metadata sections as their interests dictate. A file viewer will also be embedded in the page, which also allows for full-screen viewing and provides several new tools for users analyzing the media. Besides being able to move and spin the model within the viewer, users can also adjust lighting and other factors to focus on different areas of the model, and take custom measurements of different points on the specimen. Most exciting, for CT image series, users can scroll through the images along three axes, or convert the images to a 3D model. For some data, users will also be able to share models by embedding the file viewer in a webpage.

The MorphoSource team is very excited about our planned improvements, and plans to launch MorphoSource 2.0 in 2020. Stay tuned for the launch date, and in the meantime please visit the current site: www.morphosource.org.