{"id":5264,"date":"2017-09-29T11:40:42","date_gmt":"2017-09-29T15:40:42","guid":{"rendered":"http:\/\/blogs.library.duke.edu\/bitstreams\/?p=5264"},"modified":"2017-09-29T11:42:05","modified_gmt":"2017-09-29T15:42:05","slug":"theres-stuff-meet-filetracker","status":"publish","type":"post","link":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/","title":{"rendered":"And Then There’s The Other Stuff… Meet FileTracker"},"content":{"rendered":"

The Duke Digital Repository<\/a> is a pretty nice place if you’re a file in need of preservation and perhaps some access.\u00a0 Provided you’re well-described and your organizational relationship to other files and collections is well understood, you could hardly hope for a better home.\u00a0 But what if you’re not?\u00a0 What if you’re an important digitized file with only collection-level description?\u00a0 Or what if you’re digital reproduction of an 18th century encyclopedia created by a conservator<\/a> to supplement traditional conservation methods?\u00a0 It takes time to prepare materials for the repository.\u00a0 We try our best to preserve the materials in the repository, but we also have to think about the other stuff.<\/p>\n

We may apply different levels of preservation<\/a> to materials depending on their source, uniqueness, cost to reproduce or reacquire, and other factors, but the baseline is knowing the objects we’re maintaining are the same objects we were given.\u00a0 For that, we rely on fixity and checksums<\/a>.\u00a0 Unfortunately, it’s not easy to keep track of a couple of hundred terabytes of files from different collections, with different organizational schemes, different owners, and sometimes active intentional change.\u00a0 The hard part isn’t only knowing what has changed, but providing that information to the owners and curators of the data so they can determine if those changes are intentional and desirable.\u00a0 Seems like a lot, right?<\/p>\n

We’re used some great tools from our colleagues, notably ACE Audit Control Environment<\/a>, for scheduled fixity reporting.\u00a0 We really wanted, though, to provide reporting to data owners that was tailored to they way they thought of their data to help reduce noise (with hundreds of terabytes there can be a lot of it!) and make it easier for them to identify unintentional changes.\u00a0 So, we got work.<\/p>\n

That work is named FileTracker.\u00a0 FileTracker is a Rails application for tracking files and their fixity information.\u00a0 It’s got a nice dashboard, too.<\/p>\n

\"\"<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

What we really needed, though, was a way to disentangle the work of the monitoring application from the work of stakeholder reporting.\u00a0 The database that FileTracker generates makes it much easier to generate reports that contain the information that stakeholders want.\u00a0 For instance, one stakeholder may want to know the number of files in each directory and the difference between the present number of files and the number of files at last audit.\u00a0 We can also determine when files have been moved or renamed and not report those as missing files.<\/p>\n

If you’d like to know more, see https:\/\/github.com\/duke-libraries\/file-tracker<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

The Duke Digital Repository is a pretty nice place if you’re a file in need of preservation and perhaps some access.\u00a0 Provided you’re well-described and your organizational relationship to other files and collections is well understood, you could hardly hope for a better home.\u00a0 But what if you’re not?\u00a0 What if you’re an important digitized … Continue reading And Then There’s The Other Stuff… Meet FileTracker<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":127,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5264","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nAnd Then There's The Other Stuff... Meet FileTracker - Bitstreams: The Digital Collections Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"And Then There's The Other Stuff... Meet FileTracker - Bitstreams: The Digital Collections Blog\" \/>\n<meta property=\"og:description\" content=\"The Duke Digital Repository is a pretty nice place if you’re a file in need of preservation and perhaps some access.\u00a0 Provided you’re well-described and your organizational relationship to other files and collections is well understood, you could hardly hope for a better home.\u00a0 But what if you’re not?\u00a0 What if you’re an important digitized … Continue reading And Then There’s The Other Stuff… Meet FileTracker →\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/\" \/>\n<meta property=\"og:site_name\" content=\"Bitstreams: The Digital Collections Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/dukedigitalcoll\" \/>\n<meta property=\"article:published_time\" content=\"2017-09-29T15:40:42+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-09-29T15:42:05+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png\" \/>\n<meta name=\"author\" content=\"Jim Tuttle\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@dukedigitalcoll\" \/>\n<meta name=\"twitter:site\" content=\"@dukedigitalcoll\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jim Tuttle\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/\"},\"author\":{\"name\":\"Jim Tuttle\",\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/#\/schema\/person\/8b291f0074d379727ad84781c8bea102\"},\"headline\":\"And Then There’s The Other Stuff… Meet FileTracker\",\"datePublished\":\"2017-09-29T15:40:42+00:00\",\"dateModified\":\"2017-09-29T15:42:05+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/\"},\"wordCount\":437,\"commentCount\":0,\"image\":{\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#primaryimage\"},\"thumbnailUrl\":\"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png\",\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/\",\"url\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/\",\"name\":\"And Then There's The Other Stuff... Meet FileTracker - Bitstreams: The Digital Collections Blog\",\"isPartOf\":{\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#primaryimage\"},\"thumbnailUrl\":\"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png\",\"datePublished\":\"2017-09-29T15:40:42+00:00\",\"dateModified\":\"2017-09-29T15:42:05+00:00\",\"author\":{\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/#\/schema\/person\/8b291f0074d379727ad84781c8bea102\"},\"breadcrumb\":{\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#primaryimage\",\"url\":\"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png\",\"contentUrl\":\"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"And Then There’s The Other Stuff… Meet FileTracker\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/#website\",\"url\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/\",\"name\":\"Bitstreams: The Digital Collections Blog\",\"description\":\"Notes from the Duke University Libraries Digital Projects Team\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/#\/schema\/person\/8b291f0074d379727ad84781c8bea102\",\"name\":\"Jim Tuttle\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8356e12c7ca3de5db964d370bc469b9f947fad625a5b884cbbfb3e70c1bfd70c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8356e12c7ca3de5db964d370bc469b9f947fad625a5b884cbbfb3e70c1bfd70c?s=96&d=mm&r=g\",\"caption\":\"Jim Tuttle\"},\"url\":\"https:\/\/blogs.library.duke.edu\/bitstreams\/author\/jt112duke-edu\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"And Then There's The Other Stuff... Meet FileTracker - Bitstreams: The Digital Collections Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/","og_locale":"en_US","og_type":"article","og_title":"And Then There's The Other Stuff... Meet FileTracker - Bitstreams: The Digital Collections Blog","og_description":"The Duke Digital Repository is a pretty nice place if you’re a file in need of preservation and perhaps some access.\u00a0 Provided you’re well-described and your organizational relationship to other files and collections is well understood, you could hardly hope for a better home.\u00a0 But what if you’re not?\u00a0 What if you’re an important digitized … Continue reading And Then There’s The Other Stuff… Meet FileTracker →","og_url":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/","og_site_name":"Bitstreams: The Digital Collections Blog","article_publisher":"https:\/\/www.facebook.com\/dukedigitalcoll","article_published_time":"2017-09-29T15:40:42+00:00","article_modified_time":"2017-09-29T15:42:05+00:00","og_image":[{"url":"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png","type":"","width":"","height":""}],"author":"Jim Tuttle","twitter_card":"summary_large_image","twitter_creator":"@dukedigitalcoll","twitter_site":"@dukedigitalcoll","twitter_misc":{"Written by":"Jim Tuttle","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#article","isPartOf":{"@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/"},"author":{"name":"Jim Tuttle","@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/#\/schema\/person\/8b291f0074d379727ad84781c8bea102"},"headline":"And Then There’s The Other Stuff… Meet FileTracker","datePublished":"2017-09-29T15:40:42+00:00","dateModified":"2017-09-29T15:42:05+00:00","mainEntityOfPage":{"@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/"},"wordCount":437,"commentCount":0,"image":{"@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#primaryimage"},"thumbnailUrl":"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png","inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/","url":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/","name":"And Then There's The Other Stuff... Meet FileTracker - Bitstreams: The Digital Collections Blog","isPartOf":{"@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#primaryimage"},"image":{"@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#primaryimage"},"thumbnailUrl":"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png","datePublished":"2017-09-29T15:40:42+00:00","dateModified":"2017-09-29T15:42:05+00:00","author":{"@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/#\/schema\/person\/8b291f0074d379727ad84781c8bea102"},"breadcrumb":{"@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#primaryimage","url":"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png","contentUrl":"http:\/\/blogs.library.duke.edu\/bitstreams\/files\/2017\/09\/Selection_001.png"},{"@type":"BreadcrumbList","@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/2017\/09\/29\/theres-stuff-meet-filetracker\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blogs.library.duke.edu\/bitstreams\/"},{"@type":"ListItem","position":2,"name":"And Then There’s The Other Stuff… Meet FileTracker"}]},{"@type":"WebSite","@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/#website","url":"https:\/\/blogs.library.duke.edu\/bitstreams\/","name":"Bitstreams: The Digital Collections Blog","description":"Notes from the Duke University Libraries Digital Projects Team","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blogs.library.duke.edu\/bitstreams\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/#\/schema\/person\/8b291f0074d379727ad84781c8bea102","name":"Jim Tuttle","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blogs.library.duke.edu\/bitstreams\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8356e12c7ca3de5db964d370bc469b9f947fad625a5b884cbbfb3e70c1bfd70c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8356e12c7ca3de5db964d370bc469b9f947fad625a5b884cbbfb3e70c1bfd70c?s=96&d=mm&r=g","caption":"Jim Tuttle"},"url":"https:\/\/blogs.library.duke.edu\/bitstreams\/author\/jt112duke-edu\/"}]}},"_links":{"self":[{"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/posts\/5264","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/users\/127"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/comments?post=5264"}],"version-history":[{"count":4,"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/posts\/5264\/revisions"}],"predecessor-version":[{"id":5269,"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/posts\/5264\/revisions\/5269"}],"wp:attachment":[{"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/media?parent=5264"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/categories?post=5264"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.library.duke.edu\/bitstreams\/wp-json\/wp\/v2\/tags?post=5264"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}