BitCurator Environment
Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Links to BitCurator blog, forums and user groups, scripts library, and user and technical documentation.

BitCurator-Related Resources:

Forums and User Groups:

User and Technical Documentation:

Datasets Library

The BitCuratorEdu project team is building a library of datasets for educational, testing, and research purposes.

This document summarizes known available sample data. If you know of other examples that are already publicly available or how your own examples that you would be willing to share, please let the BitCuratorEdu project team know.

The team will continue to build this content as we gather it over the 2019-2021 project period.

General Resources about Digital Forensics Sample Data

A comprehensive review of available datasets for cyber forensics research was presented at the 2017 Digital Forensics Research Workshop.

Forensics Wiki list of forensic corpora:

Digital Corpora

This is arguably the most directly applicable and widely used source for sample data in digital forensics education.  Simson Garfinkel and his collaborators have developed several realistic corpora for digital forensics education and research, available at http://digitalcorpora.org. 

These include “scenarios,” which represent fictional but realistic events.  For example, UNC SILS frequently uses the M57-Patents Scenario for classes and a variety of continuing education offerings, including conference workshops and the Digital Archives Specialist (DAS) digital forensics courses offered through the Society of American Archivists.  The full hard drive images are of a manageable size for longer assignments and exercises that require a drive with a full operating system; the USB flash drive images are smaller and well-suited for short workshops, class exercises and demonstrations.

CD-ROM and Floppy Disk Library – Indiana University

Online collection of "nearly 5,000 CD-ROMs, DVDs, and floppy disks distributed by the GPO under the Federal Depository Library Program (FDLP). These tangible products have been received through the FDLP since the 1980s and consist of millions of individual files containing fundamental data on economics, the environment, population, and life and physical sciences."

https://webapp1.dlib.indiana.edu/virtual_disk_library/

Each disk image has an associated catalog record.  For use in classes, one approach is to share a subset of the images directly with students, so they don’t immediately encounter all of the descriptive metadata from the site.

Preservation and Access Virtual Education Laboratory (PAVEL)

The Preservation and Access Virtual Education Lab (PAVEL) project at the University of Michigan School of Information (2010-2012), funded by the National Endowment for the Humanities, developed a “virtual education laboratory featuring digital access and preservation tools.”  This includes four data sets:

Many thanks to David Wallace and Beth Yakel from the PAVEL team for allowing us to redistribute the datasets.

PAVEL data sets have the advantage of being selected for use in archival education, and they contain important supplementary information (e.g. organizational charts) for understanding their contexts of creation.  However, they do not allow for the replication of many forensic tasks.  For example, all of the email messages are stored as separate ASCII text files, rather than being embedded in a disk image or within a wrapper format such as PST, a scenario likely to be encountered during acquisition of a new collection.

Corpora Designed for Targeted System Testing

Brian Carrier’s valuable, but highly targeted Digital Forensics Tool Testing Images:

Computer Forensic Reference Data Sets (CFReDS) from the National Institute for Standards and Technology (NIST):

Open Preservation Foundation

The Open Preservation Foundation maintains a site that includes “Datasets, preservation and curation Issues with those Datasets, and Solutions to those Issues.”  The experiences of solving specific Issues are written up on Solution pages, which then link to pages in the OPF Tool Registry.  In many cases, this leads to “actual code that can be downloaded and re-used.”

National Institute of Standards and Technology

The National Institute of Standards and Technology creates and maintains data sets including disk images for testing forensic tools. Some of these are deprecated over time; the current list can be found at:

Library of Congress Labs Web Archive Data Sets

The 1000 .gov PDF Datasets provided by LC are a particularly useful dataset for exploring BitCurator functionality and can be applied to educational contexts.

Scripts Library

This is a list of scripts created by BitCurator users that can assist with digital forensics tasks and activities. If you have a script you would like added to the library, please contact us or post in our Google Group. 

Script to parse and process Guymager logs in bulk

This program is intended to help to identify floppy disk formats. It takes a folder of KryoFlux stream files as input

Relevant Resource: Floppy Disk Format Identifier Tool (blog post)

Utility to parse hfsutils output and produce DFXML for HFS-formatted disk images

Relevant Resource: Advanced Topics - Scripting in BitCurator (webinar, slides)

What are the key metadata points to extract from complex, multi-level born-digital records and later represent in EAD? This shell script extracts information from directories of digital content for use in the Descriptive Identification element wrapper.

Relevant Resource: Advanced Topics - Scripting in BitCurator (webinar, slides)

Sample scripts for acquiring and processing floppy disk images

Relevant Resource: Digital Archaeology and/or Forensics: Working with Floppy Disks from the 1980s (paper)

Scripts (bash, Python) developed to automate processes

Generate file system event timelines from a collection of disk images automatically, and collects the individual timelines into a master CSV file

Relevant Resource: Aggregating Temporal Forensic Data Across Archival Digital Media (paper)


If you would like to provide feedback for this page, please follow this link to the BitCurator Wiki Google Form for the Resources section.
  • No labels