Managing and Preserving Your Born-Digital Files

Over the past couple of months, the American Archive project team has been working with stations to contribute born-digital or previously digitized files to the American Archive. That is, the types of material we are now gathering for submission into the archive already exist on file-based media, such as Quicktime video files or .wav audio files. During the Content Inventory project, stations created inventory records for the tangible, physical media. As we have been reaching out to stations regarding born-digital, new questions arise:

  • How do we record information, or metadata, about the digital files?
  • What different types of information need to be captured?
  • What are the best practices for creating, naming, organizing, and storing digital files?

Today we want to share with you some general best practices for managing your digital media files. These suggestions may already be common practice for some stations, and if that’s the case, you’re certainly ahead of the game! Best practices for creating and managing digital media are constantly evolving. If you have any suggestions to add to this list, please share them by commenting on this post.

Metadata

Give each file a Unique Identifier, and if possible, add the unique identifier as an addendum to the file name, e.g:

Original file name: basicblack_140102.mov
Unique identifier that you assign the file name: wgbh_10467

New file name: wgbh_10476_basicblack_140102.mov

For camera generated digital files or Source Clips: DO NOT CHANGE OR RENAME THE  FILES.  Instead of changing the original file name, create a folder containing the individual file and name the folder using the Unique Identifier.

Record as much other information about the asset as possible by following the guidelines set forth in the Content Inventory Project. To download a sample Excel file with the basic metadata fields, click here.

Most importantly: Record where to find the asset on a hard drive by providing the full file path.

 Naming Digital Files

Folder and file names should only use:

  • A to Z (small case and capitalization are both acceptable
  • Numbers 0-9.
  • Underscores are acceptable but no other symbols such as !@#$%^&*() – +=|?,
  • A period is acceptable only before an application extension (.doc, .jpeg, etc)
  • No spaces within the folder and file name

Try to record essential information such as format type, date of creation and modification in the file names or through the folder structure.

File Organization

Think carefully about how best to organize your digital files in folders. Consider the best hierarchy for your digital files and whether a deep or shallow hierarchy is preferable. Files can be organized in folders according to:

  • Program Title
  • Format Generation (Master, B roll, Interview, Stock Footage, etc.)
  • Date
  • Camera Model or Digital Format (P2, Quicktime, XDCAM, MP4, DVCPro etc.)
  • Unique ID

Making Back-ups

Making back-ups of your digital files is an essential component to digitally preserving your files. Regular back-ups help to protect against hardware failure, software or media faults, viruses, power failure, or human error. Have multiple copies stored in different locations — on multiple hard drives, on your server, etc.

Data Storage

Like physical tapes, digital files are constantly at risk of being lost forever. Digital files really are only just a series of ones and zeroes — bytes of data — that must be rendered and played back. Like physical formats, digital formats can quickly become obsolete. A few suggestions:

  • create digital files using standardized formats for long-term readability
  • every two to five years, copy or migrate digital files to new hard drives, since the media storing the files are subject to degradation
  • create a schedule to regularly check the files to make sure they are still playable
  • store your digital files on two different forms of storage, such as a hard drive and a server
  • organize and clearly label your hard drives or other storage mediums
  • ensure that the areas where you are storing your hard drives and physical tapes are at a low risk of fire, flood, or other type of disaster

If you have any questions, suggestions, or resources you’d like to share about best practices for preserving born-digital media, please contribute your thoughts by commenting!

**Many thanks to WGBH Digital Archive Manager Leah Weisse for her contributions to this blog post!

This post was written by Casey E. Davis, Project Manager for the AAPB at WGBH.

Dr. Martin Luther King Memorial rally, Bascom Hill, UW-Madison, April 5, 1968

On the third Monday of every January, America honors one of the great martyrs to the cause of civil rights, Dr. Martin Luther King, Jr. To commemorate his lifetime of commitment and sacrifice, we share with you today an audio file from the American Archive of Public Broadcasting, courtesy University of Wisconsin-Madison Archive, Steenbock Library and Wisconsin Public Radio.


The day following Dr. King’s assassination in Memphis, University of Wisconsin-Madison’s Chancellor William Sewell suspended classes to mourn the assassinated civil rights leader. That afternoon, on April 5, 1968, a huge gathering of students and faculty met at the campus’s Bascom Hill to participate in memorial services. The Madison Police Department called it “the largest mass demonstration ever held in Madison,” with an estimated total of 15,000 participants.

“It has been a long time since this whole university community has felt a loss so deeply. This man more than any other challenged the forces of hatred and bigotry. This man more than any other gave us a hope that someday this nation might rise above racism and intolerance,” spoke Chancellor William Sewell. “It would be a betrayal of America if we did not learn from his death and if we did not now pledge ourselves to carry forward the spirit and the promise of his faith.”

Black student leaders Sidney Glass, Kenny Erwin, Kenny Williamson and Clara Meek were among the speakers of the memorial service. A predominantly white audience listened to them talk about their grief, racism, the importance of black solidarity. After the students had spoken, the crowd marched up State Street and around the square on UW-Madison’s campus.

The next day, hundreds of students met in classrooms and held discussion forums led by black students. In response to Dr. King’s assassination, the university pledged to create a Martin Luther King scholarship fund “to work for the elimination of racism and racial misunderstanding on campus and within the community…”

Originally recorded on 1/4″ audio tape, the audio clip we have shared today was digitized as part of the initial 40,000 hours of content preserved in the American Archive collection.

Audio clip courtesy University of Wisconsin-Madison Archives, Steenbock Library; Contribution, Wisconsin Public Radio
Image courtesy University of Wisconsin-Madison Archives, Steenbock Library
Content provided from the media collection of Wisconsin Public Broadcasting, a service of the Board of Regents of the University of Wisconsin System and the Wisconsin Educational Communications Board. All rights reserved by the particular owner of content provided. For more information, please contact 1-800-422-9707
This post was written by Casey E. Davis, Project Manager for the AAPB at WGBH.

Ann Wilkens of Wisconsin Public Television reports on the American Archive

Hello friends and colleagues!

Wisconsin Public Television is proud to have the opportunity to participate in the American Archive of Public Broadcasting. Now that hundreds of hours of our content have been made digitally accessible, we are able to actually view our old programs, enrich the metadata, and eventually share our content with the public. Here are a few observations:

• From the beginning our programming has been innovative, educational, and reflective.

• Our work with the AAPB has given us a jump start on our in-house digitizing. The hours digitized by Crawford represent about 3 percent of our collection. With some urgency we’ve been digitizing analog formats to uncompressed Quicktime files with priority on older unique items on 3/4 inch videotape and 1 inch videotape. 1 inch videotape is extremely reliable. We’ve had a handful of problems playing back 3/4 inch videotape but that too for the most part has been successful. Whew! We’ve got breathing room with our film holdings because of their stability under proper storage conditions. On the other hand, that format is the least accessible.

• Much of the programming from the mid-1950s through the mid-1970s has been lost. Viewing what we do have from that time period is hindered by the age and robustness of the format and playback equipment. So, when we were finally able to watch programs through the Archival Management System (AMS) it was like opening a treasure chest. Every program offers a glimpse at our past. And they are just plain fun to watch.

• We’ve discovered that programs we thought we produced were produced by others. As I’m sure you know, old tapes usually have very little metadata, perhaps a title and date (created? Aired? Dubbed?) on the label and that’s it. There was really no way to be certain of its provenance. After watching the digitized content through the AMS we discovered programs that were produced by other PBS or commercial stations. We will return those to you as we are able!

• What should we do with the analog tapes now that they are digitized? Good question. The usual archival practice is to always keep the original. In this case the original and the technology are going to be obsolete in the near future and the cost of digitizing analog again (if better technology comes along) is time consuming, expensive, and what would we gain visually by doing it? We need to make sure we responsibly manage the digital collections into the future. (This is the part where I wish I was a manuscript archivist.) We are at this time retaining the originals.

• We’ve elected to receive the proxy and mezzanine files from Crawford. Our aim is to incorporate the proxies in our online database (under development), use the mezzanine format [mpeg2 mxf] in our production workflow, and archive both formats to LTO. As for the preservation files (JPEG2K), we will leave those in the care of the Library of Congress. Their mandate is to care for them for the life of the Republic plus 500 years, by which time I will almost be ready to retire.

Ann Wilkens has been the media archivist at Wisconsin Public Television since 2004.

Ann Wilkens, Media Archivist at Wisconsin Public Television
Ann Wilkens, Media Archivist at Wisconsin Public Television

Onward toward the 5,000 hour goal

I hope that you all enjoyed wonderful holidays with family and friends, and my best wishes to you and yours for a happy and prosperous New Year 2014.

Now that the holidays are over, we are excited to start moving forward with the addition of 5,000 more hours of born-digital or previously digitized materials to the American Archive. Several stations have committed to contributing material, and we are already more than halfway to our goal!

If your station participated in the American Archive Pilot Project, managed by Oregon Public Broadcasting in 2009, we would love to include the materials digitized in that project in our selection. Please email me if you would like to contribute those files and/or other digital files that you would like preserved with the American Archive collection.

If your station did not participate in the Pilot Project, but you do have born-digital or previously digitized material that you would like to contribute, please send me an email letting me know an approximate number of hours of audio/video files you want to submit and whether the files were included in your Content Inventory.

The process will be very simple.

For materials included in the Content Inventory:

If the materials you want to submit were included in your Content Inventory, then you can send me a list of GUIDs (American Archive unique identifiers, which are found with in record in the AMS), or a list of local ID’s you assigned to your assets in the inventory. I can easily batch nominate the materials for you. You could also nominate the materials yourself (it’s really easy, and I can show you how).

For materials not included in the Content Inventory:

If the files you want to contribute were not included in your Content Inventory, then we would just need a CSV file that contains the same information about each asset that was asked for during the Content Inventory Project:

  • Identifier
  • Identifier Source
  • Series Title
  • Program Title
  • Genre
  • Source of Genre
  • Unique ID
  • Unique ID Source
  • Format (digital)
  • Generation
  • Duration
  • Location

[Click here to download a sample CSV file.]

Upon receipt of the CSV (or PBCore XML), we could batch ingest the metadata into the AMS and nominate the records for you.

For materials digitized in the Pilot Project:

If you are interested in contributing files digitized in the Pilot Project, then good news — we already have the metadata and can ingest it into the system for you! In this case, all you would need to do is put the files on the drive and ship it to Crawford Media Services.

What happens after the assets are nominated:

Once the materials have been nominated in the AMS, we would ship you a hard drive on which you would put the files and ship them to Crawford. Crawford would keep the file you sent as the preservation file and would create a proxy, or access file. The preservation files would be sent to the Library of Congress for long-term preservation as part of the American Archive Collection, and the access files would stream through the AMS. Rights permitting, the files would be made available on the American Archive website after it goes live by March 2015.

Please email me at casey_davis [at] wgbh [dot] org if you are interested or if you have any questions.

This post was written by Casey E. Davis, Project Manager for the AAPB at WGBH.

Happy New Year from the American Archive!

Greetings and Happy New Year! In celebration of the new year, today we are sharing a highlight from the American Archive collection, contributed by WGBH. This six-minute audio clip was taken from an episode of the Children’s Circle daily radio program, which in 1951 was launched and hosted by Nancy Harper. This episode aired on New Year’s Eve in 1952, during which Nancy held a birthday party for the new year.

Nancy Harper served as the producer and writer of Children’s Circle and eventually moved on to become an editor for the textbook companies McGraw-Hill and MacMillan. She later became associate editor of Curator, a journal published by the American Museum of Natural History. She died in 2007 at the age of 91. We appreciate her contributions to public media and are excited to share this audio clip with you today.

Originally recorded on 1/4 inch audio magnetic tape, a format typically used from the 1950s through the 1980s, the program was digitized as part of the first 40,000 hours of American Archive material. It is now being preserved for future generations at the Library of Congress.

This post was written by Casey E. Davis, Project Manager for the AAPB at WGBH.

Opportunity for Stations: BAVC’s Preservation Access Program

We have heard from many stations that are interested in seeking more funding for the digitization of their audio-visual materials. Today, I was made aware of a wonderful opportunity offered by the Bay Area Video Coalition (BAVC), a non-profit organization committed to inspiring social change by empowering media makers to tell diverse stories through art, education and technology. The BAVC Preservation Access Program partners with libraries, museums, arts organizations and artists to preserve and digitize precious works of media art and other cultural artifacts. You can learn more about BAVC at https://www.bavc.org/preservation.

Through generous funding from the National Endowment for the Arts, BAVC is offering tools for collection assessment, preservation planning support, and deeply discounted preservation transfer services through an application process.  Qualifying artists, arts organizations, and collections with arts-related content can receive a 30%- 70% subsidy on all services.

1st Round participants include Video Data Bank, VideoFreex, New Orleans Video Access Center, Media Burn Archive, The Poetry Center, Chicago Film Archives, San Francisco Media Archive, Dance Theater of Harlem, Wisconsin Center for Film & Theater, and artists Janis Lipzin, June Watanabe, Wendy Rogers, Nancy Karp, and Alfred Young.

I encourage you to learn about this opportunity submit your applications. The application for the program is due January 15, 2014, and it can be found here: https://www.bavc.org/PresAccess-application

How to view your digitized materials in the AMS

Today I wanted to share with you the step-by-step procedures for viewing your digitized materials in the Archival Management System (AMS). Visit ams.avpreserve.com to log-in. If you are a participating station and do not know your log-in credentials, email me and I can set you up with a login ID and password.

Once you have logged in to the system, you should see the dashboard. This is where you can track the progress of your digitization. Click on the “Records” tab at the top right of the page.

Image

Now you should see a table listing of all of the records you contributed during the Content Inventory Project.

Image

If Crawford Media Services (the digitization vendor) has already started on your station’s digitization, you should see a check box on the bottom of the left side bar that says “Digitized.” Click the small box next to “Digitized,” and you can filter your records to only include those that have a digitized media file attached to the record.

If you do not see the “Digitized” check box, then Crawford has not begun to digitize your materials yet. You can, however, view the records for the assets you nominated for digitization by clicking “Nomination Status” and selecting “Nominated/1st Priority.”

To view an individual record in the AMS, click the blue hyperlinked American Archive GUID, which is a unique identifier assigned to every record. If you are clicking on a record for an asset that has already been digitized, you will see a video/audio player appear above the metadata in the detailed view. And from here, you should be able to view or listen to the media.

Image

Below the video player, click “Open Proxy File.” Here, you will be able to easily pause and navigate forward and backward in the file.

AMS-dream

It may take a minute to load, and Google Chrome is the best browser to use when viewing the media. You can also download the proxy file to your computer by right-clicking within the iframe and selecting “Save As.”

If you have any questions or need any assistance navigating the AMS, please do not hesitate to email me at casey_davis [at] wgbh [dot] org.

This post was written by Casey E. Davis, Project Manager for the AAPB at WGBH.

Survey: What do you want in an export?

Hello stations! We’re working with our Archival Management System (AMS) vendor to determine metadata export functionality in the AMS. Since the Content Inventory Project, our team and our vendor have done a lot of data clean-up and normalization. We also plan to continue adding more description to the digitized materials (with permission from stations).

We want your opinion today on what type of metadata export would be useful for you. Please take a moment to complete this three question survey: https://www.surveymonkey.com/s/GSRQSBT

We sincerely appreciate your feedback!

News from the Project Team (12/9/13)

Greetings everyone!  I’m Casey Davis, the Project Manager for the American Archive. We are only one month in and are thrilled to have heard from so many stations and the general public expressing your support and interest in the initiative. We’ve been asked a lot of questions, and today I wanted to give a few updates on where we stand as we move forward in Phase One of the initiative. We have already been working very closely with the Library of Congress to begin planning for the next two years and are very excited about working toward the goals we have set.

wbez-chicago
Materials found and inventoried by WBEZ — Chicago, IL

First, let me back up a little: In 2012, we completed the Content Inventory Project, during which staff members at 120 public media stations across the United States went into their closets, browsed through their shelves, and dug into decades-old boxes to find out exactly what stuff they had been collecting for the past 50+ years. Well, that treasure hunt resulted in 2.5 MILLION inventory records, of which 40,000 hours of content are now slated for digitization and long-term preservation at the Library of Congress. And just to ease any doubts out there about the materials’ safekeeping — the Library has committed to preserving the materials for the life of the Republic plus 500 years. So, that being said, rest assured that they are safe in the hands of our friends at the LC.

iowa public television - johnston
The vault at Iowa Public Television — Johnston, Iowa
kued-salt lake
Film canisters at KUED — Salt Lake City, Utah

Stations that participated in the inventory and/or digitization projects should have access to the Archival Management System (AMS). This is where you can view all of your records contributed during the inventory project, see streaming files of your digitized materials, and keep up to date on the progress of your digitization. The AMS URL is ams.avpreserve.com. If you do not have access to the AMS, email me and I’ll get you set up in the system.

One of the first goals for the next two years is the addition of 5,000 more hours of born-digital or previously digitized materials to the archive. We’d love to include in this selection the materials that were digitized during the Pilot Project in 2009 and any other materials that stations wants to contribute. We are currently working on finalizing the documentation for this process, but feel free to go ahead and let me know if your station is interested, and we can begin discussing the next steps.

vptv
Inventoried 2″ and 1″ video tape from Vermont Public Television — Colchester, VT

Until the final American Archive website is developed, the project team will post updates and resources on this blog on a regular basis. To receive updates via email, subscribe to the blog and sign up for our American Archive email list! To learn more about our goals for the next two years and beyond, view a slide show that we presented at the annual Association of Moving Image Archivists (AMIA) conference.

If you have any questions about the project or have any suggestions on how we can improve this blog as a resource for you, please do not hesitate to contact me at Casey_Davis [at] wgbh [dot] org.

From the American Archive: President Nelson Mandela

Today, South Africa and the rest of the world mourn the loss of Nelson Mandela, the revered South African president and anti-apartheid revolutionary who spent 27 years in prison. Serving as South Africa’s first black president, he ended apartheid, the government platform, which for nearly five decades had enforced racial segregation, denying non-whites any economic or political power. During his five-year presidency, Nelson Mandela led his country to democracy, tackling racism, poverty, and inequality, and fostering reconciliation.

Upon hearing news of his death yesterday, we contacted our American Archive partnering station WHUT, located on the campus of Howard University in Washington, D.C. Today they have given us permission to share with you a video from the American Archive, featuring President Nelson Mandela at Howard University in October of 1994, the day on which he received an honorary doctorate degree from the university.

“Our cause became your cause, and so shall it remain, for us to work together to improve the quality of life of especially black people, and other disadvantaged communities — in South Africa, in Africa, in the United States, and other parts of the world.” –South Africa President Nelson Mandela, upon receiving an honorary doctorate at Howard University, October 7, 1994

This post was written by Casey E. Davis, Project Manager for the AAPB at WGBH.