Permanent Digital Storage: The hard work of preserving data
You aim the camera at your newborn child, capturing a once-in-a-lifetime moment that you hope to never forget. Looking at the image calls back the feeling of being there, holding the baby in your arms for the first time. Among all of the random pictures you snap in the course of your daily life, this is the one it matters most to you to save—ten years, fifty, a hundred. Even though the photograph freezes the memory in stasis, the picture itself is vulnerable to time’s decay—be it in a yellowing physical print or an inaccessible digital copy on an obsolete format.
Preserving information requires infinitely more work than destroying it. Data must be continually ferried from one form of storage to another to remain viable and safe from the many disasters which may befall it. In our last post, we talked about digital obsolescence and the difficulties of predicting which formats will still be readable in the near and distant future. Today we want to look at strategies that will help you approach truly permanent data storage for your personal or business files:
- Which storage formats last the longest?
- Which formats are most likely to still be readable decades from now?
- What kind of files should you be archiving?
Long-term data storage considerations
You’ve probably heard of the old engineering maxim that you can have anything fast, good or cheap, but not all three at once. With data storage, that trio might be modified to accessible, durable and cheap.
Accessibility is whether your data can be readily retrieved/restored when you need it. Does the format require specialized hardware to view? Are there technical limitations to how quickly the data can be written to or parsed (ex. tape)? Is your cloud provider’s upload speed sufficient to allow you to get what you need in a timely fashion?
Durability refers to how resistant a format is to physical shocks and damage, and also how long it is able to hold onto the information is contains without corruption.
Cheapness, you can probably put together yourself, though it includes both initial costs and maintenance.
The closest thing we have to a truly permanent solution are optical disks. Now, we’re not talking about standard CDs and DVDs here. These discs can literally rot thanks to factors like aluminum oxidization, galvanic reactions between the various layers and coating and ultraviolet light damage. Under normal use scenarios, standard discs will last a mere five to twenty years. There are, however, optical disks designed to survive much longer.
Write-once BD-R HTL (High To Low) can last for 100 to 150 years given a relatively mild environment—i.e., not on your dashboard in Phoenix. Milleniatta’s M-Disc BD-R and DVD+R write-once discs use an even more stable data layer that is rated for 10,000 years. Only its polycarbonate outer layers reduce that to a mere 1,000 years. Note that this is all theoretical, but the testing MOs were rigorous and performed by the government of France (BD-R), and the Navy for the Department of Defense (M-Disc DVD).
Perhaps your child’s face will be the one the space archaeologists find that teaches them what humans used to look like—or your company’s quarterly earnings report will be added to their stockpile of “BAFFLING EARTHLING JUNK.”
BD-Rs can house 25GB, 50GB, or 100GB of data, which is more than enough for most personal record-keeping, but quickly becomes prohibitively expensive when we start talking terabytes of information. These discs also write slowly compared to USB 3.0 transfer speeds, meaning they’re basically good for deep, secure archiving and little else.
Hard drives whip optical disks in terms of storage capacity, write-speed and cost at scale. But what you gain in accessibility and cheapness, you lose in durability. HDs age like rabbits and tend to die just as suddenly. Most drives tend to fail by about five years of regular use. On the shelf, you might get ten to twenty years before the magnetism they rely on to properly function disperses, rendering your information unrecoverable.
You can delay the inevitable by regularly rewriting your drives on an annual basis, which refreshes these magnetic reserves. In fact, the efficacy of hard drives as a long term solution has everything to do with how assiduously you take care of the data they contain. Most authorities recommend having at least three copies of all of your key data saved to separate locations to reduce the cataclysmic effects of a single storage failure.
Data must continually moved from one drive to the next over the years to ensure that it remains on a “healthy” young device, and that said device is still compatible with modern tech.
We’ve covered the issues with tape storage before, but in case you’re not in the mood for a deep-dive, here’s a sample of the challenges that come with the format:
- extremely finicky (improper handling and adverse environmental conditions are death to tape, even in comparison to HDs)
- most tape formats are linear-style drives, meaning that incremental daily or weekly backups aren’t really feasible
- replication is difficult
With that said, at about 20 years properly-stored tape lasts at as long as (if not a shade longer than) a shelved hard drive, and newer formats can backup vast amounts of data fairly quickly.
Tape is also vulnerable to format obsolescence, as there are many different formats out there and tape’s market share as a whole is diminishing. That increases the odds your format’s manufacturer may go out of business between the time you create your archive and the time you next need to access it.
In the end, tape requires the same data stewardship as hard drives do if you want to ensure your information remains viable in the long term.
Most companies today with high-volume data storage requirements house their information offsite, usually via the ever-popular cloud. Big-time operators like Apple and Microsoft charge very low rates for infrequently-accessed archival storage, and outsourcing means freeing up office space and ducking hardware costs. But cloud storage doesn’t solve the permanent data storage question—your data is still housed on a physical drive or cassette, making it vulnerable to accident, malice and the passing of time.
In effect, what you’re really doing is making someone else responsible for the work of ferrying your archives from one device to another, and hoping these experts know what they’re doing (and that you continue to have the coin to maintain their services).
What to Archive?
Higher storage volumes increase your data’s vulnerability over time, as well as your costs. Generally, you want to avoid archiving things you don’t need, like:
- Redundant files
- Unfinished documents
- Trivia (think company newsletters, office memos etc.)
File archiving solutions like ShareArchiver help to make this pruning process easier. Check out our blog on how to decide which business data to archive for some helpful tips on that front.