DocuCrunch.com » Bit rot: The new problem threatening companies’ archives

Bit rot: The new problem threatening companies’ archives

October 6, 2009 by Steve Hannaford
Posted in: Solutions, Special Report, Storage

Image2[2]

When DVDs came on the market in the mid-1990s, it wasn’t just the movie fans that were happy — IT pros thought they’d been given a great data storage medium. But are DVDs really suitable for storing information long-term?

For data managers, the existence of a standard, portable medium for archiving data was very welcome indeed. DVD was seen as safer than tapes and magnetic drives, and more universal than older optical disks. Manufacturers claimed lives of 30 to 100 years for their DVDs, certainly more permanent than anything else around. Over the past decade, millions of files were placed for safe-keeping on DVDs.

But DVDs suffer from a phenomenon called bit rot (also called bit decay, data rot, or data decay). Bit rot happens when the single bits of data get wiped out or modified. This always existed on electricity-based magnetic media, which is why programs sometimes “went bad” on hard or floppy disks. But DVDs (and CDs) have similar problems. Unlike magnetic disks, writeable DVDs store bits using a set of dyes under the surface, dyes that are “imprinted” using a laser to turn on and off specific bits.

The rate of decay can be influenced strongly by storage conditions. Humidity, temperature, light and handling can cause changes in the dyes, and therefore, DVD archives should be stored in a dark, cool, low-humidity environment with minimal handling. That’s still not the practice at many companies.

But even with the greatest care (and the best gold-plated archival DVDs), bit rot does happen. After all, it can only take a few altered bits to change a critical number or for a critical link to get altered.

The problem is that DVD technology dyes are vulnerable to a wide range of environmental changes. And what about Blu-Ray? Apparently it’s even more vulnerable, as its dyes are even closer to the surface.

Need proof? The U.S. National Archives (people who we assume know something about storing data) says: “CD/DVD experiential life expectancy is 2 to 5 years even though published life expectancies are often cited as 10 years, 25 years, or longer.”

This means that if you use DVD, you should make several backup copies, and you need to test them regularly and re-burn DVDs every few years. It’s in no way a permanent or even long-term archiving solution.

One solution you might check out comes from a new startup called Millenniata.

Millenniata writes on DVD-size media using an altered DVD writer with enhanced lasers. Instead of altering dyes, it actually writes tiny pits into the discs, below the surface. These pits are unalterable under any normal condition. The company even shows off in a video by plunging the written media into liquid nitrogen, then boiling it, then reading it good as new. (Don’t try this with your regular DVDs.)

M_WriterWhile it requires special disc media and a special drive to write them with, these disks can be read using most desktop DVD drives (the company notes that they have found some really cheap drives that don’t work very well). The “etched” bits look like regular dye bits to the reading system of standard DVD drives.

The company, which has been working with Sharp Electronics, is starting to ramp up its marketing efforts and finish testing with some major archivers (governmental and corporate). It’s also undergoing more rigorous third-party stress tests.

The list price of the disc writer is $2,500 and the list price of discs will be $15. Not cheap, but think of the cost of testing and re-burning hundreds of DVDs or the even bigger cost of losing critical data. Keep an eye on this company –- if their claims get borne out, this will revolutionize the data storage industry.

  • Share/Bookmark

DocuCrunch.com delivers the latest IT and Imaging news once a week to the inboxes of over 200,000 IT and Imaging professionals.

Click here to sign up and start your FREE subscription to DocuCrunch!

Tags: , , , , ,


19 Responses to “Bit rot: The new problem threatening companies’ archives”

  1. josephmartins Says:

    Remember Exabyte’s (acquired by Tandberg in 06 I believe) VXA tape technology? Freeze it , boil it, submerge it and it’s still readable and recoverable.

    http://www.exabyte.com/technology/tested/index.cfm

    Well it didn’t help Exabyte reach profitability and the technology never took off with tape/drive buyers (though Tandberg still sells it) Granted, this is more disaster related than bit rot, but the spirit is the same – data durability, integrity and reliability.

    Will Millenniata successfully convince optical buyers to make the switch? Only time will tell. But I’m not going to hold my breath on this one.

  2. David Says:

    I have witnessed Millenniata’s discs and I believe they are the real deal. Their ability to withstand outside pressures is incredible. Unlike Exabyte, the DVDs are forward compatible with all existing DVD readers. For companies/government with lots of data storage this is a huge plus. In addition, they have the advantage of building a carousel (or something similar) to automate the DVD retrieval process. They already have an impressive list of beta customers–so I think this company will be “the next big thing.”

  3. josephmartins Says:

    Hi David,

    I have little reason to doubt the technology. However the world is filled with interesting well-thought out, well-designed technologies that never really took off. Remember Segway? Well they still sell the vehicles but the adoption rate is far far below what Kaman projected.

    I wish the best for them, and I look forward to see how it plays out.

    But the next big thing is not the media itself. An issue I raised more than 5 years ago is far more important than media longevity and bit rot. And it is as much a problem today as it was back then.

    http://www.datamobilitygroup.com/saltworks/archives/27

    The last paragraph is the key takeaway.

  4. David W Says:

    Does anyone know if the claim that gold plated DVDs have the same problems? See this article but it sounds like these are dye-based as well?

    http://www.mam-a.com/longevity_research

  5. David Says:

    These are not dye based DVDs (Millenniata discs that is). See http://millenniata.com/products/m-arc.html.

  6. David W Says:

    Right– I guess the question I’m asking is if there is any independent research on the MAM vs Millenniata discs that people trust?

  7. David Says:

    Per the latest Millenniata newsletter, (September 16th):

    “We have been working closely for several months with the NavAir Weapons Division, a 1.1 million acre US Military Warfare testing and research facility located in China Lake, CA. NavAir has thousands of scientists and researchers working in their facilities. They want to use Millenniata’s archiving solution to convert all of their confidential military research files and studies dating back to
    1943. China Lake is currently testing our products. Their testing is the first step in our Millenniata products becoming the standard for digital data storage for the entire Department of Defense. NavAir began their military self-certification testing processes on September 8, 2009. The testing should be complete within the next few weeks. U.S. Military testing standards and approvals will far exceed any other third party testing regimens. These testing results can then be shared globally with all of our potential customers.”

    So it looks like they are coming soon.

  8. Linda A. Cross Says:

    I would like to receive further information as tests are accomplished and results available. The life of electronic storage has always been a thorn in my side and very questionable. Thank you.

  9. Mick Says:

    Is there any independent third party verification of the longevity of the discs? How do we know other than the company’s claims that this does what it says on the tin? What technical /product due diligence has anyone done?

  10. Anderson W. Says:

    As far as 3rd party/independent verifications of these discs, the movie shown at http://www.m-arcretail.com (the retail arm of Millenniata) states that the product meets or beats the ECMA standards for discs.

    Etching data in the disc as opposed to the traditional organic dye that most discs use lead me to believe that the data stored on the disc would have a much better chance of being safe and protected in terms of longevity. It looks like these discs also survive great temperature extremes with no detriment to the data quality on the disc.

  11. Jim Says:

    There is a big difference between Exabite and Millenniata. Millenniata’s discs are backward compatible and can be read on standard DVD players. Also, DVD players are in the majority of computers sold today. DVD technology will be around for years to come, because the market has a huge install base. Exabite does not have any of these advantages.

  12. josephmartins Says:

    Jim,

    It doesn’t matter if Millenniata’s discs are readable for the next 100 years.

    In my opinion, media longevity ceased to be a critical issue once durability sustainably exceeded the 10 year mark. The focus must shift to the findability, viewability and usability of the data on the discs. An intact, readable 50 year old Millenniata disc has no value if the data residing on it is rendered useless by our inability to find, view or use it. That’s where the challenge lies today.

  13. Jim Says:

    Joseph,

    Try to sell the 10 year threshold to those that have lost data in the last 5 years or less due poor quality optical or magnetic tape products (even ones that say archival quality) or viruses, power surges, mechanical or electronic failures that crash hard drives and servers.

    Regardless of the threshold it is the importance of the data that is going to determine the amount of effort expended to store it and make it accessible, whether it is 2 years or 100 years.

    In other words, an intact readable 50 year old Millenniata disc is only as valuable as the information that is stored on it. If the information is valuable, the interested party will take greater lengths keep the data in as generic form as possible and make sure that the data will be findable, viewable, and usable.

  14. Bryce Says:

    Joseph,

    you bring a good point in making data searchable – however, I would think that most archived data will be in DB format and that is why back-ups seem critical. Really, the paperless storage is still very small and so I agree that this will be a huge undertaking and need in the near future. However, that doesn’t eliminate the need to store data. Access is secondary to storage – e.g. no data means data access is futile. So, they are two eggs in a nest that both need developing. To state that the Millinniata disc is a blip on the screen is seriously an understatement. Once consumers understand that all of their digital photos/media is at risk and little Jonny won’t be in HS before his 2 year b-day photos are corrupt will drive huge market growth (no tape player required – talk about PITA searching). So while there is a bigger hole in the search/find of data, the archivability of the data comes before the searching – if that weren’t the case, so many experts wouldn’t be excited about this technology.

  15. Alistair Says:

    The China Lake report is done and you can get a copy from Millenniata. It makes the Millenniata discs look pretty good. The problem is that the discs sent to China Lake to test were cherry picked. There were discs that performed as well the Millenniata in their own trials, however, these were deliberately not sent to China Lake. This means that you can get some good quality discs for a lot less than the Millenniata discs.

  16. David Says:

    Alistair,

    Where do you get this information? You are completely making that up. Read the report, there was no “cherry picking” involved. China Lake is an independent organization that performed the most rigorous, independent testing that has been done on archival digital storage media. The entire Department of Defense of the US government has approved the Millenial disc for purchase. Unless you have cites to back up your claim of “cherry picking,” I suggest you stop spreading falsities.

    As for “good quality discs” that are a lot less than Millenniata discs, what are you talking about? All of the other discs tested did very poorly.

  17. josephmartins Says:

    Looks like I missed a few comments from December.

    Jim, there is absolutely no way to know in advance what the future holds for file formats or even the methods of storage of information. Nor can we determine decades in advance what sorts of applications will exist. Keeping information in an “as generic form as possible” is but a short term tactical band-aid that will buy a company some time but inevitably it’ll spend a fortune on ensuring outdated formats and storage methods are still usable well into the future, or migrate information into newer supported “generic” formats and improved (new form factors, new interfaces, more compact, higher density) media.

    Bryce, I am saying the information that needs to be stored, will be stored. One way or another it’ll end up stored. But I am also saying there is a point of diminishing returns on media longevity and we’ve already passed it. If anything, progress with regard to density is far more important than longevity.

    I mean seriously guys, what’s next? Media that will last 200 years? 500? Surely you understand the insanity of it? Just give us something that will reliably hold our data for a decade, perhaps two, and also provide us with the tools we’ll need to move our data to the “next big thing” when the time comes.

    And should you still wish to develop media with even greater longevity, fantastic – go for it. Just make sure it doesn’t cost more simply because it lasts longer.

  18. Bryce Says:

    Joseph,

    The problem is there is no 10 year reliable media storage. The Vatican ceased a project to digitize it’s archives because they found that after 1 year the DVD’s storing the digital data had already begun to degrade. Again, I agree that formats will change and archive standards will develop, but there will need to be some sort of backward compatibility otherwise, we better go back to microfiche and dust off the shelves for all the new film. If there isn’t any backward compatibility, searching is not the 900 ton gorrilla, data conversion is. So let’s just start saying that the M-Disc is 100% stable for 10 years. Show me a media that can do that. Further show a media that can withstand everything up to incineration and have data available.

    As for the cost, don’t you remember DVD-r’s costing $15 a pop in 1999 or 2000? To be frank, I would pay it to archive my family photos and documents. All it takes is a power surge and my HDD could be gone – even though I have 3X redundancy. It doesn’t take much.

  19. Return To Film » Blog Archive » Bit Rot… Says:

    [...] http://www.docucrunch.com/bit-rot-the-new-problem-threatening-companies-archives [...]


advertisement


Whitepapers

  • How to Select a Web Host
    November 27, 2011 by marketing

    Creating a new website?  Not sure how to choose from among all the options?  Need shared hosting, small business hosting, or VPS hosting?  Lots of email accounts? 5-star reliability rating? Fortunately, there’s information available to help. The Best Web Hosts is great resource that will help you select the best web hosting company. It features reviews, rankings, and definitions that can help make your job of selecting a new web host more effective.

  • SMART Steps Towards Workload Automation
    January 19, 2010 by Luke Marchie

    Consolidating job scheduling into a single, comprehensive workload automation solution is a critical first step to effective Workload Automation (WLA).

    Download the free whitepaper here! More…

  • Identifying and Thwarting Malicious Intrusions
    January 12, 2010 by Luke Marchie

    Identifying and Thwarting Malicious Intrusions

    The phenomenal growth in social media has opened the door for all new malicious intrusions from gangs of cyber criminals. Utilizing the trusted relationships in social networking and benefiting from immature security and content controls, hackers are seeing increased performance in their attacks.

    Download the free whitepaper here More…

  • The Security Issues with Web 2.0
    January 12, 2010 by Luke Marchie

    The collaborative benefits of Web 2.0 technologies have fueled rapid growth in online consumer markets and now are being adopted by businesses worldwide. With these technologies come new types of attack vectors.

    Download the free whitepaper here

    More…

  • Network-Critical Physical Infrastructure: Optimizing Business Value
    December 29, 2009 by Luke Marchie

    To stay competitive in today’s rapidly changing business world, companies must update the way they view the value of their investment in Network-Critical Physical Infrastructure (NCPI). No longer are simple availability and upfront costs sufficient to make adequate business decisions. Agility, or business flexibility, and low total cost of ownership have become equally important to companies that will succeed in a global, ever-changing marketplace.

    Download the free whitepaper here! More…

  • The New World of eCrime: Targeted Brand Attacks and How to Combat Them
    December 26, 2009 by Luke Marchie

    Nothing is more valuable to a business than its reputation. That is why brand attacks, which leverage a company’s valuable brand for nefarious purposes, must be battled on every possible front. Brand attacks are the new form of eCrime, and they’re being launched with new and rapidly evolving exploits, including phishing and—most recently—malware.

    Download the free whitepaper here! More…

  • DDoS: The Mother of All Cyber Threats
    December 16, 2009 by Luke Marchie

    DDoS: The Mother of All Cyber Threats

    Don’t wait until your business is targeted. A Forrester Consulting study commissioned by VeriSign revealed that nearly 75 percent of the 400 study respondents have experienced one or more DDoS attacks in the past year. Yet, most e-commerce businesses are not prepared for a large-scale DDoS attack. Could your business afford three or more hours of downtime? Avoid that revenue loss by registering for this free white paper

    Click here to download the free white paper More…

  • View more offers


    Quick Vote

    • Does your office have a color printer or copier?

      View Results

      Loading ... Loading ...

  • advertisement