The failure of the (hard drive) system

offline_blown.jpg

Late last week, upon concluding some final tweaks to the visuals and soundtrack of episode one, I set up After Effects to render a final uncompressed export. To date, all we’d exported was the compressed Flash video file currently posted on the homepage; because we’d worked so hard to post episode one on our targeted date (Independence Day), we still had some tweaks we wanted to make before outputting the final files. This was to be our full-quality export used to generate other file formats (the iTunes podcast we’re working on, for example), and was to be the file we’d always have in the archives in case of future hard drive failure.

I hit “Render,” ready to finally be done with episode one. Then, five seconds later, in the middle of the screen, a little dialogue box with a red exclamation mark popped up. Sometime between tweaking the last few keyframes and starting the export, our external hard drive—containing all 280GB of The West Side to date—turned into a doorstop. And poof. Just like that, it was gone.

Filmmaking is often a case study in Murphy’s Law, but timing-wise, this was excruciating to the point of seeming like a surreal joke. If there was any one point at which our hard drive was going to fail, wouldn’t the worst possible time be exactly when we went to output the final export for our archives? If the drive failed one hour later, we’d be fine—at least we’d have the backup file. But now, instead, we have nothing.

This post will get a bit technical as we explain how, exactly, we just lost all of our data. As someone who considers himself well-versed in the technological side of filmmaking, I feel the need to explain how we were in such a precarious position, to avoid anyone posting a comment of the “always back up your data” nature.

Fundamentally, what it comes down to is this: as a no-budget filmmaker, what you don’t have is money. What you do have, is time. So while we’ve spent countless hours over the last year writing, casting, scheduling, and location scouting for The West Side, we haven’t spent a lot money.

Some things time can’t buy, however: a redundant storage system for all our data, for example. That takes money, and if we had it, we would’ve spent it long ago. But the fact is, we’re both in our mid-twenties—actually, Zack won’t even technically be in his mid-twenties until next month—and we live in New York City, which you might have heard is kind of expensive to live in. So it’s a lack of funds, not a lack of willingness to spend funds, that left us at the onset of production with a single LaCie 500GB d2 drive (Firewire 800 interface, 2X250GB SATA drives, striped as RAID 0). Months earlier, as I formatted the array for the Mac OS, I considered mirroring the drives rather than striping them, as the unit had failed in the past (after which I replaced the dead drive). For those of you who haven’t had the pleasure of dealing with these issues, “mirroring” a drive means you have two copies of every file across two drives, so if one drive fails, you still have it on the other. “Striping” is when half of each file is written onto each drive—riskier, but it gives you twice the speed and twice the space. It was the latter consideration that prompted me to stripe the drive—and, in retrospect, we would have already run out of space if I’d mirrored them. So I’m not second-guessing that decision… although that depends on how this all plays out.

Hard drive failures go with the territory of digital filmmaking—video editing demands more of a hard drive than any other task I can think of. While I rebuilt the array less than six months ago, by no means did I think we’d get through the whole season of The West Side using this sole drive. However, until the day came when we spent the money for a new, redundant storage solution, our plan was to simply use the LaCie for editing, and upon completion of each episode, copy the critical data over to a cheap external USB hard drive as a backup. If my After Effects render had completed, we would’ve had that file to copy over. Instead, here I am writing this post, about to take the drive into a data recovery service and hope they don’t charge an outrageous amount to get our files off the drive. Unfortunately, they could easily charge more than our entire budget for episode one.

To avoid this fee, we could re-capture all the footage from the DV tapes onto a new drive (our project files from Premiere Pro and After Effects are stored on the internal hard drive, and I also have FTP backups of those). But much of the post work done on The West Side—chiefly our Foley (sound-effects) work—doesn’t exist on those DV tapes. We’d have to spend a week or two just to re-finish episode one, when instead we want to be working on getting episode two out there. Also, we edited and posted episode one in Adobe’s beta/prerelease software—more on that decision in a later blog post—and so I’m not 100% sure that everything would recapture correctly now that we have the final release of CS3.

Ultimately, I’m glad we were able to finish episode one for less than a thousand dollars. As a result, we’re not in debt and we don’t have to bother you with advertising on the site. But The West Side is no longer speculative—meaning, we have assets to protect and standards to live up to now that episode one’s live—so we’re going to have to eat a little less and spend a little more money on storage. Still, it’s a lot more tempting to spend that money on lighting equipment, lenses, or props—things that actually affect the on-screen production—instead of on a safety measure. If the choice of what to spend our money on is a matter of risk and reward, though, I suppose not having to write this post would itself be the reward.

Stay tuned.

10 comments | tweet | share on facebook

10 Comments

Jump to comment form
  1. Cecil 23 Jul 2007

    That sucks but a lesson learned. I hope this doesn’t hold up ep 2 for a long time. I’m really looking forward to it.

  2. David 23 Jul 2007

    I can turn you on to a company that does data recovery in California – they were the cheapest I found.
    Also, you can try to switch the controller board on the hard drive for another one – they say 90% of hard drive failure is fixable without even looking at the platters.

  3. steve p 24 Jul 2007

    external hard drives are cheap. Buy two 500gb firewire drives. Total cost: $300 if you buy the enclosures and drives from tigerdirect and put them together yourself.

    Do daily backups as the project progresses, or at least do backups each time any significant amount of work gets completed. Just kick off the backup right before you stop for the day.

    Alternate which of the two backup drives you use each day. So you always have day-1 and day-2 backups. For the truly paranoid, take one backup drive offsite each day in case the building burns down.

    In that case, you would have only lost a single day’s editing work instead of everything.

    Your proposed raid0+1 is NOT sufficient. You MUST backup to a totally seperate piece of hardware like an external drive. Example 1: lightning fries all your drives at once. Surge protectors do not stop all lightning no matter what they claim. Example 2: a virus eats up your disks and deletes all the files. Some will actually low level format your drive making recovery impossible.

  4. Ryan Bilsborrow-Koo 25 Jul 2007

    Thanks Steve, we will definitely take your suggestion and back up everything to a separate physical drive. Zack will keep it at his place in case we have an incident of fire/theft/lightning/four horsemen at my studio.

  5. Paul 26 Jul 2007

    Sorry to hear about the loss…. but don’t despair.

    I was reading through CamcorderInfo when I read this and I thought i’d pipe up to offer some solutions.

    I’m a Data Storage Consultant and I really cringe when I hear the accronym RAID with the number 0 being used in the same sentence.

    So for the first solution to the problem (Current Data Loss)

    “David” suggested sending for recovery, it’s a great Idea, it’s not too expensive and often gets all (atleast most) of the data back – that means don’t bugger with the disks at all if you wan’t any hope of it working, so don’t format the disks, don’t turn them on in any form etc…. really, the less handeling and use of the disks the better the chances of recovery.

    Look towards someone like Kroll Ontrack or the alikes.

    If thats too expensive, you could try a home-brew approach by using applications like EasyRecovery Pro and raid re-constructor:

    http://www.ontrackdatarecovery.com/file-recovery-software/
    http://www.runtime.org/raid.htm

    I’ve had a background with Forensic data recovery previously and have heard of suprisingly good results from home-brew recoveries using these tools. (I’ve suggested them to many people who are two tight-arsed to spend the money for a pro recovery)

    And the second solution, for the future, RAID 5, buy the extra disks, I know it’s sometimes a little slower and more expensive that RAID 0, but whats it worth to you?

    2x 300GB SATA-II disks (total $100.00) in RAID 0
    – No Redundancy – 1 disk dies, it’s all dead.
    3x 300GB SATA-II disks (total $150.00) in RAID 5
    – Redundancy – 1 disk dies it’s fine! (Just replace it before another goes or your stuffed)

    And as a rule, the more spindles you add, the better the performance. (sometimes questionable, depends on other factors.

    And Backups…. Always backup, one cheap-o 500GB disk to take off site is perfect, saves alot of tears.

    Good luck.

    Paul.

  6. Ryan Bilsborrow-Koo 26 Jul 2007

    Thanks very much, Paul.

    Currently our drive is in the hands of TekServe’s data recovery folks. It being a Mac drive and all, I wasn’t having much luck finding RAID recovery software. Plus, I have no idea what I’m doing, so we’ll see what they can pull off of it.

    Do you have any suggestions for a good (and inexpensive) RAID 5 enclosure that will work on a Mac, either eSATA or FW800?

    Thanks again.

  7. Josh Oakhurst 1 Aug 2007

    Hey Ryan-

    b U m M 3 r! I had really good experience on two failed LaCie drives with a data recovery program called Data Rescue 2. That was a while back and they may have a new version but it saved my ass on two drives that were for all intensive purposes – dead.

    look it up and good luck

  8. Christopher Sharpe 12 Aug 2007

    I feel your pain. I’ve lost 3 of the lacie d2 500GB drives this year. Let us know how TekServe does with your data recovery.

  9. libila 21 Aug 2007

    i also feel your pain. I have losst 3 of the lacie d2 500gb drives this year. Sorry but i have no used of TekServe does with your data recovery. Frankly i said that i perfer the HDRC recovery centre only, because i regular client for last two year and every time i found 100% data. They have very efficient tool for recovery. What more i am saying to you you just click the website:www.hdrconline.com

  10. pat 7 Dec 2007

    La Cie drives are guaranteed to fail. I have never met anyone who has anything good to say about them.
    When I’ve been contracted to work with an editing suit with any La Cie drives I’ve refused unless they forked out cash for reliable storage (almost any brand other than La Cie).

Leave a comment




Safari hates me

About

The West Side is an urban western. Learn more... We won a Webby Award!