A lot of people ask me, “Do I really need shared storage?” They think it’s expensive, overly complicated, and not practical for what they're doing. A couple of years ago I’d have agreed with them. Who’s ever walked into a server room, looked at the giant mess of drives and cables and thought “this is what I want, this will make my movie better”? Shared storage is changing though, and quickly becoming the only practical way to deal with the unbelievable amount of content we’re all generating. So without further ado, here are my top 10 reasons you and your team should be using shared storage for your next project:
No more sneakernet drive pyramid - I can’t tell you how many places I’ve worked where in a closet somewhere lives an ever-growing pyramid of external hard drives managed by a lone, overworked Assistant Editor desperately trying to keep an ever-growing number of staff editors and freelancers working with the same media.
It always starts innocently enough with a small group of editors who each have their own G-RAID or Pegasus. Then business picks up, more machines are added, GFX and sound are brought in, and suddenly there are cloned drives everywhere. Of course all of that data needs to be archived to other hard drives to make room for new projects and then the process starts over again. As the business grows and expands, this leads to an amazingly large, inefficient, and expensive hard drive pyramid (or modern art sculpture) that everyone likes to joke about, but is slowly crushing the souls of your editors and assistants.
Throw 4K into this boondoggle and that pyramid has just quadrupled in size and the game of “What drive is that project on?” gets even more fun. As long as you’re managing data across single user hard drives this will remain a problem, no matter how meticulous your sharpie and gaffer tape system is. If you see a pyramid growing in your office it might be time to think about shared storage.
Video conform becomes a breeze - If everyone’s working from the same storage, passing XML’s and moving from system to system as you build a larger project with lots of team members becomes WAY more efficient. Also, if everyone is working with shared GFX and SFX libraries on the server, you’re saving tons of time and not constantly running into the issue of smaller files missing from drive to drive. You no longer have to worry about media consolidation and making sure everything is in the same directory since everyone is already working from the same directory.
Connect your workflow – Imagine a world where your editor working in FCPX, Premiere, or Avid can easily send an XML or AAF to Resolve to conform to your original media, and your colorist can send out DPX or Open EXR image sequences for your VFX editors to work with in Nuke and After Effects all from the same volume from the full res files. That’s what great shared storage allows for. Think about how much time that saves compared to cloning drives and doing a reconnect every time you get a new XML or project file.
Once you’ve hit picture lock it becomes really easy for your sound department to get involved in Pro Tools and Logic and have easy access to all of your original sound files for your AAF. For anyone who has lived in a sneakernet world, not having to constantly relink and manage media is all that really needs to be said in the case for switching to shared storage.
You can take more risks – How would you work if you knew your storage could handle anything you threw at it? Would you shoot everything RAW, would you shoot at a higher frame rate, would you add more angles, would you jump into VR? If your storage worked with Mac, Windows & Linux simultaneously, allowed your VFX artists to work in 4K and could connect up more users on the fly would that expand your horizons a bit? Not only that but you wouldn't have to worry about what operating system you hard drive is formatted for. The bottom line is that it leads to less thinking about tech and more thinking about story.
It’s easier than you think – I connect to my storage with an app. With the click of a few buttons, I’m up and running, my machine’s been tuned to maximize bandwidth and I’m working. I can even turn auto-mount on when I need it.
Watch folders - Want to set up an FTP? Want to automatically upload dailies or hook into a DAM, etc...? This all gets WAY easier on shared storage, as all your editors can access the same watch folders.
Your data is safer - In the sneakernet world, unless you're in a RAID array, if your drive fails there’s a real good chance you data is gone forever. Even if it can be recovered that’s a long and expensive proposition that’ll cause you to miss your deadline anyway. With good shared storage however, there are a number of clever ways that parity can be enabled to permit drive failure with zero data loss, and zero downtime. How about an email when a drive fails, hot swappable spares, or even volumes that can be popped into another box and have all your work reappear? That's how we work in our lab, and you can do the same.
A better way to archive - You may or may not have a system for archiving in place today, and I could write a whole other article on the topic but the takeaway is this: if all of your working media and projects are in the same place, archiving it is a lot easier. The system we designed can even archive in a way similar to Apple’s Time Machine, backing up at regular intervals and letting you roll back to previous iterations of your projects and files which has saved my ass on more than one occasion.
Not to get all Inception on you, but if your working shared storage archives to more shared storage it means that you can access your archival footage as quickly and easily as your working footage. Let that sink in for a second. As for LTO, it works more efficiently than ever whether you’ve got a rack mounted device connected to your server or a desktop model connected to a client.
In the end it’s cheaper - The second you hit 4 editors, some shared storage actually becomes less expensive than the alternative. Today’s leading 64TB RAID comes in at around $8,000 which will set your team back just over $30K. They’ll more than likely want to work striped for maximum bandwidth, which means no parity or redundancy, an unfortunate but often necessary tradeoff. However for slightly less money a ShareStation Indie would give that same team of 4 editors 70 TB’s of redundant shared storage with all the perks I’ve outlined here, and none of sneakernet nightmares. Plus they’ll have room to grow in both capacity and users at any time. Throw in the amount of time lost during file copies, transcodes, and syncing and the cost/benefit of shared storage becomes even more clear.
Get your flow back – Anyone reading this who’s ever worked on a standard XSAN setup with a couple MacMini MDC's will think I’m lying, but shared storage done right can actually perform faster and with less latency than direct attached storage. Specs are one thing, but at the end of the day we are creative people and these devices are our tools. Film was a very tactile medium and your manipulations in the cut were immediate. That connection to the image is hard to replicate today working in 4K or 6K with the maddening start, wait, playback that happens thousands of times per project. The system I work on now honestly feels like I’m working from an internal SSD. I press the space bar and it plays immediately; no beachballs, no lag, and no dropped frames. Think I’m working in a proxy? I’m not. It’s that freaking fast and responsive. Am I saying shared storage will help you win you an Oscar? No, but you’ve definitely got a better shot when you’re not screaming at your timeline all day.
I hope that helps answer some questions about shared storage. I used to be a sneakernet guy, but now my workflow has totally changed and allowed me to take everything we're doing at We Make Movies to a whole new level. Stay tuned for more on my upcoming pilot, Off The Grid in the coming months.
- Sam Mestman