Sync Stories: Northeastern University’s Video Crew Alleviates Storage Nightmare

BitTorrent Sync

Sync Stories is a column dedicated to our users. Each week, we showcase a different use case for BitTorrent Sync and the personal stories behind it.

In this week’s edition: Northeastern University’s digital media producer Frank Hegyi shares how he and his video production team have used Sync to create a fast and efficient shared storage system.

From Frank: 

About 3 years ago, Northeastern University’s fundraising department decided they were spending enough on freelance video producers to justify bringing it in-house. That’s how my team was born. We’re a small team of 3, shooting branding videos, web content, video exhibitions for events, etc.

Ban The Hard Drive

When I first joined the team, our media management strategy wasn’t great. AKA there was no strategy. We just passed a bunch of hard drives around the office, which was fine for a team of 2, but with 3 it was too unwieldy. We hit our breaking point when our main editor went on vacation for 2 weeks, and I thought we lost an entire project because I was looking on the wrong hard drives. We needed a system where everything was in one place and everyone had access. It was time to invest in a shared storage solution.

The Pain of Storage Research

Anyone who has ever researched shared storage for video editing will understand the torture I endured for about 4 months. I traversed the labyrinth of network jargon the best I could. I learned the difference between NAS and SAN. I suffered through remote powerpoint presentations with sales reps because vendors refuse to advertise prices on their websites.

Why couldn’t I just buy a NAS from Best Buy? Here’s why:

  • Video files are huge. For example, this timelapse is comprised of ~10,000 raw photos. Just scrubbing through something like that requires a ton of throughput.
  • Network latency screws up the whole process, so consumer level NAS’s won’t work. The system has to be optimized for video editing.

I kicked around the idea of using a consumer NAS as a central repository and then offloading to direct attached storage while working on it, but that creates multiple versions of things, which defeats the whole purpose. The only option was traditional video shared storage.

Then There’s The Issue of Money…

I got quotes from pretty much every player in the game. In the 24-32TB range, nothing was under $15,000. It was $20-25K if you wanted a backup. My budget was $10,000, so it was close, but I didn’t want to spend ¾ of our annual budget and not even have a backup.

That wasn’t the only problem. We’re in a normal office environment, and most of these systems are rack mount units. I wasn’t going to clear space in my office for a loud-ass server rack that would buzz in my ear for the rest of my life.

Then Edward Snowden happened. I was looking for ways to replace my personal dropbox account when I stumbled across BitTorrent Sync. The next morning, my sweet direct-attached-faux-video-server was born.

Here’s how it works:

  • I set up a dedicated wired GbE LAN between the 3 workstations. Nothing fancy, just some cat6 and a TP-Link AC1750.
  • 3 direct attached raid towers, one for each workstation. In our case, 32TB Pegasus R8’s (one of which we already had).
  • Sync the 3 raid towers using BitTorrent Sync

Here are two of the synced raid towers, one attached to my laptop, and the other attached to our mac pro. As you can see in the screenshot, when I added the folder in BitTorrent Sync I pointed it to the entire volume, named ADV_VideoShare. So now anything I do on my raid is automatically cloned to the 2 other identical raids. It’s like having a 32TB central server with thunderbolt 2 connectivity and 2 automatic backups! All for less than it would have cost for a NAS/SAN with no backup.

BitTorrent Sync NEU

BitTorrent Sync NEU

BitTorrent Sync NEU

Benefits

It works really well. Here are the benefits over a traditional video NAS/SAN:

  1. It’ll work with storage that you already have. We bought the Pegasus raids, but if you’ve already got the hard drives, this solution is essentially free.
  2. We’ve got automatic triple copies of everything. This is huge.
  3. Each workstation is direct attached to their own hard drive, so no need to worry about network latency or throughput.
  4. It can work over the internet. The initial footage sync will take a long time, but after that, the only thing changing is the project file.

Sync Speeds

Sync is super fast. If the project is comprised of large video files we saturate the GbE. If the project is a collection of still images (like the timelapse), we’ll get 30-50MB/s.

Things to Consider

Here is the big thing video editors need to watch out for: BitTorrent Sync doesn’t communicate with your editing system. Some shared storage systems know when a project is open and prevent any conflicting changes from being made. With Sync, two workstations could potentially open the same project and overwrite each other’s changes.

We established a workflow that protects against the conflicting changes issue, but I can only speak to our experience with Adobe Premiere. Your mileage will vary with different workflows, so make sure to test everything thoroughly!

That pretty much sums it up. Hit me on twitter if you’ve got any thoughts, questions, or concerns @frank_hegyi.