Behind the Scenes: The Making of BitTorrent Sync

Screen Shot 2013-07-16 at 4.52.12 PM

BitTorrent Sync is a simple way to sync files across your devices; free from the speed, security, and size limits that come standard with traditional cloud-based solutions. Today, people around the world have synced over 8 petabytes of data using the app. But not so long ago, it was just an idea we were kicking around. Here’s how Sync went from whiteboard to real world, in a little under a year.

Sync started with a common problem…

Like most families, my parents and I are constantly sending each other updates. Usually, these took the form of large photos and videos. We’d share upwards of 2GB of media. So, in order to keep in touch, we had to rely on pricey premium file transfer services.

I noticed that my family was the rule, and not the exception.

I had friend who’d recently had a kid. Both he and his wife loved taking pictures of their child. They kept their photos on their home computers, but wanted to ensure that these photos were always available, and always safe. Their best option? Pay a provider to slowly synchronize the data between their home PCs. But paying for data synchronization inside your home network, between your own computers, seemed ridiculous.

So we got together and talked about our pain points. We agreed that we needed something that syncs folders over the Internet and LAN directly; a solution capable of keeping shared files updated.

…that the Internet had yet to solve.

When we figured out what we wanted to do, we were shocked to learn no product existed. I spent the next month researching every product in the market that claimed to solve all or part of the problem. The results were surprising: not a single product could effectively accomplish this simple task. We decided to do something about that.

Our work began with a Palooza hack.

For a few days each quarter, all of BitTorrent stops work on regular projects, and starts in on the fun stuff. (We call this Palooza.) We divide ourselves into teams dedicated to building out new ideas. Here’s where work on Sync began.

Within two days, we had a product capable of synchronizing new files between folders on several computers inside the LAN and over the Internet. It still required quite a bit of manual configuration at that point, but the most important detail was that it worked.

When we saw how a 300MB file was distributed among five computers inside the local network, I knew we were at the start of something big.

Building our base.

After the Palooza, BitTorrent recognized the potential of the idea, and decided that the idea needed a team. We began to make slight changes to the BitTorrent protocol so that it would be possible to exchange the list of files in several folders, detect file changes and propagate them across nodes.

Fortunately, we were building on a very solid base. The µTorrent core was developed in pure hardcoded C++ and optimized for memory and speed. We just needed to develop an additional level to convert local file system objects into torrents.

Ditching the cloud.

One of the major decisions we made in the early days of development is that Sync should not require the cloud to work. It should be completely independent of severs. There were a few reasons for this:

It’s impossible to create a free product if the company has to support a cloud environment.
The costs will grow in direct correlation to the user base. After costs reach a certain point, the team will need to make decisions about how to cover costs, and whether or not the product needs to be changed: to eliminate features, restrict usage, or change the pricing model from free to paid.

The cloud places data security and access at risk.

The second you put your data in the cloud, it leaves your control. Even when encrypted. Access to your data can be limited or even revoked for commercial or political reasons. And data stored in the cloud is more susceptible to large-scale attacks. We wanted to build a way for you and your data to sleep safely at night. In the BitTorrent Sync model, users own their own data.

Frankly, the cloud is slow.

Cloud environments require a pit stop to a central data server (which, btw, isn’t yours). Peer-to-peer systems do not. This makes them intrinsically more efficient than traditional cloud-based infrastructure. When you’re trying to sync and move files on your networks, you should be able to do so as quickly as possible, without an unnecessary pit stop or bottleneck.

On Secrets

We wanted to develop an authentication measure for Sync. So we came up with Secrets. The Secret is a fully random 160-bit key that’s impossible to brute force. You can use or generate as many Secrets as you need.

Testing the product on our CEO

Our final goal at this stage of development was to create a product that would work to address the day-to-day needs of at least three other people within BitTorrent. One of them was our CEO, Eric Klinker.

He began using it to sync photos between his family’s computers: a problem he needed a solution for, a problem we didn’t know he had, and a problem we successfully solved. He’s been a happy Sync user ever since.

Getting Sync out there.

By August, we had all of BitTorrent behind us. Now, we were ready to get it out to users. We added Sync to BitTorrent Labs: a place for anyone (everyone) to experiment with what BitTorrent’s working on. This made Sync, in its infancy, available to use, play with, and experiment on. We invited people to try Sync early, and give feedback often. This allowed us to find out more about the product and its potential. Two things we discovered early on:

1. Sync is the fastest way to pass information between two computers.

Period. Usually you think that BitTorrent is faster with a larger group of peers – a swarm. While having a larger group of peers benefits Sync too, it is also true that direct communication is always going to be faster from one peer directly to another, than when that same communication requires an additional hop through a server.

2. People need to Sync

Before we even even brought BitTorrent Sync to Labs, we created single page test website to detail Sync’s features; asking for email addresses of people interested. We did this without budget or brand support. And we expected no response. Instead, we received twenty-eight emails asking for more information. We had real people asking for information for a real solution we created.

Where we’re at now.

As soon as Sync was publicly announced, we received tremendous interest from people around the world. It reinforced an idea that motivated us throughout the development cycle: Sync provides a solution for people, everywhere.

Since its initial Alpha release, Sync has continued to grow its base of devoted and incredible users: people who are enthusiastic and creative; people who have brought forth their own awesome applications for Sync. For that, we feel lucky.

So, please join me in celebrating the team that made this product possible as we celebrate the first year of this product together. Special thanks goes to Sergey Matalytski, who was the first individual to join my team, with the belief that Sync will revolutionize the way that we share. We’re going to prove him right.

Kos
Written by: Kos

General Director of Research and Development

 Related Posts:
  • http://ho.race.hk Horace Ho

    Thank you very much for bringing us such a wonderful product! Before BitTorrent Sync, friends of mine take notebooks and HDDs to share files when we dine together. Now, we brought a few Raspberry Pi to sync files. Thank you!

  • Balint

    Just on to say a big – 8 petabytes big – thank you for this wonderful tool!

  • AlexJ

    So where really are you at?

    BitTorrent Sync is crazy amazing, no doubt, but I wish there was more actual info on its development. I subscribed to this blog just for Sync and it’s mostly about RunDMC, etc. Is there a feed JUST on Sync?

    Can’t wait for this to be on android so I can start using it!

  • Deep

    How abt transforming utorrent based on this product?

    If I have a file to share… I should be able to post it on a forum /website like torrentz.eu, along with my SyncId or QR code, then any user should be able to should be able to access that file / folder.

  • danwat1234

    Good work with the product. It’s innovation that keeps the USA strong and freedom reign.

  • frank

    Hi I just ran across BT Sync and think about using it. As I am reading available information on the tool I was surprised that you know that your customers have transferred 8 Petabytes of data with the app. How and why do you know that? I thought the transfer is strictly Peer-to Peer?
    cheers frank

  • http://friendshipdayquote.org John

    Hi, I really appreciate your blog. Great!
    Happy Friendship day messages