Twitter using BitTorrent on the Backend

February 9, 2010 – 5:37 pm

Anyone who has the battle scars that come from managing a large operational set of computers, like a large popular web site, knows all too well the challenge of deploying a new image site-wide.  Often times, this needs to happen fast (think emergency patch) and as with so many other applications on the network, the traditional client-server model breaks down at scale in the face of this challenge.

But now there is an interesting open source project from Twitter that might bring some relief to this particular problem.  The project uses BitTorrent technology and is called  Murder (per the project page, a “murder” allegedly describes a flock of crows, not some lesser crime than the accusations too-often leveled against the hard-working engineers at BitTorrent, Inc).  The distributed nature of BitTorrent means an operation that once took many dozens of minutes, now happens in less than a dozen seconds.  These efficiencies will reduce maintenance windows, site downtime and exposure to security vulnerabilities.

The folks at Twitter will be disclosing more about the project and the performance of this solution in the coming weeks.  We are thrilled to collaborate with them on this and hope that more Web Monsters out there look to this and other applications of BitTorrent in solving some of the hard problems of the Internet.

–Eric–

By Eric Klinker | Posted in Uncategorized | Tagged , , | Comments (0)

µTorrent v2.0 stable release

February 3, 2010 – 11:08 am

There are very few software products that can expect to have an impact that is measurable on the scale of the internet-at-large. Its probably not too much of a reach to claim that µTorrent v2.0 is one of those products; yesterday we released the first stable version of µTorrent v2.0

To be clear, this is not just a glib assertion: we have spent a great deal of time and energy in designing and building µTP with the expectation that it can have a very significant and positive impact for both consumers and network operators. We have also publicized the arrival of the technology extensively both in popular and technical circles.

As we have discussed previously, the importance of this milestone is that for the first time the world’s most popular BitTorrent clients will be using the new µTP (or “uTP”) protocol by default. We will shortly start updating all existing users who want it to this new stable version.

The effect of this should be very positive for end users:

(1)    A µTorrent client no longer needs to be limited to some arbitrary rate limit for fear that it will saturate the connection and crowd out all other applications – µTP  will take care of this automatically at the level of each packet transferred. On average this will likely result in actual average download speeds increasing across the board.

(2)    Because the µTP protocol is designed to mitigate congestion on networks caused by poorly configured BT clients, it is possible that ISPs will take a friendlier view towards the protocol than they often take towards the older type of BT protocol. This *may* mean less BT throttling and faster download speeds for end users. There’s some early anecdotal evidence that this is indeed the case, and certainly ISPs to whom we have presented the technology have so far reacted quite warmly.

(3)    Although there have been some concerns voiced that users “don’t want their clients to throttle themselves”, of course the reality is that if the line is congested then nothing can be transferred. You may want to drive at 80mph, but if there’s a traffic jam you’re out of luck. µTP will mean that BitTorrent transfers are no longer the cause of congestion, and will enable the connections to automatically recover from a congested state much faster.

The effect on network operators will be also hopefully quite noticeable: We anticipate that a large proportion of BitTorrent traffic will start to actively monitor packet transfer speeds for any indication that there is congestion on the line. If there is even a tiny amount of congestion, no matter what caused that congestion, µTP will immediately slow itself down and allow the congestion to clear.

Of course network congestion is not the only concern of network operators, and we will continue to support efforts to enable ISPs to communicate preferred routing policies to clients (for example through the IETF’s ALTO working group). But we hope that to the extent that µTP is successful at mitigating internet congestion, it is interpreted as an exciting win for everyone involved.

–Simon–

By Simon Morris | Posted in Uncategorized | Tagged , , , | Comments (7)

Easier setup in µTorrent 2.0 and Measurement Lab collaboration

January 25, 2010 – 10:48 am

With the release of µTorrent 2.0 Beta, there has been a lot of great coverage and discussion of µTP and what it means to be “network friendly”.  This is an important change to the BitTorrent landscape and will surely be the subject of more discussion and debate, but there are some other features in 2.0 that are worth highlighting.  One of these is the improvements to the setup guide and our first collaboration with Measurement Lab.

While BitTorrent is a relatively mature technology with a sizable global user-base, we have to admit, the learning curve can be steep for some users.  One of the reasons for this is the fact that µTorrent is one of the most powerful network applications a user is ever likely to install, and even the simplest home network is often difficult to set-up and manage.  Configuring your µTorrent client to match your network is an important step that will make for a better user experience.

In µTorrent 2.0 Beta we have sought to streamline this operation by automating the configuration process in the setup guide.  With just a click or two, users can now test their internet connection speed, see how their router handles connections, and then optimize their client to their own network.  We think these features will simplify the experience for new users and make it easier for everyone to get the best results from µTorrent.

To accurately configure µTorrent for optimal performance, it’s helpful to know the bandwidth available to the client.  Traditionally, we have expected users to go to another site to perform a test, understand the results and translate them to the client’s configuration settings.  With µTorrent 2.0 we have integrated the speed testing into the setup guide.  To do this we have partnered with Measurement Lab, which provides the testing tools and infrastructure.

Measurement Lab Collaboration

Founded by the New America Foundation’s Open Technology Institute, the PlanetLab Consortium, Google, Inc. and academic network researchers, Measurement Lab (M-Lab) is an open, distributed server platform for researchers to deploy active network measurement tools to measure real-world broadband connectivity.  M-Lab’s goal to “advance network research and empower the public with useful information about their broadband connections,” is one we at BitTorrent would like to support.

Our speed test relies on the Network Diagnostic Tool (NDT) servers hosted at M-Lab sites around the world.  The non-personally identifiable data generated by these tests (data collection info here) will be made available to researchers through M-Lab under a Creative Commons Zero license.  Given µTorrent’s substantial user-base, we are hopeful that this data will stimulate new research into the state of the Internet and support the public debate with unbiased measurement data.

M-Lab is supporting important research into how our Internet is actually performing and informing the debate on how this shared resource should be managed.  We plan introduce more M-Labs tools to users in the near future with the hope that empowering users with data about their broadband services will lead to more informed policy debate and better consumer experiences.  In the meantime, we invite you to learn a little more about your broadband connection by exploring some of the other tools at www.measurementlab.net.

By Brett Nishi | Posted in Uncategorized | Comments (2)

Point-Click-Wait or Point-Click-Watch… Will Streaming Break BitTorrent?

December 24, 2009 – 4:41 pm

The work we are currently doing to bring streaming to our popular µTorrent freeware has the potential to bring transformational improvements to the user experience for BitTorrent users. But its not without at least a little controversy, which I’d like to discuss in this post.

Roughly speaking, the controversy falls into two different areas:

(1)    “Streaming is easy”

From our point of view, streaming is not the most ambitious initiative we’ve been involved with (both µTP and Bram’s Live Streaming project are far bigger challenges).

But it is not trivial either, which is the main reason it has not happened sooner.

Our objective is to enable streaming of files distributed using popular file formats on regular BitTorrent swarms without relying on any server infrastructure and without violating any of the core rules that make the BitTorrent protocol so effective. The file will still be downloaded to your hard drive, but you will be able to play the file while it is downloading rather than waiting for the download to complete. What we’d love to eventually achieve is an experience that might be integrated seamlessly with the webtop, so that the µTorrent client can simply provide background support to a web experience which is ultimately very similar that of YouTube or Hulu, but with radically different economics.  We might call it the “µTube” vision…

What’s exciting about this is not just the fact that we have technology to achieve this, but we also have a critical mass of users who will be able to accelerate this towards broad adoption.

In short – it ain’t rocket science, but its not trivial; and its novelty is mostly in its breadth of scope rather than the fact that you will be “able to stream with BitTorrent”.

(2)    “Streaming will kill BitTorrent”

We play an important role trying to safeguard the development of standards around the BitTorrent protocol – a responsibility we take very seriously. There seem to be a surprisingly large number of pundits out there who seem “sure” that streaming will mess up BitTorrent. We’re actually quite certain that this is not the case. I might go on about the technical credentials of a team including the person who invented BitTorrent, or the fact that we have more to lose than anyone if we’re wrong – but that may overdo things. We deeply respect people’s right to be concerned, and interpret the main motivator of these concerns as fundamentally the popular enthusiasm for an incredible piece of technology.

Our streaming prototype is a careful balance of using BitTorrent fundamentals (like rarest-first piece selection and tit-for-tat reciprocity) but introducing a concept of priority windows (which correspond to the portion of media actually being viewed or listened to.) We restrict any download of consecutive pieces (i.e. non-BitTorrent behavior) to a tiny (about 15 second) portion of playback.

Added to the fact that everyone hits the “start” button at a different time, many files are an hour or longer in playback duration, so we’re not surprised that so far we’ve encountered no perceptible impact of streaming on swarm health.

The most important decision that we need to bake into the user experience is deciding the point at which a file is “streamable” – not all files will have enough seeding bandwidth to allow streaming, and some may be marginal. Before any broad deployment, we expect to be quite conservative in determining if and when the “play” button shows up in order to avoid any theoretical impact on less well-developed swarms. Furthermore, we expect people’s behavior will be self-selecting in most cases – if someone tries to stream from an under-developed swarm, it simply won’t work – so they’ll probably give up.

But the important message is that we are confident that it is completely viable to enable streaming in well-seeded swarms without having any adverse effect on those swarms.

What’s more, the basics of the client will change little – streaming will be an option to be used only in cases where it makes sense to the user, and µTorrent will remain “a (very) tiny BitTorrent client”.

–Simon–

By Simon Morris | Posted in Uncategorized | Tagged | Comments (0)

Testing µTP – is µTP actually faster than regular BitTorrent?

November 13, 2009 – 7:04 pm

Recent coverage of uTP on the popular Torrentfreak blog yielded some interesting feedback in the comments section.

There are a couple of misconceptions that I’d like to address here:

First is the idea that we designed uTP *for* the ISPs. It was not.

While we think there are substantial advantages for ISPs in the broad adoption of uTP, the protocol was actually built from the start as a way to help consumers themselves. The fact remains that when using TCP, a poorly tuned BitTorrent client may well result in an internet connection that habitually gets congested and then drops packets, then recovers and repeats the process. This is not good for anyone.

The second misconception is that uTP will somehow slow uTorrent down. This is also not true. It will certainly result in less headaches for everyone and it may even speed things up.

Our design objectives were always to leave transfer rates unchanged, and we’re still confident this is achieveable. The fact that you don’t have to manually “manage” your client or limit it to some arbitrary % of your connection should mean that in practice it will be reliably faster. What’s more, we may actually be able to make it go faster than an unlimited TCP BitTorrent client. The way to picture this is to consider cars on a highway: you can only drive at 90 mph if there’s not much other traffic. But if there’s a lot of traffic then quickly the whole system will snarl up. uTP is designed to make clients transfer at an optimal speed *without* causing a snarl up. The thrill of speeding along at 90 mph is rather lost if you keep having to slow to a crawl until things recover. By avoiding this “stop/start” we felt that uTP *should* make things go faster overall.

Early evidence is starting to come out now from researchers at the University of Washington who are performing some independent tests on uTP performance. (These results are NOT conclusive at this point, but the early indications are quite good…)

From the first graph below you can see the interaction of uTP traffic (green) with some other application competing to use the connection (red). As expected, the uTP traffic backs off immediately and is replaced by traffic from the competing application – upon completion of the competing transfer, the uTP BitTorrent traffic quickly resumes. The blue data points represent the uTP traffic holding steady against the (right vertical axis) target delay of 100ms (I’d note this is vastly lower than anything achievable with TCP BitTorrent transfers).

The uTP controller is clearly doing its job, spotting a different application trying to use bandwidth and getting out of the way, only to recover just a fast.

utp-vs-tcp2

But in many ways the more important graphs are the following…. These show you that uTP BitTorrent is just as fast as best-case TCP BitTorrent, and may even be faster…

noUTP

withUTP

Now one likely explanation for this is that the uTP overhead (a few % of the traffic which is not actual content) is included, but the TCP measurement excludes it. If this were true then probably uTP and TCP are almost identical.

But if we find that uTP traffic is indeed faster than TCP BitTorrent traffic, there are a couple of reasons why this slightly surprising conclusion might indeed be true –

Either the stop-start nature of TCP-based BitTorrent creates inefficiencies that are being optimized away using uTP.

Or else there were ISP network management measures in place which were discriminating against TCP-based BitTorrent.

Or possibly the UDP NAT-traversal techniques introduced along with uTP were resulting in far more good peers with uTP.

Or possibly something else?

Whatever the reason, this is early evidence that uTP is an even bigger win for consumers than anticipated, as well as being a positive contribution to ISPs.

Much more work remains to be done, but this is exactly the type of result we’re hoping to see more of.

–Simon–

By Simon Morris | Posted in Uncategorized | Comments (3)

Visualizing µTP

November 2, 2009 – 2:59 pm

We’ve spent a lot of time in recent posts talking about the benefits of µTP.  We’ve even talked a little bit about how it works here, though much more so in the various technical forums for the community.  But sometimes a picture is worth 2^^10 words and I think the graph below says it best.  µTP appears to be up to the task of reducing congestion.

Visualizing uTP 1

These results are taken from our QA regression tests that we run on each new version of the client that ships with µTP.  The test is a simple one.  We use a DSL line here in the office and start a client seeding on that DSL line.  We then measure the latency seen by other applications, such as VoIP, online games and web browsing, that we run concurrently over the same link.  The graph above is a histogram of those latency samples.

The green samples were taken with a client seeding on TCP and the red samples were taken with a client seeding on uTP.  (You can tell that these are engineering graphs rather than marketing ones simply enough by the fact that GREEN= bad and RED = good, but you get the picture…).  In reading the graph, remember, queuing delay (latency) is a side effect of congestion.  More latency in this test means more congestion.

With the target latency set at 100ms, µTP does a pretty good job keeping the latency felt by the other applications near the target.  TCP clearly does not and more than congests the uplink.  In the process this ruins the network for all of the adjacent applications below.

Visualizing uTP 2

While much work remains ahead of us (like picking the right target latency), it seems that µTP demonstrates some clear potential to alleviate network congestion wherever the network bottleneck happens to reside.  This has obvious benefits for users who will no longer congest themselves, benefits for publishers who want to use BitTorrent but also want to protect their brand when users seed content on their behalf, and benefits for ISPs who should see far fewer support issues with BitTorrent causing congestion and impacting other users on the network.

A win win win.

These results are taken from our QA regression tests that we run on each new version of the client that ships with µTP.  The test is a simple one.  We use a DSL line here in the office and start a client seeding on that DSL line.  We then measure the latency seen by other applications, such as VoIP, online games and web browsing, that we run concurrently over the same link.  The graph above is a histogram of those latency samples.

The green samples were taken with a client seeding on TCP and the red samples were taken with a client seeding on uTP.  (You can tell that these are engineering graphs rather than marketing ones simply enough by the fact that GREEN= bad and RED = good, but you get the picture…).  In reading the graph, remember, queuing delay (latency) is a side effect of congestion.  More latency in this test means more congestion.

With the target latency set at 100ms, µTP does a pretty good job keeping the latency felt by the other applications near the target.  TCP clearly does not and more than congests the uplink.  In the process this ruins the network for all of the adjacent applications below.

By Simon Morris | Posted in Uncategorized | Tagged , , | Comments (11)

It’s Cold Up North

October 29, 2009 – 4:32 pm

The CRTC released its long awaited decision last week on Network Neutrality in Canada and specifically set forth a framework for the continued use of Internet Traffic Management Practices (ITMPs), also known as throttling.  While the decision does not go as far as many consumer advocates would like, I believe it does provide a means by which most throttles can be avoided.

In a keynote address, CRTC Chairman Konrad von Finckenstein explained that an ITMP should be implemented only if:

1.  It addresses a justifiable purpose; for example, it is needed to prevent congestion, or disruption of time-sensitive programs.

2. It is as narrowly tailored as possible to achieve the desired result, using the least restrictive means.

3. It causes as little harm as possible to the retail customer, the application provider or the ISP that is the wholesale customer of a primary ISP.

4. And it is well advertised in advance. A full explanation must be given, describing the practice and how it will affect the user.

Fair enough, the thing has to do a particular job with a narrowly defined purpose and the ISPs must be transparent about this job.  So what might they do, you ask?  Specifically, the decision notes the “needs” currently cited by parties for the existing crop of ITMPs:

“Parties generally acknowledged that some traffic management is required to address congestion in order to ensure that all end-users receive acceptable Internet service. Parties also generally agreed that ISPs must employ ITMPs to protect the integrity of their networks from security threats.” (emphasis mine)

Address congestion and mitigate security threats.

We’ve talked at length about µTP and its design which specifically avoids causing congestion.  So on a fundamental level, this decision is good news for µTP.    There should be no need to throttle µTP in an effort to address congestion or mitigate a security threat.   And the framework will not permit it otherwise, given its discriminatory nature.

So why is everyone so glum?  First of all I think the complaint based approach the CRTC has taken is one that will tie up lots of resources (lawyers) on all sides with continued complaints.   While the framework is an incremental and far from radical step in the right direction, we wonder if the ISPs will modify their practices accordingly.  We remain ready to work with them towards this end, to help them understand µTP and its beneficial nature, as we’ve done with many ISPs here is the U.S. and elsewhere.   But while that’s our view, others find it more likely that future complaints testing these points will now need to be adjudicated.

–Eric–

By Simon Morris | Posted in Uncategorized | Tagged , , , | Comments (0)

The Internet Civil Rights Act of 2009

October 15, 2009 – 12:27 am

I was recently invited to participate in a workshop sponsored by the GIIC, an organization of telecom and technology executives who ponder large scale Internet and information infrastructure questions.   The purpose of this workshop was to consider changes to the Internet infrastructure that would allow cost transparency, an upgrade to the Internet platform that could have wide ranging implications for the economic models that currently prevail online.

While that topic is worthy of exploration on its own, some of the many areas explored during this particular gathering were in matters of policy and the recently announced rulemaking proposal by the FCC towards a principle of network neutrality.  And throughout the many discussions, it became apparent that those on the “market” side of this debate have a very difficult challenge ahead of them:  How can they frame the debate in a way that doesn’t have them come across as online versions of Strom Thurmond?  This is no small challenge given the basic issues of equality are wrapped into our national identity and any implications of inequality tug at the strings of that identity.   So as a starter, I’d suggest attempts at branding users (customers) as bandwidth hogs will not yield a useful approach toward this end.  Does society favor discrimination against those who over-eat? Even on an increasingly green political landscape, any definition of a “reasonable consumption level” is going to be, to say the very least, “sensitive”.

Some defensible ground remains in the areas of network congestion if one can stake out clear and reasonable technical arguments and actions defending those principles.  Being the target of a great many technologies currently deployed to shape, block and throttle (i.e. discriminate) it would be easy for BitTorrent to assume the role of victim in our little analogy.  I suspect there is a great deal of mileage to be had in this approach amid the fury of debate.  But this isn’t our plan.  Instead, we’ve spent considerable energy developing technologies in µTP to combat the underlying premise of discrimination, Internet congestion.    And while it remains to be seen how the market will react to these developments, whether the DPI currently deployed will be modified to discriminate against µTP as well, there is an opportunity for one side of the debate to stand behind their principles and in so doing demonstrate the potential of self regulation to the benefit of all.

–Eric–

By Eric Klinker | Posted in Uncategorized | Tagged , , , | Comments (0)

Changing the game with μTP

October 5, 2009 – 3:39 pm

μTP or “micro-Transport Protocol” is a new protocol from BitTorrent, Inc. that is at the heart of the new major release of our popular BitTorrent clients “μTorrent” and “BitTorrent Mainline”. It is going to be available as the default transport mechanism in both μTorrent v2.0 and BitTorrent v7.0. So what’s the big deal? And why do we want this to be the centerpiece of our future software?

The fact is that our BitTorrent clients have become incredibly popular with users downloading large files over the internet. So much so that some observers claim that BitTorrent traffic accounts for 30%, 50%, or even more of all Internet traffic. Regardless of the actual numbers (which we have no way of knowing), it is clear that the popularity of BitTorrent is putting such a burden on ISP networks that they sometimes react by slowing down or interfering with that traffic.

Now there is a whole “net neutrality” debate, partly about whether ISPs should be allowed to interfere with internet traffic from one particular app simply because it is “too popular” – some argue that perhaps ISPs could invest more so that supply meets demand – but this debate is not the focus here. At BitTorrent we like to be a bit more pragmatic, to assert that there is responsibility on the part of both the ISPs and authors of popular applications like BitTorrent to make sure that the internet scales smoothly to meet demand.

Which brings us back to μTP:

News of μTP started to leak to the public late last year with some wild and totally untrue reporting that we were trying to make BitTorrent more greedy and were somehow “declaring war” on users of other applications. In fact completely the opposite is true, as was subsequently acknowledged by the initial author’s follow-up article.

μTP is a completely new implementation of the BitTorrent protocol with a major new design objective – μTP is designed to be network friendly – to not swamp network connections when there are other apps trying to send and receive – and to resolve the key problem that ISPs use to justify interference with BitTorrent traffic.

If BitTorrent traffic volume is so great that it overwhelms end-users’ connections (leading to service calls from consumers whose internet doesn’t work), then μTP eliminates this problem by being better at only using bandwidth when there is no other traffic competing, and automatically slowing or stopping BitTorrent transfers before network connections seize up.

Legacy BitTorrent traffic uses the standard internet “TCP” protocol to govern when it tries to go faster or slow down. The problem with TCP is that it can only detect a problem by waiting to see if packets are dropped. Unfortunately, by the time packets are being lost, the problem is already acute and the consumers connection has already drastically slowed or stopped. TCP is a lot like trying to drive with your eyes closed. You only notice something’s wrong when you hit something.

μTP is like driving with your eyes *open* – μTP is able to see problems coming and make much more modest adjustments to ensure the problems don’t cause a car wreck. It does this by being able to detect congestion on a network based on how long a packet takes to be sent from one peer to the next. If things start to take longer, then μTP adjusts the rate of sending accordingly.

As it happens, this trick has required some very deep engineering work – the way the client talks to other clients has had to be completely re-built. As a side effect, because the new protocol so different, it is practically invisible to some of the nasty traffic shaping techniques that some ISPs have been using. We doubt whether this happy result will last for long, and nor is it the point of the technology. The point is to reduce the need for such gear rather than to evade it.

Overall, when we get μTP stable, we’re excited about the potential benefits that this could bring to ISPs by reducing the effective burdens on their networks. Although we stand to gain nothing financially from them for implementing it, we hope to maintain the lead enjoyed by μTorrent and BitTorrent Mainline software as the most popular BitTorrent clients, and hopefully demonstrate how innovation from responsible stakeholders on a neutral internet can lead to winning outcomes all-around.

– Simon–

By Simon Morris | Posted in Uncategorized | Tagged , , , | Comments (15)

Net Neutrality (CRTC-style)

October 5, 2009 – 3:36 pm

In light of the FCC’s recently proposed rulemaking around Network Neutrality, many of you might have missed a similarly lively debate in Canada a few weeks back around the traffic management practices of Canadian ISPs. Over a week of public hearings, there were some astonishing revelations around the practice of throttling BitTorrent and other P2P traffic. Michael Geist’s excellent blog covered the events in detail and of course you can get the legalese of each submission from the CRTC website here.

Since congestion was the primary justification for throttling among the ISPs, it was the perfect opportunity in our own filing to showcase μTP (“micro-transport protocol) as the solution to these problems as well generally educate the commission on the benefits and efficiencies inherent in P2P technology. If congestion is in fact the problem, μTP effectively removes that rationale for BitTorrent throttling while demonstrating the ability of good old-fashioned innovation to solve the hard problems on the Internet.

Many of the ISP filings ask the regulators to allow the market to work its magic (“don’t regulate us”). The irony of this position cannot be lost on those of us who must compete and reach consumers on these networks, and yet are subject to ad-hoc traffic management policies.  Traffic management by its very nature can skew the marketplace in favor (or to the disfavor) of any technology or business.   In recognizing the need to manage congestion on these networks, these traffic management technologies should therefore be very even-handedly applied.

We believe μTP will solve the ISPs’ greatest problems around congestion, and we would ask and hope that the ISPs will stand behind their market oriented principles (which we share as our daily reality) and refrain from throttling P2P traffic using μTP.  It would be a demonstration of good faith to the market and the power of engineers and innovation to solve a myriad of problems.

– Eric –

By Eric Klinker | Posted in Uncategorized | Tagged , , , | Comments (1)