Recent coverage of uTP on the popular Torrentfreak blog yielded some interesting feedback in the comments section.

There are a couple of misconceptions that I’d like to address here:

First is the idea that we designed uTP *for* the ISPs. It was not.

While we think there are substantial advantages for ISPs in the broad adoption of uTP, the protocol was actually built from the start as a way to help consumers themselves. The fact remains that when using TCP, a poorly tuned BitTorrent client may well result in an internet connection that habitually gets congested and then drops packets, then recovers and repeats the process. This is not good for anyone.

The second misconception is that uTP will somehow slow uTorrent down. This is also not true. It will certainly result in less headaches for everyone and it may even speed things up.

Our design objectives were always to leave transfer rates unchanged, and we’re still confident this is achieveable. The fact that you don’t have to manually “manage” your client or limit it to some arbitrary % of your connection should mean that in practice it will be reliably faster. What’s more, we may actually be able to make it go faster than an unlimited TCP BitTorrent client. The way to picture this is to consider cars on a highway: you can only drive at 90 mph if there’s not much other traffic. But if there’s a lot of traffic then quickly the whole system will snarl up. uTP is designed to make clients transfer at an optimal speed *without* causing a snarl up. The thrill of speeding along at 90 mph is rather lost if you keep having to slow to a crawl until things recover. By avoiding this “stop/start” we felt that uTP *should* make things go faster overall.

Early evidence is starting to come out now from researchers at the University of Washington who are performing some independent tests on uTP performance. (These results are NOT conclusive at this point, but the early indications are quite good…)

From the first graph below you can see the interaction of uTP traffic (green) with some other application competing to use the connection (red). As expected, the uTP traffic backs off immediately and is replaced by traffic from the competing application – upon completion of the competing transfer, the uTP BitTorrent traffic quickly resumes. The blue data points represent the uTP traffic holding steady against the (right vertical axis) target delay of 100ms (I’d note this is vastly lower than anything achievable with TCP BitTorrent transfers).

The uTP controller is clearly doing its job, spotting a different application trying to use bandwidth and getting out of the way, only to recover just a fast.

But in many ways the more important graphs are the following…. These show you that uTP BitTorrent is just as fast as best-case TCP BitTorrent, and may even be faster…

Now one likely explanation for this is that the uTP overhead (a few % of the traffic which is not actual content) is included, but the TCP measurement excludes it. If this were true then probably uTP and TCP are almost identical.

But if we find that uTP traffic is indeed faster than TCP BitTorrent traffic, there are a couple of reasons why this slightly surprising conclusion might indeed be true –

Either the stop-start nature of TCP-based BitTorrent creates inefficiencies that are being optimized away using uTP.

Or else there were ISP network management measures in place which were discriminating against TCP-based BitTorrent.

Or possibly the UDP NAT-traversal techniques introduced along with uTP were resulting in far more good peers with uTP.

Or possibly something else?

Whatever the reason, this is early evidence that uTP is an even bigger win for consumers than anticipated, as well as being a positive contribution to ISPs.

Much more work remains to be done, but this is exactly the type of result we’re hoping to see more of.

–Simon–

We’ve spent a lot of time in recent posts talking about the benefits of µTP.  We’ve even talked a little bit about how it works here, though much more so in the various technical forums for the community.  But sometimes a picture is worth 2^^10 words and I think the graph below says it best.  µTP appears to be up to the task of reducing congestion.

These results are taken from our QA regression tests that we run on each new version of the client that ships with µTP.  The test is a simple one.  We use a DSL line here in the office and start a client seeding on that DSL line.  We then measure the latency seen by other applications, such as VoIP, online games and web browsing, that we run concurrently over the same link.  The graph above is a histogram of those latency samples.

The green samples were taken with a client seeding on TCP and the red samples were taken with a client seeding on uTP.  (You can tell that these are engineering graphs rather than marketing ones simply enough by the fact that GREEN= bad and RED = good, but you get the picture…).  In reading the graph, remember, queuing delay (latency) is a side effect of congestion.  More latency in this test means more congestion.

With the target latency set at 100ms, µTP does a pretty good job keeping the latency felt by the other applications near the target.  TCP clearly does not and more than congests the uplink.  In the process this ruins the network for all of the adjacent applications below.

While much work remains ahead of us (like picking the right target latency), it seems that µTP demonstrates some clear potential to alleviate network congestion wherever the network bottleneck happens to reside.  This has obvious benefits for users who will no longer congest themselves, benefits for publishers who want to use BitTorrent but also want to protect their brand when users seed content on their behalf, and benefits for ISPs who should see far fewer support issues with BitTorrent causing congestion and impacting other users on the network.

A win win win.

The CRTC released its long awaited decision last week on Network Neutrality in Canada and specifically set forth a framework for the continued use of Internet Traffic Management Practices (ITMPs), also known as throttling.  While the decision does not go as far as many consumer advocates would like, I believe it does provide a means by which most throttles can be avoided.

In a keynote address, CRTC Chairman Konrad von Finckenstein explained that an ITMP should be implemented only if:

1.  It addresses a justifiable purpose; for example, it is needed to prevent congestion, or disruption of time-sensitive programs.

2. It is as narrowly tailored as possible to achieve the desired result, using the least restrictive means.

3. It causes as little harm as possible to the retail customer, the application provider or the ISP that is the wholesale customer of a primary ISP.

4. And it is well advertised in advance. A full explanation must be given, describing the practice and how it will affect the user.

Fair enough, the thing has to do a particular job with a narrowly defined purpose and the ISPs must be transparent about this job.  So what might they do, you ask?  Specifically, the decision notes the “needs” currently cited by parties for the existing crop of ITMPs:

“Parties generally acknowledged that some traffic management is required to address congestion in order to ensure that all end-users receive acceptable Internet service. Parties also generally agreed that ISPs must employ ITMPs to protect the integrity of their networks from security threats.” (emphasis mine)

Address congestion and mitigate security threats.

We’ve talked at length about µTP and its design which specifically avoids causing congestion.  So on a fundamental level, this decision is good news for µTP.    There should be no need to throttle µTP in an effort to address congestion or mitigate a security threat.   And the framework will not permit it otherwise, given its discriminatory nature.

So why is everyone so glum?  First of all I think the complaint based approach the CRTC has taken is one that will tie up lots of resources (lawyers) on all sides with continued complaints.   While the framework is an incremental and far from radical step in the right direction, we wonder if the ISPs will modify their practices accordingly.  We remain ready to work with them towards this end, to help them understand µTP and its beneficial nature, as we’ve done with many ISPs here is the U.S. and elsewhere.   But while that’s our view, others find it more likely that future complaints testing these points will now need to be adjudicated.

–Eric–

I was recently invited to participate in a workshop sponsored by the GIIC, an organization of telecom and technology executives who ponder large scale Internet and information infrastructure questions.   The purpose of this workshop was to consider changes to the Internet infrastructure that would allow cost transparency, an upgrade to the Internet platform that could have wide ranging implications for the economic models that currently prevail online.

While that topic is worthy of exploration on its own, some of the many areas explored during this particular gathering were in matters of policy and the recently announced rulemaking proposal by the FCC towards a principle of network neutrality.  And throughout the many discussions, it became apparent that those on the “market” side of this debate have a very difficult challenge ahead of them:  How can they frame the debate in a way that doesn’t have them come across as online versions of Strom Thurmond?  This is no small challenge given the basic issues of equality are wrapped into our national identity and any implications of inequality tug at the strings of that identity.   So as a starter, I’d suggest attempts at branding users (customers) as bandwidth hogs will not yield a useful approach toward this end.  Does society favor discrimination against those who over-eat? Even on an increasingly green political landscape, any definition of a “reasonable consumption level” is going to be, to say the very least, “sensitive”.

Some defensible ground remains in the areas of network congestion if one can stake out clear and reasonable technical arguments and actions defending those principles.  Being the target of a great many technologies currently deployed to shape, block and throttle (i.e. discriminate) it would be easy for BitTorrent to assume the role of victim in our little analogy.  I suspect there is a great deal of mileage to be had in this approach amid the fury of debate.  But this isn’t our plan.  Instead, we’ve spent considerable energy developing technologies in µTP to combat the underlying premise of discrimination, Internet congestion.    And while it remains to be seen how the market will react to these developments, whether the DPI currently deployed will be modified to discriminate against µTP as well, there is an opportunity for one side of the debate to stand behind their principles and in so doing demonstrate the potential of self regulation to the benefit of all.

–Eric–

μTP or “micro-Transport Protocol” is a new protocol from BitTorrent, Inc. that is at the heart of the new major release of our popular BitTorrent clients “μTorrent” and “BitTorrent Mainline”. It is going to be available as the default transport mechanism in both μTorrent v2.0 and BitTorrent v7.0. So what’s the big deal? And why do we want this to be the centerpiece of our future software?

The fact is that our BitTorrent clients have become incredibly popular with users downloading large files over the internet. So much so that some observers claim that BitTorrent traffic accounts for 30%, 50%, or even more of all Internet traffic. Regardless of the actual numbers (which we have no way of knowing), it is clear that the popularity of BitTorrent is putting such a burden on ISP networks that they sometimes react by slowing down or interfering with that traffic.

Now there is a whole “net neutrality” debate, partly about whether ISPs should be allowed to interfere with internet traffic from one particular app simply because it is “too popular” – some argue that perhaps ISPs could invest more so that supply meets demand – but this debate is not the focus here. At BitTorrent we like to be a bit more pragmatic, to assert that there is responsibility on the part of both the ISPs and authors of popular applications like BitTorrent to make sure that the internet scales smoothly to meet demand.

Which brings us back to μTP:

News of μTP started to leak to the public late last year with some wild and totally untrue reporting that we were trying to make BitTorrent more greedy and were somehow “declaring war” on users of other applications. In fact completely the opposite is true, as was subsequently acknowledged by the initial author’s follow-up article.

μTP is a completely new implementation of the BitTorrent protocol with a major new design objective – μTP is designed to be network friendly – to not swamp network connections when there are other apps trying to send and receive – and to resolve the key problem that ISPs use to justify interference with BitTorrent traffic.

If BitTorrent traffic volume is so great that it overwhelms end-users’ connections (leading to service calls from consumers whose internet doesn’t work), then μTP eliminates this problem by being better at only using bandwidth when there is no other traffic competing, and automatically slowing or stopping BitTorrent transfers before network connections seize up.

Legacy BitTorrent traffic uses the standard internet “TCP” protocol to govern when it tries to go faster or slow down. The problem with TCP is that it can only detect a problem by waiting to see if packets are dropped. Unfortunately, by the time packets are being lost, the problem is already acute and the consumers connection has already drastically slowed or stopped. TCP is a lot like trying to drive with your eyes closed. You only notice something’s wrong when you hit something.

μTP is like driving with your eyes *open* – μTP is able to see problems coming and make much more modest adjustments to ensure the problems don’t cause a car wreck. It does this by being able to detect congestion on a network based on how long a packet takes to be sent from one peer to the next. If things start to take longer, then μTP adjusts the rate of sending accordingly.

As it happens, this trick has required some very deep engineering work – the way the client talks to other clients has had to be completely re-built. As a side effect, because the new protocol so different, it is practically invisible to some of the nasty traffic shaping techniques that some ISPs have been using. We doubt whether this happy result will last for long, and nor is it the point of the technology. The point is to reduce the need for such gear rather than to evade it.

Overall, when we get μTP stable, we’re excited about the potential benefits that this could bring to ISPs by reducing the effective burdens on their networks. Although we stand to gain nothing financially from them for implementing it, we hope to maintain the lead enjoyed by μTorrent and BitTorrent Mainline software as the most popular BitTorrent clients, and hopefully demonstrate how innovation from responsible stakeholders on a neutral internet can lead to winning outcomes all-around.

– Simon–

In light of the FCC’s recently proposed rulemaking around Network Neutrality, many of you might have missed a similarly lively debate in Canada a few weeks back around the traffic management practices of Canadian ISPs. Over a week of public hearings, there were some astonishing revelations around the practice of throttling BitTorrent and other P2P traffic. Michael Geist’s excellent blog covered the events in detail and of course you can get the legalese of each submission from the CRTC website here.

Since congestion was the primary justification for throttling among the ISPs, it was the perfect opportunity in our own filing to showcase μTP (“micro-transport protocol) as the solution to these problems as well generally educate the commission on the benefits and efficiencies inherent in P2P technology. If congestion is in fact the problem, μTP effectively removes that rationale for BitTorrent throttling while demonstrating the ability of good old-fashioned innovation to solve the hard problems on the Internet.

Many of the ISP filings ask the regulators to allow the market to work its magic (“don’t regulate us”). The irony of this position cannot be lost on those of us who must compete and reach consumers on these networks, and yet are subject to ad-hoc traffic management policies.  Traffic management by its very nature can skew the marketplace in favor (or to the disfavor) of any technology or business.   In recognizing the need to manage congestion on these networks, these traffic management technologies should therefore be very even-handedly applied.

We believe μTP will solve the ISPs’ greatest problems around congestion, and we would ask and hope that the ISPs will stand behind their market oriented principles (which we share as our daily reality) and refrain from throttling P2P traffic using μTP.  It would be a demonstration of good faith to the market and the power of engineers and innovation to solve a myriad of problems.

– Eric –

This is the first post of our new blog from employees at BitTorrent, Inc. Our intent is to inform and participate in ongoing discussions about BitTorrent, but also to broaden those discussions to other topics that we find interesting and exciting.

BitTorrent-the-company

The word “BitTorrent” means many things to many people – to us it is a company – an employer of about 20 people (mainly engineers) who work in San Francisco on innovative content distribution technology. We are broadly interested in the magic of user-contributed-infrastructure and technology that enables users to publish, download and control their digital content in an ever-more decentralized internet. We are not a media company. We are not a political party. We’re just geeks trying to push the boundaries of technical possibility.

BitTorrent-the-technology

The incredible recognition commanded by our brand carries with it a great deal of baggage. BitTorrent is first and foremost a content delivery “protocol” (a way that computers talk to each other) invented by Bram Cohen, who is still an employee. With BitTorrent, large files can be moved around the internet without the need for large servers to serve them. The protocol requires a special piece of software on your computer (a “BitTorrent client”) as well as special lightweight servers to direct traffic. The content is “served” in small pieces from user-to-user (“peer-to-peer”) without the need for big data centers to be available to meet demand.

BitTorrent the ecosystem

The fact that the technology works so well has meant that there are a number of organizations involved in developing BitTorrent technology or derivatives of it. Likewise there are many many more involved in making it operational. It’s a bit like web servers and web browsers. Some people make the technology itself, but then an astonishing number of people make use of the technology. For better or worse, the word “BitTorrent” has become a moniker for an entire ecosystem. It is used to label both the technology, but even more what people do with the technology, and even the sometimes disruptive impact of those activities either on service providers trying to control their networks or on digital content publishers trying to control their content.

For the purposes of this blog and this website, we would stress that we certainly speak with passion on behalf of our company. We also actively promote and champion BitTorrent technology – we’d hope to be seen as one of the more qualified voices, but not the only one. But while we feel we are certainly a “relevant” voice in the broader ecosystem, there are many other voices too. Some we agree with, and some we don’t. We cannot and do not represent them all.

– Simon –