All posts by

Twitter using BitTorrent on the Backend

by eklinker on February 9, 2010 in Other with 22 Comments

Anyone who has the battle scars that come from managing a large operational set of computers, like a large popular web site, knows all too well the challenge of deploying a new image site-wide.  Often times, this needs to happen fast (think emergency patch) and as with so many other applications on the network, the traditional client-server model breaks down at scale in the face of this challenge.

But now there is an interesting open source project from Twitter that might bring some relief to this particular problem.  The project uses BitTorrent technology and is called  Murder (per the project page, a “murder” allegedly describes a flock of crows, not some lesser crime than the accusations too-often leveled against the hard-working engineers at BitTorrent, Inc).  The distributed nature of BitTorrent means an operation that once took many dozens of minutes, now happens in less than a dozen seconds.  These efficiencies will reduce maintenance windows, site downtime and exposure to security vulnerabilities.

The folks at Twitter will be disclosing more about the project and the performance of this solution in the coming weeks.  We are thrilled to collaborate with them on this and hope that more Web Monsters out there look to this and other applications of BitTorrent in solving some of the hard problems of the Internet.

–Eric–

The Internet Civil Rights Act of 2009

by eklinker on October 15, 2009 in Other with No comments

I was recently invited to participate in a workshop sponsored by the GIIC, an organization of telecom and technology executives who ponder large scale Internet and information infrastructure questions.   The purpose of this workshop was to consider changes to the Internet infrastructure that would allow cost transparency, an upgrade to the Internet platform that could have wide ranging implications for the economic models that currently prevail online.

While that topic is worthy of exploration on its own, some of the many areas explored during this particular gathering were in matters of policy and the recently announced rulemaking proposal by the FCC towards a principle of network neutrality.  And throughout the many discussions, it became apparent that those on the “market” side of this debate have a very difficult challenge ahead of them:  How can they frame the debate in a way that doesn’t have them come across as online versions of Strom Thurmond?  This is no small challenge given the basic issues of equality are wrapped into our national identity and any implications of inequality tug at the strings of that identity.   So as a starter, I’d suggest attempts at branding users (customers) as bandwidth hogs will not yield a useful approach toward this end.  Does society favor discrimination against those who over-eat? Even on an increasingly green political landscape, any definition of a “reasonable consumption level” is going to be, to say the very least, “sensitive”.

Some defensible ground remains in the areas of network congestion if one can stake out clear and reasonable technical arguments and actions defending those principles.  Being the target of a great many technologies currently deployed to shape, block and throttle (i.e. discriminate) it would be easy for BitTorrent to assume the role of victim in our little analogy.  I suspect there is a great deal of mileage to be had in this approach amid the fury of debate.  But this isn’t our plan.  Instead, we’ve spent considerable energy developing technologies in µTP to combat the underlying premise of discrimination, Internet congestion.    And while it remains to be seen how the market will react to these developments, whether the DPI currently deployed will be modified to discriminate against µTP as well, there is an opportunity for one side of the debate to stand behind their principles and in so doing demonstrate the potential of self regulation to the benefit of all.

–Eric–

Net Neutrality (CRTC-style)

by eklinker on October 5, 2009 in Other with 1 Comment

In light of the FCC’s recently proposed rulemaking around Network Neutrality, many of you might have missed a similarly lively debate in Canada a few weeks back around the traffic management practices of Canadian ISPs. Over a week of public hearings, there were some astonishing revelations around the practice of throttling BitTorrent and other P2P traffic. Michael Geist’s excellent blog covered the events in detail and of course you can get the legalese of each submission from the CRTC website here.

Since congestion was the primary justification for throttling among the ISPs, it was the perfect opportunity in our own filing to showcase μTP (“micro-transport protocol) as the solution to these problems as well generally educate the commission on the benefits and efficiencies inherent in P2P technology. If congestion is in fact the problem, μTP effectively removes that rationale for BitTorrent throttling while demonstrating the ability of good old-fashioned innovation to solve the hard problems on the Internet.

 

Many of the ISP filings ask the regulators to allow the market to work its magic (“don’t regulate us”). The irony of this position cannot be lost on those of us who must compete and reach consumers on these networks, and yet are subject to ad-hoc traffic management policies.  Traffic management by its very nature can skew the marketplace in favor (or to the disfavor) of any technology or business.   In recognizing the need to manage congestion on these networks, these traffic management technologies should therefore be very even-handedly applied.

We believe μTP will solve the ISPs’ greatest problems around congestion, and we would ask and hope that the ISPs will stand behind their market oriented principles (which we share as our daily reality) and refrain from throttling P2P traffic using μTP.  It would be a demonstration of good faith to the market and the power of engineers and innovation to solve a myriad of problems.

– Eric –

Newer Entries →