Video streaming & net neutrality

06 May 2012

UPDATE: I’ve added additional data and observations to a new post here.

—

When the Comcast/NBC Universal merger was approved by the Department of Justice, Comcast agreed to two interesting stipulations regarding the neutrality of their residential broadband service:

1. If Comcast provided capped broadband service, it would not exempt any traffic from the cap, and
2. it would not prioritize its own video streaming services over services provided by any third party.

Comcast recently announced an app for Xbox 360 for streaming content from its Xfinity TV service. The difference between this service and the others is that the Xfinity Xbox app is exempt from the monthly 250 GB bandwidth cap applied to Xfinity residential Internet service. While Comcast claims that the app “essentially acts as an additional cable box,” instead of connecting directly to a cable outlet in your home, the content is streamed over your Internet service, through your cable modem and router. I became curious about exactly how this service was implemented, given the Xbox is no different than any other device on your home network—to the outside world, it’s got the same NATted IP address as the rest of your devices. What supposedly differentiates it in implementation from Netflix and other video streaming services? I set up some equipment to passively capture the traffic between my Xbox 360 and Comcast’s streaming service, and for comparison, I then captured similar streams from the Netflix and HBO GO apps on the Xbox 360. Both of these apps count against my monthly bandwidth cap.

Background

Internet video services such as Hulu, Netflix, HBO GO, even Comcast’s own web streaming options, use third-party content delivery networks (CDN) to stream the vast majority of their content. In order to use as few network resources as possible—both to keep costs low and quality high—these providers typically attempt to get as “close” as possible to end users, often forging agreements in which they directly connect to provider networks. Typically, this results in reduced usage of backbone network resources for the cable provider. Despite this reduction, traffic receives the same prioritization as general Internet traffic. That’s how it should be—packets delivered to the Ethernet jack on my cable modem should be delivered with equal priority, regardless of whether they came from a Comcast video streaming service or a third-party one.

So how does traffic prioritization actually work? Each IP packet contains 6 bits of information in its header (the Differentiated Services Code Point, or DSCP bits) which effectively assign it a priority. These 6 bits allow for up to 64 possible different classes of traffic—think of a class as a priority levels—but most modern IP networks map these 64 potential classes into somewhere between 3 and 5 traffic classes. When each router along a path is making its forwarding decision, each IP packet is mapped into a queue based upon the DSCP bits. Because IP provides no mechanism to ensure that the network operator themselves set these bits, operators generally rewrite—change—these bits to a common value at the point of ingress from a third-party network, effectively establishing a baseline for Internet traffic. This rewriting is key to what is happening here.

Observations

Both the Xfinity app and HBO GO app are implemented using almost identical technology, from a client perspective. They both make use of a Microsoft technology called “Smooth Streaming,” using files specifically created for the Xbox 360, delivered via plain-old HTTP over IP. Netflix uses a very similar technique, streaming a proprietary container format via HTTP. The only appreciable differences between the Xfinity streaming service for Xbox and e.g., Netflix, are that the source of content is within the Comcast’s “internal” CDN  instead of on a third-party CDN, and that Comcast requires you to be using their own Internet service. (This is much more likely related to their agreements with content owners rather than any technical reason.) As you’ll see, the cap-exempt content is likely even more expensive for Comcast to deliver than the third party content!

Let’s tie this to a concrete example. I want to stream a video file from a server on the Internet to my home broadband (I’m a Comcast customer in San Francisco.) In this example, I’ve used the public domain MPEG-4 video file of President Obama’s recent address at the White House Correspondents’ Dinner.  I configure my server to set the DSCP bits for this stream to CS5 (101000). You’ll see why I chose this DSCP value in a moment.

Here’s a screenshot of the relevant bits of a packet capture of the TCP stream, captured at my server in Mountain View (so not yet subject to DSCP rewriting by Comcast):

Here’s a screenshot of the same packet in the TCP stream, captured at my house on my Comcast connection. Notice that the DSCP bits have been rewritten to CS1 (001000):

Now here’s the stream which doesn’t count against my bandwidth cap. Below is a screenshot of a packet capture of a TCP stream, captured at my house on my Comcast service, between by my Xbox 360 and the streaming infrastructure that supports the Xfinity app:

Here’s where it gets interesting. Note that this stream originates within the Comcast network (69.252.116.50, which resolves to se01-seattle-wa-seattle.se.omg-01.cdn2.comcast.net), and the packets arrive marked CS5, thus they traverse the Comcast network without being rewritten—I couldn’t get Comcast’s routers to honor my request for delivery at the same priority as their video streaming service. This is the stream which doesn’t count against my bandwidth cap.

For reference, here are similar captures from Comcast’s in-browser video streaming app. This stream, even though it’s part of the Xfinity service, does count against my bandwidth cap, and was served from 96.17.148.112, an Akamai node located in Palo Alto, and marked as CS1:

Similar capture from HBO GO (same deal–this one counts. It was served by 8.26.198.253, which is a Level(3) CDN node in San Jose). Note that packets get marked CS1:

Similar capture from Netflix (served from 208.111.173.160, which is a Limelight node in San Jose, again marked CS1):

All of these third-party streams almost certainly originate from third party providers in the Bay Area, all via direct connections to Comcast. Even though they count against my bandwidth cap, they almost certainly traverse fewer fiber route-miles and physical router ports (Comcast’s two primary costs of delivery) than the stream which originated in Seattle(!) and does not count against my cap. The only DSCP values I’ve observed inside of Comcast’s network on traffic from the Internet are CS0 and CS1, both of which are typically low-priority classifications. If Comcast were truly treating streams from their internal CDN as neutral traffic, they would apply the same DSCP remarking to that traffic.

(As an aside, I’m willing to bet this is also method by which they mark packets to be excluded from the software that counts your bandwidth usage. The traffic travels along many of the same paths as Internet service, but is accounted for differently by virtue of some magic header bits. This is not neutral.)

The bottom line: Comcast built an Internet video streaming service. In certain cases, it exempted that service from bandwidth caps despite evidence that those streams are actually more expensive to deliver. It even appears that Comcast is prioritizing its own video streams over the other services.

Unfortunately, without having visibility into the configuration of any Comcast network elements, it’s impossible to actually tell what the mapping between DSCP values and router queues actually is. But by selectively remarking traffic destined to my broadband service, it is implicitly being treated differently. And given the implementation, I’m not sure what possible argument Comcast could give to justify exempting the Xbox app from its bandwidth caps. Given this, it’s worth further investigation as to whether Comcast is in compliance with its obligations as part of its acquisition of NBC Universal.

I would urge the third-party video services to push for an independent audit of the configuration of Comcast’s network elements to determine compliance.