22 May 2012
It’s been incredible to see the interest in this story that has developed over the last few days. The response to my little hobby project (intended mostly to force myself to start writing blog posts again) has been pretty amazing to read. Comcast responded to some “chatter” (which is PR-speak for this blog, I like to think) with first a defensive rebuttal and finally some real change for consumers, which is far more than I imagined would happen.
I’ll start with the positive changes. Comcast agreed to raise the monthly data cap on for all of their subscribers by 20%. This change is long overdue. Comcast introduced data caps in October of 2008 with a promise to reevaluate the caps periodically. Only now, nearly four years later, have they made good on that promise, and only by 20%. Much more important than the increase in data cap is the implementation of a concrete policy as to what happens when you exceed your cap. The previous policy was merely to merely cut off users who repeatedly exceeded their caps. The new policy (currently in “trials”) introduces usage-based pricing above the cap threshold. Ultimately, this is more fair; as a former network operator myself, I recognize the need to manage the data usage of individual customers. But given the importance of broadband, and Comcast’s near monopoly in many markets, Comcast shouldn’t be allowed to refuse service to customers just because they watched too many hours of Netflix video.
So, that’s the good news.
But this policy change doesn’t go far enough. Comcast still asserts that their Xfinity streaming service is exempt from the cap. By merely doing this, Comcast is treating their traffic differently from other video streaming traffic and is thus violating the terms of the consent decree. In the decree there are two conditions related to treatment of network traffic, neither dependent upon the other; not until the second condition is “prioritize” even used.** Just treating the traffic differently is sufficient to violate the agreement.** Comcast confirmed my assertions in their first blog post by indicating that: a) they use DSCP values to map IP traffic into separate DOCSIS service flows, and b) these flows are exempt from the rate limits imposed on other Internet traffic. Additionally, as I’ve demonstrated in my previous post, the traffic on these separate service flows uses the same RF channels as other traffic.
Comcast goes on to say (emphasis mine):
Specifically, we provision a separate, additional bandwidth flow into the home for the use of this service — above and beyond, and distinct from, the bandwidth a customer has for his or her regular Internet access service. Our Xfinity TV content is provided through the Xbox over that separate service flow, and therefore does not use a customer’s provisioned Internet service capacity.
Let’s talk about limits versus capacity. The amount of capacity provisioned to my neighborhood is the sum of the bandwidth of each DOCSIS channel visible to my cable modem. Given the number of homes this passes—probably in the hundreds—this capacity is massively oversubscribed. That’s absolutely fine, of course; in order for this network to be financially viable, oversubscription is a given. To manage this, Comcast artificially limits the bandwidth available to my IP address to 25 Mbit/s (which is what is standard with my Blast service tier.) This is merely the CMTS imposing a logical limit on the amount of traffic that can pass to my home in any given second.
On the other hand, when using the Xfinity app on my Xbox 360, this limit does not apply. When Comcast says that the Xbox traffic “does not use a customer’s provisioned Internet service capacity,” they are, at best, stretching the truth. The Xbox traffic does not apply to my limit, but no additional capacity is made available. The same amount of capacity is used to provide service to the Xfinity Xbox app as would be used by an equivalent stream being consumed by the Netflix Xbox app, even if the limiting behavior is different. Before the introduction of the Xfinity Xbox app, all of this capacity was available for Internet service.
This distinction is important when the discussion comes to prioritization. In general, prioritization is impossible to clearly observe in the absence of congestion. My previous test clearly showed that Comcast was treating Xfinity traffic differently than Netflix traffic in terms of the artificial limit. I was unable (unwilling, rather) to create congestion for my entire neighborhood, and thus the treatment of this traffic in the face of actual downstream congestion is as yet undetermined. I believe this very narrow definition is what Comcast is using when they say they aren’t prioritizing traffic. Because I can’t measure this, I can’t verify or disprove this claim, though I remain skeptical as to whether that is the case.
Just as a note, I’ve enabled comments only for verified users. The comments section was taking a turn towards a place I was not willing to let it go. I won’t tolerate comments which attack others.
14 May 2012
Even if you don’t make it all the way through this long post, take a look at the graphs below. I think they demonstrate reasonable evidence of the prioritization of traffic in violation of the terms of the Comcast/NBCU Consent Decree.
—
Thank you to everyone who commented on my previous post here and elsewhere. Much of the feedback I received was along a few related lines:
- The DiffServ field (which contains a packet’s DSCP bits) is defined in RFC 2474. This RFC does not prescribe any specific treatment or priority to be associated with each DSCP value, so the specific values themselves do not necessarily indicate prioritization.
- Many transit providers’ routers ignore DSCP values when making routing and queuing decisions.
- DOCSIS QoS is much more complex and dynamic than the standard IP QoS features available on most routers. Under ideal conditions, DOCSIS 3.0 offers the ability to segregate specific traffic flows onto dedicated RF spectrum, so it’s possible Comcast isn’t sharing any RF spectrum between their supposed Xbox VoD service and regular Internet traffic.
All of these points are true. Comcast is the United States’ largest MSO. They operate a national IP network over which nearly all of the services they provide are delivered. Comcast openly admits to prioritizing certain kinds of traffic—like their digital voice product. It’s clear that some configuration is in place to do so. And their practice of rewriting DSCP values at the edge of their network is a strong indicator that those values have meaning to Comcast’s routers.
I also omitted any discussion of DOCSIS QoS in my previous post. My theory was that certain DSCP values signal the CMTS to map packets into different DOCSIS QoS classes. In addition, one reader of my previous post, HN commenter tonyb, suggested that Comcast might be using PacketCable Multimedia to dynamically prioritize streams and rightly pointed out that DOCSIS 3.0 offered the ability to segregate certain traffic flows onto dedicated RF spectrum. I decided it was time to dig into the CableLabs specifications to get answers and devise a set of experiments to determine whether my hypotheses were correct.
What I’ve concluded is that Comcast is using separate DOCSIS service flows to prioritize the traffic to the Xfinity Xbox app (so that I’m using consistent terminology, I’m going to call this traffic “Xfinity traffic” in the rest of the post). This separation allows them to exempt that traffic from both bandwidth cap accounting and download speed limits. It’s still plain-old HTTP delivering MP4-encoded video files, just like the other streaming services use, but additional priority is granted to the Xfinity traffic at the DOCSIS level. I still believe that DSCP values I observed in the packet headers of Xfinity traffic is the method by which Comcast signals that traffic is to be prioritized, both in their backbone and regional networks and their DOCSIS network. In addition, contrary to what has been widely speculated, the Xfinity traffic is not delivered via separate, dedicated downstream channel(s)—it uses the same downstream channels as regular Internet traffic.
As my argument about Comcast’s cost basis between streams delivered by third-party CDNs and Comcast’s “internal” CDN, this comment thread from HN commenter zbisch pointed out that I didn’t quite include enough context in my last post. To summarize: it’s rumored that all of the major CDNs (Akamai, Limelight and post-interconnection-dispute Level 3) actually pay Comcast to carry their traffic to end users. The capital and operational expenses of operating the streaming servers are borne by the third-party CDNs. The last-mile capacity required is bit-for-bit equal between third-party CDN and internal CDN. To deliver a stream from Seattle, Comcast has to haul the bits further and through more expensive router ports than a stream delivered from a third-party CDN. Hence my conclusion that, from a financial perspective, it is not cheaper for Comcast to deliver streams from their internal CDN.
Background—DOCSIS downstream QoS
Since DOCSIS 1.1, the cable standard has supported “service flows” per cable modem. A service flow is a class of traffic with an assigned priority. Until DOCSIS 3.0 was introduced, a cable modem could only attach to one 6 MHz downstream channel at a time. Prior to this, all service flows shared the same RF channel, but the CMTS would prefer higher priority service flows when scheduling downstream bandwidth. DOCSIS 3.0’s high speeds were enabled by allowing downstream modems to bond several channels together at one time in Downstream Bonding Groups. In addition, the DOCSIS 3.0 specification allows for cable operators to map certain service flows to each downstream bonding group. So in the case where both the CMTS and cable modem support DOCSIS 3.0, it is possible to dedicate specific, full 6 MHz downstream channels to certain classes of traffic. As I will demonstrate below, Comcast has not configured the service flows which carry Xfinity traffic to use dedicated RF spectrum.
Background—Packet Cable Multimedia (PCMM)
So how does the Comcast network know how to prioritize these streams?
In a 2008 filing with the FCC in which Comcast detailed the implementation of its network management practices, it disclosed the deployment of PacketCable Multimedia (PCMM) throughout its High-Speed Internet (HSI) network. PCMM is a technology implemented in both the CMTS and a policy server which allows for very fine-grained dynamic adjustment of priority of traffic. The FCC filing disclosed the use of aggregated IPDR information as a signal to deprioritize ALL traffic to a congestion-causing end user. But instead of signaling traffic as high priority by using DSCP bits in IP headers, cable operator-blessed applications can request prioritization of specific traffic by making communicating with the PCMM policy server. The PCMM server does this by pushing a “gate” to the customer’s CMTS. Each gate is a configuration record which maps a specific set of classifiers (which can be things like source IP address, source port, etc.) to a DOCSIS service flow.
While my observations lead me to believe that Comcast is mapping traffic into separate service flows solely based upon DSCP values, it is possible that they have integrated the PCMM infrastructure they built for network management into this streaming service. It’s also possible that they have deployed a DPI solution for observing and prioritizing their own video traffic, but that seems as if it would be inefficient and unreliable.
Observations
1. Xfinity traffic uses the same RF spectrum as public Internet traffic, thus it is not “separated” for the purposes of prioritization
The DOCSIS spec mandates that a simple customer-facing read-only management interface be made available at a well-known IP (which happens to be 192.168.100.1). The cable modem at my house, an Arris TM702G, exposes information via a tiny web server on this IP. (Try it, actually: If you’re on a cable connection right now, go to http://192.168.100.1/ and see your modem’s status.) One of the pieces of diagnostic information that my cable modem exposes is a count of the number of bytes received per downstream channel. Here’s a screenshot of that it looks like:
If Comcast was using dedicated downstream channels to deliver Xfinity traffic, only the “Octets” field(s) associated with those dedicated channels would increase when streaming video in the Xfinity Xbox app. Instead, like any other Internet traffic, the Xfinity traffic is directly observable to be balanced (approximately) evenly across all 4 of the downstream channels my modem has attached to.
I wrote a quick Python script to periodically poll my cable modem’s per-channel counters every five seconds and calculate the effective bandwidth of each channel. While running this script, I opened the Netflix app on my Xbox and started a stream. Then I closed the app, waited a little bit for usage to drop, then started a stream in the Xfinity app. Here’s a graph of the both the total bandwidth usage and individual downstream channel usage for both an Xfinity stream (top) and then a Netflix stream (bottom). As you can see, traffic usage was roughly equivalent and spread across each channel.
In the interest of being fully transparent, here’s the script I used to log the counter values. If you have a similar modem, you should be able to run this script and duplicate my results.
2. Xfinity traffic is prioritized over other Internet traffic and is exempt from rate limiting
I wanted to be able to observe Comcast’s traffic prioritization in action, but in the absence of actual traffic congestion, prioritization is impossible to observe, so I created a way to simulate heavy usage of my broadband connection. I wrote a short Python script which was able to saturate my Internet service’s bandwidth allocation by generating traffic—specifically by starting 24 simultaneous, repeated concurrent downloads of the an archive containing a copy of the Google Chrome web browser, direct from Google’s CDN. It’s important to note that because of TCP’s congestion avoidance algorithms, the simulated congestion did not affect anyone else’s broadband service, nor did it cause any kind of backbone congestion in Comcast’s network. It merely caused the traffic destined to my cable modem to be policed like any other download stream.
Why run 24 simultaneous downloads, then? In order to keep a single flow from unfairly using all of a downstream link’s bandwidth, a CMTS often will employ a fair-queuing algorithm to determine which packets to drop when a customer is exceeding their rate limit. This ensures that multiple flows of equal priority share the available downstream bandwidth equally. Because I have such a high service tier (download speed), simulating fewer downloads would not sufficiently degrade the available bandwidth for the purposes of observation. If I had a much lower service tier, fewer simultaneous downloads would be required.
With the congestion script written, I wrote a script to poll the interface counters of a managed switch via SNMP every second. I hooked the Xbox and the computer generating the download requests for the synthetic traffic up to separate Ethernet ports on this switch. From this script’s output, I was able to calculate both my total bandwidth usage and the amount of bandwidth used by the synthetic traffic (dark gray) and the video stream traffic (red) separately. I then tried to watch use Netflix and Xfinity streams while this synthetic traffic was being generated.
If I start a Netflix stream while the synthetic congestion is present, the video takes a long time to buffer and the quality is poor because of the contention for downstream bandwidth caused by the synthetic traffic. The plateau is me hitting my download speed rate limit. (My service tier is rated at 25 mbps. These measurements are of kbit/sec of actual Ethernet frames, not TCP payload. I believe that accounts for the difference.)
If I terminate the script generating the synthetic traffic (around 15:38:00), you’ll see that the bandwidth available to the Netflix app increases. When the Netflix app is not experiencing this congestion the video quality shows a corresponding improvement.
However, if I run the Xfinity app, the video quality is seemingly unaffected by the synthetic traffic. In addition, you’ll see that unlike the Netflix example, the amount of bandwidth available to Internet traffic is unaffected. Because the Xfinity traffic is mapped to a different DOCSIS service flow when prioritized, it is exempted from the downstream rate limit, so my total bandwidth usage stays consistently higher than my service tier would normally allow.
There’s one exception to the prioritization that I was able to find. It appears that Comcast distributes content to various CDN nodes inconsistently—after my original post went up, I began observing my content come from different nodes. I was finally seeing Comcast deliver content to me from local SF Bay Area CDN servers. All of this traffic was marked with the same CS5 DSCP value seen on the traffic from Seattle in my previous post, except for a single cache server in Burien, WA, a city outside Seattle. This one cache server was sending me traffic marked as CS1 (the same priority level as Netflix traffic), and *was* affected by the congestion caused by the synthetic traffic. This reinforces my conclusion that Comcast is using DSCP values to signal the prioritization of Xfinity at the DOCSIS MAC layer. Here’s a screenshot of one of the packets received, marked as CS1:
And here’s a similar graph as above. Since it’s not marked as CS5, it doesn’t get assigned to the same DOCSIS service flow and isn’t prioritized. Notice it looks more like the Netflix graph than the Xfinity one:
I’m willing to bet streams that come off this cache server count against your bandwidth cap, too (as the service flow ID will be wrong in the IPDR record that gets generated.)
Here’s the ugly script I wrote to log the counters to disk.
Conclusion
Hopefully these observations have served to provide some context for my previous post. If you have any constructive comments on the methodology used, I’d appreciate hearing them.
Thanks to my wonderful wife Lauren for putting up with me making a mess in the living room over the last week.
08 May 2012
From the official Comcast page justifying their bandwidth caps (emphasis mine):
Comcast has set a data consumption threshold (“threshold”) on monthly bandwidth consumption (including both upstream and downstream usage) by residential users of its high-speed Internet service (“XFINITY Internet”). Currently, that threshold allows a residential customer to send or receive up to 250 Gigabytes (GB) of data in a calendar month. This includes data in any form (including movies, photos, music, videos, e-mails, computer back-ups, or other types of files) that a customer uses his or her XFINITY Internet to send or receive over the public Internet, including data sent by one XFINITY Internet customer to another.
So a FaceTime call from my house to my neighbor’s—which never leaves even the SF metro area Comcast network, given that both of us are Comcast customers—goes over the “public Internet.”
Yet Comcast’s Xbox streams, which pass from Seattle to Sacramento to San Francisco through all of the same network elements that handle my video call (and then some!) are exempt from the bandwidth cap?
You can’t have it both ways, guys.
06 May 2012
UPDATE: I’ve added additional data and observations to a new post here.
—
When the Comcast/NBC Universal merger was approved by the Department of Justice, Comcast agreed to two interesting stipulations regarding the neutrality of their residential broadband service:
- If Comcast provided capped broadband service, it would not exempt any traffic from the cap, and
- it would not prioritize its own video streaming services over services provided by any third party.
Comcast recently announced an app for Xbox 360 for streaming content from its Xfinity TV service. The difference between this service and the others is that the Xfinity Xbox app is exempt from the monthly 250 GB bandwidth cap applied to Xfinity residential Internet service. While Comcast claims that the app “essentially acts as an additional cable box,” instead of connecting directly to a cable outlet in your home, the content is streamed over your Internet service, through your cable modem and router. I became curious about exactly how this service was implemented, given the Xbox is no different than any other device on your home network—to the outside world, it’s got the same NATted IP address as the rest of your devices. What supposedly differentiates it in implementation from Netflix and other video streaming services? I set up some equipment to passively capture the traffic between my Xbox 360 and Comcast’s streaming service, and for comparison, I then captured similar streams from the Netflix and HBO GO apps on the Xbox 360. Both of these apps count against my monthly bandwidth cap.
Background
Internet video services such as Hulu, Netflix, HBO GO, even Comcast’s own web streaming options, use third-party content delivery networks (CDN) to stream the vast majority of their content. In order to use as few network resources as possible—both to keep costs low and quality high—these providers typically attempt to get as “close” as possible to end users, often forging agreements in which they directly connect to provider networks. Typically, this results in reduced usage of backbone network resources for the cable provider. Despite this reduction, traffic receives the same prioritization as general Internet traffic. That’s how it should be—packets delivered to the Ethernet jack on my cable modem should be delivered with equal priority, regardless of whether they came from a Comcast video streaming service or a third-party one.
So how does traffic prioritization actually work? Each IP packet contains 6 bits of information in its header (the Differentiated Services Code Point, or DSCP bits) which effectively assign it a priority. These 6 bits allow for up to 64 possible different classes of traffic—think of a class as a priority levels—but most modern IP networks map these 64 potential classes into somewhere between 3 and 5 traffic classes. When each router along a path is making its forwarding decision, each IP packet is mapped into a queue based upon the DSCP bits. Because IP provides no mechanism to ensure that the network operator themselves set these bits, operators generally rewrite—change—these bits to a common value at the point of ingress from a third-party network, effectively establishing a baseline for Internet traffic. This rewriting is key to what is happening here.
Observations
Both the Xfinity app and HBO GO app are implemented using almost identical technology, from a client perspective. They both make use of a Microsoft technology called “Smooth Streaming,” using files specifically created for the Xbox 360, delivered via plain-old HTTP over IP. Netflix uses a very similar technique, streaming a proprietary container format via HTTP. The only appreciable differences between the Xfinity streaming service for Xbox and e.g., Netflix, are that the source of content is within the Comcast’s “internal” CDN instead of on a third-party CDN, and that Comcast requires you to be using their own Internet service. (This is much more likely related to their agreements with content owners rather than any technical reason.) As you’ll see, the cap-exempt content is likely even more expensive for Comcast to deliver than the third party content!
Let’s tie this to a concrete example. I want to stream a video file from a server on the Internet to my home broadband (I’m a Comcast customer in San Francisco.) In this example, I’ve used the public domain MPEG-4 video file of President Obama’s recent address at the White House Correspondents’ Dinner. I configure my server to set the DSCP bits for this stream to CS5 (101000). You’ll see why I chose this DSCP value in a moment.
Here’s a screenshot of the relevant bits of a packet capture of the TCP stream, captured at my server in Mountain View (so not yet subject to DSCP rewriting by Comcast):
Here’s a screenshot of the same packet in the TCP stream, captured at my house on my Comcast connection. Notice that the DSCP bits have been rewritten to CS1 (001000):
Now here’s the stream which doesn’t count against my bandwidth cap. Below is a screenshot of a packet capture of a TCP stream, captured at my house on my Comcast service, between by my Xbox 360 and the streaming infrastructure that supports the Xfinity app:
Here’s where it gets interesting. Note that this stream originates within the Comcast network (69.252.116.50, which resolves to se01-seattle-wa-seattle.se.omg-01.cdn2.comcast.net), and the packets arrive marked CS5, thus they traverse the Comcast network without being rewritten—I couldn’t get Comcast’s routers to honor my request for delivery at the same priority as their video streaming service. This is the stream which doesn’t count against my bandwidth cap.
For reference, here are similar captures from Comcast’s in-browser video streaming app. This stream, even though it’s part of the Xfinity service, does count against my bandwidth cap, and was served from 96.17.148.112, an Akamai node located in Palo Alto, and marked as CS1:
Similar capture from HBO GO (same deal–this one counts. It was served by 8.26.198.253, which is a Level(3) CDN node in San Jose). Note that packets get marked CS1:
Similar capture from Netflix (served from 208.111.173.160, which is a Limelight node in San Jose, again marked CS1):
All of these third-party streams almost certainly originate from third party providers in the Bay Area, all via direct connections to Comcast. Even though they count against my bandwidth cap, they almost certainly traverse fewer fiber route-miles and physical router ports (Comcast’s two primary costs of delivery) than the stream which originated in Seattle(!) and does not count against my cap. The only DSCP values I’ve observed inside of Comcast’s network on traffic from the Internet are CS0 and CS1, both of which are typically low-priority classifications. If Comcast were truly treating streams from their internal CDN as neutral traffic, they would apply the same DSCP remarking to that traffic.
(As an aside, I’m willing to bet this is also method by which they mark packets to be excluded from the software that counts your bandwidth usage. The traffic travels along many of the same paths as Internet service, but is accounted for differently by virtue of some magic header bits. This is not neutral.)
The bottom line: Comcast built an Internet video streaming service. In certain cases, it exempted that service from bandwidth caps despite evidence that those streams are actually more expensive to deliver. It even appears that Comcast is prioritizing its own video streams over the other services.
Unfortunately, without having visibility into the configuration of any Comcast network elements, it’s impossible to actually tell what the mapping between DSCP values and router queues actually is. But by selectively remarking traffic destined to my broadband service, it is implicitly being treated differently. And given the implementation, I’m not sure what possible argument Comcast could give to justify exempting the Xbox app from its bandwidth caps. Given this, it’s worth further investigation as to whether Comcast is in compliance with its obligations as part of its acquisition of NBC Universal.
I would urge the third-party video services to push for an independent audit of the configuration of Comcast’s network elements to determine compliance.