Karl has this one nailed. Just because Comcast is the bigger of the two doesn’t make it automatically wrong. L3 is trying to push the colocation costs of CDN traffic to Comcast and L3 got Called on it. Here’s a post on NANOG I found that very well illustrates what’s really going on:
Having been involved with a few peering spats in the past I know
what is said publically rarely matches the reality behind the scenes.
In this particular case my spidy sense tells me there is absolutely
something interesting behind the scenes, but the question is what.
I’d never really paid attention to how Netflix delivers its content.
It’s obviously a lot of bandwidth, and likely part of the issue
here so I thought I would investigate.
Apparently Akamai has been the primary Netflix streaming source
since March. LimeLight Networks has been a secondary provider, and
it would appear those two make up the vast majority of Netflix’s
actual streaming traffic. I can’t tell if Netflix does any streaming
out of their own ASN, but if they do it appears to be minor.
Here’s a reference from the business side of things:
http://www.fool.com/investing/general/20….
This is also part of the reason I went back to the very first message in
this thread to reply:
In a message written on Mon, Nov 29, 2010 at 05:28:18PM -0500, Patrick W. Gilmore wrote:
> > <http://www.marketwatch.com/story/level-3….
> >
> > I understand that politics is off-topic, but this policy affects operational aspects of the ‘Net.
Patrick works for Akamai, it seems likely he might know more about
what is going on. Likely he can’t discuss the details, but wanted
to seed a discussion. I’d say that worked well.
I happen to be a Comcast cable modem customer. Gooling for people
who had issues getting to Netflix streaming turned up plenty of
forum posts with traceroutes to Netflix servers on Akamai and
Limelight. I did traceroutes to about 20 of them from my cable
modem, and it’s clear Comcast and Akamai and Comcast and Limelight
are interconnected quite well. Akamai does not sell IP Transit,
and I’m thinking it is extremely unlikely that Comcast is buying
transit from Limelight. I will thus conclude that these are either
peering relationships, or that they have cut some sort of special
“CDN Interconnect” deal with Comcast.
But what about Level 3? One of my friends I was chatting with on AIM
said they thought Comcast was a Level 3 customer, at least at one time.
Google to the rescue again.
Level 3 provides fiber to Comcast (20 year deal in 2004):
http://blog.tmcnet.com/blog/tom-keating/….
Level 3 provides voice services/support to Comcast:
http://cable.tmcnet.com/news/2005/jul/11….
Perhaps the most interesting though is looking up an IP on Comcast’s
local network here in my city in L3’s looking glass:
http://lg.level3.net/bgp/bgp.cgi?site=sj….
Slightly reformatting for your viewing pleasure, along with my comments:
Community: North_America
Lclprf_100
Level3_Customer # Level 3 thinks they are a customer
United_States
San_Jose
EU_Suppress_to_Peers
Suppress_to_AS174 # Cogent
Suppress_to_AS1239 # Sprint
Suppress_to_AS1280 # ISC
Suppress_to_AS1299 # Telia
Suppress_to_AS1668 # AOL
Suppress_to_AS2828 # XO
Suppress_to_AS2914 # NTT
Suppress_to_AS3257 # TiNet
Suppress_to_AS3320 # DTAG
Suppress_to_AS3549 # GBLX
Suppress_to_AS3561 # Savvis
Suppress_to_AS3786 # LG DACOM
Suppress_to_AS4637 # Reach
Suppress_to_AS5511 # OpenTransit
Suppress_to_AS6453 # Tata
Suppress_to_AS6461 # AboveNet
Suppress_to_AS6762 # Seabone
Suppress_to_AS7018 # AT&T
Suppress_to_AS7132 # AT&T (ex SBC)
So it would appear Comcast is a transit customer of Level 3 (along with
buying a lot of other services from them). I’m going to speculate that
the list of supressed ASN’s are peers of both Level 3 and Comcast, and
Comcast is going that so those peers can’t send some traffic through
Level 3 in attempt to game the ratios on their direct connections to
Comcast.
Now a more interesting picture emerges. Let me emphasize that this is
AN EDUCATED GUESS on my part, and I can’t prove any of it.
Level 3 starts talking to Netflix, and offers them a sweetheart deal to
move traffic from Akamai to Level 3. Part of the reason they are
willing to go so low on the price to Netflix is they will get to double
dip by charging Netflix for the bits and charging Comcast for the bits,
since Comcast is a customer! But wait, they also get to triple dip,
they provide the long haul fiber to Comcast, so when Comcast needs more
capacity to get to the peering points to move the traffic that money
also goes back to Level 3! Patrick, from Akamai, is unhappy at losing
all the business, and/or bemused at the ruckus this will cause and
chooses to kick the hornets nest on NANOG.
One last thing, before we do some careful word parsing. CDN’s like
Akamai and LimeLight want to be close to the end user, and the
networks with end users want them to be close to the end user. It
doesn’t make sense to haul the bits across the country for any party
involved. Akamai does this by locating clusters inside providers
networks, LimeLight does it by provisioning bandwidth from their
data centers directly to distribution points on eyeball networks.
So let’s go back and look at the public statements now:
Level 3 said:
http://www.marketwatch.com/story/level-3….
-2010-11-29?reflink=MW_news_stmp
“On November 19, 2010, Comcast informed Level 3 that, for the first
time, it will demand a recurring fee from Level 3 to transmit Internet
online movies and other content to Comcast’s customers who request such
content.”
Comcast said:
http://blog.comcast.com/2010/11/comcast-….
“Comcast has long established and mutually acceptable commercial
arrangements with Level 3’s Content Delivery Network (CDN) competitors
in delivering the same types of traffic to our customers. Comcast
offered Level 3 the same terms it offers to Level 3’s CDN competitors
for the same traffic.”
You can make both of these statements make sense if the real situation
is that Comcast told Level 3 they needed to act like a CDN if they were
going to host Netflix. Rather than having Comcast pay as a customer,
they needed to show up in various Comcast distribution centers around
the country where they could drop traffic off “locally”. To Comcast
this is the same deal other CDN’s get, it matches their statement. To
Level 3, this means paying a fee for bandwidth to these locations, and
being that they are Comcast locations it may even mean colocation fees
or other charges directly to Comcast. Comcast said if you’re not going
to show up and do things like the CDN players then we’re going to hold
you to a reasonable peering policy, like we would anyone else making us
run the bits the old way.
The most interesting thing to me about all of this is these companies
clearly had a close relationship, fiber, voice, and IP transit all on
long term deals. If my speculation is right I’m a bit surprised Level 3
would choose to piss off such a long term large customer by bringing
Netflix to the party like this, which is one of the reasons I doubt my
speculation a bit.
But, to bring things full circle, neither of the public statements tell
the whole story here. They each tell one small nugget, the nugget that
side wants the press to run with so they can score political points.
Business is messy, and as I’ve said throughout this thread this isn’t
about peering policies or ratios, there are deeper business interests
on both sides. This article:
http://www.marketwatch.com/story/level-3….
Suggests Level 3 is adding 2.9 Terabits of capacity just for Netflix.
I’m sure a lot of that is going to Comcast users, since they are the
largest residential broadband ISP.
Messy. Very messy.
— Leo Bicknell – bicknell@ufp.org – CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/