I started getting multiple calls about issues on Comcast this afternoon.  My initial research showed it was not an IP issue, however dns was not responding on the Comcast network.  Comcast has had DNS outages before.  I tested my network by switching to Google DNS servers and i had most…but not all…of my connectivity issues resolved.  I then saw weird routing problems showing..sometimes to Google and others.  I also noticed trying to get to some of my more distant clients would jsut vanish..no packet traffic when exiting the Comcast network.  Traffic inside the Comcast network was fine.

Some more digging found that my Alaska client had zero connectivity from Comcast to their location(which is not Comcast).  I hopped onto another client’s network that does NOT use Comcast(aka my phones LTE hotspot) and i was still not able to get to this client.  I ran a trace to that client..and i noticed the issue.  Their ISP buys Tier 1 transport services not from Comcast…but from Level 3 Networks.  I also know(through years of network mapping) that Comcast prefers to use Level 3 for their tier 1 transport(when they do not have a direct peering connection).  I then tested my ability to reach my webserver from my office.  My datacenter uses Comcast as their transport to Inforelay carrier hotel which then goes to them.  I could reach my webserver without any trouble.  2 and 2 now became almost 4.  I then checked some other sites and found a status tweet from Comcast that said they are experiencing an “external” issue.  That external issue was Level 3 going down.  Comcast is a huge presence on the Internet in many areas.  At first I thought it was a Comcast issue…this was actually a Level 3 networks issue.  What happened was something called a    “route leak” and this is not the first time Level 3 has effectively taken themselves(and a host of other folks) offline.

If you want more technical details I’ll throw up a video on my YouTube channel and link it.