Last week, approximately 1,000 of our customers experienced inbound and outbound calling issues due to a problem on the Comcast network. This issue affected inbound and outbound calling for most, if not all, Comcast customers using OnSIP. Per procedure, we kept everyone in the loop with network alert updates via Twitter, Facebook, and the blog. We were happy to receive lots of feedback and quickly learn it was a Comcast routing issue. Here’s a bit more detail on the event and what you can do to prevent your Internet provider’s routing issues from affecting your phone service in the future.
What Happened
Comcast experienced a problem with their connectivity to certain networks on the Internet; these were the same networks that lie between their network and the OnSIP network. As a result, our multiple Internet Service Providers (ISPs) were unable to communicate with the Comcast network, and our customers using Comcast were thus unable to reach our servers.
Once Comcast fixed the issue, the Internet healed itself and these customers came back online. More technically, we normally have three possible routes to the Comcast network via three of our providers, XO, NTT and Level3. XO and NTT both route through Tata Communications to reach Comcast. When the problem between Tata and Comcast arose, neither provider could reach Comcast.
Our third possible route, via Level 3, has been depreferenced so severely by Comcast that BGP (the routing protocol that runs the Internet) preferred the broken routes through Tata Communications. Eventually, XO completely removed Comcast from their BGP announcements, so we used NTT exclusively until Comcast resolved the problem and normal routing was restored.
What You Can Do
The most surefire way to avoid ALL issues with your ISP that affect your VoIP service is to maintain a backup Internet service with an alternate provider. You can minimize costs by having your primary service be a business class service and your secondary a less pricey service.
For example, Michael Graves of http://www.mgraves.org does just this. “I normally route my phones via a Comcast Business Class connection. When the failure occurred this morning I routed the SIP traffic out my backup connection, which is Covad DSL. They came right back up,” said Graves. Special thanks to our customers who helped us uncover the issue and commented on our blog with suggestions.