> It seems like it would be much simpler and faster if they just took a default route from each provider and then did route probing on their top 30% prefixes in each PoP.
Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet.
> FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point.
Different game indeed. Netflix just cares about throughput, Fastly also needs low latency, hence the need for more optimizing.
"Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet."
That's not how BGP works. There's no such thing as getting the best path from the first packet in terms of latency. There's nothing in BGP that tell you about the path in terms of latency or performance. The only "best" that BGP gives you is AS path length between you and the destination. It tells you nothing about congestion/packet loss upstream. The only way to get that is out of band probing, like taking latency measurements and pref'ing routes. There is no approach beyond "after the fact"
There is nothing in this article that talks about optimization of routes.
Of course not. I only stated what Fastly needs, I did not make a statement about how BGP works.
By having full BGP feeds from all transit providers the nodes can use the AS paths as part of the heuristics to determine the best possible path (with the available information) from first packet. Obviously the heuristics will also use after the fact probe information to tune the model.
The point is that the solution proposed by the grandparent is not better than the one described in the article.
> There is nothing in this article that talks about optimization of routes.
What you're talking about works OK in a single colo or datacenter model, but it doesn't map to the PoP model. It's the PoP model that's driving them to do this. In their PoPs they've got local peers and exchanges that will handle the bulk of their traffic. The leftover stuff is only local and probably minor. They can definitely find that with flow data and optimize across a pair of default routes (basically building their own routing table). It's a pretty common practice when all you deploy is PoPs. Why do you care about a full BGP feed when you're only handling 4-10% of the Internet in a given PoP?
AS path length doesn't tell you anything about a route's performance. It's a hint and only a hint.
> What you're talking about works OK in a single colo or datacenter model, but it doesn't map to the PoP model. It's the PoP model that's driving them to do this
Could you please explain how the PoP model in this case differs from the colo/DC model?
Outside the US Fastly's PoP cover multiple countries or whole continents.
> Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet. <
Considering that Fastly deploys PoPs and not colos or datacenters, I think that the top 30% of their transit routes would be the lionshare of non-peer activity. Fastly's PoPs are generally at (or have reach to) the largest Internet exchanges for peering purposes. Peering routes are generally the best, cheapest, most concise routes you can get. You don't need a big, expensive router when you're in an exchange's peering mesh. Considering that you're only handling local traffic (by design of having multiple PoPs), peering plus two default transits will probably optimize for the traffic that matters most.
Also as mentioned, you can't really determine "best possible path" without probing. You can see what BGP tells you, but that's not always the best possible path. People that run networks know this. That's why things like FCP/IRP/RouteScience exist (for better or worse).
> Different game indeed. Netflix just cares about throughput, Fastly also needs low latency, hence the need for more optimizing. <
Sure, but you won't get that optimization from BGP. In a Fastly PoP, you're mainly dealing with in-country or in-region eyeball networks. If you buy transit from 2-3 major networks, BGP probably won't help you with those eyeballs. The path length will probably be the same (the eyeballs either peer or buy transit from large tier 1s). You generally can't trust MED to do anything useful with it, so that's out. Those are the only 2 things BGP can use to impact route selection. If path length and MED are the same, it's a coin flip as to which path to take (not really as the first installed route is the winner).
If you just took default routes and peered locally, you'd be much better off in a PoP scenario than trying to re-invent route reflectors and taking full BGP feeds.
Probing just 30% of the top prefixes is likely not enough for their use case. Probing is also an after the fact approach. Fastly needs a solution which will give best possible path from first packet.
> FWIW, the Netflix networks just take a default route from a pair of transit networks in each PoP. At their scale, they found it better to negotiate better worldwide transit deals than optimize networking at the PoP level. Different game than Fastly, but another data point.
Different game indeed. Netflix just cares about throughput, Fastly also needs low latency, hence the need for more optimizing.