Interesting approach. So Fastly is offloading the full routing table to their carriers' router(s). That's because routers that can hold full BGP tables are expensive to purchase and maintain. But to retain some form of control, they're terminating eBGP at the switch and using iBGP to disseminate (inject) the providers' route (next hop).
I feel (I don't have direct experience with this setup) like they're just offloading some compute power (therefore cost) to the hosts. So the cost is automatically spread out across their relatively massive edge nodes. A line showing a router plus support costing $100,000 looks bad in expenditures vs showing a server plus integrated routing showing $2500.
I'm curious about how this impacts Varnish considering how table look ups can be bus-expensive during odd route changes/flaps (storms). %sys must go through the roof as a result.
> So Fastly is offloading the full routing table to their carriers' router(s).
The carriers' routers will have full routing tables in any case, so Fastly is not really offloading anything. They just aren't downloading full routes to a big central router and doing routing decisions there, but rather at a host level.
> I feel (I don't have direct experience with this setup) like they're just offloading some compute power (therefore cost) to the hosts.
This appears to be the case. The routing part isn't very computationally expensive, the biggest problem at larger bitrates is moving the packets with software. But then again they are already limited by what their CDN nodes can push out, so the routing isn't really much more of a burden.
I didn't see him in the room, so that doesn't mean he wasn't there, but it sounds like Artur was paying attention to Dave Temkin's talk at NANOG (from Netlix).
Titled: "Help! My Big Expensive Router Is Really
Expensive!"
Netflix goes even cheaper/simpler and just uses default routes to a pair of transit providers. It may come as a shock to most of you, but no, Netflix is not 100% in AWS (compute, yes; network, oh hell no).
> they're just offloading some compute power (therefore cost) to the hosts
I don't know enough about networking to follow the tech, but they seem to say in the post that's exactly what they intended to do:
> The idea of dropping several millions of dollars on overly expensive networking hardware wasn’t particularly appealing to us. As systems engineers we’d much rather invest the money in commodity server hardware, which directly impacts how efficiently we can deliver content.
I feel (I don't have direct experience with this setup) like they're just offloading some compute power (therefore cost) to the hosts. So the cost is automatically spread out across their relatively massive edge nodes. A line showing a router plus support costing $100,000 looks bad in expenditures vs showing a server plus integrated routing showing $2500.
I'm curious about how this impacts Varnish considering how table look ups can be bus-expensive during odd route changes/flaps (storms). %sys must go through the roof as a result.