We run One9x on our own metal. That means every problem the cloud normally hides from you shows up on your desk, and you get to decide how to solve it.
Here’s one of them: the address the world points at has to stay fixed, and the fleet answering on it must not. That address is the whole contract with the outside world — it’s what DNS resolves to, what certificates are issued against, and what nobody should ever have to renegotiate. The gateways behind it are the opposite: they come and go with deploys, maintenance, and traffic spikes. Traffic needs to spread across whichever ones are currently alive, and stop going to the ones that aren’t.
The standard answer is to add a system. We ended up not building one.
The obvious approach, and why it bothered me
Search this problem and you get one of two answers.
The first is a managed load balancer. Fine if you’re on AWS, useless if you’re not, and the whole point of One9x is that we’re not.
The second is to build the thing yourself: run a load balancer, put a service registry next to it so the balancer knows what the current set of backends is, and run a health-check daemon so the registry knows which backends are actually alive. Consul, etcd, or something in that family, plus glue.
That’s three moving parts to solve one problem. And each one is a new failure mode. If the registry partitions, your load balancer’s view of the world goes stale. If the health checker is slow, you keep sending traffic to a dead gateway for ten or fifteen seconds. If the health checker is fast, you flap. You’ve now got a distributed systems problem sitting on top of what started as a routing problem.
The part that nagged at me: I was about to build a system whose entire job is to maintain a live list of peers and notice when one goes quiet.
That’s a solved problem. It’s been solved since 1994. It’s called BGP.
The insight
Strip BGP down to what it actually gives you:
- Session establishment. A peer connects and says “I’m here, and here’s what I can reach.”
- Keepalives. The peer has to keep proving it’s there. Go quiet past the hold timer and the session tears down on its own.
- Explicit withdrawal. A peer can say “stop sending me this” without dying.
- Peer state. The receiving end maintains an authoritative, continuously-updated view of who’s currently reachable.
- ECMP. Multiple peers announcing the same prefix means the kernel spreads traffic across all of them.
Read that list again with the word “registry” in your head instead of “router.” Session establishment is registration. Keepalives are health checks. Withdrawal is graceful deregistration. Peer state is the service registry. And ECMP is the load balancing you were going to install a load balancer for.
BGP isn’t like a service registry with health checking. It’s the same primitives with decades of production hardening and an RFC.
None of which is novel, and I’d rather say so than have it said to me. MetalLB in BGP mode is this exact pattern — announce a service VIP from every node, let ECMP spread the traffic — and Calico speaks BGP by default, so if you run Kubernetes you may already be doing this without calling it that. What I haven’t seen argued much is where the announcement should live, and that’s the rest of this post.
How it actually works
The shape is simple.
Whatever holds the public IP speaks BGP, and it does one job: listen for sessions from the gateways and program its forwarding table from what they announce. For us that’s a Linux box running BIRD. It could just as well be your router — bgp listen range <prefix> on Cisco, Arista or FRR, allow <prefix> in a JunOS group, is the same dynamic-membership primitive under a different name, and nothing on the gateway side changes by a line.
That portability is the argument, not a footnote. These are protocol primitives, not Linux ones, which is why the same design drops onto a router — and on hardware you get things the Linux version doesn’t have: resilient ECMP as a config knob, and BFD in silicon for sub-second detection instead of a hold timer measured in seconds.
Every gateway holds the service VIP as a /32 on its loopback and announces that prefix, with itself as the next hop, over a session to that listener.
When two gateways are announcing, the kernel has two equal-cost paths and hashes flows across both. When a third comes up, it announces and starts taking traffic. When one goes away, it withdraws and stops receiving traffic — and if it dies hard without withdrawing, the hold timer expires and the route is pulled anyway.
There is no registry process. There is no health-check daemon. There is no load balancer. There’s a routing table, and it’s correct by construction, because the only way to be in it is to be actively maintaining a session that says you’re alive.
The parts that aren’t obvious
Four details do most of the work, and none of them are obvious from the shape above:
The public IP is DNAT’d to the VIP; the box routes, it doesn’t proxy. Public :80 and :443 DNAT to the VIP, the kernel forwards to whichever next hop the hash picks, and conntrack un-DNATs the replies on the way back — which only works if every gateway’s default route points at that box. No SNAT, no proxy, nothing that terminates a connection between the client and the gateway.
The NAT is worth understanding as an artifact of where the public address lives — on the routing box’s own interface, rather than routed through to the gateways — and not as part of the pattern. Given a routed prefix from your provider instead, the gateways announce the public prefix directly and the whole return-path problem evaporates: no DNAT, no conntrack, no default-route requirement, and two of the three sysctls below stop mattering. Forwarding becomes stateless, which is also what makes an HA pair at that layer trivial rather than a session-sync project.
The VIP lives outside the transit subnet. It’s a routed /32, never ARP’d on the wire. That is exactly what lets every gateway hold the same address simultaneously without a conflict — and it’s why the VIP goes on lo and never on a real NIC.
merge paths on and neighbor range are the two BIRD directives that matter. Without merge paths, one announcement wins and you have failover, not ECMP. neighbor range is what removes the static fleet list: any peer in the transit subnet may open a session, so you add a gateway by booting it and remove one by killing it. Pin the import and export filters to the single VIP prefix in both directions — a member can announce that one prefix and nothing else, and the listener advertises nothing back.
Three kernel settings, two of which fail silently:
net.ipv4.fib_multipath_hash_policy=1on the routing box. The default (0) is an L3 hash — source and destination IP — so traffic does spread. It spreads per client, though, not per connection: every connection from one source address lands on the same gateway. One busy client, or everyone arriving through a single carrier-grade NAT or corporate egress, concentrates on one gateway, and with few distinct clients the distribution is simply poor. Setting1hashes the full 5-tuple, which spreads by connection and still keeps a given connection pinned to one gateway for its life.rp_filter=2on the routing box and on every gateway. The VIP is onlowhile packets for it arrive on a different interface and replies are sourced from the VIP. Strict reverse-path filtering drops that asymmetry, and it drops it without telling you.ip_forward=1on the routing box only. The gateways terminate TCP; they never forward. Leave it off there.
The listener, in config
Documentation addresses throughout: transit subnet 192.0.2.0/24, the listener at 192.0.2.1, one gateway at 192.0.2.11, the service VIP 198.51.100.10 — deliberately outside the transit subnet, as above. ASNs are from the private range.
# /etc/bird/bird.conf — on the box that holds the public IP
log syslog all;
router id 192.0.2.1;
protocol device { scan time 5; }
protocol kernel {
ipv4 {
import none;
export filter { if net = 198.51.100.10/32 then accept; else reject; };
};
merge paths on; # N announcements -> one route with N nexthops
}
protocol bgp gateways {
local 192.0.2.1 as 64512;
neighbor range 192.0.2.0/24 as 64513; # no static fleet list
dynamic name "gw"; # sessions become gw1, gw2, ...
hold time 9; # keepalive derives as 3s
passive; # gateways initiate
graceful restart off; # a dead peer must LOSE its route
ipv4 {
import filter { if net = 198.51.100.10/32 then accept; else reject; };
export none; # never advertise anything back
};
}
That half never changes. It is the same config whether the gateways announce through a sidecar or speak BGP themselves. The gateway half is where there’s a decision to make, and it’s the subject of the next section.
The part I’d actually argue about: the announcer belongs inside the gateway
This is the decision I’d defend hardest, and it’s the one that most people push back on.
The BGP lifecycle should not be a sidecar. It should not be a separate agent watching the gateway from outside and reporting on its behalf. It should be part of the gateway service itself — the process that serves traffic is the process that decides whether it’s announcing.
The reason is that every external health checker is, fundamentally, a guess. It probes /healthz and infers. It can’t see that your connection pool is exhausted, or that you’re mid-way through loading config and not ready yet, or that a dependency you need is down. It gets a 200 and calls it healthy.
The service knows. So let the service decide. It withdraws when its own internal health signal goes bad, and it withdraws on shutdown before it stops accepting connections. Nothing has to notice anything. It removes itself.
The nice side effect is that “unhealthy” and “not announcing” become the same state instead of two states that have to be kept in sync. There’s no window where the registry thinks you’re up and you know you’re not.
Start here: the cheap version
You don’t have to write a BGP speaker to get any of this, and you shouldn’t start by writing one.
Run BIRD or FRR as a config-frozen sidecar on each gateway. It announces the VIP whenever it finds it on lo, so the service’s entire contribution is adding and removing that one address — two shell-outs to ip addr, not a protocol implementation.
# /etc/bird/bird.conf — on each gateway
router id 192.0.2.11;
protocol device { scan time 5; }
protocol direct { # picks up VIP/32 when the service adds it to lo
interface "lo";
ipv4 { import filter { if net = 198.51.100.10/32 then accept; else reject; }; };
}
protocol bgp listener {
local 192.0.2.11 as 64513;
neighbor 192.0.2.1 as 64512;
hold time 9;
graceful restart off;
ipv4 {
export filter { if net = 198.51.100.10/32 then accept; else reject; };
import none;
};
}
Then bind the sidecar’s lifecycle to the service, which is the whole trick:
# /etc/systemd/system/bird.service.d/override.conf
[Unit]
BindsTo=gateway.service # gateway stops or crashes -> bird stops -> session drops
Wants=gateway.service # and starts with it; BindsTo alone never starts anything
After=gateway.service
Wants is not optional and it is the thing everyone forgets: BindsTo only ever stops the bound unit. Without Wants, BIRD sits inactive after a reboot and the gateway serves nothing, because nothing ever announced it.
That’s the whole of it, and it buys the case that matters most: the process dies, the sidecar stops with it, the session drops, and the hold timer evicts the gateway. No BGP code in your application, nothing to health-check, nothing to keep in sync. If you take one thing from this post and change nothing else, take this.
What it cannot do is tell alive from healthy. Its only signal is whether your process exists. It cannot express “I’m running fine and I should not be taking traffic right now” — and every piece of advice below depends on exactly that sentence: withdraw-on-unhealthy, withdraw-then-drain, hysteresis on a flapping signal. Those are opinions the service has, and a lifecycle-bound sidecar structurally cannot carry them.
So start there. Move the announcer in-process — replacing both blocks above with code that lives in the service — when you want the service’s own judgement to count. Not as a refactor for tidiness, but because that is what turns withdrawal from a side effect of dying into a decision the service gets to make.
What this costs you
I’d be selling you something if I stopped here. This pattern has sharp edges, and most posts about it skip them.
ECMP rehashing will reset connections. This is the big one. When the set of announcing gateways changes, the kernel’s hash buckets get recomputed, and existing flows can land on a different gateway than the one holding their state. Those connections break. Adding a gateway disrupts traffic on the ones that were already fine. If you need this to be seamless you need either resilient/consistent hashing at the routing layer or a stateless forwarding layer that can redirect misrouted packets to the right backend — which is roughly what GitHub’s GLB does. On hardware, resilient ECMP is often just a config knob — one more reason to put this on a router if you have one. Whether plain multipath is good enough is a measurement, not an opinion: put a load generator on it, change membership, and count the resets. We wrote that as an acceptance test rather than arguing about it.
Default BGP timers are far too slow. Stock keepalive is 60s with a 180s hold. If you take that default, a hard-crashed gateway keeps receiving traffic for three minutes. We run a 9s hold, which BIRD turns into a 3s keepalive, so eviction is bounded at nine seconds. If you need better than that, run BFD alongside — nine seconds is a floor you choose, not one you’re stuck with.
Turn graceful restart off, explicitly. GR exists to preserve routes when a session drops, on the assumption the peer is restarting its control plane while still forwarding. That’s the exact opposite of what you want here: a dead gateway keeps its route installed. And this is not a setting you can leave alone — BIRD’s default is aware, which honours a peer’s GR capability. Write graceful restart off on both sides and check show protocols all rather than trusting the default.
Withdrawal is not draining. Pulling the route stops new flows. In-flight requests are still in flight. The order is: withdraw, wait out a drain window, then exit. Get it backwards — sleep first, withdraw second — and you spend the whole window still accepting new connections and then cut the in-flight ones anyway.
Flapping health needs hysteresis. A health signal that oscillates will announce and withdraw in a loop and thrash the routing table. Damp it. And note that a crashlooping service with Restart=always is itself a flap source, coming and going every couple of seconds — cap the restart burst.
The box holding the public IP is still a single point of failure. BGP solved gateway-level redundancy. It did not solve that. Two boxes with VRRP or a second announced path is the honest answer, and you should be clear-eyed about whether you’ve done it.
None of these are reasons not to do this. They’re the actual engineering work, and they’re a much smaller surface than operating a registry cluster.
The wider point
I don’t think this is really a post about BGP. The reflex it’s about is this one: an infrastructure question comes up, and the answer you reach for is a product. Need service discovery? Here’s a service discovery service. Need load balancing? Here’s a load balancer. Need health checks? There’s a checkbox for that. Every answer is a new system, and every new system is another thing that can be down at 3am.
So the question I keep coming back to, and the reason we ended up here: what does the protocol already give me for free?
More often than I expected, the answer is: most of it.