Why We Chose a BGP Session Over a Service Registry

We run One9x on our own metal. That means every problem the cloud hides from you shows up on your desk.

Here’s one of them: the address the world points at has to stay fixed, and the fleet answering on it must not. That address is the contract with the outside world: DNS resolves to it, certificates are issued against it, and nobody should ever have to renegotiate it. The gateways behind it are the opposite: they come and go with deploys, maintenance, and traffic spikes. Traffic needs to spread across whichever ones are alive, and stop going to the ones that aren’t.

The standard answer is to add a system. We ended up not building one.

The obvious approach, and why it bothered me

Search this problem and you get one of two answers.

The first is a managed load balancer. Fine if you’re on AWS, useless if you’re not, and the whole point of One9x is that we’re not.

The second is to build the thing yourself: run a load balancer, put a service registry next to it so the balancer knows the current set of backends, and run a health-check daemon so the registry knows which backends are alive. Consul, etcd, or something in that family, plus glue.

That’s three moving parts for one problem, each a new failure mode. If the registry partitions, your load balancer’s view of the world goes stale. If the health checker is slow, you keep sending traffic to a dead gateway for ten or fifteen seconds. If the health checker is fast, you flap. You’ve now got a distributed systems problem on top of a routing problem.

The part that nagged at me: I was about to build a system whose job is to maintain a live list of peers and notice when one goes quiet.

That’s a solved problem. It’s been solved since 1994. It’s called BGP.

The insight

Strip BGP down to what it gives you:

• Session establishment. A peer connects and says “I’m here, and here’s what I can reach.”
• Keepalives. The peer has to keep proving it’s there. Go quiet past the hold timer and the session tears down on its own.
• Explicit withdrawal. A peer can say “stop sending me this” without dying.
• Peer state. The receiving end maintains an authoritative, live view of who’s reachable.
• ECMP. Multiple peers announcing the same prefix means the kernel spreads traffic across all of them.

Read that list again with the word “registry” in your head instead of “router.”

That last row is the loosest of the five.

Those are the same primitives, with decades of production hardening and an RFC.

None of which is novel, and I’d rather say so than have it said to me. MetalLB in BGP mode is this pattern (announce a service VIP from every node, let ECMP spread the traffic), and Calico speaks BGP by default, so if you run Kubernetes you may already be doing this without calling it that. What I haven’t seen argued much is where the announcement should live.

How it works

Whatever holds the public IP speaks BGP, and it does one job: listen for sessions from the gateways and program its forwarding table from what they announce. For us that’s a Linux box running BIRD. It could just as well be your router: bgp listen range <prefix> on Cisco, Arista or FRR, or allow <prefix> in a JunOS group, is the same dynamic-membership primitive, and nothing on the gateway side changes.

These are protocol primitives, not Linux ones, so the same design drops onto a router, and on hardware you get things the Linux version doesn’t: resilient ECMP as a config knob, and BFD in silicon for sub-second detection instead of a hold timer measured in seconds.

Every gateway holds the service VIP as a /32 on its loopback and announces that prefix, with itself as the next hop, over a session to that listener.

When two gateways are announcing, the kernel has two equal-cost paths and hashes flows across both. When a third comes up, it announces and starts taking traffic. When one goes away, it withdraws and stops receiving traffic. If it dies hard without withdrawing, the hold timer expires and the route is pulled anyway.

There is no registry process. There is no health-check daemon. There is no load balancer. There’s a routing table, and it’s correct by construction, because the only way to be in it is to be maintaining a session that says you’re alive.

The parts that aren’t obvious

Four details do most of the work:

The public IP is DNAT’d to the VIP; the box routes, it doesn’t proxy. Public :80 and :443 DNAT to the VIP, the kernel forwards to whichever next hop the hash picks, and conntrack un-DNATs the replies, which only works if every gateway’s default route points at that box. Nothing terminates a connection between client and gateway.

That NAT is an artifact of where the public address lives, not part of the pattern. Given a routed prefix instead, the gateways announce the public prefix directly and the return-path problem evaporates: no DNAT, no conntrack, no default-route requirement, and forwarding is stateless, which makes an HA pair at that layer trivial rather than a session-sync project.

The VIP lives outside the transit subnet. It’s a routed /32, never ARP’d on the wire. That is what lets every gateway hold the same address at once without a conflict, and why the VIP goes on lo and never on a real NIC.

merge paths on and neighbor range are the two BIRD directives that matter. Without merge paths, one announcement wins and you have failover, not ECMP. neighbor range removes the static fleet list: any peer in the transit subnet may open a session, so you add a gateway by booting it and remove one by killing it. Pin the import and export filters to the single VIP prefix in both directions: a member can announce that one prefix and nothing else, and the listener advertises nothing back.

Anything that can put a packet on the transit subnet can open a session and start taking production traffic. The filters bound what a peer may announce, not whether it may join. If the subnet isn’t trusted, add TCP-MD5 or GTSM.

Three kernel settings, two of which fail silently:

• net.ipv4.fib_multipath_hash_policy=1 on the routing box. The default (0) is an L3 hash on source and destination IP, so traffic spreads per client, not per connection: one busy client, or everyone behind a single CGNAT or corporate egress, lands on one gateway, and with few distinct clients the spread is poor. 1 hashes the full 5-tuple: spreads by connection, still pins a connection to one gateway for its life.
• rp_filter=2 on the routing box and on every gateway, as insurance, and not for the reason usually given. Reverse-path filtering checks a packet’s source, never its destination, so the VIP on lo is irrelevant to it: inbound traffic carries the client’s address, the route back is the default route out the same interface, and strict passes. What breaks strict is real path asymmetry, which you acquire the moment a gateway grows a second NIC. Loose costs nothing and removes the class. Two traps: the kernel takes max(conf.all, conf.<iface>), and since 0=off, 1=strict, 2=loose is not a strictness order, 2 is absorbing: all=2 silently downgrades an interface that asked for strict, and you can’t disable it on one interface while all is non-zero. And conf.default applies only to interfaces created after you set it.
• ip_forward=1 on the routing box only. The gateways terminate TCP; they never forward. Leave it off there.

All three are IPv4, and none of it carries over: v6 has its own forwarding switch and hash policy, and BIRD needs a parallel ipv6 { } block with a /128 VIP.

The listener, in config

Documentation addresses throughout: transit subnet 192.0.2.0/24, the listener at 192.0.2.1, one gateway at 192.0.2.11, the service VIP 198.51.100.10, outside the transit subnet. ASNs are from the private range.

log syslog all;
router id 192.0.2.1;

protocol device { scan time 5; }

protocol kernel {
    ipv4 {
        import none;
        export filter { if net = 198.51.100.10/32 then accept; else reject; };
    };
    merge paths on;          # N announcements -> one route with N nexthops
}

protocol bgp gateways {
    local 192.0.2.1 as 64512;
    neighbor range 192.0.2.0/24 as 64513;   # no static fleet list
    dynamic name "gw";                      # sessions become gw1, gw2, ...
    hold time 9;                            # keepalive derives as 3s
    passive;                                # gateways initiate
    graceful restart off;                   # a dead peer must LOSE its route
    ipv4 {
        import filter { if net = 198.51.100.10/32 then accept; else reject; };
        export none;                        # never advertise anything back
    };
}

That half never changes, whether the gateways announce through a sidecar or speak BGP themselves. The gateway half is where there’s a decision to make.

The part I’d argue about: the announcer belongs inside the gateway

The BGP lifecycle shouldn’t be a sidecar watching the gateway from outside and reporting on its behalf. The process that serves traffic should be the process that decides whether it’s announcing.

Every external health checker is a guess. It probes /healthz and infers. It can’t see that your connection pool is exhausted, that you’re mid-way through loading config, or that a dependency is down. It gets a 200 and calls it healthy. The service knows, so let it decide. It withdraws when its own health signal goes bad, and on shutdown before it stops accepting. Nothing has to notice anything; it removes itself.

“Unhealthy” and “not announcing” stop being two states you have to keep in sync. There’s no window where the registry thinks you’re up and you know you’re not.

Start here: the cheap version

You don’t have to write a BGP speaker to get any of this, and you shouldn’t start by writing one.

Run BIRD or FRR as a config-frozen sidecar on each gateway. It announces the VIP whenever it finds it on lo, so the service’s entire contribution is adding and removing that one address: two shell-outs to ip addr.

router id 192.0.2.11;

protocol device { scan time 5; }

protocol direct {            # picks up VIP/32 when the service adds it to lo
    interface "lo";
    ipv4 { import filter { if net = 198.51.100.10/32 then accept; else reject; }; };
}

protocol bgp listener {
    local 192.0.2.11 as 64513;
    neighbor 192.0.2.1 as 64512;
    hold time 9;
    graceful restart off;
    ipv4 {
        export filter { if net = 198.51.100.10/32 then accept; else reject; };
        import none;
    };
}

Then bind the sidecar’s lifecycle to the service:

[Unit]
BindsTo=gateway.service      # gateway stops or crashes -> bird stops -> session drops
Wants=gateway.service        # and starts with it; BindsTo alone never starts anything
After=gateway.service

Wants is the line everyone forgets: BindsTo only ever stops the bound unit. Without it, BIRD sits inactive after a reboot and the gateway serves nothing, because nothing ever announced it.

That buys the case that matters most: the process dies, the sidecar stops with it, the session drops, the hold timer evicts the gateway. No BGP code in your application, nothing to health-check, nothing to keep in sync. If you take one thing from this post, take this.

What it cannot do is tell alive from healthy. Its only signal is whether your process exists, so it cannot express “I’m running fine and I should not be taking traffic right now”. Every piece of advice below depends on that sentence: withdraw-on-unhealthy, withdraw-then-drain, hysteresis. Those are opinions the service has and a lifecycle-bound sidecar cannot carry.

So start there. Move the announcer in-process when you want the service’s own judgement to count: that is what turns withdrawal from a side effect of dying into a decision the service makes.

The protocol is not yours to write: a speaker library still owns the session, the state machine and the encoding. What moves into your service is a callback mapping your own state onto announce-or-withdraw, and the policy around it:

session = bgp.session(peer = LISTENER, local_as = 64513, peer_as = 64512)

healthy():                          # what a prober cannot see
    return origin_reachable()
       and cert_store.warm()
       and pool.free() > 0
       and not draining

announced = false
good      = 0

on startup:
    start_accepting()               # bind and serve FIRST — never announce
    announced = false               #   a socket that isn't listening yet
    good      = 0                   # the loop admits us five samples later,
                                    #   which doubles as a warm-up gate

every 1s:
    if healthy():
        good += 1
        if not announced and good >= 5:     # slow to come back
            session.announce(VIP/32)
            announced = true
    else:
        good = 0
        if announced:
            session.withdraw(VIP/32)        # instant to leave
            announced = false

on SIGTERM:
    draining = true
    session.withdraw(VIP/32)        # 1. stop new flows
    sleep(DRAIN)                    # 2. propagation + longest request
    stop_accepting()                # 3. close the listener
    exit(0)                         # 4. only now

Every line that matters is policy, not protocol: the asymmetric hysteresis, the boot and shutdown ordering, and a healthy() that reads internal state instead of answering a probe. Because admission runs through the same loop, a cold process serves for five seconds before it announces, so the warm-up gate is the hysteresis you already wrote, not a second mechanism. Announce before you bind and the routing table will hand you traffic you cannot yet answer.

What this costs you

I’d be selling you something if I stopped here.

ECMP rehashing will reset connections. This is the big one. When the set of announcing gateways changes, hash buckets get recomputed and existing flows can land on a gateway that isn’t holding their state. Those connections break, so adding a gateway disrupts traffic on the ones that were already fine. Seamless needs either resilient hashing at the routing layer or a stateless forwarding layer that redirects misrouted packets, roughly what GitHub’s GLB does. On hardware, resilient ECMP is often a config knob. On Linux there’s no such escape hatch: resilient nexthop groups (5.13+) apply to nexthop objects, and merge paths installs a classic multipath route they don’t cover. How much this costs depends on your traffic. Long-lived connections (WebSockets, streaming, large uploads) feel every membership change. Short HTTP requests mostly finish before the next one happens.

ECMP spreads flows; it does not balance load. No least-connections, no weighting, no health-aware backpressure, no retries, nothing at L7. Every announcing gateway takes an equal share of hash space regardless of what it’s doing, so one that is alive but slow (mid-GC, pool exhausted, cache cold after a restart) keeps its full share until something withdraws it, and nothing here will. That’s the alive-versus-healthy gap again, and the strongest argument for moving the announcer in-process.

Default BGP timers are far too slow. Everyone quotes a 180s hold and 60s keepalive; that’s Cisco and FRR. BIRD, the daemon in the config above, defaults to 240s and 80s, so the default costs four minutes of traffic into a hard-crashed gateway, not three. Check your own daemon. We run a 9s hold, which BIRD turns into a 3s keepalive.

In fairness to the health checker, nine seconds is no victory over ten or fifteen. The difference is that this costs no extra process and no state to go stale, and BFD takes it under a second without adding either.

Turn graceful restart off, explicitly. GR exists to preserve routes when a session drops, on the assumption the peer is restarting its control plane while still forwarding. That’s the opposite of what you want, but the obvious case isn’t the dangerous one. Hold-timer expiry sends a NOTIFICATION, which puts both ends back on normal procedures, so routes flush and GR never engages. The danger is a session dying without one: TCP reset, link loss, or the announcer SIGKILLed rather than asked to stop. There the helper keeps a dead peer’s routes for its advertised restart time, which is the killed-sidecar path this design leans on. BIRD defaults to aware, the helper role, so write graceful restart off on both sides and check show protocols all.

Withdrawal is not draining. Pulling the route stops new flows. In-flight requests are still in flight. The order is: withdraw, wait out a drain window, then exit. Get it backwards (sleep first, withdraw second) and you spend the whole window still accepting new connections and then cut the in-flight ones anyway. Size the window as route-propagation time plus your longest expected request: shorter still severs live work, longer only slows the deploy.

Draining protects your in-flight requests, but the withdrawal changes the nexthop set, which rehashes buckets on every gateway still standing. A perfectly executed rolling deploy still resets connections on the healthy peers it never touched. No ordering fixes it. It’s a property of plain multipath, and the reason resilient hashing exists.

Flapping health needs hysteresis. A health signal that oscillates will announce and withdraw in a loop and thrash the routing table, and every flap resets flows across the whole fleet, not just its own. Damp it asymmetrically: withdraw on the first bad sample, but require several consecutive good ones before re-announcing. Coming back is the direction that costs everybody else. A crashlooping service with Restart=always is itself a flap source, coming and going every couple of seconds; cap the restart burst.

The box holding the public IP is still a single point of failure. BGP solved gateway-level redundancy. It did not solve that. Two boxes with VRRP or a second announced path is the answer, and you should be clear-eyed about whether you’ve done it.

None of these are reasons not to do this. They’re the engineering work, and a much smaller surface than operating a registry cluster.

The wider point

This post is about a reflex more than about BGP: an infrastructure question comes up, and the answer you reach for is a product. Need service discovery? Here’s a service discovery service. Need load balancing? Here’s a load balancer. Need health checks? There’s a checkbox for that. Every answer is a new system, and every new system is another thing that can be down at 3am.

So the question I keep coming back to: what does the protocol already give me for free?

More often than I expected, the answer is: most of it.