Sub-millisecond WebSocket fan-out in Go: the Vedico realtime service

When I joined the Vedico: realtime fleet tracking for telecom field operations project, the dashboards polled a REST endpoint every 30 seconds. Telecom field operations don't work on a 30-second clock. This is the design of the Go service that replaced it, and the three decisions that made it fast.

Shape of the problem

• Producers: technicians' phones post GPS pings in small batches.
• Consumers: NOC dashboards, each subscribed to one or more circles (regions).
• Volume: modest in absolute terms (hundreds of producers, dozens of consumers), but with a hard requirement that an update reaching the server is on every subscribed screen now, and that one slow dashboard never slows the others.

The naive design, one goroutine per connection writing to a shared map under a mutex, works until it doesn't: the first slow client blocks the broadcast loop and every dashboard freezes together.

Decision 1: one event loop per circle

Connections are grouped by circle. Each circle owns a goroutine (its hub) and a map of subscribers guarded by a sync.RWMutex. Broadcasts for circle A never contend with circle B, and the lock is held only to snapshot the subscriber list.

type Hub struct {
    mu   sync.RWMutex
    subs map[*Client]struct{}
    in   chan Event   // telemetry + system events
}
 
func (h *Hub) run() {
    for ev := range h.in {
        h.mu.RLock()
        for c := range h.subs {
            select {
            case c.send <- ev:      // non-blocking hand-off
            default:
                c.dropAndCoalesce(ev) // slow client: coalesce, never block the loop
            }
        }
        h.mu.RUnlock()
    }
}

Decision 2: a bounded channel per client, with coalescing

Each client has a small buffered send channel and its own writer goroutine. If the buffer is full the hub does not wait; it marks the client as lagging and keeps the latest position per technician in a tiny per-client map. When the writer catches up it flushes the coalesced state. A dashboard that was away for two seconds gets current positions, not a replay of two seconds of pings.

This is the whole back-pressure story, and it's why a laptop on hotel Wi-Fi can't degrade the NOC wall.

Decision 3: multiplex, don't multi-connect

Telemetry, anomaly alerts (from the geofencing engine) and escalation events (from the SLA state machine) arrive on different internal channels. A select loop merges them into a single ordered stream per hub, so a dashboard sees "technician moved" and "technician flagged" in the order they happened, over one socket.

for {
    select {
    case p := <-telemetry:   h.in <- Event{Kind: "pos", Payload: p}
    case a := <-anomalies:   h.in <- Event{Kind: "anomaly", Payload: a}
    case e := <-escalations: h.in <- Event{Kind: "sla", Payload: e}
    case <-ctx.Done():       return
    }
}

Numbers

Measured from "event enters hub" to "bytes written to the socket" on a single small instance: p99 under one millisecond, flat as subscriber count grows, because the hot path is a channel send and a map iteration under a read lock. The network does the rest.

What I'd change

• Put the service behind the same gateway as the REST API so auth, rate limits and OpenTelemetry context are uniform. Today the socket path has its own middleware.
• Emit hub metrics (subscribers, drops, coalesce rate) to Prometheus from day one. I added them in week three, after guessing wrong about which clients lagged. The observability stack that now watches it is the lgtm-sidecar: a resource-capped observability stack for small teams.
• The geofencing and locking side of the same system (Haversine, FOR UPDATE, the five-minute grace window) deserves its own post; it leans on ideas from Designing Data-Intensive Applications more than on Go.