It started with an LLM agent documenting the boundaries for me instead of building them. It was about concurrency, and what came out of it was a comment: which call may come from which goroutine, what has to be locked first, what to watch out for. All correct. All friendly. All just text.
A boundary that lives in a comment isn’t a boundary. It’s a request. It holds as long as the comment is read and remembered. I just don’t want to remember; static linters should. Safe at any time, by any reader.
It also can’t be checked. A clear boundary is testable: the same input, the same result, over and over. An unclear one gives unclear results. I don’t like that. And safe code, in my view, is code with clear boundaries rather than comments explaining them.
“not safe for concurrent use”, “caller must hold the lock” — I’ve written lines like that myself — fewer as I got more experience — and agents without a clean harness produce tons of them. It bothered me properly for the first time, and afterwards I spent a long while thinking about what a core looks like where those comments aren’t needed at all.
More discipline doesn’t help there. What helps is a shape where the path to someone else’s data simply doesn’t exist. Then nobody needs a comment forbidding it either.
What came out of that doesn’t contain a single sync.Mutex. That mattered to me, and that’s what this is about.
What’s actually running
incus-compose will ship a DNS server soon. It pulls its records from Incus and keeps them current through the event stream. Underneath sits ievent: the chain the events from incusd walk. DNS was the occasion, though, not the goal — a checker and an operator hang off the same chain, each as its own binary.
incusd → source → debounce → log → enricher → dns → http → log
source reads the stream. debounce collapses bursts. The enricher holds a model of the fleet and patches it from the events; what it doesn’t know it reads from incusd — I’ll call one of those questions a read from here on. dns builds answers from it. http does /ready and metrics. log writes it down.
An event might sound like a work order; it’s a read order. The chain holds a picture of the fleet, and an event is a delta on it. The rest follows from that: debounce collapses because four deltas on one key describe a picture that no longer exists. Whether something is gone can only be read against a whole picture. And ChainState says whether the picture is complete.
The order of the chain is compiled in. There’s an --exclude, but it may only take away positions the binary marked optional, and a deployment doesn’t get to decide which those are.
Up to here this is an ordinary middleware chain with different names.
A plugin
type Plugin interface {
// Name identifies the plugin in logs, in metrics, and in the chain.
Name() string
// Wants declares which actions this plugin cares about.
Wants() []Want
// Setup wires the plugin, once, before anything runs.
Setup(args SetupArgs) error
// Handle receives an event.
Handle(ev *Event)
}
Four methods. The last one is the interesting one, and what sits in SetupArgs is more interesting still:
// Next is the successor's Handle — the one field that differs per position.
Next Next
next is the next plugin’s Handle. A function I call when I’m done, or when I want to send new events on to the next plugin.
Every link holds its successor itself and continues the walk itself. That’s why a plugin may work before and after next — the same shape as a chain of HTTP handlers.
The mailbox
Now the part that’s different.
// Handle runs in its parent's goroutine and must not block: it enqueues and
// returns. Next is called from the plugin's own goroutine once the work is
// done.
Handle is the mail slot. It runs in the goroutine of whoever called it. So it must not do anything that takes time. It drops the event in and leaves.
The event itself is immutable. Every With… returns a copy instead of changing the same piece of memory. So two goroutines holding the same pointer is harmless, and the question of who owns an event right now never comes up.
Here’s what that looks like in debounce:
func (p *Plugin) Handle(ev *iutil.Event) {
select {
case p.inbox <- ev:
default:
p.next(ev.WithDropped(name))
}
}
Two lines of logic. If it fits, it’s in. If it doesn’t, it moves on — with a note saying who dropped it.
The actual work happens elsewhere:
// Run holds events until told to finish. It blocks, so main owns the
// goroutine, and it returns having handed on everything it holds.
func (p *Plugin) Run(ctx context.Context) error {
open := map[string]*burst{}
for {
var due <-chan time.Time
at, ok := earliest(open)
if ok {
due = time.After(time.Until(at))
}
select {
case <-ctx.Done():
return nil
case cmd := <-p.in:
p.drain(open)
p.closeAll(open)
p.answer(ctx, cmd)
return nil
case <-due:
p.closeExpired(open)
case ev := <-p.inbox:
p.accept(open, ev)
}
}
}
A for with a select. The mailbox, a timer, a command channel, an abort. That’s all it is.
And then, at some point, this plugin calls next. From its own goroutine. Which means: it drops the event into the next plugin’s mailbox and goes back to work.
The chain is written as a call stack and runs as an assembly line.
Handing on later
That’s what the mailbox buys.
debounce collapses a burst on one key into two events: the first goes at once, the last goes when the key has been quiet for a while. Everything in between is superseded. A burst costs two reads that way instead of one per event.
“A while” is 250 milliseconds. Handle has long since returned by then. The predecessor carried on, source keeps reading the stream, and eventually the timer expires and debounce hands the last event on — a quarter of a second after it arrived.
A synchronous chain can’t do that. It can wait, but then it waits together with its caller. A link that sleeps for 250 milliseconds holds up everything behind it.
The round
The enricher walks the whole fleet, name by name, for the life of the process. GetProjectNames, GetNetworkNames, GetInstanceNames, then one read per name. Each name becomes a bare instance-updated and takes exactly the path a live event takes.
That’s the way back. A dropped connection is the obvious case, the round runs for the less obvious one: an event that never arrived at all leaves nothing behind to notice. Only a full walk catches a transition nobody was told about.
A read that found nothing new produces no event at all — it gets trashed. What comes back is compared against the last one that went out about this subject — so an unchanged instance costs a read and nothing else.
And the round works by absence:
What the model holds in a scope and the listing doesn’t name is gone.
The scope matters, and so does the moment. Which keys the model holds is recorded when the request goes out, and what is pruned is that record minus the answer. A name that comes into being while the request is in flight isn’t in it and survives — and it had to, because nothing would put it back: the next round reads it unchanged and sends nothing.
A listing that failed prunes nothing. An empty answer and a daemon that won’t answer arrive the same way, and one of them means every name in the project.
Nobody is held up by any of this. The round is paced, between the projects and between the reads inside one, and how tightly is decided by whoever assembles the plugins. That’s exactly why it may take long: there is no state in which the chain holds its breath.
What there is, is cold and warm. Cold means the fleet has never been read whole, and then debounce collapses nothing:
collapse := ev.State() == iutil.StateOk &&
ev.ChainState() == iutil.ChainWarm &&
p.wanted[ev.Action()].Debounce
The gate is on warm, never against cold. The zero value is neither, so an event nobody stamped behaves like one nothing has been read for. Before the first full round everything walks through singly, in the order it arrived — so what that round then reads is everything Incus sent, rather than whatever a window left over.
And the one who sets warm is dns. The enricher only says that a round has been all the way round, with enricher/sweep-end. Whether that means anything is decided by the last position in the chain: set at the enricher, warm would already be true while that round’s own events were still in flight.
When the mailbox is full
That default branch from earlier is the honest part. Every buffer is full eventually, and then somebody has to decide what happens.
Here the plugin that’s full decides. With its own name on it. The event doesn’t vanish, it keeps walking the chain and carries the note with it — StateDropped means “finished with, still walking so the observers behind can see it”. For log, that is, and for the metrics. Anyone who sees dropped there with a name next to it knows which plugin is sized too small. That’s all the note does today.
For DNS that’s enough: a lost event there is a record that’s stale until the next round. An operator may not lose one. What debounce collapses is harmless in that respect, since only what repeats is held back. A full mailbox drops whatever is arriving right now, and that can be a delete.
So at the end of the chain there will be a plugin that reads the note and triggers the repair for whatever was dropped. It isn’t implemented yet.
WithDropped writes down who it was and is a no-op afterwards. Whoever drops an event first stays the one on record.
Plugins that do something skip an event like that:
func (p *Plugin) Handle(ev *iutil.Event) {
if ev.State() != iutil.StateOk {
p.next(ev)
return
}
// ...
}
What is communicated outward is communicated unmistakably: the state is on the event, anyone can read it. What a plugin does with it on the inside belongs to the plugin. A type that forces the check would be a proposal that only makes sense against a plugin behaving badly — and those don’t get built here.
Clear outside, free inside.
One place is left over anyway, and it sits on Value, announcing itself — “no type can carry that rule, which is why it is written here”.
So a full mailbox reports itself by name. And a debounce collapsing two hundred events into two does exactly the same:
// Inside an open window. The first event here has nothing to supersede.
if b.ev != nil {
p.next(b.ev.WithDropped(name))
}
So nothing disappears at all. Every superseded event keeps walking the chain, with the same note a dropped one carries, and the observers behind see it. What gets collapsed isn’t the events, it’s the reads — the enricher doesn’t look anything up for an event somebody has finished with. And two hundred reads of the same object return what the last one returns anyway.
What gets held is instance-updated and network-updated — the things that repeat. Start, stop, delete and rename walk straight through, and for renames the reason sits next to them: collapsing two keeps the last OldName and loses the middle one, so the record under it would never be dropped. So an event being held can never be news about existence.
The mailbox is configurable by whoever assembles the plugins — that person knows best what buffers they need.
Handle is the actual power feature of this chain. Every plugin gets its successor’s Handle as next, and the chain needs nothing else to live. Because Handle runs in the predecessor’s goroutine, the receiver must not block it — which is exactly why a full mailbox has to drop rather than wait.
Some of them don’t have one
log has no mailbox, no goroutine, no Run. Its Setup keeps the successor and starts nothing; Handle writes the line and calls next directly, in the caller’s goroutine.
From the outside there’s no telling the two apart. Whoever calls next doesn’t know whether they just crossed a goroutine boundary or called a function that comes straight back. Both are next(ev). The same signature carries the plugin that holds an event for a quarter of a second and the one that passes it through without stopping.
And that’s why there’s no mutex
Now the part from the top.
Everything but the inbox belongs to the goroutine
Runowns.
One owner per plugin. Nobody else touches the open map in debounce. The mailbox is the only thing shared, and it’s a channel — that synchronises itself.
The hard case isn’t debounce, though, it’s the enricher. It has a worker pool, several goroutines reading from incusd at once — and not one of them touches its model. A worker reads and hands the answer back, the patch happens where the model lives. Read concurrently, written in one place, no lock.
This is exactly what I was after. A foundation where no sync.Mutex is needed any more, because there’s nothing left for two goroutines to fight over. Each owns its data, and whatever passes between them goes through a mailbox.
Communication over channels between workers beats locking their data. If a plugin is to own its data alone, then nobody else may reach in — so somebody has to send it something instead of putting it down next to it. That’s precisely what Handle does.
In a chain that runs in a worker pool this doesn’t work. No link there has its own goroutine, each borrows one, and with it shares everything another one touches too. Then a lock is all that’s left.
Same shape, a different foundation, and the mutex is gone. “Who owns this data?” is the more uncomfortable question than “where does the lock go?”, and it’s uncomfortable exactly once, at design time. After that the answer is in the shape, and there’s nothing left to remember.
And it’s checkable, as demanded above. A plugin is a function of event and own state onto events: a test is a call with a value and an assertion about what comes out the other end. The only seam where something has to be faked sits in the enricher, where Incus gets touched.
What everyone wants, up front
There’s one thing I had to build that an ordinary chain doesn’t need.
Wants() is separate from Setup(), and there’s a reason. The enricher serves the whole chain from a single read. So it has to know what everyone wants before it reads the first time — and know it completely.
If the table came into being during the wiring, every plugin would get the part of the union that happened to exist when its own Setup ran.
So the chain asks everyone first, then wires. Every plugin gets the same finished table.
The two fields fold in opposite directions and mean the same thing. For Enrich, more wins: if anyone wants an action enriched, it gets enriched. For Debounce, false wins — “the zero value vetoes for everybody”. Both times the zero value is the safe side: more work, less risk.
That’s the price of a compiled-in chain. It has to interview its own links before it can build itself.
Three ways in
Besides the mailbox, every plugin has two more channels. One in, one out, both for commands. The incoming one is unbuffered, with a reason: “a slot would let the source ask a plugin that is not listening and believe it had been heard.”
That used to be a function call. Two channels per plugin came out of it, and the two directions pull apart on purpose.
In: past the queue. When the process is to finish, source/drain goes to every plugin — on its own channel, “whatever the event inbox looks like”. A command that had to wait behind ten thousand events in a full mailbox would be useless.
Out: back into the queue. When a plugin has something to say itself — dns reporting that it now has something sensible to answer with — that doesn’t go out as a broadcast to everyone.
The source mints an event from it and puts it in at the front of the chain. It runs from position one, like every other event, “so it reaches every position and in order against the events that caused it”.
That’s the point. A side channel that goes past the queue goes past the ordering too. A ready that overtakes the events that caused it is telling an untruth.
So the one thing allowed past the queue is the one that doesn’t care about order: stopping. Everything else takes its turn.
A Command carries two independent fields, and either may be left empty. An Action mints that event; a ChainState sets what the source stamps from there on.
They say different things. An Action says what is to be done. A ChainState says what the chain currently is — one is a message, the other a level.
The source holds that level and stamps it on everything it mints. What a plugin makes of it is its own business: debounce only collapses while it’s warm, dns decides when it becomes warm, http makes its /ready out of it.
Which means http always knows how the chain is doing. Its /ready latches on warm and falls back on cold, and it starts cold — which is the truth before anything has been read. No pointer to dns, no registry, nobody it has to ask. That dns has something sensible to answer with arrives the same way: as an event, dns/ready, folded in order against whatever caused it.
And any plugin may set it — nothing checks the transition, on the same trust as everything else here. At the end of a round two plugins use one field each: the enricher sends an Action, enricher/sweep-end, and dns then sends a ChainState, warm.
Forward: from here on. There’s a third way in, and it’s the quietest. When a profile or a network changes, every instance underneath has to be read again — but the event that reported the change names none of them. So the enricher invents an instance-updated per affected instance, one that never happened, and puts it in its own line:
// One synthetic instance-updated each, put in the line here rather than at the
// head of the chain: they are this plugin's own work, and sending them round
// would have debounce collapse the set into whichever of them arrived last.
From behind it looks like any other update. What matters is where it goes in. At the head of the chain it would pass through debounce again, which sits in front of the enricher — and the set would arrive as one.
Three ways in, then, each picked by what the order is supposed to mean. Past the queue, when it doesn’t matter. At the head, when order against the causes is the point. Forward from here, when whatever sits in front must not touch it a second time.
You can see it in the names. enricher/sweep-end, source/connected, dns/ready, source/drain.
That’s for control. Data events stay unprefixed, invented ones included: the synthetic instance-updated from earlier is named exactly like one from incusd. As far as everything behind it is concerned, that is what happened.
No init()
The usual thing would be a registry. Every plugin brings a func init() that registers itself somewhere on import, and an import with _ makes sure it runs.
I know pretty well how that goes. There are 42 of them in the plugins of go-orb — forty-two times something registers itself while the program is still starting. In one place like this:
func init() {
if err := source.Plugins.Add(New()); err != nil {
panic(err)
}
}
A panic before main even runs. If that fires there’s no place to catch it and no caller to report it to.
If you buy freedom by hiding things, you buy yourself a new prison. What’s in it is in an import list. The wiring happens as a side effect. To find out why something behaves the way it does, you have to search backwards for the package that brought it along.
There isn’t a single init() in ievent and no import with an underscore. I don’t need one.
The chain is a slice. chain() returns what’s in it, in the order it’s in. You read one function and you know what the process does. And because it’s ordinary Go code rather than a generated table, the same plugin may sit in it twice — log once at the front, once at the back, and in between you can read off what a position cost.
That pays off at the second product. A checker is a second chain.go with the same imports and a different last link. With a registry the question would instead be which plugins drag themselves in on import.
It’s the same boundary as at the very start. What the process does should be readable, instead of something you have to piece together.
Plugins trust each other
There’s one sentence in the interface that matters to me:
Plugins trust each other: nothing here defends one against another, and a panic takes the process down. If a proposal only makes sense against a plugin behaving badly, it does not get built.
That’s a boundary. Inside one binary I wrote everything myself. Carelessness would be trusting unknown libraries and plugins blindly. A plugin behaving badly is a bug, and you fix bugs instead of building watchtowers around them.
Trust doesn’t mean taking your own position from somebody else, though. Every plugin accepts what arrives without checking it — and still knows for itself where it stands: debounce keeps the last chain state it saw, http its latched one. Reading your position off a shared value isn’t trust, it’s a dependency.
Across a process boundary it’s the exact opposite. The Caddy sidecar I’m planning holds no certificate. The process facing the internet has nothing anyone could take from it, and the one holding the Incus connection listens on no public port.
Full trust inside, none outside. Both on purpose.
What isn’t there at the end
I didn’t come up with the chain. A link holds its successor and hands on — that’s behind every middleware and behind defer, and it was waiting for me.
What I went looking for is what lies underneath. A next that drops something off and leaves instead of working. One owner per piece of data. A mailbox as the only door.
What comes out of that is best seen in what’s missing from the code.
sync.Mutex, init(), underscore imports — none of that is in the code of ievent.
A new plugin is a struct with four methods and one line in the list. Where it sits is none of its business. Whoever fills up decides for themselves.
And when one gets fat, it’s a chain again on the inside. dns is one, in the direction of the server and the cold store. Same shape, a different bearing.
Boring. Again. Exactly how it should be.
By René Jochum. License: CC-BY-4.0.