It started with an LLM agent documenting the boundaries for me instead of building them.
The LLM wrote correct comments about concurrency. The problem wasn’t the correctness — it was that comments can’t enforce anything.
It was about concurrency, and what came out of it was a comment: which call may come from which goroutine, what has to be locked first, what to watch out for. All correct. All friendly. All just text.
A boundary that lives in a comment isn’t a boundary. It’s a request. It holds as long as the comment is read and remembered. At three in the morning I don’t remember it.
That’s not an agent quirk. Every Go codebase is full of it — “not safe for concurrent use”, “caller must hold the lock”. I’ve written lines like that myself. But it bothered me properly for the first time, and afterwards I spent a long while thinking about what a core looks like where those comments aren’t needed at all.
More discipline doesn’t help there. What helps is a shape where the path to someone else’s data simply doesn’t exist. Then nobody needs a comment forbidding it either.
What came out of that doesn’t contain a single sync.Mutex. That mattered to me, and that’s what this is about.
What’s actually running
incus-compose ships a DNS server these days. It pulls its records from Incus and keeps them current through the event stream. Underneath sits ievent: the chain the events from incusd walk.
incusd → source → debounce → log → enricher → dns → http → log
source reads the stream. debounce collapses bursts. enricher asks incusd and fills the event in with what the daemon knows about it — I’ll call one of those questions a read from here on. dns builds answers from it. http does /ready and metrics. log writes it down.
The order is compiled in. There’s an --exclude, but it may only take away positions the binary marked optional, and a deployment doesn’t get to decide which those are.
Up to here this is an ordinary middleware chain with different names.
A plugin
type Plugin interface {
// Name identifies the plugin in logs, in metrics, and in the chain.
Name() string
// Wants declares which actions this plugin cares about.
Wants() []Want
// Setup wires the plugin, once, before anything runs.
Setup(args SetupArgs) error
// Handle receives an event.
Handle(ev *Event)
}
Four methods. The last one is the interesting one, and what sits in SetupArgs is more interesting still:
// Next is the successor's Handle — the one field that differs per position.
Next Next
next is the next plugin’s Handle. No channel, no bus, no registry. A function I call when I’m done.
Every link holds its successor itself and continues the walk itself. That’s why a plugin may work before and after next — the same shape as a chain of HTTP handlers.
The mailbox
Now the part that’s different.
// Handle runs in its parent's goroutine and must not block: it enqueues and
// returns. Next is called from the plugin's own goroutine once the work is
// done.
Handle is the mail slot. It runs in the goroutine of whoever called it. So it must not do anything that takes time. It drops the event in and leaves.
The event itself is immutable. Every With… returns a copy instead of changing the same piece of memory. So two goroutines holding the same pointer is harmless, and the question of who owns an event right now never comes up.
Here’s what that looks like in debounce:
func (p *Plugin) Handle(ev *shared.Event) {
select {
case p.inbox <- ev:
default:
p.next(ev.WithDropped(name))
}
}
Two lines of logic. If it fits, it’s in. If it doesn’t, it moves on — with a note saying who dropped it.
The actual work happens elsewhere:
// Run holds events until told to finish. It blocks, so main owns the
// goroutine, and it returns having handed on everything it holds.
func (p *Plugin) Run(ctx context.Context) error {
open := map[string]*burst{}
// sweeping is set between the brackets of a pass.
sweeping := false
for {
var due <-chan time.Time
at, ok := earliest(open)
if ok {
due = time.After(time.Until(at))
}
select {
case <-ctx.Done():
return nil
case cmd := <-p.in:
p.drain(open, sweeping)
p.closeAll(open)
p.answer(ctx, cmd)
return nil
case <-due:
p.closeExpired(open)
case ev := <-p.inbox:
sweeping = p.accept(open, ev, sweeping)
}
}
}
A for with a select. The mailbox, a timer, a command channel, an abort. That’s all it is.
And then, at some point, this plugin calls next. From its own goroutine. Which means: it drops the event into the next plugin’s mailbox and goes back to work.
The chain is written as a call stack and runs as an assembly line. There isn’t a channel anywhere in the interface.
Handing on later
That’s what the mailbox buys.
debounce collapses a burst on one key into two events: the first goes at once, the last goes when the key has been quiet for a while. Everything in between is superseded. A burst costs two reads that way instead of one per event.
“A while” is 250 milliseconds. Handle has long since returned by then. The predecessor carried on, source keeps reading the stream, and eventually the timer expires and debounce hands the last event on — a quarter of a second after it arrived.
A synchronous chain can’t do that. It can wait, but then it waits together with its caller. A link that sleeps for 250 milliseconds holds up everything behind it.
Except during a full pass
There’s one case where debounce may not hold anything back.
Every so often the enricher reads the whole fleet. The brackets around it are minted by the source: the enricher only reports, the source makes an event out of it and puts it at the front of the chain. Hence the names, source/sweep-start and source/sweep-end.
That isn’t housekeeping on the side, it’s the way back. A dropped connection is the obvious case, but it runs on a timer for the less obvious one: an event that never arrived at all leaves nothing behind to notice. Only a full read catches a transition nobody was told about.
And it works by absence:
A name a plugin holds and does not see between them is gone.
For that to work, dns folds the round into a second snapshot nobody can see yet and swaps it for the current one at source/sweep-end. Until then it keeps answering every question from the old one. A half-read fleet never reaches a client, and “no answer right now” doesn’t happen.
The contents of the round travel a different path from the brackets. What the enricher read it pushes straight forward, past debounce — the same route the fan-out takes further down. Which is what keeps the round intact: it’s made entirely of instance-updated, an action debounce would be allowed to collapse. It just never arrives there.
The brackets have to go all the way round, for the opposite reason. debounce sits in front of the enricher, and it has to react to them:
case shared.ActionSweepStart:
p.closeAll(open)
p.next(ev)
return true
Everything held goes out, then the bracket. And between the brackets nothing is collapsed any more:
collapse := ev.State() == shared.StateOk &&
!sweeping &&
p.wanted[ev.Action()].Debounce
Together those two keep anything debounce is holding from lying across a bracket.
Not because the data would go bad. A held event still passes the enricher on its way out, and the enricher reads incusd at that moment — so what finally reaches dns is current, whatever the event’s timestamp says. Nothing is lost either: dns folds the round and the stream into the same set, and the closing bracket publishes whatever is in it.
What the two rules buy is that the order stays well defined. Between the brackets, what walks is what exists, and nothing from before them is still in the air. It’s a boring guarantee and it’s the one the pass is built on.
And a pass is where two of the ways in meet. The brackets go round to the head, because a plugin sitting in front has to act on them. The contents go forward, because that same plugin would collapse them. One operation, both routes, each picked by what it needs the order to mean. The third way comes further down: a command that doesn’t care about order at all, and is allowed past everything for exactly that reason.
When the mailbox is full
That default branch from earlier is the honest part. Every buffer is full eventually, and then somebody has to decide what happens.
Here the plugin that’s full decides. With its own name on it. The event doesn’t vanish, it keeps walking the chain and carries the note with it — StateDropped means “finished with, still walking so the observers behind can see it”. For log, that is, and for the metrics. Anyone who sees dropped there with a name next to it knows which plugin is sized too small. The note isn’t meant to do more than that.
WithDropped writes down who it was and is a no-op afterwards. Whoever drops an event first stays the one on record.
Plugins that do something skip an event like that:
func (p *Plugin) Handle(ev *shared.Event) {
if ev.State() != shared.StateOk {
p.next(ev)
return
}
// ...
}
And here I have to make an exception to everything above: those four lines are a convention. They sit in the interface as a comment, and anyone who forgets them works on an event somebody finished with long ago. I haven’t found a shape that makes it impossible without bloating the four methods.
A boundary in a comment. Exactly the kind I complained about at the start. And it didn’t stay the only one: the second sits on Value, and it announces itself — “no type can carry that rule, which is why it is written here”. Honest, and it annoys me anyway.
So a full mailbox reports itself by name. And a debounce collapsing two hundred events into two does exactly the same:
// Inside an open window. The first event here has nothing to supersede.
if b.ev != nil {
p.next(b.ev.WithDropped(name))
}
So nothing disappears at all. Every superseded event keeps walking the chain, with the same note a dropped one carries, and the observers behind see it. What gets collapsed isn’t the events, it’s the reads — the enricher doesn’t look anything up for an event somebody has finished with. And two hundred reads of the same object return what the last one returns anyway.
What gets held is instance-updated and network-updated — the things that repeat. Start, stop, delete and rename walk straight through, and for renames the reason sits next to them: collapsing two keeps the last OldName and loses the middle one, so the record under it would never be dropped. So an event being held can never be news about existence.
Every plugin sets its own mailbox size. debounce takes 1024, “matched to the Incus client’s own event channel”. That isn’t a central value in some config. Whoever fills up knows best how much room they need.
That’s why the interface has Handle and not a channel. A channel would have put the decision outside.
Most of them don’t have one
log has no mailbox, no goroutine, no Run. Its Setup keeps the successor and starts nothing; Handle writes the line and calls next directly, in the caller’s goroutine.
From the outside there’s no telling the two apart. Whoever calls next doesn’t know whether they just crossed a goroutine boundary or called a function that comes straight back. Both are next(ev).
That’s the part I got stuck on. The same signature carries both: the plugin that holds an event for a quarter of a second, and the one that passes it through without stopping.
And that’s why there’s no mutex
Now the part from the top.
Everything but the inbox belongs to the goroutine
Runowns.
One owner per plugin. Nobody else touches the open map in debounce. The mailbox is the only thing shared, and it’s a channel — that synchronises itself.
This is exactly what I was after. Not the chain, I already knew that one. A foundation where no sync.Mutex is needed any more, because there’s nothing left for two goroutines to fight over. Each owns its data, and whatever passes between them goes through a channel.
The mailbox is the consequence of that, not the starting point. If a plugin is to own its data alone, then nobody else may reach in — so somebody has to send it something instead of putting it down next to it. That’s precisely what Handle does.
In a chain that runs in a worker pool this doesn’t work. No link there has its own goroutine, each borrows one, and with it shares everything another one touches too. Then a lock is all that’s left.
Same shape, a different foundation, and the mutex is gone. “Who owns this data?” is the more uncomfortable question than “where does the lock go?”, and it’s uncomfortable exactly once, at design time. After that the answer is in the shape, and there’s nothing left to remember.
What everyone wants, up front
There’s one thing I had to build that an ordinary chain doesn’t need.
Wants() is separate from Setup(), and there’s a reason. The enricher serves the whole chain from a single read. So it has to know what everyone wants before it reads the first time — and know it completely.
If the table came into being during the wiring, every plugin would get the part of the union that happened to exist when its own Setup ran.
So the chain asks everyone first, then wires. Every plugin gets the same finished table.
The two fields fold in opposite directions and mean the same thing. For Enrich, more wins: if anyone wants an action enriched, it gets enriched. For Debounce, false wins — “the zero value vetoes for everybody”.
So a plugin that says nothing gets no enrichment and prevents all collapsing. Both times the decision falls on more work and less risk, and both times the zero value is the safe side.
That’s the price of a compiled-in chain. It has to interview its own links before it can build itself.
Two channels
Besides the mailbox, every plugin has two more channels. One in, one out, both for commands. The incoming one is unbuffered, with a reason: “a slot would let the source ask a plugin that is not listening and believe it had been heard.”
That used to be a function call. Two channels per plugin came out of it, and the two directions pull apart on purpose.
In: past the queue. When the process is to finish, source/drain goes to every plugin — on its own channel, “whatever the event inbox looks like”. A command that had to wait behind ten thousand events in a full mailbox would be useless.
Drain goes through the plugins one at a time, in chain direction, and waits for the answer before asking the next. Whoever feeds someone is asked first.
Only the ones that can hold something need asking. log holds nothing: it did its work before next returned, so it’s finished before the question is even put. The source works that out by itself:
// A plugin with no goroutine is finished before it starts, decided by what it
// is rather than by what main remembers to say.
_, runs := p.(interface{ Run(context.Context) error })
if !runs {
close(pl.done)
}
No flag, no list, no configuration. A type assertion on whether the plugin has a Run.
For everyone else the answer only comes once everything really has been handed on:
// Answered only once everything has been handed on: the source asks the next
// plugin as soon as this one answers.
p.answer(ctx, cmd)
And a plugin answers with the same action back, “including for commands it does not know”. So an unknown command acknowledges instead of leaving the shutdown hanging.
Out: back into the queue. When a plugin has something to say itself — dns reporting that it now has something sensible to answer with — that doesn’t go out as a broadcast to everyone.
The source mints an event from it and puts it in at the front of the chain. It runs from position one, like every other event, “so it reaches every position and in order against the events that caused it”.
That’s the point. A side channel that goes past the queue goes past the ordering too. A ready that overtakes the events that caused it is telling an untruth.
So the one thing allowed past the queue is the one that doesn’t care about order: stopping. Everything else takes its turn.
Forward: from here on. There’s a third way in, and it’s the quietest. When a profile or a network changes, every instance underneath has to be read again — but the event that reported the change names none of them. So the enricher invents an instance-updated per affected instance, one that never happened, and puts it in its own line:
// One synthetic instance-updated each, put in the line here rather than at the
// head of the chain: they are this plugin's own work, and sending them round
// would have debounce collapse the set into whichever of them arrived last.
From behind it looks like any other update. What matters is where it goes in. At the head of the chain it would pass through debounce again, which sits in front of the enricher — and the set would arrive as one.
Three ways in, then, each picked by what the order is supposed to mean. Past the queue, when it doesn’t matter. At the head, when order against the causes is the point. Forward from here, when whatever sits in front must not touch it a second time.
You can see it in the names. source/sweep-start, source/connected, chain/ready, source/drain. The slash is deliberate — an Incus lifecycle action can’t contain one, and the prefix says who put it into the chain.
That’s for control. Data events stay unprefixed, invented ones included: the synthetic instance-updated from earlier is named exactly like one from incusd. As far as everything behind it is concerned, that is what happened.
No init()
The usual thing would be a registry. Every plugin brings a func init() that registers itself somewhere on import, and an import with _ makes sure it runs.
I know pretty well how that goes. There are 42 of them in the plugins of go-orb.
func init() {
registry.Plugins.Add(Name, Provide)
}
Forty-two times something registers itself while the program is still starting. In one place like this:
func init() {
if err := source.Plugins.Add(New()); err != nil {
panic(err)
}
}
A panic before main even runs. If that fires there’s no place to catch it and no caller to report it to.
The uncomfortable part isn’t the line. It’s that you can read the code and still not know what’s in the process. What’s in it is in an import list. The wiring happens as a side effect. To find out why something behaves the way it does, you have to search backwards for the package that brought it along.
There isn’t a single init() in ievent and not one import with an underscore. I don’t need one.
The chain is a slice. chain() returns what’s in it, in the order it’s in. You read one function and you know what the process does. And because it’s ordinary Go code rather than a generated table, the same plugin may sit in it twice — log once at the front, once at the back, and in between you can read off what a position cost.
It’s the same boundary as at the very start. What the process does should be readable, instead of something you have to piece together.
Plugins trust each other
There’s one sentence in the interface that matters to me:
Plugins trust each other: nothing here defends one against another, and a panic takes the process down. If a proposal only makes sense against a plugin behaving badly, it does not get built.
That isn’t carelessness, it’s a boundary. Inside one binary I wrote everything myself. A plugin behaving badly is a bug, and you fix bugs instead of building watchtowers around them.
Across a process boundary it’s the exact opposite. The Caddy sidecar I’m planning holds no certificate. The process facing the internet has nothing anyone could take from it, and the one holding the Incus connection listens on no public port.
Full trust inside, none outside. Both on purpose.
What isn’t there at the end
I didn’t come up with the chain. A link holds its successor and hands on — that’s behind every middleware and behind defer, and it was waiting for me.
What I went looking for is what lies underneath. A next that drops something off and leaves instead of working. One owner per piece of data. A mailbox as the only door.
What comes out of that is best seen in what’s missing from the code.
No sync.Mutex. No init(). No import with an underscore. No registry things enter themselves into.
Two comments that ask for something are left. They’re further up, I found no shape for either, and they still annoy me. The rest of the list holds.
A new plugin is a struct with four methods and one line in the list. Where it sits is none of its business. Whoever fills up decides for themselves.
Boring. Again. Exactly how it should be.
By René Jochum - LLM Assistet. License: CC-BY-4.0.