How Do You Route a Voiced Note to the Right Person Automatically?

I sell cars for a living. Twenty years in automotive retail, no tech background, no computer science degree.

Capturing a voiced note is the easy half.

You tap a mic, you speak, the words get written down. I covered that half in the voice-flag agent post. The interesting half is what happens next.

Because a note nobody reads is worse than a note you never took.

So the real question is this one. Once the agent has your words, how does it get them to the right person automatically, without you tapping through a menu to pick? That is routing. It is the part of a voice agent that decides who the note is for. This is how I built mine, the switchboard, the bugs, and the fixes.

What does it actually mean to route a note?

It means the agent decides who the note is for, and moves it there, on its own.

Picture the floor. You are between customers and something just got said that has to be dealt with later. A customer wants a price on a part. A trade needs a second look. A follow-up is owed on Thursday. Three different notes, three different people who need them, and no time to sort any of it by hand.

Routing is the agent doing that sorting. You speak one note. The agent works out which of those three it is, and sends it to the one lane that handles it. You did not pick the lane. You just talked.

The key word is intent. The agent does not route on keywords. It does not scan for the word part and fire the note at a parts inbox, because half your parts notes will never contain the word part. It reads the whole note, works out what it means, and routes on the meaning.

That is the difference between a note that reaches the right person and a note that lands in a pile nobody owns.

What are the pieces you need to route a voiced note?

Four pieces, and the router is only the last one. Each does exactly one job.

Transcribe the spoken note to text

→

Read the intent and pull structured fields

→

Match the intent to a destination rule

→

Route the flag and confirm it landed

Transcription. The audio you captured becomes plain text. You send audio, you get a string back. This part is a solved problem now, and the browser recording side of it is documented at the MDN MediaRecorder reference, with the Web Speech API as the other well-trodden path for turning speech into text on the web.

Read the intent. You hand the transcript to a model with a tight instruction. Return structured fields. What kind of note is this, who or what does it concern, and how sure are you. The model returns data, not prose, because the router needs fields it can act on, not a paragraph it has to re-read.

Match the intent to a destination. This is a rulebook. A parts intent goes to parts. A follow-up intent goes to the follow-up list. A build note goes to the build log. The rulebook is small on purpose, and it is the part you own most tightly, because it encodes who gets what in your world.

Route the flag and confirm. The router sends the note to the matched destination, then checks that it arrived and tells you it did. The confirm is not optional. It is the difference between routed and probably routed.

How do you build the router, step by step?

Here is the order I built mine in. It is small on purpose. You want the switchboard working for one intent before it works for five.

Step one. Write down every destination first. Before any code, list the real lanes. Parts. Follow-up. Build log. A human review lane for the ones the agent is not sure about. If you cannot name the lane, the agent cannot route to it. Most first routers fail here, because the builder starts matching intents before deciding where things are allowed to go.

Step two. Make the model return an intent and a confidence. Tighten the parse instruction so the model gives you a normalized intent label from your own list, the fields that intent needs, and a confidence signal. Not free text. A label you chose, so the router can match it without guessing.

Step three. Build the switchboard as a plain lookup. Intent label in, destination out. Keep it boring. A small table that says this label goes to that lane. Do not put cleverness in the router. The cleverness belongs in the parse step that reads the note. The router just moves things, and boring routers are the ones you can trust.

Step four. Add the review lane before you trust the happy path. Wire the low-confidence case first, not last. When the model is not sure, the note goes to the review lane you check once, not to a real lane it might be wrong about. Build this branch before you build the branches that feel finished, because this is the branch that keeps a wrong note from reaching a real person.

Step five. Make the last step confirm the landing. The router does not report success when it sends. It reports success when the note is visibly sitting in its destination with a record behind it. Until then the note is in flight, and anything in flight can be lost.

What breaks when you route a voiced note, and how do you fix it?

Three failures, in the order they hit me. This is the part the demos skip.

One. The router matches the word and ignores the meaning. Early on I routed on keywords, and it looked fine in testing. Then a customer note said "the guy wants to know what it runs" and it never contained a single word my rules watched for, so it fell into no lane at all. The fix is to route on the intent the model reads, not on words you hoped to catch. Let the model tell you what the note means, and match on that.

Two. The agent confidently routes to the wrong person. You say "flag the blue truck for Marcus," the transcript comes back as "flag the blue truck for parts," and the note lands in the parts lane with total confidence. Speech is messy and transcripts arrive imperfect. The fix is the review lane from step four. When confidence is low, the router does not guess, it holds the note for you to confirm. A wrong flag with confidence is worse than a note you confirm by hand.

Three. The silent misroute. This is the one that hurts. The note gets captured, the transcription succeeds, the router picks a lane, and then the delivery to that lane quietly fails. No error. You believe it is handled. It is gone. The fix is the confirm from step five. A note is not routed when the router picks a lane. It is routed when it is visibly in the lane with a record behind it.

Route on what the note means, not on the words it happens to contain. And never let the agent be confidently wrong. A note in a review lane is recoverable. A note in the wrong lane is not.

How do you keep the router small enough to ship this week?

You route one intent first. Not every note you might ever speak. One.

Pick the note you drop most, the one that has to reach a specific person, the one that happens every day. Capture it, read its intent, match it to that one destination, confirm it landed. Wire the review lane for when the agent is not sure. That is the whole v1, and it is enough to route real work off your floor.

Then run it for a week without adding a second intent. A week of real use tells you which lanes you actually needed and which were you gold-plating at the kitchen table. Almost always the thing that matters is not a second intent. It is the confirmation that the note landed, and the review lane catching the ones the agent should not have guessed.

Only after that week do you add the second intent. The switchboard you built first makes each new one cheap, because adding an intent is a new label and a new lane on a router that already works. That is the compounding that lets an operator stack grow. Five rivers, twenty-two agents, all built the same way, one small trusted piece at a time.

The reason to route on meaning and to confirm every landing, if you take nothing else from this build log, is that a note reaching the wrong person is not a small bug. It is the note being lost while everyone believes it was handled. The whole point of speaking a note instead of holding it in your head is that it reaches the person who can act on it. Routing is that promise, kept.

When a note has to travel between tools to reach its lane, the router leans on the Model Context Protocol, the open standard for how an AI conversation reaches a tool, and it is worth respecting even on a first build, because that connect-and-route layer is where the reliability lives. If you want the case for building that layer before the agents that sit on it, I wrote it out in why I built the routing layer before any of the agents.

The community where operators ship these in public, bugs and all, is the free build log at skool.com/agent-empire-4291. Come route your first voiced note with us. Build log, not a landing page.

While I sell cars for a living.

Michael