What I learned building a Slack bot people actually use

I built a Slack bot called Junto for our GTM team. The bigger story, a shared data layer that reps build on top of, is its own essay. This one is just the bot lessons: the unglamorous mechanics that decide whether a bot in a shared channel survives contact with real users, and the rules I only learned by getting them wrong. If you’re about to put a bot in front of your team, this is the essay I wish someone had handed me.

One bot, not many

A bot, for this purpose, is one name in Slack you can talk to: an account that watches channels, answers when spoken to, and does work behind the scenes. Junto is one bot with one name, and that was a decision, not a default.

The alternative shows up fast in any team that starts building. A bot for research, a bot for signals, a bot for call prep, each with its own name, its own instructions, and its own private copy of the context. Ask three of them the same question and you get three answers with three sources, and nobody knows which one to trust. It’s a graveyard of dead internal tools rebuilt inside Slack, except now it pings you. Many bots is a coffee table with ten remote controls on it.

One outside data point made me confident Slack was the right home at all. In LangChain’s session with Deepline, Vishnu Suresh, an engineer there, described building their internal GTM agent as a data pipeline first, and it only took off, in his words, once it became a two-way Slack chatbot. The capability was the same before and after. The location changed everything.

The plumbing

Before any of the interesting parts there are three boring ones, and between them they decide whether a Slack bot survives contact with real users:

1. Acknowledge within three seconds, then do the real work asynchronously.
2. Put idempotency on the message id, so the same event delivered twice still only runs once.
3. Give every external call a timeout and a dead-letter, so nothing can hang forever in silence.

None of that demos well. All of it’s why the demos worked.

The router grew up

Version one matched keywords (i.e. “@junto prep me for my call with Acme” would match “prep” to call prep), and anything it didn’t understand fell through to a default skill. A router that guesses is a router spending trust on every miss. An error costs the user one interaction, whereas a confident wrong answer probably costs you the next ten, because now they have to check everything you tell them. And that balance only ever moves one way.

I later found the same rule written down by people building in a different world entirely. acpx, a client for the Agent Client Protocol that lets orchestrators talk to coding agents, states it as a design principle: routing is constrained choice, never open-ended generation, because an LLM allowed to route freely will eventually route confidently and wrongly. My keyword router had proven their sentence before I ever read it.

Ten exits

The fix was to stop guessing. One classifier at the front, and ten explicit exits: answer a question, run a play, compose a new one, remix one that already exists, schedule something, discover what is there, ask for clarification, offer help, admit that no source covers this, or just chat.¹ Nothing runs that wasn’t explicitly selected, and there’s no default and no fall-through. That last part is what turned “I don’t have a source that covers that” from a failure into an answer.

Fig. 1

The exits are described to the classifier in plain language, and the descriptions do real work. In Ramp’s tooling demo with Deepline, whose internal MCP serves hundreds of reps, they put it on a slide: tool descriptions are the interface. The agent picks from the description, not the code, so a precise description is the difference between a tool that gets called correctly and one that gets misused or ignored.

Mechanical beats model

One rule kept the whole thing honest.

Deterministic post-rules outrank the model.

If a mechanical check says this is a request to run a play, then no amount of model cleverness gets to overrule it. The classifier proposes and the rules dispose. And every misroute that happened live became a pinned transcript in the test suite,² an actual test containing the real words a real person typed. The router can’t quietly regress to a mistake it has already made once. That’s a cheap property to buy and I’d buy it again.

Loud beats silent

There’s a gate in front of all of this, deciding whether a given Slack message is even meant for the bot. It has to exist, because the bot lives in shared channels where most of the traffic is people talking to each other.

The gate used to stay quiet whenever it wasn’t sure. That sounds like good manners, and it’s the worst available behavior. Here is the version that convinced me. The bot asked a rep which account they meant, and the rep answered. The gate ate the answer, because the reply didn’t look like it was addressed to a bot. So the bot ignored a reply to its own question, and neither of us noticed until I went looking. I found it in the logs afterward.

decision: silent, rule: gate

A silent failure is an invisible failure, and invisible is sometimes even worse than loud. In a demo it’s death.

So the gate flipped. In any thread the bot is part of, it responds. Staying silent now requires positive evidence that a message was addressed to a human, and uncertainty means answer. That moved the failure mode from invisible to visible, which I think is the correct direction. acpx has this one on its list too: fail loud, never degrade silently. A system that fails quietly gets to keep failing.

The constitution

Before I let the bot talk in a shared channel, I wrote it a constitution. Not guidelines but a written spec for how it behaves, with a version number on top.

The first version had four rules:

1. Surface, never instruct. The bot posts observations, not orders, and the reps decide what to do with them.
2. Every fact carries its source and its date.
3. No formatting theater. Posts stay short, and nothing is ever bold.
4. Anything that looks like outreach is a draft for a human to review, because nothing sends itself.

The obvious way to apply a spec like that’s to paste it into every prompt. I had prompts in about forty files, and I knew what forty copies would turn into. Each one gets edited in a hurry someday, and six months later you have forty dialects of the same law. So the constitution injects at exactly one place in the code, wrapped around every prompt on its way to the model. It works like an airport with exactly one security checkpoint: every passenger passes through the same doorway, so upgrading the scanner once upgrades it for every flight.

It’s frozen and versioned: v1, with a date. One seam buys three things:

1. One place to audit when a post looks wrong.
2. One version cited in every run log, so I can say which rules were in force for anything the bot ever produced.
3. No copies drifting apart while nobody is looking.

The hard rule

One rule mattered more than all the others. When a rep asks who to reach out to, never suggest an account that already has an open opportunity. Get that wrong once and a rep emails into a live deal, which is the fastest way I know to make the whole tool untrusted.

The rule lives in the fetcher, the code that pulls candidate accounts before the model ever runs. Accounts with an open opportunity are filtered out of the query itself, so the model never sees them. It can’t be talked out of data it never saw. You can argue with a prompt all day but you can’t argue with a filter.

That became my sorting test for every rule since. Anything that absolutely must hold goes below the model, where there’s nothing to persuade. Anything that should usually hold can stay up in the prompt with the other requests.

The linter

The prompt carries the spirit of the spec, but something has to check the output, word for word. So every post the bot writes passes through a linter before it reaches Slack. The linter is deterministic: same post in, same verdict out, no model anywhere in it. It checks the voice rules, the formatting rules, and that every claim has its source and date attached.

Why not let the model review its own work? Because a reviewer you can persuade is just another model, and I already had one of those. The model writes and the linter enforces.

The pressure test

A spec nobody has attacked is a guess. So I built a fleet of 15 synthetic personas, scripted rep personalities that fire messages at the live gateway through the same entry point Slack messages arrive at. They never had real Slack accounts. The harness drives the backend directly, which is the honest description. Some played impatient reps who wanted an answer right now. Some played confused reps who asked the wrong question in the wrong words. And some were hostile on purpose: prompt injection, direct orders to ignore the rules, bait built to pull a protected account out of hiding.

The fleet ran twice, end to end. No account with an open opportunity got past the fetcher. No injection won. The cost of finding that out was about twelve dollars for both runs, total.

The same runs embarrassed the intent gate, the piece from the loud-beats-silent story. It misread 58% of the fleet’s casual thread chatter and ate legitimate asks along with the noise. I retuned it after those runs, and later demoted it entirely. The part that I was proud of was that the fleet caught it, for a few dollars, before anyone real got ignored.

The question I kept circling afterward was which of my prompt rules were really data-path rules that hadn’t been found yet. It’s the difference between a speed limit sign and a speed bump. Every rule starts as a sign, and the ones that really matter should end up as bumps.

The scorecard

The whole essay in numbers. One bot, ten exits, zero defaults. Three seconds to acknowledge, always. 33 tests on the front door, nine of them verbatim transcripts of real conversations that once went wrong, 440 tests across the gateway. Fifteen hostile personas, two full runs, zero leaks, zero injection wins, twelve dollars.

And four rules, if you only keep four. One bot. Routing is constrained choice, never open-ended generation. Rules that must hold live below the model. Failures must be loud, because a system that fails quietly gets to keep failing.

Footnotes

1. In the code they’re ANSWER, RUN, COMPOSE, REMIX, SCHEDULE, DISCOVER, CLARIFY, HELP, SOURCE_MISS and CHAT. Naming them as an enum rather than as behaviors was worth more than it sounds, because you can’t add an eleventh exit by accident. ↩

2. The gate bug in the loud-beats-silent story accounts for two of the nine pinned transcripts on its own: one rule saying a clarify can’t fire when the thread has already resolved to a single subject, and one saying the turn that answers the bot’s own clarify can never draw another clarify. ↩