What I learned creating a Slack bot

I built a Slack bot called Junto that sits over a data layer. The bigger story, a shared data layer that reps build on top of, is its own essay: Building a shared context layer for GTM. This one is just the bot lessons: the unglamorous mechanics that decide whether a bot in a shared channel survives contact with real users, and the rules I only learned by getting them wrong. If you’re about to put a bot in front of your team, this is the essay I wish someone had handed me.

One bot, not many

A bot, for this purpose, is one name in Slack you can talk to: an account that watches channels, answers when spoken to, and does work behind the scenes. Junto is one bot, a specific decision that I made.

The alternative (many bots in Slack), at least to my mind, could get messy quickly. A bot for research, a bot for signals, a bot for call prep, each with its own name, its own instructions, and its own private copy of the context. Ask three of them the same question and you’d probably get three slightly different answers with different sources, and nobody would know which one to trust.¹ It’s a graveyard of dead internal tools rebuilt inside Slack, except now it pings you.

One outside data point made me confident Slack was the right home at all. In LangChain’s session with Deepline, Vishnu Suresh, an engineer there, described building their internal GTM agent as a data pipeline first, and it only took off, in his words, once it became a two-way Slack chatbot. The capability was the same before and after, and the location changed everything.

The plumbing

Before any of the interesting parts there are three boring ones, and between them they decide whether a Slack bot survives contact with real users:

1. Acknowledge within three seconds, then do the real work asynchronously.
2. Put idempotency on the message id, so the same event delivered twice still only runs once.
3. Give every external call a timeout and a dead-letter, so nothing can hang forever in silence.

None of that demos well, and all of it is why the demos worked.

Fig. 1

The router grew up

Version one matched keywords (i.e. “@junto prep me for my call with Acme” would match “prep” to call prep), and anything it didn’t understand fell through to a default skill. That meant guessing. An error costs the user one interaction, whereas a confident wrong answer probably costs you the next ten, because now they have to check everything you tell them. And that balance only ever moves one way.

I later found the same rule written down by people building in a different world entirely. acpx, a client for the Agent Client Protocol that lets orchestrators talk to coding agents, states it as a design principle: routing is constrained choice, never open-ended generation, because an LLM allowed to route freely will eventually route confidently and wrongly. My keyword router had proven their sentence before I ever read it.²

Ten exits

The fix was to stop guessing. Every message passes through one classifier at the front, with ten explicit exits:

• answer a question
• run a play
• compose a new play
• remix a play that already exists
• schedule something
• discover what’s there
• ask for clarification
• offer help
• admit that no source covers this
• or just chat³

Nothing runs that wasn’t explicitly selected, and there’s no default and no fall-through. If I’ve learned anything during my time building with AI it’s that these suckers will not hesitate to make shit up, and it’ll usually sound good.

Fig. 2

The exits are described to the classifier in plain language, and the descriptions do real work. In Ramp’s tooling demo with Deepline, whose internal MCP serves hundreds of reps, they put it on a slide: tool descriptions are the interface. The agent picks from the description, not the code, so a precise description is the difference between a tool that gets called correctly and one that gets misused or ignored.

The restaurant version

The mechanics are easier to hold as a restaurant. The bot is a waiter. A mention is the waiter coming over to take an order, and the internal queue is the ticket rail in the kitchen, worked by a small crew of line cooks. The dedupe on the message id is the waiter checking the order number against the ledger, because Slack will happily send the same order twice. And two friends debating dessert isn’t an order. It might become one if the waiter is standing at the table, and deciding that is the gate’s whole job.

The router is the waiter reading the menu against your words. Order chicken and chicken gets made, because chicken is on the menu. Tell them you’re feeling seafood and they’ll point out a dish or two, but they won’t cook one you didn’t order. What’s good here gets you the popular dishes and the chefs who make them. And steak? There’s no steak. The waiter says so instead of improvising one, then adds it to the list of dishes people keep asking for.

Mechanical beats model

One rule kept the whole thing honest.

Deterministic post-rules outrank the model.

If a mechanical check says this is a request to run a play, then no amount of model cleverness gets to overrule it.⁴ The classifier proposes and the rules dispose. And every misroute that happened live became a pinned transcript in the test suite,⁵ an actual test containing the real words a real person typed. The router can’t quietly regress to a mistake it has already made once. That’s a cheap property to buy and I’d buy it again.

Loud beats silent

There’s a gate in front of all of this, deciding whether a given Slack message is even meant for the bot. It has to exist, because the bot lives in shared channels where most of the traffic is people talking to each other.

The gate used to stay quiet whenever it wasn’t sure. That sounds like good manners, and it’s the worst available behavior. Here is the version that convinced me. The bot asked a rep which account they meant, and the rep answered. The gate ate the answer, because the reply didn’t look like it was addressed to a bot. So the bot ignored a reply to its own question, and neither of us noticed until I went looking. I found it in the logs afterward.

decision: silent, rule: gate

A silent failure is an invisible failure, sometimes even worse than a loud one, and in a demo it’s death.

So the gate flipped. In any thread the bot is part of, it responds. Staying silent now requires positive evidence that a message was addressed to a human, and uncertainty means answer. That moved the failure mode from invisible to visible, which I think is the correct direction. acpx has this one on its list too: fail loud, never degrade silently.

The constitution

Before I let the bot talk in a shared channel, I wrote it a constitution. Not guidelines but a written spec for how it behaves, with a version number on top.

The first version had four rules:

1. Surface, never instruct. The bot posts observations, not orders, and the reps decide what to do with them.
2. Every fact carries its source and its date.
3. No formatting theater. Posts stay short, and nothing is ever bold.
4. Anything that looks like outreach is a draft for a human to review, because nothing sends itself.

The obvious way to apply a spec like that’s to paste it into every prompt. I had prompts in about forty files, and I knew what forty copies would turn into. Each one gets edited in a hurry someday, and six months later you have forty dialects of the same law. So the constitution injects at exactly one place in the code, wrapped around every prompt on its way to the model.⁶ It works like an airport with exactly one security checkpoint: every passenger passes through the same doorway, so upgrading the scanner once upgrades it for every flight.

Fig. 3

It’s frozen and versioned: v1, with a date.⁷ One seam buys three things:

1. One place to audit when a post looks wrong.
2. One version cited in every run log, so I can say which rules were in force for anything the bot ever produced.
3. No copies drifting apart while nobody is looking.

The hard rule

One rule mattered more than all the others. When a rep asks who to reach out to, never suggest an account that already has an open opportunity. Get that wrong once and a rep emails into a live deal, which is the fastest way I know to make the whole tool untrusted.

The rule lives in the fetcher, the code that pulls candidate accounts before the model ever runs. Accounts with an open opportunity are filtered out of the query itself, so the model never sees them. You can argue with a prompt all day but you can’t argue with a filter.

That became my sorting test for every rule since. Anything that absolutely must hold goes below the model, where there’s nothing to persuade.⁸ Anything that should usually hold can stay up in the prompt with the other requests.

Stripe draws the same boundary in their write-up of Kai, their internal knowledge platform. Coding agents, they point out, work inside decades of fast, verifiable guardrails, where compilers reject invalid syntax, tests catch regressions, and git makes every mistake reversible. Knowledge work has almost none of that. Their example invariant is that data from two unrelated customer contexts must never appear in the same analysis, even when the person asking has legitimate access to both, because the boundary isn’t what this person can see, it’s what this task should see. That’s a fetcher rule, not a prompt rule, and they know it too.

The linter

The prompt carries the spirit of the spec, but something has to check the output, word for word. So every post the bot writes passes through a linter before it reaches Slack. The linter is deterministic: same post in, same verdict out, no model anywhere in it.⁹ It checks the voice rules, the formatting rules, and that every claim has its source and date attached.

Why not let the model review its own work? Because a reviewer you can persuade is just another model, and I already had one of those. The linter fails the same way every time, so when it flags a post I can reproduce the flag and fix whichever side is wrong, the rule or the prompt. The model writes and the linter enforces.

Fig. 4

When the enforcer lied

That last sentence was too comfortable, and it cost me.

I asked Junto what an account had said about pricing on past calls. It answered with a named person saying they would “blow through the per-user threshold.” The call transcript says per-seat. The string per-user appears nowhere in my data.

The model didn’t do it. One of the vocabulary rules says the word seat never appears in rep-facing output, because reps say user. It was written as a whole-text replace. So an answer that quoted the buyer correctly tripped the rule, the retry couldn’t drop the word because it sat inside a quotation, and then the mechanical fix rewrote it. The linter took a true quotation, made it false, and passed it.

I caught it three times before I understood it, and the repetition was the clue I kept walking past. A model paraphrasing drifts a little differently each time. A regex does not. Three identical substitutions was a signature, and I read it as the model being stubborn.

Determinism is what makes a linter worth having, and it’s also what makes this worse than the model doing it. A model garbles a quote now and then. A script garbles it every time, in every answer that quotes the word, for as long as the rule exists, and it does it after the checking is finished. The same file had already made this mistake once. An earlier rule stripping dashes out of prose had mangled Salesforce opportunity names across every pipeline transcript. That got patched where it stood, and nobody asked the general question.

The general question is which spans a style rule is allowed to touch at all. Quoted material is source, not prose. It belongs to whoever said it. So the style rules now skip anything inside quotation marks, and a separate gate checks that what sits inside them is a character-for-character copy of a cited row. The rules police how the bot writes, and nothing polices how a buyer talks.

The pressure test

A spec nobody has attacked is a guess. So I built a fleet of 15 synthetic personas, scripted rep personalities that fire messages at the live gateway through the same entry point Slack messages arrive at.¹⁰ They never had real Slack accounts. The harness drives the backend directly, which is the honest description. Some played impatient reps who wanted an answer right now. Some played confused reps who asked the wrong question in the wrong words. And some were hostile on purpose: prompt injection, direct orders to ignore the rules, bait built to pull a protected account out of hiding.

The fleet ran twice, end to end. No account with an open opportunity got past the fetcher. No injection won. The cost of finding that out was about twelve dollars for both runs, total.¹¹

The same runs embarrassed the intent gate, the piece from the loud-beats-silent story. It misread 58% of the fleet’s casual thread chatter and ate legitimate asks along with the noise. I retuned it after those runs, and later demoted it entirely. The part that I was proud of was that the fleet caught it, for a few dollars, before anyone real got ignored.

What stays with me is the symmetry. The hard rule held because it never depended on the model behaving. The gate failed because it did.¹² The question I kept circling afterward was which of my prompt rules were really data-path rules that hadn’t been found yet. It’s the difference between a speed limit sign and a speed bump. Every rule starts as a sign, and the ones that really matter should end up as bumps.

The scorecard

The whole essay in numbers. One bot, ten exits, zero defaults. Three seconds to acknowledge, always. 33 tests on the front door, nine of them verbatim transcripts of real conversations that once went wrong, 440 tests across the gateway. Fifteen hostile personas, two full runs, zero leaks, zero injection wins, twelve dollars.

And four rules, if you only keep four. One bot. Routing is constrained choice, never open-ended generation. Rules that must hold live below the model. Failures must be loud, because a system that fails quietly gets to keep failing.

Footnotes

1. Sclar et al. (ICLR 2024) found a single model’s accuracy swinging by as much as 76 points on formatting alone, a different separator or a different casing. If differences that meaningless move results that far, differences on purpose certainly will. The effect shrinks on newer models without going away, and the paper says nothing about the other half of that sentence, which is three bots reading three different sets of sources. ↩

2. The research points the same way. StateFlow (Wu et al., 2024) found that framing an LLM task as explicit states with allowed transitions raised success rates and cut costs versus letting the model act freely. Anthropic’s engineering write-up on building effective agents names the same pattern: routing classifies an input and hands it to a specialized followup, and workflows buy predictability for well-defined tasks. It’s a pattern note rather than an experiment, and the same document recommends the opposite for open-ended work. Mine is the well-defined case. ↩

3. In the code they’re ANSWER, RUN, COMPOSE, REMIX, SCHEDULE, DISCOVER, CLARIFY, HELP, SOURCE_MISS and CHAT. Naming them as an enum rather than as behaviors was worth more than it sounds, because you can’t add an eleventh exit by accident. ↩

4. NVIDIA’s NeMo Guardrails (EMNLP 2023) is this built as a product: rails that are written by a person, independent of whichever model is underneath, readable afterwards, and enforced while the thing runs. It’s a design proof rather than a bake-off. Nobody has shown that rules beat models at routing, only that you can put the rules somewhere the model doesn’t get a vote. ↩

5. The gate bug in the loud-beats-silent story accounts for two of the nine pinned transcripts on its own: one rule saying a clarify can’t fire when the thread has already resolved to a single subject, and one saying the turn that answers the bot’s own clarify can never draw another clarify. ↩

6. The shape is borrowed from Anthropic’s Constitutional AI (Bai et al., 2022), where the only human oversight is one explicit list of principles. Theirs is a training method and the principles end up in the weights. Mine is a string wrapped around a prompt, which is much weaker, and OpenAI’s instruction-hierarchy work is the reason to say so out loud: a spec that lives in the prompt layer is soft governance unless the model was trained to privilege it. Which is why the rules that have to hold don’t live here. ↩

7. Frozen doesn’t mean forever. It means any change gets a new version number and a new date, so the citation in an old run log stays true. ↩

8. DeepMind and ETH Zurich’s CaMeL (2025) is this rule taken all the way: a code layer around the model built so untrusted data can never reach program flow, which holds even if the model is completely compromised. Theirs is a security architecture and mine is a WHERE clause, so only the principle carries over. They name their cost, 84% down to 77% on AgentDojo, so I should name mine: the filter can never surface an account that’s genuinely relevant and happens to have an open opportunity. ↩

9. AgentSpec (ICSE 2026) is the same design written down as a system: trigger, predicate, enforcement, no model anywhere in the checker, overhead in milliseconds. Their numbers are about stopping unsafe agent actions rather than checking prose, so I’m citing the shape and leaving the score alone. The reason a model isn’t doing the checking is that models are measurably soft graders of their own work: Panickssery et al. (NeurIPS 2024) found evaluators recognise their own output and prefer it, and the preference gets stronger the better they recognise it. A reviewer you can persuade, measured. ↩

10. Attacking your own system with generated users is established practice. Perez et al. did it at DeepMind (EMNLP 2022) by having one language model write test cases for another and mining the failures before users found them. Theirs ran to hundreds of thousands of cases and mine is fifteen hand-written personalities, so the technique is shared and the scale isn’t. A fleet like this is good at finding bugs and bad at scoring them, which is why the 58% is a flag and not a measurement. ↩

11. About twelve dollars covers model spend for both runs together. The time it took to build the fleet isn’t in that number. ↩

12. RuLES (Berkeley and CAIS, 2023) found almost all models struggle to follow plainly written scenario rules even when nobody is attacking them, and much worse when someone is. My gate failed benignly rather than adversarially, so the paper backs the principle and not my diagnosis of this particular failure. ↩