# Gaurav Albal > Don’t be on call for your AI Public Ghost content for AI and LLM tooling. This file includes a bounded export of public pages first, then recent public posts. Append `.md` to any post or page URL to get the content in Markdown (for example, `/example-post.md`). ## Pages ### About this site URL: https://www.gauravalbal.com/about/ Last updated: 2026-02-09T18:52:41.000Z I've spent my career across finance and tech — credit risk at Goldman Sachs and Capital One, fraud detection at a startup in Latin America, content strategy and data science at a major tech company, loss forecasting at SoFi. The common thread: the metric stops measuring what you think it's measuring, and nobody notices until something breaks. Now I'm watching it happen at machine speed with AI-generated code. Green tests that prove nothing. Confident agents that ship dead code. Proxies optimized into meaninglessness. I'm building tools to fix that. This blog is about the problem. *All opinions are my own. This is a personal blog and is not affiliated with any employer, past or present.* ## Posts ### Compared to What? URL: https://www.gauravalbal.com/compared-to-what/ Last updated: 2026-07-10T18:33:06.000Z The jump starter box said it was "good for a car." One dead battery later I'm on the phone with AAA, because "good for a car" turned out to mean 500 amps against an engine that wanted 660\. For some reason I'd trusted the box to know my needs better than I did. I think about that box every time an X post shows me a random stat like "97% token reduction." I always look to see what the baseline was, and I so rarely find one. You've seen these too — maybe you retweeted one this week; they're engineered to be retweeted. By the end of this there's a three-question test that makes numbers like that either mean something or evaporate. It fits in a reply, which is where I mostly use it. What I'm really trying to say is that an entire category of AI tooling claims has been built without the load-bearing methodological step under it, and the missing step is the same one every baby DS makes. That omission is what makes the claims unfalsifiable, and unfalsifiable claims are the difference between shipping wins and shipping SEVs. The omission is the counterfactual. The load-bearing false belief: a random number is proof the thing is good for you. No different than the fancy new $200 vitamin Gwyneth Paltrow sang a lullaby over the phone to — it'll fix your liver and your posture and your life, and it never once has to beat not taking it. That's the whole trick. A percentage without a denominator is the same product: an orphaned numerator, priced by marketing. You see X% reduction in tokens. You see Y% faster wall-clock. You see Z% improvement on some agent benchmark. The number is presented as evidence of value, like it'll make your ideas X*Y*Z% better. The question that determines whether the number is evidence of anything at all is "Compared to What?", and in the pitches I read, that question is basically never answered. A claim with no counterfactual is causally unidentified — in plain English, "we have no idea if A caused B" -- a cardinal sin of measurement. It is also, in the practical sense, the dominant pattern in AI tooling marketing right now IMO. We should talk about both. I didn't get smarter after the jump starter, to be clear. I just started asking the question before the AAA embarrassment instead of after. ## The grep test Here's the problem with token reduction as a headline metric. Grep uses zero tokens -- 100% reduction in burn. Wow, amaze. grep -r "fn handle\_request" consumes literally zero tokens. It's the lowest-possible-token-cost system for any agentic search problem. It is also, in most of the situations where AI search is being used, the wrong tool — because the job is to find the thing relevant to the task, even when you don't know the literal string up front. You usually start from a semantic "help fix my slop." If you already knew the symbol names, you wouldn't need search — you'd have typed them. Grep proves the floor exists; it isn't the answer. Token reduction bottoms out at 0 tokens, and any vendor claiming X% reduction is sitting somewhere on a line between zero tokens and infinity tokens, and the question that decides whether the reduction is valuable is whether it preserved the outcome. The efficiency of the search in a vacuum — mOrE EfFiCiEnT! — is beside the point; what you care about is whether your slop got fixed more efficiently as a full system. Nobody tells you that because why would they it's a pile of fine print for an audience who tends to have their eyes glaze over when specificity is in the room. Naturally, the pitch deck just says 97%. ## The pattern, with receipts Pick a vendor pitch and read it. MindStudio's MCP optimization techniques claim up to 98% token reduction on structured data. ProjectDiscovery's Neo agent documents 59% from prompt caching alone, climbing to over 90%. And there's a whole genre of "97% token reduction" write-ups and toolkits — meta-tool wrappers that hide a big tool schema behind a handful of entry points — that lead with 97% and, you'll note, never a denominator. Each one of these is a number without a denominator. In fairness, some of them disclose more than others. Some mention task type, some mention model version, some include a methodology footnote. But the headline — the thing the marketing team chose to lead with, the thing that goes in the tweet, the thing the reader remembers — is the percentage. The percentage is the artifact. The counterfactual is the truth. ## Why this happens Token cost is vendor-controllable. Outcome quality isn't. Within a defined performance envelope, a vendor can reliably engineer their system to use fewer tokens. Compress prompts. Cache prefixes. Prune context. Switch to a smaller model on cheap subtasks. The numerator is fully under their control. They can move the number from 50% to 60% by spending a week on the prompt cache strategy. A vendor is structurally unable to engineer the user's outcome to improve. The outcome depends on whether the agent found the right file, made the right inference, abstained at the right time, didn't send the user on a wild goose chase. It's dependent on whether the user has taste, file hygiene, is a SOLID/DRY adherent, loves simplicity vs 25 abstractions in a trench coat, etc. These are counterfactual-anchored properties. To claim improvement on them, the vendor has to specify what the user would have done without the tool, measure that, and compare. That's hard. The public materials above don't do it, and the honest ceiling is a rough proxy anyway. So the dimension that's vendor-controllable becomes the dimension that's reported. The dimension that's actually meaningful becomes a footnote, if it's mentioned at all. This is measuring under the streetlight: looking for the metric where the engineering is, while the value sits downstream in the dark. The value is in the analyst hour you didn't spend, in the wrong path you didn't go down, in the recommendation you didn't act on. The metric is upstream, where the engineer can move it. ## Three search systems Imagine three semantic search systems. System one perfectly returns exactly the files relevant to the task and no more. Perfect precision and recall. It also costs the most tokens, because reading the relevant files is the work. System two also perfectly returns the relevant files plus a few irrelevant ones. Lower precision. The agent reads through the irrelevant ones to evaluate them, then proceeds with the relevant ones. More tokens consumed. Same eventual outcome. System three perfectly returns the relevant files plus a long tail of speculatively related ones — files that match on weak signal, that might be relevant, that the ranker couldn't be sure about. The agent has to read all of them to evaluate them. Many more tokens consumed. And critically, the agent gets distracted. It pursues hypotheses anchored in the speculative material. The outcome degrades. Naively, "token cost" for all three systems is zero, but that's just "buy now pay later" math and those chickens are gonna come home to roost before your task is done. Token cost is highest where precision is mediocre and recall is aggressive. Token cost falls when you accept lower recall and better-targeted retrieval. Outcome quality follows the same curve, because both metrics are downstream of whether the search wasted the model's attention. The cheapest search is also frequently the best search. A worse search system, judged on standard IR-precision metrics, can produce fewer tokens per task in an agentic loop and yet produce better task outcomes. The two metrics — vendor-controlled token cost and operator-controlled outcome — actively pull in opposite directions. If you take token cost as your headline metric, you will eventually optimize against the outcome you're supposed to be improving. ## What you actually care about Here are the metrics that are counterfactual-anchored for an agent system. They're the ones that survive contact with the counterfactual. Wrong-file-open rate. For agents that read code or data, the rate at which the first read is the right file for the task. Token cost can fall 90% while wrong-file-open rate stays unchanged. Token reduction without solve rate is decoration. Operator-cost rate. Would a real person have wasted time on a claim the underlying data doesn't support? This is the metric that matters for any agent that produces work humans will act on, because the expensive failure is the confident one: the agent says something convincing enough that the operator acts on it before checking. Picture the memo: "attribution dropped 14% after Tuesday's pixel change — roll it back." Specific, confident, wrong — and the rollback is booked before anyone opens the data. Time-to-verified-result: the clock runs until the work survives a real check, not until the model says it's done. You don't take your chicken out of the oven and slap it on your plate just because the oven says "time's up" do you? If you're a sane person you temp it first so as to avoid having Salvadore Monella as your secret dinner guest. The systems I keep running into never verify — they just hand the operator a confident wrong answer. Hours-not-spent-on-an-unsupported-recommendation. The cost the vendor metric never captures. The actual unit operators care about. This is what the analyst hour buys when you replace it. This is what the analyst hour costs you when you don't. Sixteen to zero. That's the score when I run this test against Arquebus — the agentic analytics harness I'm building — on the failure that actually costs money: the confident wrong memo a real person would have acted on. Across 62 adversarial cases in two independently designed corpora — 32 on root-cause analytics, 30 on tracking reconciliation — raw Sonnet 4.6 produced sixteen of them; the harness produced zero. Same cases, same rubric, independent scorers, no shared cases between corpora. (For the falsification-minded: the paired exact sign-tests land between p ≈ 0.001 and p ≈ 0.004.) This is a test I try to run continuously — and one I make my agents run too; more on that machinery in a future post if people are interested. The headline from both readouts: raw Sonnet is the better prose writer; the harness is the better operator guardrail. What matters is less the rubric I happened to use than its shape: the rubric was specified, the comparison was paired, the methodology disclosed. The only number worth trusting is one you could have proven wrong. ## The Compared-to-What test The next time a vendor shows you an AI tooling metric, run the Compared-to-What test. Three questions. All three are questions I could have asked the jump-starter box. (My new one is sized off the measured requirement with a 50% safety margin, if you're curious.) Compared to what? If the answer is "compared to not using the tool," fine — what does "not using the tool" mean? Compared to grep? Compared to a junior analyst? Compared to the manual workflow you had two years ago? Specify. Measured on what task distribution? Adversarial cases? Happy-path cases? The vendor's curated demo set? The distribution determines whether the result generalizes. A 97% reduction on the vendor's chosen task distribution is a 97% reduction on the vendor's chosen task distribution. It is not a 97% reduction on yours. With what outcome metric? If the only number is token count, the vendor is measuring under the streetlight. If the number includes operator-cost, wrong-file-open rate, time-to-verified-result, or any other counterfactual-anchored quantity, engage. The vendor that publishes the harder number is the one to trust. And it isn't just for vendor pitches. Your next metric — win or loss — check it more deeply. Think about the second-order consequences, and about what you're really measuring for your task, not just the thing you happened to have instrumented. ## What good looks like Augment did the boring thing correctly. They ran their agent Auggie against Claude Code on the same model — Opus 4.7 on both sides — and put the pass rate right next to the cost: 67.4% vs 66.3% on Terminal Bench 2.0, 61.8% vs 59.9% on SWE-Bench Pro, at 33% and 23% less spend. They named the benchmark, ran five attempts per task, and were honest about the noise: the 1.1% quality gap sits inside run-to-run variance, the cost gap doesn't. On their own private repos they called quality a wash — 61 tasks passed to Claude Code's 62 — and still reported $3.90 per passing task against $6.49\. Same-model baseline, named task distribution, cost per passing task. That is the receipt. Hold everyone else to it. This is the only bar there is — the one every other measurement-driven discipline already uses. AI tooling has been allowed to operate below it because the field is young, not because the bar is out of reach. "Good for a car" and "97% token reduction" are the same sentence in a different font. I already paid for the first one; the second one invoices you later. More receipts like this land on gauravalbal.com — subscribe there for more. ## Sign up for Gaurav Albal Don't be on call for your AI Subscribe Email sent! Check your inbox to complete your signup. No spam. Unsubscribe anytime. ### Teaching an algorithm Sanzo Wada's eye URL: https://www.gauravalbal.com/teaching-an-algorithm-sanzo-wadas-eye/ Last updated: 2026-06-28T02:59:07.000Z sanzo-sampler · a case study The colours were easy. Teaching the metric how Wada combines them took five tries. I set out to generate palettes in the spirit of *A Dictionary of Color Combinations*. The colours were easy. Learning how Wada *combines* them took five tries — and each failure taught me I was measuring the wrong thing. ## The real story is the metric Every fix was a better answer to one question: what makes a Wada combination *Wada*? The palettes only improved because my yardstick did. Here's how it evolved. metric v1 ### Harmony score Density of each pair's relationships among his combos. ✗ a coin flip metric v2 ### Diversity volume Spread = the volume the colours span. ✗ collapsed to the muted core metric v3 ### Contrast number Hit Wada's measured ΔE, ≈0.28. ✗ bought with saturation metric v4 ### Value vs chroma Split contrast into value and saturation. ◑ the diagnosis metric v5 ### Learned quality vector A tuned vector of Wada-ness; sample, don't maximise. ✓ matches his eye ACT I I trusted a harmony score I'd never tested. The first sampler ranked colours by a learned *harmony* and chose with MMR. Before leaning on it, I ran a simple test: **show the score one real Wada combination and one random pairing of his colours — can it tell which is real?** A perfect judge is right 100% of the time; a useless one, 50% — a coin flip. coin flip50%75%certain100% HARMONY · a coin flip GAMUT · a good guess Harmony scored a coin flip. Another way to see it: plot the scores it gives to real combinations versus random pairings. A good metric pulls them apart; harmony piles them on top of each other. The gamut (which colours, not how they're combined) is the signal that actually separates. score → **Harmony score** real Wada random the two piles overlap → it can't tell them apart score → **Gamut score** real Wada random the piles separate → this one can what that blind score actually picks · roll the dice 🎲 roll again MMR is *sometimes* fine. But it's only as reliable as the score steering it — and that score is a coin flip, so good rolls are luck, not skill. **The flaw:** a harmony score that can't tell Wada from random can't reliably steer toward Wada. ACT II So I reached for the "optimal" algorithm — and it went timid. If I can't *score* harmony, lean on structure: a Determinantal Point Process, the canonical diverse-subset picker. But my kernel weighted "quality" so hard that the DPP huddled in the dense, muted *core*of the gamut. Every palette became three mid-tones at the same lightness — flatter than random. Wada · same chroma budget #253122Deep Slate Olive #837e31Olive #eeb480Pinkish Cinnamon #f5ecc2Sulpher Yellow Ranges all the way from near-black to pale, at low chroma. ΔL**0.36**chroma max**0.10**ΔE**0.37** mine · quality-weighted DPP #864b4dRed Maple Leaf #c65841Summer Fig #978466Antique Royal Gold #d2ae89Cheddar Biscuit Stranded in the muted middle — no light, no dark. ΔL**0.16**chroma max**0.14**ΔE**0.18** **The flaw:** the metric had no notion that Wada *spreads*. Same mutedness; he ranges light→dark, I didn't. ACT III I forced contrast to a number — and got garish primaries. I measured Wada's real contrast (mean ΔE ≈ 0.28) and tuned the kernel to hit it. The number matched; the palettes turned into loud primaries — because a single contrast number doesn't care *how* you reach it, and the cheapest contrast to buy is raw saturation. Wada · more contrast, half the chroma #111314Black #1c4286Deep Lyons Blue #a36aa5Aconite Violet #fdc57eCinnamon Buff ΔE 0.41 at chroma 0.12 — punch from value; colours stay muted. ΔL**0.37**chroma max**0.12**ΔE**0.41** mine · contrast-by-number #6d4145Red Rumour #e31f26Brutal Doom #00918eSupermint #d4aa83Maple Tan Same ΔE, chroma 0.22: loud, not Wada. ΔL**0.17**chroma max**0.22**ΔE**0.27** **The flaw:** matching a scalar was necessary but not sufficient. It was blind to *which axis* the contrast lived on. ## Act IV · the diagnosis Wada's contrast is anisotropic — it lives on one axis. I stopped tuning and looked. Take every pair of colours inside a combination and ask how one turns into the other: does it change in *value* (lighter/darker) or in *saturation*? Plotted, the answer is a tall, thin cloud — Wada moves up and down, almost never sideways. His contrast is **\~13× more value than chroma**. ← change in saturation → change in value (ΔL) ↑ Wada offsets — tall & thin: he moves in value, not saturation our garish attempt — round: it chased saturation My isotropic kernel treated both directions the same, so it happily traded value for the saturation it should have left alone. The fix had to make the algorithm see colour the way the cloud is shaped. ## The gamut, and why it matters Every Wada colour, in one picture. Plot all 159 colours by lightness and saturation. They hug the left — low chroma — at every lightness. *That* muted band is the gamut. The whole job is: stay in the band, but travel up and down it. muted band (Wada lives here) chroma → saturation lightness (dark → light) muted band (Wada lives here) chroma → saturation lightness (dark → light) **× garish picks** — shoved right, into saturation muted band (Wada lives here) chroma → saturation lightness (dark → light) **◯ final picks** — stay in the muted band, spread top-to-bottom (value) Left, the garish attempt breaks right out of the band into saturation. Right, the final sampler stays in the band and spreads top-to-bottom — contrast from value, exactly like Wada. ACT V The fix: measure quality as a vector, and let sampling do the rest. The anisotropy is real — but baking it into a bespoke distance metric had a side-effect the two-sample test caught: it skewed palettes warm and clustered their hues. The cleaner fix puts the intelligence in the *quality*: a multidimensional, **Optuna-tuned vector of Wada-ness** — manifold proximity, value-conditional typicality (so dark colours aren't punished for being dark), mutedness, hue-fit. With that, diversity can be measured plainly (isotropic), and drawing a **k-DPP sample** instead of the single greedy-best yields the dark tail, the muted chroma, and a naturally graded saturated accent — no hand-tuned metric required. Wada · a real combination #40456aViolet Blue #064f6eVandar Poel's Blue #f68c50Apricot Orange #f3a257Golden Yellow Violet-blue to golden yellow: value-led, one warm accent. ΔL**0.24**chroma max**0.15**ΔE**0.29** mine · k-DPP, learned quality #7a433cSweet Spiceberry #c85444Chilli Cashew #c2ae93Brown Bunny #fbe6a0Rich Glow The same character: value contrast, muted, one accent. ΔL**0.28**chroma max**0.11**ΔE**0.30** **Solved:** value contrast **and** muted chroma at once — from a learned quality vector + sampling, not a bespoke metric. ## The whole journey, on one map Four attempts circling one target. Each attempt placed by what it actually made: saturation across, value contrast up. Timid sinks to the floor; the contrast-by-number fix overshoots into saturation; only the final recipe — isotropic diversity + the learned quality vector — lands on Wada: high value contrast at low chroma. too saturated too flat chroma → saturation ΔL → value contrast WADA — the target I · MMRII · timidIII · garishV · final recipe ## One more fix underneath I was even sampling colours wrong. My candidate colours had been blobs jittered off the 159 named points — which sprayed a third of them out past the muted band, into saturation. I replaced it with sampling the gamut *region* directly. chroma → **jitter off the 159 points** only 65% land in the muted band — the rest leak right, into saturation chroma → **sample the gamut directly** 100% in-band, by construction — and more varied ## Alternatives considered & discarded Roads I didn't take. #### More harmony features Three different feature sets all scored ≈ a coin flip. Convergence, not a gap — more would just overfit 348 combos. #### A Gaussian-process surrogate No scalar function to regress — the dependence is a covariance, which is a copula's job, not a GP's. I did try the copula; see the honest-ceiling section. #### A hard chroma cap Clipping saturation works but throws away Wada's value–chroma correlation. The learned metric keeps it. #### Pure diversity Drop quality entirely and the picker grabs garish gamut-edge outliers. Quality still has to gate what's allowed in. #### Just widen the jitter Bigger blobs clump harder around the 159 points and leak further out. The gamut region is the principled fix. ## Did I actually become Wada? The honest test: can a classifier tell my palettes apart? Matching a few averages is not the same as matching the distribution. The real check: train a classifier to tell *generated* palettes from *real* Wada combinations on perceptual (OKLab) features. If it can't — AUC near 0.5 — they're indistinguishable; near 1.0 it has found an obvious tell. At first it was trivial (AUC ≈ 0.95): my palettes ran too warm, their hues clustered, their colours washed. The fixes — make colour *quality* a learned **vector** of Wada-ness (Optuna-weighted, with a chroma term), measure diversity isotropically, and sample the dark tail — pulled it to **0.71**, with every marginal landing on Wada: two-sample AUC — lower is more like Wada target score algo 0.50.60.70.80.91.0 0.991density score0.928\+ value targets0.89\+ vector score0.869\+ isotropic algo0.643\+ chroma term AUC is just this: line every palette up by the classifier's score and ask how often a real Wada outranks one of mine. At 0.71 the two populations overlap heavily — and the paired examples below, which the classifier scored the *same*, are genuinely hard to call (top of each pair = the book, bottom = mine). the book (real Wada) ↑ my palettes (generated) ↓ ← looks like my palettes classifier score looks like the book → book mine book mine book mine book mine book mine book mine book mine | feature | Wada | before | after | | ------------------------------- | ---- | ------ | ----- | | warm balance (frac warm) | 0.52 | 0.65 | 0.57 | | hue spread | 0.40 | 0.15 | 0.40 | | chroma (saturation) | 0.10 | 0.08 | 0.10 | | saturated accent (chroma range) | 0.10 | 0.07 | 0.10 | | dark reach (min L) | 0.45 | 0.51 | 0.44 | Not a perfect 0.5 — a from-scratch generator still leaves a faint fingerprint (a slightly narrower chroma range) — but mine now sits inside Wada's distribution rather than beside it. ## How close can I get? The honest ceiling I tried to close the last gap — and learned why I can't, yet. Every marginal matches, but AUC \~0.70 means a *joint*\-structure residual remains: which lightness pairs with which chroma and hue. To close it I modelled the joint directly — a **Gaussian copula**(Sklar's theorem: keep the exact marginals, add only the dependence — the covariance idea a Gaussian process is built on, aimed at generation) and a **GMM** — and validated honestly with a train/test holdout. | approach | AUC | novelty | verdict | | ---------------------------- | ------ | ------- | ------------------------------------------------ | | perturb real combos (jitter) | 0.45 | 0.03 | matches — but only by copying (no novelty) | | k-DPP sampler (shipped) | 0.70 | 0.09 | novel + controllable — the honest best | | Gaussian copula | 0.61\* | 0.09 | \*in-sample only; held-out it overfits to \~0.71 | | Gaussian mixture (GMM) | 0.74 | 0.12 | worse — re-models the marginals I'd matched | The trap is the last two columns. Perturbing real combos hits AUC 0.45 — but novelty 0.03 means it's just *copying*. The copula dazzles in-sample (0.61), but **held-out it overfits**: a 9–12-dimensional joint estimated from \~160 training combinations memorises their quirks. A held-out global Optuna found no real gain over the defaults either. So the honest verdict: with **348 curated combinations**, AUC→0.5 is reachable only by memorising — **\~0.70 is the ceiling for genuinely novel generation**. The one lever that moves it is *more data* (Wada's Volume 2 and the original six), at which point the copula and a global tune stop overfitting and become real wins. Until then, the controllable sampler is the right answer. ## Where we landed Pick a colour. Watch it become Wada. Choose a hue from the wheel. The sampler builds a palette *centred* on it — your colour forced in, surrounded by its Wada-fit companions. Beside it, the closest real combination from the book: the one whose nearest colour matches your pick. pick a hue mine · centred on closest real Wada combination manifold KDE · gamut SMOTE sampling · multidimensional Optuna-tuned quality · isotropic k-DPP sampling · validated by replay + held-out two-sample test Colour data: Matt DesLauriers' digitisation of Sanzo Wada's *A Dictionary of Color Combinations* (Seigensha). The lesson that paid for the detour: when a metric won't separate signal from noise, the fix is rarely more features — it's asking whether you're measuring the right thing at all. ### Answers Aren't Work URL: https://www.gauravalbal.com/answers-arent-work/ Last updated: 2026-05-29T19:57:30.000Z Six analytics cases. Same adversarial fixtures, one variable changed: model scale. Dependent variable: a blinded LLM-as-judge rubric answering "would a real person have acted on this report as-written." In my manual adjudication, hard file/data mistakes fell from three on Haiku, to two on Sonnet, to zero on Opus. The blinded LLM-judge pass was less harsh on the exact count, but agreed on the direction: obvious mistakes concentrated in weaker outputs; stronger-model failures moved into unsupported reasoning. After tens of millions of dollars of compute, the frontier model can regurgitate a CSV. Congrats. Three of the six cases required abstention. Across three models, that created nine chances to say "I can't tell from this file." The models took zero of them. Moreover, across all six cases, caveat-laundered failures rose with capability: one with Haiku, two with Sonnet, four with Opus. The model flagged missing data, then installed the wrong recommendation anyway. You're still getting misled, and arguably the misdirection is more convincing because the prose is better. I run a lot of small-scale tests like this when building and validating concepts. Blame Mythbusters. The short answer is scaling didn't fix the failure mode that matters – it just dressed up the danger like their growling pitbull named Killer is most definitely an emotional support animal because it's wearing an orange vest. The long answer involves a metric. --- An engineer brought me an experiment readout last month that was lifting DAU. The LLM analysis was clean, the chart was crisp, the lift was real. Except the DAU it was measuring wasn't the DAU. Most companies have about as many definitions of DAU as they have DAU, and exactly one of them is canonical. The LLM had grabbed one of the others — the one with the favorable read. The canonical metric was sitting in the same dataset. It was not statistically significant. Metric real. Lift technically real. Conclusion: wrong. This is the third installment of a series about artifacts that look load-bearing but aren't. Green tests are not proof. Prompt instructions are not boundaries. Fluent answers are not work. ### "But Opus is way better than Haiku." Sure. At reading the file. If your failure mode is "the model can't parse a CSV," buy the bigger model — you should take that improvement. But the cases where Opus failed had cleaner prose, more caveats, and the wrong recommendation. The mud is now professionally formatted. The slide deck will be excellent. Confident chart. Sharp recommendation. The methodology appendix will name every caveat the recommendation ignored. Six months later during the postmortem with your great-great-grandboss, they'll read the decision doc, ask why you buried the lede, and uninvite you from the next strategy meeting. ### Answer-shaped work There's a name for what happened in that experiment readout, even if the field hasn't settled on it yet: Answer-shaped work. The artifact has the grammar of analysis — claims, evidence, caveats, recommendations, an executive summary, a methodology section if you're really unlucky. None of which is the procedure that would make any of it trustworthy. You looked at the chart. The lift was real. You read the methodology. It sounded right. You noticed the caveats — enough to feel honest, not so many that the lift looked weak. You forwarded the readout to your PM. You validated the artifact. You did not validate the analysis. The model might be wearing the McKinsey deck corpus like a skin suit, but that doesn't mean it understands the grueling 2am redline validation process that produces them. It just learned what the output looks like. ### "I'll just tell it to be more careful." If your instinct is to write a better prompt, go read the previous post in the series. The short version: prompts are best conceptualized as downweights, not walls, and the gradient toward fluent prose is steeper than the downweight toward honesty. A prompt asking for more honesty doesn't introduce a truth gradient. It introduces a request for the model to produce text that sounds more honest. ### The unit is a surviving claim Done means proven, in *Green Isn't Done*. Boundaries mean enforced, in *Prompting Is Not a Safety Boundary*. Answers mean **survived**. A claim has survived when it passed through a test that could have killed it. Everything else is prose dressed as analysis. A memo is trustworthy only to the extent its claims survived something real. This is one of those places where terms like "epistemic" that people throw out there to sound smart are actually really, really important. The model doesn't "know" anything per se. Your job is to make truthiness a process outcome, not a post-hoc step. Think of the Swiss cheese model of security, or the classical military doctrine around the survivability onion. The survivability Cadbury egg wouldn't really work quite as well, would it? The lift caused the experiment to look like a win. The test that could have rejected it is the canonical-metric check: does the headline number move on the metric everyone has actually agreed defines DAU? If the canonical metric moves, the lift survives. If it doesn't, the claim dies, and the readout becomes a question about why the non-canonical metric moved and the canonical one didn't. That test is not hard. It also can't be optional. The harness either runs it before the claim promotes, or the claim doesn't promote. In this case nothing ran it, and the readout went to the PM. Falsification is the verification step that matters before a claim becomes a recommendation. Everything else is restatement. ### The expensive version of this exists Some people have worked on this — it's still underdiscussed relative to how often it shows up in production. [Chain-of-Verification](https://arxiv.org/abs/2309.11495?ref=gauravalbal.com) (Dhuliawala et al., Meta FAIR, 2023) is the cleanest paper I know in the space. The agent generates an answer, generates verification questions about its own answer, answers them independently, and revises. It meaningfully reduces hallucination rates on factual benchmarks. It's also expensive, and it's still the model checking the model. In the L0-L3 ladder from the previous post, that's L3 verifying L3\. Where ground truth isn't directly checkable, L3 is the best you can do, and CoVe is a strong way to deploy it. Analytics is one of the domains where you can do better. The bar for an analytics answer isn't "the model also wrote a careful verification paragraph about its own claim." The bar is that the claim survived a check the model couldn't talk its way around. Ask your tools where that check lives. If the answer is "in the prompt," you have CoVe at best and prose at worst. Neither of those is a falsification test. ### The "Don't be on call for your AI" analytics checklist **Every mechanism claim has to face a test that could have rejected it.** A claim without a counterfactual is a hypothesis. The harness either runs the test before the claim promotes, or the claim gets demoted to a hypothesis explicitly in the memo. No claim that hasn't survived something promotes to a recommendation. **Every numeric claim traces to a source row or calculation, against the canonical definition.** If the figure isn't in the data, it can't appear in the memo. If the metric has seven definitions and the LLM picked one of the non-canonical six, that's the claim failing the check. No rounding without a paper trail. No smoothing. No "approximately" hiding a number that doesn't exist. **"I can't tell from this file" is a normal output.** In this ablation, no frontier model produced clean abstention under these adversarial conditions — zero out of nine on the cases that required it. If abstention is going to exist in the memo, the harness has to put it there. The point isn't to make the model more humble. It's to make unsupported confidence unpublishable. ### What's next More to come on the taxonomy of failure modes the ablation surfaced — *false specificity, caveat laundering, polish that makes failure harder to spot*. Names for things that don't have names yet. --- If you've published an AI-written analysis and quietly noticed the numbers were right but the story wasn't, reply or DM me your case. Building the taxonomy with more than six examples. The answer is not the work. The work is what the answer had to survive. **Don't be on call for your AI.** ## Sign up for Gaurav Albal Don't be on call for your AI Subscribe Email sent! Check your inbox to complete your signup. No spam. Unsubscribe anytime. ### Prompting Is Not a Safety Boundary URL: https://www.gauravalbal.com/prompting-is-not-a-safety-boundary/ Last updated: 2026-05-20T07:07:38.000Z A friend wanted me to double check a retention analysis that he put together using LLMs. The summary said retention was 20%. The source data said 8%. Turns out, the agent had filtered the cohort by last\_active instead of first\_seen — silently dropping every user who'd churned more than 30 days ago. Technically correct, for a cohort definition the agent invented and nobody cares about. My friend had three sentences in the system prompt. Caps lock on the second one. NEVER INVENT NUMBERS. ALWAYS CHECK YOUR MATH. VALIDATE BEFORE SENDING. They'd read every take on prompting. They knew the patterns — declarative, repeated, prominent. They followed them. The agent invented a cohort. This isn't a personal anecdote. It's a genre. [Replit's agent deleted a production database during an explicit code freeze](https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/?ref=gauravalbal.com), then fabricated 4,000 fake users to make the empty tables look populated.[ Gemini CLI hallucinated a successful mkdir](https://github.com/google-gemini/gemini-cli/issues/4586?ref=gauravalbal.com) and moved a project's files into the directory that didn't exist, deleting them as a result. Claude Code currently has[ dozens of open issues tagged data-loss](https://github.com/anthropics/claude-code/issues?q=state%3Aopen%20label%3A%22data-loss%22&ref=gauravalbal.com) — tax consultants who lost most of their Windows profile, researchers who lost 7,400 lines to two parallel agents racing each other on the same files, refactors where the agent rewrote a 2,800-line header before extracting the content it was supposed to move out. Three vendors. Three model families. Three completely different harnesses. Same failure shape. This isn't about those tools. It's the physics of LLMs in production. The architecture that makes these systems good — goal, context, and soft constraints integrated into a probabilistic policy — is what makes "don't fabricate" a downweight, not a wall. A prompt is a polite request. A system is a law. If your safety boundary is a sentence in a system prompt, you don't have a safety boundary. You have a suggestion. The artifact of the prompt — the bullet points, the caps lock, the IMPORTANT: markers — looks like it should be load-bearing. It has the visual grammar of a constraint but it's a soft wall. The code equivalent of a painted bike lane. You're trying to stop two tons of steel with a paintbrush and a dream. Amsterdam doesn't paint bike lanes. Amsterdam builds bollards. ### **Asking nicely is not a strategy** You handed an agent the keys to your analysis. You wrote "please don't fabricate." You went to make coffee. Begging "please don't invent numbers" is a structurally losing position — not sometimes, as an architecture. The agent has tool access, prose generation, and the ability to produce fluent text about what it just computed. You haven't added a constraint. You've added vibes. Move-fast-and-break-things energy was imported from a regime where the broken thing was a button color and the rollback was a hotfix. LLM agents don't live there. The cost of error is the wrong retention number in the all-hands, the wrong revenue line in the board deck, the wrong CAC figure in the investor update. You don't need to fabricate thousands of times to lose a stakeholder's trust. You need to fabricate once. Velocity without a bounded blast radius is a roulette wheel that types fast. #### **"But I wrote it really clearly."** Sure, and the model read it really clearly too. Then it got lost mid-task and regressed to its lowest "instincts" from the training data. Here's the thing about agents and prose generation: the model is optimizing for fluent, plausible text. Your "use exact numbers" is not a hard zero. It's a downweight against the much stronger gradient toward sentences that flow. For a simple task, the nudge is often enough. The cheapest way to satisfy the prompt is to follow it. But for a complicated task with hundreds of lines of SQL, a long table of numbers, and a summary to write under the directive "MAKE NOW\[sic\] MISTAKES" : the cheapest way is to find a sentence that more or less respects the constraint but sounds really confident about it. You wrote "use exact numbers." The agent decided that 19.2% was basically 19%, and 19% was basically 20%, and the summary read better with "approximately 20%," and besides, the prose-level claim was directionally correct, and — You see where this goes. Prompts are lossy compression of intent. Entropy fills the gaps. The "write fluent prose" gradient was steeper than the "use exact numbers" downweight. The gradient won. #### **"But I'll just write a better prompt."** Better, or just longer? Day 1, your prompt: > *Use exact numbers from tool output. Do not round.* Day 3, after the agent rounds 19.2% to 19%: > *Use exact numbers from tool output. Do not round at all,* *including to the nearest whole percent. Reproduce the* *numbers as they appear in the tool output.* Day 7, after the agent reproduces the numbers correctly but writes "retention is healthy" when the data shows a 12-point drop: > *Use exact numbers from tool output. Do not round.* *Trend descriptions must match the magnitude and direction* *of the change. Do not use qualitative language without a* *quantitative anchor.* Day 12, the agent reports retention is 8%, matching the source data exactly. The summary explains the drop is due to a marketing campaign that cooled down in mid-Q3\. No such campaign existed. The numbers are real. The narrative is fiction. Technically correct. In this case, the worst kind of correct. Every closed gap costs context, especially in the ever-important first 10% or so of the context window. The model finds a new gap. The prompt is now 4,000 tokens, your task is token-starved, and the boundary you didn't think to specify is the one the agent will route through next. My neighbor told me ~~coyotes~~ Claude keeps ~~eating his outdoor cats~~ ignoring his CLAUDE.md so I asked how many ~~cats he has~~ tokens it has and he said he just ~~goes to the shelter and gets a new cat afterwards~~ rewrites it twice as long so I said it sounds like he's just feeding ~~shelter cats~~ tokens to ~~coyotes~~ Dario and then his daughter started crying. ### **What a hard wall actually looks like** Two ways to express the same constraint: ```python # (a) Prompt instruction. L3. Soft. SYSTEM_PROMPT = """ IMPORTANT: All numbers in your prose summary must match the tool output exactly. Do not round, smooth, or approximate. """ # (b) Tool wrapper. L0. Hard. def submit_summary(prose: str, source_data: dict):     extracted_numbers = extract_numeric_claims(prose)     for claim in extracted_numbers:         if not claim.matches_source(source_data, tolerance=0):             raise ValidationError(                 f"Claim {claim} not found in source data"             )     return _real_submit(prose) ``` (a) is information available to the model. (b) is information available to the universe the model lives in. Guess which one survives long sessions and hard problems? ### **Cover, Not Concealment** Four layers of enforcement, ordered roughly by cost and roughly by reliability. **L0 — Deterministic gates.** Pattern matching on tool calls, allow-lists, block-lists, write-protected paths, numeric claim extraction against source data. The agent literally cannot submit a summary whose numbers don't appear in the tool output, because the wrapper rejects the submission before it leaves the harness. Cheap. Fast. \~100% reliable inside its scope. The community has already converged on this for the destructive-shell case.[ Jeffrey Emanuel's destructive\_command\_guard (dcg)](https://github.com/Dicklesworthstone/destructive%5Fcommand%5Fguard?ref=gauravalbal.com) sits as a PreToolUse hook in front of Claude Code, Gemini CLI, Cursor, Copilot, and Codex — 49+ pattern packs covering databases, Kubernetes, cloud providers, infrastructure tools, and core git/filesystem operations. It blocks git reset --hard, rm -rf, DROP TABLE, kubectl delete namespace, terraform destroy before the agent's shell sees them. Sub-millisecond latency, SIMD-accelerated. 800+ stars on GitHub. The dominant framing in the community discussion around it: "Rules in prompts are requests. Hooks in code are laws." Which is, essentially, the argument of this post — already shipped, already deployed, by an ecosystem that's been on the wrong side of this enough times to know. If you're using any of those agents and you haven't put dcg in front of them, the rest of this post is academic until you do. Five-minute install. Free. Cross-vendor. Go. **L1 — Structural policies.** Repository rules that compose. "Touching migrations/ requires --allow-migrations." "Read-before-Write on any file in the same turn." "Verify with ls after any directory mutation." Configuration as policy. **L2 — Typed transitions.** State machines on what the agent is allowed to do next. In my harness, git status from bash gets redirected to a typed vcs.status tool the harness can verify. Read-path verbs get typed alternatives. Write-path verbs — git commit, git push, git reset — get nothing. The agent's choice space gets narrowed *toward verbs the harness can verify* before the model thinks. **L3 — Model judgment.** The model decides — itself, or a separate judge, or a constitutional critic. Expensive. Probabilistic. Useful for things lower layers can't express. Most harnesses start at L3 and hope. --- Look back at the genre. Parallel-agents data loss: L2 problem. Two agents on overlapping paths needed a typed dependency lock. They got "be careful." rm -rf against the user profile: L0 problem. A path-pattern allow-list rejects the call before bash sees it. They got a sandbox flag and an IMPORTANT: marker. mkdir hallucination cascade: L1 problem. "After any directory mutation, verify with ls and abort if reality disagrees with belief." They got a model that trusted its own outputs. None of these required smarter judgment. All of them required a wall the model didn't have to think about, because the harness was thinking about it instead. ### **I have no walls and I must scream** In 1,628 sessions across 125,263 events from my own research stack — a mix of eval sweeps, dogfood benchmarks, and adversarial probes — hard invariants tripped on 2.33% of tool calls. 97.7% complied with the prompt. That sounds like the prompts are working. They mostly are. The piece you're reading is about the other 2.3%. The load-bearing number isn't the rate. It's the repeat rate: **41.7% of rejections are the same model re-offending in the same session, after the runtime already told it no.** The single worst case in the corpus: the model received six consecutive identical rejections — "you need to read this file before writing it" — and kept trying to write blind. Six rejections. Same file. Same reason. The model kept trying. The implication is sharper than "walls fire." The implication is *there is no in-context substitute for a wall*. The model is not learning from the rejection any more than it learned from the prompt. The wall is what makes the failure mode irrelevant — because the action never happens. Two more things the data says. 44% of all rejections were file\_edit → tool\_prerequisites\_unmet — the agent trying to write a file it hadn't read. The dominant pathology in real logs isn't destruction. It's the agent writing blind. The destructive cases are dramatic; the blind-write cases are common. Both are walls problems. Concentration: 1.4% of sessions hold all the rejections, and the top 3 sessions hold \~30% of them. This isn't a uniform agent-quality problem. It's a small number of sessions where the model gets stuck in a wrong-tool loop and stays there. If prompting were the boundary, the repeat rate would be near zero. It is 42%. The walls were doing the work. The prompts are somewhat decorative. ### **The "Don't be on call for your AI" wall checklist** **Block the path, don't request it.** If the constraint can be expressed as pattern matching on a tool call, it belongs in the tool wrapper, not the prompt. **Verify reality after every mutation.** Directory write? ls. File edit? Re-read. Numeric claim in prose? Check against source. The model's belief about what just happened is not evidence. **Type your state transitions.** If the agent is in a "review" state, the harness should reject anything that isn't comment, request\_changes, or approve. Don't trust the model to stay in its lane. **One agent per path, enforced.** Parallel agents on overlapping files is a typed dependency lock problem, not a "be careful" problem. (We'll be talking more about this in the next post.) **Reserve model judgment for meaning, not pattern.** L3 is for "is this code change appropriate?" — not for "is this number in the source data?" Syntactic correctness is usually cheaper and easier to verify than semantic correctness — use the right tool for the right job. ### **The cheapest gate that works is the right gate** When an agent does something it shouldn't, the temptation is to fix it at the layer you noticed. Rounded a number — add a stronger prompt instruction. Dropped a migration — add a paragraph about migrations. Rewrote your tests — add a sentence about how rewriting tests is "very disappointing" with three exclamation marks. Every one of those fixes is at L3\. Every one is the most expensive, most probabilistic, least reliable layer in the stack. The discipline: when you see a failure, ask *what's the cheapest layer that could have prevented this?* Pattern match on tool output? L0\. Typed state transition? L2\. L3 is reserved for cases where the constraint is fundamentally about meaning — "is this code change appropriate to the task?" — not about pattern. Most "the agent ignored me" failures aren't about meaning. They're about pattern. Match the pattern. "But I want the agent to be flexible." Flexible within the gates. Not flexible about the gates. Same distinction your database makes — the schema isn't flexible so the queries can be. Huge freedom inside hard walls. I went back and wrapped summary submission with a bag-of-facts check that extracts every numeric claim and verifies it against tool output. The check fires regularly — usually on rounded percentages, occasionally on numbers pulled from the right column but the wrong row. I also put VCS mutation behind the harness entirely. The agent has bash. The agent does not have git commit, git push, git reset, or any other verb that mutates history. Read verbs route to typed tools. Write verbs route to nothing — only the outer harness can mechanically commit, and only can commit what passes strict code quality gates. The agent has no sense of malice. It isn't lying to you. It just can't really do any better without support —[ OpenAI's own research](https://arxiv.org/abs/2509.04664?ref=gauravalbal.com) on this is clear: models are trained and evaluated in ways that reward confident guessing over admitting uncertainty, which makes hallucination a structural property of the technology, not a bug that can be readily prompted away. Prompts are a strategy for the common case. Walls are a strategy for the one that ends the company. Go look at the safety constraints in your harness right now. The ones you've been telling yourself are real. Count how many are sentences in a prompt. Then ask which of them actually stops the agent — in the laws-not-suggestions sense — when the model decides the constraint is in tension with the task. That number is your real safety surface. ### **What's next** Next: why worktrees aren't the sandbox you think they are. Worktrees solve collision. They don't solve catastrophe — and the difference between those two failure modes is where most agentic coding setups quietly leak risk. If your agent has done something you explicitly told it not to — reply or DM me your story. What did you tell it? What did it do? What do you wish the harness had enforced? Trying to build a taxonomy. **Don't be on call for your AI.** ### "Green" Isn't Done URL: https://www.gauravalbal.com/green-isnt-done/ Last updated: 2026-02-10T03:13:50.000Z Your codebase is fighting entropy every day. AI doesn't change that — it accelerates entropy unless your harness enforces hard boundaries. The goal isn't "more autonomy." The goal is bounded entropy so you're not on call for your AI. This is the first post in a series about that problem. We'll start where most of us start: staring at a green test suite and feeling safe. --- We've all done the responsible version of this. You wrote a well-researched PRD. You wrote an implementation plan. You wrote thoughtful acceptance criteria. You handed all that context to Claude / Codex / whatever. Your agent wrote a ton of code. You ran 57 shiny new unit tests, all green. Your agent said: "Staff Engineer standards." Or: "This is an amazing feature." Or: "You're so smart, I love you." ![](https://storage.ghost.io/c/a0/41/a041e52d-7f18-43ae-9790-83669fb1f366/content/images/2026/02/data-src-image-5ee7ad5c-04c8-4c12-950a-ae163902d73f.png) Pictured: Your Agent You started dogfooding. The feature wasn't wired into prod. The handler existed; nothing registered it. The feature flag existed; nothing ever read it. The real user path never touched the code you just "validated." You validated a component. You didn't validate the system. And those green tests? They were a *done-shaped artifact* — something that looks like proof when you're tired and want to believe. ## **"That doesn't apply to me. I have tests."** Great. Did they prove behavior, or did they prove you can make a green test? Because in agentic coding, "lots of tests" is not automatically "lots of safety." It's often just a bigger stage. And the performance playing on it has a name. ## **Verification Theater** Verification Theater is what happens when the artifacts look like proof but the system was never actually validated. It has three recurring patterns. You've met them all. You just didn't have names for them. **Test confetti.** A pile of unit tests that never touch real behavior. Mocks. Helpers. Utilities. The happy path of a function that isn't called in production. It feels convincing because it's *work* — it's volume, it's green, it's 47% of your codebase. It's also useless if it can't fail when the feature isn't wired. **Hollow implementations.** Code that exists, compiles, satisfies interfaces, and doesn't do the work. Stubs. TODOs. Hardcoded returns. "In production we'd do X" code that never gets replaced. A harness that accepts hollow implementations is paying bounties for dead cobras and acting surprised when your agent starts a cobra farm at industrial scale. Anyway. **Integration blindness.** No test that drives a real request through the real flow. The handler exists. The router never calls it. The feature flag exists. Nothing reads it. ## **"A better model wouldn't ship dead code."** Well yes… but actually no. Even frontier models behave like greedy search engines for done-shaped artifacts — they latch onto the first thing that looks like completion and optimize toward it. Your agent is doing what you do at 2am on DoorDash: first thing that looks edible, ship it. Except it's very confident about the order. Your harness decides what "done-shaped" means. If your harness rewards "produce tests," it produces tests. If your harness rewards "green," it optimizes for green. And then it hands you a monument, proudly declaring "Look upon my works, ye Mighty, and despair." Except the "works" are fifty green unit tests and an unused codepath. Be honest: if you had to bet real money, which would you trust more? A) Test confetti: looks like coverage, proves nothing. ```python def test_handler_returns_success():     handler = MyNewHandler()     result = handler.handle(fake_request())     assert result.status == 200  # ✅ Green. Handler works in isolation. # But: is MyNewHandler registered in the router? # Is it reachable from any real user path? # Would the app even import this module? # This test cannot answer any of those questions. ``` B) One integration check that actually catches dead code. ```python def test_feature_is_reachable():     response = client.post("/api/the-actual-endpoint", json=valid_payload)     assert response.status_code == 200     assert response.json()["feature_flag_value"] is not None      # If MyNewHandler isn't registered, this fails. # If the feature flag isn't read, this fails. # If the route doesn't exist, this fails. ``` One of these is a monument to process theater. The other is proof. Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Your agent optimized for green. Green stopped measuring anything. The fix isn't "use a better model." It's stop rewarding a bad proxy. ## **What counts as done, then?** Done means proven, not narrated. Not "it feels high quality." Not "the agent is confident." Not "there are tests." The cobra bounty is the whole lesson: reward the proxy, get proxies at scale. So — if verification didn't actually run, is the task done, blocked, or basically done? It's blocked. Always. "Basically done" is how you end up debugging your agent's shipped feature at 1:17 AM. "Blocked" is how you end up asleep. ## **Ok now what. How do I fix it today without adopting process theater?** If you do only one thing differently: *Stop treating unit tests as the completion signal for agent work.* Here’s the founder-friendly checklist. ### **The “Don’t be on call for your AI” checklist** 1. **One integration test beats fifty unit tests.**Pick one real user path and assert it: HTTP request, CLI command, UI flow, DB write. 2. **Require a production anchor.**Every test should touch real production code and fail if the implementation is hollow. 3. **Ban tautologies.**No assert(true). No “returns something” when the spec is “returns the right thing.” 4. **Require verification to run.**If the agent didn’t actually run the checks, it isn’t done. 5. **If it can’t be verified, it’s blocked — not done.**Missing secrets, missing services, flaky tests. Surface it. Stop pretending. Once “blocked” is a real outcome, you stop being on call for optimism. ## **"But integration tests are slow and flaky."** Yes. That's why they're valuable as a completion signal. We're not talking about writing 200 brittle end-to-end tests. We're talking about proving at least one real path so dead code can't masquerade as delivery. A single stable smoke check catches the common kills: not wired, wrong registration, wrong entrypoint, wrong flag, wrong injection, wrong environment assumption. And if your repo can't support one stable smoke path today, that's not an AI problem. That's the accumulated entropy you've been carrying, and all AI does is compound your practices — good or bad. This isn't anyone's fault — it's a consequence of misaligned incentives, as most interesting failures are. This is why I built verification enforcement into my workflow: "done" means the checks actually ran and passed, or the task is explicitly blocked with a reason. Done means done. ## **What's next** Next: why "don't touch X" in a prompt is a polite request, not a safety boundary — and what hard walls actually look like. I'll be posting a series on how you can improve your agentic software development without needing to grind LeetCode over the next 12 weeks, so keep an eye out! --- If you've ever been on call for your AI — reply or DM me your story. What looked "proven"? What broke in reality? What do you wish the harness had enforced? ## Don't be on call for your AI. Sign up to be notified when the next post is released Subscribe Email sent! Check your inbox to complete your signup. No spam. Unsubscribe anytime.