Engineering AI for Production · Part 2 of 5
By Gregorio Ferreira, Hybrid Mind Consulting
Technology note: this article reflects LangChain 1.3 and the LangSmith capabilities available or announced as of July 2026. Feature maturity is labelled inline where relevant. Code samples were written and import-checked against langchain==1.3.14 on Python 3.11.
Engineering AI for Production
In Part 1 of Engineering AI for Production, we looked at the retrieval layer and how measuring an embedding space can expose changes beneath a RAG system.
In Part 2, we move up the stack.
Once an agent works, how do you keep changing its prompts, models, tools and behaviour without quietly making it worse?
Most agent demos die at the first requirements change.
Not because the model is weak or the framework is wrong, but because nothing around the agent was built to absorb change. The prompt lives in a Python string. The eval is a colleague eyeballing five conversations. The “safety logic” is interleaved with the graph code.
Then week six arrives: a new document type, a model deprecation notice, a stakeholder who wants the tone changed, and every modification becomes a leap of faith.
The fix is not a smarter prompt. It is an engineering loop: the same discipline that makes an ordinary service safe to change, including version control, environments, tests and monitoring, applied to the three things that actually change in an agent system: behaviour code, prompts and quality criteria. This post walks through the loop we use in production, where LangSmith is not simply a tracing sink but part of the engineering workflow around how agent changes are measured, promoted and monitored.
Cross-cutting behaviour belongs in middleware, not in your graph
The first maintainability decision happens before any prompt is written: where does the behavior that is not the task live? Summarizing long histories, redacting PII, pausing for human approval, recovering from tool failures — every production agent needs these, and none of them is the agent's job.
LangChain 1.0 introduced create_agent with a middleware system, and the 1.3 line (May 2026) expanded the built-incatalog considerably. The point of middleware is architectural: crosscutting concerns become declarative, ordered, and individually testable, instead of being threaded through node functions where every change risks breaking the core logic.
Two properties matter here. First, the diff is small and legible. Adding an approval gate to a new tool is one line in a list. It is reviewable by another engineer without requiring them to untangle a rewrite of the control flow. Second, middleware can be changed by environment. the same agent definition can run with a stricter PII strategy in production than in development, which becomes important once promotion (below) is in place.
The rule we follow: if a behavior would survive a complete change of the agent's task, it does not belong in the task's code. It is middleware.
Prompts are deployment artefacts, not strings
Prompts change more often thancode, are edited by more kinds of people than code, and — in most teams we meet— are versioned worse than code. That inversion is the root of a lot of"the agent got worse and nobody knows why."
We treat every prompt as a versioned, deployable artifactin LangSmith's prompt registry. Each edit is a commit with an identifiablehash; the application never contains prompt text, only a reference:
The piece that completes this is prompt environments (introduced to the registry in early 2026), which replace ad-hoc tag conventions with realpromotion: a commit is promoted dev → staging → prod exactly the way acontainer image moves through environments, and per-prompt ownership controlswho can promote. The practical consequences:
- Rollback is instant and code-free. A bad prompt in production is re-pointed to the previous commit - no deploy, not hotfix branch.
- The audit question answers itself. "What exactly was the agent told on Tuesday?" is a lookup, not an archaeology project.
- Non-engineers can propose changes safely. A domain expert edits in the playground and commits to dev; nothing reaches users intil the change passes the gat tht comes next.
Evaluation is a CI concern, not a research activity
Here is the loop's core discipline: no prompt or model change is promoted without being measured against a pinned baseline. The same way you would not merge a PR that has never run the test suite, we do not promote a commit that has never run the eval suite.
Three LangSmith capabilities make this workable at normal-engineering cost rather than research-project cost:
Datasets grow from production, not from imagination. The best test cases are the conversations that actually happened — especially the ones that went wrong. Traces are curated into datasets in a few clicks or a few lines, soevery production incident becomes a permanent regression test. Over time the dataset stops resembling your launch assumptions and starts resembling your users.
Evaluator templates cover the standard dimensions. Since April 2026 LangSmith ships 30+ reusable evaluator templates (also available in the open-source open evals library) — correctness, safety dimensions like prompt-injection susceptibility, and, critically for agents, trajectory evaluators that judge the sequence of decisions rather than only the final answer. An agent that reached the right answer through three redundant tool calls and one hallucinated argument is not "passing"; final-answer checks alone will never tell you that.
Baselines are pinned, comparisons are automatic. A designated baseline experiment stays pinned as the fixed reference (a February 2026 addition), and every subsequent run is compared against it, per-metric, with deltas surfaced. This is the difference between "the new prompt feels better" and "the new prompt is +6 points on resolution correctness,−2 on tone, neutral on trajectory efficiency."
The CI wiring itself should be deliberately unremarkable:
A change that passes gets promoted to staging; a changethat regresses dies in a pull request instead of in front of users. The wholecycle — edit, evaluate, compare, promote — takes minutes, which is the propertythat makes people actually follow it.
Production closes the loop
Offline evals answer "did this change break what we know about?" Production answers the question you cannot pre-compute: "what are users doing that we never anticipated?"
Every production run is traced,and a sampled fraction flows through online evaluators — the same evaluator definitions used offline, now scoring live traffic for drift: correctness proxies, PII leakage, tool-failure rates. When an online score dips or a user files negative feedback, the offending trace is one click away from becoming a dataset example, and the loop starts again. LangSmith's Insights-style behavioral analysis helps at the aggregate level —surfacing clusters of failures you did not think to look for — but the mechanism that matters is the humble one: bad production traces become tomorrow's regression tests, automatically enough that it actually happens.
The economics of this loop are worth stating. The expensive part of agent quality is not running evaluations; it is discovering what to evaluate. Production traffic does that discovery for free, continuously. A team that captures it compounds; a team that doesn't re-learns the same failures quarterly.
One change, end to end
To make the loop concrete, here is what a routine Tuesday-afternoon change looks like when all the pieces are in place. The request: the support agent should stop offering refunds proactively and only process them when the customer explicitly asks — a classic behavioral change that touches no code.
14:05 — the change is a commit. The prompt edit happens in the playground against the current support-triage prompt; the diff is three sentences. Committing it creates a new immutable version in the registry, automatically tagged into dev. Production traffic still runs the prod commit; nothing has changed for users.
14:15 — the eval gate runs. CI pulls the dev commit and evaluates it against two datasets: the general goldens, and — because we have been burned before — a targeted dataset of 40 refund conversations, half of them harvested from production traces the last time refund behavior regressed. The comparison against the pinned baseline comes back: proactive-refund ratedown from 31% to 2% (the intended change), resolution correctness flat, tone flat, but trajectory efficiency down 4points — the new wording makes the agent double-check the refund policy tool on conversations that never mention refunds.
14:40 — one iteration. The unnecessary tool call is a wording problem; one more commit scopes the instruction. Evals re-run: efficiency recovered, all other metrics stable against baseline. The candidate is promoted to staging, where the staging environment's online evaluators watch it against a mirror of live traffic for the rest of the afternoon.
17:30 — promote and forget. Staging metrics are clean. The commit is promoted to prod — a pointer change, no deploy. The rollback plan is the previous commit, one click away. Wednesday morning, the online dashboard confirms the proactive-refund rate dropped in production; one odd trace from the evening — a customer asking about "returns" rather than"refunds" — goes into the refund dataset for next time.
Total elapsed: an afternoon, most of it waiting on staging soak. Nothing heroic happened, which is exactly the point: the loop's value is measured in how boring changes become.
The same discipline is extending beyond prompts
A prompt is only one of the artifacts that define an agent's behavior. Instructions files, policies, toneguides, worked examples — most teams scatter these across repositories, wikisand prompt text, with no versioning story at all.
The direction of travel in the ecosystem is to treat all of it the way we treat prompts. LangSmith's Context Hub (announcedMay 2026 — early-stage, evaluate accordingly) applies the same model to arbitrary behavioral context: git-like versioning, environment tags, reviewworkflows, and non-engineer editing — so a policy change proposed by acompliance officer can flow through the identical promote-with-evidence gate as an engineer's prompt change. We consider the promotion gate the transferable idea: whatever tool holds your agent's behavioral context, nothing that changes behavior should reach production without a versioned commit and an eval run.
Takeaway: a definition of done for any agent change
Our checklist, in the order the loop enforces it:
- The change is a commit — in git (code, middleware) or the prompt registry (prompt, context). No unversioned behavior change exists.
- Cross-cutting behavior went into middleware, not into task logic.
- An offline eval ran against the current golden dataset, including trajectory-level evaluators — not only final-answer checks.
- Results were compared against the pinned baseline; no unexplained regression on any gated metric
- Promotion went dev → staging → prod through the registry, not through a code deploy.
- Online evaluators cover the changed behavior in production, at a sampling rate you can afford.
- Rollback is a re-pointed commit, tested, and takes under a minute.
- Atleast one production failure from the last cycle became a dataset example.
Eight boxes. When all eight are routine, changing your agent feels like changing any well-tested service: small diffs, observable impact, reversible releases — and the second month stops being the month things quietly fall apart.
Next in the series
This is Part 2 of Engineering AI for Production, our five-part series on the engineering layers beneath production AI systems. Part 1 looked at measuring the retrieval foundation. In Part 2, we moved up the stack and looked at the development loop that allows an agent to change without turning every release into a leap of faith.
In Part 3, Ernesto Martinez takes the systems view. If tracing, evaluation, prompt management and production feedback become part of the engineering workflow, where should that platform actually run? The Platform Under the Agents: Choosing the Right Deployment Model for LangSmith compares self-hosted, hybrid and SaaS architectures, including the trade-offs around data residency, operational complexity and migration.
Engineering AI for Production
Part 1: What Is Your Embedding Space Actually Doing?
Part 2: Agents That Survive Their Second Month
Part 3: The Platform Under the Agents
Part 4: Beyond the Chatbot
Part 5: Durable by Design
References
- LangChain create_agentand middleware documentation — docs.langchain.com (LangChain 1.3, May 2026; ToolErrorMiddleware requires ≥1.3.14, July 2026)
- LangSmith prompt registry and prompt environments —docs.langchain.com/langsmith (environments: self-hosted v0.13.27 changelog, March 2026;prompt owners: v0.14.0, April 2026)
- Reusable evaluator templates / openevals 0.2 — LangChain blog, April 16, 2026
- Pinned baseline experiments — LangSmith changelog, February19, 2026
- LangSmith Context Hubannouncement — LangChain blog, May 13, 2026
.png)
