From prototype to production: Engineering AI agents with the Agent Development Lifecycle

AI agents have spent the past year moving from experimentation into engineering practice. Enterprise teams are now shipping large language model (LLM) systems that invoke tools, coordinate APIs, and respond to questions using internal knowledge.

A good demo demonstrates what an agent can do, but before release a team must answer a different question: is this agent ready for production? Whoever owns that decision needs evidence that the agent satisfies the agreed requirements, and reaching that decision consistently calls for a repeatable, affordable way to measure current behavior and to judge whether the latest change actually improved it.

In conventional software development, unit tests, continuous integration and continuous delivery (CI/CD), and passing builds provide those answers. For agents, these mechanisms are necessary but not sufficient, for three reasons — and each reason calls for additional checks. This post describes the three properties behind that gap, walks through a measured case in which an aggregate metric conceals a coverage problem, and presents the Agent Development Lifecycle (Agent DLC): a single lifecycle with two loops and one explicit standard for judging every change.

Terminology note: in this post, gold standard designates one specific artifact — the bundle of verifiable criteria, their thresholds, and the approved examples against which they are checked. Business owners approve this standard, and it forms the basis for acceptance decisions.

Three properties of agents that demand additional engineering practices

Traditional testing and CI/CD still apply to agents, but three characteristics require extra verification. First, agents are non-deterministic: the same input can yield different outputs and different execution paths. Second, edits to prompts, tool descriptions, retrieval settings, or memory configuration can shift behavior in ways that a text review cannot reveal. Third, model endpoints, knowledge bases, and external APIs evolve independently of the repository and can alter agent behavior. Table 1 summarizes the engineering practices each property calls for.

A measured example: high retrieval precision can hide missing information

This example illustrates how the choice of metrics affects diagnosis. To exercise the method end to end, the authors built a reference implementation: an internal policy assistant that answers HR policy questions over a document knowledge base. A narrow, well-scoped domain was chosen to keep the evaluation and diagnosis easy to follow. The results below were measured on this reference implementation; they are neither a customer outcome nor a benchmark result.

The accompanying AWS Samples repository contains the infrastructure deployment, the HR agent, custom evaluators, and scripts for baseline evaluation and re-evaluation after a prompt change (09-run-eval.sh and 10-optimize-prompt.sh). It also offers optional checks for judge repeatability, model comparison, cost, and latency. The scripts reproduce the workflow, and scores can vary between runs and repository versions.

Three golden questions were measured on five criteria:

• Groundedness: decompose the answer into claims and verify each against the retrieved sources.
• Chunk-level source precision: whether each retrieved chunk, taken as a whole, is necessary for the question.
• Fact-level source precision: decompose each chunk into individual facts and check whether each fact is necessary.
• Retrieval coverage: for each sub-question, whether any retrieved chunk answers it.
• Response precision: how many of the claims in the answer are necessary.

Table 2 shows the first evaluation round, measured on the reference implementation — not a customer outcome. Chunk-level precision is 1.00 for all three questions, so a release gate based only on that metric would have accepted all three answers. Fact-level precision tells a different story: 0.59, 0.53, and 0.09. For the third question, roughly nine out of ten facts in the retrieved material are irrelevant to what was asked. Chunk-level precision only asks whether a passage is on topic; it does not measure how much of the passage is relevant.

The team then changed exactly one thing — the system prompt — adding three constraints: ground strictly in retrieved content, ignore irrelevant retrieved content, and keep answers focused. The agent, knowledge base, evaluators, and questions were left untouched, so the before-and-after comparison differs by a single variable.

Table 3 captures the result of the prompt-only change: the first question improved substantially, while the third got worse. The evaluation had already suggested where to investigate before the change. In the third row of Table 2, chunk-level precision of 1.00 combined with fact-level precision of 0.09 and retrieval coverage of 0.00 pointed to a knowledge-coverage problem: the retrieved material was nominally on topic but did not answer the question. Instructing the model to ground strictly in retrieved content could not supply the missing information.

That diagnosis was verified against the source documents used in the experiment. The corpus contained no dedicated sick-leave policy document, although the leave-management policy made brief references to sick leave. The right next step was to fix policy coverage and then re-check retrieval, rather than to keep adjusting the prompt. When the source material lacks the information needed to answer a question, prompt changes alone cannot resolve the failure.

A separate result shows why groundedness must be checked alongside other criteria. The benefits-enrollment answer passed with a groundedness of 0.90, yet it contained the sentence: "I also noticed from your profile that your preferred contact method is Social Media." No such profile exists. The policy document lists contact preference as an option inside a process step, and the public, crowd-sourced, noisy FAQ corpus contains a question-answer pair whose answer is "social media". The model merged a policy option with an unrelated FAQ answer and presented the result as a fact about the user. The high groundedness score did not catch this false personal claim — passing a quality threshold does not establish that responsibility constraints have been met.

These scores can vary between runs, and a borderline Pass can become a Fail, so score differences should be interpreted together with repeated runs and the underlying evidence; evaluator reliability is addressed later in the post. This small reference-implementation example illustrates how evaluation can steer diagnosis, but it does not establish a general improvement rate or validate a production release.

Agent DLC: one lifecycle, two loops

Agent DLC connects six activities around a single explicit gold standard: Define, Build, Evaluate, Release, Observe, and Feed Back.

Evaluation-driven development has been described in earlier work, including Xia et al.'s "Evaluation-Driven Development of LLM Agents: A Process Model and Reference Architecture". Agent DLC applies that approach to delivery through six activities with explicit acceptance conditions, release authority, and evidence requirements.

The acceptance-first approach also appears in Fotor's published AWS customer case study. That team defined scalability, cost, isolation, and architectural flexibility criteria before migrating to AgentCore runtime, a capability of Amazon Bedrock AgentCore, and then validated changes against operating data. The case reports workflow overhead dropping from minutes to seconds, excluding model inference, plus support for thousands of concurrent active sessions. It documents Agent DLC-guided architecture delivery, with quality evaluation identified as the next step.

Within the lifecycle, Define translates business objectives into criteria, thresholds, and approved examples. Build designs the prompts, tools, context, memory, and runtime mechanisms that must satisfy them. Evaluate produces the evidence of whether a candidate version does so. When it does, the version proceeds to a controlled release. Observe watches it in production, and Feed Back carries what production teaches back to Define.

Figure 1 shows how each activity uses or updates the standard, which makes the standard's role in development and release explicit. Running inside that outer loop is a faster inner loop: score, inspect the path, change one thing, re-evaluate. It operates on a single candidate version and answers two questions — where to change, and whether the change helped.

The two loops do different jobs, and the difference matters. The inner loop improves the agent against a fixed standard, while the outer loop allows the standard itself to change. Discovering during development that the agent recommends products without checking inventory is an inner-loop problem. If production users ask the agent to look things up on behalf of a family member and no criterion covers that case, the outer loop is needed: define the authorization boundary and add examples, then use the inner loop to meet the updated standard. Confusing the two leads teams to fix a missing criterion with a prompt change, or to widen the standard mid-cycle, which leaves them unable to tell which change moved the score.

Both loops produce reusable assets: business criteria, approved examples, evaluators, and the record of how past failures were handled. These assets outlive the model and the framework — when the model is replaced, the replacement is evaluated against the same criteria and examples.

Define good: five dimensions, three tiers

Start from a business outcome defined by its owner. For a recommendation agent, that might be: "Recommend suitable products to returning customers while protecting their information." The requirement is then translated into verifiable criteria across five dimensions.

For each criterion, specify how it is verified and who approves it. "It must be accurate" is a goal; "every factual claim must be traceable to a retrieved source" is verifiable. Business experts should approve the examples that make expected behavior concrete.

Each criterion is assigned its own tier and, where applicable, threshold, and a high score elsewhere must never offset a hard-constraint violation. In the recommendation example, recommendation quality is a release gate under Quality, while protecting another customer's information is a hard constraint under Responsibility: one business requirement yields criteria in different dimensions and tiers, each assessed against its own acceptance condition. These criteria also determine what evidence the system must expose — evidence used to improve the agent and decide on release, and which, as a later section explains, shapes the architecture.

The inner loop: evaluate, diagnose, and improve

Score. Match the evaluator to the criterion: code for unambiguous rules, runtime telemetry for cost and latency, load tests and operating data for capacity and availability, and model-based judges — calibrated against human-reviewed examples — for criteria that require judgment. When evaluating information protection, include authorization and cross-user isolation tests alongside checks for prohibited fields. Run each example k times to assess variable behavior.

For model-based judges, repeat evaluations to measure scoring variability and compare their judgments with human-reviewed examples. Small score gains need supporting evidence before they count as improvements.

Inspect the path. Locate each failure at three levels: session, trajectory, and step. If a recommendation exposes a phone number, follow the affected session through response composition and inspect where profile fields entered the draft — the cause can sit upstream of the visible symptom, as the measured example showed.

Change one thing, then re-evaluate. Correct the identified cause and repeat the evaluation against the unchanged standard under the same conditions, verifying both that the disclosure stopped and that recommendation quality still meets its gate.

Assess release readiness separately for each task type, or intent. Release only the intents that meet their criteria, and make sure any fallback for an unreleased intent also meets the agreed release conditions.

A limited rollout requires acceptance conditions, safeguards, and stop criteria agreed before evaluation; it cannot override a hard constraint or lower a threshold after a failed result. Before release, include blocked and refused requests in the evaluation and check that each was handled as expected. Record the criteria, thresholds, evaluator versions, and the release decision so the team can trace why a version was approved. Passing these checks supports release within the tested scope; production monitoring then verifies that the agent continues to meet the standard as usage and dependencies change.

The outer loop: keep the data flowing

Record execution traces from the first test request onward so failures can be diagnosed in development and in production. Sample and evaluate production interactions to surface failures, then review them with the business owner to confirm the expected behavior. Cases already covered by existing criteria go into the regression set; a case that reveals a missing requirement triggers an update to the standard, which the inner loop then works to meet.

Run evaluations regularly to catch behavior changes caused by external dependencies. When the standard changes, compare results under both versions to distinguish an agent regression from broader test coverage, and link each result to its standard, test set, evaluator, and agent version.

Design the architecture to support evaluation

The evaluation and feedback loops depend on evidence that the architecture must provide, so work backward from each criterion to the system behavior and records needed to verify it. To distinguish a tool-selection error from an execution failure, for example, define the tool's purpose, accepted inputs, outputs, and error conditions, and expose the actual inputs, results, and errors in traces so evaluators can locate the failure. Acceptance criteria therefore shape both evaluation and architecture. With those foundations in place, start with one business task and establish a baseline.

Getting started on AWS

Choose one clearly scoped business task and aim to establish an initial evaluation baseline within a week.

• Days 1–2: the business owner and engineers agree on acceptance criteria, tiers, and thresholds; prepare 20–30 golden examples covering common requests and important edge cases, with expected behavior approved by the business owner.
• Days 3–5: implement the evaluators described above and run the golden set repeatedly; publish the baseline, including failures and the evidence needed to investigate them.
• Week 2: change one thing and re-evaluate under the same conditions; record the release decision and its supporting evidence.

After the baseline is reviewed, integrate evaluation into the release process and run it regularly to detect drift, and assign an owner to review production failures and feed confirmed cases back into the standard.

On AWS, agents can be built with models available through Amazon Bedrock. Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, assesses agent behavior with built-in evaluators and supports custom evaluators for business-specific criteria. AgentCore Observability, also an Amazon Bedrock AgentCore capability, provides execution visibility and is complemented by Amazon CloudWatch for runtime monitoring and diagnostics, while AgentCore runtime, Gateway, Identity, and Memory cover execution, tool integration, identity management, and memory management.