Stateful Orchestration for Reliable Course Production

In a previous role, I designed Temporal workflows for order fulfillment—decoupling activities, handling partial failures, implementing idempotent retries. When I started architecting content pipelines, I recognized the same coordination problems.

The tempting first move is a for-loop. Agent A calls Agent B calls Agent C. It works—until an agent timeout causes the next agent to run on stale state. A Content Drafter re-running after a Code Validator timeout picks up a previous draft's code examples and splices them into new explanatory text. The result: a lesson where the code and the explanation describe different things. It looks correct if you read either the code or the text, just not both together.

This post is part of the System Design Notes: Agentic Content Platforms for Technical Education series. The previous post in this series designed a six-agent content pipeline—interpret, draft, validate, review, check, publish. The post before that built the evaluation architecture those agents target. The current post operationalizes that pipeline with the production reliability patterns it needs to actually run, covering:

• Why agents must be stateful

• Event-driven vs loop-based orchestration

• Failure isolation patterns and their cost implications

• Idempotency for safe retries

• Content versioning and rollback

• Observability for catching silent degradation

Why Agents Must Be Stateful

Content drafts evolve through review cycles. A lesson doesn't emerge fully formed from a single LLM call—it goes through generation, evaluation, revision, re-evaluation, and possibly several more rounds. Each cycle produces state that matters:
what the evaluator flagged, what the human reviewer commented, what the previous draft looked like.

Three kinds of state accumulate across a content pipeline:

Draft evolution. Each revision builds on the previous version. If an agent crashes mid-revision, there are two options: restart from scratch (wasting the successful work) or resume from the last known good state. Without explicit state
management, you get the former by default.

Persistent feedback. Human reviewers leave comments like "the recursion example is too complex for this learner level." That feedback applies to a specific draft version. If the pipeline regenerates the lesson, should the feedback persist?
If yes, the regenerating agent needs access to that feedback. If no, you're discarding expert input.

Evaluation history. Tracking scores across draft versions reveals patterns: "Version 3 improved Conceptual Clarity from 0.72 to 0.87 but dropped Technical Correctness from 0.96 to 0.91." Without version-aware eval history, you can't detect
these regressions. Each revision looks acceptable in isolation while the lesson oscillates between passing different dimensions.

Research on multi-agent failure modes consistently finds that most failures stem from specification and coordination issues — agents misaligning on context, dropping state between handoffs,
misinterpreting upstream output — rather than technical implementation bugs. The agents work individually. The coordination between them breaks down. Explicit state management is how you prevent coordination failures from becoming content
quality failures.

Event-Driven vs Loop-Based Orchestration

Sequential orchestration chains agents in a fixed order: Content Drafter → Code Validator → Pedagogy Reviewer → Publishing Gate. Each agent calls the next.

This works when three conditions hold:

1. Clear linear dependencies — each agent needs exactly one predecessor's output

2. Predictable performance — no agent takes dramatically longer than expected

3. No retries needed — every step succeeds on the first attempt

Content pipelines violate all three. Code Validation might timeout on complex examples. The Pedagogy Reviewer might route content to human review (adding hours or days of delay). The Content Drafter might need to revise based on feedback,
jumping back in the sequence.

Event-driven orchestration inverts the control flow. Instead of Agent A calling Agent B, Agent A publishes a DraftCompleted event. Agent B subscribes to DraftCompleted events and processes them independently. A message broker mediates the
connections.

The structural advantage: integration coupling drops. In point-to-point systems, each agent discovers and manages connections to its peers — and as agents are added, the
number of potential connections grows rapidly. With a message broker, each agent maintains a single connection to the broker, regardless of how many other agents exist.

Human-in-the-loop as an event is particularly important for content pipelines. When a review step flags a lesson for human review, the sequential pipeline stalls. The reviewing agent is waiting for a response. If the system times out, it either publishes unreviewed content or drops the lesson entirely.

With events, HumanReviewRequested goes into a queue. Hours later, the reviewer publishes HumanReviewCompleted. The pipeline resumes exactly where it left off, with full context.

A fair concern: event-driven architecture adds complexity—message brokers, event schemas, eventual consistency, and message ordering (events for the same content item may arrive out of order depending on broker configuration; the version field
on each event lets consumers detect and handle this). For a pipeline with two agents and no retries, sequential orchestration is simpler and sufficient. The breakpoint comes when you need any combination of retries, HITL, or observability across more than a few agents. At that point, you're implementing event-driven patterns piecemeal inside a sequential framework. Better to start with the right abstraction.

If you implemented the previous post's pipeline in LangGraph, you're not throwing it away. LangGraph handles the graph structure and HITL patterns well. The event-driven
patterns here layer production reliability on top—retries, idempotency, failure isolation, and observability that a graph framework doesn't provide out of the box. In practice, each event handler can delegate to a LangGraph subgraph for the agent logic while the event-driven layer manages coordination and recovery.

The event types and orchestrator below show a simplified subset of the full pipeline. The previous post's six-agent design includes objective interpretation, pedagogy
review, and safety checking—each would follow the same event pattern (started/passed/failed) shown here for drafting, validation, and content-level evaluation.

from dataclasses import dataclass, field
from enum import Enum
from typing import Any
from datetime import datetime, timezone
import hashlib

class EventType(Enum):
    # Content lifecycle
    CONTENT_CREATED = "content.created"
    DRAFT_COMPLETED = "content.draft.completed"
    DRAFT_REVISION_REQUESTED = "content.draft.revision_requested"

    # Validation
    CODE_VALIDATION_STARTED = "validation.code.started"
    CODE_VALIDATION_PASSED = "validation.code.passed"
    CODE_VALIDATION_FAILED = "validation.code.failed"

    # Content-level evaluation (the assembled-lesson gate from Post 3,
    # distinct from per-agent stage-level checks)
    CONTENT_EVAL_STARTED = "evaluation.content.started"
    CONTENT_EVAL_PASSED = "evaluation.content.passed"
    CONTENT_EVAL_FAILED = "evaluation.content.failed"

    # Human-in-the-loop
    HUMAN_REVIEW_REQUESTED = "hitl.review.requested"
    HUMAN_REVIEW_COMPLETED = "hitl.review.completed"

    # Publishing
    CONTENT_PUBLISHED = "content.published"

@dataclass
class PipelineEvent:
    event_type: EventType
    content_id: str
    version: int
    payload: dict[str, Any]
    timestamp: datetime = field(
        default_factory=lambda: datetime.now(timezone.utc)
    )
    idempotency_key: str = ""

    def __post_init__(self):
        if not self.idempotency_key:
            # Two-level dedup: broker redelivery of the same
            # event (same retry_count) produces the same key
            # and is correctly deduplicated. Application-level
            # retries (orchestrator emits a new event with
            # incremented retry_count) produce a different key,
            # because they represent a distinct processing attempt.
            retry = self.payload.get("retry_count", 0)
            key_data = (
                f"{self.event_type.value}:"
                f"{self.content_id}:v{self.version}:"
                f"r{retry}"
            )
            self.idempotency_key = hashlib.sha256(
                key_data.encode()
            ).hexdigest()[:16]

The idempotency_key on every event is load-bearing. It prevents duplicate processing when the broker redelivers the same message (network ack failure, consumer restart). Note that application-level retries — where the orchestrator emits a
new event with an incremented retry_count — intentionally produce a different key, since they represent a new processing attempt with different error context.

Failure Isolation Patterns

When an agent fails—a timeout, an API error, a malformed response—the system needs a graduated response. Retrying immediately often wastes resources on transient failures that haven't resolved (though some failures, like a 429 rate limit with a
short reset window, may resolve quickly enough for a brief delay to suffice). Giving up after one failure discards content that would have succeeded on the next attempt.

The Failure Escalation Ladder

1. Transient failure → Retry with exponential backoff.

Most failures are transient: API rate limits, network blips, temporary service unavailability. Exponential backoff with jitter prevents thundering herd problems when the upstream service recovers.

2. Retry threshold reached → Attempt auto-fix.

If three retries fail, the error might be addressable without human intervention. A Code Validator timeout might succeed with a simplified prompt or a smaller model. A content-level evaluation failure on one dimension might pass if the content is regenerated with explicit constraints on that dimension.

3. Auto-fix failed → Escalate to dead-letter queue.

Content that can't be automatically recovered needs human attention. The dead-letter queue stores the failed content with enough context for a human operator to diagnose and fix the issue.

Circuit Breaker: Preventing Cascading Failures

When an external service is degraded, retry logic makes things worse. Ten agents each retrying three times against a struggling API creates 30 requests where 10 already failed. The circuit breaker pattern addresses this:

For content pipelines, the circuit breaker prevents a single degraded agent from consuming the retry budget of the entire pipeline. If the Code Validator is down, other content items don't need to discover this independently.

Dead-Letter Queue for Human Escalation

A dead-letter queue (DLQ) stores messages that can't be processed after exhausting retries. For content pipelines, the DLQ is where human operators handle edge cases the automation can't resolve.

What metadata to include in DLQ entries matters. An entry that just says "validation failed" forces the operator to reconstruct context from logs. Useful DLQ metadata:

• Original event — the full event that triggered processing

• Error chain — every error from each retry attempt, not just the last one

• Retry count and timestamps — how many attempts, how long it took

• Agent state at failure — what the agent had processed before failing

• Suggested action — auto-classified as "retry with different model," "content needs manual edit," or "upstream service issue"

Regular DLQ audits reveal systemic issues. If 40% of DLQ items are Code Validator timeouts on lessons with more than five code blocks, that's a signal to increase the validator's timeout for complex content—not a series of independent failures.

Orchestration Cost Modeling

Retry logic has a direct cost multiplier. Each retry consumes tokens — input tokens for the prompt, output tokens for the response — with zero usable output on failed attempts. Output tokens are typically priced higher than input tokens

often several times higher, depending on model and provider), making failed generations particularly expensive.

An illustrative back-of-envelope calculation (your numbers will vary based on pipeline complexity and model choice):

• 5% of pipeline runs hit retry logic

• Average 2.5 retries per failure

• Result: ~12.5% cost overhead on LLM calls from retries alone

Dead-letter items carry human cost on top of the wasted token spend. If a human operator at a rate like $50/hour spends 30 minutes diagnosing and fixing one DLQ item, that's $25 in human cost for a single content unit—potentially exceeding the
total LLM cost for generating and evaluating that content from scratch.

The trade-off is explicit: more retries improve quality (content that would have been dropped gets recovered) at the cost of higher token spend. Fewer retries reduce cost but increase DLQ volume and human workload. The right balance depends on
your content's value relative to retry costs.

The orchestrator below uses an in-memory set for processed_keys for clarity. A production implementation would use persistent storage (Redis, a database table) so that idempotency state survives restarts and works across multiple orchestrator
instances.

from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class PipelineOrchestrator:
    max_retries: int = 3
    # In-memory for illustration; use Redis or a database
    # in production for persistence across restarts
    processed_keys: set = field(default_factory=set)

    def handle_event(self, event: PipelineEvent):
        # Idempotency check
        if event.idempotency_key in self.processed_keys:
            return  # Already processed, skip

        match event.event_type:
            case EventType.DRAFT_COMPLETED:
                self._start_validation(event)

            case EventType.CODE_VALIDATION_PASSED:
                self._start_evaluation(event)

            case EventType.CODE_VALIDATION_FAILED:
                self._handle_validation_failure(event)

            case EventType.CONTENT_EVAL_PASSED:
                self._publish_content(event)

            case EventType.CONTENT_EVAL_FAILED:
                self._request_revision(event)

            case EventType.HUMAN_REVIEW_COMPLETED:
                self._apply_human_feedback(event)

        self.processed_keys.add(event.idempotency_key)

    def _handle_validation_failure(self, event: PipelineEvent):
        retry_count = event.payload.get("retry_count", 0)
        error_history = event.payload.get("error_history", [])

        if retry_count < self.max_retries:
            # Retry with accumulated error context
            self._emit(PipelineEvent(
                event_type=EventType.CODE_VALIDATION_STARTED,
                content_id=event.content_id,
                version=event.version,
                payload={
                    "retry_count": retry_count + 1,
                    "error_history": error_history + [
                        event.payload.get("error")
                    ],
                    # Error context helps the agent avoid
                    # the same failure on retry
                    "previous_errors_summary": self._summarize_errors(
                        error_history
                    ),
                },
            ))
        else:
            # Exhausted retries — escalate to DLQ
            self._send_to_dlq(event, reason="max_retries_exceeded")

    def _summarize_errors(self, errors: list) -> str:
        if not errors:
            return ""
        return f"Previous {len(errors)} attempt(s) failed: " + "; ".join(
            str(e) for e in errors
        )

    # Stubs — implementation depends on your message broker
    def _emit(self, event: PipelineEvent): ...
    def _send_to_dlq(self, event: PipelineEvent, reason: str): ...
    def _start_validation(self, event: PipelineEvent): ...
    def _start_evaluation(self, event: PipelineEvent): ...
    def _publish_content(self, event: PipelineEvent): ...
    def _request_revision(self, event: PipelineEvent): ...
    def _apply_human_feedback(self, event: PipelineEvent): ...

The error context passing is important. Each retry includes a summary of previous failures, giving the agent information to avoid the same mistake. A Code Validator that timed out on a complex nested function might succeed on retry if the error context tells it to increase its execution timeout or simplify the test harness.

A subtle gap in this design: if the orchestrator crashes between processing an event and recording its idempotency key, the event will be reprocessed on restart. For exactly-once side effects, production systems use a transactional outbox — writing the outbound event and the idempotency record in a single database transaction, so both succeed or neither does. For content pipelines where the worst case of reprocessing is a redundant LLM call, at-least-once delivery with idempotent handlers is usually sufficient.

One LLM-specific consideration for retries: even with error context, LLM non-determinism means retrying the same prompt may produce a different failure. Constraining temperature and output format on retries reduces this variance and makes error context more actionable.

Idempotency: Safe Retries Without Duplicates

Retries solve one problem (recovering from transient failures) and create another (duplicate processing). If a "create lesson" event is delivered twice—because the broker redelivered it, because a network partition caused a duplicate, because
the producer retried an ack timeout—the pipeline creates two identical lessons.

Idempotency keys make retries safe. The principle: same input + same state = same output, and the operation executes at most once.

Key Generation Strategies

Three approaches, each suited to different contexts:

1. Workflow ID + Activity ID (Temporal pattern). Temporal uses the combination of Workflow Run ID and Activity ID as a natural idempotency key. It's guaranteed consistent across retry attempts but unique among workflow executions. If your pipeline already runs on a workflow engine, you get idempotency keys for free.

2. Content-addressed hashing. Hash the request payload. Requests with identical payloads generate identical keys. This works well when the same content should always produce the same result—but fails when you want different results from the same input (e.g., generating three alternative drafts from the same prompt).

3. Client-generated UUID. The producer creates a UUID for each logical action and includes it in every request. The server checks whether that UUID has been processed before.

For content pipelines, the Temporal-style approach works best: a combination of content ID, version number, and processing step produces a key that's unique per logical operation but stable across retries of the same operation.

Server-Side Implementation

The server needs to store four things per idempotency key:

On receiving a request:

1. Check if the idempotency key exists in the store

2. If yes: verify the payload matches, return the cached response

3. If no: process the request, store the key + payload + response

A concurrency edge case: two identical requests arriving at the exact same millisecond. Both check the store, both find no existing key, both process. Distributed systems literature addresses this with locking or compare-and-set operations. For content pipelines, the practical risk is low—content creation events are rarely sub-millisecond duplicates—but the safeguard is worth implementing if your broker doesn't guarantee exactly-once delivery.

Content Versioning and Rollback

Content evolves. A lesson that passes evaluation today might regress after a revision next week—the Conceptual Clarity fix introduced a Technical Correctness issue. Without versioning, you can't compare versions, can't identify what changed, and
can't roll back.

Event Sourcing for Content

Event sourcing stores operations as an append-only sequence of events rather than overwriting current state. Instead of a lessons table with the latest content,
you have an event log where each entry describes what changed.

This gives you four capabilities:

Full audit trail. Every change is recorded: who made it (human or agent), when, and why. You can answer "why does this lesson look like this?" by reading the event history.

Time travel. Regenerate any past state by replaying events up to a point. When a learner reports an issue with a lesson, you can reconstruct the exact version they saw—even if the lesson has been revised three times since.

Rollback via compensating events. Rolling back doesn't mean deleting history. A DraftRolledBack event explicitly records the undo, preserving the full timeline. You know the rollback happened, why, and what state it restored.

Regression detection. Compare evaluation scores across versions. If version 5 scores lower than version 4 on Technical Correctness, the revision introduced a problem. Without version-aware eval history, each version is evaluated in isolation
and this pattern is invisible.

Minimum Viable Event Store

A full event sourcing implementation is a significant investment. For content pipelines, a pragmatic subset covers most needs:

from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Any

class ContentEventType(Enum):
    CREATED = "content.created"
    DRAFT_UPDATED = "content.draft_updated"
    REVIEW_REQUESTED = "content.review_requested"
    FEEDBACK_RECEIVED = "content.feedback_received"
    EVAL_SCORED = "content.eval_scored"
    PUBLISHED = "content.published"
    ROLLED_BACK = "content.rolled_back"

@dataclass
class ContentEvent:
    content_id: str
    version: int
    event_type: ContentEventType
    actor: str  # "agent:code_validator" or "human:reviewer_jane"
    payload: dict[str, Any]
    timestamp: datetime = field(
        default_factory=lambda: datetime.now(timezone.utc)
    )

@dataclass
class ContentVersionStore:
    events: list[ContentEvent] = field(default_factory=list)

    def append(self, event: ContentEvent):
        self.events.append(event)

    def get_version(self, content_id: str, version: int) -> dict:
        """Rebuild state at a specific version by replaying events."""
        state = {}
        for event in self.events:
            if (
                event.content_id == content_id
                and event.version <= version
            ):
                state = self._apply_event(state, event)
        return state

    def get_eval_history(
        self, content_id: str
    ) -> list[dict]:
        """Return eval scores across all versions for regression detection."""
        return [
            {
                "version": e.version,
                "scores": e.payload.get("scores", {}),
                "timestamp": e.timestamp,
            }
            for e in self.events
            if (
                e.content_id == content_id
                and e.event_type == ContentEventType.EVAL_SCORED
            )
        ]

    def _apply_event(
        self, state: dict, event: ContentEvent
    ) -> dict:
        match event.event_type:
            case ContentEventType.CREATED:
                return {**event.payload}
            case ContentEventType.DRAFT_UPDATED:
                return {**state, **event.payload}
            case ContentEventType.ROLLED_BACK:
                target = event.payload["rollback_to_version"]
                return self.get_version(
                    event.content_id, target
                )
            case _:
                return {
                    **state,
                    "last_event": event.event_type.value,
                }

A caveat on get_version: replaying events is O(n) per call, and the ROLLED_BACK handler calls get_version recursively—O(n²) in event stores with many rollbacks. Production systems use materialized views or periodic snapshots to avoid full
replay on every read.

When to Use Full Event Sourcing vs Simpler Versioning

Full event sourcing (append-only log, state reconstruction via replay) is warranted when you need audit trails, support branching or merging content versions, or require complex rollback logic.

A simpler versioning table—storing complete snapshots at each version number—works when you just need "show me version N" and don't need to reconstruct the transformation history between versions.

For content pipelines where evaluation history matters (regression detection, rollback with context), event sourcing pulls its weight. For simpler pipelines where content is generated once and rarely revised, a versioning table is sufficient and
easier to operate.

Observability: Seeing What Actually Happens

Here's what happens without observability:

• Day 1: Pipeline works, all evals pass, team celebrates.

• Month 1: Retry rates are climbing from 2% to 5%. Nobody notices.

• Month 3: Dead-letter queue growing. 8 items stuck in failed state. Nobody checks it because there's no alert.

• Month 6: 15% of content stuck in failed or retry-loop state. Learners seeing stale material because updates aren't completing. Team discovers the problem when a learner complains on social media.

Orchestration failures are silent by default. Unlike a web server returning 500 errors, a content pipeline that silently drops items or loops indefinitely looks fine from the outside—new content still publishes, the dashboard still shows
green—until the accumulated failures become visible to end users.

The Four Signals

OpenTelemetry defines four observability primitives:

Traces capture request paths through the pipeline. A single content item might pass through five agents across 12 events over 3 days. Without distributed tracing, correlating a Code Validation failure with the Draft that triggered it
requires log archaeology.

Metrics track quantitative trends: token usage per content unit, retry rate, DLQ size, evaluation pass rate, P99 latency per agent. Metrics answer "is the system healthy?" while traces answer "what happened to this specific content item?"

Logs record discrete events with context. Structured JSON logs (not print statements) that include content ID, event type, agent name, and processing duration. Logs answer "what did this agent do?"

Baggage passes contextual information between signals (some practitioners group traces, metrics, and logs as the "three pillars" of observability, treating baggage as cross-cutting context rather than a fourth signal). A correlation ID
attached to the initial ContentCreated event propagates through every subsequent event, trace span, and log entry for that content item. Cross-signal correlation is what makes observability actionable rather than decorative.

What to Alert On

Not every metric needs an alert. Over-alerting causes the same problem as under-alerting: the team ignores signals.

LLM-Specific Considerations

Standard infrastructure observability misses LLM-specific failure modes. Failures can happen outside the LLM call itself—in retrieval, tool calls, or post-processing. A trace that only spans the API call misses the context assembly and response
parsing where many issues originate.

For high-volume systems, sampling helps manage cost. Tracing 10-20% of requests with full detail while logging basic metrics (latency, token count, success/failure) for all traffic balances visibility with overhead.

Track cost per content unit, not just per API call. A lesson that takes three revision cycles costs three times the LLM budget of a lesson that passes on the first attempt. Per-content-unit cost tracking reveals which content types are expensive
to produce—and whether the expense is justified by quality improvement.

Build vs Buy: When Custom Orchestration Makes Sense

Three scenarios favor building custom orchestration:

Learning the patterns. Building event-driven orchestration from scratch teaches failure modes that managed services abstract away. You learn why idempotency matters when your first retry creates a duplicate lesson. You learn why circuit
breakers matter when a degraded API consumes your entire retry budget. This educational value is real—especially for teams new to production agent systems.

Domain-specific requirements. Content versioning with evaluation history, curriculum-graph-aware prerequisite checking, and pedagogically-informed retry strategies aren't features that general-purpose workflow engines prioritize. Custom
orchestration lets you build these into the core rather than bolting them on.

Full control over cost trade-offs. Managed solutions make pricing and retry decisions for you. When your pipeline's cost structure depends on the relationship between token spend, human review costs, and content value, you may want direct
control over those trade-offs.

Three scenarios favor evaluating managed solutions (Temporal):

You've validated the patterns. Once you've built custom orchestration and understand the patterns, a managed solution saves operational overhead—infrastructure management, broker maintenance, scaling. Managed solutions also provide features like Temporal's heartbeat mechanism for long-running activities, where agents can checkpoint progress and resume after crashes without re-processing.

Cross-team visibility. Managed workflow engines provide dashboards, audit logs, and role-based access that take significant effort to build from scratch.

Orchestration shouldn't be your competitive advantage. If your team's value is in content quality and curriculum design, owning orchestration infrastructure may be the wrong allocation of engineering effort.

The honest answer: build it once to understand it, then decide if managed makes sense for your scale and team.

Wrapping Up

For content pipelines with retries, HITL, or multi-agent coordination, event-driven orchestration with explicit state management is the production-grade approach. The patterns covered here—stateful agents, event-driven coordination, failure isolation, idempotency, content versioning, and observability—address the coordination failures that dominate multi-agent system breakdowns.

The core patterns:

• Stateful agents with explicit state handoffs prevent stale-state failures

• Events decouple agents, enabling retries, HITL, and independent scaling

• Idempotency keys make retries safe—same input, same state, same output

• Event sourcing provides versioning, audit trails, and regression detection

• Observability catches the silent failures that accumulate without monitoring

These patterns have operational cost. Event-driven architecture is more complex than a for-loop. Idempotency stores require maintenance. Observability infrastructure needs its own monitoring. The cost is justified when the alternative—inconsistent content, unrecoverable failures, invisible degradation—affects the learners who depend on your pipeline.

Next in the series: The next post covers curriculum drift detection—using evaluation history to detect when generated content diverges from source material over time.

Further Reading

• The Future of AI Agents Is Event-Driven | Confluent — Event-driven architecture for agent systems

• Why do Multi-Agent LLM Systems Fail | Galileo — Production failure modes and coordination breakdowns

• Activity Definition | Temporal — Idempotency and activity patterns in workflow engines

• Event Sourcing Pattern | Azure Architecture Center — Append-only event stores for audit trails and versioning

• OpenTelemetry Signals — Observability primitives for distributed systems