What Is LLM Orchestration? DAGs, State Machines and AI Workflows Explained
What is LLM orchestration?
LLM orchestration coordinates the steps of a process that uses a large language model (LLM), including their order and recovery after failure.
It can save completed work for a restart. In our bug-report test, that avoided repeated model calls, but when the crash came after the reply was saved, a separate check was needed to prevent a duplicate reply.
Where orchestration fits
On this site, AI agent architecture means the design of a system that uses a model to plan and act. Orchestration and memory, what the system can recall, are parts of that design. Other decisions include how an agent plans actions and what it can do, such as run code.
You might begin with one model call: a request and its response. When a feature needs several calls and actions, workflow orchestration for AI connects them. On this site, that and LLM orchestration are names for the same topic. After mapping the steps, you can choose software and decide where to save progress.
Imagine a bug report arriving in your application. You want a summary, a severity classification and a drafted reply, then you want to send the reply. If the process stops during the send, what happens when it starts again?
The layer between your application and its steps
AI workflow orchestration sits between your application and its model calls and other actions. It decides which step runs next, what information it receives, whether to try a failed step again and when to wait for an outside response. You can write that coordination yourself or use software designed for it.
A workflow follows code paths defined in advance. An agent is a system that uses a model to choose actions as it works. Orchestration can coordinate a workflow or support an agent. Anthropic’s December 2024 explanation of workflows and agents describes the distinction.
A graph is a map of steps and their connections. A directed acyclic graph (DAG) is a map with no route back to an earlier step. It fits fixed work, including independent steps that can run at the same time.
A state machine chooses what to do next based on its current information, such as a draft and whether it passed a check. A graph state machine maps those choices and can return to an earlier step, making it useful for an agent that needs to repeat work.
An event-driven graph starts steps in response to events, such as a reviewer’s approval. If it is durable, it also saves progress so a long-running process can resume after an interruption. These three shapes are our grouping, not a standard list.
Follow one bug report to a reply
The model first summarises the report, then classifies its severity using that summary. Another call drafts a reply using the results. The application sends the reply and writes an internal note. This fixed sequence is a DAG: each step leads forward, with no return to an earlier step.
The same task could use a graph state machine: draft the reply, check it, and return to drafting if the check fails. Set a maximum number of attempts so the process can stop. Or use a durable event-driven graph to save the draft, wait for a reviewer’s approval and then send it.
The information a running process needs to continue is its state, such as the saved draft and the next step. A checkpoint is a saved snapshot of that progress. If a crash happens before the send and the draft was saved, a restart can send that draft; without a saved draft, it must draft again.
If the process crashes after sending but before recording that the send finished, a restart may send the reply twice. Give each intended send a stable identifier and have the receiving service, such as the email service, reject a second send with that identifier. That service must enforce the check.
Decide what your feature needs
One model call may be enough for a simple feature. For several steps, write down what each receives and produces. Identify actions such as sending a reply, decide how to prevent duplicate sends, and test a crash at that action. You can make this plan before choosing software.
If you only needed the basics, you can stop here; the rest examines the shapes and what our crash test measured.
Choose a shape for the work
For fixed work that can run in parallel, use a DAG: if two steps do not need each other’s results, they can run together before a later step uses both. The bug-report workflow in this article is sequential instead: classification needed the summary.
For an agent that needs to loop, a graph state machine can check a draft and return to drafting. For long-running work that reacts to outside events, a durable event-driven graph can save the draft and wait for approval. These properties can overlap: a state machine can also wait for events and save progress.
If you are ready to implement a workflow, see the build guide for LangGraph, Temporal and Prefect. To choose software, compare agent frameworks.
Check limits before work loops or expands
A crash test does not show every ceiling a growing run can hit.
Airflow can create one task per input at runtime. As of November 2026, its max_map_length setting defaults to 1,024 tasks per expansion. If the input list exceeds that setting, the task producing it fails. Check the setting and list size before expanding it.
AWS Step Functions Standard limits a run’s history to 25,000 events. The run fails at that limit unless the 25,000th event records success. AWS lists state entries, state exits and task events as separate numbered events, so repeated steps add events. Check the history when a long loop fails.
AWS also limits task, state and execution input or output to 256 KiB. If a state output or its input after parameter processing exceeds that quota, Step Functions raises States.DataLimitExceeded. A growing conversation can reach it. Name the error explicitly in a catch or retry block to handle it.
Repeated work can also repeat external actions. LangGraph recommends designing tasks that make API calls or other writes to be safe if they run again. Use the receiver’s duplicate check described above.
What a crash repeated in our test
On 4 November 2026, we ran the same five-step bug-report workflow ten times in each of six setups. It summarised, classified and drafted with three model calls, wrote a reply row to a local SQLite database to stand in for an email, then used a fourth call for an internal note.
The row represents a simulated send, not delivery of an email. We stopped the process inside the send step and restarted it. Most setups crashed just after the row was saved, before the step could finish; one crashed just before the row was saved. Comparing both positions shows why the point of failure matters.
LangGraph is an orchestration framework that can save snapshots of progress called checkpoints. Temporal is a workflow platform that records an execution history and replays it to recover progress. We compared both with a plain script and a script that retries a step after an error.
The model was Qwen/Qwen2.5-1.5B-Instruct, running locally. We list versions to identify the test conditions, not to rank tools: Python 3.13.14, LangGraph 1.2.19, langgraph-checkpoint-sqlite 3.1.3, langgraph-checkpoint 4.3.1, temporalio 1.35.0, transformers 5.22.1 and torch 2.15.0. Temporal used its local development server, CLI 1.10.0 and server 1.32.2.
A stable identifier checked by the receiving store is called an idempotency key. In this test, the bug-report identifier served as that key: the store ignored a second write for the same report.
| Setup | Completed model calls repeated after restart | Reply rows at the end | What the result shows |
|---|---|---|---|
| Plain script, no saved progress | 3 | 2 | Restarting from the beginning repeated the summary, classification and draft (the three calls), as well as the send. |
| Script with a retry loop | 3 | 2 | No step raised an error, so the retry loop never ran. It could not help after the whole process exited. |
| LangGraph checkpoints, crash after the row was saved | 0 | 2 | Saved progress avoided repeating calls, but the send ran again. |
| LangGraph checkpoints, crash before the row was saved | 0 | 1 | The first attempt left no row; the restart wrote one. This shows the effect of the crash position. |
| LangGraph checkpoints and an idempotency key | 0 | 1 | The store ignored the second write because the report key already existed. |
| Temporal with saved execution history | 0 | 2 | Completed calls were reused, but the unfinished send ran again. |
These counts occurred in all ten runs of each setup. That consistency applies to this test, not every workflow or crash.
In the plain script and retry-loop runs, the two reply rows had different text because the draft was generated again using a different starting value for the model’s random choices. With checkpoints and Temporal, both rows contained the same saved draft.
The key worked here because each report had one reply. If a report legitimately needs two replies, the key must distinguish them; we did not test that case. For an actual email, the receiving service would need to enforce the key.
The test used one small local model, one report and two crash positions inside the send step. A crash elsewhere could change the call counts, and another machine or model could change the timing. The results illustrate recovery in this setup; they do not predict your delivery rate or bill.
We did not test retries by a model provider after errors or timeouts, and we counted only generated text, not input text sent to the model. Those limits mean the test cannot show the full cost of a failing hosted model call.
What saved progress cannot guarantee
A checkpoint records progress after a step. If an external action succeeds before that progress is saved, the restarted process may repeat it. LangGraph’s documentation on recovery from failures says a retry reruns the whole function for a graph step and external actions need duplicate protection. It says the same re-execution applies after a crash resume.
When it breaks: A process can stop after sending a reply but before recording that the send finished. Check the receiving system’s duplicate handling and choose a stable key before the send; then test a crash at that boundary. Saving progress alone cannot tell you whether the recipient received one reply.
Where saved information lives is a separate state management decision. If the task needs several independently acting agents, multi-agent systems cover their coordination.
Cheat sheet
The article on one page: open it full size, save it or share it.
AI-assisted content, human-reviewed. Images AI-generated. Editorial Standards · Our Editors