Agent Next Phase: Why Co-Design?

16 Sep, 2026

From Agent Workloads to Models, Infrastructure, Hardware, Harnesses, and Human–Agent System

AI optimization used to be easy to describe as a stack: model researchers changed the architecture, infrastructure engineers made it run faster, and hardware eventually caught up. That picture is still useful, but Agent workloads are making it increasingly incomplete.

The more interesting pattern now runs in both directions. Agent workloads expose new bottlenecks, those bottlenecks reshape model architecture and runtime systems, stable workload assumptions eventually enter hardware, and the resulting experience flows back into data, memory, evaluation, and harness design.

TL;DR

• Agents change the workload before they change the model.Workload then propagates downward through the entire stack: Model → Runtime → Hardware.
• At the same time, operational data becomes active. Sessions, trajectories, memory, replay, and evaluation increasingly influence the next execution step, which means Data is no longer just input; Data is becoming Runtime.
• This is why I increasingly use two simple ideas to think about the next phase of AI :Algorithms are data, data is Infra, Infra is algorithms.
• Agent = Model + Harness. Underneath both sits a deeper architecture question: Where should intelligence live?

What actually changes when a Chatbot becomes an Agent?

!Sources / original figures: AgentSysBench, Fig. 1 · ReAct

Sources / original figures: AgentSysBench, Fig. 1 · ReAct

A Chatbot answers a request; an Agent stays inside a task. The underlying foundation model may even be identical, but the execution pattern is not.

A conventional interaction has a clean lifecycle: a prompt arrives, the model runs, a response is returned, and the request ends. An Agent instead reasons, calls a tool, waits for an external system, receives an observation, updates its working context, and re-enters the model.

Chatbot

Prompt → Model → Response

becomes

Agent

Goal
 ↓
Reason
 ↓
Act
 ↓
Observe
 ↓
Update Context
 ↓
Reason Again
 ↺

The key change is easy to miss because it happens outside the model. A single request quietly turns into a long-running process, and the system now has to preserve state across model calls, tools, sandboxes, retrieval, network activity, and periods of inactivity.

Recent systems work is beginning to quantify this difference. AgentSysBench characterizes agentic applications as heavyweight, stateful, and heterogeneous; in several workloads, non-LLM components can dominate end-to-end latency. That is a useful reminder that “Agent serving” is not simply LLM serving with a while loop added on top. AgentSysBench, 2026

YJ’s opinionThe important transition from Chatbot to Agent is not “more intelligence per request.”It is the transition from a request-oriented system to a process-oriented system.

That distinction is the starting point for everything that follows: once the workload changes, the rest of the stack eventually has to respond.

What makes an Agent workload fundamentally different?

!Sources / original figures: AgentSysBench

Sources / original figures: AgentSysBench

Traditional LLM serving has a shape that infrastructure engineers understand extremely well:

Agent execution breaks that shape. A tool returns new information, the observation is appended to the working context, the model enters Prefill again, Decode resumes, another tool may be called, and the Agent may sit idle for minutes before continuing.

The execution path becomes closer to:

Several system pressures appear at the same time. Context gets longer, reasoning gets longer, Prefill occurs repeatedly, the request becomes stateful, and CPU, GPU, storage, retrieval, and networking all enter one latency path. The uncomfortable part is that users still expect the system to feel interactive.

This is why the workload itself becomes the first useful abstraction. AgentSysBench reports persistent state, heterogeneous resource affinity, shifting bottlenecks, control-plane overhead, and substantial cross-request redundancy in real agent traces. These are not small variations on the traditional serving problem; they are signs of a different execution regime. AgentSysBench, 2026

The practical implication is straightforward: an Agent workload is no longer simply autoregressive inference. It is a dynamic execution graph with state.

Why is workload becoming a model-design constraint?

!Sources / original figures: DeepSeek-V3, Fig. 2

Sources / original figures: DeepSeek-V3, Fig. 2

Long context initially looks like an infrastructure problem. More tokens mean more KV Cache, which means more memory capacity, more bandwidth pressure, and more data movement.

But once that pressure becomes large enough, Runtime can no longer hide it. The bottleneck leaks into the architecture.

The evolution from MHA to GQA and MLA makes this visible. More recent architectures go further, combining full attention with sparse, linear, or hybrid mechanisms because paying dense-attention cost over an ever-growing context becomes increasingly difficult to justify.

DeepSeek-V3 is a representative example. MLA is part of an architecture explicitly designed for efficient inference, while DeepSeekMoE expands total model capacity without activating every parameter for every token. The system constraint is no longer downstream from the model; it participates in shaping the model. DeepSeek-V3 Technical Report, 2024

The same pressure appears with reasoning. Longer reasoning means more Decode tokens, so techniques such as MTP, speculative decoding, efficient attention, and sparse activation become more valuable. The point is not that Agents “invented” these architectures, but that new workloads make certain bottlenecks important enough to become architecture constraints.

Workload constraints are becoming model-design constraints.

The arrow now points in both directions:

Model → Workload
Model ← Workload

And once architecture changes, Runtime cannot remain unchanged.

Why does workload eventually reshape Runtime?

!image.png

Prefill and Decode are still the cleanest example. They belong to the same autoregressive inference request, but they stress the system differently: Prefill favors large parallel computation, while Decode is far more sensitive to memory bandwidth, small-batch efficiency, and per-token latency.

That mismatch naturally motivates PD disaggregation, but Agent workloads extend the same idea into more dimensions. Long context requires KV management, repeated context makes Prefix Cache valuable, MoE creates expert-placement and All-to-All problems, tool calls introduce suspend and resume, and persistent sessions make locality and affinity increasingly useful.

At this point, Runtime is doing much more than executing a computation graph. It decides who runs, when they run, where state lives, what stays resident, what gets moved, and what can be discarded.

Those are policy decisions. A scheduler selects, a router decides, a KV manager retains or evicts, and expert placement maps sparse computation onto topology.

Infra is algorithms.

Traditional inference Runtime asks:

“How do I run this model efficiently?”

Agent Runtime increasingly has to ask:

“What is this workload trying to do?”

Once those workload patterns become stable enough, software specialization begins to reach hardwa

When do workload assumptions become Hardware?

!image.png

For a long time, accelerator design followed a fairly general rule: more compute, more memory, more bandwidth. Prefill and Decode make that “more-is-better” strategy look increasingly wasteful because they consume those resources in very different proportions.

SPAD makes the argument explicitly. Prefill is compute-bound while Decode is memory-bound, so serving both phases with the same accelerator can leave different resources underutilized. The paper therefore explores specialized Prefill and Decode chips rather than assuming one general-purpose design should serve both equally well. SPAD, 2025

Industry is moving in a similar direction. Huawei’s Ascend 950 roadmap separates 950PR for Prefill/recommendation workloads from 950DT for Decode/training, reflecting different compute, memory-capacity, and memory-bandwidth requirements. NVIDIA is exploring another form of specialization with Groq 3 LPX as an extension of Vera Rubin, pairing GPU-scale throughput with an accelerator designed for predictable, low-latency token generation in agentic workloads. Huawei Ascend 950 · NVIDIA Groq 3 LPX

Two longer-term trends start to emerge together. Models become increasingly sparse, while clusters become larger but internally more specialized.

Dense Model
    ↓
MoE / Sparse Activation

General Cluster
    ↓
Larger + Specialized Cluster

More scale does not necessarily mean more identical devices. It may mean more distinct roles inside one system.

Hardware is the executable form of workload assumptions.

Not every workload deserves dedicated silicon, but when a workload becomes important, expensive, and stable enough, its assumptions eventually become physical.

Are Algorithm, Data, and Infra still separate layers?

!image.png

At this point, the traditional stack starts to feel strangely rigid. We still talk about Algorithm, Data, and Infra as clean categories, while Agent systems continuously turn one into another.

Planning and tool use begin as behavior. Once executed at scale, they become trajectories, rollouts, preferences, and experience; those experiences can later become SFT or RL data.

Algorithms become data.

But Agent data is not passive either. Sessions, trajectories, memory, replay, rewards, and evaluation influence what the system does next.

Data becomes Infra.

And the infrastructure underneath is full of decisions: schedulers select, routers route, KV managers retain or evict, and expert placement decides where sparse computation lands.

Infra becomes algorithms.

The interesting claim is not that these concepts are literally identical. It is that their boundaries stop being stable.

Algorithms are data, data is Infra, Infra is algorithms.

This is the point where Co-Design stops looking like a performance technique and starts looking like a better mental model for the whole system.

When does Data become Runtime?

!image.png

Traditional machine learning mostly treats Data as evidence about the past: prepare a dataset, train on it, then deploy the resulting model.

Agent systems continuously generate a different kind of data: session state, trajectories, tool results, memory, rewards, replay, and evaluation. The important change is that this information does not simply wait for the next training run; it directly influences the next execution step.

A memory item determines what enters Context. A tool result changes the next action. Replay decides which experience is reconsidered. Eval determines which failures become permanent regression cases.

The transition looks roughly like:

Dataset
   ↓
Experience
   ↓
State
   ↓
Policy Input
   ↓
Runtime Decision

MemGPT anticipated part of this shift by treating context management through the mental model of operating-system virtual memory, with data moving between active context and external memory under explicit control. MemGPT, 2023

Data is no longer input. Data is becoming Runtime.

In traditional ML, Data mainly records what happened.In Agent systems, Data increasingly determines what happens next.

Once Context, Memory, Tools, State, and Eval all become part of runtime capability, a larger architectural object becomes difficult to ignore.

Why is an Agent more than a Model?

!image.png

That larger object is the Harness.

A useful simplification is:

Agent = Model + Harness.

The Model provides general capabilities such as understanding, reasoning, and generation. The Harness turns those capabilities into executable behavior by supplying Context, Tools, Memory, Session State, Runtime, Eval, and access to an environment.

This distinction matters because Agent capability increasingly cannot be explained by model capability alone. A strong Model inside a weak Harness may fail to express what it can actually do; a sophisticated Harness around an incapable Model does not magically create reasoning either.

That is why I prefer the stronger formulation:

Capability ≈ Model × Harness.

OpenAI now explicitly uses the term Harness Engineering to describe engineering around agent environments, constraints, repository legibility, evaluation, and feedback loops. Anthropic reaches a similar conclusion from long-running Agent development: harness design can materially change performance, but every harness component also encodes an assumption about what the current Model cannot do on its own. OpenAI, Harness Engineering · Anthropic, Harness Design

Agent intelligence is becoming a system property, which leads directly to the harder question.

Where should intelligence live?

!image.png

Planning can be learned by the Model or maintained explicitly by the Harness. Memory can live in weights and active context, or be externalized. Tool use can be a learned behavior, or constrained through an explicit runtime.

None of these boundaries is permanent.

A capability may belong inside the Model when generalization and flexibility matter most. The same capability may be better externalized when observability, editability, verification, recovery, or governance matter more.

SWE-agent gives a clean example. A carefully designed Agent-Computer Interface materially changes how language-model Agents navigate repositories, edit files, and execute programs. The interface is not the model, but it changes the capability of the system. SWE-agent, 2024

The real distinction is often not Model versus Harness. It is:

Co-Design is not primarily about components. Co-Design is about capability boundaries.

Once that boundary is dynamic, Model and Harness can no longer evolve independently.

Why will Model × Harness Co-Design matter?

!image.png

Agent engineering has already moved through several recognizable phases:

Prompt Engineering
       ↓
Tool Use
       ↓
Agent Frameworks
       ↓
Harness Engineering
       ↓
Model × Harness Co-Design

The temptation is to read this as an ever-growing pile of scaffolding. I think that is the wrong trend line.

Harnesses encode assumptions about what the current Model cannot reliably do. If the Model cannot plan, we add a planner; if it cannot preserve state, we add memory; if it cannot reliably route tools, we add explicit control.

But those assumptions expire.

Anthropic makes this point directly: harness components often encode assumptions about Model limitations, and those assumptions can quickly go stale as models improve. A mechanism that was load-bearing for one Model generation may become dead weight for the next. Anthropic, 2026

So:

Model_t + Harness_t
        ↓
Model_t+1 + Harness_t+1

does not imply that Harness_t+1 is larger.

Sometimes better Co-Design means deletion.

The deeper trend is:

Model and Harness are becoming a coupled architecture.

Harness Engineering becomes increasingly Model-aware, while model development may eventually become increasingly Harness-aware.

What happens when the system can modify itself?

!image.png

So far there is still one hidden assumption: Humans perform the Co-Design.

Humans change the Model, edit the Harness, modify the Tools, design the Eval, and choose which experiments survive. But all of those objects are becoming increasingly machine-editable.

Once an Agent can observe performance, propose a modification, execute it, evaluate the result, and retain the better version, the loop changes:

Observe
  ↓
Propose
  ↓
Modify
  ↓
Evaluate
  ↓
Select
  ↓
New System
  ↺

AlphaEvolve is one bounded example of this pattern: language models propose candidate programs, automated evaluators measure them, and successful candidates feed subsequent evolutionary search. Google DeepMind, AlphaEvolve

This is not the same thing as open-ended recursive self-improvement. Evaluation reliability, reward hacking, regression, safety, and rollback remain hard constraints.

But the optimization target has already expanded.

Co-Design optimizes the system.Co-Evolution lets the system participate in optimizing itself.

The interesting recursion may begin not with a Model directly training itself, but with the improvement loop becoming executable.

What does Co-Design mean for Humans?

!image.png

Once capability boundaries become the central idea, Human–Agent collaboration becomes easier to reason about.

The useful question is not “Will AI replace humans?”

It is: Which capability should live where?

Humans remain especially important for goals, judgment, taste, responsibility, and problem framing. Agents are increasingly strong at execution, iteration, persistent operation, and scale. Those are not permanent categories; they are another moving capability boundary.

HFlow provides one early example of this direction. It treats the workflow itself as an object that can evolve, while human preferences guide automated search and experience memory preserves reusable strategies. HFlow, ACL 2026

The same principle applies to learning. Traditionally, a failure updates the individual: “I will remember this next time.” An Agent-native workflow can make the same failure update the system.

Failure
  ↓
Failure Case
  ↓
Rule
  ↓
Eval
  ↓
Skill

Now the Human gains a better mental model while the Agent gains a better rule, evaluation, tool, or Harness. The experience becomes executable.

The new unit of learning is the Human-Agent System.

Not Human versus Agent, but a system that learns through both.

A new Mental Model

The old picture was hierarchical:

Model
  ↓
Infra
  ↓
Hardware

The emerging picture is cyclic.

Agent changes Workload. Workload reshapes Model. Model and Workload reshape Runtime. Stable Runtime assumptions enter Hardware. Experience becomes Data. Data becomes Runtime. Runtime and environment become Harness. Harness changes Agent capability, and the loop begins again.

Agent
 ↓
Workload
 ↓
Model
 ↓
Runtime
 ↓
Hardware

Experience
 ↓
Data
 ↓
Harness
 ↓
Agent

This is why Co-Design feels increasingly useful as a mental model. Not because every engineer needs to become a full-stack expert, but because local optimization increasingly fails to explain global behavior.

I would compress the entire argument into four lines:

Agent changes the workload.Workload reshapes the stack.Co-Design is about capability boundaries.The unit of competition is shifting from Model to System.

And once the same logic reaches learning itself:

The new unit of learning is the Human-Agent System.

The scarce capability in that world may not be knowing every layer. It may be understanding how a change at one layer propagates through the whole system, and knowing where the globally better place to solve it actually is.

That is the kind of Co-Design I think AI engineering is moving toward.

References

1. Chang et al., From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems, 2026.https://arxiv.org/abs/2608.15127

2. Pan & Li, A Survey of LLM Inference Systems, 2025.https://arxiv.org/abs/2506.21901

3. DeepSeek-AI, DeepSeek-V3 Technical Report, 2024.https://arxiv.org/abs/2412.19437

4. Zhang et al., SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference, 2025.https://arxiv.org/abs/2510.08544

5. Huawei, Ascend 950PR / 950DT, 2025.https://www.huawei.com/en/news/2025/9/hc-xu-keynote-speech

6. NVIDIA, Inside NVIDIA Groq 3 LPX, 2026.https://developer.nvidia.com/blog/inside-nvidia-groq-3-lpx-the-low-latency-inference-accelerator-for-the-nvidia-vera-rubin-platform

7. Packer et al., MemGPT: Towards LLMs as Operating Systems, 2023.https://arxiv.org/abs/2310.08560

8. Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024.https://arxiv.org/abs/2405.15793

9. OpenAI, Harness Engineering: Leveraging Codex in an Agent-First World, 2026.https://openai.com/index/harness-engineering/

10. Anthropic, Harness Design for Long-Running Application Development, 2026.

    https://www.anthropic.com/engineering/harness-design-long-running-apps

11. Google DeepMind, AlphaEvolve, 2025.

    https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/

12. Liu et al., Evolving Agentic Workflow Driven by Human-Agent Collaboration, ACL 2026.

    https://aclanthology.org/2026.findings-acl.1250/