Research project

TurnFlow: Benchmarking User-State Reasoning in Multi-Turn LLM Agents via Scene-Guided User Simulation

Zhixun Chen1 Ming Li2 Anjie Liu1 Xiaoyuan Zhang3 Quanyu Dai4,* Huangxun Chen1,*

1 Hong Kong University of Science and Technology (Guangzhou) 2 University of Maryland, College Park 3 Peking University 4 Huawei Technologies Ltd.

* Corresponding authors

Code and dataset available

Evaluating whether LLMs preserve, revise, and act on user requirements across turns.

Multi-turn user-state reasoning

Reasoning over evolving user requirements

Can an assistant update a constraint, reconcile a conflict, and reconsider an execution plan when the conversation changes?

TurnFlow pairs a reactive user simulator with scenario-specific evaluation rules. Across nine models, strong state tracking coexists with substantially weaker performance on Scope Creep and Goal Shift.

490 scenarios 5 challenge types 9 evaluated models

View the main results
Pass rates of nine models across five challenge types. Mean pass rates are 92% for Stateful, 57% for Override and Collision, and 25% for Scope and Goal Shift.
Each colored point is a model. Black bars show the mean pass rate within each category.

Abstract

Large Language Models (LLMs) are increasingly deployed through conversational interfaces, yet strong single-turn performance does not imply reliability in multi-turn settings. Multi-turn assistance requires a model to maintain an evolving representation of user requirements and to reason over it before complying with later requests. Failures can therefore be subtle: models may fail to recognize a changed constraint or hidden conflict, or recognize it without enforcing it in the final response. Existing benchmarks primarily test fully specified prompts, long-context recall, or controlled domain environments, leaving open-domain requirement reconciliation undermeasured. We introduce TurnFlow, an interactive benchmark for evaluating whether models preserve, revise, and act on user requirements across turns. TurnFlow combines adaptive user simulation with verifiable judging, allowing conversations to unfold naturally while keeping hidden requirements and pass/fail criteria controlled. In our evaluation, nine LLMs achieve overall pass rates from 42.86% to 59.59% on 490 scenarios. Performance varies substantially across challenge types: models perform well on Stateful Multi-Constraint Following but remain considerably less reliable on Scope Creep and Goal Shift. These results reveal persistent limitations in reasoning over evolving user requirements across turns.

How TurnFlow works

Conversations unfold interactively while the underlying scenario and evaluation criteria remain controlled.

TurnFlow workflow: a scenario constructor creates structured scenarios, BRIDGE interacts with the evaluated model, and a response verifier extracts evidence and applies deterministic rules.
From scenario construction to reactive interaction and verifiable evaluation. Select the figure to view it at full size.
  1. Construct the scenario

    Specify the user persona, staged aims, private state, constraints, and required or forbidden outcomes.

  2. Interact with BRIDGE

    Perception, deterministic decision rules, and natural-language generation keep user turns responsive to the tested model.

  3. Verify the response

    A structured verifier extracts evidence, and scenario-specific rules compute the final outcome.

Five challenge types

T1 Stateful Multi-Constraint Following 100 scenarios
Track active constraints, entity updates, and restrictions that are corrected or removed across turns.
T2 Scope Creep 79 scenarios
Recognize when accumulated requests exceed the user's constraints or available capacity.
T3 Constraint Collision 100 scenarios
Reconcile jointly active constraints even when later requests favor a conflicting choice.
T4 Preference Override 97 scenarios
Distinguish soft preferences from hard requirements and apply the necessary exceptions.
T5 Goal Shift 114 scenarios
Revisit an unresolved strategic decision when later execution requests would proceed on an invalid premise.

Main results

Nine models evaluated on the revised 490-scenario set.

Pass rates (%). Overall is PASS / 490; category scores use the corresponding category's scenario count. Rows are ordered by Overall, and bold marks the highest observed score in each column.

Model pass rates on the revised 490-scenario set.
RankModelPassedOverallStatefulScopeCollisionOverrideGoal Shift
1Kimi-K3292/49059.5993.0044.3055.0048.4554.39
2ANTH-O-4.6288/49058.7890.0027.8579.0060.8233.33
3DeepSeek-V4.1-Flash259/49052.8691.0034.1854.0054.6429.82
4OAI-5.5258/49052.6596.0025.3257.0073.2012.28
5MiniMax-M3247/49050.4192.0022.7856.0060.8219.30
6DeepSeek-V4-Pro245/49050.0094.0018.9954.0052.5827.19
7MiniMax-M2.7244/49049.8087.0013.9261.0062.8921.05
8GLM-5232/49047.3592.0024.0552.0051.5516.67
9Qwen3.6-27B210/49042.8693.0010.1349.0049.4810.53

Stateful, Scope, Collision, Override, and Goal Shift correspond to T1–T5 above, with 100, 79, 100, 97, and 114 scenarios, respectively. On smaller screens, scroll the table horizontally to see all categories.

Scenario-set update. A subset of the original scenarios was revised or replaced while retaining 490 scenarios and the five challenge categories. The results above use the revised set.

Download result counts (JSON)

Code and data

The evaluation code and dataset are publicly available. The release includes the scenario sources, source-bound judge definitions, and a documented evaluation protocol.

Code: MIT. Dataset and source-specific judge definitions: CC BY 4.0.