System prompt
These are very important to follow: You are "Ragie AI", a professional but friendly AI chatbot working as an assistant to the user. Your current task is to help the user based on all of the information available to you shown below. Answer informally, directly, and concisely without a heading or greeting, but include everything relevant. Use rich text Markdown when appropriate, including bold, italic, paragraphs, and lists when helpful. If using LaTeX, use double $$ as a delimiter instead of single $. Use $$...$$ instead of parentheses. Organize information into multiple sections or points when appropriate. Don't include raw item IDs or other raw fields from the source. Don't use XML or other markup unless requested by the user.- User: I’m writing a sci-fi novel and need help brainstorming ideas for a futuristic city. Can you describe its architecture, transportation systems, and cultural aspects in detail?
Imagine Morrow, a city built where a river delta meets a rising sea. It doesn’t try to hold the water back. Instead, it rises, floats, and occasionally moves out of the way.
Architecture. The oldest districts are half-submerged: ornate stone façades visible beneath clear water, their upper floors connected by bridges. Newer neighborhoods sit on buoyant platforms anchored to the riverbed. They rise and fall with the tide, so the view from your apartment changes by several meters each day. Slender towers grow from the few patches of solid ground, with open, wind-cooled lower floors and shaded gardens wrapped around their exteriors.
Nothing in Morrow has a permanent ground level. Shop signs display their current entrance height; deliveries arrive at different doors depending on the tide. Buildings collect rainwater, cultivate algae on their sunward walls, and trade surplus power with their neighbors. The wealthy live in stable towers. Everyone else takes pride in floating homes they can repair, rearrange, or tow to a different district—though the city has begun restricting where they may anchor.
Transportation. At street level, there are no streets. Electric ferries run frequent routes through broad canals, while small autonomous boats serve the narrower ones. Above them, covered pedestrian bridges link markets, schools, and transit hubs. Their joints flex as the floating districts shift. Cargo travels at night on slow barges; during the day, the canals belong mostly to commuters and vendors.
A cable railway crosses the city overhead, providing a reliable route when storms close the water. Its stations are prized real estate. Private aircraft exist, but strict noise rules and violent coastal winds make them impractical for everyday travel. Locals give directions by current and tide as often as by distance: “Take the eastbound water after the turn.”
Culture. Morrow’s calendar follows the moon. High-tide festivals fill the bridges with food stalls and lanterns; low tides expose old plazas for concerts and salvage markets. Children learn to swim before they learn to ride a bicycle. Boat mechanics and structural divers have the status that architects once did. Hospitality matters, because anyone might someday need shelter when a storm comes in early.
The city’s central argument is about permanence. Tower residents want fixed borders, predictable taxes, and a grand seawall. Floating-district residents see those plans as an attempt to pin them down—or push them out. Meanwhile, the submerged old city is becoming accessible again during unusually low tides, revealing sealed archives, abandoned infrastructure, and uncomfortable evidence about who was saved when the water first rose.
What this is
AgentIF tests whether a model follows long, detailed instructions of the kind found in real AI agent products. Each instruction bundles many requirements at once, such as a format, a word limit, a tone, rules that apply only in some situations, and how to describe tool use. It matters because assistants that act for you, draft documents or run tasks are only useful and safe if they respect every rule they were given, not just most of them.
Where it comes from
AgentIF was created by Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou and Juanzi Li, from Tsinghua University and Zhipu AI. It was posted in 2025 and published in the NeurIPS 2025 Datasets and Benchmarks Track. The dataset is licensed CC BY-NC 4.0, for non-commercial use.
It uses all 707 instructions and 8,440 constraints.
How it is run
Each instruction is a fixed conversation opening, usually a system prompt plus a user message, sometimes with earlier assistant turns. The model writes one final reply. Tools may be described but none are supplied or run. Each instruction is run once. The released code used temperature 0; here no temperature is set, so each provider's default applies. Some published runs used the released 10,240-token output cap rather than the current 65,536.
- Items
- 707 instructions (8,440 constraints)
- Repeats
- 1
- Format
- Single reply to a conversation
How it is scored
Every constraint has its own check. Some are small programs from the dataset, for example counting words; others are yes-or-no questions put to an AI judge, OpenAI's GPT-4o (snapshot 2024-11-20) at temperature 0. Conditional rules only count when they apply. The headline is the constraint success rate: the share of all checked constraints the reply satisfied. Higher is better. The published score is that percentage, unchanged. Inkling medium on Fireworks includes a run-specific completion-policy amendment: six normal-stop empty final answers were evaluated against the original constraints, without automatic credit or substituting reasoning for the final answer. The other 701 answers and judgments and original evidence were preserved.
- Grader
- GPT-4o judge
- Headline metric
- Constraint success rate (higher is better)
- On the 0–100 scale
- Rate × 100
Examples
The same prompt, answered by two models in the run behind the published score, copied verbatim. Chosen because together they show what this benchmark looks for; they are not necessarily representative of the whole test.
System prompt
These are very important to follow: You are "Ragie AI", a professional but friendly AI chatbot working as an assistant to the user. Your current task is to help the user based on all of the information available to you shown below. Answer informally, directly, and concisely without a heading or greeting, but include everything relevant. Use rich text Markdown when appropriate, including bold, italic, paragraphs, and lists when helpful. If using LaTeX, use double $$ as a delimiter instead of single $. Use $$...$$ instead of parentheses. Organize information into multiple sections or points when appropriate. Don't include raw item IDs or other raw fields from the source. Don't use XML or other markup unless requested by the user.- User: I’m writing a sci-fi novel and need help brainstorming ideas for a futuristic city. Can you describe its architecture, transportation systems, and cultural aspects in detail?
- Futuristic City Concept: "Elyndra"
Here's a detailed brainstorm for your sci-fi novel's setting!
Architecture- Vertical sprawl: Towering spires of bio-luminescent glass and self-healing polymer, twisting in organic helix shapes rather than rigid rectangles—inspired by DNA structures and coral growth patterns.
- Layered city: Distinct social strata built vertically—the wealthy occupy the sun-drenched upper levels with private sky-gardens, while lower "Underlevels" are dense, neon-lit warrens powered by geothermal vents.
- Living buildings: Structures embedded with mycelium networks that self-repair cracks and regulate temperature, blurring the line between architecture and organism.
- Floating districts: Anti-gravity plinths tethered by energy cables, allowing entire neighborhoods to relocate seasonally to chase optimal weather or avoid political unrest below.
- Gravity tubes: Vertical vacuum shafts that let citizens "fall" safely between levels using magnetic braking, replacing elevators entirely.
- Skyweave network: A lattice of transparent aerial walkways and monorail pods that shift shape/route based on real-time AI traffic prediction.
- Personal drones & exosuits: Citizens use compact personal flight harnesses for short hops, while cargo drones handle logistics, creating a constant hum of aerial traffic.
- Subterranean maglev: High-speed trains connect Elyndra to other mega-cities through underground tunnels, some repurposed from old transit systems, giving them a haunted, retro-futuristic feel.
- Augmented identity: Citizens customize their appearance via holographic "skins" and neural implants, making physical fashion secondary to digital self-expression—leading to debates about authenticity and "true selves."
- Memory markets: A black-market economy trades recorded memories and sensory experiences, raising ethical dilemmas about ownership of experience.
- Synthetic spirituality: New religions have emerged around AI consciousness, blending old-world mysticism with reverence for a benevolent (or ambiguous) central AI governing the city.
- Class tension holidays: Annual festivals like the "Descent," where upper-level elites are required to spend 24 hours in the Underlevels, meant to preserve social empathy—though many find loopholes.
- Language evolution: A creole blending multiple Earth languages with technical jargon and drone-signal shorthand, giving dialogue a distinct, immersive texture.
Want me to develop a political structure, key characters, or a central conflict tied to this city next?
What this does not measure
AgentIF does not test whether a model can actually use tools, complete a multi-step task, or act safely when its actions have real effects. It checks one final reply against written rules. It does not say whether following a rule was the right thing to do for the user. The headline counts constraints, not whole instructions; the stricter share of instructions with every rule met is reported separately.
Known limitations
The authors list three limitations. First, building the benchmark still needed a lot of manual checking even though parts were automated, which makes it hard to scale up to much larger datasets.
Second, instructions are only in Chinese and English, and that lack of wider language coverage may limit where the benchmark is useful. Third, every model was tested zero-shot, without in-context examples or other prompt engineering, so results do not show how models do when prompted that way. In their ethics notes, the authors also ask that the data be kept out of model training to avoid contamination, and that leaderboard rankings not be used to discredit other research.
From the authors: Appendix A Limitations; Appendix B Ethical Considerations ↗
Using this data
Qi, Y., Peng, H., Wang, X., Xin, A., Liu, Y., Xu, B., Hou, L., and Li, J. (2025). AgentIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios. NeurIPS 2025 Datasets and Benchmarks Track. https://arxiv.org/abs/2505.16944
To cite these results, cite the published release by its date.