Generation
456- 输入Sees
- 题面、约束、I/O 约定、限制、公开样例Statement, constraints, I/O spec, limits, public samples
- 输出Produces
- 解题程序A solution program
- 指标Metric
- pass@1(可选 pass@k):通过
score_case_ids全部测试pass@1 (optionally pass@k): passes every test inscore_case_ids
LiveCodeBench-Latest · Release 2026-11
面向 Coding Agent 的时间留出评测集:收录 2026 年 6 月至 11 月 Codeforces 与 AtCoder 比赛中的 456 道标准输入输出题,每题附完整测试、checker、程序池和五类任务定义。 A time-held-out benchmark for coding agents: 456 standard-I/O problems from Codeforces and AtCoder contests held between June and November 2026, each shipped with full tests, a checker, a program pool, and five task definitions.
收录 2026-06-01 至 2026-11-16 期间已结束比赛中的全部非交互标准输入输出题,覆盖所有难度;题目实际发布日期为 2026-06-06 至 2026-11-15。 All non-interactive standard-I/O problems from contests that finished between 2026-06-01 and 2026-11-16, across every difficulty level. Actual release dates span 2026-06-06 to 2026-11-15.
| 平台Platform | 比赛类型Contest type | 题数Problems |
|---|---|---|
| Codeforces | Div. 1 | 42 |
| Codeforces | Div. 2 | 104 |
| Codeforces | Div. 1 + 2 | 35 |
| Codeforces | Educational | 40 |
| AtCoder | ABC | 167 |
| AtCoder | ARC | 61 |
| AtCoder | AGC | 7 |
| 合计Total | 456 | |
11 月仅统计至 11-15November covers releases through Nov 15
182 道有官方 rating;另有 39 道暂无 rating182 problems with an official rating; 39 have none yet
全部 456 题;标签来自平台或由 Astra 补充,来源见 tag_detailsAll 456 problems; tags come from the platform or are added by Astra — see tag_details
uid,并通过 family_id 关联。Easy / Hard versions keep separate uids and are linked by family_id.
能力划分参考 LiveCodeBench。Tasks/ 中是按可见性规则导出的模型输入;评分所需的隐藏材料保留在各题包中,通过 task_id 对应。
The ability split follows LiveCodeBench. Tasks/ holds model-facing inputs exported under visibility rules; hidden grading material stays in each problem package and is joined via task_id.
score_case_ids 全部测试pass@1 (optionally pass@k): passes every test in score_case_idsGeneration — 每题 n 次独立采样、c 次通过:with n independent samples per problem and c passing:
pass@k = mean_over_problems(1 − C(n−c, k) / C(n, k)), n ≥ k
Repair — 修复后通过全部评分测试的实例数 / 初始确实失败的实例数。多轮修复需固定轮数或成本预算。instances passing all scoring tests after repair / instances that initially fail. Multi-round repair should fix a round or cost budget.
Prediction — 每题最多 3 份非样例输入,按 token 序列去重;构造题接受任意合法答案。up to 3 non-sample inputs per problem, deduplicated by token sequence; any valid answer is accepted for constructive problems.
Execution — 真值是指定程序的实际 stdout;1 与 1.0 不等价。ground truth is the program's actual stdout; 1 and 1.0 are different.
Testcase Gen — 输入先过 validator,再运行标准解、目标程序和 checker。预算:每实例最多 10 份输入、合计 16 MiB、generator 超时 20 s。inputs must pass the validator, then std, the target program and the checker are run. Budget: ≤ 10 inputs, 16 MiB total, 20 s generator timeout per instance.
数据采用 format 2.0,任务协议为 stdio_v2。每道题一个目录,目录名形如 atcoder_abc461_a,对应 UID atcoder:abc461_a。
Format 2.0, task protocol stdio_v2. One directory per problem, e.g. atcoder_abc461_a for UID atcoder:abc461_a.
OOD-Test/
├── README.md
├── index.html
├── Tasks/
│ ├── generation.json
│ ├── repair.json
│ ├── prediction.json
│ ├── execution.json
│ └── testcase_gen.json
└── problems/
└── <platform>_<problem_id>/
├── description.json
├── cases.json
├── tasks.json
├── expected_verdict.json
├── generator.cpp
├── validator.cpp
├── std.cpp
├── checker.cpp
└── solutions/
└── *.cpp / *.py| 文件File | 内容Contents |
|---|---|
description.json | 题面、约束、tag、rating、题解、限制、运行环境与来源Statement, constraints, tags, rating, editorial, limits, runtime, provenance |
cases.json | 完整 stdin 与标准程序的实际输出Full stdin strings and the reference program's stdout |
tasks.json | 五类任务的可见性配置与评分引用Visibility and scoring config for all five tasks |
expected_verdict.json | 每个程序在每个测试上的预期 verdictExpected verdict of every program on every test |
generator.cpp | 以 seed 为参数输出一份完整输入Takes a seed, prints one full input |
validator.cpp | 读 stdin,退出码 0 表示输入合法Reads stdin; exit code 0 means valid |
std.cpp | C++17 标准解C++17 reference solution |
checker.cpp | checker input output answer;退出码 0 表示接受; exit code 0 means accepted |
solutions/ | 额外的正确或错误程序(C++ / Python)Extra correct or buggy programs (C++ / Python) |
cases.json 示例A cases.json record{
"case_id": "t0000",
"input": "4 5\n",
"expected_output": "Yes\n",
"kind": "sample",
"coverage": "official sample",
"source": {
"origin": "source_snapshot",
"url": "https://atcoder.jp/contests/abc461/tasks/abc461_a",
"upstream_case_id": "t0000"
}
}
一份测试输入是一次完整的 stdin,可包含多个题内 test case。g0000–g0009 由 generator.cpp 以 seed 0–9 生成。构造题的保存输出可能与官网样例不同,判题请使用本题 checker.cpp。
One test input is one complete stdin and may contain several in-problem test cases. g0000–g0009 come from generator.cpp with seeds 0–9. For constructive problems the stored output may differ from the official sample, so always judge with the problem's checker.cpp.
default_visibility→tasks.<task>.visibility→case_overrides
后者覆盖前者,未开放的字段默认隐藏。题解、tag、标准解、其他程序和目标输出均可分别控制。 Later levels override earlier ones; anything not opened stays hidden. Editorials, tags, the reference solution, other programs and target outputs are controlled independently.
原题限制与数据集执行限制分别存于 limits.original 与 limits.dataset_execution。时限只对较慢的机器放宽,上限 20 秒:
Original and dataset limits are stored in limits.original and limits.dataset_execution. Time limits are only relaxed on slower machines, capped at 20 s:
multiplier = max(1, 4e8 / measured_rate) timeout_seconds = min(20, original_time_seconds × multiplier)
本批在 Intel Core Ultra 7 265K 上测得约 1.1732e9 次/秒,倍率为 1,即直接使用原题时限。 This release was built on an Intel Core Ultra 7 265K at ≈ 1.1732e9 iterations/s, so the multiplier is 1 and original time limits apply as-is.
| C++ 标准与编译参数C++ standard & flags | C++17, -O2 -pipe |
| 解答程序内存Solution memory | max(512 MiB, original),按进程虚拟地址空间, per-process virtual address space |
| 单个输出文件上限Output file limit | 64 MiB |
| 编译超时Compile timeout | 90 s |
| generator / validator / checker 超时generator / validator / checker timeout | 各 20 s20 s each |
| 判题环境Judge | upsolve-judge:1,无网络 Docker,每 worker 1 CPU, network-less Docker, 1 CPU per worker |
| testlib | b0f7477,编译时 -I third_party/testlib, compile with -I third_party/testlib |
下载数据集,并用 Python 标准库读取一道题及其 Generation 输入。 Download the dataset and read one problem plus its Generation input with the Python standard library.
hf download Sy-SU/OOD-Test --repo-type dataset --local-dir OOD-Test
import json
from pathlib import Path
root = Path("OOD-Test")
folder = root / "problems" / "atcoder_abc461_a"
def read_json(path):
with path.open(encoding="utf-8") as stream:
return json.load(stream)
description = read_json(folder / "description.json")
cases = read_json(folder / "cases.json")["cases"]
tasks = read_json(folder / "tasks.json")["tasks"]
print(description["uid"], description["title"])
print("test inputs:", len(cases))
print("generation scoring tests:", tasks["generation"]["instances"][0]["score_case_ids"])
model_inputs = read_json(root / "Tasks" / "generation.json")
task = next(item for item in model_inputs
if item["task_id"] == description["uid"] + ":generation")
print(task["description"]["statement"]["text"])
模型请求只使用 Tasks/ 中的可见输入;评测器再按 tasks.json 读取隐藏的评分材料。prediction.json 与 execution.json 各约 1.2 GB,批量处理时推荐流式解析(如 ijson)。
Build model requests from Tasks/ only; the evaluator reads hidden grading material via tasks.json. prediction.json and execution.json are ~1.2 GB each — stream-parse them (e.g. with ijson) for batch runs.
description.json 中注明。Missing generators, validators, checkers, solutions and notes were produced by Astra; each program's origin (official, adapted, or newly written) is recorded in description.json.stdio_v2 协议,结果请注明 “OOD-Test 2026-11” 及资源设置,与官方 LiveCodeBench 分数分开报告。OOD-Test uses its own tests, program pools and the stdio_v2 protocol. Please report results as “OOD-Test 2026-11” with your resource settings, separately from official LiveCodeBench scores.@misc{ood_test_2026,
title = {OOD-Test: A Time-Held-Out Benchmark for Coding Agents},
author = {Senyang Su},
year = {2026},
howpublished = {\url{https://huggingface.co/datasets/Sy-SU/OOD-Test}}
}