OOD-Test

LiveCodeBench-Latest · Release 2026-11

OOD-Test

面向 Coding Agent 的时间留出评测集:收录 2026 年 6 月至 11 月 Codeforces 与 AtCoder 比赛中的 456 道标准输入输出题,每题附完整测试、checker、程序池和五类任务定义。 A time-held-out benchmark for coding agents: 456 standard-I/O problems from Codeforces and AtCoder contests held between June and November 2026, each shipped with full tests, a checker, a program pool, and five task definitions.

456
题目(CF 221 / AtCoder 235)problems (CF 221 / AtCoder 235)
5,520
完整测试输入full test inputs
916
额外参考 / 错误程序extra reference / buggy programs
4,084
任务实例,覆盖 5 类能力task instances across 5 abilities

数据构成Composition

收录 2026-06-01 至 2026-11-16 期间已结束比赛中的全部非交互标准输入输出题,覆盖所有难度;题目实际发布日期为 2026-06-06 至 2026-11-15。 All non-interactive standard-I/O problems from contests that finished between 2026-06-01 and 2026-11-16, across every difficulty level. Actual release dates span 2026-06-06 to 2026-11-15.

平台Platform比赛类型Contest type题数Problems
CodeforcesDiv. 142
CodeforcesDiv. 2104
CodeforcesDiv. 1 + 235
CodeforcesEducational40
AtCoderABC167
AtCoderARC61
AtCoderAGC7
合计Total456

按发布月份Problems by release month

11 月仅统计至 11-15November covers releases through Nov 15

AtCoderCodeforces
表格视图Table view

Codeforces 难度分布Codeforces rating distribution

182 道有官方 rating;另有 39 道暂无 rating182 problems with an official rating; 39 have none yet

表格视图Table view

高频标签(前 12)Most frequent tags (top 12)

全部 456 题;标签来自平台或由 Astra 补充,来源见 tag_detailsAll 456 problems; tags come from the platform or are added by Astra — see tag_details

表格视图Table view
  • 共 473 道候选题,其中 16 道交互题与 1 道 output-only 题(ARC226E)未收录。Of 473 candidates, 16 interactive problems and 1 output-only problem (ARC226E) are excluded.
  • Easy / Hard 版本各自拥有独立 uid,并通过 family_id 关联。Easy / Hard versions keep separate uids and are linked by family_id.
  • “OOD” 指按题目发布时间留出。建议在报告结果时同时注明模型版本与已知训练截止日期。“OOD” means a temporal hold-out by release date. When reporting results, please include the model version and its known training cutoff.

五类任务Five tasks

能力划分参考 LiveCodeBench。Tasks/ 中是按可见性规则导出的模型输入;评分所需的隐藏材料保留在各题包中,通过 task_id 对应。 The ability split follows LiveCodeBench. Tasks/ holds model-facing inputs exported under visibility rules; hidden grading material stays in each problem package and is joined via task_id.

Generation

456
输入Sees
题面、约束、I/O 约定、限制、公开样例Statement, constraints, I/O spec, limits, public samples
输出Produces
解题程序A solution program
指标Metric
pass@1(可选 pass@k):通过 score_case_ids 全部测试pass@1 (optionally pass@k): passes every test in score_case_ids
Tasks/generation.json

Repair

456
输入Sees
题面、错误程序、失败反馈(verdict 与程序实际输出)Statement, a buggy program, failure feedback (verdict and actual output)
输出Produces
修复后的程序A fixed program
指标Metric
修复成功率;反馈测试与评分测试相互独立Repair success rate; feedback and scoring tests are disjoint
Tasks/repair.json

Prediction

1,358
输入Sees
题面与一份指定输入Statement and one given input
输出Produces
符合题意的输出A correct output for that input
指标Metric
macro / micro accuracy,由题目 checker 判定Macro / micro accuracy, judged by the problem's checker
Tasks/prediction.json · ~1.2 GB

Execution

1,358
输入Sees
一段具体程序、指定输入与运行环境A concrete program, a given input, and the runtime
输出Produces
该程序的实际 stdoutThat program's actual stdout
指标Metric
accuracy:按空白分隔的 token 序列精确匹配Accuracy: exact match on whitespace-separated tokens
Tasks/execution.json · ~1.2 GB

Testcase Gen

456
输入Sees
题面、目标程序、公开样例、生成预算Statement, target program, public samples, a generation budget
输出Produces
测试输入,或输出测试的 generatorTest inputs, or a generator that emits them
指标Metric
输入合法率、错误检出率、误杀率Input validity, bug detection rate, false-kill rate
Tasks/testcase_gen.json

评价指标Metrics

Generation — 每题 n 次独立采样、c 次通过:with n independent samples per problem and c passing:

pass@k = mean_over_problems(1 − C(n−c, k) / C(n, k)),  n ≥ k

Repair — 修复后通过全部评分测试的实例数 / 初始确实失败的实例数。多轮修复需固定轮数或成本预算。instances passing all scoring tests after repair / instances that initially fail. Multi-round repair should fix a round or cost budget.

Prediction — 每题最多 3 份非样例输入,按 token 序列去重;构造题接受任意合法答案。up to 3 non-sample inputs per problem, deduplicated by token sequence; any valid answer is accepted for constructive problems.

Execution — 真值是指定程序的实际 stdout;1 与 1.0 不等价。ground truth is the program's actual stdout; 1 and 1.0 are different.

Testcase Gen — 输入先过 validator,再运行标准解、目标程序和 checker。预算:每实例最多 10 份输入、合计 16 MiB、generator 超时 20 s。inputs must pass the validator, then std, the target program and the checker are run. Budget: ≤ 10 inputs, 16 MiB total, 20 s generator timeout per instance.

数据格式Data format

数据采用 format 2.0,任务协议为 stdio_v2。每道题一个目录,目录名形如 atcoder_abc461_a,对应 UID atcoder:abc461_a。 Format 2.0, task protocol stdio_v2. One directory per problem, e.g. atcoder_abc461_a for UID atcoder:abc461_a.

OOD-Test/
├── README.md
├── index.html
├── Tasks/
│   ├── generation.json
│   ├── repair.json
│   ├── prediction.json
│   ├── execution.json
│   └── testcase_gen.json
└── problems/
    └── <platform>_<problem_id>/
        ├── description.json
        ├── cases.json
        ├── tasks.json
        ├── expected_verdict.json
        ├── generator.cpp
        ├── validator.cpp
        ├── std.cpp
        ├── checker.cpp
        └── solutions/
            └── *.cpp / *.py
文件File内容Contents
description.json题面、约束、tag、rating、题解、限制、运行环境与来源Statement, constraints, tags, rating, editorial, limits, runtime, provenance
cases.json完整 stdin 与标准程序的实际输出Full stdin strings and the reference program's stdout
tasks.json五类任务的可见性配置与评分引用Visibility and scoring config for all five tasks
expected_verdict.json每个程序在每个测试上的预期 verdictExpected verdict of every program on every test
generator.cpp以 seed 为参数输出一份完整输入Takes a seed, prints one full input
validator.cpp读 stdin,退出码 0 表示输入合法Reads stdin; exit code 0 means valid
std.cppC++17 标准解C++17 reference solution
checker.cppchecker input output answer;退出码 0 表示接受; exit code 0 means accepted
solutions/额外的正确或错误程序(C++ / Python)Extra correct or buggy programs (C++ / Python)

cases.json 示例A cases.json record

{
  "case_id": "t0000",
  "input": "4 5\n",
  "expected_output": "Yes\n",
  "kind": "sample",
  "coverage": "official sample",
  "source": {
    "origin": "source_snapshot",
    "url": "https://atcoder.jp/contests/abc461/tasks/abc461_a",
    "upstream_case_id": "t0000"
  }
}

一份测试输入是一次完整的 stdin,可包含多个题内 test case。g0000–g0009 由 generator.cpp 以 seed 0–9 生成。构造题的保存输出可能与官网样例不同,判题请使用本题 checker.cpp。 One test input is one complete stdin and may contain several in-problem test cases. g0000–g0009 come from generator.cpp with seeds 0–9. For constructive problems the stored output may differ from the official sample, so always judge with the problem's checker.cpp.

可见性规则Visibility rules

default_visibility→tasks.<task>.visibility→case_overrides

后者覆盖前者,未开放的字段默认隐藏。题解、tag、标准解、其他程序和目标输出均可分别控制。 Later levels override earlier ones; anything not opened stays hidden. Editorials, tags, the reference solution, other programs and target outputs are controlled independently.

执行环境Execution environment

原题限制与数据集执行限制分别存于 limits.original 与 limits.dataset_execution。时限只对较慢的机器放宽,上限 20 秒: Original and dataset limits are stored in limits.original and limits.dataset_execution. Time limits are only relaxed on slower machines, capped at 20 s:

multiplier      = max(1, 4e8 / measured_rate)
timeout_seconds = min(20, original_time_seconds × multiplier)

本批在 Intel Core Ultra 7 265K 上测得约 1.1732e9 次/秒,倍率为 1,即直接使用原题时限。 This release was built on an Intel Core Ultra 7 265K at ≈ 1.1732e9 iterations/s, so the multiplier is 1 and original time limits apply as-is.

C++ 标准与编译参数C++ standard & flagsC++17, -O2 -pipe
解答程序内存Solution memorymax(512 MiB, original),按进程虚拟地址空间, per-process virtual address space
单个输出文件上限Output file limit64 MiB
编译超时Compile timeout90 s
generator / validator / checker 超时generator / validator / checker timeout各 20 s20 s each
判题环境Judgeupsolve-judge:1,无网络 Docker,每 worker 1 CPU, network-less Docker, 1 CPU per worker
testlibb0f7477,编译时 -I third_party/testlib, compile with -I third_party/testlib

快速开始Quick start

下载数据集,并用 Python 标准库读取一道题及其 Generation 输入。 Download the dataset and read one problem plus its Generation input with the Python standard library.

hf download Sy-SU/OOD-Test --repo-type dataset --local-dir OOD-Test
import json
from pathlib import Path

root = Path("OOD-Test")
folder = root / "problems" / "atcoder_abc461_a"

def read_json(path):
    with path.open(encoding="utf-8") as stream:
        return json.load(stream)

description = read_json(folder / "description.json")
cases = read_json(folder / "cases.json")["cases"]
tasks = read_json(folder / "tasks.json")["tasks"]

print(description["uid"], description["title"])
print("test inputs:", len(cases))
print("generation scoring tests:", tasks["generation"]["instances"][0]["score_case_ids"])

model_inputs = read_json(root / "Tasks" / "generation.json")
task = next(item for item in model_inputs
            if item["task_id"] == description["uid"] + ":generation")
print(task["description"]["statement"]["text"])

模型请求只使用 Tasks/ 中的可见输入;评测器再按 tasks.json 读取隐藏的评分材料。prediction.json 与 execution.json 各约 1.2 GB,批量处理时推荐流式解析(如 ijson)。 Build model requests from Tasks/ only; the evaluator reads hidden grading material via tasks.json. prediction.json and execution.json are ~1.2 GB each — stream-parse them (e.g. with ijson) for batch runs.

来源与许可Sources & license

  • 题面与题解来自 Codeforces / AtCoder 官方页面及公开数据,部分题面语料来自 CPRet 固定快照。Statements and editorials come from official Codeforces / AtCoder pages and public data, including a pinned CPRet snapshot.
  • 缺少的 generator、validator、checker、解答与说明由 Astra 补充;每个程序的来源(官方、移植改写、新编写)都在 description.json 中注明。Missing generators, validators, checkers, solutions and notes were produced by Astra; each program's origin (official, adapted, or newly written) is recorded in description.json.
  • 构建时检查了原题样例、生成输入的合法性、独立参考解的一致性、checker 的正负例以及错误程序的判定结果。The build verified official samples, generated-input validity, agreement between independent reference solutions, checker positive/negative cases, and verdicts of buggy programs.
  • 题面版权归原平台及作者所有;本数据集新增的测试、程序和标注仅供研究使用。Problem statements remain the property of their platforms and authors; the tests, programs and annotations added here are released for research use.
  • 本数据集使用自定义测试、程序池与 stdio_v2 协议,结果请注明 “OOD-Test 2026-11” 及资源设置,与官方 LiveCodeBench 分数分开报告。OOD-Test uses its own tests, program pools and the stdio_v2 protocol. Please report results as “OOD-Test 2026-11” with your resource settings, separately from official LiveCodeBench scores.

引用Citation

@misc{ood_test_2026,
  title        = {OOD-Test: A Time-Held-Out Benchmark for Coding Agents},
  author       = {Senyang Su},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/datasets/Sy-SU/OOD-Test}}
}