Context-rich tasks. State-adaptive interaction.

CastWorld three-color globe and forecasting signal

CastWorld

Benchmarking Context-Aware Agentic
Time Series Forecasting Beyond Predictive Accuracy

Evaluate how forecasting agents acquire evidence, construct and revise numerical forecasts, select final predictions, and use resources under shared task conditions.

Datasets
25
Domains
8
Core tools
49
Evaluation dimensions
4

Context-aware agentic forecasting

CastWorld Leaderboard

A / Public: 300 episodes. B / Hidden: 100 episodes. Forecast quality, process execution, behavioral trustworthiness, and resource efficiency.

The benchmark

Forecasting is a sequence of decisions.

A forecasting agent must decide what information it needs, which tools to use, whether to revise a prediction, and when to submit. CastWorld makes these decisions observable in a shared, context-aware environment.

CastWorld combines context-rich datasets, a state-adaptive forecasting harness and dual-level evaluation.
The CastWorld environment: context-rich tasks, a forecasting-native harness, and decision-centric evaluation.
01 / DATA

A world of forecasting contexts

25 real-world datasets connect target histories with external variables, events, domain knowledge, and historical cases.

02 / HARNESS

Evidence and forecasts co-evolve

New evidence can guide a revision. An intermediate forecast can reveal what to investigate next. The agent chooses how to proceed.

03 / EVALUATION

Follow decisions through to delivery

Model-level evaluation is the primary focus, assessing decisions under a common harness. Systems-level evaluation compares predictive accuracy across methods.

Benchmark results

Leaderboards

Evaluation protocol

Evaluation cohort

Evaluation level

Loading results

CastWorld evaluation results

Loading
Displayed metrics
Metric columns
CastWorld leaderboard

Beyond the final curve

Four complementary views

Total = 0.60 F + 0.20 P + 0.15 T + 0.05 E

60%

Forecast quality

Training-scaled predictive errors, with declared domain metrics where applicable.

20%

Process execution

Recorded acquisition, revision, checking, and selection among created forecasts.

15%

Behavioral trustworthiness

Permission compliance, reference integrity, and agreement between declared and executed changes.

5%

Resource efficiency

Time, tokens, and redundant behavior, conditioned on predictive adequacy.

The forecasting environment

A state-adaptive loop

Framework specification

Evidence can change a forecast. A forecast can reveal what to investigate next. CastWorld records acquisition, forecast, and execution state while the agent chooses its next action.

Acquisition state

What is known?

Information needs, acquired observations, and evidence with sources and time scopes.

Forecast state

What has been predicted?

Immutable forecast versions, the active prediction, and public check results.

Execution state

What can happen next?

Actions, errors and feedback, remaining budget, and the record of earlier decisions.

Choose the next action

Continue acquiring information, create or revise a forecast, inspect a version, return to an earlier prediction, or submit. The shared state informs the choice; the environment does not prescribe a single sequence.

Keep the numerical history

Each accepted construction becomes an immutable forecast version. Numerical changes, supporting observations, checks, and final selection remain connected in the execution record.

Evaluate after delivery

An independent scorer uses held-out targets and the recorded trajectory. Public checks guide execution without revealing the current task's future prediction error.

Forecasting-native capabilities

A shared toolkit, with domain-specific extensions

TimeSense

Time series diagnostics.

Understand the current situation

TimeFeat

Statistical and temporal feature extraction.

Quantify patterns and changes

TimeZoo

Chronos, TimesFM, Sundial, and Moirai forecasts.

Form numerical candidates

TimeKnowledge

Domain knowledge and historical analogue cases.

Ground a forecasting judgment

The registry provides 49 core tool entries, plus domain-specific diagnostics. Shared forecast operations support construction, fusion, revision, checks, and final submission.

Real-world forecasting

25 datasets across 8 domains

Dataset & field guide

The paper studies 1,310 episodes, including a shared 250-episode comparison of 15 LLMs. The public release provides separate A/B cohorts from the same 25 datasets: 300 episodes with public labels on A and 100 with organizer-held labels on B. Each cohort has independent Model and System boards awaiting evaluation.

Published CastWorld temporal and contextual dataset profiles across eight application domains.
Temporal and contextual diversity across the dataset suite.

A common context specification

Modality, temporality, semantics, and availability describe complementary attributes of context. Task packages organize numerical covariates, event timelines, text knowledge, and historical cases, with explicit field and time permissions.

Dataset domains, sampling frequencies and forecast windows
Dataset Domain Sampling History Forecast horizon

History and forecast lengths are counts of observations at the listed sampling frequency. Raw CSV and JSON tables.

Run CastWorld

From tasks to trajectories

Use the released A/B task packages, common harness, and evaluation code.

Complete usage guide
Offline smoke test
git clone https://github.com/CastWorld-Bench/CastWorld.git
cd CastWorld
python -m pip install -e .
python -m castbench.cli demo --out runs/demo