Model comparisons
The first experiment compares team-only, player-only, and combined lineup forecasts on future ODI matches. Read the experiment specification for cutoffs, shrinkage, uncertainty, independent review, and interpretation limits. Rendering reads the saved publication and does not refit models.
Saved ODI forecast comparison
Evaluation years: 2023, 2024, 2025. Models are fitted separately by competition, with past-only tuning and a fixed forecast model within each evaluation year. Lower log loss and Brier scores are better. Each match has equal weight.
This is a retrospective pilot restricted to decisive matches with fixed XIs. It is not a published ranking of current player ability.
5-year window (primary)
| Competition | Model | Matches | Log loss | Brier |
|---|---|---|---|---|
| Men | 50:50 reference | 364 | 0.693 | 0.250 |
| Men | Team only | 364 | 0.689 | 0.247 |
| Men | Players only | 364 | 0.687 | 0.247 |
| Men | Teams + players | 364 | 0.686 | 0.246 |
| Women | 50:50 reference | 194 | 0.693 | 0.250 |
| Women | Team only | 194 | 0.639 | 0.225 |
| Women | Players only | 194 | 0.622 | 0.219 |
| Women | Teams + players | 194 | 0.622 | 0.219 |
Paired differences below are model minus team-only log loss. Negative differences favor the model containing player identities.
| Competition | Model | Difference | 95% block interval | Quarters |
|---|---|---|---|---|
| Women | Players only | -0.017 | [-0.044, +0.013] | 12 |
| Women | Teams + players | -0.016 | [-0.045, +0.015] | 12 |
| Men | Players only | -0.001 | [-0.042, +0.030] | 11 |
| Men | Teams + players | -0.003 | [-0.043, +0.030] | 11 |
3-year window (sensitivity)
| Competition | Model | Matches | Log loss | Brier |
|---|---|---|---|---|
| Men | 50:50 reference | 364 | 0.693 | 0.250 |
| Men | Team only | 364 | 0.695 | 0.250 |
| Men | Players only | 364 | 0.701 | 0.252 |
| Men | Teams + players | 364 | 0.701 | 0.253 |
| Women | 50:50 reference | 194 | 0.693 | 0.250 |
| Women | Team only | 194 | 0.667 | 0.237 |
| Women | Players only | 194 | 0.630 | 0.221 |
| Women | Teams + players | 194 | 0.629 | 0.221 |
Paired differences below are model minus team-only log loss. Negative differences favor the model containing player identities.
| Competition | Model | Difference | 95% block interval | Quarters |
|---|---|---|---|---|
| Women | Players only | -0.036 | [-0.066, -0.004] | 12 |
| Women | Teams + players | -0.037 | [-0.068, -0.004] | 12 |
| Men | Players only | +0.006 | [-0.026, +0.031] | 11 |
| Men | Teams + players | +0.006 | [-0.025, +0.032] | 11 |
Intervals resample quarters, retaining every match in each sampled quarter. Few blocks, dependence across quarters, and uncertainty from training and tuning limit their interpretation. They are conditional, exploratory summaries, not evidence that individual player coefficients are identified.
The team baseline omits venue and dynamic strength. Player identities can proxy those changes; the combined model also changes team shrinkage.
Download the saved experiment
- calibration.csv
- coefficients.csv
- comparisons.csv
- diagnostics.csv
- metrics.csv
- paired losses.csv
- predictions.csv
- preparation manifest.json
- tuning.csv
- Analysis provenance
- Publication checksums
The coefficient table contains static associations in log-odds units, with exposure and co-occurrence diagnostics. It contains no WAR, player confidence intervals, or current-strength ranks. The saved analysis manifest records splits, dependencies, code hashes, and the input snapshot.
Research sequence
| Stage | Purpose | Status |
|---|---|---|
| International match and participant preparation | Stable identities, dated outcomes, and substitution flags | Implemented |
| Regularized lineup win/loss model | Outcome-based player benchmark | Implemented; saved pilot above |
| Player-only rankings by format | Five-year outcome associations for Tests, ODIs, and T20Is | Published tables; exploratory |
| Ball-level runs and wickets | Contextual contribution estimates | Specification and estimation pending |
| Time-varying player strength | Current ability and future-match forecasts | Follows the initial benchmarks |
| Win probability added | Retrospective match impact | Follows a validated probability model |
| Replacement simulation | Counterfactual lineup selection | Deferred |
Outcome-based benchmark
The player-model predictor is the sum of team A’s player effects minus the sum of team B’s effects, plus context terms that reverse direction when the teams are swapped. Shrinkage is needed because teammates often appear together. A coefficient is a relative log-odds contribution, not a number of wins or a causal effect.
The first binary experiment uses decisive matches with unambiguous fixed XIs. Preparation retains other outcomes and substitution matches with explicit flags; later specifications may use them. Run margins and wicket margins are different quantities and must not be combined as a response. A successful chase also ends when the target is reached, truncating the observed scoring margin.
National-team strength and player strength are entangled. Team effects, time windows, and priors affect how shared performance is allocated. Before interpreting individual rankings, examine lineup overlap, weakly identified players, and sensitivity to those choices. More deliveries do not create more independent match outcomes for a lineup regression.
Ball-level valuation
The model must account for the future cost of losing a wicket, as well as immediate runs. One candidate values an event as runs scored plus the change in expected remaining innings runs. Chases require target-aware valuation. Dismissal types, extras, fielding credit, shortened matches, and exposure need explicit rules before estimation. These are research specifications, not assumptions hidden in a parser.
Evaluation and publication gates
- Use chronological future-match holdouts, tuning hyperparameters within earlier training periods. Keep all deliveries from a match in the same partition.
- Compare lineup forecasts against a team-only model on the same matches. Use dated
side_orderexports with documented match-ID mappings where applicable. - Report log loss, Brier score, calibration, and uncertainty in score differences; account for dependence within series.
- Test sensitivity to regularization, time windows, team effects, replacement definitions, and sample restrictions. Show rank uncertainty and exposure.
- Assess men’s and women’s ODIs separately. Prediction quality alone does not establish that individual credit has been allocated correctly.
- Check models on simulated data with known player effects, including players who always appear together. The model must expose weak identification.
Time-varying strength becomes a published rating only after comparison with the static and team-only benchmarks. A contribution measure becomes WAR only after replacement and wins calibration are defended. Replacement probability differences need not sum to the team’s wins in a nonlinear model.