Compare assessment structures
Source:vignettes/articles/compare-assessments.Rmd
compare-assessments.RmdAn age-structured assessment combines catch, survey indices and biology to estimate abundance, recruitment and mortality. Different assumptions can explain the same observations differently. This guide uses the Northern cod example to compare three assessment structures, then inspect their diagnostics and forecasting skill.
The models match the cheatsheet. They are SCAA-, SAM- and NCAM-like examples, rather than reproductions of complete official assessments or recommended settings for another stock. For fleet catches and multiple surveys, continue with the simulated example.
Start with tidy inputs
dat <- cod_obs
print(dat, n = 2)
#>
#> ── tinyAM data ─────────────────────────────────────────────────────────────────
#>
#> ── catch: 1512 rows, 6 columns ──
#>
#> Years 1954–2025; ages 0–20
#> year age obs unit F_y_block F_a_block
#> 1 1954 0 NA thousand fish 1954-1991 0-1
#> 2 1955 0 NA thousand fish 1954-1991 0-1
#> ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
#> 1511 2024 20 0 thousand fish 2023-2025 18-20
#> 1512 2025 20 0 thousand fish 2023-2025 18-20
#>
#> ── index: 1512 rows, 7 columns ──
#>
#> Years 1954–2025; ages 0–20
#> year age obs survey samp_time q_block unit
#> 1 1954 0 NA Fall 2J3KL 0.8 0 thousand fish
#> 2 1954 1 NA Fall 2J3KL 0.8 1 thousand fish
#> ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮
#> 1511 2025 19 0 Fall 2J3KL 0.8 6-20 thousand fish
#> 1512 2025 20 0 Fall 2J3KL 0.8 6-20 thousand fish
#>
#> ── weight: 1512 rows, 4 columns ──
#>
#> Years 1954–2025; ages 0–20
#> year age obs unit
#> 1 1954 0 2.965967e-04 kg
#> 24 1955 0 2.965967e-04 kg
#> ⋮ ⋮ ⋮ ⋮ ⋮
#> 1631 2024 20 1.409146e+01 kg
#> 1654 2025 20 1.369174e+01 kg
#>
#> ── maturity: 1512 rows, 3 columns ──
#>
#> Years 1954–2025; ages 0–20
#> year age obs
#> 1 1954 0 0
#> 2 1955 0 0
#> ⋮ ⋮ ⋮ ⋮
#> 1711 2024 20 1
#> 1712 2025 20 1
#>
#> ── pop_covariates: 1512 rows, 4 columns ──
#>
#> Years 1954–2025; ages 0–20
#> year age allometric_M_assumption M_a_block
#> 1 1954 0 4.9990199 0-1
#> 2 1955 0 4.9990199 0-1
#> ⋮ ⋮ ⋮ ⋮ ⋮
#> 1511 2024 20 0.1872682 8-20
#> 1512 2025 20 0.1889190 8-20
check_tam_data(dat)
#> ✔ Input data passed validation.
#> ℹ Years 1954-2025; ages 0-20; fleets: 1; surveys: 1.Arrange your own data in the same four required tables. Catch and index contain observations; weight and maturity contain biological inputs. Each index row identifies its survey and sampling time as a fraction of the year. Optional population covariates explain recruitment, cohort survival or natural mortality. Catch-table covariates explain fishing mortality, while index-table covariates explain catchability and index precision.
The example catches and indices use thousand fish, and weights use
kg, giving biomass in tonnes. Unit columns describe supplied values;
they do not convert them. Keep unavailable observations as
NA. Genuine catch/index zeros remain in the inputs but are
excluded from the current lognormal likelihood. Biological inputs must
be complete. Read input
requirements before fitting, and the data help for provenance.
Specify three structures
First fit a statistical catch-at-age model. Annual recruitment is estimated as fixed effects, cohort survival follows the mortality calculation exactly, and F is separable into annual and age effects. Natural mortality is supplied as 0.3 per year.
scaa <- fit_tam(
dat, years = 1983:2025, ages = 2:14,
N_settings = list(process = "off", init = "exp"),
rec_settings = list(process = "off", mu_form = ~ factor(year)),
F_settings = list(process = "off",
mu_form = ~ factor(year) + factor(age), mean_ages = 5:14),
M_settings = list(process = "off",
mu_supplied = ~ I(0.3), mean_ages = 5:14),
catch_settings = list(sd_form = ~ 1),
index_settings = list(sd_form = ~ 1, q_form = ~ mono(q_block))
)Age 14 is the modeled plus group: it includes older fish retained by
the model’s biological calculations. mean_ages defines the
ages used to report average mortality. The catchability formula allows
non-decreasing steps among q_block groups under the default
log link.
Now allow cohort-survival deviations, random-walk recruitment and correlated annual F increments. This SAM-like model explains additional variation through population processes while retaining the same observations and fixed M.
sam <- update(scaa,
N_settings = list(process = "iid"),
rec_settings = list(process = "rw", mu_form = NULL),
F_settings = list(process = "cor_rw", mu_form = NULL)
)The NCAM-like alternative keeps exact cohort survival but allows temporally correlated F and M. Ages within each M block share actual M.
ncam <- update(scaa,
rec_settings = list(process = "rw", mu_form = NULL),
F_settings = list(process = "ar1", mu_form = ~ 1),
M_settings = list(process = "ar1", age_breaks = c(2, 5, 8, 14))
)Partial updates retain omitted settings. These examples differ in several deliberate assumptions; a focused investigation would also compare intermediate models that change one assumption at a time. The population equations describe process and initial-state conventions.
Inspect diagnostics before trends
check_tam(scaa, sam, ncam)
#>
#> ── scaa ────────────────────────────────────────────────────────────────────────
#> ✔ Numerical convergence: passed
#> ✔ Optimizer: 0 (relative convergence (4))
#> ✔ Max |gradient|: 0.00916 adjusted; 46.7 raw (tolerance 0.01)
#> ✔ Positive-definite Hessian: yes
#> ✔ Model structure: no redundant model terms detected
#>
#> Advisory findings (5):
#> ℹ 3 parameter(s) at a bound; symmetric uncertainty intervals may be
#> inappropriate.
#> ℹ Some age/series groups have mean standardized residuals outside -0.5 to 0.5.
#> ℹ Some age/series groups have standardized residual SDs outside 0.5 to 1.5;
#> fitted-state shrinkage also affects spread.
#> ℹ Some standardized residuals exceed three SDs in absolute value.
#> ℹ Some age/series groups have absolute adjacent-year residual correlation above
#> 0.5.
#>
#>
#> ── sam ─────────────────────────────────────────────────────────────────────────
#>
#> ✔ Numerical convergence: passed
#> ✔ Optimizer: 0 (relative convergence (4))
#> ✔ Max |gradient|: 4.62e-05 adjusted; 29.9 raw (tolerance 0.01)
#> ✔ Positive-definite Hessian: yes
#> ✔ Model structure: no redundant model terms detected
#>
#> Advisory findings (4):
#> ℹ 3 parameter(s) at a bound; symmetric uncertainty intervals may be
#> inappropriate.
#> ℹ Some age/series groups have standardized residual SDs outside 0.5 to 1.5;
#> fitted-state shrinkage also affects spread.
#> ℹ Some standardized residuals exceed three SDs in absolute value.
#> ℹ Some age/series groups have absolute adjacent-year residual correlation above
#> 0.5.
#>
#>
#> ── ncam ────────────────────────────────────────────────────────────────────────
#>
#> ✔ Numerical convergence: passed
#> ✔ Optimizer: 0 (relative convergence (4))
#> ✔ Max |gradient|: 0.000189 adjusted; 41 raw (tolerance 0.01)
#> ✔ Positive-definite Hessian: yes
#> ✔ Model structure: no redundant model terms detected
#>
#> Advisory findings (6):
#> ℹ 3 parameter(s) at a bound; symmetric uncertainty intervals may be
#> inappropriate.
#> ℹ One or more AR1 correlations exceed 0.98; examine the mean and temporal
#> process together.
#> ℹ Some age/series groups have mean standardized residuals outside -0.5 to 0.5.
#> ℹ Some age/series groups have standardized residual SDs outside 0.5 to 1.5;
#> fitted-state shrinkage also affects spread.
#> ℹ Some standardized residuals exceed three SDs in absolute value.
#> ℹ Some age/series groups have absolute adjacent-year residual correlation above
#> 0.5.
plot_model(scaa, sam, ncam)Initial states
Boundary parameters
N0: exp; R0: free
Recruitment
New youngest-age fish
Deterministic: process off
Mean ~factor(year)
Fishing mortality
Total F = sum of fleet F
Deterministic: process off
Mean ~factor(year) + factor(age)
Natural mortality
Shared population M
Deterministic: process off
Baseline ~I(0.3)
Population abundance
Survivors + recruitment
Deterministic: process off
Mean: survivors
Catch observations
Fleet catch predictions
SD ~1
Survey observations
q, sampling time and abundance
q ~mono(q_block)
Spawning biomass
Mature stock biomass
SSB = sum of mature biomass
- Initial states to Population abundance
- Recruitment to Population abundance
- Fishing mortality to Population abundance
- Natural mortality to Population abundance
- Population abundance to Survey observations
- Population abundance to Catch observations
- Fishing mortality to Catch observations
- Population abundance to Spawning biomass
Navy: annual stochastic process. Orange: no annual process deviations. Dashed arrows: active input-table predictors. Process off retains the biological component. Initial parameters and observations are separate.
Arrows show biological and observation relationships, not a computation order.
Formulas and settings
Initial states
Older-age initialization: exp
Recruitment boundary: free
Initial year: 1983
Initial parameters are separate from annual process deviations.
Equations and settingsRecruitment
Deterministic (process off)
Mean: ~factor(year)
Covariates: pop_covariates.
Equations and settingsAbundance and cohort survival
Deterministic (process off)
Mean: None
Modeled ages: 2 to 14 ; oldest age is the plus group.
Apparent gains/losses do not identify migration separately from mortality.
Equations and settingsFishing mortality
Deterministic (process off)
Mean: ~factor(year) + factor(age)
Fleets: Aggregate fishery
Age blocks: 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14
Total F is the sum of fleet F. Shared age blocks share actual F.
Equations and settingsNatural mortality
Deterministic (process off)
Supplied mean: ~I(0.3)
Mean effects: None
Age blocks: 3-14
Shared age blocks share actual M; other cells retain their mean.
Equations and settingsCatch observations
Lognormal numbers at age, conditional on population and F.
Observation SD: ~1
Supplied SD: None
Zero and missing observations are excluded from the lognormal likelihood.
Equations and settingsSurvey indices and catchability
Surveys: Fall 2J3KL
q formula: ~mono(q_block)
q link: log
Gaussian q terms: None
Observation SD: ~1
Supplied SD: None
Predictions use each survey's sampling time; q may contain Gaussian effects.
Equations and settingsSpawning biomass and projections
SSB uses stock weights, maturity and spawning timing, retaining plus-group biology.
Projections: none.
Equations and settingscheck_tam() reports numerical checks separately from
statistical advisories. Convergence does not establish that the
observations distinguish F, M and cohort variation. Warnings and failed
numerical checks are reasons to investigate, not an invitation to rank
the models immediately.
vis_tam(scaa, sam, ncam, open_file = FALSE)Explore the example dashboard to inspect these three models without fitting them first.
When supplied, Background is the first page, followed by Model structure and its Function calls tab. Without Background, Model structure opens first. Start your diagnostic review in Fisheries → Aggregate fishery and Surveys: inspect observed/predicted trajectories and residual patterns across age and year. Retained OSA residuals are displayed when available; otherwise the plots start with standardized residuals. Rendering never calculates OSAs implicitly.
Next inspect Parameters and Tables → Diagnostics for uncertainty and process behavior. Then use Trends to compare abundance and mortality. The pattern tabs show age–year structure; they can reveal differences hidden by annual totals. Standardized residuals condition on fitted states and are affected by shrinkage.
Public plotting functions reproduce these views outside the dashboard:
ssb <- get_ssb(SCAA = scaa, SAM = sam, NCAM = ncam)
plot_trend(ssb, color = ~model, colors = tam_pal(3),
ylab = "SSB (tonnes)")Check historical stability and prediction
First examine historical stability by refitting the NCAM-like model after successively withholding five terminal years. This example runs sequentially; the evaluation functions respect the caller’s future plan.
future::plan(future::sequential)
retro <- fit_retro(ncam, folds = 5)
retro
#>
#> ── Retrospectives ──────────────────────────────────────────────────────────────
#> model rho_ssb rho_recruitment rho_F_bar scored_folds failed_folds
#> ncam 0.0337 0.1281 0.02223 0 0
ssb_retro <- retro$pop$ssb
ssb_retro$terminal_year <- factor(ssb_retro$fold,
levels = sort(unique(ssb_retro$fold), decreasing = TRUE))
plot_trend(ssb_retro, color = ~terminal_year,
colors = tam_pal(nlevels(ssb_retro$terminal_year)), add_intervals = FALSE,
ylab = "SSB (tonnes)")The lines show the reference and successful peels, labeled by
terminal year. Check retro$fold_diagnostics for
unsuccessful attempts. This is a view of retrospective stability, not a
measure of SSB forecast accuracy. This run retained 5 of 5 peels.
scored_folds is zero for historical retrospectives because
only hindcasts produce forecast scores.
Hindcasts additionally predict excluded observations. The following ten-fold example is supplied for you to run; it is not executed when building this guide.
hind <- fit_hindcast(scaa, sam, ncam, folds = 10)
hind
hind$fold_diagnostics
vis_tam(hind, model = "scaa", open_file = FALSE)The printed summary includes Mohn’s rho, survey log-RMSE, MASE and scored/failed attempt counts. Rho describes retrospective change; values near zero indicate greater historical stability. Lower log-RMSE indicates smaller log-scale forecast errors. MASE below one beats keeping the last observed index value unchanged.
Default forecasts condition on the actual aggregate catch in the
withheld year. Survey observations and catch compositions remain held
out. Agreement with the imposed catch total therefore does not measure
forecasting skill. Use proj_settings = list(F_mult = 1) for
a status-quo-F alternative.
Inspect failures and compare validation coverage before interpreting score differences. Failed numerical fits are retained in diagnostics and excluded from scores; unequal coverage is not a fair ranking. These checks examine different aspects of model performance. See hindcast definitions for detailed scores and caller-controlled parallel execution.
For further comparisons, AIC and cAIC have additional assumptions, especially for intrinsic random walks and unpenalized initial states. They complement diagnostic and predictive checks.