pcntoolkit.util.evaluator

Classes

Evaluator

A class for evaluating normative model predictions.

Module Contents

class Evaluator

A class for evaluating normative model predictions.

This class implements various statistics to assess the quality of normative model predictions, including correlation coefficients, error metrics, and normality tests.

response_vars

List of response variables to evaluate

Type:

List[str]

Initialize the Evaluator.

create_statistics_group(data: pcntoolkit.dataio.norm_data.NormData) → None

Create a statistics group in the data container.

Parameters:

data (NormData) – Data container to add statistics group to

empty_statistic() → xarray.DataArray

evaluate(data: pcntoolkit.dataio.norm_data.NormData, statistics: List[str] = []) → pcntoolkit.dataio.norm_data.NormData

Evaluate model predictions using multiple statistics.

Parameters:

data (NormData) – Data container with predictions and actual values, and yhat

Returns:

Data container updated with evaluation statistics

Return type:

NormData

evaluate_R2(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate R2 for model predictions.

evaluate_bic(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Bayesian Information Criterion (BIC) for model predictions.

BIC is a criterion for model selection that statistics the trade-off between model fit and complexity.

Parameters:

data (NormData) – Data container with predictions and actual values. Must contain ‘y’ and ‘Yhat’ variables.

evaluate_expv(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Explained Variance score for model predictions.

The explained variance score statistics the proportion of variance in the target variable that is predictable from the input features.

Parameters:

data (NormData) – Data container with predictions and actual values. Must contain ‘y’ and ‘Yhat’ variables.

evaluate_kurt(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate the excess kurtosis of the z-score distribution.

Excess kurtosis measures how fat the tails of the z-score distribution are relative to a normal distribution. For a well-calibrated normative model the z-scores follow a standard normal distribution, so the ideal value is 0.

Positive values indicate fatter tails (more outliers). Negative values indicate thinner tails (less outliers).

Parameters:

data (NormData) – Data container with z-scores. Must contain the ‘Z’ variable.

evaluate_mace(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Mean Absolute Centile Error.

evaluate_mape(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Mean Absolute Percentage Error.

evaluate_mll(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Mean Log Loss (MLL) for model predictions.

MLL measures the probabilistic accuracy of the model’s predictions.

Note: In earlier PCNtoolkit releases, this metric was called NLL (Negative Log Likelihood). It is now named MLL to match the literature and avoid confusion with the different NLL used internally for BLR hyperparameter estimation.

Parameters:

data (NormData) – Data container with predictions and actual values. Must contain ‘logp’ values for the evaluated response variable.

evaluate_msll(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Mean Standardized Log Loss (MSLL) for model predictions.

MSLL compares the log loss of the model to that of a simple baseline predictor that always predicts the mean of the training data.

MSLL = MLL_model - MLL_baseline

Parameters:

data (NormData) – Data container with predictions and actual values. Must contain ‘y’, ‘Yhat’, and standard deviation predictions.

evaluate_rho(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Spearman’s rank correlation coefficient.

Parameters:

data (NormData) – Data container with predictions and actual values

evaluate_rmse(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Root Mean Square Error (RMSE) for model predictions.

Parameters:

data (NormData) – Data container with predictions and actual values. Must contain ‘y’ and ‘Yhat’ variables.

evaluate_shapiro_w(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Shapiro-Wilk test statistic for normality of residuals.

The Shapiro-Wilk test assesses whether the z-scores follow a normal distribution. A higher W statistic (closer to 1) indicates stronger normality.

Parameters:

data (NormData) – Data container with predictions and actual values. Must contain ‘zscores’ variable.

evaluate_skew(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate the skewness of the z-score distribution.

Skewness measures asymmetry of the z-score distribution. For a well-calibrated normative model the z-scores follow a standard normal distribution, so the ideal value is 0.

Positive values indicate a longer right tail. Negative values indicate a longer left tail.

Parameters:

data (NormData) – Data container with z-scores. Must contain the ‘Z’ variable.

evaluate_smse(data: pcntoolkit.dataio.norm_data.NormData) → None

Evaluate Standardized Mean Square Error (SMSE) for model predictions.

SMSE normalizes the mean squared error by the variance of the target variable, making it scale-independent.

Parameters:

data (NormData) – Data container with predictions and actual values. Must contain ‘y’ and ‘Yhat’ variables.

n_params() → int

Return the number of parameters in the model.

prepare(responsevar: str) → None

Prepare the evaluator for a specific response variable.

reset() → None

Reset the evaluator state.

response_vars: List[str] = []