pcntoolkit.normative_model#

Module providing the NormativeModel class, which is the main class for building and using normative models.

Classes#

NormativeModel

This class provides the foundation for building normative models, handling multiple

Module Contents#

class NormativeModel(template_regression_model: pcntoolkit.regression_model.regression_model.RegressionModel, savemodel: bool = True, evaluate_model: bool = True, saveresults: bool = True, saveplots: bool = True, save_dir: str | None = None, inscaler: str = 'standardize', outscaler: str = 'standardize', y_transform: str | None = None, name: str | None = None)#

This class provides the foundation for building normative models, handling multiple response variables through separate regression models. It manages data preprocessing, model fitting, prediction, and evaluation.

Parameters:

  • template_reg_model (RegressionModel) – Regression model used as a template to create all regression models.

  • savemodel (bool) – Whether to save the model.

  • evaluate_model (bool) – Whether to evaluate the model.

  • saveresults (bool) – Whether to save the results.

  • saveplots (bool) – Whether to save the plots.

  • save_dir (str) – Directory to save the model, results, and plots.

  • inscaler (str) – Input (X/covariates) scaler to use.

  • outscaler (str) – Output (Y/response_vars) scaler to use.

  • y_transform (str or None) – Optional transform applied to Y before fitting and inverted after prediction. Currently supported: - "log1p" applies log(Y+1) - "log" applies natural log(Y) This is useful for phenotypes that cannot be negative. Default is None (no transform).

  • name (str) – Name of the model

__getitem__(key: str) → pcntoolkit.regression_model.regression_model.RegressionModel#
__setitem__(key: str, value: pcntoolkit.regression_model.regression_model.RegressionModel) → None#
check_compatibility(data: pcntoolkit.dataio.norm_data.NormData) → bool#

Check if the data is compatible with the model.

Parameters:

data (NormData) – Data to check compatibility with.

Returns:

True if compatible, False otherwise

Return type:

bool

compute_baseline_logp(data: pcntoolkit.dataio.norm_data.NormData) → pcntoolkit.dataio.norm_data.NormData#

Computes the log-probability of the data under a simple Gaussian model.

The baseline model is a Gaussian with mean and standard deviation computed from the scaled Y data. This serves as a baseline model to evaluate for example the MSLL (Mean Standardized Log Loss) of our fitted model.

Parameters:

data (NormData) – Test data containing response variables (Y).

Returns:

Data with baseline_logp computed for each response variable.

Return type:

NormData

compute_centiles(data: pcntoolkit.dataio.norm_data.NormData, centiles: List[float] | numpy.ndarray | None = None, **kwargs) → pcntoolkit.dataio.norm_data.NormData#

Computes the centiles for each response variable in the data.

Parameters:

  • data (NormData) – Test data containing covariates (X) for which to generate predictions, batch effects (batch_effects), and response variables (Y).

  • centiles (np.ndarray, optional) – The centiles to compute. Defaults to [0.05, 0.25, 0.5, 0.75, 0.95].

Returns:

Prediction results containing: - Centiles: centiles of the response variables

Return type:

NormData

compute_correlation_matrix(data, bandwidth=5, covariate='age')#
compute_logp(data: pcntoolkit.dataio.norm_data.NormData) → pcntoolkit.dataio.norm_data.NormData#

Computes the log-probability of the data under the fitted model.

Parameters:

data (NormData) – Test data containing covariates (X) for which to generate predictions, batch effects (batch_effects), and response variables (Y).

Returns:

Prediction results containing: - logp: log-probability of the response variables per datapoint under the fitted model

Return type:

NormData

compute_thrivelines(data: pcntoolkit.dataio.norm_data.NormData, span: int = 5, step: int = 1, z_thrive: float = 0.0, covariate='age', **kwargs) → pcntoolkit.dataio.norm_data.NormData#

Computes the thrivelines for each responsevar in the data

compute_yhat(data: pcntoolkit.dataio.norm_data.NormData) → pcntoolkit.dataio.norm_data.NormData#

Computes the predicted values for each response variable in the data.

compute_zscores(data: pcntoolkit.dataio.norm_data.NormData) → pcntoolkit.dataio.norm_data.NormData#

Computes Z-scores for each response variable using fitted regression models.

Parameters:

data (NormData) – Test data containing covariates (X) for which to generate predictions, batch effects (batch_effects), and response variables (Y).

Returns:

Prediction results containing: - Zscores: z-scores of the response variables

Return type:

NormData

static elemwise_logp_baseline_model(y_scaled: numpy.ndarray) → numpy.ndarray#

Compute log-probability for each observation under a baseline Gaussian model.

Parameters:

y_scaled (np.ndarray) – Scaled response variable values.

Returns:

Log-probability

Return type:

np.ndarray

evaluate(data: pcntoolkit.dataio.norm_data.NormData) → None#

Evaluates the model performance on the data. This method performs the following steps: 1. Preprocesses the data

  1. Evaluates the model performance

  2. Postprocesses the data

extend(data: pcntoolkit.dataio.norm_data.NormData, save_dir: str | None = None, n_synth_samples: int | None = None) → NormativeModel#

Extends the model to a new dataset.

extend_predict(extend_data: pcntoolkit.dataio.norm_data.NormData, predict_data: pcntoolkit.dataio.norm_data.NormData, save_dir: str | None = None, n_synth_samples: int | None = None) → NormativeModel#

Extends the model to a new dataset and predicts the data.

extract_data(data: pcntoolkit.dataio.norm_data.NormData) → Tuple[xarray.DataArray, xarray.DataArray, dict[str, dict[str, int]], xarray.DataArray, xarray.DataArray]#

Returns a 5-tuple of covariates, batch effects, batch effect maps, response vars, Z-scores. If the variable is not available, returns None instead of the variable.

fit(data: pcntoolkit.dataio.norm_data.NormData) → None#

Fits a regression model for each response variable in the data.

Parameters:

data (NormData) – Training data containing covariates (X), batch effects (batch_effects), and response variables (Y). Must be a valid NormData object with properly formatted dimensions: - X: (n_samples, n_covariates) - batch_effects: (n_samples, n_batch_effects) - Y: (n_samples, n_response_vars)

fit_predict(fit_data: pcntoolkit.dataio.norm_data.NormData, predict_data: pcntoolkit.dataio.norm_data.NormData) → pcntoolkit.dataio.norm_data.NormData#

Combines model.fit and model.predict in a single operation.

classmethod from_args(**kwargs) → NormativeModel#

Create a new normative model from command line arguments.

Parameters:

args (dict[str, str]) – A dictionary of command line arguments.

Returns:

An instance of a normative model.

Return type:

NormBase

Raises:

ValueError – If the regression model specified in the arguments is unknown.

harmonize(data: pcntoolkit.dataio.norm_data.NormData, reference_batch_effect: dict[str, str] | None = None) → pcntoolkit.dataio.norm_data.NormData#

Harmonizes the data to a reference batch effect. Harmonizes to the provided reference batch effect if provided, otherwise, harmonizes to the first batch effect alphabetically.

Parameters:

  • data (NormData) – Data to harmonize.

  • reference_batch_effect (dict[str, str]) – Reference batch effect.

classmethod load(path: str, into: NormativeModel | None = None) → NormativeModel#

Load a normative model from a path.

Parameters:

  • path (str) – The path to the normative model.

  • into (NormBase, optional) – The normative model to load the data into. If None, a new normative model is created. This is useful if you want to load a normative model into an existing normative model, for example in the runner.

map_batch_effects(batch_effects: xarray.DataArray) → xarray.DataArray#
classmethod merge(save_dir: str, models: list[NormativeModel | str]) → NormativeModel#

Merges multiple models into a single model.

model_specific_evaluation() → None#

Save model-specific evaluation metrics.

postprocess(data: pcntoolkit.dataio.norm_data.NormData) → None#

Apply postprocessing to the data.

First unscales, then applies the inverse response transform (e.g. expm1).

Args:

data (NormData): Data to postprocess.

predict(data: pcntoolkit.dataio.norm_data.NormData) → pcntoolkit.dataio.norm_data.NormData#

Computes Z-scores, centiles, logp, yhat for each observation using fitted regression models.

preprocess(data: pcntoolkit.dataio.norm_data.NormData) → None#

Applies preprocessing transformations to the input data.

First applies an optional response transform (e.g. log1p), then scales.

Args:

data (NormData): Data to preprocess.

register_batch_effects(data: pcntoolkit.dataio.norm_data.NormData) → None#
register_data_info(data: pcntoolkit.dataio.norm_data.NormData) → None#
sample_batch_effects(n_samples: int) → xarray.DataArray#

Sample the batch effects from the estimated distribution.

sample_covariates(bes: xarray.DataArray, covariate_range_per_batch_effect: bool = False) → xarray.DataArray#

Sample the covariates from the estimated distribution.

Uses ranges of observed covariates matched with batch effects to create a representative sample

save(path: str | None = None) → None#

Save the model to a file.

Args:

path (str, optional): The path to save the model to. If None, the model is saved to the save_dir provided in the norm_conf.

scale_backward(data: pcntoolkit.dataio.norm_data.NormData) → None#

Scales data back to its original scale using stored scalers.

Parameters:

data (NormData) –

Data object containing arrays to be scaled back: - X : array-like, shape (n_samples, n_covariates)

Covariate data to be scaled back

  • yarray-like, shape (n_samples, n_response_vars), optional

    Response variable data to be scaled back

scale_forward(data: pcntoolkit.dataio.norm_data.NormData, overwrite: bool = False) → None#

Scales input data to standardized form using configured scalers.

Parameters:

  • data (NormData) –

    Data object containing arrays to be scaled: - X : array-like, shape (n_samples, n_covariates)

    Covariate data to be scaled

    • yarray-like, shape (n_samples, n_response_vars), optional

      Response variable data to be scaled

  • overwrite (bool, default False) – If True, creates new scalers even if they already exist. If False, uses existing scalers when available.

set_ensure_save_dirs()#

Ensures that the save directories for results and plots are created when they are not there yet (otherwise resulted in an error)

set_save_dir(save_dir: str) → None#

Override the save_dir in the norm_conf.

Args:

save_dir (str): New save directory.

synthesize(data: pcntoolkit.dataio.norm_data.NormData | None = None, n_samples: int | None = None, covariate_range_per_batch_effect=False) → pcntoolkit.dataio.norm_data.NormData#

Synthesize data from the model

Parameters:

  • data (NormData, optional) – A NormData object with X and batch_effects. If provided, used to generate the synthetic data. If the data has no batch_effects, batch_effects are sampled from the model. If the data has no X, X is sampled from the model, using the provided or sampled batch_effects. If neither X nor batch_effects are provided, the model is used to generate the synthetic data.

  • n_samples (int, optional) – Number of samples to synthesize. If this is None, the number of samples that were in the train data is used.

  • covariate_range_per_batch_effect (bool, optional) – If True, the covariate range is different for each batch effect.

to_dict()#
transfer(transfer_data: pcntoolkit.dataio.norm_data.NormData, save_dir: str | None = None, **kwargs) → NormativeModel#

Transfers the model to a new dataset.

transfer_predict(transfer_data: pcntoolkit.dataio.norm_data.NormData, predict_data: pcntoolkit.dataio.norm_data.NormData, save_dir: str | None = None, **kwargs) → NormativeModel#

Transfers the model to a new dataset and predicts the data.

batch_effect_counts = None#
batch_effect_covariate_ranges = None#
property batch_effect_dims: list[str]#

Returns the batch effect dimensions. Returns:

list[str]: The batch effect dimensions.

batch_effects_maps = None#
correlation_matrix = None#
covariate_ranges = None#
covariates = None#
evaluate_model: bool = True#
evaluator#
property has_batch_effect: bool#

Returns whether the model has a batch effect. Returns:

bool: True if the model has a batch effect, False otherwise. This currently looks at the template reg conf

inscaler: str = 'standardize'#
inscalers: dict#
is_fitted: bool = False#
property n_fit_observations: int#

Returns the number of batch effects. Returns:

int: The number of batch effects.

name: str | None = None#
outscaler: str = 'standardize'#
outscalers: dict#
regression_models: dict[str, pcntoolkit.regression_model.regression_model.RegressionModel]#
response_vars: list[str] = None#
property save_dir: str#
savemodel: bool = True#
saveplots: bool = True#
saveresults: bool = True#
template_regression_model: pcntoolkit.regression_model.regression_model.RegressionModel#
thrive_covariate = None#
unique_batch_effects = None#
y_transform: str | None = None#
On this page