pcntoolkit.dataio.norm_data

This module provides functionalities for normalizing and converting different types of data into a NormData object.

The NormData object is an xarray.Dataset that contains the data, covariates, batch effects, and response variables, and it is used by all the models in the toolkit.

Classes

NormData

A class for handling normative modeling data, extending xarray.Dataset.

Module Contents

class NormData(name: str, data_vars: xarray.core.types.DataVars, coords: Mapping[Any, Any], attrs: Mapping[Any, Any] | None = None)

Bases: xarray.Dataset

A class for handling normative modeling data, extending xarray.Dataset.

This class provides functionality for loading data for normative modeling. It supports various data formats.

Parameters:

  • name (str) – The name of the dataset

  • data_vars (DataVars) – Data variables for the dataset

  • coords (Mapping[Any, Any]) – Coordinates for the dataset

  • attrs (Mapping[Any, Any] | None, optional) – Additional attributes for the dataset, by default None

X

Covariate data

Type:

xr.DataArray

y

Response variable data

Type:

xr.DataArray

batch_effects

Batch effect data

Type:

xr.DataArray

Z

Z-score data

Type:

xr.DataArray

centiles

Centile data

Type:

xr.DataArray

Examples

>>> data = NormData.from_dataframe("my_data", df, covariates, batch_effects, response_vars)
>>> train_data, test_data = data.train_test_split([0.8, 0.2])

Initialize a NormData object.

Parameters:

  • name (str) – The name of the dataset.

  • data_vars (DataVars) – Data variables for the dataset.

  • coords (Mapping[Any, Any]) – Coordinates for the dataset.

  • attrs (Mapping[Any, Any] | None, optional) – Additional attributes for the dataset, by default None.

batch_effects_split(batch_effects: Dict[str, List[str]], names: Tuple[str, str] | None) → Tuple[NormData, NormData]

Split the data into two datasets, one with the specified batch effects and one without.

This is useful when you want to split a dataset into two smaller ones.

Parameters:

  • batch_effects (Dict[str, List[str]]) – A dictionary mapping batch effect dimensions to lists of values to split on.

  • names (Optional[Tuple[str, str]]) – The names for the two splits.

Returns:

A tuple containing the two split NormData instances.

Return type:

Tuple[NormData, NormData]

check_compatibility(other: NormData) → bool

Check if the data is compatible with another dataset.

Parameters:

other (NormData) – Another NormData instance to compare with.

Returns:

True if compatible, False otherwise

Return type:

bool

chunk(n_chunks: int) → Generator[NormData]

Split the data into n_chunks with roughly equal number of response variables

Parameters:

n_chunks (int) – The number of chunks to split the data into.

Returns:

A generator of NormData instances.

Return type:

Generator[NormData]

concatenate_string_arrays(*arrays: Any) → numpy.ndarray

Concatenate arrays of strings.

Parameters:

arrays (List[np.ndarray]) – A list of numpy arrays containing strings.

Returns:

A single concatenated numpy array of strings.

Return type:

np.ndarray

create_statistics_group() → None

Initializes a DataArray for statistics with NaN values.

This method creates a DataArray with dimensions ‘response_vars’ and ‘statistics’, where ‘response_vars’ corresponds to the response variables in the dataset, and ‘statistics’ includes statistics such as Rho, RMSE, SMSE, EXPV, MLL, and ShapiroW. The DataArray is filled with NaN values initially.

classmethod from_bids(bids_folder, config_params) → NormData
Abstractmethod:

Load a normative dataset from a BIDS dataset.

Parameters:

  • bids_folder (str) – Path to the BIDS folder.

  • config_params (dict) – Configuration parameters for loading the dataset.

Returns:

An instance of NormData.

Return type:

NormData

classmethod from_dataframe(name: str, dataframe: pandas.DataFrame, covariates: List[str] | None = None, batch_effects: List[str] | None = None, response_vars: List[str | LiteralString] | None = None, subject_ids: str | None = None, remove_Nan: bool = False, remove_outliers: bool = False, z_threshold: float = 3.0, attrs: Mapping[str, Any] | None = None) → NormData

Load a normative dataset from a pandas DataFrame.

Parameters:

  • name (str) – The name you want to give to the dataset. Will be used to name saved results.

  • dataframe (pd.DataFrame) – The pandas DataFrame to load.

  • covariates (List[str]) – The list of column names to be used as covariates in the dataset.

  • batch_effects (List[str]) – The list of column names to be used as batch effects in the dataset.

  • response_vars (List[str]) – The list of column names to be used as response variables in the dataset.

  • subject_ids (str) – The name of the column containing the subject IDs

  • attrs (Mapping[str, Any] | None, optional) – Additional attributes for the dataset, by default None.

  • remove_Nan (bool) – Whether or not to remove NAN values from the dataframe before creating of the class object. By default False

Returns:

An instance of NormData.

Return type:

NormData

classmethod from_fsl(fsl_folder, config_params) → NormData
Abstractmethod:

Load a normative dataset from a FSL file.

Parameters:

  • fsl_folder (str) – Path to the FSL folder.

  • config_params (dict) – Configuration parameters for loading the dataset.

Returns:

An instance of NormData.

Return type:

NormData

classmethod from_ndarrays(name: str, X: numpy.ndarray, Y: numpy.ndarray, batch_effects: numpy.ndarray | None = None, subject_ids: numpy.ndarray | None = None, attrs: Mapping[str, Any] | None = None, remove_outliers: bool = False, z_threshold: float = 3.0, remove_Nan: bool = False) → NormData

Create a NormData object from numpy arrays via DataFrame conversion.

classmethod from_netcdf(name: str, netcdf_path: str) → NormData

Load a normative dataset from a netcdf file.

Parameters:

  • name (str) – The name of the dataset.

  • netcdf_path (str) – The path to the netcdf file.

Returns:

An instance of NormData.

Return type:

NormData

classmethod from_paths(name: str, covariates_path: str, responses_path: str, batch_effects_path: str, **kwargs) → NormData

Load a normative dataset from a dictionary of paths.

classmethod from_xarray(name: str, xarray_dataset: xarray.Dataset) → NormData

Load a normative dataset from an xarray dataset.

Parameters:

  • name (str) – The name of the dataset.

  • xarray_dataset (xr.Dataset) – The xarray dataset to load.

Returns:

An instance of NormData.

Return type:

NormData

get_single_batch_effect() → Dict[str, List[str]]

Get a single batch effect for each dimension.

Returns:

A dictionary mapping each batch effect dimension to a list containing a single value.

Return type:

Dict[str, List[str]]

get_statistics_df() → pandas.DataFrame

Get the statistics as a pandas DataFrame.

has_registered_metadata() → bool

Check if the batch effect and covariate metadata have been registered and are non-empty.

Returns:

True if all required metadata attributes exist and are not empty, False otherwise.

Return type:

bool

kfold_split(k: int) → Generator[Tuple[numpy.typing.ArrayLike[int], numpy.typing.ArrayLike[int]], Any, Any]

Perform k-fold splitting of the data.

Parameters:

k (int) – The number of folds.

Returns:

A generator yielding training and testing indices for each fold.

Return type:

Generator[Tuple[ArrayLike[int], ArrayLike[int]], Any, Any]

load_centiles(save_dir) → None
load_logp(save_dir) → None
load_results(save_dir: str) → None

Loads the results (zscores, centiles, logp, statistics) back into the data

Args:

save_dir (str): Where the results are saved. I.e.: {save_dir}/Z_fit_test.csv

load_statistics(save_dir) → None
load_zscores(save_dir) → None
make_compatible(other: NormData)

Ensures datasets are compatible by merging the batch effects maps

merge(other: NormData, name: str | None = None) → NormData

Merge two NormData objects.

Drops all columns that are not present in both datasets.

register_batch_effects() → None

Create a mapping of batch effects to unique values.

classmethod remove_nan(dataframe: pandas.DataFrame) → pandas.DataFrame

Remove NaN values from the dataframe.

classmethod remove_outliers(dataframe: pandas.DataFrame, continuous_vars: List[str], z_threshold: float = 3.0) → pandas.DataFrame

Remove outliers from the dataframe.

save_centiles(save_dir: str) → None
save_logp(save_dir: str) → None
save_results(save_dir: str) → None

Saves the results (zscores, centiles, logp, statistics) to disk

Args:

save_dir (str): Where the results are saved. I.e.: {save_dir}/Z_fit_test.csv

save_statistics(save_dir: str) → None
save_zscores(save_dir: str) → None
scale_backward(inscalers: Dict[str, Any], outscalers: Dict[str, Any]) → None

Scale the data backward using provided scalers.

Parameters:

  • inscalers (Dict[str, Any]) – Scalers for the covariate data.

  • outscalers (Dict[str, Any]) – Scalers for the response variable data.

scale_forward(inscalers: Dict[str, Any], outscalers: Dict[str, Any]) → None

Scale the data forward in-place using provided scalers.

Parameters:

  • inscalers (Dict[str, Any]) – Scalers for the covariate data.

  • outscalers (Dict[str, Any]) – Scalers for the response variable data.

select_batch_effects(name: str, batch_effects: Dict[str, List[str]], invert: bool = False) → NormData

Select observations matching (or not matching) batch effects.

Parameters:

  • name (str) – Name to assign to the returned NormData instance.

  • batch_effects (Dict[str, List[str]]) – A dictionary mapping batch effect dimensions to lists of values to select batch effects from.

  • invert (bool, optional) – If True, return observations that do not match any of the specified batch effect values. Default is False.

Returns:

A NormData instance containing observations matching (or not matching) the specified batch effects.

Return type:

NormData

to_dataframe(dim_order: Sequence[Hashable] | None = None) → pandas.DataFrame

Convert the NormData instance to a pandas DataFrame.

Parameters:

dim_order (Sequence[Hashable] | None, optional) – The order of dimensions for the DataFrame, by default None.

Returns:

A DataFrame representation of the NormData instance.

Return type:

pd.DataFrame

to_netcdf(netcdf_path: str) → None

Save the NormData object to a netcdf file.

Parameters:

netcdf_path (str) – The path to the netcdf file.

Return type:

None

train_test_split(splits: Tuple[float, Ellipsis] | List[float] | float = 0.8, split_names: Tuple[str, Ellipsis] | None = None, random_state: int = 42) → Tuple[NormData, Ellipsis]

Split the data into training and testing datasets.

Parameters:

  • splits (Tuple[float, ] | List[float] | float) – A tuple (train_size, test_size), specifying the proportion of data for each split. Or a float specifying the proportion of data for the train set.

  • split_names (Tuple[str, ] | None, optional) – Names for the splits, by default None.

  • random_state (int , optional) – Random state for splits, by default 42.

Returns:

A tuple containing the training and testing NormData instances.

Return type:

Tuple[NormData, ]

__slots__ = ('unique_batch_effects', 'batch_effect_counts', 'batch_effect_covariate_ranges',...
property name: str

Get the name of the dataset.

Returns:

The name of the dataset.

Return type:

str

property response_var_list: xarray.DataArray