The NormData class

A key component of the PCNtoolkit is the NormData object. It is a container for the data that will be used to fit the normative model. The NormData object keeps track of the all the dimensions of your data, the features and response variables, batch effects, preprocessing steps, and more.

import copy

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

from pcntoolkit import NormData

Creating a NormData object

There are currently two easy ways to create a NormData object. 1. Load from a pandas dataframe 2. Load from numpy arrays

Here are examples of both.

# Creating a NormData object from a pandas dataframe

# Download an example dataset:
data = pd.read_csv(
    "https://raw.githubusercontent.com/predictive-clinical-neuroscience/PCNtoolkit-demo/refs/heads/main/data/fcon1000.csv"
)

# specify the column names to use
covariates = ["age"]
batch_effects = ["sex", "site"]
response_vars = ["WM-hypointensities", "Left-Lateral-Ventricle", "Brain-Stem"]

# create a NormData object
norm_data = NormData.from_dataframe(
    name="fcon1000",
    dataframe=data,
    covariates=covariates,
    batch_effects=batch_effects,
    response_vars=response_vars,
    remove_outliers=True,
    z_threshold=10,
)
norm_data.coords

Process: 75157 - 2025-11-20 13:13:40 - Removed 2 outliers for WM-hypointensities
Process: 75157 - 2025-11-20 13:13:40 - Removed 2 outliers
Process: 75157 - 2025-11-20 13:13:40 - Dataset "fcon1000" created.
    - 1076 observations
    - 1076 unique subjects
    - 1 covariates
    - 3 response variables
    - 2 batch effects:
            sex (2)
    site (23)

Coordinates:
  * observations       (observations) int64 9kB 0 1 2 3 ... 1072 1073 1074 1075
  * response_vars      (response_vars) <U22 264B 'WM-hypointensities' ... 'Br...
  * covariates         (covariates) <U3 12B 'age'
  * batch_effect_dims  (batch_effect_dims) <U4 32B 'sex' 'site'

# Creating a NormData object from numpy arrays
import numpy as np

from pcntoolkit import NormData

# synthesize some data
X = np.random.randn(100, 10)
Y = np.random.randn(100, 10)
batch_effects = np.random.randint(0, 2, 100)[:,None]
subject_ids = np.arange(100)

# Create a NormData object
np_norm_data = NormData.from_ndarrays("fcon1000", X=X, Y=Y, batch_effects=batch_effects, subject_ids=subject_ids)
np_norm_data.coords

Process: 75157 - 2025-11-20 13:18:52 - Dataset "fcon1000" created.
    - 100 observations
    - 100 unique subjects
    - 10 covariates
    - 10 response variables
    - 1 batch effects:
            batch_effect_0 (2)

Coordinates:
  * observations       (observations) int64 800B 0 1 2 3 4 5 ... 95 96 97 98 99
  * response_vars      (response_vars) <U14 560B 'response_var_0' ... 'respon...
  * covariates         (covariates) <U11 440B 'covariate_0' ... 'covariate_9'
  * batch_effect_dims  (batch_effect_dims) <U14 56B 'batch_effect_0'

As you can see, it is very simple to create a NormData object.

There is an important difference though: the coordinates of the NormData object that was created with from_dataframe have the name of the column in the dataframe, but the from_ndarrays method creates coordinates with generic names. This is why the from_dataframe method is favorable.

Casting back to a pandas dataframe

The NormData object can be cast back to a pandas dataframe using the to_dataframe method. This will return a pandas dataframe with a columnar multi-index.

df = norm_data.to_dataframe()
df.head()

	X	Y			batch_effects		subject_ids
	age	Brain-Stem	Left-Lateral-Ventricle	WM-hypointensities	sex	site	subject_ids
0	25.63	20663.2	4049.4	1686.7	1	AnnArbor_a	0
1	18.34	19954.0	9312.6	1371.1	1	AnnArbor_a	1
2	29.20	21645.2	8972.6	1414.8	1	AnnArbor_a	2
3	31.39	20790.6	6798.6	1830.6	1	AnnArbor_a	3
4	13.58	17692.6	6112.5	1642.4	1	AnnArbor_a	4

Inspecting the NormData

So let’s go over the attributes of the NormData object. Because it is a subclass of xarray.Dataset, it has all the attributes of a xarray.Dataset, but it has some additional attributes that are specific to normative modelling.

The data variables

The data variables of the NormData object are: - X: The covariates - Y: The response variables - batch_effects: The batch effects - subjects: The subject ids

And all these data variables are xarray.DataArrays, with corresponding dimensions, stored in the data_vars attribute of the NormData object.

norm_data.data_vars

Data variables:
    subject_ids    (observations) int64 9kB 0 1 2 3 4 ... 1072 1073 1074 1075
    Y              (observations, response_vars) float64 26kB 1.687e+03 ... 1...
    X              (observations, covariates) float64 9kB 25.63 18.34 ... 23.0
    batch_effects  (observations, batch_effect_dims) <U17 146kB '1' ... 'Sain...

The coordinates

Because it is a subclass of xarray.Dataset, the NormData object also holds all the coordinates of the data, found under the coords attribute.

The coordinates are: - observations: The index of the observations - response_vars: The names of the response variables - covariates: The names of the covariates - batch_effect_dims: The names of the batch effect dimensions

norm_data.coords

Coordinates:
  * observations       (observations) int64 9kB 0 1 2 3 ... 1072 1073 1074 1075
  * response_vars      (response_vars) <U22 264B 'WM-hypointensities' ... 'Br...
  * covariates         (covariates) <U3 12B 'age'
  * batch_effect_dims  (batch_effect_dims) <U4 32B 'sex' 'site'

Indexing using the coordinates

Xarrays powerful indexing methods can also be used on NormData.

Selecting a response variable

For example, to select the data for a specific response variable, you can use the response_vars coordinate:

norm_data.sel(response_vars="WM-hypointensities")

This will return a new NormData object with only the data for the response variable “WM-hypointensities”.

norm_data.sel(response_vars="WM-hypointensities")

<xarray.NormData> Size: 181kB
Dimensions:            (observations: 1076, covariates: 1, batch_effect_dims: 2)
Coordinates:
  * observations       (observations) int64 9kB 0 1 2 3 ... 1072 1073 1074 1075
    response_vars      <U22 88B 'WM-hypointensities'
  * covariates         (covariates) <U3 12B 'age'
  * batch_effect_dims  (batch_effect_dims) <U4 32B 'sex' 'site'
Data variables:
    subject_ids        (observations) int64 9kB 0 1 2 3 ... 1072 1073 1074 1075
    Y                  (observations) float64 9kB 1.687e+03 1.371e+03 ... 509.1
    X                  (observations, covariates) float64 9kB 25.63 ... 23.0
    batch_effects      (observations, batch_effect_dims) <U17 146kB '1' ... '...
Attributes:
    real_ids:                       False
    is_scaled:                      False
    name:                           fcon1000
    unique_batch_effects:           {np.str_('sex'): [np.str_('0'), np.str_('...
    batch_effect_counts:            defaultdict(<function NormData.register_b...
    covariate_ranges:               {np.str_('age'): {'mean': np.float64(28.1...
    batch_effect_covariate_ranges:  {np.str_('sex'): {np.str_('0'): {np.str_(...

xarray.NormData

• Dimensions:
  • observations: 1076
  • covariates: 1
  • batch_effect_dims: 2
• Coordinates: (4)
  • observations
    (observations)
    int64
    0 1 2 3 4 ... 1072 1073 1074 1075

    array([   0,    1,    2, ..., 1073, 1074, 1075], shape=(1076,))

  • response_vars
    ()
    <U22
    'WM-hypointensities'

    array('WM-hypointensities', dtype='<U22')

  • covariates
    (covariates)
    <U3
    'age'

    array(['age'], dtype='<U3')

  • batch_effect_dims
    (batch_effect_dims)
    <U4
    'sex' 'site'

    array(['sex', 'site'], dtype='<U4')

• Data variables: (4)
  • subject_ids
    (observations)
    int64
    0 1 2 3 4 ... 1072 1073 1074 1075

    array([   0,    1,    2, ..., 1073, 1074, 1075], shape=(1076,))

  • Y
    (observations)
    float64
    1.687e+03 1.371e+03 ... 448.3 509.1

    array([1686.7, 1371.1, 1414.8, ..., 1061. ,  448.3,  509.1], shape=(1076,))

  • X
    (observations, covariates)
    float64
    25.63 18.34 29.2 ... 27.0 29.0 23.0

    array([[25.63],
           [18.34],
           [29.2 ],
           ...,
           [27.  ],
           [29.  ],
           [23.  ]], shape=(1076, 1))

  • batch_effects
    (observations, batch_effect_dims)
    <U17
    '1' 'AnnArbor_a' ... 'SaintLouis'

    array([['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ...,
           ['1', 'SaintLouis'],
           ['0', 'SaintLouis'],
           ['0', 'SaintLouis']], shape=(1076, 2), dtype='<U17')

• Indexes: (3)
  • observations
    PandasIndex

    PandasIndex(Index([   0,    1,    2,    3,    4,    5,    6,    7,    8,    9,
           ...
           1066, 1067, 1068, 1069, 1070, 1071, 1072, 1073, 1074, 1075],
          dtype='int64', name='observations', length=1076))

  • covariates
    PandasIndex

    PandasIndex(Index(['age'], dtype='object', name='covariates'))

  • batch_effect_dims
    PandasIndex

    PandasIndex(Index(['sex', 'site'], dtype='object', name='batch_effect_dims'))

• Attributes: (7)

  real_ids :

  False

  is_scaled :

  False

  name :

  fcon1000

  unique_batch_effects :

  {np.str_('sex'): [np.str_('0'), np.str_('1')], np.str_('site'): [np.str_('AnnArbor_a'), np.str_('AnnArbor_b'), np.str_('Atlanta'), np.str_('Baltimore'), np.str_('Bangor'), np.str_('Beijing_Zang'), np.str_('Berlin_Margulies'), np.str_('Cambridge_Buckner'), np.str_('Cleveland'), np.str_('ICBM'), np.str_('Leiden_2180'), np.str_('Leiden_2200'), np.str_('Milwaukee_b'), np.str_('Munchen'), np.str_('NewYork_a'), np.str_('NewYork_a_ADHD'), np.str_('Newark'), np.str_('Oulu'), np.str_('Oxford'), np.str_('PaloAlto'), np.str_('Pittsburgh'), np.str_('Queensland'), np.str_('SaintLouis')]}

  batch_effect_counts :

  defaultdict(<function NormData.register_batch_effects.<locals>.<lambda> at 0x1685104a0>, {np.str_('sex'): {np.str_('0'): 587, np.str_('1'): 489}, np.str_('site'): {np.str_('AnnArbor_a'): 24, np.str_('AnnArbor_b'): 32, np.str_('Atlanta'): 28, np.str_('Baltimore'): 23, np.str_('Bangor'): 20, np.str_('Beijing_Zang'): 198, np.str_('Berlin_Margulies'): 26, np.str_('Cambridge_Buckner'): 198, np.str_('Cleveland'): 31, np.str_('ICBM'): 83, np.str_('Leiden_2180'): 12, np.str_('Leiden_2200'): 19, np.str_('Milwaukee_b'): 46, np.str_('Munchen'): 15, np.str_('NewYork_a'): 83, np.str_('NewYork_a_ADHD'): 25, np.str_('Newark'): 19, np.str_('Oulu'): 102, np.str_('Oxford'): 22, np.str_('PaloAlto'): 17, np.str_('Pittsburgh'): 3, np.str_('Queensland'): 19, np.str_('SaintLouis'): 31}})

  covariate_ranges :

  {np.str_('age'): {'mean': np.float64(28.160613382899626), 'min': np.float64(7.88), 'max': np.float64(79.0)}}

  batch_effect_covariate_ranges :

  {np.str_('sex'): {np.str_('0'): {np.str_('age'): {'mean': np.float64(27.88655877342419), 'min': np.float64(7.88), 'max': np.float64(79.0)}}, np.str_('1'): {np.str_('age'): {'mean': np.float64(28.48959100204499), 'min': np.float64(9.21), 'max': np.float64(78.0)}}}, np.str_('site'): {np.str_('AnnArbor_a'): {np.str_('age'): {'mean': np.float64(21.28333333333333), 'min': np.float64(13.41), 'max': np.float64(40.98)}}, np.str_('AnnArbor_b'): {np.str_('age'): {'mean': np.float64(44.40625), 'min': np.float64(19.0), 'max': np.float64(79.0)}}, np.str_('Atlanta'): {np.str_('age'): {'mean': np.float64(30.892857142857142), 'min': np.float64(22.0), 'max': np.float64(57.0)}}, np.str_('Baltimore'): {np.str_('age'): {'mean': np.float64(29.26086956521739), 'min': np.float64(20.0), 'max': np.float64(40.0)}}, np.str_('Bangor'): {np.str_('age'): {'mean': np.float64(23.4), 'min': np.float64(19.0), 'max': np.float64(38.0)}}, np.str_('Beijing_Zang'): {np.str_('age'): {'mean': np.float64(21.161616161616163), 'min': np.float64(18.0), 'max': np.float64(26.0)}}, np.str_('Berlin_Margulies'): {np.str_('age'): {'mean': np.float64(29.76923076923077), 'min': np.float64(23.0), 'max': np.float64(44.0)}}, np.str_('Cambridge_Buckner'): {np.str_('age'): {'mean': np.float64(21.03030303030303), 'min': np.float64(18.0), 'max': np.float64(30.0)}}, np.str_('Cleveland'): {np.str_('age'): {'mean': np.float64(43.54838709677419), 'min': np.float64(24.0), 'max': np.float64(60.0)}}, np.str_('ICBM'): {np.str_('age'): {'mean': np.float64(43.25301204819277), 'min': np.float64(19.0), 'max': np.float64(79.0)}}, np.str_('Leiden_2180'): {np.str_('age'): {'mean': np.float64(23.0), 'min': np.float64(20.0), 'max': np.float64(27.0)}}, np.str_('Leiden_2200'): {np.str_('age'): {'mean': np.float64(21.68421052631579), 'min': np.float64(18.0), 'max': np.float64(28.0)}}, np.str_('Milwaukee_b'): {np.str_('age'): {'mean': np.float64(53.58695652173913), 'min': np.float64(44.0), 'max': np.float64(65.0)}}, np.str_('Munchen'): {np.str_('age'): {'mean': np.float64(68.13333333333334), 'min': np.float64(63.0), 'max': np.float64(74.0)}}, np.str_('NewYork_a'): {np.str_('age'): {'mean': np.float64(24.507710843373495), 'min': np.float64(7.88), 'max': np.float64(49.16)}}, np.str_('NewYork_a_ADHD'): {np.str_('age'): {'mean': np.float64(34.9952), 'min': np.float64(20.69), 'max': np.float64(50.9)}}, np.str_('Newark'): {np.str_('age'): {'mean': np.float64(24.105263157894736), 'min': np.float64(21.0), 'max': np.float64(39.0)}}, np.str_('Oulu'): {np.str_('age'): {'mean': np.float64(21.519607843137255), 'min': np.float64(20.0), 'max': np.float64(23.0)}}, np.str_('Oxford'): {np.str_('age'): {'mean': np.float64(29.0), 'min': np.float64(20.0), 'max': np.float64(35.0)}}, np.str_('PaloAlto'): {np.str_('age'): {'mean': np.float64(32.470588235294116), 'min': np.float64(22.0), 'max': np.float64(46.0)}}, np.str_('Pittsburgh'): {np.str_('age'): {'mean': np.float64(32.333333333333336), 'min': np.float64(25.0), 'max': np.float64(47.0)}}, np.str_('Queensland'): {np.str_('age'): {'mean': np.float64(25.94736842105263), 'min': np.float64(20.0), 'max': np.float64(34.0)}}, np.str_('SaintLouis'): {np.str_('age'): {'mean': np.float64(25.096774193548388), 'min': np.float64(21.0), 'max': np.float64(29.0)}}}}

Selecting a number of observations

But we can also filter out a slice of the data. For example, to select the first 10 observations, you can use the observations coordinate:

norm_data.sel(observations=slice(0, 9))

This will return a new NormData object with only the first 10 observations.

norm_data.sel(observations=slice(0, 9))

<xarray.NormData> Size: 2kB
Dimensions:            (observations: 10, response_vars: 3, covariates: 1,
                        batch_effect_dims: 2)
Coordinates:
  * observations       (observations) int64 80B 0 1 2 3 4 5 6 7 8 9
  * response_vars      (response_vars) <U22 264B 'WM-hypointensities' ... 'Br...
  * covariates         (covariates) <U3 12B 'age'
  * batch_effect_dims  (batch_effect_dims) <U4 32B 'sex' 'site'
Data variables:
    subject_ids        (observations) int64 80B 0 1 2 3 4 5 6 7 8 9
    Y                  (observations, response_vars) float64 240B 1.687e+03 ....
    X                  (observations, covariates) float64 80B 25.63 ... 19.88
    batch_effects      (observations, batch_effect_dims) <U17 1kB '1' ... 'An...
Attributes:
    real_ids:                       False
    is_scaled:                      False
    name:                           fcon1000
    unique_batch_effects:           {np.str_('sex'): [np.str_('0'), np.str_('...
    batch_effect_counts:            defaultdict(<function NormData.register_b...
    covariate_ranges:               {np.str_('age'): {'mean': np.float64(28.1...
    batch_effect_covariate_ranges:  {np.str_('sex'): {np.str_('0'): {np.str_(...

xarray.NormData

• Dimensions:
  • observations: 10
  • response_vars: 3
  • covariates: 1
  • batch_effect_dims: 2
• Coordinates: (4)
  • observations
    (observations)
    int64
    0 1 2 3 4 5 6 7 8 9

    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

  • response_vars
    (response_vars)
    <U22
    'WM-hypointensities' ... 'Brain-...

    array(['WM-hypointensities', 'Left-Lateral-Ventricle', 'Brain-Stem'],
          dtype='<U22')

  • covariates
    (covariates)
    <U3
    'age'

    array(['age'], dtype='<U3')

  • batch_effect_dims
    (batch_effect_dims)
    <U4
    'sex' 'site'

    array(['sex', 'site'], dtype='<U4')

• Data variables: (4)
  • subject_ids
    (observations)
    int64
    0 1 2 3 4 5 6 7 8 9

    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

  • Y
    (observations, response_vars)
    float64
    1.687e+03 4.049e+03 ... 2.092e+04

    array([[ 1686.7,  4049.4, 20663.2],
           [ 1371.1,  9312.6, 19954. ],
           [ 1414.8,  8972.6, 21645.2],
           [ 1830.6,  6798.6, 20790.6],
           [ 1642.4,  6112.5, 17692.6],
           [ 2108.4,  7076.4, 20996.8],
           [ 2023.1,  4862.2, 20964.9],
           [ 2193.4,  9931.7, 21339.8],
           [ 1086. ,  6479.5, 18517.9],
           [ 1604.9,  5890.9, 20919.9]])

  • X
    (observations, covariates)
    float64
    25.63 18.34 29.2 ... 17.58 19.88

    array([[25.63],
           [18.34],
           [29.2 ],
           [31.39],
           [13.58],
           [29.21],
           [15.92],
           [21.46],
           [17.58],
           [19.88]])

  • batch_effects
    (observations, batch_effect_dims)
    <U17
    '1' 'AnnArbor_a' ... 'AnnArbor_a'

    array([['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a']], dtype='<U17')

• Indexes: (4)
  • observations
    PandasIndex

    PandasIndex(Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64', name='observations'))

  • response_vars
    PandasIndex

    PandasIndex(Index(['WM-hypointensities', 'Left-Lateral-Ventricle', 'Brain-Stem'], dtype='object', name='response_vars'))

  • covariates
    PandasIndex

    PandasIndex(Index(['age'], dtype='object', name='covariates'))

  • batch_effect_dims
    PandasIndex

    PandasIndex(Index(['sex', 'site'], dtype='object', name='batch_effect_dims'))

• Attributes: (7)

  real_ids :

  False

  is_scaled :

  False

  name :

  fcon1000

  unique_batch_effects :

  {np.str_('sex'): [np.str_('0'), np.str_('1')], np.str_('site'): [np.str_('AnnArbor_a'), np.str_('AnnArbor_b'), np.str_('Atlanta'), np.str_('Baltimore'), np.str_('Bangor'), np.str_('Beijing_Zang'), np.str_('Berlin_Margulies'), np.str_('Cambridge_Buckner'), np.str_('Cleveland'), np.str_('ICBM'), np.str_('Leiden_2180'), np.str_('Leiden_2200'), np.str_('Milwaukee_b'), np.str_('Munchen'), np.str_('NewYork_a'), np.str_('NewYork_a_ADHD'), np.str_('Newark'), np.str_('Oulu'), np.str_('Oxford'), np.str_('PaloAlto'), np.str_('Pittsburgh'), np.str_('Queensland'), np.str_('SaintLouis')]}

  batch_effect_counts :

  defaultdict(<function NormData.register_batch_effects.<locals>.<lambda> at 0x1685104a0>, {np.str_('sex'): {np.str_('0'): 587, np.str_('1'): 489}, np.str_('site'): {np.str_('AnnArbor_a'): 24, np.str_('AnnArbor_b'): 32, np.str_('Atlanta'): 28, np.str_('Baltimore'): 23, np.str_('Bangor'): 20, np.str_('Beijing_Zang'): 198, np.str_('Berlin_Margulies'): 26, np.str_('Cambridge_Buckner'): 198, np.str_('Cleveland'): 31, np.str_('ICBM'): 83, np.str_('Leiden_2180'): 12, np.str_('Leiden_2200'): 19, np.str_('Milwaukee_b'): 46, np.str_('Munchen'): 15, np.str_('NewYork_a'): 83, np.str_('NewYork_a_ADHD'): 25, np.str_('Newark'): 19, np.str_('Oulu'): 102, np.str_('Oxford'): 22, np.str_('PaloAlto'): 17, np.str_('Pittsburgh'): 3, np.str_('Queensland'): 19, np.str_('SaintLouis'): 31}})

  covariate_ranges :

  {np.str_('age'): {'mean': np.float64(28.160613382899626), 'min': np.float64(7.88), 'max': np.float64(79.0)}}

  batch_effect_covariate_ranges :

  {np.str_('sex'): {np.str_('0'): {np.str_('age'): {'mean': np.float64(27.88655877342419), 'min': np.float64(7.88), 'max': np.float64(79.0)}}, np.str_('1'): {np.str_('age'): {'mean': np.float64(28.48959100204499), 'min': np.float64(9.21), 'max': np.float64(78.0)}}}, np.str_('site'): {np.str_('AnnArbor_a'): {np.str_('age'): {'mean': np.float64(21.28333333333333), 'min': np.float64(13.41), 'max': np.float64(40.98)}}, np.str_('AnnArbor_b'): {np.str_('age'): {'mean': np.float64(44.40625), 'min': np.float64(19.0), 'max': np.float64(79.0)}}, np.str_('Atlanta'): {np.str_('age'): {'mean': np.float64(30.892857142857142), 'min': np.float64(22.0), 'max': np.float64(57.0)}}, np.str_('Baltimore'): {np.str_('age'): {'mean': np.float64(29.26086956521739), 'min': np.float64(20.0), 'max': np.float64(40.0)}}, np.str_('Bangor'): {np.str_('age'): {'mean': np.float64(23.4), 'min': np.float64(19.0), 'max': np.float64(38.0)}}, np.str_('Beijing_Zang'): {np.str_('age'): {'mean': np.float64(21.161616161616163), 'min': np.float64(18.0), 'max': np.float64(26.0)}}, np.str_('Berlin_Margulies'): {np.str_('age'): {'mean': np.float64(29.76923076923077), 'min': np.float64(23.0), 'max': np.float64(44.0)}}, np.str_('Cambridge_Buckner'): {np.str_('age'): {'mean': np.float64(21.03030303030303), 'min': np.float64(18.0), 'max': np.float64(30.0)}}, np.str_('Cleveland'): {np.str_('age'): {'mean': np.float64(43.54838709677419), 'min': np.float64(24.0), 'max': np.float64(60.0)}}, np.str_('ICBM'): {np.str_('age'): {'mean': np.float64(43.25301204819277), 'min': np.float64(19.0), 'max': np.float64(79.0)}}, np.str_('Leiden_2180'): {np.str_('age'): {'mean': np.float64(23.0), 'min': np.float64(20.0), 'max': np.float64(27.0)}}, np.str_('Leiden_2200'): {np.str_('age'): {'mean': np.float64(21.68421052631579), 'min': np.float64(18.0), 'max': np.float64(28.0)}}, np.str_('Milwaukee_b'): {np.str_('age'): {'mean': np.float64(53.58695652173913), 'min': np.float64(44.0), 'max': np.float64(65.0)}}, np.str_('Munchen'): {np.str_('age'): {'mean': np.float64(68.13333333333334), 'min': np.float64(63.0), 'max': np.float64(74.0)}}, np.str_('NewYork_a'): {np.str_('age'): {'mean': np.float64(24.507710843373495), 'min': np.float64(7.88), 'max': np.float64(49.16)}}, np.str_('NewYork_a_ADHD'): {np.str_('age'): {'mean': np.float64(34.9952), 'min': np.float64(20.69), 'max': np.float64(50.9)}}, np.str_('Newark'): {np.str_('age'): {'mean': np.float64(24.105263157894736), 'min': np.float64(21.0), 'max': np.float64(39.0)}}, np.str_('Oulu'): {np.str_('age'): {'mean': np.float64(21.519607843137255), 'min': np.float64(20.0), 'max': np.float64(23.0)}}, np.str_('Oxford'): {np.str_('age'): {'mean': np.float64(29.0), 'min': np.float64(20.0), 'max': np.float64(35.0)}}, np.str_('PaloAlto'): {np.str_('age'): {'mean': np.float64(32.470588235294116), 'min': np.float64(22.0), 'max': np.float64(46.0)}}, np.str_('Pittsburgh'): {np.str_('age'): {'mean': np.float64(32.333333333333336), 'min': np.float64(25.0), 'max': np.float64(47.0)}}, np.str_('Queensland'): {np.str_('age'): {'mean': np.float64(25.94736842105263), 'min': np.float64(20.0), 'max': np.float64(34.0)}}, np.str_('SaintLouis'): {np.str_('age'): {'mean': np.float64(25.096774193548388), 'min': np.float64(21.0), 'max': np.float64(29.0)}}}}

NormData with predictions

After fitting a model and predicting on NormData, the NormData object will have new attributes holding the predictions.

Specifically, the NormData object will be extended with new data variables:

• Z: The predicted Z scores for each response variable

• centiles: The predicted centiles

• logp: The predicted log-p-values for each response variable

• Yhat: The predicted mean of the response variable

• Y_harmonized: The harmonized response variables

• statistics: An array of statistics for each response variable

And the following new coordinates: - centile: The specific centile values - statistic: The name of the computed statistics

from pcntoolkit import BLR, NormativeModel

# We create a very simple BLR model because it is fast to fit
model = NormativeModel(BLR())
model.fit(norm_data)  # Fitting on the data also makes predictions for that data

Process: 75157 - 2025-11-20 13:18:58 - Fitting models on 3 response variables.
Process: 75157 - 2025-11-20 13:18:58 - Fitting model for WM-hypointensities.
Process: 75157 - 2025-11-20 13:18:58 - Fitting model for Left-Lateral-Ventricle.
Process: 75157 - 2025-11-20 13:18:58 - Fitting model for Brain-Stem.
Process: 75157 - 2025-11-20 13:18:58 - Making predictions on 3 response variables.
Process: 75157 - 2025-11-20 13:18:58 - Computing z-scores for 3 response variables.
Process: 75157 - 2025-11-20 13:18:58 - Computing z-scores for WM-hypointensities.
Process: 75157 - 2025-11-20 13:18:58 - Computing z-scores for Left-Lateral-Ventricle.
Process: 75157 - 2025-11-20 13:18:58 - Computing z-scores for Brain-Stem.
Process: 75157 - 2025-11-20 13:18:58 - Computing centiles for 3 response variables.
Process: 75157 - 2025-11-20 13:18:58 - Computing centiles for WM-hypointensities.
Process: 75157 - 2025-11-20 13:18:58 - Computing centiles for Left-Lateral-Ventricle.
Process: 75157 - 2025-11-20 13:18:58 - Computing centiles for Brain-Stem.
Process: 75157 - 2025-11-20 13:18:58 - Computing log-probabilities for 3 response variables.
Process: 75157 - 2025-11-20 13:18:58 - Computing log-probabilities for WM-hypointensities.
Process: 75157 - 2025-11-20 13:18:58 - Computing log-probabilities for Left-Lateral-Ventricle.
Process: 75157 - 2025-11-20 13:18:58 - Computing log-probabilities for Brain-Stem.
Process: 75157 - 2025-11-20 13:18:58 - Computing yhat for 3 response variables.
Process: 75157 - 2025-11-20 13:18:58 - Computing yhat for WM-hypointensities.
Process: 75157 - 2025-11-20 13:18:58 - Computing yhat for Left-Lateral-Ventricle.
Process: 75157 - 2025-11-20 13:18:58 - Computing yhat for Brain-Stem.

/opt/anaconda3/envs/ptk/lib/python3.12/site-packages/pcntoolkit/dataio/norm_data.py:1094: FutureWarning: The previous implementation of stack is deprecated and will be removed in a future version of pandas. See the What's New notes for pandas 2.1.0 for details. Specify future_stack=True to adopt the new implementation and silence this warning.
  subject_ids = subject_ids.stack(level="centile")

Process: 75157 - 2025-11-20 13:18:58 - Dataset "centile" created.
    - 150 observations
    - 150 unique subjects
    - 1 covariates
    - 3 response variables
    - 2 batch effects:
            sex (1)
    site (1)

Process: 75157 - 2025-11-20 13:18:58 - Computing centiles for 3 response variables.
Process: 75157 - 2025-11-20 13:18:58 - Computing centiles for WM-hypointensities.
Process: 75157 - 2025-11-20 13:18:58 - Computing centiles for Left-Lateral-Ventricle.
Process: 75157 - 2025-11-20 13:18:58 - Computing centiles for Brain-Stem.
Process: 75157 - 2025-11-20 13:18:58 - Harmonizing data on 3 response variables.
Process: 75157 - 2025-11-20 13:18:58 - Harmonizing data for WM-hypointensities.
Process: 75157 - 2025-11-20 13:18:58 - Harmonizing data for Left-Lateral-Ventricle.
Process: 75157 - 2025-11-20 13:18:58 - Harmonizing data for Brain-Stem.
Process: 75157 - 2025-11-20 13:18:59 - Saving model to:
    /Users/stijndeboer/.pcntoolkit/saves.

norm_data.data_vars

Data variables:
    subject_ids    (observations) int64 9kB 0 1 2 3 4 ... 1072 1073 1074 1075
    Y              (observations, response_vars) float64 26kB 1.687e+03 ... 1...
    X              (observations, covariates) float64 9kB 25.63 18.34 ... 23.0
    batch_effects  (observations, batch_effect_dims) <U17 146kB '1' ... 'Sain...
    Z              (observations, response_vars) float64 26kB 0.7423 ... -0.2588
    centiles       (centile, observations, response_vars) float64 129kB 152.3...
    logp           (observations, response_vars) float64 26kB -1.158 ... -0.9537
    Yhat           (observations, response_vars) float64 26kB 1.21e+03 ... 2....
    statistics     (response_vars, statistic) float64 264B 0.1172 ... 0.9965
    Y_harmonized   (observations, response_vars) float64 26kB 1.687e+03 ... 1...

norm_data.coords

Coordinates:
  * observations       (observations) int64 9kB 0 1 2 3 ... 1072 1073 1074 1075
  * response_vars      (response_vars) <U22 264B 'WM-hypointensities' ... 'Br...
  * covariates         (covariates) <U3 12B 'age'
  * batch_effect_dims  (batch_effect_dims) <U4 32B 'sex' 'site'
  * centile            (centile) float64 40B 0.05 0.25 0.5 0.75 0.95
  * statistic          (statistic) <U8 352B 'EXPV' 'MACE' ... 'SMSE' 'ShapiroW'

Indexing of predicted data

All the indexing methods can still be used, and they will also slice through the newly added data variables. So for example, to select the first 10 observations, you can use:

norm_data.sel(observations=slice(0, 9))

This will return a new NormData object with only the first 10 observations.

norm_data.sel(observations=slice(0, 9))

<xarray.NormData> Size: 5kB
Dimensions:            (observations: 10, response_vars: 3, covariates: 1,
                        batch_effect_dims: 2, centile: 5, statistic: 11)
Coordinates:
  * observations       (observations) int64 80B 0 1 2 3 4 5 6 7 8 9
  * response_vars      (response_vars) <U22 264B 'WM-hypointensities' ... 'Br...
  * covariates         (covariates) <U3 12B 'age'
  * batch_effect_dims  (batch_effect_dims) <U4 32B 'sex' 'site'
  * centile            (centile) float64 40B 0.05 0.25 0.5 0.75 0.95
  * statistic          (statistic) <U8 352B 'EXPV' 'MACE' ... 'SMSE' 'ShapiroW'
Data variables:
    subject_ids        (observations) int64 80B 0 1 2 3 4 5 6 7 8 9
    Y                  (observations, response_vars) float64 240B 1.687e+03 ....
    X                  (observations, covariates) float64 80B 25.63 ... 19.88
    batch_effects      (observations, batch_effect_dims) <U17 1kB '1' ... 'An...
    Z                  (observations, response_vars) float64 240B 0.7423 ... ...
    centiles           (centile, observations, response_vars) float64 1kB 152...
    logp               (observations, response_vars) float64 240B -1.158 ... ...
    Yhat               (observations, response_vars) float64 240B 1.21e+03 .....
    statistics         (response_vars, statistic) float64 264B 0.1172 ... 0.9965
    Y_harmonized       (observations, response_vars) float64 240B 1.687e+03 ....
Attributes:
    real_ids:                       False
    is_scaled:                      False
    name:                           fcon1000
    unique_batch_effects:           {np.str_('sex'): [np.str_('0'), np.str_('...
    batch_effect_counts:            defaultdict(<function NormData.register_b...
    covariate_ranges:               {np.str_('age'): {'mean': np.float64(28.1...
    batch_effect_covariate_ranges:  {np.str_('sex'): {np.str_('0'): {np.str_(...

xarray.NormData

• Dimensions:
  • observations: 10
  • response_vars: 3
  • covariates: 1
  • batch_effect_dims: 2
  • centile: 5
  • statistic: 11
• Coordinates: (6)
  • observations
    (observations)
    int64
    0 1 2 3 4 5 6 7 8 9

    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

  • response_vars
    (response_vars)
    <U22
    'WM-hypointensities' ... 'Brain-...

    array(['WM-hypointensities', 'Left-Lateral-Ventricle', 'Brain-Stem'],
          dtype='<U22')

  • covariates
    (covariates)
    <U3
    'age'

    array(['age'], dtype='<U3')

  • batch_effect_dims
    (batch_effect_dims)
    <U4
    'sex' 'site'

    array(['sex', 'site'], dtype='<U4')

  • centile
    (centile)
    float64
    0.05 0.25 0.5 0.75 0.95

    array([0.05, 0.25, 0.5 , 0.75, 0.95])

  • statistic
    (statistic)
    <U8
    'EXPV' 'MACE' ... 'SMSE' 'ShapiroW'

    array(['EXPV', 'MACE', 'MAPE', 'MSLL', 'NLL', 'R2', 'RMSE', 'Rho', 'Rho_p',
           'SMSE', 'ShapiroW'], dtype='<U8')

• Data variables: (10)
  • subject_ids
    (observations)
    int64
    0 1 2 3 4 5 6 7 8 9

    array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

  • Y
    (observations, response_vars)
    float64
    1.687e+03 4.049e+03 ... 2.092e+04

    array([[ 1686.7,  4049.4, 20663.2],
           [ 1371.1,  9312.6, 19954. ],
           [ 1414.8,  8972.6, 21645.2],
           [ 1830.6,  6798.6, 20790.6],
           [ 1642.4,  6112.5, 17692.6],
           [ 2108.4,  7076.4, 20996.8],
           [ 2023.1,  4862.2, 20964.9],
           [ 2193.4,  9931.7, 21339.8],
           [ 1086. ,  6479.5, 18517.9],
           [ 1604.9,  5890.9, 20919.9]])

  • X
    (observations, covariates)
    float64
    25.63 18.34 29.2 ... 17.58 19.88

    array([[25.63],
           [18.34],
           [29.2 ],
           [31.39],
           [13.58],
           [29.21],
           [15.92],
           [21.46],
           [17.58],
           [19.88]])

  • batch_effects
    (observations, batch_effect_dims)
    <U17
    '1' 'AnnArbor_a' ... 'AnnArbor_a'

    array([['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a'],
           ['1', 'AnnArbor_a']], dtype='<U17')

  • Z
    (observations, response_vars)
    float64
    0.7423 -0.8588 ... -0.2164 0.1348

    array([[ 0.74231137, -0.85877919,  0.02189509],
           [ 0.44522032,  0.68748038, -0.24940649],
           [ 0.22427048,  0.25876175,  0.40908844],
           [ 0.81287777, -0.35399385,  0.06283337],
           [ 0.99357247,  0.03846796, -1.14655481],
           [ 1.30313186, -0.21552947,  0.14927745],
           [ 1.52355424, -0.34800134,  0.15975065],
           [ 1.64146459,  0.74351599,  0.30026933],
           [ 0.02198455,  0.00355557, -0.82329652],
           [ 0.76796136, -0.21636589,  0.13481129]])

  • centiles
    (centile, observations, response_vars)
    float64
    152.3 904.6 ... 1.334e+04 2.469e+04

    array([[[ 1.52348837e+02,  9.04557999e+02,  1.65032223e+04],
            [ 2.73876824e+01, -2.05497135e+01,  1.64703336e+04],
            [ 2.13436546e+02,  1.35692658e+03,  1.65189118e+04],
            [ 2.50875616e+02,  1.63421273e+03,  1.65284010e+04],
            [-5.43642089e+01, -6.25585397e+02,  1.64482432e+04],
            [ 2.13607561e+02,  1.35819311e+03,  1.65189554e+04],
            [-1.41596036e+01, -3.28054464e+02,  1.64591635e+04],
            [ 8.09049868e+01,  3.75605185e+02,  1.64845491e+04],
            [ 1.43432619e+01, -1.17099743e+02,  1.64668392e+04],
            [ 5.38100039e+01,  1.75030170e+02,  1.64773764e+04]],
    
           [[ 7.76050569e+02,  4.78668536e+03,  1.89251183e+04],
            [ 6.51236372e+02,  3.86249247e+03,  1.88928002e+04],
            [ 8.37129588e+02,  5.23899985e+03,  1.89407741e+04],
            [ 8.74583918e+02,  5.51638098e+03,  1.89503226e+04],
            [ 5.69674000e+02,  3.25863657e+03,  1.88714456e+04],
            [ 8.37300637e+02,  5.24026658e+03,  1.89408178e+04],
            [ 6.09776206e+02,  3.55553005e+03,  1.88819683e+04],
            [ 7.04669551e+02,  4.25812368e+03,  1.89066891e+04],
            [ 6.38217254e+02,  3.76609995e+03,  1.88894040e+04],
    ...
            [ 1.51849773e+03,  9.26061624e+03,  2.22604624e+04],
            [ 1.70417457e+03,  1.06357766e+04,  2.23075963e+04],
            [ 1.74165011e+03,  1.09132898e+04,  2.23172271e+04],
            [ 1.43719882e+03,  8.65840044e+03,  2.22401307e+04],
            [ 1.70434567e+03,  1.06370437e+04,  2.23076402e+04],
            [ 1.47715868e+03,  8.95440775e+03,  2.22501007e+04],
            [ 1.57181396e+03,  9.65551942e+03,  2.22738973e+04],
            [ 1.50551379e+03,  9.16444269e+03,  2.22572028e+04],
            [ 1.54481131e+03,  9.45551922e+03,  2.22670830e+04]],
    
           [[ 2.26680936e+03,  1.40656647e+04,  2.47138834e+04],
            [ 2.14234642e+03,  1.31436584e+04,  2.46829290e+04],
            [ 2.32786761e+03,  1.45178499e+04,  2.47294586e+04],
            [ 2.36535841e+03,  1.47954581e+04,  2.47391486e+04],
            [ 2.06123703e+03,  1.25426224e+04,  2.46633330e+04],
            [ 2.32803874e+03,  1.45191172e+04,  2.47295026e+04],
            [ 2.10109448e+03,  1.28379923e+04,  2.46729055e+04],
            [ 2.19557852e+03,  1.35380379e+04,  2.46960372e+04],
            [ 2.12938778e+03,  1.30476424e+04,  2.46797676e+04],
            [ 2.16861451e+03,  1.33382782e+04,  2.46893729e+04]]])

  • logp
    (observations, response_vars)
    float64
    -1.158 -1.227 ... -0.8818 -0.9294

    array([[-1.15822243, -1.22697976, -0.92035109],
           [-0.98205551, -1.09477916, -0.95144874],
           [-0.90784404, -0.89169379, -1.00377414],
           [-1.21310502, -0.92089526, -0.92209595],
           [-1.37684181, -0.8595082 , -1.5779446 ],
           [-1.73177179, -0.8814415 , -0.9312394 ],
           [-2.04369335, -0.91915665, -0.93324669],
           [-2.23001309, -1.13473767, -0.96529295],
           [-0.88322716, -0.85851141, -1.25929608],
           [-1.17775434, -0.88179869, -0.92936108]])

  • Yhat
    (observations, response_vars)
    float64
    1.21e+03 7.485e+03 ... 2.058e+04

    array([[ 1209.5790997 ,  7485.11135948, 20608.55283429],
           [ 1084.86705068,  6561.55435498, 20576.63127352],
           [ 1270.65207844,  7937.38824645, 20624.18520356],
           [ 1308.11701498,  8214.83541241, 20633.77480824],
           [ 1003.43641236,  5958.51850568, 20555.7881145 ],
           [ 1270.82315121,  7938.65512849, 20624.22899171],
           [ 1043.46744044,  6254.96890219, 20566.03454142],
           [ 1138.24175479,  6956.82155032, 20590.29317608],
           [ 1071.86552019,  6465.27132022, 20573.30337418],
           [ 1111.21225719,  6756.65418858, 20583.3746485 ]])

  • statistics
    (response_vars, statistic)
    float64
    0.1172 0.04509 ... 0.9995 0.9965

    array([[ 1.17178773e-01,  4.50929368e-02,  3.98458621e-01,
            -6.56498543e+00,  1.35593870e+00,  1.17178773e-01,
             6.26199688e+02,  4.38152477e-03,  8.85849246e-01,
             8.82821227e-01,  8.47148187e-01],
           [ 1.57907407e-01,  5.24163569e-02,  4.10913989e-01,
            -8.44084535e+00,  1.33302553e+00,  1.57907407e-01,
             3.90104480e+03,  2.56092410e-01,  1.42121630e-17,
             8.42092593e-01,  8.87533011e-01],
           [ 5.48906457e-04,  4.38661710e-03,  9.77076127e-02,
            -7.82151261e+00,  1.41864321e+00,  5.48906457e-04,
             2.49225388e+03,  6.47556344e-02,  3.36782367e-02,
             9.99451094e-01,  9.96466623e-01]])

  • Y_harmonized
    (observations, response_vars)
    float64
    1.687e+03 4.049e+03 ... 2.092e+04

    array([[ 1686.7,  4049.4, 20663.2],
           [ 1371.1,  9312.6, 19954. ],
           [ 1414.8,  8972.6, 21645.2],
           [ 1830.6,  6798.6, 20790.6],
           [ 1642.4,  6112.5, 17692.6],
           [ 2108.4,  7076.4, 20996.8],
           [ 2023.1,  4862.2, 20964.9],
           [ 2193.4,  9931.7, 21339.8],
           [ 1086. ,  6479.5, 18517.9],
           [ 1604.9,  5890.9, 20919.9]])

• Indexes: (6)
  • observations
    PandasIndex

    PandasIndex(Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64', name='observations'))

  • response_vars
    PandasIndex

    PandasIndex(Index(['WM-hypointensities', 'Left-Lateral-Ventricle', 'Brain-Stem'], dtype='object', name='response_vars'))

  • covariates
    PandasIndex

    PandasIndex(Index(['age'], dtype='object', name='covariates'))

  • batch_effect_dims
    PandasIndex

    PandasIndex(Index(['sex', 'site'], dtype='object', name='batch_effect_dims'))

  • centile
    PandasIndex

    PandasIndex(Index([0.05, 0.25, 0.5, 0.75, 0.95], dtype='float64', name='centile'))

  • statistic
    PandasIndex

    PandasIndex(Index(['EXPV', 'MACE', 'MAPE', 'MSLL', 'NLL', 'R2', 'RMSE', 'Rho', 'Rho_p',
           'SMSE', 'ShapiroW'],
          dtype='object', name='statistic'))

• Attributes: (7)

  real_ids :

  False

  is_scaled :

  False

  name :

  fcon1000

  unique_batch_effects :

  {np.str_('sex'): [np.str_('0'), np.str_('1')], np.str_('site'): [np.str_('AnnArbor_a'), np.str_('AnnArbor_b'), np.str_('Atlanta'), np.str_('Baltimore'), np.str_('Bangor'), np.str_('Beijing_Zang'), np.str_('Berlin_Margulies'), np.str_('Cambridge_Buckner'), np.str_('Cleveland'), np.str_('ICBM'), np.str_('Leiden_2180'), np.str_('Leiden_2200'), np.str_('Milwaukee_b'), np.str_('Munchen'), np.str_('NewYork_a'), np.str_('NewYork_a_ADHD'), np.str_('Newark'), np.str_('Oulu'), np.str_('Oxford'), np.str_('PaloAlto'), np.str_('Pittsburgh'), np.str_('Queensland'), np.str_('SaintLouis')]}

  batch_effect_counts :

  defaultdict(<function NormData.register_batch_effects.<locals>.<lambda> at 0x1685104a0>, {np.str_('sex'): {np.str_('0'): 587, np.str_('1'): 489}, np.str_('site'): {np.str_('AnnArbor_a'): 24, np.str_('AnnArbor_b'): 32, np.str_('Atlanta'): 28, np.str_('Baltimore'): 23, np.str_('Bangor'): 20, np.str_('Beijing_Zang'): 198, np.str_('Berlin_Margulies'): 26, np.str_('Cambridge_Buckner'): 198, np.str_('Cleveland'): 31, np.str_('ICBM'): 83, np.str_('Leiden_2180'): 12, np.str_('Leiden_2200'): 19, np.str_('Milwaukee_b'): 46, np.str_('Munchen'): 15, np.str_('NewYork_a'): 83, np.str_('NewYork_a_ADHD'): 25, np.str_('Newark'): 19, np.str_('Oulu'): 102, np.str_('Oxford'): 22, np.str_('PaloAlto'): 17, np.str_('Pittsburgh'): 3, np.str_('Queensland'): 19, np.str_('SaintLouis'): 31}})

  covariate_ranges :

  {np.str_('age'): {'mean': np.float64(28.160613382899626), 'min': np.float64(7.88), 'max': np.float64(79.0)}}

  batch_effect_covariate_ranges :

  {np.str_('sex'): {np.str_('0'): {np.str_('age'): {'mean': np.float64(27.88655877342419), 'min': np.float64(7.88), 'max': np.float64(79.0)}}, np.str_('1'): {np.str_('age'): {'mean': np.float64(28.48959100204499), 'min': np.float64(9.21), 'max': np.float64(78.0)}}}, np.str_('site'): {np.str_('AnnArbor_a'): {np.str_('age'): {'mean': np.float64(21.28333333333333), 'min': np.float64(13.41), 'max': np.float64(40.98)}}, np.str_('AnnArbor_b'): {np.str_('age'): {'mean': np.float64(44.40625), 'min': np.float64(19.0), 'max': np.float64(79.0)}}, np.str_('Atlanta'): {np.str_('age'): {'mean': np.float64(30.892857142857142), 'min': np.float64(22.0), 'max': np.float64(57.0)}}, np.str_('Baltimore'): {np.str_('age'): {'mean': np.float64(29.26086956521739), 'min': np.float64(20.0), 'max': np.float64(40.0)}}, np.str_('Bangor'): {np.str_('age'): {'mean': np.float64(23.4), 'min': np.float64(19.0), 'max': np.float64(38.0)}}, np.str_('Beijing_Zang'): {np.str_('age'): {'mean': np.float64(21.161616161616163), 'min': np.float64(18.0), 'max': np.float64(26.0)}}, np.str_('Berlin_Margulies'): {np.str_('age'): {'mean': np.float64(29.76923076923077), 'min': np.float64(23.0), 'max': np.float64(44.0)}}, np.str_('Cambridge_Buckner'): {np.str_('age'): {'mean': np.float64(21.03030303030303), 'min': np.float64(18.0), 'max': np.float64(30.0)}}, np.str_('Cleveland'): {np.str_('age'): {'mean': np.float64(43.54838709677419), 'min': np.float64(24.0), 'max': np.float64(60.0)}}, np.str_('ICBM'): {np.str_('age'): {'mean': np.float64(43.25301204819277), 'min': np.float64(19.0), 'max': np.float64(79.0)}}, np.str_('Leiden_2180'): {np.str_('age'): {'mean': np.float64(23.0), 'min': np.float64(20.0), 'max': np.float64(27.0)}}, np.str_('Leiden_2200'): {np.str_('age'): {'mean': np.float64(21.68421052631579), 'min': np.float64(18.0), 'max': np.float64(28.0)}}, np.str_('Milwaukee_b'): {np.str_('age'): {'mean': np.float64(53.58695652173913), 'min': np.float64(44.0), 'max': np.float64(65.0)}}, np.str_('Munchen'): {np.str_('age'): {'mean': np.float64(68.13333333333334), 'min': np.float64(63.0), 'max': np.float64(74.0)}}, np.str_('NewYork_a'): {np.str_('age'): {'mean': np.float64(24.507710843373495), 'min': np.float64(7.88), 'max': np.float64(49.16)}}, np.str_('NewYork_a_ADHD'): {np.str_('age'): {'mean': np.float64(34.9952), 'min': np.float64(20.69), 'max': np.float64(50.9)}}, np.str_('Newark'): {np.str_('age'): {'mean': np.float64(24.105263157894736), 'min': np.float64(21.0), 'max': np.float64(39.0)}}, np.str_('Oulu'): {np.str_('age'): {'mean': np.float64(21.519607843137255), 'min': np.float64(20.0), 'max': np.float64(23.0)}}, np.str_('Oxford'): {np.str_('age'): {'mean': np.float64(29.0), 'min': np.float64(20.0), 'max': np.float64(35.0)}}, np.str_('PaloAlto'): {np.str_('age'): {'mean': np.float64(32.470588235294116), 'min': np.float64(22.0), 'max': np.float64(46.0)}}, np.str_('Pittsburgh'): {np.str_('age'): {'mean': np.float64(32.333333333333336), 'min': np.float64(25.0), 'max': np.float64(47.0)}}, np.str_('Queensland'): {np.str_('age'): {'mean': np.float64(25.94736842105263), 'min': np.float64(20.0), 'max': np.float64(34.0)}}, np.str_('SaintLouis'): {np.str_('age'): {'mean': np.float64(25.096774193548388), 'min': np.float64(21.0), 'max': np.float64(29.0)}}}}

Or, if we want to select only the WM-hypointensities, we can use:

norm_data.sel(response_vars="WM-hypointensities")

This will return a new NormData object with only the WM-hypointensities.

norm_data.sel(response_vars="WM-hypointensities").statistics

<xarray.DataArray 'statistics' (statistic: 11)> Size: 88B
array([ 1.17178773e-01,  4.50929368e-02,  3.98458621e-01, -6.56498543e+00,
        1.35593870e+00,  1.17178773e-01,  6.26199688e+02,  4.38152477e-03,
        8.85849246e-01,  8.82821227e-01,  8.47148187e-01])
Coordinates:
    response_vars  <U22 88B 'WM-hypointensities'
  * statistic      (statistic) <U8 352B 'EXPV' 'MACE' ... 'SMSE' 'ShapiroW'

xarray.DataArray

'statistics'

• statistic: 11

• 0.1172 0.04509 0.3985 -6.565 1.356 ... 0.004382 0.8858 0.8828 0.8471

  array([ 1.17178773e-01,  4.50929368e-02,  3.98458621e-01, -6.56498543e+00,
          1.35593870e+00,  1.17178773e-01,  6.26199688e+02,  4.38152477e-03,
          8.85849246e-01,  8.82821227e-01,  8.47148187e-01])

• Coordinates: (2)
  • response_vars
    ()
    <U22
    'WM-hypointensities'

    array('WM-hypointensities', dtype='<U22')

  • statistic
    (statistic)
    <U8
    'EXPV' 'MACE' ... 'SMSE' 'ShapiroW'

    array(['EXPV', 'MACE', 'MAPE', 'MSLL', 'NLL', 'R2', 'RMSE', 'Rho', 'Rho_p',
           'SMSE', 'ShapiroW'], dtype='<U8')

• Indexes: (1)
  • statistic
    PandasIndex

    PandasIndex(Index(['EXPV', 'MACE', 'MAPE', 'MSLL', 'NLL', 'R2', 'RMSE', 'Rho', 'Rho_p',
           'SMSE', 'ShapiroW'],
          dtype='object', name='statistic'))

• Attributes: (0)

Now we can use the to_dataframe method to cast that selection back to a pandas dataframe.

new_df = norm_data.sel(response_vars="WM-hypointensities").to_dataframe()
new_df.head()

	X	Y	Y_harmonized	Z	batch_effects		subject_ids	centiles
	age	WM-hypointensities	WM-hypointensities	WM-hypointensities	sex	site	subject_ids	(WM-hypointensities, 0.05)	(WM-hypointensities, 0.25)	(WM-hypointensities, 0.5)	(WM-hypointensities, 0.75)	(WM-hypointensities, 0.95)
0	25.63	1686.7	1686.7	0.742311	1	AnnArbor_a	0	152.348837	776.050569	1209.579100	1643.107630	2266.809363
1	18.34	1371.1	1371.1	0.445220	1	AnnArbor_a	1	27.387682	651.236372	1084.867051	1518.497730	2142.346419
2	29.20	1414.8	1414.8	0.224270	1	AnnArbor_a	2	213.436546	837.129588	1270.652078	1704.174569	2327.867611
3	31.39	1830.6	1830.6	0.812878	1	AnnArbor_a	3	250.875616	874.583918	1308.117015	1741.650112	2365.358414
4	13.58	1642.4	1642.4	0.993572	1	AnnArbor_a	4	-54.364209	569.674000	1003.436412	1437.198824	2061.237034

This should give you a pretty good overview of how to work with NormData. Most of the functionality is built on top of xarray, so if you want to learn more about xarray, you can check out the xarray documentation. However, the Xarray.DataSet class does not officially support being extended, so the API does not work completely as expected.

If you have any suggestions for improvements, please let us know!

Pre-processing and split datasets

Sometimes we have a dataset that is pre-split into train and test, and we want to use that exact data split to fit the model. We can then load the data into two NormData objects, but we have to make sure that the two datasets are compatible. This will ensure that the fitted model is applicable to both of them.

# Download an example dataset:
data = pd.read_csv(
    "https://raw.githubusercontent.com/predictive-clinical-neuroscience/PCNtoolkit-demo/refs/heads/main/data/fcon1000.csv"
)
# Create an arbitrary split as a placeholder for a predefined split ()
train, test = train_test_split(data, test_size=100)

# specify the column names to use
covariates = ["age"]
batch_effects = ["sex", "site"]
response_vars = ["WM-hypointensities", "Left-Lateral-Ventricle", "Brain-Stem"]

# create NormData objects
norm_train = NormData.from_dataframe(
    name="train", dataframe=train, covariates=covariates, batch_effects=batch_effects, response_vars=response_vars
)
norm_test = NormData.from_dataframe(
    name="test", dataframe=test, covariates=covariates, batch_effects=batch_effects, response_vars=response_vars
)

Process: 75157 - 2025-11-20 13:19:02 - Dataset "train" created.
    - 978 observations
    - 978 unique subjects
    - 1 covariates
    - 3 response variables
    - 2 batch effects:
            sex (2)
    site (23)

Process: 75157 - 2025-11-20 13:19:02 - Dataset "test" created.
    - 100 observations
    - 100 unique subjects
    - 1 covariates
    - 3 response variables
    - 2 batch effects:
            sex (2)
    site (18)

# Should print false, because the train and test split do not contain the same sites
print(norm_train.check_compatibility(norm_test))

True

norm_train.check_compatibility(norm_test)

True

norm_train.make_compatible(norm_test)

norm_train.check_compatibility(norm_test)

True