popmon.analysis package

Subpackages

Submodules

popmon.analysis.apply_func module

class popmon.analysis.apply_func.ApplyFunc(apply_to_key, store_key='', assign_to_key='', apply_funcs_key='', features=None, apply_funcs=None, metrics=None, msg='')

Bases: Module

This module applies functions to specified feature and metrics.

Extra parameters (kwargs) can be passed to the apply function.

__init__(apply_to_key, store_key='', assign_to_key='', apply_funcs_key='', features=None, apply_funcs=None, metrics=None, msg='')

Initialize an instance of ApplyFunc.

Parameters

  • apply_to_key (str) – key of the input data to apply funcs to.

  • assign_to_key (str) – key of the input data to assign function applied-output to. (optional)

  • store_key (str) – key of the output data to store in the datastore (optional)

  • apply_funcs_key (str) – key of to-be-applied functions in data to store (optional)

  • features (list) – list of features to pick up from input data and apply funcs to (optional)

  • metrics (list) – list of metrics to apply funcs to (optional)

  • msg (str) – message to print out at start of transform function. (optional)

  • apply_funcs (list) –

    functions to apply (list of dicts):

    • ’func’: function to apply

    • ’suffix’ (string, optional): suffix added to each metric. default is function name.

    • ’prefix’ (string, optional): prefix added to each metric.

    • ’features’ (list, optional): features the function is applied to. Overwrites features above

    • ’metrics’ (list, optional): metrics the function is applied to. Overwrites metrics above

    • ’entire’ (boolean, optional): apply function to the entire feature’s dataframe of metrics?

    • ’args’ (tuple, optional): args for ‘func’

    • ’kwargs’ (dict, optional): kwargs for ‘func’

add_apply_func(func, suffix=None, prefix=None, metrics=None, features=None, entire=None, *args, **kwargs)

Add function to be applied to dataframe.

Can call this function after module instantiation to add new functions.

Parameters

  • func – function to apply

  • suffix – (string, optional) suffix added to each metric. default is function name.

  • prefix – (string, optional) prefix added to each metric.

  • features – (list, optional) features the function is applied to. Overwrites features above

  • metrics – (list, optional) metrics the function is applied to. Overwrites metrics above

  • entire – (boolean, optional) apply function to the entire feature’s dataframe of metrics?

  • args – (tuple, optional) args for ‘func’

  • kwargs – (dict, optional) kwargs for ‘func’

transform(apply_to_data, assign_to_data=None, apply_funcs=None)

Apply functions to specified feature and metrics

Each feature/metric combination is treated as a pandas series

Parameters

datastore – input datastore

Returns

updated datastore

Return type

dict

popmon.analysis.apply_func.apply_func(feature, selected_metrics, df, arr)

Apply function to dataframe

Parameters

  • feature (str) – feature currently looping over

  • selected_metrics (list) – list of selected metrics to apply to

  • df – pandas data frame that function in arr is applied to

  • arr (dict) – dictionary containing the function to be applied to pandas dataframe.

Returns

dictionary with outputs of applied-to metric pd.Series

popmon.analysis.apply_func.apply_func_array(feature, metrics, apply_to_df, assign_to_df, apply_funcs, same_key)

Apply list of functions to dataframe

Split off for parallelization reasons

Parameters

  • feature (str) – feature currently looping over

  • metrics (list) – list of selected metrics to apply functions to

  • apply_to_df – pandas data frame that function in arr is applied to

  • assign_to_df – pandas data frame the output of function is assigned to

  • apply_funcs – list of functions to apply to

  • same_key – if True, merge apply_to_df and assign_to_df before returning assign_to_df

Returns

union of feature and assign_to_df

popmon.analysis.functions module

popmon.analysis.functions.expand(df, shift=1)

Implementation of expanding window that can handle non-numerical values such as histograms

Split up input array into expanding sub-arrays

Parameters

  • df (pd.DataFrame) – input dataframe to apply rolling function to.

  • shift (int) – shift of dataframe, default is 1 (optional)

  • fillvalue – default value to fill dataframe in case shift > 0 (optional)

popmon.analysis.functions.expand_norm_hist_mean_cov(df, shift=1, *args, **kwargs)

Apply expanding normalized_hist_mean_cov function

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe with column of histograms

  • shift (int) – shift of dataframe, default is 1 (optional)

  • args – args passed on to hist_sum function

  • kwargs – kwargs passed on to hist_sum function

Returns

dataframe with expanding normalized_hist_mean_cov results

popmon.analysis.functions.expanding_apply(df, func, shift=1, *args, **kwargs)

Calculate expanding apply() to all columns of a pandas dataframe

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe

  • func – function to be applied

  • shift (int) – size of shift. default is 1.

  • args – args passed on to function

  • kwargs – kwargs passed on to function

Returns

df with expanding results of function applied to all columns

popmon.analysis.functions.expanding_hist(df, shift=1, *args, **kwargs)

Apply expanding histogram sum

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe with column of histograms

  • shift (int) – shift of dataframe, default is 1 (optional)

  • args – args passed on to hist_sum function

  • kwargs – kwargs passed on to hist_sum function

Returns

dataframe with expanding hist_sum results

popmon.analysis.functions.expanding_mean(df, shift=1)

Calculate expanding mean of all numeric columns of a pandas dataframe

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe

  • shift (int) – size of shift. default is 1.

Returns

df with expanding means of columns

popmon.analysis.functions.expanding_std(df, shift=1)

Calculate expanding std of all numeric columns of a pandas dataframe

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe

  • shift (int) – size of shift. default is 1.

Returns

df with expanding std of columns

popmon.analysis.functions.hist_sum(x, hist_name='')

Return sum of histograms

Usage: df[‘hists’].apply(hist_sum) ; series.apply(hist_sum)

Parameters

  • x (pd.Series) – pandas series to extract histogram list from.

  • hist_name (str) – name of column to extract histograms from. needs to be set with axis=1 (optional)

Returns

sum histogram

popmon.analysis.functions.normalized_hist_mean_cov(x, hist_name='')

Mean normalized histogram and its covariance of list of input histograms

Usage: df[‘hists’].apply(normalized_hist_mean_cov) ; series.apply(normalized_hist_mean_cov)

Parameters

  • x (pd.Series) – pandas series to extract histogram list from.

  • hist_name (str) – name of column to extract histograms from. needs to be set with axis=1 (optional)

Returns

mean normalized histogram, covariance probability matrix

popmon.analysis.functions.pull(row, suffix_mean='_mean', suffix_std='_std', cols=None)

Calculate normalized residual (pull) for list of cols

Function can be used by ApplyFunc module.

Parameters

  • row (pd.Series) – row to apply pull function to

  • cols (list) – list of cols to calculate pull of

  • suffix_mean (str) – suffix of mean. mean column = metric + suffix_mean

  • suffix_std (str) – suffix of std. std column = metric + suffix_std

popmon.analysis.functions.relative_chi_squared(row, hist_name='histogram', suffix_mean='_mean', suffix_cov='_cov', suffix_binning='_binning')

Calculate chi squared of normalized histogram with pre-calculated mean normalized histogram

Parameters

  • row (pd.Series) – row to apply chi_squared function to.

  • hist_name (str) – name of column to extract histograms from. default is ‘histogram’ (optional)

  • suffix_mean (str) – suffix of mean. mean column = hist_name + suffix_mean (optional)

  • suffix_std (str) – suffix of std. std column = hist_name + suffix_std (optional)

  • suffix_binning (str) – suffix of binning. binning column = hist_name + suffix_binning (optional)

popmon.analysis.functions.roll(df, window, shift=1)

Implementation of rolling window that can handle non-numerical columns such as histograms

Parameters

  • df (pd.DataFrame) – input dataframe to apply rolling function to.

  • window (int) – size of rolling window

  • shift (int) – shift of dataframe, default is 1 (optional)

popmon.analysis.functions.roll_norm_hist_mean_cov(df, window, shift=1, *args, **kwargs)

Apply rolling normalized_hist_mean_cov function

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe with column of histograms

  • window (int) – size of rolling window

  • shift (int) – shift of dataframe, default is 1 (optional)

  • args – args passed on to hist_sum function

  • kwargs – kwargs passed on to hist_sum function

Returns

dataframe with rolling normalized_hist_mean_cov results

popmon.analysis.functions.rolling_apply(df, window, func, shift=1, *args, **kwargs)

Calculate rolling apply() to all columns of a pandas dataframe

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe

  • window (int) – size of rolling window.

  • func – function to be applied

  • shift (int) – size of shift. default is 1.

  • args – args passed on to function

  • kwargs – kwargs passed on to function

Returns

df with rolling results of function applied to all columns

popmon.analysis.functions.rolling_hist(df, window, shift=1, *args, **kwargs)

Apply rolling histogram sum

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe with column of histograms

  • window (int) – size of rolling window

  • shift (int) – shift of dataframe, default is 1 (optional)

  • args – args passed on to hist_sum function

  • kwargs – kwargs passed on to hist_sum function

Returns

dataframe with rolling hist_sum results

popmon.analysis.functions.rolling_lr(df, window, index=0, shift=0)

Calculate rolling scipy lin_regress() to all columns of a pandas dataframe

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe

  • window (int) – size of rolling window.

  • index (int) – index of lin_regress results to return. default is 0.

  • shift (int) – size of shift. default is 0.

Returns

df with rolling results of lin_regress() function applied to all columns

popmon.analysis.functions.rolling_lr_zscore(df, window, shift=0)

Calculate rolling z-score of scipy lin_regress() to all columns of a pandas dataframe

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe

  • window (int) – size of rolling window.

  • shift (int) – size of shift. default is 0.

Returns

df with rolling z-score results of lin_regress() function applied to all columns

popmon.analysis.functions.rolling_mean(df, window, shift=1)

Calculate rolling mean of all numeric columns of a pandas dataframe

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe

  • shift (int) – size of shift. default is 1.

  • window (int) – size of rolling window.

Returns

df with rolling mean of columns

popmon.analysis.functions.rolling_std(df, window, shift=1)

Calculate rolling std of all numeric columns of a pandas dataframe

Function can be used by ApplyFunc module.

Parameters

  • df (pd.DataFrame) – input pandas dataframe

  • shift (int) – size of shift. default is 1.

  • window (int) – size of rolling window.

Returns

df with rolling std of columns

popmon.analysis.hist_numpy module

popmon.analysis.hist_numpy.assert_similar_hists(hist_list, check_type=True, assert_type=(<class 'histogrammar.primitives.bin.Bin'>, <class 'histogrammar.primitives.sparselybin.SparselyBin'>, <class 'histogrammar.primitives.categorize.Categorize'>))

Assert consistent list of input histograms

Assert that type and dimension of all histograms in input list are the same.

Parameters

  • hist_list (list) – list of input histogram objects to check on consistency

  • assert_type (bool) – if true, also assert type consistency of histograms (besides n-dim and datatype).

popmon.analysis.hist_numpy.check_same_hists(hist1, hist2)

Check if two hists are the same

Parameters

  • hist1 – input histogram 1

  • hist2 – input histogram 2

Returns

boolean, true if two histograms are the same

popmon.analysis.hist_numpy.check_similar_hists(hist_list, check_type=True, assert_type=(<class 'histogrammar.primitives.bin.Bin'>, <class 'histogrammar.primitives.sparselybin.SparselyBin'>, <class 'histogrammar.primitives.categorize.Categorize'>))

Check consistent list of input histograms

Check that type and dimension of all histograms in input list are the same.

Parameters

  • hist_list (list) – list of input histogram objects to check on consistency

  • check_type (bool) – if true, also check type consistency of histograms (besides n-dim and datatype).

Returns

bool indicating if lists are similar

popmon.analysis.hist_numpy.get_2dgrid(hist, get_bin_labels=False)

Get filled x,y grid of first two dimensions of input histogram

Parameters

hist – input histogrammar histogram

Returns

x,y grid of first two dimensions of input histogram

popmon.analysis.hist_numpy.get_consistent_numpy_1dhists(hist_list, get_bin_labels=False, crop_range=False)

Get list of consistent numpy hists for list of sparse (or bin) input histograms

Works for sparse and bin histograms. Note: for sparse histograms, all potential bins between low and high are picked up (also unfilled).

Note: a numpy histogram is a union of lists of bin_edges and number of entries This gives the full range of bin_centers, including zeros, which is not robust against (extreme) outliers. Ideally, use this for plotting of multiple histograms only.

Parameters

  • hist_list (list) – list of input histogram objects

  • get_bin_labels (bool) – return bin labels as well, default is false.

  • crop_range (bool) – return a trimmed version of the histogram, between 5-95% quantiles.

Returns

list of consistent 1d numpy hists for list of sparse input histograms

popmon.analysis.hist_numpy.get_consistent_numpy_2dgrids(hist_list=None, get_bin_labels=False)

Get list of consistent x,y grids of first two dimensions of (sparse) input histograms

Parameters

  • hist_list (list) – list of input histogrammar histograms

  • get_bin_labels (bool) – if true, return x-keys and y-keys describing binnings of 2d-grid.

Returns

list of consistent x,y grids of first two dimensions of each input histogram in list

popmon.analysis.hist_numpy.get_consistent_numpy_entries(hist_list, get_bin_labels=False)

Get list of consistent numpy bin_entries for list of 1d input histograms

Works for categorize, sparse and bin histograms. Note: for sparse histograms, only the filled bins are picked up. (this is not the case when calling get_consistent_numpy_1dhists(), which takes all bins b/n low and high.)

Parameters

hist_list (list) – list of input histogrammar histograms

Returns

list of consistent 1d numpy arrays with bin_entries for list of input histograms

popmon.analysis.hist_numpy.get_consistent_numpy_ndgrids(hist_list=None, get_bin_labels=False, dim=3)

Get list of consistent x,y grids of first n dimensions of (sparse) input histograms

Parameters

  • hist_list (list) – list of input histogrammar histograms

  • get_bin_labels (bool) – if true, return x-keys and y-keys describing binnings of 2d-grid.

  • dim (int) – number of dimension (>= 3)

Returns

list of consistent x,y grids of first two dimensions of each input histogram in list

popmon.analysis.hist_numpy.get_contentType(hist)

Get content type of bins of histogram

Parameters

hist – input histogram

Returns

string describing content type

popmon.analysis.hist_numpy.get_ndgrid(hist, get_bin_labels=False, n_dim=2)

Get filled n-d grid of first n dimensions of input histogram

Parameters

hist – input histogrammar histogram

Returns

grid of first n dimensions of input histogram

popmon.analysis.hist_numpy.prepare_2dgrid(hist)

Get lists of all unique x and y keys

Used as input by get_2dgrid(hist).

Parameters

hist – input histogrammar histogram

Returns

two sorted lists of unique x and y keys

popmon.analysis.hist_numpy.prepare_ndgrid(hist, n_dim)

Get lists of all unique combinations of keys

Used as input by get_ndgrid(hist).

Parameters

hist – input histogrammar histogram

Returns

n sorted lists of keys

popmon.analysis.hist_numpy.set_2dgrid(hist, keys)

Set 2d grid of first two dimensions of input histogram

Used as input by get_2dgrid(hist).

Parameters

  • hist – input histogrammar histogram

  • xkeys (list) – list with unique x keys

  • ykeys (list) – list with unique y keys

Returns

filled 2d numpy grid

popmon.analysis.hist_numpy.set_ndgrid(hist, keys, n_dim)

Set n-d grid of first n dimensions of input histogram

Used as input by get_ndgrid(hist).

Parameters

  • hist – input histogrammar histogram

  • keys (list) – list with unique keys per dim

Returns

filled nd numpy grid

popmon.analysis.merge_statistics module

class popmon.analysis.merge_statistics.MergeStatistics(read_keys, store_key)

Bases: Module

Merging dictionaries of features containing dataframes with statistics as its values.

__init__(read_keys, store_key)

Initialize an instance of MergeStatistics.

Parameters

  • read_keys (list) – list of keys of input data to read from the datastore

  • store_key (str) – key of output data to store in the datastore

transform(dicts)

Central function of the module.

Typically transform() takes something from the datastore, does something to it, and puts the results back into the datastore again, to be passed on to the next module in the pipeline.

Parameters

datastore (dict) – input datastore

Returns

updated output datastore

Return type

dict