popmon.alerting package

Submodules

popmon.alerting.alerts_summary module

class popmon.alerting.alerts_summary.AlertsSummary(read_key, store_key='', features=None, ignore_features=None, combined_variable='_AGGREGATE_')

Bases: Module

The module AlertsSummary combines the alerts-summaries of all individual features

It combines the alerts-summaries of all individual features into an artificial feature “_AGGREGATE_”.

__init__(read_key, store_key='', features=None, ignore_features=None, combined_variable='_AGGREGATE_')

Initialize an instance of AlertsSummary module.

Parameters

  • read_key (str) – key of input data to read from datastore.

  • store_key (str) – key of output data to store in datastore (optional).

  • combined_variable (str) – name of artificial variable that combines all alerts. default is ‘_AGGREGATE_’.

  • features (list) – features of data frames to pick up from input data (optional).

  • ignore_features (list) – list of features to ignore (optional).

transform(data)

Central function of the module.

Typically transform() takes something from the datastore, does something to it, and puts the results back into the datastore again, to be passed on to the next module in the pipeline.

Parameters

datastore (dict) – input datastore

Returns

updated output datastore

Return type

dict

popmon.alerting.compute_tl_bounds module

class popmon.alerting.compute_tl_bounds.ComputeTLBounds(read_key, monitoring_rules=None, store_key='', features=None, ignore_features=None, apply_funcs_key='', func=None, metrics_wide=False, prefix='traffic_light_', suffix='', entire=False, **kwargs)

Bases: Module

Expand traffic light boundaries over all features and metrics

The module extracts the monitoring rules for the traffic light bounds for specific features/metrics and if those are not found - monitoring rules are extracted from the wildcard rules which are meant to be generic. Then bounds can be stored as either raw values or as directly calculated values on the statistics of the data.

__init__(read_key, monitoring_rules=None, store_key='', features=None, ignore_features=None, apply_funcs_key='', func=None, metrics_wide=False, prefix='traffic_light_', suffix='', entire=False, **kwargs)

Initialize an instance of TrafficLightBounds module.

Parameters

  • read_key (str) – key of input data to read from datastore

  • store_key (str) – key of output data to store in datastore (optional)

  • monitoring_rules (dict) – dict of traffic light bounds => key is feature:metric and value is bounds

  • features (list) – features of data frames to pick up from input data (optional)

  • ignore_features (list) – list of features to ignore (optional)

  • apply_funcs_key (str) – key of to-be-applied traffic light functions in data to store (optional)

  • func – traffic light function to apply (optional)

  • metrics_wide (bool) – if true, select wide set of metrics from columns. default is false (optional)

  • prefix (str) – prefix for traffic light variables. default is tl_ (optional)

  • suffix (str) – suffix for traffic light variables. default is ‘’ (optional)

  • entire (bool) – if True, apply function to the entire feature’s dataframe of metrics (optional)

  • kwargs – residual kwargs are passed as kwargs to ‘func’ (dict, optional)

get_description()

Returns the traffic light function as description.

transform(test_data)

Central function of the module.

Typically transform() takes something from the datastore, does something to it, and puts the results back into the datastore again, to be passed on to the next module in the pipeline.

Parameters

datastore (dict) – input datastore

Returns

updated output datastore

Return type

dict

class popmon.alerting.compute_tl_bounds.DynamicBounds(read_key, rules, store_key='', suffix_mean='_mean', suffix_std='_std')

Bases: Pipeline

Calculate dynamic traffic light bounds based on pull thresholds and dynamic mean and std.deviation.

__init__(read_key, rules, store_key='', suffix_mean='_mean', suffix_std='_std')

Initialize an instance of DynamicTrafficLightBounds.

Parameters

  • read_key (str) – key of input data to read from data store, only used to extract feature list.

  • rules (dict) – dict of traffic light bounds => key is feature:metric and value is bounds

  • store_key (str) – key of dynamic traffic light boundaries to store in data store, e.g. used for plotting.

  • suffix_mean (str) – suffix of mean. mean column = metric + suffix_mean

  • suffix_std (str) – suffix of std. std column = metric + suffix_std

transform(datastore)

Central function of the pipeline.

Calls transform() of each module in the pipeline. Typically, transform() of a module takes something from the datastore, does something to it, and puts the results back into the datastore again, to be passed on to the next module in the pipeline.

Parameters

datastore (dict) – input datastore

Returns

updated output datastore

Return type

dict

class popmon.alerting.compute_tl_bounds.StaticBounds(read_key, rules, store_key='', suffix_mean='_mean', suffix_std='_std')

Bases: Pipeline

Calculate static traffic light bounds based on pull thresholds and static mean and std.deviation.

__init__(read_key, rules, store_key='', suffix_mean='_mean', suffix_std='_std')

Initialize an instance of StaticBounds.

Parameters

  • read_key (str) – key of input data to read from data store, only used to extract feature list.

  • rules (dict) – dict of traffic light bounds => key is feature:metric and value is bounds

  • store_key (str) – key of dynamic traffic light boundaries to store in data store, e.g. used for plotting.

  • suffix_mean (str) – suffix of mean. mean column = metric + suffix_mean

  • suffix_std (str) – suffix of std. std column = metric + suffix_std

transform(datastore)

Central function of the pipeline.

Calls transform() of each module in the pipeline. Typically, transform() of a module takes something from the datastore, does something to it, and puts the results back into the datastore again, to be passed on to the next module in the pipeline.

Parameters

datastore (dict) – input datastore

Returns

updated output datastore

Return type

dict

class popmon.alerting.compute_tl_bounds.TrafficLightAlerts(read_key, store_key, rules, expanded_rules_key='')

Bases: Pipeline

Evaluate the traffic light alerts

Steps:

  • Generate static traffic light bounds by expanding the wildcarded monitoring rules

  • Apply them to profiled test statistics data

__init__(read_key, store_key, rules, expanded_rules_key='')

Initialize an instance of TrafficLightBounds.

Parameters

  • read_key (str) – key of input data to read from data store, only used to extract feature list.

  • store_key (str) – results of traffic light bounds after applied to input data, to store in data store.

  • rules (dict) – input dict of wildcard traffic light bounds => key is feature:metric and value is bounds

  • expanded_rules_key (str) – store key of expanded monitoring rules to store in data store, eg. these can be used for plotting. (optional)

transform(datastore)

Central function of the pipeline.

Calls transform() of each module in the pipeline. Typically, transform() of a module takes something from the datastore, does something to it, and puts the results back into the datastore again, to be passed on to the next module in the pipeline.

Parameters

datastore (dict) – input datastore

Returns

updated output datastore

Return type

dict

popmon.alerting.compute_tl_bounds.collect_traffic_light_bounds(monitoring_rules)

Collect traffic light boundaries

Splits generic metrics and metrics which belong to particular features and puts them in separate objects.

Parameters

monitoring_rules (dict) – dictionary of defined monitoring rules (bounds)

Returns

dict metrics_per_feature (containing features with corresponding metrics), list metrics: list of generic metrics

popmon.alerting.compute_tl_bounds.df_single_op_pull_bounds(df, red_high, yellow_high, yellow_low=0, red_low=0, suffix_mean='_mean', suffix_std='_std', cols=None)

Calculate traffic light pull bounds for list of cols on first row only

Function can be used with ApplyFunc module.

Parameters

  • df (pd.DataFrame) – df to apply this function to

  • red_high (float) – higher bound of red traffic light

  • yellow_high (float) – higher bound of yellow traffic light

  • yellow_low (float) – lower bound of yellow traffic light (optional)

  • red_low (float) – lower bound of red traffic light (optional)

  • suffix_mean (str) – suffix of mean column. default is ‘_mean’ (optional)

  • suffix_std (str) – suffix of std column. default is ‘_std’ (optional)

  • cols (list) – list of cols to calculate bounds of (optional)

popmon.alerting.compute_tl_bounds.pull_bounds(row, red_high, yellow_high, yellow_low=0, red_low=0, suffix_mean='_mean', suffix_std='_std', cols=None)

Calculate traffic light pull bounds for list of cols

Function can be used with ApplyFunc module.

Parameters

  • row (pd.Series) – row to apply this function to.

  • red_high (float) – higher bound of red traffic light

  • yellow_high (float) – higher bound of yellow traffic light

  • yellow_low (float) – lower bound of yellow traffic light (optional)

  • red_low (float) – lower bound of red traffic light (optional)

  • suffix_mean (str) – suffix of mean column. default is ‘_mean’ (optional)

  • suffix_std (str) – suffix of std column. default is ‘_std’ (optional)

  • cols (list) – list of cols to calculate bounds of (optional)

popmon.alerting.compute_tl_bounds.traffic_light(value, red_high, yellow_high, yellow_low=0, red_low=0)

Get the corresponding traffic light given a value and traffic light bounds.

Parameters

  • value (float) – value to check

  • red_high (float) – higher bound of red traffic light

  • yellow_high (float) – higher bound of yellow traffic light

  • yellow_low (float) – lower bound of yellow traffic light (optional)

  • red_low (float) – lower bound of red traffic light (optional)

Return type

int

Returns

corresponding traffic light based on the value and traffic light bounds

popmon.alerting.compute_tl_bounds.traffic_light_summary(row, cols=None, prefix='')

Make a summary of traffic light alerts present in the dataframe

Count number of green, yellow and red traffic lights.

Evaluate with df.apply(traffic_light_summary, axis=1)

Parameters

  • row (pd.Series) – row to calculate traffic light summary of.

  • cols (list) – list of cols to calculate traffic light summary of (optional)

  • prefix (str) – prefix of traffic light columns, in case cols is empty. default is "tl_"