popmon package
Subpackages
- popmon.alerting package
- popmon.analysis package
- popmon.base package
- popmon.extensions package
- popmon.hist package
- popmon.io package
- popmon.pipeline package
- Submodules
- popmon.pipeline.amazing_pipeline module
- popmon.pipeline.dataset_splitter module
- popmon.pipeline.metrics module
- popmon.pipeline.metrics_pipelines module
- popmon.pipeline.report module
- popmon.pipeline.report_pipelines module
- popmon.stats package
- popmon.stitching package
- popmon.visualization package
- Submodules
- popmon.visualization.alert_section_generator module
- popmon.visualization.histogram_section module
- popmon.visualization.overview_section module
- popmon.visualization.report_generator module
- popmon.visualization.section_generator module
- popmon.visualization.traffic_light_section_generator module
- popmon.visualization.utils module
Submodules
popmon.config module
- class popmon.config.Comparison(**data)
Bases:
BaseModel
Parameters related to comparisons
-
shift:
int
shift of time-bins in rolling/expanding window. default is 1.
-
window:
int
size of rolling window and/or trend detection. default is 10.
-
shift:
- class popmon.config.Monitoring(**data)
Bases:
BaseModel
Parameters related to monitoring
-
monitoring_rules:
Dict
[str
,List
[Union
[float
,int
]]] monitoring rules to generate traffic light alerts. The default setting is:
monitoring_rules = { "*_pull": [7, 4, -4, -7], "*_zscore": [7, 4, -4, -7], "[!p]*_unknown_labels": [0.5, 0.5, 0, 0], }
Note that the (filename based) wildcards such as * apply to all statistic names matching that pattern. For example,
"*_pull"
applies for all features to all statistics ending on “_pull”. You can also specify rules for specific features and/or statistics by leaving out wildcard and putting the feature name in front. E.g.monitoring_rules = { "featureA:*_pull": [5, 3, -3, -5], "featureA:nan": [4, 1, 0, 0], "*_pull": [7, 4, -4, -7], "nan": [8, 1, 0, 0], }
In case of multiple rules could apply for a feature’s statistic, the most specific one applies. So in case of the statistic “nan”: “featureA:nan” is used for “featureA”, and the other “nan” rule for all other features.
-
pull_rules:
Dict
[str
,List
[Union
[float
,int
]]] red and yellow (possibly dynamic) boundaries shown in plots in the report. Default is:
pull_rules = {"*_pull": [7, 4, -4, -7]}
This means that the shown yellow boundaries are at -4, +4 standard deviations around the (reference) mean, and the shown red boundaries are at -7, +7 standard deviations around the (reference) mean. Note that the (filename based) wildcards such as * apply to all statistic names matching that pattern. (The same string logic applies as for monitoring_rules.)
-
monitoring_rules:
- class popmon.config.Report(**data)
Bases:
BaseModel
Report-specific configuration
-
extended_report:
bool
if True, show all the generated statistics in the report (optional) if set to False, then smaller show_stats (see below)
-
last_n:
int
plot statistic data for last ‘n’ periods (optional)
-
online_report:
bool
Use a CDN to host resources, or embed them into the report.
-
primary_color:
str
Primary color used throughout the report
-
report_filepath:
Union
[str
,Path
,None
] the file path where to output the report (optional)
-
show_stats:
List
[str
] list of statistic name patterns to show in the report. If None, show all (optional)
-
skip_first_n:
int
in plot skip first ‘n’ periods. last_n takes precedence (optional)
-
skip_last_n:
int
in plot skip last ‘n’ periods. last_n takes precedence (optional)
-
title:
str
Report title in browser and navbar. May contain HTML.
-
tl_colors:
Dict
[str
,str
] “Configure line colors in barplots of Comparisons and Profiles section. Need to be hex format (full length)
-
extended_report:
- class popmon.config.Section(**data)
Bases:
BaseModel
Configuration for the individual sections
-
alerts:
AlertSection
Configuration related to the alerts section
-
comparisons:
ComparisonsSection
Configuration related to the comparisons section
-
histograms:
HistogramSectionModel
Configuration related to the histogram section
-
overview:
OverviewSection
Configuration related to the overview section
-
profiles:
ProfilesSection
Configuration related to the profiles section
-
traffic_lights:
TrafficLightsSection
Configuration related to the traffic lights section
-
alerts:
- class popmon.config.Settings(_env_file='<object object>', _env_file_encoding=None, _env_nested_delimiter=None, _secrets_dir=None, **values)
Bases:
ValidatedSettings
-
bin_specs:
Dict
[str
,Any
] dictionaries used for rebinning numeric or timestamp features. An example bin_specs dictionary is:
bin_specs = { "x": {"bin_width": 1, "bin_offset": 0}, "y": {"num": 10, "low": 0.0, "high": 2.0}, "x:y": [{}, {"num": 5, "low": 0.0, "high": 1.0}], }
In the bin specs for x:y, x is not provided (here) and reverts to the 1-dim setting. The ‘bin_width’, ‘bin_offset’ notation makes an open-ended histogram (for that feature) with given bin width and offset. The notation ‘num’, ‘low’, ‘high’ gives a fixed range histogram from ‘low’ to ‘high’ with ‘num’ number of bins.
-
binning:
Literal
['auto'
,'unit'
] default binning to revert to in case bin_specs not supplied. When using “auto”, semi-clever binning is automatically done.
-
comparison:
Comparison
Settings related to the comparisons
-
features:
Optional
[List
[str
]] columns to pick up from input data. (default is all features). For multi-dimensional histograms, separate the column names with a ‘:’. Example features list is:
features = ["x", "date", "date:x", "date:y", "date:x:y"]
If time_axis is set or found, and if no features provided, features becomes: [‘date:x’, ‘date:y’, ‘date:z’] etc.
-
monitoring:
Monitoring
Settings related to monitoring
-
reference_type:
Literal
['self'
,'external'
,'rolling'
,'expanding'
,'self_split'
] type of reference used for comparisons
-
time_axis:
str
name of datetime feature, used as time axis, e.g. ‘date’. (column should be timestamp, date(time) or numeric batch id) if empty string, will be auto-guessed.
-
bin_specs:
- class popmon.config.ValidatedSettings(_env_file='<object object>', _env_file_encoding=None, _env_nested_delimiter=None, _secrets_dir=None, **values)
Bases:
BaseSettings
popmon.resources module
- popmon.resources.data(name)
Return the full path filename of a shipped data file.
- Parameters
name (str) – The name of the data.
- Returns
The full path filename of the data.
- Return type
str
- Raises
FileNotFoundError – If the data cannot be found.
- popmon.resources.notebook(name)
Return the full path filename of a tutorial notebook.
- Parameters
name (str) – The name of the notebook.
- Returns
The full path filename of the notebook.
- Return type
str
- Raises
FileNotFoundError – If the notebook cannot be found.
- popmon.resources.templates_env(filename=None, **kwargs)
Return visualization templates directory environment. If filename provided, the exact template is being retrieved and provided keyword arguments - rendered accordingly.
- Parameters
filename (str) – the name of the template to get retrieved.
kwargs – residual keyword arguments which would be used for rendering
- Returns
template if a filename is provided (rendered given that keyword arguments are provided) otherwise: environment of the templates directory
popmon.utils module
- popmon.utils.filter_metrics(metrics, ignore_stat_endswith, show_stats=None)
Filter metrics by excluding based on suffix and/or including when matching a pattern
- popmon.utils.parallel(func, args_list, mode='args')
Routine for parallel processing
- popmon.utils.short_date(date)
Shorten date string to length of 22