Experiment

pnpxai.core.experiment.experiment

Experiment

Bases: Observable

A class representing an experiment for model interpretability.

Parameters:

Name	Type	Description	Default
model	Model	The machine learning model to be interpreted.	required
data	DataSource	The data used for the experiment.	required
modality	Modality	The type of modality (image, tabular, text, time series) the model is designed for.	required
explainers	Sequence[Explainer]	Explainer objects or their arguments for interpreting the model.	required
postprocessors	Optional[Sequence[Callable]]	Postprocessing functions to apply to explanations.	required
metrics	Optional[Sequence[Metric]]	Evaluation metrics used to assess model interpretability.	required
input_extractor	Optional[Callable[[Any], Any]]	Function to extract inputs from data.	None
label_extractor	Optional[Callable[[Any], Any]]	Function to extract labels from data.	None
target_extractor	Optional[Callable[[Any], Any]]	Function to extract targets from data.	None
cache_device	Optional[Union[device, str]]	Device to cache data and results.	None
target_labels	bool	True if the target is a label, False otherwise.	False

Attributes:

Name	Type	Description
modality	Modality	Object defining the modality-specific control flow of the experiment.
manager	ExperimentManager	Manager object for the experiment.
all_explainers	Sequence[Explainer]	All explainer objects used in the experiment.
all_metrics	Sequence[Metric]	All evaluation metrics used in the experiment.
errors	Sequence[Error]
is_image_task	bool	True if the modality is an image-related modality, False otherwise.
has_explanations	bool	True if the experiment has explanations, False otherwise.

model = model instance-attribute

model_device = next(self.model.parameters()).device instance-attribute

manager = ExperimentManager(data=data, cache_device=cache_device) instance-attribute

input_extractor = input_extractor if input_extractor is not None else default_input_extractor instance-attribute

label_extractor = label_extractor if label_extractor is not None else default_target_extractor instance-attribute

target_extractor = target_extractor if target_extractor is not None else default_target_extractor instance-attribute

target_labels = target_labels instance-attribute

modality = modality instance-attribute

errors property

has_explanations property

__init__(model: Model, data: DataSource, modality: Modality, explainers: Sequence[Explainer], postprocessors: Sequence[Callable], metrics: Sequence[Metric], input_extractor: Optional[Callable[[Any], Any]] = None, label_extractor: Optional[Callable[[Any], Any]] = None, target_extractor: Optional[Callable[[Any], Any]] = None, cache_device: Optional[Union[torch.device, str]] = None, target_labels: bool = False)

reset_errors()

to_device(x)

run_batch(explainer_id: int, postprocessor_id: int, metric_id: int, data_ids: Optional[Sequence[int]] = None) -> dict

Runs the experiment for selected batch of data, explainer, postprocessor and metric.

Parameters:

Name	Type	Description	Default
data_ids	Sequence[int]	A sequence of data IDs to specify the subset of data to process.	None
explainer_id	int	ID of explainer to use for the run.	required
postprocessor_id	int	ID of postprocessor to use for the run.	required
metrics_id	int	ID of metric to use for the run.	required

Returns:

Type	Description
dict	The dictionary of inputs, labels, outputs, targets, explainer, explanation, postprocessor, postprocessed, metric, and evaluation.

This method orchestrates the experiment by configuring the manager, obtaining explainer and metric instances, processing data, generating explanations, and evaluating metrics. It then caches the results in the manager, and returns back to the user.

Note: The input parameters allow for flexibility in specifying subset of data, explainer, postprocessor and metric to process.

predict_batch(data_ids: Sequence[int])

Predicts results of the experiment for selected batch of data.

Parameters:

Name	Type	Description	Default
data_ids	Sequence[int]	A sequence of data IDs to specify the subset of data to process.	required

Returns:

Type	Description
	Batched model outputs corresponding to data ids.

This method orchestrates the experiment by configuring the manager and processing data. It then caches the results in the manager, and returns back to the user.

explain_batch(data_ids: Sequence[int], explainer_id: int)

Explains selected batch of data within experiment.

Parameters:

Name	Type	Description	Default
data_ids	Sequence[int]	A sequence of data IDs to specify the subset of data to process.	required

Returns:

Type	Description
	Batched model explanations corresponding to data ids.

This method orchestrates the experiment by configuring the manager, obtaining explainer instance, processing data, and generating explanations. It then caches the results in the manager, and returns back to the user.

postprocess_batch(data_ids: List[int], explainer_id: int, postprocessor_id: int)

Postprocesses selected batch of data within experiment.

Parameters:

Name	Type	Description	Default
data_ids	Sequence[int]	A sequence of data IDs to specify the subset of data to postprocess.	required
explainer_id	int	An explainer ID to specify the explainer to use.	required
postprocessor_id	int	A postprocessor ID to specify the postprocessor to use.	required

Returns:

Type	Description
	Batched postprocessed model explanations corresponding to data ids.

This method orchestrates the experiment by configuring the manager, obtaining explainer instance, processing data, and generating explanations. It then caches the results in the manager, and returns back to the user.

evaluate_batch(data_ids: List[int], explainer_id: int, postprocessor_id: int, metric_id: int)

Evaluates selected batch of data within experiment.

Parameters:

Name	Type	Description	Default
data_ids	Sequence[int]	A sequence of data IDs to specify the subset of data to postprocess.	required
explainer_id	int	An explainer ID to specify the explainer to use.	required
postprocessor_id	int	A postprocessor ID to specify the postprocessor to use.	required
metric_id	int	A metric ID to evaluate the model explanations.	required

Returns:

Type	Description
	Batched model evaluations corresponding to data ids.

This method orchestrates the experiment by configuring the manager, obtaining explainer instance, processing data, generating explanations, and evaluating results. It then caches the results in the manager, and returns back to the user.

optimize(data_ids: Union[int, Sequence[int]], explainer_id: int, metric_id: int, direction: Literal['minimize', 'maximize'] = 'maximize', sampler: Literal['grid', 'random', 'tpe'] = 'tpe', n_trials: Optional[int] = None, timeout: Optional[float] = None, **kwargs)

Optimize experiment hyperparameters by processing data, generating explanations, evaluating with metrics, caching and retrieving the data.

Parameters:

Name	Type	Description	Default
data_ids	Union[int, Sequence[int]]	A single data ID or sequence of data IDs to specify the subset of data to process.	required
explainer_id	int	An explainer ID to specify the explainer to use.	required
metric_id	int	A metric ID to evaluate optimizer decisions.	required
direction	Literal['minimize', 'maximize']	A string to specify the direction of optimization.	'maximize'
sampler	Literal['grid', 'random', 'tpe']	A string to specify the sampler to use for optimization.	'tpe'
n_trials	Optional[int]	An integer to specify the number of trials for optimization. If none passed, the number of trials is inferred from timeout.	None
timeout	Optional[float]	A float to specify the timeout for optimization. Ignored, if n_trials is specified.	None

Returns:

Type	Description
	The Experiment instance with updated results and state.

This method orchestrates the experiment by configuring the manager, obtaining explainer and metric instances, processing data, generating explanations, and evaluating metrics. It then saves the results in the manager.

Note: The input parameters allow for flexibility in specifying subsets of data, explainers, and metrics to process. If not provided, the method processes all available data, explainers, postprocessors, and metrics.

get_inputs_flattened(data_ids: Optional[Sequence[int]] = None) -> Sequence[Tensor]

Retrieve and flatten last run input data.

Parameters:

Name	Type	Description	Default
data_ids	Optional[Sequence[int]]	A sequence of data IDs to specify the subset of data to process.	None

Returns:

Type	Description
Sequence[Tensor]	Flattened input data.

This method retrieves input data using the input extractor and flattens it for further processing.

Note: The input parameters allow for flexibility in specifying subsets of data to process. If not provided, the method processes all available data.

get_all_inputs_flattened() -> Sequence[Tensor]

Retrieve and flatten all input data.

Returns:

Type	Description
Sequence[Tensor]	Flattened input data from all available data.

This method retrieves input data from all available data points using the input extractor and flattens it.

get_labels_flattened(data_ids: Optional[Sequence[int]] = None) -> Sequence[Tensor]

Retrieve and flatten labels data.

Parameters:

Name	Type	Description	Default
data_ids	Optional[Sequence[int]]	A sequence of data IDs to specify the subset of data to process.	None

Returns:

Type	Description
Sequence[Tensor]	Flattened labels data.

This method retrieves label data using the label extractor and flattens it for further processing.

get_targets_flattened(data_ids: Optional[Sequence[int]] = None) -> Sequence[Tensor]

Retrieve and flatten target data.

Parameters:

Name	Type	Description	Default
data_ids	Optional[Sequence[int]]	A sequence of data IDs to specify the subset of data to process.	None

Returns:

Type	Description
Sequence[Tensor]	Flattened target data.

This method retrieves target data using the target extractor and flattens it for further processing.

Note: The input parameters allow for flexibility in specifying subsets of data to process. If not provided, the method processes all available data.

get_outputs_flattened(data_ids: Optional[Sequence[int]] = None) -> Sequence[Tensor]

Retrieve and flatten model outputs.

Parameters:

Name	Type	Description	Default
data_ids	Optional[Sequence[int]]	A sequence of data IDs to specify the subset of data to process.	None

Returns:

Type	Description
Sequence[Tensor]	Flattened model outputs.

This method retrieves flattened model outputs using the manager's get_flat_outputs method.

Note: The input parameters allow for flexibility in specifying subsets of data to process. If not provided, the method processes all available data.

get_explanations_flattened(data_ids: Optional[Sequence[int]] = None) -> Sequence[Sequence[Tensor]]

Retrieve and flatten explanations from all explainers.

Parameters:

Name	Type	Description	Default
data_ids	Optional[Sequence[int]]	A sequence of data IDs to specify the subset of data to process.	None

Returns:

Type	Description
Sequence[Sequence[Tensor]]	Flattened explanations from all explainers.

This method retrieves flattened explanations for each explainer using the manager's get_flat_explanations method.

Note: The input parameters allow for flexibility in specifying subsets of data to process. If not provided, the method processes all available data.

get_evaluations_flattened(data_ids: Optional[Sequence[int]] = None) -> Sequence[Sequence[Sequence[Tensor]]]

Retrieve and flatten evaluations for all explainers and metrics.

Parameters:

Name	Type	Description	Default
data_ids	Optional[Sequence[int]]	A sequence of data IDs to specify the subset of data to process.	None

Returns:

Type	Description
Sequence[Sequence[Sequence[Tensor]]]	Flattened evaluations for all explainers and metrics.

This method retrieves flattened evaluations for each explainer and metric using the manager's get_flat_evaluations method.

Note: The input parameters allow for flexibility in specifying subsets of data to process. If not provided, the method processes all available data.

get_explainers_ranks() -> Optional[Sequence[Sequence[int]]]

Calculate and return rankings for explainers based on evaluations.

Returns:

Type	Description
Optional[Sequence[Sequence[int]]]	Rankings of explainers. Returns None if rankings cannot be calculated.

This method calculates rankings for explainers based on evaluations and metric scores. It considers metric priorities and sorting preferences to produce rankings.

subscribe(callback)

fire(event)

default_input_extractor(x)

default_label_extractor(x)

default_target_extractor(y)