heart_library.metrics package

Submodules

heart_library.metrics.metrics module

Module implementing varying metrics for assessing model robustness. These fall mainly under two categories: attack-dependent and attack-independent.

class heart_library.metrics.metrics.AccuracyPerturbationMetric(benign_predictions: Sequence[ndarray[Any, dtype[float32]]], metadata: Sequence[dict[str, Any]], accuracy_type: str = 'robust', metadata_id: str | None = None)

Bases: object

A metric for easily calculating the clean and robust accuracy as well as the perturbation between clean and adversarial input.

compute() → dict[str, float]

Returns the computed metric in Tuple (clean_accuracy, robust_accuracy, average_perturbation)

Returns:

Final value from the state of the metric.

Return type:

dict[str, float]

metadata: dict[str, Any]

reset() → None

Reset the metric to default values.

update(preds_batch: Sequence[ndarray[Any, dtype[float32]]], targets_batch: Sequence[ndarray[Any, dtype[float32]]]) → None

Updates the metric value.

Parameters:

• preds_batch (Sequence[NDArray[np.float32]]) – Predicted values.

• targets_batch (Sequence[NDArray[np.float32]]) – Target values.

Raises:

KeyError – if Delta not computed for metadata.

class heart_library.metrics.metrics.BlackBoxAttackQualityMetric(attack: JaticAttack, metadata_id: str | None = None)

Bases: object

A metric for extracting the black box quality metrics.

compute() → dict[str, Any]

Returns the computed metric in dict

Returns:

Final value from the state of the metric.

Return type:

dict[str, Any]

metadata: dict[str, Any]

reset() → None

Reset the metric to default values.

update() → None

Updates the metric value.

class heart_library.metrics.metrics.HeartAccuracyMetric(is_logits: bool = True, metadata_id: str | None = None, **kwargs: Any)

Bases: object

Facilitating support for Torchmetric’s Accuracy metric.

compute() → dict[str, float]

Compute accuracy score.

Returns:

Final value from the state of the metric.

Return type:

dict[str, float]

metadata: dict[str, Any]

reset() → None

Clear contents of current metric’s cache of predictions and targets.

Returns:

None.

Return type:

_type_

update(preds_batch: Sequence[ndarray[Any, dtype[float32]]], targets_batch: Sequence[ndarray[Any, dtype[float32]]]) → None

Add predictions and targets to metric’s cache for later calculation.

Parameters:

• preds_batch (Sequence[NDArray[np.float32]]) – edictions in numpy array format.

• targets_batch (Sequence[NDArray[np.float32]]) – groundtruth targets in numpy array format.

class heart_library.metrics.metrics.HeartMAPMetric(metadata_id: str | None = None, **kwargs: Any)

Bases: object

Facilitating support for Torchmetric’s MAP metric.

Examples

We can define a MAP metric and evaluate it on a JaticPyTorchObjectDetector’s performance:

>>> from maite.workflows import evaluate
>>> from heart_library.estimators.object_detection import JaticPyTorchObjectDetector
>>> import torch
>>> import numpy
>>> from datasets import load_dataset
>>> from torchvision.transforms import transforms
>>> from copy import deepcopy

Define the JaticPyTorchObjectDetector, in this case passing in a resnet model:

>>> MEAN = [0.485, 0.456, 0.406]
>>> STD = [0.229, 0.224, 0.225]
>>> preprocessing = (MEAN, STD)

>>> detector = JaticPyTorchObjectDetector(
...     model_type="detr_resnet50_dc5",
...     input_shape=(3, 800, 800),
...     clip_values=(0, 1),
...     attack_losses=(
...         "loss_ce",
...         "loss_bbox",
...         "loss_giou",
...     ),
...     device_type="cpu",
...     optimizer=torch.nn.CrossEntropyLoss(),
...     preprocessing=preprocessing,
... )

Prepare images for detection:

>>> data = load_dataset("guydada/quickstart-coco", split="train[20:25]")
>>> preprocess = transforms.Compose([transforms.Resize(800), transforms.CenterCrop(800), transforms.ToTensor()])

>>> data = data.map(lambda x: {"image": preprocess(x["image"]), "label": None})

Execute object detection and return JaticPyTorchObjectDetectionOutput:

>>> detections = detector(data)

Define data with detections:

>>> class ImageDataset:
...     def __init__(self, images, groundtruth, threshold=0.8):
...         self.images = images
...         self.groundtruth = groundtruth
...         self.threshold = threshold
...
...     def __len__(self) -> int:
...         return len(self.images)
...
...     def __getitem__(self, ind: int) -> tuple[np.ndarray, np.ndarray, dict[str, Any]]:
...         image = np.asarray(self.images[ind]["image"]).astype(np.float32)
...         filtered_detection = self.groundtruth[ind]
...         filtered_detection.boxes = filtered_detection.boxes[filtered_detection.scores > self.threshold]
...         filtered_detection.labels = filtered_detection.labels[filtered_detection.scores > self.threshold]
...         filtered_detection.scores = filtered_detection.scores[filtered_detection.scores > self.threshold]
...         return (image, filtered_detection, None)

>>> data_with_detections = ImageDataset(data, deepcopy(detections), threshold=0.9)

Set the MAP parameters and evaluate:

>>> map_args = {
...     "box_format": "xyxy",
...     "iou_type": "bbox",
...     "iou_thresholds": [0.5],
...     "rec_thresholds": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
...     "max_detection_thresholds": [1, 10, 100],
...     "class_metrics": False,
...     "extended_summary": False,
...     "average": "macro",
... }

>>> metric = HeartMAPMetric(**map_args)

>>> results, _, metadata = evaluate(
...     model=detector,
...     dataset=data_with_detections,
...     metric=metric,
... )

>>> results["map"]
tensor(1.)

compute() → dict[str, Any]

Compute MAP scores.

Returns:

Final value from the state of the metric.

Return type:

dict[str, Any]

metadata: dict[str, Any]

reset() → None

Clear contents of current metric’s cache of predictions and targets.

Returns:

None.

Return type:

_type_

update(preds_batch: Sequence[JaticPyTorchObjectDetectionOutput], targets_batch: Sequence[JaticPyTorchObjectDetectionOutput]) → None

Add predictions and targets to metric’s cache for later calculation.

Parameters:

• preds_batch (Sequence[JaticPyTorchObjectDetectionOutput]) – predictions in ObjectDetectionTarget format.

• targets_batch (Sequence[JaticPyTorchObjectDetectionOutput]) – groundtruth targets in ObjectDetectionTarget format.

class heart_library.metrics.metrics.RobustnessBiasMetric(metadata: Sequence[dict[str, Any]], labels: ndarray[Any, dtype[float32]], interval: int = 100, metadata_id: str | None = None)

Bases: object

A metric which describes Robustness Bias for features of datasets. Currently supports only classification tasks.

compute() → dict[str, Any]

Returns the computed metric.

Returns:

Final value from the state of the metric.

Return type:

dict[str, Any]

metadata: dict[str, Any]

reset() → None

Reset the metric to default values.

update(preds_batch: Sequence[ndarray[Any, dtype[float32]]], targets_batch: Sequence[ndarray[Any, dtype[float32]]]) → None

Add predictions and targets to metric’s cache for later calculation.

Parameters:

• preds_batch (Sequence[NDArray[np.float32]]) – predictions in numpy array format.

• targets_batch (Sequence[NDArray[np.float32]]) – groundtruth targets in numpy array format.

Raises:

KeyError – if Delta not computed for metadata.

Module contents

Module providing metrics

class heart_library.metrics.AccuracyPerturbationMetric(benign_predictions: Sequence[ndarray[Any, dtype[float32]]], metadata: Sequence[dict[str, Any]], accuracy_type: str = 'robust', metadata_id: str | None = None)

Bases: object

A metric for easily calculating the clean and robust accuracy as well as the perturbation between clean and adversarial input.

compute() → dict[str, float]

Returns the computed metric in Tuple (clean_accuracy, robust_accuracy, average_perturbation)

Returns:

Final value from the state of the metric.

Return type:

dict[str, float]

metadata: dict[str, Any]

reset() → None

Reset the metric to default values.

update(preds_batch: Sequence[ndarray[Any, dtype[float32]]], targets_batch: Sequence[ndarray[Any, dtype[float32]]]) → None

Updates the metric value.

Parameters:

• preds_batch (Sequence[NDArray[np.float32]]) – Predicted values.

• targets_batch (Sequence[NDArray[np.float32]]) – Target values.

Raises:

KeyError – if Delta not computed for metadata.

class heart_library.metrics.BlackBoxAttackQualityMetric(attack: JaticAttack, metadata_id: str | None = None)

Bases: object

A metric for extracting the black box quality metrics.

compute() → dict[str, Any]

Returns the computed metric in dict

Returns:

Final value from the state of the metric.

Return type:

dict[str, Any]

metadata: dict[str, Any]

reset() → None

Reset the metric to default values.

update() → None

Updates the metric value.

class heart_library.metrics.HeartAccuracyMetric(is_logits: bool = True, metadata_id: str | None = None, **kwargs: Any)

Bases: object

Facilitating support for Torchmetric’s Accuracy metric.

compute() → dict[str, float]

Compute accuracy score.

Returns:

Final value from the state of the metric.

Return type:

dict[str, float]

metadata: dict[str, Any]

reset() → None

Clear contents of current metric’s cache of predictions and targets.

Returns:

None.

Return type:

_type_

update(preds_batch: Sequence[ndarray[Any, dtype[float32]]], targets_batch: Sequence[ndarray[Any, dtype[float32]]]) → None

Add predictions and targets to metric’s cache for later calculation.

Parameters:

• preds_batch (Sequence[NDArray[np.float32]]) – edictions in numpy array format.

• targets_batch (Sequence[NDArray[np.float32]]) – groundtruth targets in numpy array format.

class heart_library.metrics.HeartMAPMetric(metadata_id: str | None = None, **kwargs: Any)

Bases: object

Facilitating support for Torchmetric’s MAP metric.

Examples

We can define a MAP metric and evaluate it on a JaticPyTorchObjectDetector’s performance:

>>> from maite.workflows import evaluate
>>> from heart_library.estimators.object_detection import JaticPyTorchObjectDetector
>>> import torch
>>> import numpy
>>> from datasets import load_dataset
>>> from torchvision.transforms import transforms
>>> from copy import deepcopy

Define the JaticPyTorchObjectDetector, in this case passing in a resnet model:

>>> MEAN = [0.485, 0.456, 0.406]
>>> STD = [0.229, 0.224, 0.225]
>>> preprocessing = (MEAN, STD)

>>> detector = JaticPyTorchObjectDetector(
...     model_type="detr_resnet50_dc5",
...     input_shape=(3, 800, 800),
...     clip_values=(0, 1),
...     attack_losses=(
...         "loss_ce",
...         "loss_bbox",
...         "loss_giou",
...     ),
...     device_type="cpu",
...     optimizer=torch.nn.CrossEntropyLoss(),
...     preprocessing=preprocessing,
... )

Prepare images for detection:

>>> data = load_dataset("guydada/quickstart-coco", split="train[20:25]")
>>> preprocess = transforms.Compose([transforms.Resize(800), transforms.CenterCrop(800), transforms.ToTensor()])

>>> data = data.map(lambda x: {"image": preprocess(x["image"]), "label": None})

Execute object detection and return JaticPyTorchObjectDetectionOutput:

>>> detections = detector(data)

Define data with detections:

>>> class ImageDataset:
...     def __init__(self, images, groundtruth, threshold=0.8):
...         self.images = images
...         self.groundtruth = groundtruth
...         self.threshold = threshold
...
...     def __len__(self) -> int:
...         return len(self.images)
...
...     def __getitem__(self, ind: int) -> tuple[np.ndarray, np.ndarray, dict[str, Any]]:
...         image = np.asarray(self.images[ind]["image"]).astype(np.float32)
...         filtered_detection = self.groundtruth[ind]
...         filtered_detection.boxes = filtered_detection.boxes[filtered_detection.scores > self.threshold]
...         filtered_detection.labels = filtered_detection.labels[filtered_detection.scores > self.threshold]
...         filtered_detection.scores = filtered_detection.scores[filtered_detection.scores > self.threshold]
...         return (image, filtered_detection, None)

>>> data_with_detections = ImageDataset(data, deepcopy(detections), threshold=0.9)

Set the MAP parameters and evaluate:

>>> map_args = {
...     "box_format": "xyxy",
...     "iou_type": "bbox",
...     "iou_thresholds": [0.5],
...     "rec_thresholds": [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0],
...     "max_detection_thresholds": [1, 10, 100],
...     "class_metrics": False,
...     "extended_summary": False,
...     "average": "macro",
... }

>>> metric = HeartMAPMetric(**map_args)

>>> results, _, metadata = evaluate(
...     model=detector,
...     dataset=data_with_detections,
...     metric=metric,
... )

>>> results["map"]
tensor(1.)

compute() → dict[str, Any]

Compute MAP scores.

Returns:

Final value from the state of the metric.

Return type:

dict[str, Any]

metadata: dict[str, Any]

reset() → None

Clear contents of current metric’s cache of predictions and targets.

Returns:

None.

Return type:

_type_

update(preds_batch: Sequence[JaticPyTorchObjectDetectionOutput], targets_batch: Sequence[JaticPyTorchObjectDetectionOutput]) → None

Add predictions and targets to metric’s cache for later calculation.

Parameters:

• preds_batch (Sequence[JaticPyTorchObjectDetectionOutput]) – predictions in ObjectDetectionTarget format.

• targets_batch (Sequence[JaticPyTorchObjectDetectionOutput]) – groundtruth targets in ObjectDetectionTarget format.

class heart_library.metrics.RobustnessBiasMetric(metadata: Sequence[dict[str, Any]], labels: ndarray[Any, dtype[float32]], interval: int = 100, metadata_id: str | None = None)

Bases: object

A metric which describes Robustness Bias for features of datasets. Currently supports only classification tasks.

compute() → dict[str, Any]

Returns the computed metric.

Returns:

Final value from the state of the metric.

Return type:

dict[str, Any]

metadata: dict[str, Any]

reset() → None

Reset the metric to default values.

update(preds_batch: Sequence[ndarray[Any, dtype[float32]]], targets_batch: Sequence[ndarray[Any, dtype[float32]]]) → None

Add predictions and targets to metric’s cache for later calculation.

Parameters:

• preds_batch (Sequence[NDArray[np.float32]]) – predictions in numpy array format.

• targets_batch (Sequence[NDArray[np.float32]]) – groundtruth targets in numpy array format.

Raises:

KeyError – if Delta not computed for metadata.