Defining custom metric functions¶
It is possible to define custom metric functions for Abed. This can be done by defining your metrics in a separate Python file. We will illustrate how to define a custom metric function with a practical example. In this example, we are interested in the sparsity hitrate of an output vector, the proportion of correctly identified zeros and correctly identified non-zero elements in the vector.
Assume our method outputs the following data:
beta_true beta_pred
0.1 0.0
1.2 1.0
0 0.1
0.3 0.2
0 0.0
0 0.5
0.7 0.5
Thus, in 4 out of 7 elements the sparsity is correctly identified (2nd, 4th,
5th, and 7th element). We can compute this with the following Python code,
which we place in execs/metrics.py
:
import numpy as np
from sklearn.metrics import accuracy_score
def sparsity_hitrate(true, pred):
""" Compute the sparsity hitrate of a vector. """
np_true = np.array(true)
np_pred = np.array(pred)
return accuracy_score(np_true == 0, np_pred == 0)
To use this metric in Abed’s output, we add the following to the Abed settings file:
import imp
mymetrics = imp.load_source('mymetrics', './execs/metrics.py')
This loads the metrics file as a module in the Abed settings file. Thus, in
the METRICS
setting, we can add:
METRICS = {
'HIT': {
'metric': mymetrics.sparsity_hitrate,
'best': max
}
}