|
In NeuroSolutions, 2-class problems can either be encoded into 2 output PEs (e.g., Detection-> -1/1; Non-detection->1/-1) or 1 output PE (e.g., Detection->1; Non-detection->-1). For the former case, the output with the highest value corresponds to the predicted class. In the latter case, any values above a threshold would be classified as one class and any values below would be classified as the other class. By default, this threshold is set to 0.
Receiver Operating Characteristic (ROC) matrices are used to show how changing the detection threshold affects detections versus false alarms. If the threshold is set too high then the system will miss too many detections. Conversely, if the threshold is set too low then there will be too many false alarms. Below is an example of an ROC matrix graphed as an ROC curve.
Figure 1: Example ROC Curve
In NeuroSolutions, a ROC matrix is created by attaching a MatrixViewer probe to the ROC access point of the ErrorCriterion component (or selecting the “ROC” checkbox within the “Probe Configuration” panel of the NeuralBuilder). The matrix contains three columns: 1) the detection threshold, 2) the percentage of detections classified correctly, and 3) the percentage of non-detections incorrectly classified as detections (i.e., false alarms). The number of thresholds (i.e., rows of the matrix) is defined within the Error Criteria inspector (i.e., right-click on the rightmost Axon and select “Properties”).
To use the information contained within the ROC matrix, you first find the threshold that meets the needs of your particular application. For example, if your application were cancer diagnosis, you may want to find the threshold that gives you 100% detections, but has the fewest false alarms. You can then change the “Confusion Threshold” setting within the Error Criteria inspector so that any attached Confusion Matrices will report the classifications based on this new threshold instead of 0.
At the present time there is no way to produce a graph in NeuroSolutions like the one shown in the figure above. However, you can export the data for the ROC matrix using a DataWriter probe, and then graph those points using an external graphing program such as Excel.
|
