A Comparison Study of Cost-Sensitive Classifier Evaluation

Abstract

Performance evaluation plays an important role in the rule induction and classification process. Classic evaluation measures have been extensively studied in the past. In recent years, cost-sensitive classification has received much attention. In a typical classification task, all types of classification errors are treated equally. In many practical cases, not all errors are equal. Therefore, it is critical to build a cost-sensitive classifier to minimize the expected cost. Much work has been done with regard to this issue. On the other hand, cost-sensitive classifier evaluation received less attention, and had only been investigated in specific classification tasks. The goal of my project is to investigate different aspects of this problem. I review 5 existing cost-sensitive evaluation measures and compare their similarities and differences. I find that the cost-sensitive measures can provide consistent evaluation results comparing to classic evaluation measures in most cases. However, when applying different cost values to the evaluation, the differences between the performances of each algorithm change. It is reasonable to conclude that the evaluation results could change dramatically when certain cost values applied. Moreover, by using cost curves to visualize the classification results, performance and performance differences of different classifiers can be easily seen.