Big data analysis by AI is utilized in various fields, and is an important verification means in predicting the effectiveness of molecular models in drug discovery research.However, in the case of detection of a specific molecule, for example, it is often reported that the detection success rate in an experiment is much lower than the prior prediction by a computer model.The root cause was attributed to the computer model, a statistical indicator that overestimated the performance of AI.Until now, several types of indicators such as TPR (True Positive Rate) and ACC (Accuracy) have been used as AI performance evaluation indicators, but these indicators really evaluate AI performance correctly. It was considered that it was made.

　In this research, we developed a method to accurately evaluate the performance of AI using statistical indicators.This method verifies the characteristics and effectiveness of each index such as TPR and ACC by statistical analysis using the distribution function (iCDF).The results of the verification showed that regardless of the AI ​​technology, there is a high probability that a high evaluation value can be obtained in TPR, ACC, etc., and it is highly likely that it will not lead to the effectiveness of actual application.Furthermore, it was found that the characteristics of the evaluation index itself should be thoroughly examined by a new method before developing AI and conducting evaluation experiments.

The method developed this time can be applied to AI in any field.It is expected to contribute to the development of "robust" AI that can accurately classify any data set in proof-of-concept experiments.

Paper information:[Molecular Informatics] Classifiers and their Metrics Quantified