NeurIPS 2020

FrugalML: How to use ML Prediction APIs more accurately and cheaply

Meta Review

This paper addresses an important praactica problem arising in the use of machine learning APIs. Each API has some predictive accuracy and quality score (confidence) but also has an assigned cost, which we'd like to minimize. The authors give a method to accomplish this: a base API is chosen based on learnt conditional accuracies which might be overruled by an add-on API if the quality score is not sufficiently high. The optimal strategy is generated via solving a stated optimization problem. The paper presents some neat experiments with this method on computer vision and NLP datasets with real-world APIs. These appear promising in that the generated strategy reduces costs while still achieving high predictive accuracies. The reviewers were impressed with the solution to the practical problem and the high quality writing. I recommend this paper for acceptance.