ML-Inception: understanding where and why models work (and don’t work)
Carlos Soares FEUP
A subgroup discovery-based method has recently been proposed to understand the behavior of models in the (original) feature space. The subgroups identified represent areas of feature space where the model obtains better or worse predictive performance than on average. For instance, in the marketing domain, the approach extracts subgroups such as: for customers with higher income and who are younger, the random forest achieves higher accuracy than on average. Here, we propose the use of metalearning to analyze those subgroups on the metafeature space, where they are characterized in a domain-independent way, using statistical and information theoretic properties. We then use association rules to relate characteristics of the subgroups to improvement or degradation of the performance of models. For instance, in the same domain, the approach extracts rules such as: when the class entropy decreases and the mutual information increases in the subgroup data, the random forest achieves lower accuracy. We illustrate the approach with some empirical results.
Carlos Soares is an Associate Professor at the Faculty of Engineering of U. Porto, where he holds the positions of Subdirector of the Dep. of Informatics Engineering, Director of the Ph.D. programme on Informatics Engineering and Adjunct Director of the M.Sc. programme on Data Science and Engineering. Carlos teaches at the Porto Business School, where he is the co-Director of the executive programme on Business Intelligence & Analytics. He is also an External Advisor for Intelligent Systems at Fraunhofer Portugal AICOS, a researcher at LIACC and a collaborator at LIAAD-INESC TEC. The focus of his research is on metalearning/autoML but he has a general interest in Data Science. He has participated in 20+ national and international R&ID as well as consulting projects. Carlos regularly collaborates with companies, including Feedzai, Accenture and InovRetail. He has published/edited several books and 150+ papers in journals and conferences, (90+/125+ indexed by ISI/Scopus) and supervised 10+/50+ Ph.D./M.Sc. thesis. Recent participation in the organization of events, includes ECML PKDD 2015, IDA 2016 and Discovery Science 2021, as programme co-chair. In 2009, he was awarded the Scientific Merit and Excellence Award of the Portuguese AI Association.