EvalRS @ CIKM 2022
A Rounded Evaluation of Recommender Systems

Thank you for joining the challenge!

Here you can read the full paper : evaluation scripts and artifacts are shared publicly in our repository.

Standard and Behavioral Testing

We aim for a rounded testing of recommender system, to award not only point-wise metrics but also patterns that are important for humans.

Training data, evaluation scripts and rules can be found in the official challenge repository; relevant literature and background information about the challenge and relevant industry use cases can be found in the challenge paper pre-print.

EvalRS Leaderboard (Phase 2)

Note: Values for individual metrics (e.g. Hit-Rate) reflect their un-normalized value as per Phase 1. Score reflects the aggregated score inclusive of normalization.

Workshop Schedule

When What
Welcome 9:00-9:05
Dietmar Jannach (virtual) + Q&A 9:05-9:45
EvalRS presentation and awards 9:45-10:10
10:10-10:30 Item-based Variational Auto-encoder for Fair Music Recommendation
10:30-10:45 Coffee break
10:45-10:55 Lighting talk (virtual): Triplet Losses-based Matrix Factorization for Robust Recommendations
10:55-11:05 Lighting talk (virtual): Bias Mitigation in Recommender Systems to Improve Diversity
11:05-11:15 Lighting talk (in-person): Track2Vec: Fairness Music Recommendation with a GPU-Free Customizable-Driven Framework
11:15-12:00 Michael D. Ekstrand (in-person) + Q&A



Prof. Dietmar Jannach

Title: Multi-Objective Recommender Systems

Abstract: Optimizing recommendations for a single objective, e.g., prediction accuracy, may be too limiting in certain applications. Instead, it is often important not only to consider multiple quality factors of recommendations, e.g., diversity, but to also take the perspectives of multiple stakeholders into account. In this talk, we will review different approaches from the literature that aim to consider multiple objectives in the recommendation process. Furthermore, we will outline open challenges and future directions in this area.


Dietmar Jannach is a professor of computer science at the University of Klagenfurt, Austria. His main research theme is related to the application of intelligent system technology to practical problems and the development of methods for building knowledge-intensive software applications. In recent years, he worked on various topics in the area of recommender systems. In this area, he also published the first international textbook on the topic.


Prof. Michael D. Ekstrand

Title: Do You Want To Hunt A Kraken? Mapping and Expanding Recommendation Fairness

Abstract: Fair recommendation (and related problems, such as fair information retrieval) is a complex, multi-faceted problem. Significant progress has been made in recent years on identifying and measuring important forms of unfair recommendation, but there are still many ways recommender systems can replicate, exacerbate, or mitigate potentially discriminatory harms that need careful study.


Michael Ekstrand is an associate professor of computer science at Boise State University, where he co-directs the People and Information Research Team. His research blends information retrieval, human-computer interaction, machine learning, and algorithmic fairness to try to make information access systems good for everyone they affect. He received his Ph.D in 2014 from the University of Minnesota and an NSF CAREER award on measuring fairness in recommender systems in 2018, and leads the LensKit open-source software project for enabling high-velocity reproducible research in recommender systems, among many other activities.


Main Prize

5k USD total prize amount for winning teams

Support Prize

Free tickets at CIKM for the best systems (first) authored by students


Proudly sponsored by these amazing folks.

Important Dates

What When
Registration / Challenge Start August 5
We accept design papers starting September 10
Deadline paper submission October 1 (UPDATED)
Acceptance decision October 4 (UPDATED)
Final paper submission after review October 10
Challenge End (stop submission) October 10
Workshop during CIKM October 21 (UPDATED)



Jacopo Tagliabue

New York University/South Park Commons


Ciro Greco

South Park Commons


Tobias Schnabel

Microsoft Research


Patrick John Chia



Federico Bianchi

Stanford University


Giuseppe Attanasio

Bocconi University


Gabriel de Souza P. Moreira