A Rounded Evaluation of Recommender Systems


Training data, evaluation scripts and rules can be found in the official challenge repository; relevant literature and background information about the challenge and relevant industry use cases can be found in the challenge paper pre-print.


To sign up for the challenge, you should fill the form below with your official e-mail, your organization (university or company), first and last name of the team lead and a nickname for the leaderboard. You will receive a confirmation e-mail from the system including your user id and AWS write-only credentials to upload your json files to the challenge bucket - please use the information responsibly. After submitting the form, you will be sent back to this page: please wait some minutes and re-check your e-mail, including your spam folder before re-submitting. We suggest adding evalrs2022@gmail.com to your list of trusted senders.   Registration for the challenge is now closed.

This web-app is hosted on a devoted AWS account and all data will be destroyed at the end of the Data Challenge: if you want to use the same serverless back-end to run a lightweight leaderboard website, please get in touch.

For the submission format and the general rules of the contest, please consult the relevant section in the README

EvalRS Leaderboard (Phase 2)

Note: Values for individual metrics (e.g. Hit-Rate) reflect their un-normalized value as per Phase 1. Score reflects the aggregated score inclusive of normalization.

Some charts to better understand how models perform

We are plotting MRR with other metrics, to see the impact of the "main" metric onto the others. Note that MRED is multiplied by -1 in this plot to give a better idea of the relationships. Thus HIGHER MRED means higher unequal performance across partitions.

More charts available on the charts page!

Go to Charts Page

MRR and Country MRED

MRR and Gender MRED

EvalRS Leaderboard (Phase 1)