EvalRS2023

A Well-Rounded Evaluation of Recommender Systems. The Second Edition of the workshop on well-rounding the evaluation of recommender systems, covering fairness, robustness and trustworthiness.

KDD, August 7th, Long Beach (CA), US

Workshop

Can We Build Recommender Systems That Are More Robust, Trustworthy And Well-Rounded?

The ubiquity of personalized recommendations has led to a surge of interest in recommender systems research; however, evaluating model performance across use cases is still difficult.


Building on the success of EvalRS 2022 and Reclist, we are back with a broader scope and a new interactive format: our workshop focuses on multi-faceted evaluation for recommender systems, mixing a traditional research track (with prizes for innovative papers) and a hackathon , to live and breath the challenges with working code (with prizes for the best projects).

Important: Registration to KDD is required to attend the workshop, but submitting papers is not required to participate in the hackathon. The hackathon and social event is an affiliated event organized with the help of our generous sponsors, Snap , Costanoa and by Bauplan Labs .

hackathon

"Talk is cheap, show me the code": assemble a team, complete a project, and compete for the prizes!

We'll host a pizza hackathon night, where participants will pursue innovative projects for the rounded evaluation of recommender systems!


A new, open dataset, with notebooks, models, and tests is already available if you want to get into the workshop mood early: check the official repository to get started (yes, you can start your project before the workshop if you desire to do so!). As usual, everything goes back to the community as open source contributions.


Important: The hackathon and social event is an affiliated event organized with the help of our generous sponsors. To participate in the hackathon and win the prizes, you're not required to submit a paper to the workshop: if you can't be in Long Beach, write us an email, as we welcome hybrid participation from teams that can't come in person.


Bauplan Logo Snap Logo Snap Logo

Accepted Papers

Title Link
Realistic but Non-Identifiable Synthetic User Data Generation Link
Disentangling the Performance Puzzle of Multimodal-aware Recommender Systems Link
Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenders Link
Evaluating Recommendation Systems Using the Power of Embeddings Link
Metric@CustomerN: Evaluating Metrics at a Customer Level in E-Commerce Link

Schedule

August 7th at KDD 2023

Time What
1.00 - 1.30 (PM, Pacific Time) KeyNote Luca Belli
1.30 - 2.00 Papers Presentations
2.00 - 4.30 hackathon Part I
4:30 - 5:00 KeyNote Joey Robinson
Break Break
6:00-8:00 After Party hackathon Part II

KeyNotes

Joey Robinson, Manager, Web Ad Ranking @ Snap


Bio : Joey is an engineering manager at Snap, where he leads the Web Ad Ranking team. His team owns the development of the CTR and CVR models used for predicting web engagements for Snapchat ads. Prior to Snap, he led the machine learning team at an AI startup (Scaled Inference) and studied computer science and machine learning at Carnegie Mellon University.


Abstract : Ranking and recommendation systems power a variety of Snapchat features, including spotlight, discover, maps, search, lenses, ads, and friending. For all of these products, model evaluation and analysis tooling play a central role at each stage of the model development lifecycle. In this talk, we will discuss the evolution of Snapchat's unified model evaluation system and the key design decisions that allowed it to grow to support more than a dozen ML use cases at scale. We will also discuss the importance of extending model evaluation beyond a single accuracy number, and how a flexible evaluation system can allow engineers to ask and answer questions about their models that help to guide future model improvements.

Luca Belli, founder of Sator Labs, UC Berkeley Tech Policy Fellow.


Bio : Luca Belli is the founder of Sator Labs, a Visiting Nist AI Fellow and a UC Berkeley Tech Policy Fellow. Previously he was the co-founder and Research Lead for Twitter’s Machine learning Ethics, Transparency and Accountability (META) team where he guided industry leading approaches for responsible ML practices and product changes. Before that he operated as a Data Science and Machine Learning Engineer at Conversant and WolframAlpha. His research interests lie at the intersection of feedback loops, algorithmic amplification (with a special eye on politics), and algorithmic audits. He holds a Ph.D. in Math from Tor Vergata University in Rome.


Organizers

EvalRS is brought to you by the following people

Federico Bianchi

Federico Bianchi

Stanford

Patrick John Chia

Patrick John Chia

Coveo

Jacopo Tagliabue

Jacopo Tagliabue

NYU/Bauplan

Claudio Pomo

Claudio Pomo

Politecnico di Bari

Gabriel Moreira

Gabriel Moreira

NVIDIA

Ciro Greco

Ciro Greco

Bauplan

Davide Eynard

Davide Eynard

Mozilla AI

Fahd Husain

Fahd Husain

Mozilla AI

Fahd Husain

Aaron Gonzales

Mozilla AI