EvalRS 2023 at KDD

Workshop

Can We Build Recommender Systems That Are More Robust, Trustworthy And Well-Rounded?

The ubiquity of personalized recommendations has led to a surge of interest in recommender systems research; however, evaluating model performance across use cases is still difficult.

Building on the success of EvalRS 2022 and Reclist, we are back with a broader scope and a new interactive format: our workshop focuses on multi-faceted evaluation for recommender systems, mixing a traditional research track (with prizes for innovative papers) and a hackathon , to live and breath the challenges with working code (with prizes for the best projects).

Important: Registration to KDD is required to attend the workshop, but submitting papers is not required to participate in the hackathon. The hackathon and social event is an affiliated event organized with the help of our generous sponsors, Snap , Costanoa and by Bauplan Labs .

hackathon

"Talk is cheap, show me the code": assemble a team, complete a project, and compete for the prizes!

We'll host a pizza hackathon night, where participants will pursue innovative projects for the rounded evaluation of recommender systems!

A new, open dataset, with notebooks, models, and tests is already available if you want to get into the workshop mood early: check the official repository to get started (yes, you can start your project before the workshop if you desire to do so!). As usual, everything goes back to the community as open source contributions.

Important: The hackathon and social event is an affiliated event organized with the help of our generous sponsors. To participate in the hackathon and win the prizes, you're not required to submit a paper to the workshop: if you can't be in Long Beach, write us an email, as we welcome hybrid participation from teams that can't come in person.

Accepted Papers

Title	Link
Realistic but Non-Identifiable Synthetic User Data Generation	Link
Disentangling the Performance Puzzle of Multimodal-aware Recommender Systems	Link
Bridging Offline-Online Evaluation with a Time-dependent and Popularity Bias-free Offline Metric for Recommenders	Link
Evaluating Recommendation Systems Using the Power of Embeddings	Link
Metric@CustomerN: Evaluating Metrics at a Customer Level in E-Commerce	Link

Schedule

August 7th at KDD 2023

Time	What
1.00 - 1.30 (PM, Pacific Time)	KeyNote Luca Belli
1.30 - 2.00	Papers Presentations
2.00 - 4.30	hackathon Part I
4:30 - 5:00	KeyNote Joey Robinson
Break	Break
6:00-8:00	After Party hackathon Part II

KeyNotes

Joey Robinson, Manager, Web Ad Ranking @ Snap

Bio : Joey is an engineering manager at Snap, where he leads the Web Ad Ranking team. His team owns the development of the CTR and CVR models used for predicting web engagements for Snapchat ads. Prior to Snap, he led the machine learning team at an AI startup (Scaled Inference) and studied computer science and machine learning at Carnegie Mellon University.

Abstract : Ranking and recommendation systems power a variety of Snapchat features, including spotlight, discover, maps, search, lenses, ads, and friending. For all of these products, model evaluation and analysis tooling play a central role at each stage of the model development lifecycle. In this talk, we will discuss the evolution of Snapchat's unified model evaluation system and the key design decisions that allowed it to grow to support more than a dozen ML use cases at scale. We will also discuss the importance of extending model evaluation beyond a single accuracy number, and how a flexible evaluation system can allow engineers to ask and answer questions about their models that help to guide future model improvements.

Luca Belli, founder of Sator Labs, UC Berkeley Tech Policy Fellow.

Bio : Luca Belli is the founder of Sator Labs, a Visiting Nist AI Fellow and a UC Berkeley Tech Policy Fellow. Previously he was the co-founder and Research Lead for Twitter’s Machine learning Ethics, Transparency and Accountability (META) team where he guided industry leading approaches for responsible ML practices and product changes. Before that he operated as a Data Science and Machine Learning Engineer at Conversant and WolframAlpha. His research interests lie at the intersection of feedback loops, algorithmic amplification (with a special eye on politics), and algorithmic audits. He holds a Ph.D. in Math from Tor Vergata University in Rome.