RecSys Challenge 2025 by Synerise

About | Task | Evaluation | Dataset | Download | Timeline

About

The challenge is designed to promote a unified approach to behavior modeling. Many modern enterprises rely on machine learning and predictive analytics for improved business decisions. Common predictive tasks in such organizations include recommendation, propensity prediction, churn prediction, user lifetime value prediction, and many others. A central piece of information that is used for these predictive tasks are logs of past behavior of users e.g., what they bought, what they added to their shopping cart, which pages they visited. Rather than treating these tasks as separate problems, we propose a unified modeling approach.

To achieve this, we introduce the concept of Universal Behavioral Profiles—user representations that encode essential aspects of each individual’s past interactions. These profiles are designed to be universally applicable across multiple predictive tasks, such as churn prediction and product recommendations. By developing representations that capture fundamental patterns in user behavior, we enable models to generalize effectively across different applications.

Challenge Task

The objective of this challenge is to develop Universal Behavioral Profiles based on the provided data, which includes various types of events such as product buy, add to cart, remove from cart, page visit, and search query. These user representations will be evaluated based on their ability to generalize across a range of predictive tasks. The task of the challenge participants is to submit user representations, which will serve as inputs to a simple neural network architecture. Based on the submitted representations, models will be trained on several tasks, including some that are disclosed to participants, called "open tasks," as well as additional hidden tasks. The final performance score will aggregate results from all tasks. We iterate model training and evaluation automatically upon submission, the task of the participants is to submit universal user representations.

Open Tasks:

Hidden Tasks:

In addition to the open tasks, the challenge includes hidden tasks, which remain undisclosed during the competition. The purpose of these tasks is to ensure that submitted Universal Behavioral Profiles are capable of generalization rather than being fine-tuned for specific known objectives. Similar to the open tasks, the hidden tasks focus on predicting user behavior based on the submitted representations, but they introduce new contexts that participants are not explicitly optimizing for.

After the competition concludes, the hidden tasks will be disclosed along with the corresponding code, allowing participants to replicate results.

Evaluation

The primary metric by which we measure model performance is AUROC. Additionally, the performance of category propensity and product propensity models is evaluated based on the novelty and diversity of the results. In these cases, the task’s score is derived as a weighted sum of all metrics, specifically 0.8 × AUROC + 0.1 × Novelty + 0.1 × Diversity.

For each task, a leaderboard is created based on the respective task scores. The final score, which evaluates the overall quality of user representations and their ability to generalize, is determined by aggregating ranks from all per-task leaderboards using Borda count method. In this approach, each model's rank in a task leaderboard is converted into points, where a model ranked k-th among N participants receives (N - k) points. The final ranking is based on the total points accumulated across all tasks, ensuring that models performing well consistently across multiple tasks achieve a higher overall score.

DataSet

The challenge organizers will publish an anonymized dataset containing real-world user interaction logs. All recorded interactions can be utilized to create Universal Behavioral Profiles; however, participants will be required to submit behavioral profiles only for a subset of users, which will be used for model training and evaluation.

The data will consist of four types of events and product attributes:
product_buy

add_to_cart

remove_from_cart

product_properties

page_visit

search_query

Data Format

We provide a data folder containing event files and two subdirectories: `input` and `target`.  

1. Event and properties files

The event data is divided into five Parquet files. Each file corresponds to a different type of user interaction available in the dataset

Product properties are stored in:

2. `input` directory

This directory stores a NumPy file containing a subset of 1,000,000 client_ids for which Universal Behavioral Profiles should be generated:

- relevant_clients.npy

3. `target` directory

This directory stores the labels for propensity tasks. For each propensity task, target category names are stored in NumPy files:

- propensity_category.npy: Contains a subset of 100 categories for which the model is asked to provide predictions

- popularity_propensity_category.npy: Contains popularity scores for categories from the propensity_category.npy file. Scores are used to compute the Novelty measure.

- propensity_sku.npy: Contains a subset of 100 products for which the model is asked to provide predictions

- popularity_propensity_sku.npy: Contains popularity scores for products from the propensity_sku.npy file. These scores are used to compute the Novelty measure.

- active_clients.npy: Contains a subset of relevant clients with at least one product_buy event in history (data available for the participants). Active clients are used to compute churn target.

Download

Universal Behavioral Modeling Dataset © 2025 by Synerise SA is licensed under Creative Commons Attribution-NonCommercial 4.0 International. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc/4.0/

Download dataset

We provide a competition repository that includes a baseline solution and a training pipeline, allowing participants to run experiments on open tasks:

Repository

Timeline

Add to calendar

10 March, 2025
Start RecSys Challenge Release dataset

10 April, 2025
Submission System Open Leaderboard live

15 June, 2025

End RecSys Challenge

20 June, 2025
Final Leaderboard & Winners EasyChair open for submissions

26 June, 2025
Code Upload Upload code of the final predictions

7 July, 2025

Paper Submission Due

24 July, 2025

Paper Acceptance Notifications

2 August, 2025

Camera-Ready Papers

September 2025

RecSys Challenge Workshop @ ACM RecSys 2025