RecSys Challenge 2025 by Synerise
About | Task | Evaluation | Dataset | Download | Timeline
About
The challenge is designed to promote a unified approach to behavior modeling. Many modern enterprises rely on machine learning and predictive analytics for improved business decisions. Common predictive tasks in such organizations include recommendation, propensity prediction, churn prediction, user lifetime value prediction, and many others. A central piece of information that is used for these predictive tasks are logs of past behavior of users e.g., what they bought, what they added to their shopping cart, which pages they visited. Rather than treating these tasks as separate problems, we propose a unified modeling approach.
To achieve this, we introduce the concept of Universal Behavioral Profiles—user representations that encode essential aspects of each individual’s past interactions. These profiles are designed to be universally applicable across multiple predictive tasks, such as churn prediction and product recommendations. By developing representations that capture fundamental patterns in user behavior, we enable models to generalize effectively across different applications.
Challenge Task
The objective of this challenge is to develop Universal Behavioral Profiles based on the provided data, which includes various types of events such as product buy, add to cart, remove from cart, page visit, and search query. These user representations will be evaluated based on their ability to generalize across a range of predictive tasks. The task of the challenge participants is to submit user representations, which will serve as inputs to a simple neural network architecture. Based on the submitted representations, models will be trained on several tasks, including some that are disclosed to participants, called "open tasks," as well as additional hidden tasks. The final performance score will aggregate results from all tasks. We iterate model training and evaluation automatically upon submission, the task of the participants is to submit universal user representations.
Open Tasks:
- Churn Prediction: Using the submitted user representations, the model will predict whether an active user (defined as one with at least one product-buy event) will churn, i.e., make no purchases within the next 14 days.
- Product Propensity: Using the submitted user representations, the model will predict products which a user is most likely to purchase within the next 14 days, from a predefined subset of items.
- Category Propensity: Using the submitted user representations, the model will predict the categories in which a user is most likely to make a purchase within the next 14 days, from a predefined subset of categories.
Hidden Tasks:
In addition to the open tasks, the challenge includes hidden tasks, which remain undisclosed during the competition. The purpose of these tasks is to ensure that submitted Universal Behavioral Profiles are capable of generalization rather than being fine-tuned for specific known objectives. Similar to the open tasks, the hidden tasks focus on predicting user behavior based on the submitted representations, but they introduce new contexts that participants are not explicitly optimizing for.
After the competition concludes, the hidden tasks will be disclosed along with the corresponding code, allowing participants to replicate results.
Evaluation
The primary metric by which we measure model performance is AUROC. Additionally, the performance of category propensity and product propensity models is evaluated based on the novelty and diversity of the results. In these cases, the task’s score is derived as a weighted sum of all metrics, specifically 0.8 × AUROC + 0.1 × Novelty + 0.1 × Diversity.
For each task, a leaderboard is created based on the respective task scores. The final score, which evaluates the overall quality of user representations and their ability to generalize, is determined by aggregating ranks from all per-task leaderboards using Borda count method. In this approach, each model's rank in a task leaderboard is converted into points, where a model ranked k-th among N participants receives (N - k) points. The final ranking is based on the total points accumulated across all tasks, ensuring that models performing well consistently across multiple tasks achieve a higher overall score.
DataSet
The challenge organizers will publish an anonymized dataset containing real-world user interaction logs. All recorded interactions can be utilized to create Universal Behavioral Profiles; however, participants will be required to submit behavioral profiles only for a subset of users, which will be used for model training and evaluation.
The data will consist of four types of events and product attributes:
product_buy
- client_id (int64): Numeric ID of the client (user).
- timestamp (object): Date and time of the event in the format YYYY-MM-DD HH:mm:ss.
- sku (int64): Numeric ID of the item.
add_to_cart
- client_id (int64): Numeric ID of the client (user).
- timestamp (object): Date and time of the event in the format YYYY-MM-DD HH:mm:ss.
- sku (int64): Numeric ID of the item.
remove_from_cart
- client_id (int64): Numeric ID of the client (user).
- timestamp (object): Date and time of the event in the format YYYY-MM-DD HH:mm:ss.
- sku (int64): Numeric ID of the item.
product_properties
- sku (int64): Numeric ID of the item.
- category (int64): Numeric ID of the item category.
- price (int64): Numeric ID of the item's price bucket.
- embedding (object): A textual embedding of a product name, compressed using the product quantization method.
page_visit
- client_id (int64): Numeric ID of the client.
- timestamp (object): Date and time of the event in the format YYYY-MM-DD HH:mm
- url (int64): Numeric ID of a visited URL. The explicit information about what (e.g., which item) is presented on a particular page is not provided.
search_query
- client_id (int64): Numeric ID of the client.
- timestamp (object): Date and time of the event in the format YYYY-MM-DD HH:mm:ss.
- query (object): The textual embedding of the search query, compressed using the product quantization method.
Data Format
We provide a data folder containing event files and two subdirectories: `input` and `target`.
1. Event and properties files
The event data is divided into five Parquet files. Each file corresponds to a different type of user interaction available in the dataset
- product_buy.parquet
- add_to_cart.parquet
- remove_from_cart.parquet
- page_visit.parquet
- search_query.parquet
Product properties are stored in:
- product_properties.parquet
2. `input` directory
This directory stores a NumPy file containing a subset of 1,000,000 client_ids for which Universal Behavioral Profiles should be generated:
- relevant_clients.npy
3. `target` directory
This directory stores the labels for propensity tasks. For each propensity task, target category names are stored in NumPy files:
- propensity_category.npy: Contains a subset of 100 categories for which the model is asked to provide predictions
- popularity_propensity_category.npy: Contains popularity scores for categories from the propensity_category.npy file. Scores are used to compute the Novelty measure.
- propensity_sku.npy: Contains a subset of 100 products for which the model is asked to provide predictions
- popularity_propensity_sku.npy: Contains popularity scores for products from the propensity_sku.npy file. These scores are used to compute the Novelty measure.
- active_clients.npy: Contains a subset of relevant clients with at least one product_buy event in history (data available for the participants). Active clients are used to compute churn target.
Download
Universal Behavioral Modeling Dataset © 2025 by Synerise SA is licensed under Creative Commons Attribution-NonCommercial 4.0 International. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc/4.0/
Download dataset
We provide a competition repository that includes a baseline solution and a training pipeline, allowing participants to run experiments on open tasks:
Timeline
10 March, 2025
Start RecSys Challenge Release dataset
10 April, 2025
Submission System Open Leaderboard live
15 June, 2025
End RecSys Challenge
20 June, 2025
Final Leaderboard & Winners EasyChair open for submissions
26 June, 2025
Code Upload Upload code of the final predictions
7 July, 2025
Paper Submission Due
24 July, 2025
Paper Acceptance Notifications
2 August, 2025
Camera-Ready Papers
September 2025
RecSys Challenge Workshop @ ACM RecSys 2025