Reproducibility and Simulation of Interactive retrieval experiments

General information

• Duration: From 1. 1. 2023 until 28. 2. 2026
• Involved Persons:
  • Prof. Dr.-Ing. Norbert Fuhr
  • Jana Friese, M.Sc.
• Fund: DFG

Description

Although reproducibility of experiments is a topic with increasing importance in information retrieval (IR), there has been hardly any work on reproducing interactive IR experiments. The main idea of our project is to describe actual users by quantitative modeling of their behavior and their content-related decisions during retrieval sessions. These model parameters can then form the basis for comparing two interactive IR experiments with regard to reproducibility, and can also be used for simulating interactive IR, e. g. for evaluating new systems or system variants without (or before) carrying out actual user experiments.

As part of the project, we will define a new framework model for reproducibility of interactive IR experiments, develop new metrics for reproducibility and session quality, design new simulation methods integrating behavior and content-related decisions, develop an open source framework for reproduction and simulation of interactive IR, and perform extensive experiments using existing IR log data as well as observation data from new user experiments.

Publications

• Engelmann, B., Breuer, T., Friese, J.I., Schaer, P. and Fuhr, N., 2024, March. Context-Driven Interactive Query Simulations Based on Generative Large Language Models. In European Conference on Information Retrieval (pp. 173-188). Cham: Springer Nature Switzerland.