Algorithms for User Simulation
Faithful simulators of human behavior can help us train models that better interact and collaborate with people. This post explores some design decisions behind building user simulators and makes connections to prior work in computational cognitive science.
To me, the core difference between user simulation and multi-turn question answering is that the user simulator needs to capture how many people could act. A chat assistant can have a consistent personality and behave similarly across contexts, but a user simulator should preserve the variance from different contexts and produce different behavior trajectories.
I’ll discuss three axes of modeling decisions for user modeling: evaluation and reward modeling, training objective, and input context. I’ll wrap up with why I think user modeling is cool.
But first, some preliminaries.
Preliminaries
Let \(X\) be a human behavior trajectory: a sequence of actions, messages, decisions.
Let \(y\) be an outcome, such as whether a person buys milk or their rating of how annoying a chatbot is, from one to five.
User simulation aims to learn \(p(X)\), or a generative model of human behavior. The starting point for learning \(p(X)\) is often a pretrained language model, which is then fine-tuned on human behavior trajectories using cross entropy loss (supervised fine-tuning, SFT). For a representative training pipeline, see OdysSim from @nlpxuhui et al.
Evaluation and reward modeling
To go beyond SFT training data, we can do reinforcement learning (RL) in the scenarios where we want to model users. To do RL, we need to construct a reward function.
One way to learn a reward function is using the RLHF pipeline. For a particular context or persona, we collect pairs of behavior trajectories and ask a human annotator which one is more realistic. If the annotator prefers trajectory \(X_1\) to \(X_2\), we can turn this preference into scalar scores for both trajectories using a linking function such as Bradley-Terry and train a regression model to predict these scalar scores.
Alternatively, we could construct a reward function by writing down a rubric. We specify several properties a trajectory should satisfy, such as whether the user makes plausible mistakes, whether their tone is consistent with context, and so on. Recent work by @vijaytarian et al., among others, have shown that rubric-based reward functions work better for alignment. This approach is also plausible for user modeling, and maps nicely onto multi-objective RL, which I’ll discuss in the next section.
Finally, instead of reinforcement learning, which relies on a scalar reward function, we can try a combination of verbal reinforcement learning and on-policy distillation. Instead of getting a scalar reward from an LLM judge, we can instead just ask the judge to critique a behavior trajectory. Conditioning on this natural language feedback will change the policy’s probability distribution, and we can convert this change into a training signal using on-policy distillation.
In all of these cases, the reward or feedback function has to generalize to many people. If we trained a reward model just based on my own preferences, then any user simulator trained with the reward model would converge towards acting like me. But as shown by works in cognitive science like Centaur, one gets faithful predictors of human behavior by fine-tuning language models on large datasets of human behavior. Relatedly, establishing scaling laws for human behavior prediction would be useful for user simulation.
Training objective
A first pass would be SFT + RL. But one has to be careful here, because vanilla policy gradient is not designed maintain diversity in actions—it often collapses to some argmax policy. This is fine and perhaps even preferable for a chat assistant, but not desirable for modeling a population that will do different things in the same situation. So what to do?
One possibility is to accept vanilla policy gradient collapsing and condition the user simulation on context: learn \(p(X \mid c)\) instead of \(p(X)\). If you describe people in sufficient detail in the context, then you can (ideally) push all interpersonal variation into the context and focus on making your user simulator take the optimal action, conditioned on personal information. This might make sense if you have very detailed information about people, or can generate very realistic synthetic contexts (see next section).
However, pushing all the variance into the context doesn’t work for situations where one person might have several plausible actions, or where the context disambiguates down to a group of people. In these cases, there is uncertainty in what should the user simulator should do. One can choose training objectives that emphasize lower likelihood but good trajectories, such as MaxRL or Delightful Policy Gradient. The simplest possible thing is to do some entropy regularization tuning, which is already built into PPO.
Another principled approach is to directly optimize your policy to produce sets of actions. Recent algorithms from @AntChen_ (ROSA) and @RyanBoldi (VPO) produce sets of trajectories intended to maximize different kinds of rewards. In the user modeling case, we can train models to model several users simultaneously, given uncertainty over what a user would do. Ultimately, I think a combination of both approaches—conditioning on diverse context and optimizing the policy itself for diversity—will be helpful in training better user simulators.
Input context
Finally, we come to generating contexts, or the seed prompts from which people should be simulated. A starting point is to collect either real data or descriptions of people. The success of using real data depends on how weird or uncommon the behavior is that you want to simulate—it could be expensive or take a long time to sample naturally.
Alternatively, one could set up a data generating process from which you can eventually recover the real data. For example, in @alexisjross et al. they simulate errors of human students by generating incorrect answers, backfilling misconceptions to match the incorrect answer, and checking that the language model’s answer conditioned on the misconception matches the original incorrect answer. One could imagine that this process eventually recovers all possible errors students could have on a set of questions, including the errors collected from real students.
A synthetic data generating process might not yield the same data distribution as real data, but 1) this might not matter and 2) can be ameliorated using techniques like importance sampling. To the first point, some situations don’t need the simulated distribution to be exactly the same as human data. For example, if I’m stress testing a website with simulated human users, then perhaps I just need coverage of edge cases. Second, you can try to match a real data distribution more closely by doing resampling; example from computational social science here.
The research direction of setting up data generating processes that cover human behavior is exciting to me because it connects with recent work in open-endedness and autoresearch. Cognitive science is inherently also open-ended, because we’re trying to model people and culture, both of which change and evolve over time. In this way, building open-ended models of human cognition seems directionally correct.
Why working on user modeling is fun
A person’s capacity for working memory is roughly seven, plus or minus two. For example, if you ask people to memorize a run of digits, the average comes out to around seven. It turns out that working memory is even more limited than that, and the raw number of “slots” you can fill in working memory is around four. But people can hold seven items in memory by creating chunks of concepts. For example, if you asked me to memorize the span of digits 4 0 4 2 0 0 0, I might chunk the span of digits into 404, my hometown area code, and 2000 the year.
People have unique experiences, which will cause different parts of reality to stand out and be memorable. So what makes a random span of digits memorable differs from person to person. Thus, modeling human memory, and human behavior in general, is a complex challenge that often must grapple with the subject’s past. These challenges are inherited by user simulation, a field where I believe computational cognitive science can have significant real world impact.
Lots to do. If you’re interested in reading more about simulating human memory, see this project led by Qihan Wang and @NickATomlin.
P.S. This post discusses training one big simulator model and how not to lose variance as you use it to simulate different people. I didn’t discuss approaches where you use many different simulator models, such as training a LoRA or doing continual learning per person.