Paper summary: RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

Problem

A companion AI that talks with someone for months should come to understand them. Real conversations are private, so existing benchmarks use invented people and questions.

Method

RealCompanion releases ten real relationships between people and an AI companion: 27,218 messages over up to 120 days. For each person there’s the full conversation, a profile, a persona, and chat and question test items. Every label points to the supporting messages, and chat labels include the reasoning behind them.

Key results

From the abstract, three findings:

1. People rarely refer back. Only 3.4% of messages depend on something said earlier, and then usually far back (a median of 2,157 messages). Looking at recent messages finds the needed one 95.9% of the time overall, but only 2.2% when it’s far back.
2. AI systems can’t tell when the past matters. Detectors barely beat chance, and labelling earlier messages as “memories” makes models bring up the past 10–14 points more often, even when nothing from the past is needed.
3. AI reads more into a person than they revealed. Agent systems rebuilding personas add things the person never said.

Limitations

There’s no dedicated limitations section in the extracted text. Ten relationships is a small, specific sample, a trade-off the authors make for real data.