Question Types in Long-Term Conversation Benchmarks
LoCoMo
≈ 300 turns
≈ 9K tokens / conversation
session 1
session 2
session 3
…
weeks apart
questions asked after the whole conversation
scored by F1 and BLEU-1 against a gold answer
GVD
15 virtual users × 10 days · at least two topics per day · an LLM judge scores retrieval accuracy, correctness, coherence
single-hop
p1 park
p2 squirrels
p3 run
one page
Where did the user go
last week?
needs · one page
answer ·
the wetland park
multi-hop
p1 park
p2 squirrels
p3 run
two pages, linked
What did the user see at the
place where they went running?
needs · two pages joined
answer ·
squirrels
temporal
p1 park
p2 squirrels
p3 run
T · week 1
+ today’s date
How many weeks ago did the
user visit the park?
needs · a page plus its timestamp
answer ·
three weeks ago
open-domain
p1 park
p2 squirrels
p3 run
+ world knowledge · + preferences
What would be a good gift
for the user?
needs · pages plus outside knowledge
answer ·
running shoes
03 / 25