Recent Work in Hierarchical Document Retrieval
& HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering

2026. 09. presenter: Joohyung Yun
PART 1
Motivation
Part 1 Motivation · why trees
Part 2 A Tour of Tree-Based Search · the lineage
Part 3 HiKEY · deep dive (ACL 2026)
01 / 32

Question Answering over a Document Collection

Q. How often should the battery of model X100 be inspected? what to read? hundreds of product manuals Install Guide X200 Manual Warranty X300 Manual Service Guide Parts List Safety Rules FAQ X100 Manual … × hundreds Inside the X100 manual: the path to the answer 📕 X100 User Manual └ 5. Maintenance └ 5.3 Battery └ Table: inspection intervals part interval cooling fan12 months battery6 months answer = one document → one section → one table cell
  • Goal · find the evidence in a document pile and answer · this example recurs through the deck
02 / 32

Flat Retrieval by Chunking and Similarity Matching

documents with structure + hierarchy ① chunk uniform chunks hierarchy thrown away ✂ ② embed semantic space · similar meaning = nearby points ★ ★ query and its top-k nearest chunks ③ hand over Reader (LLM) reads k chunks, writes the answer
  • A single criterion · is the chunk semantically close to the question
  • Simple and fast · still the most widely used baseline
03 / 32

Failure Modes of Flat Retrieval

① A chunk carries no context

X100 › Maintenance › Battery ✂ "… inspection interval: 6 months …" which model? which part?

the severed heading path was its identity

② Evidence is scattered

table: 6 months §2 caution: halve it in high heat they answer only together, yet travel apart

worse when the answer spans table, figure, and text

③ Ambiguity at corpus scale

X100 · battery inspection 6 months X200 · battery inspection 12 months X300 · battery inspection 3 months ?

hundreds of docs × hundreds of chunks · near-identical sentences compete

  • Common cause · structure is discarded and chunks are judged in isolation
04 / 32

Searching by the Document Hierarchy

how a person searches Table of Contents 1. Introduction 2. Safety Precautions 3. Installation 4. Usage 5. Maintenance ← 5.3 Battery ← Table: intervals ← arrival reads only headings, rules out whole branches = the same strategy, mechanized: tree traversal X100 2. Safety 4. Usage 5. Maint. 5.1 Clean 5.3 Battery Table ✓ not opened = never read
  • Choose branches from headings and summaries alone · reading shrinks sharply
  • A ToC is just the document's hierarchy · tree-based search = this strategy, mechanized
05 / 32

A Common Frame: Build, Traverse, Pack

① BUILD give each document a hierarchy (tree) with headings and summaries offline · once per corpus ② TRAVERSE per question, narrow the tree down to the places worth reading online · per question ③ PACK bundle the chosen evidence to fit the reader budget budget gauge → reader writes the answer every method that follows = a different way to fill these three boxes
06 / 32
PART 2
A Tour of Tree-Based Search
Part 1 Motivation · why trees
Part 2 A Tour of Tree-Based Search · the lineage
Part 3 HiKEY · deep dive (ACL 2026)
07 / 32

Two Design Axes from Build and Traverse

BUILD TRAVERSE PACK decided at BUILD Q1 · Where does the hierarchy (tree) come from? a hierarchy must exist before anyone can traverse it · BUILD has to get one from somewhere the document's own hierarchy headings become the tree, as-is + free · faithful to the author − only as good as its formatting machine-built (summaries · learned) cluster + summarize (generated) + works on unstructured text − generated: cost · can drift BUILD TRAVERSE PACK decided at TRAVERSE Q2 · Who decides where to go next? at every fork someone must compare the branches with the question and choose score-based traversal (word overlap · embeddings) q 0.92 0.15 the bigger number wins · no thinking + cheap · 0 LLM calls · scales − matches only, cannot reason LLM-guided traversal (reads and reasons) LLM reads both children, then chooses + reasons · follows references − calls per query: slow, costly each method answers both questions → next slide places all seven on this 2×2 map
08 / 32

Design Space of Tree-Based Search

uses the document's own hierarchy machine builds the hierarchy (summaries · learning) score-based traversal word overlap · semantic sim · 0 LLM calls per query LLM-guided traversal reads and reasons its way, per query HiKEY the only multimodal one today's paper TreeRAG PageIndex RDR2 trains the traversal RAPTOR MemWalker score-based traversal LLM-guided traversal TIMELINE MemWalker arXiv 2023 RAPTOR ICLR 2024 PageIndex open-source 2025 TreeRAG ACL Findings 2025 RDR2 EMNLP 2025 HiKEY ACL 2026
09 / 32

RAPTOR: Summary Trees by Recursive Clustering

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
BUILD · the machine writes the hierarchy chunks (embedded) cluster summary A cluster summary B top summary cluster similar ones, LLM summarizes · repeat ↺ TRAVERSE · it barely traverses flatten all levels into one pool (collapsed) top-k by similarity · summaries (rounded) mixed with chunks (square) q
  • The starting point (ICLR 2024) · summaries at several resolutions become searchable
  • Irony · it builds a tree, yet flat similarity over the pool worked better
10 / 32

TreeRAG: Bidirectional Tree Traversal

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
document section 1 section 2 chunk chunk chunk ★ chunk top→down · follow the hierarchy bottom→up · chunk first up one hop, then all leaves under it q similarity grabs a chunk first
  • Two directions · up from the hit chunk, then down to the leaves under its parent
  • Lost context is repaired at BUILD: ancestor titles are prefixed into every chunk before embedding
11 / 32

TreeRAG: Retrieval Procedure

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
0 · intent check LLM, once per query "does it ask to list or summarize?" yes no · skip the traversal entirely: the initial top-K chunks go straight to the reader 1 · initial retrieval (top-K) root § A § B chunk chunk hit ★ chunk embeddings hit leaf chunks each chunk was embedded with its ancestor-title prefix (from BUILD) 2 · up one hop → 3 · all its leaves root ✗ no higher § A § B chunk chunk hit ★ sibling stops at the immediate parent collects every leaf beneath it repeated for each of the K hits · no iteration 4 · re-rank → final chunk set hit chunk ★ sibling chunk sibling chunk leaf chunks only, re-ranked parents · summaries never returned → handed to the reader a single pass, no iteration: each hit gets one hop up + one leaf expansion, then re-rank and stop · nothing is revisited
12 / 32

MemWalker: LLM Navigation over Summary Trees

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
root summary branch summary A branch summary B segment 1 segment 2 segment 3 segment 4 wrong turn, back up LLM reads child summaries, picks one q original goal: reading documents longer than the reader's context window
  • The traversal goes to an LLM · read summaries, reason, descend · back up on a wrong turn
  • The cost · several LLM calls per question (one per step)
13 / 32

MemWalker: Navigation Procedure

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
0 · build the memory tree seg 1 seg 2 seg 3 … summ A summ B root summ = LLM(children) · recursed to a root offline · query-independent 1 · branch selection at every node, from the root root · LLM summ A summ B seg 1 seg 2 seg 3 seg 4 enter sees the children's summaries + the query Reasoning: B mentions the war… Action: 1 → enter summ B menu at a non-leaf: enter child 0…N or revert · one LLM call per node ↻ not a leaf → select again, one level down at a leaf 2 · at a leaf: read the real segment root summ A summ B seg 1 seg 2 seg 3 ★ seg 4 [working memory] rides the path commit −2 → answer the leaf holds the full segment, not a summary commit ends the walk · revert climbs back and enters a sibling revert −1 → select again at the parent output · the answer answer text + the path as explanation a reader, not a retriever: no chunk set is handed to a separate model terminates when the LLM commits at a leaf, or after 3 unparsable replies ("no answer") · a single path, no beam not tool-calling: two zero-shot prompts (branch-selection · leaf) fix an output format · the LLM writes Reasoning / Action as plain text, parsed each step
14 / 32

PageIndex: Retrieval by Reasoning over the Hierarchy

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
the hierarchy index · fetched whole title pages X100 User Manual1-52 1. Introduction1-4 2. Safety5-9 2.1 Operating Env.7-8 5. Maintenance30-44 5.1 Cleaning30-33 5.2 Filter34-37 5.3 Battery38-41 5.4 Firmware42-44 6. FAQ45-48 every node carries its own page range a parent range = union of its children 1 · reason over it → which pages to read LLM sees titles + ranges · no page text yet Thinking: intervals live under Maintenance; heat may matter → check Operating Env. too picks titles copies their stored range 2.1 Operating Env. 7-8 5.3 Battery 38-41 get_page_content("7-8, 38-41") the numbers are copied from the index, never guessed one call carries several ranges = several sections choosing a parent instead (5. Maintenance → 30-44) pulls every subsection 2 · read those pages, then judge p.7-8 p.38-41 · 5.3 Battery fan 12 mo battery 6 mo found: battery → 6 months and from p.7-8 · halve it in high heat enough to answer? the LLM itself decides · no scorer it may also follow a reference it just read ("see Appendix G") back into step 1 output enough → answer "every 6 months, 3 in high heat" + the sections it opened 5.3 Battery · 2.1 Operating Env. as citations sections, not chunks a reader, not a retriever not enough ↻ pick other titles, call again state between rounds = the tool history already in the prompt · what it read stays there, there is no note store
  • Reasoning, not similarity · an LLM matches titles, page numbers are looked up · zero embeddings
15 / 32

RDR2: Routing over Document Structure

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
Retrieve top-k DocumentRoute Read one document at a time, in a loop the document tree as it stands now 0: =Wonder Woman in other media= 2: ==Live action== 3: ===Theatrical releases=== 4: Gal Gadot portrays Diana… 5: ===Television=== 8: ==Animation== 10: ==Video games== every heading is always there · only passages open and close open now: the passage retrieval landed on, plus its siblings ▸ collapsed heading ▾ opened heading one step of the router ★ the router is the only trained part reads the question + the tree on the left [EXP] 5 open heading 5, close what was open EXPLORE [ANS] 4 take passage 4 out, the tree is untouched HARVEST [REF] −1 not chosen this step — would end this document STOP an action is ⟨tag, node id⟩ · several per step only [EXP] rewrites the tree → only [ANS] writes to the pile below ↓ after [EXP] 5 · the same tree, rewritten 0: =Wonder Woman in other media= 2: ==Live action== 3: ===Theatrical releases=== ← closed 5: ===Television=== ← opened 6: Cathy Lee Crosby portrays… 7: Lynda Carter portrays… 8: ==Animation== 10: ==Video games== identical headings · one window of text swapped for another the tree never grows, so the prompt stays a fixed size kept passages outside the tree what [ANS] pulls out of the tree, step after step — the tree never holds it 4: Gal Gadot portrays… 6, 7 next grows until [REF] Reader one call · q + every pile answer
16 / 32

RDR2: Training the Router

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
the obstacle · standard RAG data gives a question and an answer, but no routing trajectory to imitate action curation · manufacture the trajectories from the question alone ① a question "who played wonder woman?" the answer is never used so any question set works ② build the state off-the-shelf retriever → top-k their originating documents → retrieval subtree s the same state the router sees at inference ③ a teacher writes the actions DeepSeek-v3 · sees (q, s) [EXP] 5: ===Television=== [ANS] 6,7: Cathy Lee Crosby… [REF] −1 one step's worth of actions, not a full trajectory ④ the training set ⟨ q , s , A ⟩ ASQA training questions 23,827 samples no human annotation anywhere supervised fine-tuning · plain next-token prediction ℒ = − log P(A | q, s) cross-entropy on the output tokens only Llama-3.1-8B-Instruct + LoRA · 3.5 epochs trained on single steps, never on whole trajectories the model learns "what to do in this state"; the iteration is left to inference the output format carries the node id "[ACTION] id: text_prefix" — every action is bound to a real node
17 / 32

Comparison of Tree-Based Search Methods

① a tree, no traversal→② score-based, both ways→③ LLM-guided instead→④ traversal learned→⑤ multimodal
MethodWho writes the hierarchyWho traverses itLLM calls / queryTable · figure evidence
RAPTORmachine · pyramid of cluster summariesno traversal · flat similarity over all levels0✗
TreeRAGthe document's own hierarchysimilarity · bidirectional (up + down)0✗
MemWalkermachine · segment summariesLLM reasoning + backtrackingper step✗
PageIndexthe document's own hierarchyLLM reasoning (no vectors)iterative✗
RDR2the document's own hierarchya trained LLM navigatorcalls✗
HiKEYthe document's own hierarchy (auto-parsed)scores, two stages · pick docs → pick sections0 (reader only)✓ first-class
  • One blank remains · documents whose key evidence sits in tables and figures → Part 3
18 / 32
PART 3
HiKEY: Hierarchical Multimodal Retrieval
Part 1 Motivation · why trees
Part 2 A Tour of Tree-Based Search · the lineage
Part 3 HiKEY · deep dive (ACL 2026)
19 / 32

Two Bottlenecks in Industrial Document QA

① Can't find it wrong document from the start

Q. X100 battery inspection interval? ? ? ? thousands of docs · similar titles · similar sentences

when a question needs several docs, missing one = failure

② Can't assemble it evidence fragments even when found

even inside one document text p.12 table p.39 figure p.7 reader budget all of it overflows, cutting loses evidence

industrial docs = long PDFs · tables and figures carry the evidence · budget is finite

  • HiKEY's diagnosis · both are symptoms of search that ignores document structure
20 / 32

Overview of HiKEY

BUILD documents → trees, auto-parsed two kinds of index cards doc cards · section cards offline, once TRAVERSE ① Pick Documents thousands → a few candidates score = word overlap + semantic sim target = title + heading paths X100 Service FAQ TRAVERSE ② Pick Sections candidates → top-K sections section score = its single best- matching unit (text·table·figure) 5.3 Battery › table ★ PACK assemble within budget key unit + heading path + neighboring tables·figures → one reader call → answer zero LLM calls during search · narrowing is pure scoring → cheap and fast
21 / 32

Building Document Trees and Index Cards

a PDF page paragraph table figure auto-parse tree · every unit gets its heading path X100 Manual 2. Safety 5. Maintenance 5.3 Battery text · "consum…" table · intervals figure · location a caption-less figure gets its preceding paragraph attached as context two kinds of search cards Doc card · one per document X100 User Manual ├ 2. Safety └ 5. Maintenance › 5.3 Battery … every path = title + the whole hierarchy skeleton · paths only, no contents Section card · one per section § 5.3 Battery · path: X100 › 5. Maintenance text "consum…" table intervals figure location = its name (path) + every component beneath it, each searchable components keep their native form · tables & figures never flattened to text
  • Structure becomes a first-class search signal · heading paths live inside the cards
  • Tables and figures become first-class targets · the blank in the lineage, filled
22 / 32

Traversal Stage 1: Selecting Candidate Documents

inputs color code · the query · doc cards · pass Q. X100 battery inspection interval? doc cards title + heading paths every document · × hundreds score(doc, q) BM25 word overlap → 0.88 dense emb. meaning close → 0.94 α⊕ 0.92 α : 1−α both min-max normalized before the blend every doc scored, then ranked X100 Manual · "…5.3 Battery…" 0.92 Service Guide · "…battery handling…" 0.71 cutoff · these pass ✓ X200 Manual · similar, wrong model 0.55 Install Guide · no battery at all 0.18 the query word "battery" meets the heading path directly · structure boosts the score output only a few pass ✓ X100 Manual ✓ Service Guide thousands → a few
  • Thousands → a few · the cheapest computation prunes the largest space
23 / 32

Traversal Stage 2: Scoring Sections

input · Stage-1 output ✓ X100 Manual ✓ Service Guide section cards text · tables · figures every section of the candidates non-candidates: never opened score(section, q) text unit BM25 ⊕α dense the Stage-1 blend, reused 0.64 visual emb. the unit as an image nearby text BM25 ⊕α dense on ctx γ⊕ 0.95 0.41 max over units S_sec = 0.95 ★ the winning unit is called the section's anchor ★ one decisive hit is enough, no averaging S_final = λ·S_doc ⊕ (1−λ)·S_sec the Stage-1 doc score is fused back in output · top-K sections X100 › 5.3 Battery › table ★ 0.95 X100 › 2. Safety › table ★ (heat caution) 0.62 top-K pass ✓ · K = 2 · each with its anchor ★ X100 › 5.1 Cleaning 0.22 Service Guide › 1. Overview 0.10 final rank = doc score ⊕ section score → handed to PACK
24 / 32

Packing Evidence under a Reader Budget

input · Stage-2 output + budget X100 › 5.3 Battery › table ★ section + anchor · 0.95 X100 › 2. Safety › table ★ section + anchor · 0.62 the top-K sections · here K = 2 reader budget a fixed number of tokens note: rule ③ may pull visuals from outside the selected sections pack(sections, budget) ① anchor + heading path = the Stage-2 winner ★ +2.1K tok ② sibling tbl·fig only same parent section as ★ layout decides · +1.4K tok ③ distant visuals high sim(unit, ★) · corpus-wide similarity decides · +0.9K tok budget left? yes → append ✓ no → stop ✂ slack greedy, per selected section · one pass budget shrinks → ③ then ② dropped, ① survives e.g. inside the §5.3 card ✗ not tbl·fig ① = ★ ② ✓ outside · by sim to ★ ③ ✓ · .81 ✗ · .22 output · what the reader receives packed context · serialized, ≤ budget ① X100 › 5. Maint. › 5.3 Battery 0.3K ① intervals · battery 6 mo ★ 1.8K ② battery location 1.4K ① X100 › 2. Safety 0.2K ① heat caution · halve it ★ 0.7K ③ §6 charging chart · similar 0.9K total ≈ 5.3K of 16K heading paths ride along · provenance visible to the reader q Reader (LVLM) one call · q + context "every 6 months, 3 in high heat" one prompt, one call · nothing else is sent
25 / 32

HiKEY versus the Surveyed Methods

Their betWhat they claimedWhat HiKEY does instead
RAPTOR · generated summary pyramidmulti-level summaries make retrieval smarteruses the hierarchy the document already has · nothing generated on the retrieval path
TreeRAG · bidirectional traversaltraversing upward recovers the lost contextcontext is restored at PACK: ancestor headings ship with the evidence, no traversal needed
MemWalker · PageIndex · LLM navigationreasoning picks the right branchsame coarse-to-fine narrowing by plain scores · zero LLM calls per query
RDR2 · learned routingtraining makes the traversal accurateno training · works out of the box on a new corpus
left open by all of themtables and figures as evidencefirst-class multimodal units + budget-aware packing · the genuinely new part
Every ingredient looks simple because it is borrowed. The contribution is where the pieces sit: structure enters at BUILD and PACK instead of a smarter TRAVERSAL, and the whole system runs on one reader call. The experiments have to show the combination wins.
26 / 32
PART 3
HiKEY: Experimental Results
Part 1 Motivation · why trees
Part 2 A Tour of Tree-Based Search · the lineage
Part 3 HiKEY · deep dive (ACL 2026)
27 / 32

Experimental Setup

Benchmarks · corpus-level: one joint index

M3DocVQAmulti-page PDFs · evidence in text, tables, figures · multi-hop
FRAMES-PDFWikipedia questions re-rendered as PDFs · one query needs several documents

Competitors · 14 baselines

text chunkspage / length / structural chunking · LumberChunker · MultiDocFusion …
text graphRAPTOR (from Part 2) · HopRAG
page-level MMM3DocRAG · VDocRAG
MM graphMoLoRAG · SimpleDoc

Metrics

retrievalRecall · MRR · Hit · All (every needed doc found · miss one = fail)
QAEM (exact match) · ANLS · ROUGE-L · METEOR
Fair control · every method gets the same reader (Qwen2.5-VL 7B), the same 16K token budget, and the same encoders (keyword + text embedding + visual embedding) → a difference between rows is a difference of method, not of budget or encoder
28 / 32

Document-Level Retrieval Accuracy

Did it find the right documents · avg recall (mean of two benchmarks)

HiKEY
79.1
MultiDocFusionbest of text chunks
74.6
HopRAGbest of text graph
74.1
RAPTORtext graph · Part 2
73.4
SimpleDocbest of MM graph
69.0
VDocRAGbest of page-level MM
67.8

Did it find all of them · All (miss one = fail)

HiKEY
59.9
MultiDocFusionbest of text chunks
52.6
HopRAGbest of text graph
51.8
RAPTORtext graph · Part 2
50.9
SimpleDocbest of MM graph
46.0
VDocRAGbest of page-level MM
44.9

one bar per family: the strongest of the 14 baselines (+ RAPTOR from Part 2) · full 14-row tables are in the paper

On the multi-document benchmark (FRAMES), All improves by +11.4pt · hierarchy-path routing misses fewer of the scattered documents
29 / 32

Token-Budget Robustness and End-to-End QA

Does it survive a tighter budget document recall@10

BudgetHiKEYMultiDocFusionpage methods
0.5K tokens78.874.3cannot adapt
(whole pages)
1K tokens79.174.6
2K tokens81.376.8

the Pack stage at work · still best at 1/32 of the full budget

Do the answers improve EM · exact match, mean of two benchmarks

HiKEY
19.0
MultiDocFusionbest of text chunks
15.4
HopRAGbest of text graph
15.3
SimpleDocbest of MM graph
15.3
RAPTORtext graph · Part 2
15.0
VDocRAGbest of page-level MM
14.5

largest gaps on table/figure-grounded questions and multi-hop (3+) questions

30 / 32

Ablation Study

document recall (Avg R@10) when one design choice is removed · bar length = the value on a 0–100 axis

hierarchy in the index · does the Doc card need it?

body text only
81.2
+ titles, concatenated
84.6
separate hierarchy field
88.6

hierarchy as an explicit signal: +7.4 over no hierarchy

two-stage routing · is coarse-to-fine needed?

sections only
76.1
documents only
87.7
docs → sections
88.6

skipping the document stage collapses recall (−12.5)

search scope · does narrowing help at all?

whole corpus, no routing
81.0
candidate docs only
87.9
+ anchor subtree
88.6

routing itself is worth +7.6

score signals · does each one earn its place?

keywords only
87.6
+ text embeddings
88.2
+ visual embeddings
88.6

each signal adds a little · the visual one completes it

No latency or cost experiment in the paper · the efficiency case rests on zero query-time LLM calls (by design) and the budget robustness of the previous slide; offline DHP indexing overhead is acknowledged as a limitation, without numbers
31 / 32

Takeaways

The hierarchy is a free signal

Documents already carry the structure that should steer search. This is why the lineage moved from chunk similarity to hierarchy traversal.

Narrow, then bundle

HiKEY's answer: narrow in two stages, documents → sections (zero LLM calls), then fill the budget core-first. A combination tuned to the industrial setting: thousands of docs, tables and figures, a finite reader budget.

The open question (discussion)

Whichever direction the traversal takes, and however it is trained, it happens over headings and summaries. A fact they fail to carry is invisible to every method here. Who breaks this limit?

One line · document-QA search has become a hierarchy-traversal problem, and HiKEY is where that traversal converges for multimodal industrial documents: cheap and robust
32 / 32