LILaC · EMNLP 2025 · Worked Example

How a question about a photo two hops away gets answered

LILaC retrieves from documents that mix paragraphs, tables and images. It builds a two-layer component graph offline, decomposes the query into modality-specific subqueries online, and walks the graph edge by edge, scoring each edge with late interaction over fine-grained subcomponents. Step through one query below.

❓

In the mausoleum built by the Mughal emperor who married Mumtaz Mahal, how many slender minarets surround the central dome?

ANSWERFour

1 / 10

SetupThree Wikipedia-style documents, each a sequence of paragraph / image / table components; one paragraph hyperlinks to the other two pages.

Offline · 1Every component becomes a two-level component tree: sentences under a paragraph, header+row segments under a table, detected objects under an image.

Offline · 2Coarse nodes are linked by a clique inside each document and by hyperlink edges across documents; then every node is embedded once.

Online · 1The query is decomposed by an LLM into three subqueries, each classified into a modality that becomes the embedding instruction.

Online · 2Seed: the top-b (b = 2) coarse nodes by similarity to the whole query — one nearest-neighbour search over V₀, nothing else.

Online · 3From the seeds, every incident edge is a candidate; each edge gets a relevance score computed now, from fine-grained nodes — no edge embeddings exist.

Online · 4Edge-level late interaction: an edge is scored by matching each subquery against the subcomponents hanging off both endpoints, then summing the per-subquery maxima. (Similarities here are illustrative.)

Online · 5Beam search over edges: keep the b best edges, make their endpoints the new candidates, repeat for n_i iterations. The search never scores more than a few dozen edges.

ResultThe final subgraph's nodes are returned as a ranked list; the multimodal LLM answers from the Taj Mahal photo.

✦ Why it worksGranularity, structure and modality are handled by three separate mechanisms — and the expensive one (the LLM) runs once per query, not per hop.

Navigate with ← →, the buttons, the dots, or swipe on touch · 10 frames

How a question about a photo two hops away gets answered

SetupThree Wikipedia-style documents, each a sequence of paragraph / image / table components; one paragraph hyperlinks to the other two pages.

Offline · 1Every component becomes a two-level component tree: sentences under a paragraph, header+row segments under a table, detected objects under an image.

Offline · 2Coarse nodes are linked by a clique inside each document and by hyperlink edges across documents; then every node is embedded once.

Online · 1The query is decomposed by an LLM into three subqueries, each classified into a modality that becomes the embedding instruction.

Online · 2Seed: the top-b (b = 2) coarse nodes by similarity to the whole query — one nearest-neighbour search over V₀, nothing else.

Online · 3From the seeds, every incident edge is a candidate; each edge gets a relevance score computed now, from fine-grained nodes — no edge embeddings exist.

Online · 4Edge-level late interaction: an edge is scored by matching each subquery against the subcomponents hanging off both endpoints, then summing the per-subquery maxima. (Similarities here are illustrative.)

Online · 5Beam search over edges: keep the b best edges, make their endpoints the new candidates, repeat for ni iterations. The search never scores more than a few dozen edges.

ResultThe final subgraph's nodes are returned as a ranked list; the multimodal LLM answers from the Taj Mahal photo.

✦ Why it worksGranularity, structure and modality are handled by three separate mechanisms — and the expensive one (the LLM) runs once per query, not per hop.

Online · 5Beam search over edges: keep the b best edges, make their endpoints the new candidates, repeat for n_i iterations. The search never scores more than a few dozen edges.