HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning

Awais Rauf¹ Ahmed Hassan² Greg Slabaugh³

¹Queen's University Belfast · ²AMD · ³Queen Mary University of London

Paper Code Trajectories BibTeX

TL;DR: We study how to improve the video understanding of LLMs. Vision-language models perform fine-grained perception and multi-step temporal reasoning jointly in latent space, limiting their ability to plan and execute multi-step strategies over long videos. We show that decoupling the two and reformulating long video understanding as iterative programmatic probing of a hierarchically segmented video yields substantial improvements. With GPT-5-mini and Qwen3-VL-2B, HPP reaches 66.4% accuracy on LongVideoBench while using up to 16× fewer tokens than standard frame sampling.

Method

A reasoning LLM starts with minimal information, devises a strategy, and executes it using code that probes the video through probing functions.

HPP overview: an LLM with minimal context writes code in a REPL that calls search, perception, sampling and structure functions over the video, then answers with FINAL()

1Information-Density-Aware Hierarchy

Shot boundaries and a motion score per shot come straight from the compressed bitstream (I-frames, motion vectors, residuals), without decoding frames. Shots are grouped into scenes and clusters for coarse-to-fine navigation.

Raw frames are converted to shots using compressed-video I-frames, motion vectors and residuals

2Late-Interaction Semantic Retrieval

The LLM searches shots and scenes with text queries. Frames are sampled in proportion to motion, and scores combine token–frame MaxSim with global similarity, which separates visually similar shots.

Late-interaction search: motion-adaptive frame sampling, MaxSim between text and frame tokens plus global similarity

3Structured Probing with On-Demand Perception

Probing functions (search, sample a shot, zoom, frames in a range, clips) localize evidence coarse to fine. The 2B VLM is called only on the frames the LLM picks, and the reasoning stays in code.

Coarse-to-fine probing: scene-level search, shot-level refinement, dense frame sampling and VLM verification

Probing Example

More Trajectories →

Results

1) HPP with GPT-5-mini and Qwen3-VL-2B

LongVideoBench	EgoSchema	MLVU	Video-MME
66.4	70.4	69.3	71.8

Accuracy (%).

2) Decoupling Perception from Reasoning Along Two Axes

(a) Better Perception, Better Results

HPP with Kimi-K2.6 as the LLM

VLM	Avg. accuracy	Δ
2B	70.3	+10.9
38B	73.5	+7.8

(b) Stronger Coding LLM, Better Results

HPP with Qwen3-VL-2B as the VLM

LLM	Avg. accuracy	Δ
MiniMax-M2.5	59.2	+1.8
Kimi-K2.6	68.3	+10.9
Claude Opus 4.6	76.3	+18.9

Accuracy (%) on a 600-question LongVideoBench subset. Δ: gain over the same VLM used on its own.

3) Resource Usage

HPP uses up to 16× fewer tokens than LLM frame sampling on hour-long videos.

Tokens per query for LLM frame sampling and HPP across 15 sub-tasks, ordered by video length — **Tokens per query** across 15 sub-tasks, ordered by video length.

BibTeX

@inproceedings{rauf2026hpp, title = {HPP: Hierarchical Programmatic Probing for Long Video Understanding by Decoupling Perception and Reasoning}, author = {Rauf, Awais and Hassan, Ahmed and Slabaugh, Greg}, booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, year = {2026} }