Agentic Cinematography in Dynamic Scenes

Input: a source video and a camera instruction. Output: the video re-shot along the requested trajectory. Two VLM agents, each with its own system prompt and structured output, drive the loop. A cinematographer proposes trajectories from the instruction and the scene evidence; a trajectory critic accepts a proposal or returns actionable feedback for the next round, until acceptance or the iteration budget runs out.

A feed-forward 4D reconstruction model (π³) lifts the video into a point cloud with and estimates the source cameras.

Both agents see the scene as top-down orthographic canvases at sparse keyframes. Their isotropic scale maps equal pixels to equal world distances, so a trajectory authored in pixels lifts back into the reconstruction with only a height. Each canvas is annotated with camera symbols, the subject's bounding box and motion path, and pose metadata.

The cinematographer receives the camera instruction, the first source frame, the canvases and metadata, and, on refinement rounds, the previous proposal with the critic's feedback. It authors a sparse trajectory: five JSON fields at each keyframe. The first key is pinned to the source camera, and the keys are interpolated linearly per field into a dense per-frame sequence.

This 2.5D parameterization avoids generating SE(3) poses directly: pitch follows from the two heights, yaw from the horizontal displacement, and roll is fixed to zero.

The critic judges the proposal against the same instruction with its own evidence: the canvases extended with the proposed trajectory, and renders of the reconstruction from the proposed cameras.

The critic either accepts the proposal or rejects it with concrete feedback such as "raise t_34 camera_height by 0.4". The accepted trajectory is the reshoot's target.

Re-aiming a crash zoom. The instruction: do a Tarantino zoom on the punch. The first pass zooms in on the woman throwing the punch and misses the impact and the man's reaction. The critic moves the look-at point for the zoom, and the updated trajectory lands on the contact point.

Keeping a zoom on target. The instruction: zoom in on the newspaper. The first pass tilts down as it zooms and runs into space the reconstruction never covered, so the render goes black from t_24 on. The critic asks for a higher look-at point, and the updated trajectory stays on the paper.

Accepting a first proposal. The instruction: zoom in on the man. The first proposal already narrows the field of view onto the man on the bench, and the critic accepts it without feedback.

Catching an under-covered trajectory. The instruction: track the tennis player while increasing the height of the camera. The first pass raises the camera but pitches it down into the court, so the player leaves the top of the frame and the render goes almost black after the first frames. The critic asks for a higher look-at point, and the updated trajectory rises while keeping the player in frame.

• Backbone dependence: open-weight models score lower, so the headline numbers rely on a frontier model behind a commercial API, which also complicates exact replication.
• A narrow critic: it sees no reference trajectory and only edits sparse keys, so it fixes framing but cannot tell that a move is too short, and its edits shrink the path.
• Coupled parameterization: pitch is derived from the two heights and their ground distance, so it cannot be set independently of placement, and it drifts between keyframes because the fields interpolate linearly while pitch is nonlinear in them.
• Benchmark mismatch: DataDoP trajectories are the source shot's own path, not a reshoot, and the loop runs from a single frame. The numbers measure trajectory authoring capabilities from scratch but not for the reshooting task.

Works this thesis builds on or compares against:

Vista4D: video reshooting with 4D point clouds. Our video model and the source of the loop's 4D grounding.

GenDoP: auto-regressive camera trajectory generation and the DataDoP benchmark.

LAMP: language-assisted motion planning; fine-tunes an LLM to write camera-motion programs.

E.T. the Exceptional Trajectories: text-to-camera-trajectory generation with character awareness; origin of the CLaTr embedding used for scoring.

π³: permutation-equivariant visual geometry learning; the 4D reconstruction model behind scene preparation.

Vista4D, like video generation models in general, cannot check whether its output is plausible or fix artifacts such as smearing and warped geometry. This stage adds two pieces. First, a VLM video critic reviews the generated video frame by frame and returns a verdict. Second, a negative verdict gates a repair of the 4D conditioning the video was generated from, followed by one regeneration. It is a single pass, not a closed loop: it supplies the failure detector and the conditioning-level repair an iterative version would need, but runs each exactly once.

The critic samples frames from the generated video and receives four inputs: the sampled frames, one source anchor image that fixes scene and subject identity, the target trajectory in text (per sampled frame, as translation and rotation relative to the first frame), and the camera instruction. Each frame is scored on seven qualities: anchor identity consistency, temporal coherence, artifact freedom, sharpness and detail, geometric plausibility, camera trajectory adherence, and reconstruction utility, and labeled good or bad. Repair runs if and only if at least one sampled frame is bad. A good video is returned unmodified; a bad one is repaired, re-rendered and regenerated once, and the regenerated video is not reviewed again.

The repair targets the conditioning rather than the video model, on the assumption that a more complete scene memory yields a better video, and it starts from the most visible gap: the dynamic subject is hollow, because the memory holds only what the single source view saw.

The repair replaces the monocular subject with one reconstructed from multi-view priors, reusing three parts of FreeOrbit4D. SV4D 2.0 takes the masked subject video and generates four more views from fixed angles, giving five views per frame with the real one. VGGT lifts the five views into one point cloud per frame, each point tagged by its source view. FreeOrbit4D's cleanup then removes outliers and smooths the centroid trajectory with a constant-velocity forward-backward filter. The background is not borrowed; it comes from the memory itself.

Fitting the subject into the scene. VGGT's point clouds come with their own scale and focal length. Instead of registering two point clouds in 3D, each fused point is transferred through the pixel it occupies in the shared source view, since both reconstructions were built from the same frame. Only the depth needs converting: a per-frame scale λ_t, the median ratio of VGGT's source-view depth to the memory's depth over pixels both agree belong to the subject, smoothed over time, rescales each point's depth before it is placed back with the memory's own camera pose and intrinsics. Rendered along the target trajectory, points from the real source view take priority over generated ones where they overlap. The old subject points are discarded, the new ones inserted, and the memory is re-rendered for one regeneration.

The stage is evaluated on 57 trajectories authored by the loop over sequences from DAVIS, Lyra and Vista4D, weighted toward orbits and wide lateral moves. Vista4D's 384p checkpoint generates 49 frames along a frozen trajectory with a four-step distilled sampler and a fixed seed, so a baseline and its repaired counterpart differ only in the 4D conditioning. Scene preparation uses π³ and SAM 3; the repair uses SV4D 2.0 and VGGT. The critic flagged 19 of the 57 shots, dominated by orbits and large lateral moves, and every comparison below is paired within those 19.

The two families move in opposite directions. Every camera-control metric except translation error improves; every fidelity and quality metric falls, though only slightly, with motion smoothness and overall consistency essentially flat. Part of the drop is built into the measurement: CLIP-SIM, DINO-SIM and the VBench consistency scores reward agreement with the source frames, and the repair exists to put new content where the source has nothing to say. Aesthetic and imaging quality are not source-referenced and fall too; the SV4D 2.0 prior is softer and more cartoonish than the source. The fidelity numbers are also misleading on their own: in the half orbit above, the baseline earns its richer texture by failing the task, showing frontal views instead of following the trajectory.

• Single pass: one review, one repair, no re-review of the regenerated video and no stopping criterion of its own.
• Frame-level review misses defects: the critic labels frames but does not say why, so it gates the conditioning repair even when the fault lies elsewhere, and it accepts frames with defects a viewer would catch, as above.
• Subject only: static regions the source camera never saw stay empty in the memory, so the video model must hallucinate them without geometric support.
• Unvalidated reviewer, no held-out views: verdicts were never compared with human labels, and the 57 trajectories are our own with no ground-truth target views, so the numbers are not comparable with published results.