Shoot a Blockout First, Then Hand It to AI: My AI Video Workflow After 17 Iterations

Bottom line first: AI video is uncontrollable because it lacks a deterministic spatial-temporal reference. I validated a local workflow over 17 versions — render a blockout in Blender (about 10 seconds per 5-second shot) → H3 reference-to-video (6–8 minutes per 5 seconds) → photoreal final, roughly 7 minutes end to end for a single clip. The root cause of the artifacts turned out to be a single thin cylinder.

1. The Problem: Prompts Can't Control Motion or Camera

Write a prompt like "two men trade blows, camera orbits, push in on the third second", and the model gives you a different result every time: action beats drift, camera pacing is random, and it does not cut when it should. Rewrite it ten times and you get ten variants.

It is not that the prompt is not good enough — language is suited to describing "what it is", not "at which second, along which path it moves".

Film has had the answer for a long time: previz — lock down blocking, camera and pacing with grey geometry first, then shoot the real shot. This approach comes from a public article about an "AI director", and I replicated it locally: except that the "real shoot" was handed to H3.

2. The Workflow: Three Steps, Three Pieces

Step one: Blender blockout (scripted, no GUI)

Build the scene and lay out the camera path with a bpy script, then render headless. Parameters:

• Engine Workbench, color OBJECT+FLAT (flat solid color)
• 960×544 / 24fps / 120 frames (5 seconds), render time about 10 seconds
• Scene rules (to prevent artifacts): no thin cylindrical arms, light-colored sky, slow down the action

A note on installation: on this ARM machine, installing Blender was harder than using Blender — the source build stalled at 88% on dependency compilation, and I eventually got 4.0.2 running by unpacking 55 debs (about 92MB) with apt-get download + dpkg -x; BLENDER_SYSTEM_SCRIPTS must be set, or it core dumps.

Step two: Character reference images

Two photoreal character images (matching the two characters in the blockout), generated by the local Qwen image model at about 30 seconds each. These were only added in the fifth version — without reference images there is no photographic feel: a pure blockout drags its "grey model texture" into the final, and the first four versions all came out as ink-wash illustration.

Step three: H3 reference-to-video

Feed the blockout video and the two reference images into MiniMax H3 reference build (ComfyUI, turbo 4-step LoRA):

• Blockout video → extract frames → ref_videos.ref_video_0 (spatial-temporal reference)
• Character images → ref_images.ref_image_0/1 (appearance reference)
• Output 960×544 / 5 seconds / with native audio, taking 6–8 minutes

The prompt template I actually used:

Use the two men from <Picture 1>/<Picture 2> as the fighters (faces, hair, costumes match exactly).
<Video 1> is a grey clay previz: copy ONLY its motion, blocking and camera.
Clean frame: no ink, no brush strokes, no black smoke, no particle effects, no text.
Preserve character positions, motion trajectories, camera movement and shot timing from <Video 1>.

Three key sentences: the men from Picture 1/2 are the characters (locking appearance); Video 1 is the blockout, so copy only its motion, blocking and camera; the frame must be clean — no ink, no brush strokes, no smoke, no particle effects, no text.

3. Chasing the Artifact: 13 Controlled Variables, the Culprit Was a Cylinder

With reference images added the frame "looked right", but a black blob appeared: like thick ink, like black smoke, only on fast-motion frames, in a drifting position. Changing the seed and writing "no ink" did nothing.

From version 5 to version 17, 13 versions were spent chasing it. The method was dumb but effective — control the variables:

• Blockout shadows too heavy? Switch to flat solid color — artifact still there
• Dark sky being misread? Switch to a light color — still there
• Arms swinging too fast? Freeze them — still there
• Two people too close? Pull them apart — still there

After all four hypotheses died, I ran a more direct experiment: delete the arms from the blockout entirely — the artifact immediately dropped by more than half.

The root cause was clear: that thin cylindrical arm in the blockout was read by H3 as a pen or a weapon, so it "dutifully" added a black effect stroke. It was not drawing wrong — it was adding drama.

The final fix came in three pieces:

1. Remove the separate arms from the blockout — use shoulder blocks instead and let the model grow the arms itself
2. Slow the action down — from fast strikes to tai-chi-style pushing hands
3. Explicitly exclude effects in the prompt — no ink / no brush strokes / no black smoke / no particle effects

Version 17: zero artifacts, photoreal, with motion and camera matching my blockout.

4. The Time Budget and the Limits

• Blockout render: about 10 seconds per 5-second shot (Workbench engine)
• H3 reference-to-video: 6–8 minutes per 5 seconds (reference tokens participate in every sampling step, so it is slower than pure text-to-video)
• Single clip, full chain: about 7 minutes; cost = electricity (everything local, 0 API cost)

The marginal cost is roughly the electricity bill, which is why I dared run 17 versions back to back chasing the artifact; billed against a cloud plan, every version would be real money.

Limits: one case — 5 seconds, one shot, a wuxia scene. Long takes, multiple cuts and complex action are not covered yet; change the scene or the model version and the results may differ.

5. Reusable Checklist

1. Build the blockout with scripts (bpy + headless), not by hand in the GUI — iterations need to be reproducible
2. Render with Workbench + flat solid color: a 5-second shot renders in about 10 seconds
3. Keep three things out of the blockout: thin cylindrical arms, dark sky, fast action — all of them trigger repainting
4. Character reference images are mandatory, otherwise the frame will not look photoreal (the first four ink-wash versions were the lesson)
5. The three-part prompt: reference images lock appearance + "copy ONLY the motion from Video 1" + an effects exclusion list
6. When artifacts appear, debug with controlled variables, in this order: arm animation → character overlap → arm geometry → dark regions
7. Fix priority: remove arm geometry > slow motion > exclude effects in the prompt (changing the seed or lightening the sky only reduces it, never eliminates it)

In one sentence, this workflow is: give geometry control over space and time, and give language and reference images control over style.