By Jieyu Bai

Generative AI tools are now making it much more easier to build an image. Runway, a video generation tool, allow users use text to describe character, settings, composition, camera movement, even action pacing, and combine multiple short clips into longer video.

However, in Runway’s long video production guideline, it still start with the story structure, characters, scenes, storyboard, and shot planning, instead of merely hitting the “generate” button.

This brings a more specific question than whether AI will replace the filmmakers:

If AI could handle more and more production tasks, then what exactly is left for filmmakers to do? The answer might not be the “less work”, but rather “different work”.

A visual planning board used to organise references, characters, scenes and shot ideas during the filmmaking process. Photograph by Jieyu Bai.

In the past, a large part of the professional skills in film were focused on how to bring an idea to life. Now, as the cost of creating a single image has decreased, the work of filmmakers is more concentrated on another issue:

Why should this frame exist?

Step 1: Before generation:

Creator needs to conclude more thoughts in advance

In traditional film production process, it usually involves a clear division of professional roles.

The screen writer first come up with the characters, the conflicts, and story progresses. Then the director considers how these written elements should be translated into shots. Moreover, the cinematographer, art director, and actors continue to implement the director’s vision in their respective fields.

AI-assisted filmmaking might have changed this workflow. When a creator faces the AI generative tool, the creator can not just tell AI that a girl just received a breakup text.

The creator needs to further answer the following questions:

Hand-drawn annotations and visual design by Jieyu Bai.

Where is the girl? What is she wearing? Is it daytime or nighttime? Should the shot be a wide shot or a close-up? Does the girl look at her phone first or does she let the audience see the text message first? Should her expression change immediately or pause for a few seconds?

In the past, these questions need to be answered jointly by the screen writer, director, cinemaphotographer, arts and actors during the production process. But now, before generating, it needs to be answered by a creator in advance.

Therefore, for AI filmmakers, they are not simply dealing with a new production tool, but rather a more complect and centralized creative role.

Hand-drawn annotations and visual design by Jieyu Bai.

Step 2: During generation:

Managing the uncertainties

Even if the initial plan is already quite clear, the creators still face a specific problem:

The AI model does not consistently and automatically replicate the same world as envisioned by creators.

When a same character was generated twice, there might be slight variations in its facial features. The same clothing piece might suddenly change color. Or in one shot, the character was standing on the left side of the window, but in the next shot, the position of the window might change.

Reference image: AYu. used under permission, original source: xhslink.cn/o/AnfOsc1XURh.

On the one hand, the creators continuously add constraints to make the model closer to the preset story world. On the other hand, they also constantly review and eliminate the generated version that might deviate.

When AI could quickly provide a large number of possibilities, the creators need to deal with: which results can be accepted? which ones need to be re-generated? which ones, beautiful but violate the consistency of the characters?

Therefore, a core task for the AI filmmakers is not simply telling AI what to do.

Prompting and continuity checking together form a new kind of creative labor: uncertainty management.

Concept, layout and graphic design by Jieyu Bai. Created for this multimedia essay.

This also explains why although the faster the AI generates, it does not necessarily mean that the creators’ works are less.

Form this perspective, AI builds variation, while the filmmakers continuously condense the open possibilities into a coherent story world featuring characters, space, style, and emotions.

Step 3: After generation:

Constructing meaning

After the AI completes the generation process, even if all the shots have remained consistent in terms of characters, scenes, and visual style, the filmmaker still does not have a finished film version, but a series of independent and varying raw materials.

The meaning of a story emerges from the relationships between shots, instead of merely lying within a single shot.

Take an example of a girl receiving a breakup text message. The generative AI model could generate a series of shots, including a wide shot of a coffee shop, the text message on her phone, a close-up of the girl’s face, and the action that she slowly put down the phone.

When it comes to the construction of its visual meaning, if the text message is shown first and then the girl’s expression revealed, the audience will first understand the event and then observe the character’s reaction. However, if the girl’s expression is shown first and the content of the phone is revealed later, the same material will lead to a suspense.

Thus, after the AI generation process completed, the crucial task for the filmmakers is to determine how the audience should interpret these materials.

Concept, comparative slides and layout by Jieyu Bai. Created in Canva for this multimedia essay.

This judgment is also reflected in the editing process: which shots appear first, which information needs to be delayed, how long the reaction should last, when to add sound, and when to remain silent.

Even for the expression of the same character, it might merely be a piece of information if it lasts for just one second, while it also could turn into an emotion if it lasts for five seconds.

Therefore, although AI can generate shots, it will not automatically determine the order in which these shots should appear. It is the filmmaker’s arrangements of sequences, rhythm, and meaning, which truly turns the generated materials into a story.

The work of filmmakers has not disappeared, but redefined

From before the generation, during the generation, and after the generation, the generative AI model does not simply remove the filmmaker from the production process. Instead, it has changed the location where professional judgement occurs.

AI has mainly lowered the operational barriers, not automatically replacing the creative judgment.

One could even say that when some of the tasks that were previously distributed among the screenwriter, director, cinematographer, artist, and post-production team began to be concentrated in the hands of a single creator, the role of the filmmaker became even more comprehensive and centralized.