New Video Editing and Generation Tools
Novella has been an on-and-off project centered around using AI and LLMs to assist video editing and production.
Since September 2023, I've had a few phases of experiments:
- A text based video editor (similar to Descript) for editing pre-existing video files, experimenting with AI agents to automate the process.
- A full text based editor for turning writing into videos using image/video/voice generation.
- A simple CLI tool for auto-cutting long talking videos into shorter snippets.
Each of these have been fun to explore!
Novella v1
The first editor - April 2023 to September 2023
This first version was a three step process: transcribe your videos, generate a story structure from the content, and then edit the transcripts to follow that story structure.
On the left you could semantically search your media, in the middle you could add media and edit the transcripts, and on the right you could switch between "chapters" of the full video.
As you can see my goal was to create a video of me building this software using the software itself, which didn't pan out. The hardest aspect was real time playback and quickly splicing and playing sub-clips.
System architecture diagram, all running locally:
Experimenting with Agents - September 2023 to October 2023
Then I spent lots of time working with the ReAct agent framework (this is when AI agents we're a fairly new thing) to attempt editing videos automatically within the editor. This was fraught with issues and worked ~30% of the time, but dear lord was that satisfying when it did.
This view showed a readout of the editing agents processing steps.
As per usual, between the spaghetti code and new opportunities, I paused there for some time.
Novella v2
A simplified editor - October 2024
I had some extra time so I tried to rebuild the application using what I learned (minus the agent experiments). In this version the text timeline corresponded with a horizontal timeline and I nailed real time playback.
Novella v3
A generative editor - March 2025 to May 2025
I decided to revisit the idea of AI assisted filmmaking, this time taking it from a generative angle. Around this time (with the release of Google Veo3, Higgsfield, and others) high quality video models we're becoming accessible.
I love video essays on YouTube, so I thought a tool to turn writing into videos would be great. But more generally, to me it's obvious the trend of shortening the gap between script and film.
So this was a jab at that idea. An integrated generative filmmaking editor, all text based, with the goal of merging pre-production to post-production in one tool.
Here's an early demo video:
Other opportunities came on my plate, and I'm not particularly interested in creating an AI shlop generator (despite being potentially very lucrative), so I put this particular project on pause.
Let me know if you want to see it as a real product, maybe I'll come back to this.
Novella v3 CLI
Cut long talking videos into shorter clips - May 2025
This is a simple tool to automatically edit down a video transcript using LLMs, then apply those text edits to the original video file. For me this is useful to create short summary clips from long ranty brainstorm videos.
