Follow one thing all the way
- Did
- Followed a single font from the browser to the finished video and wrote down what each step called it.
- Why
- Each step was right on its own. The fault only showed between them.
A case study · client under NDA
I'm Arick Vigas, a software engineer. I spent four weeks with a video product where fixes did not hold. This is how I work when I walk into a system like that.
The client and the product are not named. I signed an NDA, so this page is about my method and about general facts of rendering video on servers. It leaves out their system.
Short on time?
The job
Make a video pipeline stop breaking before launch.
The product let a person take a ready-made video, change the words and pictures in a web browser, and get a finished video back from the cloud.
Behind the Render button sat After Effects, the program studios use to animate video. It is made for a designer sitting at a desk. Here it had to run alone on a server, for strangers, all day.
When I arrived, almost everything around that button was broken. Bugs were fixed and came back, again and again. The team was tired.
One
Pick one thing. Follow it all the way.
The most visible bug was text. Some videos came back with empty gaps where the words should be.
So I picked one font and followed it through every step, from the moment a person chose it in the browser to the moment the video was drawn. At each step I wrote down what that step called the font and where it looked for it.
A web font is known by its family name. A browser does not show the exact internal name.
MontserratFonts are looked up by an exact name. The two kinds of name do not always match.
Montserrat-SemiBoldThe font a person picked in a browser also has to be installed on the machine that draws the video.
installed, or nothing to draw withEvery step was right on its own. Between them the font got lost.
Once I had the whole path on paper, the fix was no longer a patch. I built a new font process, one path from the browser to the render machine.
The server decides the exact font. Before every render it checks that each piece of text has its letters. If a letter is missing, the font is swapped before rendering, and the person is told.
Two
Read everything first. Find the hidden rocks.
The font was one bug. The render system itself was the bigger problem.
Before I proposed anything, I researched the whole area. How do other teams run After Effects on servers? Which open tools exist? Where does each one break?
I was looking for the hidden rocks, the problems you only hit after you have committed. That research led to one decision: stop repairing the home-made render runner and move to an established open-source one.
After Effects is a desktop program. On a server it wants a machine to itself and one job at a time.
Three
Prove it small. Then switch everything.
Management was nervous about changing the heart of the product so close to launch. I asked for a few days to build a small proof and be sure the move would work.
In a few days I showed it running, and it was more stable than what we had. Then we moved everything over.
The new runner has hooks, places where you can attach your own steps. We used those to keep our own logic, and patched a lot of our code to fit.
The result
Bugs started to close, and stay closed.
The whole system steadied. Bugs that used to come back were closed for good, many of them. The mood in the team got better. The product opened to its first users on the new pipeline.
Not everything went to plan. One protective layer around the renderer caused a bug of its own, and the fix was to remove it. The less code touches After Effects, the better.
The last problem
The machines cost too much.
Renting cloud machines to run After Effects was extremely expensive. The team was paying thousands of dollars a month for them, and the public launch showed gaps in the unit economics.
Even a second environment for testing was too expensive for them at that stage.
In my research I found that Mac minis were a great replacement. If you plan to build something like this, try rendering on Apple's M-series machines and measure how they do.
For anyone doing this
A case study · client under NDA
I joined a video product where fixes did not hold. In four weeks I rebuilt how fonts reach the video and moved rendering to an open-source runner. Bugs started to close and stay closed, and the product opened to its first users.
Text came out blank, renders got stuck, and bugs came back after they were fixed. A launch was close.
Traced one font through every step and built a new font process. Researched the options, proved a new render runner in a few days, then moved everything to it.
The system steadied. Bugs closed for good, the team's mood improved, and the product launched on the new pipeline.
Cost. Rented cloud machines for After Effects were extremely expensive, and the public launch showed gaps in the unit economics. In my research Mac minis were a great replacement. Anyone building this should try Apple M-series machines and measure.
Give every job one ID across the whole system in the first week. Without it I spent days guessing which step had failed.