Client under NDA

A case study · client under NDA

I'm Arick Vigas, a software engineer. I spent four weeks with a video product where fixes did not hold. This is how I work when I walk into a system like that.

The client and the product are not named. I signed an NDA, so this page is about my method and about general facts of rendering video on servers. It leaves out their system.

Short on time?

The job

What was I hired to do?

Make a video pipeline stop breaking before launch.

  1. Pick a video
  2. Change the words
  3. Press Render
  4. Get your video

The product let a person take a ready-made video, change the words and pictures in a web browser, and get a finished video back from the cloud.

Behind the Render button sat After Effects, the program studios use to animate video. It is made for a designer sitting at a desk. Here it had to run alone on a server, for strangers, all day.

When I arrived, almost everything around that button was broken. Bugs were fixed and came back, again and again. The team was tired.

One

Where do you start when everything is broken?

Pick one thing. Follow it all the way.

The most visible bug was text. Some videos came back with empty gaps where the words should be.

So I picked one font and followed it through every step, from the moment a person chose it in the browser to the moment the video was drawn. At each step I wrote down what that step called the font and where it looked for it.

One video, three languages
EnglishHello
EspañolHola
Arabic مرحبا
Illustration of the general problem. When a font has no letter to draw, you get an empty box.
In the browser

A web font is known by its family name. A browser does not show the exact internal name.

Montserrat
In After Effects

Fonts are looked up by an exact name. The two kinds of name do not always match.

Montserrat-SemiBold
On the render machine

The font a person picked in a browser also has to be installed on the machine that draws the video.

installed, or nothing to draw with

Every step was right on its own. Between them the font got lost.

Once I had the whole path on paper, the fix was no longer a patch. I built a new font process, one path from the browser to the render machine.

The server decides the exact font. Before every render it checks that each piece of text has its letters. If a letter is missing, the font is swapped before rendering, and the person is told.

Two

How do you decide what to replace?

Read everything first. Find the hidden rocks.

The font was one bug. The render system itself was the bigger problem.

Before I proposed anything, I researched the whole area. How do other teams run After Effects on servers? Which open tools exist? Where does each one break?

I was looking for the hidden rocks, the problems you only hit after you have committed. That research led to one decision: stop repairing the home-made render runner and move to an established open-source one.

After Effects is a desktop program. On a server it wants a machine to itself and one job at a time.

Three

What if people are nervous?

Prove it small. Then switch everything.

  1. Research
  2. Small proof
  3. Full switch
  4. Extend

Management was nervous about changing the heart of the product so close to launch. I asked for a few days to build a small proof and be sure the move would work.

In a few days I showed it running, and it was more stable than what we had. Then we moved everything over.

The new runner has hooks, places where you can attach your own steps. We used those to keep our own logic, and patched a lot of our code to fit.

The result

Did it work?

Bugs started to close, and stay closed.

The whole system steadied. Bugs that used to come back were closed for good, many of them. The mood in the team got better. The product opened to its first users on the new pipeline.

Not everything went to plan. One protective layer around the renderer caused a bug of its own, and the fix was to remove it. The less code touches After Effects, the better.

The last problem

What was still unsolved?

The machines cost too much.

Renting cloud machines to run After Effects was extremely expensive. The team was paying thousands of dollars a month for them, and the public launch showed gaps in the unit economics.

Even a second environment for testing was too expensive for them at that stage.

In my research I found that Mac minis were a great replacement. If you plan to build something like this, try rendering on Apple's M-series machines and measure how they do.

For anyone doing this

What would I tell a team starting today?

  • Give After Effects a machine to itself, and one job at a time.
  • Let the server choose the font. Never the browser.
  • Keep the same fonts on every render machine.
  • Give every job one ID you can follow everywhere.
  • Try the open-source runners before you write your own.
  • Try rendering on Apple M-series machines, and measure.