My bodybuilding documentary, Stronger, was shot during the interim while Canon was still scratching their heads about why 24p should be available on the 5D mark II. The thing is that I still want it to have that film motion 24p look, but at 30 frames/second the footage looks like video. What's worse is that the Canon firmware I was using at the time only supported 30.0 frames/second, not even the broadcast/DVD standard of 29.97 frames/second. So...what to do?
Last spring I met Shane Hurlbut and he showed a small group of us a sizzle reel of his feature film converted from 30p to 24p via the Twixtor software. The conversion looked downright decent. My budget is obviously much, much smaller (i.e. non-existant). After a little research I found that Adobe After Effects CS5 has a feature called "Pixel Motion," which isn't exactly glaringly obvious at first, but does work to a certain degree. Pixel Motion analyzes the motion in the footage and generates in-between frames, thus doing frame rate conversion.
But first; a tangent...By default After Effects remaps from one frame rate to another by choosing the nearest source frame and duplicating it if necessary. We see this all the time with DVD's of film and 3:2 pulldown (in my case it would be 2:3 pullup?). This sort of processing can lead to choppy motion because you're either duplicating frames or throwing frames away. You won't see the difference in the still frames of the video, only when the video starts moving.
Another option is to use a process called frame blending, which overlaps two frames of video at one source frame rate (i.e. 30.0 frames/second) to create a new in-between frame timed to the desired frame rate (i.e. 23.976 frames/second). Premier Pro does this by default (After Effects can do it too) when you dump footage into the edit time line and the footage frame rate doesn't match the frame rate for the edit sequence. The frame blending algorithm basically overlays the nearest two source frames of your footage and sets the opacity of each source frame proportion to the time distance to the new frame of video. For instance if the 24p frame is 1/3rd of the way from source frame#1 and (obviously) 2/3rd's of the way to source frame#2, then the 24p frame is made up of 66.6% of source frame #1 and 33.3% of source frame #2. Does it work? ...yeah, kinda. The result of frame blending is shown in the first picture below. You can see that there are two frames overlaid and the result is a blurred in-between 24p frame. Any video with motion will blur in order to create the in between 24p frame. It doesn't look great when looking at stills, but the effect works for the most part when the video is in motion due to our persistance of vision. (Kinda looks like a 3D movie, huh?)
The second aforementioned "Pixel Motion" algorithm isn't available in Premier Pro, but is available in After Effects. Just to cut to the chase - the picture below shows the result of an After Effects render with Pixel Motion. Pixel Motion is similar to Twixtor and Apple's Motion software ("Optical Flow") algorithm. Does it work...again, yeah kinda. For the purposes of this post I'm just going to talk about how to enable "Pixel Motion. The idea here is that the software analyzes the two nearest frames of source footage and creates a morphed in-between frame that in theory represents what the picture would look like if it was shot at the edit sequence frame rate. Massive difference in quality, eh?
OK, so how does Pixel Motion work? First, create an After Effects project with the desired frame rate. In this case it's 23.976 frames/second (i.e. 24p). Then import your footage (File->Import) and drag it into the timeline. See the two red arrows in the picture below. The upper one enables frame blending when you click on the icon. The lower one enables either frame blending or Pixel Motion. When you click on the lower check box enough times such that it's a smooth slash from lower left to upper right then Pixel Motion is enabled. In fact, if you keep clicking on this box until you see a jaggie line from upper left to lower right you'll see the frame blending algorithm and you can easily compare the two algorithms. When the check box is blank that means frame duplication is in effect.
The final action is to render out your video in After Effects. I found that if I just imported the After Effects project into Premier that is not only takes a long, long, long time to render out the video but it also has a bug at edits where the wrong underlying track on the edit time line can appear for short bursts with Premier Pro CS5 (version 5.0.3 bug!). The safest thing was to just render out and take the degradation of second generation footage. I don't have enough storage to do raw TIFF sequences, so high quality H.264 did an OK job for me. I could then take the video and import it back into Premier to do my editing.
The major question is, does it look like film 24p motion? Well...again, kinda. Sometimes it looks film-ish, sometimes it looks a little video-ie. However, I'll put the emphasis on the "little" here. In my opinion it looks a lot more like 24p film motion now. One key thing that I didn't know to do at the time is to use a 1/50th shutter speed on the camera when shooting at 30p if I want to convert the footage. Motion blur is a component of the 24p film look, however most of this film was shot with 1/60th shutter speed. What I wish I knew back then...
So... is Pixel Motion perfect? It's very good, but not perfect by a long shot. Neither is Twixtor nor Optical Flow. I'm going to cover that subject with part 2 of this tutorial. I'll also cover how I got around a major problem with Pixel Motion, or at least sufficiently hid the flaw.