Oct 10

Scene change detection with ffmpeg (avconv) to extract meaningful thumbnails

Category: Linux,multimedia   — Published by goeszen on October 10, 2013 at 10:51 pm

When you want to get a grasp of what a video is about, a visual timeline is quite helpful.
YouTube has integrated such a feature into their player - when you hover of the timeline, scrubbing along the seek fader, a small popup comes up and gives you a glimpse into a video at a certain offset. Well, that's YouTube...

In a desktop environment some NLE applications extract video timelines. And in non webapp-based video sharing the Media Player Classic has gathered some fame for offering the compilation of a simple storyboard thumbnailing.

But when it comes to long videos with long scenes, extracting a thumbnail every x seconds
doesn't make a lot of sense as many images simply contain stuff slightly different as in the
frame before. When a human observer would segment such a video, s/he would certainly segment it along editorial cuts, or scene changes.

The good news now are that we can employ automated tools to do just that. Ffmpeg (avconv) has all the stuff we need in its libavfilter library to extract still images from a video not every x
seconds but every time the video image significantly changes (what more or less equals a scene change). This magic is commonly known as scene change detection.

One approach is to let the video codec do the hard work. Most codecs employ a "key frame" identification technique, that means "this frame is a new scene" (I-frame) and subsequent images change only slightly from that "master frame" and so the codec can store the difference only (B-Frames) instead of full frames.
Now, in ffmpeg Stefano Sabatini's select filter goes through the video and triggers a save of the respective frame only when it encounters an I-Frame / key-frame (probably the beginning of a new scene)

ffmpeg -vf "select='eq(pict_type,I)'" -i somevideo.mp4 -vsync 0 -f image2 /tmp/thumbnails-%02d.jpg

This approach works, but isn't perfect. That why Clément Boesch added scene, a scene detection subfilter for select. If your build of ffmpeg has this code, you can call it like this:

ffmpeg -vf "-vf select='gt(scene\,0.9)'" -i somevideo.mp4 -vsync 0 -f image2 /tmp/thumbnails-%02d.jpg

In case your ffmpeg/avconv complains about "Undefined constant or missing '(' in 'scene'" you probably don't have the scene detection code in your compile of libavfilter/vf_select.c.

If ffmpeg/avconv complains about ""Missing ')' or too many args in ..." you probably used the examples from the docs, with an escaped comma, no surrounding quotes etc. Make sure that you include the whole "filtergraph" in quotes, like -vf "<filters>" where <filters> has our "select=..." stuff, including single quotes.

Leave a Reply