info: http://qa.digitalcollections.nypl.org/items/510d47dd-f272-a3d9-e040-e00a18064a99
FFmpeg: Extract foreground [moving] objects from video
This is a somewhat crude implementation, but given the right source material, an acceptable result can be generated. It is based on FFmpeg's 'maskedmerge' filter, which takes three input streams: a background, an overlay, and a mask (which is used to manipulate the pixels of the overlay layer).
ffmpeg \
-i background.png \
-i video.mkv \
-filter_complex \
"
color=#00ff00:size=1280x720 [matte];
[1:0] format=rgb24, split[mask][video];
[0:0][mask] blend=all_mode=difference,
curves=m='0/0 .1/0 .2/1 1/1',
format=gray,
smartblur=1,
eq=brightness=30:contrast=3,
eq=brightness=50:contrast=2,
eq=brightness=-10:contrast=50,
smartblur=3,
format=rgb24 [mask];
[matte][video][mask] maskedmerge,format=rgb24
" \
-shortest \
-pix_fmt yuv422p \
result.mkv
For this process, a still background image is needed. An extracted frame from the video will do, or if the background is constantly obscured, it may be necessary to manually create a clean image from multiple frames (stacking multiple frames may produce better results too).
The background image is 'difference' blended with the video, to produce the mask which will be used with the 'maskedmerge' filter. This video stream is then converted to grayscale and adjusted to maximise the contrast levels. [N.B. The video format changes multiple times with different filter effects, and so 'format=rgb24' is set in each filterchain for colour compatibility.]
The curves and equilisation filtering is a bit hard to explain, and due to to lack of a real time preview, somewhat "hit and miss". Basically, a 'threshold' filter is being built, where just black and white areas are created. The eq/curve filters here progressively squeeze the tones together in such a way that only the wanted areas are solid white. This will change for each project, and the shown filter chain has been progressive "hacked together" for this specific video.[N.B. 'maskedmerge' interprets tonality as levels of pixel opacity in the overlay layer]
The first 'smartblur' filter fills out (dilates) the areas to create more solid structures in the mask. The second 'smartblur' filter blends the edges of the mask to create a softer cutout. Additional 'smartblur' filters can be used on the background and on the video stream it is blended with, which will act as a noise filter to cull stray momentary differences.
The final element is a new background for the extracted elements to sit upon. In this example, a simple green matte is generated. This, along with the created mask, and original video, are are provided as input for the 'maskedmerge' filter.
There are many ways this can be implemented, adjusted, and improved. In the example above, everything is done within one filtergraph, but it can be separated out into multiple passes (this would be useful for manually fixing errors in the mask). [N.B. Timing can be an issue when running this all in a single filtergraph (where the mask layer didn't match up with the overlay). 29.97fps videos proved particularly troublesome. Repeated use of 'setpts=PTS' in filter graph might help, but it this case, it was fixed by converting the video to 25fps beforehand.]
UPDATE: 2020-05-05
There is some recurring confusion over what I wrote about stacking multiple frames for the background image. It's really not that important; it's just something to help create a more general/average background image by image stacking.
The image-stacking process is just to create a cleaner background image to work with. The idea is to remove momentary anomalies by averaging frames together. The benefits may be negligible though. Image-stacking can be done many ways. I created a quick demo for you using FFmpeg:
— oioiiooixiii (@oioiiooixiii) November 4, 2019
# Image stacking with FFmpeg usinf 'tmix' filter. # More info on 'tmix' filter: https://ffmpeg.org/ffmpeg-filters.html#tmix ffmpeg -i background-frame%d.png -vf tmix=frames=3 stacked.png # Image stacking is also possible with ImageMagick convert *.png -evaluate-sequence mean stacked.png
ffmpeg maskedmerge: https://ffmpeg.org/ffmpeg-filters.html#maskedmerge
source video: ぷに (Puni) https://www.youtube.com/watch?v=B0o8cQa-Kd8
Discussion of technique on twitter: https://twitter.com/alihaydarglc/status/982950986175209472
FFmpeg: Create a video composite of colourised macroblock motion-vectors
# Generate video motion vectors, in various colours, and merge together
# NB: Includes fixed 'curve' filters for issue outlined in blog post
ffplay \
-flags2 +export_mvs \
-i video.mkv \
-vf \
"
split=3 [original][original1][vectors];
[vectors] codecview=mv=pf+bf+bb [vectors];
[vectors][original] blend=all_mode=difference128,
eq=contrast=7:brightness=-0.3,
split=3 [yellow][pink][black];
[yellow] curves=r='0/0 0.1/0.5 1/1':
g='0/0 0.1/0.5 1/1':
b='0/0 0.4/0.5 1/1' [yellow];
[pink] curves=r='0/0 0.1/0.5 1/1':
g='0/0 0.1/0.3 1/1':
b='0/0 0.1/0.3 1/1' [pink];
[original1][yellow] blend=all_expr=if(gt(X\,Y*(W/H))\,A\,B) [yellorig];
[pink][black] blend=all_expr=if(gt(X\,Y*(W/H))\,A\,B) [pinkblack];
[pinkblack][yellorig]blend=all_expr=if(gt(X\,W-Y*(W/H))\,A\,B)
"
# Process:
# 1: Three copies of input video are made
# 2: Motion vectors are applied to one stream
# 3: The result of #2 is 'difference128' blended with an original video stream
# The brightness and contrast are adjusted to improve clarity
# Three copies of this vectors result are made
# 4: Curves are applied to one vectors stream to create yellow colour
# 5: Curves are applied to another vectors stream to create pink colour
# 6: Original video stream and yellow vectors are combined diagonally
# 7: Pink vectors stream and original vectors stream are combined diagonally
# 8: The results of #6 and #7 are combined diagonally (opposite direction)
NB: At time of writing, the latest version of FFmpeg (N-81396-g0d8b6a1) has a bug (feature?) where upper and lower bounds of 'curves' filter must be set for accurate results. This is contrary to what's written in official documentation.
alternate version:
see related: http://oioiiooixiii.blogspot.com/2016/04/ffmpeg-display-and-isolate-macroblock.html
source video: 足太ぺんた (Asibuto Penta) https://www.youtube.com/watch?v=Djdm7NaQheU
FFmpeg: Video Stabilisation using 'libvidstab'
It is possible to stablise video with standard FFmpeg using the 'deshake' filter, which can produce satisfactory results¹. Another option is to use FFmpeg with the 'vid.stab' library.
In the video above, a side-by-side comparison is made between the original video, and the 'vid.stab' stablised version. The subject matter remains still, while the video content floats around the frame. This is achieved by setting the 'zoom' to a negative value, 'optzoom' to 0, and setting 'relative' to 1. This is not typically desired in most instances, as it creates unusual framing. However, it does mean that no picture information is lost in the process. Note also, how missing information is replaced by the content of previous frames². The other option is to leave these areas black. Further settings information found at Georg Martius's website.
I've created a Bash script to aid in setting the values for video stabilisation (link at end of post). It was intended as a way of getting to grips with different settings, rather than a final application. It initialises a crude interface using Zenity, however all options can be set with this quickly, and it will build complete filters for the first and second pass. It also creates a video using FFmpeg's default values for MKV files. It produces a rudimentary log file, as follows:
**** Wed Jul 6 17:02:07 IST 2016 **** ARRAY VALUES: |10||||||||200|||0|||1|-50|0|||| vidstabdetect=result=transforms.trf:shakiness=10:accuracy=15:stepsize=6:mincontrast=0.3:tripod=0:show=0 vidstabtransform=input=transforms.trf:smoothing=200:optalgo=gauss:maxshift=-1:maxangle=0:crop=keep:invert=0:relative=1:zoom=-50:optzoom=0:zoomspeed=0.25:interpol=bilinear:tripod=0:debug=0 # Info: # 1: Time and date of specific filtering. The filter choices of each run on a video gets added to the same log file. # 2: Clearly shows the user specified values for filtering (blanks between '|' symbols indicate default value used) # 3: Filtergraph used for first pass # 4: Filtergraph used for second pass
To use 'vib.stab' features in FFmpeg, FFmpeg must be compiled using the following procedures (correct as of this post's date)
# Using the FFmpeg compilation method for GNU/Linux, found here # https://trac.ffmpeg.org/wiki/CompilationGuide/Ubuntu # Add (or complete) the following to pre-compile procedures # --------------------------------------------------------- cd ~/ffmpeg_sources wget -O vid-stab-master.tar.gz https://github.com/georgmartius/vid.stab/tarball/master tar xzvf vid-stab-master.tar.gz cd *vid.stab* cmake . make sudo make install # --------------------------------------------------------- # When compiling FFmpeg, include '--enable-libvidstab' in './configure PATH' # Create necessary symlinks to 'libvidstab.so' automatically by running sudo ldconfig
On a final note, vid.stab refuses to work with videos of certain pixel formats, so I encoded all test video as 'yuv420p' which worked without a problem.
¹ Deshake has an advantage over vid.stab, in that it allows setting a region for motion search.
² In some other systems (like 'Deshaker' for VirtualDub ) missing frame information can be interpolated from bi-directional frame analysis.
bash script: http://oioiiooixiii.blogspot.com/p/context-download-binbashset-e-x-script.html
vid.stab home: http://public.hronopik.de/vid.stab/features.php
vid.stab github: https://github.com/georgmartius/vid.stab
initial reading: https://www.epifocal.net/blog/video-stabilization-with-ffmpeg
source video: えんそく (Ensoku) https://www.youtube.com/watch?v=jWdQMgBlXEo
North Korea is a serious threat to America...
"Can you see the imperialists, dear Leader?" - "No, not a sign of them. Our tireless rhetoric has frightened them away!"
America could flatten North Korea in a heartbeat; could flatten it before North Korea got the kindling smoldering in one of it's glorified fireworks; but it hasn't. The debate goes back and forth: America won't attack a country with nuclear weapons - America won't remove a country that provides it a perfect excuse to keep a massive military presence in the area.
context: http://fair.org/uncategorized/north-korea-rattles-sabres-meanwhile-u-s-pretends-to-drop-nuclear-bombs-on-them/
Subscribe to:
Posts
(
Atom
)