Recherche avancée

Sur d’autres sites (6894)

• ffmpeg : crop video into two grayscale sub-videos ; guarantee monotonical frames ; and get timestamps

  13 mars 2021, par lurix66

  The need

  


  Hello, I need to extract two regions of a .h264 video file via the crop filter into two files. The output videos need to be monochrome and extension .mp4. The encoding (or format ?) should guarantee that video frames are organized monotonically. Finally, I need to get the timestamps for both files (which I'd bet are the same timestamps that I would get from the input file, see below).

  


  In the end I will be happy to do everything in one command via an elegant one liner (via a complex filter I guess), but I start doing it in multiple steps to break it down in simpler problems.

  


  In this path I get into many difficulties and despite having searched in many places I don't seem to find solutions that work. Unfortunately I'm no expert of ffmpeg or video conversion, so the more I search, the more details I discover, the less I solve problems.

  


  Below you find some of my attempts to work with the following options :

  


  

  • -filter:v "crop=400:ih:260:0,format=gray" to do the crop and the monochrome conversion

  


  • -vf showinfo possibly combined with -vsync 0 or -copyts to get the timestamps via stderr redirection &> filename

  


  • -c:v mjpeg to force monotony of frames (are there other ways ?)

  


  


  1. cropping each region and obtaining monochrome videos

  


  $ ffmpeg -y -hide_banner -i inVideo.h264 -filter:v "crop=400:ih:260:0,format=gray" outL.mp4
$ ffmpeg -y -hide_banner -i inVideo.h264 -filter:v "crop=400:ih:1280:0,format=gray" outR.mp4


  


  The issue here is that in the output files the frames are not organized monotonically (I don't understand why ; how come would that make sense in any video format ? I can't say if that comes from the input file).

  


  EDIT. Maybe it is not frames, but packets, as returned by av .demux() method that are not monotonic (see below "instructions to reproduce...")

  


  I have got the advice to do a ffmpeg -i outL.mp4 outL.mjpeg after, but this produces two videos that look very pixellated (at least playing them with ffplay) despite being surprisingly 4x bigger than the input. Needless to say, I need both monotonic frames and lossless conversion.

  


  EDIT. I acknowledge the advice to specify -q:v 1 ; this fixes the pixellation effect but produces a file even bigger, 12x in size. Is it necessary ? (see below "instructions to reproduce...")

  


  2. getting the timestamps

  


  I found this piece of advice, but I don't want to generate hundreds of image files, so I tried the following :

  


  $ ffmpeg -y -hide_banner -i outL.mp4 -vf showinfo -vsync 0 &>tsL.txt
$ ffmpeg -y -hide_banner -i outR.mp4 -vf showinfo -vsync 0 &>tsR.txt


  


  The issue here is that I don't get any output because ffmpeg claims it needs an output file.

  


  The need to produce an output file, and the doubt that the timestamps could be lost in the previous conversions, leads me back to making a first attempt of a one liner, where I am testing also the -copyts option, and the forcing the encoding with -c:v mjpeg option as per the advice mentioned above (don't know if in the right position though)

  


  ffmpeg -y -hide_banner -i testTex2.h264 -copyts -filter:v "crop=400:ih:1280:0,format=gray" -vf showinfo -c:v mjpeg eyeL.mp4 &>tsL.txt


  


  This does not work because surprisingly the output .mp4 I get is the same as the input. If instead I put the -vf showinfo option just before the stderr redirection, I get no redirected output

  


  ffmpeg -y -hide_banner -i testTex2.h264 -copyts -filter:v "crop=400:ih:260:0,format=gray" -c:v mjpeg outR.mp4 -vf showinfo dummy.mp4 &>tsR.txt


  


  In this case I get the desired timestamps output (too much : I will need some solution to grab only the pts and pts_time data out of it) but I have to produce a big dummy file. The worst thing is anyway, that the mjpeg encoding produces a low resolution very pixellated video again

  


  I admit that the logic how to place the options and the output files on the command line is obscure to me. Possible combinations are many, and the more options I try the more complicated it gets, and I am not getting much closer to the solution.

  


  3. [EDIT] instructions how to reproduce this

  


  

  • get a .h264 video

  


  • turn it into .mp by ffmpeg command $ ffmpeg -i inVideo.h264 out.mp4

  


  • run the following python cell in a jupyter-notebook

  


  • see that the packets timestamps have diffs greater and less than zero

  


  


  %matplotlib inline
import av
import numpy as np
import matplotlib.pyplot as mpl

fname, ext="outL.direct", "mp4"

cont=av.open(f"{fname}.{ext}")
pk_pts=np.array([p.pts for p in cont.demux(video=0) if p.pts is not None])

cont=av.open(f"{fname}.{ext}")
fm_pts=np.array([f.pts for f in cont.decode(video=0) if f.pts is not None])

print(pk_pts.shape,fm_pts.shape)

mpl.subplot(211)
mpl.plot(np.diff(pk_pts))

mpl.subplot(212)
mpl.plot(np.diff(fm_pts))


  


  

  • finally create also the mjpeg encoded files in various ways, and check packets monotony with the same script (see also file size)

  


  


  $ ffmpeg -i inVideo.h264 out.mjpeg
$ ffmpeg -i inVideo.h264 -c:v mjpeg out.c_mjpeg.mp4
$ ffmpeg -i inVideo.h264 -c:v mjpeg -q:v 1 out.c_mjpeg_q1.mp4


  


  Finally, the question

  


  What is a working way / the right way to do it ?

  


  Any hints, even about single steps and how to rightly combine them will be appreciated. Also, I am not limited tio the command line, and I would be able to try some more programmatic solution in python (jupyter notebook) instead of the command line if someone points me in that direction.

  


• Visualizing Call Graphs Using Gephi

  1er septembre 2014, par Multimedia Mike — General

  When I was at university studying computer science, I took a basic chemistry course. During an accompanying lab, the teaching assistant chatted me up and asked about my major. He then said, “Computer science ? Well, that’s just typing stuff, right ?”

  My impulsive retort : “Sure, and chemistry is just about mixing together liquids and coming up with different colored liquids, as seen on the cover of my high school chemistry textbook, right ?”

  
  
  

  In fact, pure computer science has precious little to do with typing (as is joked in CS circles, computer science is about computers in the same way that astronomy is about telescopes). However, people who study computer science often pursue careers as programmers, or to put it in fancier professional language, software engineers.

  So, what’s a software engineer’s job ? Isn’t it just typing ? That’s where I’ve been going with this overly long setup. After thinking about it for long enough, I like to say that a software engineer’s trade is managing complexity.

  A few years ago, I discovered Gephi, an open source tool for graph and data visualization. It looked neat but I didn’t have much use for it at the time. Recently, however, I was trying to get a better handle on a large codebase. I.e., I was trying to manage the project’s complexity. And then I thought of Gephi again.

  Prior Work
  One way to get a grip on a large C codebase is to instrument it for profiling and extract details from the profiler. On Linux systems, this means compiling and linking the code using the -pg flag. After running the executable, there will be a gmon.out file which is post-processed using the gprof command.

  GNU software development tools have a reputation for being rather powerful and flexible, but also extremely raw. This first hit home when I was learning how to use the GNU tool for code coverage — gcov — and the way it outputs very raw data that you need to massage with other tools in order to get really useful intelligence.

  And so it is with gprof output. The output gives you a list of functions sorted by the amount of processing time spent in each. Then it gives you a flattened call tree. This is arranged as “during the profiled executions, function c was called by functions a and b and called functions d, e, and f ; function d was called by function c and called functions g and h”.

  How can this call tree data be represented in a more instructive manner that is easier to navigate ? My first impulse (and I don’t think I’m alone in this) is to convert the gprof call tree into a representation suitable for interpretation by Graphviz. Unfortunately, doing so tends to generate some enormous and unwieldy static images.

  Feeding gprof Data To Gephi
  I learned of Gephi a few years ago and recalled it when I developed an interest in gaining better perspective on a large base of alien C code. To understand what this codebase is doing for a particular use case, instrument it with gprof, gather execution data, and then study the code paths.

  How could I feed the gprof data into Gephi ? Gephi supports numerous graphing formats including an XML-based format named GEXF.

  Thus, the challenge becomes converting gprof output to GEXF.

  Which I did.

  Demonstration
  I have been absent from FFmpeg development for a long time, which is a pity because a lot of interesting development has occurred over the last 2-3 years after a troubling period of stagnation. I know that 2 big video codec developments have been HEVC (next in the line of MPEG codecs) and VP9 (heir to VP8’s throne). FFmpeg implements them both now.

  I decided I wanted to study the code flow of VP9. So I got the latest FFmpeg code from git and built it using the options "--extra-cflags=-pg --extra-ldflags=-pg". Annoyingly, I also needed to specify "--disable-asm" because gcc complains of some register allocation snafus when compiling inline ASM in profiling mode (and this is on x86_64). No matter ; ASM isn’t necessary for understanding overall code flow.

  After compiling, the binary ‘ffmpeg_g’ will have symbols and be instrumented for profiling. I grabbed a sample from this VP9 test vector set and went to work.

  ./ffmpeg_g -i vp90-2-00-quantizer-00.webm -f null /dev/null
  gprof ./ffmpeg_g > vp9decode.txt
  convert-gprof-to-gexf.py vp9decode.txt > /bigdisk/vp9decode.gexf
  

  Gephi loads vp9decode.gexf with no problem. Using Gephi, however, can be a bit challenging if one is not versed in any data exploration jargon. I recommend this Gephi getting starting guide in slide deck form. Here’s what the default graph looks like :

  
  
  

  Not very pretty or helpful. BTW, that beefy arrow running from mid-top to lower-right is the call from decode_coeffs_b -> iwht_iwht_4x4_add_c. There were 18774 from the former to the latter in this execution. Right now, the edge thicknesses correlate to number of calls between the nodes, which I’m not sure is the best representation.

  Following the tutorial slide deck, I at least learned how to enable the node labels (function symbols in this case) and apply a layout algorithm. The tutorial shows the force atlas layout. Here’s what the node neighborhood looks like for probing file type :

  
  
  

  Okay, so that’s not especially surprising– avprobe_input_format3 calls all of the *_probe functions in order to automatically determine input type. Let’s find that decode_coeffs_b function and see what its neighborhood looks like :

  
  
  

  That’s not very useful. Perhaps another algorithm might help. I select the Fruchterman–Reingold algorithm instead and get a slightly more coherent representation of the decoding node neighborhood :

  
  
  

  Further Work
  Obviously, I’m just getting started with this data exploration topic. One thing I would really appreciate in such a tool is the ability to interactively travel the graph since that’s what I’m really hoping to get out of this experiment– watching the code flows.

  Perhaps someone else can find better use cases for visualizing call graph data. Thus, I have published the source code for this tool at Github.

• Dreamcast Finds

  15 avril 2022, par Multimedia Mike — Sega Dreamcast

  Pursuant to my recent post about finally understanding how Sega Dreamcast GD-ROM rips are structured, I was able to prepare the contents of various demo discs in a manner that makes exploration easy via the Internet Archive. This is due to the way that IA makes it easy to browse archives such as ZIP or ISO files (anything that 7zip knows how to unpack), and also presents the audio tracks for native playback directly through the web browser.

  These are some of the interesting things I have found while perusing the various Dreamcast sampler discs.

  Multimedia Formats
  First and foremost : Multimedia-wise, SFD and ADX files abound on all the discs. SFD files are Sofdec, a middleware format used for a lot of FMV on Dreamcast games. These were little more than MPEG video files with a non-MPEG (ADPCM instead) audio codec. VLC will usually play the video portions of these files but has trouble detecting the audio. It’s not for lack of audio codec support because it can play the ADX files just fine.
  

  It should be noted that Dreamcast Magazine Disc 11 has an actual .mpg file (as opposed to a .sfd file) that has proper MPEG audio instead instead of ADX ADPCM.

  The only other multimedia format I know of that was used in any Dreamcast games was 4XM, used on Alone In The Dark : The New Nightmare. I wrote a simple C tool a long time to recover these files from a disc image I extracted myself. Rather than interpreting the ISO-9660 filesystem, the tool just crawled through the binary blob searching for ‘4XMV’ file signatures and using length data within the files for extraction.

  Also, there are plentiful PVR files (in reference to the PowerVR2 GPU hardware that the DC uses) which ‘file’ dutifully identifies as “Sega PVR image”. There are probably tools to view them. It doesn’t appear to be a complicated format.

  Scripting
  I was fascinated to see Lua files on at least one of the discs. It turns out that MDK 2 leverages the language, as several other games do. But it was still interesting to see the .lua files show up in the Dreamcast version as well.

  That Windows CE Logo
  Every Sega Dreamcast is famously emblazoned with a logo mentioning Microsoft Windows CE :

  
  
  

  It has confused many folks. It also confused me until this exploratory exercise. Many would wonder if the Dreamcast booted up into some Windows CE OS environment that then ran the game, but that certainly wasn’t it. Indeed, Dreamcast was one of the last consoles that really didn’t have any kind of hypervisor operating system managing everything.

  I found a file called rt2dc.exe on one sampler disc. At first, I suspected that this was a development utility for Windows to convert some “RT” graphical format into a format more suitable for the Dreamcast. Then, ‘file’ told me that it was actually a Windows EXE but compiled for the Hitachi SH-4 CPU (the brain inside the DC). Does the conversion utility run on the Dreamcast itself ? Then I analyzed the strings inside the binary and saw references to train stations. That’s when it started to click for me that this was the binary executable for the demo version of Railroad Tycoon 2 : Gold Edition, hence “rt2dc.exe”. Still, this provides some insight about whether Dreamcast “runs” Windows. This binary was built against a series of Windows CE libraries. The symbols also imply DirectX compatibility.

  Here is a page with more info about the WinCE/DirectX variant for the Sega Dreamcast. It seems that this was useful for closing the gap between PC and DC ports of games (i.e., being able to re-use more code between the 2 platforms). I guess this was part of what made Dreamcast a dry run for the DirectXbox (later Xbox).

  Here is a list of all the Dreamcast games that are known to use Windows CE.

  Suddenly, I am curious if tools such as IDA Pro or Ghidra can possibly open up Windows CE binaries that contain SH-4 code. Not that I’m particularly interested in reverse engineering any algorithms locked up in Dreamcast land.

  Tomb Raider Easter Egg
  The volume 6 sampler disc has a demo of Tomb Raider : The Last Revelation. While inspecting the strings, I found an Easter egg. I was far from the first person to discover it, though, as seen on this The Cutting Room Floor wiki page (look under “Developer Message”). It looks like I am the first person to notice it on the Dreamcast version. It shows up at offset 0xE3978 in the Dreamcast (demo version) binary, if anyone with permissions wants to update the page.

  Web Browser
  Then there’s the Web Browser for Sega Dreamcast. It seemed to be included on a lot of these sampler discs. But only mentioning the web browser undersells it– the thing also bundled an email client and an IRC client. It’s important to remember that the Dreamcast also had a keyboard peripheral.

  I need to check the timeline for when the web browser first became available vs. when the MIL-CD hack became known. My thinking is that there is no way that the web browser program didn’t have some security issues– buffer overflows and the like. It seems like this would have been a good method of breaking the security of the system.

  Ironically, I suddenly can think of a reason why one might want to use advanced reverse engineering tools on Dreamcast binaries, something I struggled with just a few paragraphs ago.

  Odds ‘n Ends
  It’s always fun to find plain text files among video game assets and speculating on the precise meaning… while also marveling how long people have been struggling to correctly spell “length”.

  Internationalization via plain text files.

  Another game (Slave Zero) saw fit to zip its assets. Maybe this was to save space in order to fit everything on the magazine sampler disc. Quizzically, this didn’t really save an appreciable amount of space.

  Finally, all the discs have an audio track 2 that advises that the disc must be played in a Dreamcast console. Not unusual. However, volume 4 also has a Japanese lady saying the same thing on track 4. This is odd because track 4 is one of the GD area audio tracks and is not accessible with normal CD hardware. Further, she identifies the disc as a “Windows CE disc”.

  The post Dreamcast Finds first appeared on Breaking Eggs And Making Omelettes.