Sur d’autres sites (3960)

• Revision 6ce718eb18 : Merge "End of orientation zero group experiment" into experimental

  22 avril 2013, par Deb Mukherjee

  Changed Paths : Modify /vp9/decoder/vp9_decodframe.c Modify /vp9/encoder/vp9_encodemb.c Modify /vp9/encoder/vp9_rdopt.c Merge "End of orientation zero group experiment" into experimental

• Writing frames from camera using skvideo.io.FFmpegWriter

  28 juin 2018, par Liam Deacon

  I’m trying to finely control the video encoding of camera image frames captured on the fly using skvideo.io.FFmpegWriter and cv2.VideoCapture, e.g.

  from skvideo import io
  
  import cv2
  
  
  
  fps = 60
  
  stream = cv2.VideoCapture(0)                    # 0 is for /dev/video0
  
  print("fps: {}".format(stream.set(cv2.CAP_PROP_FPS, fps)))
  
  
  
  stream.set(cv2.CAP_PROP_FRAME_WIDTH, 1920)
  
  stream.set(cv2.CAP_PROP_FRAME_HEIGHT, 1080)
  
  print("bit_depth: {}".format(stream.set(cv2.CAP_PROP_FORMAT, cv2.CV_8U)))
  
  
  
  video = io.FFmpegWriter('/tmp/test_ffmpeg.avi', 
  
              inputdict={'-r': fps, '-width': 1920, '-height': 1080},
  
              outputdict={'-r': fps, '-vcodec': 'libx264', '-pix_fmt': 'h264'}
  
  )
  
  
  
  try:
  
      for i in range(fps*10):  # 10s of video
  
          ret, frame = stream.read()
  
          video.writeFrame(frame)
  
  finally:
  
      stream.release()
  
  
  
  try:
  
      video.close()
  
  except:
  
      pass

  However, I get the following exception (in Jupyter notebook) :

  ---------------------------------------------------------------------------
  
  TypeError                                 Traceback (most recent call last)
  
  
  
   in <module>()
  
       18     while range(fps*10):
  
       19         ret, frame = stream.read()
  
  ---> 20         video.writeFrame(frame)
  
       21 except BaseException as err:
  
       22     raise err
  
  
  
  /usr/local/lib/python3.6/site-packages/skvideo/io/ffmpeg.py in writeFrame(self, im)
  
      446         T, M, N, C = vid.shape
  
      447         if not self.warmStarted:
  
  --> 448             self._warmStart(M, N, C)
  
      449 
  
      450         # Ensure that ndarray image is in uint8
  
  
  
  /usr/local/lib/python3.6/site-packages/skvideo/io/ffmpeg.py in _warmStart(self, M, N, C)
  
      412         cmd = [_FFMPEG_PATH + "/" + _FFMPEG_APPLICATION, "-y"] + iargs + ["-i", "-"] + oargs + [self._filename]
  
      413 
  
  --> 414         self._cmd = " ".join(cmd)
  
      415 
  
      416         # Launch process
  
  
  
  TypeError: sequence item 3: expected str instance, int found
  
  </module>

  Changing this to video.writeFrame(frame.tostring()) results in ValueError: Improper data input, leaving me stumped.

  How should I go about writing each frame (as returned by OpenCV) to my FFmpegWriter instance ?

  EDIT

  The code works fine if I remove inputdict and outputdict from the io.FFmpegWriter call, however this defeats the purpose for me as I need fine control over the video encoding (I am experimenting with lossless/near-lossless compression of the raw video captured from the camera and trying to establish the best compromise in terms of compression vs fidelity for my needs).

• How to force A/V sync using mkvmerge and external time-codes ?

  19 avril 2017, par b..

  Background

  I’m working on a project where video and audio are algorithmic interpretations of an MKV source file where I use ffmpeg -ss and -t to extract a particular region of audio and video to separate files. I use scene changes in the video in the audio process (i.e. the audio changes on video scene change), so sync is crucial.

  Audio is 48khz, using 512 sample blocks.
  Video is 23.976fps (I also tried 24).

  I store the frame onset of sceneChanges in a file in terms of cumulative blocks :

  blocksPerFrame = (48000 / 512) / 23.976
  
  sceneOnsetBlock = sceneOnsetFrame*blocksPerFrame

  I use these blocks in my audio code to treat the samples associated with each scene as a group.

  When I combine the audio and video back together (currently using ffmpeg to generate mp4(v) mp3(a) in an MKV container), the audio and video start off in sync but increasingly drifts until it ends up being 35 seconds off. The worst part is that the audio lag is nonlinear ! By non-linear, I mean that if I plot the lag against the location of that lag in time, I don’t get a line, but what you see in the image below). I can’t just shift or scale the audio to fit the video because of this nonlinearity. I cannot figure out the cause of this nonlinearly increasing audio delay ; I’ve double and triple checked my math.

  

  Since I know the exact timing of scene changes, I should be able to generate "external timecodes" (from the blocks above) for mkvmerge to perfectly sync the output !

  Subquestions :

  1. Is this the best approach (beyond trying to figure out what went wrong in the first place) ? As I’m using my video frames as a
     reference, if I use the scene changes as timecodes for the audio,
     will it force the video to match the audio or vice versa ? I’m much less concerned with the duration than the sync. The video was much more laborious to produce, so I’d rather loose some sound than some frames.

  2. I’m not clear on what numbers to use in the timecodes file.
     According to mkvmerge documentation "For video this is exactly
     one frame, for audio this is one packet of the specific audio type."
     Since I’m using MP3, what is the packet size ? Ideally, I could specify a packetsize (in the audio-encoder ?) that matches my block size (512) to keep things consistent and simple. Can I do this with ffmpeg ?

  Thank you !