Recherche avancée

Sur d’autres sites (5257)

• ffmpeg streaming via rtp reorders streams

  19 juin 2023, par konovification

  I'm streaming a video using ffmpeg -i bbb.mp4 -c:v copy -c:a copy -f rtp_mpegts "rtp://239.1.1.1:8000". The command line output is :

  


  ffmpeg version 4.3.6-0+deb11u1 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 10 (Debian 10.2.1-6)
  configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  WARNING: library configuration mismatch
  avcodec     configuration: --prefix=/usr --extra-version=0+deb11u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared --enable-version3 --disable-doc --disable-programs --enable-libaribb24 --enable-liblensfun --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libtesseract --enable-libvo_amrwbenc
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Guessed Channel Layout for Input Stream #0.1 : 5.1
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    composer        : Sacha Goedegebure
    encoder         : Lavf58.45.100
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
  Duration: 00:10:34.64, start: 0.000000, bitrate: 4195 kb/s
    Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 4094 kb/s, 25 fps, 25 tbr, 12800 tbn, 50 tbc (default)
    Metadata:
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 96 kb/s (default)
    Metadata:
      handler_name    : GPAC ISO Audio Handler
Output #0, rtp_mpegts, to 'rtp://239.1.1.1:8000':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    composer        : Sacha Goedegebure
    genre           : Animation
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    encoder         : Lavf58.45.100
    Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 4094 kb/s, 25 fps, 25 tbr, 90k tbn, 25 tbc (default)
    Metadata:
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, 5.1, fltp, 96 kb/s (default)
    Metadata:
      handler_name    : GPAC ISO Audio Handler
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help


  


  When I run ffprobe rtp://239.1.1.1:8000 (without restarting the stream), around one in ten times I get

  


  Input #0, rtp, from 'rtp://239.1.1.1:8000':
  Duration: N/A, start: 411.533978, bitrate: N/A
  Program 1 
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:1: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:0: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 384 kb/s


  


  Meaning that the video and audio streams swapped indices. I'm writing a program which uses libav where I rely on this to not happen. What is the reason for this and can it be fixed ?

  


• Python cv2 script that scans a giant image to a video. Why do I need pad two extra lines

  27 avril 2022, par Mahrarena

  I wrote a script that scans a giant image to make a video. Normally I just post my scripts straight to my Code Review account, but this script is ugly, needs to be refactored, implements only horizontal scrolling and most importantly I just fixed a bug but I don't completely understand why it works.

  


  Example :

  


  Original image (Google Drive)

  


  Video Output (Google Drive)

  


  As you can see from the video, everything is working properly except the fact that I don't know how it works.

  


  Full working code

  


  ---

  


  import cv2
import numpy as np
import random
import rpack
from fractions import Fraction
from math import prod

def resize_guide(image_size, target_area):
    aspect_ratio = Fraction(*image_size).limit_denominator()
    horizontal = aspect_ratio.numerator
    vertical = aspect_ratio.denominator
    unit_length = (target_area/(horizontal*vertical))**.5
    return (int(horizontal*unit_length), int(vertical*unit_length))

fourcc = cv2.VideoWriter_fourcc(*'mp4v')
FRAME = np.zeros((1080, 1920, 3), dtype=np.uint8)

def new_frame():
    return np.ndarray.copy(FRAME)

def center(image):
    frame = new_frame()
    h, w = image.shape[:2]
    yoff = round((1080-h)/2)
    xoff = round((1920-w)/2)
    frame[yoff:yoff+h, xoff:xoff+w] = image
    return frame

def image_scanning(file, fps=60, pan_increment=64, horizontal_increment=8):
    image = cv2.imread(file)
    height, width = image.shape[:2]
    assert width*height >= 1920*1080
    video_writer = cv2.VideoWriter(file+'.mp4', fourcc, fps, (1920, 1080))
    fit_height = True
    if height < 1080:
        width = width*1080/height
        image = cv2.resize(image, (width, 1080), interpolation = cv2.INTER_AREA)
    aspect_ratio = width / height
    zooming_needed = False
    if 4/9 <= aspect_ratio <= 16/9:
        new_width = round(width*1080/height)
        fit = cv2.resize(image, (new_width, 1080), interpolation = cv2.INTER_AREA)
        zooming_needed = True
    
    elif 16/9 < aspect_ratio <= 32/9:
        new_height = round(height*1920/width)
        fit = cv2.resize(image, (1920, new_height), interpolation = cv2.INTER_AREA)
        fit_height = False
        zooming_needed = True
    
    centered = center(fit)
    for i in range(fps):
        video_writer.write(centered)
    if fit_height:
        xoff = round((1920 - new_width)/2)
        while xoff:
            if xoff - pan_increment >= 0:
                xoff -= pan_increment
            else:
                xoff = 0
            frame = new_frame()
            frame[0:1080, xoff:xoff+new_width] = fit
            video_writer.write(frame)
    else:
        yoff = round((1080 - new_height)/2)
        while yoff:
            if yoff - pan_increment >= 0:
                yoff -= pan_increment
            else:
                yoff = 0
            frame = new_frame()
            frame[yoff:yoff+new_height, 0:1920] = fit
            video_writer.write(frame)
    
    if zooming_needed:
        if fit_height:
            width_1, height_1 = new_width, 1080
        else:
            width_1, height_1 = 1920, new_height
        new_area = width_1 * height_1
        original_area = width * height
        area_diff = original_area - new_area
        unit_diff = area_diff / fps
        for i in range(1, fps+1):
            zoomed = cv2.resize(image, resize_guide((width_1, height_1), new_area+unit_diff*i), interpolation=cv2.INTER_AREA)
            zheight, zwidth = zoomed.shape[:2]
            zheight = min(zheight, 1080)
            zwidth = min(zwidth, 1920)
            frame = new_frame()
            frame[0:zheight, 0:zwidth] = zoomed[0:zheight, 0:zwidth]
            video_writer.write(frame)
    
    if (width - 1920) % horizontal_increment:
        new_width = ((width - 1920) // horizontal_increment + 1) * horizontal_increment + 1920
        frame = np.zeros([height, new_width, 3], dtype=np.uint8)
        frame[0:height, 0:width] = image
        width = new_width
        image = frame
    
    if height % 1080:
        new_height = (height // 1080 + 2) * 1080
        frame = np.zeros([new_height, width, 3], dtype=np.uint8)
        frame[0:height, 0:width] = image
        height = new_height - 1080
        image = frame
    
    y, x = 0, 0
    for y in range(0, height, 1080):
        for x in range(0, width-1920, horizontal_increment):
            frame = image[y:y+1080, x:x+1920]
            video_writer.write(frame)
        x = width - 1920
        frame = image[y:y+1080, x:x+1920]
        for i in range(round(fps/3)):
            video_writer.write(frame)
    cv2.destroyAllWindows()
    video_writer.release()
    del video_writer


  


  I don't know why I need to pad two extra lines instead of one, meaning if I change this :

  


      if height % 1080:
        new_height = (height // 1080 + 2) * 1080
        frame = np.zeros([new_height, width, 3], dtype=np.uint8)
        frame[0:height, 0:width] = image
        height = new_height - 1080
        image = frame


  


  To this :

  


      if height % 1080:
        new_height = (height // 1080 + 1) * 1080
        frame = np.zeros([new_height, width, 3], dtype=np.uint8)
        frame[0:height, 0:width] = image
        height = new_height
        image = frame


  


  The program raises exceptions :

  


  OpenCV: FFMPEG: tag 0x34363268/&#x27;h264&#x27; is not supported with codec id 27 and format &#x27;mp4 / MP4 (MPEG-4 Part 14)&#x27;&#xA;OpenCV: FFMPEG: fallback to use tag 0x31637661/&#x27;avc1&#x27;&#xA;---------------------------------------------------------------------------&#xA;error                                     Traceback (most recent call last)&#xA; in <module>&#xA;----> 1 image_scanning("D:/collages/91f53ebcea2a.png")&#xA;&#xA; in image_scanning(file, fps, pan_increment, horizontal_increment, fast_decrement)&#xA;    122                     x &#x2B;= horizontal_increment&#xA;    123                     frame = image[y:y&#x2B;1080, x:x&#x2B;1920]&#xA;--> 124                     video_writer.write(frame)&#xA;    125     cv2.destroyAllWindows()&#xA;    126     video_writer.release()&#xA;&#xA;error: Unknown C&#x2B;&#x2B; exception from OpenCV code&#xA;</module>

  


  I guess it was caused by indexing error because the last line would not have enough pixels so padding the height of the image to a multiple of 1080 should work.

  


  But that's not the case, I need to pad two lines, why is that ? I really don't understand why it is working.

  


  ---

  


  No, I really wrote all of it, I understand all the principles, the ideas are all mine, but there is one small problem in implementation. I don't know why I need extra pixels in the bottom to make it work, because if I don't pad the height to a multiple of 1080, I can't get the bottom line, the lowest potion of height % 1080 would be lost.

  


  If I tried to get the lowest part, the program will raise exceptions even if I pad the height to a multiple of 1080, I think it is related to indexing but I don't fully understand it, turns out I need to pad the height and add extra pixels, even 1 pixel would work.

  


  I don't know why it raises exceptions and how add extra pixels got rid of the exception, but I understand everything else perfectly clear, after all I wrote it.

  


  There's a bug in my program, I don't know what caused it, and I want you to help me debugging, and that's the entire point of the question !

  


• Decoding MediaRecorder produced webm stream

  15 août 2019, par sgmg

  I am trying to decode a video stream from the browser using the ffmpeg API. The stream is produced by the webcam and recorded with MediaRecorder as webm format. What I ultimately need is a vector of opencv cv::Mat objects for further processing.

  I have written a C++ webserver using the uWebsocket library. The video stream is sent via websocket from the browser to the server once per second. On the server, I append the received data to my custom buffer and decode it with the ffmpeg API.

  If I just save the data on the disk and later I play it with a media player, it works fine. So, whatever the browser sends is a valid video.

  I do not think that I correctly understand how should the custom IO behave with network streaming as nothing seems to be working.

  The custom buffer :

   struct Buffer
  
      {
  
          std::vector data;
  
          int currentPos = 0;
  
      };

  The readAVBuffer method for custom IO

  int MediaDecoder::readAVBuffer(void* opaque, uint8_t* buf, int buf_size)
  
  {
  
      MediaDecoder::Buffer* mbuf = (MediaDecoder::Buffer*)opaque;
  
      int count = 0;
  
      for(int i=0;icurrentPos;
  
          if(index >= (int)mbuf->data.size())
  
          {
  
              break;
  
          }
  
          count++;
  
          buf[i] = mbuf->data.at(index);
  
      }
  
      if(count > 0) mbuf->currentPos+=count;
  
  
  
      std::cout << "read : "<currentPos<<", buff size:"<data.size() << std::endl;
  
      if(count <= 0) return AVERROR(EAGAIN); //is this error that should be returned? It cannot be EOF since we're not done yet, most likely
  
      return count;
  
  }

  The big decode method, that’s supposed to return whatever frames it could read

  std::vector MediaDecoder::decode(const char* data, size_t length)
  
  {
  
      std::vector frames;
  
      //add data to the buffer
  
      for(size_t i=0;i/do not invoke the decoders until we have 1MB of data
  
      if(((buf.data.size() - buf.currentPos) < 1*1024*1024) && !initializedCodecs) return frames;
  
  
  
      std::cout << "decoding data length "</initialize ffmpeg objects. Custom I/O, format, decoder, etc.
  
      {       
  
          //these are just members of the class 
  
          avioCtxPtr = std::unique_ptr(
  
                      avio_alloc_context((uint8_t*)av_malloc(4096),4096,0,&buf,&readAVBuffer,nullptr,nullptr),
  
                      avio_context_deleter());
  
          if(!avioCtxPtr)
  
          {
  
              std::cerr << "Could not create IO buffer" << std::endl;
  
              return frames;
  
          }                
  
  
  
          fmt_ctx = std::unique_ptr(avformat_alloc_context(),
  
                                                                            avformat_context_deleter());
  
          fmt_ctx->pb = avioCtxPtr.get();
  
          fmt_ctx->flags |= AVFMT_FLAG_CUSTOM_IO ;
  
          //fmt_ctx->max_analyze_duration = 2 * AV_TIME_BASE; // read 2 seconds of data
  
          {
  
              AVFormatContext *fmtCtxRaw = fmt_ctx.get();            
  
              if (avformat_open_input(&fmtCtxRaw, "", nullptr, nullptr) < 0) {
  
                  std::cerr << "Could not open movie" << std::endl;
  
                  return frames;
  
              }
  
          }
  
          if (avformat_find_stream_info(fmt_ctx.get(), nullptr) < 0) {
  
              std::cerr << "Could not find stream information" << std::endl;
  
              return frames;
  
          }
  
          if((video_stream_idx = av_find_best_stream(fmt_ctx.get(), AVMEDIA_TYPE_VIDEO, -1, -1, nullptr, 0)) < 0)
  
          {
  
              std::cerr << "Could not find video stream" << std::endl;
  
              return frames;
  
          }
  
          AVStream *video_stream = fmt_ctx->streams[video_stream_idx];
  
          AVCodec *dec = avcodec_find_decoder(video_stream->codecpar->codec_id);
  
  
  
          video_dec_ctx = std::unique_ptr (avcodec_alloc_context3(dec),
  
                                                                                avcodec_context_deleter());
  
          if (!video_dec_ctx)
  
          {
  
              std::cerr << "Failed to allocate the video codec context" << std::endl;
  
              return frames;
  
          }
  
          avcodec_parameters_to_context(video_dec_ctx.get(),video_stream->codecpar);
  
          video_dec_ctx->thread_count = 1;
  
         /* video_dec_ctx->max_b_frames = 0;
  
          video_dec_ctx->frame_skip_threshold = 10;*/
  
  
  
          AVDictionary *opts = nullptr;
  
          av_dict_set(&opts, "refcounted_frames", "1", 0);
  
          av_dict_set(&opts, "deadline", "1", 0);
  
          av_dict_set(&opts, "auto-alt-ref", "0", 0);
  
          av_dict_set(&opts, "lag-in-frames", "1", 0);
  
          av_dict_set(&opts, "rc_lookahead", "1", 0);
  
          av_dict_set(&opts, "drop_frame", "1", 0);
  
          av_dict_set(&opts, "error-resilient", "1", 0);
  
  
  
          int width = video_dec_ctx->width;
  
          videoHeight = video_dec_ctx->height;
  
  
  
          if(avcodec_open2(video_dec_ctx.get(), dec, &opts) < 0)
  
          {
  
              std::cerr << "Failed to open the video codec context" << std::endl;
  
              return frames;
  
          }
  
  
  
          AVPixelFormat  pFormat = AV_PIX_FMT_BGR24;
  
          img_convert_ctx = std::unique_ptr(sws_getContext(width, videoHeight,
  
                                           video_dec_ctx->pix_fmt,   width, videoHeight, pFormat,
  
                                           SWS_BICUBIC, nullptr, nullptr,nullptr),swscontext_deleter());
  
  
  
          frame = std::unique_ptr(av_frame_alloc(),avframe_deleter());
  
          frameRGB = std::unique_ptr(av_frame_alloc(),avframe_deleter());
  
  
  
  
  
          int numBytes = av_image_get_buffer_size(pFormat, width, videoHeight,32 /*https://stackoverflow.com/questions/35678041/what-is-linesize-alignment-meaning*/);
  
          std::unique_ptr imageBuffer((uint8_t *) av_malloc(numBytes*sizeof(uint8_t)),avbuffer_deleter());
  
          av_image_fill_arrays(frameRGB->data,frameRGB->linesize,imageBuffer.get(),pFormat,width,videoHeight,32);
  
          frameRGB->width = width;
  
          frameRGB->height = videoHeight;
  
  
  
          initializedCodecs = true;
  
      }    
  
      AVPacket pkt;
  
      av_init_packet(&pkt);
  
      pkt.data = nullptr;
  
      pkt.size = 0;
  
  
  
      int read_frame_return = 0;
  
      while ( (read_frame_return=av_read_frame(fmt_ctx.get(), &pkt)) >= 0)
  
      {
  
          readFrame(&frames,&pkt,video_dec_ctx.get(),frame.get(),img_convert_ctx.get(),
  
                    videoHeight,frameRGB.get());
  
          //if(cancelled) break;
  
      }
  
      avioCtxPtr->eof_reached = 0;
  
      avioCtxPtr->error = 0;
  
  
  
  
  
      //flush
  
     // readFrame(frames.get(),nullptr,video_dec_ctx.get(),frame.get(),
  
       //         img_convert_ctx.get(),videoHeight,frameRGB.get());
  
  
  
      avioCtxPtr->eof_reached = 0;
  
      avioCtxPtr->error = 0;
  
  
  
      if(frames->size() <= 0)
  
      {
  
          std::cout << "buffer pos: "<code>

  What I would expect to happen would be for a continuous extraction of cv::Mat frames as I feed it more and more data. What actually happens is that after the the buffer is fully read I see :

  [matroska,webm @ 0x507b450] Read error at pos. 1278266 (0x13813a)
  
  [matroska,webm @ 0x507b450] Seek to desired resync point failed. Seeking to earliest point available instead.

  And then no more bytes are read from the buffer even if later I increase the size of it.

  There is something terribly wrong I’m doing here and I don’t understand what.