Sur d’autres sites (5165)

• Why does use of H264 in sender/receiver pipelines introduce just HUGE delay ?

  24 janvier 2012, par Serguey Zefirov

  When I try to create pipeline that uses H264 to transmit video, I get some enormous delay, up to 10 seconds to transmit video from my machine to... my machine ! This is unacceptable for my goals and I'd like to consult StackOverflow over what I (or someone else) do wrong.

  I took pipelines from gstrtpbin documentation page and slightly modified them to use Speex :

  This is sender pipeline :
  # !/bin/sh

  gst-launch -v gstrtpbin name=rtpbin \
  
          v4l2src ! ffmpegcolorspace ! ffenc_h263 ! rtph263ppay ! rtpbin.send_rtp_sink_0 \
  
                    rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=5000                            \
  
                    rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=5001 sync=false async=false    \
  
                    udpsrc port=5005 ! rtpbin.recv_rtcp_sink_0                           \
  
          pulsesrc ! audioconvert ! audioresample  ! audio/x-raw-int,rate=16000 !    \
  
                    speexenc bitrate=16000 ! rtpspeexpay ! rtpbin.send_rtp_sink_1                   \
  
                    rtpbin.send_rtp_src_1 ! udpsink host=127.0.0.1 port=5002                            \
  
                    rtpbin.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=5003 sync=false async=false    \
  
                    udpsrc port=5007 ! rtpbin.recv_rtcp_sink_1

  Receiver pipeline :

   !/bin/sh

  gst-launch -v\
  
      gstrtpbin name=rtpbin                                          \
  
      udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)H263-1998" \
  
              port=5000 ! rtpbin.recv_rtp_sink_0                                \
  
          rtpbin. ! rtph263pdepay ! ffdec_h263 ! xvimagesink                    \
  
       udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0                               \
  
       rtpbin.send_rtcp_src_0 ! udpsink port=5005 sync=false async=false        \
  
      udpsrc caps="application/x-rtp,media=(string)audio, clock-rate=(int)16000, encoding-name=(string)SPEEX, encoding-params=(string)1, payload=(int)110" \
  
              port=5002 ! rtpbin.recv_rtp_sink_1                                \
  
          rtpbin. ! rtpspeexdepay ! speexdec ! audioresample ! audioconvert ! alsasink \
  
       udpsrc port=5003 ! rtpbin.recv_rtcp_sink_1                               \
  
       rtpbin.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=5007 sync=false async=false

  Those pipelines, a combination of H263 and Speex, work fine enough. I snap my fingers near camera and micropohne and then I see movement and hear sound at the same time.

  Then I changed pipelines to use H264 along the video path.

  The sender becomes :
  # !/bin/sh

  gst-launch -v gstrtpbin name=rtpbin \
  
          v4l2src ! ffmpegcolorspace ! x264enc bitrate=300 ! rtph264pay ! rtpbin.send_rtp_sink_0 \
  
                    rtpbin.send_rtp_src_0 ! udpsink host=127.0.0.1 port=5000                            \
  
                    rtpbin.send_rtcp_src_0 ! udpsink host=127.0.0.1 port=5001 sync=false async=false    \
  
                    udpsrc port=5005 ! rtpbin.recv_rtcp_sink_0                           \
  
          pulsesrc ! audioconvert ! audioresample  ! audio/x-raw-int,rate=16000 !    \
  
                    speexenc bitrate=16000 ! rtpspeexpay ! rtpbin.send_rtp_sink_1                   \
  
                    rtpbin.send_rtp_src_1 ! udpsink host=127.0.0.1 port=5002                            \
  
                    rtpbin.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=5003 sync=false async=false    \
  
                    udpsrc port=5007 ! rtpbin.recv_rtcp_sink_1

  And receiver becomes :
  # !/bin/sh

  gst-launch -v\
  
      gstrtpbin name=rtpbin                                          \
  
      udpsrc caps="application/x-rtp,media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" \
  
              port=5000 ! rtpbin.recv_rtp_sink_0                                \
  
          rtpbin. ! rtph264depay ! ffdec_h264 ! xvimagesink                    \
  
       udpsrc port=5001 ! rtpbin.recv_rtcp_sink_0                               \
  
       rtpbin.send_rtcp_src_0 ! udpsink port=5005 sync=false async=false        \
  
      udpsrc caps="application/x-rtp,media=(string)audio, clock-rate=(int)16000, encoding-name=(string)SPEEX, encoding-params=(string)1, payload=(int)110" \
  
              port=5002 ! rtpbin.recv_rtp_sink_1                                \
  
          rtpbin. ! rtpspeexdepay ! speexdec ! audioresample ! audioconvert ! alsasink \
  
       udpsrc port=5003 ! rtpbin.recv_rtcp_sink_1                               \
  
       rtpbin.send_rtcp_src_1 ! udpsink host=127.0.0.1 port=5007 sync=false async=false

  This is what happen under Ubuntu 10.04. I didn't noticed such huge delays on Ubuntu 9.04 - the delays there was in range 2-3 seconds, AFAIR.

• Resampling audio using libswresample from 48000 to 44100

  9 août 2017, par Ruben Sanchez Castellano

  I’m trying to resample a decoded audio frame from 48KHz to 44.1KHz using the libswresample API. The code I have is the following :

  // 'frame' is the original decoded audio frame
  
  AVFrame *output_frame = av_frame_alloc();
  
  
  
  // Without this, there is no sound at all at the output (PTS stuff I guess)
  
  av_frame_copy_props(output_frame, frame);
  
  
  
  output_frame->channel_layout = audioStream->codec->channel_layout;
  
  output_frame->sample_rate = audioStream->codec->sample_rate;
  
  output_frame->format = audioStream->codec->sample_fmt;
  
  
  
  SwrContext *swr;
  
  // Configure resampling context
  
  swr = swr_alloc_set_opts(NULL,  // we're allocating a new context
  
                           AV_CH_LAYOUT_STEREO,  // out_ch_layout
  
                           AV_SAMPLE_FMT_FLTP,     // out_sample_fmt
  
                           44100,                // out_sample_rate
  
                           AV_CH_LAYOUT_STEREO,  // in_ch_layout
  
                           AV_SAMPLE_FMT_FLTP,   // in_sample_fmt
  
                           48000,                // in_sample_rate
  
                           0,                    // log_offset
  
                           NULL);                // log_ctx
  
  // Initialize resampling context
  
  swr_init(swr);
  
  
  
  // Perform conversion
  
  swr_convert_frame(swr, output_frame, frame);
  
  
  
  // Close resampling context
  
  swr_close(swr);
  
  swr_free(&swr);
  
  // Free the original frame and replace it with the new one
  
  av_frame_unref(frame);
  
  return output_frame;

  With this code I’m able to hear the audio at the output but it is also noisy. From what I read, this code without the av_frame_copy_props() should be enough but it is not working for some reason. Any ideas ?

  EDIT : The input stream encodes the audio using AAC and the number of samples is 1024. But, after conversion, the number of samples is 925.

  EDIT : I tried doing it in reverse. Since my app receives streams from any sources, some audio streams are 48KHz and some others 44.1KHz. So I tried resampling from 44.1 to 48 to avoid resampling loss. But now the frames has more than 1024 samples each one and the encoding fails.

  EDIT : I tried using libavfilter instead with the following filter chain :

  int init_filter_graph(AVStream *audio_st) {
  
  // create new graph
  
  filter_graph = avfilter_graph_alloc();
  
  if (!filter_graph) {
  
      av_log(NULL, AV_LOG_ERROR, "unable to create filter graph: out of memory\n");
  
      return -1;
  
  }
  
  
  
  AVFilter *abuffer = avfilter_get_by_name("abuffer");
  
  AVFilter *aformat = avfilter_get_by_name("aformat");
  
  AVFilter *asetnsamples = avfilter_get_by_name("asetnsamples");
  
  AVFilter *abuffersink = avfilter_get_by_name("abuffersink");
  
  
  
  int err;
  
  // create abuffer filter
  
  AVCodecContext *avctx = audio_st->codec;
  
  AVRational time_base = audio_st->time_base;
  
  snprintf(strbuf, sizeof(strbuf),
  
           "time_base=%d/%d:sample_rate=%d:sample_fmt=%s:channel_layout=0x%" PRIx64,
  
           time_base.num, time_base.den, avctx->sample_rate,
  
           av_get_sample_fmt_name(avctx->sample_fmt),
  
           avctx->channel_layout);
  
  fprintf(stderr, "abuffer: %s\n", strbuf);
  
  err = avfilter_graph_create_filter(&abuffer_ctx, abuffer,
  
                                     NULL, strbuf, NULL, filter_graph);
  
  if (err < 0) {
  
      av_log(NULL, AV_LOG_ERROR, "error initializing abuffer filter\n");
  
      return err;
  
  }
  
  // create aformat filter
  
  snprintf(strbuf, sizeof(strbuf),
  
           "sample_fmts=%s:sample_rates=%d:channel_layouts=0x%" PRIx64,
  
           av_get_sample_fmt_name(AV_SAMPLE_FMT_FLTP), 44100,
  
           AV_CH_LAYOUT_STEREO);
  
  fprintf(stderr, "aformat: %s\n", strbuf);
  
  err = avfilter_graph_create_filter(&aformat_ctx, aformat,
  
                                     NULL, strbuf, NULL, filter_graph);
  
  if (err < 0) {
  
      av_log(NULL, AV_LOG_ERROR, "unable to create aformat filter\n");
  
      return err;
  
  }
  
  // create asetnsamples filter
  
  snprintf(strbuf, sizeof(strbuf),
  
           "n=1024:p=0");
  
  fprintf(stderr, "asetnsamples: %s\n", strbuf);
  
  err = avfilter_graph_create_filter(&asetnsamples_ctx, asetnsamples,
  
                                     NULL, strbuf, NULL, filter_graph);
  
  if (err < 0) {
  
      av_log(NULL, AV_LOG_ERROR, "unable to create asetnsamples filter\n");
  
      return err;
  
  }
  
  // create abuffersink filter
  
  err = avfilter_graph_create_filter(&abuffersink_ctx, abuffersink,
  
                                     NULL, NULL, NULL, filter_graph);
  
  if (err < 0) {
  
      av_log(NULL, AV_LOG_ERROR, "unable to create aformat filter\n");
  
      return err;
  
  }
  
  
  
  // connect inputs and outputs
  
  if (err >= 0) err = avfilter_link(abuffer_ctx, 0, aformat_ctx, 0);
  
  if (err >= 0) err = avfilter_link(aformat_ctx, 0, asetnsamples_ctx, 0);
  
  if (err >= 0) err = avfilter_link(asetnsamples_ctx, 0, abuffersink_ctx, 0);
  
  if (err < 0) {
  
      av_log(NULL, AV_LOG_ERROR, "error connecting filters\n");
  
      return err;
  
  }
  
  err = avfilter_graph_config(filter_graph, NULL);
  
  if (err < 0) {
  
      av_log(NULL, AV_LOG_ERROR, "error configuring the filter graph\n");
  
      return err;
  
  }
  
  return 0;
  
  }

  Now the resulting frame has 1024 samples but the audio is still choppy.

• The neutering of Google Code-In 2011

  23 octobre 2011, par Dark Shikari — development, GCI, google, x264

  Posting this from the Google Summer of Code Mentor Summit, at a session about Google Code-In !

  Google Code-In is the most innovative open-source program I’ve ever seen. It provided a way for students who had never done open source — or never even done programming — to get involved in open source work. It made it easy for people who weren’t sure of their ability, who didn’t know whether they could do open source, to get involved and realize that yes, they too could do amazing work — whether code useful to millions of people, documentation to make the code useful, translations to make it accessible, and more. Hundreds of students had a great experience, learned new things, and many stayed around in open source projects afterwards because they enjoyed it so much !

  x264 benefitted greatly from Google Code-In. Most of the high bit depth assembly code was written through GCI — literally man-weeks of work by an professional developer, done by high-schoolers who had never written assembly before ! Furthermore, we got loads of bugs fixed in ffmpeg/libav, a regression test tool, and more. And best of all, we gained a new developer : Daniel Kang, who is now a student at MIT, an x264 and libav developer, and has gotten paid work applying the skills he learned in Google Code-In !

  Some students in GCI complained about the system being “unfair”. Task difficulties were inconsistent and there were many ways to game the system to get lots of points. Some people complained about Daniel — he was completing a staggering number of tasks, so they must be too easy. Yet many of the other students considered these tasks too hard. I mean, I’m asking high school students to write hundreds of lines of complicated assembly code in one of the world’s most complicated instruction sets, and optimize it to meet extremely strict code-review standards ! Of course, there may have been valid complaints about other projects : I did hear from many students talking about gaming the system and finding the easiest, most “profitable” tasks. Though, with the payout capped at $500, the only prize for gaming the system is a high rank on the points list.

  According to people at the session, in an effort to make GCI more “fair”, Google has decided to change the system. There are two big changes they’re making.

  Firstly, Google is requiring projects to submit tasks on only two dates : the start, and the halfway point. But in Google Code-In, we certainly had no idea at the start what types of tasks would be the most popular — or new ideas that came up over time. Often students would come up with ideas for tasks, which we could then add ! A waterfall-style plan-everything-in-advance model does not work for real-world coding. The halfway point addition may solve this somewhat, but this is still going to dramatically reduce the number of ideas that can be proposed as tasks.

  Secondly, Google is requiring projects to submit at least 5 tasks of each category just to apply. Quality assurance, translation, documentation, coding, outreach, training, user interface, and research. For large projects like Gnome, this is easy : they can certainly come up with 5 for each on such a large, general project. But often for a small, focused project, some of these are completely irrelevant. This rules out a huge number of smaller projects that just don’t have relevant work in all these categories. x264 may be saved here : as we work under the Videolan umbrella, we’ll likely be able to fudge enough tasks from Videolan to cover the gaps. But for hundreds of other organizations, they are going to be out of luck. It would make more sense to require, say, 5 out of 8 of the categories, to allow some flexibility, while still encouraging interesting non-coding tasks.

  For example, what’s “user interface” for a software library with a stable API, say, a libc ? Can you make 5 tasks out of it that are actually useful ?

  If x264 applied on its own, could you come up with 5 real, meaningful tasks in each category for it ? It might be possible, but it’d require a lot of stretching.

  How many smaller or more-focused projects do you think are going to give up and not apply because of this ?

  Is GCI supposed to be something for everyone, or just or Gnome, KDE, and other megaprojects ?