Sur d’autres sites (5809)

• iOS Objective C Opus audio stream kAudioFormatOpus — specification / conversion query

  16 avril 2022, par deffodeffo

  I'm working on a React Native voice app wanting to record straight to an Opus stream.

  


  On the iOS side at device level, I'm working in Objective C and I'm using an AVAudioSession recorder with formatID set to kAudioFormatOpus. The recorder captures audio data in the specified format and passes packets for upstreaming to the React Native app.

  


  This is all working well and emitting a stream of audio data when I run my code on an iOS Simulator or real device.

  


  My problem is when I receive the audio stream in the backend, I find I'm unable to reliably decode the Opus stream using ffmpeg. (I wish to decode to PCM.)

  


  Whilst I could post up specific code to show what I'm doing in more detail, my question at this stage is more generic in nature :

  


  Is anyone familiar with the format of Opus audio stream that is generated by iOS AudioSession recorder under audio format kAudioFormatOpus ? Is anyone able to suggest proven conversion techniques ? (e.g. some ffmpeg commands), or else hook me up with some links to format specs so I can figure out what is going on here ?

  


  The Apple Developer documentation contains zero useful information : https://developer.apple.com/documentation/coreaudiotypes/1572096-audio_data_format_identifiers/kaudioformatopus?changes=la__2&language=objc

  


  I have looked all over the internet, but I'm unable to find any useful spec info that corresponds to the stream that AVAudioSession recorder is outputting. I have seen a few posts which say Apple have not fully complied with the Opus spec, but I don't know enough about the proper structure of Opus to ascertain this for myself.

  


  Any help would be very much appreciated.

  


  Thanks all

  


• Can pyAudioAnalysis be used on a live http audio stream ?

  30 mars 2022, par Tompo

  I am trying to use pyAudioAnalysis to analyse an audio stream in real-time from a HTTP stream. My goal is to use the Zero Crossing Rate (ZCR) and other methods in this library to identify events in the stream.

  


  pyAudioAnalysis only supports input from a file but converting a http stream to a .wav will create a large overhead and temporary file management I would like to avoid.

  


  My method is as follows :

  


  Using ffmpeg I was able to get the raw audio bytes into a subprocess pipe.

  


  try:
    song = subprocess.Popen(["ffmpeg", "-i", "https://media-url/example", "-acodec", "pcm_s16le", "-ac", "1", "-f", "wav", "pipe:1"],
                            stdout=subprocess.PIPE)


  


  I then buffered this data using pyAudio with the hope of being able to use the bytes in pyAudioAnalysis

  


  CHUNK = 65536

p = pyaudio.PyAudio()

stream = p.open(format=pyaudio.paInt16,
                channels=1,
                rate=44100,
                output=True)

data = song.stdout.read(CHUNK)

while len(data) > 0:
    stream.write(data)
    data = song.stdout.read(CHUNK)


  


  However, inputting this data output into AudioBasicIO.read_audio_generic() produces an empty numpy array.

  


  Is there a valid solution to this problem without temporary file creation ?

  


• Why android video player that based on ffmpeg can't sync video' time and audio's time ?

  6 juillet 2021, par ZSpirytus

  Background

  


  Recently, I use ffmpeg to write my first Android video player. But video channel's time is faster than audio channel's time about 2 3 times.

  


  Question

  


  Why android video player audio and video is out of sync ? Video is about faster than audio 2 3 times. Thanks for your reading and answers.

  


  Code

  


  In short, I use PacketDispatcher to read AVPacket from http hlv source :

  


  void PacketDispatcher::RealDispatch() {
    while (GetStatus() != DISPATCHER_STOP) {
        ...

        AVPacket *av_packet = av_packet_alloc();
        int ret = av_read_frame(av_format_context, av_packet);

        // PacketDispatcher is who read the AVPacket from http hlv source 
        // and dispatch to decoder by its stream index.
        decoder_map[av_packet->stream_index]->Push(av_packet);
    }
}


  


  And then, Decoder written by Producer-Consumer Pattern, Decoder maintain a queue that store all the AVPacket received from PacketDispatcher. The code like this :

  


  // write to the queue
void BaseDecoder::Push(AVPacket *av_packet) {
    ...
    av_packet_queue.push(av_packet);
    ...
}

// real decode logic
void BaseDecoder::RealDecode() {
    ...
    while (true) {
        // read frame from codec
        AVFrame *av_frame = av_frame_alloc();
        ret = avcodec_receive_frame(av_codec_ctx, av_frame);

        void *decode_result = DecodeFrame(av_frame);
        // send to render
        m_render->Render(decode_result);
    }
}


  


  And then, I do rendering logic in Render. Render also written by Producer-Consumer Pattern, it maintain a queue that store AVFrame received from Decoder, the code like this :

  


  // write AVFrame
void BaseRender::Render(void *frame_data) {
    ...
    frame_queue.push(frame_data);
    ...
}

// render to surface or Open SL
void BaseRender::RealRender() {
    if (m_render_synchronizer && m_render_synchronizer->Sync(frame_data)) {
        continue;
    }
}


  


  Finally, the synchronizer will decide to sleep time or drop video frame according to the frame pts, frame pts is :

  


  frame_data->pts = av_frame->best_effort_timestamp * av_q2d(GetTimeBase());


  


  Also, video extra delay is :

  


  frame_data->video_extra_delay = av_frame->repeat_pict * 1.0 / fps * 2.0;


  


  RenderSynchronizer code like this :

  


  bool RenderSynchronizer::Sync(void *frame_data) {&#xA;    auto base_frame_data = static_cast<baseframedata>(frame_data);&#xA;    if (base_frame_data->media_type == AVMEDIA_TYPE_AUDIO) {&#xA;        audio_pts = pcm_data->pts;&#xA;        return false;&#xA;    } else if (base_frame_data->media_type == AVMEDIA_TYPE_VIDEO) {&#xA;        video_pts = rgba_data->pts;&#xA;        return ReceiveVideoFrame(static_cast<rgbadata>(frame_data));&#xA;    }&#xA;    return false;&#xA;}&#xA;&#xA;bool RenderSynchronizer::ReceiveVideoFrame(RGBAData *rgba_data) {&#xA;    if (audio_pts &lt;= 0 || video_pts &lt;= 0) {&#xA;        return false;&#xA;    }&#xA;&#xA;    double diff = video_pts - audio_pts;&#xA;    if (diff > 0) {&#xA;        if (diff > 1) {&#xA;            av_usleep((unsigned int) (rgba_data->extra_delay * 1000000.0));&#xA;        } else {&#xA;            av_usleep((unsigned int) ((diff &#x2B; rgba_data->extra_delay) * 1000000.0));&#xA;        }&#xA;        return false;&#xA;    } else if (diff &lt; 0) {&#xA;        LOGD(TAG, "drop video frame");&#xA;        return true;&#xA;    } else {&#xA;        return false;&#xA;    }&#xA;}&#xA;</rgbadata></baseframedata>