Recherche avancée
Médias (91)
-
MediaSPIP Simple : futur thème graphique par défaut ?
26 septembre 2013, par
Mis à jour : Octobre 2013
Langue : français
Type : Video
-
avec chosen
13 septembre 2013, par
Mis à jour : Septembre 2013
Langue : français
Type : Image
-
sans chosen
13 septembre 2013, par
Mis à jour : Septembre 2013
Langue : français
Type : Image
-
config chosen
13 septembre 2013, par
Mis à jour : Septembre 2013
Langue : français
Type : Image
-
SPIP - plugins - embed code - Exemple
2 septembre 2013, par
Mis à jour : Septembre 2013
Langue : français
Type : Image
-
GetID3 - Bloc informations de fichiers
9 avril 2013, par
Mis à jour : Mai 2013
Langue : français
Type : Image
Autres articles (25)
-
Selection of projects using MediaSPIP
2 mai 2011, parThe examples below are representative elements of MediaSPIP specific uses for specific projects.
MediaSPIP farm @ Infini
The non profit organizationInfini develops hospitality activities, internet access point, training, realizing innovative projects in the field of information and communication technologies and Communication, and hosting of websites. It plays a unique and prominent role in the Brest (France) area, at the national level, among the half-dozen such association. Its members (...) -
Keeping control of your media in your hands
13 avril 2011, parThe vocabulary used on this site and around MediaSPIP in general, aims to avoid reference to Web 2.0 and the companies that profit from media-sharing.
While using MediaSPIP, you are invited to avoid using words like "Brand", "Cloud" and "Market".
MediaSPIP is designed to facilitate the sharing of creative media online, while allowing authors to retain complete control of their work.
MediaSPIP aims to be accessible to as many people as possible and development is based on expanding the (...) -
Sélection de projets utilisant MediaSPIP
29 avril 2011, parLes exemples cités ci-dessous sont des éléments représentatifs d’usages spécifiques de MediaSPIP pour certains projets.
Vous pensez avoir un site "remarquable" réalisé avec MediaSPIP ? Faites le nous savoir ici.
Ferme MediaSPIP @ Infini
L’Association Infini développe des activités d’accueil, de point d’accès internet, de formation, de conduite de projets innovants dans le domaine des Technologies de l’Information et de la Communication, et l’hébergement de sites. Elle joue en la matière un rôle unique (...)
Sur d’autres sites (4428)
-
Read multiple frames in demuxer
22 décembre 2024, par AyconI use
FFmpeg.AutoGen. This is C# wrapper offfmpeg(7.0) C++ library for reading mediafiles and generate stream of frames for other application.
I want get n frames and hold pointers in memory indefinitely.
However, I am completely confused trying to figure out which API is deprecated and how I can tell ffmpeg to hold the pointers in memory until I tell it to.

I don't want to copy the frame after receiving it if I can avoid it.

I tried many ways.

My last attempt was to receive the frames usingffmpeg.avcodec_send_packet(),ffmpeg.av_read_frame()andffmpeg.avcodec_receive_frame()functions as it is specified in the current manual.

My code fragment for reading frames :

using Core.Backends.FFmpeg.Helpers;
using Core.Backends.FFmpeg.UnsafeWrappers;
using Core.Enums;
using Core.Interfaces;
using FFmpeg.AutoGen.Abstractions;
using System.Diagnostics;
using System.Drawing;

namespace Core.Backends.FFmpeg.Internal;

internal class Video : IVideo
{
 private readonly AVFormatHandler p_format;
 private readonly AVCodecHandler p_codec;
 private readonly AVPacketWrapper p_packet;
 private readonly FramesPool p_framesPool;
 private readonly FramesPool p_framesBufferPool;
 private bool p_disposedValue;

 public Video(VideoMetadata _videoMetadata, AVFormatHandler _format, AVCodecHandler _codec, int _bufferizedFramesCount = 1)
 {
 Duration = _videoMetadata.Duration;
 FrameRate = _videoMetadata.FrameRate;
 FrameSize = _videoMetadata.FrameSize;
 PixelFormat = _videoMetadata.PixelFormat;
 SelectedStreamID = _videoMetadata.SelectedStreamID;
 p_format = _format;
 p_codec = _codec;
 p_packet = new AVPacketWrapper();
 var frame = new AVFrameWrapper(p_format, p_packet);
 p_framesPool = new(frame, _bufferizedFramesCount);
 p_framesBufferPool = new(frame, _bufferizedFramesCount);
 }

 /// <inheritdoc></inheritdoc>
 public long Duration { get; init; }

 /// <inheritdoc></inheritdoc>
 public (int num, int den) FrameRate { get; init; }

 /// <inheritdoc></inheritdoc>
 public Size FrameSize { get; init; }

 /// <inheritdoc></inheritdoc>
 public PixelFormat PixelFormat { get; init; }

 private int SelectedStreamID { get; init; }

 private unsafe int SendPacket(AVPacketWrapper? _packet)
 {
 if (_packet == null)
 return ffmpeg.avcodec_send_packet(p_codec.AVCodecContextPointer, null);

 return ffmpeg.avcodec_send_packet(p_codec.AVCodecContextPointer, _packet.AVPacketPointer);
 }

 private unsafe bool IsSelectedStream(AVPacketWrapper _packet)
 {
 return _packet.AVPacketPointer->stream_index == SelectedStreamID;
 }

 private unsafe int ReadFrame(AVPacketWrapper _packet)
 {
 return ffmpeg.av_read_frame(p_format.AVFormatPointer, _packet.AVPacketPointer);
 }

 private static unsafe void UnrefPacket(AVPacketWrapper _packet) => ffmpeg.av_packet_unref(_packet.AVPacketPointer);

 private IEnumerable<int> ReadToSelectedStream(AVPacketWrapper _packet)
 {
 do
 {
 UnrefPacket(p_packet);
 yield return ReadFrame(_packet);
 } while (!IsSelectedStream(_packet));
 }

 private unsafe void FlushBuffers() => ffmpeg.avcodec_flush_buffers(p_codec.AVCodecContextPointer);

 private IEnumerable<avpacketwrapper> GetNextPacketPrivate()
 {
 try
 {
 while (true)
 {
 foreach (int errorCodeRead in ReadToSelectedStream(p_packet))
 {
 if (errorCodeRead == ffmpeg.AVERROR_EOF)
 break;

 errorCodeRead.ThrowInvalidOperationExceptionIfError();
 }

 int errorCodeSend = SendPacket(p_packet);

 if (errorCodeSend == ffmpeg.AVERROR(ffmpeg.EAGAIN))
 {
 yield return p_packet;
 continue;
 }

 if (errorCodeSend == ffmpeg.AVERROR_EOF)
 {
 yield return p_packet;
 break;
 }

 errorCodeSend.ThrowInvalidOperationExceptionIfError();

 yield return p_packet;
 }

 // Last iteration special case handling
 int errorCodeSendLast = SendPacket(null);

 if (errorCodeSendLast != ffmpeg.AVERROR_EOF)
 errorCodeSendLast.ThrowInvalidOperationExceptionIfError();

 yield return p_packet;
 }
 finally
 {
 UnrefPacket(p_packet);
 FlushBuffers();
 }
 }

 private unsafe int ReceiveFrame(AVFrameWrapper _frame)
 {
 return ffmpeg.avcodec_receive_frame(p_codec.AVCodecContextPointer, _frame.AVFramePointer);
 }

 private unsafe AVFrameWrapper HWFrameCopyIfRequired(AVCodecHandler _codec, AVFrameWrapper _frame, AVFrameWrapper _buffer)
 {
 if (_codec.AVCodecContextPointer->hw_device_ctx != null)
 {
 int errorCode = ffmpeg.av_hwframe_transfer_data(_buffer.AVFramePointer, _frame.AVFramePointer, flags: 0);
 errorCode.ThrowInvalidOperationExceptionIfError();
 return _buffer;
 }

 return _frame;
 }

 private IEnumerable GetNextFramePrivate(AVFrameWrapper _fresh_frame, AVFrameWrapper _fresh_frameBuffer)
 {
 int readCode;

 while (true)
 {
 readCode = ReceiveFrame(_fresh_frame);

 if (readCode == ffmpeg.AVERROR(ffmpeg.EAGAIN) || readCode == ffmpeg.AVERROR_EOF)
 yield break;

 readCode.ThrowInvalidOperationExceptionIfError();
 
 yield return HWFrameCopyIfRequired(p_codec, _fresh_frame, _fresh_frameBuffer);
 }
 }

 private static void RefreshFrames
 (
 IEnumerator<avframewrapper> _framesEnumerator,
 IEnumerator<avframewrapper> _framesBufferEnumerator,
 out AVFrameWrapper _frame,
 out AVFrameWrapper _frameBuffer
 )
 {
 // Catch fresh frame from pool
 Debug.Assert(_framesEnumerator.MoveNext(), "Пул фреймов никогда не должен завершать предоставление фреймов.");
 _frame = _framesEnumerator.Current;

 // Catch fresh frame buffer from pool
 Debug.Assert(_framesBufferEnumerator.MoveNext(), "Пул фреймов никогда не должен завершать предоставление фреймов.");
 _frameBuffer = _framesBufferEnumerator.Current;
 }

 /// <inheritdoc></inheritdoc>
 public IEnumerable GetNextFrame()
 {
 IEnumerator<avframewrapper> framesEnumerator = p_framesPool.GetNextFrame().GetEnumerator();
 IEnumerator<avframewrapper> framesBufferEnumerator = p_framesBufferPool.GetNextFrame().GetEnumerator();
 RefreshFrames(framesEnumerator, framesBufferEnumerator, out AVFrameWrapper fresh_frame, out AVFrameWrapper fresh_frameBuffer);
 foreach (var packet in GetNextPacketPrivate())
 foreach (var frame in GetNextFramePrivate(fresh_frame, fresh_frameBuffer))
 {
 yield return frame;
 RefreshFrames(framesEnumerator, framesBufferEnumerator, out fresh_frame, out fresh_frameBuffer);
 }
 }

 protected virtual void Dispose(bool disposing)
 {
 if (!p_disposedValue)
 {
 if (disposing)
 {
 }

 p_packet.Dispose();
 p_framesPool.Flush();
 p_framesBufferPool.Flush();

 p_disposedValue = true;
 }
 }

 ~Video()
 {
 Dispose(disposing: false);
 }

 public void Dispose()
 {
 Dispose(disposing: true);
 GC.SuppressFinalize(this);
 }
}
</avframewrapper></avframewrapper></avframewrapper></avframewrapper></avpacketwrapper></int>

My
FramesPoolclass :

using Core.Backends.FFmpeg.UnsafeWrappers;
using FFmpeg.AutoGen.Abstractions;

namespace Core.Backends.FFmpeg.Internal;

internal class FramesPool
{
 private readonly AVFrameWrapper p_frameWrapper;
 private readonly Queue<avframewrapper> p_frames;
 private readonly int p_count;

 public FramesPool(AVFrameWrapper _initframeWrapper, int _count = 1)
 {
 p_frameWrapper = _initframeWrapper;
 p_frames = new(_count);
 p_frames.Enqueue(p_frameWrapper);
 p_count = _count;
 }

 private static unsafe void UnrefFrame(AVFrameWrapper _frame) => ffmpeg.av_frame_unref(_frame.AVFramePointer);

 public IEnumerable<avframewrapper> GetNextFrame()
 {
 // First frame case
 UnrefFrame(p_frameWrapper);
 yield return p_frameWrapper;

 while (true)
 {
 if (p_frames.Count < p_count)
 {
 var new_frame = p_frameWrapper.Clone();
 p_frames.Enqueue(new_frame);
 yield return new_frame;
 }
 else
 {
 var frame = p_frames.Dequeue();
 UnrefFrame(frame);
 yield return frame;
 p_frames.Enqueue(frame);
 }
 }
 }

 public void Flush()
 {
 foreach(var frame in p_frames)
 {
 UnrefFrame(frame);
 frame.Dispose();
 }

 p_frames.Clear();
 }
}
</avframewrapper></avframewrapper>

Additional calls, among others :


ffmpeg.avformat_alloc_context();
ffmpeg.avformat_open_input(pptr, p_filePath, null, null);
ffmpeg.av_hwdevice_ctx_create(&p_avCodecHandler!.AVCodecContextPointer->hw_device_ctx, strongDevice, null, null, 0);
ffmpeg.av_find_best_stream(/*args*/);
ffmpeg.avcodec_alloc_context3(p_avCodec);
ffmpeg.avcodec_parameters_to_context(/*args*/);
ffmpeg.avcodec_open2(/*args*/);


Function
ffmpeg.av_hwframe_transfer_data(_buffer.AVFramePointer, _frame.AVFramePointer, flags: 0);returns "-22" (message : "Invalid argument")

Please, help me)


-
FFMpeg gpl (ffmpeg-4.2.1-win64-dev_and_shared) version give different decode result (cmd query vs code)
31 mai 2021, par Aleksey Timoshchenko*all the source img I put on my google drive due to SO restrictions (just click on links provided in the text)


The problem is that for
.h264I use two implementations (cmd query and code) (depends on the tasks) that give me different results and I don't see any reason for this.

Before all I would like to give an explanation, I have
.bmpbayer image, then I do debayering and compress it to.h264(compress .h264) with the script

ffmpeg -y -hide_banner -i orig_bayer.bmp -vf format=gray -f rawvideo pipe: | ffmpeg -hide_banner -y -framerate 30 -f rawvideo -pixel_format bayer_rggb8 -video_size 4096x3000 -i pipe: -c:v hevc_nvenc -qp 0 -pix_fmt yuv444p res.h264



Then the first cmd query implementation of decoding (image result)


ffmpeg -y -i res.h264 -vframes 1 -f image2 gpl_script_res.bmp -hide_banner



The second implementation is code one and takes more lines (image result here)


Init ffmpeg


bool FFmpegDecoder::InitFFmpeg()
{
 m_pAVPkt = av_packet_alloc();
 m_pAVFrame = av_frame_alloc();
 m_pAVFrameRGB = av_frame_alloc();

 m_pAVFormatCtx = avformat_alloc_context();
 m_pIoCtx->initAVFormatContext(m_pAVFormatCtx);

 if (avformat_open_input(&m_pAVFormatCtx, "", nullptr, nullptr) != 0)
 {
 printf("FFmpegDecoder::InitFFmpeg: error in avformat_open_input\n");
 return false;
 }

 if (avformat_find_stream_info(m_pAVFormatCtx, nullptr) < 0)
 {
 printf("FFmpegDecoder::InitFFmpeg: error in avformat_find_stream_info\n");
 return false;
 }

 //av_dump_format(ctx_format, 0, "", false);
 for (int i = 0; i < (int)m_pAVFormatCtx->nb_streams; i++)
 {
 if (m_pAVFormatCtx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO)
 {
 m_streamIdx = i;
 m_pAVStream = m_pAVFormatCtx->streams[i];
 break;
 }
 }
 if (m_pAVStream == nullptr)
 {
 printf("FFmpegDecoder::InitFFmpeg: failed to find video stream\n");
 return false;
 }

 m_pAVCodec = avcodec_find_decoder(m_pAVStream->codecpar->codec_id);
 if (!m_pAVCodec)
 {
 printf("FFmpegDecoder::InitFFmpeg: error in avcodec_find_decoder\n");
 return false;
 }

 m_pAVCodecCtx = avcodec_alloc_context3(m_pAVCodec);
 if (!m_pAVCodecCtx)
 {
 printf("FFmpegDecoder::InitFFmpeg: error in avcodec_alloc_context3\n");
 return false;
 }

 if (avcodec_parameters_to_context(m_pAVCodecCtx, m_pAVStream->codecpar) < 0)
 {
 printf("FFmpegDecoder::InitFFmpeg: error in avcodec_parameters_to_context\n");
 return false;
 }

 if (avcodec_open2(m_pAVCodecCtx, m_pAVCodec, nullptr) < 0)
 {
 printf("FFmpegDecoder::InitFFmpeg: error in avcodec_open2\n");
 return false;
 }

 m_pAVFrameRGB->format = AV_PIX_FMT_BGR24;
 m_pAVFrameRGB->width = m_pAVCodecCtx->width;
 m_pAVFrameRGB->height = m_pAVCodecCtx->height;
 if (av_frame_get_buffer(m_pAVFrameRGB, 32) != 0)
 {
 printf("FFmpegDecoder::InitFFmpeg: error in av_frame_get_buffer\n");
 return false;
 }

 m_streamRotationDegrees = GetAVStreamRotation(m_pAVStream);
 m_estimatedFramesCount = 0;
 assert(m_pAVFormatCtx->nb_streams > 0);
 if (m_pAVFormatCtx->nb_streams > 0)
 {
 m_estimatedFramesCount = m_pAVFormatCtx->streams[0]->nb_frames;
 }

 return InitConvertColorSpace(); 
}

bool FFmpegDecoder::InitConvertColorSpace()
{
 // Init converter from YUV420p to BGR:
 m_pSwsCtxConvertImg = sws_getContext(m_pAVCodecCtx->width, m_pAVCodecCtx->height, m_pAVCodecCtx->pix_fmt, m_pAVCodecCtx->width, m_pAVCodecCtx->height, AV_PIX_FMT_RGB24, SWS_FAST_BILINEAR, NULL, NULL, NULL);
 if (!m_pSwsCtxConvertImg)
 {
 printf("FFmpegDecoder::InitFFmpeg: error in sws_getContext\n");
 return false;
 }
 return true;
}


Decoding impl


bool FFmpegDecoder::DecodeContinue(int firstFrameIdx, int maxNumFrames)
{
 if (firstFrameIdx == FIRST_FRAME_IDX_BEGINNING)
 {
 firstFrameIdx = 0;
 }

 auto lastReportedFrameIdxTillNow = GetLastReportedFrameIdx();

 if (GetLastDecodedFrameIdx() >= 0 && firstFrameIdx <= GetLastDecodedFrameIdx())
 {
 printf("FFmpegDecoder::DecodeContinue FAILED: firstFrameIdx (%d) already passed decoded (Last decoded idx: %d)\n", firstFrameIdx, GetLastDecodedFrameIdx());
 return false;
 }

 bool bRes;
 int nRet = 0;
 bool bDecodeShouldEnd = false;

 if (m_pAVPkt != nullptr)
 {
 while (nRet >= 0)
 {
 m_bCurrentAVPktSentToCodec = false;
 nRet = av_read_frame(m_pAVFormatCtx, m_pAVPkt);
 if (nRet < 0)
 {
 break;
 }
 if (m_pAVPkt->stream_index == m_streamIdx)
 {
 bRes = DecodeCurrentAVPkt(firstFrameIdx, maxNumFrames, bDecodeShouldEnd);
 if (!bRes || m_bRequestedAbort)
 {
 av_packet_unref(m_pAVPkt);
 return false;
 }

 if (bDecodeShouldEnd)
 {
 av_packet_unref(m_pAVPkt);
 return true;
 }
 }

 av_packet_unref(m_pAVPkt);
 }
 m_bCurrentAVPktSentToCodec = false;
 m_pAVPkt = nullptr;
 }

 // drain:
 bRes = DecodeCurrentAVPkt(firstFrameIdx, maxNumFrames, bDecodeShouldEnd);
 if (!bRes)
 {
 return false;
 }

 if (lastReportedFrameIdxTillNow == GetLastReportedFrameIdx())
 {
 printf("FFmpegDecoder::DecodeContinue(firstFrameIdx==%d, maxNumFrames==%d) FAILED: no new frame was decoded\n", firstFrameIdx, maxNumFrames);
 return false;
 }

 return true;
}


bool FFmpegDecoder::DecodeCurrentAVPkt(int firstFrameIdx, int maxNumFrames, bool & bDecodeShouldEnd)
{
 bDecodeShouldEnd = false;

 int ret = 0;
 if (m_bCurrentAVPktSentToCodec == false)
 {
 ret = avcodec_send_packet(m_pAVCodecCtx, m_pAVPkt);
 m_bCurrentAVPktSentToCodec = true;
 if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
 {
 printf("FFmpegDecoder::DecodeFrameImp: error EAGAIN/AVERROR_EOF in avcodec_send_packet\n");
 return false;
 }
 if (ret < 0)
 {
 if (ret == AVERROR_INVALIDDATA)
 {
 printf("FFmpegDecoder::DecodeFrameImp: error (%d - AVERROR_INVALIDDATA) in avcodec_send_packet\n", ret);
 }
 else
 {
 printf("FFmpegDecoder::DecodeFrameImp: error (%d) in avcodec_send_packet\n", ret);
 }
 return false;
 }
 }

 ret = 0;
 while (ret >= 0)
 {
 ret = avcodec_receive_frame(m_pAVCodecCtx, m_pAVFrame);
 if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)
 {
 break;
 }

 IncrementLastDecodedFrameIdx();
 if (GetLastDecodedFrameIdx() < firstFrameIdx)
 {
 printf("FFmpegDecoder::DecodeCurrentAVPkt ignoring frame idx %d\n", GetLastDecodedFrameIdx());
 continue; // we don't need this frame
 }

 AVFrame * theFrame = m_pAVFrame; // default

 for (int j = 0; j < m_pAVFrame->nb_side_data; j++)
 {
 AVFrameSideData *sd = m_pAVFrame->side_data[j];
 if (sd->type == AV_FRAME_DATA_DISPLAYMATRIX) {
 auto ddd = av_display_rotation_get((int32_t *)sd->data);
 }
 }

 if (m_pSwsCtxConvertImg != nullptr)
 {
 {
 if (sws_scale(m_pSwsCtxConvertImg, theFrame->data, theFrame->linesize, 0, theFrame->height, m_pAVFrameRGB->data, m_pAVFrameRGB->linesize) == 0)
 {
 printf("FFmpegDecoder::DecodeFrameImp: error in sws_scale\n");
 return false;
 }
 }
 int numChannels = 3;
 FFmpegDecoderCallback::EPixelFormat pixFormat = FFmpegDecoderCallback::EPixelFormat::RGB;
 // Report frame to the client and update last reported frame idx:
 m_pCB->FFmpegDecoderCallback_HandleFrame(m_reqId, GetLastDecodedFrameIdx(), m_pAVFrameRGB->width, m_pAVFrameRGB->height, m_pAVFrameRGB->linesize[0], pixFormat, numChannels, m_pAVFrameRGB->data[0]);
 m_lastReportedFrameIdx = GetLastDecodedFrameIdx(); 
 }

 if (maxNumFrames != MAX_NUM_FRAMES_INFINITE && GetLastDecodedFrameIdx() >= (firstFrameIdx + maxNumFrames - 1))
 {
 bDecodeShouldEnd = true;
 return true;
 }
 }
 return true;
}

/*virtual*/ void FFmpegOneDecoderCtx::FFmpegDecoderCallback_HandleFrame(int reqId, int frameIdx0based, int width, int height, int widthStepBytes, EPixelFormat pixFormat, int numChannels, void * pData) /*override*/
{
 // We don't have metadata json => return the frame as is:
 m_pLastFrame->create(height, width, CV_8UC3);
 *m_pLastFrame = cv::Scalar(0, 0, 0);
 unsigned char * pSrc = reinterpret_cast<unsigned char="char">(pData);
 unsigned char *pDst = m_pLastFrame->data;
 auto dstStride = m_pLastFrame->step[0];
 for (int y = 0; y < height; ++y)
 {
 memcpy(pDst + y * dstStride, pSrc + y * widthStepBytes, numChannels*width);
 }
}
</unsigned>

And eventually usage looks like this


//>>>Get Frame
 FFmpegMultiDecoder decoder;
 decoder.HandleRequest_GetFrame(nullptr, filename, 1, image);
 //<<<

 //>>> Collor conversion from BGR to RGB
 cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
 //<<<
 bool isOk = cv::imwrite(save_location, image);


The problem is that if you try to open two final decompressed images


- 

- one with code https://drive.google.com/file/d/1sfTnqvHKQ2DUy0uP8POZXDw2-u3oIRfZ/view?usp=sharing
- one with cmd query https://drive.google.com/file/d/1cwsOltk3DVtK86eLeyhiYjNeEXj0msES/view?usp=sharing






and try to flip from one image to other you'll see that image I got by cmd query a little bit brighter than that I got by code.


What is a possible problem here ?


If I missed smth feel free to ask.


-
Vulkan image data to AVFrames and to video
12 avril 2024, par W4zab1I am trying to encode Vulkan image data into video with MPEG4 format. For some reason the output videofile is corrupted. FFProbe shows discontinuity in timestamps, and the frames are corrupted.
First I prepare my video encoder

Then I get FrameEnded events from my engine where I can get the image data from the vulkan swapchain.

I then convert the image data from vulkan to AVFrames (RGBA to YUV420P), then I pass the frames into queue.

This queue is then handled in another thread, where the frames are processed, and written into video.

I am bit of a noob with ffmpeg, so there can be some code that does not make sense.

This seems right straight forward logic, but there is probably some problems with codec params, way I am converting the imagedata to AVFrame, or something of that sort.

The videofile still gets created, and has some data in it (it is > 0 bytes, and longer the recording, bigger the filesize).

There is no errors from ffmpeg with log_level set to DEBUG.

struct FrameData {
 AVFrame* frame;
 int frame_index;
};

class EventListenerVideoCapture : public VEEventListenerGLFW {
private:
 AVFormatContext* format_ctx = nullptr;
 AVCodec* video_codec = nullptr;
 AVCodecContext* codec_context = nullptr;
 AVStream* video_stream = nullptr;
 AVDictionary* muxer_opts = nullptr;
 int frame_index = 0;

 std::queue frame_queue;
 std::mutex queue_mtx;
 std::condition_variable queue_cv;
 std::atomic<bool> stop_processing{ false };
 std::thread video_processing_thread;
 int prepare_video_encoder()
 {
 av_log_set_level(AV_LOG_DEBUG);
 // Add video stream to format context
 avformat_alloc_output_context2(&format_ctx, nullptr, nullptr, "video.mpg");
 video_stream = avformat_new_stream(format_ctx, NULL);
 video_codec = (AVCodec*)avcodec_find_encoder(AV_CODEC_ID_MPEG4);
 codec_context = avcodec_alloc_context3(video_codec);
 if (!format_ctx) { std::cerr << "Error: Failed to allocate format context" << std::endl; system("pause"); }
 if (!video_stream) { std::cerr << "Error: Failed to create new stream" << std::endl; system("pause"); }
 if (!video_codec) { std::cerr << "Error: Failed to find video codec" << std::endl; system("pause"); }
 if (!codec_context) { std::cerr << "Error: Failed to allocate codec context" << std::endl; system("pause"); }

 if (avio_open(&format_ctx->pb, "video.mpg", AVIO_FLAG_WRITE) < 0) { std::cerr << "Error: Failed to open file for writing!" << std::endl; return -1; }

 av_opt_set(codec_context->priv_data, "preset", "fast", 0);

 codec_context->codec_id = AV_CODEC_ID_MPEG4;
 codec_context->codec_type = AVMEDIA_TYPE_VIDEO;
 codec_context->pix_fmt = AV_PIX_FMT_YUV420P;
 codec_context->width = getWindowPointer()->getExtent().width;
 codec_context->height = getWindowPointer()->getExtent().height;
 codec_context->bit_rate = 1000 * 1000; // Bitrate
 codec_context->time_base = { 1, 30 }; // 30 FPS
 codec_context->gop_size = 10;

 av_dict_set(&muxer_opts, "movflags", "faststart", 0);

 //Unecessary? Since the params are copied anyways
 video_stream->time_base = codec_context->time_base;

 //Try to open codec after changes
 //copy codec_context params to videostream
 //and write headers to format_context
 if (avcodec_open2(codec_context, video_codec, NULL) < 0) { std::cerr << "Error: Could not open codec!" << std::endl; return -1; }
 if (avcodec_parameters_from_context(video_stream->codecpar, codec_context) < 0) { std::cerr << "Error: Could not copy params from context to stream!" << std::endl; return -1; };
 if (avformat_write_header(format_ctx, &muxer_opts) < 0) { std::cerr << "Error: Failed to write output file headers!" << std::endl; return -1; }
 return 0;
 }

 void processFrames() {
 while (!stop_processing) {
 FrameData* frameData = nullptr;
 {
 std::unique_lock lock(queue_mtx);
 queue_cv.wait(lock, [&]() { return !frame_queue.empty() || stop_processing; });

 if (stop_processing && frame_queue.empty())
 break;

 frameData = frame_queue.front();
 frame_queue.pop();
 }

 if (frameData) {
 encodeAndWriteFrame(frameData);
 AVFrame* frame = frameData->frame;
 av_frame_free(&frame); // Free the processed frame
 delete frameData;
 }
 }
 }

 void encodeAndWriteFrame(FrameData* frameData) {

 // Validation
 if (!frameData->frame) { std::cerr << "Error: Frame was null! " << std::endl; return; }
 if (frameData->frame->format != codec_context->pix_fmt) { std::cerr << "Error: Frame format mismatch!" << std::endl; return; }
 if ( av_frame_get_buffer(frameData->frame, 0) < 0) { std::cerr << "Error allocating frame buffer: " << std::endl; return; }
 if (!codec_context) return;

 AVPacket* pkt = av_packet_alloc();
 if (!pkt) { std::cerr << "Error: Failed to allocate AVPacket" << std::endl; system("pause"); }

 int ret = avcodec_send_frame(codec_context, frameData->frame);
 if (ret < 0) { 
 std::cerr << "Error receiving packet from codec: " << ret << std::endl;
 delete frameData;
 av_packet_free(&pkt); return; 
 }

 while (ret >= 0) {
 ret = avcodec_receive_packet(codec_context, pkt);

 //Error checks
 if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) { break; }
 else if (ret < 0) { std::cerr << "Error receiving packet from codec: " << ret << std::endl; av_packet_free(&pkt); return; }
 if (!video_stream) { std::cerr << "Error: video stream is null!" << std::endl; av_packet_free(&pkt); return; }
 
 int64_t frame_duration = codec_context->time_base.den / codec_context->time_base.num;
 pkt->stream_index = video_stream->index;
 pkt->duration = frame_duration;
 pkt->pts = frameData->frame_index * frame_duration;

 int write_ret = av_interleaved_write_frame(format_ctx, pkt);
 if (write_ret < 0) { std::cerr << "Error: failed to write a frame! " << write_ret << std::endl;}

 av_packet_unref(pkt);
 }

 av_packet_free(&pkt);

 }

protected:
 virtual void onFrameEnded(veEvent event) override {
 // Get the image data from vulkan
 VkExtent2D extent = getWindowPointer()->getExtent();
 uint32_t imageSize = extent.width * extent.height * 4;
 VkImage image = getEnginePointer()->getRenderer()->getSwapChainImage();

 uint8_t *dataImage = new uint8_t[imageSize];
 
 vh::vhBufCopySwapChainImageToHost(getEnginePointer()->getRenderer()->getDevice(),
 getEnginePointer()->getRenderer()->getVmaAllocator(),
 getEnginePointer()->getRenderer()->getGraphicsQueue(),
 getEnginePointer()->getRenderer()->getCommandPool(),
 image, VK_FORMAT_R8G8B8A8_UNORM,
 VK_IMAGE_ASPECT_COLOR_BIT, VK_IMAGE_LAYOUT_PRESENT_SRC_KHR,
 dataImage, extent.width, extent.height, imageSize);
 
 // Create AVFrame for the converted image data
 AVFrame* frame = av_frame_alloc();
 if (!frame) { std::cout << "Could not allocate memory for frame!" << std::endl; return; }

 frame->format = AV_PIX_FMT_YUV420P;
 frame->width = extent.width;
 frame->height = extent.height;
 if (av_frame_get_buffer(frame, 0) < 0) { std::cerr << "Failed to allocate frame buffer! " << std::endl; return;} ;

 // Prepare context for converting from RGBA to YUV420P
 SwsContext* sws_ctx = sws_getContext(
 extent.width, extent.height, AV_PIX_FMT_RGBA,
 extent.width, extent.height, AV_PIX_FMT_YUV420P,
 SWS_BILINEAR, nullptr, nullptr, nullptr);

 // Convert the vulkan image data to AVFrame
 uint8_t* src_data[1] = { dataImage };
 int src_linesize[1] = { extent.width * 4 };
 int scale_ret = sws_scale(sws_ctx, src_data, src_linesize, 0, extent.height,
 frame->data, frame->linesize);

 if (scale_ret <= 0) { std::cerr << "Failed to scale the image to frame" << std::endl; return; }

 sws_freeContext(sws_ctx);
 delete[] dataImage;

 // Add frame to the queue
 {
 std::lock_guard lock(queue_mtx);

 FrameData* frameData = new FrameData;
 frameData->frame = frame;
 frameData->frame_index = frame_index;
 frame_queue.push(frameData);

 frame_index++;
 }

 // Notify processing thread
 queue_cv.notify_one();
 }

public:
 EventListenerVideoCapture(std::string name) : VEEventListenerGLFW(name) {
 //Prepare the video encoder
 int ret = prepare_video_encoder();
 if (ret < 0)
 {
 std::cerr << "Failed to prepare video encoder! " << std::endl;
 exit(-1);
 }
 else
 {
 // Start video processing thread
 video_processing_thread = std::thread(&EventListenerVideoCapture::processFrames, this);
 }
 }

 ~EventListenerVideoCapture() {
 // Stop video processing thread
 stop_processing = true;
 queue_cv.notify_one(); // Notify processing thread to stop

 if (video_processing_thread.joinable()) {
 video_processing_thread.join();
 }

 // Flush codec and close output file
 avcodec_send_frame(codec_context, nullptr);
 av_write_trailer(format_ctx);

 av_dict_free(&muxer_opts);
 avio_closep(&format_ctx->pb);
 avcodec_free_context(&codec_context);
 avformat_free_context(format_ctx);
 }
};

</bool>

I have tried changing the codec params, debugging and printing the videoframe data with no success.


Syndiquer tout le site
