Sur d’autres sites (3027)

• How does FFmpeg determine the “attached pic” and “timed thumbnails” dispositions of an MP4 track ?

  6 mars, par obskyr

  The Issue

  


  FFmpeg has a concept of “dispositions” – a property that describes the purpose of a stream in a media file. For example, here are the streams in a file I have lying around, with the dispositions emphasized :

  


    Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,&#xA;fltp, 251 kb/s <strong><em>(default)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Audio&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A),&#xA;yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr,&#xA;600 tbn <strong><em>(default) (attached pic) (timed thumbnails)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Video&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:2[0x3](und): Data: bin_data (text / 0x74786574)&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Text&#xA;&#xA;  Stream #0:3[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/&#xA;unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn <strong><em>(attached pic)</em></strong>

  


  However, if I make any modification to this file’s chapter markers using the C++ library MP4v2 (even just re-saving the existing ones : auto f = MP4Modify("test.m4a"); MP4Chapter_t* chapterList; uint32_t chapterCount; MP4GetChapters(f, &chapterList, &chapterCount); MP4SetChapters(f, chapterList, chapterCount); MP4Close(f);), some of these dispositions are removed :

  


    Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,&#xA;fltp, 251 kb/s <strong><em>(default)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Audio&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A),&#xA;yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr,&#xA;600 tbn <strong><em>(default)</em></strong> <kbd>← “attached pic” and “timed thumbnails” removed!</kbd>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Video&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:2[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/&#xA;unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn <strong><em>(attached pic)</em></strong>&#xA;&#xA;  Stream #0:3[0x4](und): Data: bin_data (text / 0x74786574)&#xA;  <kbd>This stream was moved to the end, but that’s intended behavior. It contains chapter titles, and we just edited the chapters.</kbd>&#xA;      Metadata:&#xA;        creation_time   : 2025-03-05T09:56:31.000000Z

  


  It also renders the file unplayable in MPC-HC (but not in VLC !), which is apparently a bug in MP4v2. I’m currently investigating that bug to report and potentially fix it, but that’s a separate issue – in my journey there, I’m wracking my brain trying to understand what it is that MP4v2 changes to make FFmpeg stop reporting the “attached pic” and “timed thumbnails” dispositions. I’ve explored the before-and-afters in MP4 Box, and I can’t for the life of me find which atom it is that differs in a relevant way.

  


  (I’d love to share the files, but unfortunately the contents are under copyright – if anyone knows of a way to remove the audio from an MP4 file without changing anything else, let me know and I’ll upload dummied-out versions. Without them, I can’t really ask about the issue directly. I can at least show you the files’ respective atom trees, but I’m not sure how relevant that is.)

  


  The Question

  


  I thought I’d read FFmpeg’s source code to find out how it determines dispositions for MP4 streams, but of course, FFmpeg is very complex. Could someone who’s more familiar with C and/or FFmpeg’s codebase help me sleuth out how FFmpeg determines dispositions for MP4 files (in particular, “attached pic” and “timed thumbnails”) ?

  


  Some Thoughts…

  


  

  • I figure searching for “attached_pic” might be a good start ?

  


  • Could the MP4 muxer movenc.c be helpful ?

  


  • I’d imagine what we’d really like to look at is the MP4 demuxing process, as it’s during demuxing that FFmpeg determines dispositions from the data in the file. After poring over the code for hours, however, I’ve been utterly unable to find where that happens.

  


  


• How does FFmpeg determine the dispositions of an MP4 track ?

  5 mars, par obskyr

  The Issue

  


  FFmpeg has a concept of “dispositions” – a property that describes the purpose of a stream in a media file. For example, here are the streams in a file I have lying around, with the dispositions emphasized :

  


    Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,&#xA;fltp, 251 kb/s <strong><em>(default)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Audio&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A),&#xA;yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr,&#xA;600 tbn <strong><em>(default) (attached pic) (timed thumbnails)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Video&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:2[0x3](und): Data: bin_data (text / 0x74786574)&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Text&#xA;&#xA;  Stream #0:3[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/&#xA;unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn <strong><em>(attached pic)</em></strong>

  


  However, if I make any modification to this file’s chapter markers using the C++ library MP4v2 (even just re-saving the existing ones : auto f = MP4Modify("test.m4a"); MP4Chapter_t* chapterList; uint32_t chapterCount; MP4GetChapters(f, &chapterList, &chapterCount); MP4SetChapters(f, chapterList, chapterCount); MP4Close(f);), some of these dispositions are removed :

  


    Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,&#xA;fltp, 251 kb/s <strong><em>(default)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Audio&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A),&#xA;yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr,&#xA;600 tbn <strong><em>(default)</em></strong> <kbd>← “attached pic” and “timed thumbnails” removed!</kbd>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Video&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:2[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/&#xA;unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn <strong><em>(attached pic)</em></strong>&#xA;&#xA;  Stream #0:3[0x4](und): Data: bin_data (text / 0x74786574)&#xA;  <kbd>This stream was moved to the end, but that’s intended behavior. It contains chapter titles, and we just edited the chapters.</kbd>&#xA;      Metadata:&#xA;        creation_time   : 2025-03-05T09:56:31.000000Z

  


  It also renders the file unplayable in MPC-HC (but not in VLC !), which is apparently a bug in MP4v2. I’m currently investigating that bug to report and potentially fix it, but that’s a separate issue – in my journey there, I’m wracking my brain trying to understand what it is that MP4v2 changes to make FFmpeg stop reporting the “attached pic” and “timed thumbnails” dispositions. I’ve explored the before-and-afters in MP4 Box, and I can’t for the life of me find which atom it is that differs in a relevant way.

  


  (I’d love to share the files, but unfortunately the contents are under copyright – if anyone knows of a way to remove the audio from an MP4 file without changing anything else, let me know and I’ll upload dummied-out versions. Without them, I can’t really ask about the issue directly. I can at least show you the files’ respective atom trees, but I’m not sure how relevant that is.)

  


  The Question

  


  I thought I’d read FFmpeg’s source code to find out how it determines dispositions for MP4 streams, but of course, FFmpeg is very complex. Could someone who’s more familiar with C and/or FFmpeg’s codebase help me sleuth out how FFmpeg determines dispositions for MP4 files (in particular, “attached pic” and “timed thumbnails”) ?

  


  Some Thoughts…

  


  

  • I figure searching for “attached_pic” might be a good start ?

  


  • Could the MP4 muxer movenc.c be helpful ?

  


  • I’d imagine what we’d really like to look at is the MP4 demuxing process, as it’s during demuxing that FFmpeg determines dispositions from the data in the file. After poring over the code for hours, however, I’ve been utterly unable to find where that happens.

  


  


• FFmpeg unexpected behavior using -loop flag

  7 décembre 2020, par all jazz

  Dear hackers of the world !

  


  I've been trying to use the beloved FFmpeg library to create a video from an image loop and audio using the famous Docker FFmpeg image, but it has been driving me crazy not producing the expected results (the results that I get when I run the ffmpeg command with the equivalent version on my Macbook).

  


  Here is the command :

  


  docker run -v $(pwd):$(pwd) -w $(pwd) jrottenberg/ffmpeg:4.3-alpine \
    -y \
    -stats \
    -loop 1 -i files/image.jpg \
    -i files/a.mp3 \
    -c:v libx265 -pix_fmt yuv420p10 \
    -c:a aac \
    -movflags +faststart \
    -shortest \
    -f mp4 test.mp4


  


  It should create a test.mp4 with the provided audio and image that is ready to be uploaded to the unfortunate Youtube.

  


  When I do this, the video seems to be lacking moov atoms (if I try to analyse it). Strangely enough, if I run this two times using the Docker image (overriding the same file), the video file will magically start to work.

  


  I also tried using different ffmpeg os images and versions. It seems that ffmpeg docummentation and code repo could also benefit from some care and love.

  


  What else I could do to get this fixed ?

  


  Here is the output from the console :

  


          -y \
        -stats \
        -loop 1 -i files/image.jpg \
        -i files/a.mp3 \
        -c:v libx265 -pix_fmt yuv420p10 \
        -c:a aac \
        -movflags +faststart \
        -shortest \
        -f mp4 test30.mp4
ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 6.4.0 (Alpine 6.4.0)
  configuration: --disable-debug --disable-doc --disable-ffplay --enable-shared --enable-avresample --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-gpl --enable-libass --enable-fontconfig --enable-libfreetype --enable-libvidstab --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libxcb --enable-libx265 --enable-libxvid --enable-libx264 --enable-nonfree --enable-openssl --enable-libfdk_aac --enable-postproc --enable-small --enable-version3 --enable-libbluray --enable-libzmq --extra-libs=-ldl --prefix=/opt/ffmpeg --enable-libopenjpeg --enable-libkvazaar --enable-libaom --extra-libs=-lpthread --enable-libsrt --enable-libaribb24 --extra-cflags=-I/opt/ffmpeg/include --extra-ldflags=-L/opt/ffmpeg/lib
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  7.100 /  5.  7.100
  libswresample   3.  7.100 /  3.  7.100
  libpostproc    55.  7.100 / 55.  7.100
Input #0, image2, from 'files/image.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 34300 kb/s
    Stream #0:0: Video: mjpeg, gray(bt470bg/unknown/unknown), 500x500 [SAR 240:240 DAR 1:1], 25 fps, 25 tbr, 25 tbn, 25 tbc
Input #1, mp3, from 'files/a.mp3':
  Metadata:
    title           : Visions
    artist          : Hattori Hanzo
    album           : Visions
    encoded_by      : Fission
    encoder         : Lavf58.45.100
    TLEN            : 16039
    track           : 1
  Duration: 00:00:16.04, start: 0.000000, bitrate: 199 kb/s
    Stream #1:0: Audio: mp3, 44100 Hz, stereo, fltp, 191 kb/s
    Stream #1:1: Video: mjpeg, yuvj444p(pc, bt470bg/unknown/unknown), 500x500 [SAR 300:300 DAR 1:1], 90k tbr, 90k tbn, 90k tbc (attached pic)
    Metadata:
      comment         : Other
Stream mapping:
  Stream #0:0 -> #0:0 (mjpeg (native) -> hevc (libx265))
  Stream #1:0 -> #0:1 (mp3 (mp3float) -> aac (native))
Press [q] to stop, [?] for help
x265 [info]: HEVC encoder version 3.1.1+1-04b37fdfd2dc
x265 [info]: build info [Linux][GCC 6.4.0][64 bit] 10bit
x265 [info]: using cpu capabilities: MMX2 SSE2Fast LZCNT SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
x265 [info]: Main 10 profile, Level-3 (Main tier)
x265 [info]: Thread pool created using 8 threads
x265 [info]: Slices                              : 1
x265 [info]: frame threads / pool features       : 3 / wpp(8 rows)
x265 [warning]: Source height < 720p; disabling lookahead-slices
x265 [info]: Coding QT: max CU size, min CU size : 64 / 8
x265 [info]: Residual QT: max TU size, max depth : 32 / 1 inter / 1 intra
x265 [info]: ME / range / subpel / merge         : hex / 57 / 2 / 3
x265 [info]: Keyframe min / max / scenecut / bias: 25 / 250 / 40 / 5.00
x265 [info]: Lookahead / bframes / badapt        : 20 / 4 / 2
x265 [info]: b-pyramid / weightp / weightb       : 1 / 1 / 0
x265 [info]: References / ref-limit  cu / depth  : 3 / off / on
x265 [info]: AQ: mode / str / qg-size / cu-tree  : 2 / 1.0 / 32 / 1
x265 [info]: Rate Control / qCompress            : CRF-28.0 / 0.60
x265 [info]: tools: rd=3 psy-rd=2.00 early-skip rskip signhide tmvp b-intra
x265 [info]: tools: strong-intra-smoothing deblock sao
Output #0, mp4, to 'test30.mp4':
  Metadata:
    encoder         : Lavf58.45.100
    Stream #0:0: Video: hevc (libx265) (hev1 / 0x31766568), yuv420p10le(progressive), 500x500 [SAR 1:1 DAR 1:1], q=-1--1, 25 fps, 12800 tbn, 25 tbc
    Metadata:
      encoder         : Lavc58.91.100 libx265
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
    Stream #0:1: Audio: aac (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc58.91.100 aac
frame=   52 fps=0.0 q=33.0 size=       0kB time=00:00:00.80 bitrate=   0.4kbits/frame=  120 fps=119 q=33.0 size=       0kB time=00:00:03.52 bitrate=   0.1kbits/frame=  190 fps=125 q=33.0 size=       0kB time=00:00:06.33 bitrate=   0.1kbits/frame=  257 fps=127 q=33.0 size=       0kB time=00:00:09.00 bitrate=   0.0kbits/frame=  303 fps=120 q=35.0 size=     256kB time=00:00:10.86 bitrate= 193.0kbits/frame=  373 fps=123 q=36.0 size=     256kB time=00:00:13.65 bitrate= 153.6kbits/[mp4 @ 0x55d481bc6980] Starting second pass: moving the moov atom to the beginning of the file
frame=  432 fps=121 q=36.0 Lsize=     379kB time=00:00:17.16 bitrate= 180.8kbits/s speed= 4.8x
video:107kB audio:255kB subtitle:0kB other streams:0kB global headers:2kB muxing overhead: 4.667185%
x265 [info]: frame I:      2, Avg QP:23.55  kb/s: 8634.80
x265 [info]: frame P:    147, Avg QP:33.00  kb/s: 13.49
x265 [info]: frame B:    283, Avg QP:35.71  kb/s: 8.06
x265 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x265 [info]: consecutive B-frames: 34.2% 10.1% 20.8% 1.3% 33.6%

encoded 432 frames in 3.56s (121.32 fps), 49.84 kb/s, Avg QP:34.73
[aac @ 0x55d481b23ac0] Qavg: 563.168```