Sur d’autres sites (4894)

• How to improve web camera streaming latency to v4l2loopback device with ffmpeg ?

  11 mars, par Made by Moses

  I'm trying to stream my iPhone camera to my PC on LAN.

  


  What I've done :

  


  


  1. HTTP server with html page and streaming script :

     


     I use WebSockets here and maybe WebRTC is better choice but it seems like network latency is good enough

     


  


  


  async function beginCameraStream() {
  const mediaStream = await navigator.mediaDevices.getUserMedia({
    video: { facingMode: "user" },
  });

  websocket = new WebSocket(SERVER_URL);

  websocket.onopen = () => {
    console.log("WS connected");

    const options = { mimeType: "video/mp4", videoBitsPerSecond: 1_000_000 };
    mediaRecorder = new MediaRecorder(mediaStream, options);

    mediaRecorder.ondataavailable = async (event) => {
      // to measure latency I prepend timestamp to the actual video bytes chunk
      const timestamp = Date.now();
      const timestampBuffer = new ArrayBuffer(8);
      const dataView = new DataView(timestampBuffer);
      dataView.setBigUint64(0, BigInt(timestamp), true);
      const data = await event.data.bytes();

      const result = new Uint8Array(data.byteLength + 8);
      result.set(new Uint8Array(timestampBuffer), 0);
      result.set(data, 8);

      websocket.send(result);
    };

    mediaRecorder.start(100); // Collect 100ms chunks
  };
}


  


  


  2. Server to process video chunks

     


  


  


  import { serve } from "bun";
import { Readable } from "stream";

const V4L2LOOPBACK_DEVICE = "/dev/video10";

export const setupFFmpeg = (v4l2device) => {
  // prettier-ignore
  return spawn("ffmpeg", [
    '-i', 'pipe:0',           // Read from stdin
    '-pix_fmt', 'yuv420p',    // Pixel format
    '-r', '30',               // Target 30 fps
    '-f', 'v4l2',             // Output format
    v4l2device, // Output to v4l2loopback device
  ]);
};

export class FfmpegStream extends Readable {
  _read() {
    // This is called when the stream wants more data
    // We push data when we get chunks
  }
}

function main() {
  const ffmpeg = setupFFmpeg(V4L2LOOPBACK_DEVICE);
  serve({
    port: 8000,
    fetch(req, server) {
      if (server.upgrade(req)) {
        return; // Upgraded to WebSocket
      }
    },
    websocket: {
      open(ws) {
        console.log("Client connected");
        const stream = new FfmpegStream();
        stream.pipe(ffmpeg?.stdin);

        ws.data = {
          stream,
          received: 0,
        };
      },
      async message(ws, message) {
        const view = new DataView(message.buffer, 0, 8);
        const ts = Number(view.getBigUint64(0, true));
        ws.data.received += message.byteLength;
        const chunk = new Uint8Array(message.buffer, 8, message.byteLength - 8);

        ws.data.stream.push(chunk);

        console.log(
          [
            `latency: ${Date.now() - ts} ms`,
            `chunk: ${message.byteLength}`,
            `total: ${ws.data.received}`,
          ].join(" | "),
        );
      },
    },
  });
}

main();


  


  After I try to open the v4l2loopback device

  


  cvlc v4l2:///dev/video10


  


  picture is delayed for at least 1.5 sec which is unacceptable for my project.

  


  Thoughts :

  


  

  • Problem doesn't seems to be with network latency

  


  


  latency: 140 ms | chunk: 661 Bytes | total: 661 Bytes
latency: 206 ms | chunk: 16.76 KB | total: 17.41 KB
latency: 141 ms | chunk: 11.28 KB | total: 28.68 KB
latency: 141 ms | chunk: 13.05 KB | total: 41.74 KB
latency: 199 ms | chunk: 11.39 KB | total: 53.13 KB
latency: 141 ms | chunk: 16.94 KB | total: 70.07 KB
latency: 139 ms | chunk: 12.67 KB | total: 82.74 KB
latency: 142 ms | chunk: 13.14 KB | total: 95.88 KB


  


   150ms is actually too much for 15KB on LAN but there can some issue with my router

  


  

  • As far as I can tell it neither ties to ffmpeg throughput :

  


  


  Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'pipe:0':
  Metadata:
    major_brand     : iso5
    minor_version   : 1
    compatible_brands: isomiso5hlsf
    creation_time   : 2025-03-09T17:16:49.000000Z
  Duration: 00:00:01.38, start:
0.000000, bitrate: N/A
    Stream #0:0(und): Video: h264 (Baseline) (avc1 / 0x31637661), yuvj420p(pc), 1280x720, 4012 kb/s, 57.14 fps, 29.83 tbr, 600 tbn, 1200 tbc (default)
    Metadata:
      rotate          : 90
      creation_time   : 2025-03-09T17:16:49.000000Z
      handler_name    : Core Media Video
    Side data:
      displaymatrix: rotation of -90.00 degrees

Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))

[swscaler @ 0x55d8d0b83100] deprecated pixel format used, make sure you did set range correctly

Output #0, video4linux2,v4l2, to '/dev/video10':
  Metadata:
    major_brand     : iso5
    minor_version   : 1
    compatible_brands: isomiso5hlsf
    encoder         : Lavf58.45.100

Stream #0:0(und): Video: rawvideo (I420 / 0x30323449), yuv420p, 720x1280, q=2-31, 663552 kb/s, 60 fps, 60 tbn, 60 tbc (default)
    Metadata:
      encoder         : Lavc58.91.100 rawvideo
      creation_time   : 2025-03-09T17:16:49.000000Z
      handler_name    : Core Media Video
    Side data:
      displaymatrix: rotation of -0.00 degrees

frame=   99 fps=0.0 q=-0.0 size=N/A time=00:00:01.65 bitrate=N/A dup=50 drop=0 speed=2.77x
frame=  137 fps=114 q=-0.0 size=N/A time=00:00:02.28 bitrate=N/A dup=69 drop=0 speed=1.89x
frame=  173 fps= 98 q=-0.0 size=N/A time=00:00:02.88 bitrate=N/A dup=87 drop=0 speed=1.63x
frame=  210 fps= 86 q=-0.0 size=N/A time=00:00:03.50 bitrate=N/A dup=105 drop=0 speed=1.44x
frame=  249 fps= 81 q=-0.0 size=N/A time=00:00:04.15 bitrate=N/A dup=125 drop=0 speed=1.36
frame=  279 fps= 78 q=-0.0 size=N/A time=00:00:04.65 bitrate=N/A dup=139 drop=0 speed=1.31x


  


  


  • I also tried to write the video stream directly to video.mp4 file and immediately open it with vlc but it only can be successfully opened after 1.5 sec.

    


  


  • I've tried to use OBS v4l2 input source instead of vlc but the latency is the same

    


  


  


  Update №1

  


  When i try to stream actual .mp4 file to ffmpeg it works almost immediately with 0.2sec delay to spin up the ffmpeg itself :

  


  cat video.mp4 | ffmpeg -re -i pipe:0 -pix_fmt yuv420p -f v4l2 /dev/video10 & ; sleep 0.2 && cvlc v4l2:///dev/video10


  


  So the problem is apparently with streaming process

  


• How does FFmpeg determine the “attached pic” and “timed thumbnails” dispositions of an MP4 track ?

  6 mars, par obskyr

  The Issue

  


  FFmpeg has a concept of “dispositions” – a property that describes the purpose of a stream in a media file. For example, here are the streams in a file I have lying around, with the dispositions emphasized :

  


    Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,&#xA;fltp, 251 kb/s <strong><em>(default)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Audio&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A),&#xA;yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr,&#xA;600 tbn <strong><em>(default) (attached pic) (timed thumbnails)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Video&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:2[0x3](und): Data: bin_data (text / 0x74786574)&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Text&#xA;&#xA;  Stream #0:3[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/&#xA;unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn <strong><em>(attached pic)</em></strong>

  


  However, if I make any modification to this file’s chapter markers using the C++ library MP4v2 (even just re-saving the existing ones : auto f = MP4Modify("test.m4a"); MP4Chapter_t* chapterList; uint32_t chapterCount; MP4GetChapters(f, &chapterList, &chapterCount); MP4SetChapters(f, chapterList, chapterCount); MP4Close(f);), some of these dispositions are removed :

  


    Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,&#xA;fltp, 251 kb/s <strong><em>(default)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Audio&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A),&#xA;yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr,&#xA;600 tbn <strong><em>(default)</em></strong> <kbd>← “attached pic” and “timed thumbnails” removed!</kbd>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Video&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:2[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/&#xA;unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn <strong><em>(attached pic)</em></strong>&#xA;&#xA;  Stream #0:3[0x4](und): Data: bin_data (text / 0x74786574)&#xA;  <kbd>This stream was moved to the end, but that’s intended behavior. It contains chapter titles, and we just edited the chapters.</kbd>&#xA;      Metadata:&#xA;        creation_time   : 2025-03-05T09:56:31.000000Z

  


  It also renders the file unplayable in MPC-HC (but not in VLC !), which is apparently a bug in MP4v2. I’m currently investigating that bug to report and potentially fix it, but that’s a separate issue – in my journey there, I’m wracking my brain trying to understand what it is that MP4v2 changes to make FFmpeg stop reporting the “attached pic” and “timed thumbnails” dispositions. I’ve explored the before-and-afters in MP4 Box, and I can’t for the life of me find which atom it is that differs in a relevant way.

  


  (I’d love to share the files, but unfortunately the contents are under copyright – if anyone knows of a way to remove the audio from an MP4 file without changing anything else, let me know and I’ll upload dummied-out versions. Without them, I can’t really ask about the issue directly. I can at least show you the files’ respective atom trees, but I’m not sure how relevant that is.)

  


  The Question

  


  I thought I’d read FFmpeg’s source code to find out how it determines dispositions for MP4 streams, but of course, FFmpeg is very complex. Could someone who’s more familiar with C and/or FFmpeg’s codebase help me sleuth out how FFmpeg determines dispositions for MP4 files (in particular, “attached pic” and “timed thumbnails”) ?

  


  Some Thoughts…

  


  

  • I figure searching for “attached_pic” might be a good start ?

  


  • Could the MP4 muxer movenc.c be helpful ?

  


  • I’d imagine what we’d really like to look at is the MP4 demuxing process, as it’s during demuxing that FFmpeg determines dispositions from the data in the file. After poring over the code for hours, however, I’ve been utterly unable to find where that happens.

  


  


• How does FFmpeg determine the dispositions of an MP4 track ?

  5 mars, par obskyr

  The Issue

  


  FFmpeg has a concept of “dispositions” – a property that describes the purpose of a stream in a media file. For example, here are the streams in a file I have lying around, with the dispositions emphasized :

  


    Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,&#xA;fltp, 251 kb/s <strong><em>(default)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Audio&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A),&#xA;yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr,&#xA;600 tbn <strong><em>(default) (attached pic) (timed thumbnails)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Video&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:2[0x3](und): Data: bin_data (text / 0x74786574)&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Text&#xA;&#xA;  Stream #0:3[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/&#xA;unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn <strong><em>(attached pic)</em></strong>

  


  However, if I make any modification to this file’s chapter markers using the C++ library MP4v2 (even just re-saving the existing ones : auto f = MP4Modify("test.m4a"); MP4Chapter_t* chapterList; uint32_t chapterCount; MP4GetChapters(f, &chapterList, &chapterCount); MP4SetChapters(f, chapterList, chapterCount); MP4Close(f);), some of these dispositions are removed :

  


    Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo,&#xA;fltp, 251 kb/s <strong><em>(default)</em></strong>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Audio&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:1[0x2](und): Video: mjpeg (Baseline) (jpeg / 0x6765706A),&#xA;yuvj420p(pc, bt470bg/unknown/unknown), 1024x1024, 0 kb/s, 0.0006 fps, 3.08 tbr,&#xA;600 tbn <strong><em>(default)</em></strong> <kbd>← “attached pic” and “timed thumbnails” removed!</kbd>&#xA;      Metadata:&#xA;        creation_time   : 2021-11-10T20:14:06.000000Z&#xA;        handler_name    : Core Media Video&#xA;        vendor_id       : [0][0][0][0]&#xA;&#xA;  Stream #0:2[0x0]: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/&#xA;unknown), 1024x1024 [SAR 144:144 DAR 1:1], 90k tbr, 90k tbn <strong><em>(attached pic)</em></strong>&#xA;&#xA;  Stream #0:3[0x4](und): Data: bin_data (text / 0x74786574)&#xA;  <kbd>This stream was moved to the end, but that’s intended behavior. It contains chapter titles, and we just edited the chapters.</kbd>&#xA;      Metadata:&#xA;        creation_time   : 2025-03-05T09:56:31.000000Z

  


  It also renders the file unplayable in MPC-HC (but not in VLC !), which is apparently a bug in MP4v2. I’m currently investigating that bug to report and potentially fix it, but that’s a separate issue – in my journey there, I’m wracking my brain trying to understand what it is that MP4v2 changes to make FFmpeg stop reporting the “attached pic” and “timed thumbnails” dispositions. I’ve explored the before-and-afters in MP4 Box, and I can’t for the life of me find which atom it is that differs in a relevant way.

  


  (I’d love to share the files, but unfortunately the contents are under copyright – if anyone knows of a way to remove the audio from an MP4 file without changing anything else, let me know and I’ll upload dummied-out versions. Without them, I can’t really ask about the issue directly. I can at least show you the files’ respective atom trees, but I’m not sure how relevant that is.)

  


  The Question

  


  I thought I’d read FFmpeg’s source code to find out how it determines dispositions for MP4 streams, but of course, FFmpeg is very complex. Could someone who’s more familiar with C and/or FFmpeg’s codebase help me sleuth out how FFmpeg determines dispositions for MP4 files (in particular, “attached pic” and “timed thumbnails”) ?

  


  Some Thoughts…

  


  

  • I figure searching for “attached_pic” might be a good start ?

  


  • Could the MP4 muxer movenc.c be helpful ?

  


  • I’d imagine what we’d really like to look at is the MP4 demuxing process, as it’s during demuxing that FFmpeg determines dispositions from the data in the file. After poring over the code for hours, however, I’ve been utterly unable to find where that happens.