Recherche avancée

Sur d’autres sites (1928)

• Methods For Retaining State

  26 décembre 2011, par Multimedia Mike — General, evernote, organization

  I jump around between projects. A lot. Over the years, I have employed various methods for retaining state or context as I switch to a different project. Here’s a quick survey and a general classification of their effectiveness.

  Good

  • Evernote : This is a cloud-based note-taking service that has a web client, Mac and Windows clients, and clients for just about ever mobile platform out there. I have an account and access it via the web interface as as the Windows, iOS, and Android clients. I really like it.

  
  
  

  Okay

  • Series of text files : I have been doing this for a very long time. I have many little note-filled directories here and there that are consistently migrated to new machines but generally forgotten about. This isn’t a terrible method but can be unwieldy when you work on lots of different machines. I’m still tracking down all these directories and importing them into Evernote.

  Bad

  • Layout of desktop windows : I have a habit of working on one project in a set of windows on one desktop space and another project in a second set of windows in another space, etc. Oh, this makes me shudder just thinking about it, mostly because of living in constant fear of a power failure or some other inadvertent reset (darn you, default config’d Windows Update) that wipes the state clean (sure, all of the work might have been saved, but I was relying on those windows to be set up in just the right manner to remind me of all the things I was working on). These days, I force myself to reboot at least once a week so I can’t get too deep into this habit. When it’s time to change projects, I write up exactly what I was doing and where I left off and stick it in Evernote.
  • Open browser windows : I guess it’s common to have many, many tabs open in one’s web browser in this day and age. Like many, I use open tabs as a stack of items to read. The state problem comes when a few of the open tabs represent TODO items. Then I start living in fear that the browser might crash or be restarted in an unexpected way and I struggle to recall what 3-5 important TODO items were that I had opened in separate tabs (on top of a stack of less important items). Again, I try to shut down the browser frequently in order to break this tendency. TODO items are better filed in Evernote.
  • Unsaved data in a text editor : Okay, this is just sloppy on my part, shoving temporary data into a text editor window thinking it’s supremely ephemeral. The problem comes when it’s linked to one of the many tasks on my desktop that might be bumped down a few priority levels ; when finally returning to the context-free data, I’m at a loss to explain what it’s for. Evernote gets it, once more, with a more thorough description of what was going on.
  • Email inbox : I make an effort to ensure that my email inbox has the fewest number of messages possible. Once things are dealt with, they get filed away elsewhere. This implies that things in my inbox require action. Some things have a habit of hanging around, though. Longer items now get described in better detail and filed away in Evernote.
  • Classic paper : Thanks to Derek in the comments for reminding me of this one. Paper is a reliable standby but it can get unwieldy when Post-It Notes litter your work area. Further, it can be problematic when you have multiple physical work areas.
  • Shell history : Another method I rely on entirely too often. This is when I count on a recipe of command line incantations living on in the history buffer of my Unix shell (generally Bash). What sequence of git commands allowed me to do XYZ ? Let’s check the shell history– I sure hope it’s still in there.

  Conclusion
  I guess what I’m trying to say here is that I really like Evernote. If you have similar troubles with retaining state, try it out. I hear there are many other services similar to it with slightly varying feature sets (people rave about Microsoft OneNote). So there are plenty of options and something out there is surely a fit.

  Evernote has a free tier and a premium tier. For my meager note-taking needs, I don’t come anywhere close to the free tier’s limit but I decided to pay for a premium subscription simply because I feel like I derive so much value from the service.

  One downside, however, is that I seem to be doing a lot less blogging since I got on Evernote earlier this year (though it is where I author most of these posts nowadays ; I especially like that I have a notebook labeled “Posted” whose incrementing count reminds me that I am getting some stuff out there). I originally started this blog as a sort of technical journal in order to organize notes and projects in a central location. It’s strange to think that if Evernote existed in 2005, I might never have had a reason to start this blog.

• Improve ffmpeg x11grab screen capture performance

  10 janvier 2020, par Toby Eggitt

  I have been doing screen-only (no sound) capture using ffmpeg with libx264 for the encoding quite successfully on an old machine built around a Core2 Quad Q6600 processor. I now need to include audio in this, but the fans on this ancient machine are too loud. So, I found a fanless motherboard (https://www.asrock.com/mb/Intel/J5005-ITX/index.asp) that has an Intel Pentium Silver J5005 processor and decided to use this instead. The CPU’s benchmarks put it in a similar bracket to the Q6600, and the general performance seems to be significantly better, presumably at least in part because it’s now using DDR4 memory that’s faster access.

  However, the machine fails horribly at the screen capture. It’s missing frames all over the place ; I actually end up with video that’s missing almost half the frames, and plays back at about double speed. Also, any audio is just messed up so badly I can hardly think how to describe it, best I can come up with is that I get perhaps a quarter second of sound then a few seconds pause (the video meanwhile is actually still playing back, albeit with no sense of time).

  Some things occur to me that might be the cause, or cure, of my troubles, some of which I might be able to fix, others not so much. What other things should I try ? (I’d prefer to avoid simply throwing money at the issue with random ideas that are baseless !)

  1) perhaps the CPU lacks some "extensions" to the instruction set (I recall years ago some CPUs gaining MMX extensions") so that the CPU is fast at mundane computing but sucks at video encoding.

  2) perhaps the fact that the old machine had a dedicated graphics card, while this new one is sharing main memory with the graphics system means that reading the screen pixels is much slower.

  3) perhaps the fact that this new machine has a single DDR4 memory stick in it means that I’m forcing all the memory reads and writes for the computations through the same memory as is holding the screen, and that’s too much (implying that adding an additional memory stick might jus possibly help ?)

  4) perhaps there’s some bios setting that would allow more efficient sharing of video memory ?

  5) my favorite, perhaps there’s a better compression library that I could use to get decent quality screen capture with much less CPU usage.

  I should also note that I have tried this with -threads 0, and the CPU usage hovers between 100% and 200% ; around 100% when the screen is static, and rising as I move windows around and otherwise create more output.

  6) the motherboard claims to have some kind of hardware video encoder built into it. I haven’t paid this any attention to this point, as I assumed it was for the purpose of taking HDMI input and encoding it, but maybe there’s a way to use this, if so, what libraries might I need to get ffmpeg to do this.

  Edits :

  • This is an off the shelf ffmpeg. I’m certainly willing to try building it myself if I have some idea what I should do different.
  • The motherboard claims to have hardware encoders, but I’m struggling to find out what they are (seems like it’s an Intel chip called "UHD Graphics 605" but nothing I can find suggests ffmpeg can work with that)

  • command line right now has been (without audio) :

    ffmpeg  -video_size 1280x720   -f x11grab  -i ${DISPLAY}+100,100  -vcodec libx264  -f alsa -i pulse -acodec ac3 -threads 0  ./video$(date +%F-%H-%M-%S).mp4

  Log from a short recording session is :

  ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
  
    built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  
    configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  
    libavutil      55. 78.100 / 55. 78.100
  
    libavcodec     57.107.100 / 57.107.100
  
    libavformat    57. 83.100 / 57. 83.100
  
    libavdevice    57. 10.100 / 57. 10.100
  
    libavfilter     6.107.100 /  6.107.100
  
    libavresample   3.  7.  0 /  3.  7.  0
  
    libswscale      4.  8.100 /  4.  8.100
  
    libswresample   2.  9.100 /  2.  9.100
  
    libpostproc    54.  7.100 / 54.  7.100
  
  [x11grab @ 0x561a723e5ac0] Stream #0: not enough frames to estimate rate; consider increasing probesize
  
  Input #0, x11grab, from ':0+100,100':
  
    Duration: N/A, start: 1578693116.465807, bitrate: N/A
  
      Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1280x720, 29.97 fps, 1000k tbr, 1000k tbn, 1000k tbc
  
  Unknown decoder 'libx264'
  
  simon@studio:~$ ffmpeg  -video_size 1280x720   -f x11grab  -i ${DISPLAY}+100,100  -vcodec libx264 -threads 0  ./video$(date +%F-%H-%M-%S).mp4
  
  ffmpeg version 3.4.6-0ubuntu0.18.04.1 Copyright (c) 2000-2019 the FFmpeg developers
  
    built with gcc 7 (Ubuntu 7.3.0-16ubuntu3)
  
    configuration: --prefix=/usr --extra-version=0ubuntu0.18.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  
    libavutil      55. 78.100 / 55. 78.100
  
    libavcodec     57.107.100 / 57.107.100
  
    libavformat    57. 83.100 / 57. 83.100
  
    libavdevice    57. 10.100 / 57. 10.100
  
    libavfilter     6.107.100 /  6.107.100
  
    libavresample   3.  7.  0 /  3.  7.  0
  
    libswscale      4.  8.100 /  4.  8.100
  
    libswresample   2.  9.100 /  2.  9.100
  
    libpostproc    54.  7.100 / 54.  7.100
  
  [x11grab @ 0x558225bc29a0] Stream #0: not enough frames to estimate rate; consider increasing probesize
  
  Input #0, x11grab, from ':0+100,100':
  
    Duration: N/A, start: 1578693132.513351, bitrate: N/A
  
      Stream #0:0: Video: rawvideo (BGR[0] / 0x524742), bgr0, 1280x720, 29.97 fps, 1000k tbr, 1000k tbn, 1000k tbc
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
  
  Press [q] to stop, [?] for help
  
  [libx264 @ 0x558225bcd360] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
  
  [libx264 @ 0x558225bcd360] profile High 4:4:4 Predictive, level 3.1, 4:4:4 8-bit
  
  [libx264 @ 0x558225bcd360] 264 - core 152 r2854 e9a5903 - H.264/MPEG-4 AVC codec - Copyleft 2003-2017 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x1:0x111 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=4 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
  
  Output #0, mp4, to './video2020-01-10-14-52-12.mp4':
  
    Metadata:
  
      encoder         : Lavf57.83.100
  
      Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv444p, 1280x720, q=-1--1, 29.97 fps, 30k tbn, 29.97 tbc
  
      Metadata:
  
        encoder         : Lavc57.107.100 libx264
  
      Side data:
  
        cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: -1
  
  Past duration 0.806847 too large     256kB time=00:00:00.43 bitrate=4835.3kbits/s dup=16 drop=0 speed=0.207x    
  
  frame=  371 fps= 29 q=-1.0 Lsize=     639kB time=00:00:12.27 bitrate= 426.6kbits/s dup=16 drop=14 speed=0.971x    
  
  video:634kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.813096%
  
  [libx264 @ 0x558225bcd360] frame I:2     Avg QP:18.16  size:221502
  
  [libx264 @ 0x558225bcd360] frame P:93    Avg QP:14.97  size:  2007
  
  [libx264 @ 0x558225bcd360] frame B:276   Avg QP:20.13  size:    69
  
  [libx264 @ 0x558225bcd360] consecutive B-frames:  0.8%  0.0%  0.0% 99.2%
  
  [libx264 @ 0x558225bcd360] mb I  I16..4: 44.6%  0.0% 55.4%
  
  [libx264 @ 0x558225bcd360] mb P  I16..4:  0.2%  0.0%  0.3%  P16..4:  0.7%  0.1%  0.1%  0.0%  0.0%    skip:98.5%
  
  [libx264 @ 0x558225bcd360] mb B  I16..4:  0.0%  0.0%  0.0%  B16..8:  1.0%  0.0%  0.0%  direct: 0.0%  skip:99.0%  L0:50.9% L1:49.0% BI: 0.1%
  
  [libx264 @ 0x558225bcd360] coded y,u,v intra: 41.3% 37.5% 37.4% inter: 0.1% 0.0% 0.0%
  
  [libx264 @ 0x558225bcd360] i16 v,h,dc,p: 58% 41%  1%  0%
  
  [libx264 @ 0x558225bcd360] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 33% 30% 14%  2%  4%  4%  5%  3%  5%
  
  [libx264 @ 0x558225bcd360] Weighted P-Frames: Y:0.0% UV:0.0%
  
  [libx264 @ 0x558225bcd360] ref P L0: 59.2%  8.8% 25.5%  6.5%
  
  [libx264 @ 0x558225bcd360] ref B L0: 59.4% 39.0%  1.6%
  
  [libx264 @ 0x558225bcd360] ref B L1: 96.5%  3.5%
  
  [libx264 @ 0x558225bcd360] kb/s:419.29
  
  Exiting normally, received signal 2.

• Video Frame-by-Frame Deraining with MFDNet in Python

  28 octobre 2024, par JimmyHu

  As this CodeReview question mentioned, I am trying to modify the code to process frame-by-frame rain streaks removal in a video. FFmpeg package is used in this code.

  


  import argparse
import os
import time

import cv2
import ffmpeg
import numpy as np
import torch
from skimage import img_as_ubyte
from torch.utils.data import DataLoader
from tqdm import tqdm

import utils
from data_RGB import get_test_data
from MFDNet import HPCNet as mfdnet


def process_video_frame_by_frame(input_file, output_file, model_restoration):
    """
    Decodes a video frame by frame, processes each frame,
    and re-encodes to a new video.

    Args:
        input_file: Path to the input video file.
        output_file: Path to the output video file.
    """
    try:
        # Probe for video information
        probe = ffmpeg.probe(input_file)
        video_stream = next((stream for stream in probe['streams'] if stream['codec_type'] == 'video'), None)
        width = int(video_stream['width'])
        height = int(video_stream['height'])

        # Input
        process1 = (
            ffmpeg
            .input(input_file)
            .output('pipe:', format='rawvideo', pix_fmt='rgb24')
            .run_async(pipe_stdout=True)
        )

        # Output
        process2 = (
            ffmpeg
            .input('pipe:', format='rawvideo', pix_fmt='rgb24', s='{}x{}'.format(width, height))
            .output(output_file, vcodec='libx264', pix_fmt='yuv420p')
            .overwrite_output()
            .run_async(pipe_stdin=True)
        )

        # Process frame (deraining processing)
        while in_bytes := process1.stdout.read(width * height * 3):
            in_frame = torch.frombuffer(in_bytes, dtype=torch.uint8).float().reshape((1, 3, width, height))
            restored = model_restoration(torch.div(in_frame, 255).to(device='cuda'))
            restored = torch.clamp(restored[0], 0, 1)
            restored = restored.cpu().detach().numpy()
            restored *= 255
            out_frame = restored
            np.reshape(out_frame, (3, width, height))

            # Encode and write the frame
            process2.stdin.write(
                out_frame
                .astype(np.uint8)
                .tobytes()
            )
            
        # Close streams
        process1.stdout.close()
        process2.stdin.close()
        process1.wait()
        process2.wait()

    except ffmpeg.Error as e:
        print('stdout:', e.stdout.decode('utf8'))
        print('stderr:', e.stderr.decode('utf8'))

if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Image Deraining using MPRNet')

    parser.add_argument('--weights', default='./checkpoints/checkpoints_mfd.pth', type=str,
                        help='Path to weights')
    parser.add_argument('--gpus', default='0', type=str, help='CUDA_VISIBLE_DEVICES')

    args = parser.parse_args()

    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
    os.environ["CUDA_VISIBLE_DEVICES"] = args.gpus

    model_restoration = mfdnet()
    utils.load_checkpoint(model_restoration, args.weights)
    print("===>Testing using weights: ", args.weights)

    model_restoration.eval().cuda()
    
    input_video = "Input_video.mp4"
    output_video = 'output_video.mp4'

    process_video_frame_by_frame(input_video, output_video, model_restoration)


  


  Let's focus on the while loop part :

  


  The version of the code snippet above can be executed without error. In the next step, I am trying to follow 301_Moved_Permanently's answer to make the usage of torch.save. Therefore, the contents of while loop comes as the following code :

  


          # Process frame (deraining processing)
        while in_bytes := process1.stdout.read(width * height * 3):
            in_frame = torch.frombuffer(in_bytes, dtype=torch.uint8).float().reshape((1, 3, width, height))
            restored = model_restoration(torch.div(in_frame, 255).to(device='cuda'))
            restored = torch.clamp(restored[0], 0, 1)
            out_frame = torch.mul(restored.cpu().detach(), 255).reshape(3, width, height).byte()
            torch.save(out_frame, process2.stdin)


  


  Out of memory error happened with the following message :

  


  


  torch.OutOfMemoryError : CUDA out of memory. Tried to allocate 676.00 MiB. GPU 0 has a total capacity of 23.99 GiB of which 0 bytes is free. Of the allocated memory 84.09 GiB is allocated by PyTorch, and 1.21 GiB is reserved by PyTorch but unallocated.

  


  


  To diagnostics the error, I removed the last two lines of code :

  


          # Process frame (deraining processing)
        while in_bytes := process1.stdout.read(width * height * 3):
            in_frame = torch.frombuffer(in_bytes, dtype=torch.uint8).float().reshape((1, 3, width, height))
            restored = model_restoration(torch.div(in_frame, 255).to(device='cuda'))
            restored = torch.clamp(restored[0], 0, 1)


  


  The out of memory error still happened. This is weird to me. My understanding of the executable version code, the line restored = restored.cpu().detach().numpy() is to transfer the restored data in GPU memory to main memory and then convert it to numpy format. Why I remove this line of code then out of memory error happened ?

  


  The hardware and software specification I used is as follows :

  


  


  • CPU : 12th Gen Intel(R) Core(TM) i9-12900K 3.20 GHz

    


  


  • RAM : 128 GB (128 GB usable)

    


  


  • Graphic card : NVIDIA GeForce RTX 4090

    


  


  • OS : Windows 11 Pro 22H2, OS build : 22621.4317

    


  


  • Pytorch version :

    


    > python -c "import torch; print(torch.__version__)"
2.5.0+cu124