Recherche avancée

Sur d’autres sites (6941)

• lavu/x86 : add FFT assembly

  10 avril 2021, par Lynne

  lavu/x86 : add FFT assembly
  This commit adds a pure x86 assembly SIMD version of the FFT in libavutil/tx.
  
  The design of this pure assembly FFT is pretty unconventional.
  On the lowest level, instead of splitting the complex numbers into
  
  real and imaginary parts, we keep complex numbers together but split
  
  them in terms of parity. This saves a number of shuffles in each transform,
  
  but more importantly, it splits each transform into two independent
  
  paths, which we process using separate registers in parallel.
  
  This allows us to keep all units saturated and lets us use all available
  
  registers to avoid dependencies.
  
  Moreover, it allows us to double the granularity of our per-load permutation,
  
  skipping many expensive lookups and allowing us to use just 4 loads per register,
  
  rather than 8, or in case FMA3 (and by extension, AVX2), use the vgatherdpd
  
  instruction, which is at least as fast as 4 separate loads on old hardware,
  
  and quite a bit faster on modern CPUs).
  Higher up, we go for a bottom-up construction of large transforms, foregoing
  
  the traditional per-transform call-return recursion chains. Instead, we always
  
  start at the bottom-most basis transform (in this case, a 32-point transform),
  
  and continue constructing larger and larger transforms until we return to the
  
  top-most transform.
  
  This way, we only touch the stack 3 times per a complete target transform :
  
  once for the 1/2 length transform and two times for the 1/4 length transform.
  The combination algorithm we use is a standard Split-Radix algorithm,
  
  as used in our C code. Although a version with less operations exists
  
  (Steven G. Johnson and Matteo Frigo's "A modified split-radix FFT with fewer
  
  arithmetic operations", IEEE Trans. Signal Process. 55 (1), 111–119 (2007),
  
  which is the one FFTW uses), it only has 2% less operations and requires at least 4x
  
  the binary code (due to it needing 4 different paths to do a single transform).
  
  That version also has other issues which prevent it from being implemented
  
  with SIMD code as efficiently, which makes it lose the marginal gains it offered,
  
  and cannot be performed bottom-up, requiring many recursive call-return chains,
  
  whose overhead adds up.
  We go through a lot of effort to minimize load/stores by keeping as much in
  
  registers in between construcring transforms. This saves us around 32 cycles,
  
  on paper, but in reality a lot more due to load/store aliasing (a load from a
  
  memory location cannot be issued while there's a store pending, and there are
  
  only so many (2 for Zen 3) load/store units in a CPU).
  
  Also, we interleave coefficients during the last stage to save on a store+load
  
  per register.
  Each of the smallest, basis transforms (4, 8 and 16-point in our case)
  
  has been extremely optimized. Our 8-point transform is barely 20 instructions
  
  in total, beating our old implementation 8-point transform by 1 instruction.
  
  Our 2x8-point transform is 23 instructions, beating our old implementation by
  
  6 instruction and needing 50% less cycles. Our 16-point transform's combination
  
  code takes slightly more instructions than our old implementation, but makes up
  
  for it by requiring a lot less arithmetic operations.
  Overall, the transform was optimized for the timings of Zen 3, which at the
  
  time of writing has the most IPC from all documented CPUs. Shuffles were
  
  preferred over arithmetic operations due to their 1/0.5 latency/throughput.
  On average, this code is 30% faster than our old libavcodec implementation.
  
  It's able to trade blows with the previously-untouchable FFTW on small transforms,
  
  and due to its tiny size and better prediction, outdoes FFTW on larger transforms
  
  by 11% on the largest currently supported size.
  

  • [DH] libavutil/tx.c
  • [DH] libavutil/tx_priv.h
  • [DH] libavutil/x86/Makefile
  • [DH] libavutil/x86/tx_float.asm
  • [DH] libavutil/x86/tx_float_init.c

• Combine Audio and Images in Stream

  19 décembre 2017, par SenorContento

  I would like to be able to create images on the fly and also create audio on the fly too and be able to combine them together into an rtmp stream (for Twitch or YouTube). The goal is to accomplish this in Python 3 as that is the language my bot is written in. Bonus points for not having to save to disk.

  So far, I have figured out how to stream to rtmp servers using ffmpeg by loading a PNG image and playing it on loop as well as loading a mp3 and then combining them together in the stream. The problem is I have to load at least one of them from file.

  I know I can use Moviepy to create videos, but I cannot figure out whether or not I can stream the video from Moviepy to ffmpeg or directly to rtmp. I think that I have to generate a lot of really short clips and send them, but I want to know if there’s an existing solution.

  There’s also OpenCV which I hear can stream to rtmp, but cannot handle audio.

  A redacted version of an ffmpeg command I have successfully tested with is

  ffmpeg -loop 1 -framerate 15 -i ScreenRover.png -i "Song-Stereo.mp3" -c:v libx264 -preset fast -pix_fmt yuv420p -threads 0 -f flv rtmp://SITE-SUCH-AS-TWITCH/.../STREAM-KEY

  or

  cat Song-Stereo.mp3 | ffmpeg -loop 1 -framerate 15 -i ScreenRover.png -i - -c:v libx264 -preset fast -pix_fmt yuv420p -threads 0 -f flv rtmp://SITE-SUCH-AS-TWITCH/.../STREAM-KEY

  I know these commands are not set up properly for smooth streaming, the result manages to screw up both Twitch’s and Youtube’s player and I will have to figure out how to fix that.

  The problem with this is I don’t think I can stream both the image and the audio at once when creating them on the spot. I have to load one of them from the hard drive. This becomes a problem when trying to react to a command or user chat or anything else that requires live reactions. I also do not want to destroy my hard drive by constantly saving to it.

  As for the python code, what I have tried so far in order to create a video is the following code. This still saves to the HD and is not responsive in realtime, so this is not very useful to me. The video itself is okay, with the one exception that as time passes on, the clock the qr code says versus the video’s clock start to spread apart farther and farther as the video gets closer to the end. I can work around that limitation if it shows up while live streaming.

  def make_frame(t):
  
    img = qrcode.make("Hello! The second is %s!" % t)
  
    return numpy.array(img.convert("RGB"))
  
  
  
  clip = mpy.VideoClip(make_frame, duration=120)
  
  clip.write_gif("test.gif",fps=15)
  
  
  
  gifclip = mpy.VideoFileClip("test.gif")
  
  gifclip.set_duration(120).write_videofile("test.mp4",fps=15)

  My goal is to be able to produce something along the psuedo-code of

  original_video = qrcode_generator("I don't know, a clock, pyotp, today's news sources, just anything that can be generated on the fly!")
  
  original_video.overlay_text(0,0,"This is some sample text, the left two are coordinates, the right three are font, size, and color", Times_New_Roman, 12, Blue)
  
  original_video.add_audio(sine_wave_generator(0,180,2)) # frequency min-max, seconds
  
  
  
  # NOTICE - I did not add any time measurements to the actual video itself. The whole point is this is a live stream and not a video clip, so the time frame would be now. The 2 seconds list above is for our psuedo sine wave generator to know how long the audio clip should be, not for the actual streaming library.
  
  
  
  stream.send_to_rtmp_server(original_video) # Doesn't matter if ffmpeg or some native library

  The above example is what I am looking for in terms of video creation in Python and then streaming. I am not trying to create a clip and then stream it later, I am trying to have the program be able to respond to outside events and then update it’s stream to do whatever it wants. It is sort of like a chat bot, but with video instead of text.

  def track_movement(...):
  
    ...
  
    return ...
  
  
  
  original_video = user_submitted_clip(chat.lastVideoMessage)
  
  original_video.overlay_text(0,0,"The robot watches the user's movements and puts a blue square around it.", Times_New_Roman, 12, Blue)
  
  original_video.add_audio(sine_wave_generator(0,180,2)) # frequency min-max, seconds
  
  
  
  # It would be awesome if I could also figure out how to perform advance actions such as tracking movements or pulling a face out of a clip and then applying effects to it on the fly. I know OpenCV can track movements and I hear that it can work with streams, but I cannot figure out how that works. Any help would be appreciated! Thanks!

  Because I forgot to add the imports, here are some useful imports I have in my file !

  import pyotp
  
  import qrcode
  
  from io import BytesIO
  
  from moviepy import editor as mpy

  The library, pyotp, is for generating one time pad authenticator codes, qrcode is for the qr codes, BytesIO is used for virtual files, and moviepy is what I used to generate the GIF and MP4. I believe BytesIO might be useful for piping data to the streaming service, but how that happens, depends entirely on how data is sent to the service, whether it be ffmpeg over command line (from subprocess import Popen, PIPE) or it be a native library.

• How to save animations with tight layout, transparency and in high quality with matplotlib ?

  29 novembre 2019, par mapf

  I am trying to implement an option in my GUI to save an image sequence displayed using matplotlib. The code looks something like this :

  import matplotlib.pyplot as plt
  
  from matplotlib.backends.backend_qt5agg import \
  
      FigureCanvasQTAgg as FigureCanvas
  
  from matplotlib.animation import FuncAnimation
  
  from PIL import Image
  
  
  
  
  
  plt.rcParams['savefig.bbox'] = 'tight' 
  
  
  
  
  
  class Printer:
  
      def __init__(self, data):
  
          self.fig, self.ax = plt.subplots()
  
          self.canvas = FigureCanvas(self.fig)
  
  
  
          # some irrelevant color adjustment here
  
          #self.ax.spines['bottom'].set_color('#f9f2d7')
  
          #self.ax.spines['top'].set_color('#f9f2d7')
  
          #self.ax.spines['right'].set_color('#f9f2d7')
  
          #self.ax.spines['left'].set_color('#f9f2d7')
  
          #self.ax.tick_params(axis='both', colors='#f9f2d7')
  
          #self.ax.yaxis.label.set_color('#f9f2d7')
  
          #self.ax.xaxis.label.set_color('#f9f2d7')
  
          #self.fig.subplots_adjust(left=0.1, right=0.975, bottom=0.09, top=0.98)
  
          self.fig.patch.set_alpha(0)
  
          self.fig.patch.set_visible(False)
  
          self.canvas.setStyleSheet("background-color:transparent;")
  
          self.fig.set_size_inches(10, 10, True)
  
          self.fig.tight_layout()
  
  
  
          self.data = data
  
          self.image_artist = self.ax.imshow(data[0])
  
  
  
      def animate(self, i):
  
          self.image_artist.set_data(self.data[i])
  
          self.canvas.draw()
  
  
  
  
  
  def save_animation():
  
      data = [
  
          Image.open("test000.png"),
  
          Image.open("test001.png"),
  
      ]
  
      file = 'test.gif'
  
      printer = Printer(data)
  
  
  
      ani = FuncAnimation(
  
          printer.fig, printer.animate, interval=100, frames=len(data),
  
      )
  
      # writer = animation.writers['pillow'](bitrate=1000)
  
      ani.save(
  
          file, writer='pillow', savefig_kwargs={'transparent': True, 'bbox_inches': 'tight'}
  
      )
  
  
  
  
  
  save_animation()

  Transparency :

  As you can see I have already tried several different approaches as suggested elsewhere (1, 2), but didn’t manage to find a solution. All of the settings and arguments patch.set_alpha(0), patch.set_visible(False), canvas.setStyleSheet("background-color:transparent;"), savefig_kwargs={'transparent': True} seem to have no effect at all on the transparency. I found this post but I didn’t get the code to work (for one I had to comment out this %matplotlib inline, but then I ended up getting some error during the MovieWriter.cleanup out = TextIOWrapper(BytesIO(out)).read() TypeError: a bytes-like object is required, not 'str'). Here, it was suggested that this is actually a bug, but the proposed workaroud doesn’t work for me since I would have to rely on third-party software. There also exists this bug report which was supposedly solved, so maybe it is unrelated.

  Tight layout

  I actually couldn’t really find much on this, but all the things I tried (plt.rcParams['savefig.bbox'] = 'tight', fig.tight_layout(), savefig_kwargs={'bbox_inches': 'tight'}) don’t have any effect or are even actively discarded in the case of the bbox_inches argument. How does this work ?

  High quality

  Since I cannot use ImageMagick and can’t get ffmpeg to work (more on this below), I rely on pillow to save my animation. But the only argument in terms of quality that I can pass on seems to be the bitrate, which doesn’t have any effect. The files still have the same size and the animation still looks like mush. The only way that I found to increase the resolution was to use fig.set_size_inches(10, 10, True), but this still doesn’t improve the overall quality of the animation. It still looks bad. I saw that you can pass on codec and extra_args so maybe that is something that might help, but I have no idea how to use these because I couldn’t find a list with allowed arguments.

  ffmpeg

  I can’t get ffmpeg to work. I installed the python package from here and can import it into a python session but I don’t know how I can get matplotlib to use that. I also got ffmpeg from here (Windows 64-bit version) and set the plt.rcParams['animation.ffmpeg_path'] to where I saved the files (there was no intaller to run, not sure if I did it correctly). But this didn’t help either. Also this is of course also third-party software, so if somebody else were to use my code/program it wouldn’t work.