Recherche avancée

Sur d’autres sites (6273)

• The New Samples Regime

  1er décembre 2011, par Multimedia Mike — General

  A little while ago, I got a big head over the fact that I owned and controlled the feared and revered MPlayer samples archive. This is the repository that retains more than a decade of multimedia samples.

  Conflict
  Where once there was one multimedia project (FFmpeg), there are now 2 (also Libav). There were various political and technical snafus regarding the previous infrastructure. I volunteered to take over hosting the vast samples archive (53 GB at the time) at samples.mplayerhq.hu (s.mphq for this post).

  However, a brand new server is online at samples.libav.org (s.libav for this post).

  Policies
  The server at s.libav will be the authoritative samples repository going forward. Why does s.libav receive the honor ? Mostly by virtue of having more advanced features. My simple (yet bandwidth-rich) web hosting plan does not provide for rsync or anonymous FTP services, both of which have traditionally been essential for the samples server. In the course of hosting s.mphq for the past few months, a few more discrepancies have come to light– apparently, the symlinks weren’t properly replicated. And perhaps most unusual is that if a directory contains a README file, it won’t be displayed in the directory listing (which frustrated me greatly when I couldn’t find this README file that I carefully and lovingly crafted years ago).

  The s.mphq archive will continue to exist — nay, must exist — going forward since there are years’ worth of web links pointing into it. I’ll likely set up a mirroring script that periodically (daily) rsyncs from s.libav to my local machine and then uses lftp (the best facility I have available) to mirror the files up to s.mphq.

  Also, since we’re starting fresh with a new upload directory, I think we need to be far more ruthless about policing its content. This means making sure that anything that is uploaded has an accompanying file which explains why it’s there and ideally links the sample to a bug report somewhere. No explanation = sample terminated.

  RSS
  I think it would be nifty to have an RSS feed that shows the latest samples to appear in the repository. I figure that I can use the Unix ‘find’ command on my local repository in concert with something like PyRSS2Gen to accomplish this goal.

  Monetization
  In the few months that I have been managing the repository, I have had numerous requests for permission to leech the entire collection in one recursive web-suck. These requests often from commercial organizations who wish to test their multimedia product on a large corpus of diverse samples. Personally, I believe the archive makes a rather poor corpus for such an endeavor, but so be it. Go ahead ; hosting this archive barely makes a dent in my fairly low-end web hosting plan. However, at least one person indicated that it might be easier to mail a hard drive to me, have me copy it, and send it back.

  This got me thinking about monetization opportunities. Perhaps, I should provide a service to send HDs filled with samples for the cost of the HD, shipping, and a small donation to the multimedia projects. I immediately realized that that is precisely the point at which the vast multimedia samples archive — with all of its media of questionable fair use status — would officially run afoul of copyright laws.

  Which brings me to…

  Clean Up
  I think we need to clean up some samples, starting with the ones that were marked not-readable in the old repository. Apparently, some ‘samples’ were, e.g., full anime videos and were responsible for a large bandwidth burden when linked from various sources.

  We multimedia nerds are a hoarding lot, never willing to throw anything away. This will probably the most challenging proposal to implement.

• Anomalie #2953 : changement de statut au survol impossible

  8 mars 2013, par chan kalan

  alors peut-être qu’on peut régler ça comme ça ? la règle existe déjà, donc l’overflow dans la page plan seulement .plan .menu-items .item border-bottom : 0 none ; border-top : 1px solid #DDDDDD ; overflow : visible ; padding-left : 4px ; ou alors pour éviter de toucher l’overflow, juste un padding + fort ? (...)

• Cuda Memory Management : re-using device memory from C calls (multithreaded, ffmpeg), but failing on cudaMemcpy

  4 mars 2013, par Nuke Stollak

  I'm trying to CUDA-fy my ffmpeg filter that was taking over 90% of the CPU time, according to gprof. I first went from one core to OpenMP on 4 cores and got a 3.8x increase in frames encoded per second, but it's still too slow. CUDA seemed like the next natural step.

  I've gotten a modest (20% ?) increase by replacing one of my filter's functions with a CUDA kernel call, and just to get things up and running, I was cudaMalloc'ing and cudaMemcpy'ing on each frame. I suspected I would get better results if I weren't doing this each frame, so before I go ahead and move the rest of my code to CUDA, I wanted to fix this by allocating the memory before my filter is called and freeing it afterwards, but the device memory isn't having it. I'm only storing the device memory locations outside of code that knows about CUDA ; I'm not trying to use the data there, just save it for the next time I call a CUDA-aware function that needs it.

  Here's where I am so far :

  Environment : the last AMI Linux on EC2's GPU Cluster, latest updates installed. Everything is fairly standard.

  My filter is split into two files : vf_myfilter.c (compiled by gcc, like almost every other file in ffmpeg) and vf_myfilter_cu.cu (compiled by nvcc). My Makefile's link step includes -lcudart and both .o files. I build vf_myfilter_cu.o using (as one line)

  nvcc -I. -I./ -I/opt/nvidia/cuda/include $(CPPFLAGS) 
  
       -Xcompiler "$(CFLAGS)" 
  
        -c -o libfilter/vf_myfilter_cu.o libfilter/vf_myfilter_cu.cu

  When the variables (set by configure) are expanded, here's what I get, again all in one line but split up here for easier reading. I just noticed the duplicate include path directives, but it shouldn't hurt.

  nvcc -I. -I./ -I/opt/nvidia/cuda/include -I. -I./ -D_ISOC99_SOURCE 
  
      -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112
  
      -D_XOPEN_SOURCE=600 -DHAVE_AV_CONFIG_H 
  
      -XCompiler "-fopenmp -std=c99 -fomit-frame-pointer -pthread -g 
  
                  -Wdeclaration-after-statment -Wall -Wno-parentheses 
  
                  -Wno-switch -Wno-format-zero-length -Wdisabled-optimization  
  
                  -Wpointer-arith -Wredundant-decls -Wno-pointer-sign 
  
                  -Wwrite-strings -Wtype-limits -Wundef -Wmissing-prototypes
  
                  -Wno-pointer-to-int-case -Wstrict-prototypes -O3 -fno-math-errno 
  
                  -fno-signed-zeros -fno-tree-vectorize 
  
                  -Werror=implicit-function-declaration -Werror=missing-prototypes 
  
                  -Werror=vla " 
  
      -c -o libavfilter/vf_myfilter_cu.o libavfilter/vf_myfilter_cu.cu

  vf_myfilter.c calls three functions from vf_myfilter_cu.cu file which handle memory and call the CUDA kernel code. I thought I would be able to save the device pointers from my memory initialization, which runs once per ffmpeg run, and re-use that space each time I called the wrapper for my kernel function, but when I cudaMemcpy from my host memory to my device memory that I stored, it fails with cudaInvalidValue. If I cudaMalloc my device memory on every frame, I'm fine.

  I plan on using pinned host memory, once I have everything up in CUDA code and have minimized the number of times I need to return to the main ffmpeg code.

  Steps taken :

  First sign of trouble : search the web. I found Passing a pointer to device memory between classes in CUDA and printed out the pointers at various places in my execution to ensure that the device memory values were the same everywhere, and they are. FWIW, they seem to start around 0x90010000.

  ffmpeg's configure gave me -pthreads, so I checked to see if my filter was being called from multiple threads according to how can I tell if pthread_self is the main (first) thread in the process ? and checking syscall(SYS_gettid) == getpid() to ensure that I'm not calling CUDA from different threads—I'm indeed in the primary thread at every step, according to those two funcs. I am still using OpenMP later around some for loops in the main .c filter function, but the calls to CUDA don't occur in those loops.

  Code Overview :

  ffmpeg provides me a MyfilterContext structure pointer on each frame, as well as on the filter's config_input and uninit routines (called once per file), so I added some *host_var and *dev_var variables (a few of each, float and unsigned char).

  There is a whole lot of code I skipped for this post, but most of it has to do with my algorithm and details involved in writing an ffmpeg filter. I'm actually using about 6 host variables and 7 device variables right now, but for demonstration I limited it to one of each.

  Here is, broadly, what my vf_myfilter.c looks like.

  // declare my functions from vf_myfilter_cu.cu 
  
  extern void cudaMyInit(unsigned char **dev_var, size_t mysize);
  
  extern void cudaMyUninit(unsigned char *dev_var);
  
  extern void cudaMyFunction(unsigned char *host_var, unsigned char *dev_var, size_t mysize);
  
  
  
  // part of the MyFilterContext structure, which ffmpeg keeps track of for me.
  
  typedef struct {
  
      unsigned char *host_var;
  
      unsigned char *dev_var;
  
  } MyFilterContext;
  
  
  
  // ffmpeg calls this function once per file, before any frames are processed.
  
  static int config_input(AVFilterLink *inlink) {
  
          // how ffmpeg passes me my context, fairly standard.
  
      MyfilterContext * myContext = inlink->dst->priv; 
  
          // compute the size one video plane of one frame of video
  
      size_t mysize = sizeof(unsigned char) * inlink->w * inlink->h;
  
          // av_mallocz is a malloc wrapper provided and required by ffmpeg
  
      myContext->host_var = (unsigned char*) av_mallocz(size);
  
          // Here's where I attempt to allocate my device memory.
  
      cudaMyInit( & myContext->dev_var, mysize);  
  
  }
  
  
  
  // Called once per frame of video
  
  static int filter_frame(AVFilterLink *inlink, AVFilterBufferRef *frame) {
  
      MyFilterContext *myContext = inlink->dst->priv;
  
  
  
      // sanity check to make sure that this isn't part of the multithreaded code
  
      if ( syscall(SYS_gettid) == getpid() ) 
  
          av_log(.... ); // This line never runs, so it's not threaded?
  
  
  
      // ...fill host_var with data from frame, 
  
      // set mysize to the size of the buffer
  
  
  
      // Call my wrapper function defined in the .cu file
  
      cudaMyFunction(myContext->host_var, myContext->dev_var, mysize);
  
  
  
      // ... take the results from host_var and apply them to frame
  
      // ... and return the processed frame to ffmpeg 
  
  }
  
  
  
  // called after everything else has happened:  free up the memory.
  
  static av_cold void uninit(AVFilterContext *ctx) {
  
      MyFilterContext *myContext = ctx->priv;
  
      // free my host_var
  
      if(myContext->host_var!=NULL) {
  
          av_free(myContext->host_var);
  
          myContext->host_var=NULL;
  
      }
  
      // free my dev_var
  
      cudaMyUninit(myContext->dev_var);
  
  }

  Here is, broadly, what my vf_myfilter_cu.cu looks like :

  // my kernel function that does the work.
  
  __global__ void myfunc(unsigned char *dev_var, size_t mysize) {
  
      // find the offset for this particular GPU thread to process
  
      // exit this function if the block/thread combo points to somewhere
  
      //     outside the frame
  
      // make sure we're less than mysize bytes from the beginning of dev_var
  
      // do things to dev_var[some_offset]
  
  } 
  
  // Allocate the device memory
  
  extern "C" void cudaMyInit(unsigned char **dev_var, size_t mysize) {
  
      if(cudaMalloc( (void**) dev_var, mysize) != cudaSuccess) {
  
          printf("Cannot allocate the memory\n");
  
      }
  
  }
  
  
  
  // Free the device memory.
  
  extern "C" void cudaMyUninit(unsigned char *dev_var) {
  
      cudaFree(dev_var);
  
  }
  
  
  
  // Copy data from the host to the device,
  
  // Call the kernel function, and 
  
  // Copy data from the device to the host.
  
  extern "C" void cudaMyFunction(
  
          unsigned char *host_var, 
  
          unsigned char *dev_var, 
  
          size_t mysize         ) 
  
  {
  
      cudaError_t cres;
  
  
  
      // dev_works is what I want to get rid of, but 
  
      // to make sure that there's not something more obvious going 
  
      // on, I made sure that my cudaMemcpy works if I'm allocating
  
      // the device memory in every frame.
  
      unsigned char *dev_works;  
  
      if(cudaMalloc( (void **) &dev_works, mysize)!=cudaSuccess) { 
  
          // I don't see this message
  
          printf("failed at per-frame malloc\n");
  
      }
  
  
  
      // THIS PART WORKS, copying host_var to dev_works
  
      cres=cudaMemcpy( (void *) dev_works, host_var, mysize, cudaMemcpyHostToDevice);
  
      if(cres!=cudaSuccess) {
  
          if(cres==cudaErrorInvalidValue) {
  
              // I don't see this message.
  
              printf("cudaErrorInvalidValue at per-frame cudaMemcpy\n");
  
          }
  
      }
  
  
  
      // THIS PART FAILS, copying host_var to dev_var
  
      cres=cudaMemcpy( (void *) dev_var, host_var, mysize, cudaMemcpyHostToDevice);
  
      if(cres!=cudaSuccess) {
  
          if(cres==cudaErrorInvalidValue) {
  
              // this is the error code that prints.
  
              printf("cudaErrorInvalidValue at per-frame cudaMemcpy\n");
  
          }
  
          // I check for other error codes, but they're not being hit.
  
      }
  
  
  
      // and this works with dev_works
  
      myfunc<<>>(dev_works, mysize);
  
  
  
      if(cudaMemcpy(host_var, dev_works, mysize, cudaMemcpyDeviceToHost)!=cudaSuccess) {
  
          // I don't see this message.
  
          printf("Failed to copy post-kernel func\n");
  
      }
  
  
  
      cudaFree(dev_works);
  
  
  
  }

  Any ideas ?