Recherche avancée

Sur d’autres sites (3587)

• Overthinking My Search Engine Problem

  31 décembre 2013, par Multimedia Mike — General

  I wrote a search engine for my Game Music Appreciation website, because the site would have been significantly less valuable without it (and I would eventually realize that the search feature is probably the most valuable part of this endeavor). I came up with a search solution that was a bit sketchy, but worked… until it didn’t. I thought of a fix but still searched for more robust and modern solutions (where ‘modern’ is defined as something that doesn’t require compiling a C program into a static CGI script and hoping that it works on a server I can’t debug on).

  Finally, I realized that I was overthinking the problem– did you know that a bunch of relational database management systems (RDBMSs) support full text search (FTS) ? Okay, maybe you did, but I didn’t know this.

  Problem Statement
  My goal is to enable users to search the metadata (title, composer, copyright, other tags) attached to various games. To do this, I want to index a series of contrived documents that describe the metadata. 2 examples of these contrived documents, interesting because both of these games have very different titles depending on region, something the search engine needs to account for :

  system : Nintendo NES
  game : Snoopy’s Silly Sports Spectacular
  author : None ; copyright : 1988 Kemco ; dumped by : None
  additional tags : Donald Duck.nsf Donald Duck
  system : Super Nintendo
  
  game : Arcana
  
  author : Jun Ishikawa, Hirokazu Ando ; copyright : 1992 HAL Laboratory ; dumped by : Datschge
  
  additional tags : card.rsn.gamemusic Card Master Cardmaster
  

  The index needs to map these documents to various pieces of game music and the search solution needs to efficiently search these documents and find the various game music entries that match a user’s request.

  Now that I’ve been looking at it for long enough, I’m able to express the problem surprisingly succinctly. If I had understood that much originally, this probably would have been simpler.

  First Solution & Breakage
  My original solution was based on SWISH-E. The CGI script was a C program that statically linked the SWISH-E library into a binary that miraculously ran on my web provider. At least, it ran until it decided to stop working a month ago when I added a new feature unrelated to search. It was a very bizarre problem, the details of which would probably bore you to tears. But if you care, the details are all there in the Stack Overflow question I asked on the matter.

  While no one could think of a direct answer to the problem, I eventually thought of a roundabout fix. The problem seemed to pertain to the static linking. Since I couldn’t count on the relevant SWISH-E library to be on my host’s system, I uploaded the shared library to the same directory as the CGI script and used dlopen()/dlsym() to fetch the functions I needed. It worked again, but I didn’t know for how long.

  Searching For A Hosted Solution
  I know that anything is possible in this day and age ; while my web host is fairly limited, there are lots of solutions for things like this and you can deploy any technology you want, and for reasonable prices. I figured that there must be a hosted solution out there.

  I have long wanted a compelling reason to really dive into Amazon Web Services (AWS) and this sounded like a good opportunity. After all, my script works well enough ; if I could just find a simple Linux box out there where I could install the SWISH-E library and compile the CGI script, I should be good to go. AWS has a free tier and I started investigating this approach. But it seems like a rabbit hole with a lot of moving pieces necessary for such a simple task.

  I had heard that AWS had something in this area. Sure enough, it’s called CloudSearch. However, I’m somewhat discouraged by the fact that it would cost me around $75 per month to run the smallest type of search instance which is at the core of the service.

  Finally, I came to another platform called Heroku. It’s supposed to be super-scalable while having a free tier for hobbyists. I started investigating FTS on Heroku and found this article which recommends using the FTS capabilities of their standard hosted PostgreSQL solution. However, the free tier of Postgres hosting only allows for 10,000 rows of data. Right now, my database has about 5400 rows. I expect it to easily overflow the 10,000 limit as soon as I incorporate the C64 SID music corpus.

  However, this Postgres approach planted a seed.

  RDBMS Revelation
  I have 2 RDBMSs available on my hosting plan– MySQL and SQLite (the former is a separate service while SQLite is built into PHP). I quickly learned that both have FTS capabilities. Since I like using SQLite so much, I elected to leverage its FTS functionality. And it’s just this simple :

  CREATE VIRTUAL TABLE gamemusic_metadata_fts USING fts3
  ( content TEXT, game_id INT, title TEXT ) ;
  SELECT id, title FROM gamemusic_metadata_fts WHERE content MATCH "arcana" ;
  
  479|Arcana
  

  The ‘content’ column gets the metadata pseudo-documents. The SQL gets wrapped up in a little PHP so that it queries this small database and turns the result into JSON. The script is then ready as a drop-in replacement for the previous script.

• It’s January 28th : Let’s celebrate Data Privacy !

  29 janvier 2018, par Matomo Core Team

  It is a special Sunday here for us at Matomo, as today is international Data Privacy Day. The day was created in 2007 to raise awareness of the importance of data privacy for people and businesses worldwide.

  

  What is data privacy about ?

  Personal data refers to any data which is collected and can be linked to an individual human being such as phone records, credit card transactions, GPS position, IP addresses, browsing history… So basically, personal data refers to your identity online. That is why you should be highly concerned about sending your personal data (or your customers’ personal data) away. It is important to be aware of who is collecting the information and how it is being used.

  What big changes are happening in 2018 ?

  New privacy regulations GDPR comes into play next May 2018 : GDPR will bring about some changes (in the right direction) by making people and businesses aware of what data privacy means, and what they should be doing to protect their customers’ privacy. With these new regulations, data privacy awareness is reaching a critical milestone this year.

  How can I protect my privacy ?

  Here are a few tips to protect your privacy :

  • Educate yourself on the importance of privacy : the more informed you are the better.
  • Use open source solutions where you can keep full control of your own data (such as NextCloud instead of Dropbox and of course Matomo instead of Google Analytics),
  • Experiment with different online services to protect your data privacy, for example using an alternative search engine (such as DuckDuckGo instead of Google) or an alternative email provider (such as ProtonMail).

  What’s coming next for Matomo and Privacy ?

  Here at Matomo, we are building the leading decentralised open web analytics platform. We’re currently working on new sets of privacy features to make compliance with GDPR a breeze. Stay tuned here to be notified when we launch the new privacy compliance tools !

  And in case you’ve missed this important info, you may be interested in :

  • Configure Privacy Settings in Matomo
  • 11 ways Matomo Analytics helps you to protect your visitors privacy

   

  The post It’s January 28th : Let’s celebrate Data Privacy ! appeared first on Analytics Platform - Matomo.

• UVC webcam with ffplay outputs only noise ?

  1er mars 2024, par Abdulla Masud

  (My end goal is to use a UVC webcam with esp32 or raspberry pi. I was hoping to learn while doing some fun projects.)

  


  I have an old UVC webcam (Creative model ct6840) but I can't seem to get it to work with ffplay. I have tried looking through the documentation and other questions here but nothing is working for me. So far I have only been able to achieve a noisy-jittery output.

  


  Running ffplay -f rawvideo -video_size 670x480 /dev/video1, I get :

  


  

  


  Can someone help me understand how to make the camera work with ffplay ?

  


  The following is the information of my webcam :

  


  $ ffmpeg -f v4l2 -list_formats all -i /dev/video2

  


  [video4linux2,v4l2 @ 0x17eb3c0] Compressed: Unsupported :          GSPCA OV511 : 320x240 640x480


  


  $ v4l-info /dev/video2

  


  ### v4l2 device info [/dev/video2] ###
general info
    VIDIOC_QUERYCAP
    driver                  : "ov519"
    card                    : "USB Camera (05a9:0511)"
    bus_info                : "usb-0000:00:14.0-8.2"
    version                 : 6.1.79
    capabilities            : 0x85200001 [VIDEO_CAPTURE,?,READWRITE,STREAMING,(null)]

standards

inputs
    VIDIOC_ENUMINPUT(0)
    index                   : 0
    name                    : "ov519"
    type                    : CAMERA
    audioset                : 0
    tuner                   : 0
    std                     : 0x0 []
    status                  : 0x0 []

video capture
    VIDIOC_ENUM_FMT(0,VIDEO_CAPTURE)
    index                   : 0
    type                    : VIDEO_CAPTURE
    flags                   : 1
    description             : "GSPCA OV511"
    pixelformat             : 0x3131354f [O511]
    VIDIOC_G_FMT(VIDEO_CAPTURE)
    type                    : VIDEO_CAPTURE
    fmt.pix.width           : 640
    fmt.pix.height          : 480
    fmt.pix.pixelformat     : 0x3131354f [O511]
    fmt.pix.field           : NONE
    fmt.pix.bytesperline    : 640
    fmt.pix.sizeimage       : 614400
    fmt.pix.colorspace      : JPEG
    fmt.pix.priv            : 4276996862

controls
    VIDIOC_QUERYCTRL(BASE+0)
    id                      : 9963776
    type                    : INTEGER
    name                    : "Brightness"
    minimum                 : 0
    maximum                 : 255
    step                    : 1
    default_value           : 127
    flags                   : 48
    VIDIOC_QUERYCTRL(BASE+1)
    id                      : 9963777
    type                    : INTEGER
    name                    : "Contrast"
    minimum                 : 0
    maximum                 : 255
    step                    : 1
    default_value           : 127
    flags                   : 32
    VIDIOC_QUERYCTRL(BASE+2)
    id                      : 9963778
    type                    : INTEGER
    name                    : "Saturation"
    minimum                 : 0
    maximum                 : 255
    step                    : 1
    default_value           : 127
    flags                   : 32
    VIDIOC_QUERYCTRL(BASE+24)
    id                      : 9963800
    type                    : MENU
    name                    : "Power Line Frequency"
    minimum                 : 0
    maximum                 : 2
    step                    : 1
    default_value           : 0
    flags                   : 0
    VIDIOC_QUERYCTRL(BASE+32)
    id                      : 9963808
    type                    : BOOLEAN
    name                    : "Brightness, Automatic"
    minimum                 : 0
    maximum                 : 1
    step                    : 1
    default_value           : 1
    flags                   : 8


  


  Can someone guide me here please ? Any advice will be greatly appreciated 

  


  (P.S. the camera works perfectly with "guvcview" gtk application but since I want to use the camera with raspberry pi, I want it to work with ffplay...)