Recherche avancée

Sur d’autres sites (5564)

• Parsing The Clue Chronicles

  30 décembre 2018, par Multimedia Mike — Game Hacking

  A long time ago, I procured a 1999 game called Clue Chronicles : Fatal Illusion, based on the classic board game Clue, a.k.a. Cluedo. At the time, I was big into collecting old, unloved PC games so that I could research obscure multimedia formats.

  
  
  

  Surveying the 3 CD-ROMs contained in the box packaging revealed only Smacker (SMK) videos for full motion video which was nothing new to me or the multimedia hacking community at the time. Studying the mix of data formats present on the discs, I found a selection of straightforward formats such as WAV for audio and BMP for still images. I generally find myself more fascinated by how computer games are constructed rather than by playing them, and this mix of files has always triggered a strong “I could implement a new engine for this !” feeling in me, perhaps as part of the ScummVM project which already provides the core infrastructure for reimplementing engines for 2D adventure games.

  Tying all of the assets together is a custom high-level programming language. I have touched on this before in a blog post over a decade ago. The scripts are in a series of files bearing the extension .ini (usually reserved for configuration scripts, but we’ll let that slide). A representative sample of such a script can be found here :

  clue-chronicles-scarlet-1.txt

  What Is This Language ?
  At the time I first analyzed this language, I was still primarily a C/C++-minded programmer, with a decent amount of Perl experience as a high level language, and had just started to explore Python. I assessed this language to be “mildly object oriented with C++-type comments (‘//’) and reliant upon a number of implicit library functions”. Other people saw other properties. When I look at it nowadays, it reminds me a bit more of JavaScript than C++. I think it’s sort of a Rorschach test for programming languages.

  Strangely, I sort of had this fear that I would put a lot of effort into figuring out how to parse out the language only for someone to come along and point out that it’s a well-known yet academic language that already has a great deal of supporting code and libraries available as open source. Google for “spanish dolphins far side comic” for an illustration of the feeling this would leave me with.

  It doesn’t matter in the end. Even if such libraries exist, how easy would they be to integrate into something like ScummVM ? Time to focus on a workable approach to understanding and processing the format.

  Problem Scope
  So I set about to see if I can write a program to parse the language seen in these INI files. Some questions :

  1. How large is the corpus of data that I need to be sure to support ?
  2. What parsing approach should I take ?
  3. What is the exact language format ?
  4. Other hidden challenges ?

  To figure out how large the data corpus is, I counted all of the INI files on all of the discs. There are 138 unique INI files between the 3 discs. However, there are 146 unique INI files after installation. This leads to a hidden challenge described a bit later.

  What parsing approach should I take ? I worried a bit too much that I might not be doing this the “right” way. I’m trying to ignore doubts like this, like how “SQL Shame” blocked me on a task for a little while a few years ago as I concerned myself that I might not be using the purest, most elegant approach to the problem. I know I covered language parsing a lot time ago in university computer science education and there is a lot of academic literature to the matter. But sometimes, you just have to charge in and experiment and prototype and see what falls out. In doing so, I expect to have a better understanding of the problems that need to solved and the right questions to ask, not unlike that time that I wrote a continuous integration system from scratch because I didn’t actually know that “continuous integration” was the keyword I needed.

  Next, what is the exact language format ? I realized that parsing the language isn’t the first and foremost problem here– I need to know exactly what the language is. I need to know what the grammar are keywords are. In essence, I need to reverse engineer the language before I write a proper parser for it. I guess that fits in nicely with the historical aim of this blog (reverse engineering).

  Now, about the hidden challenges– I mentioned that there are 8 more INI files after the game installs itself. Okay, so what’s the big deal ? For some reason, all of the INI files are in plaintext on the CD-ROM but get compressed (apparently, according to file size ratios) when installed to the hard drive. This includes those 8 extra INI files. I thought to look inside the CAB installation archive file on the CD-ROM and the files were there… but all in compressed form. I suspect that one of the files forms the “root” of the program and is the launching point for the game.

  Parsing Approach
  I took a stab at parsing an INI file. My approach was to first perform lexical analysis on the file and create a list of 4 types : symbols, numbers, strings, and language elements ([]{}()=., :). Apparently, this is the kind of thing that Lex/Flex are good at. This prototyping tool is written in Python, but when I port this to ScummVM, it might be useful to call upon the services of Lex/Flex, or another lexical analyzer, for there are many. I have a feeling it will be easier to use better tools when I understand the full structure of the language based on the data available.
  
  The purpose of this tool is to explore all the possibilities of the existing corpus of INI files. To that end, I ran all 138 of the plaintext files through it, collected all of the symbols, and massaged the results, assuming that the symbols that occurred most frequently are probably core language features. These are all the symbols which occur more than 1000 times among all the scripts :

     6248 false
     5734 looping
     4390 scripts
     3877 layer
     3423 sequentialscript
     3408 setactive
     3360 file
     3257 thescreen
     3239 true
     3008 autoplay
     2914 offset
     2599 transparent
     2441 text
     2361 caption
     2276 add
     2205 ge
     2197 smackanimation
     2196 graphicscript
     2196 graphic
     1977 setstate
     1642 state
     1611 skippable
     1576 desc
     1413 delayscript
     1298 script
     1267 seconds
     1019 rect
  

  About That Compression
  I have sorted out at least these few details of the compression :

  bytes 0-3    "COMP" (a pretty strong sign that this is, in fact, compressed data)
  bytes 4-11   unknown
  bytes 12-15  size of uncompressed data
  bytes 16-19  size of compressed data (filesize - 20)
  bytes 20-    compressed payload
  

  The compression ratios are on the same order of gzip. I was hoping that it was stock zlib data. However, I have been unable to prove this. I wrote a Python script that scrubbed through the first 100 bytes of payload data and tried to get Python’s zlib.decompress to initialize– no luck. It’s frustrating to know that I’ll have to reverse engineer a compression algorithm that deals with just 8 total text files if I want to see this effort through to fruition.

  Update, January 15, 2019
  Some folks expressed interest in trying to sort out the details of the compression format. So I have posted a followup in which I post some samples and go into deeper details about things I have tried :

  Reverse Engineering Clue Chronicles Compression

  The post Parsing The Clue Chronicles first appeared on Breaking Eggs And Making Omelettes.

• Combining three or more videos with ffmpeg and the xfade filter

  25 février 2021, par sil

  As of 2020 ffmpeg has the xfade filter which can combine videos with a transition. Combining two videos is easy enough :

  


  ffmpeg -i vid1.mp4 -i vid2.mp4 \
  -filter_complex [0][1]xfade=transition=pixelize:duration=1:offset=4,format=yuv420p" \
  out.mp4


  


  But I don't understand how to combine three videos (so that v1 fades into v2 and v2 then fades into v3. I tried something like this :

  


  ffmpeg -i vid1.mp4 -i vid2.mp4 -i vid3.mp4 \
  -filter_complex [0][1]xfade=transition=pixelize:duration=1:offset=4,format=yuv420p[0n1];[0n1][2]xfade=transition=pixelize:duration=1:offset=9,format=yuv420p" \
  out.mp4


  


  but that doesn't work. My idea was that 0 and 1, or vid1 and vid2, would be combined into a [0n1] stream with a transition by xfade, and then that 0n1 stream could be combined with vid3 with another filter. As far as I can tell, this includes the first two videos but not the third. What this of course means is that I don't understand how to specify a filtergraph correctly !

  


  How should I use xfade to combine 3 or more videos with transitions between them ?

  


  A full example is as follows. Here I'll use three images (because then issues with combining videos at different frame rates are avoided), and smash them all to 500x500 for ease (in the final version they would be letterboxed to keep resolution and so on).

  


  ffmpeg \
  -loop 1 -t 5 -i tests/p1.jpg \
  -loop 1 -t 5 -i tests/p2.jpg \
  -loop 1 -t 5 -i tests/p3.jpg \
  -filter_complex "[0]scale=500:500[s0];[1]scale=500:500[s1];[2]scale=500:500[s2];[s0][s1]xfade=transition=pixelize:duration=1:offset=4,format=yuv420p[s01];[s01][s2]xfade=transition=pixelize:duration=1:offset=9,format=yuv420p" out.mp4


  


  I would expect this to create a video which was :

  


  

  • 4 seconds of p1.jpg

  


  • a pixelise transition into p2.jpg lasting 1 second

  


  • 4 seconds of p2.jpg

  


  • a pixelise transition into p3.jpg lasting 1 second

  


  • 4 seconds of p3.jpg

  


  


  but what I actually get is

  


  

  • 4 seconds of p1.jpg

  


  • a pixelise transition into p2.jpg lasting 1 second

  


  • 4 seconds of p2.jpg

  


  


  and then the video ends. p3 is not included at all.

  


  The output is as follows :

  


  ffmpeg version N-53546-g5eb4405fc5-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 8 (Debian 8.3.0-6)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
  libavutil      56. 56.100 / 56. 56.100
  libavcodec     58. 97.100 / 58. 97.100
  libavformat    58. 49.100 / 58. 49.100
  libavdevice    58. 11.101 / 58. 11.101
  libavfilter     7. 87.100 /  7. 87.100
  libswscale      5.  8.100 /  5.  8.100
  libswresample   3.  8.100 /  3.  8.100
  libpostproc    55.  8.100 / 55.  8.100
Input #0, image2, from 'tests/p1.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 44845 kb/s
    Stream #0:0: Video: mjpeg (Baseline), yuvj444p(pc, bt470bg/unknown/unknown), 820x1270 [SAR 150:150 DAR 82:127], 25 fps, 25 tbr, 25 tbn, 25 tbc
Input #1, image2, from 'tests/p2.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 22325 kb/s
    Stream #1:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 960x600 [SAR 1:1 DAR 8:5], 25 fps, 25 tbr, 25 tbn, 25 tbc
Input #2, image2, from 'tests/p3.jpg':
  Duration: 00:00:00.04, start: 0.000000, bitrate: 15266 kb/s
    Stream #2:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 728x669 [SAR 96:96 DAR 728:669], 25 fps, 25 tbr, 25 tbn, 25 tbc
File 'out.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 (mjpeg) -> scale
  Stream #1:0 (mjpeg) -> scale
  Stream #2:0 (mjpeg) -> scale
  format -> Stream #0:0 (libx264)
Press [q] to stop, [?] for help
[swscaler @ 0x8228040] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x8258640] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x827df40] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x829f800] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x82c13c0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x82e8340] deprecated pixel format used, make sure you did set range correctly
[libx264 @ 0x77b7600] using SAR=82/127
[libx264 @ 0x77b7600] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x77b7600] profile High, level 3.0, 4:2:0, 8-bit
[libx264 @ 0x77b7600] 264 - core 161 r3018 db0d417 - H.264/MPEG-4 AVC codec - Copyleft 2003-2020 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf58.49.100
    Stream #0:0: Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 500x500 [SAR 82:127 DAR 82:127], q=-1--1, 25 fps, 12800 tbn, 25 tbc (default)
    Metadata:
      encoder         : Lavc58.97.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=  101 fps=100 q=28.0 size=       0kB time=00:00:01.92 bitrate=   0.2kbits/s speed=[swscaler @ 0x8291d80] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
[swscaler @ 0x82b3000] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
[swscaler @ 0x82fc200] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
frame=  153 fps=101 q=28.0 size=       0kB time=00:00:04.00 bitrate=   0.1kbits/s speed=[swscaler @ 0x82fc200] deprecated pixel format used, make sure you did set range correctly
    Last message repeated 2 times
frame=  225 fps=111 q=28.0 size=     256kB time=00:00:06.88 bitrate= 304.9kbits/s dup=0 frame=  225 fps= 95 q=-1.0 Lsize=     267kB time=00:00:08.88 bitrate= 245.9kbits/s dup=0 drop=125 speed=3.74x    
video:263kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.293847%
[libx264 @ 0x77b7600] frame I:2     Avg QP:21.03  size: 19928
[libx264 @ 0x77b7600] frame P:66    Avg QP:21.18  size:  2453
[libx264 @ 0x77b7600] frame B:157   Avg QP:29.72  size:   427
[libx264 @ 0x77b7600] consecutive B-frames:  5.3%  4.4%  1.3% 88.9%
[libx264 @ 0x77b7600] mb I  I16..4: 29.0% 27.8% 43.2%
[libx264 @ 0x77b7600] mb P  I16..4: 10.8%  3.2%  7.5%  P16..4:  2.6%  0.9%  0.3%  0.0%  0.0%    skip:74.8%
[libx264 @ 0x77b7600] mb B  I16..4:  1.1%  0.5%  1.6%  B16..8:  1.0%  0.5%  0.1%  direct: 0.3%  skip:94.9%  L0:37.4% L1:43.5% BI:19.1%
[libx264 @ 0x77b7600] 8x8 transform intra:16.5% inter:55.3%
[libx264 @ 0x77b7600] coded y,uvDC,uvAC intra: 31.4% 71.7% 31.8% inter: 1.1% 1.4% 0.1%
[libx264 @ 0x77b7600] i16 v,h,dc,p: 45% 46%  9%  1%
[libx264 @ 0x77b7600] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 30% 27%  6%  1%  1%  2%  1%  7%
[libx264 @ 0x77b7600] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 56% 31%  9%  1%  1%  1%  1%  1%  1%
[libx264 @ 0x77b7600] i8c dc,h,v,p: 46% 27% 23%  4%
[libx264 @ 0x77b7600] Weighted P-Frames: Y:10.6% UV:7.6%
[libx264 @ 0x77b7600] ref P L0: 61.9% 17.1% 13.9%  6.8%  0.3%
[libx264 @ 0x77b7600] ref B L0: 76.3% 22.8%  0.9%
[libx264 @ 0x77b7600] ref B L1: 97.6%  2.4%
[libx264 @ 0x77b7600] kb/s:238.88


  


• Piwik Analytics and becoming a Piwik Certified Professional

  10 juillet 2017, par Piwik Core Team — About

  Digital Analytics software

  Piwik Analytics is the leading open source digital analytics software, offering users around the world an opportunity to liberate their analytics. Most recently, they have introduced the Piwik Certified Professional certification exam which now allows users to become qualified in Piwik Analytics software on an individual level to gain a deeper understanding of Piwik. In this blog post I will guide you through the topics that are covered during the exam and provide you with advice on taking the official Piwik Certified Professional exam.

  

  Piwik certification exam

  Taking the exam will cost you a maximum investment of 60 minutes of your time, besides learning all materials of course. The exam consists of 55 multiple choice questions with four answers to choose from. The score needed to pass is 80% (44 questions answered correctly) and the cost is 50 USD total. An earned certificate is valid for 18 months, before these eighteen months are over a person should pass the exam again in order to retain the certified status.

  Learning topics

  The exam consists of two sections. The main section is focused on the Piwik Analytics software itself while the second part relates to digital analytics in general. All topics and content covered about Piwik Analytics is available through the official Piwik user guides. The second section tests your experience as a digital analyst, online marketer or any other function title in which you work with Piwik Analytics. In this case, the general digital analytics questions should be quite straightforward and easy to answer, and cover only a fraction of the total questions in the exam (around 10% with 5-7 general questions). An outline of all exam topics are listed below :

  • A Tour of Piwik
  • Track Goals and Measure Conversions
  • Event Tracking
  • Content Tracking
  • Ecommerce Analytics
  • Row Evolution – View and compare historical data
  • Segmentation – Compare segments of visitors
  • Visitors Maps – World, region, city
  • Real Time Visitor World Map
  • Real Time Analytics
  • The Visitor Profile
  • Site speed and Page speed
  • Site Search Tracking and Reporting
    
  • Transitions – Analyze the previous and following actions of your visitors for each page
  • Page Overlay
  • Custom Variables Analytics
  • Custom Dimensions
  • User ID
  • Annotating your data
  • Tracking Campaigns
  • URL Builder for Marketing Campaign Tracking

  The best way to prepare for the exam is read the entire Piwik user guides. You should definitely read the “Analytics Features” section since most questions of the Piwik Certified Professional – Digital Analytics exam that will be asked come from these sections. Furthermore you should be able to find your way around in Piwik at a basic level which means you know what the reports mean and where to find certain information. In addition, some basic knowledge regarding the settings is useful too. The exam is definitely not a technical implementation exam so no coding knowledge or any other deeply technical knowledge regarding Piwik is required.

  Finally, some general questions will be asked regarding digital analytics covering topics about KPI’s and the role of the analyst within an organization. While Piwik provides some links to articles by Avinash Kaushik covering these topics, you will not be able to learn these topics just by reading. When you have some experience with digital analytics you should be able to answer these general digital analytics questions with common sense and (even basic) experience as a digital analyst, analytics consultant, online marketer or any other related job whereby you work with Piwik.

  Taking the exam

  With 55 questions to be answered in 60 minutes the key to passing the exam is to keep moving. You have about 1 minute and 5 seconds to answer each question. This means that you should focus on the easiest questions first and return later to the questions that are a bit more challenging to answer. Keep an eye on the timer that will be displayed in the exam window. When the time expires or you click ‘Finish test’ your exam will end and be automatically submitted for review. Remember to first check all questions and answers before you click on the ‘Finish test’ button. If you click too soon and you still haven’t answered all questions, all unanswered questions will be marked as incorrect.

  During the test, no hard copy or online materials may be referenced. As you can imagine, it is almost impossible to check if users reference these kind of materials. However, be aware of the penalty system that is in place during the test. When a user leaves from the active test screen to another screen (i.e. a different browser tab) the screen turns red and provides a warning count when the user returns to the test screen again. You will have three warnings, after this your test will be submitted and graded as false. Furthermore, keep in mind you will have to do the test in one go and cannot pause and come back another time.

  Practice makes perfect

  Below I have included some example questions that could be asked during the exam. These questions do not necessarily represent how Piwik will test you on these topics.

  • What is the default report date that is selected by Piwik ?
  • Why would someone flatten a report in Piwik ?
  • Why would a user especially use the Page Overlay report ?
  • What are the three main Ecommerce interactions tracked with Piwik ?
  • What is an example of an anonymized IP address in Piwik ?

  Passing the Piwik Certified Professional – Digital Analytics exam

  Directly after submitting the exam you will receive a notification telling you whether or not you have passed the exam. If you pass, you will be able to download your personal certification right away. A report of your exam performance will also be available. This report lists the amount of correct answers and total questions by topic. The report with your exam performance is also available if you did not pass the exam. The certificate is valid for 18 months from the date of successful completion.

  Sometimes the difference between passing and failing can be a matter of how you interpret some of Piwik’s questions. There are several tricky questions included, so be sure to pay attention to detail on every question. If you fail, you may take the exam again. You will have to pay the 50 USD fee for each try, so do your best to pass it the first time.

  → Register to become a Piwik Certified Professional.

  We wish you the best of luck and happy analytics !