Recherche avancée

Sur d’autres sites (7837)

• Reverse Engineering Italian Literature

  1er juillet 2014, par Multimedia Mike — Reverse Engineering

  Some time ago, Diego “Flameeyes” Pettenò tried his hand at reverse engineering a set of really old CD-ROMs containing even older Italian literature. The goal of this RE endeavor would be to extract the useful literature along with any structural metadata (chapters, etc.) and convert it to a more open format suitable for publication at, e.g., Project Gutenberg or Archive.org.

  Unfortunately, the structure of the data thwarted the more simplistic analysis attempts (like inspecting for blocks of textual data). This will require deeper RE techniques. Further frustrating the effort, however, is the fact that the binaries that implement the reading program are written for the now-archaic Windows 3.1 operating system.

  In pursuit of this RE goal, I recently thought of a way to glean more intelligence using DOSBox.

  Prior Work
  There are 6 discs in the full set (distributed along with 6 sequential issues of a print magazine named L’Espresso). Analysis of the contents of the various discs reveals that many of the files are the same on each disc. It was straightforward to identify the set of files which are unique on each disc. This set of files all end with the extension “LZn”, where n = 1..6 depending on the disc number. Further, the root directory of each disc has a file indicating the sequence number (1..6) of the CD. Obviously, these are the interesting targets.

  The LZ file extensions stand out to an individual skilled in the art of compression– could it be a variation of the venerable LZ compression ? That’s actually unlikely because LZ — also seen as LIZ — stands for Letteratura Italiana Zanichelli (Zanichelli’s Italian Literature).

  The Unix ‘file’ command was of limited utility, unable to plausibly identify any of the files.

  Progress was stalled.

  Saying Hello To An Old Frenemy
  I have been showing this screenshot to younger coworkers to see if any of them recognize it :

  
  
  

  Not a single one has seen it before. Senior computer citizen status : Confirmed.

  I recently watched an Ancient DOS Games video about Windows 3.1 games. This episode showed Windows 3.1 running under DOSBox. I had heard this was possible but that it took a little work to get running. I had a hunch that someone else had probably already done the hard stuff so I took to the BitTorrent networks and quickly found a download that had the goods ready to go– a directory of Windows 3.1 files that just had to be dropped into a DOSBox directory and they would be ready to run.

  Aside : Running OS software procured from a BitTorrent network ? Isn’t that an insane security nightmare ? I’m not too worried since it effectively runs under a sandboxed virtual machine, courtesy of DOSBox. I suppose there’s the risk of trojan’d OS software infecting binaries that eventually leave the sandbox.

  Using DOSBox Like ‘strace’
  strace is a tool available on some Unix systems, including Linux, which is able to monitor the system calls that a program makes. In reverse engineering contexts, it can be useful to monitor an opaque, binary program to see the names of the files it opens and how many bytes it reads, and from which locations. I have written examples of this before (wow, almost 10 years ago to the day ; now I feel old for the second time in this post).

  Here’s the pitch : Make DOSBox perform as strace in order to serve as a platform for reverse engineering Windows 3.1 applications. I formed a mental model about how DOSBox operates — abstracted file system classes with methods for opening and reading files — and then jumped into the source code. Sure enough, the code was exactly as I suspected and a few strategic print statements gave me the data I was looking for.

  Eventually, I even took to running DOSBox under the GNU Debugger (GDB). This hasn’t proven especially useful yet, but it has led to an absurd level of nesting :

  
  
  

  The target application runs under Windows 3.1, which is running under DOSBox, which is running under GDB. This led to a crazy situation in which DOSBox had the mouse focus when a GDB breakpoint was triggered. At this point, DOSBox had all desktop input focus and couldn’t surrender it because it wasn’t running. I had no way to interact with the Linux desktop and had to reboot the computer. The next time, I took care to only use the keyboard to navigate the application and trigger the breakpoint and not allow DOSBox to consume the mouse focus.

  New Intelligence
  
  By instrumenting the local file class (virtual HD files) and the ISO file class (CD-ROM files), I was able to watch which programs and dynamic libraries are loaded and which data files the code cares about. I was able to narrow down the fact that the most interesting programs are called LEGGENDO.EXE (‘reading’) and LEGGENDA.EXE (‘legend’ ; this has been a great Italian lesson as well as RE puzzle). The first calls the latter, which displays this view of the data we are trying to get at :

  
  
  

  When first run, the program takes an interest in a file called DBBIBLIO (‘database library’, I suspect) :

  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x0
  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x151
  === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x2A2
  [...]
  

  While we were unable to sort out all of the data files in our cursory investigation, a few things were obvious. The structure of this file looked to contain 336-byte records. Turns out I was off by 1– the records are actually 337 bytes each. The count of records read from disc is equal to the number of items shown in the UI.

  Next, the program is interested in a few more files :

  *** isoFile() : ’DEPOSITO\BLOKCTC.LZ1’, offset 0x27D6000, 2911488 bytes large
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 96 bytes ; read 96 bytes from pos 0x0
  *** isoFile() : ’DEPOSITO\BLOKCTX0.LZ1’, offset 0x2A9D000, 17152 bytes large
  === Read(’DEPOSITO\BLOKCTX0.LZ1’) : req 128 bytes ; read 128 bytes from pos 0x0
  === Seek(’DEPOSITO\BLOKCTX0.LZ1’) : seek 384 (0x180) bytes, type 0
  === Read(’DEPOSITO\BLOKCTX0.LZ1’) : req 256 bytes ; read 256 bytes from pos 0x180
  === Seek(’DEPOSITO\BLOKCTC.LZ1’) : seek 1152 (0x480) bytes, type 0
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 32 bytes ; read 32 bytes from pos 0x480
  === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 1504 bytes ; read 1504 bytes from pos 0x4A0
  [...]

  Eventually, it becomes obvious that BLOKCTC has the juicy meat. There are 32-byte records followed by variable-length encoded text sections. Since there is no text to be found in these files, the text is either compressed, encrypted, or both. Some rough counting (the program seems to disable copy/paste, which thwarts more precise counting), indicates that the text size is larger than the data chunks being read from disc, so compression seems likely. Encryption isn’t out of the question (especially since the program deems it necessary to disable copy and pasting of this public domain literary data), and if it’s in use, that means the key is being read from one of these files.

  Blocked On Disassembly
  So I’m a bit blocked right now. I know exactly where the data lives, but it’s clear that I need to reverse engineer some binary code. The big problem is that I have no idea how to disassemble Windows 3.1 binaries. These are NE-type executable files. Disassemblers abound for MZ files (MS-DOS executables) and PE files (executables for Windows 95 and beyond). NE files get no respect. It’s difficult (but not impossible) to even find data about the format anymore, and details are incomplete. It should be noted, however, the DOSBox-as-strace method described here lends insight into how Windows 3.1 processes NE-type EXEs. You can’t get any more authoritative than that.

  So far, I have tried the freeware version of IDA Pro. Unfortunately, I haven’t been able to get the program to work on my Windows machine for a long time. Even if I could, I can’t find any evidence that it actually supports NE files (the free version specifically mentions MZ and PE, but does not mention NE or LE).

  I found an old copy of Borland’s beloved Turbo Assembler and Debugger package. It has Turbo Debugger for Windows, both regular and 32-bit versions. Unfortunately, the normal version just hangs Windows 3.1 in DOSBox. The 32-bit Turbo Debugger loads just fine but can’t load the NE file.

  I’ve also wondered if DOSBox contains any advanced features for trapping program execution and disassembling. I haven’t looked too deeply into this yet.

  Future Work
  NE files seem to be the executable format that time forgot. I have a crazy brainstorm about repacking NE files as MZ executables so that they could be taken apart with an MZ disassembler. But this will take some experimenting.

  If anyone else has any ideas about ripping open these binaries, I would appreciate hearing them.

  And I guess I shouldn’t be too surprised to learn that all the literature in this corpus is already freely available and easily downloadable anyway. But you shouldn’t be too surprised if that doesn’t discourage me from trying to crack the format that’s keeping this particular copy of the data locked up.

• avfilter/avf_showspectrum : add terrain color map

  8 novembre 2018, par Paul B Mahol

  avfilter/avf_showspectrum : add terrain color map
  

  • [DH] doc/filters.texi
  • [DH] libavfilter/avf_showspectrum.c

• CJEU rules US cloud servers don’t comply with GDPR and what this means for web analytics

  17 juillet 2020, par Jake Thornton
  Breaking news : On July 16, 2020, the Court of Justice of the European Union (CJEU) has ruled that any cloud services hosted in the US are incapable of complying with the GDPR and EU privacy laws.

  In August 2016, the EU-US Privacy Shield framework came into effect, which “protects the fundamental rights of anyone in the EU whose personal data is transferred to the United States for commercial purposes. It allows the free transfer of data to companies that are certified in the US under the Privacy Shield.” – European Commission website

  However after today’s CJEU ruling, this Privacy Shield framework became invalidated due to significant differences between EU and US privacy laws.

  European privacy law activist Max Schrems summarises with “The Court clarified for a second time now that there is a clash between EU privacy law and US surveillance law. As the EU will not change its fundamental rights to please the NSA, the only way to overcome this clash is for the US to introduce solid privacy rights for all people – including foreigners. Surveillance reform thereby becomes crucial for the business interests of Silicon Valley.” – noyb website

  Today’s ruling also continues to spark concern into the legitimacy of US privacy laws which doesn’t fully protect people’s personal data when hosted on cloud servers based in the US.

  

  How will this affect you ?

  For any business operating a website in the EU or if you have traffic coming to your website from EU visitors, you need to know what data you’re capturing and where this data is being stored.

  Here’s what Maja Smoltczyk (Berlin’s Commissioner for Data Protection and Freedom of Information) says :

  Controllers who transfer personal data to the USA, especially when using cloud-based services, are now required to switch immediately to service providers based in the European Union or a country that can
  ensure an adequate level of data protection. The CJEU has made it refreshingly clear that data exports are not just financial decisions, as people’s fundamental rights must also be considered as a matter of priority. This ruling will put
  an end to the transfer of personal data to the USA for the sake of convenience or to cut costs.

  The controller is you (not Google) and by transferring data to the US you are at risk of being fined up to €20 million or 4% of your annual worldwide turnover for not being GDPR compliant. 

  It’s you who has to take action, not Google or other US companies. The court’s decision has immediate effect. While we assume there will be a grace period, companies should act now as finding and implementing alternatives solution can take a while. 

  Can no data be exported outside the EU anymore ?

  Data can still be exported outside the EU if an adequate level of data protection is guaranteed. This is the case for some trading partners of the EU such as New Zealand, Japan, Switzerland, and Canada. They have been certified by the EU as having a comparable level of privacy protection and therefore demonstrate adequacy at a country level.

  Necessary data can still flow to countries like the US too. This is for example the case when someone books a hotel in the US or when sending an email to someone in the US. Backups for disaster recovery and most other reasons don’t qualify as necessary.

  In all other cases you can still send data to countries like the US if you get explicit and informed consent from a user. Meaning the user has been informed about all possible risks of sending the data to the US and who can access the data (for example the US government).

  How this affects Google Analytics and Google Tag Manager users

  If your website is using Google Analytics, the safest bet is to deactivate it immediately. Otherwise, you must ask for consent from everyone who visits your website and inform them that the data will be processed in the United States under less strict privacy laws and all associated risks. If you don’t, you could be liable to privacy law infringements and face being fined for not complying with the GDPR. This also applies to Google Tag Manager as it transfers the IP address to the US which is considered personal data under the GDPR.

  Consent needs to be :

  • Freely given (the user must have a choice to not give consent and be able to opt out at any time) 
  • Informed (you need to disclose who is processing the data, what data is processed, where the data will be stored and how to opt out) 
  • Specific (consent is only valid for the specific informed purpose) 
  • Unambiguous (for example pre-ticked boxes or similar aren’t allowed)

  

  If users don’t give you consent, you are not allowed to track them using Google Analytics or any other US based cloud solution.

  Update August 19, 2020

  A month after this ruling, over 100 complaints have been filed against websites for continuing to send data to the US via Google Analytics or Facebook, by the European privacy campaign group noyb. It’s clear Google and Facebook fall under US surveillance laws such as FISA 702 and the court clearly ruled these companies cannot rely on SCCs to transfer data to the US. Anyone still using Google Analytics is now at risk of facing fines and compensation damages. 

  How this affects Matomo users

  Our cloud servers are based in Germany.

  Matomo On-Premise users choose the location of their data themselves. If the servers are located in the EU nothing changes. If the servers are located outside the EU and the website targets EU users and tracks personal data, then you need to assess whether you are required to ask for tracking consent.

  If the data is stored inside the EU you can use Matomo without asking for any consent and you can continue tracking users even if they reject a consent screen which greatly increases the quality of your data.

  Want to avoid informing users about transferring their data to the US and all associated risks ?

  Try Matomo now for free ! No credit card required.

  Download Matomo