Recherche avancée

Sur d’autres sites (5946)

• MediaCodec hardware decoder much slower with different server configuration ?

  19 octobre 2013, par mathieujofis

  I've been using the Android MediaCodec in order to (hardware) decode H.264 frames on my Galaxy S4 coming from a Live 555 RTSP live (real-time) stream. After changing my Live 555 server configuration from using ffmpeg (with x264) to encode frames, to using strictly x264 to encode frames, the time to decode frames with MediaCodec takes much longer. Basically, MediaCodec can't keep up with the stream, and displays the video in slow motion, getting slower and slower as time goes on. Going back to ffmpeg isn't a solution for me, because I need the ability to encode into discrete NAL units, rather than a whole frame like ffmpeg does.

  I was wondering if this was either : A) An issue with the way my server is encoding NAL units, or B) An issue with my Android client, specifically the way it is receiving and decoding NAL units.

  My encoding configuration with x264 is :

  x264_param_default_preset(&param,"ultrafast", "zerolatency:fastdecode");
  
  param.i_threads = 1;
  
  param.i_bframe = 1;
  
  param.i_width = image_width;
  
  param.i_height = image_height;
  
  param.i_fps_num = 60;
  
  param.i_fps_den = 1;
  
  param.i_keyint_max = 10;
  
  
  
  param.rc.i_rc_method = X264_RC_ABR;
  
  param.rc.i.bitrate = 6000;
  
  param.i_sps_id = 7;
  
  param.b_repeat_headers = 1;
  
  param.b_annexb = 0;

  My Android MediaCodec client is set up as follows :

  I receive each individual NAL unit on a separate Live 555 RTSP client thread. Each NAL is put into a queue along with its size and presentation time. A separate decoder thread grabs NALs from this queue, and if there are none available, waits until there are.

  Some notes :

  What I generally see happen is the queue starts filling up with NALs, instead of staying close to empty. So, I know that the decoder thread is not working fast enough. I don't think this is an inherent problem with decoding on an Android phone (for example, processing limitations) because it does the same thing for very low bitrates— also again, it DID work when I was using ffmpeg to encode. If I omit certain NAL units, the decoder can start to keep up. Since I'm using Cyanogenmod 10.1, bumping up the minimum CPU frequency helps, too.

  Edit :

  Here is a log of the Android client, as well as a log highlighting the garbage collector specifically—

  Entire Logcat :

  10-15 16:40:03.955: D/DecodeActivity(18859): INFO_OUTPUT_BUFFERS_CHANGED
  
  10-15 16:40:03.995: E/OMX-VDEC-1080P(288): Sync frame received
  
  10-15 16:40:03.995: E/OMX-VDEC-1080P(288):  No color conversion required
  
  10-15 16:40:03.995: E/OMX-VDEC-1080P(288): Get_parameter: OMX_IndexParamPortDefinition: nPortIndex (1), nFrameWidth (1280), nFrameHeight (720), nStride (1280), nSliceHeight (736), nBitrate (-1073741824), xFramerate (0x1e), nBufferSize (1433600), nBufferCountMin (4), nBufferCountActual (8), bBuffersContiguous (1918394328), nBufferAlignment (1075643347), bEnabled (1), bPopulated (1), eCompressionFormat (0x0), eColorFormat (0x7fa30c03)
  
  10-15 16:40:03.995: D/DecodeActivity(18859): New format {height=720, what=1869968451, color-format=2141391875, slice-height=736, crop-left=0, width=1280, crop-bottom=719, crop-top=0, mime=video/raw, stride=1280, crop-right=1279}
  
  10-15 16:40:04.005: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4966) fps(201.369308)
  
  10-15 16:40:04.015: W/IInputConnectionWrapper(1069): showStatusIcon on inactive InputConnection
  
  10-15 16:40:04.025: I/ActivityManager(698): Displayed com.mathieu.alloclient.javadecoder/.MainActivity: +617ms
  
  10-15 16:40:04.025: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4766) fps(209.819550)
  
  10-15 16:40:04.125: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4733) fps(211.282486)
  
  10-15 16:40:04.445: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4711) fps(212.269150)
  
  10-15 16:40:04.495: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4700) fps(212.765961)
  
  10-15 16:40:05.676: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4688) fps(213.310577)
  
  10-15 16:40:06.087: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 1ms
  
  10-15 16:40:06.207: D/dalvikvm(698): GC_EXPLICIT freed 4120K, 39% free 24216K/39664K, paused 7ms+9ms, total 117ms
  
  10-15 16:40:06.537: D/ALSADevice(288): standby: handle 0x40024450 h 0x0
  
  10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _verb value Inactive
  
  10-15 16:40:06.577: D/alsa_ucm(288): Set mixer controls for HiFi Lowlatency enable 0
  
  10-15 16:40:06.577: D/alsa_ucm(288): Setting mixer control: SLIMBUS_0_RX Audio Mixer MultiMedia5, value: 0
  
  10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _disdev value Line
  
  10-15 16:40:06.577: D/alsa_ucm(288): disdev: device Line not enabled, no need to disable
  
  10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _disdev value Speaker
  
  10-15 16:40:06.577: D/alsa_ucm(288): Set mixer controls for Speaker enable 0
  
  10-15 16:40:06.577: D/alsa_ucm(288): Setting mixer control: RX5 MIX1 INP1, value: ZERO
  
  10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: RX5 MIX1 INP2, value: ZERO
  
  10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: LINEOUT2 Volume, value: 0
  
  10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: LINEOUT4 Volume, value: 0
  
  10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: RX5 Digital Volume, value: 0
  
  10-15 16:40:06.587: D/AudioUsbALSA(288): exitPlaybackThread, mproxypfdPlayback: -1
  
  10-15 16:40:06.587: D/AudioUsbALSA(288): closeDevice handle 0x0
  
  10-15 16:40:06.587: D/AudioUsbALSA(288): closeDevice handle 0x0
  
  10-15 16:40:17.638: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4677) fps(213.812271)
  
  10-15 16:40:17.698: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4644) fps(215.331604)
  
  10-15 16:40:20.681: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4633) fps(215.842865)
  
  10-15 16:40:21.111: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4611) fps(216.872696)
  
  10-15 16:40:25.746: D/dalvikvm(698): GC_CONCURRENT freed 5829K, 41% free 23778K/39664K, paused 10ms+24ms, total 165ms
  
  10-15 16:40:28.448: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.800000 hotplug_avg_load_dw: 74
  
  10-15 16:40:28.448: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:243.000000
  
  10-15 16:40:30.841: W/SystemClock(698): time going backwards: prev 16555345563411(ioctl) vs now 16555345441341(ioctl), tid=764
  
  10-15 16:40:44.684: D/dalvikvm(698): GC_CONCURRENT freed 5302K, 41% free 23774K/39664K, paused 6ms+9ms, total 107ms
  
  10-15 16:40:57.467: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4577) fps(218.483719)
  
  10-15 16:41:14.383: D/dalvikvm(698): GC_CONCURRENT freed 5371K, 41% free 23768K/39664K, paused 7ms+8ms, total 146ms
  
  10-15 16:41:14.403: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.500000 hotplug_avg_load_dw: 89
  
  10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
  
  10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
  
  10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:922.000000
  
  10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 48
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_HELLO
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: SENDING --> SAHARA_HELLO_RESPONSE
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: sahara_mode                         = 2
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: m_comm->sahara_hello_packet_rx.mode = 2
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: helloRx.mode                        = 2
  
  10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 64
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_MEMORY_DEBUG
  
  10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 116
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: 0x46980000, len=000C0000, "m9kefs1", ""
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_MEMORY_REGION
  
  10-15 16:41:27.466: E/kickstart(862): EVENT: Saving "/dev/block/platform/msm_sdcc.1/by-name/m9kefs1"
  
  10-15 16:41:27.526: E/kickstart(862): Total bytes received so far: 786548
  
  10-15 16:41:27.526: E/kickstart(862): EVENT: Received: 786432 bytes
  
  10-15 16:41:27.526: E/kickstart(862): EVENT: Writing to disk
  
  10-15 16:41:27.526: E/kickstart(862): EVENT: Successfully wrote to disk
  
  10-15 16:41:27.526: E/kickstart(862): Received file "m9kefs1"
  
  10-15 16:41:27.576: E/kickstart(862): Sync finish Received file "m9kefs1"
  
  10-15 16:41:27.576: E/kickstart(862): 786432 bytes transferred in 0.106s (7.10 MBps)
  
  10-15 16:41:27.576: E/kickstart(862): EVENT: num_debug_entries not >=0
  
  10-15 16:41:27.576: E/kickstart(862): Successfully downloaded files from target 
  
  10-15 16:41:27.576: E/kickstart(862): EVENT: SENDING --> SAHARA_RESET
  
  10-15 16:41:27.576: E/kickstart(862): Total bytes received so far: 786556
  
  10-15 16:41:27.576: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_RESET_RESP
  
  10-15 16:41:27.576: E/kickstart(862): Sahara protocol completed
  
  10-15 16:41:27.576: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_HELLO
  
  10-15 16:41:27.746: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.100000 hotplug_avg_load_dw: 79
  
  10-15 16:41:27.746: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
  
  10-15 16:41:49.598: D/dalvikvm(698): GC_CONCURRENT freed 5339K, 41% free 23775K/39664K, paused 10ms+7ms, total 134ms
  
  10-15 16:42:17.225: I/ActivityManager(698): Start proc com.cyanogenmod.lockclock for service com.cyanogenmod.lockclock/.weather.WeatherUpdateService: pid=18954 uid=10028 gids={50028, 3003, 1028}
  
  10-15 16:42:17.865: D/WeatherXmlParser(18954): Weather updated: WeatherInfo for Santa Barbara@ Tue Oct 15 16:42:17 PDT 2013: Fair(34), temperature 29°C, low 11°, high 27°, humidity 14%, wind 11km/h at W
  
  10-15 16:42:17.945: I/ActivityManager(698): No longer want com.google.android.apps.uploader (pid 13565): empty #17
  
  10-15 16:42:24.021: D/dalvikvm(698): GC_CONCURRENT freed 5277K, 41% free 23770K/39664K, paused 8ms+11ms, total 106ms
  
  10-15 16:42:35.983: W/ThrottleService(698): unable to find stats for iface rmnet0
  
  10-15 16:42:59.476: D/dalvikvm(698): GC_CONCURRENT freed 5345K, 41% free 23770K/39664K, paused 6ms+13ms, total 168ms
  
  10-15 16:43:02.098: E/kickstart(862): Total bytes received so far: 48
  
  10-15 16:43:02.098: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_HELLO
  
  10-15 16:43:02.098: E/kickstart(862): EVENT: SENDING --> SAHARA_HELLO_RESPONSE
  
  10-15 16:43:02.098: E/kickstart(862): EVENT: sahara_mode                         = 2
  
  10-15 16:43:02.098: E/kickstart(862): EVENT: m_comm->sahara_hello_packet_rx.mode = 2
  
  10-15 16:43:02.098: E/kickstart(862): EVENT: helloRx.mode                        = 2
  
  10-15 16:43:02.098: E/kickstart(862): Total bytes received so far: 64
  
  10-15 16:43:02.098: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_MEMORY_DEBUG
  
  10-15 16:43:02.108: E/kickstart(862): Total bytes received so far: 116
  
  10-15 16:43:02.108: E/kickstart(862): EVENT: 0x46980000, len=000C0000, "m9kefs2", ""
  
  10-15 16:43:02.108: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_MEMORY_REGION
  
  10-15 16:43:02.108: E/kickstart(862): EVENT: Saving "/dev/block/platform/msm_sdcc.1/by-name/m9kefs2"
  
  10-15 16:43:02.158: E/kickstart(862): Total bytes received so far: 786548
  
  10-15 16:43:02.158: E/kickstart(862): EVENT: Received: 786432 bytes
  
  10-15 16:43:02.158: E/kickstart(862): EVENT: Writing to disk
  
  10-15 16:43:02.168: E/kickstart(862): EVENT: Successfully wrote to disk
  
  10-15 16:43:02.168: E/kickstart(862): Received file "m9kefs2"
  
  10-15 16:43:02.218: E/kickstart(862): Sync finish Received file "m9kefs2"
  
  10-15 16:43:02.218: E/kickstart(862): 786432 bytes transferred in 0.113s (6.65 MBps)
  
  10-15 16:43:02.218: E/kickstart(862): EVENT: num_debug_entries not >=0
  
  10-15 16:43:02.218: E/kickstart(862): Successfully downloaded files from target 
  
  10-15 16:43:02.218: E/kickstart(862): EVENT: SENDING --> SAHARA_RESET
  
  10-15 16:43:02.218: E/kickstart(862): Total bytes received so far: 786556
  
  10-15 16:43:02.218: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_RESET_RESP
  
  10-15 16:43:02.218: E/kickstart(862): Sahara protocol completed
  
  10-15 16:43:02.218: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_HELLO
  
  10-15 16:43:17.563: W/SystemClock(698): time going backwards: prev 16722067121029(ioctl) vs now 16722066968441(ioctl), tid=764
  
  10-15 16:43:34.610: D/dalvikvm(698): GC_CONCURRENT freed 5359K, 41% free 23772K/39664K, paused 9ms+9ms, total 113ms
  
  10-15 16:43:36.452: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.900000 hotplug_avg_load_dw: 100
  
  10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
  
  10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
  
  10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:194.000000
  
  10-15 16:44:09.965: D/dalvikvm(698): GC_CONCURRENT freed 5369K, 41% free 23779K/39664K, paused 8ms+8ms, total 103ms
  
  10-15 16:44:09.965: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 30ms
  
  10-15 16:44:10.605: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.800000 hotplug_avg_load_dw: 133
  
  10-15 16:44:10.605: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
  
  10-15 16:44:12.707: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.100000 hotplug_avg_load_dw: 116
  
  10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
  
  10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
  
  10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
  
  10-15 16:44:14.008: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:1.700000 hotplug_avg_load_dw: 140
  
  10-15 16:44:14.008: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:240.000000
  
  10-15 16:44:14.759: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.900000 hotplug_avg_load_dw: 132
  
  10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
  
  10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
  
  10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
  
  10-15 16:44:15.360: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.300000 hotplug_avg_load_dw: 139
  
  10-15 16:44:15.360: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:243.000000
  
  10-15 16:44:15.560: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.900000 hotplug_avg_load_dw: 133
  
  10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
  
  10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
  
  10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:95.000000
  
  10-15 16:44:16.361: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.500000 hotplug_avg_load_dw: 132
  
  10-15 16:44:16.361: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:242.000000
  
  10-15 16:44:16.661: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.700000 hotplug_avg_load_dw: 127
  
  10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
  
  10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
  
  10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
  
  10-15 16:44:20.605: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:1.700000 hotplug_avg_load_dw: 122
  
  10-15 16:44:20.605: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
  
  10-15 16:44:20.615: W/ProcessStats(698): Skipping unknown process pid 19038
  
  10-15 16:44:20.615: W/ProcessStats(698): Skipping unknown process pid 19041

  Garbage collector entries in Logcat :

  10-15 16:40:06.087: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 1ms
  
  10-15 16:40:06.207: D/dalvikvm(698): GC_EXPLICIT freed 4120K, 39% free 24216K/39664K,   paused 7ms+9ms, total 117ms
  
  10-15 16:40:25.746: D/dalvikvm(698): GC_CONCURRENT freed 5829K, 41% free 23778K/39664K, paused 10ms+24ms, total 165ms
  
  10-15 16:40:44.684: D/dalvikvm(698): GC_CONCURRENT freed 5302K, 41% free 23774K/39664K, paused 6ms+9ms, total 107ms
  
  10-15 16:41:14.383: D/dalvikvm(698): GC_CONCURRENT freed 5371K, 41% free 23768K/39664K, paused 7ms+8ms, total 146ms
  
  10-15 16:41:49.598: D/dalvikvm(698): GC_CONCURRENT freed 5339K, 41% free 23775K/39664K, paused 10ms+7ms, total 134ms
  
  10-15 16:42:24.021: D/dalvikvm(698): GC_CONCURRENT freed 5277K, 41% free 23770K/39664K, paused 8ms+11ms, total 106ms
  
  10-15 16:42:59.476: D/dalvikvm(698): GC_CONCURRENT freed 5345K, 41% free 23770K/39664K, paused 6ms+13ms, total 168ms
  
  10-15 16:43:34.610: D/dalvikvm(698): GC_CONCURRENT freed 5359K, 41% free 23772K/39664K, paused 9ms+9ms, total 113ms
  
  10-15 16:44:09.965: D/dalvikvm(698): GC_CONCURRENT freed 5369K, 41% free 23779K/39664K, paused 8ms+8ms, total 103ms
  
  10-15 16:44:09.965: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 30ms
  
  10-15 16:44:24.389: D/dalvikvm(698): GC_EXPLICIT freed 3618K, 41% free 23768K/39664K, paused 12ms+11ms, total 123ms

• Use Google Analytics and risk fines, after CJEU ruling on Privacy Shield

  27 août 2020, par Joselyn Khor — Privacy
  EU websites using Google Analytics and Facebook are being targeted by European privacy group noyb after the invalidation of the Privacy Shield. They filed a complaint against 101 websites for continuing to send data to the US. 

  “A quick analysis of the HTML source code of major EU webpages shows that many companies still use Google Analytics or Facebook Connect one month after a major judgment by the Court of Justice of the European Union (CJEU) - despite both companies clearly falling under US surveillance laws, such as FISA 702. Neither Facebook nor Google seem to have a legal basis for the data transfers.”

  

  The Privacy Shield previously allowed for EU data to be transferred to the US. However, this was invalidated by the Court of Justice of the European Union (CJEU) on July 16, 2020. The CJEU deemed it illegal for any websites to transfer the personal data of European citizens to the US. 

  They also made it clear in a press release that “data subjects can claim compensation for inadmissible data exports (marginal no. 143 of the judgment). This should in particular include non-material damage (“compensation for pain and suffering”) and must be of a deterrent amount under European law.” Which puts extra financial pressure on websites to take the new ruling seriously.

  Immediate action is required after Google Privacy Shield invalidation

  The Berlin Commissioner for Data Protection and Freedom of Information therefore calls on all those responsible under its supervision to observe the decision of the ECJ [CJEU]. Those responsible who transfer personal data to the USA - especially when using cloud services - are now required to immediately switch to service providers in the European Union or in a country with an adequate level of data protection.

  As the ruling is effective immediately, there’s a pressing need for websites using Google Analytics to act, or face getting fined.

  What does this mean for you ?

  If you’re using Google Analytics the safest bet is to stop using it immediately. 

  "Neither Google Analytics nor Facebook Connect are necessary for the operation of these websites and could therefore have been replaced or at least deactivated in the meantime."

  If you still need to use it, then you’ll need to inform your visitors via a clear consent screen. This banner needs to make clear their personal data will be sent to the US, and to educate them about any potential risk related to this. They will then need to explicitly agree to this. 

  Another downside of cookie consent screens is that you may also suffer a damaging loss of visitors. After implementing cookie consent best practices, the UK’s data regulator the Information Commissioner’s Office (ICO) found a 90% drop in traffic, “implying a ninety percent drop in opt-in rates.”

  With an acceptance rate for such consent screens being lower than 10% your analytics becomes guesswork rather than science. 

  Looking for a privacy-respecting alternative to Google Analytics ?

  Privacy compliant Matomo Analytics is one of the best Google Analytics alternatives availalble. 

  With Matomo you’re able to continue using analytics without facing the wrath of both the GDPR and the CJEU. Matomo On-Premise lets you choose where your data is stored, so you can ensure no data is processed in the US. 

  Matomo is privacy-friendly and can be tweaked to comply with all privacy laws. Including the GDPR, HIPAA, CCPA and PECR. The benefits of this include : not needing to use tracking or cookie consent screens (like with GA) ; and avoiding fines because no personal data is collected. You also get 100% accurate data and the ability to protect your user’s privacy.

  Download Matomo

  

  Is your EU business at risk of being fined for using Google Analytics ?

  Email

  Get our assessment guide

• How to not process any personal data with Matomo and what it means for you

  22 avril 2018, par InnoCraft

  Disclaimer : this blog post has been written by digital analysts, not lawyers. The purpose of this article is to explain how to not process any personal data with Matomo in order to avoid going through the GDPR compliance process with Matomo analytics. This work comes from our interpretation of different sources : the official GDPR text and the UK privacy commission : ICO resources. It cannot be considered as a professional legal advice. So as GDPR, this information is subject to change. GDPR may be also known as RGPD in French, Spanish, Portuguese, Datenschutz-Grundverordnung, DS-GVO in German, Algemene verordening gegevensbescherming in Dutch, Regolamento generale sulla protezione dei dati in Italian.

  Are you looking for a way to not process any personal data with Matomo ? If the answer is yes, you are at the right place. From our understanding, if you are not processing personal data, then you shouldn’t be concerned about GDPR. Our inspiration came from this official reference :

  “The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable. This Regulation does not therefore concern the processing of such anonymous information, including for statistical or research purposes.“

  In this blog post we are going to see how you can configure Matomo in order to not process any personal data and what the consequences are.

  Which data is considered as personal according to GDPR ?

  From : eur-lex.europa.eu

  (1) “‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’) ; an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person ;”

  (30) “Natural persons may be associated with online identifiers provided by their devices, applications, tools and protocols, such as internet protocol addresses, cookie identifiers or other identifiers such as radio frequency identification tags. This may leave traces which, in particular when combined with unique identifiers and other information received by the servers, may be used to create profiles of the natural persons and identify them.”

  So according to your Matomo configuration, it may leave some traces within the following data :

  1. IP addresses
  2. Cookies identifiers
  3. Page URL or page titles
  4. User ID and Custom “personal” data
  5. Ecommerce order IDs
  6. Location
  7. Heatmaps & Session Recordings

  Let’s see each of them in more detail.

  1. IP addresses

  IP addresses can indirectly identify an individual. It can also give a good approximation of an individual’s location.

  IP addresses are therefore considered as personal data which means you need to anonymize them. To do so, a feature is available within Matomo, where you can anonymize the IP. We recommend you to anonymize at least the last two bytes :

  

  See our configuration guide for more information

  What are the consequences of using this feature ?

  When applying IP anonymization on two bytes, you will no longer be able to see the full IP in the UI.

  Moreover, there is a small chance that 2 different visitors with the same device and software configuration will be identified as the same visitor if the anonymised IP address is the same for both.

  2. Cookies

  It is not clear for us yet if all cookies are considered equal under GDPR. At this stage it is too early to make a definite decision.

  Did you know ? Matomo lets you optionally disable the creation of cookies by adding an extra line of code to your tracking code see below.

  

  See our configuration guide for more information

  What are the consequences of using this feature ?

  Matomo is using a few first party cookies, and the following cookies may hold personal data :

  • _pk_id : contains a visitor id used to identify unique visitors
  • _pk_ref : to identify from where they came from

  If Matomo cannot set cookies, it will use a technique called Fingerprint. It is based on several metadata such as the operating system, browser, browser plugins, IP address, browser language ; just to name a few to identify a unique visitor. As this feature is less accurate than the one using cookies, the number of visitors and visits will be affected.

  3. Page URLs and page titles

  URLs are not mentioned within the official GDPR text. However, we know that according to the different CMS you use, some of them may have URLs including personal identifiers.

  For example :

  

  As a result, you need to find a way to anonymize this data.

  There are several ways you can perform this action according to your website. If your website is adding the personal data through query parameters, you can define a rule to exclude them from Matomo.

  If the personal data are not included within query parameters, you can use the “setCustomURL” feature and write your code as follow :

  

  See our developer documentation for more information

  If you are also processing personal data within the title tag, you can use the following function : “setDocumentTitle”.

  What are the consequences of using this feature ?

  By anonymizing the URLs containing personal data, some of your  URLs will be grouped together.

  4. User ID and custom personal data

  User ID is a feature (a tracking code needs to be added) which allows you to identify the same user across different devices.

  A User ID needs a corresponding database in order to link a user across different devices, it can be an email, a username, a name, a random number… All those data are either direct or non direct online identifiers and are therefore under the scope of GDPR.

  It will be the same situation if you are using custom variables and/or custom dimensions in order to push personal data to the system.

  To continue using the User ID feature but not recording personal data, you can consider using a hash function which will anonymize/convert your actual User ID into something like “3jrj3j34434834urj33j3”.

  Alternatively, you can enable the feature “Anonymise User IDs”. This feature will be available starting in Matomo 3.5.0 :

  

  What are the consequences of using this feature ?

  Under GDPR, User ID is personal data. Anonymizing the User ID using a hash function or our built-in functionality make the User Id pseudo-anonymous, which means it can’t be easily identified to a specific user. As a result, you will still get accurate visits and unique visitors metrics, and the Visitor Profile, but without tracking the original User ID which is personal data.

  5. Ecommerce order IDs

  Order IDs are the reference number assigned to the products/services bought by your customers. As this information can be crossed with your internal database, it is considered as an online identifier and is therefore under the scope of GDPR. As for User ID, you can anonymize order IDs using our built-in functionality to Anonymise Order IDs (see section 4. about User Id).

  What are the consequences of anonymizing order ID ?

  It really depends on your former use of order IDs. If you were not using them in the past then you should not see any difference.

  6. Location

  Based on the IP address of a visitor, Matomo can detect the visitors location. Location data is problematic for privacy as this technology has become quite accurate and can detect not only the city a visitor is from, but sometimes an even more precise position of a visitor.

  

  In order to not leave any accurate traces, we strongly recommend you to enable the IP anonymization feature. Next, you need to enable the setting “Also use the anonymized IP address when enriching visits”. You find this setting directly below the IP anonymization. This is important as otherwise the full IP address will be used to geolocate a visitor.

  

  What are the consequences of anonymizing location data ?

  The more bytes you anonymize from the IP, the more anonymized your location will be. When you remove two bytes as suggested, the city and region location reports will not be as accurate. In some cases even the country may not be detected correctly anymore.

  7. Heatmaps & Session Recordings

  Heatmaps & Session Recording is a premium feature in Matomo allowing you to see where users click, hover, type and scroll. With session recordings you can then replay their actions in a video.

  Heatmaps & Session Recordings are under the scope of GDPR as they can disclose in some specific cases (for example : filling a contact form) personal data :

  

  To avoid this, Matomo will anonymize all keystrokes which a user enters into a form field unless you specifically whitelist a field. Many fields that could contain personal data, such as a credit card, phone number, email address, password, social security number, and more are always anonymized and not recorded.

  See our configuration guide for more information

  Note that a page may still show personal information within the page as part of regular content (not a form element). For example an address, or the profile page of a forum user. We have added a feature which allows you to set an HTML attribute “data-matomo-mask” to anonymize any personal content shown in the UI.

  What are the consequences of using this feature ?

  Mainly, you will not be able to see in plain text what people are entering into your forms.

  What should you do with past data ?

  Once more, we have to say that we are not lawyers. So do not take our answers as legal advice. From : ec.europa.eu/newsroom/article29/document.cfm ?doc_id=50053

  “For example, as the GDPR requires that a controller must be able to demonstrate that valid consent was obtained, all presumed consents of which no references are kept will automatically be below the consent standard of the GDPR and will need to be renewed.”

  Our interpretation is that, if you were previously relying on consent, unless you can demonstrate that valid consent was obtained, you need to get the consent back (which is almost impossible) or you need to anonymize or remove that data.

  To anonymize previously tracked data, we are actively working on a feature to do just that directly within Matomo. Alternatively, you may also set up the deletion of logs after a certain amount of time.

  We really hope you enjoyed reading this article. GDPR is still on the go and we are pretty sure you have a lot of questions about it. You probably would like to share our vision about it. So do not hesitate to ask us through our contact form to see how we are interpreting GDPR at Matomo and InnoCraft.

  The post How to not process any personal data with Matomo and what it means for you appeared first on Analytics Platform - Matomo.