Recherche avancée

Sur d’autres sites (3642)

• Anomalie #3217 (Nouveau) : Caractères spéciaux d’expression régulière présents dans une chaîne de ...

  22 mai 2014, par Pascal Verrier

  Bonjour,

  J’ai une boucle au sein de l’affichage d’un forum qui permet d’afficher les autres posts similaires ; pour cela, l’argument recherche est renseigné avec le titre du forum en cours d’affichage.

  Cela fonctionne bien sur la plupart des pages sauf sur un forum ayant le titre suivant : Sortie du kit compatible SPIP 3 ??

  J’ai une erreur MySQL 1139 :
  

  Got error ’repetition-operator operand invalid’ from regexp
  SELECT t.id_forum, t.titre, t.texte, t.auteur, t.email_auteur, t.nom_site, t.url_site FROM spip_forum AS t WHERE t.titre REGEXP ’Sortie du kit compatible SPIP 3 ??|Sortie|compatible|SPIP’ OR ...
  

  Ce sont les ?? qui coincent car manifestament ils sont interprétés en tant que métacaractères dans la regex, n’ayant pas été échappés.

  J’ai fait un autre essai en mettant d’autres opérateurs de regex : Sortie kit[0-9]* compatib(ilité|le) SPIP 3 ??, qui confirme ce que je pensais :
  

  Erreur SQL 1139
  Got error ’repetition-operator operand invalid’ from regexp
  SELECT ... WHERE t.titre REGEXP ’Sortie kit[0-9]* compatib(ilite|le) SPIP 3 ??|Sortie|kit[0-9]*|compatib(ilite|le)|SPIP’ OR ...
  

  à savoir la chaîne de recherche n’est pas prétraitée, et les caractères spéciaux sont repris tels quels, d’où le plantage MySQL.

  Je propose donc les corrections ci-joint pour ecrire/inc/rechercher.php (commentaires avec pverrier).

• ffmpeg "Manifest too large" when downloading youtube video

  22 avril 2024, par ynn

  TL ;DR

  


  I'm trying to programmatically download a part of a YouTube video. The widely known procedure doesn't work for some videos and I'd like to overcome this situation.

  


  ---

  


  Context

  


  I'm trying to programmatically download a part of a YouTube video. As described in How to download portion of video with youtube-dl command ?, you can achieve this by the following commands.

  


  #Converts a human-readable URL to longer URLs for internal use.
~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=POrfo478HRI'
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes
https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes

#Stores the result to a variable.
~ $ url_video_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=299&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278%2C298%2C299%2C302%2C303&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=video%2Fmp4&gir=yes&clen=9205054989&dur=17931.000&lmt=1597347724488304&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgYDqTm0CnLfG0kxsYNrAdtAUYB7alnowiaBArU8R5bBwCIQDeCRDJHFO_PWcnbeFaJvip80deuboN4Pi1x3eRhJBxlQ%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Stores the result to a variable.
~ $ url_audio_part='https://r1---sn-3pm76n76.googlevideo.com/videoplayback?expire=1597593009&ei=UQE5X7bXEYqO4QLs17KACQ&ip=10.100.238.99&id=o-AD0HfadoeimIbuf9t8Anru5X9V7cER08YyOz4iKZOCHL&itag=251&source=youtube&requiressl=yes&mh=NV&mm=31%2C26&mn=sn-3pm76n76%2Csn-oguesnze&ms=au%2Conr&mv=m&mvi=1&pl=17&initcwndbps=908750&vprv=1&mime=audio%2Fwebm&gir=yes&clen=268945126&dur=17931.001&lmt=1597348111385641&mt=1597571322&fvip=1&keepalive=yes&fexp=23883098&c=WEB&txp=7311222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAK8XGu3vHucHwqERZ7TQniuvKEd-NTbMkwZbu8EXJ5E3AiEAiS7OYFOcsIrD2xp-AJNzucj1H9ZKMlmkCl_sU7__dZo%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKTkyTIGVkbWWVx4lqFz77rOdG1FzkM5XuTnYOqtHT4oAiEA9u1R47jPz-mjj4DRXpqqqpUbfaFMW36KbG_pR2jovFM%3D&ratebypass=yes'

#Downloads the video for one minute from 12:45.
~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
frame= 3600 fps= 49 q=-1.0 Lsize=   27047kB time=00:00:59.99 bitrate=3692.9kbits/s speed=0.813x


  


  This does work perfectly.

  


  ---

  


  Problem

  


  For some videos^†1, the internal URLs are of the form https://manifest.googlevideo.com/... instead of https://RANDOM-STRING.googlevideo.com/... as in the example above.

  


  ~ $ youtube-dl --get-url 'https://www.youtube.com/watch?v=KRMfnvLODZQ'
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D
https://manifest.googlevideo.com/api/manifest/dash/expire/1597593866/ei/qgQ5X5maN8-ilQSu15W4CQ/ip/10.100.238.99/id/KRMfnvLODZQ.1/source/yt_live_broadcast/requiressl/yes/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/force_finished/1/vprv/1/keepalive/yes/fexp/23883098/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Chfr%2Cas%2Cforce_finished%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRQIhANYKQvKTN8BRmhp7--4tQp9aqRp_qIGjh472BTYLyJ62AiAautFF86xGDHOe7mYlYaZ1W86EM4k2e6UiJS9Q91dMog%3D%3D


  


  And downloading such videos always fails.

  


  ~ $ ffmpeg    -ss 12:45 -i "${url_video_part}"    -ss 12:45 -i "${url_audio_part}"    -t 1:00 output.mkv
[dash @ 0x55fa6a1a4400] Manifest too large: 3776722
https://manifest.googlevideo.com/...: Invalid data found when processing input


  


  Please note downloading full of such videos using youtube-dl alone succeeds.

  


  _{†1 : As far as we tested, recently (e.g. yesterday) uploaded videos have such URLs and when we retry the same youtube-dl --get-url command after a day or so, we get "innocent" URLs like in the first example.}

  


  ---

  


  Questions

  


  How can I download parts of such videos

  


  


  • by specifying additional options for ffmpeg,

    


  


  • by somehow forcefully retrieving internal URLs of the form https://RANDOM-STRING.googlevideo.com/...,

    


  


  • or by something which I have no idea at all ?

    


  


  


  My current workaround is just to wait for a day or two until the internal URLs change, which is much slower than downloading the full video and then cropping it.

  


  ---

  


  Environments

  


  ~ $ lsb_release -d
Description:    Arch Linux

~ $ ffmpeg -version
ffmpeg version n4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
built with gcc 10.1.0 (GCC)
configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libdav1d --enable-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librav1e --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-nvdec --enable-nvenc --enable-omx --enable-shared --enable-version3
libavutil      56. 51.100 / 56. 51.100
libavcodec     58. 91.100 / 58. 91.100
libavformat    58. 45.100 / 58. 45.100
libavdevice    58. 10.100 / 58. 10.100
libavfilter     7. 85.100 /  7. 85.100
libswscale      5.  7.100 /  5.  7.100
libswresample   3.  7.100 /  3.  7.100
libpostproc    55.  7.100 / 55.  7.100

~ $ youtube-dl --version
2020.07.28


  


• Understanding Data Processing Agreements and How They Affect GDPR Compliance

  9 octobre 2023, par Erin — GDPR

  The General Data Protection Regulation (GDPR) impacts international organisations that conduct business or handle personal data in the European Union (EU), and they must know how to stay compliant.

  One way of ensuring GDPR compliance is through implementing a data processing agreement (DPA). Most businesses overlook DPAs when considering ways of maintaining user data security. So, what exactly is a DPA’s role in ensuring GDPR compliance ?

  In this article, we’ll discuss DPAs, their advantages, which data protection laws require them and the clauses that make up a DPA. We’ll also discuss the consequences of non-compliance and how you can maintain GDPR compliance using Matomo.

  What is a data processing agreement ?

  A data processing agreement, data protection agreement or data processing addendum is a contractual agreement between a data controller (a company) and a data processor (a third-party service provider.) It defines each party’s rights and obligations regarding data protection.

  A DPA also defines the responsibilities of the controller and the processor and sets out the terms they’ll use for data processing. For instance, when MHP/Team SI sought the services of Matomo (a data processor) to get reliable and compliant web analytics, a DPA helped to outline their responsibilities and liabilities.

  A DPA is one of the basic requirements for GDPR compliance. The GDPR is an EU regulation concerning personal data protection and security. The GDPR is binding on any company that actively collects data from EU residents or citizens, regardless of their location.

  As a business, you need to know what goes into a DPA to identify possible liabilities that may arise if you don’t comply with European data protection laws. For example, having a recurrent security incident can lead to data breaches as you process customer personal data.

  The average data breach cost for 2023 is $4.45 million. This amount includes regulatory fines, containment costs and business losses. As such, a DPA can help you assess the organisational security measures of your data processing methods and define the protocol for reporting a data breach.

  Why is a DPA essential for your business ?

  If your company processes personal data from your customers, such as contact details, you need a DPA to ensure compliance with data security laws like GDPR. You’ll also need a DPA to hire a third party to process your data, e.g., through web analytics or cloud storage.

  But what are the benefits of having a DPA in place ?

  

  A key benefit of signing a DPA is it outlines business terms with a third-party data processor and guarantees compliance with the relevant data privacy laws. A DPA also helps to create an accountability framework between you and your data processor by establishing contractual obligations.

  Additionally, a DPA helps to minimise the risk of unauthorised access to sensitive data. A DPA defines organisational measures that help protect the rights of individuals and safeguard personal data against unauthorised disclosure. Overall, before choosing a data processor, having a DPA ensures that they are capable, compliant and qualified.

  More than 120 countries have already adopted some form of international data protection laws to protect their citizens and their data better. Hence, knowing which laws require a DPA and how you can better ensure compliance is important.

  Which data protection laws require a DPA ?

  Regulatory bodies enact data protection laws to grant consumers greater control over their data and how businesses use it. These laws ensure transparency in data processing and compliance for businesses.

  

  The following are some of the relevant data privacy laws that require you to have a DPA :

  • UK GDPR
  • Brazil LGPD
  • EU GDPR
  • Dubai PDPA
  • Colorado CPA
  • California CCPA/CPRA
  • Virginia VCDPA
  • Connecticut DPA
  • South African POPIA
  • Thailand PDPA

  Companies that don’t adhere to these data protection obligations usually face liabilities such as fines and penalties. With a DPA, you can set clear expectations regarding data processing between you and your customers.

  Review and update any DPAs with third-party processors to ensure compliance with GDPR and the laws we mentioned above. Additionally, confirm that all the relevant clauses are present for compliance with relevant data privacy laws. 

  So, what key data processing clauses should you have in your DPA ? Let’s take a closer look in the next section.

  Key clauses in a data processing agreement

  GDPR provides some general recommendations for what you should state in a DPA.

  

  Here are the elements you should include :

  Data processing specifications

  Your DPA should address the specific business purposes for data processing, the duration of processing and the categories of data under processing. It should also clearly state the party responsible for maintaining GDPR compliance and who the data subjects are, including their location and nationality.

  Your DPA should also address the data processor and controller’s responsibilities concerning data deletion and contract termination.

  Role of processor

  Your DPA should clearly state what your data processor is responsible for and liable for. Some key responsibilities include record keeping, reporting breaches and maintaining data security.

  Other roles of your data processor include providing you with audit opportunities and cooperating with data protection authorities during inquiries. If you decide to end your contract, the data processor is responsible for deleting or returning data, depending on your agreement.

  Role of controller

  Your DPA should inform the responsibilities of the data controller, which typically include issuing processing instructions to the data processor and directing them on how to handle data processing.

  Your DPA should let you define the lawful data processes the data processor should follow and how you’ll uphold the data protection rights of individuals’ sensitive data.

  Organisational and technical specifications

  Your DPA should define specifications such as how third-party processors encrypt, access and test personal data. It should also include specifications on how the data processor and controller will maintain ongoing data security through various factors such as :

  • State of the technology : Do ‌third-party processors have reliable technology, and can they ensure data security within their systems ?
  • Costs of implementation : Does the data controller’s budget allow them to seek third-party services from industry-leading providers who can guarantee a certain level of security ?
  • Variances in users’ personal freedom : Are there privacy policies and opt-out forms for users to express how they want companies to use their sensitive data ?

  Moreover, your DPA should define how you and your data processor will ensure the confidentiality, availability and integrity of data processing services and systems.

  What are the penalties for DPA GDPR non-compliance ?

  Regulators use GDPR’s stiff fines to encourage data controllers and third-party processors to follow‌ best data security practices. One way of maintaining compliance is through drafting up a DPA with your data processor.

  The DPA should clearly outline the necessary legal requirements and include all the relevant clauses mentioned above. Understand what goes into this agreement since data protection authorities can hold your business accountable for a breach — even if a processor’s error caused it.

  Data protection authorities can issue penalties now that the GDPR is in place. For example, according to Article 83 of the GDPR, penalties for data or privacy breaches or non-compliance can amount to up to €20 million or 4% of your annual revenue.

  There are two tiers of fines : tier one and tier two. Violations related to data processors typically attract fines on the tier-one level. Tier one fines can cost your business €10 million or 2% of your company’s global revenue.

  Tier-two fines result from infringement of the right to forget and the right to privacy of your consumer. Tier-two fines can cost your business up to €20 million or 4% of your company’s global revenue.

  GDPR fines make non-compliance an expensive mistake for businesses of all sizes. As such, signing a DPA with any party that acts as a data processor for your business can help you remain GDPR-compliant.

  How a DPA can help your business remain GDPR compliant

  A DPA can help your business define and adhere to lawful data processes.

  

  So, in what other ways can a DPA help you to remain compliant with GDPR ? Let’s take a look !

  1. Assess data processor’s compliance

  Having a DPA helps ensure that the data processor you are working with is GDPR-compliant. You should check if they have a DPA and confirm the processor’s terms of service and legal basis.

  For example, if you want an alternative to Google Analytics that’s GDPR compliant, then you can opt for Matomo. Matomo features a DPA, which you can agree to when you sign up for web analytics services or later.

  2. Establish lawful data processes

  A DPA can also help you review your data processes to ensure they’re GDPR compliant. For example, by defining lawful data processes, you better understand personally identifiable information (PII) and how it relates to data privacy.

  Further, you can allow users to opt out of sharing their data. As such, Matomo can help you to enable Do Not Track preferences on your website.

  

  With this feature, users are given the option to opt in or out of tracking via a toggle in their respective browsers.

  Indeed, establishing lawful data processes helps you define the specific business purposes for collecting and processing personal data. By doing so, you get to notify your users why you need their data and get their consent to process it by including a GDPR-compliant privacy policy on your website.

  3. Anonymise your data

  Global privacy laws like GDPR and ePrivacy mandate companies to display cookie banners or seek consent before tracking visitors’ data. You can either include a cookie consent banner on your site or stop tracking cookies to follow the applicable regulations.

  Further, you can enable cookie-less tracking or easily let users opt out. For example, you can use Matomo without a cookie consent banner, exempting it from many countries’ privacy rules.

  Additionally, through a DPA, you can define organisational measures that define how you’ll anonymise all your users’ data. Matomo can help you anonymise IP addresses, and we recommend that you at least anonymise the last two bytes.

  

  As one of the few web analytics tools you can use to collect data without tracking consent, Matomo also has the French Data Protection Authority (CNIL) approval.

  4. Assess the processor’s bandwidth

  Having a DPA can help you implement data retention policies that show clear retention periods. Such policies are useful when ending a contract with a third-party service provider and determining how they should handle your data.

  A DPA also helps you ensure the processor has the necessary technology to store personal data securely. You can conduct an audit to understand possible vulnerabilities and your data processor’s technological capacity.

  5. Obtain legal counsel

  When drafting a DPA, it’s important to get a consultation on what is needed to ensure complete compliance. Obtaining legal counsel points you in the right direction so you don’t make any mistakes that may lead to non-compliance.

  Conclusion

  Businesses that process users’ data are subject to several DPA contract requirements under GDPR. One of the most important is having DPAs with every third-party provider that helps them perform data processing.

  It’s important to stay updated on GDPR requirements for compliance. As such, Matomo can help you maintain lawful data processes. Matomo gives you complete control over your data and complies with GDPR requirements.

  To get started with Matomo, you can sign up for a 21-day free trial. No credit card required.

  Disclaimer

  We are not lawyers and don’t claim to be. The information provided here is to help give an introduction to GDPR. We encourage every business and website to take data privacy seriously and discuss these issues with your lawyer if you have any concerns.