Merge branch 'master' into totalwebcasting

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,60 @@
+## Please follow the guide below
+
+- You will be asked some questions and requested to provide some information, please read them **carefully** and answer honestly
+- Put an `x` into all the boxes [ ] relevant to your *issue* (like this: `[x]`)
+- Use the *Preview* tab to see what your issue will actually look like
+
+---
+
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2017.12.31*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2017.12.31**
+
+### Before submitting an *issue* make sure you have:
+- [ ] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
+- [ ] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
+
+### What is the purpose of your *issue*?
+- [ ] Bug report (encountered problems with youtube-dl)
+- [ ] Site support request (request for adding support for a new site)
+- [ ] Feature request (request for a new functionality)
+- [ ] Question
+- [ ] Other
+
+---
+
+### The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your *issue*
+
+---
+
+### If the purpose of this *issue* is a *bug report*, *site support request* or you are not completely sure provide the full verbose output as follows:
+
+Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl -v <your command line>`), copy the **whole** output and insert it here. It should look similar to one below (replace it with **your** log inserted between triple ```):
+
+```
+[debug] System config: []
+[debug] User config: []
+[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
+[debug] youtube-dl version 2017.12.31
+[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
+[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
+[debug] Proxy map: {}
+...
+<end of log>
+```
+
+---
+
+### If the purpose of this *issue* is a *site support request* please provide all kinds of example URLs support for which should be included (replace following example URLs by **yours**):
+- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
+- Single video: https://youtu.be/BaW_jenozKc
+- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
+
+Note that **youtube-dl does not support sites dedicated to [copyright infringement](https://github.com/rg3/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+
+---
+
+### Description of your *issue*, suggested solution and other information
+
+Explanation of your *issue* in arbitrary form goes here. Please make sure the [description is worded well enough to be understood](https://github.com/rg3/youtube-dl#is-the-description-of-the-issue-itself-sufficient). Provide as much context and examples as possible.
+If work on your *issue* requires account credentials please provide them or explain how one can obtain them.

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -1,63 +0,0 @@
----
-name: Broken site support
-about: Report broken or misfunctioning site
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
-- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar issues including closed ones
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2021.12.17
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-```
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -1,54 +0,0 @@
----
-name: Site support request
-about: Request support for a new site
-title: ''
-labels: 'site-support-request'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that none of provided URLs violate any copyrights
-- [ ] I've searched the bugtracker for similar site support requests including closed ones
-
-
-## Example URLs
-
-<!--
-Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
--->
-
-- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
-- Single video: https://youtu.be/BaW_jenozKc
-- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
-
-
-## Description
-
-<!--
-Provide any additional information.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -1,37 +0,0 @@
----
-name: Site feature request
-about: Request a new functionality for a site
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
-- [ ] I've searched the bugtracker for similar site feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your site feature request in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -1,65 +0,0 @@
----
-name: Bug report
-about: Report a bug unrelated to any particular site or extractor
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
-- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Read bugs section in FAQ: http://yt-dl.org/reporting
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar bug reports including closed ones
-- [ ] I've read bugs section in FAQ
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2021.12.17
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-```
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -1,38 +0,0 @@
----
-name: Feature request
-about: Request a new functionality unrelated to any particular site or extractor
-title: ''
-labels: 'request'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
-- [ ] I've searched the bugtracker for similar feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE/6_question.md

@@ -1,38 +0,0 @@
----
-name: Ask question
-about: Ask youtube-dl related question
-title: ''
-labels: 'question'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- Look through the README (http://yt-dl.org/readme) and FAQ (http://yt-dl.org/faq) for similar questions
-- Search the bugtracker for similar questions: http://yt-dl.org/search-issues
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm asking a question
-- [ ] I've looked through the README and FAQ for similar questions
-- [ ] I've searched the bugtracker for similar questions including closed ones
-
-
-## Question
-
-<!--
-Ask your question in an arbitrary form. Please make sure it's worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient.
--->
-
-WRITE QUESTION HERE

.github/ISSUE_TEMPLATE/config.yml

@@ -1 +0,0 @@
-blank_issues_enabled: false

.github/ISSUE_TEMPLATE_tmpl.md

@@ -0,0 +1,60 @@
+## Please follow the guide below
+
+- You will be asked some questions and requested to provide some information, please read them **carefully** and answer honestly
+- Put an `x` into all the boxes [ ] relevant to your *issue* (like this: `[x]`)
+- Use the *Preview* tab to see what your issue will actually look like
+
+---
+
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *%(version)s*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **%(version)s**
+
+### Before submitting an *issue* make sure you have:
+- [ ] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
+- [ ] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
+
+### What is the purpose of your *issue*?
+- [ ] Bug report (encountered problems with youtube-dl)
+- [ ] Site support request (request for adding support for a new site)
+- [ ] Feature request (request for a new functionality)
+- [ ] Question
+- [ ] Other
+
+---
+
+### The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your *issue*
+
+---
+
+### If the purpose of this *issue* is a *bug report*, *site support request* or you are not completely sure provide the full verbose output as follows:
+
+Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl -v <your command line>`), copy the **whole** output and insert it here. It should look similar to one below (replace it with **your** log inserted between triple ```):
+
+```
+[debug] System config: []
+[debug] User config: []
+[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
+[debug] youtube-dl version %(version)s
+[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
+[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
+[debug] Proxy map: {}
+...
+<end of log>
+```
+
+---
+
+### If the purpose of this *issue* is a *site support request* please provide all kinds of example URLs support for which should be included (replace following example URLs by **yours**):
+- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
+- Single video: https://youtu.be/BaW_jenozKc
+- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
+
+Note that **youtube-dl does not support sites dedicated to [copyright infringement](https://github.com/rg3/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
+
+---
+
+### Description of your *issue*, suggested solution and other information
+
+Explanation of your *issue* in arbitrary form goes here. Please make sure the [description is worded well enough to be understood](https://github.com/rg3/youtube-dl#is-the-description-of-the-issue-itself-sufficient). Provide as much context and examples as possible.
+If work on your *issue* requires account credentials please provide them or explain how one can obtain them.

.github/ISSUE_TEMPLATE_tmpl/1_broken_site.md

@@ -1,63 +0,0 @@
----
-name: Broken site support
-about: Report broken or misfunctioning site
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
-- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar issues including closed ones
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version %(version)s
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-```
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md

@@ -1,54 +0,0 @@
----
-name: Site support request
-about: Request support for a new site
-title: ''
-labels: 'site-support-request'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
-- Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that none of provided URLs violate any copyrights
-- [ ] I've searched the bugtracker for similar site support requests including closed ones
-
-
-## Example URLs
-
-<!--
-Provide all kinds of example URLs support for which should be included. Replace following example URLs by yours.
--->
-
-- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
-- Single video: https://youtu.be/BaW_jenozKc
-- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc
-
-
-## Description
-
-<!--
-Provide any additional information.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md

@@ -1,37 +0,0 @@
----
-name: Site feature request
-about: Request a new functionality for a site
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar site feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've searched the bugtracker for similar site feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your site feature request in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/4_bug_report.md

@@ -1,65 +0,0 @@
----
-name: Bug report
-about: Report a bug unrelated to any particular site or extractor
-title: ''
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
-- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
-- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Read bugs section in FAQ: http://yt-dl.org/reporting
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've checked that all provided URLs are alive and playable in a browser
-- [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
-- [ ] I've searched the bugtracker for similar bug reports including closed ones
-- [ ] I've read bugs section in FAQ
-
-
-## Verbose log
-
-<!--
-Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
-Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
- [debug] System config: []
- [debug] User config: []
- [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
- [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version %(version)s
- [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
- [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
- [debug] Proxy map: {}
- <more lines>
--->
-
-```
-PASTE VERBOSE LOG HERE
-```
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
-If work on your issue requires account credentials please provide them or explain how one can obtain them.
--->
-
-WRITE DESCRIPTION HERE

.github/ISSUE_TEMPLATE_tmpl/5_feature_request.md

@@ -1,38 +0,0 @@
----
-name: Feature request
-about: Request a new functionality unrelated to any particular site or extractor
-title: ''
-labels: 'request'
----
-
-<!--
-
-######################################################################
-  WARNING!
-  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
-######################################################################
-
--->
-
-
-## Checklist
-
-<!--
-Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is %(version)s. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
-- Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
-- Finally, put x into all relevant boxes (like this [x])
--->
-
-- [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dl version **%(version)s**
-- [ ] I've searched the bugtracker for similar feature requests including closed ones
-
-
-## Description
-
-<!--
-Provide an explanation of your issue in an arbitrary form. Please make sure the description is worded well enough to be understood, see https://github.com/ytdl-org/youtube-dl#is-the-description-of-the-issue-itself-sufficient. Provide any additional information, suggested solution and as much context and examples as possible.
--->
-
-WRITE DESCRIPTION HERE

.github/PULL_REQUEST_TEMPLATE.md

@@ -7,10 +7,8 @@
 ---
 
 ### Before submitting a *pull request* make sure you have:
-- [ ] [Searched](https://github.com/ytdl-org/youtube-dl/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
-- [ ] Read [adding new extractor tutorial](https://github.com/ytdl-org/youtube-dl#adding-support-for-a-new-site)
-- [ ] Read [youtube-dl coding conventions](https://github.com/ytdl-org/youtube-dl#youtube-dl-coding-conventions) and adjusted the code to meet them
-- [ ] Covered the code with tests (note that PRs without tests will be REJECTED)
+- [ ] At least skimmed through [adding new extractor tutorial](https://github.com/rg3/youtube-dl#adding-support-for-a-new-site) and [youtube-dl coding conventions](https://github.com/rg3/youtube-dl#youtube-dl-coding-conventions) sections
+- [ ] [Searched](https://github.com/rg3/youtube-dl/search?q=is%3Apr&type=Issues) the bugtracker for similar pull requests
 - [ ] Checked the code with [flake8](https://pypi.python.org/pypi/flake8)
 
 ### In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under [Unlicense](http://unlicense.org/). Check one of the following options:

.github/workflows/ci.yml

@@ -1,81 +0,0 @@
-name: CI
-on: [push, pull_request]
-jobs:
-  tests:
-    name: Tests
-    runs-on: ${{ matrix.os }}
-    strategy:
-      fail-fast: true
-      matrix:
-        os: [ubuntu-18.04]
-        # TODO: python 2.6
-        python-version: [2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, pypy-2.7, pypy-3.6, pypy-3.7]
-        python-impl: [cpython]
-        ytdl-test-set: [core, download]
-        run-tests-ext: [sh]
-        include:
-        # python 3.2 is only available on windows via setup-python
-        - os: windows-2019
-          python-version: 3.2
-          python-impl: cpython
-          ytdl-test-set: core
-          run-tests-ext: bat
-        - os: windows-2019
-          python-version: 3.2
-          python-impl: cpython
-          ytdl-test-set: download
-          run-tests-ext: bat
-        # jython
-        - os: ubuntu-18.04
-          python-impl: jython
-          ytdl-test-set: core
-          run-tests-ext: sh
-        - os: ubuntu-18.04
-          python-impl: jython
-          ytdl-test-set: download
-          run-tests-ext: sh
-    steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v2
-      if: ${{ matrix.python-impl == 'cpython' }}
-      with:
-        python-version: ${{ matrix.python-version }}
-    - name: Set up Java 8
-      if: ${{ matrix.python-impl == 'jython' }}
-      uses: actions/setup-java@v1
-      with:
-        java-version: 8
-    - name: Install Jython
-      if: ${{ matrix.python-impl == 'jython' }}
-      run: |
-        wget https://repo1.maven.org/maven2/org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar -O jython-installer.jar
-        java -jar jython-installer.jar -s -d "$HOME/jython"
-        echo "$HOME/jython/bin" >> $GITHUB_PATH
-    - name: Install nose
-      if: ${{ matrix.python-impl != 'jython' }}
-      run: pip install nose
-    - name: Install nose (Jython)
-      if: ${{ matrix.python-impl == 'jython' }}
-      # Working around deprecation of support for non-SNI clients at PyPI CDN (see https://status.python.org/incidents/hzmjhqsdjqgb)
-      run: |
-        wget https://files.pythonhosted.org/packages/99/4f/13fb671119e65c4dce97c60e67d3fd9e6f7f809f2b307e2611f4701205cb/nose-1.3.7-py2-none-any.whl
-        pip install nose-1.3.7-py2-none-any.whl
-    - name: Run tests
-      continue-on-error: ${{ matrix.ytdl-test-set == 'download' || matrix.python-impl == 'jython' }}
-      env:
-        YTDL_TEST_SET: ${{ matrix.ytdl-test-set }}
-      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
-  flake8:
-    name: Linter
-    runs-on: ubuntu-latest
-    steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python
-      uses: actions/setup-python@v2
-      with:
-        python-version: 3.9
-    - name: Install flake8
-      run: pip install flake8
-    - name: Run flake8
-      run: flake8 .

.gitignore

@@ -47,7 +47,3 @@ youtube-dl.zsh
 *.iml
 
 tmp/
-venv/
-
-# VS Code related files
-.vscode

.travis.yml

@@ -0,0 +1,27 @@
+language: python
+python:
+  - "2.6"
+  - "2.7"
+  - "3.2"
+  - "3.3"
+  - "3.4"
+  - "3.5"
+  - "3.6"
+  - "pypy"
+  - "pypy3"
+sudo: false
+env:
+  - YTDL_TEST_SET=core
+  - YTDL_TEST_SET=download
+matrix:
+  include:
+    - env: JYTHON=true; YTDL_TEST_SET=core
+    - env: JYTHON=true; YTDL_TEST_SET=download
+  fast_finish: true
+  allow_failures:
+    - env: YTDL_TEST_SET=download
+    - env: JYTHON=true; YTDL_TEST_SET=core
+    - env: JYTHON=true; YTDL_TEST_SET=download
+before_install:
+  - if [ "$JYTHON" == "true" ]; then ./devscripts/install_jython.sh; export PATH="$HOME/jython/bin:$PATH"; fi
+script: ./devscripts/run_tests.sh

AUTHORS

@@ -231,19 +231,3 @@ John Dong
 Tatsuyuki Ishi
 Daniel Weber
 Kay Bouché
-Yang Hongbo
-Lei Wang
-Petr Novák
-Leonardo Taccari
-Martin Weinelt
-Surya Oktafendri
-TingPing
-Alexandre Macabies
-Bastian de Groot
-Niklas Haas
-András Veres-Szentkirályi
-Enes Solak
-Nathan Rossi
-Thomas van der Berg
-Luca Cherubin
-Adrian Heine

CONTRIBUTING.md

@@ -42,11 +42,11 @@ Before reporting any issue, type `youtube-dl -U`. This should report that you're
 
 ###  Is the issue already documented?
 
-Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/ytdl-org/youtube-dl/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
+Make sure that someone has not already opened the issue you're trying to open. Search at the top of the window or browse the [GitHub Issues](https://github.com/rg3/youtube-dl/search?type=Issues) of this repository. If there is an issue, feel free to write something along the lines of "This affects me as well, with version 2015.01.01. Here is some more information on the issue: ...". While some issues may be old, a new post into them often spurs rapid activity.
 
 ###  Why are existing options not enough?
 
-Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/ytdl-org/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
+Before requesting a new feature, please have a quick peek at [the list of supported options](https://github.com/rg3/youtube-dl/blob/master/README.md#options). Many feature requests are for features that actually exist already! Please, absolutely do show off your work in the issue report and detail how the existing similar options do *not* solve your problem.
 
 ###  Is there enough context in your bug report?
 
@@ -70,7 +70,7 @@ It may sound strange, but some bug reports we receive are completely unrelated t
 
 # DEVELOPER INSTRUCTIONS
 
-Most users do not need to build youtube-dl and can [download the builds](https://ytdl-org.github.io/youtube-dl/download.html) or get them from their distribution.
+Most users do not need to build youtube-dl and can [download the builds](https://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
 
 To run youtube-dl as a developer, you don't need to build anything either. Simply execute
 
@@ -98,7 +98,7 @@ If you want to add support for a new site, first of all **make sure** this site
 
 After you have ensured this site is distributing its content legally, you can follow this quick list (assuming your service is called `yourextractor`):
 
-1. [Fork this repository](https://github.com/ytdl-org/youtube-dl/fork)
+1. [Fork this repository](https://github.com/rg3/youtube-dl/fork)
 2. Check out the source code with:
 
         git clone git@github.com:YOUR_GITHUB_USERNAME/youtube-dl.git
@@ -150,22 +150,18 @@ After you have ensured this site is distributing its content legally, you can fo
                 # TODO more properties (see youtube_dl/extractor/common.py)
             }
     ```
-5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/extractors.py). This makes the extractor available for use, as long as the class ends with `IE`.
+5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc. Note that tests with `only_matching` key in test's dict are not counted in.
-7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://flake8.pycqa.org/en/latest/index.html#quickstart):
-
-        $ flake8 youtube_dl/extractor/yourextractor.py
-
-9. Make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
-10. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
+7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
+8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
+9. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
 
         $ git add youtube_dl/extractor/extractors.py
         $ git add youtube_dl/extractor/yourextractor.py
         $ git commit -m '[yourextractor] Add new extractor'
         $ git push origin yourextractor
 
-11. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
+10. Finally, [create a pull request](https://help.github.com/articles/creating-a-pull-request). We'll then review and merge it.
 
 In any case, thank you very much for your contributions!
 
@@ -177,7 +173,7 @@ Extractors are very fragile by nature since they depend on the layout of the sou
 
 ### Mandatory and optional metafields
 
-For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by an [information dictionary](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L94-L303) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl:
+For extraction to work youtube-dl relies on metadata your extractor extracts and provides to youtube-dl expressed by an [information dictionary](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L75-L257) or simply *info dict*. Only the following meta fields in the *info dict* are considered mandatory for a successful extraction process by youtube-dl:
 
  - `id` (media identifier)
  - `title` (media title)
@@ -185,7 +181,7 @@ For extraction to work youtube-dl relies on metadata your extractor extracts and
 
 In fact only the last option is technically mandatory (i.e. if you can't figure out the download location of the media the extraction does not make any sense). But by convention youtube-dl also treats `id` and `title` as mandatory. Thus the aforementioned metafields are the critical data that the extraction does not make any sense without and if any of them fail to be extracted then the extractor is considered completely broken.
 
-[Any field](https://github.com/ytdl-org/youtube-dl/blob/7f41a598b3fba1bcab2817de64a08941200aa3c8/youtube_dl/extractor/common.py#L188-L303) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
+[Any field](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L149-L257) apart from the aforementioned ones are considered **optional**. That means that extraction should be **tolerant** to situations when sources for these fields can potentially be unavailable (even if they are always available at the moment) and **future-proof** in order not to break the extraction of general purpose mandatory fields.
 
 #### Example
 
@@ -261,33 +257,11 @@ title = meta.get('title') or self._og_search_title(webpage)
 
 This code will try to extract from `meta` first and if it fails it will try extracting `og:title` from a `webpage`.
 
-### Regular expressions
+### Make regular expressions flexible
 
-#### Don't capture groups you don't use
-
-Capturing group must be an indication that it's used somewhere in the code. Any group that is not used must be non capturing.
-
-##### Example
-
-Don't capture id attribute name here since you can't use it for anything anyway.
-
-Correct:
-
-```python
-r'(?:id|ID)=(?P<id>\d+)'
-```
-
-Incorrect:
-```python
-r'(id|ID)=(?P<id>\d+)'
-```
-
-
-#### Make regular expressions relaxed and flexible
-
-When using regular expressions try to write them fuzzy, relaxed and flexible, skipping insignificant parts that are more likely to change, allowing both single and double quotes for quoted values and so on.
+When using regular expressions try to write them fuzzy and flexible.
  
-##### Example
+#### Example
 
 Say you need to extract `title` from the following HTML code:
 
@@ -320,115 +294,7 @@ title = self._search_regex(
     webpage, 'title', group='title')
 ```
 
-### Long lines policy
+### Use safe conversion functions
 
-There is a soft limit to keep lines of code under 80 characters long. This means it should be respected if possible and if it does not make readability and code maintenance worse.
-
-For example, you should **never** split long string literals like URLs or some other often copied entities over multiple lines to fit this limit:
-
-Correct:
-
-```python
-'https://www.youtube.com/watch?v=FqZTN594JQw&list=PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4'
-```
-
-Incorrect:
-
-```python
-'https://www.youtube.com/watch?v=FqZTN594JQw&list='
-'PLMYEtVRpaqY00V9W81Cwmzp6N6vZqfUKD4'
-```
-
-### Inline values
-
-Extracting variables is acceptable for reducing code duplication and improving readability of complex expressions. However, you should avoid extracting variables used only once and moving them to opposite parts of the extractor file, which makes reading the linear flow difficult.
-
-#### Example
-
-Correct:
-
-```python
-title = self._html_search_regex(r'<title>([^<]+)</title>', webpage, 'title')
-```
-
-Incorrect:
-
-```python
-TITLE_RE = r'<title>([^<]+)</title>'
-# ...some lines of code...
-title = self._html_search_regex(TITLE_RE, webpage, 'title')
-```
-
-### Collapse fallbacks
-
-Multiple fallback values can quickly become unwieldy. Collapse multiple fallback values into a single expression via a list of patterns.
-
-#### Example
-
-Good:
-
-```python
-description = self._html_search_meta(
-    ['og:description', 'description', 'twitter:description'],
-    webpage, 'description', default=None)
-```
-
-Unwieldy:
-
-```python
-description = (
-    self._og_search_description(webpage, default=None)
-    or self._html_search_meta('description', webpage, default=None)
-    or self._html_search_meta('twitter:description', webpage, default=None))
-```
-
-Methods supporting list of patterns are: `_search_regex`, `_html_search_regex`, `_og_search_property`, `_html_search_meta`.
-
-### Trailing parentheses
-
-Always move trailing parentheses after the last argument.
-
-#### Example
-
-Correct:
-
-```python
-    lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'],
-    list)
-```
-
-Incorrect:
-
-```python
-    lambda x: x['ResultSet']['Result'][0]['VideoUrlSet']['VideoUrl'],
-    list,
-)
-```
-
-### Use convenience conversion and parsing functions
-
-Wrap all extracted numeric data into safe functions from [`youtube_dl/utils.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/utils.py): `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
-
-Use `url_or_none` for safe URL processing.
-
-Use `try_get` for safe metadata extraction from parsed JSON.
-
-Use `unified_strdate` for uniform `upload_date` or any `YYYYMMDD` meta field extraction, `unified_timestamp` for uniform `timestamp` extraction, `parse_filesize` for `filesize` extraction, `parse_count` for count meta fields extraction, `parse_resolution`, `parse_duration` for `duration` extraction, `parse_age_limit` for `age_limit` extraction. 
-
-Explore [`youtube_dl/utils.py`](https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/utils.py) for more useful convenience functions.
-
-#### More examples
-
-##### Safely extract optional description from parsed JSON
-```python
-description = try_get(response, lambda x: x['result']['video'][0]['summary'], compat_str)
-```
-
-##### Safely extract more optional metadata
-```python
-video = try_get(response, lambda x: x['result']['video'][0], dict) or {}
-description = video.get('summary')
-duration = float_or_none(video.get('durationMs'), scale=1000)
-view_count = int_or_none(video.get('views'))
-```
+Wrap all extracted numeric data into safe functions from `utils`: `int_or_none`, `float_or_none`. Use them for string to number conversions as well.
 

Makefile

@@ -1,7 +1,7 @@
 all: youtube-dl README.md CONTRIBUTING.md README.txt youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish supportedsites
 
 clean:
-	rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz youtube-dl.zsh youtube-dl.fish youtube_dl/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png CONTRIBUTING.md.tmp youtube-dl youtube-dl.exe
+	rm -rf youtube-dl.1.temp.md youtube-dl.1 youtube-dl.bash-completion README.txt MANIFEST build/ dist/ .coverage cover/ youtube-dl.tar.gz youtube-dl.zsh youtube-dl.fish youtube_dl/extractor/lazy_extractors.py *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png CONTRIBUTING.md.tmp ISSUE_TEMPLATE.md.tmp youtube-dl youtube-dl.exe
 	find . -name "*.pyc" -delete
 	find . -name "*.class" -delete
 
@@ -14,9 +14,6 @@ PYTHON ?= /usr/bin/env python
 # set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local
 SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then echo /etc; else echo $(PREFIX)/etc; fi)
 
-# set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2
-MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi)
-
 install: youtube-dl youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish
 	install -d $(DESTDIR)$(BINDIR)
 	install -m 755 youtube-dl $(DESTDIR)$(BINDIR)
@@ -78,22 +75,18 @@ README.md: youtube_dl/*.py youtube_dl/*/*.py
 CONTRIBUTING.md: README.md
 	$(PYTHON) devscripts/make_contributing.py README.md CONTRIBUTING.md
 
-issuetemplates: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.md .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md .github/ISSUE_TEMPLATE_tmpl/4_bug_report.md .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md youtube_dl/version.py
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/1_broken_site.md .github/ISSUE_TEMPLATE/1_broken_site.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/2_site_support_request.md .github/ISSUE_TEMPLATE/2_site_support_request.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/3_site_feature_request.md .github/ISSUE_TEMPLATE/3_site_feature_request.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/4_bug_report.md .github/ISSUE_TEMPLATE/4_bug_report.md
-	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl/5_feature_request.md .github/ISSUE_TEMPLATE/5_feature_request.md
+.github/ISSUE_TEMPLATE.md: devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl.md  youtube_dl/version.py
+	$(PYTHON) devscripts/make_issue_template.py .github/ISSUE_TEMPLATE_tmpl.md .github/ISSUE_TEMPLATE.md
 
 supportedsites:
 	$(PYTHON) devscripts/make_supportedsites.py docs/supportedsites.md
 
 README.txt: README.md
-	pandoc -f $(MARKDOWN) -t plain README.md -o README.txt
+	pandoc -f markdown -t plain README.md -o README.txt
 
 youtube-dl.1: README.md
 	$(PYTHON) devscripts/prepare_manpage.py youtube-dl.1.temp.md
-	pandoc -s -f $(MARKDOWN) -t man youtube-dl.1.temp.md -o youtube-dl.1
+	pandoc -s -f markdown -t man youtube-dl.1.temp.md -o youtube-dl.1
 	rm -f youtube-dl.1.temp.md
 
 youtube-dl.bash-completion: youtube_dl/*.py youtube_dl/*/*.py devscripts/bash-completion.in

devscripts/buildserver.py

@@ -322,7 +322,7 @@ class GITBuilder(GITInfoBuilder):
 
 
 class YoutubeDLBuilder(object):
-    authorizedUsers = ['fraca7', 'phihag', 'rg3', 'FiloSottile', 'ytdl-org']
+    authorizedUsers = ['fraca7', 'phihag', 'rg3', 'FiloSottile']
 
     def __init__(self, **kwargs):
         if self.repoName != 'youtube-dl':

devscripts/check-porn.py

@@ -45,12 +45,12 @@ for test in gettestcases():
 
         RESULT = ('.' + domain + '\n' in LIST or '\n' + domain + '\n' in LIST)
 
-    if RESULT and ('info_dict' not in test or 'age_limit' not in test['info_dict']
-                   or test['info_dict']['age_limit'] != 18):
+    if RESULT and ('info_dict' not in test or 'age_limit' not in test['info_dict'] or
+                   test['info_dict']['age_limit'] != 18):
         print('\nPotential missing age_limit check: {0}'.format(test['name']))
 
-    elif not RESULT and ('info_dict' in test and 'age_limit' in test['info_dict']
-                         and test['info_dict']['age_limit'] == 18):
+    elif not RESULT and ('info_dict' in test and 'age_limit' in test['info_dict'] and
+                         test['info_dict']['age_limit'] == 18):
         print('\nPotential false negative: {0}'.format(test['name']))
 
     else:

devscripts/create-github-release.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 from __future__ import unicode_literals
 
+import base64
 import io
 import json
 import mimetypes
@@ -14,6 +15,7 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from youtube_dl.compat import (
     compat_basestring,
+    compat_input,
     compat_getpass,
     compat_print,
     compat_urllib_request,
@@ -25,8 +27,8 @@ from youtube_dl.utils import (
 
 
 class GitHubReleaser(object):
-    _API_URL = 'https://api.github.com/repos/ytdl-org/youtube-dl/releases'
-    _UPLOADS_URL = 'https://uploads.github.com/repos/ytdl-org/youtube-dl/releases/%s/assets?name=%s'
+    _API_URL = 'https://api.github.com/repos/rg3/youtube-dl/releases'
+    _UPLOADS_URL = 'https://uploads.github.com/repos/rg3/youtube-dl/releases/%s/assets?name=%s'
     _NETRC_MACHINE = 'github.com'
 
     def __init__(self, debuglevel=0):
@@ -38,20 +40,28 @@ class GitHubReleaser(object):
         try:
             info = netrc.netrc().authenticators(self._NETRC_MACHINE)
             if info is not None:
-                self._token = info[2]
+                self._username = info[0]
+                self._password = info[2]
                 compat_print('Using GitHub credentials found in .netrc...')
                 return
             else:
                 compat_print('No GitHub credentials found in .netrc')
         except (IOError, netrc.NetrcParseError):
             compat_print('Unable to parse .netrc')
-        self._token = compat_getpass(
-            'Type your GitHub PAT (personal access token) and press [Return]: ')
+        self._username = compat_input(
+            'Type your GitHub username or email address and press [Return]: ')
+        self._password = compat_getpass(
+            'Type your GitHub password and press [Return]: ')
 
     def _call(self, req):
         if isinstance(req, compat_basestring):
             req = sanitized_Request(req)
-        req.add_header('Authorization', 'token %s' % self._token)
+        # Authorizing manually since GitHub does not response with 401 with
+        # WWW-Authenticate header set (see
+        # https://developer.github.com/v3/#basic-authentication)
+        b64 = base64.b64encode(
+            ('%s:%s' % (self._username, self._password)).encode('utf-8')).decode('ascii')
+        req.add_header('Authorization', 'Basic %s' % b64)
         response = self._opener.open(req).read().decode('utf-8')
         return json.loads(response)
 

devscripts/gh-pages/generate-download.py

@@ -1,22 +1,27 @@
 #!/usr/bin/env python3
 from __future__ import unicode_literals
 
+import hashlib
+import urllib.request
 import json
 
 versions_info = json.load(open('update/versions.json'))
 version = versions_info['latest']
-version_dict = versions_info['versions'][version]
+URL = versions_info['versions'][version]['bin'][0]
+
+data = urllib.request.urlopen(URL).read()
 
 # Read template page
 with open('download.html.in', 'r', encoding='utf-8') as tmplf:
     template = tmplf.read()
 
+sha256sum = hashlib.sha256(data).hexdigest()
 template = template.replace('@PROGRAM_VERSION@', version)
-template = template.replace('@PROGRAM_URL@', version_dict['bin'][0])
-template = template.replace('@PROGRAM_SHA256SUM@', version_dict['bin'][1])
-template = template.replace('@EXE_URL@', version_dict['exe'][0])
-template = template.replace('@EXE_SHA256SUM@', version_dict['exe'][1])
-template = template.replace('@TAR_URL@', version_dict['tar'][0])
-template = template.replace('@TAR_SHA256SUM@', version_dict['tar'][1])
+template = template.replace('@PROGRAM_URL@', URL)
+template = template.replace('@PROGRAM_SHA256SUM@', sha256sum)
+template = template.replace('@EXE_URL@', versions_info['versions'][version]['exe'][0])
+template = template.replace('@EXE_SHA256SUM@', versions_info['versions'][version]['exe'][1])
+template = template.replace('@TAR_URL@', versions_info['versions'][version]['tar'][0])
+template = template.replace('@TAR_SHA256SUM@', versions_info['versions'][version]['tar'][1])
 with open('download.html', 'w', encoding='utf-8') as dlf:
     dlf.write(template)

devscripts/gh-pages/update-copyright.py

@@ -13,7 +13,7 @@ year = str(datetime.datetime.now().year)
 for fn in glob.glob('*.html*'):
     with io.open(fn, encoding='utf-8') as f:
         content = f.read()
-    newc = re.sub(r'(?P<copyright>Copyright © 2011-)(?P<year>[0-9]{4})', 'Copyright © 2011-' + year, content)
+    newc = re.sub(r'(?P<copyright>Copyright © 2006-)(?P<year>[0-9]{4})', 'Copyright © 2006-' + year, content)
     if content != newc:
         tmpFn = fn + '.part'
         with io.open(tmpFn, 'wt', encoding='utf-8') as outf:

devscripts/gh-pages/update-feed.py

@@ -10,7 +10,7 @@ import textwrap
 atom_template = textwrap.dedent("""\
     <?xml version="1.0" encoding="utf-8"?>
     <feed xmlns="http://www.w3.org/2005/Atom">
-        <link rel="self" href="http://ytdl-org.github.io/youtube-dl/update/releases.atom" />
+        <link rel="self" href="http://rg3.github.io/youtube-dl/update/releases.atom" />
         <title>youtube-dl releases</title>
         <id>https://yt-dl.org/feed/youtube-dl-updates-feed</id>
         <updated>@TIMESTAMP@</updated>
@@ -21,7 +21,7 @@ entry_template = textwrap.dedent("""
     <entry>
         <id>https://yt-dl.org/feed/youtube-dl-updates-feed/youtube-dl-@VERSION@</id>
         <title>New version @VERSION@</title>
-        <link href="http://ytdl-org.github.io/youtube-dl" />
+        <link href="http://rg3.github.io/youtube-dl" />
         <content type="xhtml">
             <div xmlns="http://www.w3.org/1999/xhtml">
                 Downloads available at <a href="https://yt-dl.org/downloads/@VERSION@/">https://yt-dl.org/downloads/@VERSION@/</a>

devscripts/install_jython.sh

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+wget http://central.maven.org/maven2/org/python/jython-installer/2.7.1/jython-installer-2.7.1.jar
+java -jar jython-installer-2.7.1.jar -s -d "$HOME/jython"
+$HOME/jython/bin/jython -m pip install nose

devscripts/make_lazy_extractors.py

@@ -61,7 +61,7 @@ def build_lazy_ie(ie, name):
     return s
 
 
-# find the correct sorting and add the required base classes so that subclasses
+# find the correct sorting and add the required base classes so that sublcasses
 # can be correctly created
 classes = _ALL_CLASSES[:-1]
 ordered_cls = []

devscripts/release.sh

@@ -78,8 +78,8 @@ sed -i "s/__version__ = '.*'/__version__ = '$version'/" youtube_dl/version.py
 sed -i "s/<unreleased>/$version/" ChangeLog
 
 /bin/echo -e "\n### Committing documentation, templates and youtube_dl/version.py..."
-make README.md CONTRIBUTING.md issuetemplates supportedsites
-git add README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE/1_broken_site.md .github/ISSUE_TEMPLATE/2_site_support_request.md .github/ISSUE_TEMPLATE/3_site_feature_request.md .github/ISSUE_TEMPLATE/4_bug_report.md .github/ISSUE_TEMPLATE/5_feature_request.md .github/ISSUE_TEMPLATE/6_question.md docs/supportedsites.md youtube_dl/version.py ChangeLog
+make README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md supportedsites
+git add README.md CONTRIBUTING.md .github/ISSUE_TEMPLATE.md docs/supportedsites.md youtube_dl/version.py ChangeLog
 git commit $gpg_sign_commits -m "release $version"
 
 /bin/echo -e "\n### Now tagging, signing and pushing..."
@@ -96,7 +96,7 @@ git push origin "$version"
 REV=$(git rev-parse HEAD)
 make youtube-dl youtube-dl.tar.gz
 read -p "VM running? (y/n) " -n 1
-wget "http://$buildserver/build/ytdl-org/youtube-dl/youtube-dl.exe?rev=$REV" -O youtube-dl.exe
+wget "http://$buildserver/build/rg3/youtube-dl/youtube-dl.exe?rev=$REV" -O youtube-dl.exe
 mkdir -p "build/$version"
 mv youtube-dl youtube-dl.exe "build/$version"
 mv youtube-dl.tar.gz "build/$version/youtube-dl-$version.tar.gz"

devscripts/run_tests.bat

@@ -1,17 +0,0 @@
-@echo off
-
-rem Keep this list in sync with the `offlinetest` target in Makefile
-set DOWNLOAD_TESTS="age_restriction^|download^|iqiyi_sdk_interpreter^|socks^|subtitles^|write_annotations^|youtube_lists^|youtube_signature"
-
-if "%YTDL_TEST_SET%" == "core" (
-    set test_set="-I test_("%DOWNLOAD_TESTS%")\.py"
-    set multiprocess_args=""
-) else if "%YTDL_TEST_SET%" == "download" (
-    set test_set="-I test_(?!"%DOWNLOAD_TESTS%").+\.py"
-    set multiprocess_args="--processes=4 --process-timeout=540"
-) else (
-    echo YTDL_TEST_SET is not set or invalid
-    exit /b 1
-)
-
-nosetests test --verbose %test_set:"=% %multiprocess_args:"=%

devscripts/show-downloads-statistics.py

@@ -24,7 +24,7 @@ total_bytes = 0
 
 for page in itertools.count(1):
     releases = json.loads(compat_urllib_request.urlopen(
-        'https://api.github.com/repos/ytdl-org/youtube-dl/releases?page=%s' % page
+        'https://api.github.com/repos/rg3/youtube-dl/releases?page=%s' % page
     ).read().decode('utf-8'))
 
     if not releases:

setup.cfg

@@ -2,5 +2,5 @@
 universal = True
 
 [flake8]
-exclude = youtube_dl/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git,venv
-ignore = E402,E501,E731,E741,W503
+exclude = youtube_dl/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git
+ignore = E402,E501,E731

setup.py

@@ -104,7 +104,7 @@ setup(
     version=__version__,
     description=DESCRIPTION,
     long_description=LONG_DESCRIPTION,
-    url='https://github.com/ytdl-org/youtube-dl',
+    url='https://github.com/rg3/youtube-dl',
     author='Ricardo Garcia',
     author_email='ytdl@yt-dl.org',
     maintainer='Sergey M.',
@@ -124,8 +124,6 @@ setup(
         'Development Status :: 5 - Production/Stable',
         'Environment :: Console',
         'License :: Public Domain',
-        'Programming Language :: Python',
-        'Programming Language :: Python :: 2',
         'Programming Language :: Python :: 2.6',
         'Programming Language :: Python :: 2.7',
         'Programming Language :: Python :: 3',
@@ -134,13 +132,6 @@ setup(
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
-        'Programming Language :: Python :: 3.8',
-        'Programming Language :: Python :: Implementation',
-        'Programming Language :: Python :: Implementation :: CPython',
-        'Programming Language :: Python :: Implementation :: IronPython',
-        'Programming Language :: Python :: Implementation :: Jython',
-        'Programming Language :: Python :: Implementation :: PyPy',
     ],
 
     cmdclass={'build_lazy_extractors': build_lazy_extractors},

test/helper.py

@@ -7,7 +7,6 @@ import json
 import os.path
 import re
 import types
-import ssl
 import sys
 
 import youtube_dl.extractor
@@ -128,12 +127,6 @@ def expect_value(self, got, expected, field):
         self.assertTrue(
             contains_str in got,
             'field %s (value: %r) should contain %r' % (field, got, contains_str))
-    elif isinstance(expected, compat_str) and re.match(r'^lambda \w+:', expected):
-        fn = eval(expected)
-        suite = expected.split(':', 1)[1].strip()
-        self.assertTrue(
-            fn(got),
-            'Expected field %s to meet condition %s, but value %r failed ' % (field, suite, got))
     elif isinstance(expected, type):
         self.assertTrue(
             isinstance(got, expected),
@@ -143,7 +136,7 @@ def expect_value(self, got, expected, field):
     elif isinstance(expected, list) and isinstance(got, list):
         self.assertEqual(
             len(expected), len(got),
-            'Expected a list of length %d, but got a list of length %d for field %s' % (
+            'Expect a list of length %d, but got a list of length %d for field %s' % (
                 len(expected), len(got), field))
         for index, (item_got, item_expected) in enumerate(zip(got, expected)):
             type_got = type(item_got)
@@ -159,27 +152,15 @@ def expect_value(self, got, expected, field):
                 isinstance(got, compat_str),
                 'Expected field %s to be a unicode object, but got value %r of type %r' % (field, got, type(got)))
             got = 'md5:' + md5(got)
-        elif isinstance(expected, compat_str) and re.match(r'^(?:min|max)?count:\d+', expected):
+        elif isinstance(expected, compat_str) and expected.startswith('mincount:'):
             self.assertTrue(
                 isinstance(got, (list, dict)),
                 'Expected field %s to be a list or a dict, but it is of type %s' % (
                     field, type(got).__name__))
-            op, _, expected_num = expected.partition(':')
-            expected_num = int(expected_num)
-            if op == 'mincount':
-                assert_func = assertGreaterEqual
-                msg_tmpl = 'Expected %d items in field %s, but only got %d'
-            elif op == 'maxcount':
-                assert_func = assertLessEqual
-                msg_tmpl = 'Expected maximum %d items in field %s, but got %d'
-            elif op == 'count':
-                assert_func = assertEqual
-                msg_tmpl = 'Expected exactly %d items in field %s, but got %d'
-            else:
-                assert False
-            assert_func(
+            expected_num = int(expected.partition(':')[2])
+            assertGreaterEqual(
                 self, len(got), expected_num,
-                msg_tmpl % (expected_num, field, len(got)))
+                'Expected %d items in field %s, but only got %d' % (expected_num, field, len(got)))
             return
         self.assertEqual(
             expected, got,
@@ -255,20 +236,6 @@ def assertGreaterEqual(self, got, expected, msg=None):
         self.assertTrue(got >= expected, msg)
 
 
-def assertLessEqual(self, got, expected, msg=None):
-    if not (got <= expected):
-        if msg is None:
-            msg = '%r not less than or equal to %r' % (got, expected)
-        self.assertTrue(got <= expected, msg)
-
-
-def assertEqual(self, got, expected, msg=None):
-    if not (got == expected):
-        if msg is None:
-            msg = '%r not equal to %r' % (got, expected)
-        self.assertTrue(got == expected, msg)
-
-
 def expect_warnings(ydl, warnings_re):
     real_warning = ydl.report_warning
 
@@ -277,12 +244,3 @@ def expect_warnings(ydl, warnings_re):
             real_warning(w)
 
     ydl.report_warning = _report_warning
-
-
-def http_server_port(httpd):
-    if os.name == 'java' and isinstance(httpd.socket, ssl.SSLSocket):
-        # In Jython SSLSocket is not a subclass of socket.socket
-        sock = httpd.socket.sock
-    else:
-        sock = httpd.socket
-    return sock.getsockname()[1]

test/parameters.json

@@ -18,6 +18,7 @@
     "noprogress": false, 
     "outtmpl": "%(id)s.%(ext)s", 
     "password": null, 
+    "playlistend": -1, 
     "playliststart": 1, 
     "prefer_free_formats": false, 
     "quiet": false, 
@@ -36,7 +37,7 @@
     "writeinfojson": true, 
     "writesubtitles": false,
     "allsubtitles": false,
-    "listsubtitles": false,
+    "listssubtitles": false,
     "socket_timeout": 20,
     "fixup": "never"
 }

test/test_InfoExtractor.py

@@ -9,30 +9,11 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL, expect_dict, expect_value, http_server_port
-from youtube_dl.compat import compat_etree_fromstring, compat_http_server
+from test.helper import FakeYDL, expect_dict, expect_value
+from youtube_dl.compat import compat_etree_fromstring
 from youtube_dl.extractor.common import InfoExtractor
 from youtube_dl.extractor import YoutubeIE, get_info_extractor
 from youtube_dl.utils import encode_data_uri, strip_jsonp, ExtractorError, RegexNotFoundError
-import threading
-
-
-TEAPOT_RESPONSE_STATUS = 418
-TEAPOT_RESPONSE_BODY = "<h1>418 I'm a teapot</h1>"
-
-
-class InfoExtractorTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
-    def log_message(self, format, *args):
-        pass
-
-    def do_GET(self):
-        if self.path == '/teapot':
-            self.send_response(TEAPOT_RESPONSE_STATUS)
-            self.send_header('Content-Type', 'text/html; charset=utf-8')
-            self.end_headers()
-            self.wfile.write(TEAPOT_RESPONSE_BODY.encode())
-        else:
-            assert False
 
 
 class TestIE(InfoExtractor):
@@ -61,7 +42,6 @@ class TestInfoExtractor(unittest.TestCase):
             <meta content='Foo' property=og:foobar>
             <meta name="og:test1" content='foo > < bar'/>
             <meta name="og:test2" content="foo >//< bar"/>
-            <meta property=og-test3 content='Ill-formatted opengraph'/>
             '''
         self.assertEqual(ie._og_search_title(html), 'Foo')
         self.assertEqual(ie._og_search_description(html), 'Some video\'s description ')
@@ -70,7 +50,6 @@ class TestInfoExtractor(unittest.TestCase):
         self.assertEqual(ie._og_search_property('foobar', html), 'Foo')
         self.assertEqual(ie._og_search_property('test1', html), 'foo > < bar')
         self.assertEqual(ie._og_search_property('test2', html), 'foo >//< bar')
-        self.assertEqual(ie._og_search_property('test3', html), 'Ill-formatted opengraph')
         self.assertEqual(ie._og_search_property(('test0', 'test1'), html), 'foo > < bar')
         self.assertRaises(RegexNotFoundError, ie._og_search_property, 'test0', html, None, fatal=True)
         self.assertRaises(RegexNotFoundError, ie._og_search_property, ('test0', 'test00'), html, None, fatal=True)
@@ -98,55 +77,6 @@ class TestInfoExtractor(unittest.TestCase):
         self.assertRaises(RegexNotFoundError, ie._html_search_meta, 'z', html, None, fatal=True)
         self.assertRaises(RegexNotFoundError, ie._html_search_meta, ('z', 'x'), html, None, fatal=True)
 
-    def test_search_json_ld_realworld(self):
-        # https://github.com/ytdl-org/youtube-dl/issues/23306
-        expect_dict(
-            self,
-            self.ie._search_json_ld(r'''<script type="application/ld+json">
-{
-"@context": "http://schema.org/",
-"@type": "VideoObject",
-"name": "1 On 1 With Kleio",
-"url": "https://www.eporner.com/hd-porn/xN49A1cT3eB/1-On-1-With-Kleio/",
-"duration": "PT0H12M23S",
-"thumbnailUrl": ["https://static-eu-cdn.eporner.com/thumbs/static4/7/78/780/780814/9_360.jpg", "https://imggen.eporner.com/780814/1920/1080/9.jpg"],
-"contentUrl": "https://gvideo.eporner.com/xN49A1cT3eB/xN49A1cT3eB.mp4",
-"embedUrl": "https://www.eporner.com/embed/xN49A1cT3eB/1-On-1-With-Kleio/",
-"image": "https://static-eu-cdn.eporner.com/thumbs/static4/7/78/780/780814/9_360.jpg",
-"width": "1920",
-"height": "1080",
-"encodingFormat": "mp4",
-"bitrate": "6617kbps",
-"isFamilyFriendly": "False",
-"description": "Kleio Valentien",
-"uploadDate": "2015-12-05T21:24:35+01:00",
-"interactionStatistic": {
-"@type": "InteractionCounter",
-"interactionType": { "@type": "http://schema.org/WatchAction" },
-"userInteractionCount": 1120958
-}, "aggregateRating": {
-"@type": "AggregateRating",
-"ratingValue": "88",
-"ratingCount": "630",
-"bestRating": "100",
-"worstRating": "0"
-}, "actor": [{
-"@type": "Person",
-"name": "Kleio Valentien",
-"url": "https://www.eporner.com/pornstar/kleio-valentien/"
-}]}
-</script>''', None),
-            {
-                'title': '1 On 1 With Kleio',
-                'description': 'Kleio Valentien',
-                'url': 'https://gvideo.eporner.com/xN49A1cT3eB/xN49A1cT3eB.mp4',
-                'timestamp': 1449347075,
-                'duration': 743.0,
-                'view_count': 1120958,
-                'width': 1920,
-                'height': 1080,
-            })
-
     def test_download_json(self):
         uri = encode_data_uri(b'{"foo": "blah"}', 'application/json')
         self.assertEqual(self.ie._download_json(uri, None), {'foo': 'blah'})
@@ -156,196 +86,6 @@ class TestInfoExtractor(unittest.TestCase):
         self.assertRaises(ExtractorError, self.ie._download_json, uri, None)
         self.assertEqual(self.ie._download_json(uri, None, fatal=False), None)
 
-    def test_parse_html5_media_entries(self):
-        # inline video tag
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://127.0.0.1/video.html',
-                r'<html><video src="/vid.mp4" /></html>', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://127.0.0.1/vid.mp4',
-                }],
-            })
-
-        # from https://www.r18.com/
-        # with kpbs in label
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.r18.com/',
-                r'''
-                <video id="samplevideo_amateur" class="js-samplevideo video-js vjs-default-skin vjs-big-play-centered" controls preload="auto" width="400" height="225" poster="//pics.r18.com/digital/amateur/mgmr105/mgmr105jp.jpg">
-                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_sm_w.mp4" type="video/mp4"  res="240" label="300kbps">
-                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dm_w.mp4" type="video/mp4"  res="480" label="1000kbps">
-                    <source id="video_source" src="https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dmb_w.mp4" type="video/mp4"  res="740" label="1500kbps">
-                    <p>Your browser does not support the video tag.</p>
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_sm_w.mp4',
-                    'ext': 'mp4',
-                    'format_id': '300kbps',
-                    'height': 240,
-                    'tbr': 300,
-                }, {
-                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dm_w.mp4',
-                    'ext': 'mp4',
-                    'format_id': '1000kbps',
-                    'height': 480,
-                    'tbr': 1000,
-                }, {
-                    'url': 'https://awscc3001.r18.com/litevideo/freepv/m/mgm/mgmr105/mgmr105_dmb_w.mp4',
-                    'ext': 'mp4',
-                    'format_id': '1500kbps',
-                    'height': 740,
-                    'tbr': 1500,
-                }],
-                'thumbnail': '//pics.r18.com/digital/amateur/mgmr105/mgmr105jp.jpg'
-            })
-
-        # from https://www.csfd.cz/
-        # with width and height
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.csfd.cz/',
-                r'''
-                <video width="770" height="328" preload="none" controls  poster="https://img.csfd.cz/files/images/film/video/preview/163/344/163344118_748d20.png?h360" >
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327358_eac647.mp4" type="video/mp4" width="640" height="360">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327360_3d2646.mp4" type="video/mp4" width="1280" height="720">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327356_91f258.mp4" type="video/mp4" width="1920" height="1080">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327359_962b4a.webm" type="video/webm" width="640" height="360">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327361_6feee0.webm" type="video/webm" width="1280" height="720">
-                    <source src="https://video.csfd.cz/files/videos/157/750/157750813/163327357_8ab472.webm" type="video/webm" width="1920" height="1080">
-                    <track src="https://video.csfd.cz/files/subtitles/163/344/163344115_4c388b.srt" type="text/x-srt" kind="subtitles" srclang="cs" label="cs">
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327358_eac647.mp4',
-                    'ext': 'mp4',
-                    'width': 640,
-                    'height': 360,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327360_3d2646.mp4',
-                    'ext': 'mp4',
-                    'width': 1280,
-                    'height': 720,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327356_91f258.mp4',
-                    'ext': 'mp4',
-                    'width': 1920,
-                    'height': 1080,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327359_962b4a.webm',
-                    'ext': 'webm',
-                    'width': 640,
-                    'height': 360,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327361_6feee0.webm',
-                    'ext': 'webm',
-                    'width': 1280,
-                    'height': 720,
-                }, {
-                    'url': 'https://video.csfd.cz/files/videos/157/750/157750813/163327357_8ab472.webm',
-                    'ext': 'webm',
-                    'width': 1920,
-                    'height': 1080,
-                }],
-                'subtitles': {
-                    'cs': [{'url': 'https://video.csfd.cz/files/subtitles/163/344/163344115_4c388b.srt'}]
-                },
-                'thumbnail': 'https://img.csfd.cz/files/images/film/video/preview/163/344/163344118_748d20.png?h360'
-            })
-
-        # from https://tamasha.com/v/Kkdjw
-        # with height in label
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://tamasha.com/v/Kkdjw',
-                r'''
-                <video crossorigin="anonymous">
-                        <source src="https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4" type="video/mp4" label="AUTO" res="0"/>
-                                <source src="https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4" type="video/mp4"
-                                        label="240p" res="240"/>
-                                <source src="https://s-v2.tamasha.com/statics/videos_file/20/00/Kkdjw_200041c66f657fc967db464d156eafbc1ed9fe6f_n_144.mp4" type="video/mp4"
-                                        label="144p" res="144"/>
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4',
-                }, {
-                    'url': 'https://s-v2.tamasha.com/statics/videos_file/19/8f/Kkdjw_198feff8577d0057536e905cce1fb61438dd64e0_n_240.mp4',
-                    'ext': 'mp4',
-                    'format_id': '240p',
-                    'height': 240,
-                }, {
-                    'url': 'https://s-v2.tamasha.com/statics/videos_file/20/00/Kkdjw_200041c66f657fc967db464d156eafbc1ed9fe6f_n_144.mp4',
-                    'ext': 'mp4',
-                    'format_id': '144p',
-                    'height': 144,
-                }]
-            })
-
-        # from https://www.directvnow.com
-        # with data-src
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.directvnow.com',
-                r'''
-                <video id="vid1" class="header--video-masked active" muted playsinline>
-                    <source data-src="https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4" type="video/mp4" />
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'ext': 'mp4',
-                    'url': 'https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4',
-                }]
-            })
-
-        # from https://www.directvnow.com
-        # with data-src
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.directvnow.com',
-                r'''
-                <video id="vid1" class="header--video-masked active" muted playsinline>
-                    <source data-src="https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4" type="video/mp4" />
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://cdn.directv.com/content/dam/dtv/prod/website_directvnow-international/videos/DTVN_hdr_HBO_v3.mp4',
-                    'ext': 'mp4',
-                }]
-            })
-
-        # from https://www.klarna.com/uk/
-        # with data-video-src
-        expect_dict(
-            self,
-            self.ie._parse_html5_media_entries(
-                'https://www.directvnow.com',
-                r'''
-                <video loop autoplay muted class="responsive-video block-kl__video video-on-medium">
-                    <source src="" data-video-desktop data-video-src="https://www.klarna.com/uk/wp-content/uploads/sites/11/2019/01/KL062_Smooth3_0_DogWalking_5s_920x080_.mp4" type="video/mp4" />
-                </video>
-                ''', None)[0],
-            {
-                'formats': [{
-                    'url': 'https://www.klarna.com/uk/wp-content/uploads/sites/11/2019/01/KL062_Smooth3_0_DogWalking_5s_920x080_.mp4',
-                    'ext': 'mp4',
-                }],
-            })
-
     def test_extract_jwplayer_data_realworld(self):
         # from http://www.suffolk.edu/sjc/
         expect_dict(
@@ -440,7 +180,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
     def test_parse_m3u8_formats(self):
         _TEST_CASES = [
             (
-                # https://github.com/ytdl-org/youtube-dl/issues/11507
+                # https://github.com/rg3/youtube-dl/issues/11507
                 # http://pluzz.francetv.fr/videos/le_ministere.html
                 'pluzz_francetv_11507',
                 'http://replayftv-vh.akamaihd.net/i/streaming-adaptatif_france-dom-tom/2017/S16/J2/156589847-58f59130c1f52-,standard1,standard2,standard3,standard4,standard5,.mp4.csmil/master.m3u8?caption=2017%2F16%2F156589847-1492488987.m3u8%3Afra%3AFrancais&audiotrack=0%3Afra%3AFrancais',
@@ -502,7 +242,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                 }]
             ),
             (
-                # https://github.com/ytdl-org/youtube-dl/issues/11995
+                # https://github.com/rg3/youtube-dl/issues/11995
                 # http://teamcoco.com/video/clueless-gamer-super-bowl-for-honor
                 'teamcoco_11995',
                 'http://ak.storage-w.teamcococdn.com/cdn/2017-02/98599/ed8f/main.m3u8',
@@ -576,7 +316,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                 }]
             ),
             (
-                # https://github.com/ytdl-org/youtube-dl/issues/12211
+                # https://github.com/rg3/youtube-dl/issues/12211
                 # http://video.toggle.sg/en/series/whoopie-s-world/ep3/478601
                 'toggle_mobile_12211',
                 'http://cdnapi.kaltura.com/p/2082311/sp/208231100/playManifest/protocol/http/entryId/0_89q6e8ku/format/applehttp/tags/mobile_sd/f/a.m3u8',
@@ -738,64 +478,7 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'width': 1280,
                     'height': 720,
                 }]
-            ),
-            (
-                # https://github.com/ytdl-org/youtube-dl/issues/18923
-                # https://www.ted.com/talks/boris_hesser_a_grassroots_healthcare_revolution_in_africa
-                'ted_18923',
-                'http://hls.ted.com/talks/31241.m3u8',
-                [{
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/audio/600k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '600k-Audio',
-                    'vcodec': 'none',
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/audio/600k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '68',
-                    'vcodec': 'none',
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/64k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '163',
-                    'acodec': 'none',
-                    'width': 320,
-                    'height': 180,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/180k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '481',
-                    'acodec': 'none',
-                    'width': 512,
-                    'height': 288,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/320k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '769',
-                    'acodec': 'none',
-                    'width': 512,
-                    'height': 288,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/450k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '984',
-                    'acodec': 'none',
-                    'width': 512,
-                    'height': 288,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/600k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '1255',
-                    'acodec': 'none',
-                    'width': 640,
-                    'height': 360,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/950k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '1693',
-                    'acodec': 'none',
-                    'width': 853,
-                    'height': 480,
-                }, {
-                    'url': 'http://hls.ted.com/videos/BorisHesser_2018S/video/1500k.m3u8?nobumpers=true&uniqueId=76011e2b',
-                    'format_id': '2462',
-                    'acodec': 'none',
-                    'width': 1280,
-                    'height': 720,
-                }]
-            ),
+            )
         ]
 
         for m3u8_file, m3u8_url, expected_formats in _TEST_CASES:
@@ -809,12 +492,11 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
     def test_parse_mpd_formats(self):
         _TEST_CASES = [
             (
-                # https://github.com/ytdl-org/youtube-dl/issues/13919
+                # https://github.com/rg3/youtube-dl/issues/13919
                 # Also tests duplicate representation ids, see
-                # https://github.com/ytdl-org/youtube-dl/issues/15111
+                # https://github.com/rg3/youtube-dl/issues/15111
                 'float_duration',
-                'http://unknown/manifest.mpd',  # mpd_url
-                None,  # mpd_base_url
+                'http://unknown/manifest.mpd',
                 [{
                     'manifest_url': 'http://unknown/manifest.mpd',
                     'ext': 'm4a',
@@ -892,10 +574,9 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'height': 1080,
                 }]
             ), (
-                # https://github.com/ytdl-org/youtube-dl/pull/14844
+                # https://github.com/rg3/youtube-dl/pull/14844
                 'urls_only',
-                'http://unknown/manifest.mpd',  # mpd_url
-                None,  # mpd_base_url
+                'http://unknown/manifest.mpd',
                 [{
                     'manifest_url': 'http://unknown/manifest.mpd',
                     'ext': 'mp4',
@@ -974,68 +655,22 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                     'width': 1920,
                     'height': 1080,
                 }]
-            ), (
-                # https://github.com/ytdl-org/youtube-dl/issues/20346
-                # Media considered unfragmented even though it contains
-                # Initialization tag
-                'unfragmented',
-                'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',  # mpd_url
-                'https://v.redd.it/hw1x7rcg7zl21',  # mpd_base_url
-                [{
-                    'url': 'https://v.redd.it/hw1x7rcg7zl21/audio',
-                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
-                    'ext': 'm4a',
-                    'format_id': 'AUDIO-1',
-                    'format_note': 'DASH audio',
-                    'container': 'm4a_dash',
-                    'acodec': 'mp4a.40.2',
-                    'vcodec': 'none',
-                    'tbr': 129.87,
-                    'asr': 48000,
-
-                }, {
-                    'url': 'https://v.redd.it/hw1x7rcg7zl21/DASH_240',
-                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
-                    'ext': 'mp4',
-                    'format_id': 'VIDEO-2',
-                    'format_note': 'DASH video',
-                    'container': 'mp4_dash',
-                    'acodec': 'none',
-                    'vcodec': 'avc1.4d401e',
-                    'tbr': 608.0,
-                    'width': 240,
-                    'height': 240,
-                    'fps': 30,
-                }, {
-                    'url': 'https://v.redd.it/hw1x7rcg7zl21/DASH_360',
-                    'manifest_url': 'https://v.redd.it/hw1x7rcg7zl21/DASHPlaylist.mpd',
-                    'ext': 'mp4',
-                    'format_id': 'VIDEO-1',
-                    'format_note': 'DASH video',
-                    'container': 'mp4_dash',
-                    'acodec': 'none',
-                    'vcodec': 'avc1.4d401e',
-                    'tbr': 804.261,
-                    'width': 360,
-                    'height': 360,
-                    'fps': 30,
-                }]
             )
         ]
 
-        for mpd_file, mpd_url, mpd_base_url, expected_formats in _TEST_CASES:
+        for mpd_file, mpd_url, expected_formats in _TEST_CASES:
             with io.open('./test/testdata/mpd/%s.mpd' % mpd_file,
                          mode='r', encoding='utf-8') as f:
                 formats = self.ie._parse_mpd_formats(
                     compat_etree_fromstring(f.read().encode('utf-8')),
-                    mpd_base_url=mpd_base_url, mpd_url=mpd_url)
+                    mpd_url=mpd_url)
                 self.ie._sort_formats(formats)
                 expect_value(self, formats, expected_formats, None)
 
     def test_parse_f4m_formats(self):
         _TEST_CASES = [
             (
-                # https://github.com/ytdl-org/youtube-dl/issues/14660
+                # https://github.com/rg3/youtube-dl/issues/14660
                 'custom_base_url',
                 'http://api.new.livestream.com/accounts/6115179/events/6764928/videos/144884262.f4m',
                 [{
@@ -1059,74 +694,6 @@ jwplayer("mediaplayer").setup({"abouttext":"Visit Indie DB","aboutlink":"http:\/
                 self.ie._sort_formats(formats)
                 expect_value(self, formats, expected_formats, None)
 
-    def test_parse_xspf(self):
-        _TEST_CASES = [
-            (
-                'foo_xspf',
-                'https://example.org/src/foo_xspf.xspf',
-                [{
-                    'id': 'foo_xspf',
-                    'title': 'Pandemonium',
-                    'description': 'Visit http://bigbrother404.bandcamp.com',
-                    'duration': 202.416,
-                    'formats': [{
-                        'manifest_url': 'https://example.org/src/foo_xspf.xspf',
-                        'url': 'https://example.org/src/cd1/track%201.mp3',
-                    }],
-                }, {
-                    'id': 'foo_xspf',
-                    'title': 'Final Cartridge (Nichico Twelve Remix)',
-                    'description': 'Visit http://bigbrother404.bandcamp.com',
-                    'duration': 255.857,
-                    'formats': [{
-                        'manifest_url': 'https://example.org/src/foo_xspf.xspf',
-                        'url': 'https://example.org/%E3%83%88%E3%83%A9%E3%83%83%E3%82%AF%E3%80%80%EF%BC%92.mp3',
-                    }],
-                }, {
-                    'id': 'foo_xspf',
-                    'title': 'Rebuilding Nightingale',
-                    'description': 'Visit http://bigbrother404.bandcamp.com',
-                    'duration': 287.915,
-                    'formats': [{
-                        'manifest_url': 'https://example.org/src/foo_xspf.xspf',
-                        'url': 'https://example.org/src/track3.mp3',
-                    }, {
-                        'manifest_url': 'https://example.org/src/foo_xspf.xspf',
-                        'url': 'https://example.com/track3.mp3',
-                    }]
-                }]
-            ),
-        ]
-
-        for xspf_file, xspf_url, expected_entries in _TEST_CASES:
-            with io.open('./test/testdata/xspf/%s.xspf' % xspf_file,
-                         mode='r', encoding='utf-8') as f:
-                entries = self.ie._parse_xspf(
-                    compat_etree_fromstring(f.read().encode('utf-8')),
-                    xspf_file, xspf_url=xspf_url, xspf_base_url=xspf_url)
-                expect_value(self, entries, expected_entries, None)
-                for i in range(len(entries)):
-                    expect_dict(self, entries[i], expected_entries[i])
-
-    def test_response_with_expected_status_returns_content(self):
-        # Checks for mitigations against the effects of
-        # <https://bugs.python.org/issue15002> that affect Python 3.4.1+, which
-        # manifest as `_download_webpage`, `_download_xml`, `_download_json`,
-        # or the underlying `_download_webpage_handle` returning no content
-        # when a response matches `expected_status`.
-
-        httpd = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), InfoExtractorTestRequestHandler)
-        port = http_server_port(httpd)
-        server_thread = threading.Thread(target=httpd.serve_forever)
-        server_thread.daemon = True
-        server_thread.start()
-
-        (content, urlh) = self.ie._download_webpage_handle(
-            'http://127.0.0.1:%d/teapot' % port, None,
-            expected_status=TEAPOT_RESPONSE_STATUS)
-        self.assertEqual(content, TEAPOT_RESPONSE_BODY)
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_YoutubeDL.py

@@ -239,76 +239,6 @@ class TestFormatSelection(unittest.TestCase):
         downloaded = ydl.downloaded_info_dicts[0]
         self.assertEqual(downloaded['format_id'], 'vid-vcodec-dot')
 
-    def test_format_selection_string_ops(self):
-        formats = [
-            {'format_id': 'abc-cba', 'ext': 'mp4', 'url': TEST_URL},
-            {'format_id': 'zxc-cxz', 'ext': 'webm', 'url': TEST_URL},
-        ]
-        info_dict = _make_result(formats)
-
-        # equals (=)
-        ydl = YDL({'format': '[format_id=abc-cba]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'abc-cba')
-
-        # does not equal (!=)
-        ydl = YDL({'format': '[format_id!=abc-cba]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'zxc-cxz')
-
-        ydl = YDL({'format': '[format_id!=abc-cba][format_id!=zxc-cxz]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
-        # starts with (^=)
-        ydl = YDL({'format': '[format_id^=abc]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'abc-cba')
-
-        # does not start with (!^=)
-        ydl = YDL({'format': '[format_id!^=abc]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'zxc-cxz')
-
-        ydl = YDL({'format': '[format_id!^=abc][format_id!^=zxc]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
-        # ends with ($=)
-        ydl = YDL({'format': '[format_id$=cba]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'abc-cba')
-
-        # does not end with (!$=)
-        ydl = YDL({'format': '[format_id!$=cba]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'zxc-cxz')
-
-        ydl = YDL({'format': '[format_id!$=cba][format_id!$=cxz]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
-        # contains (*=)
-        ydl = YDL({'format': '[format_id*=bc-cb]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'abc-cba')
-
-        # does not contain (!*=)
-        ydl = YDL({'format': '[format_id!*=bc-cb]'})
-        ydl.process_ie_result(info_dict.copy())
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(downloaded['format_id'], 'zxc-cxz')
-
-        ydl = YDL({'format': '[format_id!*=abc][format_id!*=zxc]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
-        ydl = YDL({'format': '[format_id!*=-]'})
-        self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
-
     def test_youtube_format_selection(self):
         order = [
             '38', '37', '46', '22', '45', '35', '44', '18', '34', '43', '6', '5', '17', '36', '13',
@@ -411,7 +341,7 @@ class TestFormatSelection(unittest.TestCase):
         # For extractors with incomplete formats (all formats are audio-only or
         # video-only) best and worst should fallback to corresponding best/worst
         # video-only or audio-only formats (as per
-        # https://github.com/ytdl-org/youtube-dl/pull/5556)
+        # https://github.com/rg3/youtube-dl/pull/5556)
         formats = [
             {'format_id': 'low', 'ext': 'mp3', 'preference': 1, 'vcodec': 'none', 'url': TEST_URL},
             {'format_id': 'high', 'ext': 'mp3', 'preference': 2, 'vcodec': 'none', 'url': TEST_URL},
@@ -442,7 +372,7 @@ class TestFormatSelection(unittest.TestCase):
         self.assertRaises(ExtractorError, ydl.process_ie_result, info_dict.copy())
 
     def test_format_selection_issue_10083(self):
-        # See https://github.com/ytdl-org/youtube-dl/issues/10083
+        # See https://github.com/rg3/youtube-dl/issues/10083
         formats = [
             {'format_id': 'regular', 'height': 360, 'url': TEST_URL},
             {'format_id': 'video', 'height': 720, 'acodec': 'none', 'url': TEST_URL},
@@ -464,7 +394,6 @@ class TestFormatSelection(unittest.TestCase):
         assert_syntax_error('+bestaudio')
         assert_syntax_error('bestvideo+')
         assert_syntax_error('/')
-        assert_syntax_error('bestvideo+bestvideo+bestaudio')
 
     def test_format_filtering(self):
         formats = [
@@ -633,20 +562,13 @@ class TestYoutubeDL(unittest.TestCase):
             'title2': '%PATH%',
         }
 
-        def fname(templ, na_placeholder='NA'):
-            params = {'outtmpl': templ}
-            if na_placeholder != 'NA':
-                params['outtmpl_na_placeholder'] = na_placeholder
-            ydl = YoutubeDL(params)
+        def fname(templ):
+            ydl = YoutubeDL({'outtmpl': templ})
             return ydl.prepare_filename(info)
         self.assertEqual(fname('%(id)s.%(ext)s'), '1234.mp4')
         self.assertEqual(fname('%(id)s-%(width)s.%(ext)s'), '1234-NA.mp4')
-        NA_TEST_OUTTMPL = '%(uploader_date)s-%(width)d-%(id)s.%(ext)s'
-        # Replace missing fields with 'NA' by default
-        self.assertEqual(fname(NA_TEST_OUTTMPL), 'NA-NA-1234.mp4')
-        # Or by provided placeholder
-        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder='none'), 'none-none-1234.mp4')
-        self.assertEqual(fname(NA_TEST_OUTTMPL, na_placeholder=''), '--1234.mp4')
+        # Replace missing fields with 'NA'
+        self.assertEqual(fname('%(uploader_date)s-%(id)s.%(ext)s'), 'NA-1234.mp4')
         self.assertEqual(fname('%(height)d.%(ext)s'), '1080.mp4')
         self.assertEqual(fname('%(height)6d.%(ext)s'), '  1080.mp4')
         self.assertEqual(fname('%(height)-6d.%(ext)s'), '1080  .mp4')
@@ -824,15 +746,11 @@ class TestYoutubeDL(unittest.TestCase):
             'webpage_url': 'http://example.com',
         }
 
-        def get_downloaded_info_dicts(params):
-            ydl = YDL(params)
-            # make a deep copy because the dictionary and nested entries
-            # can be modified
-            ydl.process_ie_result(copy.deepcopy(playlist))
-            return ydl.downloaded_info_dicts
-
         def get_ids(params):
-            return [int(v['id']) for v in get_downloaded_info_dicts(params)]
+            ydl = YDL(params)
+            # make a copy because the dictionary can be modified
+            ydl.process_ie_result(playlist.copy())
+            return [int(v['id']) for v in ydl.downloaded_info_dicts]
 
         result = get_ids({})
         self.assertEqual(result, [1, 2, 3, 4])
@@ -864,24 +782,8 @@ class TestYoutubeDL(unittest.TestCase):
         result = get_ids({'playlist_items': '2-4,3-4,3'})
         self.assertEqual(result, [2, 3, 4])
 
-        # Tests for https://github.com/ytdl-org/youtube-dl/issues/10591
-        # @{
-        result = get_downloaded_info_dicts({'playlist_items': '2-4,3-4,3'})
-        self.assertEqual(result[0]['playlist_index'], 2)
-        self.assertEqual(result[1]['playlist_index'], 3)
-
-        result = get_downloaded_info_dicts({'playlist_items': '2-4,3-4,3'})
-        self.assertEqual(result[0]['playlist_index'], 2)
-        self.assertEqual(result[1]['playlist_index'], 3)
-        self.assertEqual(result[2]['playlist_index'], 4)
-
-        result = get_downloaded_info_dicts({'playlist_items': '4,2'})
-        self.assertEqual(result[0]['playlist_index'], 4)
-        self.assertEqual(result[1]['playlist_index'], 2)
-        # @}
-
     def test_urlopen_no_file_protocol(self):
-        # see https://github.com/ytdl-org/youtube-dl/issues/8227
+        # see https://github.com/rg3/youtube-dl/issues/8227
         ydl = YDL()
         self.assertRaises(compat_urllib_error.URLError, ydl.urlopen, 'file:///etc/passwd')
 
@@ -927,95 +829,6 @@ class TestYoutubeDL(unittest.TestCase):
         self.assertEqual(downloaded['extractor'], 'testex')
         self.assertEqual(downloaded['extractor_key'], 'TestEx')
 
-    # Test case for https://github.com/ytdl-org/youtube-dl/issues/27064
-    def test_ignoreerrors_for_playlist_with_url_transparent_iterable_entries(self):
-
-        class _YDL(YDL):
-            def __init__(self, *args, **kwargs):
-                super(_YDL, self).__init__(*args, **kwargs)
-
-            def trouble(self, s, tb=None):
-                pass
-
-        ydl = _YDL({
-            'format': 'extra',
-            'ignoreerrors': True,
-        })
-
-        class VideoIE(InfoExtractor):
-            _VALID_URL = r'video:(?P<id>\d+)'
-
-            def _real_extract(self, url):
-                video_id = self._match_id(url)
-                formats = [{
-                    'format_id': 'default',
-                    'url': 'url:',
-                }]
-                if video_id == '0':
-                    raise ExtractorError('foo')
-                if video_id == '2':
-                    formats.append({
-                        'format_id': 'extra',
-                        'url': TEST_URL,
-                    })
-                return {
-                    'id': video_id,
-                    'title': 'Video %s' % video_id,
-                    'formats': formats,
-                }
-
-        class PlaylistIE(InfoExtractor):
-            _VALID_URL = r'playlist:'
-
-            def _entries(self):
-                for n in range(3):
-                    video_id = compat_str(n)
-                    yield {
-                        '_type': 'url_transparent',
-                        'ie_key': VideoIE.ie_key(),
-                        'id': video_id,
-                        'url': 'video:%s' % video_id,
-                        'title': 'Video Transparent %s' % video_id,
-                    }
-
-            def _real_extract(self, url):
-                return self.playlist_result(self._entries())
-
-        ydl.add_info_extractor(VideoIE(ydl))
-        ydl.add_info_extractor(PlaylistIE(ydl))
-        info = ydl.extract_info('playlist:')
-        entries = info['entries']
-        self.assertEqual(len(entries), 3)
-        self.assertTrue(entries[0] is None)
-        self.assertTrue(entries[1] is None)
-        self.assertEqual(len(ydl.downloaded_info_dicts), 1)
-        downloaded = ydl.downloaded_info_dicts[0]
-        self.assertEqual(entries[2], downloaded)
-        self.assertEqual(downloaded['url'], TEST_URL)
-        self.assertEqual(downloaded['title'], 'Video Transparent 2')
-        self.assertEqual(downloaded['id'], '2')
-        self.assertEqual(downloaded['extractor'], 'Video')
-        self.assertEqual(downloaded['extractor_key'], 'Video')
-
-    def test_default_times(self):
-        """Test addition of missing upload/release/_date from /release_/timestamp"""
-        info = {
-            'id': '1234',
-            'url': TEST_URL,
-            'title': 'Title',
-            'ext': 'mp4',
-            'timestamp': 1631352900,
-            'release_timestamp': 1632995931,
-        }
-
-        params = {'simulate': True, }
-        ydl = FakeYDL(params)
-        out_info = ydl.process_ie_result(info)
-        self.assertTrue(isinstance(out_info['upload_date'], compat_str))
-        self.assertEqual(out_info['upload_date'], '20210911')
-        self.assertTrue(isinstance(out_info['release_date'], compat_str))
-        self.assertEqual(out_info['release_date'], '20210930')
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_YoutubeDLCookieJar.py

@@ -1,51 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-
-from __future__ import unicode_literals
-
-import os
-import re
-import sys
-import tempfile
-import unittest
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from youtube_dl.utils import YoutubeDLCookieJar
-
-
-class TestYoutubeDLCookieJar(unittest.TestCase):
-    def test_keep_session_cookies(self):
-        cookiejar = YoutubeDLCookieJar('./test/testdata/cookies/session_cookies.txt')
-        cookiejar.load(ignore_discard=True, ignore_expires=True)
-        tf = tempfile.NamedTemporaryFile(delete=False)
-        try:
-            cookiejar.save(filename=tf.name, ignore_discard=True, ignore_expires=True)
-            temp = tf.read().decode('utf-8')
-            self.assertTrue(re.search(
-                r'www\.foobar\.foobar\s+FALSE\s+/\s+TRUE\s+0\s+YoutubeDLExpiresEmpty\s+YoutubeDLExpiresEmptyValue', temp))
-            self.assertTrue(re.search(
-                r'www\.foobar\.foobar\s+FALSE\s+/\s+TRUE\s+0\s+YoutubeDLExpires0\s+YoutubeDLExpires0Value', temp))
-        finally:
-            tf.close()
-            os.remove(tf.name)
-
-    def test_strip_httponly_prefix(self):
-        cookiejar = YoutubeDLCookieJar('./test/testdata/cookies/httponly_cookies.txt')
-        cookiejar.load(ignore_discard=True, ignore_expires=True)
-
-        def assert_cookie_has_value(key):
-            self.assertEqual(cookiejar._cookies['www.foobar.foobar']['/'][key].value, key + '_VALUE')
-
-        assert_cookie_has_value('HTTPONLY_COOKIE')
-        assert_cookie_has_value('JS_ACCESSIBLE_COOKIE')
-
-    def test_malformed_cookies(self):
-        cookiejar = YoutubeDLCookieJar('./test/testdata/cookies/malformed_cookies.txt')
-        cookiejar.load(ignore_discard=True, ignore_expires=True)
-        # Cookies should be empty since all malformed cookie file entries
-        # will be ignored
-        self.assertFalse(cookiejar._cookies)
-
-
-if __name__ == '__main__':
-    unittest.main()

test/test_aes.py

@@ -8,7 +8,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from youtube_dl.aes import aes_decrypt, aes_encrypt, aes_cbc_decrypt, aes_cbc_encrypt, aes_decrypt_text, aes_ecb_encrypt
+from youtube_dl.aes import aes_decrypt, aes_encrypt, aes_cbc_decrypt, aes_cbc_encrypt, aes_decrypt_text
 from youtube_dl.utils import bytes_to_intlist, intlist_to_bytes
 import base64
 
@@ -44,27 +44,20 @@ class TestAES(unittest.TestCase):
     def test_decrypt_text(self):
         password = intlist_to_bytes(self.key).decode('utf-8')
         encrypted = base64.b64encode(
-            intlist_to_bytes(self.iv[:8])
-            + b'\x17\x15\x93\xab\x8d\x80V\xcdV\xe0\t\xcdo\xc2\xa5\xd8ksM\r\xe27N\xae'
+            intlist_to_bytes(self.iv[:8]) +
+            b'\x17\x15\x93\xab\x8d\x80V\xcdV\xe0\t\xcdo\xc2\xa5\xd8ksM\r\xe27N\xae'
         ).decode('utf-8')
         decrypted = (aes_decrypt_text(encrypted, password, 16))
         self.assertEqual(decrypted, self.secret_msg)
 
         password = intlist_to_bytes(self.key).decode('utf-8')
         encrypted = base64.b64encode(
-            intlist_to_bytes(self.iv[:8])
-            + b'\x0b\xe6\xa4\xd9z\x0e\xb8\xb9\xd0\xd4i_\x85\x1d\x99\x98_\xe5\x80\xe7.\xbf\xa5\x83'
+            intlist_to_bytes(self.iv[:8]) +
+            b'\x0b\xe6\xa4\xd9z\x0e\xb8\xb9\xd0\xd4i_\x85\x1d\x99\x98_\xe5\x80\xe7.\xbf\xa5\x83'
         ).decode('utf-8')
         decrypted = (aes_decrypt_text(encrypted, password, 32))
         self.assertEqual(decrypted, self.secret_msg)
 
-    def test_ecb_encrypt(self):
-        data = bytes_to_intlist(self.secret_msg)
-        encrypted = intlist_to_bytes(aes_ecb_encrypt(data, self.key))
-        self.assertEqual(
-            encrypted,
-            b'\xaa\x86]\x81\x97>\x02\x92\x9d\x1bR[[L/u\xd3&\xd1(h\xde{\x81\x94\xba\x02\xae\xbd\xa6\xd0:')
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_all_urls.py

@@ -31,17 +31,16 @@ class TestAllURLsMatching(unittest.TestCase):
 
     def test_youtube_playlist_matching(self):
         assertPlaylist = lambda url: self.assertMatch(url, ['youtube:playlist'])
-        assertTab = lambda url: self.assertMatch(url, ['youtube:tab'])
         assertPlaylist('ECUl4u3cNGP61MdtwGTqZA0MreSaDybji8')
         assertPlaylist('UUBABnxM4Ar9ten8Mdjj1j0Q')  # 585
         assertPlaylist('PL63F0C78739B09958')
-        assertTab('https://www.youtube.com/playlist?list=UUBABnxM4Ar9ten8Mdjj1j0Q')
-        assertTab('https://www.youtube.com/course?list=ECUl4u3cNGP61MdtwGTqZA0MreSaDybji8')
-        assertTab('https://www.youtube.com/playlist?list=PLwP_SiAcdui0KVebT0mU9Apz359a4ubsC')
-        assertTab('https://www.youtube.com/watch?v=AV6J6_AeFEQ&playnext=1&list=PL4023E734DA416012')  # 668
+        assertPlaylist('https://www.youtube.com/playlist?list=UUBABnxM4Ar9ten8Mdjj1j0Q')
+        assertPlaylist('https://www.youtube.com/course?list=ECUl4u3cNGP61MdtwGTqZA0MreSaDybji8')
+        assertPlaylist('https://www.youtube.com/playlist?list=PLwP_SiAcdui0KVebT0mU9Apz359a4ubsC')
+        assertPlaylist('https://www.youtube.com/watch?v=AV6J6_AeFEQ&playnext=1&list=PL4023E734DA416012')  # 668
         self.assertFalse('youtube:playlist' in self.matching_ies('PLtS2H6bU1M'))
         # Top tracks
-        assertTab('https://www.youtube.com/playlist?list=MCUS.20142101')
+        assertPlaylist('https://www.youtube.com/playlist?list=MCUS.20142101')
 
     def test_youtube_matching(self):
         self.assertTrue(YoutubeIE.suitable('PLtS2H6bU1M'))
@@ -52,24 +51,36 @@ class TestAllURLsMatching(unittest.TestCase):
         self.assertMatch('http://www.cleanvideosearch.com/media/action/yt/watch?videoId=8v_4O44sfjM', ['youtube'])
 
     def test_youtube_channel_matching(self):
-        assertChannel = lambda url: self.assertMatch(url, ['youtube:tab'])
+        assertChannel = lambda url: self.assertMatch(url, ['youtube:channel'])
         assertChannel('https://www.youtube.com/channel/HCtnHdj3df7iM')
         assertChannel('https://www.youtube.com/channel/HCtnHdj3df7iM?feature=gb_ch_rec')
         assertChannel('https://www.youtube.com/channel/HCtnHdj3df7iM/videos')
 
     def test_youtube_user_matching(self):
-        self.assertMatch('http://www.youtube.com/NASAgovVideo/videos', ['youtube:tab'])
+        self.assertMatch('http://www.youtube.com/NASAgovVideo/videos', ['youtube:user'])
 
     def test_youtube_feeds(self):
-        self.assertMatch('https://www.youtube.com/feed/library', ['youtube:tab'])
-        self.assertMatch('https://www.youtube.com/feed/history', ['youtube:tab'])
-        self.assertMatch('https://www.youtube.com/feed/watch_later', ['youtube:tab'])
-        self.assertMatch('https://www.youtube.com/feed/subscriptions', ['youtube:tab'])
+        self.assertMatch('https://www.youtube.com/feed/watch_later', ['youtube:watchlater'])
+        self.assertMatch('https://www.youtube.com/feed/subscriptions', ['youtube:subscriptions'])
+        self.assertMatch('https://www.youtube.com/feed/recommended', ['youtube:recommended'])
+        self.assertMatch('https://www.youtube.com/my_favorites', ['youtube:favorites'])
+
+    def test_youtube_show_matching(self):
+        self.assertMatch('http://www.youtube.com/show/airdisasters', ['youtube:show'])
 
     def test_youtube_search_matching(self):
         self.assertMatch('http://www.youtube.com/results?search_query=making+mustard', ['youtube:search_url'])
         self.assertMatch('https://www.youtube.com/results?baz=bar&search_query=youtube-dl+test+video&filters=video&lclk=video', ['youtube:search_url'])
 
+    def test_youtube_extract(self):
+        assertExtractId = lambda url, id: self.assertEqual(YoutubeIE.extract_id(url), id)
+        assertExtractId('http://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
+        assertExtractId('https://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
+        assertExtractId('https://www.youtube.com/watch?feature=player_embedded&v=BaW_jenozKc', 'BaW_jenozKc')
+        assertExtractId('https://www.youtube.com/watch_popup?v=BaW_jenozKc', 'BaW_jenozKc')
+        assertExtractId('http://www.youtube.com/watch?v=BaW_jenozKcsharePLED17F32AD9753930', 'BaW_jenozKc')
+        assertExtractId('BaW_jenozKc', 'BaW_jenozKc')
+
     def test_facebook_matching(self):
         self.assertTrue(FacebookIE.suitable('https://www.facebook.com/Shiniknoh#!/photo.php?v=10153317450565268'))
         self.assertTrue(FacebookIE.suitable('https://www.facebook.com/cindyweather?fref=ts#!/photo.php?v=10152183998945793'))
@@ -99,7 +110,7 @@ class TestAllURLsMatching(unittest.TestCase):
         self.assertMatch('https://vimeo.com/user7108434/videos', ['vimeo:user'])
         self.assertMatch('https://vimeo.com/user21297594/review/75524534/3c257a1b5d', ['vimeo:review'])
 
-    # https://github.com/ytdl-org/youtube-dl/issues/1930
+    # https://github.com/rg3/youtube-dl/issues/1930
     def test_soundcloud_not_matching_sets(self):
         self.assertMatch('http://soundcloud.com/floex/sets/gone-ep', ['soundcloud:set'])
 
@@ -108,10 +119,16 @@ class TestAllURLsMatching(unittest.TestCase):
         self.assertMatch('http://tatianamaslanydaily.tumblr.com/post/54196191430', ['Tumblr'])
 
     def test_pbs(self):
-        # https://github.com/ytdl-org/youtube-dl/issues/2350
+        # https://github.com/rg3/youtube-dl/issues/2350
         self.assertMatch('http://video.pbs.org/viralplayer/2365173446/', ['pbs'])
         self.assertMatch('http://video.pbs.org/widget/partnerplayer/980042464/', ['pbs'])
 
+    def test_yahoo_https(self):
+        # https://github.com/rg3/youtube-dl/issues/2701
+        self.assertMatch(
+            'https://screen.yahoo.com/smartwatches-latest-wearable-gadgets-163745379-cbs.html',
+            ['Yahoo'])
+
     def test_no_duplicated_ie_names(self):
         name_accu = collections.defaultdict(list)
         for ie in self.ies:

test/test_cache.py

@@ -3,18 +3,17 @@
 
 from __future__ import unicode_literals
 
+import shutil
+
 # Allow direct execution
 import os
 import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-import shutil
 
 from test.helper import FakeYDL
 from youtube_dl.cache import Cache
-from youtube_dl.utils import version_tuple
-from youtube_dl.version import __version__
 
 
 def _is_empty(d):
@@ -55,17 +54,6 @@ class TestCache(unittest.TestCase):
         self.assertFalse(os.path.exists(self.test_dir))
         self.assertEqual(c.load('test_cache', 'k.'), None)
 
-    def test_cache_validation(self):
-        ydl = FakeYDL({
-            'cachedir': self.test_dir,
-        })
-        c = Cache(ydl)
-        obj = {'x': 1, 'y': ['ä', '\\a', True]}
-        c.store('test_cache', 'k.', obj)
-        self.assertEqual(c.load('test_cache', 'k.', min_ver='1970.01.01'), obj)
-        new_version = '.'.join(('%d' % ((v + 1) if i == 0 else v, )) for i, v in enumerate(version_tuple(__version__)))
-        self.assertIs(c.load('test_cache', 'k.', min_ver=new_version), None)
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_compat.py

@@ -11,10 +11,8 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 
 from youtube_dl.compat import (
-    compat_casefold,
     compat_getenv,
     compat_setenv,
-    compat_etree_Element,
     compat_etree_fromstring,
     compat_expanduser,
     compat_shlex_split,
@@ -41,7 +39,7 @@ class TestCompat(unittest.TestCase):
 
     def test_compat_expanduser(self):
         old_home = os.environ.get('HOME')
-        test_str = r'C:\Documents and Settings\тест\Application Data'
+        test_str = 'C:\Documents and Settings\тест\Application Data'
         compat_setenv('HOME', test_str)
         self.assertEqual(compat_expanduser('~'), test_str)
         compat_setenv('HOME', old_home or '')
@@ -92,12 +90,6 @@ class TestCompat(unittest.TestCase):
         self.assertEqual(compat_shlex_split('-option "one\ntwo" \n -flag'), ['-option', 'one\ntwo', '-flag'])
         self.assertEqual(compat_shlex_split('-val 中文'), ['-val', '中文'])
 
-    def test_compat_etree_Element(self):
-        try:
-            compat_etree_Element.items
-        except AttributeError:
-            self.fail('compat_etree_Element is not a type')
-
     def test_compat_etree_fromstring(self):
         xml = '''
             <root foo="bar" spam="中文">
@@ -119,21 +111,9 @@ class TestCompat(unittest.TestCase):
 <smil xmlns="http://www.w3.org/2001/SMIL20/Language"></smil>'''
         compat_etree_fromstring(xml)
 
-    def test_compat_struct_unpack(self):
+    def test_struct_unpack(self):
         self.assertEqual(compat_struct_unpack('!B', b'\x00'), (0,))
 
-    def test_compat_casefold(self):
-        if hasattr(compat_str, 'casefold'):
-            # don't bother to test str.casefold() (again)
-            return
-        # thanks https://bugs.python.org/file24232/casefolding.patch
-        self.assertEqual(compat_casefold('hello'), 'hello')
-        self.assertEqual(compat_casefold('hELlo'), 'hello')
-        self.assertEqual(compat_casefold('ß'), 'ss')
-        self.assertEqual(compat_casefold('fi'), 'fi')
-        self.assertEqual(compat_casefold('\u03a3'), '\u03c3')
-        self.assertEqual(compat_casefold('A\u0345\u03a3'), 'a\u03b9\u03c3')
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_download.py

@@ -33,7 +33,6 @@ from youtube_dl.compat import (
 from youtube_dl.utils import (
     DownloadError,
     ExtractorError,
-    error_to_compat_str,
     format_bytes,
     UnavailableVideoError,
 )
@@ -93,36 +92,35 @@ class TestDownload(unittest.TestCase):
 def generator(test_case, tname):
 
     def test_template(self):
-        ie = youtube_dl.extractor.get_info_extractor(test_case['name'])()
-        other_ies = [get_info_extractor(ie_key)() for ie_key in test_case.get('add_ie', [])]
+        ie = youtube_dl.extractor.get_info_extractor(test_case['name'])
+        other_ies = [get_info_extractor(ie_key) for ie_key in test_case.get('add_ie', [])]
         is_playlist = any(k.startswith('playlist') for k in test_case)
         test_cases = test_case.get(
             'playlist', [] if is_playlist else [test_case])
 
         def print_skipping(reason):
             print('Skipping %s: %s' % (test_case['name'], reason))
-            self.skipTest(reason)
-
         if not ie.working():
             print_skipping('IE marked as not _WORKING')
+            return
 
         for tc in test_cases:
             info_dict = tc.get('info_dict', {})
             if not (info_dict.get('id') and info_dict.get('ext')):
-                raise Exception('Test definition (%s) requires both \'id\' and \'ext\' keys present to define the output file' % (tname, ))
+                raise Exception('Test definition incorrect. The output file cannot be known. Are both \'id\' and \'ext\' keys present?')
 
         if 'skip' in test_case:
             print_skipping(test_case['skip'])
-
+            return
         for other_ie in other_ies:
             if not other_ie.working():
                 print_skipping('test depends on %sIE, marked as not WORKING' % other_ie.ie_key())
+                return
 
         params = get_params(test_case.get('params', {}))
         params['outtmpl'] = tname + '_' + params['outtmpl']
         if is_playlist and 'playlist' not in test_case:
             params.setdefault('extract_flat', 'in_playlist')
-            params.setdefault('playlistend', test_case.get('playlist_mincount'))
             params.setdefault('skip_download', True)
 
         ydl = YoutubeDL(params, auto_init=False)
@@ -162,9 +160,7 @@ def generator(test_case, tname):
                 except (DownloadError, ExtractorError) as err:
                     # Check if the exception is not a network related one
                     if not err.exc_info[0] in (compat_urllib_error.URLError, socket.timeout, UnavailableVideoError, compat_http_client.BadStatusLine) or (err.exc_info[0] == compat_HTTPError and err.exc_info[1].code == 503):
-                        msg = getattr(err, 'msg', error_to_compat_str(err))
-                        err.msg = '%s (%s)' % (msg, tname, )
-                        raise err
+                        raise
 
                     if try_num == RETRIES:
                         report_warning('%s failed due to network errors, skipping...' % tname)

test/test_downloader_http.py

@@ -1,115 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-from __future__ import unicode_literals
-
-# Allow direct execution
-import os
-import re
-import sys
-import unittest
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-from test.helper import http_server_port, try_rm
-from youtube_dl import YoutubeDL
-from youtube_dl.compat import compat_http_server
-from youtube_dl.downloader.http import HttpFD
-from youtube_dl.utils import encodeFilename
-import threading
-
-TEST_DIR = os.path.dirname(os.path.abspath(__file__))
-
-
-TEST_SIZE = 10 * 1024
-
-
-class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
-    def log_message(self, format, *args):
-        pass
-
-    def send_content_range(self, total=None):
-        range_header = self.headers.get('Range')
-        start = end = None
-        if range_header:
-            mobj = re.search(r'^bytes=(\d+)-(\d+)', range_header)
-            if mobj:
-                start = int(mobj.group(1))
-                end = int(mobj.group(2))
-        valid_range = start is not None and end is not None
-        if valid_range:
-            content_range = 'bytes %d-%d' % (start, end)
-            if total:
-                content_range += '/%d' % total
-            self.send_header('Content-Range', content_range)
-        return (end - start + 1) if valid_range else total
-
-    def serve(self, range=True, content_length=True):
-        self.send_response(200)
-        self.send_header('Content-Type', 'video/mp4')
-        size = TEST_SIZE
-        if range:
-            size = self.send_content_range(TEST_SIZE)
-        if content_length:
-            self.send_header('Content-Length', size)
-        self.end_headers()
-        self.wfile.write(b'#' * size)
-
-    def do_GET(self):
-        if self.path == '/regular':
-            self.serve()
-        elif self.path == '/no-content-length':
-            self.serve(content_length=False)
-        elif self.path == '/no-range':
-            self.serve(range=False)
-        elif self.path == '/no-range-no-content-length':
-            self.serve(range=False, content_length=False)
-        else:
-            assert False
-
-
-class FakeLogger(object):
-    def debug(self, msg):
-        pass
-
-    def warning(self, msg):
-        pass
-
-    def error(self, msg):
-        pass
-
-
-class TestHttpFD(unittest.TestCase):
-    def setUp(self):
-        self.httpd = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), HTTPTestRequestHandler)
-        self.port = http_server_port(self.httpd)
-        self.server_thread = threading.Thread(target=self.httpd.serve_forever)
-        self.server_thread.daemon = True
-        self.server_thread.start()
-
-    def download(self, params, ep):
-        params['logger'] = FakeLogger()
-        ydl = YoutubeDL(params)
-        downloader = HttpFD(ydl, params)
-        filename = 'testfile.mp4'
-        try_rm(encodeFilename(filename))
-        self.assertTrue(downloader.real_download(filename, {
-            'url': 'http://127.0.0.1:%d/%s' % (self.port, ep),
-        }))
-        self.assertEqual(os.path.getsize(encodeFilename(filename)), TEST_SIZE)
-        try_rm(encodeFilename(filename))
-
-    def download_all(self, params):
-        for ep in ('regular', 'no-content-length', 'no-range', 'no-range-no-content-length'):
-            self.download(params, ep)
-
-    def test_regular(self):
-        self.download_all({})
-
-    def test_chunked(self):
-        self.download_all({
-            'http_chunk_size': 1000,
-        })
-
-
-if __name__ == '__main__':
-    unittest.main()

test/test_execution.py

@@ -39,16 +39,6 @@ class TestExecution(unittest.TestCase):
         _, stderr = p.communicate()
         self.assertFalse(stderr)
 
-    def test_lazy_extractors(self):
-        try:
-            subprocess.check_call([sys.executable, 'devscripts/make_lazy_extractors.py', 'youtube_dl/extractor/lazy_extractors.py'], cwd=rootDir, stdout=_DEV_NULL)
-            subprocess.check_call([sys.executable, 'test/test_all_urls.py'], cwd=rootDir, stdout=_DEV_NULL)
-        finally:
-            try:
-                os.remove('youtube_dl/extractor/lazy_extractors.py')
-            except (IOError, OSError):
-                pass
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_http.py

@@ -8,7 +8,6 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import http_server_port
 from youtube_dl import YoutubeDL
 from youtube_dl.compat import compat_http_server, compat_urllib_request
 import ssl
@@ -17,6 +16,15 @@ import threading
 TEST_DIR = os.path.dirname(os.path.abspath(__file__))
 
 
+def http_server_port(httpd):
+    if os.name == 'java' and isinstance(httpd.socket, ssl.SSLSocket):
+        # In Jython SSLSocket is not a subclass of socket.socket
+        sock = httpd.socket.sock
+    else:
+        sock = httpd.socket
+    return sock.getsockname()[1]
+
+
 class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
     def log_message(self, format, *args):
         pass
@@ -39,7 +47,7 @@ class HTTPTestRequestHandler(compat_http_server.BaseHTTPRequestHandler):
                 self.end_headers()
                 return
 
-            new_url = 'http://127.0.0.1:%d/中文.html' % http_server_port(self.server)
+            new_url = 'http://localhost:%d/中文.html' % http_server_port(self.server)
             self.send_response(302)
             self.send_header(b'Location', new_url.encode('utf-8'))
             self.end_headers()
@@ -66,7 +74,7 @@ class FakeLogger(object):
 class TestHTTP(unittest.TestCase):
     def setUp(self):
         self.httpd = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), HTTPTestRequestHandler)
+            ('localhost', 0), HTTPTestRequestHandler)
         self.port = http_server_port(self.httpd)
         self.server_thread = threading.Thread(target=self.httpd.serve_forever)
         self.server_thread.daemon = True
@@ -78,15 +86,15 @@ class TestHTTP(unittest.TestCase):
             return
 
         ydl = YoutubeDL({'logger': FakeLogger()})
-        r = ydl.extract_info('http://127.0.0.1:%d/302' % self.port)
-        self.assertEqual(r['entries'][0]['url'], 'http://127.0.0.1:%d/vid.mp4' % self.port)
+        r = ydl.extract_info('http://localhost:%d/302' % self.port)
+        self.assertEqual(r['entries'][0]['url'], 'http://localhost:%d/vid.mp4' % self.port)
 
 
 class TestHTTPS(unittest.TestCase):
     def setUp(self):
         certfn = os.path.join(TEST_DIR, 'testcert.pem')
         self.httpd = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), HTTPTestRequestHandler)
+            ('localhost', 0), HTTPTestRequestHandler)
         self.httpd.socket = ssl.wrap_socket(
             self.httpd.socket, certfile=certfn, server_side=True)
         self.port = http_server_port(self.httpd)
@@ -99,11 +107,11 @@ class TestHTTPS(unittest.TestCase):
             ydl = YoutubeDL({'logger': FakeLogger()})
             self.assertRaises(
                 Exception,
-                ydl.extract_info, 'https://127.0.0.1:%d/video.html' % self.port)
+                ydl.extract_info, 'https://localhost:%d/video.html' % self.port)
 
         ydl = YoutubeDL({'logger': FakeLogger(), 'nocheckcertificate': True})
-        r = ydl.extract_info('https://127.0.0.1:%d/video.html' % self.port)
-        self.assertEqual(r['entries'][0]['url'], 'https://127.0.0.1:%d/vid.mp4' % self.port)
+        r = ydl.extract_info('https://localhost:%d/video.html' % self.port)
+        self.assertEqual(r['entries'][0]['url'], 'https://localhost:%d/vid.mp4' % self.port)
 
 
 def _build_proxy_handler(name):
@@ -124,23 +132,23 @@ def _build_proxy_handler(name):
 class TestProxy(unittest.TestCase):
     def setUp(self):
         self.proxy = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), _build_proxy_handler('normal'))
+            ('localhost', 0), _build_proxy_handler('normal'))
         self.port = http_server_port(self.proxy)
         self.proxy_thread = threading.Thread(target=self.proxy.serve_forever)
         self.proxy_thread.daemon = True
         self.proxy_thread.start()
 
         self.geo_proxy = compat_http_server.HTTPServer(
-            ('127.0.0.1', 0), _build_proxy_handler('geo'))
+            ('localhost', 0), _build_proxy_handler('geo'))
         self.geo_port = http_server_port(self.geo_proxy)
         self.geo_proxy_thread = threading.Thread(target=self.geo_proxy.serve_forever)
         self.geo_proxy_thread.daemon = True
         self.geo_proxy_thread.start()
 
     def test_proxy(self):
-        geo_proxy = '127.0.0.1:{0}'.format(self.geo_port)
+        geo_proxy = 'localhost:{0}'.format(self.geo_port)
         ydl = YoutubeDL({
-            'proxy': '127.0.0.1:{0}'.format(self.port),
+            'proxy': 'localhost:{0}'.format(self.port),
             'geo_verification_proxy': geo_proxy,
         })
         url = 'http://foo.com/bar'
@@ -154,7 +162,7 @@ class TestProxy(unittest.TestCase):
 
     def test_proxy_with_idn(self):
         ydl = YoutubeDL({
-            'proxy': '127.0.0.1:{0}'.format(self.port),
+            'proxy': 'localhost:{0}'.format(self.port),
         })
         url = 'http://中文.tw/'
         response = ydl.urlopen(url).read().decode('utf-8')

test/test_jsinterp.py

@@ -8,12 +8,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-import math
-import re
-
-from youtube_dl.compat import compat_re_Pattern
-
-from youtube_dl.jsinterp import JS_Undefined, JSInterpreter
+from youtube_dl.jsinterp import JSInterpreter
 
 
 class TestJSInterpreter(unittest.TestCase):
@@ -24,9 +19,6 @@ class TestJSInterpreter(unittest.TestCase):
         jsi = JSInterpreter('function x3(){return 42;}')
         self.assertEqual(jsi.call_function('x3'), 42)
 
-        jsi = JSInterpreter('function x3(){42}')
-        self.assertEqual(jsi.call_function('x3'), None)
-
         jsi = JSInterpreter('var x5 = function(){return 42;}')
         self.assertEqual(jsi.call_function('x5'), 42)
 
@@ -53,32 +45,14 @@ class TestJSInterpreter(unittest.TestCase):
         jsi = JSInterpreter('function f(){return 1 << 5;}')
         self.assertEqual(jsi.call_function('f'), 32)
 
-        jsi = JSInterpreter('function f(){return 2 ** 5}')
-        self.assertEqual(jsi.call_function('f'), 32)
-
         jsi = JSInterpreter('function f(){return 19 & 21;}')
         self.assertEqual(jsi.call_function('f'), 17)
 
         jsi = JSInterpreter('function f(){return 11 >> 2;}')
         self.assertEqual(jsi.call_function('f'), 2)
 
-        jsi = JSInterpreter('function f(){return []? 2+3: 4;}')
-        self.assertEqual(jsi.call_function('f'), 5)
-
-        jsi = JSInterpreter('function f(){return 1 == 2}')
-        self.assertEqual(jsi.call_function('f'), False)
-
-        jsi = JSInterpreter('function f(){return 0 && 1 || 2;}')
-        self.assertEqual(jsi.call_function('f'), 2)
-
-        jsi = JSInterpreter('function f(){return 0 ?? 42;}')
-        self.assertEqual(jsi.call_function('f'), 0)
-
-        jsi = JSInterpreter('function f(){return "life, the universe and everything" < 42;}')
-        self.assertFalse(jsi.call_function('f'))
-
     def test_array_access(self):
-        jsi = JSInterpreter('function f(){var x = [1,2,3]; x[0] = 4; x[0] = 5; x[2.0] = 7; return x;}')
+        jsi = JSInterpreter('function f(){var x = [1,2,3]; x[0] = 4; x[0] = 5; x[2] = 7; return x;}')
         self.assertEqual(jsi.call_function('f'), [5, 2, 7])
 
     def test_parens(self):
@@ -88,10 +62,6 @@ class TestJSInterpreter(unittest.TestCase):
         jsi = JSInterpreter('function f(){return (1 + 2) * 3;}')
         self.assertEqual(jsi.call_function('f'), 9)
 
-    def test_quotes(self):
-        jsi = JSInterpreter(r'function f(){return "a\"\\("}')
-        self.assertEqual(jsi.call_function('f'), r'a"\(')
-
     def test_assignments(self):
         jsi = JSInterpreter('function f(){var x = 20; x = 30 + 1; return x;}')
         self.assertEqual(jsi.call_function('f'), 31)
@@ -134,277 +104,13 @@ class TestJSInterpreter(unittest.TestCase):
         }''')
         self.assertEqual(jsi.call_function('x'), [20, 20, 30, 40, 50])
 
-    def test_builtins(self):
-        jsi = JSInterpreter('''
-        function x() { return NaN }
-        ''')
-        self.assertTrue(math.isnan(jsi.call_function('x')))
-
-        jsi = JSInterpreter('''
-        function x() { return new Date('Wednesday 31 December 1969 18:01:26 MDT') - 0; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 86000)
-        jsi = JSInterpreter('''
-        function x(dt) { return new Date(dt) - 0; }
-        ''')
-        self.assertEqual(jsi.call_function('x', 'Wednesday 31 December 1969 18:01:26 MDT'), 86000)
-
     def test_call(self):
         jsi = JSInterpreter('''
         function x() { return 2; }
-        function y(a) { return x() + (a?a:0); }
+        function y(a) { return x() + a; }
         function z() { return y(3); }
         ''')
         self.assertEqual(jsi.call_function('z'), 5)
-        self.assertEqual(jsi.call_function('y'), 2)
-
-    def test_for_loop(self):
-        # function x() { a=0; for (i=0; i-10; i++) {a++} a }
-        jsi = JSInterpreter('''
-        function x() { a=0; for (i=0; i-10; i++) {a++} return a }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 10)
-
-    def test_switch(self):
-        jsi = JSInterpreter('''
-        function x(f) { switch(f){
-            case 1:f+=1;
-            case 2:f+=2;
-            case 3:f+=3;break;
-            case 4:f+=4;
-            default:f=0;
-        } return f }
-        ''')
-        self.assertEqual(jsi.call_function('x', 1), 7)
-        self.assertEqual(jsi.call_function('x', 3), 6)
-        self.assertEqual(jsi.call_function('x', 5), 0)
-
-    def test_switch_default(self):
-        jsi = JSInterpreter('''
-        function x(f) { switch(f){
-            case 2: f+=2;
-            default: f-=1;
-            case 5:
-            case 6: f+=6;
-            case 0: break;
-            case 1: f+=1;
-        } return f }
-        ''')
-        self.assertEqual(jsi.call_function('x', 1), 2)
-        self.assertEqual(jsi.call_function('x', 5), 11)
-        self.assertEqual(jsi.call_function('x', 9), 14)
-
-    def test_try(self):
-        jsi = JSInterpreter('''
-        function x() { try{return 10} catch(e){return 5} }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 10)
-
-    def test_catch(self):
-        jsi = JSInterpreter('''
-        function x() { try{throw 10} catch(e){return 5} }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 5)
-
-    def test_finally(self):
-        jsi = JSInterpreter('''
-        function x() { try{throw 10} finally {return 42} }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 42)
-        jsi = JSInterpreter('''
-        function x() { try{throw 10} catch(e){return 5} finally {return 42} }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 42)
-
-    def test_nested_try(self):
-        jsi = JSInterpreter('''
-        function x() {try {
-            try{throw 10} finally {throw 42}
-            } catch(e){return 5} }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 5)
-
-    def test_for_loop_continue(self):
-        jsi = JSInterpreter('''
-        function x() { a=0; for (i=0; i-10; i++) { continue; a++ } return a }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 0)
-
-    def test_for_loop_break(self):
-        jsi = JSInterpreter('''
-        function x() { a=0; for (i=0; i-10; i++) { break; a++ } return a }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 0)
-
-    def test_for_loop_try(self):
-        jsi = JSInterpreter('''
-        function x() {
-            for (i=0; i-10; i++) { try { if (i == 5) throw i} catch {return 10} finally {break} };
-            return 42 }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 42)
-
-    def test_literal_list(self):
-        jsi = JSInterpreter('''
-        function x() { return [1, 2, "asdf", [5, 6, 7]][3] }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [5, 6, 7])
-
-    def test_comma(self):
-        jsi = JSInterpreter('''
-        function x() { a=5; a -= 1, a+=3; return a }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 7)
-        jsi = JSInterpreter('''
-        function x() { a=5; return (a -= 1, a+=3, a); }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 7)
-
-        jsi = JSInterpreter('''
-        function x() { return (l=[0,1,2,3], function(a, b){return a+b})((l[1], l[2]), l[3]) }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 5)
-
-    def test_void(self):
-        jsi = JSInterpreter('''
-        function x() { return void 42; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), None)
-
-    def test_return_function(self):
-        jsi = JSInterpreter('''
-        function x() { return [1, function(){return 1}][1] }
-        ''')
-        self.assertEqual(jsi.call_function('x')([]), 1)
-
-    def test_null(self):
-        jsi = JSInterpreter('''
-        function x() { return null; }
-        ''')
-        self.assertIs(jsi.call_function('x'), None)
-
-        jsi = JSInterpreter('''
-        function x() { return [null > 0, null < 0, null == 0, null === 0]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [False, False, False, False])
-
-        jsi = JSInterpreter('''
-        function x() { return [null >= 0, null <= 0]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [True, True])
-
-    def test_undefined(self):
-        jsi = JSInterpreter('''
-        function x() { return undefined === undefined; }
-        ''')
-        self.assertTrue(jsi.call_function('x'))
-
-        jsi = JSInterpreter('''
-        function x() { return undefined; }
-        ''')
-        self.assertIs(jsi.call_function('x'), JS_Undefined)
-
-        jsi = JSInterpreter('''
-        function x() { let v; return v; }
-        ''')
-        self.assertIs(jsi.call_function('x'), JS_Undefined)
-
-        jsi = JSInterpreter('''
-        function x() { return [undefined === undefined, undefined == undefined, undefined < undefined, undefined > undefined]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [True, True, False, False])
-
-        jsi = JSInterpreter('''
-        function x() { return [undefined === 0, undefined == 0, undefined < 0, undefined > 0]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [False, False, False, False])
-
-        jsi = JSInterpreter('''
-        function x() { return [undefined >= 0, undefined <= 0]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [False, False])
-
-        jsi = JSInterpreter('''
-        function x() { return [undefined > null, undefined < null, undefined == null, undefined === null]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [False, False, True, False])
-
-        jsi = JSInterpreter('''
-        function x() { return [undefined === null, undefined == null, undefined < null, undefined > null]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [False, True, False, False])
-
-        jsi = JSInterpreter('''
-        function x() { let v; return [42+v, v+42, v**42, 42**v, 0**v]; }
-        ''')
-        for y in jsi.call_function('x'):
-            self.assertTrue(math.isnan(y))
-
-        jsi = JSInterpreter('''
-        function x() { let v; return v**0; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), 1)
-
-        jsi = JSInterpreter('''
-        function x() { let v; return [v>42, v<=42, v&&42, 42&&v]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [False, False, JS_Undefined, JS_Undefined])
-
-        jsi = JSInterpreter('function x(){return undefined ?? 42; }')
-        self.assertEqual(jsi.call_function('x'), 42)
-
-    def test_object(self):
-        jsi = JSInterpreter('''
-        function x() { return {}; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), {})
-
-        jsi = JSInterpreter('''
-        function x() { let a = {m1: 42, m2: 0 }; return [a["m1"], a.m2]; }
-        ''')
-        self.assertEqual(jsi.call_function('x'), [42, 0])
-
-        jsi = JSInterpreter('''
-        function x() { let a; return a?.qq; }
-        ''')
-        self.assertIs(jsi.call_function('x'), JS_Undefined)
-
-        jsi = JSInterpreter('''
-        function x() { let a = {m1: 42, m2: 0 }; return a?.qq; }
-        ''')
-        self.assertIs(jsi.call_function('x'), JS_Undefined)
-
-    def test_regex(self):
-        jsi = JSInterpreter('''
-        function x() { let a=/,,[/,913,/](,)}/; }
-        ''')
-        self.assertIs(jsi.call_function('x'), None)
-
-        jsi = JSInterpreter('''
-        function x() { let a=/,,[/,913,/](,)}/; return a; }
-        ''')
-        self.assertIsInstance(jsi.call_function('x'), compat_re_Pattern)
-
-        jsi = JSInterpreter('''
-        function x() { let a=/,,[/,913,/](,)}/i; return a; }
-        ''')
-        self.assertEqual(jsi.call_function('x').flags & ~re.U, re.I)
-
-    def test_char_code_at(self):
-        jsi = JSInterpreter('function x(i){return "test".charCodeAt(i)}')
-        self.assertEqual(jsi.call_function('x', 0), 116)
-        self.assertEqual(jsi.call_function('x', 1), 101)
-        self.assertEqual(jsi.call_function('x', 2), 115)
-        self.assertEqual(jsi.call_function('x', 3), 116)
-        self.assertEqual(jsi.call_function('x', 4), None)
-        self.assertEqual(jsi.call_function('x', 'not_a_number'), 116)
-
-    def test_bitwise_operators_overflow(self):
-        jsi = JSInterpreter('function x(){return -524999584 << 5}')
-        self.assertEqual(jsi.call_function('x'), 379882496)
-
-        jsi = JSInterpreter('function x(){return 1236566549 << 5}')
-        self.assertEqual(jsi.call_function('x'), 915423904)
 
 
 if __name__ == '__main__':

test/test_postprocessors.py

@@ -14,4 +14,4 @@ from youtube_dl.postprocessor import MetadataFromTitlePP
 class TestMetadataFromTitle(unittest.TestCase):
     def test_format_to_regex(self):
         pp = MetadataFromTitlePP(None, '%(title)s - %(artist)s')
-        self.assertEqual(pp._titleregex, r'(?P<title>.+)\ \-\ (?P<artist>.+)')
+        self.assertEqual(pp._titleregex, '(?P<title>.+)\ \-\ (?P<artist>.+)')

test/test_subtitles.py

@@ -26,6 +26,7 @@ from youtube_dl.extractor import (
     ThePlatformIE,
     ThePlatformFeedIE,
     RTVEALaCartaIE,
+    FunnyOrDieIE,
     DemocracynowIE,
 )
 
@@ -38,9 +39,6 @@ class BaseTestSubtitles(unittest.TestCase):
         self.DL = FakeYDL()
         self.ie = self.IE()
         self.DL.add_info_extractor(self.ie)
-        if not self.IE.working():
-            print('Skipping: %s marked as not _WORKING' % self.IE.ie_key())
-            self.skipTest('IE marked as not _WORKING')
 
     def getInfoDict(self):
         info_dict = self.DL.extract_info(self.url, download=False)
@@ -59,21 +57,6 @@ class BaseTestSubtitles(unittest.TestCase):
 
 
 class TestYoutubeSubtitles(BaseTestSubtitles):
-    # Available subtitles for QRS8MkLhQmM:
-    # Language formats
-    # ru       vtt, ttml, srv3, srv2, srv1, json3
-    # fr       vtt, ttml, srv3, srv2, srv1, json3
-    # en       vtt, ttml, srv3, srv2, srv1, json3
-    # nl       vtt, ttml, srv3, srv2, srv1, json3
-    # de       vtt, ttml, srv3, srv2, srv1, json3
-    # ko       vtt, ttml, srv3, srv2, srv1, json3
-    # it       vtt, ttml, srv3, srv2, srv1, json3
-    # zh-Hant  vtt, ttml, srv3, srv2, srv1, json3
-    # hi       vtt, ttml, srv3, srv2, srv1, json3
-    # pt-BR    vtt, ttml, srv3, srv2, srv1, json3
-    # es-MX    vtt, ttml, srv3, srv2, srv1, json3
-    # ja       vtt, ttml, srv3, srv2, srv1, json3
-    # pl       vtt, ttml, srv3, srv2, srv1, json3
     url = 'QRS8MkLhQmM'
     IE = YoutubeIE
 
@@ -82,60 +65,41 @@ class TestYoutubeSubtitles(BaseTestSubtitles):
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertEqual(len(subtitles.keys()), 13)
-        self.assertEqual(md5(subtitles['en']), 'ae1bd34126571a77aabd4d276b28044d')
-        self.assertEqual(md5(subtitles['it']), '0e0b667ba68411d88fd1c5f4f4eab2f9')
+        self.assertEqual(md5(subtitles['en']), '3cb210999d3e021bd6c7f0ea751eab06')
+        self.assertEqual(md5(subtitles['it']), '6d752b98c31f1cf8d597050c7a2cb4b5')
         for lang in ['fr', 'de']:
             self.assertTrue(subtitles.get(lang) is not None, 'Subtitles for \'%s\' not extracted' % lang)
 
-    def _test_subtitles_format(self, fmt, md5_hash, lang='en'):
-        self.DL.params['writesubtitles'] = True
-        self.DL.params['subtitlesformat'] = fmt
-        subtitles = self.getSubtitles()
-        self.assertEqual(md5(subtitles[lang]), md5_hash)
-
     def test_youtube_subtitles_ttml_format(self):
-        self._test_subtitles_format('ttml', 'c97ddf1217390906fa9fbd34901f3da2')
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['subtitlesformat'] = 'ttml'
+        subtitles = self.getSubtitles()
+        self.assertEqual(md5(subtitles['en']), 'e306f8c42842f723447d9f63ad65df54')
 
     def test_youtube_subtitles_vtt_format(self):
-        self._test_subtitles_format('vtt', 'ae1bd34126571a77aabd4d276b28044d')
-
-    def test_youtube_subtitles_json3_format(self):
-        self._test_subtitles_format('json3', '688dd1ce0981683867e7fe6fde2a224b')
-
-    def _test_automatic_captions(self, url, lang):
-        self.url = url
-        self.DL.params['writeautomaticsub'] = True
-        self.DL.params['subtitleslangs'] = [lang]
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['subtitlesformat'] = 'vtt'
         subtitles = self.getSubtitles()
-        self.assertTrue(subtitles[lang] is not None)
+        self.assertEqual(md5(subtitles['en']), '3cb210999d3e021bd6c7f0ea751eab06')
 
     def test_youtube_automatic_captions(self):
-        # Available automatic captions for 8YoUxe5ncPo:
-        # Language formats (all in vtt, ttml, srv3, srv2, srv1, json3)
-        # gu, zh-Hans, zh-Hant, gd, ga, gl, lb, la, lo, tt, tr,
-        # lv, lt, tk, th, tg, te, fil, haw, yi, ceb, yo, de, da,
-        # el, eo, en, eu, et, es, ru, rw, ro, bn, be, bg, uk, jv,
-        # bs, ja, or, xh, co, ca, cy, cs, ps, pt, pa, vi, pl, hy,
-        # hr, ht, hu, hmn, hi, ha, mg, uz, ml, mn, mi, mk, ur,
-        # mt, ms, mr, ug, ta, my, af, sw, is, am,
-        #                                         *it*, iw, sv, ar,
-        # su, zu, az, id, ig, nl, no, ne, ny, fr, ku, fy, fa, fi,
-        # ka, kk, sr, sq, ko, kn, km, st, sk, si, so, sn, sm, sl,
-        # ky, sd
-        # ...
-        self._test_automatic_captions('8YoUxe5ncPo', 'it')
+        self.url = '8YoUxe5ncPo'
+        self.DL.params['writeautomaticsub'] = True
+        self.DL.params['subtitleslangs'] = ['it']
+        subtitles = self.getSubtitles()
+        self.assertTrue(subtitles['it'] is not None)
 
-    @unittest.skip('ASR subs all in all supported langs now')
     def test_youtube_translated_subtitles(self):
-        # This video has a subtitles track, which can be translated (#4555)
-        self._test_automatic_captions('Ky9eprVWzlI', 'it')
+        # This video has a subtitles track, which can be translated
+        self.url = 'Ky9eprVWzlI'
+        self.DL.params['writeautomaticsub'] = True
+        self.DL.params['subtitleslangs'] = ['it']
+        subtitles = self.getSubtitles()
+        self.assertTrue(subtitles['it'] is not None)
 
     def test_youtube_nosubtitles(self):
         self.DL.expect_warning('video doesn\'t have subtitles')
-        # Available automatic captions for 8YoUxe5ncPo:
-        # ...
-        # 8YoUxe5ncPo has no subtitles
-        self.url = '8YoUxe5ncPo'
+        self.url = 'n5BB19UTcdA'
         self.DL.params['writesubtitles'] = True
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
@@ -165,7 +129,6 @@ class TestDailymotionSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
-@unittest.skip('IE broken')
 class TestTedSubtitles(BaseTestSubtitles):
     url = 'http://www.ted.com/talks/dan_dennett_on_our_consciousness.html'
     IE = TEDIE
@@ -190,19 +153,18 @@ class TestVimeoSubtitles(BaseTestSubtitles):
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertEqual(set(subtitles.keys()), set(['de', 'en', 'es', 'fr']))
-        self.assertEqual(md5(subtitles['en']), '386cbc9320b94e25cb364b97935e5dd1')
-        self.assertEqual(md5(subtitles['fr']), 'c9b69eef35bc6641c0d4da8a04f9dfac')
+        self.assertEqual(md5(subtitles['en']), '8062383cf4dec168fc40a088aa6d5888')
+        self.assertEqual(md5(subtitles['fr']), 'b6191146a6c5d3a452244d853fde6dc8')
 
     def test_nosubtitles(self):
         self.DL.expect_warning('video doesn\'t have subtitles')
-        self.url = 'http://vimeo.com/68093876'
+        self.url = 'http://vimeo.com/56015672'
         self.DL.params['writesubtitles'] = True
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertFalse(subtitles)
 
 
-@unittest.skip('IE broken')
 class TestWallaSubtitles(BaseTestSubtitles):
     url = 'http://vod.walla.co.il/movie/2705958/the-yes-men'
     IE = WallaIE
@@ -224,7 +186,6 @@ class TestWallaSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
-@unittest.skip('IE broken')
 class TestCeskaTelevizeSubtitles(BaseTestSubtitles):
     url = 'http://www.ceskatelevize.cz/ivysilani/10600540290-u6-uzasny-svet-techniky'
     IE = CeskaTelevizeIE
@@ -246,7 +207,6 @@ class TestCeskaTelevizeSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
-@unittest.skip('IE broken')
 class TestLyndaSubtitles(BaseTestSubtitles):
     url = 'http://www.lynda.com/Bootstrap-tutorials/Using-exercise-files/110885/114408-4.html'
     IE = LyndaIE
@@ -259,7 +219,6 @@ class TestLyndaSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '09bbe67222259bed60deaa26997d73a7')
 
 
-@unittest.skip('IE broken')
 class TestNPOSubtitles(BaseTestSubtitles):
     url = 'http://www.npo.nl/nos-journaal/28-08-2014/POW_00722860'
     IE = NPOIE
@@ -272,9 +231,8 @@ class TestNPOSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['nl']), 'fc6435027572b63fb4ab143abd5ad3f4')
 
 
-@unittest.skip('IE broken')
 class TestMTVSubtitles(BaseTestSubtitles):
-    url = 'http://www.cc.com/video-clips/p63lk0/adam-devine-s-house-party-chasing-white-swans'
+    url = 'http://www.cc.com/video-clips/kllhuv/stand-up-greg-fitzsimmons--uncensored---too-good-of-a-mother'
     IE = ComedyCentralIE
 
     def getInfoDict(self):
@@ -285,7 +243,7 @@ class TestMTVSubtitles(BaseTestSubtitles):
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertEqual(set(subtitles.keys()), set(['en']))
-        self.assertEqual(md5(subtitles['en']), '78206b8d8a0cfa9da64dc026eea48961')
+        self.assertEqual(md5(subtitles['en']), 'b9f6ca22a6acf597ec76f61749765e65')
 
 
 class TestNRKSubtitles(BaseTestSubtitles):
@@ -296,31 +254,22 @@ class TestNRKSubtitles(BaseTestSubtitles):
         self.DL.params['writesubtitles'] = True
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
-        self.assertEqual(set(subtitles.keys()), set(['nb-ttv']))
-        self.assertEqual(md5(subtitles['nb-ttv']), '67e06ff02d0deaf975e68f6cb8f6a149')
+        self.assertEqual(set(subtitles.keys()), set(['no']))
+        self.assertEqual(md5(subtitles['no']), '544fa917d3197fcbee64634559221cc2')
 
 
 class TestRaiPlaySubtitles(BaseTestSubtitles):
+    url = 'http://www.raiplay.it/video/2014/04/Report-del-07042014-cb27157f-9dd0-4aee-b788-b1f67643a391.html'
     IE = RaiPlayIE
 
-    def test_subtitles_key(self):
-        self.url = 'http://www.raiplay.it/video/2014/04/Report-del-07042014-cb27157f-9dd0-4aee-b788-b1f67643a391.html'
+    def test_allsubtitles(self):
         self.DL.params['writesubtitles'] = True
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertEqual(set(subtitles.keys()), set(['it']))
         self.assertEqual(md5(subtitles['it']), 'b1d90a98755126b61e667567a1f6680a')
 
-    def test_subtitles_array_key(self):
-        self.url = 'https://www.raiplay.it/video/2020/12/Report---04-01-2021-2e90f1de-8eee-4de4-ac0e-78d21db5b600.html'
-        self.DL.params['writesubtitles'] = True
-        self.DL.params['allsubtitles'] = True
-        subtitles = self.getSubtitles()
-        self.assertEqual(set(subtitles.keys()), set(['it']))
-        self.assertEqual(md5(subtitles['it']), '4b3264186fbb103508abe5311cfcb9cd')
 
-
-@unittest.skip('IE broken - DRM only')
 class TestVikiSubtitles(BaseTestSubtitles):
     url = 'http://www.viki.com/videos/1060846v-punch-episode-18'
     IE = VikiIE
@@ -347,7 +296,6 @@ class TestThePlatformSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '97e7670cbae3c4d26ae8bcc7fdd78d4b')
 
 
-@unittest.skip('IE broken')
 class TestThePlatformFeedSubtitles(BaseTestSubtitles):
     url = 'http://feed.theplatform.com/f/7wvmTC/msnbc_video-p-test?form=json&pretty=true&range=-40&byGuid=n_hardball_5biden_140207'
     IE = ThePlatformFeedIE
@@ -374,6 +322,18 @@ class TestRtveSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['es']), '69e70cae2d40574fb7316f31d6eb7fca')
 
 
+class TestFunnyOrDieSubtitles(BaseTestSubtitles):
+    url = 'http://www.funnyordie.com/videos/224829ff6d/judd-apatow-will-direct-your-vine'
+    IE = FunnyOrDieIE
+
+    def test_allsubtitles(self):
+        self.DL.params['writesubtitles'] = True
+        self.DL.params['allsubtitles'] = True
+        subtitles = self.getSubtitles()
+        self.assertEqual(set(subtitles.keys()), set(['en']))
+        self.assertEqual(md5(subtitles['en']), 'c5593c193eacd353596c11c2d4f9ecc4')
+
+
 class TestDemocracynowSubtitles(BaseTestSubtitles):
     url = 'http://www.democracynow.org/shows/2015/7/3'
     IE = DemocracynowIE
@@ -383,7 +343,7 @@ class TestDemocracynowSubtitles(BaseTestSubtitles):
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertEqual(set(subtitles.keys()), set(['en']))
-        self.assertEqual(md5(subtitles['en']), 'a3cc4c0b5eadd74d9974f1c1f5101045')
+        self.assertEqual(md5(subtitles['en']), 'acaca989e24a9e45a6719c9b3d60815c')
 
     def test_subtitles_in_page(self):
         self.url = 'http://www.democracynow.org/2015/7/3/this_flag_comes_down_today_bree'
@@ -391,7 +351,7 @@ class TestDemocracynowSubtitles(BaseTestSubtitles):
         self.DL.params['allsubtitles'] = True
         subtitles = self.getSubtitles()
         self.assertEqual(set(subtitles.keys()), set(['en']))
-        self.assertEqual(md5(subtitles['en']), 'a3cc4c0b5eadd74d9974f1c1f5101045')
+        self.assertEqual(md5(subtitles['en']), 'acaca989e24a9e45a6719c9b3d60815c')
 
 
 if __name__ == '__main__':

test/test_swfinterp.py

@@ -34,8 +34,8 @@ def _make_testfunc(testfile):
     def test_func(self):
         as_file = os.path.join(TEST_DIR, testfile)
         swf_file = os.path.join(TEST_DIR, test_id + '.swf')
-        if ((not os.path.exists(swf_file))
-                or os.path.getmtime(swf_file) < os.path.getmtime(as_file)):
+        if ((not os.path.exists(swf_file)) or
+                os.path.getmtime(swf_file) < os.path.getmtime(as_file)):
             # Recompile
             try:
                 subprocess.check_call([

test/test_utils.py

@@ -12,18 +12,14 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 # Various small unit tests
 import io
-import itertools
 import json
-import re
 import xml.etree.ElementTree
 
 from youtube_dl.utils import (
     age_restricted,
     args_to_str,
     encode_base_n,
-    caesar,
     clean_html,
-    clean_podcast_url,
     date_from_str,
     DateRange,
     detect_exe_version,
@@ -37,21 +33,15 @@ from youtube_dl.utils import (
     ExtractorError,
     find_xpath_attr,
     fix_xml_ampersands,
-    float_or_none,
     get_element_by_class,
     get_element_by_attribute,
     get_elements_by_class,
     get_elements_by_attribute,
-    get_first,
     InAdvancePagedList,
-    int_or_none,
     intlist_to_bytes,
     is_html,
-    join_nonempty,
     js_to_json,
-    LazyList,
     limit_length,
-    merge_dicts,
     mimetype2ext,
     month_by_name,
     multipart_encode,
@@ -63,29 +53,21 @@ from youtube_dl.utils import (
     parse_filesize,
     parse_count,
     parse_iso8601,
-    parse_resolution,
-    parse_bitrate,
     pkcs1pad,
     read_batch_urls,
     sanitize_filename,
     sanitize_path,
-    sanitize_url,
     expand_path,
     prepend_extension,
     replace_extension,
     remove_start,
     remove_end,
     remove_quotes,
-    rot47,
     shell_quote,
     smuggle_url,
     str_to_int,
     strip_jsonp,
-    strip_or_none,
-    subtitles_filename,
     timeconvert,
-    traverse_obj,
-    try_call,
     unescapeHTML,
     unified_strdate,
     unified_timestamp,
@@ -93,13 +75,11 @@ from youtube_dl.utils import (
     uppercase_escape,
     lowercase_escape,
     url_basename,
-    url_or_none,
     base_url,
     urljoin,
     urlencode_postdata,
     urshift,
     update_url_query,
-    variadic,
     version_tuple,
     xpath_with_ns,
     xpath_element,
@@ -120,18 +100,12 @@ from youtube_dl.compat import (
     compat_getenv,
     compat_os_name,
     compat_setenv,
-    compat_str,
     compat_urlparse,
     compat_parse_qs,
 )
 
 
 class TestUtil(unittest.TestCase):
-
-    # yt-dlp shim
-    def assertCountEqual(self, expected, got, msg='count should be the same'):
-        return self.assertEqual(len(tuple(expected)), len(tuple(got)), msg=msg)
-
     def test_timeconvert(self):
         self.assertTrue(timeconvert('') is None)
         self.assertTrue(timeconvert('bougrg') is None)
@@ -202,7 +176,7 @@ class TestUtil(unittest.TestCase):
 
         self.assertEqual(sanitize_filename(
             'ÂÃÄÀÁÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖŐØŒÙÚÛÜŰÝÞßàáâãäåæçèéêëìíîïðñòóôõöőøœùúûüűýþÿ', restricted=True),
-            'AAAAAAAECEEEEIIIIDNOOOOOOOOEUUUUUYTHssaaaaaaaeceeeeiiiionooooooooeuuuuuythy')
+            'AAAAAAAECEEEEIIIIDNOOOOOOOOEUUUUUYPssaaaaaaaeceeeeiiiionooooooooeuuuuuypy')
 
     def test_sanitize_ids(self):
         self.assertEqual(sanitize_filename('_n_cd26wFpw', is_id=True), '_n_cd26wFpw')
@@ -245,12 +219,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(sanitize_path('./abc'), 'abc')
         self.assertEqual(sanitize_path('./../abc'), '..\\abc')
 
-    def test_sanitize_url(self):
-        self.assertEqual(sanitize_url('//foo.bar'), 'http://foo.bar')
-        self.assertEqual(sanitize_url('httpss://foo.bar'), 'https://foo.bar')
-        self.assertEqual(sanitize_url('rmtps://foo.bar'), 'rtmps://foo.bar')
-        self.assertEqual(sanitize_url('https://foo.bar'), 'https://foo.bar')
-
     def test_expand_path(self):
         def env(var):
             return '%{0}%'.format(var) if sys.platform == 'win32' else '${0}'.format(var)
@@ -279,11 +247,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(replace_extension('.abc', 'temp'), '.abc.temp')
         self.assertEqual(replace_extension('.abc.ext', 'temp'), '.abc.temp')
 
-    def test_subtitles_filename(self):
-        self.assertEqual(subtitles_filename('abc.ext', 'en', 'vtt'), 'abc.en.vtt')
-        self.assertEqual(subtitles_filename('abc.ext', 'en', 'vtt', 'ext'), 'abc.en.vtt')
-        self.assertEqual(subtitles_filename('abc.unexpected_ext', 'en', 'vtt', 'ext'), 'abc.unexpected_ext.en.vtt')
-
     def test_remove_start(self):
         self.assertEqual(remove_start(None, 'A - '), None)
         self.assertEqual(remove_start('A - B', 'A - '), 'B')
@@ -357,8 +320,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(unified_strdate('July 15th, 2013'), '20130715')
         self.assertEqual(unified_strdate('September 1st, 2013'), '20130901')
         self.assertEqual(unified_strdate('Sep 2nd, 2013'), '20130902')
-        self.assertEqual(unified_strdate('November 3rd, 2019'), '20191103')
-        self.assertEqual(unified_strdate('October 23rd, 2005'), '20051023')
 
     def test_unified_timestamps(self):
         self.assertEqual(unified_timestamp('December 21, 2010'), 1292889600)
@@ -383,10 +344,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(unified_timestamp('2017-03-30T17:52:41Q'), 1490896361)
         self.assertEqual(unified_timestamp('Sep 11, 2013 | 5:49 AM'), 1378878540)
         self.assertEqual(unified_timestamp('December 15, 2017 at 7:49 am'), 1513324140)
-        self.assertEqual(unified_timestamp('2018-03-14T08:32:43.1493874+00:00'), 1521016363)
-        self.assertEqual(unified_timestamp('December 31 1969 20:00:01 EDT'), 1)
-        self.assertEqual(unified_timestamp('Wednesday 31 December 1969 18:01:26 MDT'), 86)
-        self.assertEqual(unified_timestamp('12/31/1969 20:01:18 EDT', False), 78)
 
     def test_determine_ext(self):
         self.assertEqual(determine_ext('http://example.com/foo/bar.mp4/?download'), 'mp4')
@@ -394,7 +351,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(determine_ext('http://example.com/foo/bar.nonext/?download', None), None)
         self.assertEqual(determine_ext('http://example.com/foo/bar/mp4?download', None), None)
         self.assertEqual(determine_ext('http://example.com/foo/bar.m3u8//?download'), 'm3u8')
-        self.assertEqual(determine_ext('foobar', None), None)
 
     def test_find_xpath_attr(self):
         testxml = '''<root>
@@ -499,30 +455,9 @@ class TestUtil(unittest.TestCase):
             shell_quote(args),
             """ffmpeg -i 'ñ€ß'"'"'.mp4'""" if compat_os_name != 'nt' else '''ffmpeg -i "ñ€ß'.mp4"''')
 
-    def test_float_or_none(self):
-        self.assertEqual(float_or_none('42.42'), 42.42)
-        self.assertEqual(float_or_none('42'), 42.0)
-        self.assertEqual(float_or_none(''), None)
-        self.assertEqual(float_or_none(None), None)
-        self.assertEqual(float_or_none([]), None)
-        self.assertEqual(float_or_none(set()), None)
-
-    def test_int_or_none(self):
-        self.assertEqual(int_or_none('42'), 42)
-        self.assertEqual(int_or_none(''), None)
-        self.assertEqual(int_or_none(None), None)
-        self.assertEqual(int_or_none([]), None)
-        self.assertEqual(int_or_none(set()), None)
-
     def test_str_to_int(self):
         self.assertEqual(str_to_int('123,456'), 123456)
         self.assertEqual(str_to_int('123.456'), 123456)
-        self.assertEqual(str_to_int(523), 523)
-        # Python 3 has no long
-        if sys.version_info < (3, 0):
-            eval('self.assertEqual(str_to_int(123456L), 123456)')
-        self.assertEqual(str_to_int('noninteger'), None)
-        self.assertEqual(str_to_int([]), None)
 
     def test_url_basename(self):
         self.assertEqual(url_basename('http://foo.de/'), '')
@@ -560,23 +495,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(urljoin('http://foo.de/', ''), None)
         self.assertEqual(urljoin('http://foo.de/', ['foobar']), None)
         self.assertEqual(urljoin('http://foo.de/a/b/c.txt', '.././../d.txt'), 'http://foo.de/d.txt')
-        self.assertEqual(urljoin('http://foo.de/a/b/c.txt', 'rtmp://foo.de'), 'rtmp://foo.de')
-        self.assertEqual(urljoin(None, 'rtmp://foo.de'), 'rtmp://foo.de')
-
-    def test_url_or_none(self):
-        self.assertEqual(url_or_none(None), None)
-        self.assertEqual(url_or_none(''), None)
-        self.assertEqual(url_or_none('foo'), None)
-        self.assertEqual(url_or_none('http://foo.de'), 'http://foo.de')
-        self.assertEqual(url_or_none('https://foo.de'), 'https://foo.de')
-        self.assertEqual(url_or_none('http$://foo.de'), None)
-        self.assertEqual(url_or_none('http://foo.de'), 'http://foo.de')
-        self.assertEqual(url_or_none('//foo.de'), '//foo.de')
-        self.assertEqual(url_or_none('s3://foo.de'), None)
-        self.assertEqual(url_or_none('rtmpte://foo.de'), 'rtmpte://foo.de')
-        self.assertEqual(url_or_none('mms://foo.de'), 'mms://foo.de')
-        self.assertEqual(url_or_none('rtspu://foo.de'), 'rtspu://foo.de')
-        self.assertEqual(url_or_none('ftps://foo.de'), 'ftps://foo.de')
 
     def test_parse_age_limit(self):
         self.assertEqual(parse_age_limit(None), None)
@@ -591,8 +509,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(parse_age_limit('PG-13'), 13)
         self.assertEqual(parse_age_limit('TV-14'), 14)
         self.assertEqual(parse_age_limit('TV-MA'), 17)
-        self.assertEqual(parse_age_limit('TV14'), 14)
-        self.assertEqual(parse_age_limit('TV_G'), 0)
 
     def test_parse_duration(self):
         self.assertEqual(parse_duration(None), None)
@@ -744,17 +660,6 @@ class TestUtil(unittest.TestCase):
             self.assertEqual(dict_get(d, ('b', 'c', key, )), None)
             self.assertEqual(dict_get(d, ('b', 'c', key, ), skip_false_values=False), false_value)
 
-    def test_merge_dicts(self):
-        self.assertEqual(merge_dicts({'a': 1}, {'b': 2}), {'a': 1, 'b': 2})
-        self.assertEqual(merge_dicts({'a': 1}, {'a': 2}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': 1}, {'a': None}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': 1}, {'a': ''}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': 1}, {}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': None}, {'a': 1}), {'a': 1})
-        self.assertEqual(merge_dicts({'a': ''}, {'a': 1}), {'a': ''})
-        self.assertEqual(merge_dicts({'a': ''}, {'a': 'abc'}), {'a': 'abc'})
-        self.assertEqual(merge_dicts({'a': None}, {'a': ''}, {'a': 'abc'}), {'a': 'abc'})
-
     def test_encode_compat_str(self):
         self.assertEqual(encode_compat_str(b'\xd1\x82\xd0\xb5\xd1\x81\xd1\x82', 'utf-8'), 'тест')
         self.assertEqual(encode_compat_str('тест', 'utf-8'), 'тест')
@@ -788,22 +693,6 @@ class TestUtil(unittest.TestCase):
         d = json.loads(stripped)
         self.assertEqual(d, {'status': 'success'})
 
-        stripped = strip_jsonp('({"status": "success"});')
-        d = json.loads(stripped)
-        self.assertEqual(d, {'status': 'success'})
-
-    def test_strip_or_none(self):
-        self.assertEqual(strip_or_none(' abc'), 'abc')
-        self.assertEqual(strip_or_none('abc '), 'abc')
-        self.assertEqual(strip_or_none(' abc '), 'abc')
-        self.assertEqual(strip_or_none('\tabc\t'), 'abc')
-        self.assertEqual(strip_or_none('\n\tabc\n\t'), 'abc')
-        self.assertEqual(strip_or_none('abc'), 'abc')
-        self.assertEqual(strip_or_none(''), '')
-        self.assertEqual(strip_or_none(None), None)
-        self.assertEqual(strip_or_none(42), None)
-        self.assertEqual(strip_or_none([]), None)
-
     def test_uppercase_escape(self):
         self.assertEqual(uppercase_escape('aä'), 'aä')
         self.assertEqual(uppercase_escape('\\U0001d550'), '𝕐')
@@ -826,8 +715,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(mimetype2ext('text/vtt'), 'vtt')
         self.assertEqual(mimetype2ext('text/vtt;charset=utf-8'), 'vtt')
         self.assertEqual(mimetype2ext('text/html; charset=utf-8'), 'html')
-        self.assertEqual(mimetype2ext('audio/x-wav'), 'wav')
-        self.assertEqual(mimetype2ext('audio/x-wav;codec=pcm'), 'wav')
 
     def test_month_by_name(self):
         self.assertEqual(month_by_name(None), None)
@@ -859,19 +746,6 @@ class TestUtil(unittest.TestCase):
             'vcodec': 'h264',
             'acodec': 'aac',
         })
-        self.assertEqual(parse_codecs('av01.0.05M.08'), {
-            'vcodec': 'av01.0.05M.08',
-            'acodec': 'none',
-        })
-        self.assertEqual(parse_codecs('theora, vorbis'), {
-            'vcodec': 'theora',
-            'acodec': 'vorbis',
-        })
-        self.assertEqual(parse_codecs('unknownvcodec, unknownacodec'), {
-            'vcodec': 'unknownvcodec',
-            'acodec': 'unknownacodec',
-        })
-        self.assertEqual(parse_codecs('unknown'), {})
 
     def test_escape_rfc3986(self):
         reserved = "!*'();:@&=+$,/?#[]"
@@ -940,9 +814,6 @@ class TestUtil(unittest.TestCase):
         inp = '''{"duration": "00:01:07"}'''
         self.assertEqual(js_to_json(inp), '''{"duration": "00:01:07"}''')
 
-        inp = '''{segments: [{"offset":-3.885780586188048e-16,"duration":39.75000000000001}]}'''
-        self.assertEqual(js_to_json(inp), '''{"segments": [{"offset":-3.885780586188048e-16,"duration":39.75000000000001}]}''')
-
     def test_js_to_json_edgecases(self):
         on = js_to_json("{abc_def:'1\\'\\\\2\\\\\\'3\"4'}")
         self.assertEqual(json.loads(on), {"abc_def": "1'\\2\\'3\"4"})
@@ -960,28 +831,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(d['x'], 1)
         self.assertEqual(d['y'], 'a')
 
-        # Just drop ! prefix for now though this results in a wrong value
-        on = js_to_json('''{
-            a: !0,
-            b: !1,
-            c: !!0,
-            d: !!42.42,
-            e: !!![],
-            f: !"abc",
-            g: !"",
-            !42: 42
-        }''')
-        self.assertEqual(json.loads(on), {
-            'a': 0,
-            'b': 1,
-            'c': 0,
-            'd': 42.42,
-            'e': [],
-            'f': "abc",
-            'g': "",
-            '42': 42
-        })
-
         on = js_to_json('["abc", "def",]')
         self.assertEqual(json.loads(on), ['abc', 'def'])
 
@@ -1036,19 +885,6 @@ class TestUtil(unittest.TestCase):
         on = js_to_json('{/*comment\n*/42/*comment\n*/:/*comment\n*/42/*comment\n*/}')
         self.assertEqual(json.loads(on), {'42': 42})
 
-        on = js_to_json('{42:4.2e1}')
-        self.assertEqual(json.loads(on), {'42': 42.0})
-
-        on = js_to_json('{ "0x40": "0x40" }')
-        self.assertEqual(json.loads(on), {'0x40': '0x40'})
-
-        on = js_to_json('{ "040": "040" }')
-        self.assertEqual(json.loads(on), {'040': '040'})
-
-    def test_js_to_json_malformed(self):
-        self.assertEqual(js_to_json('42a1'), '42"a1"')
-        self.assertEqual(js_to_json('42a-1'), '42"a"-1')
-
     def test_extract_attributes(self):
         self.assertEqual(extract_attributes('<e x="y">'), {'x': 'y'})
         self.assertEqual(extract_attributes("<e x='y'>"), {'x': 'y'})
@@ -1129,23 +965,6 @@ class TestUtil(unittest.TestCase):
         self.assertEqual(parse_count('1.1kk '), 1100000)
         self.assertEqual(parse_count('1.1kk views'), 1100000)
 
-    def test_parse_resolution(self):
-        self.assertEqual(parse_resolution(None), {})
-        self.assertEqual(parse_resolution(''), {})
-        self.assertEqual(parse_resolution('1920x1080'), {'width': 1920, 'height': 1080})
-        self.assertEqual(parse_resolution('1920×1080'), {'width': 1920, 'height': 1080})
-        self.assertEqual(parse_resolution('1920 x 1080'), {'width': 1920, 'height': 1080})
-        self.assertEqual(parse_resolution('720p'), {'height': 720})
-        self.assertEqual(parse_resolution('4k'), {'height': 2160})
-        self.assertEqual(parse_resolution('8K'), {'height': 4320})
-
-    def test_parse_bitrate(self):
-        self.assertEqual(parse_bitrate(None), None)
-        self.assertEqual(parse_bitrate(''), None)
-        self.assertEqual(parse_bitrate('300kbps'), 300)
-        self.assertEqual(parse_bitrate('1500kbps'), 1500)
-        self.assertEqual(parse_bitrate('300 kbps'), 300)
-
     def test_version_tuple(self):
         self.assertEqual(version_tuple('1'), (1,))
         self.assertEqual(version_tuple('10.23.344'), (10, 23, 344))
@@ -1224,18 +1043,6 @@ ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
         self.assertFalse(match_str(
             'like_count > 100 & dislike_count <? 50 & description',
             {'like_count': 190, 'dislike_count': 10}))
-        self.assertTrue(match_str('is_live', {'is_live': True}))
-        self.assertFalse(match_str('is_live', {'is_live': False}))
-        self.assertFalse(match_str('is_live', {'is_live': None}))
-        self.assertFalse(match_str('is_live', {}))
-        self.assertFalse(match_str('!is_live', {'is_live': True}))
-        self.assertTrue(match_str('!is_live', {'is_live': False}))
-        self.assertTrue(match_str('!is_live', {'is_live': None}))
-        self.assertTrue(match_str('!is_live', {}))
-        self.assertTrue(match_str('title', {'title': 'abc'}))
-        self.assertTrue(match_str('title', {'title': ''}))
-        self.assertFalse(match_str('!title', {'title': 'abc'}))
-        self.assertFalse(match_str('!title', {'title': ''}))
 
     def test_parse_dfxp_time_expr(self):
         self.assertEqual(parse_dfxp_time_expr(None), None)
@@ -1430,20 +1237,6 @@ Line 1
         self.assertRaises(ValueError, encode_base_n, 0, 70)
         self.assertRaises(ValueError, encode_base_n, 0, 60, custom_table)
 
-    def test_caesar(self):
-        self.assertEqual(caesar('ace', 'abcdef', 2), 'cea')
-        self.assertEqual(caesar('cea', 'abcdef', -2), 'ace')
-        self.assertEqual(caesar('ace', 'abcdef', -2), 'eac')
-        self.assertEqual(caesar('eac', 'abcdef', 2), 'ace')
-        self.assertEqual(caesar('ace', 'abcdef', 0), 'ace')
-        self.assertEqual(caesar('xyz', 'abcdef', 2), 'xyz')
-        self.assertEqual(caesar('abc', 'acegik', 2), 'ebg')
-        self.assertEqual(caesar('ebg', 'acegik', -2), 'abc')
-
-    def test_rot47(self):
-        self.assertEqual(rot47('youtube-dl'), r'J@FEF36\5=')
-        self.assertEqual(rot47('YOUTUBE-DL'), r'*~&%&qt\s{')
-
     def test_urshift(self):
         self.assertEqual(urshift(3, 1), 1)
         self.assertEqual(urshift(-3, 1), 2147483646)
@@ -1488,319 +1281,6 @@ Line 1
         self.assertEqual(get_elements_by_attribute('class', 'foo', html), [])
         self.assertEqual(get_elements_by_attribute('class', 'no-such-foo', html), [])
 
-    def test_clean_podcast_url(self):
-        self.assertEqual(clean_podcast_url('https://www.podtrac.com/pts/redirect.mp3/chtbl.com/track/5899E/traffic.megaphone.fm/HSW7835899191.mp3'), 'https://traffic.megaphone.fm/HSW7835899191.mp3')
-        self.assertEqual(clean_podcast_url('https://play.podtrac.com/npr-344098539/edge1.pod.npr.org/anon.npr-podcasts/podcast/npr/waitwait/2020/10/20201003_waitwait_wwdtmpodcast201003-015621a5-f035-4eca-a9a1-7c118d90bc3c.mp3'), 'https://edge1.pod.npr.org/anon.npr-podcasts/podcast/npr/waitwait/2020/10/20201003_waitwait_wwdtmpodcast201003-015621a5-f035-4eca-a9a1-7c118d90bc3c.mp3')
-
-    def test_LazyList(self):
-        it = list(range(10))
-
-        self.assertEqual(list(LazyList(it)), it)
-        self.assertEqual(LazyList(it).exhaust(), it)
-        self.assertEqual(LazyList(it)[5], it[5])
-
-        self.assertEqual(LazyList(it)[5:], it[5:])
-        self.assertEqual(LazyList(it)[:5], it[:5])
-        self.assertEqual(LazyList(it)[::2], it[::2])
-        self.assertEqual(LazyList(it)[1::2], it[1::2])
-        self.assertEqual(LazyList(it)[5::-1], it[5::-1])
-        self.assertEqual(LazyList(it)[6:2:-2], it[6:2:-2])
-        self.assertEqual(LazyList(it)[::-1], it[::-1])
-
-        self.assertTrue(LazyList(it))
-        self.assertFalse(LazyList(range(0)))
-        self.assertEqual(len(LazyList(it)), len(it))
-        self.assertEqual(repr(LazyList(it)), repr(it))
-        self.assertEqual(compat_str(LazyList(it)), compat_str(it))
-
-        self.assertEqual(list(LazyList(it, reverse=True)), it[::-1])
-        self.assertEqual(list(reversed(LazyList(it))[::-1]), it)
-        self.assertEqual(list(reversed(LazyList(it))[1:3:7]), it[::-1][1:3:7])
-
-    def test_LazyList_laziness(self):
-
-        def test(ll, idx, val, cache):
-            self.assertEqual(ll[idx], val)
-            self.assertEqual(ll._cache, list(cache))
-
-        ll = LazyList(range(10))
-        test(ll, 0, 0, range(1))
-        test(ll, 5, 5, range(6))
-        test(ll, -3, 7, range(10))
-
-        ll = LazyList(range(10), reverse=True)
-        test(ll, -1, 0, range(1))
-        test(ll, 3, 6, range(10))
-
-        ll = LazyList(itertools.count())
-        test(ll, 10, 10, range(11))
-        ll = reversed(ll)
-        test(ll, -15, 14, range(15))
-
-    def test_try_call(self):
-        def total(*x, **kwargs):
-            return sum(x) + sum(kwargs.values())
-
-        self.assertEqual(try_call(None), None,
-                         msg='not a fn should give None')
-        self.assertEqual(try_call(lambda: 1), 1,
-                         msg='int fn with no expected_type should give int')
-        self.assertEqual(try_call(lambda: 1, expected_type=int), 1,
-                         msg='int fn with expected_type int should give int')
-        self.assertEqual(try_call(lambda: 1, expected_type=dict), None,
-                         msg='int fn with wrong expected_type should give None')
-        self.assertEqual(try_call(total, args=(0, 1, 0, ), expected_type=int), 1,
-                         msg='fn should accept arglist')
-        self.assertEqual(try_call(total, kwargs={'a': 0, 'b': 1, 'c': 0}, expected_type=int), 1,
-                         msg='fn should accept kwargs')
-        self.assertEqual(try_call(lambda: 1, expected_type=dict), None,
-                         msg='int fn with no expected_type should give None')
-        self.assertEqual(try_call(lambda x: {}, total, args=(42, ), expected_type=int), 42,
-                         msg='expect first int result with expected_type int')
-
-    def test_variadic(self):
-        self.assertEqual(variadic(None), (None, ))
-        self.assertEqual(variadic('spam'), ('spam', ))
-        self.assertEqual(variadic('spam', allowed_types=dict), 'spam')
-
-    def test_traverse_obj(self):
-        _TEST_DATA = {
-            100: 100,
-            1.2: 1.2,
-            'str': 'str',
-            'None': None,
-            '...': Ellipsis,
-            'urls': [
-                {'index': 0, 'url': 'https://www.example.com/0'},
-                {'index': 1, 'url': 'https://www.example.com/1'},
-            ],
-            'data': (
-                {'index': 2},
-                {'index': 3},
-            ),
-            'dict': {},
-        }
-
-        # Test base functionality
-        self.assertEqual(traverse_obj(_TEST_DATA, ('str',)), 'str',
-                         msg='allow tuple path')
-        self.assertEqual(traverse_obj(_TEST_DATA, ['str']), 'str',
-                         msg='allow list path')
-        self.assertEqual(traverse_obj(_TEST_DATA, (value for value in ("str",))), 'str',
-                         msg='allow iterable path')
-        self.assertEqual(traverse_obj(_TEST_DATA, 'str'), 'str',
-                         msg='single items should be treated as a path')
-        self.assertEqual(traverse_obj(_TEST_DATA, None), _TEST_DATA)
-        self.assertEqual(traverse_obj(_TEST_DATA, 100), 100)
-        self.assertEqual(traverse_obj(_TEST_DATA, 1.2), 1.2)
-
-        # Test Ellipsis behavior
-        self.assertCountEqual(traverse_obj(_TEST_DATA, Ellipsis),
-                              (item for item in _TEST_DATA.values() if item is not None),
-                              msg='`...` should give all values except `None`')
-        self.assertCountEqual(traverse_obj(_TEST_DATA, ('urls', 0, Ellipsis)), _TEST_DATA['urls'][0].values(),
-                              msg='`...` selection for dicts should select all values')
-        self.assertEqual(traverse_obj(_TEST_DATA, (Ellipsis, Ellipsis, 'url')),
-                         ['https://www.example.com/0', 'https://www.example.com/1'],
-                         msg='nested `...` queries should work')
-        self.assertCountEqual(traverse_obj(_TEST_DATA, (Ellipsis, Ellipsis, 'index')), range(4),
-                              msg='`...` query result should be flattened')
-
-        # Test function as key
-        self.assertEqual(traverse_obj(_TEST_DATA, lambda x, y: x == 'urls' and isinstance(y, list)),
-                         [_TEST_DATA['urls']],
-                         msg='function as query key should perform a filter based on (key, value)')
-        self.assertCountEqual(traverse_obj(_TEST_DATA, lambda _, x: isinstance(x[0], compat_str)), {'str'},
-                              msg='exceptions in the query function should be caught')
-
-        # Test alternative paths
-        self.assertEqual(traverse_obj(_TEST_DATA, 'fail', 'str'), 'str',
-                         msg='multiple `paths` should be treated as alternative paths')
-        self.assertEqual(traverse_obj(_TEST_DATA, 'str', 100), 'str',
-                         msg='alternatives should exit early')
-        self.assertEqual(traverse_obj(_TEST_DATA, 'fail', 'fail'), None,
-                         msg='alternatives should return `default` if exhausted')
-        self.assertEqual(traverse_obj(_TEST_DATA, (Ellipsis, 'fail'), 100), 100,
-                         msg='alternatives should track their own branching return')
-        self.assertEqual(traverse_obj(_TEST_DATA, ('dict', Ellipsis), ('data', Ellipsis)), list(_TEST_DATA['data']),
-                         msg='alternatives on empty objects should search further')
-
-        # Test branch and path nesting
-        self.assertEqual(traverse_obj(_TEST_DATA, ('urls', (3, 0), 'url')), ['https://www.example.com/0'],
-                         msg='tuple as key should be treated as branches')
-        self.assertEqual(traverse_obj(_TEST_DATA, ('urls', [3, 0], 'url')), ['https://www.example.com/0'],
-                         msg='list as key should be treated as branches')
-        self.assertEqual(traverse_obj(_TEST_DATA, ('urls', ((1, 'fail'), (0, 'url')))), ['https://www.example.com/0'],
-                         msg='double nesting in path should be treated as paths')
-        self.assertEqual(traverse_obj(['0', [1, 2]], [(0, 1), 0]), [1],
-                         msg='do not fail early on branching')
-        self.assertCountEqual(traverse_obj(_TEST_DATA, ('urls', ((1, ('fail', 'url')), (0, 'url')))),
-                              ['https://www.example.com/0', 'https://www.example.com/1'],
-                              msg='triple nesting in path should be treated as branches')
-        self.assertEqual(traverse_obj(_TEST_DATA, ('urls', ('fail', (Ellipsis, 'url')))),
-                         ['https://www.example.com/0', 'https://www.example.com/1'],
-                         msg='ellipsis as branch path start gets flattened')
-
-        # Test dictionary as key
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: 100, 1: 1.2}), {0: 100, 1: 1.2},
-                         msg='dict key should result in a dict with the same keys')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: ('urls', 0, 'url')}),
-                         {0: 'https://www.example.com/0'},
-                         msg='dict key should allow paths')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: ('urls', (3, 0), 'url')}),
-                         {0: ['https://www.example.com/0']},
-                         msg='tuple in dict path should be treated as branches')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: ('urls', ((1, 'fail'), (0, 'url')))}),
-                         {0: ['https://www.example.com/0']},
-                         msg='double nesting in dict path should be treated as paths')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: ('urls', ((1, ('fail', 'url')), (0, 'url')))}),
-                         {0: ['https://www.example.com/1', 'https://www.example.com/0']},
-                         msg='triple nesting in dict path should be treated as branches')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: 'fail'}), {},
-                         msg='remove `None` values when dict key')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: 'fail'}, default=Ellipsis), {0: Ellipsis},
-                         msg='do not remove `None` values if `default`')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: 'dict'}), {0: {}},
-                         msg='do not remove empty values when dict key')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: 'dict'}, default=Ellipsis), {0: {}},
-                         msg='do not remove empty values when dict key and a default')
-        self.assertEqual(traverse_obj(_TEST_DATA, {0: ('dict', Ellipsis)}), {0: []},
-                         msg='if branch in dict key not successful, return `[]`')
-
-        # Testing default parameter behavior
-        _DEFAULT_DATA = {'None': None, 'int': 0, 'list': []}
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, 'fail'), None,
-                         msg='default value should be `None`')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, 'fail', 'fail', default=Ellipsis), Ellipsis,
-                         msg='chained fails should result in default')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, 'None', 'int'), 0,
-                         msg='should not short cirquit on `None`')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, 'fail', default=1), 1,
-                         msg='invalid dict key should result in `default`')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, 'None', default=1), 1,
-                         msg='`None` is a deliberate sentinel and should become `default`')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, ('list', 10)), None,
-                         msg='`IndexError` should result in `default`')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, (Ellipsis, 'fail'), default=1), 1,
-                         msg='if branched but not successful return `default` if defined, not `[]`')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, (Ellipsis, 'fail'), default=None), None,
-                         msg='if branched but not successful return `default` even if `default` is `None`')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, (Ellipsis, 'fail')), [],
-                         msg='if branched but not successful return `[]`, not `default`')
-        self.assertEqual(traverse_obj(_DEFAULT_DATA, ('list', Ellipsis)), [],
-                         msg='if branched but object is empty return `[]`, not `default`')
-
-        # Testing expected_type behavior
-        _EXPECTED_TYPE_DATA = {'str': 'str', 'int': 0}
-        self.assertEqual(traverse_obj(_EXPECTED_TYPE_DATA, 'str', expected_type=compat_str), 'str',
-                         msg='accept matching `expected_type` type')
-        self.assertEqual(traverse_obj(_EXPECTED_TYPE_DATA, 'str', expected_type=int), None,
-                         msg='reject non matching `expected_type` type')
-        self.assertEqual(traverse_obj(_EXPECTED_TYPE_DATA, 'int', expected_type=lambda x: compat_str(x)), '0',
-                         msg='transform type using type function')
-        self.assertEqual(traverse_obj(_EXPECTED_TYPE_DATA, 'str',
-                                      expected_type=lambda _: 1 / 0), None,
-                         msg='wrap expected_type function in try_call')
-        self.assertEqual(traverse_obj(_EXPECTED_TYPE_DATA, Ellipsis, expected_type=compat_str), ['str'],
-                         msg='eliminate items that expected_type fails on')
-
-        # Test get_all behavior
-        _GET_ALL_DATA = {'key': [0, 1, 2]}
-        self.assertEqual(traverse_obj(_GET_ALL_DATA, ('key', Ellipsis), get_all=False), 0,
-                         msg='if not `get_all`, return only first matching value')
-        self.assertEqual(traverse_obj(_GET_ALL_DATA, Ellipsis, get_all=False), [0, 1, 2],
-                         msg='do not overflatten if not `get_all`')
-
-        # Test casesense behavior
-        _CASESENSE_DATA = {
-            'KeY': 'value0',
-            0: {
-                'KeY': 'value1',
-                0: {'KeY': 'value2'},
-            },
-            # FULLWIDTH LATIN CAPITAL LETTER K
-            '\uff2bey': 'value3',
-        }
-        self.assertEqual(traverse_obj(_CASESENSE_DATA, 'key'), None,
-                         msg='dict keys should be case sensitive unless `casesense`')
-        self.assertEqual(traverse_obj(_CASESENSE_DATA, 'keY',
-                                      casesense=False), 'value0',
-                         msg='allow non matching key case if `casesense`')
-        self.assertEqual(traverse_obj(_CASESENSE_DATA, '\uff4bey',  # FULLWIDTH LATIN SMALL LETTER K
-                                      casesense=False), 'value3',
-                         msg='allow non matching Unicode key case if `casesense`')
-        self.assertEqual(traverse_obj(_CASESENSE_DATA, (0, ('keY',)),
-                                      casesense=False), ['value1'],
-                         msg='allow non matching key case in branch if `casesense`')
-        self.assertEqual(traverse_obj(_CASESENSE_DATA, (0, ((0, 'keY'),)),
-                                      casesense=False), ['value2'],
-                         msg='allow non matching key case in branch path if `casesense`')
-
-        # Test traverse_string behavior
-        _TRAVERSE_STRING_DATA = {'str': 'str', 1.2: 1.2}
-        self.assertEqual(traverse_obj(_TRAVERSE_STRING_DATA, ('str', 0)), None,
-                         msg='do not traverse into string if not `traverse_string`')
-        self.assertEqual(traverse_obj(_TRAVERSE_STRING_DATA, ('str', 0),
-                                      _traverse_string=True), 's',
-                         msg='traverse into string if `traverse_string`')
-        self.assertEqual(traverse_obj(_TRAVERSE_STRING_DATA, (1.2, 1),
-                                      _traverse_string=True), '.',
-                         msg='traverse into converted data if `traverse_string`')
-        self.assertEqual(traverse_obj(_TRAVERSE_STRING_DATA, ('str', Ellipsis),
-                                      _traverse_string=True), list('str'),
-                         msg='`...` branching into string should result in list')
-        self.assertEqual(traverse_obj(_TRAVERSE_STRING_DATA, ('str', (0, 2)),
-                                      _traverse_string=True), ['s', 'r'],
-                         msg='branching into string should result in list')
-        self.assertEqual(traverse_obj(_TRAVERSE_STRING_DATA, ('str', lambda _, x: x),
-                                      _traverse_string=True), list('str'),
-                         msg='function branching into string should result in list')
-
-        # Test is_user_input behavior
-        _IS_USER_INPUT_DATA = {'range8': list(range(8))}
-        self.assertEqual(traverse_obj(_IS_USER_INPUT_DATA, ('range8', '3'),
-                                      _is_user_input=True), 3,
-                         msg='allow for string indexing if `is_user_input`')
-        self.assertCountEqual(traverse_obj(_IS_USER_INPUT_DATA, ('range8', '3:'),
-                                           _is_user_input=True), tuple(range(8))[3:],
-                              msg='allow for string slice if `is_user_input`')
-        self.assertCountEqual(traverse_obj(_IS_USER_INPUT_DATA, ('range8', ':4:2'),
-                                           _is_user_input=True), tuple(range(8))[:4:2],
-                              msg='allow step in string slice if `is_user_input`')
-        self.assertCountEqual(traverse_obj(_IS_USER_INPUT_DATA, ('range8', ':'),
-                                           _is_user_input=True), range(8),
-                              msg='`:` should be treated as `...` if `is_user_input`')
-        with self.assertRaises(TypeError, msg='too many params should result in error'):
-            traverse_obj(_IS_USER_INPUT_DATA, ('range8', ':::'), _is_user_input=True)
-
-        # Test re.Match as input obj
-        mobj = re.match(r'^0(12)(?P<group>3)(4)?$', '0123')
-        self.assertEqual(traverse_obj(mobj, Ellipsis), [x for x in mobj.groups() if x is not None],
-                         msg='`...` on a `re.Match` should give its `groups()`')
-        self.assertEqual(traverse_obj(mobj, lambda k, _: k in (0, 2)), ['0123', '3'],
-                         msg='function on a `re.Match` should give groupno, value starting at 0')
-        self.assertEqual(traverse_obj(mobj, 'group'), '3',
-                         msg='str key on a `re.Match` should give group with that name')
-        self.assertEqual(traverse_obj(mobj, 2), '3',
-                         msg='int key on a `re.Match` should give group with that name')
-        self.assertEqual(traverse_obj(mobj, 'gRoUp', casesense=False), '3',
-                         msg='str key on a `re.Match` should respect casesense')
-        self.assertEqual(traverse_obj(mobj, 'fail'), None,
-                         msg='failing str key on a `re.Match` should return `default`')
-        self.assertEqual(traverse_obj(mobj, 'gRoUpS', casesense=False), None,
-                         msg='failing str key on a `re.Match` should return `default`')
-        self.assertEqual(traverse_obj(mobj, 8), None,
-                         msg='failing int key on a `re.Match` should return `default`')
-
-    def test_get_first(self):
-        self.assertEqual(get_first([{'a': None}, {'a': 'spam'}], 'a'), 'spam')
-
-    def test_join_nonempty(self):
-        self.assertEqual(join_nonempty('a', 'b'), 'a-b')
-        self.assertEqual(join_nonempty(
-            'a', 'b', 'c', 'd',
-            from_dict={'a': 'c', 'c': [], 'b': 'd', 'd': None}), 'c-d')
-
 
 if __name__ == '__main__':
     unittest.main()

test/test_youtube_chapters.py

@@ -0,0 +1,275 @@
+#!/usr/bin/env python
+# coding: utf-8
+from __future__ import unicode_literals
+
+# Allow direct execution
+import os
+import sys
+import unittest
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from test.helper import expect_value
+from youtube_dl.extractor import YoutubeIE
+
+
+class TestYoutubeChapters(unittest.TestCase):
+
+    _TEST_CASES = [
+        (
+            # https://www.youtube.com/watch?v=A22oy8dFjqc
+            # pattern: 00:00 - <title>
+            '''This is the absolute ULTIMATE experience of Queen's set at LIVE AID, this is the best video mixed to the absolutely superior stereo radio broadcast. This vastly superior audio mix takes a huge dump on all of the official mixes. Best viewed in 1080p. ENJOY! ***MAKE SURE TO READ THE DESCRIPTION***<br /><a href="#" onclick="yt.www.watch.player.seekTo(00*60+36);return false;">00:36</a> - Bohemian Rhapsody<br /><a href="#" onclick="yt.www.watch.player.seekTo(02*60+42);return false;">02:42</a> - Radio Ga Ga<br /><a href="#" onclick="yt.www.watch.player.seekTo(06*60+53);return false;">06:53</a> - Ay Oh!<br /><a href="#" onclick="yt.www.watch.player.seekTo(07*60+34);return false;">07:34</a> - Hammer To Fall<br /><a href="#" onclick="yt.www.watch.player.seekTo(12*60+08);return false;">12:08</a> - Crazy Little Thing Called Love<br /><a href="#" onclick="yt.www.watch.player.seekTo(16*60+03);return false;">16:03</a> - We Will Rock You<br /><a href="#" onclick="yt.www.watch.player.seekTo(17*60+18);return false;">17:18</a> - We Are The Champions<br /><a href="#" onclick="yt.www.watch.player.seekTo(21*60+12);return false;">21:12</a> - Is This The World We Created...?<br /><br />Short song analysis:<br /><br />- "Bohemian Rhapsody": Although it's a short medley version, it's one of the best performances of the ballad section, with Freddie nailing the Bb4s with the correct studio phrasing (for the first time ever!).<br /><br />- "Radio Ga Ga": Although it's missing one chorus, this is one of - if not the best - the best versions ever, Freddie nails all the Bb4s and sounds very clean! Spike Edney's Roland Jupiter 8 also really shines through on this mix, compared to the DVD releases!<br /><br />- "Audience Improv": A great improv, Freddie sounds strong and confident. You gotta love when he sustains that A4 for 4 seconds!<br /><br />- "Hammer To Fall": Despite missing a verse and a chorus, it's a strong version (possibly the best ever). Freddie sings the song amazingly, and even ad-libs a C#5 and a C5! Also notice how heavy Brian's guitar sounds compared to the thin DVD mixes - it roars!<br /><br />- "Crazy Little Thing Called Love": A great version, the crowd loves the song, the jam is great as well! Only downside to this is the slight feedback issues.<br /><br />- "We Will Rock You": Although cut down to the 1st verse and chorus, Freddie sounds strong. He nails the A4, and the solo from Dr. May is brilliant!<br /><br />- "We Are the Champions": Perhaps the high-light of the performance - Freddie is very daring on this version, he sustains the pre-chorus Bb4s, nails the 1st C5, belts great A4s, but most importantly: He nails the chorus Bb4s, in all 3 choruses! This is the only time he has ever done so! It has to be said though, the last one sounds a bit rough, but that's a side effect of belting high notes for the past 18 minutes, with nodules AND laryngitis!<br /><br />- "Is This The World We Created... ?": Freddie and Brian perform a beautiful version of this, and it is one of the best versions ever. It's both sad and hilarious that a couple of BBC engineers are talking over the song, one of them being completely oblivious of the fact that he is interrupting the performance, on live television... Which was being televised to almost 2 billion homes.<br /><br /><br />All rights go to their respective owners!<br />-----Copyright Disclaimer Under Section 107 of the Copyright Act 1976, allowance is made for fair use for purposes such as criticism, comment, news reporting, teaching, scholarship, and research. Fair use is a use permitted by copyright statute that might otherwise be infringing. Non-profit, educational or personal use tips the balance in favor of fair use''',
+            1477,
+            [{
+                'start_time': 36,
+                'end_time': 162,
+                'title': 'Bohemian Rhapsody',
+            }, {
+                'start_time': 162,
+                'end_time': 413,
+                'title': 'Radio Ga Ga',
+            }, {
+                'start_time': 413,
+                'end_time': 454,
+                'title': 'Ay Oh!',
+            }, {
+                'start_time': 454,
+                'end_time': 728,
+                'title': 'Hammer To Fall',
+            }, {
+                'start_time': 728,
+                'end_time': 963,
+                'title': 'Crazy Little Thing Called Love',
+            }, {
+                'start_time': 963,
+                'end_time': 1038,
+                'title': 'We Will Rock You',
+            }, {
+                'start_time': 1038,
+                'end_time': 1272,
+                'title': 'We Are The Champions',
+            }, {
+                'start_time': 1272,
+                'end_time': 1477,
+                'title': 'Is This The World We Created...?',
+            }]
+        ),
+        (
+            # https://www.youtube.com/watch?v=ekYlRhALiRQ
+            # pattern: <num>. <title> 0:00
+            '1.  Those Beaten Paths of Confusion <a href="#" onclick="yt.www.watch.player.seekTo(0*60+00);return false;">0:00</a><br />2.  Beyond the Shadows of Emptiness & Nothingness <a href="#" onclick="yt.www.watch.player.seekTo(11*60+47);return false;">11:47</a><br />3.  Poison Yourself...With Thought <a href="#" onclick="yt.www.watch.player.seekTo(26*60+30);return false;">26:30</a><br />4.  The Agents of Transformation <a href="#" onclick="yt.www.watch.player.seekTo(35*60+57);return false;">35:57</a><br />5.  Drowning in the Pain of Consciousness <a href="#" onclick="yt.www.watch.player.seekTo(44*60+32);return false;">44:32</a><br />6.  Deny the Disease of Life <a href="#" onclick="yt.www.watch.player.seekTo(53*60+07);return false;">53:07</a><br /><br />More info/Buy: http://crepusculonegro.storenvy.com/products/257645-cn-03-arizmenda-within-the-vacuum-of-infinity<br /><br />No copyright is intended. The rights to this video are assumed by the owner and its affiliates.',
+            4009,
+            [{
+                'start_time': 0,
+                'end_time': 707,
+                'title': '1. Those Beaten Paths of Confusion',
+            }, {
+                'start_time': 707,
+                'end_time': 1590,
+                'title': '2. Beyond the Shadows of Emptiness & Nothingness',
+            }, {
+                'start_time': 1590,
+                'end_time': 2157,
+                'title': '3. Poison Yourself...With Thought',
+            }, {
+                'start_time': 2157,
+                'end_time': 2672,
+                'title': '4. The Agents of Transformation',
+            }, {
+                'start_time': 2672,
+                'end_time': 3187,
+                'title': '5. Drowning in the Pain of Consciousness',
+            }, {
+                'start_time': 3187,
+                'end_time': 4009,
+                'title': '6. Deny the Disease of Life',
+            }]
+        ),
+        (
+            # https://www.youtube.com/watch?v=WjL4pSzog9w
+            # pattern: 00:00 <title>
+            '<a href="https://arizmenda.bandcamp.com/merch/despairs-depths-descended-cd" class="yt-uix-servicelink  " data-target-new-window="True" data-servicelink="CDAQ6TgYACITCNf1raqT2dMCFdRjGAod_o0CBSj4HQ" data-url="https://arizmenda.bandcamp.com/merch/despairs-depths-descended-cd" rel="nofollow noopener" target="_blank">https://arizmenda.bandcamp.com/merch/...</a><br /><br /><a href="#" onclick="yt.www.watch.player.seekTo(00*60+00);return false;">00:00</a> Christening Unborn Deformities <br /><a href="#" onclick="yt.www.watch.player.seekTo(07*60+08);return false;">07:08</a> Taste of Purity<br /><a href="#" onclick="yt.www.watch.player.seekTo(16*60+16);return false;">16:16</a> Sculpting Sins of a Universal Tongue<br /><a href="#" onclick="yt.www.watch.player.seekTo(24*60+45);return false;">24:45</a> Birth<br /><a href="#" onclick="yt.www.watch.player.seekTo(31*60+24);return false;">31:24</a> Neves<br /><a href="#" onclick="yt.www.watch.player.seekTo(37*60+55);return false;">37:55</a> Libations in Limbo',
+            2705,
+            [{
+                'start_time': 0,
+                'end_time': 428,
+                'title': 'Christening Unborn Deformities',
+            }, {
+                'start_time': 428,
+                'end_time': 976,
+                'title': 'Taste of Purity',
+            }, {
+                'start_time': 976,
+                'end_time': 1485,
+                'title': 'Sculpting Sins of a Universal Tongue',
+            }, {
+                'start_time': 1485,
+                'end_time': 1884,
+                'title': 'Birth',
+            }, {
+                'start_time': 1884,
+                'end_time': 2275,
+                'title': 'Neves',
+            }, {
+                'start_time': 2275,
+                'end_time': 2705,
+                'title': 'Libations in Limbo',
+            }]
+        ),
+        (
+            # https://www.youtube.com/watch?v=o3r1sn-t3is
+            # pattern: <title> 00:00 <note>
+            'Download this show in MP3: <a href="http://sh.st/njZKK" class="yt-uix-servicelink  " data-url="http://sh.st/njZKK" data-target-new-window="True" data-servicelink="CDAQ6TgYACITCK3j8_6o2dMCFVDCGAoduVAKKij4HQ" rel="nofollow noopener" target="_blank">http://sh.st/njZKK</a><br /><br />Setlist:<br />I-E-A-I-A-I-O <a href="#" onclick="yt.www.watch.player.seekTo(00*60+45);return false;">00:45</a><br />Suite-Pee <a href="#" onclick="yt.www.watch.player.seekTo(4*60+26);return false;">4:26</a>  (Incomplete)<br />Attack <a href="#" onclick="yt.www.watch.player.seekTo(5*60+31);return false;">5:31</a> (First live performance since 2011)<br />Prison Song <a href="#" onclick="yt.www.watch.player.seekTo(8*60+42);return false;">8:42</a><br />Know <a href="#" onclick="yt.www.watch.player.seekTo(12*60+32);return false;">12:32</a> (First live performance since 2011)<br />Aerials <a href="#" onclick="yt.www.watch.player.seekTo(15*60+32);return false;">15:32</a><br />Soldier Side - Intro <a href="#" onclick="yt.www.watch.player.seekTo(19*60+13);return false;">19:13</a><br />B.Y.O.B. <a href="#" onclick="yt.www.watch.player.seekTo(20*60+09);return false;">20:09</a><br />Soil <a href="#" onclick="yt.www.watch.player.seekTo(24*60+32);return false;">24:32</a><br />Darts <a href="#" onclick="yt.www.watch.player.seekTo(27*60+48);return false;">27:48</a><br />Radio/Video <a href="#" onclick="yt.www.watch.player.seekTo(30*60+38);return false;">30:38</a><br />Hypnotize <a href="#" onclick="yt.www.watch.player.seekTo(35*60+05);return false;">35:05</a><br />Temper <a href="#" onclick="yt.www.watch.player.seekTo(38*60+08);return false;">38:08</a> (First live performance since 1999)<br />CUBErt <a href="#" onclick="yt.www.watch.player.seekTo(41*60+00);return false;">41:00</a><br />Needles <a href="#" onclick="yt.www.watch.player.seekTo(42*60+57);return false;">42:57</a><br />Deer Dance <a href="#" onclick="yt.www.watch.player.seekTo(46*60+27);return false;">46:27</a><br />Bounce <a href="#" onclick="yt.www.watch.player.seekTo(49*60+38);return false;">49:38</a><br />Suggestions <a href="#" onclick="yt.www.watch.player.seekTo(51*60+25);return false;">51:25</a><br />Psycho <a href="#" onclick="yt.www.watch.player.seekTo(53*60+52);return false;">53:52</a><br />Chop Suey! <a href="#" onclick="yt.www.watch.player.seekTo(58*60+13);return false;">58:13</a><br />Lonely Day <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+01*60+15);return false;">1:01:15</a><br />Question! <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+04*60+14);return false;">1:04:14</a><br />Lost in Hollywood <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+08*60+10);return false;">1:08:10</a><br />Vicinity of Obscenity  <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+13*60+40);return false;">1:13:40</a>(First live performance since 2012)<br />Forest <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+16*60+17);return false;">1:16:17</a><br />Cigaro <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+20*60+02);return false;">1:20:02</a><br />Toxicity <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+23*60+57);return false;">1:23:57</a>(with Chino Moreno)<br />Sugar <a href="#" onclick="yt.www.watch.player.seekTo(1*3600+27*60+53);return false;">1:27:53</a>',
+            5640,
+            [{
+                'start_time': 45,
+                'end_time': 266,
+                'title': 'I-E-A-I-A-I-O',
+            }, {
+                'start_time': 266,
+                'end_time': 331,
+                'title': 'Suite-Pee (Incomplete)',
+            }, {
+                'start_time': 331,
+                'end_time': 522,
+                'title': 'Attack (First live performance since 2011)',
+            }, {
+                'start_time': 522,
+                'end_time': 752,
+                'title': 'Prison Song',
+            }, {
+                'start_time': 752,
+                'end_time': 932,
+                'title': 'Know (First live performance since 2011)',
+            }, {
+                'start_time': 932,
+                'end_time': 1153,
+                'title': 'Aerials',
+            }, {
+                'start_time': 1153,
+                'end_time': 1209,
+                'title': 'Soldier Side - Intro',
+            }, {
+                'start_time': 1209,
+                'end_time': 1472,
+                'title': 'B.Y.O.B.',
+            }, {
+                'start_time': 1472,
+                'end_time': 1668,
+                'title': 'Soil',
+            }, {
+                'start_time': 1668,
+                'end_time': 1838,
+                'title': 'Darts',
+            }, {
+                'start_time': 1838,
+                'end_time': 2105,
+                'title': 'Radio/Video',
+            }, {
+                'start_time': 2105,
+                'end_time': 2288,
+                'title': 'Hypnotize',
+            }, {
+                'start_time': 2288,
+                'end_time': 2460,
+                'title': 'Temper (First live performance since 1999)',
+            }, {
+                'start_time': 2460,
+                'end_time': 2577,
+                'title': 'CUBErt',
+            }, {
+                'start_time': 2577,
+                'end_time': 2787,
+                'title': 'Needles',
+            }, {
+                'start_time': 2787,
+                'end_time': 2978,
+                'title': 'Deer Dance',
+            }, {
+                'start_time': 2978,
+                'end_time': 3085,
+                'title': 'Bounce',
+            }, {
+                'start_time': 3085,
+                'end_time': 3232,
+                'title': 'Suggestions',
+            }, {
+                'start_time': 3232,
+                'end_time': 3493,
+                'title': 'Psycho',
+            }, {
+                'start_time': 3493,
+                'end_time': 3675,
+                'title': 'Chop Suey!',
+            }, {
+                'start_time': 3675,
+                'end_time': 3854,
+                'title': 'Lonely Day',
+            }, {
+                'start_time': 3854,
+                'end_time': 4090,
+                'title': 'Question!',
+            }, {
+                'start_time': 4090,
+                'end_time': 4420,
+                'title': 'Lost in Hollywood',
+            }, {
+                'start_time': 4420,
+                'end_time': 4577,
+                'title': 'Vicinity of Obscenity (First live performance since 2012)',
+            }, {
+                'start_time': 4577,
+                'end_time': 4802,
+                'title': 'Forest',
+            }, {
+                'start_time': 4802,
+                'end_time': 5037,
+                'title': 'Cigaro',
+            }, {
+                'start_time': 5037,
+                'end_time': 5273,
+                'title': 'Toxicity (with Chino Moreno)',
+            }, {
+                'start_time': 5273,
+                'end_time': 5640,
+                'title': 'Sugar',
+            }]
+        ),
+        (
+            # https://www.youtube.com/watch?v=PkYLQbsqCE8
+            # pattern: <num> - <title> [<latinized title>] 0:00:00
+            '''Затемно (Zatemno) is an Obscure Black Metal Band from Russia.<br /><br />"Во прах (Vo prakh)'' Into The Ashes", Debut mini-album released may 6, 2016, by Death Knell Productions<br />Released on 6 panel digipak CD, limited to 100 copies only<br />And digital format on Bandcamp<br /><br />Tracklist<br /><br />1 - Во прах [Vo prakh] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+00*60+00);return false;">0:00:00</a><br />2 - Искупление [Iskupleniye] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+08*60+10);return false;">0:08:10</a><br />3 - Из серпов луны...[Iz serpov luny] <a href="#" onclick="yt.www.watch.player.seekTo(0*3600+14*60+30);return false;">0:14:30</a><br /><br />Links:<br /><a href="https://deathknellprod.bandcamp.com/album/--2" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://deathknellprod.bandcamp.com/album/--2" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://deathknellprod.bandcamp.com/a...</a><br /><a href="https://www.facebook.com/DeathKnellProd/" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://www.facebook.com/DeathKnellProd/" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://www.facebook.com/DeathKnellProd/</a><br /><br /><br />I don't have any right about this artifact, my only intention is to spread the music of the band, all rights are reserved to the Затемно (Zatemno) and his producers, Death Knell Productions.<br /><br />------------------------------------------------------------------<br /><br />Subscribe for more videos like this.<br />My link: <a href="https://web.facebook.com/AttackOfTheDragons" class="yt-uix-servicelink  " data-target-new-window="True" data-url="https://web.facebook.com/AttackOfTheDragons" data-servicelink="CC8Q6TgYACITCNP234Kr2dMCFcNxGAodQqsIwSj4HQ" target="_blank" rel="nofollow noopener">https://web.facebook.com/AttackOfTheD...</a>''',
+            1138,
+            [{
+                'start_time': 0,
+                'end_time': 490,
+                'title': '1 - Во прах [Vo prakh]',
+            }, {
+                'start_time': 490,
+                'end_time': 870,
+                'title': '2 - Искупление [Iskupleniye]',
+            }, {
+                'start_time': 870,
+                'end_time': 1138,
+                'title': '3 - Из серпов луны...[Iz serpov luny]',
+            }]
+        ),
+        (
+            # https://www.youtube.com/watch?v=xZW70zEasOk
+            # time point more than duration
+            '''● LCS Spring finals: Saturday and Sunday from <a href="#" onclick="yt.www.watch.player.seekTo(13*60+30);return false;">13:30</a> outside the venue! <br />● PAX East: Fri, Sat & Sun - more info in tomorrows video on the main channel!''',
+            283,
+            []
+        ),
+    ]
+
+    def test_youtube_chapters(self):
+        for description, duration, expected_chapters in self._TEST_CASES:
+            ie = YoutubeIE()
+            expect_value(
+                self, ie._extract_chapters(description, duration),
+                expected_chapters, None)
+
+
+if __name__ == '__main__':
+    unittest.main()

test/test_youtube_lists.py

@@ -1,5 +1,4 @@
 #!/usr/bin/env python
-# -*- coding: utf-8 -*-
 from __future__ import unicode_literals
 
 # Allow direct execution
@@ -10,10 +9,10 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from test.helper import FakeYDL
 
+
 from youtube_dl.extractor import (
-    YoutubeIE,
     YoutubePlaylistIE,
-    YoutubeTabIE,
+    YoutubeIE,
 )
 
 
@@ -25,40 +24,47 @@ class TestYoutubeLists(unittest.TestCase):
     def test_youtube_playlist_noplaylist(self):
         dl = FakeYDL()
         dl.params['noplaylist'] = True
-        dl.params['format'] = 'best'
         ie = YoutubePlaylistIE(dl)
         result = ie.extract('https://www.youtube.com/watch?v=FXxLjLQi3Fg&list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re')
         self.assertEqual(result['_type'], 'url')
-        result = dl.extract_info(result['url'], download=False, ie_key=result.get('ie_key'), process=False)
         self.assertEqual(YoutubeIE().extract_id(result['url']), 'FXxLjLQi3Fg')
 
+    def test_youtube_course(self):
+        dl = FakeYDL()
+        ie = YoutubePlaylistIE(dl)
+        # TODO find a > 100 (paginating?) videos course
+        result = ie.extract('https://www.youtube.com/course?list=ECUl4u3cNGP61MdtwGTqZA0MreSaDybji8')
+        entries = list(result['entries'])
+        self.assertEqual(YoutubeIE().extract_id(entries[0]['url']), 'j9WZyLZCBzs')
+        self.assertEqual(len(entries), 25)
+        self.assertEqual(YoutubeIE().extract_id(entries[-1]['url']), 'rYefUsYuEp0')
+
     def test_youtube_mix(self):
         dl = FakeYDL()
-        dl.params['format'] = 'best'
-        ie = YoutubeTabIE(dl)
-        result = dl.extract_info('https://www.youtube.com/watch?v=tyITL_exICo&list=RDCLAK5uy_kLWIr9gv1XLlPbaDS965-Db4TrBoUTxQ8',
-                                 download=False, ie_key=ie.ie_key(), process=True)
-        entries = (result or {}).get('entries', [{'id': 'not_found', }])
-        self.assertTrue(len(entries) >= 25)
+        ie = YoutubePlaylistIE(dl)
+        result = ie.extract('https://www.youtube.com/watch?v=W01L70IGBgE&index=2&list=RDOQpdSVF_k_w')
+        entries = result['entries']
+        self.assertTrue(len(entries) >= 50)
         original_video = entries[0]
-        self.assertEqual(original_video['id'], 'tyITL_exICo')
+        self.assertEqual(original_video['id'], 'OQpdSVF_k_w')
 
-    def test_youtube_flat_playlist_extraction(self):
+    def test_youtube_toptracks(self):
+        print('Skipping: The playlist page gives error 500')
+        return
+        dl = FakeYDL()
+        ie = YoutubePlaylistIE(dl)
+        result = ie.extract('https://www.youtube.com/playlist?list=MCUS')
+        entries = result['entries']
+        self.assertEqual(len(entries), 100)
+
+    def test_youtube_flat_playlist_titles(self):
         dl = FakeYDL()
         dl.params['extract_flat'] = True
-        ie = YoutubeTabIE(dl)
-        result = ie.extract('https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc')
+        ie = YoutubePlaylistIE(dl)
+        result = ie.extract('https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re')
         self.assertIsPlaylist(result)
-        entries = list(result['entries'])
-        self.assertTrue(len(entries) == 1)
-        video = entries[0]
-        self.assertEqual(video['_type'], 'url')
-        self.assertEqual(video['ie_key'], 'Youtube')
-        self.assertEqual(video['id'], 'BaW_jenozKc')
-        self.assertEqual(video['url'], 'BaW_jenozKc')
-        self.assertEqual(video['title'], 'youtube-dl test video "\'/\\ä↭𝕐')
-        self.assertEqual(video['duration'], 10)
-        self.assertEqual(video['uploader'], 'Philipp Hagemeister')
+        for entry in result['entries']:
+            self.assertTrue(entry.get('title'))
 
 
 if __name__ == '__main__':

test/test_youtube_misc.py

@@ -1,26 +0,0 @@
-#!/usr/bin/env python
-from __future__ import unicode_literals
-
-# Allow direct execution
-import os
-import sys
-import unittest
-sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-
-
-from youtube_dl.extractor import YoutubeIE
-
-
-class TestYoutubeMisc(unittest.TestCase):
-    def test_youtube_extract(self):
-        assertExtractId = lambda url, id: self.assertEqual(YoutubeIE.extract_id(url), id)
-        assertExtractId('http://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
-        assertExtractId('https://www.youtube.com/watch?&v=BaW_jenozKc', 'BaW_jenozKc')
-        assertExtractId('https://www.youtube.com/watch?feature=player_embedded&v=BaW_jenozKc', 'BaW_jenozKc')
-        assertExtractId('https://www.youtube.com/watch_popup?v=BaW_jenozKc', 'BaW_jenozKc')
-        assertExtractId('http://www.youtube.com/watch?v=BaW_jenozKcsharePLED17F32AD9753930', 'BaW_jenozKc')
-        assertExtractId('BaW_jenozKc', 'BaW_jenozKc')
-
-
-if __name__ == '__main__':
-    unittest.main()

test/test_youtube_signature.py

@@ -12,213 +12,111 @@ import io
 import re
 import string
 
-from youtube_dl.compat import compat_str, compat_urlretrieve
-
 from test.helper import FakeYDL
 from youtube_dl.extractor import YoutubeIE
-from youtube_dl.jsinterp import JSInterpreter
+from youtube_dl.compat import compat_str, compat_urlretrieve
 
-_SIG_TESTS = [
+_TESTS = [
     (
         'https://s.ytimg.com/yts/jsbin/html5player-vflHOr_nV.js',
+        'js',
         86,
         '>=<;:/.-[+*)(\'&%$#"!ZYX0VUTSRQPONMLKJIHGFEDCBA\\yxwvutsrqponmlkjihgfedcba987654321',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-vfldJ8xgI.js',
+        'js',
         85,
         '3456789a0cdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRS[UVWXYZ!"#$%&\'()*+,-./:;<=>?@',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-vfle-mVwz.js',
+        'js',
         90,
         ']\\[@?>=<;:/.-,+*)(\'&%$#"hZYXWVUTSRQPONMLKJIHGFEDCBAzyxwvutsrqponmlkjiagfedcb39876',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vfl0Cbn9e.js',
+        'js',
         84,
         'O1I3456789abcde0ghijklmnopqrstuvwxyzABCDEFGHfJKLMN2PQRSTUVW@YZ!"#$%&\'()*+,-./:;<=',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflXGBaUN.js',
+        'js',
         '2ACFC7A61CA478CD21425E5A57EBD73DDC78E22A.2094302436B2D377D14A3BBA23022D023B8BC25AA',
         'A52CB8B320D22032ABB3A41D773D2B6342034902.A22E87CDD37DBE75A5E52412DC874AC16A7CFCA2',
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflBb0OQx.js',
+        'js',
         84,
         '123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQ0STUVWXYZ!"#$%&\'()*+,@./:;<=>'
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vfl9FYC6l.js',
+        'js',
         83,
         '123456789abcdefghijklmnopqr0tuvwxyzABCDETGHIJKLMNOPQRS>UVWXYZ!"#$%&\'()*+,-./:;<=F'
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflCGk6yw/html5player.js',
+        'js',
         '4646B5181C6C3020DF1D9C7FCFEA.AD80ABF70C39BD369CCCAE780AFBB98FA6B6CB42766249D9488C288',
         '82C8849D94266724DC6B6AF89BBFA087EACCD963.B93C07FBA084ACAEFCF7C9D1FD0203C6C1815B6B'
     ),
     (
         'https://s.ytimg.com/yts/jsbin/html5player-en_US-vflKjOTVq/html5player.js',
+        'js',
         '312AA52209E3623129A412D56A40F11CB0AF14AE.3EE09501CB14E3BCDC3B2AE808BF3F1D14E7FBF12',
         '112AA5220913623229A412D56A40F11CB0AF14AE.3EE0950FCB14EEBCDC3B2AE808BF331D14E7FBF3',
     )
 ]
 
-_NSIG_TESTS = [
-    (
-        'https://www.youtube.com/s/player/9216d1f7/player_ias.vflset/en_US/base.js',
-        'SLp9F5bwjAdhE9F-', 'gWnb9IK2DJ8Q1w',
-    ),
-    (
-        'https://www.youtube.com/s/player/f8cb7a3b/player_ias.vflset/en_US/base.js',
-        'oBo2h5euWy6osrUt', 'ivXHpm7qJjJN',
-    ),
-    (
-        'https://www.youtube.com/s/player/2dfe380c/player_ias.vflset/en_US/base.js',
-        'oBo2h5euWy6osrUt', '3DIBbn3qdQ',
-    ),
-    (
-        'https://www.youtube.com/s/player/f1ca6900/player_ias.vflset/en_US/base.js',
-        'cu3wyu6LQn2hse', 'jvxetvmlI9AN9Q',
-    ),
-    (
-        'https://www.youtube.com/s/player/8040e515/player_ias.vflset/en_US/base.js',
-        'wvOFaY-yjgDuIEg5', 'HkfBFDHmgw4rsw',
-    ),
-    (
-        'https://www.youtube.com/s/player/e06dea74/player_ias.vflset/en_US/base.js',
-        'AiuodmaDDYw8d3y4bf', 'ankd8eza2T6Qmw',
-    ),
-    (
-        'https://www.youtube.com/s/player/5dd88d1d/player-plasma-ias-phone-en_US.vflset/base.js',
-        'kSxKFLeqzv_ZyHSAt', 'n8gS8oRlHOxPFA',
-    ),
-    (
-        'https://www.youtube.com/s/player/324f67b9/player_ias.vflset/en_US/base.js',
-        'xdftNy7dh9QGnhW', '22qLGxrmX8F1rA',
-    ),
-    (
-        'https://www.youtube.com/s/player/4c3f79c5/player_ias.vflset/en_US/base.js',
-        'TDCstCG66tEAO5pR9o', 'dbxNtZ14c-yWyw',
-    ),
-    (
-        'https://www.youtube.com/s/player/c81bbb4a/player_ias.vflset/en_US/base.js',
-        'gre3EcLurNY2vqp94', 'Z9DfGxWP115WTg',
-    ),
-    (
-        'https://www.youtube.com/s/player/1f7d5369/player_ias.vflset/en_US/base.js',
-        'batNX7sYqIJdkJ', 'IhOkL_zxbkOZBw',
-    ),
-    (
-        'https://www.youtube.com/s/player/009f1d77/player_ias.vflset/en_US/base.js',
-        '5dwFHw8aFWQUQtffRq', 'audescmLUzI3jw',
-    ),
-    (
-        'https://www.youtube.com/s/player/dc0c6770/player_ias.vflset/en_US/base.js',
-        '5EHDMgYLV6HPGk_Mu-kk', 'n9lUJLHbxUI0GQ',
-    ),
-    (
-        'https://www.youtube.com/s/player/c2199353/player_ias.vflset/en_US/base.js',
-        '5EHDMgYLV6HPGk_Mu-kk', 'AD5rgS85EkrE7',
-    ),
-    (
-        'https://www.youtube.com/s/player/113ca41c/player_ias.vflset/en_US/base.js',
-        'cgYl-tlYkhjT7A', 'hI7BBr2zUgcmMg',
-    ),
-    (
-        'https://www.youtube.com/s/player/c57c113c/player_ias.vflset/en_US/base.js',
-        '-Txvy6bT5R6LqgnQNx', 'dcklJCnRUHbgSg',
-    ),
-    (
-        'https://www.youtube.com/s/player/5a3b6271/player_ias.vflset/en_US/base.js',
-        'B2j7f_UPT4rfje85Lu_e', 'm5DmNymaGQ5RdQ',
-    ),
-]
-
-
-class TestPlayerInfo(unittest.TestCase):
-    def test_youtube_extract_player_info(self):
-        PLAYER_URLS = (
-            ('https://www.youtube.com/s/player/4c3f79c5/player_ias.vflset/en_US/base.js', '4c3f79c5'),
-            ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/en_US/base.js', '64dddad9'),
-            ('https://www.youtube.com/s/player/64dddad9/player_ias.vflset/fr_FR/base.js', '64dddad9'),
-            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-phone-en_US.vflset/base.js', '64dddad9'),
-            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-phone-de_DE.vflset/base.js', '64dddad9'),
-            ('https://www.youtube.com/s/player/64dddad9/player-plasma-ias-tablet-en_US.vflset/base.js', '64dddad9'),
-            # obsolete
-            ('https://www.youtube.com/yts/jsbin/player_ias-vfle4-e03/en_US/base.js', 'vfle4-e03'),
-            ('https://www.youtube.com/yts/jsbin/player_ias-vfl49f_g4/en_US/base.js', 'vfl49f_g4'),
-            ('https://www.youtube.com/yts/jsbin/player_ias-vflCPQUIL/en_US/base.js', 'vflCPQUIL'),
-            ('https://www.youtube.com/yts/jsbin/player-vflzQZbt7/en_US/base.js', 'vflzQZbt7'),
-            ('https://www.youtube.com/yts/jsbin/player-en_US-vflaxXRn1/base.js', 'vflaxXRn1'),
-            ('https://s.ytimg.com/yts/jsbin/html5player-en_US-vflXGBaUN.js', 'vflXGBaUN'),
-            ('https://s.ytimg.com/yts/jsbin/html5player-en_US-vflKjOTVq/html5player.js', 'vflKjOTVq'),
-        )
-        for player_url, expected_player_id in PLAYER_URLS:
-            player_id = YoutubeIE._extract_player_info(player_url)
-            self.assertEqual(player_id, expected_player_id)
-
 
 class TestSignature(unittest.TestCase):
     def setUp(self):
         TEST_DIR = os.path.dirname(os.path.abspath(__file__))
-        self.TESTDATA_DIR = os.path.join(TEST_DIR, 'testdata/sigs')
+        self.TESTDATA_DIR = os.path.join(TEST_DIR, 'testdata')
         if not os.path.exists(self.TESTDATA_DIR):
             os.mkdir(self.TESTDATA_DIR)
 
-    def tearDown(self):
-        try:
-            for f in os.listdir(self.TESTDATA_DIR):
-                os.remove(f)
-        except OSError:
-            pass
 
+def make_tfunc(url, stype, sig_input, expected_sig):
+    m = re.match(r'.*-([a-zA-Z0-9_-]+)(?:/watch_as3|/html5player)?\.[a-z]+$', url)
+    assert m, '%r should follow URL format' % url
+    test_id = m.group(1)
 
-def t_factory(name, sig_func, url_pattern):
-    def make_tfunc(url, sig_input, expected_sig):
-        m = url_pattern.match(url)
-        assert m, '%r should follow URL format' % url
-        test_id = m.group('id')
+    def test_func(self):
+        basename = 'player-%s.%s' % (test_id, stype)
+        fn = os.path.join(self.TESTDATA_DIR, basename)
 
-        def test_func(self):
-            basename = 'player-{0}-{1}.js'.format(name, test_id)
-            fn = os.path.join(self.TESTDATA_DIR, basename)
+        if not os.path.exists(fn):
+            compat_urlretrieve(url, fn)
 
-            if not os.path.exists(fn):
-                compat_urlretrieve(url, fn)
+        ydl = FakeYDL()
+        ie = YoutubeIE(ydl)
+        if stype == 'js':
             with io.open(fn, encoding='utf-8') as testf:
                 jscode = testf.read()
-            self.assertEqual(sig_func(jscode, sig_input), expected_sig)
+            func = ie._parse_sig_js(jscode)
+        else:
+            assert stype == 'swf'
+            with open(fn, 'rb') as testf:
+                swfcode = testf.read()
+            func = ie._parse_sig_swf(swfcode)
+        src_sig = (
+            compat_str(string.printable[:sig_input])
+            if isinstance(sig_input, int) else sig_input)
+        got_sig = func(src_sig)
+        self.assertEqual(got_sig, expected_sig)
 
-        test_func.__name__ = str('test_{0}_js_{1}'.format(name, test_id))
-        setattr(TestSignature, test_func.__name__, test_func)
-    return make_tfunc
+    test_func.__name__ = str('test_signature_' + stype + '_' + test_id)
+    setattr(TestSignature, test_func.__name__, test_func)
 
 
-def signature(jscode, sig_input):
-    func = YoutubeIE(FakeYDL())._parse_sig_js(jscode)
-    src_sig = (
-        compat_str(string.printable[:sig_input])
-        if isinstance(sig_input, int) else sig_input)
-    return func(src_sig)
-
-
-def n_sig(jscode, sig_input):
-    funcname = YoutubeIE(FakeYDL())._extract_n_function_name(jscode)
-    return JSInterpreter(jscode).call_function(funcname, sig_input)
-
-
-make_sig_test = t_factory(
-    'signature', signature, re.compile(r'.*-(?P<id>[a-zA-Z0-9_-]+)(?:/watch_as3|/html5player)?\.[a-z]+$'))
-for test_spec in _SIG_TESTS:
-    make_sig_test(*test_spec)
-
-make_nsig_test = t_factory(
-    'nsig', n_sig, re.compile(r'.+/player/(?P<id>[a-zA-Z0-9_-]+)/.+.js$'))
-for test_spec in _NSIG_TESTS:
-    make_nsig_test(*test_spec)
+for test_spec in _TESTS:
+    make_tfunc(*test_spec)
 
 
 if __name__ == '__main__':

test/testdata/cookies/httponly_cookies.txt

@@ -1,6 +0,0 @@
-# Netscape HTTP Cookie File
-# http://curl.haxx.se/rfc/cookie_spec.html
-# This is a generated file!  Do not edit.
-
-#HttpOnly_www.foobar.foobar	FALSE	/	TRUE	2147483647	HTTPONLY_COOKIE	HTTPONLY_COOKIE_VALUE
-www.foobar.foobar	FALSE	/	TRUE	2147483647	JS_ACCESSIBLE_COOKIE	JS_ACCESSIBLE_COOKIE_VALUE

test/testdata/cookies/malformed_cookies.txt

@@ -1,9 +0,0 @@
-# Netscape HTTP Cookie File
-# http://curl.haxx.se/rfc/cookie_spec.html
-# This is a generated file!  Do not edit.
-
-# Cookie file entry with invalid number of fields - 6 instead of 7
-www.foobar.foobar	FALSE	/	FALSE	0	COOKIE
-
-# Cookie file entry with invalid expires at
-www.foobar.foobar	FALSE	/	FALSE	1.7976931348623157e+308	COOKIE	VALUE

test/testdata/cookies/session_cookies.txt

@@ -1,6 +0,0 @@
-# Netscape HTTP Cookie File
-# http://curl.haxx.se/rfc/cookie_spec.html
-# This is a generated file!  Do not edit.
-
-www.foobar.foobar	FALSE	/	TRUE		YoutubeDLExpiresEmpty	YoutubeDLExpiresEmptyValue
-www.foobar.foobar	FALSE	/	TRUE	0	YoutubeDLExpires0	YoutubeDLExpires0Value

test/testdata/m3u8/ted_18923.m3u8

@@ -1,28 +0,0 @@
-#EXTM3U
-#EXT-X-VERSION:4
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=1255659,PROGRAM-ID=1,CODECS="avc1.42c01e,mp4a.40.2",RESOLUTION=640x360
-/videos/BorisHesser_2018S/video/600k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=163154,PROGRAM-ID=1,CODECS="avc1.42c00c,mp4a.40.2",RESOLUTION=320x180
-/videos/BorisHesser_2018S/video/64k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=481701,PROGRAM-ID=1,CODECS="avc1.42c015,mp4a.40.2",RESOLUTION=512x288
-/videos/BorisHesser_2018S/video/180k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=769968,PROGRAM-ID=1,CODECS="avc1.42c015,mp4a.40.2",RESOLUTION=512x288
-/videos/BorisHesser_2018S/video/320k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=984037,PROGRAM-ID=1,CODECS="avc1.42c015,mp4a.40.2",RESOLUTION=512x288
-/videos/BorisHesser_2018S/video/450k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=1693925,PROGRAM-ID=1,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=853x480
-/videos/BorisHesser_2018S/video/950k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=2462469,PROGRAM-ID=1,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=1280x720
-/videos/BorisHesser_2018S/video/1500k.m3u8?nobumpers=true&uniqueId=76011e2b
-#EXT-X-STREAM-INF:AUDIO="600k",BANDWIDTH=68101,PROGRAM-ID=1,CODECS="mp4a.40.2",DEFAULT=YES
-/videos/BorisHesser_2018S/audio/600k.m3u8?nobumpers=true&uniqueId=76011e2b
-
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=74298,PROGRAM-ID=1,CODECS="avc1.42c00c",RESOLUTION=320x180,URI="/videos/BorisHesser_2018S/video/64k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=216200,PROGRAM-ID=1,CODECS="avc1.42c015",RESOLUTION=512x288,URI="/videos/BorisHesser_2018S/video/180k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=304717,PROGRAM-ID=1,CODECS="avc1.42c015",RESOLUTION=512x288,URI="/videos/BorisHesser_2018S/video/320k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=350933,PROGRAM-ID=1,CODECS="avc1.42c015",RESOLUTION=512x288,URI="/videos/BorisHesser_2018S/video/450k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=495850,PROGRAM-ID=1,CODECS="avc1.42c01e",RESOLUTION=640x360,URI="/videos/BorisHesser_2018S/video/600k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=810750,PROGRAM-ID=1,CODECS="avc1.4d401f",RESOLUTION=853x480,URI="/videos/BorisHesser_2018S/video/950k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-#EXT-X-I-FRAME-STREAM-INF:BANDWIDTH=1273700,PROGRAM-ID=1,CODECS="avc1.640028",RESOLUTION=1280x720,URI="/videos/BorisHesser_2018S/video/1500k_iframe.m3u8?nobumpers=true&uniqueId=76011e2b"
-
-#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="600k",LANGUAGE="en",NAME="Audio",AUTOSELECT=YES,DEFAULT=YES,URI="/videos/BorisHesser_2018S/audio/600k.m3u8?nobumpers=true&uniqueId=76011e2b",BANDWIDTH=614400

test/testdata/mpd/unfragmented.mpd

@@ -1,28 +0,0 @@
-<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
-<MPD mediaPresentationDuration="PT54.915S" minBufferTime="PT1.500S" profiles="urn:mpeg:dash:profile:isoff-on-demand:2011" type="static" xmlns="urn:mpeg:dash:schema:mpd:2011">
-    <Period duration="PT54.915S">
-        <AdaptationSet segmentAlignment="true" subsegmentAlignment="true" subsegmentStartsWithSAP="1">
-            <Representation bandwidth="804261" codecs="avc1.4d401e" frameRate="30" height="360" id="VIDEO-1" mimeType="video/mp4" startWithSAP="1" width="360">
-                <BaseURL>DASH_360</BaseURL>
-                <SegmentBase indexRange="915-1114" indexRangeExact="true">
-                    <Initialization range="0-914"/>
-                </SegmentBase>
-            </Representation>
-            <Representation bandwidth="608000" codecs="avc1.4d401e" frameRate="30" height="240" id="VIDEO-2" mimeType="video/mp4" startWithSAP="1" width="240">
-                <BaseURL>DASH_240</BaseURL>
-                <SegmentBase indexRange="913-1112" indexRangeExact="true">
-                    <Initialization range="0-912"/>
-                </SegmentBase>
-            </Representation>
-        </AdaptationSet>
-        <AdaptationSet>
-            <Representation audioSamplingRate="48000" bandwidth="129870" codecs="mp4a.40.2" id="AUDIO-1" mimeType="audio/mp4" startWithSAP="1">
-                <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2"/>
-                <BaseURL>audio</BaseURL>
-                <SegmentBase indexRange="832-1007" indexRangeExact="true">
-                    <Initialization range="0-831"/>
-                </SegmentBase>
-            </Representation>
-        </AdaptationSet>
-    </Period>
-</MPD>

test/testdata/xspf/foo_xspf.xspf

@@ -1,34 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<playlist version="1" xmlns="http://xspf.org/ns/0/">
-    <date>2018-03-09T18:01:43Z</date>
-    <trackList>
-        <track>
-            <location>cd1/track%201.mp3</location>
-            <title>Pandemonium</title>
-            <creator>Foilverb</creator>
-            <annotation>Visit http://bigbrother404.bandcamp.com</annotation>
-            <album>Pandemonium EP</album>
-            <trackNum>1</trackNum>
-            <duration>202416</duration>
-        </track>
-        <track>
-            <location>../%E3%83%88%E3%83%A9%E3%83%83%E3%82%AF%E3%80%80%EF%BC%92.mp3</location>
-            <title>Final Cartridge (Nichico Twelve Remix)</title>
-            <annotation>Visit http://bigbrother404.bandcamp.com</annotation>
-            <creator>Foilverb</creator>
-            <album>Pandemonium EP</album>
-            <trackNum>2</trackNum>
-            <duration>255857</duration>
-        </track>
-        <track>
-            <location>track3.mp3</location>
-            <location>https://example.com/track3.mp3</location>
-            <title>Rebuilding Nightingale</title>
-            <annotation>Visit http://bigbrother404.bandcamp.com</annotation>
-            <creator>Foilverb</creator>
-            <album>Pandemonium EP</album>
-            <trackNum>3</trackNum>
-            <duration>287915</duration>
-        </track>
-    </trackList>
-</playlist>

youtube-dl.plugin.zsh

@@ -7,7 +7,7 @@
 # https://github.com/zsh-users/antigen
 
 # Install youtube-dl:
-# antigen bundle ytdl-org/youtube-dl
+# antigen bundle rg3/youtube-dl
 # Bundles installed by antigen are available for use immediately.
 
 # Update youtube-dl (and all other antigen bundles):

youtube_dl/__init__.py

@@ -48,7 +48,7 @@ from .YoutubeDL import YoutubeDL
 def _real_main(argv=None):
     # Compatibility fixes for Windows
     if sys.platform == 'win32':
-        # https://github.com/ytdl-org/youtube-dl/issues/820
+        # https://github.com/rg3/youtube-dl/issues/820
         codecs.register(lambda name: codecs.lookup('utf-8') if name == 'cp65001' else None)
 
     workaround_optparse_bug9161()
@@ -94,7 +94,7 @@ def _real_main(argv=None):
             if opts.verbose:
                 write_string('[debug] Batch file urls: ' + repr(batch_urls) + '\n')
         except IOError:
-            sys.exit('ERROR: batch file %s could not be read' % opts.batchfile)
+            sys.exit('ERROR: batch file could not be read')
     all_urls = batch_urls + [url.strip() for url in args]  # batch_urls are already striped in read_batch_urls
     _enc = preferredencoding()
     all_urls = [url.decode(_enc, 'ignore') if isinstance(url, bytes) else url for url in all_urls]
@@ -166,8 +166,6 @@ def _real_main(argv=None):
     if opts.max_sleep_interval is not None:
         if opts.max_sleep_interval < 0:
             parser.error('max sleep interval must be positive or 0')
-        if opts.sleep_interval is None:
-            parser.error('min sleep interval must be specified, use --min-sleep-interval')
         if opts.max_sleep_interval < opts.sleep_interval:
             parser.error('max sleep interval must be greater than or equal to min sleep interval')
     else:
@@ -193,11 +191,6 @@ def _real_main(argv=None):
         if numeric_buffersize is None:
             parser.error('invalid buffer size specified')
         opts.buffersize = numeric_buffersize
-    if opts.http_chunk_size is not None:
-        numeric_chunksize = FileDownloader.parse_bytes(opts.http_chunk_size)
-        if not numeric_chunksize:
-            parser.error('invalid http chunk size specified')
-        opts.http_chunk_size = numeric_chunksize
     if opts.playliststart <= 0:
         raise ValueError('Playlist start must be positive')
     if opts.playlistend not in (-1, None) and opts.playlistend < opts.playliststart:
@@ -230,14 +223,14 @@ def _real_main(argv=None):
     if opts.allsubtitles and not opts.writeautomaticsub:
         opts.writesubtitles = True
 
-    outtmpl = ((opts.outtmpl is not None and opts.outtmpl)
-               or (opts.format == '-1' and opts.usetitle and '%(title)s-%(id)s-%(format)s.%(ext)s')
-               or (opts.format == '-1' and '%(id)s-%(format)s.%(ext)s')
-               or (opts.usetitle and opts.autonumber and '%(autonumber)s-%(title)s-%(id)s.%(ext)s')
-               or (opts.usetitle and '%(title)s-%(id)s.%(ext)s')
-               or (opts.useid and '%(id)s.%(ext)s')
-               or (opts.autonumber and '%(autonumber)s-%(id)s.%(ext)s')
-               or DEFAULT_OUTTMPL)
+    outtmpl = ((opts.outtmpl is not None and opts.outtmpl) or
+               (opts.format == '-1' and opts.usetitle and '%(title)s-%(id)s-%(format)s.%(ext)s') or
+               (opts.format == '-1' and '%(id)s-%(format)s.%(ext)s') or
+               (opts.usetitle and opts.autonumber and '%(autonumber)s-%(title)s-%(id)s.%(ext)s') or
+               (opts.usetitle and '%(title)s-%(id)s.%(ext)s') or
+               (opts.useid and '%(id)s.%(ext)s') or
+               (opts.autonumber and '%(autonumber)s-%(id)s.%(ext)s') or
+               DEFAULT_OUTTMPL)
     if not os.path.splitext(outtmpl)[1] and opts.extractaudio:
         parser.error('Cannot download a video and extract audio into the same'
                      ' file! Use "{0}.%(ext)s" instead of "{0}" as the output'
@@ -340,7 +333,6 @@ def _real_main(argv=None):
         'format': opts.format,
         'listformats': opts.listformats,
         'outtmpl': outtmpl,
-        'outtmpl_na_placeholder': opts.outtmpl_na_placeholder,
         'autonumber_size': opts.autonumber_size,
         'autonumber_start': opts.autonumber_start,
         'restrictfilenames': opts.restrictfilenames,
@@ -354,7 +346,6 @@ def _real_main(argv=None):
         'keep_fragments': opts.keep_fragments,
         'buffersize': opts.buffersize,
         'noresizebuffer': opts.noresizebuffer,
-        'http_chunk_size': opts.http_chunk_size,
         'continuedl': opts.continue_dl,
         'noprogress': opts.noprogress,
         'progress_with_newline': opts.progress_with_newline,
@@ -433,7 +424,6 @@ def _real_main(argv=None):
         'config_location': opts.config_location,
         'geo_bypass': opts.geo_bypass,
         'geo_bypass_country': opts.geo_bypass_country,
-        'geo_bypass_ip_block': opts.geo_bypass_ip_block,
         # just for deprecation check
         'autonumber': opts.autonumber if opts.autonumber is True else None,
         'usetitle': opts.usetitle if opts.usetitle is True else None,

youtube_dl/aes.py

@@ -1,25 +1,13 @@
 from __future__ import unicode_literals
 
+import base64
 from math import ceil
 
-from .compat import compat_b64decode
 from .utils import bytes_to_intlist, intlist_to_bytes
 
 BLOCK_SIZE_BYTES = 16
 
 
-def pkcs7_padding(data):
-    """
-    PKCS#7 padding
-
-    @param {int[]} data        cleartext
-    @returns {int[]}           padding data
-    """
-
-    remaining_length = BLOCK_SIZE_BYTES - len(data) % BLOCK_SIZE_BYTES
-    return data + [remaining_length] * remaining_length
-
-
 def aes_ctr_decrypt(data, key, counter):
     """
     Decrypt with aes in counter mode
@@ -88,7 +76,8 @@ def aes_cbc_encrypt(data, key, iv):
     previous_cipher_block = iv
     for i in range(block_count):
         block = data[i * BLOCK_SIZE_BYTES: (i + 1) * BLOCK_SIZE_BYTES]
-        block = pkcs7_padding(block)
+        remaining_length = BLOCK_SIZE_BYTES - len(block)
+        block += [remaining_length] * remaining_length
         mixed_block = xor(block, previous_cipher_block)
 
         encrypted_block = aes_encrypt(mixed_block, expanded_key)
@@ -99,28 +88,6 @@ def aes_cbc_encrypt(data, key, iv):
     return encrypted_data
 
 
-def aes_ecb_encrypt(data, key):
-    """
-    Encrypt with aes in ECB mode. Using PKCS#7 padding
-
-    @param {int[]} data        cleartext
-    @param {int[]} key         16/24/32-Byte cipher key
-    @returns {int[]}           encrypted data
-    """
-    expanded_key = key_expansion(key)
-    block_count = int(ceil(float(len(data)) / BLOCK_SIZE_BYTES))
-
-    encrypted_data = []
-    for i in range(block_count):
-        block = data[i * BLOCK_SIZE_BYTES: (i + 1) * BLOCK_SIZE_BYTES]
-        block = pkcs7_padding(block)
-
-        encrypted_block = aes_encrypt(block, expanded_key)
-        encrypted_data += encrypted_block
-
-    return encrypted_data
-
-
 def key_expansion(data):
     """
     Generate key schedule
@@ -213,7 +180,7 @@ def aes_decrypt_text(data, password, key_size_bytes):
     """
     NONCE_LENGTH_BYTES = 8
 
-    data = bytes_to_intlist(compat_b64decode(data))
+    data = bytes_to_intlist(base64.b64decode(data.encode('utf-8')))
     password = bytes_to_intlist(password.encode('utf-8'))
 
     key = password[:key_size_bytes] + [0] * (key_size_bytes - len(password))
@@ -336,7 +303,7 @@ def xor(data1, data2):
 
 
 def rijndael_mul(a, b):
-    if (a == 0 or b == 0):
+    if(a == 0 or b == 0):
         return 0
     return RIJNDAEL_EXP_TABLE[(RIJNDAEL_LOG_TABLE[a] + RIJNDAEL_LOG_TABLE[b]) % 0xFF]
 

youtube_dl/cache.py

@@ -10,21 +10,12 @@ import traceback
 
 from .compat import compat_getenv
 from .utils import (
-    error_to_compat_str,
     expand_path,
-    is_outdated_version,
-    try_get,
     write_json_file,
 )
-from .version import __version__
 
 
 class Cache(object):
-
-    _YTDL_DIR = 'youtube-dl'
-    _VERSION_KEY = _YTDL_DIR + '_version'
-    _DEFAULT_VERSION = '2021.12.17'
-
     def __init__(self, ydl):
         self._ydl = ydl
 
@@ -32,7 +23,7 @@ class Cache(object):
         res = self._ydl.params.get('cachedir')
         if res is None:
             cache_root = compat_getenv('XDG_CACHE_HOME', '~/.cache')
-            res = os.path.join(cache_root, self._YTDL_DIR)
+            res = os.path.join(cache_root, 'youtube-dl')
         return expand_path(res)
 
     def _get_cache_fn(self, section, key, dtype):
@@ -59,22 +50,13 @@ class Cache(object):
             except OSError as ose:
                 if ose.errno != errno.EEXIST:
                     raise
-            write_json_file({self._VERSION_KEY: __version__, 'data': data}, fn)
+            write_json_file(data, fn)
         except Exception:
             tb = traceback.format_exc()
             self._ydl.report_warning(
                 'Writing cache to %r failed: %s' % (fn, tb))
 
-    def _validate(self, data, min_ver):
-        version = try_get(data, lambda x: x[self._VERSION_KEY])
-        if not version:  # Backward compatibility
-            data, version = {'data': data}, self._DEFAULT_VERSION
-        if not is_outdated_version(version, min_ver or '0', assume_new=False):
-            return data['data']
-        self._ydl.to_screen(
-            'Discarding old cache from version {version} (needs {min_ver})'.format(**locals()))
-
-    def load(self, section, key, dtype='json', default=None, min_ver=None):
+    def load(self, section, key, dtype='json', default=None):
         assert dtype in ('json',)
 
         if not self.enabled:
@@ -84,12 +66,12 @@ class Cache(object):
         try:
             try:
                 with io.open(cache_fn, 'r', encoding='utf-8') as cachef:
-                    return self._validate(json.load(cachef), min_ver)
+                    return json.load(cachef)
             except ValueError:
                 try:
                     file_size = os.path.getsize(cache_fn)
                 except (OSError, IOError) as oe:
-                    file_size = error_to_compat_str(oe)
+                    file_size = str(oe)
                 self._ydl.report_warning(
                     'Cache retrieval from %s failed (%s)' % (cache_fn, file_size))
         except IOError:

youtube_dl/compat.py

@@ -1,7 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import base64
 import binascii
 import collections
 import ctypes
@@ -21,23 +20,6 @@ import subprocess
 import sys
 import xml.etree.ElementTree
 
-# deal with critical unicode/str things first
-try:
-    # Python 2
-    compat_str, compat_basestring, compat_chr = (
-        unicode, basestring, unichr
-    )
-    from .casefold import casefold as compat_casefold
-except NameError:
-    compat_str, compat_basestring, compat_chr = (
-        str, str, chr
-    )
-    compat_casefold = lambda s: s.casefold()
-
-try:
-    import collections.abc as compat_collections_abc
-except ImportError:
-    import collections as compat_collections_abc
 
 try:
     import urllib.request as compat_urllib_request
@@ -74,31 +56,11 @@ try:
 except ImportError:  # Python 2
     import cookielib as compat_cookiejar
 
-if sys.version_info[0] == 2:
-    class compat_cookiejar_Cookie(compat_cookiejar.Cookie):
-        def __init__(self, version, name, value, *args, **kwargs):
-            if isinstance(name, compat_str):
-                name = name.encode()
-            if isinstance(value, compat_str):
-                value = value.encode()
-            compat_cookiejar.Cookie.__init__(self, version, name, value, *args, **kwargs)
-else:
-    compat_cookiejar_Cookie = compat_cookiejar.Cookie
-
 try:
     import http.cookies as compat_cookies
 except ImportError:  # Python 2
     import Cookie as compat_cookies
 
-if sys.version_info[0] == 2:
-    class compat_cookies_SimpleCookie(compat_cookies.SimpleCookie):
-        def load(self, rawdata):
-            if isinstance(rawdata, compat_str):
-                rawdata = str(rawdata)
-            return super(compat_cookies_SimpleCookie, self).load(rawdata)
-else:
-    compat_cookies_SimpleCookie = compat_cookies.SimpleCookie
-
 try:
     import html.entities as compat_html_entities
 except ImportError:  # Python 2
@@ -2371,7 +2333,7 @@ except ImportError:  # Python <3.4
 
         # HTMLParseError has been deprecated in Python 3.3 and removed in
         # Python 3.5. Introducing dummy exception for Python >3.5 for compatible
-        # and uniform cross-version exception handling
+        # and uniform cross-version exceptiong handling
         class compat_HTMLParseError(Exception):
             pass
 
@@ -2386,6 +2348,11 @@ try:
 except ImportError:
     import BaseHTTPServer as compat_http_server
 
+try:
+    compat_str = unicode  # Python 2
+except NameError:
+    compat_str = str
+
 try:
     from urllib.parse import unquote_to_bytes as compat_urllib_parse_unquote_to_bytes
     from urllib.parse import unquote as compat_urllib_parse_unquote
@@ -2396,7 +2363,7 @@ except ImportError:  # Python 2
 
     # HACK: The following are the correct unquote_to_bytes, unquote and unquote_plus
     # implementations from cpython 3.4.3's stdlib. Python 2's version
-    # is apparently broken (see https://github.com/ytdl-org/youtube-dl/pull/6244)
+    # is apparently broken (see https://github.com/rg3/youtube-dl/pull/6244)
 
     def compat_urllib_parse_unquote_to_bytes(string):
         """unquote_to_bytes('abc%20def') -> b'abc def'."""
@@ -2516,11 +2483,22 @@ except ImportError:  # Python < 3.4
 
             return compat_urllib_response.addinfourl(io.BytesIO(data), headers, url)
 
+try:
+    compat_basestring = basestring  # Python 2
+except NameError:
+    compat_basestring = str
+
+try:
+    compat_chr = unichr  # Python 2
+except NameError:
+    compat_chr = chr
+
 try:
     from xml.etree.ElementTree import ParseError as compat_xml_parse_error
 except ImportError:  # Python 2.6
     from xml.parsers.expat import ExpatError as compat_xml_parse_error
 
+
 etree = xml.etree.ElementTree
 
 
@@ -2529,15 +2507,6 @@ class _TreeBuilder(etree.TreeBuilder):
         pass
 
 
-try:
-    # xml.etree.ElementTree.Element is a method in Python <=2.6 and
-    # the following will crash with:
-    #  TypeError: isinstance() arg 2 must be a class, type, or tuple of classes and types
-    isinstance(None, xml.etree.ElementTree.Element)
-    from xml.etree.ElementTree import Element as compat_etree_Element
-except TypeError:  # Python <=2.6
-    from xml.etree.ElementTree import _ElementInterface as compat_etree_Element
-
 if sys.version_info[0] >= 3:
     def compat_etree_fromstring(text):
         return etree.XML(text, parser=etree.XMLParser(target=_TreeBuilder()))
@@ -2670,9 +2639,9 @@ else:
 
 try:
     args = shlex.split('中文')
-    assert (isinstance(args, list)
-            and isinstance(args[0], compat_str)
-            and args[0] == '中文')
+    assert (isinstance(args, list) and
+            isinstance(args[0], compat_str) and
+            args[0] == '中文')
     compat_shlex_split = shlex.split
 except (AssertionError, UnicodeEncodeError):
     # Working around shlex issue with unicode strings on some python 2
@@ -2775,17 +2744,6 @@ else:
         compat_expanduser = os.path.expanduser
 
 
-if compat_os_name == 'nt' and sys.version_info < (3, 8):
-    # os.path.realpath on Windows does not follow symbolic links
-    # prior to Python 3.8 (see https://bugs.python.org/issue9949)
-    def compat_realpath(path):
-        while os.path.islink(path):
-            path = os.path.abspath(os.readlink(path))
-        return path
-else:
-    compat_realpath = os.path.realpath
-
-
 if sys.version_info < (3, 0):
     def compat_print(s):
         from .utils import preferredencoding
@@ -2828,12 +2786,6 @@ except NameError:  # Python 3
     compat_numeric_types = (int, float, complex)
 
 
-try:
-    compat_integer_types = (int, long)
-except NameError:  # Python 3
-    compat_integer_types = (int, )
-
-
 if sys.version_info < (2, 7):
     def compat_socket_create_connection(address, timeout, source_address=None):
         host, port = address
@@ -2860,7 +2812,7 @@ else:
     compat_socket_create_connection = socket.create_connection
 
 
-# Fix https://github.com/ytdl-org/youtube-dl/issues/4223
+# Fix https://github.com/rg3/youtube-dl/issues/4223
 # See http://bugs.python.org/issue9161 for what is broken
 def workaround_optparse_bug9161():
     op = optparse.OptionParser()
@@ -2887,7 +2839,6 @@ else:
     _terminal_size = collections.namedtuple('terminal_size', ['columns', 'lines'])
 
     def compat_get_terminal_size(fallback=(80, 24)):
-        from .utils import process_communicate_or_kill
         columns = compat_getenv('COLUMNS')
         if columns:
             columns = int(columns)
@@ -2904,7 +2855,7 @@ else:
                 sp = subprocess.Popen(
                     ['stty', 'size'],
                     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-                out, err = process_communicate_or_kill(sp)
+                out, err = sp.communicate()
                 _lines, _columns = map(int, out.split())
             except Exception:
                 _columns, _lines = _terminal_size(*fallback)
@@ -2945,42 +2896,9 @@ except TypeError:
         if isinstance(spec, compat_str):
             spec = spec.encode('ascii')
         return struct.unpack(spec, *args)
-
-    class compat_Struct(struct.Struct):
-        def __init__(self, fmt):
-            if isinstance(fmt, compat_str):
-                fmt = fmt.encode('ascii')
-            super(compat_Struct, self).__init__(fmt)
 else:
     compat_struct_pack = struct.pack
     compat_struct_unpack = struct.unpack
-    if platform.python_implementation() == 'IronPython' and sys.version_info < (2, 7, 8):
-        class compat_Struct(struct.Struct):
-            def unpack(self, string):
-                if not isinstance(string, buffer):  # noqa: F821
-                    string = buffer(string)  # noqa: F821
-                return super(compat_Struct, self).unpack(string)
-    else:
-        compat_Struct = struct.Struct
-
-
-# compat_map/filter() returning an iterator, supposedly the
-# same versioning as for zip below
-try:
-    from future_builtins import map as compat_map
-except ImportError:
-    try:
-        from itertools import imap as compat_map
-    except ImportError:
-        compat_map = map
-
-try:
-    from future_builtins import filter as compat_filter
-except ImportError:
-    try:
-        from itertools import ifilter as compat_filter
-    except ImportError:
-        compat_filter = filter
 
 try:
     from future_builtins import zip as compat_zip
@@ -2990,97 +2908,11 @@ except ImportError:  # not 2.6+ or is 3.x
     except ImportError:
         compat_zip = zip
 
-
-# method renamed between Py2/3
-try:
-    from itertools import zip_longest as compat_itertools_zip_longest
-except ImportError:
-    from itertools import izip_longest as compat_itertools_zip_longest
-
-
-# new class in collections
-try:
-    from collections import ChainMap as compat_collections_chain_map
-    # Py3.3's ChainMap is deficient
-    if sys.version_info < (3, 4):
-        raise ImportError
-except ImportError:
-    # Py <= 3.3
-    class compat_collections_chain_map(compat_collections_abc.MutableMapping):
-
-        maps = [{}]
-
-        def __init__(self, *maps):
-            self.maps = list(maps) or [{}]
-
-        def __getitem__(self, k):
-            for m in self.maps:
-                if k in m:
-                    return m[k]
-            raise KeyError(k)
-
-        def __setitem__(self, k, v):
-            self.maps[0].__setitem__(k, v)
-            return
-
-        def __contains__(self, k):
-            return any((k in m) for m in self.maps)
-
-        def __delitem(self, k):
-            if k in self.maps[0]:
-                del self.maps[0][k]
-                return
-            raise KeyError(k)
-
-        def __delitem__(self, k):
-            self.__delitem(k)
-
-        def __iter__(self):
-            return itertools.chain(*reversed(self.maps))
-
-        def __len__(self):
-            return len(iter(self))
-
-        # to match Py3, don't del directly
-        def pop(self, k, *args):
-            if self.__contains__(k):
-                off = self.__getitem__(k)
-                self.__delitem(k)
-                return off
-            elif len(args) > 0:
-                return args[0]
-            raise KeyError(k)
-
-        def new_child(self, m=None, **kwargs):
-            m = m or {}
-            m.update(kwargs)
-            return compat_collections_chain_map(m, *self.maps)
-
-        @property
-        def parents(self):
-            return compat_collections_chain_map(*(self.maps[1:]))
-
-
-# Pythons disagree on the type of a pattern (RegexObject, _sre.SRE_Pattern, Pattern, ...?)
-compat_re_Pattern = type(re.compile(''))
-# and on the type of a match
-compat_re_Match = type(re.match('a', 'a'))
-
-
-if sys.version_info < (3, 3):
-    def compat_b64decode(s, *args, **kwargs):
-        if isinstance(s, compat_str):
-            s = s.encode('ascii')
-        return base64.b64decode(s, *args, **kwargs)
-else:
-    compat_b64decode = base64.b64decode
-
-
 if platform.python_implementation() == 'PyPy' and sys.pypy_version_info < (5, 4, 0):
     # PyPy2 prior to version 5.4.0 expects byte strings as Windows function
     # names, see the original PyPy issue [1] and the youtube-dl one [2].
     # 1. https://bitbucket.org/pypy/pypy/issues/2360/windows-ctypescdll-typeerror-function-name
-    # 2. https://github.com/ytdl-org/youtube-dl/pull/4392
+    # 2. https://github.com/rg3/youtube-dl/pull/4392
     def compat_ctypes_WINFUNCTYPE(*args, **kwargs):
         real = ctypes.WINFUNCTYPE(*args, **kwargs)
 
@@ -3098,23 +2930,14 @@ __all__ = [
     'compat_HTMLParseError',
     'compat_HTMLParser',
     'compat_HTTPError',
-    'compat_Struct',
-    'compat_b64decode',
     'compat_basestring',
-    'compat_casefold',
     'compat_chr',
-    'compat_collections_abc',
-    'compat_collections_chain_map',
     'compat_cookiejar',
-    'compat_cookiejar_Cookie',
     'compat_cookies',
-    'compat_cookies_SimpleCookie',
     'compat_ctypes_WINFUNCTYPE',
-    'compat_etree_Element',
     'compat_etree_fromstring',
     'compat_etree_register_namespace',
     'compat_expanduser',
-    'compat_filter',
     'compat_get_terminal_size',
     'compat_getenv',
     'compat_getpass',
@@ -3123,19 +2946,13 @@ __all__ = [
     'compat_http_client',
     'compat_http_server',
     'compat_input',
-    'compat_integer_types',
     'compat_itertools_count',
-    'compat_itertools_zip_longest',
     'compat_kwargs',
-    'compat_map',
     'compat_numeric_types',
     'compat_ord',
     'compat_os_name',
     'compat_parse_qs',
     'compat_print',
-    'compat_re_Match',
-    'compat_re_Pattern',
-    'compat_realpath',
     'compat_setenv',
     'compat_shlex_quote',
     'compat_shlex_split',

youtube_dl/downloader/__init__.py

@@ -1,31 +1,22 @@
 from __future__ import unicode_literals
 
-from ..utils import (
-    determine_protocol,
-)
-
-
-def get_suitable_downloader(info_dict, params={}):
-    info_dict['protocol'] = determine_protocol(info_dict)
-    info_copy = info_dict.copy()
-    return _get_suitable_downloader(info_copy, params)
-
-
-# Some of these require get_suitable_downloader
 from .common import FileDownloader
-from .dash import DashSegmentsFD
 from .f4m import F4mFD
 from .hls import HlsFD
 from .http import HttpFD
 from .rtmp import RtmpFD
+from .dash import DashSegmentsFD
 from .rtsp import RtspFD
 from .ism import IsmFD
-from .niconico import NiconicoDmcFD
 from .external import (
     get_external_downloader,
     FFmpegFD,
 )
 
+from ..utils import (
+    determine_protocol,
+)
+
 PROTOCOL_MAP = {
     'rtmp': RtmpFD,
     'm3u8_native': HlsFD,
@@ -35,12 +26,13 @@ PROTOCOL_MAP = {
     'f4m': F4mFD,
     'http_dash_segments': DashSegmentsFD,
     'ism': IsmFD,
-    'niconico_dmc': NiconicoDmcFD,
 }
 
 
-def _get_suitable_downloader(info_dict, params={}):
+def get_suitable_downloader(info_dict, params={}):
     """Get the downloader class that can handle the info dict."""
+    protocol = determine_protocol(info_dict)
+    info_dict['protocol'] = protocol
 
     # if (info_dict.get('start_time') or info_dict.get('end_time')) and not info_dict.get('requested_formats') and FFmpegFD.can_download(info_dict):
     #     return FFmpegFD
@@ -50,11 +42,7 @@ def _get_suitable_downloader(info_dict, params={}):
         ed = get_external_downloader(external_downloader)
         if ed.can_download(info_dict):
             return ed
-        # Avoid using unwanted args since external_downloader was rejected
-        if params.get('external_downloader_args'):
-            params['external_downloader_args'] = None
 
-    protocol = info_dict['protocol']
     if protocol.startswith('m3u8') and info_dict.get('is_live'):
         return FFmpegFD
 

youtube_dl/downloader/common.py

@@ -45,12 +45,10 @@ class FileDownloader(object):
     min_filesize:       Skip files smaller than this size
     max_filesize:       Skip files larger than this size
     xattr_set_filesize: Set ytdl.filesize user xattribute with expected size.
+                        (experimental)
     external_downloader_args:  A list of additional command-line arguments for the
                         external downloader.
     hls_use_mpegts:     Use the mpegts container for HLS videos.
-    http_chunk_size:    Size of a chunk for chunk-based HTTP downloading. May be
-                        useful for bypassing bandwidth throttling imposed by
-                        a webserver (experimental)
 
     Subclasses of this one must re-define the real_download method.
     """
@@ -176,9 +174,7 @@ class FileDownloader(object):
             return
         speed = float(byte_counter) / elapsed
         if speed > rate_limit:
-            sleep_time = float(byte_counter) / rate_limit - elapsed
-            if sleep_time > 0:
-                time.sleep(sleep_time)
+            time.sleep(max((byte_counter // rate_limit) - elapsed, 0))
 
     def temp_name(self, filename):
         """Returns a temporary filename for the given filename."""
@@ -250,13 +246,12 @@ class FileDownloader(object):
             if self.params.get('noprogress', False):
                 self.to_screen('[download] Download completed')
             else:
-                msg_template = '100%%'
-                if s.get('total_bytes') is not None:
-                    s['_total_bytes_str'] = format_bytes(s['total_bytes'])
-                    msg_template += ' of %(_total_bytes_str)s'
+                s['_total_bytes_str'] = format_bytes(s['total_bytes'])
                 if s.get('elapsed') is not None:
                     s['_elapsed_str'] = self.format_seconds(s['elapsed'])
-                    msg_template += ' in %(_elapsed_str)s'
+                    msg_template = '100%% of %(_total_bytes_str)s in %(_elapsed_str)s'
+                else:
+                    msg_template = '100%% of %(_total_bytes_str)s'
                 self._report_progress_status(
                     msg_template % s, is_last_line=True)
 
@@ -332,15 +327,15 @@ class FileDownloader(object):
         """
 
         nooverwrites_and_exists = (
-            self.params.get('nooverwrites', False)
-            and os.path.exists(encodeFilename(filename))
+            self.params.get('nooverwrites', False) and
+            os.path.exists(encodeFilename(filename))
         )
 
         if not hasattr(filename, 'write'):
             continuedl_and_exists = (
-                self.params.get('continuedl', True)
-                and os.path.isfile(encodeFilename(filename))
-                and not self.params.get('nopart', False)
+                self.params.get('continuedl', True) and
+                os.path.isfile(encodeFilename(filename)) and
+                not self.params.get('nopart', False)
             )
 
             # Check file already present

youtube_dl/downloader/dash.py

@@ -2,10 +2,7 @@ from __future__ import unicode_literals
 
 from .fragment import FragmentFD
 from ..compat import compat_urllib_error
-from ..utils import (
-    DownloadError,
-    urljoin,
-)
+from ..utils import urljoin
 
 
 class DashSegmentsFD(FragmentFD):
@@ -53,21 +50,13 @@ class DashSegmentsFD(FragmentFD):
                 except compat_urllib_error.HTTPError as err:
                     # YouTube may often return 404 HTTP error for a fragment causing the
                     # whole download to fail. However if the same fragment is immediately
-                    # retried with the same request data this usually succeeds (1-2 attempts
+                    # retried with the same request data this usually succeeds (1-2 attemps
                     # is usually enough) thus allowing to download the whole file successfully.
                     # To be future-proof we will retry all fragments that fail with any
                     # HTTP error.
                     count += 1
                     if count <= fragment_retries:
                         self.report_retry_fragment(err, frag_index, count, fragment_retries)
-                except DownloadError:
-                    # Don't retry fragment if error occurred during HTTP downloading
-                    # itself since it has own retry settings
-                    if not fatal:
-                        self.report_skip_fragment(frag_index)
-                        break
-                    raise
-
             if count > fragment_retries:
                 if not fatal:
                     self.report_skip_fragment(frag_index)

youtube_dl/downloader/external.py

@@ -1,10 +1,9 @@
 from __future__ import unicode_literals
 
 import os.path
-import re
 import subprocess
 import sys
-import time
+import re
 
 from .common import FileDownloader
 from ..compat import (
@@ -22,7 +21,6 @@ from ..utils import (
     handle_youtubedl_headers,
     check_executable,
     is_outdated_version,
-    process_communicate_or_kill,
 )
 
 
@@ -32,7 +30,6 @@ class ExternalFD(FileDownloader):
         tmpfilename = self.temp_name(filename)
 
         try:
-            started = time.time()
             retval = self._call_downloader(tmpfilename, info_dict)
         except KeyboardInterrupt:
             if not info_dict.get('is_live'):
@@ -44,20 +41,15 @@ class ExternalFD(FileDownloader):
             self.to_screen('[%s] Interrupted by user' % self.get_basename())
 
         if retval == 0:
-            status = {
+            fsize = os.path.getsize(encodeFilename(tmpfilename))
+            self.to_screen('\r[%s] Downloaded %s bytes' % (self.get_basename(), fsize))
+            self.try_rename(tmpfilename, filename)
+            self._hook_progress({
+                'downloaded_bytes': fsize,
+                'total_bytes': fsize,
                 'filename': filename,
                 'status': 'finished',
-                'elapsed': time.time() - started,
-            }
-            if filename != '-':
-                fsize = os.path.getsize(encodeFilename(tmpfilename))
-                self.to_screen('\r[%s] Downloaded %s bytes' % (self.get_basename(), fsize))
-                self.try_rename(tmpfilename, filename)
-                status.update({
-                    'downloaded_bytes': fsize,
-                    'total_bytes': fsize,
-                })
-            self._hook_progress(status)
+            })
             return True
         else:
             self.to_stderr('\n')
@@ -105,7 +97,7 @@ class ExternalFD(FileDownloader):
 
         p = subprocess.Popen(
             cmd, stderr=subprocess.PIPE)
-        _, stderr = process_communicate_or_kill(p)
+        _, stderr = p.communicate()
         if p.returncode != 0:
             self.to_stderr(stderr.decode('utf-8', 'replace'))
         return p.returncode
@@ -122,11 +114,7 @@ class CurlFD(ExternalFD):
         cmd += self._valueless_option('--silent', 'noprogress')
         cmd += self._valueless_option('--verbose', 'verbose')
         cmd += self._option('--limit-rate', 'ratelimit')
-        retry = self._option('--retry', 'retries')
-        if len(retry) == 2:
-            if retry[1] in ('inf', 'infinite'):
-                retry[1] = '2147483647'
-            cmd += retry
+        cmd += self._option('--retry', 'retries')
         cmd += self._option('--max-filesize', 'max_filesize')
         cmd += self._option('--interface', 'source_address')
         cmd += self._option('--proxy', 'proxy')
@@ -142,7 +130,7 @@ class CurlFD(ExternalFD):
 
         # curl writes the progress to stderr so don't capture it.
         p = subprocess.Popen(cmd)
-        process_communicate_or_kill(p)
+        p.communicate()
         return p.returncode
 
 
@@ -165,12 +153,6 @@ class WgetFD(ExternalFD):
         cmd = [self.exe, '-O', tmpfilename, '-nv', '--no-cookies']
         for key, val in info_dict['http_headers'].items():
             cmd += ['--header', '%s: %s' % (key, val)]
-        cmd += self._option('--limit-rate', 'ratelimit')
-        retry = self._option('--tries', 'retries')
-        if len(retry) == 2:
-            if retry[1] in ('inf', 'infinite'):
-                retry[1] = '0'
-            cmd += retry
         cmd += self._option('--bind-address', 'source_address')
         cmd += self._option('--proxy', 'proxy')
         cmd += self._valueless_option('--no-check-certificate', 'nocheckcertificate')
@@ -195,7 +177,6 @@ class Aria2cFD(ExternalFD):
         cmd += self._option('--interface', 'source_address')
         cmd += self._option('--all-proxy', 'proxy')
         cmd += self._bool_option('--check-certificate', 'nocheckcertificate', 'false', 'true', '=')
-        cmd += self._bool_option('--remote-time', 'updatetime', 'true', 'false', '=')
         cmd += ['--', info_dict['url']]
         return cmd
 
@@ -241,7 +222,7 @@ class FFmpegFD(ExternalFD):
             # setting -seekable prevents ffmpeg from guessing if the server
             # supports seeking(by adding the header `Range: bytes=0-`), which
             # can cause problems in some cases
-            # https://github.com/ytdl-org/youtube-dl/issues/11800#issuecomment-275037127
+            # https://github.com/rg3/youtube-dl/issues/11800#issuecomment-275037127
             # http://trac.ffmpeg.org/ticket/6125#comment:10
             args += ['-seekable', '1' if seekable else '0']
 
@@ -291,7 +272,6 @@ class FFmpegFD(ExternalFD):
             tc_url = info_dict.get('tc_url')
             flash_version = info_dict.get('flash_version')
             live = info_dict.get('rtmp_live', False)
-            conn = info_dict.get('rtmp_conn')
             if player_url is not None:
                 args += ['-rtmp_swfverify', player_url]
             if page_url is not None:
@@ -306,11 +286,6 @@ class FFmpegFD(ExternalFD):
                 args += ['-rtmp_flashver', flash_version]
             if live:
                 args += ['-rtmp_live', 'live']
-            if isinstance(conn, list):
-                for entry in conn:
-                    args += ['-rtmp_conn', entry]
-            elif isinstance(conn, compat_str):
-                args += ['-rtmp_conn', conn]
 
         args += ['-i', url, '-c', 'copy']
 
@@ -337,17 +312,14 @@ class FFmpegFD(ExternalFD):
         proc = subprocess.Popen(args, stdin=subprocess.PIPE, env=env)
         try:
             retval = proc.wait()
-        except BaseException as e:
-            # subprocess.run would send the SIGKILL signal to ffmpeg and the
+        except KeyboardInterrupt:
+            # subprocces.run would send the SIGKILL signal to ffmpeg and the
             # mp4 file couldn't be played, but if we ask ffmpeg to quit it
             # produces a file that is playable (this is mostly useful for live
             # streams). Note that Windows is not affected and produces playable
-            # files (see https://github.com/ytdl-org/youtube-dl/issues/8300).
-            if isinstance(e, KeyboardInterrupt) and sys.platform != 'win32':
-                process_communicate_or_kill(proc, b'q')
-            else:
-                proc.kill()
-                proc.wait()
+            # files (see https://github.com/rg3/youtube-dl/issues/8300).
+            if sys.platform != 'win32':
+                proc.communicate(b'q')
             raise
         return retval
 

youtube_dl/downloader/f4m.py

@@ -1,12 +1,12 @@
 from __future__ import division, unicode_literals
 
+import base64
 import io
 import itertools
 import time
 
 from .fragment import FragmentFD
 from ..compat import (
-    compat_b64decode,
     compat_etree_fromstring,
     compat_urlparse,
     compat_urllib_error,
@@ -238,8 +238,8 @@ def write_metadata_tag(stream, metadata):
 
 
 def remove_encrypted_media(media):
-    return list(filter(lambda e: 'drmAdditionalHeaderId' not in e.attrib
-                                 and 'drmAdditionalHeaderSetId' not in e.attrib,
+    return list(filter(lambda e: 'drmAdditionalHeaderId' not in e.attrib and
+                                 'drmAdditionalHeaderSetId' not in e.attrib,
                        media))
 
 
@@ -267,8 +267,8 @@ class F4mFD(FragmentFD):
         media = doc.findall(_add_ns('media'))
         if not media:
             self.report_error('No media found')
-        for e in (doc.findall(_add_ns('drmAdditionalHeader'))
-                  + doc.findall(_add_ns('drmAdditionalHeaderSet'))):
+        for e in (doc.findall(_add_ns('drmAdditionalHeader')) +
+                  doc.findall(_add_ns('drmAdditionalHeaderSet'))):
             # If id attribute is missing it's valid for all media nodes
             # without drmAdditionalHeaderId or drmAdditionalHeaderSetId attribute
             if 'id' not in e.attrib:
@@ -312,7 +312,7 @@ class F4mFD(FragmentFD):
             boot_info = self._get_bootstrap_from_url(bootstrap_url)
         else:
             bootstrap_url = None
-            bootstrap = compat_b64decode(node.text)
+            bootstrap = base64.b64decode(node.text.encode('ascii'))
             boot_info = read_bootstrap_info(bootstrap)
         return boot_info, bootstrap_url
 
@@ -324,8 +324,8 @@ class F4mFD(FragmentFD):
         urlh = self.ydl.urlopen(self._prepare_url(info_dict, man_url))
         man_url = urlh.geturl()
         # Some manifests may be malformed, e.g. prosiebensat1 generated manifests
-        # (see https://github.com/ytdl-org/youtube-dl/issues/6215#issuecomment-121704244
-        # and https://github.com/ytdl-org/youtube-dl/issues/7823)
+        # (see https://github.com/rg3/youtube-dl/issues/6215#issuecomment-121704244
+        # and https://github.com/rg3/youtube-dl/issues/7823)
         manifest = fix_xml_ampersands(urlh.read().decode('utf-8', 'ignore')).strip()
 
         doc = compat_etree_fromstring(manifest)
@@ -349,7 +349,7 @@ class F4mFD(FragmentFD):
         live = boot_info['live']
         metadata_node = media.find(_add_ns('metadata'))
         if metadata_node is not None:
-            metadata = compat_b64decode(metadata_node.text)
+            metadata = base64.b64decode(metadata_node.text.encode('ascii'))
         else:
             metadata = None
 
@@ -409,7 +409,7 @@ class F4mFD(FragmentFD):
                             # In tests, segments may be truncated, and thus
                             # FlvReader may not be able to parse the whole
                             # chunk. If so, write the segment as is
-                            # See https://github.com/ytdl-org/youtube-dl/issues/9214
+                            # See https://github.com/rg3/youtube-dl/issues/9214
                             dest_stream.write(down_data)
                             break
                         raise

youtube_dl/downloader/fragment.py

@@ -74,14 +74,9 @@ class FragmentFD(FileDownloader):
         return not ctx['live'] and not ctx['tmpfilename'] == '-'
 
     def _read_ytdl_file(self, ctx):
-        assert 'ytdl_corrupt' not in ctx
         stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'r')
-        try:
-            ctx['fragment_index'] = json.loads(stream.read())['downloader']['current_fragment']['index']
-        except Exception:
-            ctx['ytdl_corrupt'] = True
-        finally:
-            stream.close()
+        ctx['fragment_index'] = json.loads(stream.read())['downloader']['current_fragment']['index']
+        stream.close()
 
     def _write_ytdl_file(self, ctx):
         frag_index_stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'w')
@@ -97,15 +92,12 @@ class FragmentFD(FileDownloader):
 
     def _download_fragment(self, ctx, frag_url, info_dict, headers=None):
         fragment_filename = '%s-Frag%d' % (ctx['tmpfilename'], ctx['fragment_index'])
-        fragment_info_dict = {
+        success = ctx['dl'].download(fragment_filename, {
             'url': frag_url,
             'http_headers': headers or info_dict.get('http_headers'),
-        }
-        success = ctx['dl'].download(fragment_filename, fragment_info_dict)
+        })
         if not success:
             return False, None
-        if fragment_info_dict.get('filetime'):
-            ctx['fragment_filetime'] = fragment_info_dict.get('filetime')
         down, frag_sanitized = sanitize_open(fragment_filename, 'rb')
         ctx['fragment_filename_sanitized'] = frag_sanitized
         frag_content = down.read()
@@ -166,17 +158,11 @@ class FragmentFD(FileDownloader):
         if self.__do_ytdl_file(ctx):
             if os.path.isfile(encodeFilename(self.ytdl_filename(ctx['filename']))):
                 self._read_ytdl_file(ctx)
-                is_corrupt = ctx.get('ytdl_corrupt') is True
-                is_inconsistent = ctx['fragment_index'] > 0 and resume_len == 0
-                if is_corrupt or is_inconsistent:
-                    message = (
-                        '.ytdl file is corrupt' if is_corrupt else
-                        'Inconsistent state of incomplete fragment download')
+                if ctx['fragment_index'] > 0 and resume_len == 0:
                     self.report_warning(
-                        '%s. Restarting from the beginning...' % message)
+                        'Inconsistent state of incomplete fragment download. '
+                        'Restarting from the beginning...')
                     ctx['fragment_index'] = resume_len = 0
-                    if 'ytdl_corrupt' in ctx:
-                        del ctx['ytdl_corrupt']
                     self._write_ytdl_file(ctx)
             else:
                 self._write_ytdl_file(ctx)
@@ -193,13 +179,12 @@ class FragmentFD(FileDownloader):
         })
 
     def _start_frag_download(self, ctx):
-        resume_len = ctx['complete_frags_downloaded_bytes']
         total_frags = ctx['total_frags']
         # This dict stores the download progress, it's updated by the progress
         # hook
         state = {
             'status': 'downloading',
-            'downloaded_bytes': resume_len,
+            'downloaded_bytes': ctx['complete_frags_downloaded_bytes'],
             'fragment_index': ctx['fragment_index'],
             'fragment_count': total_frags,
             'filename': ctx['filename'],
@@ -223,8 +208,8 @@ class FragmentFD(FileDownloader):
             frag_total_bytes = s.get('total_bytes') or 0
             if not ctx['live']:
                 estimated_size = (
-                    (ctx['complete_frags_downloaded_bytes'] + frag_total_bytes)
-                    / (state['fragment_index'] + 1) * total_frags)
+                    (ctx['complete_frags_downloaded_bytes'] + frag_total_bytes) /
+                    (state['fragment_index'] + 1) * total_frags)
                 state['total_bytes_estimate'] = estimated_size
 
             if s['status'] == 'finished':
@@ -238,8 +223,8 @@ class FragmentFD(FileDownloader):
                 state['downloaded_bytes'] += frag_downloaded_bytes - ctx['prev_frag_downloaded_bytes']
                 if not ctx['live']:
                     state['eta'] = self.calc_eta(
-                        start, time_now, estimated_size - resume_len,
-                        state['downloaded_bytes'] - resume_len)
+                        start, time_now, estimated_size,
+                        state['downloaded_bytes'])
                 state['speed'] = s.get('speed') or ctx.get('speed')
                 ctx['speed'] = state['speed']
                 ctx['prev_frag_downloaded_bytes'] = frag_downloaded_bytes
@@ -256,23 +241,12 @@ class FragmentFD(FileDownloader):
             if os.path.isfile(ytdl_filename):
                 os.remove(ytdl_filename)
         elapsed = time.time() - ctx['started']
-
-        if ctx['tmpfilename'] == '-':
-            downloaded_bytes = ctx['complete_frags_downloaded_bytes']
-        else:
-            self.try_rename(ctx['tmpfilename'], ctx['filename'])
-            if self.params.get('updatetime', True):
-                filetime = ctx.get('fragment_filetime')
-                if filetime:
-                    try:
-                        os.utime(ctx['filename'], (time.time(), filetime))
-                    except Exception:
-                        pass
-            downloaded_bytes = os.path.getsize(encodeFilename(ctx['filename']))
+        self.try_rename(ctx['tmpfilename'], ctx['filename'])
+        fsize = os.path.getsize(encodeFilename(ctx['filename']))
 
         self._hook_progress({
-            'downloaded_bytes': downloaded_bytes,
-            'total_bytes': downloaded_bytes,
+            'downloaded_bytes': fsize,
+            'total_bytes': fsize,
             'filename': ctx['filename'],
             'status': 'finished',
             'elapsed': elapsed,

youtube_dl/downloader/hls.py

@@ -42,13 +42,11 @@ class HlsFD(FragmentFD):
             # no segments will definitely be appended to the end of the playlist.
             # r'#EXT-X-PLAYLIST-TYPE:EVENT',  # media segments may be appended to the end of
             #                                 # event media playlists [4]
-            r'#EXT-X-MAP:',  # media initialization [5]
 
             # 1. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.4
             # 2. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.2
             # 3. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.2
             # 4. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.3.5
-            # 5. https://tools.ietf.org/html/draft-pantos-http-live-streaming-17#section-4.3.2.5
         )
         check_results = [not re.search(feature, manifest) for feature in UNSUPPORTED_FEATURES]
         is_aes128_enc = '#EXT-X-KEY:METHOD=AES-128' in manifest
@@ -66,7 +64,7 @@ class HlsFD(FragmentFD):
         s = urlh.read().decode('utf-8', 'ignore')
 
         if not self.can_download(s, info_dict):
-            if info_dict.get('extra_param_to_segment_url') or info_dict.get('_decryption_key_url'):
+            if info_dict.get('extra_param_to_segment_url'):
                 self.report_error('pycrypto not found. Please install it.')
                 return False
             self.report_warning(
@@ -77,13 +75,8 @@ class HlsFD(FragmentFD):
                 fd.add_progress_hook(ph)
             return fd.real_download(filename, info_dict)
 
-        def is_ad_fragment_start(s):
-            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s
-                    or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',ad'))
-
-        def is_ad_fragment_end(s):
-            return (s.startswith('#ANVATO-SEGMENT-INFO') and 'type=master' in s
-                    or s.startswith('#UPLYNK-SEGMENT') and s.endswith(',segment'))
+        def anvato_ad(s):
+            return s.startswith('#ANVATO-SEGMENT-INFO') and 'type=ad' in s
 
         media_frags = 0
         ad_frags = 0
@@ -93,13 +86,12 @@ class HlsFD(FragmentFD):
             if not line:
                 continue
             if line.startswith('#'):
-                if is_ad_fragment_start(line):
+                if anvato_ad(line):
+                    ad_frags += 1
                     ad_frag_next = True
-                elif is_ad_fragment_end(line):
-                    ad_frag_next = False
                 continue
             if ad_frag_next:
-                ad_frags += 1
+                ad_frag_next = False
                 continue
             media_frags += 1
 
@@ -130,6 +122,7 @@ class HlsFD(FragmentFD):
             if line:
                 if not line.startswith('#'):
                     if ad_frag_next:
+                        ad_frag_next = False
                         continue
                     frag_index += 1
                     if frag_index <= ctx['fragment_index']:
@@ -143,7 +136,7 @@ class HlsFD(FragmentFD):
                     count = 0
                     headers = info_dict.get('http_headers', {})
                     if byte_range:
-                        headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'] - 1)
+                        headers['Range'] = 'bytes=%d-%d' % (byte_range['start'], byte_range['end'])
                     while count <= fragment_retries:
                         try:
                             success, frag_content = self._download_fragment(
@@ -154,8 +147,8 @@ class HlsFD(FragmentFD):
                         except compat_urllib_error.HTTPError as err:
                             # Unavailable (possibly temporary) fragments may be served.
                             # First we try to retry then either skip or abort.
-                            # See https://github.com/ytdl-org/youtube-dl/issues/10165,
-                            # https://github.com/ytdl-org/youtube-dl/issues/10448).
+                            # See https://github.com/rg3/youtube-dl/issues/10165,
+                            # https://github.com/rg3/youtube-dl/issues/10448).
                             count += 1
                             if count <= fragment_retries:
                                 self.report_retry_fragment(err, frag_index, count, fragment_retries)
@@ -171,13 +164,9 @@ class HlsFD(FragmentFD):
                     if decrypt_info['METHOD'] == 'AES-128':
                         iv = decrypt_info.get('IV') or compat_struct_pack('>8xq', media_sequence)
                         decrypt_info['KEY'] = decrypt_info.get('KEY') or self.ydl.urlopen(
-                            self._prepare_url(info_dict, info_dict.get('_decryption_key_url') or decrypt_info['URI'])).read()
-                        # Don't decrypt the content in tests since the data is explicitly truncated and it's not to a valid block
-                        # size (see https://github.com/ytdl-org/youtube-dl/pull/27660). Tests only care that the correct data downloaded,
-                        # not what it decrypts to.
-                        if not test:
-                            frag_content = AES.new(
-                                decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
+                            self._prepare_url(info_dict, decrypt_info['URI'])).read()
+                        frag_content = AES.new(
+                            decrypt_info['KEY'], AES.MODE_CBC, iv).decrypt(frag_content)
                     self._append_fragment(ctx, frag_content)
                     # We only download the first fragment during the test
                     if test:
@@ -206,10 +195,8 @@ class HlsFD(FragmentFD):
                         'start': sub_range_start,
                         'end': sub_range_start + int(splitted_byte_range[0]),
                     }
-                elif is_ad_fragment_start(line):
+                elif anvato_ad(line):
                     ad_frag_next = True
-                elif is_ad_fragment_end(line):
-                    ad_frag_next = False
 
         self._finish_frag_download(ctx)
 

youtube_dl/downloader/http.py

@@ -4,18 +4,13 @@ import errno
 import os
 import socket
 import time
-import random
 import re
 
 from .common import FileDownloader
-from ..compat import (
-    compat_str,
-    compat_urllib_error,
-)
+from ..compat import compat_urllib_error
 from ..utils import (
     ContentTooShortError,
     encodeFilename,
-    int_or_none,
     sanitize_open,
     sanitized_Request,
     write_xattr,
@@ -43,26 +38,21 @@ class HttpFD(FileDownloader):
         add_headers = info_dict.get('http_headers')
         if add_headers:
             headers.update(add_headers)
+        basic_request = sanitized_Request(url, None, headers)
+        request = sanitized_Request(url, None, headers)
 
         is_test = self.params.get('test', False)
-        chunk_size = self._TEST_FILE_SIZE if is_test else (
-            info_dict.get('downloader_options', {}).get('http_chunk_size')
-            or self.params.get('http_chunk_size') or 0)
+
+        if is_test:
+            request.add_header('Range', 'bytes=0-%s' % str(self._TEST_FILE_SIZE - 1))
 
         ctx.open_mode = 'wb'
         ctx.resume_len = 0
-        ctx.data_len = None
-        ctx.block_size = self.params.get('buffersize', 1024)
-        ctx.start_time = time.time()
-        ctx.chunk_size = None
 
         if self.params.get('continuedl', True):
             # Establish possible resume length
             if os.path.isfile(encodeFilename(ctx.tmpfilename)):
-                ctx.resume_len = os.path.getsize(
-                    encodeFilename(ctx.tmpfilename))
-
-        ctx.is_resume = ctx.resume_len > 0
+                ctx.resume_len = os.path.getsize(encodeFilename(ctx.tmpfilename))
 
         count = 0
         retries = self.params.get('retries', 0)
@@ -74,93 +64,50 @@ class HttpFD(FileDownloader):
             def __init__(self, source_error):
                 self.source_error = source_error
 
-        class NextFragment(Exception):
-            pass
-
-        def set_range(req, start, end):
-            range_header = 'bytes=%d-' % start
-            if end:
-                range_header += compat_str(end)
-            req.add_header('Range', range_header)
-
         def establish_connection():
-            ctx.chunk_size = (random.randint(int(chunk_size * 0.95), chunk_size)
-                              if not is_test and chunk_size else chunk_size)
-            if ctx.resume_len > 0:
-                range_start = ctx.resume_len
-                if ctx.is_resume:
-                    self.report_resuming_byte(ctx.resume_len)
+            if ctx.resume_len != 0:
+                self.report_resuming_byte(ctx.resume_len)
+                request.add_header('Range', 'bytes=%d-' % ctx.resume_len)
                 ctx.open_mode = 'ab'
-            elif ctx.chunk_size > 0:
-                range_start = 0
-            else:
-                range_start = None
-            ctx.is_resume = False
-            range_end = range_start + ctx.chunk_size - 1 if ctx.chunk_size else None
-            if range_end and ctx.data_len is not None and range_end >= ctx.data_len:
-                range_end = ctx.data_len - 1
-            has_range = range_start is not None
-            ctx.has_range = has_range
-            request = sanitized_Request(url, None, headers)
-            if has_range:
-                set_range(request, range_start, range_end)
             # Establish connection
             try:
-                try:
-                    ctx.data = self.ydl.urlopen(request)
-                except (compat_urllib_error.URLError, ) as err:
-                    # reason may not be available, e.g. for urllib2.HTTPError on python 2.6
-                    reason = getattr(err, 'reason', None)
-                    if isinstance(reason, socket.timeout):
-                        raise RetryDownload(err)
-                    raise err
+                ctx.data = self.ydl.urlopen(request)
                 # When trying to resume, Content-Range HTTP header of response has to be checked
                 # to match the value of requested Range HTTP header. This is due to a webservers
                 # that don't support resuming and serve a whole file with no Content-Range
                 # set in response despite of requested Range (see
-                # https://github.com/ytdl-org/youtube-dl/issues/6057#issuecomment-126129799)
-                if has_range:
+                # https://github.com/rg3/youtube-dl/issues/6057#issuecomment-126129799)
+                if ctx.resume_len > 0:
                     content_range = ctx.data.headers.get('Content-Range')
                     if content_range:
-                        content_range_m = re.search(r'bytes (\d+)-(\d+)?(?:/(\d+))?', content_range)
+                        content_range_m = re.search(r'bytes (\d+)-', content_range)
                         # Content-Range is present and matches requested Range, resume is possible
-                        if content_range_m:
-                            if range_start == int(content_range_m.group(1)):
-                                content_range_end = int_or_none(content_range_m.group(2))
-                                content_len = int_or_none(content_range_m.group(3))
-                                accept_content_len = (
-                                    # Non-chunked download
-                                    not ctx.chunk_size
-                                    # Chunked download and requested piece or
-                                    # its part is promised to be served
-                                    or content_range_end == range_end
-                                    or content_len < range_end)
-                                if accept_content_len:
-                                    ctx.data_len = content_len
-                                    return
+                        if content_range_m and ctx.resume_len == int(content_range_m.group(1)):
+                            return
                     # Content-Range is either not present or invalid. Assuming remote webserver is
                     # trying to send the whole file, resume is not possible, so wiping the local file
                     # and performing entire redownload
                     self.report_unable_to_resume()
                     ctx.resume_len = 0
                     ctx.open_mode = 'wb'
-                ctx.data_len = int_or_none(ctx.data.info().get('Content-length', None))
                 return
             except (compat_urllib_error.HTTPError, ) as err:
-                if err.code == 416:
+                if (err.code < 500 or err.code >= 600) and err.code != 416:
+                    # Unexpected HTTP error
+                    raise
+                elif err.code == 416:
                     # Unable to resume (requested range not satisfiable)
                     try:
                         # Open the connection again without the range header
-                        ctx.data = self.ydl.urlopen(
-                            sanitized_Request(url, None, headers))
+                        ctx.data = self.ydl.urlopen(basic_request)
                         content_length = ctx.data.info()['Content-Length']
                     except (compat_urllib_error.HTTPError, ) as err:
                         if err.code < 500 or err.code >= 600:
                             raise
                     else:
                         # Examine the reported length
-                        if (content_length is not None
-                                and (ctx.resume_len - 100 < int(content_length) < ctx.resume_len + 100)):
+                        if (content_length is not None and
+                                (ctx.resume_len - 100 < int(content_length) < ctx.resume_len + 100)):
                             # The file had already been fully downloaded.
                             # Explanation to the above condition: in issue #175 it was revealed that
                             # YouTube sometimes adds or removes a few bytes from the end of the file,
@@ -183,9 +130,6 @@ class HttpFD(FileDownloader):
                             ctx.resume_len = 0
                             ctx.open_mode = 'wb'
                             return
-                elif err.code < 500 or err.code >= 600:
-                    # Unexpected HTTP error
-                    raise
                 raise RetryDownload(err)
             except socket.error as err:
                 if err.errno != errno.ECONNRESET:
@@ -216,7 +160,7 @@ class HttpFD(FileDownloader):
                     return False
 
             byte_counter = 0 + ctx.resume_len
-            block_size = ctx.block_size
+            block_size = self.params.get('buffersize', 1024)
             start = time.time()
 
             # measure time over whole while-loop, so slow_down() and best_block_size() work together properly
@@ -224,28 +168,24 @@ class HttpFD(FileDownloader):
             before = start  # start measuring
 
             def retry(e):
-                to_stdout = ctx.tmpfilename == '-'
-                if ctx.stream is not None:
-                    if not to_stdout:
-                        ctx.stream.close()
-                    ctx.stream = None
-                ctx.resume_len = byte_counter if to_stdout else os.path.getsize(encodeFilename(ctx.tmpfilename))
+                if ctx.tmpfilename != '-':
+                    ctx.stream.close()
+                ctx.stream = None
+                ctx.resume_len = os.path.getsize(encodeFilename(ctx.tmpfilename))
                 raise RetryDownload(e)
 
             while True:
                 try:
                     # Download and write
-                    data_block = ctx.data.read(block_size if data_len is None else min(block_size, data_len - byte_counter))
+                    data_block = ctx.data.read(block_size if not is_test else min(block_size, data_len - byte_counter))
                 # socket.timeout is a subclass of socket.error but may not have
                 # errno set
                 except socket.timeout as e:
                     retry(e)
                 except socket.error as e:
-                    # SSLError on python 2 (inherits socket.error) may have
-                    # no errno set but this error message
-                    if e.errno in (errno.ECONNRESET, errno.ETIMEDOUT) or getattr(e, 'message', None) == 'The read operation timed out':
-                        retry(e)
-                    raise
+                    if e.errno not in (errno.ECONNRESET, errno.ETIMEDOUT):
+                        raise
+                    retry(e)
 
                 byte_counter += len(data_block)
 
@@ -293,30 +233,25 @@ class HttpFD(FileDownloader):
 
                 # Progress message
                 speed = self.calc_speed(start, now, byte_counter - ctx.resume_len)
-                if ctx.data_len is None:
+                if data_len is None:
                     eta = None
                 else:
-                    eta = self.calc_eta(start, time.time(), ctx.data_len - ctx.resume_len, byte_counter - ctx.resume_len)
+                    eta = self.calc_eta(start, time.time(), data_len - ctx.resume_len, byte_counter - ctx.resume_len)
 
                 self._hook_progress({
                     'status': 'downloading',
                     'downloaded_bytes': byte_counter,
-                    'total_bytes': ctx.data_len,
+                    'total_bytes': data_len,
                     'tmpfilename': ctx.tmpfilename,
                     'filename': ctx.filename,
                     'eta': eta,
                     'speed': speed,
-                    'elapsed': now - ctx.start_time,
+                    'elapsed': now - start,
                 })
 
-                if data_len is not None and byte_counter == data_len:
+                if is_test and byte_counter == data_len:
                     break
 
-            if not is_test and ctx.chunk_size and ctx.data_len is not None and byte_counter < ctx.data_len:
-                ctx.resume_len = byte_counter
-                # ctx.block_size = block_size
-                raise NextFragment()
-
             if ctx.stream is None:
                 self.to_stderr('\n')
                 self.report_error('Did not get any data blocks')
@@ -341,7 +276,7 @@ class HttpFD(FileDownloader):
                 'total_bytes': byte_counter,
                 'filename': ctx.filename,
                 'status': 'finished',
-                'elapsed': time.time() - ctx.start_time,
+                'elapsed': time.time() - start,
             })
 
             return True
@@ -355,8 +290,6 @@ class HttpFD(FileDownloader):
                 if count <= retries:
                     self.report_retry(e.source_error, count, retries)
                 continue
-            except NextFragment:
-                continue
             except SucceedDownload:
                 return True
 

youtube_dl/downloader/ism.py

@@ -1,27 +1,25 @@
 from __future__ import unicode_literals
 
 import time
+import struct
 import binascii
 import io
 
 from .fragment import FragmentFD
-from ..compat import (
-    compat_Struct,
-    compat_urllib_error,
-)
+from ..compat import compat_urllib_error
 
 
-u8 = compat_Struct('>B')
-u88 = compat_Struct('>Bx')
-u16 = compat_Struct('>H')
-u1616 = compat_Struct('>Hxx')
-u32 = compat_Struct('>I')
-u64 = compat_Struct('>Q')
+u8 = struct.Struct(b'>B')
+u88 = struct.Struct(b'>Bx')
+u16 = struct.Struct(b'>H')
+u1616 = struct.Struct(b'>Hxx')
+u32 = struct.Struct(b'>I')
+u64 = struct.Struct(b'>Q')
 
-s88 = compat_Struct('>bx')
-s16 = compat_Struct('>h')
-s1616 = compat_Struct('>hxx')
-s32 = compat_Struct('>i')
+s88 = struct.Struct(b'>bx')
+s16 = struct.Struct(b'>h')
+s1616 = struct.Struct(b'>hxx')
+s32 = struct.Struct(b'>i')
 
 unity_matrix = (s32.pack(0x10000) + s32.pack(0) * 3) * 2 + s32.pack(0x40000000)
 
@@ -141,12 +139,12 @@ def write_piff_header(stream, params):
         sample_entry_payload += u16.pack(0x18)  # depth
         sample_entry_payload += s16.pack(-1)  # pre defined
 
-        codec_private_data = binascii.unhexlify(params['codec_private_data'].encode('utf-8'))
+        codec_private_data = binascii.unhexlify(params['codec_private_data'])
         if fourcc in ('H264', 'AVC1'):
             sps, pps = codec_private_data.split(u32.pack(1))[1:]
             avcc_payload = u8.pack(1)  # configuration version
             avcc_payload += sps[1:4]  # avc profile indication + profile compatibility + avc level indication
-            avcc_payload += u8.pack(0xfc | (params.get('nal_unit_length_field', 4) - 1))  # complete representation (1) + reserved (11111) + length size minus one
+            avcc_payload += u8.pack(0xfc | (params.get('nal_unit_length_field', 4) - 1))  # complete represenation (1) + reserved (11111) + length size minus one
             avcc_payload += u8.pack(1)  # reserved (0) + number of sps (0000001)
             avcc_payload += u16.pack(len(sps))
             avcc_payload += sps

youtube_dl/downloader/niconico.py

@@ -1,66 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-try:
-    import threading
-except ImportError:
-    threading = None
-
-from .common import FileDownloader
-from ..downloader import get_suitable_downloader
-from ..extractor.niconico import NiconicoIE
-from ..utils import sanitized_Request
-
-
-class NiconicoDmcFD(FileDownloader):
-    """ Downloading niconico douga from DMC with heartbeat """
-
-    FD_NAME = 'niconico_dmc'
-
-    def real_download(self, filename, info_dict):
-        self.to_screen('[%s] Downloading from DMC' % self.FD_NAME)
-
-        ie = NiconicoIE(self.ydl)
-        info_dict, heartbeat_info_dict = ie._get_heartbeat_info(info_dict)
-
-        fd = get_suitable_downloader(info_dict, params=self.params)(self.ydl, self.params)
-        for ph in self._progress_hooks:
-            fd.add_progress_hook(ph)
-
-        if not threading:
-            self.to_screen('[%s] Threading for Heartbeat not available' % self.FD_NAME)
-            return fd.real_download(filename, info_dict)
-
-        success = download_complete = False
-        timer = [None]
-        heartbeat_lock = threading.Lock()
-        heartbeat_url = heartbeat_info_dict['url']
-        heartbeat_data = heartbeat_info_dict['data'].encode()
-        heartbeat_interval = heartbeat_info_dict.get('interval', 30)
-
-        request = sanitized_Request(heartbeat_url, heartbeat_data)
-
-        def heartbeat():
-            try:
-                self.ydl.urlopen(request).read()
-            except Exception:
-                self.to_screen('[%s] Heartbeat failed' % self.FD_NAME)
-
-            with heartbeat_lock:
-                if not download_complete:
-                    timer[0] = threading.Timer(heartbeat_interval, heartbeat)
-                    timer[0].start()
-
-        heartbeat_info_dict['ping']()
-        self.to_screen('[%s] Heartbeat with %d second interval ...' % (self.FD_NAME, heartbeat_interval))
-        try:
-            heartbeat()
-            if type(fd).__name__ == 'HlsFD':
-                info_dict.update(ie._extract_m3u8_formats(info_dict['url'], info_dict['id'])[0])
-            success = fd.real_download(filename, info_dict)
-        finally:
-            if heartbeat_lock:
-                with heartbeat_lock:
-                    timer[0].cancel()
-                    download_complete = True
-            return success

youtube_dl/downloader/rtmp.py

@@ -29,73 +29,69 @@ class RtmpFD(FileDownloader):
             proc = subprocess.Popen(args, stderr=subprocess.PIPE)
             cursor_in_new_line = True
             proc_stderr_closed = False
-            try:
-                while not proc_stderr_closed:
-                    # read line from stderr
-                    line = ''
-                    while True:
-                        char = proc.stderr.read(1)
-                        if not char:
-                            proc_stderr_closed = True
-                            break
-                        if char in [b'\r', b'\n']:
-                            break
-                        line += char.decode('ascii', 'replace')
-                    if not line:
-                        # proc_stderr_closed is True
-                        continue
-                    mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec \(([0-9]{1,2}\.[0-9])%\)', line)
+            while not proc_stderr_closed:
+                # read line from stderr
+                line = ''
+                while True:
+                    char = proc.stderr.read(1)
+                    if not char:
+                        proc_stderr_closed = True
+                        break
+                    if char in [b'\r', b'\n']:
+                        break
+                    line += char.decode('ascii', 'replace')
+                if not line:
+                    # proc_stderr_closed is True
+                    continue
+                mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec \(([0-9]{1,2}\.[0-9])%\)', line)
+                if mobj:
+                    downloaded_data_len = int(float(mobj.group(1)) * 1024)
+                    percent = float(mobj.group(2))
+                    if not resume_percent:
+                        resume_percent = percent
+                        resume_downloaded_data_len = downloaded_data_len
+                    time_now = time.time()
+                    eta = self.calc_eta(start, time_now, 100 - resume_percent, percent - resume_percent)
+                    speed = self.calc_speed(start, time_now, downloaded_data_len - resume_downloaded_data_len)
+                    data_len = None
+                    if percent > 0:
+                        data_len = int(downloaded_data_len * 100 / percent)
+                    self._hook_progress({
+                        'status': 'downloading',
+                        'downloaded_bytes': downloaded_data_len,
+                        'total_bytes_estimate': data_len,
+                        'tmpfilename': tmpfilename,
+                        'filename': filename,
+                        'eta': eta,
+                        'elapsed': time_now - start,
+                        'speed': speed,
+                    })
+                    cursor_in_new_line = False
+                else:
+                    # no percent for live streams
+                    mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec', line)
                     if mobj:
                         downloaded_data_len = int(float(mobj.group(1)) * 1024)
-                        percent = float(mobj.group(2))
-                        if not resume_percent:
-                            resume_percent = percent
-                            resume_downloaded_data_len = downloaded_data_len
                         time_now = time.time()
-                        eta = self.calc_eta(start, time_now, 100 - resume_percent, percent - resume_percent)
-                        speed = self.calc_speed(start, time_now, downloaded_data_len - resume_downloaded_data_len)
-                        data_len = None
-                        if percent > 0:
-                            data_len = int(downloaded_data_len * 100 / percent)
+                        speed = self.calc_speed(start, time_now, downloaded_data_len)
                         self._hook_progress({
-                            'status': 'downloading',
                             'downloaded_bytes': downloaded_data_len,
-                            'total_bytes_estimate': data_len,
                             'tmpfilename': tmpfilename,
                             'filename': filename,
-                            'eta': eta,
+                            'status': 'downloading',
                             'elapsed': time_now - start,
                             'speed': speed,
                         })
                         cursor_in_new_line = False
-                    else:
-                        # no percent for live streams
-                        mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec', line)
-                        if mobj:
-                            downloaded_data_len = int(float(mobj.group(1)) * 1024)
-                            time_now = time.time()
-                            speed = self.calc_speed(start, time_now, downloaded_data_len)
-                            self._hook_progress({
-                                'downloaded_bytes': downloaded_data_len,
-                                'tmpfilename': tmpfilename,
-                                'filename': filename,
-                                'status': 'downloading',
-                                'elapsed': time_now - start,
-                                'speed': speed,
-                            })
-                            cursor_in_new_line = False
-                        elif self.params.get('verbose', False):
-                            if not cursor_in_new_line:
-                                self.to_screen('')
-                            cursor_in_new_line = True
-                            self.to_screen('[rtmpdump] ' + line)
-                if not cursor_in_new_line:
-                    self.to_screen('')
-                return proc.wait()
-            except BaseException:  # Including KeyboardInterrupt
-                proc.kill()
-                proc.wait()
-                raise
+                    elif self.params.get('verbose', False):
+                        if not cursor_in_new_line:
+                            self.to_screen('')
+                        cursor_in_new_line = True
+                        self.to_screen('[rtmpdump] ' + line)
+            proc.wait()
+            if not cursor_in_new_line:
+                self.to_screen('')
+            return proc.returncode
 
         url = info_dict['url']
         player_url = info_dict.get('player_url')
@@ -167,15 +163,7 @@ class RtmpFD(FileDownloader):
         RD_INCOMPLETE = 2
         RD_NO_CONNECT = 3
 
-        started = time.time()
-
-        try:
-            retval = run_rtmpdump(args)
-        except KeyboardInterrupt:
-            if not info_dict.get('is_live'):
-                raise
-            retval = RD_SUCCESS
-            self.to_screen('\n[rtmpdump] Interrupted by user')
+        retval = run_rtmpdump(args)
 
         if retval == RD_NO_CONNECT:
             self.report_error('[rtmpdump] Could not connect to RTMP server.')
@@ -183,7 +171,7 @@ class RtmpFD(FileDownloader):
 
         while retval in (RD_INCOMPLETE, RD_FAILED) and not test and not live:
             prevsize = os.path.getsize(encodeFilename(tmpfilename))
-            self.to_screen('[rtmpdump] Downloaded %s bytes' % prevsize)
+            self.to_screen('[rtmpdump] %s bytes' % prevsize)
             time.sleep(5.0)  # This seems to be needed
             args = basic_args + ['--resume']
             if retval == RD_FAILED:
@@ -200,14 +188,13 @@ class RtmpFD(FileDownloader):
                 break
         if retval == RD_SUCCESS or (test and retval == RD_INCOMPLETE):
             fsize = os.path.getsize(encodeFilename(tmpfilename))
-            self.to_screen('[rtmpdump] Downloaded %s bytes' % fsize)
+            self.to_screen('[rtmpdump] %s bytes' % fsize)
             self.try_rename(tmpfilename, filename)
             self._hook_progress({
                 'downloaded_bytes': fsize,
                 'total_bytes': fsize,
                 'filename': filename,
                 'status': 'finished',
-                'elapsed': time.time() - started,
             })
             return True
         else:

youtube_dl/extractor/abc.py

@@ -13,7 +13,6 @@ from ..utils import (
     int_or_none,
     parse_iso8601,
     try_get,
-    unescapeHTML,
     update_url_query,
 )
 
@@ -105,22 +104,21 @@ class ABCIE(InfoExtractor):
 
 class ABCIViewIE(InfoExtractor):
     IE_NAME = 'abc.net.au:iview'
-    _VALID_URL = r'https?://iview\.abc\.net\.au/(?:[^/]+/)*video/(?P<id>[^/?#]+)'
+    _VALID_URL = r'https?://iview\.abc\.net\.au/programs/[^/]+/(?P<id>[^/?#]+)'
     _GEO_COUNTRIES = ['AU']
 
     # ABC iview programs are normally available for 14 days only.
     _TESTS = [{
-        'url': 'https://iview.abc.net.au/show/gruen/series/11/video/LE1927H001S00',
-        'md5': '67715ce3c78426b11ba167d875ac6abf',
+        'url': 'http://iview.abc.net.au/programs/call-the-midwife/ZW0898A003S00',
+        'md5': 'cde42d728b3b7c2b32b1b94b4a548afc',
         'info_dict': {
-            'id': 'LE1927H001S00',
+            'id': 'ZW0898A003S00',
             'ext': 'mp4',
-            'title': "Series 11 Ep 1",
-            'series': "Gruen",
-            'description': 'md5:52cc744ad35045baf6aded2ce7287f67',
-            'upload_date': '20190925',
+            'title': 'Series 5 Ep 3',
+            'description': 'md5:e0ef7d4f92055b86c4f33611f180ed79',
+            'upload_date': '20171228',
             'uploader_id': 'abc1',
-            'timestamp': 1569445289,
+            'timestamp': 1514499187,
         },
         'params': {
             'skip_download': True,
@@ -129,16 +127,17 @@ class ABCIViewIE(InfoExtractor):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_params = self._download_json(
-            'https://iview.abc.net.au/api/programs/' + video_id, video_id)
-        title = unescapeHTML(video_params.get('title') or video_params['seriesTitle'])
-        stream = next(s for s in video_params['playlist'] if s.get('type') in ('program', 'livestream'))
+        webpage = self._download_webpage(url, video_id)
+        video_params = self._parse_json(self._search_regex(
+            r'videoParams\s*=\s*({.+?});', webpage, 'video params'), video_id)
+        title = video_params.get('title') or video_params['seriesTitle']
+        stream = next(s for s in video_params['playlist'] if s.get('type') == 'program')
 
-        house_number = video_params.get('episodeHouseNumber') or video_id
-        path = '/auth/hls/sign?ts={0}&hn={1}&d=android-tablet'.format(
+        house_number = video_params.get('episodeHouseNumber')
+        path = '/auth/hls/sign?ts={0}&hn={1}&d=android-mobile'.format(
             int(time.time()), house_number)
         sig = hmac.new(
-            b'android.content.res.Resources',
+            'android.content.res.Resources'.encode('utf-8'),
             path.encode('utf-8'), hashlib.sha256).hexdigest()
         token = self._download_webpage(
             'http://iview.abc.net.au{0}&sig={1}'.format(path, sig), video_id)
@@ -148,7 +147,7 @@ class ABCIViewIE(InfoExtractor):
                 'hdnea': token,
             })
 
-        for sd in ('720', 'sd', 'sd-low'):
+        for sd in ('sd', 'sd-low'):
             sd_url = try_get(
                 stream, lambda x: x['streams']['hls'][sd], compat_str)
             if not sd_url:
@@ -168,26 +167,18 @@ class ABCIViewIE(InfoExtractor):
                 'ext': 'vtt',
             }]
 
-        is_live = video_params.get('livestream') == '1'
-        if is_live:
-            title = self._live_title(title)
-
         return {
             'id': video_id,
             'title': title,
-            'description': video_params.get('description'),
-            'thumbnail': video_params.get('thumbnail'),
+            'description': self._html_search_meta(['og:description', 'twitter:description'], webpage),
+            'thumbnail': self._html_search_meta(['og:image', 'twitter:image:src'], webpage),
             'duration': int_or_none(video_params.get('eventDuration')),
             'timestamp': parse_iso8601(video_params.get('pubDate'), ' '),
-            'series': unescapeHTML(video_params.get('seriesTitle')),
+            'series': video_params.get('seriesTitle'),
             'series_id': video_params.get('seriesHouseNumber') or video_id[:7],
-            'season_number': int_or_none(self._search_regex(
-                r'\bSeries\s+(\d+)\b', title, 'season number', default=None)),
-            'episode_number': int_or_none(self._search_regex(
-                r'\bEp\s+(\d+)\b', title, 'episode number', default=None)),
-            'episode_id': house_number,
+            'episode_number': int_or_none(self._html_search_meta('episodeNumber', webpage, default=None)),
+            'episode': self._html_search_meta('episode_title', webpage, default=None),
             'uploader_id': video_params.get('channel'),
             'formats': formats,
             'subtitles': subtitles,
-            'is_live': is_live,
         }

youtube_dl/extractor/abcnews.py

@@ -1,28 +1,24 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
+import calendar
 import re
+import time
 
 from .amp import AMPIE
 from .common import InfoExtractor
-from ..utils import (
-    parse_duration,
-    parse_iso8601,
-    try_get,
-)
+from .youtube import YoutubeIE
+from ..compat import compat_urlparse
 
 
 class AbcNewsVideoIE(AMPIE):
     IE_NAME = 'abcnews:video'
     _VALID_URL = r'''(?x)
                     https?://
+                        abcnews\.go\.com/
                         (?:
-                            abcnews\.go\.com/
-                            (?:
-                                (?:[^/]+/)*video/(?P<display_id>[0-9a-z-]+)-|
-                                video/(?:embed|itemfeed)\?.*?\bid=
-                            )|
-                            fivethirtyeight\.abcnews\.go\.com/video/embed/\d+/
+                            [^/]+/video/(?P<display_id>[0-9a-z-]+)-|
+                            video/embed\?.*?\bid=
                         )
                         (?P<id>\d+)
                     '''
@@ -37,8 +33,6 @@ class AbcNewsVideoIE(AMPIE):
             'description': 'George Stephanopoulos goes one-on-one with Iranian Foreign Minister Dr. Javad Zarif.',
             'duration': 180,
             'thumbnail': r're:^https?://.*\.jpg$',
-            'timestamp': 1380454200,
-            'upload_date': '20130929',
         },
         'params': {
             # m3u8 download
@@ -50,12 +44,6 @@ class AbcNewsVideoIE(AMPIE):
     }, {
         'url': 'http://abcnews.go.com/2020/video/2020-husband-stands-teacher-jail-student-affairs-26119478',
         'only_matching': True,
-    }, {
-        'url': 'http://abcnews.go.com/video/itemfeed?id=46979033',
-        'only_matching': True,
-    }, {
-        'url': 'https://abcnews.go.com/GMA/News/video/history-christmas-story-67894761',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
@@ -76,23 +64,28 @@ class AbcNewsIE(InfoExtractor):
     _VALID_URL = r'https?://abcnews\.go\.com/(?:[^/]+/)+(?P<display_id>[0-9a-z-]+)/story\?id=(?P<id>\d+)'
 
     _TESTS = [{
-        # Youtube Embeds
-        'url': 'https://abcnews.go.com/Entertainment/peter-billingsley-child-actor-christmas-story-hollywood-power/story?id=51286501',
+        'url': 'http://abcnews.go.com/Blotter/News/dramatic-video-rare-death-job-america/story?id=10498713#.UIhwosWHLjY',
         'info_dict': {
-            'id': '51286501',
-            'title': "Peter Billingsley: From child actor in 'A Christmas Story' to Hollywood power player",
-            'description': 'Billingsley went from a child actor to Hollywood power player.',
+            'id': '10498713',
+            'ext': 'flv',
+            'display_id': 'dramatic-video-rare-death-job-america',
+            'title': 'Occupational Hazards',
+            'description': 'Nightline investigates the dangers that lurk at various jobs.',
+            'thumbnail': r're:^https?://.*\.jpg$',
+            'upload_date': '20100428',
+            'timestamp': 1272412800,
         },
-        'playlist_count': 5,
+        'add_ie': ['AbcNewsVideo'],
     }, {
         'url': 'http://abcnews.go.com/Entertainment/justin-timberlake-performs-stop-feeling-eurovision-2016/story?id=39125818',
         'info_dict': {
-            'id': '38897857',
+            'id': '39125818',
             'ext': 'mp4',
+            'display_id': 'justin-timberlake-performs-stop-feeling-eurovision-2016',
             'title': 'Justin Timberlake Drops Hints For Secret Single',
             'description': 'Lara Spencer reports the buzziest stories of the day in "GMA" Pop News.',
-            'upload_date': '20160505',
-            'timestamp': 1462442280,
+            'upload_date': '20160515',
+            'timestamp': 1463329500,
         },
         'params': {
             # m3u8 download
@@ -104,55 +97,49 @@ class AbcNewsIE(InfoExtractor):
     }, {
         'url': 'http://abcnews.go.com/Technology/exclusive-apple-ceo-tim-cook-iphone-cracking-software/story?id=37173343',
         'only_matching': True,
-    }, {
-        # inline.type == 'video'
-        'url': 'http://abcnews.go.com/Technology/exclusive-apple-ceo-tim-cook-iphone-cracking-software/story?id=37173343',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
-        story_id = self._match_id(url)
-        webpage = self._download_webpage(url, story_id)
-        story = self._parse_json(self._search_regex(
-            r"window\['__abcnews__'\]\s*=\s*({.+?});",
-            webpage, 'data'), story_id)['page']['content']['story']['everscroll'][0]
-        article_contents = story.get('articleContents') or {}
+        mobj = re.match(self._VALID_URL, url)
+        display_id = mobj.group('display_id')
+        video_id = mobj.group('id')
 
-        def entries():
-            featured_video = story.get('featuredVideo') or {}
-            feed = try_get(featured_video, lambda x: x['video']['feed'])
-            if feed:
-                yield {
-                    '_type': 'url',
-                    'id': featured_video.get('id'),
-                    'title': featured_video.get('name'),
-                    'url': feed,
-                    'thumbnail': featured_video.get('images'),
-                    'description': featured_video.get('description'),
-                    'timestamp': parse_iso8601(featured_video.get('uploadDate')),
-                    'duration': parse_duration(featured_video.get('duration')),
-                    'ie_key': AbcNewsVideoIE.ie_key(),
-                }
+        webpage = self._download_webpage(url, video_id)
+        video_url = self._search_regex(
+            r'window\.abcnvideo\.url\s*=\s*"([^"]+)"', webpage, 'video URL')
+        full_video_url = compat_urlparse.urljoin(url, video_url)
 
-            for inline in (article_contents.get('inlines') or []):
-                inline_type = inline.get('type')
-                if inline_type == 'iframe':
-                    iframe_url = try_get(inline, lambda x: x['attrs']['src'])
-                    if iframe_url:
-                        yield self.url_result(iframe_url)
-                elif inline_type == 'video':
-                    video_id = inline.get('id')
-                    if video_id:
-                        yield {
-                            '_type': 'url',
-                            'id': video_id,
-                            'url': 'http://abcnews.go.com/video/embed?id=' + video_id,
-                            'thumbnail': inline.get('imgSrc') or inline.get('imgDefault'),
-                            'description': inline.get('description'),
-                            'duration': parse_duration(inline.get('duration')),
-                            'ie_key': AbcNewsVideoIE.ie_key(),
-                        }
+        youtube_url = YoutubeIE._extract_url(webpage)
 
-        return self.playlist_result(
-            entries(), story_id, article_contents.get('headline'),
-            article_contents.get('subHead'))
+        timestamp = None
+        date_str = self._html_search_regex(
+            r'<span[^>]+class="timestamp">([^<]+)</span>',
+            webpage, 'timestamp', fatal=False)
+        if date_str:
+            tz_offset = 0
+            if date_str.endswith(' ET'):  # Eastern Time
+                tz_offset = -5
+                date_str = date_str[:-3]
+            date_formats = ['%b. %d, %Y', '%b %d, %Y, %I:%M %p']
+            for date_format in date_formats:
+                try:
+                    timestamp = calendar.timegm(time.strptime(date_str.strip(), date_format))
+                except ValueError:
+                    continue
+            if timestamp is not None:
+                timestamp -= tz_offset * 3600
+
+        entry = {
+            '_type': 'url_transparent',
+            'ie_key': AbcNewsVideoIE.ie_key(),
+            'url': full_video_url,
+            'id': video_id,
+            'display_id': display_id,
+            'timestamp': timestamp,
+        }
+
+        if youtube_url:
+            entries = [entry, self.url_result(youtube_url, ie=YoutubeIE.ie_key())]
+            return self.playlist_result(entries)
+
+        return entry

youtube_dl/extractor/abcotvs.py

@@ -4,30 +4,29 @@ from __future__ import unicode_literals
 import re
 
 from .common import InfoExtractor
-from ..compat import compat_str
 from ..utils import (
-    dict_get,
     int_or_none,
-    try_get,
+    parse_iso8601,
 )
 
 
 class ABCOTVSIE(InfoExtractor):
     IE_NAME = 'abcotvs'
     IE_DESC = 'ABC Owned Television Stations'
-    _VALID_URL = r'https?://(?P<site>abc(?:7(?:news|ny|chicago)?|11|13|30)|6abc)\.com(?:(?:/[^/]+)*/(?P<display_id>[^/]+))?/(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:abc(?:7(?:news|ny|chicago)?|11|13|30)|6abc)\.com(?:/[^/]+/(?P<display_id>[^/]+))?/(?P<id>\d+)'
     _TESTS = [
         {
             'url': 'http://abc7news.com/entertainment/east-bay-museum-celebrates-vintage-synthesizers/472581/',
             'info_dict': {
-                'id': '472548',
+                'id': '472581',
                 'display_id': 'east-bay-museum-celebrates-vintage-synthesizers',
                 'ext': 'mp4',
-                'title': 'East Bay museum celebrates synthesized music',
+                'title': 'East Bay museum celebrates vintage synthesizers',
                 'description': 'md5:24ed2bd527096ec2a5c67b9d5a9005f3',
                 'thumbnail': r're:^https?://.*\.jpg$',
-                'timestamp': 1421118520,
+                'timestamp': 1421123075,
                 'upload_date': '20150113',
+                'uploader': 'Jonathan Bloom',
             },
             'params': {
                 # m3u8 download
@@ -38,63 +37,39 @@ class ABCOTVSIE(InfoExtractor):
             'url': 'http://abc7news.com/472581',
             'only_matching': True,
         },
-        {
-            'url': 'https://6abc.com/man-75-killed-after-being-struck-by-vehicle-in-chester/5725182/',
-            'only_matching': True,
-        },
     ]
-    _SITE_MAP = {
-        '6abc': 'wpvi',
-        'abc11': 'wtvd',
-        'abc13': 'ktrk',
-        'abc30': 'kfsn',
-        'abc7': 'kabc',
-        'abc7chicago': 'wls',
-        'abc7news': 'kgo',
-        'abc7ny': 'wabc',
-    }
 
     def _real_extract(self, url):
-        site, display_id, video_id = re.match(self._VALID_URL, url).groups()
-        display_id = display_id or video_id
-        station = self._SITE_MAP[site]
+        mobj = re.match(self._VALID_URL, url)
+        video_id = mobj.group('id')
+        display_id = mobj.group('display_id') or video_id
 
-        data = self._download_json(
-            'https://api.abcotvs.com/v2/content', display_id, query={
-                'id': video_id,
-                'key': 'otv.web.%s.story' % station,
-                'station': station,
-            })['data']
-        video = try_get(data, lambda x: x['featuredMedia']['video'], dict) or data
-        video_id = compat_str(dict_get(video, ('id', 'publishedKey'), video_id))
-        title = video.get('title') or video['linkText']
+        webpage = self._download_webpage(url, display_id)
 
-        formats = []
-        m3u8_url = video.get('m3u8')
-        if m3u8_url:
-            formats = self._extract_m3u8_formats(
-                video['m3u8'].split('?')[0], display_id, 'mp4', m3u8_id='hls', fatal=False)
-        mp4_url = video.get('mp4')
-        if mp4_url:
-            formats.append({
-                'abr': 128,
-                'format_id': 'https',
-                'height': 360,
-                'url': mp4_url,
-                'width': 640,
-            })
+        m3u8 = self._html_search_meta(
+            'contentURL', webpage, 'm3u8 url', fatal=True).split('?')[0]
+
+        formats = self._extract_m3u8_formats(m3u8, display_id, 'mp4')
         self._sort_formats(formats)
 
-        image = video.get('image') or {}
+        title = self._og_search_title(webpage).strip()
+        description = self._og_search_description(webpage).strip()
+        thumbnail = self._og_search_thumbnail(webpage)
+        timestamp = parse_iso8601(self._search_regex(
+            r'<div class="meta">\s*<time class="timeago" datetime="([^"]+)">',
+            webpage, 'upload date', fatal=False))
+        uploader = self._search_regex(
+            r'rel="author">([^<]+)</a>',
+            webpage, 'uploader', default=None)
 
         return {
             'id': video_id,
             'display_id': display_id,
             'title': title,
-            'description': dict_get(video, ('description', 'caption'), try_get(video, lambda x: x['meta']['description'])),
-            'thumbnail': dict_get(image, ('source', 'dynamicSource')),
-            'timestamp': int_or_none(video.get('date')),
-            'duration': int_or_none(video.get('length')),
+            'description': description,
+            'thumbnail': thumbnail,
+            'timestamp': timestamp,
+            'uploader': uploader,
             'formats': formats,
         }
 

youtube_dl/extractor/acast.py

@@ -2,125 +2,98 @@
 from __future__ import unicode_literals
 
 import re
+import functools
 
 from .common import InfoExtractor
+from ..compat import compat_str
 from ..utils import (
-    clean_html,
-    clean_podcast_url,
     int_or_none,
-    parse_iso8601,
+    unified_timestamp,
+    OnDemandPagedList,
 )
 
 
-class ACastBaseIE(InfoExtractor):
-    def _extract_episode(self, episode, show_info):
-        title = episode['title']
-        info = {
-            'id': episode['id'],
-            'display_id': episode.get('episodeUrl'),
-            'url': clean_podcast_url(episode['url']),
-            'title': title,
-            'description': clean_html(episode.get('description') or episode.get('summary')),
-            'thumbnail': episode.get('image'),
-            'timestamp': parse_iso8601(episode.get('publishDate')),
-            'duration': int_or_none(episode.get('duration')),
-            'filesize': int_or_none(episode.get('contentLength')),
-            'season_number': int_or_none(episode.get('season')),
-            'episode': title,
-            'episode_number': int_or_none(episode.get('episode')),
-        }
-        info.update(show_info)
-        return info
-
-    def _extract_show_info(self, show):
-        return {
-            'creator': show.get('author'),
-            'series': show.get('title'),
-        }
-
-    def _call_api(self, path, video_id, query=None):
-        return self._download_json(
-            'https://feeder.acast.com/api/v1/shows/' + path, video_id, query=query)
-
-
-class ACastIE(ACastBaseIE):
+class ACastIE(InfoExtractor):
     IE_NAME = 'acast'
-    _VALID_URL = r'''(?x)
-                    https?://
-                        (?:
-                            (?:(?:embed|www)\.)?acast\.com/|
-                            play\.acast\.com/s/
-                        )
-                        (?P<channel>[^/]+)/(?P<id>[^/#?]+)
-                    '''
+    _VALID_URL = r'https?://(?:www\.)?acast\.com/(?P<channel>[^/]+)/(?P<id>[^/#?]+)'
     _TESTS = [{
+        # test with one bling
+        'url': 'https://www.acast.com/condenasttraveler/-where-are-you-taipei-101-taiwan',
+        'md5': 'ada3de5a1e3a2a381327d749854788bb',
+        'info_dict': {
+            'id': '57de3baa-4bb0-487e-9418-2692c1277a34',
+            'ext': 'mp3',
+            'title': '"Where Are You?": Taipei 101, Taiwan',
+            'timestamp': 1196172000,
+            'upload_date': '20071127',
+            'description': 'md5:a0b4ef3634e63866b542e5b1199a1a0e',
+            'duration': 211,
+        }
+    }, {
+        # test with multiple blings
         'url': 'https://www.acast.com/sparpodcast/2.raggarmordet-rosterurdetforflutna',
-        'md5': 'f5598f3ad1e4776fed12ec1407153e4b',
+        'md5': 'e87d5b8516cd04c0d81b6ee1caca28d0',
         'info_dict': {
             'id': '2a92b283-1a75-4ad8-8396-499c641de0d9',
             'ext': 'mp3',
             'title': '2. Raggarmordet - Röster ur det förflutna',
-            'description': 'md5:a992ae67f4d98f1c0141598f7bebbf67',
             'timestamp': 1477346700,
             'upload_date': '20161024',
+            'description': 'md5:4f81f6d8cf2e12ee21a321d8bca32db4',
             'duration': 2766,
-            'creator': 'Anton Berg & Martin Johnson',
-            'series': 'Spår',
-            'episode': '2. Raggarmordet - Röster ur det förflutna',
         }
-    }, {
-        'url': 'http://embed.acast.com/adambuxton/ep.12-adam-joeschristmaspodcast2015',
-        'only_matching': True,
-    }, {
-        'url': 'https://play.acast.com/s/rattegangspodden/s04e09styckmordetihelenelund-del2-2',
-        'only_matching': True,
-    }, {
-        'url': 'https://play.acast.com/s/sparpodcast/2a92b283-1a75-4ad8-8396-499c641de0d9',
-        'only_matching': True,
     }]
 
     def _real_extract(self, url):
         channel, display_id = re.match(self._VALID_URL, url).groups()
-        episode = self._call_api(
-            '%s/episodes/%s' % (channel, display_id),
-            display_id, {'showInfo': 'true'})
-        return self._extract_episode(
-            episode, self._extract_show_info(episode.get('show') or {}))
+        cast_data = self._download_json(
+            'https://play-api.acast.com/splash/%s/%s' % (channel, display_id), display_id)
+        e = cast_data['result']['episode']
+        return {
+            'id': compat_str(e['id']),
+            'display_id': display_id,
+            'url': e['mediaUrl'],
+            'title': e['name'],
+            'description': e.get('description'),
+            'thumbnail': e.get('image'),
+            'timestamp': unified_timestamp(e.get('publishingDate')),
+            'duration': int_or_none(e.get('duration')),
+        }
 
 
-class ACastChannelIE(ACastBaseIE):
+class ACastChannelIE(InfoExtractor):
     IE_NAME = 'acast:channel'
-    _VALID_URL = r'''(?x)
-                    https?://
-                        (?:
-                            (?:www\.)?acast\.com/|
-                            play\.acast\.com/s/
-                        )
-                        (?P<id>[^/#?]+)
-                    '''
-    _TESTS = [{
-        'url': 'https://www.acast.com/todayinfocus',
+    _VALID_URL = r'https?://(?:www\.)?acast\.com/(?P<id>[^/#?]+)'
+    _TEST = {
+        'url': 'https://www.acast.com/condenasttraveler',
         'info_dict': {
-            'id': '4efc5294-5385-4847-98bd-519799ce5786',
-            'title': 'Today in Focus',
-            'description': 'md5:c09ce28c91002ce4ffce71d6504abaae',
+            'id': '50544219-29bb-499e-a083-6087f4cb7797',
+            'title': 'Condé Nast Traveler Podcast',
+            'description': 'md5:98646dee22a5b386626ae31866638fbd',
         },
-        'playlist_mincount': 200,
-    }, {
-        'url': 'http://play.acast.com/s/ft-banking-weekly',
-        'only_matching': True,
-    }]
+        'playlist_mincount': 20,
+    }
+    _API_BASE_URL = 'https://www.acast.com/api/'
+    _PAGE_SIZE = 10
 
     @classmethod
     def suitable(cls, url):
         return False if ACastIE.suitable(url) else super(ACastChannelIE, cls).suitable(url)
 
+    def _fetch_page(self, channel_slug, page):
+        casts = self._download_json(
+            self._API_BASE_URL + 'channels/%s/acasts?page=%s' % (channel_slug, page),
+            channel_slug, note='Download page %d of channel data' % page)
+        for cast in casts:
+            yield self.url_result(
+                'https://www.acast.com/%s/%s' % (channel_slug, cast['url']),
+                'ACast', cast['id'])
+
     def _real_extract(self, url):
-        show_slug = self._match_id(url)
-        show = self._call_api(show_slug, show_slug)
-        show_info = self._extract_show_info(show)
-        entries = []
-        for episode in (show.get('episodes') or []):
-            entries.append(self._extract_episode(episode, show_info))
-        return self.playlist_result(
-            entries, show.get('id'), show.get('title'), show.get('description'))
+        channel_slug = self._match_id(url)
+        channel_data = self._download_json(
+            self._API_BASE_URL + 'channels/%s' % channel_slug, channel_slug)
+        entries = OnDemandPagedList(functools.partial(
+            self._fetch_page, channel_slug), self._PAGE_SIZE)
+        return self.playlist_result(entries, compat_str(
+            channel_data['id']), channel_data['name'], channel_data.get('description'))

youtube_dl/extractor/addanime.py

@@ -0,0 +1,95 @@
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..compat import (
+    compat_HTTPError,
+    compat_str,
+    compat_urllib_parse_urlencode,
+    compat_urllib_parse_urlparse,
+)
+from ..utils import (
+    ExtractorError,
+    qualities,
+)
+
+
+class AddAnimeIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:\w+\.)?add-anime\.net/(?:watch_video\.php\?(?:.*?)v=|video/)(?P<id>[\w_]+)'
+    _TESTS = [{
+        'url': 'http://www.add-anime.net/watch_video.php?v=24MR3YO5SAS9',
+        'md5': '72954ea10bc979ab5e2eb288b21425a0',
+        'info_dict': {
+            'id': '24MR3YO5SAS9',
+            'ext': 'mp4',
+            'description': 'One Piece 606',
+            'title': 'One Piece 606',
+        },
+        'skip': 'Video is gone',
+    }, {
+        'url': 'http://add-anime.net/video/MDUGWYKNGBD8/One-Piece-687',
+        'only_matching': True,
+    }]
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+
+        try:
+            webpage = self._download_webpage(url, video_id)
+        except ExtractorError as ee:
+            if not isinstance(ee.cause, compat_HTTPError) or \
+               ee.cause.code != 503:
+                raise
+
+            redir_webpage = ee.cause.read().decode('utf-8')
+            action = self._search_regex(
+                r'<form id="challenge-form" action="([^"]+)"',
+                redir_webpage, 'Redirect form')
+            vc = self._search_regex(
+                r'<input type="hidden" name="jschl_vc" value="([^"]+)"/>',
+                redir_webpage, 'redirect vc value')
+            av = re.search(
+                r'a\.value = ([0-9]+)[+]([0-9]+)[*]([0-9]+);',
+                redir_webpage)
+            if av is None:
+                raise ExtractorError('Cannot find redirect math task')
+            av_res = int(av.group(1)) + int(av.group(2)) * int(av.group(3))
+
+            parsed_url = compat_urllib_parse_urlparse(url)
+            av_val = av_res + len(parsed_url.netloc)
+            confirm_url = (
+                parsed_url.scheme + '://' + parsed_url.netloc +
+                action + '?' +
+                compat_urllib_parse_urlencode({
+                    'jschl_vc': vc, 'jschl_answer': compat_str(av_val)}))
+            self._download_webpage(
+                confirm_url, video_id,
+                note='Confirming after redirect')
+            webpage = self._download_webpage(url, video_id)
+
+        FORMATS = ('normal', 'hq')
+        quality = qualities(FORMATS)
+        formats = []
+        for format_id in FORMATS:
+            rex = r"var %s_video_file = '(.*?)';" % re.escape(format_id)
+            video_url = self._search_regex(rex, webpage, 'video file URLx',
+                                           fatal=False)
+            if not video_url:
+                continue
+            formats.append({
+                'format_id': format_id,
+                'url': video_url,
+                'quality': quality(format_id),
+            })
+        self._sort_formats(formats)
+        video_title = self._og_search_title(webpage)
+        video_description = self._og_search_description(webpage)
+
+        return {
+            '_type': 'video',
+            'id': video_id,
+            'formats': formats,
+            'title': video_title,
+            'description': video_description
+        }

youtube_dl/extractor/adn.py

@@ -2,99 +2,55 @@
 from __future__ import unicode_literals
 
 import base64
-import binascii
 import json
 import os
-import random
 
 from .common import InfoExtractor
 from ..aes import aes_cbc_decrypt
-from ..compat import (
-    compat_HTTPError,
-    compat_b64decode,
-    compat_ord,
-)
+from ..compat import compat_ord
 from ..utils import (
     bytes_to_intlist,
-    bytes_to_long,
     ExtractorError,
     float_or_none,
-    int_or_none,
     intlist_to_bytes,
-    long_to_bytes,
-    pkcs1pad,
+    srt_subtitles_timecode,
     strip_or_none,
-    try_get,
-    unified_strdate,
-    urlencode_postdata,
+    urljoin,
 )
 
 
 class ADNIE(InfoExtractor):
-    IE_DESC = 'Animation Digital Network'
-    _VALID_URL = r'https?://(?:www\.)?(?:animation|anime)digitalnetwork\.fr/video/[^/]+/(?P<id>\d+)'
-    _TESTS = [{
-        'url': 'https://animationdigitalnetwork.fr/video/fruits-basket/9841-episode-1-a-ce-soir',
-        'md5': '1c9ef066ceb302c86f80c2b371615261',
-        'info_dict': {
-            'id': '9841',
-            'ext': 'mp4',
-            'title': 'Fruits Basket - Episode 1',
-            'description': 'md5:14be2f72c3c96809b0ca424b0097d336',
-            'series': 'Fruits Basket',
-            'duration': 1437,
-            'release_date': '20190405',
-            'comment_count': int,
-            'average_rating': float,
-            'season_number': 1,
-            'episode': 'À ce soir !',
-            'episode_number': 1,
-        },
-        'skip': 'Only available in region (FR, ...)',
-    }, {
+    IE_DESC = 'Anime Digital Network'
+    _VALID_URL = r'https?://(?:www\.)?animedigitalnetwork\.fr/video/[^/]+/(?P<id>\d+)'
+    _TEST = {
         'url': 'http://animedigitalnetwork.fr/video/blue-exorcist-kyoto-saga/7778-episode-1-debut-des-hostilites',
-        'only_matching': True,
-    }]
-
-    _NETRC_MACHINE = 'animationdigitalnetwork'
-    _BASE = 'animationdigitalnetwork.fr'
-    _API_BASE_URL = 'https://gw.api.' + _BASE + '/'
-    _PLAYER_BASE_URL = _API_BASE_URL + 'player/'
-    _HEADERS = {}
-    _LOGIN_ERR_MESSAGE = 'Unable to log in'
-    _RSA_KEY = (0x9B42B08905199A5CCE2026274399CA560ECB209EE9878A708B1C0812E1BB8CB5D1FB7441861147C1A1F2F3A0476DD63A9CAC20D3E983613346850AA6CB38F16DC7D720FD7D86FC6E5B3D5BBC72E14CD0BF9E869F2CEA2CCAD648F1DCE38F1FF916CEFB2D339B64AA0264372344BC775E265E8A852F88144AB0BD9AA06C1A4ABB, 65537)
-    _POS_ALIGN_MAP = {
-        'start': 1,
-        'end': 3,
-    }
-    _LINE_ALIGN_MAP = {
-        'middle': 8,
-        'end': 4,
+        'md5': 'e497370d847fd79d9d4c74be55575c7a',
+        'info_dict': {
+            'id': '7778',
+            'ext': 'mp4',
+            'title': 'Blue Exorcist - Kyôto Saga - Épisode 1',
+            'description': 'md5:2f7b5aa76edbc1a7a92cedcda8a528d5',
+        }
     }
+    _BASE_URL = 'http://animedigitalnetwork.fr'
 
-    @staticmethod
-    def _ass_subtitles_timecode(seconds):
-        return '%01d:%02d:%02d.%02d' % (seconds / 3600, (seconds % 3600) / 60, seconds % 60, (seconds % 1) * 100)
-
-    def _get_subtitles(self, sub_url, video_id):
-        if not sub_url:
+    def _get_subtitles(self, sub_path, video_id):
+        if not sub_path:
             return None
 
         enc_subtitles = self._download_webpage(
-            sub_url, video_id, 'Downloading subtitles location', fatal=False) or '{}'
-        subtitle_location = (self._parse_json(enc_subtitles, video_id, fatal=False) or {}).get('location')
-        if subtitle_location:
-            enc_subtitles = self._download_webpage(
-                subtitle_location, video_id, 'Downloading subtitles data',
-                fatal=False, headers={'Origin': 'https://' + self._BASE})
+            urljoin(self._BASE_URL, sub_path),
+            video_id, fatal=False, headers={
+                'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0',
+            })
         if not enc_subtitles:
             return None
 
-        # http://animationdigitalnetwork.fr/components/com_vodvideo/videojs/adn-vjs.min.js
+        # http://animedigitalnetwork.fr/components/com_vodvideo/videojs/adn-vjs.min.js
         dec_subtitles = intlist_to_bytes(aes_cbc_decrypt(
-            bytes_to_intlist(compat_b64decode(enc_subtitles[24:])),
-            bytes_to_intlist(binascii.unhexlify(self._K + '7fac1178830cfe0c')),
-            bytes_to_intlist(compat_b64decode(enc_subtitles[:24]))
+            bytes_to_intlist(base64.b64decode(enc_subtitles[24:])),
+            bytes_to_intlist(b'\x1b\xe0\x29\x61\x38\x94\x24\x00\x12\xbd\xc5\x80\xac\xce\xbe\xb0'),
+            bytes_to_intlist(base64.b64decode(enc_subtitles[:24]))
         ))
         subtitles_json = self._parse_json(
             dec_subtitles[:-compat_ord(dec_subtitles[-1])].decode(),
@@ -104,27 +60,23 @@ class ADNIE(InfoExtractor):
 
         subtitles = {}
         for sub_lang, sub in subtitles_json.items():
-            ssa = '''[Script Info]
-ScriptType:V4.00
-[V4 Styles]
-Format: Name,Fontname,Fontsize,PrimaryColour,SecondaryColour,TertiaryColour,BackColour,Bold,Italic,BorderStyle,Outline,Shadow,Alignment,MarginL,MarginR,MarginV,AlphaLevel,Encoding
-Style: Default,Arial,18,16777215,16777215,16777215,0,-1,0,1,1,0,2,20,20,20,0,0
-[Events]
-Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
-            for current in sub:
-                start, end, text, line_align, position_align = (
+            srt = ''
+            for num, current in enumerate(sub):
+                start, end, text = (
                     float_or_none(current.get('startTime')),
                     float_or_none(current.get('endTime')),
-                    current.get('text'), current.get('lineAlign'),
-                    current.get('positionAlign'))
+                    current.get('text'))
                 if start is None or end is None or text is None:
                     continue
-                alignment = self._POS_ALIGN_MAP.get(position_align, 2) + self._LINE_ALIGN_MAP.get(line_align, 0)
-                ssa += os.linesep + 'Dialogue: Marked=0,%s,%s,Default,,0,0,0,,%s%s' % (
-                    self._ass_subtitles_timecode(start),
-                    self._ass_subtitles_timecode(end),
-                    '{\\a%d}' % alignment if alignment != 2 else '',
-                    text.replace('\n', '\\N').replace('<i>', '{\\i1}').replace('</i>', '{\\i0}'))
+                srt += os.linesep.join(
+                    (
+                        '%d' % num,
+                        '%s --> %s' % (
+                            srt_subtitles_timecode(start),
+                            srt_subtitles_timecode(end)),
+                        text,
+                        os.linesep,
+                    ))
 
             if sub_lang == 'vostf':
                 sub_lang = 'fr'
@@ -132,115 +84,44 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
                 'ext': 'json',
                 'data': json.dumps(sub),
             }, {
-                'ext': 'ssa',
-                'data': ssa,
+                'ext': 'srt',
+                'data': srt,
             }])
         return subtitles
 
-    def _real_initialize(self):
-        username, password = self._get_login_info()
-        if not username:
-            return
-        try:
-            url = self._API_BASE_URL + 'authentication/login'
-            access_token = (self._download_json(
-                url, None, 'Logging in', self._LOGIN_ERR_MESSAGE, fatal=False,
-                data=urlencode_postdata({
-                    'password': password,
-                    'rememberMe': False,
-                    'source': 'Web',
-                    'username': username,
-                })) or {}).get('accessToken')
-            if access_token:
-                self._HEADERS = {'authorization': 'Bearer ' + access_token}
-        except ExtractorError as e:
-            message = None
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
-                resp = self._parse_json(
-                    self._webpage_read_content(e.cause, url, username),
-                    username, fatal=False) or {}
-                message = resp.get('message') or resp.get('code')
-            self.report_warning(message or self._LOGIN_ERR_MESSAGE)
-
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_base_url = self._PLAYER_BASE_URL + 'video/%s/' % video_id
-        player = self._download_json(
-            video_base_url + 'configuration', video_id,
-            'Downloading player config JSON metadata',
-            headers=self._HEADERS)['player']
-        options = player['options']
+        webpage = self._download_webpage(url, video_id)
+        player_config = self._parse_json(self._search_regex(
+            r'playerConfig\s*=\s*({.+});', webpage, 'player config'), video_id)
 
-        user = options['user']
-        if not user.get('hasAccess'):
-            self.raise_login_required()
+        video_info = {}
+        video_info_str = self._search_regex(
+            r'videoInfo\s*=\s*({.+});', webpage,
+            'video info', fatal=False)
+        if video_info_str:
+            video_info = self._parse_json(
+                video_info_str, video_id, fatal=False) or {}
 
-        token = self._download_json(
-            user.get('refreshTokenUrl') or (self._PLAYER_BASE_URL + 'refresh/token'),
-            video_id, 'Downloading access token', headers={
-                'x-player-refresh-token': user['refreshToken']
-            }, data=b'')['token']
-
-        links_url = try_get(options, lambda x: x['video']['url']) or (video_base_url + 'link')
-        self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
-        message = bytes_to_intlist(json.dumps({
-            'k': self._K,
-            't': token,
-        }))
-
-        # Sometimes authentication fails for no good reason, retry with
-        # a different random padding
-        links_data = None
-        for _ in range(3):
-            padded_message = intlist_to_bytes(pkcs1pad(message, 128))
-            n, e = self._RSA_KEY
-            encrypted_message = long_to_bytes(pow(bytes_to_long(padded_message), e, n))
-            authorization = base64.b64encode(encrypted_message).decode()
-
-            try:
-                links_data = self._download_json(
-                    links_url, video_id, 'Downloading links JSON metadata', headers={
-                        'X-Player-Token': authorization
-                    }, query={
-                        'freeWithAds': 'true',
-                        'adaptive': 'false',
-                        'withMetadata': 'true',
-                        'source': 'Web'
-                    })
-                break
-            except ExtractorError as e:
-                if not isinstance(e.cause, compat_HTTPError):
-                    raise e
-
-                if e.cause.code == 401:
-                    # This usually goes away with a different random pkcs1pad, so retry
-                    continue
-
-                error = self._parse_json(
-                    self._webpage_read_content(e.cause, links_url, video_id),
-                    video_id, fatal=False) or {}
-                message = error.get('message')
-                if e.cause.code == 403 and error.get('code') == 'player-bad-geolocation-country':
-                    self.raise_geo_restricted(msg=message)
-                raise ExtractorError(message)
-        else:
-            raise ExtractorError('Giving up retrying')
-
-        links = links_data.get('links') or {}
-        metas = links_data.get('metadata') or {}
-        sub_url = (links.get('subtitles') or {}).get('all')
-        video_info = links_data.get('video') or {}
-        title = metas['title']
+        options = player_config.get('options') or {}
+        metas = options.get('metas') or {}
+        title = metas.get('title') or video_info['title']
+        links = player_config.get('links') or {}
+        error = None
+        if not links:
+            links_url = player_config['linksurl']
+            links_data = self._download_json(urljoin(
+                self._BASE_URL, links_url), video_id)
+            links = links_data.get('links') or {}
+            error = links_data.get('error')
 
         formats = []
-        for format_id, qualities in (links.get('streaming') or {}).items():
+        for format_id, qualities in links.items():
             if not isinstance(qualities, dict):
                 continue
-            for quality, load_balancer_url in qualities.items():
+            for load_balancer_url in qualities.values():
                 load_balancer_data = self._download_json(
-                    load_balancer_url, video_id,
-                    'Downloading %s %s JSON metadata' % (format_id, quality),
-                    fatal=False) or {}
+                    load_balancer_url, video_id, fatal=False) or {}
                 m3u8_url = load_balancer_data.get('location')
                 if not m3u8_url:
                     continue
@@ -251,26 +132,19 @@ Format: Marked,Start,End,Style,Name,MarginL,MarginR,MarginV,Effect,Text'''
                     for f in m3u8_formats:
                         f['language'] = 'fr'
                 formats.extend(m3u8_formats)
+        if not error:
+            error = options.get('error')
+        if not formats and error:
+            raise ExtractorError('%s said: %s' % (self.IE_NAME, error), expected=True)
         self._sort_formats(formats)
 
-        video = (self._download_json(
-            self._API_BASE_URL + 'video/%s' % video_id, video_id,
-            'Downloading additional video metadata', fatal=False) or {}).get('video') or {}
-        show = video.get('show') or {}
-
         return {
             'id': video_id,
             'title': title,
-            'description': strip_or_none(metas.get('summary') or video.get('summary')),
-            'thumbnail': video_info.get('image') or player.get('image'),
+            'description': strip_or_none(metas.get('summary') or video_info.get('resume')),
+            'thumbnail': video_info.get('image'),
             'formats': formats,
-            'subtitles': self.extract_subtitles(sub_url, video_id),
-            'episode': metas.get('subtitle') or video.get('name'),
-            'episode_number': int_or_none(video.get('shortNumber')),
-            'series': show.get('title'),
-            'season_number': int_or_none(video.get('season')),
-            'duration': int_or_none(video_info.get('duration') or video.get('duration')),
-            'release_date': unified_strdate(video.get('releaseDate')),
-            'average_rating': float_or_none(video.get('rating') or metas.get('rating')),
-            'comment_count': int_or_none(video.get('commentsCount')),
+            'subtitles': self.extract_subtitles(player_config.get('subtitles'), video_id),
+            'episode': metas.get('subtitle') or video_info.get('videoTitle'),
+            'series': video_info.get('playlistTitle'),
         }

youtube_dl/extractor/adobeconnect.py

@@ -1,37 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..compat import (
-    compat_parse_qs,
-    compat_urlparse,
-)
-
-
-class AdobeConnectIE(InfoExtractor):
-    _VALID_URL = r'https?://\w+\.adobeconnect\.com/(?P<id>[\w-]+)'
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        title = self._html_search_regex(r'<title>(.+?)</title>', webpage, 'title')
-        qs = compat_parse_qs(self._search_regex(r"swfUrl\s*=\s*'([^']+)'", webpage, 'swf url').split('?')[1])
-        is_live = qs.get('isLive', ['false'])[0] == 'true'
-        formats = []
-        for con_string in qs['conStrings'][0].split(','):
-            formats.append({
-                'format_id': con_string.split('://')[0],
-                'app': compat_urlparse.quote('?' + con_string.split('?')[1] + 'flvplayerapp/' + qs['appInstance'][0]),
-                'ext': 'flv',
-                'play_path': 'mp4:' + qs['streamName'][0],
-                'rtmp_conn': 'S:' + qs['ticket'][0],
-                'rtmp_live': is_live,
-                'url': con_string,
-            })
-
-        return {
-            'id': video_id,
-            'title': self._live_title(title) if is_live else title,
-            'formats': formats,
-            'is_live': is_live,
-        }

youtube_dl/extractor/adobepass.py

@@ -25,11 +25,6 @@ MSO_INFO = {
         'username_field': 'username',
         'password_field': 'password',
     },
-    'ATT': {
-        'name': 'AT&T U-verse',
-        'username_field': 'userid',
-        'password_field': 'password',
-    },
     'ATTOTT': {
         'name': 'DIRECTV NOW',
         'username_field': 'email',
@@ -1330,8 +1325,8 @@ class AdobePassIE(InfoExtractor):
     _DOWNLOADING_LOGIN_PAGE = 'Downloading Provider Login Page'
 
     def _download_webpage_handle(self, *args, **kwargs):
-        headers = self.geo_verification_headers()
-        headers.update(kwargs.get('headers', {}))
+        headers = kwargs.get('headers', {})
+        headers.update(self.geo_verification_headers())
         kwargs['headers'] = headers
         return super(AdobePassIE, self)._download_webpage_handle(
             *args, **compat_kwargs(kwargs))

youtube_dl/extractor/adobetv.py

@@ -1,119 +1,25 @@
 from __future__ import unicode_literals
 
-import functools
 import re
 
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
-    float_or_none,
-    int_or_none,
-    ISO639Utils,
-    OnDemandPagedList,
     parse_duration,
-    str_or_none,
-    str_to_int,
     unified_strdate,
+    str_to_int,
+    int_or_none,
+    float_or_none,
+    ISO639Utils,
+    determine_ext,
 )
 
 
 class AdobeTVBaseIE(InfoExtractor):
-    def _call_api(self, path, video_id, query, note=None):
-        return self._download_json(
-            'http://tv.adobe.com/api/v4/' + path,
-            video_id, note, query=query)['data']
-
-    def _parse_subtitles(self, video_data, url_key):
-        subtitles = {}
-        for translation in video_data.get('translations', []):
-            vtt_path = translation.get(url_key)
-            if not vtt_path:
-                continue
-            lang = translation.get('language_w3c') or ISO639Utils.long2short(translation['language_medium'])
-            subtitles.setdefault(lang, []).append({
-                'ext': 'vtt',
-                'url': vtt_path,
-            })
-        return subtitles
-
-    def _parse_video_data(self, video_data):
-        video_id = compat_str(video_data['id'])
-        title = video_data['title']
-
-        s3_extracted = False
-        formats = []
-        for source in video_data.get('videos', []):
-            source_url = source.get('url')
-            if not source_url:
-                continue
-            f = {
-                'format_id': source.get('quality_level'),
-                'fps': int_or_none(source.get('frame_rate')),
-                'height': int_or_none(source.get('height')),
-                'tbr': int_or_none(source.get('video_data_rate')),
-                'width': int_or_none(source.get('width')),
-                'url': source_url,
-            }
-            original_filename = source.get('original_filename')
-            if original_filename:
-                if not (f.get('height') and f.get('width')):
-                    mobj = re.search(r'_(\d+)x(\d+)', original_filename)
-                    if mobj:
-                        f.update({
-                            'height': int(mobj.group(2)),
-                            'width': int(mobj.group(1)),
-                        })
-                if original_filename.startswith('s3://') and not s3_extracted:
-                    formats.append({
-                        'format_id': 'original',
-                        'preference': 1,
-                        'url': original_filename.replace('s3://', 'https://s3.amazonaws.com/'),
-                    })
-                    s3_extracted = True
-            formats.append(f)
-        self._sort_formats(formats)
-
-        return {
-            'id': video_id,
-            'title': title,
-            'description': video_data.get('description'),
-            'thumbnail': video_data.get('thumbnail'),
-            'upload_date': unified_strdate(video_data.get('start_date')),
-            'duration': parse_duration(video_data.get('duration')),
-            'view_count': str_to_int(video_data.get('playcount')),
-            'formats': formats,
-            'subtitles': self._parse_subtitles(video_data, 'vtt'),
-        }
-
-
-class AdobeTVEmbedIE(AdobeTVBaseIE):
-    IE_NAME = 'adobetv:embed'
-    _VALID_URL = r'https?://tv\.adobe\.com/embed/\d+/(?P<id>\d+)'
-    _TEST = {
-        'url': 'https://tv.adobe.com/embed/22/4153',
-        'md5': 'c8c0461bf04d54574fc2b4d07ac6783a',
-        'info_dict': {
-            'id': '4153',
-            'ext': 'flv',
-            'title': 'Creating Graphics Optimized for BlackBerry',
-            'description': 'md5:eac6e8dced38bdaae51cd94447927459',
-            'thumbnail': r're:https?://.*\.jpg$',
-            'upload_date': '20091109',
-            'duration': 377,
-            'view_count': int,
-        },
-    }
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-
-        video_data = self._call_api(
-            'episode/' + video_id, video_id, {'disclosure': 'standard'})[0]
-        return self._parse_video_data(video_data)
+    _API_BASE_URL = 'http://tv.adobe.com/api/v4/'
 
 
 class AdobeTVIE(AdobeTVBaseIE):
-    IE_NAME = 'adobetv'
     _VALID_URL = r'https?://tv\.adobe\.com/(?:(?P<language>fr|de|es|jp)/)?watch/(?P<show_urlname>[^/]+)/(?P<id>[^/]+)'
 
     _TEST = {
@@ -136,33 +42,45 @@ class AdobeTVIE(AdobeTVBaseIE):
         if not language:
             language = 'en'
 
-        video_data = self._call_api(
-            'episode/get', urlname, {
-                'disclosure': 'standard',
-                'language': language,
-                'show_urlname': show_urlname,
-                'urlname': urlname,
-            })[0]
-        return self._parse_video_data(video_data)
+        video_data = self._download_json(
+            self._API_BASE_URL + 'episode/get/?language=%s&show_urlname=%s&urlname=%s&disclosure=standard' % (language, show_urlname, urlname),
+            urlname)['data'][0]
+
+        formats = [{
+            'url': source['url'],
+            'format_id': source.get('quality_level') or source['url'].split('-')[-1].split('.')[0] or None,
+            'width': int_or_none(source.get('width')),
+            'height': int_or_none(source.get('height')),
+            'tbr': int_or_none(source.get('video_data_rate')),
+        } for source in video_data['videos']]
+        self._sort_formats(formats)
+
+        return {
+            'id': compat_str(video_data['id']),
+            'title': video_data['title'],
+            'description': video_data.get('description'),
+            'thumbnail': video_data.get('thumbnail'),
+            'upload_date': unified_strdate(video_data.get('start_date')),
+            'duration': parse_duration(video_data.get('duration')),
+            'view_count': str_to_int(video_data.get('playcount')),
+            'formats': formats,
+        }
 
 
 class AdobeTVPlaylistBaseIE(AdobeTVBaseIE):
-    _PAGE_SIZE = 25
+    def _parse_page_data(self, page_data):
+        return [self.url_result(self._get_element_url(element_data)) for element_data in page_data]
 
-    def _fetch_page(self, display_id, query, page):
-        page += 1
-        query['page'] = page
-        for element_data in self._call_api(
-                self._RESOURCE, display_id, query, 'Download Page %d' % page):
-            yield self._process_data(element_data)
-
-    def _extract_playlist_entries(self, display_id, query):
-        return OnDemandPagedList(functools.partial(
-            self._fetch_page, display_id, query), self._PAGE_SIZE)
+    def _extract_playlist_entries(self, url, display_id):
+        page = self._download_json(url, display_id)
+        entries = self._parse_page_data(page['data'])
+        for page_num in range(2, page['paging']['pages'] + 1):
+            entries.extend(self._parse_page_data(
+                self._download_json(url + '&page=%d' % page_num, display_id)['data']))
+        return entries
 
 
 class AdobeTVShowIE(AdobeTVPlaylistBaseIE):
-    IE_NAME = 'adobetv:show'
     _VALID_URL = r'https?://tv\.adobe\.com/(?:(?P<language>fr|de|es|jp)/)?show/(?P<id>[^/]+)'
 
     _TEST = {
@@ -174,31 +92,26 @@ class AdobeTVShowIE(AdobeTVPlaylistBaseIE):
         },
         'playlist_mincount': 136,
     }
-    _RESOURCE = 'episode'
-    _process_data = AdobeTVBaseIE._parse_video_data
+
+    def _get_element_url(self, element_data):
+        return element_data['urls'][0]
 
     def _real_extract(self, url):
         language, show_urlname = re.match(self._VALID_URL, url).groups()
         if not language:
             language = 'en'
-        query = {
-            'disclosure': 'standard',
-            'language': language,
-            'show_urlname': show_urlname,
-        }
+        query = 'language=%s&show_urlname=%s' % (language, show_urlname)
 
-        show_data = self._call_api(
-            'show/get', show_urlname, query)[0]
+        show_data = self._download_json(self._API_BASE_URL + 'show/get/?%s' % query, show_urlname)['data'][0]
 
         return self.playlist_result(
-            self._extract_playlist_entries(show_urlname, query),
-            str_or_none(show_data.get('id')),
-            show_data.get('show_name'),
-            show_data.get('show_description'))
+            self._extract_playlist_entries(self._API_BASE_URL + 'episode/?%s' % query, show_urlname),
+            compat_str(show_data['id']),
+            show_data['show_name'],
+            show_data['show_description'])
 
 
 class AdobeTVChannelIE(AdobeTVPlaylistBaseIE):
-    IE_NAME = 'adobetv:channel'
     _VALID_URL = r'https?://tv\.adobe\.com/(?:(?P<language>fr|de|es|jp)/)?channel/(?P<id>[^/]+)(?:/(?P<category_urlname>[^/]+))?'
 
     _TEST = {
@@ -208,30 +121,24 @@ class AdobeTVChannelIE(AdobeTVPlaylistBaseIE):
         },
         'playlist_mincount': 96,
     }
-    _RESOURCE = 'show'
 
-    def _process_data(self, show_data):
-        return self.url_result(
-            show_data['url'], 'AdobeTVShow', str_or_none(show_data.get('id')))
+    def _get_element_url(self, element_data):
+        return element_data['url']
 
     def _real_extract(self, url):
         language, channel_urlname, category_urlname = re.match(self._VALID_URL, url).groups()
         if not language:
             language = 'en'
-        query = {
-            'channel_urlname': channel_urlname,
-            'language': language,
-        }
+        query = 'language=%s&channel_urlname=%s' % (language, channel_urlname)
         if category_urlname:
-            query['category_urlname'] = category_urlname
+            query += '&category_urlname=%s' % category_urlname
 
         return self.playlist_result(
-            self._extract_playlist_entries(channel_urlname, query),
+            self._extract_playlist_entries(self._API_BASE_URL + 'show/?%s' % query, channel_urlname),
             channel_urlname)
 
 
-class AdobeTVVideoIE(AdobeTVBaseIE):
-    IE_NAME = 'adobetv:video'
+class AdobeTVVideoIE(InfoExtractor):
     _VALID_URL = r'https?://video\.tv\.adobe\.com/v/(?P<id>\d+)'
 
     _TEST = {
@@ -253,36 +160,38 @@ class AdobeTVVideoIE(AdobeTVBaseIE):
 
         video_data = self._parse_json(self._search_regex(
             r'var\s+bridge\s*=\s*([^;]+);', webpage, 'bridged data'), video_id)
-        title = video_data['title']
 
-        formats = []
-        sources = video_data.get('sources') or []
-        for source in sources:
-            source_src = source.get('src')
-            if not source_src:
-                continue
-            formats.append({
-                'filesize': int_or_none(source.get('kilobytes') or None, invscale=1000),
-                'format_id': '-'.join(filter(None, [source.get('format'), source.get('label')])),
-                'height': int_or_none(source.get('height') or None),
-                'tbr': int_or_none(source.get('bitrate') or None),
-                'width': int_or_none(source.get('width') or None),
-                'url': source_src,
-            })
+        formats = [{
+            'format_id': '%s-%s' % (determine_ext(source['src']), source.get('height')),
+            'url': source['src'],
+            'width': int_or_none(source.get('width')),
+            'height': int_or_none(source.get('height')),
+            'tbr': int_or_none(source.get('bitrate')),
+        } for source in video_data['sources']]
         self._sort_formats(formats)
 
         # For both metadata and downloaded files the duration varies among
         # formats. I just pick the max one
         duration = max(filter(None, [
             float_or_none(source.get('duration'), scale=1000)
-            for source in sources]))
+            for source in video_data['sources']]))
+
+        subtitles = {}
+        for translation in video_data.get('translations', []):
+            lang_id = translation.get('language_w3c') or ISO639Utils.long2short(translation['language_medium'])
+            if lang_id not in subtitles:
+                subtitles[lang_id] = []
+            subtitles[lang_id].append({
+                'url': translation['vttPath'],
+                'ext': 'vtt',
+            })
 
         return {
             'id': video_id,
             'formats': formats,
-            'title': title,
+            'title': video_data['title'],
             'description': video_data.get('description'),
-            'thumbnail': video_data.get('video', {}).get('poster'),
+            'thumbnail': video_data['video'].get('poster'),
             'duration': duration,
-            'subtitles': self._parse_subtitles(video_data, 'vttPath'),
+            'subtitles': subtitles,
         }

youtube_dl/extractor/adultswim.py

@@ -1,19 +1,12 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import json
 import re
 
 from .turner import TurnerBaseIE
 from ..utils import (
-    determine_ext,
-    float_or_none,
     int_or_none,
-    mimetype2ext,
-    parse_age_limit,
-    parse_iso8601,
     strip_or_none,
-    try_get,
 )
 
 
@@ -27,8 +20,8 @@ class AdultSwimIE(TurnerBaseIE):
             'ext': 'mp4',
             'title': 'Rick and Morty - Pilot',
             'description': 'Rick moves in with his daughter\'s family and establishes himself as a bad influence on his grandson, Morty.',
-            'timestamp': 1543294800,
-            'upload_date': '20181127',
+            'timestamp': 1493267400,
+            'upload_date': '20170427',
         },
         'params': {
             # m3u8 download
@@ -49,7 +42,6 @@ class AdultSwimIE(TurnerBaseIE):
             # m3u8 download
             'skip_download': True,
         },
-        'skip': '404 Not Found',
     }, {
         'url': 'http://www.adultswim.com/videos/decker/inside-decker-a-new-hero/',
         'info_dict': {
@@ -68,9 +60,9 @@ class AdultSwimIE(TurnerBaseIE):
     }, {
         'url': 'http://www.adultswim.com/videos/attack-on-titan',
         'info_dict': {
-            'id': 'attack-on-titan',
+            'id': 'b7A69dzfRzuaXIECdxW8XQ',
             'title': 'Attack on Titan',
-            'description': 'md5:41caa9416906d90711e31dc00cb7db7e',
+            'description': 'md5:6c8e003ea0777b47013e894767f5e114',
         },
         'playlist_mincount': 12,
     }, {
@@ -85,118 +77,83 @@ class AdultSwimIE(TurnerBaseIE):
             # m3u8 download
             'skip_download': True,
         },
-        'skip': '404 Not Found',
     }]
 
     def _real_extract(self, url):
         show_path, episode_path = re.match(self._VALID_URL, url).groups()
         display_id = episode_path or show_path
-        query = '''query {
-  getShowBySlug(slug:"%s") {
-    %%s
-  }
-}''' % show_path
-        if episode_path:
-            query = query % '''title
-    getVideoBySlug(slug:"%s") {
-      _id
-      auth
-      description
-      duration
-      episodeNumber
-      launchDate
-      mediaID
-      seasonNumber
-      poster
-      title
-      tvRating
-    }''' % episode_path
-            ['getVideoBySlug']
-        else:
-            query = query % '''metaDescription
-    title
-    videos(first:1000,sort:["episode_number"]) {
-      edges {
-        node {
-           _id
-           slug
-        }
-      }
-    }'''
-        show_data = self._download_json(
-            'https://www.adultswim.com/api/search', display_id,
-            data=json.dumps({'query': query}).encode(),
-            headers={'Content-Type': 'application/json'})['data']['getShowBySlug']
-        if episode_path:
-            video_data = show_data['getVideoBySlug']
-            video_id = video_data['_id']
-            episode_title = title = video_data['title']
-            series = show_data.get('title')
-            if series:
-                title = '%s - %s' % (series, title)
-            info = {
-                'id': video_id,
-                'title': title,
-                'description': strip_or_none(video_data.get('description')),
-                'duration': float_or_none(video_data.get('duration')),
-                'formats': [],
-                'subtitles': {},
-                'age_limit': parse_age_limit(video_data.get('tvRating')),
-                'thumbnail': video_data.get('poster'),
-                'timestamp': parse_iso8601(video_data.get('launchDate')),
-                'series': series,
-                'season_number': int_or_none(video_data.get('seasonNumber')),
-                'episode': episode_title,
-                'episode_number': int_or_none(video_data.get('episodeNumber')),
-            }
+        webpage = self._download_webpage(url, display_id)
+        initial_data = self._parse_json(self._search_regex(
+            r'AS_INITIAL_DATA(?:__)?\s*=\s*({.+?});',
+            webpage, 'initial data'), display_id)
 
-            auth = video_data.get('auth')
-            media_id = video_data.get('mediaID')
-            if media_id:
-                info.update(self._extract_ngtv_info(media_id, {
-                    # CDN_TOKEN_APP_ID from:
-                    # https://d2gg02c3xr550i.cloudfront.net/assets/asvp.e9c8bef24322d060ef87.bundle.js
-                    'appId': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhcHBJZCI6ImFzLXR2ZS1kZXNrdG9wLXB0enQ2bSIsInByb2R1Y3QiOiJ0dmUiLCJuZXR3b3JrIjoiYXMiLCJwbGF0Zm9ybSI6ImRlc2t0b3AiLCJpYXQiOjE1MzI3MDIyNzl9.BzSCk-WYOZ2GMCIaeVb8zWnzhlgnXuJTCu0jGp_VaZE',
-                }, {
-                    'url': url,
-                    'site_name': 'AdultSwim',
-                    'auth_required': auth,
-                }))
+        is_stream = show_path == 'streams'
+        if is_stream:
+            if not episode_path:
+                episode_path = 'live-stream'
 
-            if not auth:
-                extract_data = self._download_json(
-                    'https://www.adultswim.com/api/shows/v1/videos/' + video_id,
-                    video_id, query={'fields': 'stream'}, fatal=False) or {}
-                assets = try_get(extract_data, lambda x: x['data']['video']['stream']['assets'], list) or []
-                for asset in assets:
-                    asset_url = asset.get('url')
-                    if not asset_url:
+            video_data = next(stream for stream_path, stream in initial_data['streams'].items() if stream_path == episode_path)
+            video_id = video_data.get('stream')
+
+            if not video_id:
+                entries = []
+                for episode in video_data.get('archiveEpisodes', []):
+                    episode_url = episode.get('url')
+                    if not episode_url:
                         continue
-                    ext = determine_ext(asset_url, mimetype2ext(asset.get('mime_type')))
-                    if ext == 'm3u8':
-                        info['formats'].extend(self._extract_m3u8_formats(
-                            asset_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
-                    elif ext == 'f4m':
-                        continue
-                        # info['formats'].extend(self._extract_f4m_formats(
-                        #     asset_url, video_id, f4m_id='hds', fatal=False))
-                    elif ext in ('scc', 'ttml', 'vtt'):
-                        info['subtitles'].setdefault('en', []).append({
-                            'url': asset_url,
-                        })
-            self._sort_formats(info['formats'])
-
-            return info
+                    entries.append(self.url_result(
+                        episode_url, 'AdultSwim', episode.get('id')))
+                return self.playlist_result(
+                    entries, video_data.get('id'), video_data.get('title'),
+                    strip_or_none(video_data.get('description')))
         else:
-            entries = []
-            for edge in show_data.get('videos', {}).get('edges', []):
-                video = edge.get('node') or {}
-                slug = video.get('slug')
-                if not slug:
-                    continue
-                entries.append(self.url_result(
-                    'http://adultswim.com/videos/%s/%s' % (show_path, slug),
-                    'AdultSwim', video.get('_id')))
-            return self.playlist_result(
-                entries, show_path, show_data.get('title'),
-                strip_or_none(show_data.get('metaDescription')))
+            show_data = initial_data['show']
+
+            if not episode_path:
+                entries = []
+                for video in show_data.get('videos', []):
+                    slug = video.get('slug')
+                    if not slug:
+                        continue
+                    entries.append(self.url_result(
+                        'http://adultswim.com/videos/%s/%s' % (show_path, slug),
+                        'AdultSwim', video.get('id')))
+                return self.playlist_result(
+                    entries, show_data.get('id'), show_data.get('title'),
+                    strip_or_none(show_data.get('metadata', {}).get('description')))
+
+            video_data = show_data['sluggedVideo']
+            video_id = video_data['id']
+
+        info = self._extract_cvp_info(
+            'http://www.adultswim.com/videos/api/v0/assets?platform=desktop&id=' + video_id,
+            video_id, {
+                'secure': {
+                    'media_src': 'http://androidhls-secure.cdn.turner.com/adultswim/big',
+                    'tokenizer_src': 'http://www.adultswim.com/astv/mvpd/processors/services/token_ipadAdobe.do',
+                },
+            }, {
+                'url': url,
+                'site_name': 'AdultSwim',
+                'auth_required': video_data.get('auth'),
+            })
+
+        info.update({
+            'id': video_id,
+            'display_id': display_id,
+            'description': info.get('description') or strip_or_none(video_data.get('description')),
+        })
+        if not is_stream:
+            info.update({
+                'duration': info.get('duration') or int_or_none(video_data.get('duration')),
+                'timestamp': info.get('timestamp') or int_or_none(video_data.get('launch_date')),
+                'season_number': info.get('season_number') or int_or_none(video_data.get('season_number')),
+                'episode': info['title'],
+                'episode_number': info.get('episode_number') or int_or_none(video_data.get('episode_number')),
+            })
+
+            info['series'] = video_data.get('collection_title') or info.get('series')
+            if info['series'] and info['series'] != info['title']:
+                info['title'] = '%s - %s' % (info['series'], info['title'])
+
+        return info

youtube_dl/extractor/aenetworks.py

@@ -1,148 +1,70 @@
-# coding: utf-8
 from __future__ import unicode_literals
 
 import re
 
 from .theplatform import ThePlatformIE
 from ..utils import (
-    ExtractorError,
-    GeoRestrictedError,
-    int_or_none,
+    smuggle_url,
     update_url_query,
-    urlencode_postdata,
+    unescapeHTML,
+    extract_attributes,
+    get_element_by_attribute,
+)
+from ..compat import (
+    compat_urlparse,
 )
 
 
 class AENetworksBaseIE(ThePlatformIE):
-    _BASE_URL_REGEX = r'''(?x)https?://
-        (?:(?:www|play|watch)\.)?
-        (?P<domain>
-            (?:history(?:vault)?|aetv|mylifetime|lifetimemovieclub)\.com|
-            fyi\.tv
-        )/'''
-    _THEPLATFORM_KEY = '43jXaGRQud'
-    _THEPLATFORM_SECRET = 'S10BPXHMlb'
-    _DOMAIN_MAP = {
-        'history.com': ('HISTORY', 'history'),
-        'aetv.com': ('AETV', 'aetv'),
-        'mylifetime.com': ('LIFETIME', 'lifetime'),
-        'lifetimemovieclub.com': ('LIFETIMEMOVIECLUB', 'lmc'),
-        'fyi.tv': ('FYI', 'fyi'),
-        'historyvault.com': (None, 'historyvault'),
-        'biography.com': (None, 'biography'),
-    }
-
-    def _extract_aen_smil(self, smil_url, video_id, auth=None):
-        query = {'mbr': 'true'}
-        if auth:
-            query['auth'] = auth
-        TP_SMIL_QUERY = [{
-            'assetTypes': 'high_video_ak',
-            'switch': 'hls_high_ak'
-        }, {
-            'assetTypes': 'high_video_s3'
-        }, {
-            'assetTypes': 'high_video_s3',
-            'switch': 'hls_high_fastly',
-        }]
-        formats = []
-        subtitles = {}
-        last_e = None
-        for q in TP_SMIL_QUERY:
-            q.update(query)
-            m_url = update_url_query(smil_url, q)
-            m_url = self._sign_url(m_url, self._THEPLATFORM_KEY, self._THEPLATFORM_SECRET)
-            try:
-                tp_formats, tp_subtitles = self._extract_theplatform_smil(
-                    m_url, video_id, 'Downloading %s SMIL data' % (q.get('switch') or q['assetTypes']))
-            except ExtractorError as e:
-                if isinstance(e, GeoRestrictedError):
-                    raise
-                last_e = e
-                continue
-            formats.extend(tp_formats)
-            subtitles = self._merge_subtitles(subtitles, tp_subtitles)
-        if last_e and not formats:
-            raise last_e
-        self._sort_formats(formats)
-        return {
-            'id': video_id,
-            'formats': formats,
-            'subtitles': subtitles,
-        }
-
-    def _extract_aetn_info(self, domain, filter_key, filter_value, url):
-        requestor_id, brand = self._DOMAIN_MAP[domain]
-        result = self._download_json(
-            'https://feeds.video.aetnd.com/api/v2/%s/videos' % brand,
-            filter_value, query={'filter[%s]' % filter_key: filter_value})['results'][0]
-        title = result['title']
-        video_id = result['id']
-        media_url = result['publicUrl']
-        theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
-            r'https?://link\.theplatform\.com/s/([^?]+)', media_url, 'theplatform_path'), video_id)
-        info = self._parse_theplatform_metadata(theplatform_metadata)
-        auth = None
-        if theplatform_metadata.get('AETN$isBehindWall'):
-            resource = self._get_mvpd_resource(
-                requestor_id, theplatform_metadata['title'],
-                theplatform_metadata.get('AETN$PPL_pplProgramId') or theplatform_metadata.get('AETN$PPL_pplProgramId_OLD'),
-                theplatform_metadata['ratings'][0]['rating'])
-            auth = self._extract_mvpd_auth(
-                url, video_id, requestor_id, resource)
-        info.update(self._extract_aen_smil(media_url, video_id, auth))
-        info.update({
-            'title': title,
-            'series': result.get('seriesName'),
-            'season_number': int_or_none(result.get('tvSeasonNumber')),
-            'episode_number': int_or_none(result.get('tvSeasonEpisodeNumber')),
-        })
-        return info
+    _THEPLATFORM_KEY = 'crazyjava'
+    _THEPLATFORM_SECRET = 's3cr3t'
 
 
 class AENetworksIE(AENetworksBaseIE):
     IE_NAME = 'aenetworks'
-    IE_DESC = 'A+E Networks: A&E, Lifetime, History.com, FYI Network and History Vault'
-    _VALID_URL = AENetworksBaseIE._BASE_URL_REGEX + r'''(?P<id>
-        shows/[^/]+/season-\d+/episode-\d+|
-        (?:
-            (?:movie|special)s/[^/]+|
-            (?:shows/[^/]+/)?videos
-        )/[^/?#&]+
-    )'''
+    IE_DESC = 'A+E Networks: A&E, Lifetime, History.com, FYI Network'
+    _VALID_URL = r'''(?x)
+                    https?://
+                        (?:www\.)?
+                        (?P<domain>
+                            (?:history|aetv|mylifetime|lifetimemovieclub)\.com|
+                            fyi\.tv
+                        )/
+                        (?:
+                            shows/(?P<show_path>[^/]+(?:/[^/]+){0,2})|
+                            movies/(?P<movie_display_id>[^/]+)(?:/full-movie)?|
+                            specials/(?P<special_display_id>[^/]+)/full-special
+                        )
+                    '''
     _TESTS = [{
         'url': 'http://www.history.com/shows/mountain-men/season-1/episode-1',
+        'md5': 'a97a65f7e823ae10e9244bc5433d5fe6',
         'info_dict': {
             'id': '22253814',
             'ext': 'mp4',
-            'title': 'Winter is Coming',
+            'title': 'Winter Is Coming',
             'description': 'md5:641f424b7a19d8e24f26dea22cf59d74',
             'timestamp': 1338306241,
             'upload_date': '20120529',
             'uploader': 'AENE-NEW',
         },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
         'add_ie': ['ThePlatform'],
-        'skip': 'This video is only available for users of participating TV providers.',
+    }, {
+        'url': 'http://www.history.com/shows/ancient-aliens/season-1',
+        'info_dict': {
+            'id': '71889446852',
+        },
+        'playlist_mincount': 5,
+    }, {
+        'url': 'http://www.mylifetime.com/shows/atlanta-plastic',
+        'info_dict': {
+            'id': 'SERIES4317',
+            'title': 'Atlanta Plastic',
+        },
+        'playlist_mincount': 2,
     }, {
         'url': 'http://www.aetv.com/shows/duck-dynasty/season-9/episode-1',
-        'info_dict': {
-            'id': '600587331957',
-            'ext': 'mp4',
-            'title': 'Inlawful Entry',
-            'description': 'md5:57c12115a2b384d883fe64ca50529e08',
-            'timestamp': 1452634428,
-            'upload_date': '20160112',
-            'uploader': 'AENE-NEW',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-        'add_ie': ['ThePlatform'],
+        'only_matching': True
     }, {
         'url': 'http://www.fyi.tv/shows/tiny-house-nation/season-1/episode-8',
         'only_matching': True
@@ -153,137 +75,95 @@ class AENetworksIE(AENetworksBaseIE):
         'url': 'http://www.mylifetime.com/movies/center-stage-on-pointe/full-movie',
         'only_matching': True
     }, {
-        'url': 'https://watch.lifetimemovieclub.com/movies/10-year-reunion/full-movie',
+        'url': 'https://www.lifetimemovieclub.com/movies/a-killer-among-us',
         'only_matching': True
     }, {
         'url': 'http://www.history.com/specials/sniper-into-the-kill-zone/full-special',
         'only_matching': True
-    }, {
-        'url': 'https://www.aetv.com/specials/hunting-jonbenets-killer-the-untold-story/preview-hunting-jonbenets-killer-the-untold-story',
-        'only_matching': True
-    }, {
-        'url': 'http://www.history.com/videos/history-of-valentines-day',
-        'only_matching': True
-    }, {
-        'url': 'https://play.aetv.com/shows/duck-dynasty/videos/best-of-duck-dynasty-getting-quack-in-shape',
-        'only_matching': True
     }]
+    _DOMAIN_TO_REQUESTOR_ID = {
+        'history.com': 'HISTORY',
+        'aetv.com': 'AETV',
+        'mylifetime.com': 'LIFETIME',
+        'lifetimemovieclub.com': 'LIFETIMEMOVIECLUB',
+        'fyi.tv': 'FYI',
+    }
 
     def _real_extract(self, url):
-        domain, canonical = re.match(self._VALID_URL, url).groups()
-        return self._extract_aetn_info(domain, 'canonical', '/' + canonical, url)
+        domain, show_path, movie_display_id, special_display_id = re.match(self._VALID_URL, url).groups()
+        display_id = show_path or movie_display_id or special_display_id
+        webpage = self._download_webpage(url, display_id)
+        if show_path:
+            url_parts = show_path.split('/')
+            url_parts_len = len(url_parts)
+            if url_parts_len == 1:
+                entries = []
+                for season_url_path in re.findall(r'(?s)<li[^>]+data-href="(/shows/%s/season-\d+)"' % url_parts[0], webpage):
+                    entries.append(self.url_result(
+                        compat_urlparse.urljoin(url, season_url_path), 'AENetworks'))
+                if entries:
+                    return self.playlist_result(
+                        entries, self._html_search_meta('aetn:SeriesId', webpage),
+                        self._html_search_meta('aetn:SeriesTitle', webpage))
+                else:
+                    # single season
+                    url_parts_len = 2
+            if url_parts_len == 2:
+                entries = []
+                for episode_item in re.findall(r'(?s)<[^>]+class="[^"]*(?:episode|program)-item[^"]*"[^>]*>', webpage):
+                    episode_attributes = extract_attributes(episode_item)
+                    episode_url = compat_urlparse.urljoin(
+                        url, episode_attributes['data-canonical'])
+                    entries.append(self.url_result(
+                        episode_url, 'AENetworks',
+                        episode_attributes.get('data-videoid') or episode_attributes.get('data-video-id')))
+                return self.playlist_result(
+                    entries, self._html_search_meta('aetn:SeasonId', webpage))
 
-
-class AENetworksListBaseIE(AENetworksBaseIE):
-    def _call_api(self, resource, slug, brand, fields):
-        return self._download_json(
-            'https://yoga.appsvcs.aetnd.com/graphql',
-            slug, query={'brand': brand}, data=urlencode_postdata({
-                'query': '''{
-  %s(slug: "%s") {
-    %s
-  }
-}''' % (resource, slug, fields),
-            }))['data'][resource]
-
-    def _real_extract(self, url):
-        domain, slug = re.match(self._VALID_URL, url).groups()
-        _, brand = self._DOMAIN_MAP[domain]
-        playlist = self._call_api(self._RESOURCE, slug, brand, self._FIELDS)
-        base_url = 'http://watch.%s' % domain
-
-        entries = []
-        for item in (playlist.get(self._ITEMS_KEY) or []):
-            doc = self._get_doc(item)
-            canonical = doc.get('canonical')
-            if not canonical:
-                continue
-            entries.append(self.url_result(
-                base_url + canonical, AENetworksIE.ie_key(), doc.get('id')))
-
-        description = None
-        if self._PLAYLIST_DESCRIPTION_KEY:
-            description = playlist.get(self._PLAYLIST_DESCRIPTION_KEY)
-
-        return self.playlist_result(
-            entries, playlist.get('id'),
-            playlist.get(self._PLAYLIST_TITLE_KEY), description)
-
-
-class AENetworksCollectionIE(AENetworksListBaseIE):
-    IE_NAME = 'aenetworks:collection'
-    _VALID_URL = AENetworksBaseIE._BASE_URL_REGEX + r'(?:[^/]+/)*(?:list|collections)/(?P<id>[^/?#&]+)/?(?:[?#&]|$)'
-    _TESTS = [{
-        'url': 'https://watch.historyvault.com/list/america-the-story-of-us',
-        'info_dict': {
-            'id': '282',
-            'title': 'America The Story of Us',
-        },
-        'playlist_mincount': 12,
-    }, {
-        'url': 'https://watch.historyvault.com/shows/america-the-story-of-us-2/season-1/list/america-the-story-of-us',
-        'only_matching': True
-    }, {
-        'url': 'https://www.historyvault.com/collections/mysteryquest',
-        'only_matching': True
-    }]
-    _RESOURCE = 'list'
-    _ITEMS_KEY = 'items'
-    _PLAYLIST_TITLE_KEY = 'display_title'
-    _PLAYLIST_DESCRIPTION_KEY = None
-    _FIELDS = '''id
-    display_title
-    items {
-      ... on ListVideoItem {
-        doc {
-          canonical
-          id
+        query = {
+            'mbr': 'true',
+            'assetTypes': 'high_video_s3'
         }
-      }
-    }'''
-
-    def _get_doc(self, item):
-        return item.get('doc') or {}
-
-
-class AENetworksShowIE(AENetworksListBaseIE):
-    IE_NAME = 'aenetworks:show'
-    _VALID_URL = AENetworksBaseIE._BASE_URL_REGEX + r'shows/(?P<id>[^/?#&]+)/?(?:[?#&]|$)'
-    _TESTS = [{
-        'url': 'http://www.history.com/shows/ancient-aliens',
-        'info_dict': {
-            'id': 'SERIES1574',
-            'title': 'Ancient Aliens',
-            'description': 'md5:3f6d74daf2672ff3ae29ed732e37ea7f',
-        },
-        'playlist_mincount': 150,
-    }]
-    _RESOURCE = 'series'
-    _ITEMS_KEY = 'episodes'
-    _PLAYLIST_TITLE_KEY = 'title'
-    _PLAYLIST_DESCRIPTION_KEY = 'description'
-    _FIELDS = '''description
-    id
-    title
-    episodes {
-      canonical
-      id
-    }'''
-
-    def _get_doc(self, item):
-        return item
+        video_id = self._html_search_meta('aetn:VideoID', webpage)
+        media_url = self._search_regex(
+            [r"media_url\s*=\s*'(?P<url>[^']+)'",
+             r'data-media-url=(?P<url>(?:https?:)?//[^\s>]+)',
+             r'data-media-url=(["\'])(?P<url>(?:(?!\1).)+?)\1'],
+            webpage, 'video url', group='url')
+        theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
+            r'https?://link\.theplatform\.com/s/([^?]+)', media_url, 'theplatform_path'), video_id)
+        info = self._parse_theplatform_metadata(theplatform_metadata)
+        if theplatform_metadata.get('AETN$isBehindWall'):
+            requestor_id = self._DOMAIN_TO_REQUESTOR_ID[domain]
+            resource = self._get_mvpd_resource(
+                requestor_id, theplatform_metadata['title'],
+                theplatform_metadata.get('AETN$PPL_pplProgramId') or theplatform_metadata.get('AETN$PPL_pplProgramId_OLD'),
+                theplatform_metadata['ratings'][0]['rating'])
+            query['auth'] = self._extract_mvpd_auth(
+                url, video_id, requestor_id, resource)
+        info.update(self._search_json_ld(webpage, video_id, fatal=False))
+        media_url = update_url_query(media_url, query)
+        media_url = self._sign_url(media_url, self._THEPLATFORM_KEY, self._THEPLATFORM_SECRET)
+        formats, subtitles = self._extract_theplatform_smil(media_url, video_id)
+        self._sort_formats(formats)
+        info.update({
+            'id': video_id,
+            'formats': formats,
+            'subtitles': subtitles,
+        })
+        return info
 
 
 class HistoryTopicIE(AENetworksBaseIE):
     IE_NAME = 'history:topic'
     IE_DESC = 'History.com Topic'
-    _VALID_URL = r'https?://(?:www\.)?history\.com/topics/[^/]+/(?P<id>[\w+-]+?)-video'
+    _VALID_URL = r'https?://(?:www\.)?history\.com/topics/(?:[^/]+/)?(?P<topic_id>[^/]+)(?:/[^/]+(?:/(?P<video_display_id>[^/?#]+))?)?'
     _TESTS = [{
-        'url': 'https://www.history.com/topics/valentines-day/history-of-valentines-day-video',
+        'url': 'http://www.history.com/topics/valentines-day/history-of-valentines-day/videos/bet-you-didnt-know-valentines-day?m=528e394da93ae&s=undefined&f=1&free=false',
         'info_dict': {
             'id': '40700995724',
             'ext': 'mp4',
-            'title': "History of Valentine’s Day",
+            'title': "Bet You Didn't Know: Valentine's Day",
             'description': 'md5:7b57ea4829b391995b405fa60bd7b5f7',
             'timestamp': 1375819729,
             'upload_date': '20130806',
@@ -294,49 +174,63 @@ class HistoryTopicIE(AENetworksBaseIE):
             'skip_download': True,
         },
         'add_ie': ['ThePlatform'],
+    }, {
+        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history/videos',
+        'info_dict':
+        {
+            'id': 'world-war-i-history',
+            'title': 'World War I History',
+        },
+        'playlist_mincount': 23,
+    }, {
+        'url': 'http://www.history.com/topics/world-war-i-history/videos',
+        'only_matching': True,
+    }, {
+        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history',
+        'only_matching': True,
+    }, {
+        'url': 'http://www.history.com/topics/world-war-i/world-war-i-history/speeches',
+        'only_matching': True,
     }]
 
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        return self.url_result(
-            'http://www.history.com/videos/' + display_id,
-            AENetworksIE.ie_key())
-
-
-class HistoryPlayerIE(AENetworksBaseIE):
-    IE_NAME = 'history:player'
-    _VALID_URL = r'https?://(?:www\.)?(?P<domain>(?:history|biography)\.com)/player/(?P<id>\d+)'
-    _TESTS = []
+    def theplatform_url_result(self, theplatform_url, video_id, query):
+        return {
+            '_type': 'url_transparent',
+            'id': video_id,
+            'url': smuggle_url(
+                update_url_query(theplatform_url, query),
+                {
+                    'sig': {
+                        'key': self._THEPLATFORM_KEY,
+                        'secret': self._THEPLATFORM_SECRET,
+                    },
+                    'force_smil_url': True
+                }),
+            'ie_key': 'ThePlatform',
+        }
 
     def _real_extract(self, url):
-        domain, video_id = re.match(self._VALID_URL, url).groups()
-        return self._extract_aetn_info(domain, 'id', video_id, url)
+        topic_id, video_display_id = re.match(self._VALID_URL, url).groups()
+        if video_display_id:
+            webpage = self._download_webpage(url, video_display_id)
+            release_url, video_id = re.search(r"_videoPlayer.play\('([^']+)'\s*,\s*'[^']+'\s*,\s*'(\d+)'\)", webpage).groups()
+            release_url = unescapeHTML(release_url)
 
-
-class BiographyIE(AENetworksBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?biography\.com/video/(?P<id>[^/?#&]+)'
-    _TESTS = [{
-        'url': 'https://www.biography.com/video/vincent-van-gogh-full-episode-2075049808',
-        'info_dict': {
-            'id': '30322987',
-            'ext': 'mp4',
-            'title': 'Vincent Van Gogh - Full Episode',
-            'description': 'A full biography about the most influential 20th century painter, Vincent Van Gogh.',
-            'timestamp': 1311970571,
-            'upload_date': '20110729',
-            'uploader': 'AENE-NEW',
-        },
-        'params': {
-            # m3u8 download
-            'skip_download': True,
-        },
-        'add_ie': ['ThePlatform'],
-    }]
-
-    def _real_extract(self, url):
-        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
-        player_url = self._search_regex(
-            r'<phoenix-iframe[^>]+src="(%s)' % HistoryPlayerIE._VALID_URL,
-            webpage, 'player URL')
-        return self.url_result(player_url, HistoryPlayerIE.ie_key())
+            return self.theplatform_url_result(
+                release_url, video_id, {
+                    'mbr': 'true',
+                    'switch': 'hls',
+                    'assetTypes': 'high_video_ak',
+                })
+        else:
+            webpage = self._download_webpage(url, topic_id)
+            entries = []
+            for episode_item in re.findall(r'<a.+?data-release-url="[^"]+"[^>]*>', webpage):
+                video_attributes = extract_attributes(episode_item)
+                entries.append(self.theplatform_url_result(
+                    video_attributes['data-release-url'], video_attributes['data-id'], {
+                        'mbr': 'true',
+                        'switch': 'hls',
+                        'assetTypes': 'high_video_ak',
+                    }))
+            return self.playlist_result(entries, topic_id, get_element_by_attribute('class', 'show-title', webpage))

youtube_dl/extractor/afreecatv.py

@@ -9,8 +9,6 @@ from ..utils import (
     determine_ext,
     ExtractorError,
     int_or_none,
-    url_or_none,
-    urlencode_postdata,
     xpath_text,
 )
 
@@ -30,7 +28,6 @@ class AfreecaTVIE(InfoExtractor):
                         )
                         (?P<id>\d+)
                     '''
-    _NETRC_MACHINE = 'afreecatv'
     _TESTS = [{
         'url': 'http://live.afreecatv.com:8079/app/index.cgi?szType=read_ucc_bbs&szBjId=dailyapril&nStationNo=16711924&nBbsNo=18605867&nTitleNo=36164052&szSkin=',
         'md5': 'f72c89fe7ecc14c1b5ce506c4996046e',
@@ -142,22 +139,22 @@ class AfreecaTVIE(InfoExtractor):
             'skip_download': True,
         },
     }, {
-        # PARTIAL_ADULT
-        'url': 'http://vod.afreecatv.com/PLAYER/STATION/32028439',
+        # adult video
+        'url': 'http://vod.afreecatv.com/PLAYER/STATION/26542731',
         'info_dict': {
-            'id': '20180327_27901457_202289533_1',
+            'id': '20171001_F1AE1711_196617479_1',
             'ext': 'mp4',
-            'title': '[생]빨개요♥ (part 1)',
+            'title': '[생]서아 초심 찾기 방송 (part 1)',
             'thumbnail': 're:^https?://(?:video|st)img.afreecatv.com/.*$',
-            'uploader': '[SA]서아',
+            'uploader': 'BJ서아',
             'uploader_id': 'bjdyrksu',
-            'upload_date': '20180327',
-            'duration': 3601,
+            'upload_date': '20171001',
+            'duration': 3600,
+            'age_limit': 18,
         },
         'params': {
             'skip_download': True,
         },
-        'expected_warnings': ['adult content'],
     }, {
         'url': 'http://www.afreecatv.com/player/Player.swf?szType=szBjId=djleegoon&nStationNo=11273158&nBbsNo=13161095&nTitleNo=36327652',
         'only_matching': True,
@@ -175,107 +172,25 @@ class AfreecaTVIE(InfoExtractor):
             video_key['part'] = int(m.group('part'))
         return video_key
 
-    def _real_initialize(self):
-        self._login()
-
-    def _login(self):
-        username, password = self._get_login_info()
-        if username is None:
-            return
-
-        login_form = {
-            'szWork': 'login',
-            'szType': 'json',
-            'szUid': username,
-            'szPassword': password,
-            'isSaveId': 'false',
-            'szScriptVar': 'oLoginRet',
-            'szAction': '',
-        }
-
-        response = self._download_json(
-            'https://login.afreecatv.com/app/LoginAction.php', None,
-            'Logging in', data=urlencode_postdata(login_form))
-
-        _ERRORS = {
-            -4: 'Your account has been suspended due to a violation of our terms and policies.',
-            -5: 'https://member.afreecatv.com/app/user_delete_progress.php',
-            -6: 'https://login.afreecatv.com/membership/changeMember.php',
-            -8: "Hello! AfreecaTV here.\nThe username you have entered belongs to \n an account that requires a legal guardian's consent. \nIf you wish to use our services without restriction, \nplease make sure to go through the necessary verification process.",
-            -9: 'https://member.afreecatv.com/app/pop_login_block.php',
-            -11: 'https://login.afreecatv.com/afreeca/second_login.php',
-            -12: 'https://member.afreecatv.com/app/user_security.php',
-            0: 'The username does not exist or you have entered the wrong password.',
-            -1: 'The username does not exist or you have entered the wrong password.',
-            -3: 'You have entered your username/password incorrectly.',
-            -7: 'You cannot use your Global AfreecaTV account to access Korean AfreecaTV.',
-            -10: 'Sorry for the inconvenience. \nYour account has been blocked due to an unauthorized access. \nPlease contact our Help Center for assistance.',
-            -32008: 'You have failed to log in. Please contact our Help Center.',
-        }
-
-        result = int_or_none(response.get('RESULT'))
-        if result != 1:
-            error = _ERRORS.get(result, 'You have failed to log in.')
-            raise ExtractorError(
-                'Unable to login: %s said: %s' % (self.IE_NAME, error),
-                expected=True)
-
     def _real_extract(self, url):
         video_id = self._match_id(url)
 
-        webpage = self._download_webpage(url, video_id)
-
-        if re.search(r'alert\(["\']This video has been deleted', webpage):
-            raise ExtractorError(
-                'Video %s has been deleted' % video_id, expected=True)
-
-        station_id = self._search_regex(
-            r'nStationNo\s*=\s*(\d+)', webpage, 'station')
-        bbs_id = self._search_regex(
-            r'nBbsNo\s*=\s*(\d+)', webpage, 'bbs')
-        video_id = self._search_regex(
-            r'nTitleNo\s*=\s*(\d+)', webpage, 'title', default=video_id)
-
-        partial_view = False
-        for _ in range(2):
-            query = {
+        video_xml = self._download_xml(
+            'http://afbbs.afreecatv.com:8080/api/video/get_video_info.php',
+            video_id, query={
                 'nTitleNo': video_id,
-                'nStationNo': station_id,
-                'nBbsNo': bbs_id,
-            }
-            if partial_view:
-                query['partialView'] = 'SKIP_ADULT'
-            video_xml = self._download_xml(
-                'http://afbbs.afreecatv.com:8080/api/video/get_video_info.php',
-                video_id, 'Downloading video info XML%s'
-                % (' (skipping adult)' if partial_view else ''),
-                video_id, headers={
-                    'Referer': url,
-                }, query=query)
+                'partialView': 'SKIP_ADULT',
+            })
 
-            flag = xpath_text(video_xml, './track/flag', 'flag', default=None)
-            if flag and flag == 'SUCCEED':
-                break
-            if flag == 'PARTIAL_ADULT':
-                self._downloader.report_warning(
-                    'In accordance with local laws and regulations, underage users are restricted from watching adult content. '
-                    'Only content suitable for all ages will be downloaded. '
-                    'Provide account credentials if you wish to download restricted content.')
-                partial_view = True
-                continue
-            elif flag == 'ADULT':
-                error = 'Only users older than 19 are able to watch this video. Provide account credentials to download this content.'
-            else:
-                error = flag
+        flag = xpath_text(video_xml, './track/flag', 'flag', default=None)
+        if flag and flag != 'SUCCEED':
             raise ExtractorError(
-                '%s said: %s' % (self.IE_NAME, error), expected=True)
-        else:
-            raise ExtractorError('Unable to download video info')
+                '%s said: %s' % (self.IE_NAME, flag), expected=True)
 
-        video_element = video_xml.findall(compat_xpath('./track/video'))[-1]
+        video_element = video_xml.findall(compat_xpath('./track/video'))[1]
         if video_element is None or video_element.text is None:
-            raise ExtractorError(
-                'Video %s does not exist' % video_id, expected=True)
+            raise ExtractorError('Specified AfreecaTV video does not exist',
+                                 expected=True)
 
         video_url = video_element.text.strip()
 
@@ -305,7 +220,7 @@ class AfreecaTVIE(InfoExtractor):
             file_elements = video_element.findall(compat_xpath('./file'))
             one = len(file_elements) == 1
             for file_num, file_element in enumerate(file_elements, start=1):
-                file_url = url_or_none(file_element.text)
+                file_url = file_element.text
                 if not file_url:
                     continue
                 key = file_element.get('key', '')

youtube_dl/extractor/aliexpress.py

@@ -18,7 +18,7 @@ class AliExpressLiveIE(InfoExtractor):
             'id': '2800002704436634',
             'ext': 'mp4',
             'title': 'CASIMA7.22',
-            'thumbnail': r're:https?://.*\.jpg',
+            'thumbnail': r're:http://.*\.jpg',
             'uploader': 'CASIMA Official Store',
             'timestamp': 1500717600,
             'upload_date': '20170722',

youtube_dl/extractor/aljazeera.py

@@ -1,16 +1,13 @@
 from __future__ import unicode_literals
 
-import json
-import re
-
 from .common import InfoExtractor
 
 
 class AlJazeeraIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?aljazeera\.com/(?P<type>program/[^/]+|(?:feature|video)s)/\d{4}/\d{1,2}/\d{1,2}/(?P<id>[^/?&#]+)'
+    _VALID_URL = r'https?://(?:www\.)?aljazeera\.com/(?:programmes|video)/.*?/(?P<id>[^/]+)\.html'
 
     _TESTS = [{
-        'url': 'https://www.aljazeera.com/program/episode/2014/9/19/deliverance',
+        'url': 'http://www.aljazeera.com/programmes/the-slum/2014/08/deliverance-201482883754237240.html',
         'info_dict': {
             'id': '3792260579001',
             'ext': 'mp4',
@@ -23,34 +20,14 @@ class AlJazeeraIE(InfoExtractor):
         'add_ie': ['BrightcoveNew'],
         'skip': 'Not accessible from Travis CI server',
     }, {
-        'url': 'https://www.aljazeera.com/videos/2017/5/11/sierra-leone-709-carat-diamond-to-be-auctioned-off',
-        'only_matching': True,
-    }, {
-        'url': 'https://www.aljazeera.com/features/2017/8/21/transforming-pakistans-buses-into-art',
+        'url': 'http://www.aljazeera.com/video/news/2017/05/sierra-leone-709-carat-diamond-auctioned-170511100111930.html',
         'only_matching': True,
     }]
-    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/%s/%s_default/index.html?videoId=%s'
+    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/665003303001/default_default/index.html?videoId=%s'
 
     def _real_extract(self, url):
-        post_type, name = re.match(self._VALID_URL, url).groups()
-        post_type = {
-            'features': 'post',
-            'program': 'episode',
-            'videos': 'video',
-        }[post_type.split('/')[0]]
-        video = self._download_json(
-            'https://www.aljazeera.com/graphql', name, query={
-                'operationName': 'SingleArticleQuery',
-                'variables': json.dumps({
-                    'name': name,
-                    'postType': post_type,
-                }),
-            }, headers={
-                'wp-site': 'aje',
-            })['data']['article']['video']
-        video_id = video['id']
-        account_id = video.get('accountId') or '665003303001'
-        player_id = video.get('playerId') or 'BkeSH5BDb'
-        return self.url_result(
-            self.BRIGHTCOVE_URL_TEMPLATE % (account_id, player_id, video_id),
-            'BrightcoveNew', video_id)
+        program_name = self._match_id(url)
+        webpage = self._download_webpage(url, program_name)
+        brightcove_id = self._search_regex(
+            r'RenderPagesVideo\(\'(.+?)\'', webpage, 'brightcove id')
+        return self.url_result(self.BRIGHTCOVE_URL_TEMPLATE % brightcove_id, 'BrightcoveNew', brightcove_id)

youtube_dl/extractor/alsace20tv.py

@@ -1,89 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..utils import (
-    clean_html,
-    dict_get,
-    get_element_by_class,
-    int_or_none,
-    unified_strdate,
-    url_or_none,
-)
-
-
-class Alsace20TVIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?alsace20\.tv/(?:[\w-]+/)+[\w-]+-(?P<id>[\w]+)'
-    _TESTS = [{
-        'url': 'https://www.alsace20.tv/VOD/Actu/JT/Votre-JT-jeudi-3-fevrier-lyNHCXpYJh.html',
-        # 'md5': 'd91851bf9af73c0ad9b2cdf76c127fbb',
-        'info_dict': {
-            'id': 'lyNHCXpYJh',
-            'ext': 'mp4',
-            'description': 'md5:fc0bc4a0692d3d2dba4524053de4c7b7',
-            'title': 'Votre JT du jeudi 3 février',
-            'upload_date': '20220203',
-            'thumbnail': r're:https?://.+\.jpg',
-            'duration': 1073,
-            'view_count': int,
-        },
-        'params': {
-            'format': 'bestvideo',
-        },
-    }]
-
-    def _extract_video(self, video_id, url=None):
-        info = self._download_json(
-            'https://www.alsace20.tv/visionneuse/visio_v9_js.php?key=%s&habillage=0&mode=html' % (video_id, ),
-            video_id) or {}
-        title = info['titre']
-
-        formats = []
-        for res, fmt_url in (info.get('files') or {}).items():
-            formats.extend(
-                self._extract_smil_formats(fmt_url, video_id, fatal=False)
-                if '/smil:_' in fmt_url
-                else self._extract_mpd_formats(fmt_url, video_id, mpd_id=res, fatal=False))
-        self._sort_formats(formats)
-
-        webpage = (url and self._download_webpage(url, video_id, fatal=False)) or ''
-        thumbnail = url_or_none(dict_get(info, ('image', 'preview', )) or self._og_search_thumbnail(webpage))
-        upload_date = self._search_regex(r'/(\d{6})_', thumbnail, 'upload_date', default=None)
-        upload_date = unified_strdate('20%s-%s-%s' % (upload_date[:2], upload_date[2:4], upload_date[4:])) if upload_date else None
-        return {
-            'id': video_id,
-            'title': title,
-            'formats': formats,
-            'description': clean_html(get_element_by_class('wysiwyg', webpage)),
-            'upload_date': upload_date,
-            'thumbnail': thumbnail,
-            'duration': int_or_none(self._og_search_property('video:duration', webpage) if webpage else None),
-            'view_count': int_or_none(info.get('nb_vues')),
-        }
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        return self._extract_video(video_id, url)
-
-
-class Alsace20TVEmbedIE(Alsace20TVIE):
-    _VALID_URL = r'https?://(?:www\.)?alsace20\.tv/emb/(?P<id>[\w]+)'
-    _TESTS = [{
-        'url': 'https://www.alsace20.tv/emb/lyNHCXpYJh',
-        # 'md5': 'd91851bf9af73c0ad9b2cdf76c127fbb',
-        'info_dict': {
-            'id': 'lyNHCXpYJh',
-            'ext': 'mp4',
-            'title': 'Votre JT du jeudi 3 février',
-            'upload_date': '20220203',
-            'thumbnail': r're:https?://.+\.jpg',
-            'view_count': int,
-        },
-        'params': {
-            'format': 'bestvideo',
-        },
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        return self._extract_video(video_id)

youtube_dl/extractor/amara.py

@@ -1,103 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from .youtube import YoutubeIE
-from .vimeo import VimeoIE
-from ..utils import (
-    int_or_none,
-    parse_iso8601,
-    update_url_query,
-)
-
-
-class AmaraIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?amara\.org/(?:\w+/)?videos/(?P<id>\w+)'
-    _TESTS = [{
-        # Youtube
-        'url': 'https://amara.org/en/videos/jVx79ZKGK1ky/info/why-jury-trials-are-becoming-less-common/?tab=video',
-        'md5': 'ea10daf2b6154b8c1ecf9922aca5e8ae',
-        'info_dict': {
-            'id': 'h6ZuVdvYnfE',
-            'ext': 'mp4',
-            'title': 'Why jury trials are becoming less common',
-            'description': 'md5:a61811c319943960b6ab1c23e0cbc2c1',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'subtitles': dict,
-            'upload_date': '20160813',
-            'uploader': 'PBS NewsHour',
-            'uploader_id': 'PBSNewsHour',
-            'timestamp': 1549639570,
-        }
-    }, {
-        # Vimeo
-        'url': 'https://amara.org/en/videos/kYkK1VUTWW5I/info/vimeo-at-ces-2011',
-        'md5': '99392c75fa05d432a8f11df03612195e',
-        'info_dict': {
-            'id': '18622084',
-            'ext': 'mov',
-            'title': 'Vimeo at CES 2011!',
-            'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'subtitles': dict,
-            'timestamp': 1294763658,
-            'upload_date': '20110111',
-            'uploader': 'Sam Morrill',
-            'uploader_id': 'sammorrill'
-        }
-    }, {
-        # Direct Link
-        'url': 'https://amara.org/en/videos/s8KL7I3jLmh6/info/the-danger-of-a-single-story/',
-        'md5': 'd3970f08512738ee60c5807311ff5d3f',
-        'info_dict': {
-            'id': 's8KL7I3jLmh6',
-            'ext': 'mp4',
-            'title': 'The danger of a single story',
-            'description': 'md5:d769b31139c3b8bb5be9177f62ea3f23',
-            'thumbnail': r're:^https?://.*\.jpg$',
-            'subtitles': dict,
-            'upload_date': '20091007',
-            'timestamp': 1254942511,
-        }
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        meta = self._download_json(
-            'https://amara.org/api/videos/%s/' % video_id,
-            video_id, query={'format': 'json'})
-        title = meta['title']
-        video_url = meta['all_urls'][0]
-
-        subtitles = {}
-        for language in (meta.get('languages') or []):
-            subtitles_uri = language.get('subtitles_uri')
-            if not (subtitles_uri and language.get('published')):
-                continue
-            subtitle = subtitles.setdefault(language.get('code') or 'en', [])
-            for f in ('json', 'srt', 'vtt'):
-                subtitle.append({
-                    'ext': f,
-                    'url': update_url_query(subtitles_uri, {'format': f}),
-                })
-
-        info = {
-            'url': video_url,
-            'id': video_id,
-            'subtitles': subtitles,
-            'title': title,
-            'description': meta.get('description'),
-            'thumbnail': meta.get('thumbnail'),
-            'duration': int_or_none(meta.get('duration')),
-            'timestamp': parse_iso8601(meta.get('created')),
-        }
-
-        for ie in (YoutubeIE, VimeoIE):
-            if ie.suitable(video_url):
-                info.update({
-                    '_type': 'url_transparent',
-                    'ie_key': ie.ie_key(),
-                })
-                break
-
-        return info

youtube_dl/extractor/amcnetworks.py

@@ -1,8 +1,6 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-import re
-
 from .theplatform import ThePlatformIE
 from ..utils import (
     int_or_none,
@@ -13,22 +11,25 @@ from ..utils import (
 
 
 class AMCNetworksIE(ThePlatformIE):
-    _VALID_URL = r'https?://(?:www\.)?(?P<site>amc|bbcamerica|ifc|(?:we|sundance)tv)\.com/(?P<id>(?:movies|shows(?:/[^/]+)+)/[^/?#&]+)'
+    _VALID_URL = r'https?://(?:www\.)?(?:amc|bbcamerica|ifc|wetv)\.com/(?:movies|shows(?:/[^/]+)+)/(?P<id>[^/?#]+)'
     _TESTS = [{
-        'url': 'https://www.bbcamerica.com/shows/the-graham-norton-show/videos/tina-feys-adorable-airline-themed-family-dinner--51631',
+        'url': 'http://www.ifc.com/shows/maron/season-04/episode-01/step-1',
+        'md5': '',
         'info_dict': {
-            'id': '4Lq1dzOnZGt0',
+            'id': 's3MX01Nl4vPH',
             'ext': 'mp4',
-            'title': "The Graham Norton Show - Season 28 - Tina Fey's Adorable Airline-Themed Family Dinner",
-            'description': "It turns out child stewardesses are very generous with the wine! All-new episodes of 'The Graham Norton Show' premiere Fridays at 11/10c on BBC America.",
-            'upload_date': '20201120',
-            'timestamp': 1605904350,
+            'title': 'Maron - Season 4 - Step 1',
+            'description': 'In denial about his current situation, Marc is reluctantly convinced by his friends to enter rehab. Starring Marc Maron and Constance Zimmer.',
+            'age_limit': 17,
+            'upload_date': '20160505',
+            'timestamp': 1462468831,
             'uploader': 'AMCN',
         },
         'params': {
             # m3u8 download
             'skip_download': True,
         },
+        'skip': 'Requires TV provider accounts',
     }, {
         'url': 'http://www.bbcamerica.com/shows/the-hunt/full-episodes/season-1/episode-01-the-hardest-challenge',
         'only_matching': True,
@@ -50,38 +51,33 @@ class AMCNetworksIE(ThePlatformIE):
     }, {
         'url': 'http://www.wetv.com/shows/la-hair/videos/season-05/episode-09-episode-9-2/episode-9-sneak-peek-3',
         'only_matching': True,
-    }, {
-        'url': 'https://www.sundancetv.com/shows/riviera/full-episodes/season-1/episode-01-episode-1',
-        'only_matching': True,
     }]
-    _REQUESTOR_ID_MAP = {
-        'amc': 'AMC',
-        'bbcamerica': 'BBCA',
-        'ifc': 'IFC',
-        'sundancetv': 'SUNDANCE',
-        'wetv': 'WETV',
-    }
 
     def _real_extract(self, url):
-        site, display_id = re.match(self._VALID_URL, url).groups()
-        requestor_id = self._REQUESTOR_ID_MAP[site]
-        properties = self._download_json(
-            'https://content-delivery-gw.svc.ds.amcn.com/api/v2/content/amcn/%s/url/%s' % (requestor_id.lower(), display_id),
-            display_id)['data']['properties']
+        display_id = self._match_id(url)
+        webpage = self._download_webpage(url, display_id)
         query = {
             'mbr': 'true',
             'manifest': 'm3u',
         }
-        tp_path = 'M_UwQC/media/' + properties['videoPid']
-        media_url = 'https://link.theplatform.com/s/' + tp_path
-        theplatform_metadata = self._download_theplatform_metadata(tp_path, display_id)
+        media_url = self._search_regex(
+            r'window\.platformLinkURL\s*=\s*[\'"]([^\'"]+)',
+            webpage, 'media url')
+        theplatform_metadata = self._download_theplatform_metadata(self._search_regex(
+            r'link\.theplatform\.com/s/([^?]+)',
+            media_url, 'theplatform_path'), display_id)
         info = self._parse_theplatform_metadata(theplatform_metadata)
         video_id = theplatform_metadata['pid']
         title = theplatform_metadata['title']
         rating = try_get(
             theplatform_metadata, lambda x: x['ratings'][0]['rating'])
-        video_category = properties.get('videoCategory')
-        if video_category and video_category.endswith('-Auth'):
+        auth_required = self._search_regex(
+            r'window\.authRequired\s*=\s*(true|false);',
+            webpage, 'auth required')
+        if auth_required == 'true':
+            requestor_id = self._search_regex(
+                r'window\.requestor_id\s*=\s*[\'"]([^\'"]+)',
+                webpage, 'requestor id')
             resource = self._get_mvpd_resource(
                 requestor_id, title, video_id, rating)
             query['auth'] = self._extract_mvpd_auth(