Release 2021.07.24

Based on: https://github.com/ytdl-org/youtube-dl/pull/29487 by glenn-slayden

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.07. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running yt-dlp version **2021.07.07**
+- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -44,7 +44,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.07.07
+ [debug] yt-dlp version 2021.07.21
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.07. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://github.com/yt-dlp/yt-dlp. yt-dlp does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -29,7 +29,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running yt-dlp version **2021.07.07**
+- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.07. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.07.07**
+- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -21,7 +21,7 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.07. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in https://github.com/yt-dlp/yt-dlp.
 - Search the bugtracker for similar issues: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
@@ -30,7 +30,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running yt-dlp version **2021.07.07**
+- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -46,7 +46,7 @@ Add the `-v` flag to your command line you run yt-dlp with (`yt-dlp -v <your com
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKc']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] yt-dlp version 2021.07.07
+ [debug] yt-dlp version 2021.07.21
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -21,13 +21,13 @@ assignees: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of yt-dlp:
-- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.07. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of yt-dlp. Run `yt-dlp --version` and ensure your version is 2021.07.21. If it's not, see https://github.com/yt-dlp/yt-dlp on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: https://github.com/yt-dlp/yt-dlp. DO NOT post duplicates.
 - Finally, put x into all relevant boxes like this [x] (Dont forget to delete the empty space)
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running yt-dlp version **2021.07.07**
+- [ ] I've verified that I'm running yt-dlp version **2021.07.21**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

.github/workflows/core.yml

@@ -10,7 +10,7 @@ jobs:
       matrix:
         os: [ubuntu-18.04]
         # py3.9 is in quick-test
-        python-version: [3.7, 3.8, pypy-3.6, pypy-3.7]
+        python-version: [3.7, 3.8, 3.10-dev, pypy-3.6, pypy-3.7]
         run-tests-ext: [sh]
         include:
         # atleast one of the tests must be in windows
@@ -23,11 +23,9 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install nose
-      run: pip install nose
+    - name: Install pytest
+      run: pip install pytest
     - name: Run tests
       continue-on-error: False
-      env:
-        YTDL_TEST_SET: core
-      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
+      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }} core
   # Linter is in quick-test

.github/workflows/download.yml

@@ -9,7 +9,7 @@ jobs:
       fail-fast: true
       matrix:
         os: [ubuntu-18.04]
-        python-version: [3.7, 3.8, 3.9, pypy-3.6, pypy-3.7]
+        python-version: [3.7, 3.8, 3.9, 3.10-dev, pypy-3.6, pypy-3.7]
         run-tests-ext: [sh]
         include:
         - os: windows-latest
@@ -21,10 +21,8 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install nose
-      run: pip install nose
+    - name: Install pytest
+      run: pip install pytest
     - name: Run tests
       continue-on-error: true
-      env:
-        YTDL_TEST_SET: download
-      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }}
+      run: ./devscripts/run_tests.${{ matrix.run-tests-ext }} download

.github/workflows/quick-test.yml

@@ -12,11 +12,9 @@ jobs:
       with:
         python-version: 3.9
     - name: Install test requirements
-      run: pip install nose pycryptodome
+      run: pip install pytest pycryptodome
     - name: Run tests
-      env:
-        YTDL_TEST_SET: core
-      run: ./devscripts/run_tests.sh
+      run: ./devscripts/run_tests.sh core
   flake8:
     name: Linter
     if: "!contains(github.event.head_commit.message, 'ci skip all')"
@@ -30,4 +28,4 @@ jobs:
     - name: Install flake8
       run: pip install flake8
     - name: Run flake8
-      run: flake8 .
+      run: flake8 .

.gitignore

@@ -45,6 +45,7 @@ cookies.txt
 # Python
 *.pyc
 *.pyo
+.pytest_cache
 wine-py2exe/
 py2exe.log
 build/
@@ -79,6 +80,7 @@ README.txt
 *.tar.gz
 *.zsh
 *.spec
+test/testdata/player-*.js
 
 # Binary
 /youtube-dl

CONTRIBUTING.md

@@ -81,16 +81,17 @@ To run the test, simply invoke your favorite test runner, or execute a test file
     python -m unittest discover
     python test/test_download.py
     nosetests
+    pytest
 
 See item 6 of [new extractor tutorial](#adding-support-for-a-new-site) for how to run extractor specific test cases.
 
 If you want to create a build of youtube-dl yourself, you'll need
 
-* python
+* python3
 * make (only GNU make is supported)
 * pandoc
 * zip
-* nosetests
+* pytest
 
 ### Adding support for a new site
 

Changelog.md

@@ -18,6 +18,20 @@
 
 -->
 
+
+### 2021.07.24
+
+* [youtube:tab] Extract video duration early
+* [downloader] Pass `info_dict` to `progress_hook`s
+* [youtube] Fix age-gated videos for API clients when cookies are supplied by [colethedj](https://github.com/colethedj)
+* [youtube] Disable `get_video_info` age-gate workaround - This endpoint seems to be completely dead
+* [youtube] Try all clients even if age-gated
+* [youtube] Fix subtitles only being extracted from the first client
+* [youtube] Simplify `_get_text`
+* [cookies] bugfix for microsoft edge on macOS
+* [cookies] Handle `sqlite` `ImportError` gracefully by [mbway](https://github.com/mbway)
+* [cookies] Handle errors when importing `keyring`
+
 ### 2021.07.21
 
 * **Add option `--cookies-from-browser`** to load cookies from a browser by [mbway](https://github.com/mbway)
@@ -53,6 +67,7 @@
 * [youtube:tab] Fix channels tab
 * [youtube:tab] Extract playlist availability by [colethedj](https://github.com/colethedj)
 * **[youtube:comments] Move comment extraction to new API** by [colethedj](https://github.com/colethedj)
+    * Adds extractor-args `comment_sort` (`top`/`new`), `max_comments`, `max_comment_depth`
 * [youtube:comments] Fix `is_favorited`, improve `like_count` parsing by [colethedj](https://github.com/colethedj)
 * [BravoTV] Improve metadata extraction by [kevinoconnor7](https://github.com/kevinoconnor7)
 * [crunchyroll:playlist] Force http

Makefile

@@ -13,7 +13,7 @@ pypi-files: AUTHORS Changelog.md LICENSE README.md README.txt supportedsites com
 .PHONY: all clean install test tar pypi-files completions ot offlinetest codetest supportedsites
 
 clean-test:
-	rm -rf *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.frag *.frag.urls *.frag.aria2
+	rm -rf *.dump *.part* *.ytdl *.info.json *.mp4 *.m4a *.flv *.mp3 *.avi *.mkv *.webm *.3gp *.wav *.ape *.swf *.jpg *.png *.frag *.frag.urls *.frag.aria2 test/testdata/player-*.js
 clean-dist:
 	rm -rf yt-dlp.1.temp.md yt-dlp.1 README.txt MANIFEST build/ dist/ .coverage cover/ yt-dlp.tar.gz completions/ yt_dlp/extractor/lazy_extractors.py *.spec CONTRIBUTING.md.tmp yt-dlp yt-dlp.exe yt_dlp.egg-info/ AUTHORS .mailmap
 clean-cache:
@@ -49,23 +49,11 @@ codetest:
 	flake8 .
 
 test:
-	#nosetests --with-coverage --cover-package=yt_dlp --cover-html --verbose --processes 4 test
-	nosetests --verbose test
+	$(PYTHON) -m pytest
 	$(MAKE) codetest
 
-# Keep this list in sync with devscripts/run_tests.sh
 offlinetest: codetest
-	$(PYTHON) -m nose --verbose test \
-		--exclude test_age_restriction.py \
-		--exclude test_download.py \
-		--exclude test_iqiyi_sdk_interpreter.py \
-		--exclude test_overwrites.py \
-		--exclude test_socks.py \
-		--exclude test_subtitles.py \
-		--exclude test_write_annotations.py \
-		--exclude test_youtube_lists.py \
-		--exclude test_youtube_signature.py \
-		--exclude test_post_hooks.py
+	$(PYTHON) -m pytest -k "not download"
 
 yt-dlp: yt_dlp/*.py yt_dlp/*/*.py
 	mkdir -p zip

README.md

@@ -75,7 +75,7 @@ The major new features from the latest release of [blackjack4494/yt-dlc](https:/
     * All Feeds (`:ytfav`, `:ytwatchlater`, `:ytsubs`, `:ythistory`, `:ytrec`) supports downloading multiple pages of content
     * Search (`ytsearch:`, `ytsearchdate:`), search URLs and in-channel search works
     * Mixes supports downloading multiple pages of content
-    * Partial workarounds for age-gate and throttling issues
+    * Partial workaround for throttling issue
     * Redirect channel's home URL automatically to `/video` to preserve the old behaviour
     * `255kbps` audio is extracted from youtube music if premium cookies are given
     * Youtube music Albums, channels etc can be downloaded
@@ -215,7 +215,7 @@ You can also build the executable without any version info or metadata by using:
 Note that pyinstaller [does not support](https://github.com/pyinstaller/pyinstaller#requirements-and-tested-platforms) Python installed from the Windows store without using a virtual environment
 
 **For Unix**:
-You will need the required build tools: `python`, `make` (GNU), `pandoc`, `zip`, `nosetests`  
+You will need the required build tools: `python`, `make` (GNU), `pandoc`, `zip`, `pytest`  
 Then simply run `make`. You can also run `make yt-dlp` instead to compile only the binary without updating any of the additional files
 
 **Note**: In either platform, `devscripts\update-version.py` can be used to automatically update the version number

devscripts/run_tests.bat

@@ -1,17 +1,16 @@
+@setlocal
 @echo off
+cd /d %~dp0..
 
-rem Keep this list in sync with the `offlinetest` target in Makefile
-set DOWNLOAD_TESTS="age_restriction^|download^|iqiyi_sdk_interpreter^|socks^|subtitles^|write_annotations^|youtube_lists^|youtube_signature^|post_hooks"
-
-if "%YTDL_TEST_SET%" == "core" (
-    set test_set="-I test_("%DOWNLOAD_TESTS%")\.py"
-    set multiprocess_args=""
-) else if "%YTDL_TEST_SET%" == "download" (
-    set test_set="-I test_(?!"%DOWNLOAD_TESTS%").+\.py"
-    set multiprocess_args="--processes=4 --process-timeout=540"
+if ["%~1"]==[""] (
+    set "test_set="
+) else if ["%~1"]==["core"] (
+    set "test_set=-k "not download""
+) else if ["%~1"]==["download"] (
+    set "test_set=-k download"
 ) else (
-    echo YTDL_TEST_SET is not set or invalid
+    echo.Invalid test type "%~1". Use "core" ^| "download"
     exit /b 1
 )
 
-nosetests test --verbose %test_set:"=% %multiprocess_args:"=%
+pytest %test_set%

devscripts/run_tests.sh

@@ -1,22 +1,15 @@
-#!/bin/bash
+#!/bin/sh
 
-# Keep this list in sync with the `offlinetest` target in Makefile
-DOWNLOAD_TESTS="age_restriction|download|iqiyi_sdk_interpreter|overwrites|socks|subtitles|write_annotations|youtube_lists|youtube_signature|post_hooks"
+if [ -z $1 ]; then
+    test_set='test'
+elif [ $1 = 'core' ]; then
+    test_set='not download'
+elif [ $1 = 'download' ]; then
+    test_set='download'
+else
+    echo 'Invalid test type "'$1'". Use "core" | "download"'
+    exit 1
+fi
 
-test_set=""
-multiprocess_args=""
-
-case "$YTDL_TEST_SET" in
-    core)
-        test_set="-I test_($DOWNLOAD_TESTS)\.py"
-    ;;
-    download)
-        test_set="-I test_(?!$DOWNLOAD_TESTS).+\.py"
-        multiprocess_args="--processes=4 --process-timeout=540"
-    ;;
-    *)
-        break
-    ;;
-esac
-
-nosetests test --verbose $test_set $multiprocess_args
+echo python3 -m pytest -k $test_set
+python3 -m pytest -k "$test_set"

pytest.ini

@@ -0,0 +1,4 @@
+[pytest]
+addopts = -ra -v --strict-markers
+markers =
+    download

test/helper.py

@@ -22,6 +22,14 @@ from yt_dlp.utils import (
 )
 
 
+if "pytest" in sys.modules:
+    import pytest
+    is_download_test = pytest.mark.download
+else:
+    def is_download_test(testClass):
+        return testClass
+
+
 def get_params(override=None):
     PARAMETERS_FILE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
                                    "parameters.json")

test/parameters.json

@@ -1,4 +1,5 @@
 {
+    "check_formats": false,
     "consoletitle": false,
     "continuedl": true,
     "forcedescription": false,

test/test_InfoExtractor.py

@@ -35,13 +35,13 @@ class InfoExtractorTestRequestHandler(compat_http_server.BaseHTTPRequestHandler)
             assert False
 
 
-class TestIE(InfoExtractor):
+class DummyIE(InfoExtractor):
     pass
 
 
 class TestInfoExtractor(unittest.TestCase):
     def setUp(self):
-        self.ie = TestIE(FakeYDL())
+        self.ie = DummyIE(FakeYDL())
 
     def test_ie_key(self):
         self.assertEqual(get_info_extractor(YoutubeIE.ie_key()), YoutubeIE)

test/test_age_restriction.py

@@ -7,8 +7,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import try_rm
-
+from test.helper import try_rm, is_download_test
 
 from yt_dlp import YoutubeDL
 
@@ -32,6 +31,7 @@ def _download_restricted(url, filename, age):
     return res
 
 
+@is_download_test
 class TestAgeRestriction(unittest.TestCase):
     def _assert_restricted(self, url, filename, age, old_age=None):
         self.assertTrue(_download_restricted(url, filename, old_age))

test/test_download.py

@@ -10,12 +10,13 @@ sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from test.helper import (
     assertGreaterEqual,
+    expect_info_dict,
     expect_warnings,
     get_params,
     gettestcases,
-    expect_info_dict,
-    try_rm,
+    is_download_test,
     report_warning,
+    try_rm,
 )
 
 
@@ -64,6 +65,7 @@ def _file_md5(fn):
 defs = gettestcases()
 
 
+@is_download_test
 class TestDownload(unittest.TestCase):
     # Parallel testing in nosetests. See
     # http://nose.readthedocs.org/en/latest/doc_tests/test_multiprocess/multiprocess.html

test/test_iqiyi_sdk_interpreter.py

@@ -8,7 +8,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL
+from test.helper import FakeYDL, is_download_test
 from yt_dlp.extractor import IqiyiIE
 
 
@@ -31,6 +31,7 @@ class WarningLogger(object):
         pass
 
 
+@is_download_test
 class TestIqiyiSDKInterpreter(unittest.TestCase):
     def test_iqiyi_sdk_interpreter(self):
         '''

test/test_post_hooks.py

@@ -7,7 +7,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import get_params, try_rm
+from test.helper import get_params, try_rm, is_download_test
 import yt_dlp.YoutubeDL
 from yt_dlp.utils import DownloadError
 
@@ -22,6 +22,7 @@ TEST_ID = 'gr51aVj-mLg'
 EXPECTED_NAME = 'gr51aVj-mLg'
 
 
+@is_download_test
 class TestPostHooks(unittest.TestCase):
     def setUp(self):
         self.stored_name_1 = None

test/test_socks.py

@@ -14,6 +14,7 @@ import subprocess
 from test.helper import (
     FakeYDL,
     get_params,
+    is_download_test,
 )
 from yt_dlp.compat import (
     compat_str,
@@ -21,6 +22,7 @@ from yt_dlp.compat import (
 )
 
 
+@is_download_test
 class TestMultipleSocks(unittest.TestCase):
     @staticmethod
     def _check_params(attrs):
@@ -76,6 +78,7 @@ class TestMultipleSocks(unittest.TestCase):
             params['secondary_server_ip'])
 
 
+@is_download_test
 class TestSocks(unittest.TestCase):
     _SKIP_SOCKS_TEST = True
 

test/test_subtitles.py

@@ -7,7 +7,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL, md5
+from test.helper import FakeYDL, md5, is_download_test
 
 
 from yt_dlp.extractor import (
@@ -30,6 +30,7 @@ from yt_dlp.extractor import (
 )
 
 
+@is_download_test
 class BaseTestSubtitles(unittest.TestCase):
     url = None
     IE = None
@@ -55,6 +56,7 @@ class BaseTestSubtitles(unittest.TestCase):
         return dict((l, sub_info['data']) for l, sub_info in subtitles.items())
 
 
+@is_download_test
 class TestYoutubeSubtitles(BaseTestSubtitles):
     url = 'QRS8MkLhQmM'
     IE = YoutubeIE
@@ -111,6 +113,7 @@ class TestYoutubeSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestDailymotionSubtitles(BaseTestSubtitles):
     url = 'http://www.dailymotion.com/video/xczg00'
     IE = DailymotionIE
@@ -134,6 +137,7 @@ class TestDailymotionSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestTedSubtitles(BaseTestSubtitles):
     url = 'http://www.ted.com/talks/dan_dennett_on_our_consciousness.html'
     IE = TEDIE
@@ -149,6 +153,7 @@ class TestTedSubtitles(BaseTestSubtitles):
             self.assertTrue(subtitles.get(lang) is not None, 'Subtitles for \'%s\' not extracted' % lang)
 
 
+@is_download_test
 class TestVimeoSubtitles(BaseTestSubtitles):
     url = 'http://vimeo.com/76979871'
     IE = VimeoIE
@@ -170,6 +175,7 @@ class TestVimeoSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestWallaSubtitles(BaseTestSubtitles):
     url = 'http://vod.walla.co.il/movie/2705958/the-yes-men'
     IE = WallaIE
@@ -191,6 +197,7 @@ class TestWallaSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestCeskaTelevizeSubtitles(BaseTestSubtitles):
     url = 'http://www.ceskatelevize.cz/ivysilani/10600540290-u6-uzasny-svet-techniky'
     IE = CeskaTelevizeIE
@@ -212,6 +219,7 @@ class TestCeskaTelevizeSubtitles(BaseTestSubtitles):
         self.assertFalse(subtitles)
 
 
+@is_download_test
 class TestLyndaSubtitles(BaseTestSubtitles):
     url = 'http://www.lynda.com/Bootstrap-tutorials/Using-exercise-files/110885/114408-4.html'
     IE = LyndaIE
@@ -224,6 +232,7 @@ class TestLyndaSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '09bbe67222259bed60deaa26997d73a7')
 
 
+@is_download_test
 class TestNPOSubtitles(BaseTestSubtitles):
     url = 'http://www.npo.nl/nos-journaal/28-08-2014/POW_00722860'
     IE = NPOIE
@@ -236,6 +245,7 @@ class TestNPOSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['nl']), 'fc6435027572b63fb4ab143abd5ad3f4')
 
 
+@is_download_test
 class TestMTVSubtitles(BaseTestSubtitles):
     url = 'http://www.cc.com/video-clips/p63lk0/adam-devine-s-house-party-chasing-white-swans'
     IE = ComedyCentralIE
@@ -251,6 +261,7 @@ class TestMTVSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '78206b8d8a0cfa9da64dc026eea48961')
 
 
+@is_download_test
 class TestNRKSubtitles(BaseTestSubtitles):
     url = 'http://tv.nrk.no/serie/ikke-gjoer-dette-hjemme/DMPV73000411/sesong-2/episode-1'
     IE = NRKTVIE
@@ -263,6 +274,7 @@ class TestNRKSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['no']), '544fa917d3197fcbee64634559221cc2')
 
 
+@is_download_test
 class TestRaiPlaySubtitles(BaseTestSubtitles):
     IE = RaiPlayIE
 
@@ -283,6 +295,7 @@ class TestRaiPlaySubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['it']), '4b3264186fbb103508abe5311cfcb9cd')
 
 
+@is_download_test
 class TestVikiSubtitles(BaseTestSubtitles):
     url = 'http://www.viki.com/videos/1060846v-punch-episode-18'
     IE = VikiIE
@@ -295,6 +308,7 @@ class TestVikiSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '53cb083a5914b2d84ef1ab67b880d18a')
 
 
+@is_download_test
 class TestThePlatformSubtitles(BaseTestSubtitles):
     # from http://www.3playmedia.com/services-features/tools/integrations/theplatform/
     # (see http://theplatform.com/about/partners/type/subtitles-closed-captioning/)
@@ -309,6 +323,7 @@ class TestThePlatformSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '97e7670cbae3c4d26ae8bcc7fdd78d4b')
 
 
+@is_download_test
 class TestThePlatformFeedSubtitles(BaseTestSubtitles):
     url = 'http://feed.theplatform.com/f/7wvmTC/msnbc_video-p-test?form=json&pretty=true&range=-40&byGuid=n_hardball_5biden_140207'
     IE = ThePlatformFeedIE
@@ -321,6 +336,7 @@ class TestThePlatformFeedSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['en']), '48649a22e82b2da21c9a67a395eedade')
 
 
+@is_download_test
 class TestRtveSubtitles(BaseTestSubtitles):
     url = 'http://www.rtve.es/alacarta/videos/los-misterios-de-laura/misterios-laura-capitulo-32-misterio-del-numero-17-2-parte/2428621/'
     IE = RTVEALaCartaIE
@@ -335,6 +351,7 @@ class TestRtveSubtitles(BaseTestSubtitles):
         self.assertEqual(md5(subtitles['es']), '69e70cae2d40574fb7316f31d6eb7fca')
 
 
+@is_download_test
 class TestDemocracynowSubtitles(BaseTestSubtitles):
     url = 'http://www.democracynow.org/shows/2015/7/3'
     IE = DemocracynowIE

test/test_write_annotations.py

@@ -8,7 +8,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import get_params, try_rm
+from test.helper import get_params, try_rm, is_download_test
 
 
 import io
@@ -38,6 +38,7 @@ ANNOTATIONS_FILE = TEST_ID + '.annotations.xml'
 EXPECTED_ANNOTATIONS = ['Speech bubble', 'Note', 'Title', 'Spotlight', 'Label']
 
 
+@is_download_test
 class TestAnnotations(unittest.TestCase):
     def setUp(self):
         # Clear old files

test/test_youtube_lists.py

@@ -7,7 +7,7 @@ import sys
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from test.helper import FakeYDL
+from test.helper import FakeYDL, is_download_test
 
 
 from yt_dlp.extractor import (
@@ -17,6 +17,7 @@ from yt_dlp.extractor import (
 )
 
 
+@is_download_test
 class TestYoutubeLists(unittest.TestCase):
     def assertIsPlaylist(self, info):
         """Make sure the info has '_type' set to 'playlist'"""

test/test_youtube_signature.py

@@ -12,7 +12,7 @@ import io
 import re
 import string
 
-from test.helper import FakeYDL
+from test.helper import FakeYDL, is_download_test
 from yt_dlp.extractor import YoutubeIE
 from yt_dlp.compat import compat_str, compat_urlretrieve
 
@@ -65,6 +65,7 @@ _TESTS = [
 ]
 
 
+@is_download_test
 class TestPlayerInfo(unittest.TestCase):
     def test_youtube_extract_player_info(self):
         PLAYER_URLS = (
@@ -87,6 +88,7 @@ class TestPlayerInfo(unittest.TestCase):
             self.assertEqual(player_id, expected_player_id)
 
 
+@is_download_test
 class TestSignature(unittest.TestCase):
     def setUp(self):
         TEST_DIR = os.path.dirname(os.path.abspath(__file__))

tox.ini

@@ -1,5 +1,7 @@
 [tox]
 envlist = py26,py27,py33,py34,py35
+
+# Needed?
 [testenv]
 deps =
    nose

yt_dlp/YoutubeDL.py

@@ -322,6 +322,7 @@ class YoutubeDL(object):
                        progress, with a dictionary with the entries
                        * status: One of "downloading", "error", or "finished".
                                  Check this first and ignore unknown values.
+                       * info_dict: The extracted info_dict
 
                        If status is one of "downloading", or "finished", the
                        following properties may also be present:

yt_dlp/cookies.py

@@ -2,7 +2,6 @@ import ctypes
 import json
 import os
 import shutil
-import sqlite3
 import struct
 import subprocess
 import sys
@@ -16,6 +15,7 @@ from yt_dlp.compat import (
     compat_cookiejar_Cookie,
 )
 from yt_dlp.utils import (
+    bug_reports_message,
     bytes_to_intlist,
     expand_path,
     intlist_to_bytes,
@@ -23,6 +23,15 @@ from yt_dlp.utils import (
     YoutubeDLCookieJar,
 )
 
+try:
+    import sqlite3
+    SQLITE_AVAILABLE = True
+except ImportError:
+    # although sqlite3 is part of the standard library, it is possible to compile python without
+    # sqlite support. See: https://github.com/yt-dlp/yt-dlp/issues/544
+    SQLITE_AVAILABLE = False
+
+
 try:
     from Crypto.Cipher import AES
     CRYPTO_AVAILABLE = True
@@ -32,8 +41,17 @@ except ImportError:
 try:
     import keyring
     KEYRING_AVAILABLE = True
+    KEYRING_UNAVAILABLE_REASON = f'due to unknown reasons{bug_reports_message()}'
 except ImportError:
     KEYRING_AVAILABLE = False
+    KEYRING_UNAVAILABLE_REASON = (
+        'as the `keyring` module is not installed. '
+        'Please install by running `python3 -m pip install keyring`. '
+        'Depending on your platform, additional packages may be required '
+        'to access the keyring; see  https://pypi.org/project/keyring')
+except Exception as _err:
+    KEYRING_AVAILABLE = False
+    KEYRING_UNAVAILABLE_REASON = 'as the `keyring` module could not be initialized: %s' % _err
 
 
 CHROMIUM_BASED_BROWSERS = {'brave', 'chrome', 'chromium', 'edge', 'opera', 'vivaldi'}
@@ -90,6 +108,10 @@ def extract_cookies_from_browser(browser_name, profile=None, logger=YDLLogger())
 
 def _extract_firefox_cookies(profile, logger):
     logger.info('Extracting cookies from firefox')
+    if not SQLITE_AVAILABLE:
+        logger.warning('Cannot extract cookies from firefox without sqlite3 support. '
+                       'Please use a python interpreter compiled with sqlite3 support')
+        return YoutubeDLCookieJar()
 
     if profile is None:
         search_root = _firefox_browser_dir()
@@ -179,7 +201,7 @@ def _get_chromium_based_browser_settings(browser_name):
         'brave': 'Brave',
         'chrome': 'Chrome',
         'chromium': 'Chromium',
-        'edge': 'Mirosoft Edge' if sys.platform == 'darwin' else 'Chromium',
+        'edge': 'Microsoft Edge' if sys.platform == 'darwin' else 'Chromium',
         'opera': 'Opera' if sys.platform == 'darwin' else 'Chromium',
         'vivaldi': 'Vivaldi' if sys.platform == 'darwin' else 'Chrome',
     }[browser_name]
@@ -195,6 +217,12 @@ def _get_chromium_based_browser_settings(browser_name):
 
 def _extract_chrome_cookies(browser_name, profile, logger):
     logger.info('Extracting cookies from {}'.format(browser_name))
+
+    if not SQLITE_AVAILABLE:
+        logger.warning(('Cannot extract cookies from {} without sqlite3 support. '
+                        'Please use a python interpreter compiled with sqlite3 support').format(browser_name))
+        return YoutubeDLCookieJar()
+
     config = _get_chromium_based_browser_settings(browser_name)
 
     if profile is None:
@@ -322,10 +350,7 @@ class LinuxChromeCookieDecryptor(ChromeCookieDecryptor):
 
         elif version == b'v11':
             if self._v11_key is None:
-                self._logger.warning('cannot decrypt cookie as the `keyring` module is not installed. '
-                                     'Please install by running `python3 -m pip install keyring`. '
-                                     'Note that depending on your platform, additional packages may be required '
-                                     'to access the keyring, see  https://pypi.org/project/keyring', only_once=True)
+                self._logger.warning(f'cannot decrypt cookie {KEYRING_UNAVAILABLE_REASON}', only_once=True)
                 return None
             return _decrypt_aes_cbc(ciphertext, self._v11_key, self._logger)
 

yt_dlp/downloader/common.py

@@ -1,5 +1,6 @@
 from __future__ import division, unicode_literals
 
+import copy
 import os
 import re
 import sys
@@ -360,7 +361,7 @@ class FileDownloader(object):
                     'filename': filename,
                     'status': 'finished',
                     'total_bytes': os.path.getsize(encodeFilename(filename)),
-                })
+                }, info_dict)
                 return True, False
 
         if subtitle is False:
@@ -388,7 +389,16 @@ class FileDownloader(object):
         """Real download process. Redefine in subclasses."""
         raise NotImplementedError('This method must be implemented by subclasses')
 
-    def _hook_progress(self, status):
+    def _hook_progress(self, status, info_dict):
+        if not self._progress_hooks:
+            return
+        info_dict = dict(info_dict)
+        for key in ('__original_infodict', '__postprocessors'):
+            info_dict.pop(key, None)
+        # youtube-dl passes the same status object to all the hooks.
+        # Some third party scripts seems to be relying on this.
+        # So keep this behavior if possible
+        status['info_dict'] = copy.deepcopy(info_dict)
         for ph in self._progress_hooks:
             ph(status)
 

yt_dlp/downloader/dash.py

@@ -29,7 +29,7 @@ class DashSegmentsFD(FragmentFD):
         if real_downloader:
             self._prepare_external_frag_download(ctx)
         else:
-            self._prepare_and_start_frag_download(ctx)
+            self._prepare_and_start_frag_download(ctx, info_dict)
 
         fragments_to_download = []
         frag_index = 0

yt_dlp/downloader/external.py

@@ -67,7 +67,7 @@ class ExternalFD(FileDownloader):
                     'downloaded_bytes': fsize,
                     'total_bytes': fsize,
                 })
-            self._hook_progress(status)
+            self._hook_progress(status, info_dict)
             return True
         else:
             self.to_stderr('\n')

yt_dlp/downloader/f4m.py

@@ -380,7 +380,7 @@ class F4mFD(FragmentFD):
 
         base_url_parsed = compat_urllib_parse_urlparse(base_url)
 
-        self._start_frag_download(ctx)
+        self._start_frag_download(ctx, info_dict)
 
         frag_index = 0
         while fragments_list:
@@ -434,6 +434,6 @@ class F4mFD(FragmentFD):
                     msg = 'Missed %d fragments' % (fragments_list[0][1] - (frag_i + 1))
                     self.report_warning(msg)
 
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
 
         return True

yt_dlp/downloader/fragment.py

@@ -83,9 +83,9 @@ class FragmentFD(FileDownloader):
         headers = info_dict.get('http_headers')
         return sanitized_Request(url, None, headers) if headers else url
 
-    def _prepare_and_start_frag_download(self, ctx):
+    def _prepare_and_start_frag_download(self, ctx, info_dict):
         self._prepare_frag_download(ctx)
-        self._start_frag_download(ctx)
+        self._start_frag_download(ctx, info_dict)
 
     def __do_ytdl_file(self, ctx):
         return not ctx['live'] and not ctx['tmpfilename'] == '-' and not self.params.get('_no_ytdl_file')
@@ -219,7 +219,7 @@ class FragmentFD(FileDownloader):
             'complete_frags_downloaded_bytes': resume_len,
         })
 
-    def _start_frag_download(self, ctx):
+    def _start_frag_download(self, ctx, info_dict):
         resume_len = ctx['complete_frags_downloaded_bytes']
         total_frags = ctx['total_frags']
         # This dict stores the download progress, it's updated by the progress
@@ -248,6 +248,7 @@ class FragmentFD(FileDownloader):
             time_now = time.time()
             state['elapsed'] = time_now - start
             frag_total_bytes = s.get('total_bytes') or 0
+            s['fragment_info_dict'] = s.pop('info_dict', {})
             if not ctx['live']:
                 estimated_size = (
                     (ctx['complete_frags_downloaded_bytes'] + frag_total_bytes)
@@ -270,13 +271,13 @@ class FragmentFD(FileDownloader):
                 state['speed'] = s.get('speed') or ctx.get('speed')
                 ctx['speed'] = state['speed']
                 ctx['prev_frag_downloaded_bytes'] = frag_downloaded_bytes
-            self._hook_progress(state)
+            self._hook_progress(state, info_dict)
 
         ctx['dl'].add_progress_hook(frag_progress_hook)
 
         return start
 
-    def _finish_frag_download(self, ctx):
+    def _finish_frag_download(self, ctx, info_dict):
         ctx['dest_stream'].close()
         if self.__do_ytdl_file(ctx):
             ytdl_filename = encodeFilename(self.ytdl_filename(ctx['filename']))
@@ -303,7 +304,7 @@ class FragmentFD(FileDownloader):
             'filename': ctx['filename'],
             'status': 'finished',
             'elapsed': elapsed,
-        })
+        }, info_dict)
 
     def _prepare_external_frag_download(self, ctx):
         if 'live' not in ctx:
@@ -421,5 +422,5 @@ class FragmentFD(FileDownloader):
                 if not result:
                     return False
 
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
         return True

yt_dlp/downloader/hls.py

@@ -133,7 +133,7 @@ class HlsFD(FragmentFD):
         if real_downloader:
             self._prepare_external_frag_download(ctx)
         else:
-            self._prepare_and_start_frag_download(ctx)
+            self._prepare_and_start_frag_download(ctx, info_dict)
 
         extra_state = ctx.setdefault('extra_state', {})
 

yt_dlp/downloader/http.py

@@ -177,7 +177,7 @@ class HttpFD(FileDownloader):
                                 'status': 'finished',
                                 'downloaded_bytes': ctx.resume_len,
                                 'total_bytes': ctx.resume_len,
-                            })
+                            }, info_dict)
                             raise SucceedDownload()
                         else:
                             # The length does not match, we start the download over
@@ -310,7 +310,7 @@ class HttpFD(FileDownloader):
                     'eta': eta,
                     'speed': speed,
                     'elapsed': now - ctx.start_time,
-                })
+                }, info_dict)
 
                 if data_len is not None and byte_counter == data_len:
                     break
@@ -357,7 +357,7 @@ class HttpFD(FileDownloader):
                 'filename': ctx.filename,
                 'status': 'finished',
                 'elapsed': time.time() - ctx.start_time,
-            })
+            }, info_dict)
 
             return True
 

yt_dlp/downloader/ism.py

@@ -246,7 +246,7 @@ class IsmFD(FragmentFD):
             'total_frags': len(segments),
         }
 
-        self._prepare_and_start_frag_download(ctx)
+        self._prepare_and_start_frag_download(ctx, info_dict)
 
         extra_state = ctx.setdefault('extra_state', {
             'ism_track_written': False,
@@ -284,6 +284,6 @@ class IsmFD(FragmentFD):
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
                 return False
 
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
 
         return True

yt_dlp/downloader/mhtml.py

@@ -122,7 +122,7 @@ body > figure > img {
             'total_frags': len(fragments),
         }
 
-        self._prepare_and_start_frag_download(ctx)
+        self._prepare_and_start_frag_download(ctx, info_dict)
 
         extra_state = ctx.setdefault('extra_state', {
             'header_written': False,
@@ -198,5 +198,5 @@ body > figure > img {
 
         ctx['dest_stream'].write(
             b'--%b--\r\n\r\n' % frag_boundary.encode('us-ascii'))
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
         return True

yt_dlp/downloader/rtmp.py

@@ -66,7 +66,7 @@ class RtmpFD(FileDownloader):
                             'eta': eta,
                             'elapsed': time_now - start,
                             'speed': speed,
-                        })
+                        }, info_dict)
                         cursor_in_new_line = False
                     else:
                         # no percent for live streams
@@ -82,7 +82,7 @@ class RtmpFD(FileDownloader):
                                 'status': 'downloading',
                                 'elapsed': time_now - start,
                                 'speed': speed,
-                            })
+                            }, info_dict)
                             cursor_in_new_line = False
                         elif self.params.get('verbose', False):
                             if not cursor_in_new_line:
@@ -208,7 +208,7 @@ class RtmpFD(FileDownloader):
                 'filename': filename,
                 'status': 'finished',
                 'elapsed': time.time() - started,
-            })
+            }, info_dict)
             return True
         else:
             self.to_stderr('\n')

yt_dlp/downloader/rtsp.py

@@ -39,7 +39,7 @@ class RtspFD(FileDownloader):
                 'total_bytes': fsize,
                 'filename': filename,
                 'status': 'finished',
-            })
+            }, info_dict)
             return True
         else:
             self.to_stderr('\n')

yt_dlp/downloader/youtube_live_chat.py

@@ -140,7 +140,7 @@ class YoutubeLiveChatFD(FragmentFD):
                 self.report_error('giving up after %s fragment retries' % fragment_retries)
                 return False, None, None, None
 
-        self._prepare_and_start_frag_download(ctx)
+        self._prepare_and_start_frag_download(ctx, info_dict)
 
         success, raw_fragment = dl_fragment(info_dict['url'])
         if not success:
@@ -196,7 +196,7 @@ class YoutubeLiveChatFD(FragmentFD):
             if test:
                 break
 
-        self._finish_frag_download(ctx)
+        self._finish_frag_download(ctx, info_dict)
         return True
 
     @staticmethod

yt_dlp/extractor/youtube.py

@@ -691,7 +691,7 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
                 alert_type = alert.get('type')
                 if not alert_type:
                     continue
-                message = cls._get_text(alert.get('text'))
+                message = cls._get_text(alert, 'text')
                 if message:
                     yield alert_type, message
 
@@ -721,23 +721,26 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
         return badges
 
     @staticmethod
-    def _get_text(data, getter=None, max_runs=None):
-        for get in variadic(getter):
-            d = try_get(data, get) if get is not None else data
-            text = try_get(d, lambda x: x['simpleText'], compat_str)
-            if text:
-                return text
-            runs = try_get(d, lambda x: x['runs'], list) or []
-            if not runs and isinstance(d, list):
-                runs = d
+    def _get_text(data, *path_list, max_runs=None):
+        for path in path_list or [None]:
+            if path is None:
+                obj = [data]
+            else:
+                obj = traverse_obj(data, path, default=[])
+                if not any(key is ... or isinstance(key, (list, tuple)) for key in variadic(path)):
+                    obj = [obj]
+            for item in obj:
+                text = try_get(item, lambda x: x['simpleText'], compat_str)
+                if text:
+                    return text
+                runs = try_get(item, lambda x: x['runs'], list) or []
+                if not runs and isinstance(item, list):
+                    runs = item
 
-            def get_runs(runs):
-                for run in runs[:min(len(runs), max_runs or len(runs))]:
-                    yield try_get(run, lambda x: x['text'], compat_str) or ''
-
-            text = ''.join(get_runs(runs))
-            if text:
-                return text
+                runs = runs[:min(len(runs), max_runs or len(runs))]
+                text = ''.join(traverse_obj(runs, (..., 'text'), expected_type=str, default=[]))
+                if text:
+                    return text
 
     def _extract_response(self, item_id, query, note='Downloading API JSON', headers=None,
                           ytcfg=None, check_get_keys=None, ep='browse', fatal=True, api_hostname=None,
@@ -804,15 +807,16 @@ class YoutubeBaseInfoExtractor(InfoExtractor):
 
     def _extract_video(self, renderer):
         video_id = renderer.get('videoId')
-        title = self._get_text(renderer.get('title'))
-        description = self._get_text(renderer.get('descriptionSnippet'))
-        duration = parse_duration(self._get_text(renderer.get('lengthText')))
-        view_count_text = self._get_text(renderer.get('viewCountText')) or ''
+        title = self._get_text(renderer, 'title')
+        description = self._get_text(renderer, 'descriptionSnippet')
+        duration = parse_duration(self._get_text(
+            renderer, 'lengthText', ('thumbnailOverlays', ..., 'thumbnailOverlayTimeStatusRenderer', 'text')))
+        view_count_text = self._get_text(renderer, 'viewCountText') or ''
         view_count = str_to_int(self._search_regex(
             r'^([\d,]+)', re.sub(r'\s', '', view_count_text),
             'view count', default=None))
 
-        uploader = self._get_text(renderer, (lambda x: x['ownerText'], lambda x: x['shortBylineText']))
+        uploader = self._get_text(renderer, 'ownerText', 'shortBylineText')
 
         return {
             '_type': 'url',
@@ -2028,8 +2032,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             data,
             ('engagementPanels', ..., 'engagementPanelSectionListRenderer', 'content', 'macroMarkersListRenderer', 'contents'),
             expected_type=list, default=[])
-        chapter_time = lambda chapter: parse_duration(self._get_text(chapter.get('timeDescription')))
-        chapter_title = lambda chapter: self._get_text(chapter.get('title'))
+        chapter_time = lambda chapter: parse_duration(self._get_text(chapter, 'timeDescription'))
+        chapter_title = lambda chapter: self._get_text(chapter, 'title')
 
         return next((
             filter(None, (
@@ -2083,14 +2087,14 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         if not comment_id:
             return
 
-        text = self._get_text(comment_renderer.get('contentText'))
+        text = self._get_text(comment_renderer, 'contentText')
 
         # note: timestamp is an estimate calculated from the current time and time_text
-        time_text = self._get_text(comment_renderer.get('publishedTimeText')) or ''
+        time_text = self._get_text(comment_renderer, 'publishedTimeText') or ''
         time_text_dt = self.parse_time_text(time_text)
         if isinstance(time_text_dt, datetime.datetime):
             timestamp = calendar.timegm(time_text_dt.timetuple())
-        author = self._get_text(comment_renderer.get('authorText'))
+        author = self._get_text(comment_renderer, 'authorText')
         author_id = try_get(comment_renderer,
                             lambda x: x['authorEndpoint']['browseEndpoint']['browseId'], compat_str)
 
@@ -2125,7 +2129,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             for content in contents:
                 comments_header_renderer = try_get(content, lambda x: x['commentsHeaderRenderer'])
                 expected_comment_count = parse_count(self._get_text(
-                    comments_header_renderer, (lambda x: x['countText'], lambda x: x['commentsCount']), max_runs=1))
+                    comments_header_renderer, 'countText', 'commentsCount', max_runs=1))
 
                 if expected_comment_count:
                     comment_counts[1] = expected_comment_count
@@ -2343,7 +2347,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             'playbackContext': {
                 'contentPlaybackContext': context
             },
-            'contentCheckOk': True
+            'contentCheckOk': True,
+            'racyCheckOk': True
         }
 
     @staticmethod
@@ -2389,21 +2394,22 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         ) or None
 
     def _extract_age_gated_player_response(self, client, video_id, ytcfg, identity_token, player_url, initial_pr):
-        gvi_client = self._YT_CLIENTS.get(f'_{client}_agegate')
-        if not gvi_client:
+        # get_video_info endpoint seems to be completely dead
+        gvi_client = None  # self._YT_CLIENTS.get(f'_{client}_agegate')
+        if gvi_client:
+            pr = self._parse_json(traverse_obj(
+                compat_parse_qs(self._download_webpage(
+                    self.http_scheme() + '//www.youtube.com/get_video_info', video_id,
+                    'Refetching age-gated %s info webpage' % gvi_client.lower(),
+                    'unable to download video info webpage', fatal=False,
+                    query=self._get_video_info_params(video_id, client=gvi_client))),
+                ('player_response', 0), expected_type=str) or '{}', video_id)
+            if pr:
+                return pr
+            self.report_warning('Falling back to embedded-only age-gate workaround')
+
+        if not self._YT_CLIENTS.get(f'_{client}_embedded'):
             return
-
-        pr = self._parse_json(traverse_obj(
-            compat_parse_qs(self._download_webpage(
-                self.http_scheme() + '//www.youtube.com/get_video_info', video_id,
-                'Refetching age-gated %s info webpage' % gvi_client.lower(),
-                'unable to download video info webpage', fatal=False,
-                query=self._get_video_info_params(video_id, client=gvi_client))),
-            ('player_response', 0), expected_type=str) or '{}', video_id)
-        if pr:
-            return pr
-
-        self.report_warning('Falling back to embedded-only age-gate workaround')
         embed_webpage = None
         if client == 'web' and 'configs' not in self._configuration_arg('player_skip'):
             embed_webpage = self._download_webpage(
@@ -2442,12 +2448,9 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                 webpage, self._YT_INITIAL_PLAYER_RESPONSE_RE,
                 video_id, 'initial player response')
 
-        age_gated = False
         for client in clients:
             player_ytcfg = master_ytcfg if client == 'web' else {}
-            if age_gated:
-                pr = None
-            elif client == 'web' and initial_pr:
+            if client == 'web' and initial_pr:
                 pr = initial_pr
             else:
                 if client == 'web_music' and 'configs' not in self._configuration_arg('player_skip'):
@@ -2459,8 +2462,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                     client, video_id, player_ytcfg or master_ytcfg, player_ytcfg, identity_token, player_url, initial_pr)
             if pr:
                 yield pr
-            if age_gated or traverse_obj(pr, ('playabilityStatus', 'reason')) in self._AGE_GATE_REASONS:
-                age_gated = True
+            if traverse_obj(pr, ('playabilityStatus', 'reason')) in self._AGE_GATE_REASONS:
                 pr = self._extract_age_gated_player_response(
                     client, video_id, player_ytcfg or master_ytcfg, identity_token, player_url, initial_pr)
                 if pr:
@@ -2847,7 +2849,14 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
             'release_timestamp': live_starttime,
         }
 
-        pctr = get_first(player_responses, ('captions', 'playerCaptionsTracklistRenderer'), expected_type=dict)
+        pctr = traverse_obj(player_responses, (..., 'captions', 'playerCaptionsTracklistRenderer'), expected_type=dict)
+        # Converted into dicts to remove duplicates
+        captions = {
+            sub.get('baseUrl'): sub
+            for sub in traverse_obj(pctr, (..., 'captionTracks', ...), default=[])}
+        translation_languages = {
+            lang.get('languageCode'): lang.get('languageName')
+            for lang in traverse_obj(pctr, (..., 'translationLanguages', ...), default=[])}
         subtitles = {}
         if pctr:
             def process_language(container, base_url, lang_code, sub_name, query):
@@ -2862,8 +2871,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         'name': sub_name,
                     })
 
-            for caption_track in (pctr.get('captionTracks') or []):
-                base_url = caption_track.get('baseUrl')
+            for base_url, caption_track in captions.items():
                 if not base_url:
                     continue
                 if caption_track.get('kind') != 'asr':
@@ -2874,18 +2882,17 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         continue
                     process_language(
                         subtitles, base_url, lang_code,
-                        try_get(caption_track, lambda x: x['name']['simpleText']),
+                        traverse_obj(caption_track, ('name', 'simpleText')),
                         {})
                     continue
                 automatic_captions = {}
-                for translation_language in (pctr.get('translationLanguages') or []):
-                    translation_language_code = translation_language.get('languageCode')
-                    if not translation_language_code:
+                for trans_code, trans_name in translation_languages.items():
+                    if not trans_code:
                         continue
                     process_language(
-                        automatic_captions, base_url, translation_language_code,
-                        self._get_text(translation_language.get('languageName'), max_runs=1),
-                        {'tlang': translation_language_code})
+                        automatic_captions, base_url, trans_code,
+                        self._get_text(trans_name, max_runs=1),
+                        {'tlang': trans_code})
                 info['automatic_captions'] = automatic_captions
         info['subtitles'] = subtitles
 
@@ -2998,10 +3005,7 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         })
                 vsir = content.get('videoSecondaryInfoRenderer')
                 if vsir:
-                    info['channel'] = self._get_text(try_get(
-                        vsir,
-                        lambda x: x['owner']['videoOwnerRenderer']['title'],
-                        dict))
+                    info['channel'] = self._get_text(vsir, ('owner', 'videoOwnerRenderer', 'title'))
                     rows = try_get(
                         vsir,
                         lambda x: x['metadataRowContainer']['metadataRowContainerRenderer']['rows'],
@@ -3016,8 +3020,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                         mrr_title = mrr.get('title')
                         if not mrr_title:
                             continue
-                        mrr_title = self._get_text(mrr['title'])
-                        mrr_contents_text = self._get_text(mrr['contents'][0])
+                        mrr_title = self._get_text(mrr, 'title')
+                        mrr_contents_text = self._get_text(mrr, ('contents', 0))
                         if mrr_title == 'License':
                             info['license'] = mrr_contents_text
                         elif not multiple_songs:
@@ -3589,7 +3593,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             renderer = self._extract_basic_item_renderer(item)
             if not isinstance(renderer, dict):
                 continue
-            title = self._get_text(renderer.get('title'))
+            title = self._get_text(renderer, 'title')
 
             # playlist
             playlist_id = renderer.get('playlistId')
@@ -3649,7 +3653,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
             # will not work
             if skip_channels and '/channels?' in shelf_url:
                 return
-            title = self._get_text(shelf_renderer, lambda x: x['title'])
+            title = self._get_text(shelf_renderer, 'title')
             yield self.url_result(shelf_url, video_title=title)
         # Shelf may not contain shelf URL, fallback to extraction from content
         for entry in self._shelf_entries_from_content(shelf_renderer):
@@ -4023,8 +4027,7 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
                 renderer_dict, lambda x: x['privacyDropdownItemRenderer']['isSelected'], bool) or False
             if not is_selected:
                 continue
-            label = self._get_text(
-                try_get(renderer_dict, lambda x: x['privacyDropdownItemRenderer']['label'], dict) or [])
+            label = self._get_text(renderer_dict, ('privacyDropdownItemRenderer', 'label'))
             if label:
                 badge_labels.add(label.lower())
                 break

yt_dlp/utils.py

@@ -3964,7 +3964,7 @@ def detect_exe_version(output, version_re=None, unrecognized='present'):
         return unrecognized
 
 
-class LazyList(collections.Sequence):
+class LazyList(collections.abc.Sequence):
     ''' Lazy immutable list from an iterable
     Note that slices of a LazyList are lists and not LazyList'''
 
@@ -6313,4 +6313,4 @@ def traverse_dict(dictn, keys, casesense=True):
 
 
 def variadic(x, allowed_types=(str, bytes)):
-    return x if isinstance(x, collections.Iterable) and not isinstance(x, allowed_types) else (x,)
+    return x if isinstance(x, collections.abc.Iterable) and not isinstance(x, allowed_types) else (x,)

yt_dlp/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.07.07'
+__version__ = '2021.07.21'