Over the past year, YouTube has significantly retricted access to its non-public APIs. Most notably, this includes the unauthenticated API that's triggered when playing a video. This applies to all types of devices, whether it's a web browser, iOS, Android, or even a smart TV.
On the server hosting this blog, I have public applications available here. In particular youtube_player is a web frontend that makes it easy to retrieve your latest subscription videos. In the background, I'm using the official YouTube Public API to retrieve subscriptions, but to download YouTube videos I'm using the yt-dlp.
yt-dlp
Tools like yt-dlp, which allow you to download YouTube videos, have back-engineered the non-public youtube API used by YouTube clients. You can see the source code used to extract information and video URL from a YouTube video : https://github.com/yt-dlp/yt-dlp/tree/master/yt_dlp/extractor/youtube
If you are trying to download a video with yt-dlp from a server hosted in a datacenter. -For example in my case where my server is at Oracle Cloud-. YouTube blocked access to the platform from non-residential IPs. You receive an error like "Sign in to confirm you’re not a bot".
ubuntu@arfevrier:~$ yt-dlp -F "https://www.youtube.com/watch?v=xvFZjo5PgG0"
[youtube] Extracting URL: https://www.youtube.com/watch?v=xvFZjo5PgG0
[youtube] xvFZjo5PgG0: Downloading webpage
[youtube] xvFZjo5PgG0: Downloading tv client config
[youtube] xvFZjo5PgG0: Downloading tv player API JSON
[youtube] xvFZjo5PgG0: Downloading ios player API JSON
ERROR: [youtube] xvFZjo5PgG0: Sign in to confirm you’re not a bot. Use --cookies-from-browser or --cookies for the authentication. See https://github.com/yt-dlp/yt-dlp/wiki/FAQ#how-do-i-pass-cookies-to-yt-dlp for how to manually pass cookies. Also see https://github.com/yt-dlp/yt-dlp/wiki/Extractors#exporting-youtube-cookies for tips on effectively exporting YouTube cookies
So my concern is that my application is publicly available. I don't want to set up a personal account for yt-dlp access. The often-recommended option of using residential proxies isn't a solution either. I'd like a free solution that only uses public tools.
This is where Cloudflare WARP comes in. Rather unknown, Cloudflare offers a free VPN in conjunction with its public DNS one⁴. And what's interesting with this VPN is that YouTube considers cloudflare IPs not to be blacklisted. Meaning we can use the public YouTube APIs without limit.
Running Wireguard VPN inside a Docker container
Setup Cloudflare WARP VPN to run inside a container and create an HTTP proxy that uses this VPN connection
The problem with WARP is that it only offers Wireguard-based VPN. It is not a service that exposes a public proxy. Normally, you need to install the Cloudflare CLI on your machine, which sets up a Wireguard VPN connection. But in my case, I don't want all my server's traffic to go through Cloudflare's servers. I want to restrict this usage to yt-dlp only. And the great thing about Wireguard is that it's completely container-ready 😃.
So, the idea is to set up WARP inside a Docker container. Then, set up a SQUID proxy in order to expose the WARP proxy to my local network interface.
There is the project named warproxy which does this all for you:
kingcc
But in my case I want to do this myself, because I want to extend WARP VPN for other Docker containers that need the VPN connection. So we're not going to use this utility, and redo the HTTP proxy management ourselves.
Here's my philosophy for hosting my application.
- I try to have as few applications as possible installed directly on the host system. To minimize incompatibility between tools.
- This way, the entire application environment is available in a single docker compose file. I have no prerequisites on the host system hosting the application.
- I have another VPN Wireguard server running on the server. The network interface created by Docker makes it possible to separate these two usages.
- I only want YouTube traffic to go through Cloudflare. The rest of the server's traffic must go through the default route.
- We don't know if this solution is stable over time. Containers allow the solution to evolve rapidly.
- I think it's a pretty nice solution, using these network features of containers.
To sum up, we need:
- Cloudflare WARP CLI to generate Wireguard client. I will use the unofficial wgcf project. Because this provides more export options for Wireguard configuration.
ViRb3
- A Wireguard container client. Here the linuxserver edition of wireguard is perfect for the job: https://hub.docker.com/r/linuxserver/wireguard
- A proxy. I'm going with Squid: https://hub.docker.com/r/ubuntu/squid
The first step is to build a Docker image with the wireguard client and wgcf tool available inside.
Some explanation of the Dockerfile code:
- 7-8: Download wgcf to the local binary folder
- 9: Create the folder /config/wg_confs used by wireguard to read client configuration. This is specific to how Linuxserver have configured their container image.
- 10: Linuxserver are using s6-overlay to manage the lifecycle of the service they package inside containers. Here s6 is managing the run of wireguard client inside the container. That's why I'm adding a new "init-config" script to initialize Cloudflare WARP configuration. This "init-config" script is running before the service Wireguard is started.
- 11-13: Add to the script the command to generate wireguard client. Then the image container automatically runs all wireguard client configuration inside: /config/wg_confs/*.conf (https://hub.docker.com/r/linuxserver/wireguard#client-mode)
- 16: Add a health check to check every 30s if the WARp connection is down. The API (https://www.cloudflare.com/cdn-cgi/trace) displays the status of WARP status. This is in case of the wireguard VPN disconnects.
The Dockerfile build by Docker Compose:
Some explanation of the Docker Compose configuration:
- 7: Capabilities required for wireguard.
- 13-14: External port needs to be defined inside wireguard container. Because it holds the network.
- 24: Set network as the wireguard network. Squid and Wireguard containers are sharing the same network stack.
- 29: Squid config, nothing special here: https://gist.github.com/arfevrier/018ec0b1ad215d714680ead8498ff805
You can now build and run your configuration:
$ docker compose build
$ docker compose run -d
$ docker compose logs -f
With the container running, I set up yt-dlp inside my server to use squid. The proxy, as it shares the network stack of wireguard, uses Cloudflare WARP to anwser to YouTube API requests. Meaning that the IP address seen by YouTube is the Cloudflare WARP one.
$ ubuntu@arfevrier:~$ https_proxy="http://127.0.0.1:8080" yt-dlp -F "https://www.youtube.com/watch?v=xvFZjo5PgG0"
[youtube] Extracting URL: https://www.youtube.com/watch?v=xvFZjo5PgG0
[youtube] xvFZjo5PgG0: Downloading webpage
[youtube] xvFZjo5PgG0: Downloading tv client config
[youtube] xvFZjo5PgG0: Downloading tv player API JSON
[youtube] xvFZjo5PgG0: Downloading ios player API JSON
[youtube] xvFZjo5PgG0: Downloading m3u8 information
[...]
This is how to bypass the Youtube retriction in order to still have my youtube_player application available.
That's how this post ends, sharing a simple method for continuing to use yt-dlp tools to download YouTube videos. Cloudflare Warp is a free tool at the moment, and it's powerful enough to take advantage of Cloudflare's interconnection network. Google will certainly continue to restrict access to its video content. It may be necessary to find a new, innovative solution to continue enjoying YouTube's free features.
This use case was also a way for me to propose a powerful method of taking advantage of the network isolation of containers. It's perfectly possible to encapsulate a VPN in a container. This leaves the host machine's network configuration unaffected, and allows the VPN connection to be shared with other application containers.
