Browse documentation
Guides
Compare features, map your existing settings and move a video dataset workflow to managed cloud delivery.
On this page
Choose by the work you want to own
- Keep yt-dlp for local files, custom FFmpeg processing, plugins or an existing worker you want to operate.
- Use FlashData for highly concurrent YouTube collections delivered to cloud storage, with built-in YouTube error handling and a console or API for tracking jobs.
- Need a finished dataset? Use managed dataset delivery when you also want help with source selection and packaging.
Compare video and data features
| Feature | yt-dlp | FlashData |
|---|---|---|
| Video quality | Select and filter source formats, including 4K and 8K. | Up to 8K; best, exact, minimum or maximum resolution settings. |
| Video and audio | Separate streams, FFmpeg merging and custom post-processing. | Video + audio, video-only or audio-only jobs. Video: MP4, WebM, MKV. Audio: M4A, MP3, Opus. |
| Format selection | -f selectors and -S sorting. | yt_dlp_format accepts a format selector; it overrides quality, FPS and audio-language selection. |
| Clips | --download-sections with FFmpeg; time ranges or chapters. | start_at and end_at; one interval for every video in a Bulk job. |
| Metadata | --write-info-json saves a local sidecar. | upload_metadata=True delivers a JSON sidecar beside the media; metadata bytes are not charged as download volume. |
| Captions and transcripts | Retrieve supplied or automatic tracks; save VTT, SRT or other available formats. | Separate YouTube Data requests return timestamps and text as JSON. Select language and caption origin explicitly. |
Compare collection and delivery features
| Feature | yt-dlp | FlashData |
|---|---|---|
| Batch input | --batch-file accepts a list of URLs. | Bulk Jobs accepts a UTF-8 text file of YouTube IDs, up to 2 GiB. |
| Storage | Local files; connect an uploader for cloud storage. | Direct delivery to S3, S3-compatible storage, GCS, Azure, OSS, COS, BOS or TOS. |
| Progress | CLI progress, Python hooks and your own queue or dashboard. | Saved job IDs, Bulk progress, completed/failed/invalid counts and console history. |
| Pause and resume | Partial-file resume and archives; your process supervisor keeps workers alive. | Pause new Bulk submissions and resume the saved job. Accepted work can finish while paused. |
| Delivery records | Files and info JSON; add your own upload receipts and checksums. | Media location, file size and MD5 in result records, plus metadata-file details when requested. |
| Deduplication | --download-archive tracks completed IDs for a download policy. | Your application filters delivered IDs using its inventory and saved Bulk results. |
Keep the delivery records. Bulk result parts are retrievable for 24 hours after completion. Save them in your dataset inventory; media retention follows your own storage policy.
Let FlashData handle YouTube errors at scale
Run downloads at near-unlimited concurrency. FlashData handles YouTube 403s, bot checks, session renewal and 429s internally, so your team can focus on the delivered files.
| Problem | With your yt-dlp workers | With FlashData |
|---|---|---|
| YouTube changes break extraction | Update nightly and check Deno, EJS and your worker image. | FlashData maintains the retrieval stack. Your application submits IDs and output settings. |
| 403s or persistent bot checks | Reduce concurrency, test healthy exit IPs and keep one route through extraction and transfer. | Automatic bot handling, proxy selection and recovery resolve these errors inside FlashData. |
| Browser sessions need renewal | Valid cookies can help small jobs; you maintain and renew those sessions. | FlashData manages access and session handling internally. No browser cookies to supply, rotate or renew. |
| YouTube 429s as traffic increases | Cool down the exit IP and coordinate request pacing across workers. | FlashData handles YouTube throttling, request routing and recovery internally, including during large concurrent collections. |
| High concurrent download volume | Size worker pools, maintain healthy proxy capacity and coordinate request pacing. | Scale concurrent downloads through FlashData's managed retrieval infrastructure, with bot handling and recovery built in. |
Map your existing yt-dlp settings
| Existing setting | FlashData setting | Migration detail |
|---|---|---|
-f / --format | yt_dlp_format | Use a format selector, not a CLI command. It overrides video_quality, fps and audio_language; output format and clipping still apply. |
--merge-output-format mp4 | download_type=audio_video, output_format=mp4 | Choose the required codec policy in the selector when your decoder needs particular codecs. |
-x --audio-format | download_type=audio, output_format | Choose M4A, MP3 or Opus; audio_quality accepts best or a supported bitrate string. |
--download-sections | start_at, end_at | Split collections into separate jobs when clip intervals differ. |
--write-info-json | upload_metadata=True | Consume the returned metadata-file reference; adapt readers to its schema. |
--write-subs / --write-auto-subs | youtube.captions(..., language=..., auto=...) | A separate request. false selects uploader-provided captions; true selects automatic captions. |
-o and an upload hook | storage.object_prefix | Choose a bucket prefix. Keep custom naming and downstream processing in your application. |
--batch-file and --download-archive | Bulk IDs file and your delivery inventory | Expand URLs/playlists into IDs and remove completed items before upload. Archives are not imported automatically. |
Move the remaining queue in five steps
- Reconcile completed work. Match your archive against actual media, uploads and required caption files. Keep a separate list of items needing only caption or upload repair.
- Prepare the remaining IDs. Expand playlists and channels, remove duplicates, and save one YouTube ID per line. Group videos by output and caption policy.
- Choose output and storage. In Bulk Jobs, select YouTube Download, attach the file, configure video + audio, MP4 and your quality or format selector, then enable metadata if needed.
- Start and save the Bulk ID. Use a new object prefix for this run. Follow Completed, Failed and Invalid counts; pause or resume this same job when needed.
- Collect and join the results. Save the manifest and result parts, then join caption results by video ID. Create a new retry file only for reviewed failures.
jNQXAC9IVRw
dQw4w9WgXcQOne output policy per Bulk job. Preserve your existing files and archive. The example below establishes the media, metadata and caption record for one item before you apply that policy to a collection.
Example: a video dataset item
Output: best available video + audio merged to MP4, source metadata, and available automatic English captions with timestamps. Both workflows use the same video ID and format selector.
| Artifact | Local yt-dlp workflow | FlashData workflow |
|---|---|---|
| Video + audio | MP4 under dataset-local/VIDEO_ID/. | MP4 in your bucket; its location and properties appear in delivery.json. |
| Metadata | A local .info.json sidecar. | A JSON sidecar in the same bucket, referenced by the media result. |
| Captions | An available English automatic-caption track saved as VTT. | A separate captions request; the example saves tracks and errors in local captions.json. |
| Dataset inventory | Join files, archive and caption availability by ID. | Local item.json joins the video ID, accepted job, media result and caption status. |
Caption origin is explicit. This example requests automatic captions. For uploader-provided captions, use --write-subs instead of --write-auto-subs, and set auto=False in the API call.
1. Run the local yt-dlp equivalent
python -m yt_dlp --js-runtimes deno --impersonate="" --no-playlist \
-f "bestvideo+bestaudio/best" --merge-output-format mp4 --remux-video mp4 \
--write-info-json --write-auto-subs --sub-langs "en" --sub-format "vtt/best" \
--sleep-subtitles 5 --concurrent-fragments 1 -P dataset-local -o "%(id)s/%(id)s.%(ext)s" \
"https://www.youtube.com/watch?v=jNQXAC9IVRw"Check captions separately. No matching track and a failed caption request require different follow-up. Keep the media already saved; do not replay a completed download to repair text.
2. Run the FlashData dataset example
Media and metadata go to your bucket. The Python example saves delivery records, caption JSON and the joined item record locally. Your dataset publisher can upload these JSON records alongside the media.
python -m pip install "flashdata>=1.2.0,<2"import json
from pathlib import Path
from flashdata import FlashData
def save_json(path, value):
path.write_text(json.dumps(value, indent=2), encoding="utf-8")
def collect_dataset_item(client, video_id, storage, directory="dataset"):
folder = Path(directory) / video_id
folder.mkdir(parents=True, exist_ok=True)
receipt = folder / "download-job-id.txt"
if not receipt.exists():
# Reserve the checkpoint before submitting; never overwrite an old ID.
with receipt.open("x", encoding="utf-8") as checkpoint:
job = client.youtube.download(
video_id,
storage=storage,
download_type="audio_video",
output_format="mp4",
yt_dlp_format="bestvideo+bestaudio/best",
upload_metadata=True,
)
checkpoint.write(job["id"])
job_id = receipt.read_text(encoding="utf-8").strip()
if not job_id:
raise RuntimeError("Submission outcome unknown; inspect job history first")
# A rerun waits for the saved job instead of downloading the video again.
client.jobs.wait(job_id, timeout=900)
delivery = client.jobs.result(job_id)
save_json(folder / "delivery.json", delivery)
# Captions are a separate YouTube Data request, with their own usage charge.
response = client.youtube.captions(video_id, language=["en"], auto=True)
captions = response["results"][0]
save_json(folder / "captions.json", captions)
item = {
"video_id": video_id,
"download_job_id": job_id,
"media": delivery["results"][0],
"captions_file": "captions.json",
"caption_status": "available" if captions["tracks"] else "needs_review",
}
save_json(folder / "item.json", item)
return item
if __name__ == "__main__":
storage = {
"storage_type": "s3",
"storage_url": "your-media-bucket",
"object_prefix": "datasets/ytdlp-migration/run-01",
"region": "us-east-1",
"access_key": "your-storage-access-key",
"secret_key": "your-storage-secret-key",
}
with FlashData(api_key="your-flashdata-api-key") as client:
item = collect_dataset_item(client, "jNQXAC9IVRw", storage)
print(json.dumps(item, indent=2))
3. Verify the item before publishing it
| Check | What to record |
|---|---|
| Media | Read the delivered object with your storage client. Check file size, MD5, duration, video/audio streams and decoder compatibility. |
| Metadata | Preserve the source video ID and the metadata-file reference. Keep that relationship when renaming objects. |
| Captions | Check tracks and errors in captions.json. Store actual languageCode, isAutoGenerated and timestamped segments. |
| Missing or failed captions | needs_review means no track was returned. language_unavailable needs a different caption policy or transcription; retrieval_failed can be retried separately. |
| Dataset index | Join media and captions by video ID. Publish only items that meet your collection's required fields. |
A caption failure does not erase the media result. The script saves delivery.json before requesting captions. If that request raises an error, rerun with the same directory to reuse the accepted download job.
Recover a stopped run without downloading again
- Client stopped while waiting: rerun managed_dataset.py with the same directory; it reads the saved ID and resumes polling.
- Download job failed: inspect its error and the original source before starting a replacement job.
- Media succeeded, captions failed: keep delivery.json and the media. Repeat the caption stage; the saved media job is reused.
- Submission outcome is unknown: an empty checkpoint deliberately blocks another submission. Find the original job in history and reconcile its ID first.
- Scaling to Bulk: save the Bulk ID and result parts, keep successful media IDs, and submit a new file only for reviewed failures.
Compare the cost of the completed collection
| Option | What you pay for |
|---|---|
| yt-dlp | $0 software usage fee, plus your actual compute, temporary disk, network/proxies and operating time. |
| FlashData PAYG | $0.10 per GB delivered ($100 per TB), with no monthly subscription. 100 GB uses $10 of credits. |
| FlashData subscriptions | Lower published usage rates with monthly fees and included shared credits. Business publishes $0.04/GB with a $1,999 monthly fee. |
PAYG volume pricing can be discussed. For recurring collections, compare your expected delivered GB with the available plan rates and included credits. Count usage already covered by a subscription only once.
Companion metadata JSON is excluded from download-volume charges. Caption requests use YouTube Data pricing; destination storage is separate.
Would a different local interface solve the problem?
- Python integration: yt-dlp already exposes YoutubeDL; the Python guide shows direct calls without shell commands.
- A desktop interface: a GUI can simplify manual work, while its downloader engine still needs updates.
- A self-hosted HTTP wrapper: clients get an API, while your team continues to operate extraction, queues and delivery.