Skip to content
EN
Get API key
Browse documentation

Examples

Download a video to your storage

Deliver one MP4 to your Amazon S3 bucket, then retrieve and verify the file with Python or Bash.

On this page

Before you start

Use owner access with youtube_download permission and available credits. Download is not included in temporary-agent trial access. You need an existing Amazon S3 bucket and a video you are authorized to download.

Choose Python 3.10+ with requests and boto3, or Bash with cURL, jq, OpenSSL and AWS CLI v2. On Windows, use Git Bash for the Bash version.

CredentialUsed for
FLASHDATA_API_KEYSubmit the job and retrieve its status and delivery record.
FLASHDATA_STORAGE_ACCESS_KEY / FLASHDATA_STORAGE_SECRET_KEYAn S3 upload identity sent to FlashData for writing to your chosen bucket and prefix.
Your local AWS profile or roleRead the private object with boto3 or AWS CLI in step 4. It needs s3:GetObject access.

1. Choose the video and destination

Create a private download-video folder outside your repository. Save request.json there and open a terminal in that folder. Replace storage_url with your bucket name and region with its AWS region; use an 11-character video ID.

This recipe delivers video with audio as MP4, at up to 720p, under recipes/clip-001/. Keep the bucket private. Use a fresh prefix for a separate download so another job cannot overwrite this object's path.

download-video/request.json
{
  "source": "youtube_download",
  "video_id": "jNQXAC9IVRw",
  "params": {
    "download_type": "audio_video",
    "video_quality": "<=720",
    "output_format": "mp4",
    "storage": {
      "storage_type": "s3",
      "storage_url": "your-media-bucket",
      "region": "us-east-1",
      "object_prefix": "recipes/clip-001"
    }
  }
}

2. Submit one download job

Choose one version and save its script beside request.json before running it. The script reads the three environment variables above or asks for them at hidden prompts. Agents should supply them from their secret store. Do not put keys in request.json.

A new job-001 folder sends one async request. It saves the request without upload credentials, a timestamp, the HTTP response and the job ID. Reusing that folder stops submission; it does not create a second job.

Install in your Python environment
python -m pip install requests boto3
download-video/submit.py
# Save as download-video/submit.py.
import getpass
import json
import os
import re
import sys
from datetime import datetime, timezone
from pathlib import Path

import requests

ROOT = Path(__file__).resolve().parent


def read_json(path):
    return json.loads(path.read_text(encoding="utf-8-sig"))


def secret(name):
    value = os.environ.get(name) or getpass.getpass(name + ": ")
    if not value.strip():
        raise ValueError("Missing " + name)
    return value


def main():
    os.umask(0o077)
    folder = sys.argv[1]
    if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
        raise ValueError("Use a simple output folder name, such as job-001.")
    directory = ROOT / folder
    if directory.exists():
        raise ValueError("Keep this attempt. Run collect.py with the same folder.")
    request = read_json(ROOT / "request.json")
    storage = request["params"]["storage"]
    if (request["source"] != "youtube_download"
            or not re.fullmatch(r"[A-Za-z0-9_-]{11}", request["video_id"])
            or request["params"]["download_type"] != "audio_video"
            or request["params"]["output_format"] != "mp4"
            or storage["storage_type"] != "s3"
            or set(storage) != {"storage_type", "storage_url", "region", "object_prefix"}
            or not re.fullmatch(r"[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]", storage["storage_url"])
            or not re.fullmatch(r"[a-z0-9-]+", storage["region"])
            or not re.fullmatch(r"[A-Za-z0-9_-]+(/[A-Za-z0-9_-]+)*", storage["object_prefix"])):
        raise ValueError("Check the video, MP4 options and Amazon S3 destination.")
    key = secret("FLASHDATA_API_KEY")
    access = secret("FLASHDATA_STORAGE_ACCESS_KEY")
    secret_key = secret("FLASHDATA_STORAGE_SECRET_KEY")
    directory.mkdir(mode=0o700)
    # Save the plan BEFORE sending. Keep upload credentials out of this file.
    (directory / "request.json").write_text(json.dumps(request, indent=2), encoding="utf-8")
    (directory / "submitted-at.txt").write_text(
        datetime.now(timezone.utc).isoformat(), encoding="utf-8")
    storage.update(access_key=access, secret_key=secret_key)
    response = requests.post(
        "https://data.flashdata.dev/v1/queries", json=request,
        headers={"X-API-Key": key, "Accept": "application/json"},
        timeout=(10, 120), allow_redirects=False)
    (directory / "submission.json").write_bytes(response.content)
    (directory / "submission-headers.txt").write_text(
        "\n".join(f"{k}: {v}" for k, v in response.headers.items()), encoding="utf-8")
    (directory / "submission-http.txt").write_text(str(response.status_code))
    if response.status_code not in (200, 201, 202):
        raise ValueError(f"HTTP {response.status_code}. Inspect the saved attempt.")
    receipt = response.json()
    if (receipt.get("source") != "youtube_download"
            or not re.fullmatch(r"[A-Za-z0-9_-]+", receipt.get("id", ""))):
        raise ValueError("Missing job ID. Check History before submitting again.")
    print("Submitted:", receipt["id"], "— keep", directory)


if __name__ == "__main__":
    try:
        main()
    except (IndexError, KeyError, OSError, ValueError, requests.RequestException) as error:
        print("Network request failed. Keep this attempt and inspect its saved files."
              if isinstance(error, requests.RequestException) else str(error), file=sys.stderr)
        sys.exit(1)
Run from download-video/
python submit.py job-001

Expected receipt

Success prints Submitted: job_… and saves submission.json. The receipt confirms that work was accepted; it is not the completed delivery or final charge.

job-001/submission.json · abbreviated
{
  "id": "job_example",
  "source": "youtube_download",
  "status": "pending",
  "usage": {
    "credits": 0
  }
}

3. Wait for delivery and save its record

Save collect.py beside submit.py, or collect.sh beside submit.sh. Run with the same job-001 folder. It checks the saved job once. For pending or processing, wait 10 seconds and run the same command again.

When completed, it saves delivery.json. A signed redirect here retrieves the JSON delivery record, not the MP4; the script follows it without forwarding your FlashData key. A saved delivery.json is reused without another API request.

download-video/collect.py
# Save as download-video/collect.py, beside submit.py.
import os
import re
import sys
from urllib.parse import urlsplit

import requests
from submit import ROOT, read_json, secret


def main():
    os.umask(0o077)
    folder = sys.argv[1]
    if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
        raise ValueError("Use the saved output folder name.")
    directory = ROOT / folder
    if (directory / "delivery.json").exists():
        print("Already saved:", directory / "delivery.json")
        return
    receipt = read_json(directory / "submission.json")
    job_id = receipt["id"]
    if receipt["source"] != "youtube_download" or not re.fullmatch(r"[A-Za-z0-9_-]+", job_id):
        raise ValueError("Inspect submission.json before continuing.")
    headers = {"X-API-Key": secret("FLASHDATA_API_KEY"), "Accept": "application/json"}
    url = "https://api.flashdata.dev/v2/jobs/" + job_id

    def get(url, name, headers):
        response = requests.get(url, headers=headers, timeout=(10, 120), allow_redirects=False)
        (directory / name).write_bytes(response.content)
        (directory / (name + ".headers")).write_text(
            "\n".join(f"{k}: {v}" for k, v in response.headers.items()), encoding="utf-8")
        (directory / (name + ".http")).write_text(str(response.status_code))
        return response

    status = get(url, "job-status.json", headers)
    if status.status_code != 200:
        raise ValueError(f"HTTP {status.status_code}. Inspect job-status.json and its headers.")
    job = status.json()["job"]
    if job["id"] != job_id or job["source"] != "youtube_download":
        raise ValueError("The job does not match the saved receipt.")
    if job["status"] in ("pending", "processing"):
        print(job["status"] + ". Wait 10 seconds, then run this command again.")
        return
    if job["status"] != "completed":
        raise ValueError("Job " + job["status"] + ". Inspect job-status.json; do not resubmit.")
    result = get(url + "/results", "delivery.pending.json", headers)
    if result.status_code == 302:
        location = result.headers.get("Location", "")
        parsed = urlsplit(location)
        if parsed.scheme != "https" or not parsed.hostname or parsed.username or parsed.password:
            raise ValueError("Expected an HTTPS result URL.")
        # This URL retrieves the JSON delivery record, not the video.
        # Never forward the FlashData key to the signed storage URL.
        result = get(location, "delivery.pending.json", {})
    if result.status_code != 200:
        raise ValueError(f"HTTP {result.status_code}. Run collect.py again after resolving it.")
    delivery = result.json()
    if (delivery.get("source") != "youtube_download" or delivery.get("status") != "done"
            or not isinstance(delivery.get("results"), list) or len(delivery["results"]) != 1):
        raise ValueError("Inspect delivery.pending.json before using the result.")
    (directory / "delivery.pending.json").replace(directory / "delivery.json")
    print("Saved:", directory / "delivery.json")
    print("Usage:", delivery.get("usage"))


if __name__ == "__main__":
    try:
        main()
    except (IndexError, KeyError, OSError, ValueError, requests.RequestException) as error:
        print("Network request failed. Keep this attempt and inspect its saved files."
              if isinstance(error, requests.RequestException) else str(error), file=sys.stderr)
        sys.exit(1)
Check the saved job
python collect.py job-001

What delivery.json contains

The result includes file_url, file_size in bytes and md5. usage reports the completed job's charge. Keep the full record. The file URL names your S3 object; it does not make a private bucket publicly readable.

job-001/delivery.json · illustrative fields
{
  "source": "youtube_download",
  "status": "done",
  "results": [
    {
      "file_url": "https://your-media-bucket.s3.us-east-1.amazonaws.com/recipes/clip-001/jNQXAC9IVRw.mp4",
      "file_size": 1048576,
      "md5": "0123456789abcdef0123456789abcdef"
    }
  ]
}

4. Read the private object and verify it

Save the verification script beside the other scripts. Use your configured local AWS identity with read access to the object. The script checks that the delivery belongs to the planned S3 bucket, prefix and video filename, then downloads it to media.mp4.part.

Only a matching byte count and MD5 produce media.mp4 and verified.json. Running verification again checks an existing media.mp4 locally. This step reads your S3 object; it does not submit another FlashData download.

download-video/verify.py
# Save as download-video/verify.py, beside submit.py.
import hashlib
import json
import os
import re
import sys
from contextlib import closing
from datetime import datetime, timezone
from urllib.parse import urlsplit

import boto3
from submit import ROOT, read_json


def main():
    os.umask(0o077)
    folder = sys.argv[1]
    if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
        raise ValueError("Use the saved output folder name.")
    directory = ROOT / folder
    request = read_json(directory / "request.json")
    receipt = read_json(directory / "submission.json")
    job = read_json(directory / "job-status.json")["job"]
    delivery = read_json(directory / "delivery.json")
    if (request["source"] != "youtube_download" or receipt["source"] != "youtube_download"
            or job["id"] != receipt["id"] or job["source"] != "youtube_download"
            or job["status"] != "completed" or delivery.get("source") != "youtube_download"
            or delivery.get("status") != "done" or len(delivery["results"]) != 1):
        raise ValueError("Check the saved request, receipt, status and delivery.")
    storage = request["params"]["storage"]
    bucket, region, prefix = (storage[k] for k in ("storage_url", "region", "object_prefix"))
    file = delivery["results"][0]
    parsed = urlsplit(file["file_url"])
    # This recipe uses an ASCII prefix and the MP4 filename generated for the video.
    object_key = prefix + "/" + request["video_id"] + ".mp4"
    if (storage["storage_type"] != "s3" or parsed.scheme != "https"
            or parsed.netloc != bucket + ".s3." + region + ".amazonaws.com"
            or parsed.path != "/" + object_key or parsed.query or parsed.fragment
            or type(file.get("file_size")) is not int or file["file_size"] <= 0
            or not re.fullmatch(r"[A-Fa-f0-9]{32}", file.get("md5", ""))):
        raise ValueError("Delivery URL, byte count or MD5 does not match this recipe's request.")
    local = directory / "media.mp4"
    pending = directory / "media.mp4.part"
    if not local.exists():
        # Boto3 uses your local AWS read identity, not the upload key sent to FlashData.
        with pending.open("xb") as output:
            s3 = boto3.client("s3", region_name=region)
            with closing(s3.get_object(Bucket=bucket, Key=object_key)["Body"]) as stream:
                while chunk := stream.read(1024 * 1024):
                    output.write(chunk)
        local = pending
    size, digest = 0, hashlib.md5(usedforsecurity=False)
    with local.open("rb") as stream:
        while chunk := stream.read(1024 * 1024):
            size += len(chunk)
            digest.update(chunk)
    if size != file["file_size"] or digest.hexdigest() != file["md5"].lower():
        raise ValueError("Size or MD5 mismatch. Keep the local file for inspection.")
    final = directory / "media.mp4"
    if local == pending:
        pending.rename(final)
    record = {
        "job_id": receipt["id"], "video_id": request["video_id"],
        "bucket": bucket, "object_key": object_key, "bytes": size, "md5": digest.hexdigest(),
        "verified_at": datetime.now(timezone.utc).isoformat(), "usage": delivery.get("usage"),
    }
    (directory / "verified.json").write_text(json.dumps(record, indent=2), encoding="utf-8")
    print("Verified:", final, "—", size, "bytes; MD5 matches.")


if __name__ == "__main__":
    try:
        main()
    except Exception as error:
        print(str(error), file=sys.stderr)
        sys.exit(1)
Verify the saved delivery
python verify.py job-001

Successful result

The byte count above is illustrative. You now have the MP4 in your bucket, a verified local copy, and verified.json tying the file to its job ID, video ID, object key, checksum and reported usage.

Terminal
Verified: .../job-001/media.mp4 — 1048576 bytes; MD5 matches.

Resume without a second download

What happenedNext action
Submission timed out or no job ID was savedKeep job-001 and check History using its timestamp and video. Do not delete the folder to force another submission. Recover the original job before deciding whether a new request is needed.
Pending / processingWait 10 seconds, then rerun collect with job-001. It only reads the existing job.
Job failedInspect job-status.json for the error. Fix the video or storage configuration before explicitly starting a new job in a new folder and prefix.
401 / 403 / 402 / 429Check the key, permission or balance. For 429, read Retry-After in the saved headers and wait. The scripts do not automatically retry submission.
Fetching the result failed or its signed URL expiredRerun collect for job-001 to request the existing result again.
S3 AccessDeniedFix the local AWS read identity or bucket policy. FlashData upload access and local read access are separate.
Partial file, wrong path, byte count or MD5Keep the saved delivery and file for inspection. For a fresh S3 read after resolving the cause, move media.mp4.part aside, then rerun verify. Do not rerun submit.