Browse documentation
Examples
Deliver one MP4 to your Amazon S3 bucket, then retrieve and verify the file with Python or Bash.
On this page
Before you start
Use owner access with youtube_download permission and available credits. Download is not included in temporary-agent trial access. You need an existing Amazon S3 bucket and a video you are authorized to download.
Choose Python 3.10+ with requests and boto3, or Bash with cURL, jq, OpenSSL and AWS CLI v2. On Windows, use Git Bash for the Bash version.
| Credential | Used for |
|---|---|
| FLASHDATA_API_KEY | Submit the job and retrieve its status and delivery record. |
| FLASHDATA_STORAGE_ACCESS_KEY / FLASHDATA_STORAGE_SECRET_KEY | An S3 upload identity sent to FlashData for writing to your chosen bucket and prefix. |
| Your local AWS profile or role | Read the private object with boto3 or AWS CLI in step 4. It needs s3:GetObject access. |
1. Choose the video and destination
Create a private download-video folder outside your repository. Save request.json there and open a terminal in that folder. Replace storage_url with your bucket name and region with its AWS region; use an 11-character video ID.
This recipe delivers video with audio as MP4, at up to 720p, under recipes/clip-001/. Keep the bucket private. Use a fresh prefix for a separate download so another job cannot overwrite this object's path.
{
"source": "youtube_download",
"video_id": "jNQXAC9IVRw",
"params": {
"download_type": "audio_video",
"video_quality": "<=720",
"output_format": "mp4",
"storage": {
"storage_type": "s3",
"storage_url": "your-media-bucket",
"region": "us-east-1",
"object_prefix": "recipes/clip-001"
}
}
}2. Submit one download job
Choose one version and save its script beside request.json before running it. The script reads the three environment variables above or asks for them at hidden prompts. Agents should supply them from their secret store. Do not put keys in request.json.
A new job-001 folder sends one async request. It saves the request without upload credentials, a timestamp, the HTTP response and the job ID. Reusing that folder stops submission; it does not create a second job.
python -m pip install requests boto3# Save as download-video/submit.py.
import getpass
import json
import os
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
import requests
ROOT = Path(__file__).resolve().parent
def read_json(path):
return json.loads(path.read_text(encoding="utf-8-sig"))
def secret(name):
value = os.environ.get(name) or getpass.getpass(name + ": ")
if not value.strip():
raise ValueError("Missing " + name)
return value
def main():
os.umask(0o077)
folder = sys.argv[1]
if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
raise ValueError("Use a simple output folder name, such as job-001.")
directory = ROOT / folder
if directory.exists():
raise ValueError("Keep this attempt. Run collect.py with the same folder.")
request = read_json(ROOT / "request.json")
storage = request["params"]["storage"]
if (request["source"] != "youtube_download"
or not re.fullmatch(r"[A-Za-z0-9_-]{11}", request["video_id"])
or request["params"]["download_type"] != "audio_video"
or request["params"]["output_format"] != "mp4"
or storage["storage_type"] != "s3"
or set(storage) != {"storage_type", "storage_url", "region", "object_prefix"}
or not re.fullmatch(r"[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]", storage["storage_url"])
or not re.fullmatch(r"[a-z0-9-]+", storage["region"])
or not re.fullmatch(r"[A-Za-z0-9_-]+(/[A-Za-z0-9_-]+)*", storage["object_prefix"])):
raise ValueError("Check the video, MP4 options and Amazon S3 destination.")
key = secret("FLASHDATA_API_KEY")
access = secret("FLASHDATA_STORAGE_ACCESS_KEY")
secret_key = secret("FLASHDATA_STORAGE_SECRET_KEY")
directory.mkdir(mode=0o700)
# Save the plan BEFORE sending. Keep upload credentials out of this file.
(directory / "request.json").write_text(json.dumps(request, indent=2), encoding="utf-8")
(directory / "submitted-at.txt").write_text(
datetime.now(timezone.utc).isoformat(), encoding="utf-8")
storage.update(access_key=access, secret_key=secret_key)
response = requests.post(
"https://data.flashdata.dev/v1/queries", json=request,
headers={"X-API-Key": key, "Accept": "application/json"},
timeout=(10, 120), allow_redirects=False)
(directory / "submission.json").write_bytes(response.content)
(directory / "submission-headers.txt").write_text(
"\n".join(f"{k}: {v}" for k, v in response.headers.items()), encoding="utf-8")
(directory / "submission-http.txt").write_text(str(response.status_code))
if response.status_code not in (200, 201, 202):
raise ValueError(f"HTTP {response.status_code}. Inspect the saved attempt.")
receipt = response.json()
if (receipt.get("source") != "youtube_download"
or not re.fullmatch(r"[A-Za-z0-9_-]+", receipt.get("id", ""))):
raise ValueError("Missing job ID. Check History before submitting again.")
print("Submitted:", receipt["id"], "— keep", directory)
if __name__ == "__main__":
try:
main()
except (IndexError, KeyError, OSError, ValueError, requests.RequestException) as error:
print("Network request failed. Keep this attempt and inspect its saved files."
if isinstance(error, requests.RequestException) else str(error), file=sys.stderr)
sys.exit(1)python submit.py job-001curl --version
jq --version
openssl version
aws --version#!/usr/bin/env bash
# Save as download-video/submit.sh.
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
folder=${1:?Usage: bash submit.sh job-001}
[[ "$folder" =~ ^[A-Za-z0-9_-]+$ ]] || exit 2
[[ ! -e "$folder" ]] || { echo 'Keep this attempt. Run collect.sh with the same folder.' >&2; exit 1; }
jq -e '
.source == "youtube_download" and (.video_id | test("^[A-Za-z0-9_-]{11}$"))
and .params.download_type == "audio_video" and .params.output_format == "mp4"
and (.params.storage | .storage_type == "s3"
and (keys == ["object_prefix","region","storage_type","storage_url"])
and (.storage_url | test("^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$"))
and (.region | test("^[a-z0-9-]+$"))
and (.object_prefix | test("^[A-Za-z0-9_-]+(/[A-Za-z0-9_-]+)*$")))
' request.json >/dev/null
for name in FLASHDATA_API_KEY FLASHDATA_STORAGE_ACCESS_KEY FLASHDATA_STORAGE_SECRET_KEY; do
if [[ -z "${!name:-}" ]]; then
read -r -s -p "$name: " "$name"; printf '\n'
fi
[[ -n "${!name}" ]] || exit 1
export "$name"
done
mkdir -- "$folder"
# Save the plan before sending; upload credentials are added only to the pipe.
cp request.json "$folder/request.json"
date -u '+%Y-%m-%dT%H:%M:%SZ' > "$folder/submitted-at.txt"
jq '.params.storage += {
access_key: env.FLASHDATA_STORAGE_ACCESS_KEY,
secret_key: env.FLASHDATA_STORAGE_SECRET_KEY
}' "$folder/request.json" |
curl --silent --show-error --connect-timeout 10 --max-time 120 \
--request POST 'https://data.flashdata.dev/v1/queries' \
--header "X-API-Key: $FLASHDATA_API_KEY" --header 'Accept: application/json' \
--header 'Content-Type: application/json' --data-binary @- \
--dump-header "$folder/submission-headers.txt" \
--output "$folder/submission.json" --write-out '%{http_code}' > "$folder/submission-http.txt"
status=$(cat "$folder/submission-http.txt")
[[ "$status" =~ ^20[012]$ ]] || { echo "HTTP $status. Inspect the saved attempt." >&2; exit 1; }
jq -er 'if .source == "youtube_download" and (.id | test("^[A-Za-z0-9_-]+$"))
then "Submitted: " + .id else error("Missing job ID. Check History before resubmitting.") end' \
"$folder/submission.json"bash submit.sh job-001Expected receipt
Success prints Submitted: job_… and saves submission.json. The receipt confirms that work was accepted; it is not the completed delivery or final charge.
{
"id": "job_example",
"source": "youtube_download",
"status": "pending",
"usage": {
"credits": 0
}
}3. Wait for delivery and save its record
Save collect.py beside submit.py, or collect.sh beside submit.sh. Run with the same job-001 folder. It checks the saved job once. For pending or processing, wait 10 seconds and run the same command again.
When completed, it saves delivery.json. A signed redirect here retrieves the JSON delivery record, not the MP4; the script follows it without forwarding your FlashData key. A saved delivery.json is reused without another API request.
# Save as download-video/collect.py, beside submit.py.
import os
import re
import sys
from urllib.parse import urlsplit
import requests
from submit import ROOT, read_json, secret
def main():
os.umask(0o077)
folder = sys.argv[1]
if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
raise ValueError("Use the saved output folder name.")
directory = ROOT / folder
if (directory / "delivery.json").exists():
print("Already saved:", directory / "delivery.json")
return
receipt = read_json(directory / "submission.json")
job_id = receipt["id"]
if receipt["source"] != "youtube_download" or not re.fullmatch(r"[A-Za-z0-9_-]+", job_id):
raise ValueError("Inspect submission.json before continuing.")
headers = {"X-API-Key": secret("FLASHDATA_API_KEY"), "Accept": "application/json"}
url = "https://api.flashdata.dev/v2/jobs/" + job_id
def get(url, name, headers):
response = requests.get(url, headers=headers, timeout=(10, 120), allow_redirects=False)
(directory / name).write_bytes(response.content)
(directory / (name + ".headers")).write_text(
"\n".join(f"{k}: {v}" for k, v in response.headers.items()), encoding="utf-8")
(directory / (name + ".http")).write_text(str(response.status_code))
return response
status = get(url, "job-status.json", headers)
if status.status_code != 200:
raise ValueError(f"HTTP {status.status_code}. Inspect job-status.json and its headers.")
job = status.json()["job"]
if job["id"] != job_id or job["source"] != "youtube_download":
raise ValueError("The job does not match the saved receipt.")
if job["status"] in ("pending", "processing"):
print(job["status"] + ". Wait 10 seconds, then run this command again.")
return
if job["status"] != "completed":
raise ValueError("Job " + job["status"] + ". Inspect job-status.json; do not resubmit.")
result = get(url + "/results", "delivery.pending.json", headers)
if result.status_code == 302:
location = result.headers.get("Location", "")
parsed = urlsplit(location)
if parsed.scheme != "https" or not parsed.hostname or parsed.username or parsed.password:
raise ValueError("Expected an HTTPS result URL.")
# This URL retrieves the JSON delivery record, not the video.
# Never forward the FlashData key to the signed storage URL.
result = get(location, "delivery.pending.json", {})
if result.status_code != 200:
raise ValueError(f"HTTP {result.status_code}. Run collect.py again after resolving it.")
delivery = result.json()
if (delivery.get("source") != "youtube_download" or delivery.get("status") != "done"
or not isinstance(delivery.get("results"), list) or len(delivery["results"]) != 1):
raise ValueError("Inspect delivery.pending.json before using the result.")
(directory / "delivery.pending.json").replace(directory / "delivery.json")
print("Saved:", directory / "delivery.json")
print("Usage:", delivery.get("usage"))
if __name__ == "__main__":
try:
main()
except (IndexError, KeyError, OSError, ValueError, requests.RequestException) as error:
print("Network request failed. Keep this attempt and inspect its saved files."
if isinstance(error, requests.RequestException) else str(error), file=sys.stderr)
sys.exit(1)python collect.py job-001#!/usr/bin/env bash
# Save as download-video/collect.sh.
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
folder=${1:?Usage: bash collect.sh job-001}
[[ "$folder" =~ ^[A-Za-z0-9_-]+$ ]] || exit 2
if [[ -f "$folder/delivery.json" ]]; then
echo "Already saved: $folder/delivery.json"
exit 0
fi
id=$(jq -er 'select(.source == "youtube_download") | .id
| select(test("^[A-Za-z0-9_-]+$"))' "$folder/submission.json")
if [[ -z "${FLASHDATA_API_KEY:-}" ]]; then
read -r -s -p 'FLASHDATA_API_KEY: ' FLASHDATA_API_KEY; printf '\n'
fi
: "${FLASHDATA_API_KEY:?Set your key}"
url="https://api.flashdata.dev/v2/jobs/$id"
status=$(curl --silent --show-error --connect-timeout 10 --max-time 120 \
--header "X-API-Key: $FLASHDATA_API_KEY" --header 'Accept: application/json' \
--dump-header "$folder/job-status.json.headers" --output "$folder/job-status.json" \
--write-out '%{http_code}' "$url")
printf '%s' "$status" > "$folder/job-status.json.http"
[[ "$status" == 200 ]] || { echo "HTTP $status. Inspect job-status.json and its headers." >&2; exit 1; }
state=$(jq -er --arg id "$id" 'select(.job.id == $id and .job.source == "youtube_download")
| .job.status' "$folder/job-status.json")
case "$state" in
pending|processing) echo "$state. Wait 10 seconds, then run this command again."; exit 0 ;;
completed) ;;
*) echo "Job $state. Inspect job-status.json; do not resubmit." >&2; exit 1 ;;
esac
status=$(curl --silent --show-error --connect-timeout 10 --max-time 120 \
--header "X-API-Key: $FLASHDATA_API_KEY" --header 'Accept: application/json' \
--dump-header "$folder/result-headers.txt" --output "$folder/delivery.pending.json" \
--write-out '%{http_code}' "$url/results")
printf '%s' "$status" > "$folder/result-http.txt"
if [[ "$status" == 302 ]]; then
location=$(awk 'tolower($1)=="location:" {sub(/\r$/, "", $2); print $2}' "$folder/result-headers.txt")
[[ "$location" =~ ^https://[^/@[:space:]]+(/|$) ]] || { echo 'Expected an HTTPS result URL.' >&2; exit 1; }
# Retrieve the JSON delivery record without the FlashData key. Do not add -L.
status=$(curl --silent --show-error --compressed --proto '=https' \
--connect-timeout 10 --max-time 120 --dump-header "$folder/storage-result-headers.txt" \
--output "$folder/delivery.pending.json" --write-out '%{http_code}' "$location")
printf '%s' "$status" > "$folder/storage-result-http.txt"
fi
[[ "$status" == 200 ]] || { echo "HTTP $status. Resolve the error, then run collect.sh again." >&2; exit 1; }
jq -e '.source == "youtube_download" and .status == "done"
and (.results | type == "array" and length == 1)' "$folder/delivery.pending.json" >/dev/null
mv "$folder/delivery.pending.json" "$folder/delivery.json"
echo "Saved: $folder/delivery.json"
jq '{usage}' "$folder/delivery.json"bash collect.sh job-001What delivery.json contains
The result includes file_url, file_size in bytes and md5. usage reports the completed job's charge. Keep the full record. The file URL names your S3 object; it does not make a private bucket publicly readable.
{
"source": "youtube_download",
"status": "done",
"results": [
{
"file_url": "https://your-media-bucket.s3.us-east-1.amazonaws.com/recipes/clip-001/jNQXAC9IVRw.mp4",
"file_size": 1048576,
"md5": "0123456789abcdef0123456789abcdef"
}
]
}4. Read the private object and verify it
Save the verification script beside the other scripts. Use your configured local AWS identity with read access to the object. The script checks that the delivery belongs to the planned S3 bucket, prefix and video filename, then downloads it to media.mp4.part.
Only a matching byte count and MD5 produce media.mp4 and verified.json. Running verification again checks an existing media.mp4 locally. This step reads your S3 object; it does not submit another FlashData download.
# Save as download-video/verify.py, beside submit.py.
import hashlib
import json
import os
import re
import sys
from contextlib import closing
from datetime import datetime, timezone
from urllib.parse import urlsplit
import boto3
from submit import ROOT, read_json
def main():
os.umask(0o077)
folder = sys.argv[1]
if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
raise ValueError("Use the saved output folder name.")
directory = ROOT / folder
request = read_json(directory / "request.json")
receipt = read_json(directory / "submission.json")
job = read_json(directory / "job-status.json")["job"]
delivery = read_json(directory / "delivery.json")
if (request["source"] != "youtube_download" or receipt["source"] != "youtube_download"
or job["id"] != receipt["id"] or job["source"] != "youtube_download"
or job["status"] != "completed" or delivery.get("source") != "youtube_download"
or delivery.get("status") != "done" or len(delivery["results"]) != 1):
raise ValueError("Check the saved request, receipt, status and delivery.")
storage = request["params"]["storage"]
bucket, region, prefix = (storage[k] for k in ("storage_url", "region", "object_prefix"))
file = delivery["results"][0]
parsed = urlsplit(file["file_url"])
# This recipe uses an ASCII prefix and the MP4 filename generated for the video.
object_key = prefix + "/" + request["video_id"] + ".mp4"
if (storage["storage_type"] != "s3" or parsed.scheme != "https"
or parsed.netloc != bucket + ".s3." + region + ".amazonaws.com"
or parsed.path != "/" + object_key or parsed.query or parsed.fragment
or type(file.get("file_size")) is not int or file["file_size"] <= 0
or not re.fullmatch(r"[A-Fa-f0-9]{32}", file.get("md5", ""))):
raise ValueError("Delivery URL, byte count or MD5 does not match this recipe's request.")
local = directory / "media.mp4"
pending = directory / "media.mp4.part"
if not local.exists():
# Boto3 uses your local AWS read identity, not the upload key sent to FlashData.
with pending.open("xb") as output:
s3 = boto3.client("s3", region_name=region)
with closing(s3.get_object(Bucket=bucket, Key=object_key)["Body"]) as stream:
while chunk := stream.read(1024 * 1024):
output.write(chunk)
local = pending
size, digest = 0, hashlib.md5(usedforsecurity=False)
with local.open("rb") as stream:
while chunk := stream.read(1024 * 1024):
size += len(chunk)
digest.update(chunk)
if size != file["file_size"] or digest.hexdigest() != file["md5"].lower():
raise ValueError("Size or MD5 mismatch. Keep the local file for inspection.")
final = directory / "media.mp4"
if local == pending:
pending.rename(final)
record = {
"job_id": receipt["id"], "video_id": request["video_id"],
"bucket": bucket, "object_key": object_key, "bytes": size, "md5": digest.hexdigest(),
"verified_at": datetime.now(timezone.utc).isoformat(), "usage": delivery.get("usage"),
}
(directory / "verified.json").write_text(json.dumps(record, indent=2), encoding="utf-8")
print("Verified:", final, "—", size, "bytes; MD5 matches.")
if __name__ == "__main__":
try:
main()
except Exception as error:
print(str(error), file=sys.stderr)
sys.exit(1)python verify.py job-001#!/usr/bin/env bash
# Save as download-video/verify.sh.
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
folder=${1:?Usage: bash verify.sh job-001}
[[ "$folder" =~ ^[A-Za-z0-9_-]+$ ]] || exit 2
jq -e --slurpfile r "$folder/request.json" --slurpfile s "$folder/submission.json" \
--slurpfile j "$folder/job-status.json" '
$r[0] as $r | $s[0] as $s | $j[0].job as $j
| $r.params.storage as $store
| ($store.object_prefix + "/" + $r.video_id + ".mp4") as $key
| .results[0] as $f
| $r.source == "youtube_download" and $s.source == "youtube_download"
and $j.id == $s.id and $j.source == "youtube_download" and $j.status == "completed"
and .source == "youtube_download" and .status == "done"
and (.results | type == "array" and length == 1)
and $store.storage_type == "s3"
and $f.file_url == ("https://" + $store.storage_url + ".s3." + $store.region + ".amazonaws.com/" + $key)
and ($f.file_size | type == "number" and . > 0 and . == floor)
and ($f.md5 | test("^[A-Fa-f0-9]{32}$"))
' "$folder/delivery.json" >/dev/null
bucket=$(jq -er '.params.storage.storage_url' "$folder/request.json")
region=$(jq -er '.params.storage.region' "$folder/request.json")
key=$(jq -er '.params.storage.object_prefix + "/" + .video_id + ".mp4"' "$folder/request.json")
local_file="$folder/media.mp4"
if [[ ! -e "$local_file" ]]; then
local_file="$folder/media.mp4.part"
[[ ! -e "$local_file" ]] || { echo 'Keep the partial file; inspect it before retrying.' >&2; exit 1; }
# AWS CLI uses your configured read identity; no FlashData key is sent to S3.
aws s3 cp "s3://$bucket/$key" "$local_file" --region "$region" --only-show-errors
fi
bytes=$(wc -c < "$local_file" | tr -d '[:space:]')
md5=$(openssl dgst -md5 "$local_file" | awk '{print tolower($NF)}')
expected_bytes=$(jq -er '.results[0].file_size' "$folder/delivery.json")
expected_md5=$(jq -er '.results[0].md5 | ascii_downcase' "$folder/delivery.json")
[[ "$bytes" == "$expected_bytes" && "$md5" == "$expected_md5" ]] ||
{ echo 'Size or MD5 mismatch. Keep the local file for inspection.' >&2; exit 1; }
if [[ "$local_file" != "$folder/media.mp4" ]]; then
mv "$local_file" "$folder/media.mp4"
fi
jq --slurpfile r "$folder/request.json" --slurpfile s "$folder/submission.json" \
--arg key "$key" --arg md5 "$md5" --argjson bytes "$bytes" '
{job_id: $s[0].id, video_id: $r[0].video_id, bucket: $r[0].params.storage.storage_url,
object_key: $key, bytes: $bytes, md5: $md5, verified_at: (now | todateiso8601), usage}
' "$folder/delivery.json" > "$folder/verified.json"
echo "Verified: $folder/media.mp4 — $bytes bytes; MD5 matches."bash verify.sh job-001Successful result
The byte count above is illustrative. You now have the MP4 in your bucket, a verified local copy, and verified.json tying the file to its job ID, video ID, object key, checksum and reported usage.
Verified: .../job-001/media.mp4 — 1048576 bytes; MD5 matches.Resume without a second download
| What happened | Next action |
|---|---|
| Submission timed out or no job ID was saved | Keep job-001 and check History using its timestamp and video. Do not delete the folder to force another submission. Recover the original job before deciding whether a new request is needed. |
| Pending / processing | Wait 10 seconds, then rerun collect with job-001. It only reads the existing job. |
| Job failed | Inspect job-status.json for the error. Fix the video or storage configuration before explicitly starting a new job in a new folder and prefix. |
| 401 / 403 / 402 / 429 | Check the key, permission or balance. For 429, read Retry-After in the saved headers and wait. The scripts do not automatically retry submission. |
| Fetching the result failed or its signed URL expired | Rerun collect for job-001 to request the existing result again. |
| S3 AccessDenied | Fix the local AWS read identity or bucket policy. FlashData upload access and local read access are separate. |
| Partial file, wrong path, byte count or MD5 | Keep the saved delivery and file for inspection. For a fresh S3 read after resolving the cause, move media.mp4.part aside, then rerun verify. Do not rerun submit. |