浏览文档
示例
使用 Python 或 Bash 将一个 MP4 交付到你的 Amazon S3 存储桶,再取回文件并验证。
本页内容
开始前准备
需要具有 youtube_download 权限的所有者访问及可用额度。临时智能体试用不包含下载。准备已有的 Amazon S3 存储桶,以及你有权下载的视频。
选择 Python 3.10+、requests 和 boto3,或 Bash、cURL、jq、OpenSSL 和 AWS CLI v2。Windows 的 Bash 版本请使用 Git Bash。
| 凭据 | 用途 |
|---|---|
| FLASHDATA_API_KEY | 提交任务并获取状态和交付记录。 |
| FLASHDATA_STORAGE_ACCESS_KEY / FLASHDATA_STORAGE_SECRET_KEY | 发送给 FlashData 的 S3 上传身份,用于写入指定存储桶及前缀。 |
| 本地 AWS 配置档或角色 | 在步骤 4 使用 boto3 或 AWS CLI 读取私有对象,需要 s3:GetObject 权限。 |
1. 选择视频和目标存储
在代码仓库外创建私有 download-video 文件夹,将 request.json 保存到其中,并在该文件夹打开终端。将 storage_url 替换为存储桶名称,region 替换为其 AWS 区域;视频使用 11 位 ID。
本例将带音频的视频以最高 720p 的 MP4 交付到 recipes/clip-001/ 下。保持存储桶私有。另一次下载请使用新前缀,避免其他任务覆盖此对象路径。
{
"source": "youtube_download",
"video_id": "jNQXAC9IVRw",
"params": {
"download_type": "audio_video",
"video_quality": "<=720",
"output_format": "mp4",
"storage": {
"storage_type": "s3",
"storage_url": "your-media-bucket",
"region": "us-east-1",
"object_prefix": "recipes/clip-001"
}
}
}2. 提交一个下载任务
选择一种版本,将脚本保存到 request.json 旁,再运行。脚本读取上方三个环境变量,或通过隐藏输入提示获取。智能体应从密钥存储提供这些变量。不要将密钥写入 request.json。
新建 job-001 文件夹时发送一次异步请求,保存不含上传凭据的请求、时间戳、HTTP 响应及任务 ID。复用该文件夹会停止提交,不会创建第二个任务。
python -m pip install requests boto3# Save as download-video/submit.py.
import getpass
import json
import os
import re
import sys
from datetime import datetime, timezone
from pathlib import Path
import requests
ROOT = Path(__file__).resolve().parent
def read_json(path):
return json.loads(path.read_text(encoding="utf-8-sig"))
def secret(name):
value = os.environ.get(name) or getpass.getpass(name + ": ")
if not value.strip():
raise ValueError("Missing " + name)
return value
def main():
os.umask(0o077)
folder = sys.argv[1]
if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
raise ValueError("Use a simple output folder name, such as job-001.")
directory = ROOT / folder
if directory.exists():
raise ValueError("Keep this attempt. Run collect.py with the same folder.")
request = read_json(ROOT / "request.json")
storage = request["params"]["storage"]
if (request["source"] != "youtube_download"
or not re.fullmatch(r"[A-Za-z0-9_-]{11}", request["video_id"])
or request["params"]["download_type"] != "audio_video"
or request["params"]["output_format"] != "mp4"
or storage["storage_type"] != "s3"
or set(storage) != {"storage_type", "storage_url", "region", "object_prefix"}
or not re.fullmatch(r"[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]", storage["storage_url"])
or not re.fullmatch(r"[a-z0-9-]+", storage["region"])
or not re.fullmatch(r"[A-Za-z0-9_-]+(/[A-Za-z0-9_-]+)*", storage["object_prefix"])):
raise ValueError("Check the video, MP4 options and Amazon S3 destination.")
key = secret("FLASHDATA_API_KEY")
access = secret("FLASHDATA_STORAGE_ACCESS_KEY")
secret_key = secret("FLASHDATA_STORAGE_SECRET_KEY")
directory.mkdir(mode=0o700)
# Save the plan BEFORE sending. Keep upload credentials out of this file.
(directory / "request.json").write_text(json.dumps(request, indent=2), encoding="utf-8")
(directory / "submitted-at.txt").write_text(
datetime.now(timezone.utc).isoformat(), encoding="utf-8")
storage.update(access_key=access, secret_key=secret_key)
response = requests.post(
"https://data.flashdata.dev/v1/queries", json=request,
headers={"X-API-Key": key, "Accept": "application/json"},
timeout=(10, 120), allow_redirects=False)
(directory / "submission.json").write_bytes(response.content)
(directory / "submission-headers.txt").write_text(
"\n".join(f"{k}: {v}" for k, v in response.headers.items()), encoding="utf-8")
(directory / "submission-http.txt").write_text(str(response.status_code))
if response.status_code not in (200, 201, 202):
raise ValueError(f"HTTP {response.status_code}. Inspect the saved attempt.")
receipt = response.json()
if (receipt.get("source") != "youtube_download"
or not re.fullmatch(r"[A-Za-z0-9_-]+", receipt.get("id", ""))):
raise ValueError("Missing job ID. Check History before submitting again.")
print("Submitted:", receipt["id"], "— keep", directory)
if __name__ == "__main__":
try:
main()
except (IndexError, KeyError, OSError, ValueError, requests.RequestException) as error:
print("Network request failed. Keep this attempt and inspect its saved files."
if isinstance(error, requests.RequestException) else str(error), file=sys.stderr)
sys.exit(1)python submit.py job-001curl --version
jq --version
openssl version
aws --version#!/usr/bin/env bash
# Save as download-video/submit.sh.
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
folder=${1:?Usage: bash submit.sh job-001}
[[ "$folder" =~ ^[A-Za-z0-9_-]+$ ]] || exit 2
[[ ! -e "$folder" ]] || { echo 'Keep this attempt. Run collect.sh with the same folder.' >&2; exit 1; }
jq -e '
.source == "youtube_download" and (.video_id | test("^[A-Za-z0-9_-]{11}$"))
and .params.download_type == "audio_video" and .params.output_format == "mp4"
and (.params.storage | .storage_type == "s3"
and (keys == ["object_prefix","region","storage_type","storage_url"])
and (.storage_url | test("^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$"))
and (.region | test("^[a-z0-9-]+$"))
and (.object_prefix | test("^[A-Za-z0-9_-]+(/[A-Za-z0-9_-]+)*$")))
' request.json >/dev/null
for name in FLASHDATA_API_KEY FLASHDATA_STORAGE_ACCESS_KEY FLASHDATA_STORAGE_SECRET_KEY; do
if [[ -z "${!name:-}" ]]; then
read -r -s -p "$name: " "$name"; printf '\n'
fi
[[ -n "${!name}" ]] || exit 1
export "$name"
done
mkdir -- "$folder"
# Save the plan before sending; upload credentials are added only to the pipe.
cp request.json "$folder/request.json"
date -u '+%Y-%m-%dT%H:%M:%SZ' > "$folder/submitted-at.txt"
jq '.params.storage += {
access_key: env.FLASHDATA_STORAGE_ACCESS_KEY,
secret_key: env.FLASHDATA_STORAGE_SECRET_KEY
}' "$folder/request.json" |
curl --silent --show-error --connect-timeout 10 --max-time 120 \
--request POST 'https://data.flashdata.dev/v1/queries' \
--header "X-API-Key: $FLASHDATA_API_KEY" --header 'Accept: application/json' \
--header 'Content-Type: application/json' --data-binary @- \
--dump-header "$folder/submission-headers.txt" \
--output "$folder/submission.json" --write-out '%{http_code}' > "$folder/submission-http.txt"
status=$(cat "$folder/submission-http.txt")
[[ "$status" =~ ^20[012]$ ]] || { echo "HTTP $status. Inspect the saved attempt." >&2; exit 1; }
jq -er 'if .source == "youtube_download" and (.id | test("^[A-Za-z0-9_-]+$"))
then "Submitted: " + .id else error("Missing job ID. Check History before resubmitting.") end' \
"$folder/submission.json"bash submit.sh job-001预期回执
成功后输出 Submitted: job_… 并保存 submission.json。回执表示任务已被接受,不代表交付完成或最终扣费。
{
"id": "job_example",
"source": "youtube_download",
"status": "pending",
"usage": {
"credits": 0
}
}3. 等待交付并保存记录
将 collect.py 保存到 submit.py 旁,或将 collect.sh 保存到 submit.sh 旁。使用相同的 job-001 文件夹运行。它检查一次已保存任务;如果状态为 pending 或 processing,等待 10 秒后运行相同命令。
状态为 completed 时保存 delivery.json。此处的签名重定向用于获取 JSON 交付记录,不是 MP4;脚本不会向重定向地址转发 FlashData 密钥。已有 delivery.json 时直接复用,不再调用 API。
# Save as download-video/collect.py, beside submit.py.
import os
import re
import sys
from urllib.parse import urlsplit
import requests
from submit import ROOT, read_json, secret
def main():
os.umask(0o077)
folder = sys.argv[1]
if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
raise ValueError("Use the saved output folder name.")
directory = ROOT / folder
if (directory / "delivery.json").exists():
print("Already saved:", directory / "delivery.json")
return
receipt = read_json(directory / "submission.json")
job_id = receipt["id"]
if receipt["source"] != "youtube_download" or not re.fullmatch(r"[A-Za-z0-9_-]+", job_id):
raise ValueError("Inspect submission.json before continuing.")
headers = {"X-API-Key": secret("FLASHDATA_API_KEY"), "Accept": "application/json"}
url = "https://api.flashdata.dev/v2/jobs/" + job_id
def get(url, name, headers):
response = requests.get(url, headers=headers, timeout=(10, 120), allow_redirects=False)
(directory / name).write_bytes(response.content)
(directory / (name + ".headers")).write_text(
"\n".join(f"{k}: {v}" for k, v in response.headers.items()), encoding="utf-8")
(directory / (name + ".http")).write_text(str(response.status_code))
return response
status = get(url, "job-status.json", headers)
if status.status_code != 200:
raise ValueError(f"HTTP {status.status_code}. Inspect job-status.json and its headers.")
job = status.json()["job"]
if job["id"] != job_id or job["source"] != "youtube_download":
raise ValueError("The job does not match the saved receipt.")
if job["status"] in ("pending", "processing"):
print(job["status"] + ". Wait 10 seconds, then run this command again.")
return
if job["status"] != "completed":
raise ValueError("Job " + job["status"] + ". Inspect job-status.json; do not resubmit.")
result = get(url + "/results", "delivery.pending.json", headers)
if result.status_code == 302:
location = result.headers.get("Location", "")
parsed = urlsplit(location)
if parsed.scheme != "https" or not parsed.hostname or parsed.username or parsed.password:
raise ValueError("Expected an HTTPS result URL.")
# This URL retrieves the JSON delivery record, not the video.
# Never forward the FlashData key to the signed storage URL.
result = get(location, "delivery.pending.json", {})
if result.status_code != 200:
raise ValueError(f"HTTP {result.status_code}. Run collect.py again after resolving it.")
delivery = result.json()
if (delivery.get("source") != "youtube_download" or delivery.get("status") != "done"
or not isinstance(delivery.get("results"), list) or len(delivery["results"]) != 1):
raise ValueError("Inspect delivery.pending.json before using the result.")
(directory / "delivery.pending.json").replace(directory / "delivery.json")
print("Saved:", directory / "delivery.json")
print("Usage:", delivery.get("usage"))
if __name__ == "__main__":
try:
main()
except (IndexError, KeyError, OSError, ValueError, requests.RequestException) as error:
print("Network request failed. Keep this attempt and inspect its saved files."
if isinstance(error, requests.RequestException) else str(error), file=sys.stderr)
sys.exit(1)python collect.py job-001#!/usr/bin/env bash
# Save as download-video/collect.sh.
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
folder=${1:?Usage: bash collect.sh job-001}
[[ "$folder" =~ ^[A-Za-z0-9_-]+$ ]] || exit 2
if [[ -f "$folder/delivery.json" ]]; then
echo "Already saved: $folder/delivery.json"
exit 0
fi
id=$(jq -er 'select(.source == "youtube_download") | .id
| select(test("^[A-Za-z0-9_-]+$"))' "$folder/submission.json")
if [[ -z "${FLASHDATA_API_KEY:-}" ]]; then
read -r -s -p 'FLASHDATA_API_KEY: ' FLASHDATA_API_KEY; printf '\n'
fi
: "${FLASHDATA_API_KEY:?Set your key}"
url="https://api.flashdata.dev/v2/jobs/$id"
status=$(curl --silent --show-error --connect-timeout 10 --max-time 120 \
--header "X-API-Key: $FLASHDATA_API_KEY" --header 'Accept: application/json' \
--dump-header "$folder/job-status.json.headers" --output "$folder/job-status.json" \
--write-out '%{http_code}' "$url")
printf '%s' "$status" > "$folder/job-status.json.http"
[[ "$status" == 200 ]] || { echo "HTTP $status. Inspect job-status.json and its headers." >&2; exit 1; }
state=$(jq -er --arg id "$id" 'select(.job.id == $id and .job.source == "youtube_download")
| .job.status' "$folder/job-status.json")
case "$state" in
pending|processing) echo "$state. Wait 10 seconds, then run this command again."; exit 0 ;;
completed) ;;
*) echo "Job $state. Inspect job-status.json; do not resubmit." >&2; exit 1 ;;
esac
status=$(curl --silent --show-error --connect-timeout 10 --max-time 120 \
--header "X-API-Key: $FLASHDATA_API_KEY" --header 'Accept: application/json' \
--dump-header "$folder/result-headers.txt" --output "$folder/delivery.pending.json" \
--write-out '%{http_code}' "$url/results")
printf '%s' "$status" > "$folder/result-http.txt"
if [[ "$status" == 302 ]]; then
location=$(awk 'tolower($1)=="location:" {sub(/\r$/, "", $2); print $2}' "$folder/result-headers.txt")
[[ "$location" =~ ^https://[^/@[:space:]]+(/|$) ]] || { echo 'Expected an HTTPS result URL.' >&2; exit 1; }
# Retrieve the JSON delivery record without the FlashData key. Do not add -L.
status=$(curl --silent --show-error --compressed --proto '=https' \
--connect-timeout 10 --max-time 120 --dump-header "$folder/storage-result-headers.txt" \
--output "$folder/delivery.pending.json" --write-out '%{http_code}' "$location")
printf '%s' "$status" > "$folder/storage-result-http.txt"
fi
[[ "$status" == 200 ]] || { echo "HTTP $status. Resolve the error, then run collect.sh again." >&2; exit 1; }
jq -e '.source == "youtube_download" and .status == "done"
and (.results | type == "array" and length == 1)' "$folder/delivery.pending.json" >/dev/null
mv "$folder/delivery.pending.json" "$folder/delivery.json"
echo "Saved: $folder/delivery.json"
jq '{usage}' "$folder/delivery.json"bash collect.sh job-001delivery.json 包含什么
结果包含 file_url、以字节为单位的 file_size 和 md5。usage 报告已完成任务的费用。保留完整记录。文件 URL 标识你的 S3 对象,不会让私有存储桶变为公开可读。
{
"source": "youtube_download",
"status": "done",
"results": [
{
"file_url": "https://your-media-bucket.s3.us-east-1.amazonaws.com/recipes/clip-001/jNQXAC9IVRw.mp4",
"file_size": 1048576,
"md5": "0123456789abcdef0123456789abcdef"
}
]
}4. 读取私有对象并验证
将验证脚本保存到其他脚本旁。使用已配置且具有该对象读取权限的本地 AWS 身份。脚本先核对交付是否对应计划中的 S3 存储桶、前缀和视频文件名,再将文件下载为 media.mp4.part。
只有字节数及 MD5 一致时,才生成 media.mp4 和 verified.json。再次验证时在本地检查已有 media.mp4。此步骤读取你的 S3 对象,不会再次提交 FlashData 下载。
# Save as download-video/verify.py, beside submit.py.
import hashlib
import json
import os
import re
import sys
from contextlib import closing
from datetime import datetime, timezone
from urllib.parse import urlsplit
import boto3
from submit import ROOT, read_json
def main():
os.umask(0o077)
folder = sys.argv[1]
if not re.fullmatch(r"[A-Za-z0-9_-]+", folder):
raise ValueError("Use the saved output folder name.")
directory = ROOT / folder
request = read_json(directory / "request.json")
receipt = read_json(directory / "submission.json")
job = read_json(directory / "job-status.json")["job"]
delivery = read_json(directory / "delivery.json")
if (request["source"] != "youtube_download" or receipt["source"] != "youtube_download"
or job["id"] != receipt["id"] or job["source"] != "youtube_download"
or job["status"] != "completed" or delivery.get("source") != "youtube_download"
or delivery.get("status") != "done" or len(delivery["results"]) != 1):
raise ValueError("Check the saved request, receipt, status and delivery.")
storage = request["params"]["storage"]
bucket, region, prefix = (storage[k] for k in ("storage_url", "region", "object_prefix"))
file = delivery["results"][0]
parsed = urlsplit(file["file_url"])
# This recipe uses an ASCII prefix and the MP4 filename generated for the video.
object_key = prefix + "/" + request["video_id"] + ".mp4"
if (storage["storage_type"] != "s3" or parsed.scheme != "https"
or parsed.netloc != bucket + ".s3." + region + ".amazonaws.com"
or parsed.path != "/" + object_key or parsed.query or parsed.fragment
or type(file.get("file_size")) is not int or file["file_size"] <= 0
or not re.fullmatch(r"[A-Fa-f0-9]{32}", file.get("md5", ""))):
raise ValueError("Delivery URL, byte count or MD5 does not match this recipe's request.")
local = directory / "media.mp4"
pending = directory / "media.mp4.part"
if not local.exists():
# Boto3 uses your local AWS read identity, not the upload key sent to FlashData.
with pending.open("xb") as output:
s3 = boto3.client("s3", region_name=region)
with closing(s3.get_object(Bucket=bucket, Key=object_key)["Body"]) as stream:
while chunk := stream.read(1024 * 1024):
output.write(chunk)
local = pending
size, digest = 0, hashlib.md5(usedforsecurity=False)
with local.open("rb") as stream:
while chunk := stream.read(1024 * 1024):
size += len(chunk)
digest.update(chunk)
if size != file["file_size"] or digest.hexdigest() != file["md5"].lower():
raise ValueError("Size or MD5 mismatch. Keep the local file for inspection.")
final = directory / "media.mp4"
if local == pending:
pending.rename(final)
record = {
"job_id": receipt["id"], "video_id": request["video_id"],
"bucket": bucket, "object_key": object_key, "bytes": size, "md5": digest.hexdigest(),
"verified_at": datetime.now(timezone.utc).isoformat(), "usage": delivery.get("usage"),
}
(directory / "verified.json").write_text(json.dumps(record, indent=2), encoding="utf-8")
print("Verified:", final, "—", size, "bytes; MD5 matches.")
if __name__ == "__main__":
try:
main()
except Exception as error:
print(str(error), file=sys.stderr)
sys.exit(1)python verify.py job-001#!/usr/bin/env bash
# Save as download-video/verify.sh.
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
folder=${1:?Usage: bash verify.sh job-001}
[[ "$folder" =~ ^[A-Za-z0-9_-]+$ ]] || exit 2
jq -e --slurpfile r "$folder/request.json" --slurpfile s "$folder/submission.json" \
--slurpfile j "$folder/job-status.json" '
$r[0] as $r | $s[0] as $s | $j[0].job as $j
| $r.params.storage as $store
| ($store.object_prefix + "/" + $r.video_id + ".mp4") as $key
| .results[0] as $f
| $r.source == "youtube_download" and $s.source == "youtube_download"
and $j.id == $s.id and $j.source == "youtube_download" and $j.status == "completed"
and .source == "youtube_download" and .status == "done"
and (.results | type == "array" and length == 1)
and $store.storage_type == "s3"
and $f.file_url == ("https://" + $store.storage_url + ".s3." + $store.region + ".amazonaws.com/" + $key)
and ($f.file_size | type == "number" and . > 0 and . == floor)
and ($f.md5 | test("^[A-Fa-f0-9]{32}$"))
' "$folder/delivery.json" >/dev/null
bucket=$(jq -er '.params.storage.storage_url' "$folder/request.json")
region=$(jq -er '.params.storage.region' "$folder/request.json")
key=$(jq -er '.params.storage.object_prefix + "/" + .video_id + ".mp4"' "$folder/request.json")
local_file="$folder/media.mp4"
if [[ ! -e "$local_file" ]]; then
local_file="$folder/media.mp4.part"
[[ ! -e "$local_file" ]] || { echo 'Keep the partial file; inspect it before retrying.' >&2; exit 1; }
# AWS CLI uses your configured read identity; no FlashData key is sent to S3.
aws s3 cp "s3://$bucket/$key" "$local_file" --region "$region" --only-show-errors
fi
bytes=$(wc -c < "$local_file" | tr -d '[:space:]')
md5=$(openssl dgst -md5 "$local_file" | awk '{print tolower($NF)}')
expected_bytes=$(jq -er '.results[0].file_size' "$folder/delivery.json")
expected_md5=$(jq -er '.results[0].md5 | ascii_downcase' "$folder/delivery.json")
[[ "$bytes" == "$expected_bytes" && "$md5" == "$expected_md5" ]] ||
{ echo 'Size or MD5 mismatch. Keep the local file for inspection.' >&2; exit 1; }
if [[ "$local_file" != "$folder/media.mp4" ]]; then
mv "$local_file" "$folder/media.mp4"
fi
jq --slurpfile r "$folder/request.json" --slurpfile s "$folder/submission.json" \
--arg key "$key" --arg md5 "$md5" --argjson bytes "$bytes" '
{job_id: $s[0].id, video_id: $r[0].video_id, bucket: $r[0].params.storage.storage_url,
object_key: $key, bytes: $bytes, md5: $md5, verified_at: (now | todateiso8601), usage}
' "$folder/delivery.json" > "$folder/verified.json"
echo "Verified: $folder/media.mp4 — $bytes bytes; MD5 matches."bash verify.sh job-001成功结果
上方字节数仅为示意。此时存储桶中已有 MP4,本地有经验证的副本,verified.json 记录文件对应的任务 ID、视频 ID、对象键、校验和及返回的用量。
Verified: .../job-001/media.mp4 — 1048576 bytes; MD5 matches.恢复操作,避免重复下载
| 遇到的情况 | 下一步 |
|---|---|
| 提交超时或未保存任务 ID | 保留 job-001,根据时间戳和视频检查请求历史。不要删除文件夹来强行重新提交。先恢复原任务,再判断是否需要新请求。 |
| pending / processing | 等待 10 秒,再使用 job-001 运行 collect。它只读取已有任务。 |
| 任务失败 | 检查 job-status.json 中的错误。修复视频或存储配置后,明确使用新文件夹及前缀启动新任务。 |
| 401 / 403 / 402 / 429 | 检查密钥、权限或余额。遇到 429 时读取已保存响应头中的 Retry-After 并等待。脚本不会自动重试提交。 |
| 获取结果失败或签名 URL 已过期 | 使用 job-001 重新运行 collect,再次获取已有结果。 |
| S3 AccessDenied | 修复本地 AWS 读取身份或存储桶策略。FlashData 上传权限与本地读取权限彼此独立。 |
| 部分文件、路径、字节数或 MD5 不匹配 | 保留交付记录及文件以供检查。解决原因后,如需重新读取 S3,将 media.mp4.part 移到其他位置,再运行 verify。不要重新运行 submit。 |