Browse documentation
Examples
Turn a YouTube ID file into metadata JSON and CSV, with one outcome per video. Use Python or Bash with the FlashData CLI.
On this page
Before you start
Use FlashData CLI v1.2.1 on PATH, plus Python 3.10+ (standard library only), or Bash with jq and OpenSSL. Both versions use the CLI for upload, job recovery and result-part retrieval. On Windows, use Git Bash for the Bash version.
You need an owner API key with youtube_metadata permission and available credits. Reuse your saved owner login, or run auth login once and enter the key at its hidden prompt. Agents can supply FLASHDATA_API_KEY from their secret store.
These examples use the default CLI profile and state directory. Keep that same account and saved state for recovery. If auth status shows temporary access, claim it before starting Bulk.
flashdata auth loginflashdata --version
flashdata auth status --json
flashdata account balance --json
flashdata account pricing --source youtube_metadata --request-count 2 --json
flashdata account limits --json1. Prepare the video IDs
Create a bulk-enrichment folder and save video-ids.txt there. Open a terminal in that folder. Use one 11-character ID per line, without a header, blank lines or full YouTube URLs.
Choose one preparation script and save it beside video-ids.txt. It rejects invalid and duplicate IDs, then saves a normalized input snapshot, metadata options and the run ID under prepared/. The input order is preserved.
jNQXAC9IVRw
dQw4w9WgXcQ
# Save as bulk-enrichment/prepare.py. Python 3.10+, standard library only.
import hashlib
import json
import os
import re
import sys
from pathlib import Path
from subprocess import run, CalledProcessError, DEVNULL
ROOT = Path(__file__).resolve().parent
try:
os.umask(0o077)
run_id = sys.argv[1]
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.:@+-]{0,159}", run_id):
raise ValueError("Choose a valid, stable run ID.")
ids = (ROOT / "video-ids.txt").read_text(encoding="utf-8-sig").splitlines()
ids = [line.strip() for line in ids]
if not ids or any(not re.fullmatch(r"[A-Za-z0-9_-]{11}", value) for value in ids):
raise ValueError("Use one 11-character video ID per line, with no blank rows or URLs.")
if len(set(ids)) != len(ids):
raise ValueError("Remove duplicate IDs before submitting; they would be processed separately.")
data = ("\n".join(ids) + "\n").encode("utf-8")
prepared = ROOT / "prepared"
prepared.mkdir(mode=0o700) # Refuse to overwrite an existing input snapshot.
(prepared / "video-ids.txt").write_bytes(data)
(prepared / "options.json").write_text(json.dumps({
"include_formats": False, "include_chapters": False,
}, indent=2), encoding="utf-8")
if os.name == "nt":
# The CLI requires a private options file, including on Windows.
command = """
$ErrorActionPreference = 'Stop'
$path = [System.IO.Path]::GetFullPath("./prepared/options.json")
$user = [System.Security.Principal.WindowsIdentity]::GetCurrent().User
$acl = [System.Security.AccessControl.FileSecurity]::new()
$acl.SetAccessRuleProtection($true, $false)
$acl.AddAccessRule([System.Security.AccessControl.FileSystemAccessRule]::new($user, 'FullControl', 'Allow'))
[System.IO.File]::SetAccessControl($path, $acl)
"""
run(["powershell.exe", "-NoProfile", "-NonInteractive", "-Command", command],
cwd=ROOT, stdin=DEVNULL, check=True, capture_output=True)
plan = {"run_id": run_id, "source": "youtube_metadata", "input_count": len(ids),
"input_sha256": hashlib.sha256(data).hexdigest()}
(prepared / "plan.json").write_text(json.dumps(plan, indent=2), encoding="utf-8")
print("Prepared", len(ids), "unique videos for", run_id)
except (IndexError, OSError, ValueError, CalledProcessError) as error:
print(str(error), file=sys.stderr)
sys.exit(1)python prepare.py videos-001jq --version
openssl version#!/usr/bin/env bash
# Save as bulk-enrichment/prepare.sh.
set -euo pipefail
trap 'exit 1' ERR
umask 077
cd -- "$(dirname -- "$0")"
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*) jq() { command jq --binary "$@"; } ;;
esac
run_id=${1:?Usage: bash prepare.sh videos-001}
[[ "$run_id" =~ ^[A-Za-z0-9][A-Za-z0-9_.:@+-]{0,159}$ ]] || exit 2
# Validate before creating the snapshot; normalize BOM, CRLF and surrounding spaces.
ids=$(jq -Rsc '
sub("^\uFEFF"; "") | split("\n")
| if .[-1] == "" then .[:-1] else . end
| map(gsub("^\\s+|\\s+$"; ""))
| if length == 0 or any(.[]; test("^[A-Za-z0-9_-]{11}$") | not)
then error("Use one 11-character ID per line, with no blank rows or URLs.")
elif length != (unique | length)
then error("Remove duplicate IDs before submitting.")
else . end
' video-ids.txt)
mkdir prepared
printf '%s' "$ids" | jq -r '.[]' > prepared/video-ids.txt
printf '%s\n' '{"include_formats":false,"include_chapters":false}' > prepared/options.json
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*)
powershell.exe -NoProfile -NonInteractive -Command '
$ErrorActionPreference = "Stop"
$path = [System.IO.Path]::GetFullPath("./prepared/options.json")
$user = [System.Security.Principal.WindowsIdentity]::GetCurrent().User
$acl = [System.Security.AccessControl.FileSecurity]::new()
$acl.SetAccessRuleProtection($true, $false)
$acl.AddAccessRule([System.Security.AccessControl.FileSystemAccessRule]::new($user, "FullControl", "Allow"))
[System.IO.File]::SetAccessControl($path, $acl)
' </dev/null ;;
esac
hash=$(openssl dgst -sha256 prepared/video-ids.txt | awk '{print $NF}')
printf '%s' "$ids" | jq --arg run "$run_id" --arg hash "$hash" '
{run_id:$run, source:"youtube_metadata", input_count:length, input_sha256:$hash}
' > prepared/plan.json
jq '"Prepared \(.input_count) unique videos for \(.run_id)"' -r prepared/plan.jsonbash prepare.sh videos-001What is prepared
Expect Prepared 2 unique videos for videos-001. Keep prepared/ unchanged. Metadata uses include_formats: false and include_chapters: false. Each accepted video is a separate billable item; duplicate lines would be processed separately, so this recipe stops before submitting them.
Use a run ID you have not used for a different task. Retain videos-001 for this task's retries and recovery. A new run ID is for an intentional new dataset or refresh.
2. Submit and collect the results
Save process.py or process.sh in bulk-enrichment before running it. The CLI creates one Bulk job, uploads the prepared IDs, starts processing and saves every result part into bulk-results.ndjson. The local wait stops after 30 minutes; accepted server work continues.
Each attempt gets its own folder under attempts/, with the CLI receipt, error and saved run. ready.txt points to the latest complete result. Exit 2 means complete results include failed videos; continue to export and inspect those rows.
# Save as bulk-enrichment/process.py. Requires the FlashData CLI on PATH.
import hashlib
import json
import os
import sys
import tempfile
from pathlib import Path
from subprocess import run, DEVNULL
ROOT = Path(__file__).resolve().parent
def read_json(path):
return json.loads(path.read_text(encoding="utf-8"))
def main(action):
os.umask(0o077)
plan = read_json(ROOT / "prepared/plan.json")
ids = ROOT / "prepared/video-ids.txt"
if hashlib.sha256(ids.read_bytes()).hexdigest() != plan["input_sha256"]:
raise ValueError("The prepared input changed. Restore it before recovery.")
# Use the same default profile and state directory as auth login/status.
flags = ["--json", "--non-interactive"]
def cli(*args):
return run(["flashdata", *args, *flags], cwd=ROOT, stdin=DEVNULL,
capture_output=True, text=True, encoding="utf-8")
if action == "status":
result = cli("runs", "show", plan["run_id"])
if result.returncode:
raise ValueError(result.stderr)
saved = json.loads(result.stdout)["data"]
print(json.dumps({k: saved.get(k) for k in
("id", "remoteId", "status", "stage", "errorCode")}, indent=2))
if saved.get("remoteId"):
result = cli("bulk", "get", saved["remoteId"])
print(result.stdout or result.stderr)
return result.returncode
return 0
if action not in ("submit", "recover"):
raise ValueError("Choose submit, status or recover.")
attempts = ROOT / "attempts"
attempts.mkdir(exist_ok=True)
directory = Path(tempfile.mkdtemp(prefix="run-", dir=attempts))
output = str(directory / "bulk-results.ndjson")
if action == "submit":
args = ["bulk", "run", "--source", plan["source"], "--input", str(ids),
"--options", str(ROOT / "prepared/options.json"), "--run-id", plan["run_id"]]
else:
args = ["runs", "recover", plan["run_id"]]
result = cli(*args, "--output", output, "--timeout", "30m")
(directory / "receipt.json").write_text(result.stdout, encoding="utf-8")
(directory / "error.json").write_text(result.stderr, encoding="utf-8")
(directory / "exit-code.txt").write_text(str(result.returncode))
print("Attempt:", directory.relative_to(ROOT))
# bulk_item_failures still includes a complete result file with failed rows.
code = json.loads(result.stderr).get("error", {}).get("code") if result.stderr.strip() else None
if result.returncode and code != "bulk_item_failures":
raise ValueError("Collection stopped. Inspect error.json; recover the same run.")
saved = cli("runs", "show", plan["run_id"])
(directory / "run.json").write_text(saved.stdout, encoding="utf-8")
if saved.returncode or not Path(output).is_file():
raise ValueError("Could not save the run record and results together.")
record = json.loads(saved.stdout)
data = record.get("data", {})
if (not record.get("ok") or data.get("status") != "succeeded"
or data.get("id") != plan["run_id"] or data.get("source") != plan["source"]
or data.get("inputFileHash") != plan["input_sha256"]):
raise ValueError("Inspect the run/input binding before exporting.")
pointer = ROOT / "ready.next"
pointer.write_text(directory.relative_to(ROOT).as_posix(), encoding="utf-8")
pointer.replace(ROOT / "ready.txt")
print("Results ready for export." if code is None else "Results ready; inspect failed video rows.")
return 2 if code == "bulk_item_failures" else 0
if __name__ == "__main__":
try:
sys.exit(main(sys.argv[1]))
except (IndexError, KeyError, OSError, ValueError) as error:
print(str(error), file=sys.stderr)
sys.exit(1)python process.py submit#!/usr/bin/env bash
# Save as bulk-enrichment/process.sh. Requires the FlashData CLI on PATH.
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*) jq() { command jq --binary "$@"; } ;;
esac
action=${1:?Usage: bash process.sh submit|status|recover}
plan=prepared/plan.json
run_id=$(jq -er '.run_id' "$plan")
hash=$(openssl dgst -sha256 prepared/video-ids.txt | awk '{print $NF}')
jq -e --arg hash "$hash" '.source == "youtube_metadata" and .input_sha256 == $hash' "$plan" >/dev/null
# Keep the same default profile and state directory as auth login/status.
cli=(flashdata --json --non-interactive)
if [[ "$action" == status ]]; then
saved=$("${cli[@]}" runs show "$run_id")
printf '%s' "$saved" | jq '.data | {id,remoteId,status,stage,errorCode}'
remote=$(printf '%s' "$saved" | jq -r '.data.remoteId // empty')
[[ -z "$remote" ]] || "${cli[@]}" bulk get "$remote"
exit 0
fi
case "$action" in
submit) args=(bulk run --source youtube_metadata --input prepared/video-ids.txt
--options prepared/options.json --run-id "$run_id") ;;
recover) args=(runs recover "$run_id") ;;
*) echo 'Choose submit, status or recover.' >&2; exit 2 ;;
esac
mkdir -p attempts
directory=$(mktemp -d attempts/run-XXXXXX)
exit_code=0
"${cli[@]}" "${args[@]}" --output "$directory/bulk-results.ndjson" --timeout 30m \
> "$directory/receipt.json" 2> "$directory/error.json" || exit_code=$?
printf '%s' "$exit_code" > "$directory/exit-code.txt"
echo "Attempt: $directory"
code=$(jq -r '.error.code // empty' "$directory/error.json")
if [[ "$exit_code" != 0 && "$code" != bulk_item_failures ]]; then
echo "Collection stopped. Inspect $directory/error.json; recover the same run." >&2
exit 1
fi
"${cli[@]}" runs show "$run_id" > "$directory/run.json"
test -f "$directory/bulk-results.ndjson"
jq -e --arg run "$run_id" --arg hash "$hash" '
.ok and .data.status == "succeeded" and .data.id == $run
and .data.source == "youtube_metadata" and .data.inputFileHash == $hash
' "$directory/run.json" >/dev/null
printf '%s\n' "$directory" > ready.next
mv ready.next ready.txt
echo 'Results ready for export.'
[[ "$code" != bulk_item_failures ]] || { echo 'Inspect failed video rows.'; exit 2; }bash process.sh submitCheck progress from another terminal
Run from bulk-enrichment with the same CLI account. The status action shows the saved run and reads its remote Bulk job when an ID is available. It does not create or resume work.
python process.py statusbash process.sh status3. Reconcile and export
Save export.py or export.sh beside the other scripts, then run it. It checks the saved run and input hash, then matches every result by zero-based index and video_id. Result parts may arrive out of order. Missing, duplicate or mismatched rows stop the export.
The exporter retains the full metadata object, error and reported usage for each row. Failed videos stay visible with metadata: null. This small example loads the results into memory; use the streaming audit in Bulk file processing for a large dataset.
# Save as bulk-enrichment/export.py. Local files only; no API calls.
import csv
import hashlib
import json
import os
import re
import sys
from pathlib import Path
ROOT = Path(__file__).resolve().parent
try:
os.umask(0o077)
directory = (ROOT / "ready.txt").read_text(encoding="utf-8").strip()
if not re.fullmatch(r"attempts/run-[A-Za-z0-9_-]+", directory):
raise ValueError("Use the ready.txt written by process.py or process.sh.")
directory = ROOT / directory
def read_json(path):
return json.loads(path.read_text(encoding="utf-8"))
plan = read_json(ROOT / "prepared/plan.json")
raw = (ROOT / "prepared/video-ids.txt").read_bytes()
ids = raw.decode("utf-8").splitlines()
saved = read_json(directory / "run.json")
run = saved["data"]
digest = hashlib.sha256(raw).hexdigest()
if (not saved.get("ok") or run.get("status") != "succeeded" or run.get("kind") != "bulk:run"
or run.get("id") != plan["run_id"] or run.get("source") != "youtube_metadata"
or run.get("inputFileHash") != digest or plan["input_sha256"] != digest
or plan["input_count"] != len(ids) or not run.get("remoteId")):
raise ValueError("The saved run does not match this input file.")
# This recipe loads a small file into memory to validate it before writing exports.
records = {}
with (directory / "bulk-results.ndjson").open(encoding="utf-8") as source:
for line in source:
row = json.loads(line)
index = row.get("index")
if (type(index) is not int or not 0 <= index < len(ids) or index in records
or row.get("video_id") != ids[index] or row.get("source") != "youtube_metadata"
or row.get("status") not in ("completed", "failed")):
raise ValueError("Unknown, duplicate or mismatched result row.")
metadata = None
if row["status"] == "completed":
results = row.get("results")
if (not isinstance(results, list) or len(results) != 1
or not isinstance(results[0], dict) or results[0].get("videoId") != ids[index]):
raise ValueError("Completed metadata does not match its video.")
metadata = results[0]
records[index] = {
"index": index, "video_id": ids[index], "status": row["status"],
"metadata": metadata, "error": row.get("error"), "usage": row.get("usage"),
}
if len(records) != len(ids):
raise ValueError("Missing result rows. Recover the same run before exporting.")
videos = [records[index] for index in range(len(ids))]
failed = [v["video_id"] for v in videos if v["status"] == "failed"]
output = ROOT / "export"
output.mkdir(mode=0o700) # Never overwrite an earlier export.
(output / "videos.json").write_text(json.dumps(videos, ensure_ascii=False, indent=2), encoding="utf-8")
def cell(value):
if value is None:
return ""
if not isinstance(value, str):
value = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
# Keep remote text from being interpreted as a spreadsheet formula.
return "'" + value if re.match(r"^[ \t\r\n]*[=+@-]|^[\t\r\n]", value) else value
with (output / "videos.csv").open("x", encoding="utf-8", newline="") as target:
writer = csv.writer(target)
writer.writerow(["index", "video_id", "status", "title", "duration", "error"])
for video in videos:
metadata = video["metadata"] or {}
writer.writerow([cell(v) for v in (video["index"], video["video_id"], video["status"],
metadata.get("title"), metadata.get("duration"), video["error"])])
(output / "failed-video-ids.txt").write_bytes("".join(v + "\n" for v in failed).encode("utf-8"))
summary = {"run_id": plan["run_id"], "bulk_id": run["remoteId"], "input_count": len(ids),
"completed": len(ids) - len(failed), "failed": len(failed), "coverage_complete": True}
(output / "summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
print(json.dumps(summary, indent=2))
print("Saved export/videos.json, videos.csv and failed-video-ids.txt")
sys.exit(2 if failed else 0)
except (OSError, ValueError, KeyError, TypeError) as error:
print(str(error), file=sys.stderr)
sys.exit(1)python export.py#!/usr/bin/env bash
# Save as bulk-enrichment/export.sh. Local files only; no API calls.
set -euo pipefail
trap 'exit 1' ERR
umask 077
cd -- "$(dirname -- "$0")"
case "$(uname -s)" in
MINGW*|MSYS*|CYGWIN*) jq() { command jq --binary "$@"; } ;;
esac
directory=$(cat ready.txt)
[[ "$directory" =~ ^attempts/run-[A-Za-z0-9_-]+$ ]] || exit 1
hash=$(openssl dgst -sha256 prepared/video-ids.txt | awk '{print $NF}')
jq -e --slurpfile plan prepared/plan.json --arg hash "$hash" '
.ok and .data.status == "succeeded" and .data.kind == "bulk:run"
and .data.id == $plan[0].run_id and .data.source == "youtube_metadata"
and .data.inputFileHash == $hash and $plan[0].input_sha256 == $hash
and (.data.remoteId | type == "string" and length > 0)
' "$directory/run.json" >/dev/null
# Validate the complete small-sample result before creating export/.
videos=$(jq -sc --rawfile input prepared/video-ids.txt --slurpfile plan prepared/plan.json '
($input | split("\n") | .[:-1]) as $ids
| if $plan[0].input_count != ($ids | length) or length != ($ids | length)
or any(.[]; (.index | type != "number" or . != floor)
or .source != "youtube_metadata" or (.status != "completed" and .status != "failed"))
then error("Missing rows or unexpected source/status/index.")
else sort_by(.index) end
| if map(.index) != [range(0; $ids | length)]
or any(.[]; .video_id != $ids[.index])
then error("Unknown, duplicate or mismatched rows.")
elif any(.[]; .status == "completed" and
((.results | type != "array" or length != 1) or .results[0].videoId != .video_id))
then error("Completed metadata does not match its video.")
else map({index, video_id, status,
metadata:(if .status == "completed" then .results[0] else null end), error, usage}) end
' "$directory/bulk-results.ndjson")
mkdir export
printf '%s\n' "$videos" | jq . > export/videos.json
jq -r '
def cell: if . == null then ""
else (if type == "string" then . else tojson end)
| if test("^[ \\t\\r\\n]*[=+@-]|^[\\t\\r\\n]") then "'"'"'" + . else . end end;
(["index","video_id","status","title","duration","error"],
(.[] | [.index,.video_id,.status,.metadata.title,.metadata.duration,.error] | map(cell)))
| @csv
' export/videos.json > export/videos.csv
jq -r '.[] | select(.status == "failed") | .video_id' export/videos.json > export/failed-video-ids.txt
jq --slurpfile plan prepared/plan.json --slurpfile run "$directory/run.json" '
{run_id:$plan[0].run_id, bulk_id:$run[0].data.remoteId, input_count:length,
completed:([.[] | select(.status == "completed")] | length),
failed:([.[] | select(.status == "failed")] | length), coverage_complete:true}
' export/videos.json > export/summary.json
cat export/summary.json
echo 'Saved export/videos.json, videos.csv and failed-video-ids.txt'
jq -e '.failed == 0' export/summary.json >/dev/null || exit 2bash export.shUse the exported files
Success prints the summary and saved filenames. Export exits 0 when every video succeeded, 2 when some failed, or 1 when validation or writing failed. An existing export/ is preserved; read it locally instead of rerunning submission.
| File under export/ | Use it for |
|---|---|
| videos.json | Import into your app or agent workflow. Rows keep input order, metadata, errors and usage. |
| videos.csv | Review titles, durations and failures in a spreadsheet. Formula-like text is prefixed with an apostrophe; JSON retains the original text. |
| summary.json | Check coverage and the completed/failed counts, linked to the run ID and Bulk ID. |
| failed-video-ids.txt | Review only confirmed failed videos before deciding on another paid request. Empty when all succeeded. |
python -m json.tool export/summary.json
python -m json.tool export/videos.jsonjq . export/summary.json
jq . export/videos.jsonExample outcome with a failed video
Illustrative videos.json excerpt. A completed Bulk job can contain failed videos. Do not treat a failed video as an empty successful result or silently drop it from your dataset.
[
{
"index": 0,
"video_id": "jNQXAC9IVRw",
"status": "completed",
"metadata": {
"videoId": "jNQXAC9IVRw",
"title": "Example video",
"duration": 19
},
"error": null,
"usage": null
},
{
"index": 1,
"video_id": "dQw4w9WgXcQ",
"status": "failed",
"metadata": null,
"error": {
"code": "example_video_unavailable"
},
"usage": null
}
]Recover the same job
If waiting stopped or a result part could not be read, run recover. It collects the existing job into a fresh attempt folder and updates ready.txt only after complete retrieval. It does not upload or submit replacement work. Then run the exporter if export/ has not yet been created.
| Saved outcome | Next action |
|---|---|
| bulk_item_failures | Export and review failed-video-ids.txt. Do not resubmit all successful videos. |
| partial_result | Preserve the partial file. Recover the same run to retrieve all parts before exporting. |
| running / finalizing | Recover later using the same run ID, profile and state directory. |
| paused | Read pauseReason in status, fix the cause, then explicitly resume that Bulk ID using the controls guide. |
| awaiting_upload / bulk_input_required | Inspect the saved draft. Upload the original prepared file and start the existing Bulk ID; recovery does not replay these actions. |
| Uncertain submission or no remote ID | Keep the run and inspect the saved error and Bulk Jobs before retrying. A new run ID can create another job. |
| Wrong source, missing rows or input hash mismatch | No export is accepted. Check the prepared snapshot, saved run and result file; do not invent missing metadata. |
python process.py status
python process.py recoverbash process.sh status
bash process.sh recoverSave this snapshot or refresh it deliberately
Save completed results within the 24-hour retention window. Your exports remain local files for reuse. To refresh metadata later, use a new working folder and run ID, and review the new request cost first.