跳转到正文
中文
获取 API 密钥
浏览文档

示例

补全视频 ID 文件

将 YouTube ID 文件转换为元数据 JSON 和 CSV,每个视频保留一项结果。使用 Python 或 Bash 配合 FlashData CLI。

本页内容

开始前准备

将 FlashData CLI v1.2.1 加入 PATH,并准备 Python 3.10+(仅使用标准库),或 Bash、jq 和 OpenSSL。两种版本均由 CLI 处理上传、任务恢复及结果分片获取。Windows 的 Bash 版本请使用 Git Bash。

需要具有 youtube_metadata 权限的所有者 API Key 和可用额度。复用已保存的所有者登录,或运行一次 auth login 并在隐藏输入提示中输入密钥。智能体可从密钥存储提供 FLASHDATA_API_KEY。

本例使用默认 CLI 配置档和状态目录。恢复时保持相同账户及已保存状态。若 auth status 显示临时访问,请先认领,再使用 Bulk。

仅在未保存所有者登录时
flashdata auth login
检查访问及本例两个视频的费用
flashdata --version
flashdata auth status --json
flashdata account balance --json
flashdata account pricing --source youtube_metadata --request-count 2 --json
flashdata account limits --json

1. 准备视频 ID

创建 bulk-enrichment 文件夹,将 video-ids.txt 保存到其中,并在该文件夹打开终端。每行使用一个 11 位 ID,不要添加表头、空行或完整 YouTube URL。

选择一种准备脚本,将其保存到 video-ids.txt 旁。它拒绝无效及重复 ID,再将规范化输入快照、元数据选项及运行 ID 保存到 prepared/ 下,并保留输入顺序。

bulk-enrichment/video-ids.txt
jNQXAC9IVRw
dQw4w9WgXcQ
bulk-enrichment/prepare.py
# Save as bulk-enrichment/prepare.py. Python 3.10+, standard library only.
import hashlib
import json
import os
import re
import sys
from pathlib import Path
from subprocess import run, CalledProcessError, DEVNULL

ROOT = Path(__file__).resolve().parent
try:
    os.umask(0o077)
    run_id = sys.argv[1]
    if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.:@+-]{0,159}", run_id):
        raise ValueError("Choose a valid, stable run ID.")
    ids = (ROOT / "video-ids.txt").read_text(encoding="utf-8-sig").splitlines()
    ids = [line.strip() for line in ids]
    if not ids or any(not re.fullmatch(r"[A-Za-z0-9_-]{11}", value) for value in ids):
        raise ValueError("Use one 11-character video ID per line, with no blank rows or URLs.")
    if len(set(ids)) != len(ids):
        raise ValueError("Remove duplicate IDs before submitting; they would be processed separately.")
    data = ("\n".join(ids) + "\n").encode("utf-8")
    prepared = ROOT / "prepared"
    prepared.mkdir(mode=0o700)  # Refuse to overwrite an existing input snapshot.
    (prepared / "video-ids.txt").write_bytes(data)
    (prepared / "options.json").write_text(json.dumps({
        "include_formats": False, "include_chapters": False,
    }, indent=2), encoding="utf-8")
    if os.name == "nt":
        # The CLI requires a private options file, including on Windows.
        command = """
$ErrorActionPreference = 'Stop'
$path = [System.IO.Path]::GetFullPath("./prepared/options.json")
$user = [System.Security.Principal.WindowsIdentity]::GetCurrent().User
$acl = [System.Security.AccessControl.FileSecurity]::new()
$acl.SetAccessRuleProtection($true, $false)
$acl.AddAccessRule([System.Security.AccessControl.FileSystemAccessRule]::new($user, 'FullControl', 'Allow'))
[System.IO.File]::SetAccessControl($path, $acl)
"""
        run(["powershell.exe", "-NoProfile", "-NonInteractive", "-Command", command],
            cwd=ROOT, stdin=DEVNULL, check=True, capture_output=True)
    plan = {"run_id": run_id, "source": "youtube_metadata", "input_count": len(ids),
            "input_sha256": hashlib.sha256(data).hexdigest()}
    (prepared / "plan.json").write_text(json.dumps(plan, indent=2), encoding="utf-8")
    print("Prepared", len(ids), "unique videos for", run_id)
except (IndexError, OSError, ValueError, CalledProcessError) as error:
    print(str(error), file=sys.stderr)
    sys.exit(1)
在 bulk-enrichment/ 中运行
python prepare.py videos-001

准备后的文件

预期输出 Prepared 2 unique videos for videos-001。保留 prepared/ 不变。元数据选项为 include_formats: false、include_chapters: false。每个被接受的视频独立计费;重复行会分别处理,因此本例在提交前阻止重复项。

使用未被其他任务占用的运行 ID。本任务的重试与恢复保留 videos-001。仅在明确创建新数据集或刷新时使用新的运行 ID。

2. 提交并收集结果

运行前,将 process.py 或 process.sh 保存到 bulk-enrichment 中。CLI 创建一个 Bulk 任务,上传已准备的 ID,启动处理,并将全部结果分片保存到 bulk-results.ndjson。30 分钟后本地等待停止,服务端已接受的工作继续执行。

每次尝试在 attempts/ 下使用独立文件夹,保存 CLI 回执、错误及运行记录。ready.txt 指向最近一次完整结果。退出码 2 表示完整结果中包含失败视频;继续导出并检查这些行。

bulk-enrichment/process.py
# Save as bulk-enrichment/process.py. Requires the FlashData CLI on PATH.
import hashlib
import json
import os
import sys
import tempfile
from pathlib import Path
from subprocess import run, DEVNULL

ROOT = Path(__file__).resolve().parent


def read_json(path):
    return json.loads(path.read_text(encoding="utf-8"))


def main(action):
    os.umask(0o077)
    plan = read_json(ROOT / "prepared/plan.json")
    ids = ROOT / "prepared/video-ids.txt"
    if hashlib.sha256(ids.read_bytes()).hexdigest() != plan["input_sha256"]:
        raise ValueError("The prepared input changed. Restore it before recovery.")
    # Use the same default profile and state directory as auth login/status.
    flags = ["--json", "--non-interactive"]
    def cli(*args):
        return run(["flashdata", *args, *flags], cwd=ROOT, stdin=DEVNULL,
                   capture_output=True, text=True, encoding="utf-8")
    if action == "status":
        result = cli("runs", "show", plan["run_id"])
        if result.returncode:
            raise ValueError(result.stderr)
        saved = json.loads(result.stdout)["data"]
        print(json.dumps({k: saved.get(k) for k in
                          ("id", "remoteId", "status", "stage", "errorCode")}, indent=2))
        if saved.get("remoteId"):
            result = cli("bulk", "get", saved["remoteId"])
            print(result.stdout or result.stderr)
            return result.returncode
        return 0
    if action not in ("submit", "recover"):
        raise ValueError("Choose submit, status or recover.")
    attempts = ROOT / "attempts"
    attempts.mkdir(exist_ok=True)
    directory = Path(tempfile.mkdtemp(prefix="run-", dir=attempts))
    output = str(directory / "bulk-results.ndjson")
    if action == "submit":
        args = ["bulk", "run", "--source", plan["source"], "--input", str(ids),
                "--options", str(ROOT / "prepared/options.json"), "--run-id", plan["run_id"]]
    else:
        args = ["runs", "recover", plan["run_id"]]
    result = cli(*args, "--output", output, "--timeout", "30m")
    (directory / "receipt.json").write_text(result.stdout, encoding="utf-8")
    (directory / "error.json").write_text(result.stderr, encoding="utf-8")
    (directory / "exit-code.txt").write_text(str(result.returncode))
    print("Attempt:", directory.relative_to(ROOT))
    # bulk_item_failures still includes a complete result file with failed rows.
    code = json.loads(result.stderr).get("error", {}).get("code") if result.stderr.strip() else None
    if result.returncode and code != "bulk_item_failures":
        raise ValueError("Collection stopped. Inspect error.json; recover the same run.")
    saved = cli("runs", "show", plan["run_id"])
    (directory / "run.json").write_text(saved.stdout, encoding="utf-8")
    if saved.returncode or not Path(output).is_file():
        raise ValueError("Could not save the run record and results together.")
    record = json.loads(saved.stdout)
    data = record.get("data", {})
    if (not record.get("ok") or data.get("status") != "succeeded"
            or data.get("id") != plan["run_id"] or data.get("source") != plan["source"]
            or data.get("inputFileHash") != plan["input_sha256"]):
        raise ValueError("Inspect the run/input binding before exporting.")
    pointer = ROOT / "ready.next"
    pointer.write_text(directory.relative_to(ROOT).as_posix(), encoding="utf-8")
    pointer.replace(ROOT / "ready.txt")
    print("Results ready for export." if code is None else "Results ready; inspect failed video rows.")
    return 2 if code == "bulk_item_failures" else 0


if __name__ == "__main__":
    try:
        sys.exit(main(sys.argv[1]))
    except (IndexError, KeyError, OSError, ValueError) as error:
        print(str(error), file=sys.stderr)
        sys.exit(1)
处理已准备文件
python process.py submit

在另一终端检查进度

从 bulk-enrichment 使用相同 CLI 账户运行。status 操作显示已保存运行,并在存在 ID 时读取远端 Bulk 任务;它不会创建或恢复处理。

读取当前进度
python process.py status

3. 核对并导出

将 export.py 或 export.sh 保存到其他脚本旁,再运行。它检查已保存运行及输入哈希,再按从零开始的 index 和 video_id 匹配每项结果。结果分片可能乱序到达;缺失、重复或不匹配的行会使导出停止。

导出脚本保留每行的完整 metadata 对象、error 及返回的 usage。失败视频以 metadata: null 保留。此小型示例将结果加载到内存;大型数据集请使用 Bulk 文件处理指南中的流式核对。

bulk-enrichment/export.py
# Save as bulk-enrichment/export.py. Local files only; no API calls.
import csv
import hashlib
import json
import os
import re
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parent
try:
    os.umask(0o077)
    directory = (ROOT / "ready.txt").read_text(encoding="utf-8").strip()
    if not re.fullmatch(r"attempts/run-[A-Za-z0-9_-]+", directory):
        raise ValueError("Use the ready.txt written by process.py or process.sh.")
    directory = ROOT / directory
    def read_json(path):
        return json.loads(path.read_text(encoding="utf-8"))
    plan = read_json(ROOT / "prepared/plan.json")
    raw = (ROOT / "prepared/video-ids.txt").read_bytes()
    ids = raw.decode("utf-8").splitlines()
    saved = read_json(directory / "run.json")
    run = saved["data"]
    digest = hashlib.sha256(raw).hexdigest()
    if (not saved.get("ok") or run.get("status") != "succeeded" or run.get("kind") != "bulk:run"
            or run.get("id") != plan["run_id"] or run.get("source") != "youtube_metadata"
            or run.get("inputFileHash") != digest or plan["input_sha256"] != digest
            or plan["input_count"] != len(ids) or not run.get("remoteId")):
        raise ValueError("The saved run does not match this input file.")
    # This recipe loads a small file into memory to validate it before writing exports.
    records = {}
    with (directory / "bulk-results.ndjson").open(encoding="utf-8") as source:
        for line in source:
            row = json.loads(line)
            index = row.get("index")
            if (type(index) is not int or not 0 <= index < len(ids) or index in records
                    or row.get("video_id") != ids[index] or row.get("source") != "youtube_metadata"
                    or row.get("status") not in ("completed", "failed")):
                raise ValueError("Unknown, duplicate or mismatched result row.")
            metadata = None
            if row["status"] == "completed":
                results = row.get("results")
                if (not isinstance(results, list) or len(results) != 1
                        or not isinstance(results[0], dict) or results[0].get("videoId") != ids[index]):
                    raise ValueError("Completed metadata does not match its video.")
                metadata = results[0]
            records[index] = {
                "index": index, "video_id": ids[index], "status": row["status"],
                "metadata": metadata, "error": row.get("error"), "usage": row.get("usage"),
            }
    if len(records) != len(ids):
        raise ValueError("Missing result rows. Recover the same run before exporting.")
    videos = [records[index] for index in range(len(ids))]
    failed = [v["video_id"] for v in videos if v["status"] == "failed"]
    output = ROOT / "export"
    output.mkdir(mode=0o700)  # Never overwrite an earlier export.
    (output / "videos.json").write_text(json.dumps(videos, ensure_ascii=False, indent=2), encoding="utf-8")
    def cell(value):
        if value is None:
            return ""
        if not isinstance(value, str):
            value = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
        # Keep remote text from being interpreted as a spreadsheet formula.
        return "'" + value if re.match(r"^[ \t\r\n]*[=+@-]|^[\t\r\n]", value) else value
    with (output / "videos.csv").open("x", encoding="utf-8", newline="") as target:
        writer = csv.writer(target)
        writer.writerow(["index", "video_id", "status", "title", "duration", "error"])
        for video in videos:
            metadata = video["metadata"] or {}
            writer.writerow([cell(v) for v in (video["index"], video["video_id"], video["status"],
                            metadata.get("title"), metadata.get("duration"), video["error"])])
    (output / "failed-video-ids.txt").write_bytes("".join(v + "\n" for v in failed).encode("utf-8"))
    summary = {"run_id": plan["run_id"], "bulk_id": run["remoteId"], "input_count": len(ids),
               "completed": len(ids) - len(failed), "failed": len(failed), "coverage_complete": True}
    (output / "summary.json").write_text(json.dumps(summary, indent=2), encoding="utf-8")
    print(json.dumps(summary, indent=2))
    print("Saved export/videos.json, videos.csv and failed-video-ids.txt")
    sys.exit(2 if failed else 0)
except (OSError, ValueError, KeyError, TypeError) as error:
    print(str(error), file=sys.stderr)
    sys.exit(1)
本地导出 · 不调用 API
python export.py

使用导出文件

成功后输出摘要及已保存文件名。全部视频成功时导出退出码为 0,部分失败为 2,验证或写入失败为 1。已有 export/ 会被保留;直接在本地读取,不要重新提交。

export/ 下的文件用途
videos.json导入应用或智能体工作流。每行保留输入顺序、元数据、错误及用量。
videos.csv在电子表格中查看标题、时长及失败项。类似公式的文本添加单引号前缀;JSON 保留原文本。
summary.json检查覆盖情况及成功/失败数量,并关联运行 ID 和 Bulk ID。
failed-video-ids.txt决定是否再次付费请求前,仅检查已确认失败的视频。全部成功时为空。
读取已保存摘要及元数据
python -m json.tool export/summary.json
python -m json.tool export/videos.json

包含失败视频的结果示例

以下为 videos.json 字段示意。已完成的 Bulk 任务可能包含失败视频。不要将失败视频当作空的成功结果,也不要从数据集中静默丢弃。

export/videos.json
[
  {
    "index": 0,
    "video_id": "jNQXAC9IVRw",
    "status": "completed",
    "metadata": {
      "videoId": "jNQXAC9IVRw",
      "title": "Example video",
      "duration": 19
    },
    "error": null,
    "usage": null
  },
  {
    "index": 1,
    "video_id": "dQw4w9WgXcQ",
    "status": "failed",
    "metadata": null,
    "error": {
      "code": "example_video_unavailable"
    },
    "usage": null
  }
]

恢复同一个任务

若等待停止或无法读取某个结果分片,运行 recover。它将已有任务收集到新的尝试文件夹,仅在完整获取后更新 ready.txt;不会上传或提交替代任务。如果尚未创建 export/,随后运行导出脚本。

已保存结果下一步
bulk_item_failures导出并检查 failed-video-ids.txt,不要重新提交全部成功视频。
partial_result保留部分结果文件。恢复同一运行,获取全部分片后再导出。
running / finalizing稍后使用相同运行 ID、配置档及状态目录恢复。
paused读取 status 中的 pauseReason,解决原因后按控制指南明确恢复该 Bulk ID。
awaiting_upload / bulk_input_required检查已保存草稿,上传原始已准备文件并启动已有 Bulk ID;recover 不会重放这些操作。
提交结果不确定或无远端 ID保留运行记录,重试前检查已保存错误及 Bulk Jobs。新的运行 ID 可能创建另一个任务。
来源错误、缺行或输入哈希不匹配不会接受该导出。检查已准备快照、保存运行及结果文件,不要编造缺失元数据。
检查并恢复已有工作
python process.py status
python process.py recover

保留快照或明确刷新

在完成后的 24 小时保留期内保存结果。导出文件可在本地复用。以后刷新元数据时,使用新的工作文件夹及运行 ID,并先查看新请求的费用。