浏览文档
指南
测量一次搜索,汇总已保存样本,或记录媒体交付到你的存储所需的耗时。
本页内容
选择测量指标
| 指标 | 开始 → 结束 |
|---|---|
| 搜索响应时间 | 开始 HTTP 请求 → 读取完整响应体。使用下方第 1–3 步。 |
| 媒体交付吞吐量 | 交付媒体字节数 ÷ 从启动 CLI 到保存结果的秒数,包括访问检查、提交、轮询及结果导出。 |
| 异步 API 耗时 | 分别记录接收耗时(请求 → 任务 ID)及完成耗时(请求 → 完成状态)。使用异步流程,并保留轮询间隔。 |
1. 准备搜索样本
使用 Python 3.10+,并在 FLASHDATA_API_KEY 中设置具有 google_search 权限的密钥,无需额外 Python 包。如果解释器名为 python3,请替换下方的 python。
创建工作目录,将下方脚本全部保存在此目录并在其中执行命令。每个新搜索样本发送一次查询,不自动重试,适用正常查询定价。
mkdir flashdata-benchmark
cd flashdata-benchmark
python --version保存请求
在 flashdata-benchmark/ 中将以下内容保存为 request.json。
{
"source": "google_search",
"query": "battery recycling research",
"params": {
"num": 5
}
}2. 测量完整响应
将 measure.py 保存在 request.json 旁。脚本把请求复制到新的样本目录,再记录完整响应耗时、HTTP 状态及返回的请求 ID 和 Retry-After。套接字操作超时为 120 秒。如果样本目录已存在,会在发送请求前停止。
import hashlib
import json
import os
import platform
import sys
import time
from datetime import datetime, timezone
from http.client import HTTPException
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import HTTPRedirectHandler, Request, build_opener
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
if len(sys.argv) != 4:
raise SystemExit("Usage: python measure.py request.json NEW_DIRECTORY CLIENT_REGION")
key = os.environ.get("FLASHDATA_API_KEY")
if not key:
raise SystemExit("Set FLASHDATA_API_KEY first")
payload = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8-sig"))
if not isinstance(payload, dict) or payload.get("source") != "google_search" \
or set(payload) - {"source", "query", "params"} \
or not isinstance(payload.get("query"), str) or not payload["query"].strip() \
or not isinstance(payload.get("params", {}), dict):
raise SystemExit("This example measures one realtime google_search request")
body = json.dumps(payload, sort_keys=True, allow_nan=False).encode()
settings = json.dumps({"source": payload["source"], "params": payload.get("params", {})},
sort_keys=True, allow_nan=False).encode()
folder = Path(sys.argv[2])
try:
folder.mkdir(mode=0o700)
except FileExistsError:
raise SystemExit("Sample directory already exists; no request sent. Inspect its saved files.")
(folder / "request.json").write_bytes(body)
request = Request("https://data.flashdata.dev/v1/queries/realtime", data=body,
headers={"X-API-Key": key, "Content-Type": "application/json", "Accept": "application/json"})
report = {"started_at": datetime.now(timezone.utc).isoformat(),
"region": sys.argv[3], "source": payload["source"], "mode": "realtime",
"input_sha256": hashlib.sha256(body).hexdigest(),
"settings_sha256": hashlib.sha256(settings).hexdigest(), "concurrency": 1,
"python_version": platform.python_version(), "timeout_seconds": 120,
"http_status": None, "request_id": None, "retry_after": None,
"elapsed_ms": None, "outcome": "unknown"}
(folder / "timing.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
raw = b""
start = time.perf_counter()
try:
try:
response = build_opener(NoRedirect()).open(request, timeout=120)
except HTTPError as error:
response = error
with response:
report["http_status"] = response.status
report["request_id"] = response.headers.get("X-Request-Id")
report["retry_after"] = response.headers.get("Retry-After")
raw = response.read()
except (URLError, OSError, HTTPException):
report["error"] = "transport_error_or_timeout"
report["elapsed_ms"] = round((time.perf_counter() - start) * 1000, 3)
try:
result = json.loads(raw)
except (ValueError, UnicodeDecodeError):
result = {}
if report["http_status"] is not None and "error" not in report:
complete = report["http_status"] == 200 and isinstance(result, dict) \
and result.get("status") == "done" and isinstance(result.get("results"), list)
report["outcome"] = "completed" if complete else "failed"
(folder / "response.json").write_bytes(raw)
(folder / "timing.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
print(json.dumps(report, indent=2))
raise SystemExit(0 if report["outcome"] == "completed" else 1)执行并检查
将 client-region 替换为客户端机器的区域,例如 eu-west-1。第一条命令执行一次请求,第二条命令仅读取已保存的耗时。
python measure.py request.json sample-01 client-region
python -m json.tool sample-01/timing.json检查已保存文件
成功时退出码为 0,记录 http_status: 200 和 outcome: completed。响应体包含 status: done 和 results 数组。failed 或 unknown 的退出码为 1,请先检查文件,再决定是否进行新的查询。
| 文件 | 检查内容 |
|---|---|
| sample-01/request.json | 提交前保存的完整输入。 |
| sample-01/response.json | 原始响应体;HTTP 错误可能返回非 JSON 内容。 |
| sample-01/timing.json | elapsed_ms、outcome、http_status、request_id 及 retry_after。elapsed_ms 为 null 表示进程未完成记录。 |
3. 汇总已保存样本
将 summarize.py 保存在 measure.py 旁。脚本读取本地样本,拒绝重复目录、不完整耗时,以及请求设置、客户端区域或 Python 版本不一致的记录。
p50/p95 使用已完成响应的最近秩百分位数计算。失败及未知结果仍计入总数。只有一个成功样本时,p50 和 p95 相同;没有成功样本时,两者均为 null。
import json
import math
import sys
from pathlib import Path
if len(sys.argv) < 2:
raise SystemExit("Usage: python summarize.py SAMPLE_DIRECTORY [SAMPLE_DIRECTORY ...]")
folders = [Path(value).resolve() for value in sys.argv[1:]]
if len(set(folders)) != len(folders):
raise SystemExit("Pass each sample directory only once")
rows = [json.loads((folder / "timing.json").read_text(encoding="utf-8")) for folder in folders]
fields = ("source", "mode", "region", "concurrency", "timeout_seconds", "settings_sha256", "python_version")
for row in rows:
if not isinstance(row, dict) or any(field not in row for field in (*fields, "input_sha256")):
raise SystemExit("Use timing.json files written by this measure.py")
elapsed = row.get("elapsed_ms")
if row.get("outcome") not in ("completed", "failed", "unknown") \
or type(elapsed) not in (int, float) or not math.isfinite(elapsed) or elapsed < 0:
raise SystemExit("Incomplete or invalid timing record; inspect it before summarizing")
if len({json.dumps([row[field] for field in fields], sort_keys=True) for row in rows}) != 1:
raise SystemExit("Compare runs separately when source, mode, region or settings differ")
times = sorted(row["elapsed_ms"] for row in rows if row["outcome"] == "completed")
inputs = len({row["input_sha256"] for row in rows})
def percentile(percent):
return times[math.ceil(len(times) * percent / 100) - 1] if times else None
print(json.dumps({"context": {field: rows[0][field] for field in fields},
"attempted": len(rows), "completed": len(times),
"failed": sum(row["outcome"] == "failed" for row in rows),
"unknown": sum(row["outcome"] == "unknown" for row in rows),
"success_rate": len(times) / len(rows), "distinct_inputs": inputs,
"input_pattern": "repeated_input" if inputs == 1 else "varied_inputs",
"successful_latency_ms": {"p50": percentile(50), "p95": percentile(95)}}, indent=2))python summarize.py sample-01按需增加样本
下方第一条命令会再发送一次搜索。保持 client-region 及请求设置相同。测量不同输入时,只修改 request.json 中的 query,再运行命令。报告会标明输入模式并统计不同输入数量。
由智能体执行评估时,先确定样本数及 Credits 预算。顺序执行样本,此示例记录 concurrency: 1。将汇总报告与样本目录一起保存。
python measure.py request.json sample-02 client-regionpython summarize.py sample-01 sample-02 > summary.json
python -m json.tool summary.json通过 CLI 测量媒体交付
此独立示例将一个 MP3 交付到你的存储。安装 CLI,使用具备 youtube_download 权限的已保存所有者配置,并保留其状态目录。计时脚本复用该配置,忽略 FLASHDATA_API_KEY,适用正常下载定价。
在 flashdata-benchmark/ 中,将目标存储保存为私有 storage.json。此 S3 示例需填入你的存储桶、区域及存储凭据;如果已有私有配置文件,可直接复用。
{
"storage_type": "s3",
"storage_url": "YOUR_BUCKET",
"access_key": "YOUR_ACCESS_KEY",
"secret_key": "YOUR_SECRET_KEY",
"region": "us-east-1"
}保存交付计时脚本
将 measure-download.py 保存在 storage.json 旁。脚本在启动 CLI 前保存新的运行 ID,等待命令最多 10 分钟,并记录至结果导出结束的耗时。复用已有样本目录会在启动 CLI 前停止。
import json
import os
import shutil
import subprocess
import sys
import time
import uuid
from datetime import datetime, timezone
from pathlib import Path
if len(sys.argv) not in (6, 7):
raise SystemExit("Usage: python measure-download.py VIDEO_ID storage.json NEW_DIRECTORY PROFILE CLIENT_REGION [STATE_DIRECTORY]")
video, storage, directory, profile, region = sys.argv[1:6]
binary = shutil.which("flashdata")
if not binary or not Path(storage).is_file():
raise SystemExit("Install flashdata on PATH and create the private storage.json first")
folder = Path(directory)
try:
folder.mkdir(mode=0o700)
except FileExistsError:
raise SystemExit("Sample directory already exists; inspect timing.json and recover its run_id.")
report = {"run_id": "perf-" + uuid.uuid4().hex, "profile": profile,
"started_at": datetime.now(timezone.utc).isoformat(),
"source": "youtube_download", "video_id": video, "output_format": "mp3",
"region": region, "metric": "cli_start_to_result_saved",
"timeout_seconds": 600, "elapsed_seconds": None, "outcome": "unknown"}
timing = folder / "timing.json"
timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
print("Saved run_id in " + str(timing), flush=True)
command = [binary, "youtube", "download", video, "--format", "mp3",
"--storage-config", str(Path(storage).resolve()), "--mode", "async", "--wait",
"--run-id", report["run_id"], "--output", str(folder / "delivery.json"),
"--profile", profile, "--timeout", "10m", "--json", "--non-interactive"]
if len(sys.argv) == 7:
report["state_directory"] = str(Path(sys.argv[6]).resolve())
command.extend(["--state-dir", report["state_directory"]])
timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
environment = dict(os.environ)
environment.pop("FLASHDATA_API_KEY", None) # Reuse the named profile's stored key.
start = time.perf_counter()
with (folder / "cli.json").open("wb") as output, (folder / "cli-error.json").open("wb") as errors:
try:
completed = subprocess.run(command, env=environment, stdout=output, stderr=errors,
timeout=620, check=False)
report["exit_code"] = completed.returncode
report["outcome"] = "completed" if completed.returncode == 0 else "not_completed"
except (OSError, subprocess.TimeoutExpired):
report["error"] = "process_error_or_timeout"
report["elapsed_seconds"] = time.perf_counter() - start
timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
print(json.dumps(report, indent=2))
raise SystemExit(0 if report["outcome"] == "completed" else 1)执行一次交付
将 research 替换为你的已保存配置名,将 client-region 替换为当前机器的区域。视频 ID 可换为你有权处理的视频。完成后会在 download-01/ 中保存 delivery.json、timing.json 及 CLI 输出文件。
命令使用 CLI 默认状态目录。若配置使用自定义目录,在 client-region 后将该路径作为最后一个参数传入。
python measure-download.py jNQXAC9IVRw storage.json download-01 research client-region
python -m json.tool download-01/timing.json通过保存的结果计算吞吐量
在 flashdata-benchmark/ 中,将 throughput.py 保存在 measure-download.py 旁。脚本仅读取 download-01/ 中已完成的交付结果及计时记录,不发送请求。吞吐量为媒体 file_size 之和除以 elapsed_seconds;Mbps 按字节数 × 8 ÷ 秒数 ÷ 1,000,000 计算。
import json
import math
import sys
from pathlib import Path
if len(sys.argv) != 2:
raise SystemExit("Usage: python throughput.py SAMPLE_DIRECTORY")
folder = Path(sys.argv[1])
timing = json.loads((folder / "timing.json").read_text(encoding="utf-8"))
if not isinstance(timing, dict) or timing.get("outcome") != "completed" \
or timing.get("metric") != "cli_start_to_result_saved":
raise SystemExit("Use a completed sample from measure-download.py")
elapsed = timing.get("elapsed_seconds")
if type(elapsed) not in (int, float) or not math.isfinite(elapsed) or elapsed <= 0:
raise SystemExit("Missing or invalid elapsed_seconds")
result = json.loads((folder / "delivery.json").read_text(encoding="utf-8-sig"))
if not isinstance(result, dict) or result.get("status") not in ("done", "completed") \
or result.get("source") != "youtube_download":
raise SystemExit("Use the completed download result, not a submission or job-status response")
files = result.get("results")
if not isinstance(files, list) or not files or any(not isinstance(item, dict) for item in files):
raise SystemExit("No delivered media records")
sizes = [item.get("file_size") for item in files]
if any(type(size) not in (int, float) or not math.isfinite(size) or size <= 0 for size in sizes):
raise SystemExit("Missing or invalid media file_size")
total = sum(sizes)
print(json.dumps({"run_id": timing["run_id"], "metric": timing["metric"],
"delivered_media_bytes": total, "elapsed_seconds": elapsed,
"bytes_per_second": total / elapsed, "megabits_per_second": total * 8 / elapsed / 1_000_000}, indent=2))python throughput.py download-01交付未完成时
读取 download-01/cli-error.json 及 timing.json 中的 run_id。使用该 ID 及原配置执行 runs show 和 runs recover,恢复已接收的任务。本次尝试不计入已完成吞吐量汇总,不应重新计时或再次提交下载。
python -m json.tool download-01/timing.json
python -c "from pathlib import Path; print(Path('download-01/cli-error.json').read_text(encoding='utf-8'))"报告 Bulk 集合
汇总完成媒体记录的 file_size,并除以集合总耗时秒数。同时记录已接收、完成及失败数量、轮询间隔和暂停情况。报告中保留来源 ID、输出参数及目标区域。从存储桶读取媒体另行测量。