Skip to content
EN
Get API key
Browse documentation

Guides

Measure API latency and download throughput

Run one measured search, summarize saved samples, or time media delivery to your storage.

On this page

Choose the measurement

MetricStart → finish
Search response timeStart the HTTP request → read the complete body. Use steps 1–3 below.
Media delivery throughputDelivered media bytes ÷ CLI start-to-result seconds. Includes access checks, submission, polling and result export.
Async API timingsRecord acceptance (request → job ID) separately from completion (request → completed status). Use the async workflow and retain its polling interval.

1. Prepare a search sample

Use Python 3.10+ and a key with google_search permission in FLASHDATA_API_KEY. No extra Python packages are needed. Use python3 instead of python if that is your interpreter name.

Create a working folder. Save all scripts below in this folder and run their commands here. Each new search sample sends one query without automatic retries; normal query pricing applies.

Terminal · create the working folder
mkdir flashdata-benchmark
cd flashdata-benchmark
python --version

Save the request

Save this as request.json inside flashdata-benchmark/.

request.json
{
  "source": "google_search",
  "query": "battery recycling research",
  "params": {
    "num": 5
  }
}

2. Time the full response

Save measure.py beside request.json. It copies the request into a new sample folder, then records the complete response time, HTTP status, request ID and Retry-After when returned. Socket operations use a 120-second timeout. An existing sample folder stops execution before sending a request.

measure.py
import hashlib
import json
import os
import platform
import sys
import time
from datetime import datetime, timezone
from http.client import HTTPException
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import HTTPRedirectHandler, Request, build_opener

class NoRedirect(HTTPRedirectHandler):
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None

if len(sys.argv) != 4:
    raise SystemExit("Usage: python measure.py request.json NEW_DIRECTORY CLIENT_REGION")
key = os.environ.get("FLASHDATA_API_KEY")
if not key:
    raise SystemExit("Set FLASHDATA_API_KEY first")
payload = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8-sig"))
if not isinstance(payload, dict) or payload.get("source") != "google_search" \
        or set(payload) - {"source", "query", "params"} \
        or not isinstance(payload.get("query"), str) or not payload["query"].strip() \
        or not isinstance(payload.get("params", {}), dict):
    raise SystemExit("This example measures one realtime google_search request")
body = json.dumps(payload, sort_keys=True, allow_nan=False).encode()
settings = json.dumps({"source": payload["source"], "params": payload.get("params", {})},
    sort_keys=True, allow_nan=False).encode()
folder = Path(sys.argv[2])
try:
    folder.mkdir(mode=0o700)
except FileExistsError:
    raise SystemExit("Sample directory already exists; no request sent. Inspect its saved files.")
(folder / "request.json").write_bytes(body)
request = Request("https://data.flashdata.dev/v1/queries/realtime", data=body,
    headers={"X-API-Key": key, "Content-Type": "application/json", "Accept": "application/json"})
report = {"started_at": datetime.now(timezone.utc).isoformat(),
    "region": sys.argv[3], "source": payload["source"], "mode": "realtime",
    "input_sha256": hashlib.sha256(body).hexdigest(),
    "settings_sha256": hashlib.sha256(settings).hexdigest(), "concurrency": 1,
    "python_version": platform.python_version(), "timeout_seconds": 120,
    "http_status": None, "request_id": None, "retry_after": None,
    "elapsed_ms": None, "outcome": "unknown"}
(folder / "timing.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
raw = b""
start = time.perf_counter()
try:
    try:
        response = build_opener(NoRedirect()).open(request, timeout=120)
    except HTTPError as error:
        response = error
    with response:
        report["http_status"] = response.status
        report["request_id"] = response.headers.get("X-Request-Id")
        report["retry_after"] = response.headers.get("Retry-After")
        raw = response.read()
except (URLError, OSError, HTTPException):
    report["error"] = "transport_error_or_timeout"
report["elapsed_ms"] = round((time.perf_counter() - start) * 1000, 3)
try:
    result = json.loads(raw)
except (ValueError, UnicodeDecodeError):
    result = {}
if report["http_status"] is not None and "error" not in report:
    complete = report["http_status"] == 200 and isinstance(result, dict) \
        and result.get("status") == "done" and isinstance(result.get("results"), list)
    report["outcome"] = "completed" if complete else "failed"
(folder / "response.json").write_bytes(raw)
(folder / "timing.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
print(json.dumps(report, indent=2))
raise SystemExit(0 if report["outcome"] == "completed" else 1)

Run and inspect

Replace client-region with your client machine's region, such as eu-west-1. Run the first command once; the second command only reads the saved timing.

One search, then a local read
python measure.py request.json sample-01 client-region
python -m json.tool sample-01/timing.json

Check the saved files

Success exits with code 0 and records http_status: 200 and outcome: completed. The response body has status: done and a results array. failed or unknown exits with code 1; inspect the files before deciding on another query.

FileWhat to check
sample-01/request.jsonThe exact input saved before submission.
sample-01/response.jsonRaw response body; an HTTP error may return non-JSON content.
sample-01/timing.jsonelapsed_ms, outcome, http_status, request_id and retry_after. A null elapsed_ms means the process did not finish recording.

3. Summarize saved samples

Save summarize.py beside measure.py. It reads local samples and rejects duplicate folders, incomplete timings and mismatched request settings, client region or Python version.

p50/p95 use nearest-rank percentiles of completed responses. Failed and unknown attempts remain in the totals. One successful sample gives the same p50 and p95; no successful samples gives null for both.

summarize.py
import json
import math
import sys
from pathlib import Path

if len(sys.argv) < 2:
    raise SystemExit("Usage: python summarize.py SAMPLE_DIRECTORY [SAMPLE_DIRECTORY ...]")
folders = [Path(value).resolve() for value in sys.argv[1:]]
if len(set(folders)) != len(folders):
    raise SystemExit("Pass each sample directory only once")
rows = [json.loads((folder / "timing.json").read_text(encoding="utf-8")) for folder in folders]
fields = ("source", "mode", "region", "concurrency", "timeout_seconds", "settings_sha256", "python_version")
for row in rows:
    if not isinstance(row, dict) or any(field not in row for field in (*fields, "input_sha256")):
        raise SystemExit("Use timing.json files written by this measure.py")
    elapsed = row.get("elapsed_ms")
    if row.get("outcome") not in ("completed", "failed", "unknown") \
            or type(elapsed) not in (int, float) or not math.isfinite(elapsed) or elapsed < 0:
        raise SystemExit("Incomplete or invalid timing record; inspect it before summarizing")
if len({json.dumps([row[field] for field in fields], sort_keys=True) for row in rows}) != 1:
    raise SystemExit("Compare runs separately when source, mode, region or settings differ")
times = sorted(row["elapsed_ms"] for row in rows if row["outcome"] == "completed")
inputs = len({row["input_sha256"] for row in rows})
def percentile(percent):
    return times[math.ceil(len(times) * percent / 100) - 1] if times else None
print(json.dumps({"context": {field: rows[0][field] for field in fields},
    "attempted": len(rows), "completed": len(times),
    "failed": sum(row["outcome"] == "failed" for row in rows),
    "unknown": sum(row["outcome"] == "unknown" for row in rows),
    "success_rate": len(times) / len(rows), "distinct_inputs": inputs,
    "input_pattern": "repeated_input" if inputs == 1 else "varied_inputs",
    "successful_latency_ms": {"p50": percentile(50), "p95": percentile(95)}}, indent=2))
Read the first sample
python summarize.py sample-01

Add samples deliberately

The first command below sends one more search. Keep the same client-region value and request settings. To measure varied inputs, change only query in request.json before running it. The report labels the input pattern and counts distinct inputs.

For an agent-run evaluation, agree on a sample count and Credit budget first. Run samples sequentially; this example records concurrency: 1. Save the summary with the sample folders.

One additional search
python measure.py request.json sample-02 client-region
Combine saved samples · no new requests
python summarize.py sample-01 sample-02 > summary.json
python -m json.tool summary.json

Measure media delivery with the CLI

This separate example delivers one MP3 to your storage. Install the CLI, use a saved owner profile with youtube_download permission, and keep its state directory available. The timer reuses that profile and ignores FLASHDATA_API_KEY. Normal download pricing applies.

In flashdata-benchmark/, save your destination as a private storage.json. This S3 example needs your bucket, region and storage credentials; use your existing private file if you already have one.

storage.json
{
  "storage_type": "s3",
  "storage_url": "YOUR_BUCKET",
  "access_key": "YOUR_ACCESS_KEY",
  "secret_key": "YOUR_SECRET_KEY",
  "region": "us-east-1"
}

Save the delivery timer

Save measure-download.py beside storage.json. It saves a new run ID before starting the CLI, waits up to 10 minutes for the command, and records elapsed time through result export. Reusing a sample folder stops before launching the CLI.

measure-download.py
import json
import os
import shutil
import subprocess
import sys
import time
import uuid
from datetime import datetime, timezone
from pathlib import Path

if len(sys.argv) not in (6, 7):
    raise SystemExit("Usage: python measure-download.py VIDEO_ID storage.json NEW_DIRECTORY PROFILE CLIENT_REGION [STATE_DIRECTORY]")
video, storage, directory, profile, region = sys.argv[1:6]
binary = shutil.which("flashdata")
if not binary or not Path(storage).is_file():
    raise SystemExit("Install flashdata on PATH and create the private storage.json first")
folder = Path(directory)
try:
    folder.mkdir(mode=0o700)
except FileExistsError:
    raise SystemExit("Sample directory already exists; inspect timing.json and recover its run_id.")
report = {"run_id": "perf-" + uuid.uuid4().hex, "profile": profile,
    "started_at": datetime.now(timezone.utc).isoformat(),
    "source": "youtube_download", "video_id": video, "output_format": "mp3",
    "region": region, "metric": "cli_start_to_result_saved",
    "timeout_seconds": 600, "elapsed_seconds": None, "outcome": "unknown"}
timing = folder / "timing.json"
timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
print("Saved run_id in " + str(timing), flush=True)
command = [binary, "youtube", "download", video, "--format", "mp3",
    "--storage-config", str(Path(storage).resolve()), "--mode", "async", "--wait",
    "--run-id", report["run_id"], "--output", str(folder / "delivery.json"),
    "--profile", profile, "--timeout", "10m", "--json", "--non-interactive"]
if len(sys.argv) == 7:
    report["state_directory"] = str(Path(sys.argv[6]).resolve())
    command.extend(["--state-dir", report["state_directory"]])
    timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
environment = dict(os.environ)
environment.pop("FLASHDATA_API_KEY", None)  # Reuse the named profile's stored key.
start = time.perf_counter()
with (folder / "cli.json").open("wb") as output, (folder / "cli-error.json").open("wb") as errors:
    try:
        completed = subprocess.run(command, env=environment, stdout=output, stderr=errors,
            timeout=620, check=False)
        report["exit_code"] = completed.returncode
        report["outcome"] = "completed" if completed.returncode == 0 else "not_completed"
    except (OSError, subprocess.TimeoutExpired):
        report["error"] = "process_error_or_timeout"
report["elapsed_seconds"] = time.perf_counter() - start
timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
print(json.dumps(report, indent=2))
raise SystemExit(0 if report["outcome"] == "completed" else 1)

Run one delivery

Replace research with your saved profile name and client-region with this machine's region. The video ID can be replaced with one you can process. A completed run saves download-01/delivery.json, timing.json and CLI output files.

The command uses the CLI's default state directory. If your profile uses a custom directory, append its path as the last argument, after client-region.

One MP3 delivery
python measure-download.py jNQXAC9IVRw storage.json download-01 research client-region
python -m json.tool download-01/timing.json

Calculate throughput from saved results

Save throughput.py in flashdata-benchmark/, beside measure-download.py. It reads the completed delivery and timer from download-01/ without sending requests. Throughput is the sum of media file_size values divided by elapsed_seconds; Mbps uses bytes × 8 ÷ seconds ÷ 1,000,000.

throughput.py
import json
import math
import sys
from pathlib import Path

if len(sys.argv) != 2:
    raise SystemExit("Usage: python throughput.py SAMPLE_DIRECTORY")
folder = Path(sys.argv[1])
timing = json.loads((folder / "timing.json").read_text(encoding="utf-8"))
if not isinstance(timing, dict) or timing.get("outcome") != "completed" \
        or timing.get("metric") != "cli_start_to_result_saved":
    raise SystemExit("Use a completed sample from measure-download.py")
elapsed = timing.get("elapsed_seconds")
if type(elapsed) not in (int, float) or not math.isfinite(elapsed) or elapsed <= 0:
    raise SystemExit("Missing or invalid elapsed_seconds")
result = json.loads((folder / "delivery.json").read_text(encoding="utf-8-sig"))
if not isinstance(result, dict) or result.get("status") not in ("done", "completed") \
        or result.get("source") != "youtube_download":
    raise SystemExit("Use the completed download result, not a submission or job-status response")
files = result.get("results")
if not isinstance(files, list) or not files or any(not isinstance(item, dict) for item in files):
    raise SystemExit("No delivered media records")
sizes = [item.get("file_size") for item in files]
if any(type(size) not in (int, float) or not math.isfinite(size) or size <= 0 for size in sizes):
    raise SystemExit("Missing or invalid media file_size")
total = sum(sizes)
print(json.dumps({"run_id": timing["run_id"], "metric": timing["metric"],
    "delivered_media_bytes": total, "elapsed_seconds": elapsed,
    "bytes_per_second": total / elapsed, "megabits_per_second": total * 8 / elapsed / 1_000_000}, indent=2))
Read the saved measurement
python throughput.py download-01

If delivery did not finish

Read download-01/cli-error.json and the run_id in timing.json. Use runs show and runs recover with that ID and the original profile. Recovery retrieves the accepted job; keep this attempt out of completed-throughput summaries instead of restarting the timer or submitting another download.

Inspect the saved attempt · local reads
python -m json.tool download-01/timing.json
python -c "from pathlib import Path; print(Path('download-01/cli-error.json').read_text(encoding='utf-8'))"

Report a Bulk collection

Sum file_size for completed media records and divide by the collection's total elapsed seconds. Also record accepted, completed and failed counts, polling interval and pauses. Keep source IDs, output options and destination region with the report. Media retrieval from your bucket is a separate measurement.