Browse documentation
Guides
Run one measured search, summarize saved samples, or time media delivery to your storage.
On this page
Choose the measurement
| Metric | Start → finish |
|---|---|
| Search response time | Start the HTTP request → read the complete body. Use steps 1–3 below. |
| Media delivery throughput | Delivered media bytes ÷ CLI start-to-result seconds. Includes access checks, submission, polling and result export. |
| Async API timings | Record acceptance (request → job ID) separately from completion (request → completed status). Use the async workflow and retain its polling interval. |
1. Prepare a search sample
Use Python 3.10+ and a key with google_search permission in FLASHDATA_API_KEY. No extra Python packages are needed. Use python3 instead of python if that is your interpreter name.
Create a working folder. Save all scripts below in this folder and run their commands here. Each new search sample sends one query without automatic retries; normal query pricing applies.
mkdir flashdata-benchmark
cd flashdata-benchmark
python --versionSave the request
Save this as request.json inside flashdata-benchmark/.
{
"source": "google_search",
"query": "battery recycling research",
"params": {
"num": 5
}
}2. Time the full response
Save measure.py beside request.json. It copies the request into a new sample folder, then records the complete response time, HTTP status, request ID and Retry-After when returned. Socket operations use a 120-second timeout. An existing sample folder stops execution before sending a request.
import hashlib
import json
import os
import platform
import sys
import time
from datetime import datetime, timezone
from http.client import HTTPException
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import HTTPRedirectHandler, Request, build_opener
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
if len(sys.argv) != 4:
raise SystemExit("Usage: python measure.py request.json NEW_DIRECTORY CLIENT_REGION")
key = os.environ.get("FLASHDATA_API_KEY")
if not key:
raise SystemExit("Set FLASHDATA_API_KEY first")
payload = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8-sig"))
if not isinstance(payload, dict) or payload.get("source") != "google_search" \
or set(payload) - {"source", "query", "params"} \
or not isinstance(payload.get("query"), str) or not payload["query"].strip() \
or not isinstance(payload.get("params", {}), dict):
raise SystemExit("This example measures one realtime google_search request")
body = json.dumps(payload, sort_keys=True, allow_nan=False).encode()
settings = json.dumps({"source": payload["source"], "params": payload.get("params", {})},
sort_keys=True, allow_nan=False).encode()
folder = Path(sys.argv[2])
try:
folder.mkdir(mode=0o700)
except FileExistsError:
raise SystemExit("Sample directory already exists; no request sent. Inspect its saved files.")
(folder / "request.json").write_bytes(body)
request = Request("https://data.flashdata.dev/v1/queries/realtime", data=body,
headers={"X-API-Key": key, "Content-Type": "application/json", "Accept": "application/json"})
report = {"started_at": datetime.now(timezone.utc).isoformat(),
"region": sys.argv[3], "source": payload["source"], "mode": "realtime",
"input_sha256": hashlib.sha256(body).hexdigest(),
"settings_sha256": hashlib.sha256(settings).hexdigest(), "concurrency": 1,
"python_version": platform.python_version(), "timeout_seconds": 120,
"http_status": None, "request_id": None, "retry_after": None,
"elapsed_ms": None, "outcome": "unknown"}
(folder / "timing.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
raw = b""
start = time.perf_counter()
try:
try:
response = build_opener(NoRedirect()).open(request, timeout=120)
except HTTPError as error:
response = error
with response:
report["http_status"] = response.status
report["request_id"] = response.headers.get("X-Request-Id")
report["retry_after"] = response.headers.get("Retry-After")
raw = response.read()
except (URLError, OSError, HTTPException):
report["error"] = "transport_error_or_timeout"
report["elapsed_ms"] = round((time.perf_counter() - start) * 1000, 3)
try:
result = json.loads(raw)
except (ValueError, UnicodeDecodeError):
result = {}
if report["http_status"] is not None and "error" not in report:
complete = report["http_status"] == 200 and isinstance(result, dict) \
and result.get("status") == "done" and isinstance(result.get("results"), list)
report["outcome"] = "completed" if complete else "failed"
(folder / "response.json").write_bytes(raw)
(folder / "timing.json").write_text(json.dumps(report, indent=2), encoding="utf-8")
print(json.dumps(report, indent=2))
raise SystemExit(0 if report["outcome"] == "completed" else 1)Run and inspect
Replace client-region with your client machine's region, such as eu-west-1. Run the first command once; the second command only reads the saved timing.
python measure.py request.json sample-01 client-region
python -m json.tool sample-01/timing.jsonCheck the saved files
Success exits with code 0 and records http_status: 200 and outcome: completed. The response body has status: done and a results array. failed or unknown exits with code 1; inspect the files before deciding on another query.
| File | What to check |
|---|---|
| sample-01/request.json | The exact input saved before submission. |
| sample-01/response.json | Raw response body; an HTTP error may return non-JSON content. |
| sample-01/timing.json | elapsed_ms, outcome, http_status, request_id and retry_after. A null elapsed_ms means the process did not finish recording. |
3. Summarize saved samples
Save summarize.py beside measure.py. It reads local samples and rejects duplicate folders, incomplete timings and mismatched request settings, client region or Python version.
p50/p95 use nearest-rank percentiles of completed responses. Failed and unknown attempts remain in the totals. One successful sample gives the same p50 and p95; no successful samples gives null for both.
import json
import math
import sys
from pathlib import Path
if len(sys.argv) < 2:
raise SystemExit("Usage: python summarize.py SAMPLE_DIRECTORY [SAMPLE_DIRECTORY ...]")
folders = [Path(value).resolve() for value in sys.argv[1:]]
if len(set(folders)) != len(folders):
raise SystemExit("Pass each sample directory only once")
rows = [json.loads((folder / "timing.json").read_text(encoding="utf-8")) for folder in folders]
fields = ("source", "mode", "region", "concurrency", "timeout_seconds", "settings_sha256", "python_version")
for row in rows:
if not isinstance(row, dict) or any(field not in row for field in (*fields, "input_sha256")):
raise SystemExit("Use timing.json files written by this measure.py")
elapsed = row.get("elapsed_ms")
if row.get("outcome") not in ("completed", "failed", "unknown") \
or type(elapsed) not in (int, float) or not math.isfinite(elapsed) or elapsed < 0:
raise SystemExit("Incomplete or invalid timing record; inspect it before summarizing")
if len({json.dumps([row[field] for field in fields], sort_keys=True) for row in rows}) != 1:
raise SystemExit("Compare runs separately when source, mode, region or settings differ")
times = sorted(row["elapsed_ms"] for row in rows if row["outcome"] == "completed")
inputs = len({row["input_sha256"] for row in rows})
def percentile(percent):
return times[math.ceil(len(times) * percent / 100) - 1] if times else None
print(json.dumps({"context": {field: rows[0][field] for field in fields},
"attempted": len(rows), "completed": len(times),
"failed": sum(row["outcome"] == "failed" for row in rows),
"unknown": sum(row["outcome"] == "unknown" for row in rows),
"success_rate": len(times) / len(rows), "distinct_inputs": inputs,
"input_pattern": "repeated_input" if inputs == 1 else "varied_inputs",
"successful_latency_ms": {"p50": percentile(50), "p95": percentile(95)}}, indent=2))python summarize.py sample-01Add samples deliberately
The first command below sends one more search. Keep the same client-region value and request settings. To measure varied inputs, change only query in request.json before running it. The report labels the input pattern and counts distinct inputs.
For an agent-run evaluation, agree on a sample count and Credit budget first. Run samples sequentially; this example records concurrency: 1. Save the summary with the sample folders.
python measure.py request.json sample-02 client-regionpython summarize.py sample-01 sample-02 > summary.json
python -m json.tool summary.jsonMeasure media delivery with the CLI
This separate example delivers one MP3 to your storage. Install the CLI, use a saved owner profile with youtube_download permission, and keep its state directory available. The timer reuses that profile and ignores FLASHDATA_API_KEY. Normal download pricing applies.
In flashdata-benchmark/, save your destination as a private storage.json. This S3 example needs your bucket, region and storage credentials; use your existing private file if you already have one.
{
"storage_type": "s3",
"storage_url": "YOUR_BUCKET",
"access_key": "YOUR_ACCESS_KEY",
"secret_key": "YOUR_SECRET_KEY",
"region": "us-east-1"
}Save the delivery timer
Save measure-download.py beside storage.json. It saves a new run ID before starting the CLI, waits up to 10 minutes for the command, and records elapsed time through result export. Reusing a sample folder stops before launching the CLI.
import json
import os
import shutil
import subprocess
import sys
import time
import uuid
from datetime import datetime, timezone
from pathlib import Path
if len(sys.argv) not in (6, 7):
raise SystemExit("Usage: python measure-download.py VIDEO_ID storage.json NEW_DIRECTORY PROFILE CLIENT_REGION [STATE_DIRECTORY]")
video, storage, directory, profile, region = sys.argv[1:6]
binary = shutil.which("flashdata")
if not binary or not Path(storage).is_file():
raise SystemExit("Install flashdata on PATH and create the private storage.json first")
folder = Path(directory)
try:
folder.mkdir(mode=0o700)
except FileExistsError:
raise SystemExit("Sample directory already exists; inspect timing.json and recover its run_id.")
report = {"run_id": "perf-" + uuid.uuid4().hex, "profile": profile,
"started_at": datetime.now(timezone.utc).isoformat(),
"source": "youtube_download", "video_id": video, "output_format": "mp3",
"region": region, "metric": "cli_start_to_result_saved",
"timeout_seconds": 600, "elapsed_seconds": None, "outcome": "unknown"}
timing = folder / "timing.json"
timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
print("Saved run_id in " + str(timing), flush=True)
command = [binary, "youtube", "download", video, "--format", "mp3",
"--storage-config", str(Path(storage).resolve()), "--mode", "async", "--wait",
"--run-id", report["run_id"], "--output", str(folder / "delivery.json"),
"--profile", profile, "--timeout", "10m", "--json", "--non-interactive"]
if len(sys.argv) == 7:
report["state_directory"] = str(Path(sys.argv[6]).resolve())
command.extend(["--state-dir", report["state_directory"]])
timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
environment = dict(os.environ)
environment.pop("FLASHDATA_API_KEY", None) # Reuse the named profile's stored key.
start = time.perf_counter()
with (folder / "cli.json").open("wb") as output, (folder / "cli-error.json").open("wb") as errors:
try:
completed = subprocess.run(command, env=environment, stdout=output, stderr=errors,
timeout=620, check=False)
report["exit_code"] = completed.returncode
report["outcome"] = "completed" if completed.returncode == 0 else "not_completed"
except (OSError, subprocess.TimeoutExpired):
report["error"] = "process_error_or_timeout"
report["elapsed_seconds"] = time.perf_counter() - start
timing.write_text(json.dumps(report, indent=2), encoding="utf-8")
print(json.dumps(report, indent=2))
raise SystemExit(0 if report["outcome"] == "completed" else 1)Run one delivery
Replace research with your saved profile name and client-region with this machine's region. The video ID can be replaced with one you can process. A completed run saves download-01/delivery.json, timing.json and CLI output files.
The command uses the CLI's default state directory. If your profile uses a custom directory, append its path as the last argument, after client-region.
python measure-download.py jNQXAC9IVRw storage.json download-01 research client-region
python -m json.tool download-01/timing.jsonCalculate throughput from saved results
Save throughput.py in flashdata-benchmark/, beside measure-download.py. It reads the completed delivery and timer from download-01/ without sending requests. Throughput is the sum of media file_size values divided by elapsed_seconds; Mbps uses bytes × 8 ÷ seconds ÷ 1,000,000.
import json
import math
import sys
from pathlib import Path
if len(sys.argv) != 2:
raise SystemExit("Usage: python throughput.py SAMPLE_DIRECTORY")
folder = Path(sys.argv[1])
timing = json.loads((folder / "timing.json").read_text(encoding="utf-8"))
if not isinstance(timing, dict) or timing.get("outcome") != "completed" \
or timing.get("metric") != "cli_start_to_result_saved":
raise SystemExit("Use a completed sample from measure-download.py")
elapsed = timing.get("elapsed_seconds")
if type(elapsed) not in (int, float) or not math.isfinite(elapsed) or elapsed <= 0:
raise SystemExit("Missing or invalid elapsed_seconds")
result = json.loads((folder / "delivery.json").read_text(encoding="utf-8-sig"))
if not isinstance(result, dict) or result.get("status") not in ("done", "completed") \
or result.get("source") != "youtube_download":
raise SystemExit("Use the completed download result, not a submission or job-status response")
files = result.get("results")
if not isinstance(files, list) or not files or any(not isinstance(item, dict) for item in files):
raise SystemExit("No delivered media records")
sizes = [item.get("file_size") for item in files]
if any(type(size) not in (int, float) or not math.isfinite(size) or size <= 0 for size in sizes):
raise SystemExit("Missing or invalid media file_size")
total = sum(sizes)
print(json.dumps({"run_id": timing["run_id"], "metric": timing["metric"],
"delivered_media_bytes": total, "elapsed_seconds": elapsed,
"bytes_per_second": total / elapsed, "megabits_per_second": total * 8 / elapsed / 1_000_000}, indent=2))python throughput.py download-01If delivery did not finish
Read download-01/cli-error.json and the run_id in timing.json. Use runs show and runs recover with that ID and the original profile. Recovery retrieves the accepted job; keep this attempt out of completed-throughput summaries instead of restarting the timer or submitting another download.
python -m json.tool download-01/timing.json
python -c "from pathlib import Path; print(Path('download-01/cli-error.json').read_text(encoding='utf-8'))"Report a Bulk collection
Sum file_size for completed media records and divide by the collection's total elapsed seconds. Also record accepted, completed and failed counts, polling interval and pauses. Keep source IDs, output options and destination region with the report. Media retrieval from your bucket is a separate measurement.