Skip to content
EN
Get API key
Browse documentation

Examples

Search the web and save source links

Run one Google search. Save unique source links in result order, together with the query and usage.

On this page

Before you start

Use a key with google_search permission and available credits. Choose Python 3.10+ with requests, or Bash with cURL 7.76+ and jq. On Windows, use Git Bash for the Bash path.

1. Prepare the query

Create a web-search folder. Save request.json there, then open a terminal in that folder. This example requests up to five organic results; one new run submits one billable query.

web-search/request.json
{
  "source": "google_search",
  "query": "battery recycling research",
  "params": {
    "num": 5
  }
}

2. Save and run the script

Choose one version and save its script beside request.json. It reads FLASHDATA_API_KEY, or prompts for your key without displaying it. An agent should supply the environment variable from its secret store.

Run with search-001 as the output folder. The script saves the exact request before sending it. Running again with the same folder reads the saved response without calling the API.

Install in your Python environment
python -m pip install requests
web-search/search.py
import argparse
import getpass
import json
import os
from datetime import datetime, timezone
from pathlib import Path

import requests

parser = argparse.ArgumentParser()
parser.add_argument("folder", help="New output folder, or an existing saved search")
out = Path(parser.parse_args().folder)

if not out.exists():
    body = Path("request.json").read_bytes()
    request = json.loads(body)
    if not isinstance(request, dict) or request.get("source") != "google_search" or not request.get("query"):
        raise SystemExit("request.json must contain google_search and a query")
    key = os.environ.get("FLASHDATA_API_KEY") or getpass.getpass("FlashData API key: ")
    if not key.strip():
        raise SystemExit("An API key is required")
    out.mkdir()  # A concurrent run cannot submit into the same folder.
    (out / "request.json").write_bytes(body)
    (out / "requested-at.txt").write_text(
        datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"), encoding="utf-8"
    )
    try:
        response = requests.post(
            "https://data.flashdata.dev/v1/queries/realtime",
            headers={"X-API-Key": key, "Content-Type": "application/json",
                     "Accept": "application/json"},
            data=body, timeout=120, allow_redirects=False,
        )
    except requests.RequestException:
        raise SystemExit(f"Request interrupted. Keep {out} and check History before retrying.")
    (out / "result.json").write_bytes(response.content)
    (out / "response-headers.txt").write_text(
        "\n".join(f"{k}: {v}" for k, v in response.headers.items()), encoding="utf-8"
    )
    (out / "http-status.txt").write_text(str(response.status_code), encoding="utf-8")
else:
    print(f"Reading saved response from {out}; no API call.")

if not (out / "http-status.txt").exists():
    raise SystemExit("Incomplete attempt. Keep the folder and check History before retrying.")
status = (out / "http-status.txt").read_text().strip()
if status != "200":
    raise SystemExit(f"HTTP {status}: inspect {out}/result.json and response-headers.txt")

data = json.loads((out / "result.json").read_bytes())
if not isinstance(data, dict) or data.get("source") != "google_search" or data.get("status") != "done":
    raise SystemExit("Search did not finish successfully; inspect result.json")
pages = data.get("results")
if not isinstance(pages, list) or any(
    not isinstance(page, dict) or not isinstance(page.get("organic"), list) for page in pages
):
    raise SystemExit("Expected results[].organic arrays; inspect result.json")

sources, seen = [], set()
for page in pages:
    for row in page["organic"]:
        if not isinstance(row, dict):
            raise SystemExit("Expected organic result objects; inspect result.json")
        url = row.get("link")
        if isinstance(url, str) and url.strip() and url not in seen:
            seen.add(url)
            sources.append({field: row.get(field) for field in ("title", "link", "snippet")})
record = {
    "request": json.loads((out / "request.json").read_bytes()),
    "requested_at": (out / "requested-at.txt").read_text().strip(),
    "usage": data.get("usage"), "source_count": len(sources), "result_file": "result.json",
}
for filename, value in (("sources.json", sources), ("record.json", record)):
    (out / filename).write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
print(f"Saved {len(sources)} sources to {out}/sources.json")
Run from web-search/
python search.py search-001

3. Read the saved sources

Success prints Saved N sources to search-001/sources.json. The array keeps the first occurrence of each exact link in Google's returned order. Rows without a usable link are omitted; missing title or snippet values remain null.

record.json contains the submitted query, request time and returned usage. Keep result.json as the full response. Reading these files is local; an empty sources array is a valid outcome.

Read both files
python -m json.tool search-001/sources.json
python -m json.tool search-001/record.json

Example output

Illustrative sources.json; live titles, links, snippets and counts vary. Read the linked pages before using their snippets as evidence for an answer.

web-search/search-001/sources.json
[
  {
    "title": "Battery recycling research",
    "link": "https://example.org/recycling",
    "snippet": "An example search excerpt."
  }
]

Try a more focused query

To focus on a domain, change query in request.json to the example below. Choose a new output folder for the new query; search-001 always refers to its original saved request.

web-search/request.json
{
  "source": "google_search",
  "query": "site:energy.gov battery recycling research",
  "params": {
    "num": 5
  }
}
New query · another billable request
python search.py search-002

If the search needs attention

What happenedWhat to do
401 / 403 / 402Check the key, google_search permission and balance, respectively. Keep this attempt; use a new output folder only when you are ready to submit again.
429Read Retry-After in response-headers.txt and wait. The scripts do not retry automatically.
Timeout, 5xx or an incomplete folderKeep the saved files and check History before creating another attempt. Rerunning the same folder makes no new request.
Invalid JSON, failed status or unexpected fieldsInspect result.json before extracting sources. An invalid response must not be treated as an empty successful search.
No useful sourcesReview the saved query. Make it more specific or remove an overly narrow filter, then choose a new folder for another query.

Save inspect_search.py in web-search/ before running it.

web-search/inspect_search.py
from pathlib import Path

for name in ("http-status.txt", "response-headers.txt", "result.json"):
    path = Path("search-001") / name
    print(f"--- {name} ---")
    print(path.read_text(encoding="utf-8", errors="replace") if path.exists() else "Not saved")
Inspect locally
python inspect_search.py