Browse documentation
Guides
Run background work for an application or agent, resume from its saved job ID and collect every batch outcome.
On this page
Continue work after your process stops
Use a background job for a queued application task or an agent workflow that can finish later. This walkthrough runs one Google search, saves its receipt and resumes from that receipt in another process.
| Keep | Purpose |
|---|---|
| Job receipt | submission.json — keep the accepted job ID with your application task and original input. |
| Saved response | result.json — validate its source and results before importing it. |
| Batch outcomes | batch-results/outcomes.json — one entry per input, with its job ID and result file. |
Before you start
Use an API key with google_search access and available credits. Choose CLI or HTTP for submission, waiting and retrieval. The Bash examples use jq; HTTP also needs cURL 7.76+. Run in a new working folder.
Reuse your authenticated CLI profile or FLASHDATA_API_KEY. Confirm data.accessVerified is true. Set FLASHDATA_PROFILE for a named profile so later scripts use it too. If no key is configured, use the CLI authentication guide first.
flashdata auth status --json
flashdata account balance --json
flashdata account limits --jsonKeep this terminal open for the remaining commands. Agents should read the key from their secret store or environment.
# Bash: paste the key into a hidden prompt, not into this command.
read -r -s -p "FlashData API key: " FLASHDATA_API_KEY
printf '\n'
export FLASHDATA_API_KEYRun a background search
This walkthrough submits one Google Search. Status checks and result retrieval do not submit another query. Reuse the same credential throughout.
1. Submit once and keep the receipt
The submission.json file is your receipt. HTTP returns an id, source, status and self/results links; CLI saves the receipt under submission and also returns remoteId. HTTP 200 means accepted, not finished.
flashdata search "battery recycling research" --num 5 \
--mode async --run-id async-first --output submission.json --json(
set -euC
: "${FLASHDATA_API_KEY:?Set FLASHDATA_API_KEY first}"
test ! -e submission.json || { echo "Use the saved receipt or a new filename." >&2; exit 1; }
curl --fail-with-body --silent --show-error \
--request POST 'https://data.flashdata.dev/v1/queries' \
--header "X-API-Key: $FLASHDATA_API_KEY" \
--header 'Content-Type: application/json' \
--data '{"source":"google_search","query":"battery recycling research","params":{"num":5}}' \
--output submission.json
jq '{id, source, status, links}' submission.json
)2. Wait for the existing job
Continue when the job is completed. If it failed, inspect its error before submitting anything else.
A successful wait returns data.status: completed in CLI JSON.
(
set -euC
JOB_ID=$(jq -er '.remoteId // .id // .submission.id' submission.json)
flashdata jobs wait "$JOB_ID" --timeout 2m --json
)This checks once. Repeat the GET after a short delay while job.status is pending or processing. On 429, wait for Retry-After from job-headers.txt.
(
set -euC
: "${FLASHDATA_API_KEY:?Set FLASHDATA_API_KEY first}"
JOB_ID=$(jq -er '(.remoteId // .id // .submission.id) | select(test("^[A-Za-z0-9_-]+$"))' submission.json)
STATUS_URL="https://api.flashdata.dev/v2/jobs/$JOB_ID"
curl --fail-with-body --silent --show-error \
--header "X-API-Key: $FLASHDATA_API_KEY" \
--max-time 30 --dump-header job-headers.txt --output job.json "$STATUS_URL"
jq -e --arg id "$JOB_ID" '.job.id == $id' job.json >/dev/null
jq '.job | {id, source, status, error}' job.json
){
"job": {
"id": "EXAMPLE_JOB_ID",
"source": "google_search",
"status": "completed",
"error": null
}
}3. Retrieve and save the response
Save result.json locally. The completed job and its result body are separate records: this search's body contains source: google_search, a results array and usage. The validation below accepts the stored result's done or completed success status.
(
set -euC
JOB_ID=$(jq -er '.remoteId // .id // .submission.id' submission.json)
flashdata jobs result "$JOB_ID" --output result.json --json
jq '{source, status, results, usage}' result.json
)The default JSON result redirects to signed storage. This code decompresses the stored result and also accepts a direct JSON response. Only the FlashData endpoint receives your API key.
(
set -euC
: "${FLASHDATA_API_KEY:?Set FLASHDATA_API_KEY first}"
JOB_ID=$(jq -er '(.remoteId // .id // .submission.id) | select(test("^[A-Za-z0-9_-]+$"))' submission.json)
SOURCE=$(jq -er '(.submission // .).source' submission.json)
jq -e --arg id "$JOB_ID" --arg source "$SOURCE" \
'.job.id == $id and .job.source == $source and .job.status == "completed"' job.json >/dev/null
test ! -e result.json || { echo "Choose a new output filename." >&2; exit 1; }
RESULT_URL="https://api.flashdata.dev/v2/jobs/$JOB_ID/results"
STATUS=$(curl --silent --show-error \
--header "X-API-Key: $FLASHDATA_API_KEY" \
--max-time 120 --dump-header result-headers.txt --output result.pending \
--write-out '%{http_code}' "$RESULT_URL")
case "$STATUS" in
200) ;;
302)
LOCATION=$(awk 'tolower($1) == "location:" { sub(/\r$/, "", $2); print $2 }' result-headers.txt)
# The signed storage URL is fetched WITHOUT the FlashData API key.
curl --fail-with-body --silent --show-error --compressed --max-time 120 --proto '=https' \
--output result.pending "$LOCATION" ;;
*) echo "Result HTTP $STATUS; inspect result.pending and result-headers.txt." >&2; exit 1 ;;
esac
jq -e --arg source "$SOURCE" '.source == $source and (.status == "done" or .status == "completed")
and (.results | type == "array")' result.pending >/dev/null
cat result.pending > result.json
jq '{id, source, status, results, usage}' result.json
)4. Use the result in your application
Validate the saved response, then extract title, link and snippet for a search view or the agent's next step. A successful empty organic array means no matching items; a missing field or failed body needs investigation.
| Application task | Transition |
|---|---|
| Waiting | Persist the accepted job ID before returning control to the user or agent. |
| Ready | Mark ready after the result is saved and validated. Use your task ID as the import/upsert key. |
| Needs attention | Keep the job ID and error. A polling timeout or retrieval failure is not a request to submit again. |
jq -e '.source == "google_search" and (.status == "done" or .status == "completed")
and (.results | type == "array")
and all(.results[]; (.organic | type == "array"))' result.json > /dev/null &&
jq -c '.results[] | .organic[] | {title, link, snippet}' result.jsonResume in another process
Open a fresh terminal in the same folder, with the same credential/profile. These commands read the job in submission.json and save to a new filename. They never submit work. Reusing the same CLI run ID requires its original state directory; HTTP itself does not deduplicate POSTs using that local run ID.
(
set -eu
JOB_ID=$(jq -er '.remoteId // .id // .submission.id' submission.json)
flashdata jobs get "$JOB_ID" --json
flashdata jobs wait "$JOB_ID" --timeout 2m --json
flashdata jobs result "$JOB_ID" --output recovered-result.json --json
)Submit several queries
Batch submits independent jobs in one request. This two-query example needs capacity for at least two queries and two pending jobs; a temporary trial is limited to one. Save the array below as batch.json in a separate working folder.
[
{
"source": "google_search",
"query": "battery recycling research",
"params": {
"num": 5
}
},
{
"source": "google_search",
"query": "lithium battery recycling methods",
"params": {
"num": 5
}
}
]flashdata batch submit --input batch.json \
--run-id batch-first --output batch-submission.json --json(
set -euC
: "${FLASHDATA_API_KEY:?Set FLASHDATA_API_KEY first}"
test ! -e batch-submission.json || { echo "Keep the saved batch receipt." >&2; exit 1; }
jq '{queries: .}' batch.json > batch-request.json
curl --fail-with-body --silent --show-error \
--request POST 'https://data.flashdata.dev/v1/queries/batch' \
--header "X-API-Key: $FLASHDATA_API_KEY" \
--header 'Content-Type: application/json' \
--data-binary @batch-request.json --output batch-submission.json
jq '.queries[] | {id, source, status}' batch-submission.json
)Save collect-batch.py
Create collect-batch.py in the Batch working folder beside batch.json and batch-submission.json. Copy the complete code below into it and save it.
Keep batch.json, batch-submission.json and batch-results/ together. The script stops if the saved inputs or receipts change. It reuses completed local files and never submits replacement queries after a read error.
# collect-batch.py — Python 3.10+ and FlashData CLI; no new submissions.
import json
import re
import subprocess
import sys
from pathlib import Path
def read(path):
return json.loads(Path(path).read_text(encoding='utf-8-sig'))
def cli(*args):
result = subprocess.run(['flashdata', *args, '--json', '--non-interactive', '--timeout', '2m'],
check=True, capture_output=True, text=True, encoding='utf-8')
return json.loads(result.stdout)['data']
inputs = read('batch.json')
saved = read('batch-submission.json')
receipts = saved.get('submission', saved)['queries']
if not isinstance(inputs, list) or not inputs or len(inputs) != len(receipts):
raise ValueError('Keep the original batch input and matching receipt.')
ids = [r.get('id', '') for r in receipts]
if len(set(ids)) != len(ids) or any(not re.fullmatch(r'[A-Za-z0-9_-]+', x) for x in ids):
raise ValueError('Invalid or duplicate job IDs.')
folder = Path('batch-results')
folder.mkdir(exist_ok=True)
binding = json.dumps({'inputs': inputs, 'receipts': receipts}, sort_keys=True)
try:
with (folder / 'input-receipts.json').open('x', encoding='utf-8') as file:
file.write(binding)
except FileExistsError:
if (folder / 'input-receipts.json').read_text(encoding='utf-8') != binding:
raise ValueError('Inputs or receipts changed. Restore the original snapshot.')
outcomes = [{'index': index, 'input': request, 'job_id': receipt['id'], 'status': 'unread'}
for index, (request, receipt) in enumerate(zip(inputs, receipts))]
for index, (request, receipt) in enumerate(zip(inputs, receipts)):
job_id = receipt['id']
file = folder / (job_id + '.json')
row = {'index': index, 'input': request, 'job_id': job_id, 'status': 'unread'}
try:
if not file.exists():
try:
job = cli('jobs', 'wait', job_id)
except subprocess.CalledProcessError:
job = cli('jobs', 'get', job_id)
if job['id'] != job_id or job['source'] != request['source']:
raise ValueError('Job does not match the saved receipt/input.')
row.update(status=job['status'], error=job.get('error'))
if job['status'] == 'completed':
cli('jobs', 'result', job_id, '--output', str(file))
if file.exists():
result = read(file)
if (result.get('source') != request['source']
or result.get('status') not in ['done', 'completed']
or not isinstance(result.get('results'), list)):
raise ValueError('Inspect the saved result before importing it.')
row.update(status='completed', result_file=str(file), error=None)
except (OSError, ValueError, KeyError, subprocess.CalledProcessError) as error:
row.update(status='read_error', error=getattr(error, 'stderr', None) or str(error))
outcomes[index] = row
pending = folder / 'outcomes.next'
pending.write_text(json.dumps(outcomes, ensure_ascii=False, indent=2), encoding='utf-8')
pending.replace(folder / 'outcomes.json')
print(index, job_id, row['status'])
print('Saved batch-results/outcomes.json. No queries were submitted.')
sys.exit(0 if all(row['status'] == 'completed' for row in outcomes) else 2)Track each accepted job
After saving collect-batch.py, run it from the Batch working folder with Python 3.10+ and the authenticated CLI. It works with either submission method above. It reads every receipt, waits up to two minutes per unfinished job and saves each successful response. Run one collector at a time; rerun it to continue the same jobs.
Expect two entries in batch-results/outcomes.json for the sample. Import completed entries by input index and job ID. Failed, pending and read_error entries remain visible; exit code 2 means review or resume them. The collector sends no new queries.
python3 collect-batch.pypython collect-batch.pyjq '.[] | {index, job_id, status, result_file, error}' batch-results/outcomes.jsonReceive a completion webhook
Add callback_url beside source, query and params in a single request or each Batch entry. Replace the example URL with your receiver. Events report completion or failure; retrieve the result separately.
- Persist the event and return 2xx promptly. Deduplicate delivery_id (also sent as X-Delivery-Id).
- The event is unsigned. Match its id and source to your saved receipt, then authenticate a status read using the original receipt's self link. Do not trust URLs supplied by an unverified event.
- On completed, fetch and save the result. On failed, inspect the job error. Keep polling as a fallback if no event arrives.
{
"source": "google_search",
"query": "battery recycling research",
"params": {
"num": 5
},
"callback_url": "https://your-app.example/webhooks/flashdata"
}{
"delivery_id": "completion:EXAMPLE_JOB_ID",
"id": "EXAMPLE_JOB_ID",
"source": "google_search",
"status": "completed",
"links": [
{
"rel": "self",
"href": "https://api.flashdata.dev/v2/jobs/EXAMPLE_JOB_ID",
"method": "GET"
},
{
"rel": "results",
"href": "https://api.flashdata.dev/v2/jobs/EXAMPLE_JOB_ID/results",
"method": "GET"
}
]
}Callback URL requirements
Use a public HTTPS URL on port 443, at most 2,048 characters, without credentials or a fragment. Every resolved IP must be public. Omit callback_url, or use null or an empty string, to disable notifications.
Recover an existing job
| Situation | What to do |
|---|---|
| Submission timed out | Inspect runs show async-first and account History. runs recover can retrieve accepted job IDs; if no ID was saved, reconcile the submission before sending another POST. |
| pending / processing | Keep checking the existing job. These states are not failures. |
| failed / 422 | Read job.error, correct the cause and decide whether to submit new work. |
| 429 | Honor Retry-After. Reduce submission concurrency if pending-job capacity is full. |
| 404 / 409 | 404: check the ID and submitting account. 409: results are not ready yet. |
| 410 | Stored result JSON expires 24 hours after completion. Use your saved copy. Media already delivered to your storage follows that storage's retention policy. |
Running a YouTube Download job?
Download uses async or Batch and needs an owner API key with youtube_download access plus destination storage. Its result JSON describes media already delivered to your bucket; the results endpoint does not stream video bytes. Use the Download guide to configure storage and verify the delivered file.
Hosts and response formats
All four use X-API-Key. Use the same key as the submission. Status and result reads are free, but rate-limited. The examples use JSON; choose Accept and X-Respond-With in the builder for other result representations. A signed storage download uses its URL, without the FlashData key.
| Operation | Endpoint |
|---|---|
| Submit one job | POST https://data.flashdata.dev/v1/queries |
| Submit a batch | POST https://data.flashdata.dev/v1/queries/batch |
| Read status | GET https://api.flashdata.dev/v2/jobs/{id} |
| Read results | GET https://api.flashdata.dev/v2/jobs/{id}/results |
Supported sources and modes
Use the builder below for the source's current parameters. Google Search and YouTube Data support background jobs. YouTube Download uses background jobs and destination storage. For a large file of video IDs with shared options, use the Bulk guide.
Build a workflow for your source
Choose a source, edit its parameters and copy the generated workflow in your language. Enable a callback to reveal the receiver example. This builder previews code and sample responses; requests run only when you execute the code or use the console.
Build a request
Advanced parameters
Starts at 1. Each page is a separate request.
Google time filter, such as qdr:d for the past day.
Completion webhook
Webhook payload, verification and retriesMarkdown applies to completed results. Receipts and job status keep their original fields.
https://data.flashdata.dev/v1/queries