# Async & batch requests

Run background work for an application or agent, resume from its saved job ID and collect every batch outcome.

Canonical reference: https://flashdata.dev/docs/guides/async-batch
Reviewed: 2026-11-05 by FlashData

<a id="choose"></a>

## Continue work after your process stops

Use a background job for a queued application task or an agent workflow that can finish later. This walkthrough runs one Google search, saves its receipt and resumes from that receipt in another process.

| Keep | Purpose |
| --- | --- |
| Job receipt | submission.json — keep the accepted job ID with your application task and original input. |
| Saved response | result.json — validate its source and results before importing it. |
| Batch outcomes | batch-results/outcomes.json — one entry per input, with its job ID and result file. |

- [Realtime versus async](https://flashdata.dev/docs/reference/realtime-and-async/reference.md#choose)
- [Batch versus Bulk](https://flashdata.dev/docs/reference/batch-and-bulk/reference.md#choose)
- [Open the workflow builder](https://flashdata.dev/docs/guides/async-batch/reference.md#customize)

<a id="prepare"></a>

## Before you start

Use an API key with google_search access and available credits. Choose CLI or HTTP for submission, waiting and retrieval. The Bash examples use jq; HTTP also needs cURL 7.76+. Run in a new working folder.

### CLI

Reuse your authenticated CLI profile or FLASHDATA_API_KEY. Confirm data.accessVerified is true. Set FLASHDATA_PROFILE for a named profile so later scripts use it too. If no key is configured, use the CLI authentication guide first.

#### Free access and account checks

```bash
flashdata auth status --json
flashdata account balance --json
flashdata account limits --json
```

- [Install and authenticate the CLI](https://flashdata.dev/docs/guides/flashdata-cli/reference.md#existing-key)

### HTTP / cURL

Keep this terminal open for the remaining commands. Agents should read the key from their secret store or environment.

#### Set your API key

```bash
# Bash: paste the key into a hidden prompt, not into this command.
read -r -s -p "FlashData API key: " FLASHDATA_API_KEY
printf '\n'
export FLASHDATA_API_KEY
```

- [Create an API key](https://flashdata.dev/app/api-keys)

- [Agent trial and claiming](https://flashdata.dev/docs/guides/agent-onboarding/reference.md)
- [Check pricing](https://flashdata.dev/docs/guides/credits-and-billing/reference.md)

<a id="job-lifecycle"></a>

## Run a background search

This walkthrough submits one Google Search. Status checks and result retrieval do not submit another query. Reuse the same credential throughout.


<a id="async-batch"></a>

## 1. Submit once and keep the receipt

The submission.json file is your receipt. HTTP returns an id, source, status and self/results links; CLI saves the receipt under submission and also returns remoteId. HTTP 200 means accepted, not finished.

### CLI

#### Bash · FlashData CLI

```bash
flashdata search "battery recycling research" --num 5 \
  --mode async --run-id async-first --output submission.json --json
```


### HTTP / cURL

#### Bash · cURL + jq

```bash
(
set -euC
: "${FLASHDATA_API_KEY:?Set FLASHDATA_API_KEY first}"
test ! -e submission.json || { echo "Use the saved receipt or a new filename." >&2; exit 1; }
curl --fail-with-body --silent --show-error \
  --request POST 'https://data.flashdata.dev/v1/queries' \
  --header "X-API-Key: $FLASHDATA_API_KEY" \
  --header 'Content-Type: application/json' \
  --data '{"source":"google_search","query":"battery recycling research","params":{"num":5}}' \
  --output submission.json
jq '{id, source, status, links}' submission.json
)
```



<a id="job-status"></a>

## 2. Wait for the existing job

Continue when the job is completed. If it failed, inspect its error before submitting anything else.

### CLI

A successful wait returns data.status: completed in CLI JSON.

#### Bash · FlashData CLI

```bash
(
set -euC
JOB_ID=$(jq -er '.remoteId // .id // .submission.id' submission.json)
flashdata jobs wait "$JOB_ID" --timeout 2m --json
)
```


### HTTP / cURL

This checks once. Repeat the GET after a short delay while job.status is pending or processing. On 429, wait for Retry-After from job-headers.txt.

#### Bash · cURL + jq

```bash
(
set -euC
: "${FLASHDATA_API_KEY:?Set FLASHDATA_API_KEY first}"
JOB_ID=$(jq -er '(.remoteId // .id // .submission.id) | select(test("^[A-Za-z0-9_-]+$"))' submission.json)
STATUS_URL="https://api.flashdata.dev/v2/jobs/$JOB_ID"
curl --fail-with-body --silent --show-error \
  --header "X-API-Key: $FLASHDATA_API_KEY" \
  --max-time 30 --dump-header job-headers.txt --output job.json "$STATUS_URL"
jq -e --arg id "$JOB_ID" '.job.id == $id' job.json >/dev/null
jq '.job | {id, source, status, error}' job.json
)
```

#### Completed status · illustrative HTTP response

```json
{
  "job": {
    "id": "EXAMPLE_JOB_ID",
    "source": "google_search",
    "status": "completed",
    "error": null
  }
}
```



<a id="job-results"></a>

## 3. Retrieve and save the response

Save result.json locally. The completed job and its result body are separate records: this search's body contains source: google_search, a results array and usage. The validation below accepts the stored result's done or completed success status.

### CLI

#### Bash · FlashData CLI

```bash
(
set -euC
JOB_ID=$(jq -er '.remoteId // .id // .submission.id' submission.json)
flashdata jobs result "$JOB_ID" --output result.json --json
jq '{source, status, results, usage}' result.json
)
```


### HTTP / cURL

The default JSON result redirects to signed storage. This code decompresses the stored result and also accepts a direct JSON response. Only the FlashData endpoint receives your API key.

#### Bash · cURL + jq

```bash
(
set -euC
: "${FLASHDATA_API_KEY:?Set FLASHDATA_API_KEY first}"
JOB_ID=$(jq -er '(.remoteId // .id // .submission.id) | select(test("^[A-Za-z0-9_-]+$"))' submission.json)
SOURCE=$(jq -er '(.submission // .).source' submission.json)
jq -e --arg id "$JOB_ID" --arg source "$SOURCE" \
  '.job.id == $id and .job.source == $source and .job.status == "completed"' job.json >/dev/null
test ! -e result.json || { echo "Choose a new output filename." >&2; exit 1; }
RESULT_URL="https://api.flashdata.dev/v2/jobs/$JOB_ID/results"
STATUS=$(curl --silent --show-error \
  --header "X-API-Key: $FLASHDATA_API_KEY" \
  --max-time 120 --dump-header result-headers.txt --output result.pending \
  --write-out '%{http_code}' "$RESULT_URL")
case "$STATUS" in
  200) ;;
  302)
    LOCATION=$(awk 'tolower($1) == "location:" { sub(/\r$/, "", $2); print $2 }' result-headers.txt)
    # The signed storage URL is fetched WITHOUT the FlashData API key.
    curl --fail-with-body --silent --show-error --compressed --max-time 120 --proto '=https' \
      --output result.pending "$LOCATION" ;;
  *) echo "Result HTTP $STATUS; inspect result.pending and result-headers.txt." >&2; exit 1 ;;
esac
jq -e --arg source "$SOURCE" '.source == $source and (.status == "done" or .status == "completed")
  and (.results | type == "array")' result.pending >/dev/null
cat result.pending > result.json
jq '{id, source, status, results, usage}' result.json
)
```


- [Inspect and save search results](https://flashdata.dev/docs/guides/google-search-guide/reference.md#results)

<a id="use-result"></a>

## 4. Use the result in your application

Validate the saved response, then extract title, link and snippet for a search view or the agent's next step. A successful empty organic array means no matching items; a missing field or failed body needs investigation.

| Application task | Transition |
| --- | --- |
| Waiting | Persist the accepted job ID before returning control to the user or agent. |
| Ready | Mark ready after the result is saved and validated. Use your task ID as the import/upsert key. |
| Needs attention | Keep the job ID and error. A polling timeout or retrieval failure is not a request to submit again. |

### Read the local response · no API calls

```bash
jq -e '.source == "google_search" and (.status == "done" or .status == "completed")
  and (.results | type == "array")
  and all(.results[]; (.organic | type == "array"))' result.json > /dev/null &&
jq -c '.results[] | .organic[] | {title, link, snippet}' result.json
```

- [Archive and reuse the response](https://flashdata.dev/docs/guides/save-results/reference.md)

<a id="resume"></a>

## Resume in another process

Open a fresh terminal in the same folder, with the same credential/profile. These commands read the job in submission.json and save to a new filename. They never submit work. Reusing the same CLI run ID requires its original state directory; HTTP itself does not deduplicate POSTs using that local run ID.

### CLI + jq · recover the saved job

```bash
(
set -eu
JOB_ID=$(jq -er '.remoteId // .id // .submission.id' submission.json)
flashdata jobs get "$JOB_ID" --json
flashdata jobs wait "$JOB_ID" --timeout 2m --json
flashdata jobs result "$JOB_ID" --output recovered-result.json --json
)
```

- [Unknown submission outcome](https://flashdata.dev/docs/guides/agent-recovery/reference.md#recover)

<a id="batch"></a>

## Submit several queries

Batch submits independent jobs in one request. This two-query example needs capacity for at least two queries and two pending jobs; a temporary trial is limited to one. Save the array below as batch.json in a separate working folder.

### batch.json

```json
[
  {
    "source": "google_search",
    "query": "battery recycling research",
    "params": {
      "num": 5
    }
  },
  {
    "source": "google_search",
    "query": "lithium battery recycling methods",
    "params": {
      "num": 5
    }
  }
]
```

### CLI

#### Bash · FlashData CLI

```bash
flashdata batch submit --input batch.json \
  --run-id batch-first --output batch-submission.json --json
```


### HTTP / cURL

#### Bash · cURL + jq

```bash
(
set -euC
: "${FLASHDATA_API_KEY:?Set FLASHDATA_API_KEY first}"
test ! -e batch-submission.json || { echo "Keep the saved batch receipt." >&2; exit 1; }
jq '{queries: .}' batch.json > batch-request.json
curl --fail-with-body --silent --show-error \
  --request POST 'https://data.flashdata.dev/v1/queries/batch' \
  --header "X-API-Key: $FLASHDATA_API_KEY" \
  --header 'Content-Type: application/json' \
  --data-binary @batch-request.json --output batch-submission.json
jq '.queries[] | {id, source, status}' batch-submission.json
)
```



<a id="batch-collector"></a>

### Save collect-batch.py

Create collect-batch.py in the Batch working folder beside batch.json and batch-submission.json. Copy the complete code below into it and save it.

Keep batch.json, batch-submission.json and batch-results/ together. The script stops if the saved inputs or receipts change. It reuses completed local files and never submits replacement queries after a read error.

#### collect-batch.py

```python
# collect-batch.py — Python 3.10+ and FlashData CLI; no new submissions.
import json
import re
import subprocess
import sys
from pathlib import Path


def read(path):
    return json.loads(Path(path).read_text(encoding='utf-8-sig'))


def cli(*args):
    result = subprocess.run(['flashdata', *args, '--json', '--non-interactive', '--timeout', '2m'],
                            check=True, capture_output=True, text=True, encoding='utf-8')
    return json.loads(result.stdout)['data']


inputs = read('batch.json')
saved = read('batch-submission.json')
receipts = saved.get('submission', saved)['queries']
if not isinstance(inputs, list) or not inputs or len(inputs) != len(receipts):
    raise ValueError('Keep the original batch input and matching receipt.')
ids = [r.get('id', '') for r in receipts]
if len(set(ids)) != len(ids) or any(not re.fullmatch(r'[A-Za-z0-9_-]+', x) for x in ids):
    raise ValueError('Invalid or duplicate job IDs.')
folder = Path('batch-results')
folder.mkdir(exist_ok=True)
binding = json.dumps({'inputs': inputs, 'receipts': receipts}, sort_keys=True)
try:
    with (folder / 'input-receipts.json').open('x', encoding='utf-8') as file:
        file.write(binding)
except FileExistsError:
    if (folder / 'input-receipts.json').read_text(encoding='utf-8') != binding:
        raise ValueError('Inputs or receipts changed. Restore the original snapshot.')

outcomes = [{'index': index, 'input': request, 'job_id': receipt['id'], 'status': 'unread'}
            for index, (request, receipt) in enumerate(zip(inputs, receipts))]
for index, (request, receipt) in enumerate(zip(inputs, receipts)):
    job_id = receipt['id']
    file = folder / (job_id + '.json')
    row = {'index': index, 'input': request, 'job_id': job_id, 'status': 'unread'}
    try:
        if not file.exists():
            try:
                job = cli('jobs', 'wait', job_id)
            except subprocess.CalledProcessError:
                job = cli('jobs', 'get', job_id)
            if job['id'] != job_id or job['source'] != request['source']:
                raise ValueError('Job does not match the saved receipt/input.')
            row.update(status=job['status'], error=job.get('error'))
            if job['status'] == 'completed':
                cli('jobs', 'result', job_id, '--output', str(file))
        if file.exists():
            result = read(file)
            if (result.get('source') != request['source']
                    or result.get('status') not in ['done', 'completed']
                    or not isinstance(result.get('results'), list)):
                raise ValueError('Inspect the saved result before importing it.')
            row.update(status='completed', result_file=str(file), error=None)
    except (OSError, ValueError, KeyError, subprocess.CalledProcessError) as error:
        row.update(status='read_error', error=getattr(error, 'stderr', None) or str(error))
    outcomes[index] = row
    pending = folder / 'outcomes.next'
    pending.write_text(json.dumps(outcomes, ensure_ascii=False, indent=2), encoding='utf-8')
    pending.replace(folder / 'outcomes.json')
    print(index, job_id, row['status'])
print('Saved batch-results/outcomes.json. No queries were submitted.')
sys.exit(0 if all(row['status'] == 'completed' for row in outcomes) else 2)
```


<a id="batch-results"></a>

### Track each accepted job

After saving collect-batch.py, run it from the Batch working folder with Python 3.10+ and the authenticated CLI. It works with either submission method above. It reads every receipt, waits up to two minutes per unfinished job and saves each successful response. Run one collector at a time; rerun it to continue the same jobs.

Expect two entries in batch-results/outcomes.json for the sample. Import completed entries by input index and job ID. Failed, pending and read_error entries remain visible; exit code 2 means review or resume them. The collector sends no new queries.

#### Bash

```bash
python3 collect-batch.py
```

#### PowerShell 7

```powershell
python collect-batch.py
```

#### Read each outcome · jq

```bash
jq '.[] | {index, job_id, status, result_file, error}' batch-results/outcomes.json
```

- [Batch result model](https://flashdata.dev/docs/reference/batch-and-bulk/reference.md#batch-results)
- [Keep application record IDs](https://flashdata.dev/docs/guides/enrich-youtube-videos/reference.md)

<a id="webhooks"></a>

## Receive a completion webhook

Add callback_url beside source, query and params in a single request or each Batch entry. Replace the example URL with your receiver. Events report completion or failure; retrieve the result separately.

1. Persist the event and return 2xx promptly. Deduplicate delivery_id (also sent as X-Delivery-Id).
2. The event is unsigned. Match its id and source to your saved receipt, then authenticate a status read using the original receipt's self link. Do not trust URLs supplied by an unverified event.
3. On completed, fetch and save the result. On failed, inspect the job error. Keep polling as a fallback if no event arrives.

### Request body with a callback

```json
{
  "source": "google_search",
  "query": "battery recycling research",
  "params": {
    "num": 5
  },
  "callback_url": "https://your-app.example/webhooks/flashdata"
}
```

### Event body · illustrative

```json
{
  "delivery_id": "completion:EXAMPLE_JOB_ID",
  "id": "EXAMPLE_JOB_ID",
  "source": "google_search",
  "status": "completed",
  "links": [
    {
      "rel": "self",
      "href": "https://api.flashdata.dev/v2/jobs/EXAMPLE_JOB_ID",
      "method": "GET"
    },
    {
      "rel": "results",
      "href": "https://api.flashdata.dev/v2/jobs/EXAMPLE_JOB_ID/results",
      "method": "GET"
    }
  ]
}
```

- [Generate a receiver in the workflow builder](https://flashdata.dev/docs/guides/async-batch/reference.md#customize)

<a id="callbacks"></a>

### Callback URL requirements

Use a public HTTPS URL on port 443, at most 2,048 characters, without credentials or a fragment. Every resolved IP must be public. Omit callback_url, or use null or an empty string, to disable notifications.


<a id="limits"></a>

## Recover an existing job

| Situation | What to do |
| --- | --- |
| Submission timed out | Inspect runs show async-first and account History. runs recover can retrieve accepted job IDs; if no ID was saved, reconcile the submission before sending another POST. |
| pending / processing | Keep checking the existing job. These states are not failures. |
| failed / 422 | Read job.error, correct the cause and decide whether to submit new work. |
| 429 | Honor Retry-After. Reduce submission concurrency if pending-job capacity is full. |
| 404 / 409 | 404: check the ID and submitting account. 409: results are not ready yet. |
| 410 | Stored result JSON expires 24 hours after completion. Use your saved copy. Media already delivered to your storage follows that storage's retention policy. |

- [Rate limits and usage](https://flashdata.dev/docs/guides/usage-and-limits/reference.md)
- [CLI run recovery](https://flashdata.dev/docs/guides/cli-automation/reference.md#recover)

<a id="delivery"></a>

## Running a YouTube Download job?

Download uses async or Batch and needs an owner API key with youtube_download access plus destination storage. Its result JSON describes media already delivered to your bucket; the results endpoint does not stream video bytes. Use the Download guide to configure storage and verify the delivered file.

- [Configure storage and download](https://flashdata.dev/docs/guides/youtube-download-guide/reference.md)
- [Verify delivered media](https://flashdata.dev/docs/guides/youtube-download-guide/reference.md#verify-media)

<a id="api-hosts"></a>

## Hosts and response formats

All four use X-API-Key. Use the same key as the submission. Status and result reads are free, but rate-limited. The examples use JSON; choose Accept and X-Respond-With in the builder for other result representations. A signed storage download uses its URL, without the FlashData key.

| Operation | Endpoint |
| --- | --- |
| Submit one job | POST https://data.flashdata.dev/v1/queries |
| Submit a batch | POST https://data.flashdata.dev/v1/queries/batch |
| Read status | GET https://api.flashdata.dev/v2/jobs/{id} |
| Read results | GET https://api.flashdata.dev/v2/jobs/{id}/results |

- [Realtime and async protocol](https://flashdata.dev/docs/reference/realtime-and-async/reference.md)
- [Batch request protocol](https://flashdata.dev/docs/reference/batch-and-bulk/reference.md)
- [Response format rules](https://flashdata.dev/docs/guides/response-formats/reference.md)

<a id="supported-sources"></a>

## Supported sources and modes

Use the builder below for the source's current parameters. Google Search and YouTube Data support background jobs. YouTube Download uses background jobs and destination storage. For a large file of video IDs with shared options, use the Bulk guide.

- [Request modes](https://flashdata.dev/docs/reference/realtime-and-async/reference.md#choose)
- [Bulk file workflow](https://flashdata.dev/docs/guides/bulk/reference.md)

<a id="customize"></a>

## Build a workflow for your source

Choose a source, edit its parameters and copy the generated workflow in your language. Enable a callback to reveal the receiver example. This builder previews code and sample responses; requests run only when you execute the code or use the console.

- [Source parameters and schemas](https://flashdata.dev/docs/reference/reference.md)
