Skip to content
EN
Get API key
Browse documentation

Guides

Bulk file processing

Process a file of video IDs, retrieve every result part and check coverage before importing the dataset into your application.

On this page

From an ID file to a checked dataset

Use this workflow for a video catalog, research collection or ingestion queue. One file uses one source and the same options for every ID. Start with the two-video metadata sample, then substitute your own file.

Files produced by this workflow
video-ids.txt → Bulk job → bulk-results.ndjson
  → audit/completed.ndjson       results for your application
  → audit/failed-video-ids.txt   confirmed failures to review
  → audit/missing.ndjson        inputs whose results are missing
  → audit/summary.json          coverage and outcome counts

Before you start

Install the CLI and use an existing profile or FLASHDATA_API_KEY with youtube_metadata permission. Confirm data.accessVerified: true and review available balance, pricing for two requests and current limits. For your own file, use its ID count when checking pricing. Each processed ID is a separate query, not one flat charge for the file.

Use Bash with jq or PowerShell 7 for steps 1–3. The optional import audit in step 4 uses Python 3.10+ and its standard library. The CLI itself requires neither Node.js nor Python.

Free access and account checks
flashdata auth status --json
flashdata account balance --json
flashdata account pricing --source youtube_metadata --request-count 2
flashdata account limits

1. Prepare the IDs

Run in a new working folder. Use UTF-8 text with one 11-character video ID per line, not a YouTube URL. Maximum file size is 2 GiB. Remove blank lines and duplicates: the CLI rejects malformed input, and repeated IDs are processed separately.

Bash
# Run in a new working directory.
(
  set -euC
  printf '%s\n' 'jNQXAC9IVRw' 'dQw4w9WgXcQ' > video-ids.txt
)
cat video-ids.txt

2. Run the file

The CLI creates a job, uploads the file, starts processing and saves every result part into one NDJSON file. Use a new output path. The local run ID bulk-first keeps the remote job receipt for recovery.

Bash
flashdata bulk run --source youtube_metadata \
  --input video-ids.txt --run-id bulk-first \
  --output bulk-results.ndjson --timeout 30m --json

3. Inspect the results

Open bulk-results.ndjson. Each line is a JSON record with index, video_id and status. Successful rows contain a results array; failed rows contain an error. Join records to the input by zero-based index, not file order. The Bash command uses jq.

One successful row · abbreviated example
{"index":0,"video_id":"jNQXAC9IVRw","source":"youtube_metadata","status":"completed","results":[{"videoId":"jNQXAC9IVRw","title":"Example video title"}]}
Bash
# Bash + jq: read each record without loading the whole file.
jq -c '{index, video_id, status, title: .results[0].title, error}' bulk-results.ndjson

4. Check coverage before importing

Save audit-bulk.py below. Run it against the unchanged input file and all retrieved result parts. It checks every index, video ID and source, rejects duplicate result rows, and separates confirmed failures from missing results. For another source, add --source SOURCE_ID.

Expect inputRows: 2, completed: 2, failed: 0, missing: 0 and allSucceeded: true for the successful sample. Only completed rows belong in the application's result import. Keep the audit summary and unsuccessful outcomes alongside them.

Save audit-bulk.py

Create audit-bulk.py beside video-ids.txt and bulk-results.ndjson in the working folder. Copy the complete code below into it and save it.

The script reads files line by line and uses a local SQLite index instead of loading the entire dataset into memory. It requires a validated input file of video IDs. Each audit uses a new output directory; partial files remain separate until parsing finishes.

audit-bulk.py
# audit-bulk.py — Python 3.10+, standard library only; no network requests.
import argparse
import json
import re
import sqlite3
import sys
from contextlib import ExitStack
from pathlib import Path

parser = argparse.ArgumentParser()
parser.add_argument('--input', required=True)
parser.add_argument('--results', required=True)
parser.add_argument('--out', required=True)
parser.add_argument('--source', default='youtube_metadata')
args = parser.parse_args()
out = Path(args.out)
try:
    out.mkdir()  # Keep any previous audit. Use a new directory for another attempt.
    with ExitStack() as stack:
        db = sqlite3.connect(out / 'index.sqlite')
        stack.callback(db.close)
        db.execute('CREATE TABLE inputs (idx INTEGER PRIMARY KEY, video_id TEXT, outcome TEXT)')
        count = 0
        with open(args.input, encoding='utf-8-sig') as inputs:
            for index, line in enumerate(inputs):
                video_id = line.strip()
                if not re.fullmatch(r'[A-Za-z0-9_-]{11}', video_id):
                    raise ValueError('Invalid ID on input line ' + str(index + 1))
                db.execute('INSERT INTO inputs VALUES (?, ?, NULL)', (index, video_id))
                count += 1
        if not count:
            raise ValueError('The input file is empty.')
        db.commit()
        files = {name: stack.enter_context(open(out / (name + '.part'), 'x', encoding='utf-8'))
                 for name in ['completed.ndjson', 'failed.ndjson', 'missing.ndjson', 'failed-video-ids.txt']}
        completed = failed = 0
        with open(args.results, encoding='utf-8-sig') as results:
            for line_number, line in enumerate(results, 1):
                if not line.strip():
                    continue
                row = json.loads(line)
                if not isinstance(row, dict) or type(row.get('index')) is not int:
                    raise ValueError('Invalid result index on line ' + str(line_number))
                expected = db.execute('SELECT video_id, outcome FROM inputs WHERE idx = ?',
                                      (row['index'],)).fetchone()
                if (not expected or row.get('video_id') != expected[0]
                        or row.get('source') != args.source or expected[1] is not None):
                    raise ValueError('Unknown, duplicate or mismatched result on line ' + str(line_number))
                status = row.get('status')
                if status not in ['completed', 'failed']:
                    raise ValueError('Unexpected item status on line ' + str(line_number))
                if status == 'completed':
                    if not isinstance(row.get('results'), list):
                        raise ValueError('Completed row is missing its results array.')
                    if args.source == 'youtube_metadata' and (len(row['results']) != 1
                            or not isinstance(row['results'][0], dict)
                            or row['results'][0].get('videoId') != expected[0]):
                        raise ValueError('Metadata does not belong to the requested video.')
                    completed += 1
                else:
                    failed += 1
                db.execute('UPDATE inputs SET outcome = ? WHERE idx = ?', (status, row['index']))
                files[status + '.ndjson'].write(json.dumps(row, ensure_ascii=False) + '\n')
        db.commit()
        missing = 0
        for index, video_id in db.execute('SELECT idx, video_id FROM inputs WHERE outcome IS NULL ORDER BY idx'):
            files['missing.ndjson'].write(json.dumps({'index': index, 'video_id': video_id,
                                                     'status': 'missing_result'}) + '\n')
            missing += 1
        for (video_id,) in db.execute("SELECT video_id FROM inputs WHERE outcome = 'failed' GROUP BY video_id ORDER BY MIN(idx)"):
            files['failed-video-ids.txt'].write(video_id + '\n')
        summary = {'source': args.source, 'inputRows': count, 'completed': completed,
                   'failed': failed, 'missing': missing, 'coverageComplete': missing == 0,
                   'allSucceeded': failed == 0 and missing == 0}
    for name in files:
        (out / (name + '.part')).rename(out / name)
    with open(out / 'summary.json', 'x', encoding='utf-8') as output:
        json.dump(summary, output, indent=2)
    print(json.dumps(summary, indent=2))
    sys.exit(0 if summary['allSucceeded'] else 2)
except Exception as error:
    print(str(error), file=sys.stderr)
    print('Audit not complete. Preserve this directory and inspect the inputs/results.', file=sys.stderr)
    sys.exit(1)

Run the import audit

Run from the folder containing the script, original input and downloaded results. The audit reads local files only.

Bash
python3 audit-bulk.py --input video-ids.txt --results bulk-results.ndjson --out audit

Import completed rows and keep the audit

File / signalUse it this way
completed.ndjsonJoin by index to the original file. Use video_id for a catalog key; retain the job/snapshot identity when storing refresh history.
failed.ndjson + failed-video-ids.txtReview errors before deliberately retrying these IDs. Successful and missing items are excluded from the failed-ID list.
missing.ndjsonRecover all result parts from the existing job, then audit the recovered file in a new directory. Missing output is not a confirmed failed query.
summary.jsoncoverageComplete checks that every input has an outcome. allSucceeded also requires no failed videos. The audit exits 2 for either failures or missing rows.
Exit 1 / no summary.jsonMalformed or mismatched data: inspect the saved files. Do not publish the partial audit as a complete dataset.

If some videos fail

A completed Bulk job can contain failed videos. The CLI saves those rows too, then exits with bulk_item_failures. Inspect their errors before deciding whether to retry individual IDs; do not resubmit the entire file automatically.

Show only unsuccessful records · jq
jq -c 'select(.status != "completed") | {index, video_id, status, error}' bulk-results.ndjson

Track progress and capacity

Read the saved job while the original command runs. The CLI wraps status under data; direct HTTP wraps it under job. Check with a delay and backoff, and honor Retry-After on 429. A local timeout or closed terminal does not cancel remote work.

FieldRead it as
input.scannedBytes / input.bytesFile scan progress, not completed queries or uploaded media.
input.completeAll input has been read; queries and result packaging may still be running.
counts.accepted / completed / failedSubmitted IDs, successful videos and failed videos. Outstanding = accepted − completed − failed.
counts.invalid / resultsInvalid input lines / records packaged so far. HTTP can accept files containing invalid lines; the CLI validates input before upload.
deliveryCustomer data delivery: uploadedParts, retryCount, storageType and customerStorage. Separate from each video's media upload.
Bash
(
set -eu
BULK_ID=$(flashdata runs show bulk-first --json | jq -er '.data.remoteId')
flashdata bulk get "$BULK_ID" --json
)

Know when to wait or act

Bulk submits gradually within your account's maxPendingJobs capacity, shared with other jobs. Insufficient credits or an inactive subscription can pause the job. Check pauseReason before resuming.

StateNext action
awaiting_uploadUpload the file, then start the existing job.
running / finalizingWait. Finalizing can include unfinished queries, packaging and storage delivery.
pausedResolve pauseReason and resume the same job.
completedDownload all result parts and inspect failed rows.
failed / cancelledInspect job.error and the saved receipt. Accepted work may still finish; no completed manifest is published.

Resume an interrupted workflow

After a local timeout or interrupted result download, recover bulk-first. This reads the existing remote job; it does not submit the dataset again. Use a new output filename. A partial_result error may leave a partial local file; keep it separate from the recovered result.

Use the same CLI profile and state directory throughout. For a named profile, add --profile NAME to every command.

SituationFix
awaiting_uploadruns recover does not replay an upload or start. Check the saved ID, upload status and original file before explicitly continuing.
pausedResolve the reported cause, then use bulk resume with the saved ID. Recovery does not resume paused work automatically.
401 / 403Restore valid access to the submitting account and source. Reuse the same job after authentication is fixed.
409 / 410409: results are not ready. 410: the stored results expired; use your saved copy.
Expired upload URLConfirm the old job never started before creating a new one. An uncertain submission is a reason to read its saved ID first.
Recover the saved run
flashdata runs show bulk-first --json
flashdata runs recover bulk-first --output recovered-results.ndjson --timeout 30m --json

Pause or resume a job

Replace BULK_ID with the ID from runs show bulk-first. Pause stops new submissions and result packaging; accepted queries continue. Resolve pauseReason before resuming. For changed options or storage credentials, update the existing job through the Bulk API.

Pause
flashdata bulk pause BULK_ID --json
Resume after fixing the cause
flashdata bulk resume BULK_ID --json

Cancel permanently

Only cancel when you intend to abandon the job. Cancellation does not undo accepted queries or refund completed work. Counts can still change, and no completed manifest is published. Pause is reversible.

Cancel the chosen job
flashdata bulk cancel BULK_ID --json

Retrieve every result part

For an existing completed job, the CLI follows its manifest and saves every result part into one NDJSON file. Use a new output path and run the import audit on the combined file. Download results describe delivered media; they are not MP4 or audio bytes.

Fetch all parts from an existing job
flashdata bulk results BULK_ID --output saved-results.ndjson --timeout 30m --json

Deliver to your storage

Save your provider credentials as storage.json in the working folder before running this command. Use the provider fields linked below; keep the file private. The CLI's --storage-config reads this JSON file and selects result_storage for data or storage for media. --options also takes a file path, not inline JSON.

Run with the saved storage configuration

Add storage to the metadata workflow
flashdata bulk run --source youtube_metadata --input video-ids.txt --storage-config storage.json --run-id bulk-with-storage --output stored-results.ndjson --timeout 30m --json