Browse documentation
Guides
Process a file of video IDs, retrieve every result part and check coverage before importing the dataset into your application.
On this page
From an ID file to a checked dataset
Use this workflow for a video catalog, research collection or ingestion queue. One file uses one source and the same options for every ID. Start with the two-video metadata sample, then substitute your own file.
video-ids.txt → Bulk job → bulk-results.ndjson
→ audit/completed.ndjson results for your application
→ audit/failed-video-ids.txt confirmed failures to review
→ audit/missing.ndjson inputs whose results are missing
→ audit/summary.json coverage and outcome countsBefore you start
Install the CLI and use an existing profile or FLASHDATA_API_KEY with youtube_metadata permission. Confirm data.accessVerified: true and review available balance, pricing for two requests and current limits. For your own file, use its ID count when checking pricing. Each processed ID is a separate query, not one flat charge for the file.
Use Bash with jq or PowerShell 7 for steps 1–3. The optional import audit in step 4 uses Python 3.10+ and its standard library. The CLI itself requires neither Node.js nor Python.
flashdata auth status --json
flashdata account balance --json
flashdata account pricing --source youtube_metadata --request-count 2
flashdata account limits1. Prepare the IDs
Run in a new working folder. Use UTF-8 text with one 11-character video ID per line, not a YouTube URL. Maximum file size is 2 GiB. Remove blank lines and duplicates: the CLI rejects malformed input, and repeated IDs are processed separately.
# Run in a new working directory.
(
set -euC
printf '%s\n' 'jNQXAC9IVRw' 'dQw4w9WgXcQ' > video-ids.txt
)
cat video-ids.txtif (Test-Path -LiteralPath video-ids.txt) { throw 'Keep the existing file or use a new folder.' }
$idsFile = Join-Path (Get-Location) 'video-ids.txt'
$idsText = [string]::Join([char]10, @('jNQXAC9IVRw', 'dQw4w9WgXcQ', ''))
[IO.File]::WriteAllText($idsFile, $idsText, [Text.UTF8Encoding]::new($false))
Get-Content -LiteralPath video-ids.txt2. Run the file
The CLI creates a job, uploads the file, starts processing and saves every result part into one NDJSON file. Use a new output path. The local run ID bulk-first keeps the remote job receipt for recovery.
flashdata bulk run --source youtube_metadata \
--input video-ids.txt --run-id bulk-first \
--output bulk-results.ndjson --timeout 30m --jsonflashdata bulk run --source youtube_metadata `
--input video-ids.txt --run-id bulk-first `
--output bulk-results.ndjson --timeout 30m --json3. Inspect the results
Open bulk-results.ndjson. Each line is a JSON record with index, video_id and status. Successful rows contain a results array; failed rows contain an error. Join records to the input by zero-based index, not file order. The Bash command uses jq.
{"index":0,"video_id":"jNQXAC9IVRw","source":"youtube_metadata","status":"completed","results":[{"videoId":"jNQXAC9IVRw","title":"Example video title"}]}# Bash + jq: read each record without loading the whole file.
jq -c '{index, video_id, status, title: .results[0].title, error}' bulk-results.ndjsonGet-Content -LiteralPath bulk-results.ndjson | ForEach-Object {
$row = $_ | ConvertFrom-Json
$title = if ($row.results) { $row.results[0].title } else { $null }
[pscustomobject]@{ index = $row.index; video_id = $row.video_id; status = $row.status; title = $title; error = $row.error }
}4. Check coverage before importing
Save audit-bulk.py below. Run it against the unchanged input file and all retrieved result parts. It checks every index, video ID and source, rejects duplicate result rows, and separates confirmed failures from missing results. For another source, add --source SOURCE_ID.
Expect inputRows: 2, completed: 2, failed: 0, missing: 0 and allSucceeded: true for the successful sample. Only completed rows belong in the application's result import. Keep the audit summary and unsuccessful outcomes alongside them.
Save audit-bulk.py
Create audit-bulk.py beside video-ids.txt and bulk-results.ndjson in the working folder. Copy the complete code below into it and save it.
The script reads files line by line and uses a local SQLite index instead of loading the entire dataset into memory. It requires a validated input file of video IDs. Each audit uses a new output directory; partial files remain separate until parsing finishes.
# audit-bulk.py — Python 3.10+, standard library only; no network requests.
import argparse
import json
import re
import sqlite3
import sys
from contextlib import ExitStack
from pathlib import Path
parser = argparse.ArgumentParser()
parser.add_argument('--input', required=True)
parser.add_argument('--results', required=True)
parser.add_argument('--out', required=True)
parser.add_argument('--source', default='youtube_metadata')
args = parser.parse_args()
out = Path(args.out)
try:
out.mkdir() # Keep any previous audit. Use a new directory for another attempt.
with ExitStack() as stack:
db = sqlite3.connect(out / 'index.sqlite')
stack.callback(db.close)
db.execute('CREATE TABLE inputs (idx INTEGER PRIMARY KEY, video_id TEXT, outcome TEXT)')
count = 0
with open(args.input, encoding='utf-8-sig') as inputs:
for index, line in enumerate(inputs):
video_id = line.strip()
if not re.fullmatch(r'[A-Za-z0-9_-]{11}', video_id):
raise ValueError('Invalid ID on input line ' + str(index + 1))
db.execute('INSERT INTO inputs VALUES (?, ?, NULL)', (index, video_id))
count += 1
if not count:
raise ValueError('The input file is empty.')
db.commit()
files = {name: stack.enter_context(open(out / (name + '.part'), 'x', encoding='utf-8'))
for name in ['completed.ndjson', 'failed.ndjson', 'missing.ndjson', 'failed-video-ids.txt']}
completed = failed = 0
with open(args.results, encoding='utf-8-sig') as results:
for line_number, line in enumerate(results, 1):
if not line.strip():
continue
row = json.loads(line)
if not isinstance(row, dict) or type(row.get('index')) is not int:
raise ValueError('Invalid result index on line ' + str(line_number))
expected = db.execute('SELECT video_id, outcome FROM inputs WHERE idx = ?',
(row['index'],)).fetchone()
if (not expected or row.get('video_id') != expected[0]
or row.get('source') != args.source or expected[1] is not None):
raise ValueError('Unknown, duplicate or mismatched result on line ' + str(line_number))
status = row.get('status')
if status not in ['completed', 'failed']:
raise ValueError('Unexpected item status on line ' + str(line_number))
if status == 'completed':
if not isinstance(row.get('results'), list):
raise ValueError('Completed row is missing its results array.')
if args.source == 'youtube_metadata' and (len(row['results']) != 1
or not isinstance(row['results'][0], dict)
or row['results'][0].get('videoId') != expected[0]):
raise ValueError('Metadata does not belong to the requested video.')
completed += 1
else:
failed += 1
db.execute('UPDATE inputs SET outcome = ? WHERE idx = ?', (status, row['index']))
files[status + '.ndjson'].write(json.dumps(row, ensure_ascii=False) + '\n')
db.commit()
missing = 0
for index, video_id in db.execute('SELECT idx, video_id FROM inputs WHERE outcome IS NULL ORDER BY idx'):
files['missing.ndjson'].write(json.dumps({'index': index, 'video_id': video_id,
'status': 'missing_result'}) + '\n')
missing += 1
for (video_id,) in db.execute("SELECT video_id FROM inputs WHERE outcome = 'failed' GROUP BY video_id ORDER BY MIN(idx)"):
files['failed-video-ids.txt'].write(video_id + '\n')
summary = {'source': args.source, 'inputRows': count, 'completed': completed,
'failed': failed, 'missing': missing, 'coverageComplete': missing == 0,
'allSucceeded': failed == 0 and missing == 0}
for name in files:
(out / (name + '.part')).rename(out / name)
with open(out / 'summary.json', 'x', encoding='utf-8') as output:
json.dump(summary, output, indent=2)
print(json.dumps(summary, indent=2))
sys.exit(0 if summary['allSucceeded'] else 2)
except Exception as error:
print(str(error), file=sys.stderr)
print('Audit not complete. Preserve this directory and inspect the inputs/results.', file=sys.stderr)
sys.exit(1)Run the import audit
Run from the folder containing the script, original input and downloaded results. The audit reads local files only.
python3 audit-bulk.py --input video-ids.txt --results bulk-results.ndjson --out auditpython audit-bulk.py --input video-ids.txt --results bulk-results.ndjson --out auditImport completed rows and keep the audit
| File / signal | Use it this way |
|---|---|
| completed.ndjson | Join by index to the original file. Use video_id for a catalog key; retain the job/snapshot identity when storing refresh history. |
| failed.ndjson + failed-video-ids.txt | Review errors before deliberately retrying these IDs. Successful and missing items are excluded from the failed-ID list. |
| missing.ndjson | Recover all result parts from the existing job, then audit the recovered file in a new directory. Missing output is not a confirmed failed query. |
| summary.json | coverageComplete checks that every input has an outcome. allSucceeded also requires no failed videos. The audit exits 2 for either failures or missing rows. |
| Exit 1 / no summary.json | Malformed or mismatched data: inspect the saved files. Do not publish the partial audit as a complete dataset. |
If some videos fail
A completed Bulk job can contain failed videos. The CLI saves those rows too, then exits with bulk_item_failures. Inspect their errors before deciding whether to retry individual IDs; do not resubmit the entire file automatically.
jq -c 'select(.status != "completed") | {index, video_id, status, error}' bulk-results.ndjsonTrack progress and capacity
Read the saved job while the original command runs. The CLI wraps status under data; direct HTTP wraps it under job. Check with a delay and backoff, and honor Retry-After on 429. A local timeout or closed terminal does not cancel remote work.
| Field | Read it as |
|---|---|
| input.scannedBytes / input.bytes | File scan progress, not completed queries or uploaded media. |
| input.complete | All input has been read; queries and result packaging may still be running. |
| counts.accepted / completed / failed | Submitted IDs, successful videos and failed videos. Outstanding = accepted − completed − failed. |
| counts.invalid / results | Invalid input lines / records packaged so far. HTTP can accept files containing invalid lines; the CLI validates input before upload. |
| delivery | Customer data delivery: uploadedParts, retryCount, storageType and customerStorage. Separate from each video's media upload. |
(
set -eu
BULK_ID=$(flashdata runs show bulk-first --json | jq -er '.data.remoteId')
flashdata bulk get "$BULK_ID" --json
)$run = flashdata runs show bulk-first --json | ConvertFrom-Json
if ($LASTEXITCODE -ne 0 -or -not $run.data.remoteId) { throw 'Read the saved run before continuing.' }
flashdata bulk get $run.data.remoteId --jsonKnow when to wait or act
Bulk submits gradually within your account's maxPendingJobs capacity, shared with other jobs. Insufficient credits or an inactive subscription can pause the job. Check pauseReason before resuming.
| State | Next action |
|---|---|
| awaiting_upload | Upload the file, then start the existing job. |
| running / finalizing | Wait. Finalizing can include unfinished queries, packaging and storage delivery. |
| paused | Resolve pauseReason and resume the same job. |
| completed | Download all result parts and inspect failed rows. |
| failed / cancelled | Inspect job.error and the saved receipt. Accepted work may still finish; no completed manifest is published. |
Resume an interrupted workflow
After a local timeout or interrupted result download, recover bulk-first. This reads the existing remote job; it does not submit the dataset again. Use a new output filename. A partial_result error may leave a partial local file; keep it separate from the recovered result.
Use the same CLI profile and state directory throughout. For a named profile, add --profile NAME to every command.
| Situation | Fix |
|---|---|
| awaiting_upload | runs recover does not replay an upload or start. Check the saved ID, upload status and original file before explicitly continuing. |
| paused | Resolve the reported cause, then use bulk resume with the saved ID. Recovery does not resume paused work automatically. |
| 401 / 403 | Restore valid access to the submitting account and source. Reuse the same job after authentication is fixed. |
| 409 / 410 | 409: results are not ready. 410: the stored results expired; use your saved copy. |
| Expired upload URL | Confirm the old job never started before creating a new one. An uncertain submission is a reason to read its saved ID first. |
flashdata runs show bulk-first --json
flashdata runs recover bulk-first --output recovered-results.ndjson --timeout 30m --jsonPause or resume a job
Replace BULK_ID with the ID from runs show bulk-first. Pause stops new submissions and result packaging; accepted queries continue. Resolve pauseReason before resuming. For changed options or storage credentials, update the existing job through the Bulk API.
flashdata bulk pause BULK_ID --jsonflashdata bulk resume BULK_ID --jsonCancel permanently
Only cancel when you intend to abandon the job. Cancellation does not undo accepted queries or refund completed work. Counts can still change, and no completed manifest is published. Pause is reversible.
flashdata bulk cancel BULK_ID --jsonRetrieve every result part
For an existing completed job, the CLI follows its manifest and saves every result part into one NDJSON file. Use a new output path and run the import audit on the combined file. Download results describe delivered media; they are not MP4 or audio bytes.
flashdata bulk results BULK_ID --output saved-results.ndjson --timeout 30m --jsonDeliver to your storage
Save your provider credentials as storage.json in the working folder before running this command. Use the provider fields linked below; keep the file private. The CLI's --storage-config reads this JSON file and selects result_storage for data or storage for media. --options also takes a file path, not inline JSON.
Run with the saved storage configuration
flashdata bulk run --source youtube_metadata --input video-ids.txt --storage-config storage.json --run-id bulk-with-storage --output stored-results.ndjson --timeout 30m --json