Browse documentation
Guides
Add metadata to a playlist, reading list or application dataset while keeping your original record IDs and every outcome.
On this page
From your records to an enriched dataset
Start with three records referring to two videos. Query each distinct video once, then join the metadata back to all three records. This workflow uses a small Batch request and keeps unsuccessful items visible.
| Output fields | Purpose |
|---|---|
| record_id · input | Your original record and fields, in the original order. |
| video_id · query_index · job_id | Trace each record to the query that produced it. |
| status · metadata · result_file | A completed result, or an explicit pending/failed/read-error outcome. |
1. Check your tools and access
Use FlashData CLI v1.2.1 on PATH and Node.js 22+. Run in a new folder with your existing owner credential and youtube_metadata permission. Bash and PowerShell use the same commands below.
Keep the same account and profile throughout. For a named profile, set FLASHDATA_PROFILE in this terminal so the collector also uses it. Agents can load FLASHDATA_API_KEY from their secret store.
flashdata --version
node --version
flashdata auth status --json
mkdir video-enrichment
cd video-enrichmentCheck balance and capacity
Confirm owner access, youtube_metadata permission and room for two batch items and pending jobs. If temporary access is still active, claim it or use the single-video recipe. Review the dry-run estimate in step 3 before submitting.
flashdata account balance --json
flashdata account limits --json2. Prepare your input records
Save this as videos.json. Replace the values with your own record IDs and 11-character YouTube video IDs. Additional fields stay in the local mapping and are not sent to FlashData. Repeated video IDs are allowed; record_id must be unique.
[
{
"record_id": "lesson-intro",
"video_id": "jNQXAC9IVRw",
"collection": "Course"
},
{
"record_id": "playlist-song",
"video_id": "dQw4w9WgXcQ",
"collection": "Playlist"
},
{
"record_id": "saved-intro",
"video_id": "jNQXAC9IVRw",
"collection": "Reading list"
}
]Save prepare.mjs
Create prepare.mjs beside videos.json in the video-enrichment folder. Copy the complete code below into it and save it.
The script validates every row before creating prepared/. It deduplicates video IDs for this metadata request while retaining your original records and order.
// prepare.mjs — local preparation; sends no API requests.
import { mkdir, readFile, writeFile } from 'node:fs/promises';
const rows = JSON.parse((await readFile('videos.json', 'utf8')).replace(/^\uFEFF/, ''));
if (!Array.isArray(rows) || rows.length === 0) throw new Error('Supply a nonempty array of records.');
const recordIds = new Set(), videoIndexes = new Map(), queries = [];
const mapping = rows.map((input, index) => {
if (!input || typeof input !== 'object' || Array.isArray(input)
|| typeof input.record_id !== 'string' || !input.record_id.trim()
|| recordIds.has(input.record_id) || typeof input.video_id !== 'string'
|| !/^[A-Za-z0-9_-]{11}$/.test(input.video_id)) {
throw new Error('Invalid or duplicate record_id, or invalid video_id at row ' + (index + 1));
}
recordIds.add(input.record_id);
if (!videoIndexes.has(input.video_id)) {
videoIndexes.set(input.video_id, queries.length);
queries.push({ source: 'youtube_metadata', video_id: input.video_id,
params: { include_formats: false } });
}
return { input, query_index: videoIndexes.get(input.video_id) };
});
// Existing preparation is kept intact. Use it to resume, or choose a new folder.
await mkdir('prepared');
await writeFile('prepared/batch.json', JSON.stringify(queries, null, 2), { flag: 'wx' });
await writeFile('prepared/input-map.json', JSON.stringify(mapping, null, 2), { flag: 'wx' });
console.log(JSON.stringify({ inputRecords: rows.length, uniqueQueries: queries.length,
reusedVideoIds: rows.length - queries.length }, null, 2));Build the query list locally
After saving prepare.mjs, run it once. Expect inputRecords: 3, uniqueQueries: 2 and reusedVideoIds: 1. Keep prepared/batch.json and prepared/input-map.json unchanged for this run. If prepared/ already exists, reuse it to resume; use a new folder for a different dataset.
node prepare.mjs3. Preview and submit the unique videos
The dry run checks the actual query count, current pricing and account limits. Review it, then submit once. The sample has two billable queries, even though the input has three records.
Keep submission.json and the CLI state directory. Reuse the same run ID and input after an interruption. A new run ID creates new work; inspect an uncertain submission before retrying.
flashdata batch submit --input prepared/batch.json --dry-run --jsonflashdata batch submit --input prepared/batch.json --run-id video-metadata-001 --output submission.json --json4. Collect and join the outcomes
Save collect.mjs below, then run it. The collector waits up to two minutes per unfinished job, verifies that results belong to the requested video, and joins each outcome to the original records. Run one collector at a time.
Run it again to resume the same job IDs. It reuses validated local results and sends no new queries. Exit code 1 means some outcomes still need attention; inspect them instead of resubmitting the list.
Save collect.mjs
Create collect.mjs in that same video-enrichment folder, beside submission.json and prepared/. Copy the complete code below into it and save it.
// collect.mjs — reads saved jobs; never submits another query.
import { execFileSync } from 'node:child_process';
import { mkdir, readFile, writeFile, rename } from 'node:fs/promises';
const readJSON = async file => JSON.parse((await readFile(file, 'utf8')).replace(/^\uFEFF/, ''));
const cli = (...args) => JSON.parse(execFileSync('flashdata', [
...args, '--json', '--non-interactive', '--timeout', '2m',
], { encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'], maxBuffer: 16 * 1024 * 1024 }));
const inputs = await readJSON('prepared/batch.json');
const mapping = await readJSON('prepared/input-map.json');
const saved = await readJSON('submission.json');
const receipts = (saved.submission ?? saved).queries;
if (!Array.isArray(inputs) || !inputs.length || !Array.isArray(receipts)
|| receipts.length !== inputs.length || new Set(receipts.map(r => r.id)).size !== receipts.length
|| inputs.some(q => q.source !== 'youtube_metadata') || !Array.isArray(mapping) || !mapping.length
|| mapping.some(row => !Number.isInteger(row.query_index)
|| inputs[row.query_index]?.video_id !== row.input?.video_id)) {
throw new Error('Input/receipt mismatch. Keep the original prepared files and receipt.');
}
await mkdir('results', { recursive: true });
const binding = JSON.stringify({ inputs, mapping, receipts });
try { await writeFile('results/input-receipts.json', binding, { flag: 'wx' }); }
catch (error) {
if (error.code !== 'EEXIST') throw error;
if (await readFile('results/input-receipts.json', 'utf8') !== binding) {
throw new Error('Inputs or receipts changed. Restore the original files.');
}
}
const rows = inputs.map((input, query_index) => ({ query_index,
video_id: input.video_id, job_id: receipts[query_index].id, status: 'unread' }));
const saveRows = async (path, values) => {
await writeFile(path + '.next', values.map(row => JSON.stringify(row)).join('\n') + '\n');
await rename(path + '.next', path);
};
for (const row of rows) {
try {
if (!/^[a-zA-Z0-9_-]+$/.test(row.job_id ?? '')) throw new Error('Invalid job ID.');
const file = 'results/' + row.job_id + '.json';
let result;
try { result = await readJSON(file); }
catch (error) { if (error.code !== 'ENOENT') throw error; }
if (!result) {
let job;
try { job = cli('jobs', 'wait', row.job_id).data; }
catch { job = cli('jobs', 'get', row.job_id).data; }
if (job.videoId !== row.video_id || job.source !== 'youtube_metadata') {
throw new Error('Job/input mismatch. Inspect the saved input and receipt.');
}
row.status = job.status;
row.error = job.error ?? null;
if (job.status === 'completed') {
cli('jobs', 'result', row.job_id, '--output', file);
result = await readJSON(file);
}
}
if (result) {
if (result.status !== 'done' || result.source !== 'youtube_metadata'
|| result.results?.length !== 1 || result.results[0].videoId !== row.video_id) {
throw new Error('Result/video mismatch. Inspect the saved JSON before continuing.');
}
Object.assign(row, { status: 'completed', metadata: result.results[0], error: null,
credits: result.usage?.credits ?? null, result_file: file });
}
} catch (error) {
row.status = 'read_error';
row.error = error.stderr?.toString().trim() || error.message;
}
await saveRows('metadata.ndjson', rows); // One outcome and charge per unique query.
await saveRows('enriched.ndjson', mapping.map(({ input, query_index }) => {
const { credits, ...outcome } = rows[query_index]; // Do not duplicate a charge across input records.
return { record_id: input.record_id, input, ...outcome };
}));
}
const failed = rows.filter(row => ['failed', 'cancelled'].includes(row.status));
await writeFile('failed-videos.json', JSON.stringify(failed, null, 2));
console.table(rows.map(({ video_id, job_id, status }) => ({ video_id, job_id, status })));
console.log('Saved enriched.ndjson, metadata.ndjson and results/*.json.');
if (rows.some(row => row.status !== 'completed')) process.exitCode = 1;Run or resume the collector
Run from video-enrichment after saving collect.mjs. Keep the original CLI profile and state directory.
node collect.mjsExpected files
| File | Use it for |
|---|---|
| enriched.ndjson | One row per original record, including duplicates and failures. Import by record_id. |
| metadata.ndjson | One row per unique query, with job ID, result file and actual credits. Count usage here once. |
| results/*.json | Raw successful responses plus the saved input/receipt binding. |
| failed-videos.json | Confirmed failed/cancelled jobs for review. Pending and read-error jobs are kept out of this list. |
5. Use the enriched records in your application
Save inspect.mjs in video-enrichment, beside enriched.ndjson, then run it from that folder. Expect three original records and two unique queries. Upsert completed rows by record_id, retain your input fields, and keep noncompleted rows visible in your import report. Read title and duration from metadata; use video_id for links or a later captions request.
// inspect.mjs — inspect your dataset without another API call.
import { readFile } from 'node:fs/promises';
const rows = (await readFile('enriched.ndjson', 'utf8')).trim().split('\n').map(JSON.parse);
console.table(rows.map(row => ({ record_id: row.record_id, video_id: row.video_id,
collection: row.input.collection, status: row.status, title: row.metadata?.title ?? null })));
console.log('Original records:', rows.length);
console.log('Unique queries:', new Set(rows.map(row => row.query_index)).size);node inspect.mjsContinue from the item that needs attention
Save results within their 24-hour retention period. For recurring refreshes, use a new snapshot folder and run ID only when you intend to pay for fresh data; keep record_id stable in your application.
| Outcome | Next action |
|---|---|
| pending / processing | Run the collector again later. Keep the same job IDs. |
| failed / cancelled | Review failed-videos.json and the job error. Correct the cause before intentionally requesting that video again. |
| read_error | Fix access, input binding or the saved result. Keep the raw file for inspection; a read error does not mean a new query is needed. |
| Submission outcome unknown | Inspect the local run and History before submitting again. |
flashdata runs show video-metadata-001 --json