Browse documentation
Guides
Download an MP4 to private S3 storage, resume after a restart and give your application access to the verified file.
On this page
What you’ll build
One durable item moves from a video ID to an application-ready file. Keep each stage on disk so a new process can continue from the last confirmed result.
item.json
→ submit once → saved run + job ID
→ recover the job → delivery.json
→ verify S3 bytes → media.mp4 + ready.json
→ authorize app user → short-lived download URLBefore you start
Choose one example path for all six steps. Both use an installed FlashData CLI with a saved owner profile and youtube_download permission, private S3 storage.json for uploads, and an AWS identity that can read the delivered object.
Use Python 3.10+ in your Python environment. The scripts call the CLI to preserve its run journal, and Boto3 to read S3 and sign links. Bash, jq, Node.js and AWS CLI are not needed. Run the commands below with python, or python3 if that is your interpreter name.
python --version
flashdata --version
python -m pip install boto3Install Bash, jq, Node.js 22+ and AWS CLI v2. On Windows, run Bash commands in Git Bash. Install the Node S3 signing packages in step 6.
Save the runtime settings
Create a private item folder outside source control. Save the settings for your chosen path and edit the absolute paths and profile name. Each process uses that saved account and run journal. Storage credentials stay in private storage.json. Run one process per item at a time.
Save pipeline.json. On Windows, forward-slash paths such as C:/private/flashdata-state work in JSON.
{
"stateDir": "/absolute/path/to/flashdata-state",
"storageConfig": "/absolute/path/to/private/storage.json",
"profile": "default"
}# pipeline.env — edit these paths for your existing installation.
export FLASHDATA_STATE_DIR='/absolute/path/to/flashdata-state'
export FLASHDATA_STORAGE_CONFIG='/absolute/path/to/private/storage.json'
export FLASHDATA_PROFILE='default'
# Use the saved profile for every process.
unset FLASHDATA_API_KEY1. Save one durable item
Save item.json in that folder. Replace the video and S3 destination; this example requests one MP4 at up to 720p. Keep its runId for retries. A different planned output needs a new runId and object prefix.
{
"runId": "media:clip-001:720p:v1",
"request": {
"source": "youtube_download",
"video_id": "jNQXAC9IVRw",
"params": {
"download_type": "audio_video",
"video_quality": "<=720",
"output_format": "mp4"
}
},
"destination": {
"bucket": "your-media-bucket",
"region": "us-east-1",
"prefix": "media/clip-001-v1"
}
}2. Save the script and check access
Create the file below before running the checks. It validates the S3 destination and keeps a receipt for each submission attempt.
Create the pipeline script
Create pipeline.py in the same folder as pipeline.json and item.json. Copy the complete code below into that file and save it. You will reuse it for checks, submission and recovery in steps 2–4.
# pipeline.py — Python 3.10+ and the FlashData CLI on PATH.
import argparse
import json
import os
import subprocess
import sys
import tempfile
from pathlib import Path
ROOT = Path(__file__).resolve().parent
def read_json(path):
return json.loads(Path(path).read_text(encoding='utf-8-sig'))
def delivery_directory():
# Recovery updates this pointer only after the delivery is saved successfully.
return ROOT / (ROOT / 'delivery-dir.txt').read_text(encoding='utf-8').strip()
def main(action):
os.umask(0o077)
settings = read_json(ROOT / 'pipeline.json')
for name in ['stateDir', 'storageConfig']:
if not Path(settings[name]).is_absolute():
raise ValueError('Set an absolute path for ' + name)
item = read_json(ROOT / 'item.json')
run_id = item['runId']
env = dict(os.environ)
env.pop('FLASHDATA_API_KEY', None) # Reuse the saved owner profile.
flags = ['--profile', settings['profile'], '--state-dir', settings['stateDir'],
'--json', '--non-interactive']
def cli(*args, payload=None, attempt=None):
result = subprocess.run(['flashdata', *args, *flags], cwd=ROOT, env=env,
input=json.dumps(payload) if payload is not None else None,
capture_output=True, text=True, encoding='utf-8')
if attempt:
(attempt / 'receipt.json').write_text(result.stdout, encoding='utf-8')
(attempt / 'error.json').write_text(result.stderr, encoding='utf-8')
if result.returncode:
if attempt:
raise RuntimeError('Keep ' + str(attempt) + '; inspect its error and saved run.')
raise RuntimeError(result.stderr.strip() or 'FlashData command failed.')
response = json.loads(result.stdout)
if not response.get('ok'):
raise RuntimeError('Inspect the CLI response before continuing.')
return response
def new_attempt(stage):
attempts = ROOT / 'attempts'
attempts.mkdir(exist_ok=True)
return Path(tempfile.mkdtemp(prefix=stage + '.', dir=attempts))
if action == 'check':
access = cli('auth', 'status')['data']
identity = access['identity']
if (not access.get('accessVerified')
or identity['identity'].get('recurringAllowed') is not True
or 'youtube_download' not in identity.get('permittedSources', [])):
raise ValueError('Owner access with youtube_download is required.')
print(json.dumps({'accountId': identity['identity']['accountId'],
'permittedSources': identity['permittedSources']}, indent=2))
for args in [('account', 'balance'),
('account', 'pricing', '--source', 'youtube_download', '--request-count', '1'),
('account', 'limits')]:
print(json.dumps(cli(*args)['data'], indent=2))
elif action in ['dry-run', 'submit']:
storage = read_json(settings['storageConfig'])
destination = item['destination']
if (item['request']['source'] != 'youtube_download'
or storage.get('storage_type') != 's3'
or storage.get('storage_url') != destination['bucket']
or storage.get('region') != destination['region']):
raise ValueError('Check item.json and the S3 destination in storage.json.')
request = item['request']
request['params']['storage'] = {**storage, 'object_prefix': destination['prefix']}
args = ['query', '--input', '-', '--mode', 'async', '--run-id', run_id]
if action == 'dry-run':
response = cli(*args, '--dry-run', payload=request)
if response['data'].get('dryRun') is not True:
raise ValueError('Expected a dry-run result.')
print(json.dumps({'ok': True, 'dryRun': True}))
else:
attempt = new_attempt('submit')
response = cli(*args, '--output', str(attempt / 'submission.json'),
payload=request, attempt=attempt)
print(json.dumps({'runId': run_id, 'cached': response.get('cached'),
'receipt': str(attempt / 'submission.json')}, indent=2))
elif action == 'inspect':
run = cli('runs', 'show', run_id)['data']
print(json.dumps({'runId': run_id, 'status': run['status'], 'jobId': run.get('remoteId'),
'errorCode': run.get('errorCode'),
'candidateJobIds': run.get('candidateRemoteIds')}, indent=2))
elif action == 'recover':
attempt = new_attempt('recover')
cli('runs', 'recover', run_id, '--timeout', '10m',
'--output', str(attempt / 'delivery.json'), attempt=attempt)
run = cli('runs', 'show', run_id)
(attempt / 'run.json').write_text(json.dumps(run, indent=2), encoding='utf-8')
delivery = read_json(attempt / 'delivery.json')
if delivery.get('source') != 'youtube_download' or delivery.get('status') != 'done':
raise ValueError('Inspect the saved delivery before verifying.')
pointer = ROOT / 'delivery-dir.next'
pointer.write_text(attempt.relative_to(ROOT).as_posix(), encoding='utf-8')
pointer.replace(ROOT / 'delivery-dir.txt')
print('Delivery files:', attempt)
elif action == 'delivery':
delivery = read_json(delivery_directory() / 'delivery.json')
print(json.dumps({'status': delivery['status'], 'file': delivery['results'][0],
'usage': delivery.get('usage')}, indent=2))
if __name__ == '__main__':
sys.stdout.reconfigure(encoding='utf-8')
sys.stderr.reconfigure(encoding='utf-8')
parser = argparse.ArgumentParser()
parser.add_argument('action', choices=['check', 'dry-run', 'submit', 'inspect', 'recover', 'delivery'])
try:
main(parser.parse_args().action)
except (OSError, ValueError, KeyError, RuntimeError) as error:
print(str(error), file=sys.stderr)
sys.exit(1)Create request-media.sh in the same folder as pipeline.env and item.json. Copy the complete code below into that file and save it.
#!/usr/bin/env bash
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
. ./pipeline.env
: "${FLASHDATA_STATE_DIR:?Set your existing absolute CLI state directory}"
: "${FLASHDATA_STORAGE_CONFIG:?Set the absolute path to private storage.json}"
action=${1:-dry-run}
[[ "$action" == dry-run || "$action" == submit ]] || exit 2
run_id=$(jq -er '.runId | select(type == "string" and length > 0)' item.json)
request() {
jq -en --slurpfile item item.json --slurpfile storage "$FLASHDATA_STORAGE_CONFIG" '
$item[0] as $i | $storage[0] as $s
| if $i.request.source != "youtube_download"
or $s.storage_type != "s3" or $s.storage_url != $i.destination.bucket
or $s.region != $i.destination.region
then error("Check item.json and the S3 destination in storage.json")
else $i.request | .params.storage = ($s + {object_prefix: $i.destination.prefix}) end'
}
args=(query --input - --mode async --run-id "$run_id"
--profile "${FLASHDATA_PROFILE:-default}" --state-dir "$FLASHDATA_STATE_DIR"
--json --non-interactive)
if [[ "$action" == dry-run ]]; then
request | flashdata "${args[@]}" --dry-run
else
mkdir -p attempts
attempt=$(mktemp -d attempts/submit.XXXXXX)
if ! request | flashdata "${args[@]}" --output "$attempt/submission.json" \
> "$attempt/receipt.json" 2> "$attempt/error.json"; then
cat "$attempt/error.json" >&2
printf 'Keep %s; inspect the saved run before retrying.\n' "$attempt" >&2
exit 1
fi
jq '{runId, cached, data}' "$attempt/receipt.json"
printf 'Saved receipt: %s/submission.json\n' "$attempt"
fiCheck your saved account
Run this free check from the item folder after saving the script. Missing owner access or youtube_download permission stops the check. Confirm the account ID, balance, per-TB rate and limits. The final charge depends on uploaded bytes.
python pipeline.py checkset -euo pipefail
. ./pipeline.env
: "${FLASHDATA_STATE_DIR:?Set your existing absolute CLI state directory}"
: "${FLASHDATA_STORAGE_CONFIG:?Set the absolute path to private storage.json}"
flashdata --version
jq --version
node --version
aws --version
flashdata auth status --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive |
jq -e 'if .ok and .data.accessVerified
and (.data.identity.identity.recurringAllowed == true)
and (.data.identity.permittedSources | index("youtube_download") != null)
then {accountId: .data.identity.identity.accountId,
permittedSources: .data.identity.permittedSources}
else error("Owner access with youtube_download is required.") end'
flashdata account balance --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive
flashdata account pricing --source youtube_download --request-count 1 \
--profile "${FLASHDATA_PROFILE:-default}" --state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive
flashdata account limits --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactiveValidate without a download
After saving the helper, run its dry run. Python prints ok: true and dryRun: true; Bash prints true. This validates input and access without submitting media work or reserving credits.
python pipeline.py dry-runset -euo pipefail
bash request-media.sh dry-run | jq -e '.ok and .data.dryRun'3. Submit and retain the job ID
This submits one paid background download. Expect status: accepted and a jobId in the local run. Keep the profile, state directory and item.json for recovery. The media is ready only after step 5.
python pipeline.py submitpython pipeline.py inspectbash request-media.sh submitset -euo pipefail
. ./pipeline.env
RUN_ID=$(jq -er '.runId' item.json)
flashdata runs show "$RUN_ID" --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive |
jq '{runId, status: .data.status, jobId: .data.remoteId,
errorCode: .data.errorCode, candidateJobIds: .data.candidateRemoteIds}'If the run is already saved or uncertain
| CLI state | Action |
|---|---|
| accepted + jobId | Recover this run in step 4. Keep the job ID if the process stops. |
| succeeded | Recover/export its saved response, then verify storage. Do not submit a new item. |
| Unknown outcome / no confirmed job ID | Keep the run, receipt and error. Inspect History or candidate IDs before deciding on another submission. |
4. Recover the delivery record
Run from the item folder in any new terminal. Recovery waits for the original job and prints a directory containing delivery.json, run.json and receipts. A 10-minute timeout stops local waiting; repeat recovery to resume the same job.
Resume the saved job
Reuse pipeline.py saved in step 2. No additional script is needed.
python pipeline.py recoverFirst create recover-media.sh beside item.json and paste in the code below. Save the file, then run the recovery command beneath it.
#!/usr/bin/env bash
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
. ./pipeline.env
: "${FLASHDATA_STATE_DIR:?Set your existing absolute CLI state directory}"
RUN_ID=$(jq -er '.runId' item.json)
mkdir -p attempts
DELIVERY_DIR=$(mktemp -d attempts/recover.XXXXXX)
if ! flashdata runs recover "$RUN_ID" --timeout 10m \
--profile "${FLASHDATA_PROFILE:-default}" --state-dir "$FLASHDATA_STATE_DIR" \
--output "$DELIVERY_DIR/delivery.json" --json --non-interactive \
> "$DELIVERY_DIR/receipt.json" 2> "$DELIVERY_DIR/error.json"; then
cat "$DELIVERY_DIR/error.json" >&2
printf 'Keep %s; recover the same run after resolving the error.\n' "$DELIVERY_DIR" >&2
exit 1
fi
flashdata runs show "$RUN_ID" --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive > "$DELIVERY_DIR/run.json"
jq -e '.source == "youtube_download" and .status == "done"' \
"$DELIVERY_DIR/delivery.json" >/dev/null
printf '%s\n' "$DELIVERY_DIR"set -euo pipefail
DELIVERY_DIR=$(bash recover-media.sh)
printf 'Delivery files: %s\n' "$DELIVERY_DIR"Inspect the delivered object
Expect one S3 object URL, a positive file size and an MD5. Private objects need your AWS read identity. Python saves the successful directory in delivery-dir.txt for later scripts. Bash keeps DELIVERY_DIR; in a new terminal, set it to the printed directory.
python pipeline.py deliveryjq '{status, file: .results[0], usage}' "$DELIVERY_DIR/delivery.json"5. Verify bytes before marking ready
The verifier checks the original input, S3 bucket and prefix, streams the private object to disk, and compares its byte count and MD5. Only matching files produce verified/ready.json. Python reads delivery-dir.txt; Node uses DELIVERY_DIR. Neither submits a FlashData query.
Save the verification script
This verifier handles standard Amazon S3 object URLs. For custom endpoints or another provider, use its authenticated object read and retain the same destination, byte-count and checksum checks.
Create verify-media.py beside pipeline.py and copy the complete code below into it. Save both files in the item folder.
# verify-media.py — uses Boto3's configured AWS read identity.
import hashlib
import json
import os
import re
import sys
from contextlib import closing
from datetime import datetime, timezone
from urllib.parse import unquote, urlsplit
import boto3
from pipeline import ROOT, delivery_directory, read_json
def main():
os.umask(0o077)
directory = delivery_directory()
item = read_json(ROOT / 'item.json')
saved = read_json(directory / 'run.json')
raw = (directory / 'delivery.json').read_bytes()
delivery = json.loads(raw)
run = saved.get('data', {})
if (not saved.get('ok') or run.get('id') != item['runId']
or run.get('status') != 'succeeded' or not run.get('remoteId')
or delivery.get('source') != 'youtube_download' or delivery.get('status') != 'done'
or not isinstance(delivery.get('results'), list) or len(delivery['results']) != 1):
raise ValueError('Inspect the saved run and completed delivery before verifying.')
file = delivery['results'][0]
bucket, region, prefix = (item['destination'][key] for key in ['bucket', 'region', 'prefix'])
original = run.get('input', {})
storage = original.get('params', {}).get('storage', {})
if (original.get('source') != 'youtube_download' or original.get('video_id') != item['request']['video_id']
or any(original.get('params', {}).get(key) != item['request']['params'][key]
for key in ['download_type', 'video_quality', 'output_format'])
or storage.get('storage_url') != bucket or storage.get('region') != region
or storage.get('object_prefix') != prefix):
raise ValueError('item.json changed after submission. Inspect the original run.')
url = urlsplit(file['file_url'])
hosts = [bucket + '.s3.' + region + '.amazonaws.com', bucket + '.s3.amazonaws.com']
key = unquote(url.path[1:], errors='strict')
if (url.scheme != 'https' or url.netloc not in hosts or url.query or url.fragment
or not prefix or not key.startswith(prefix + '/')):
raise ValueError('The delivery does not match the planned Amazon S3 destination.')
if (type(file.get('file_size')) is not int or file['file_size'] <= 0
or not re.fullmatch(r'[a-fA-F0-9]{32}', file.get('md5', ''))):
raise ValueError('A positive file_size and MD5 are required. Keep the manifest.')
verification = directory / 'verified'
verification.mkdir(mode=0o700) # Stop if an earlier verification already exists.
local_file = verification / 'media.mp4'
s3 = boto3.client('s3', region_name=region)
response = s3.get_object(Bucket=bucket, Key=key)
size, digest = 0, hashlib.md5(usedforsecurity=False)
with closing(response['Body']) as stream, local_file.open('xb') as output:
while chunk := stream.read(1024 * 1024):
output.write(chunk)
size += len(chunk)
digest.update(chunk)
md5 = digest.hexdigest()
if size != file['file_size'] or md5 != file['md5'].lower():
raise ValueError('Byte count or MD5 mismatch. Keep the file; do not mark it ready.')
ready = {'runId': item['runId'], 'jobId': run['remoteId'], 'videoId': item['request']['video_id'],
'bucket': bucket, 'region': region, 'objectKey': key,
'storageVersionId': response.get('VersionId'), 'bytes': size, 'md5': md5,
'verifiedAt': datetime.now(timezone.utc).isoformat(), 'localFile': str(local_file.resolve()),
'deliverySha256': hashlib.sha256(raw).hexdigest(), 'usage': delivery.get('usage')}
with (verification / 'ready.json').open('x', encoding='utf-8') as output:
json.dump(ready, output, indent=2)
print('Delivery verified:', verification / 'ready.json')
print(json.dumps(ready, indent=2))
if __name__ == '__main__':
sys.stdout.reconfigure(encoding='utf-8')
sys.stderr.reconfigure(encoding='utf-8')
try:
main()
except Exception as error:
print(str(error), file=sys.stderr)
sys.exit(1)Create verify-media.mjs beside item.json and copy the complete code below into it.
import { readFile, mkdir, writeFile } from 'node:fs/promises';
import { createReadStream } from 'node:fs';
import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { resolve, join } from 'node:path';
try {
const directory = process.argv[2];
if (!directory) throw new Error('Pass the directory printed by the recovery step.');
const readJson = async path => JSON.parse(await readFile(path, 'utf8'));
const item = await readJson('item.json');
const run = await readJson(join(directory, 'run.json'));
const raw = await readFile(join(directory, 'delivery.json'));
const delivery = JSON.parse(raw);
if (!run.ok || run.data?.id !== item.runId || run.data?.status !== 'succeeded'
|| !run.data?.remoteId || delivery.source !== 'youtube_download'
|| delivery.status !== 'done' || delivery.results?.length !== 1) {
throw new Error('Inspect the saved run and completed delivery before verifying.');
}
const file = delivery.results[0];
const { bucket, region, prefix } = item.destination;
const input = run.data.input;
const storage = input?.params?.storage;
if (input?.source !== 'youtube_download' || input.video_id !== item.request.video_id
|| ['download_type', 'video_quality', 'output_format'].some(
key => input.params?.[key] !== item.request.params[key])
|| storage?.storage_url !== bucket || storage.region !== region
|| storage.object_prefix !== prefix) {
throw new Error('item.json changed after submission. Inspect the original run.');
}
const url = new URL(file.file_url);
const hosts = [bucket + '.s3.' + region + '.amazonaws.com', bucket + '.s3.amazonaws.com'];
const objectKey = decodeURIComponent(url.pathname.slice(1));
if (url.protocol !== 'https:' || !hosts.includes(url.host) || url.username
|| url.password || url.search || url.hash || !prefix
|| !objectKey.startsWith(prefix + '/')) {
throw new Error('The delivery does not match the planned Amazon S3 destination.');
}
if (!Number.isSafeInteger(file.file_size) || file.file_size <= 0
|| !/^[a-f0-9]{32}$/i.test(file.md5 ?? '')) {
throw new Error('A positive file_size and MD5 are required. Keep the manifest.');
}
const verification = join(directory, 'verified');
await mkdir(verification, { mode: 0o700 }); // Existing attempt: stop without overwriting.
const localFile = join(verification, 'media.mp4');
let metadata;
try {
metadata = JSON.parse(execFileSync('aws', ['s3api', 'get-object',
'--bucket', bucket, '--key', objectKey, '--region', region,
'--output', 'json', '--no-cli-pager', '--no-cli-auto-prompt', localFile],
{ encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }));
} catch {
throw new Error('S3 read failed. Keep this attempt and check the AWS read identity.');
}
let bytes = 0;
const hash = createHash('md5');
for await (const chunk of createReadStream(localFile)) {
bytes += chunk.length;
hash.update(chunk);
}
const md5 = hash.digest('hex');
if (bytes !== file.file_size || md5 !== file.md5.toLowerCase()) {
throw new Error('Byte count or MD5 mismatch. Keep the file; do not mark it ready.');
}
const ready = { runId: item.runId, jobId: run.data.remoteId,
videoId: item.request.video_id, bucket, region, objectKey, storageVersionId: metadata.VersionId ?? null,
bytes, md5, verifiedAt: new Date().toISOString(), localFile: resolve(localFile),
deliverySha256: createHash('sha256').update(raw).digest('hex'), usage: delivery.usage };
const readyFile = join(verification, 'ready.json');
await writeFile(readyFile, JSON.stringify(ready, null, 2), { flag: 'wx', mode: 0o600 });
console.log('Delivery verified:', readyFile);
} catch (error) {
console.error(error instanceof Error ? error.message : 'Inspect the saved attempt.');
process.exitCode = 1;
}Run the verification
After saving the script, run it from the item folder.
python verify-media.pyset -euo pipefail
: "${DELIVERY_DIR:?Set the directory printed by recovery}"
node verify-media.mjs "$DELIVERY_DIR"
jq '{runId, jobId, bucket, region, objectKey, bytes, md5, storageVersionId}' \
"$DELIVERY_DIR/verified/ready.json"Hand off ready.json to your application
Expect Delivery verified and a record containing the run/job IDs, bucket, region, object key, verified size and MD5. This is your application’s record, created by the script. Archive it with delivery.json and the original item; it survives the API result’s retention window.
For transcription, indexing or other processing, pass the verified localFile to your worker. For a user-facing download, continue with step 6.
6. Give your application controlled access
In your backend, authorize the user and look up that user’s ready record in your own store. Then create a short-lived S3 download URL. Use the saved destination; do not accept a bucket or object key supplied by the browser.
Save the download-link script
The SDK uses the backend’s AWS read identity. No FlashData query is sent.
The URL names the verified object version when S3 versioning provides one. Without a version ID, keep each output key immutable after verification. An identity reading a specific version needs s3:GetObjectVersion; an unversioned read needs s3:GetObject.
Create create-download-link.py beside pipeline.py and copy the complete code below into it. It uses the Boto3 package installed at the start.
# create-download-link.py — run in your backend after authorizing the user.
import json
import os
import sys
import boto3
from botocore.config import Config
from pipeline import delivery_directory, read_json
def main():
os.umask(0o077)
# In an app, load this user's verified ready record from your own store.
directory = delivery_directory() / 'verified'
ready = read_json(directory / 'ready.json')
if any(not isinstance(ready.get(key), str) or not ready[key]
for key in ['bucket', 'region', 'objectKey', 'verifiedAt']):
raise ValueError('Use the ready.json produced by the verification step.')
destination = directory / 'access.json'
if destination.exists():
raise FileExistsError('Keep the existing access.json or choose a new output filename.')
params = {'Bucket': ready['bucket'], 'Key': ready['objectKey']}
version = ready.get('storageVersionId')
if version and version != 'null':
params['VersionId'] = version
s3 = boto3.client('s3', region_name=ready['region'], config=Config(signature_version='s3v4'))
url = s3.generate_presigned_url('get_object', Params=params, ExpiresIn=300, HttpMethod='GET')
with destination.open('x', encoding='utf-8') as output:
json.dump({'url': url, 'expiresInSeconds': 300}, output, indent=2)
# Do not log the signed URL; return it only to the authorized user.
print('Access record saved:', destination)
if __name__ == '__main__':
sys.stdout.reconfigure(encoding='utf-8')
sys.stderr.reconfigure(encoding='utf-8')
try:
main()
except Exception as error:
print(str(error), file=sys.stderr)
sys.exit(1)Install the signing packages in the item folder, then save the source below as create-download-link.mjs in that folder.
test -f package.json || npm init -y
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner// create-download-link.mjs — run in your backend with its AWS read identity.
import { readFile } from 'node:fs/promises';
import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
try {
const path = process.argv[2];
if (!path) throw new Error('Pass the verified ready.json file.');
const ready = JSON.parse(await readFile(path, 'utf8'));
if (['bucket', 'region', 'objectKey', 'verifiedAt'].some(
key => typeof ready[key] !== 'string' || !ready[key])) {
throw new Error('Use the ready.json produced by the verification step.');
}
// Load this record from your own store after authorizing the requesting user.
const version = ready.storageVersionId;
const command = new GetObjectCommand({
Bucket: ready.bucket, Key: ready.objectKey,
...(version && version !== 'null' ? { VersionId: version } : {}),
});
const client = new S3Client({ region: ready.region });
const url = await getSignedUrl(client, command, { expiresIn: 300 });
console.log(JSON.stringify({ url, expiresInSeconds: 300 }, null, 2));
} catch (error) {
console.error(error instanceof Error ? error.message : 'Could not create a download link.');
process.exitCode = 1;
}Return the link to the authorized user
Expect a private access.json containing url and expiresInSeconds: 300. Your backend returns only the URL to the authorized user. Anyone holding it can download the file for up to five minutes; it may expire earlier with the signing credentials.
Create a new link from the saved ready record when the user next requests access. Keep the object key and version in your database, rather than storing the expiring URL as the asset address. This local example stops if access.json already exists.
python create-download-link.py(
set -euo pipefail
umask 077
: "${DELIVERY_DIR:?Set the directory printed by recovery}"
# Keep the signed link in a private file; do not log it or paste it into chat.
set -C
node create-download-link.mjs "$DELIVERY_DIR/verified/ready.json" \
> "$DELIVERY_DIR/verified/access.json"
)Restart from the last confirmed stage
| Situation | Next action |
|---|---|
| Accepted job / local wait timed out | Repeat runs recover with the same runId into a fresh attempt directory. |
| S3 access denied or interrupted read | Fix the AWS read identity or connection. Recover to a fresh directory and verify again; keep the previous partial file. |
| Byte count / MD5 / destination mismatch | Stop. Compare the manifest, object version and planned destination before using the media. |
| verified/ already exists | Read its ready.json, or inspect its incomplete verification. The script will not overwrite it. |
| Application download URL expired | Authorize access again and sign the saved object/version. Do not submit another FlashData download. |
| run_id_input_conflict | The request changed. Recover the original run; use a new runId only for an intentional new output. |
| 401 / 403 / 402 / 429 | Fix access, permissions or balance; follow Retry-After. Inspect saved work before retrying a submission. |
Apply the same stages to a queue
- Start with one in-flight item. Read current limits before increasing application concurrency.
- Keep one durable runId for each video/output version, with a distinct destination prefix. Store the account, job ID and final outcome in your application database.
- Persist the CLI state directory across process restarts. Separate machines need shared application coordination; a local run journal does not deduplicate independent state directories.
- Use ready.json as the handoff to downstream processing. Retain failed items individually so successful deliveries are reused.