Skip to content
EN
Get API key
Browse documentation

Guides

Build a media delivery pipeline

Download an MP4 to private S3 storage, resume after a restart and give your application access to the verified file.

On this page

What you’ll build

One durable item moves from a video ID to an application-ready file. Keep each stage on disk so a new process can continue from the last confirmed result.

Pipeline and saved records
item.json
  → submit once         → saved run + job ID
  → recover the job     → delivery.json
  → verify S3 bytes     → media.mp4 + ready.json
  → authorize app user  → short-lived download URL

Before you start

Choose one example path for all six steps. Both use an installed FlashData CLI with a saved owner profile and youtube_download permission, private S3 storage.json for uploads, and an AWS identity that can read the delivered object.

Use Python 3.10+ in your Python environment. The scripts call the CLI to preserve its run journal, and Boto3 to read S3 and sign links. Bash, jq, Node.js and AWS CLI are not needed. Run the commands below with python, or python3 if that is your interpreter name.

Install the Python dependency
python --version
flashdata --version
python -m pip install boto3

Save the runtime settings

Create a private item folder outside source control. Save the settings for your chosen path and edit the absolute paths and profile name. Each process uses that saved account and run journal. Storage credentials stay in private storage.json. Run one process per item at a time.

Save pipeline.json. On Windows, forward-slash paths such as C:/private/flashdata-state work in JSON.

pipeline.json
{
  "stateDir": "/absolute/path/to/flashdata-state",
  "storageConfig": "/absolute/path/to/private/storage.json",
  "profile": "default"
}

1. Save one durable item

Save item.json in that folder. Replace the video and S3 destination; this example requests one MP4 at up to 720p. Keep its runId for retries. A different planned output needs a new runId and object prefix.

item.json
{
  "runId": "media:clip-001:720p:v1",
  "request": {
    "source": "youtube_download",
    "video_id": "jNQXAC9IVRw",
    "params": {
      "download_type": "audio_video",
      "video_quality": "<=720",
      "output_format": "mp4"
    }
  },
  "destination": {
    "bucket": "your-media-bucket",
    "region": "us-east-1",
    "prefix": "media/clip-001-v1"
  }
}

2. Save the script and check access

Create the file below before running the checks. It validates the S3 destination and keeps a receipt for each submission attempt.

Create the pipeline script

Create pipeline.py in the same folder as pipeline.json and item.json. Copy the complete code below into that file and save it. You will reuse it for checks, submission and recovery in steps 2–4.

pipeline.py
# pipeline.py — Python 3.10+ and the FlashData CLI on PATH.
import argparse
import json
import os
import subprocess
import sys
import tempfile
from pathlib import Path

ROOT = Path(__file__).resolve().parent


def read_json(path):
    return json.loads(Path(path).read_text(encoding='utf-8-sig'))


def delivery_directory():
    # Recovery updates this pointer only after the delivery is saved successfully.
    return ROOT / (ROOT / 'delivery-dir.txt').read_text(encoding='utf-8').strip()


def main(action):
    os.umask(0o077)
    settings = read_json(ROOT / 'pipeline.json')
    for name in ['stateDir', 'storageConfig']:
        if not Path(settings[name]).is_absolute():
            raise ValueError('Set an absolute path for ' + name)
    item = read_json(ROOT / 'item.json')
    run_id = item['runId']
    env = dict(os.environ)
    env.pop('FLASHDATA_API_KEY', None)  # Reuse the saved owner profile.
    flags = ['--profile', settings['profile'], '--state-dir', settings['stateDir'],
             '--json', '--non-interactive']

    def cli(*args, payload=None, attempt=None):
        result = subprocess.run(['flashdata', *args, *flags], cwd=ROOT, env=env,
                                input=json.dumps(payload) if payload is not None else None,
                                capture_output=True, text=True, encoding='utf-8')
        if attempt:
            (attempt / 'receipt.json').write_text(result.stdout, encoding='utf-8')
            (attempt / 'error.json').write_text(result.stderr, encoding='utf-8')
        if result.returncode:
            if attempt:
                raise RuntimeError('Keep ' + str(attempt) + '; inspect its error and saved run.')
            raise RuntimeError(result.stderr.strip() or 'FlashData command failed.')
        response = json.loads(result.stdout)
        if not response.get('ok'):
            raise RuntimeError('Inspect the CLI response before continuing.')
        return response

    def new_attempt(stage):
        attempts = ROOT / 'attempts'
        attempts.mkdir(exist_ok=True)
        return Path(tempfile.mkdtemp(prefix=stage + '.', dir=attempts))

    if action == 'check':
        access = cli('auth', 'status')['data']
        identity = access['identity']
        if (not access.get('accessVerified')
                or identity['identity'].get('recurringAllowed') is not True
                or 'youtube_download' not in identity.get('permittedSources', [])):
            raise ValueError('Owner access with youtube_download is required.')
        print(json.dumps({'accountId': identity['identity']['accountId'],
                          'permittedSources': identity['permittedSources']}, indent=2))
        for args in [('account', 'balance'),
                     ('account', 'pricing', '--source', 'youtube_download', '--request-count', '1'),
                     ('account', 'limits')]:
            print(json.dumps(cli(*args)['data'], indent=2))
    elif action in ['dry-run', 'submit']:
        storage = read_json(settings['storageConfig'])
        destination = item['destination']
        if (item['request']['source'] != 'youtube_download'
                or storage.get('storage_type') != 's3'
                or storage.get('storage_url') != destination['bucket']
                or storage.get('region') != destination['region']):
            raise ValueError('Check item.json and the S3 destination in storage.json.')
        request = item['request']
        request['params']['storage'] = {**storage, 'object_prefix': destination['prefix']}
        args = ['query', '--input', '-', '--mode', 'async', '--run-id', run_id]
        if action == 'dry-run':
            response = cli(*args, '--dry-run', payload=request)
            if response['data'].get('dryRun') is not True:
                raise ValueError('Expected a dry-run result.')
            print(json.dumps({'ok': True, 'dryRun': True}))
        else:
            attempt = new_attempt('submit')
            response = cli(*args, '--output', str(attempt / 'submission.json'),
                           payload=request, attempt=attempt)
            print(json.dumps({'runId': run_id, 'cached': response.get('cached'),
                              'receipt': str(attempt / 'submission.json')}, indent=2))
    elif action == 'inspect':
        run = cli('runs', 'show', run_id)['data']
        print(json.dumps({'runId': run_id, 'status': run['status'], 'jobId': run.get('remoteId'),
                          'errorCode': run.get('errorCode'),
                          'candidateJobIds': run.get('candidateRemoteIds')}, indent=2))
    elif action == 'recover':
        attempt = new_attempt('recover')
        cli('runs', 'recover', run_id, '--timeout', '10m',
            '--output', str(attempt / 'delivery.json'), attempt=attempt)
        run = cli('runs', 'show', run_id)
        (attempt / 'run.json').write_text(json.dumps(run, indent=2), encoding='utf-8')
        delivery = read_json(attempt / 'delivery.json')
        if delivery.get('source') != 'youtube_download' or delivery.get('status') != 'done':
            raise ValueError('Inspect the saved delivery before verifying.')
        pointer = ROOT / 'delivery-dir.next'
        pointer.write_text(attempt.relative_to(ROOT).as_posix(), encoding='utf-8')
        pointer.replace(ROOT / 'delivery-dir.txt')
        print('Delivery files:', attempt)
    elif action == 'delivery':
        delivery = read_json(delivery_directory() / 'delivery.json')
        print(json.dumps({'status': delivery['status'], 'file': delivery['results'][0],
                          'usage': delivery.get('usage')}, indent=2))


if __name__ == '__main__':
    sys.stdout.reconfigure(encoding='utf-8')
    sys.stderr.reconfigure(encoding='utf-8')
    parser = argparse.ArgumentParser()
    parser.add_argument('action', choices=['check', 'dry-run', 'submit', 'inspect', 'recover', 'delivery'])
    try:
        main(parser.parse_args().action)
    except (OSError, ValueError, KeyError, RuntimeError) as error:
        print(str(error), file=sys.stderr)
        sys.exit(1)

Check your saved account

Run this free check from the item folder after saving the script. Missing owner access or youtube_download permission stops the check. Confirm the account ID, balance, per-TB rate and limits. The final charge depends on uploaded bytes.

Check the environment and account
python pipeline.py check

Validate without a download

After saving the helper, run its dry run. Python prints ok: true and dryRun: true; Bash prints true. This validates input and access without submitting media work or reserving credits.

Free dry run
python pipeline.py dry-run

3. Submit and retain the job ID

This submits one paid background download. Expect status: accepted and a jobId in the local run. Keep the profile, state directory and item.json for recovery. The media is ready only after step 5.

Submit once
python pipeline.py submit
Inspect local state · no new query
python pipeline.py inspect

If the run is already saved or uncertain

CLI stateAction
accepted + jobIdRecover this run in step 4. Keep the job ID if the process stops.
succeededRecover/export its saved response, then verify storage. Do not submit a new item.
Unknown outcome / no confirmed job IDKeep the run, receipt and error. Inspect History or candidate IDs before deciding on another submission.

4. Recover the delivery record

Run from the item folder in any new terminal. Recovery waits for the original job and prints a directory containing delivery.json, run.json and receipts. A 10-minute timeout stops local waiting; repeat recovery to resume the same job.

Resume the saved job

Reuse pipeline.py saved in step 2. No additional script is needed.

Recover the existing job · no new submission
python pipeline.py recover

Inspect the delivered object

Expect one S3 object URL, a positive file size and an MD5. Private objects need your AWS read identity. Python saves the successful directory in delivery-dir.txt for later scripts. Bash keeps DELIVERY_DIR; in a new terminal, set it to the printed directory.

Delivery record
python pipeline.py delivery

5. Verify bytes before marking ready

The verifier checks the original input, S3 bucket and prefix, streams the private object to disk, and compares its byte count and MD5. Only matching files produce verified/ready.json. Python reads delivery-dir.txt; Node uses DELIVERY_DIR. Neither submits a FlashData query.

Save the verification script

This verifier handles standard Amazon S3 object URLs. For custom endpoints or another provider, use its authenticated object read and retain the same destination, byte-count and checksum checks.

Create verify-media.py beside pipeline.py and copy the complete code below into it. Save both files in the item folder.

verify-media.py
# verify-media.py — uses Boto3's configured AWS read identity.
import hashlib
import json
import os
import re
import sys
from contextlib import closing
from datetime import datetime, timezone
from urllib.parse import unquote, urlsplit

import boto3
from pipeline import ROOT, delivery_directory, read_json


def main():
    os.umask(0o077)
    directory = delivery_directory()
    item = read_json(ROOT / 'item.json')
    saved = read_json(directory / 'run.json')
    raw = (directory / 'delivery.json').read_bytes()
    delivery = json.loads(raw)
    run = saved.get('data', {})
    if (not saved.get('ok') or run.get('id') != item['runId']
            or run.get('status') != 'succeeded' or not run.get('remoteId')
            or delivery.get('source') != 'youtube_download' or delivery.get('status') != 'done'
            or not isinstance(delivery.get('results'), list) or len(delivery['results']) != 1):
        raise ValueError('Inspect the saved run and completed delivery before verifying.')
    file = delivery['results'][0]
    bucket, region, prefix = (item['destination'][key] for key in ['bucket', 'region', 'prefix'])
    original = run.get('input', {})
    storage = original.get('params', {}).get('storage', {})
    if (original.get('source') != 'youtube_download' or original.get('video_id') != item['request']['video_id']
            or any(original.get('params', {}).get(key) != item['request']['params'][key]
                   for key in ['download_type', 'video_quality', 'output_format'])
            or storage.get('storage_url') != bucket or storage.get('region') != region
            or storage.get('object_prefix') != prefix):
        raise ValueError('item.json changed after submission. Inspect the original run.')
    url = urlsplit(file['file_url'])
    hosts = [bucket + '.s3.' + region + '.amazonaws.com', bucket + '.s3.amazonaws.com']
    key = unquote(url.path[1:], errors='strict')
    if (url.scheme != 'https' or url.netloc not in hosts or url.query or url.fragment
            or not prefix or not key.startswith(prefix + '/')):
        raise ValueError('The delivery does not match the planned Amazon S3 destination.')
    if (type(file.get('file_size')) is not int or file['file_size'] <= 0
            or not re.fullmatch(r'[a-fA-F0-9]{32}', file.get('md5', ''))):
        raise ValueError('A positive file_size and MD5 are required. Keep the manifest.')
    verification = directory / 'verified'
    verification.mkdir(mode=0o700)  # Stop if an earlier verification already exists.
    local_file = verification / 'media.mp4'
    s3 = boto3.client('s3', region_name=region)
    response = s3.get_object(Bucket=bucket, Key=key)
    size, digest = 0, hashlib.md5(usedforsecurity=False)
    with closing(response['Body']) as stream, local_file.open('xb') as output:
        while chunk := stream.read(1024 * 1024):
            output.write(chunk)
            size += len(chunk)
            digest.update(chunk)
    md5 = digest.hexdigest()
    if size != file['file_size'] or md5 != file['md5'].lower():
        raise ValueError('Byte count or MD5 mismatch. Keep the file; do not mark it ready.')
    ready = {'runId': item['runId'], 'jobId': run['remoteId'], 'videoId': item['request']['video_id'],
             'bucket': bucket, 'region': region, 'objectKey': key,
             'storageVersionId': response.get('VersionId'), 'bytes': size, 'md5': md5,
             'verifiedAt': datetime.now(timezone.utc).isoformat(), 'localFile': str(local_file.resolve()),
             'deliverySha256': hashlib.sha256(raw).hexdigest(), 'usage': delivery.get('usage')}
    with (verification / 'ready.json').open('x', encoding='utf-8') as output:
        json.dump(ready, output, indent=2)
    print('Delivery verified:', verification / 'ready.json')
    print(json.dumps(ready, indent=2))


if __name__ == '__main__':
    sys.stdout.reconfigure(encoding='utf-8')
    sys.stderr.reconfigure(encoding='utf-8')
    try:
        main()
    except Exception as error:
        print(str(error), file=sys.stderr)
        sys.exit(1)

Run the verification

After saving the script, run it from the item folder.

Verify and read the handoff record
python verify-media.py

Hand off ready.json to your application

Expect Delivery verified and a record containing the run/job IDs, bucket, region, object key, verified size and MD5. This is your application’s record, created by the script. Archive it with delivery.json and the original item; it survives the API result’s retention window.

For transcription, indexing or other processing, pass the verified localFile to your worker. For a user-facing download, continue with step 6.

6. Give your application controlled access

In your backend, authorize the user and look up that user’s ready record in your own store. Then create a short-lived S3 download URL. Use the saved destination; do not accept a bucket or object key supplied by the browser.

The SDK uses the backend’s AWS read identity. No FlashData query is sent.

The URL names the verified object version when S3 versioning provides one. Without a version ID, keep each output key immutable after verification. An identity reading a specific version needs s3:GetObjectVersion; an unversioned read needs s3:GetObject.

Create create-download-link.py beside pipeline.py and copy the complete code below into it. It uses the Boto3 package installed at the start.

create-download-link.py
# create-download-link.py — run in your backend after authorizing the user.
import json
import os
import sys

import boto3
from botocore.config import Config
from pipeline import delivery_directory, read_json


def main():
    os.umask(0o077)
    # In an app, load this user's verified ready record from your own store.
    directory = delivery_directory() / 'verified'
    ready = read_json(directory / 'ready.json')
    if any(not isinstance(ready.get(key), str) or not ready[key]
           for key in ['bucket', 'region', 'objectKey', 'verifiedAt']):
        raise ValueError('Use the ready.json produced by the verification step.')
    destination = directory / 'access.json'
    if destination.exists():
        raise FileExistsError('Keep the existing access.json or choose a new output filename.')
    params = {'Bucket': ready['bucket'], 'Key': ready['objectKey']}
    version = ready.get('storageVersionId')
    if version and version != 'null':
        params['VersionId'] = version
    s3 = boto3.client('s3', region_name=ready['region'], config=Config(signature_version='s3v4'))
    url = s3.generate_presigned_url('get_object', Params=params, ExpiresIn=300, HttpMethod='GET')
    with destination.open('x', encoding='utf-8') as output:
        json.dump({'url': url, 'expiresInSeconds': 300}, output, indent=2)
    # Do not log the signed URL; return it only to the authorized user.
    print('Access record saved:', destination)


if __name__ == '__main__':
    sys.stdout.reconfigure(encoding='utf-8')
    sys.stderr.reconfigure(encoding='utf-8')
    try:
        main()
    except Exception as error:
        print(str(error), file=sys.stderr)
        sys.exit(1)

Expect a private access.json containing url and expiresInSeconds: 300. Your backend returns only the URL to the authorized user. Anyone holding it can download the file for up to five minutes; it may expire earlier with the signing credentials.

Create a new link from the saved ready record when the user next requests access. Keep the object key and version in your database, rather than storing the expiring URL as the asset address. This local example stops if access.json already exists.

Create a five-minute link
python create-download-link.py

Restart from the last confirmed stage

SituationNext action
Accepted job / local wait timed outRepeat runs recover with the same runId into a fresh attempt directory.
S3 access denied or interrupted readFix the AWS read identity or connection. Recover to a fresh directory and verify again; keep the previous partial file.
Byte count / MD5 / destination mismatchStop. Compare the manifest, object version and planned destination before using the media.
verified/ already existsRead its ready.json, or inspect its incomplete verification. The script will not overwrite it.
Application download URL expiredAuthorize access again and sign the saved object/version. Do not submit another FlashData download.
run_id_input_conflictThe request changed. Recover the original run; use a new runId only for an intentional new output.
401 / 403 / 402 / 429Fix access, permissions or balance; follow Retry-After. Inspect saved work before retrying a submission.

Apply the same stages to a queue

  1. Start with one in-flight item. Read current limits before increasing application concurrency.
  2. Keep one durable runId for each video/output version, with a distinct destination prefix. Store the account, job ID and final outcome in your application database.
  3. Persist the CLI state directory across process restarts. Separate machines need shared application coordination; a local run journal does not deduplicate independent state directories.
  4. Use ready.json as the handoff to downstream processing. Retain failed items individually so successful deliveries are reused.