浏览文档
指南
将 MP4 下载到私有 S3 存储,在重启后恢复任务,并让应用访问已验证的文件。
本页内容
你将构建什么
将一项持久任务从视频 ID 转为应用可用的文件。把每个阶段保存在磁盘上,使新进程能从最后确认的结果继续。
item.json
→ 提交一次 → 保存运行记录及任务 ID
→ 恢复任务 → delivery.json
→ 验证 S3 文件 → media.mp4 + ready.json
→ 确认用户权限 → 短期下载 URL开始前准备
为全部六步选择同一示例方式。两种方式都需要已安装的 FlashData CLI、具有 youtube_download 权限的已保存所有者配置档、用于上传的私有 S3 storage.json,以及可读取交付对象的 AWS 身份。
在 Python 环境中使用 Python 3.10+。脚本通过 CLI 保留运行日志,通过 Boto3 读取 S3 并签名链接,无需 Bash、jq、Node.js 或 AWS CLI。下方命令使用 python;若解释器名为 python3,请相应替换。
python --version
flashdata --version
python -m pip install boto3安装 Bash、jq、Node.js 22+ 和 AWS CLI v2。Windows 使用 Git Bash 执行 Bash 命令,第 6 步安装 Node S3 签名包。
保存运行环境设置
在版本控制之外创建私有任务目录。保存所选方式的设置,修改绝对路径及配置档名。每个进程使用相同账户及运行日志,存储凭据保留在私有 storage.json 中。每项任务同时只运行一个进程。
保存 pipeline.json。Windows 可在 JSON 中使用 C:/private/flashdata-state 这样的正斜杠路径。
{
"stateDir": "/absolute/path/to/flashdata-state",
"storageConfig": "/absolute/path/to/private/storage.json",
"profile": "default"
}# pipeline.env — edit these paths for your existing installation.
export FLASHDATA_STATE_DIR='/absolute/path/to/flashdata-state'
export FLASHDATA_STORAGE_CONFIG='/absolute/path/to/private/storage.json'
export FLASHDATA_PROFILE='default'
# Use the saved profile for every process.
unset FLASHDATA_API_KEY1. 保存一项持久任务
在任务目录保存 item.json,替换视频及 S3 目标。此示例请求一个最高 720p 的 MP4。重试使用同一 runId;不同预期输出使用新的 runId 及对象前缀。
{
"runId": "media:clip-001:720p:v1",
"request": {
"source": "youtube_download",
"video_id": "jNQXAC9IVRw",
"params": {
"download_type": "audio_video",
"video_quality": "<=720",
"output_format": "mp4"
}
},
"destination": {
"bucket": "your-media-bucket",
"region": "us-east-1",
"prefix": "media/clip-001-v1"
}
}2. 保存脚本并检查访问
先创建下方文件,再运行检查。脚本验证 S3 目标,并为每次提交尝试保留回执。
创建流水线脚本
在 pipeline.json 及 item.json 所在目录创建 pipeline.py。将下方完整代码复制到文件中并保存。第 2–4 步复用此文件进行检查、提交及恢复。
# pipeline.py — Python 3.10+ and the FlashData CLI on PATH.
import argparse
import json
import os
import subprocess
import sys
import tempfile
from pathlib import Path
ROOT = Path(__file__).resolve().parent
def read_json(path):
return json.loads(Path(path).read_text(encoding='utf-8-sig'))
def delivery_directory():
# Recovery updates this pointer only after the delivery is saved successfully.
return ROOT / (ROOT / 'delivery-dir.txt').read_text(encoding='utf-8').strip()
def main(action):
os.umask(0o077)
settings = read_json(ROOT / 'pipeline.json')
for name in ['stateDir', 'storageConfig']:
if not Path(settings[name]).is_absolute():
raise ValueError('Set an absolute path for ' + name)
item = read_json(ROOT / 'item.json')
run_id = item['runId']
env = dict(os.environ)
env.pop('FLASHDATA_API_KEY', None) # Reuse the saved owner profile.
flags = ['--profile', settings['profile'], '--state-dir', settings['stateDir'],
'--json', '--non-interactive']
def cli(*args, payload=None, attempt=None):
result = subprocess.run(['flashdata', *args, *flags], cwd=ROOT, env=env,
input=json.dumps(payload) if payload is not None else None,
capture_output=True, text=True, encoding='utf-8')
if attempt:
(attempt / 'receipt.json').write_text(result.stdout, encoding='utf-8')
(attempt / 'error.json').write_text(result.stderr, encoding='utf-8')
if result.returncode:
if attempt:
raise RuntimeError('Keep ' + str(attempt) + '; inspect its error and saved run.')
raise RuntimeError(result.stderr.strip() or 'FlashData command failed.')
response = json.loads(result.stdout)
if not response.get('ok'):
raise RuntimeError('Inspect the CLI response before continuing.')
return response
def new_attempt(stage):
attempts = ROOT / 'attempts'
attempts.mkdir(exist_ok=True)
return Path(tempfile.mkdtemp(prefix=stage + '.', dir=attempts))
if action == 'check':
access = cli('auth', 'status')['data']
identity = access['identity']
if (not access.get('accessVerified')
or identity['identity'].get('recurringAllowed') is not True
or 'youtube_download' not in identity.get('permittedSources', [])):
raise ValueError('Owner access with youtube_download is required.')
print(json.dumps({'accountId': identity['identity']['accountId'],
'permittedSources': identity['permittedSources']}, indent=2))
for args in [('account', 'balance'),
('account', 'pricing', '--source', 'youtube_download', '--request-count', '1'),
('account', 'limits')]:
print(json.dumps(cli(*args)['data'], indent=2))
elif action in ['dry-run', 'submit']:
storage = read_json(settings['storageConfig'])
destination = item['destination']
if (item['request']['source'] != 'youtube_download'
or storage.get('storage_type') != 's3'
or storage.get('storage_url') != destination['bucket']
or storage.get('region') != destination['region']):
raise ValueError('Check item.json and the S3 destination in storage.json.')
request = item['request']
request['params']['storage'] = {**storage, 'object_prefix': destination['prefix']}
args = ['query', '--input', '-', '--mode', 'async', '--run-id', run_id]
if action == 'dry-run':
response = cli(*args, '--dry-run', payload=request)
if response['data'].get('dryRun') is not True:
raise ValueError('Expected a dry-run result.')
print(json.dumps({'ok': True, 'dryRun': True}))
else:
attempt = new_attempt('submit')
response = cli(*args, '--output', str(attempt / 'submission.json'),
payload=request, attempt=attempt)
print(json.dumps({'runId': run_id, 'cached': response.get('cached'),
'receipt': str(attempt / 'submission.json')}, indent=2))
elif action == 'inspect':
run = cli('runs', 'show', run_id)['data']
print(json.dumps({'runId': run_id, 'status': run['status'], 'jobId': run.get('remoteId'),
'errorCode': run.get('errorCode'),
'candidateJobIds': run.get('candidateRemoteIds')}, indent=2))
elif action == 'recover':
attempt = new_attempt('recover')
cli('runs', 'recover', run_id, '--timeout', '10m',
'--output', str(attempt / 'delivery.json'), attempt=attempt)
run = cli('runs', 'show', run_id)
(attempt / 'run.json').write_text(json.dumps(run, indent=2), encoding='utf-8')
delivery = read_json(attempt / 'delivery.json')
if delivery.get('source') != 'youtube_download' or delivery.get('status') != 'done':
raise ValueError('Inspect the saved delivery before verifying.')
pointer = ROOT / 'delivery-dir.next'
pointer.write_text(attempt.relative_to(ROOT).as_posix(), encoding='utf-8')
pointer.replace(ROOT / 'delivery-dir.txt')
print('Delivery files:', attempt)
elif action == 'delivery':
delivery = read_json(delivery_directory() / 'delivery.json')
print(json.dumps({'status': delivery['status'], 'file': delivery['results'][0],
'usage': delivery.get('usage')}, indent=2))
if __name__ == '__main__':
sys.stdout.reconfigure(encoding='utf-8')
sys.stderr.reconfigure(encoding='utf-8')
parser = argparse.ArgumentParser()
parser.add_argument('action', choices=['check', 'dry-run', 'submit', 'inspect', 'recover', 'delivery'])
try:
main(parser.parse_args().action)
except (OSError, ValueError, KeyError, RuntimeError) as error:
print(str(error), file=sys.stderr)
sys.exit(1)在 pipeline.env 及 item.json 所在目录创建 request-media.sh,将下方完整代码复制到文件中并保存。
#!/usr/bin/env bash
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
. ./pipeline.env
: "${FLASHDATA_STATE_DIR:?Set your existing absolute CLI state directory}"
: "${FLASHDATA_STORAGE_CONFIG:?Set the absolute path to private storage.json}"
action=${1:-dry-run}
[[ "$action" == dry-run || "$action" == submit ]] || exit 2
run_id=$(jq -er '.runId | select(type == "string" and length > 0)' item.json)
request() {
jq -en --slurpfile item item.json --slurpfile storage "$FLASHDATA_STORAGE_CONFIG" '
$item[0] as $i | $storage[0] as $s
| if $i.request.source != "youtube_download"
or $s.storage_type != "s3" or $s.storage_url != $i.destination.bucket
or $s.region != $i.destination.region
then error("Check item.json and the S3 destination in storage.json")
else $i.request | .params.storage = ($s + {object_prefix: $i.destination.prefix}) end'
}
args=(query --input - --mode async --run-id "$run_id"
--profile "${FLASHDATA_PROFILE:-default}" --state-dir "$FLASHDATA_STATE_DIR"
--json --non-interactive)
if [[ "$action" == dry-run ]]; then
request | flashdata "${args[@]}" --dry-run
else
mkdir -p attempts
attempt=$(mktemp -d attempts/submit.XXXXXX)
if ! request | flashdata "${args[@]}" --output "$attempt/submission.json" \
> "$attempt/receipt.json" 2> "$attempt/error.json"; then
cat "$attempt/error.json" >&2
printf 'Keep %s; inspect the saved run before retrying.\n' "$attempt" >&2
exit 1
fi
jq '{runId, cached, data}' "$attempt/receipt.json"
printf 'Saved receipt: %s/submission.json\n' "$attempt"
fi检查已保存的账户
保存脚本后,在任务目录运行此免费检查。缺少所有者访问或 youtube_download 权限时会停止。确认账户 ID、余额、每 TB 费率及限额。最终费用取决于上传字节数。
python pipeline.py checkset -euo pipefail
. ./pipeline.env
: "${FLASHDATA_STATE_DIR:?Set your existing absolute CLI state directory}"
: "${FLASHDATA_STORAGE_CONFIG:?Set the absolute path to private storage.json}"
flashdata --version
jq --version
node --version
aws --version
flashdata auth status --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive |
jq -e 'if .ok and .data.accessVerified
and (.data.identity.identity.recurringAllowed == true)
and (.data.identity.permittedSources | index("youtube_download") != null)
then {accountId: .data.identity.identity.accountId,
permittedSources: .data.identity.permittedSources}
else error("Owner access with youtube_download is required.") end'
flashdata account balance --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive
flashdata account pricing --source youtube_download --request-count 1 \
--profile "${FLASHDATA_PROFILE:-default}" --state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive
flashdata account limits --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive验证而不下载
保存辅助脚本后执行预检查。Python 输出 ok: true 及 dryRun: true;Bash 输出 true。此步骤验证输入及访问,不提交媒体任务,也不预留 Credits。
python pipeline.py dry-runset -euo pipefail
bash request-media.sh dry-run | jq -e '.ok and .data.dryRun'3. 提交并保留任务 ID
此步骤提交一次付费后台下载。本地运行中应出现 status: accepted 及 jobId。保留配置档、状态目录及 item.json,以便恢复。第 5 步完成后媒体才可供应用使用。
python pipeline.py submitpython pipeline.py inspectbash request-media.sh submitset -euo pipefail
. ./pipeline.env
RUN_ID=$(jq -er '.runId' item.json)
flashdata runs show "$RUN_ID" --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive |
jq '{runId, status: .data.status, jobId: .data.remoteId,
errorCode: .data.errorCode, candidateJobIds: .data.candidateRemoteIds}'4. 恢复交付记录
在任意新终端进入任务目录后执行。恢复流程等待原始任务,输出包含 delivery.json、run.json 及回执的目录。10 分钟超时仅停止本地等待,重复恢复即可继续相同任务。
继续已保存的任务
复用第 2 步保存的 pipeline.py,无需创建其他脚本。
python pipeline.py recover先在 item.json 旁创建 recover-media.sh,粘贴下方代码并保存,再执行代码下方的恢复命令。
#!/usr/bin/env bash
set -euo pipefail
umask 077
cd -- "$(dirname -- "$0")"
. ./pipeline.env
: "${FLASHDATA_STATE_DIR:?Set your existing absolute CLI state directory}"
RUN_ID=$(jq -er '.runId' item.json)
mkdir -p attempts
DELIVERY_DIR=$(mktemp -d attempts/recover.XXXXXX)
if ! flashdata runs recover "$RUN_ID" --timeout 10m \
--profile "${FLASHDATA_PROFILE:-default}" --state-dir "$FLASHDATA_STATE_DIR" \
--output "$DELIVERY_DIR/delivery.json" --json --non-interactive \
> "$DELIVERY_DIR/receipt.json" 2> "$DELIVERY_DIR/error.json"; then
cat "$DELIVERY_DIR/error.json" >&2
printf 'Keep %s; recover the same run after resolving the error.\n' "$DELIVERY_DIR" >&2
exit 1
fi
flashdata runs show "$RUN_ID" --profile "${FLASHDATA_PROFILE:-default}" \
--state-dir "$FLASHDATA_STATE_DIR" --json --non-interactive > "$DELIVERY_DIR/run.json"
jq -e '.source == "youtube_download" and .status == "done"' \
"$DELIVERY_DIR/delivery.json" >/dev/null
printf '%s\n' "$DELIVERY_DIR"set -euo pipefail
DELIVERY_DIR=$(bash recover-media.sh)
printf 'Delivery files: %s\n' "$DELIVERY_DIR"检查交付对象
预期得到一个 S3 对象 URL、正的文件大小及 MD5,私有对象需 AWS 读取身份。Python 将成功目录保存到 delivery-dir.txt,供后续脚本使用。Bash 保留 DELIVERY_DIR,换用新终端时将它设为刚输出的目录。
python pipeline.py deliveryjq '{status, file: .results[0], usage}' "$DELIVERY_DIR/delivery.json"5. 标记就绪前验证文件
验证脚本检查原输入、S3 Bucket 及前缀,将私有对象流式写入磁盘,再比对字节数及 MD5。只有匹配文件才生成 verified/ready.json。Python 读取 delivery-dir.txt,Node 使用 DELIVERY_DIR。两者都不提交 FlashData 查询。
保存验证脚本
此验证器处理标准 Amazon S3 对象 URL。自定义端点或其他提供商应使用相应的认证对象读取,并保留相同的目标、字节数及校验和检查。
在 pipeline.py 旁创建 verify-media.py,复制下方完整代码。将两个文件保存在任务目录。
# verify-media.py — uses Boto3's configured AWS read identity.
import hashlib
import json
import os
import re
import sys
from contextlib import closing
from datetime import datetime, timezone
from urllib.parse import unquote, urlsplit
import boto3
from pipeline import ROOT, delivery_directory, read_json
def main():
os.umask(0o077)
directory = delivery_directory()
item = read_json(ROOT / 'item.json')
saved = read_json(directory / 'run.json')
raw = (directory / 'delivery.json').read_bytes()
delivery = json.loads(raw)
run = saved.get('data', {})
if (not saved.get('ok') or run.get('id') != item['runId']
or run.get('status') != 'succeeded' or not run.get('remoteId')
or delivery.get('source') != 'youtube_download' or delivery.get('status') != 'done'
or not isinstance(delivery.get('results'), list) or len(delivery['results']) != 1):
raise ValueError('Inspect the saved run and completed delivery before verifying.')
file = delivery['results'][0]
bucket, region, prefix = (item['destination'][key] for key in ['bucket', 'region', 'prefix'])
original = run.get('input', {})
storage = original.get('params', {}).get('storage', {})
if (original.get('source') != 'youtube_download' or original.get('video_id') != item['request']['video_id']
or any(original.get('params', {}).get(key) != item['request']['params'][key]
for key in ['download_type', 'video_quality', 'output_format'])
or storage.get('storage_url') != bucket or storage.get('region') != region
or storage.get('object_prefix') != prefix):
raise ValueError('item.json changed after submission. Inspect the original run.')
url = urlsplit(file['file_url'])
hosts = [bucket + '.s3.' + region + '.amazonaws.com', bucket + '.s3.amazonaws.com']
key = unquote(url.path[1:], errors='strict')
if (url.scheme != 'https' or url.netloc not in hosts or url.query or url.fragment
or not prefix or not key.startswith(prefix + '/')):
raise ValueError('The delivery does not match the planned Amazon S3 destination.')
if (type(file.get('file_size')) is not int or file['file_size'] <= 0
or not re.fullmatch(r'[a-fA-F0-9]{32}', file.get('md5', ''))):
raise ValueError('A positive file_size and MD5 are required. Keep the manifest.')
verification = directory / 'verified'
verification.mkdir(mode=0o700) # Stop if an earlier verification already exists.
local_file = verification / 'media.mp4'
s3 = boto3.client('s3', region_name=region)
response = s3.get_object(Bucket=bucket, Key=key)
size, digest = 0, hashlib.md5(usedforsecurity=False)
with closing(response['Body']) as stream, local_file.open('xb') as output:
while chunk := stream.read(1024 * 1024):
output.write(chunk)
size += len(chunk)
digest.update(chunk)
md5 = digest.hexdigest()
if size != file['file_size'] or md5 != file['md5'].lower():
raise ValueError('Byte count or MD5 mismatch. Keep the file; do not mark it ready.')
ready = {'runId': item['runId'], 'jobId': run['remoteId'], 'videoId': item['request']['video_id'],
'bucket': bucket, 'region': region, 'objectKey': key,
'storageVersionId': response.get('VersionId'), 'bytes': size, 'md5': md5,
'verifiedAt': datetime.now(timezone.utc).isoformat(), 'localFile': str(local_file.resolve()),
'deliverySha256': hashlib.sha256(raw).hexdigest(), 'usage': delivery.get('usage')}
with (verification / 'ready.json').open('x', encoding='utf-8') as output:
json.dump(ready, output, indent=2)
print('Delivery verified:', verification / 'ready.json')
print(json.dumps(ready, indent=2))
if __name__ == '__main__':
sys.stdout.reconfigure(encoding='utf-8')
sys.stderr.reconfigure(encoding='utf-8')
try:
main()
except Exception as error:
print(str(error), file=sys.stderr)
sys.exit(1)在 item.json 旁创建 verify-media.mjs,复制下方完整代码并保存。
import { readFile, mkdir, writeFile } from 'node:fs/promises';
import { createReadStream } from 'node:fs';
import { createHash } from 'node:crypto';
import { execFileSync } from 'node:child_process';
import { resolve, join } from 'node:path';
try {
const directory = process.argv[2];
if (!directory) throw new Error('Pass the directory printed by the recovery step.');
const readJson = async path => JSON.parse(await readFile(path, 'utf8'));
const item = await readJson('item.json');
const run = await readJson(join(directory, 'run.json'));
const raw = await readFile(join(directory, 'delivery.json'));
const delivery = JSON.parse(raw);
if (!run.ok || run.data?.id !== item.runId || run.data?.status !== 'succeeded'
|| !run.data?.remoteId || delivery.source !== 'youtube_download'
|| delivery.status !== 'done' || delivery.results?.length !== 1) {
throw new Error('Inspect the saved run and completed delivery before verifying.');
}
const file = delivery.results[0];
const { bucket, region, prefix } = item.destination;
const input = run.data.input;
const storage = input?.params?.storage;
if (input?.source !== 'youtube_download' || input.video_id !== item.request.video_id
|| ['download_type', 'video_quality', 'output_format'].some(
key => input.params?.[key] !== item.request.params[key])
|| storage?.storage_url !== bucket || storage.region !== region
|| storage.object_prefix !== prefix) {
throw new Error('item.json changed after submission. Inspect the original run.');
}
const url = new URL(file.file_url);
const hosts = [bucket + '.s3.' + region + '.amazonaws.com', bucket + '.s3.amazonaws.com'];
const objectKey = decodeURIComponent(url.pathname.slice(1));
if (url.protocol !== 'https:' || !hosts.includes(url.host) || url.username
|| url.password || url.search || url.hash || !prefix
|| !objectKey.startsWith(prefix + '/')) {
throw new Error('The delivery does not match the planned Amazon S3 destination.');
}
if (!Number.isSafeInteger(file.file_size) || file.file_size <= 0
|| !/^[a-f0-9]{32}$/i.test(file.md5 ?? '')) {
throw new Error('A positive file_size and MD5 are required. Keep the manifest.');
}
const verification = join(directory, 'verified');
await mkdir(verification, { mode: 0o700 }); // Existing attempt: stop without overwriting.
const localFile = join(verification, 'media.mp4');
let metadata;
try {
metadata = JSON.parse(execFileSync('aws', ['s3api', 'get-object',
'--bucket', bucket, '--key', objectKey, '--region', region,
'--output', 'json', '--no-cli-pager', '--no-cli-auto-prompt', localFile],
{ encoding: 'utf8', stdio: ['ignore', 'pipe', 'pipe'] }));
} catch {
throw new Error('S3 read failed. Keep this attempt and check the AWS read identity.');
}
let bytes = 0;
const hash = createHash('md5');
for await (const chunk of createReadStream(localFile)) {
bytes += chunk.length;
hash.update(chunk);
}
const md5 = hash.digest('hex');
if (bytes !== file.file_size || md5 !== file.md5.toLowerCase()) {
throw new Error('Byte count or MD5 mismatch. Keep the file; do not mark it ready.');
}
const ready = { runId: item.runId, jobId: run.data.remoteId,
videoId: item.request.video_id, bucket, region, objectKey, storageVersionId: metadata.VersionId ?? null,
bytes, md5, verifiedAt: new Date().toISOString(), localFile: resolve(localFile),
deliverySha256: createHash('sha256').update(raw).digest('hex'), usage: delivery.usage };
const readyFile = join(verification, 'ready.json');
await writeFile(readyFile, JSON.stringify(ready, null, 2), { flag: 'wx', mode: 0o600 });
console.log('Delivery verified:', readyFile);
} catch (error) {
console.error(error instanceof Error ? error.message : 'Inspect the saved attempt.');
process.exitCode = 1;
}运行文件验证
保存脚本后,在任务目录运行。
python verify-media.pyset -euo pipefail
: "${DELIVERY_DIR:?Set the directory printed by recovery}"
node verify-media.mjs "$DELIVERY_DIR"
jq '{runId, jobId, bucket, region, objectKey, bytes, md5, storageVersionId}' \
"$DELIVERY_DIR/verified/ready.json"将 ready.json 交给应用
预期输出 Delivery verified,以及包含运行及任务 ID、Bucket、区域、对象 Key、已验证大小和 MD5 的记录。这是脚本创建的应用记录。将它与 delivery.json 及原始任务一同归档;API 结果过期后仍可保留。
转录、索引或其他处理任务可将已验证的 localFile 交给应用的处理进程。若需供用户下载,继续第 6 步。
6. 让应用受控访问文件
在后端确认用户有权访问,再从自己的存储读取该用户对应的 ready 记录,随后创建短期 S3 下载 URL。使用已保存的目标,不接受浏览器提供的 Bucket 或对象 Key。
保存下载链接脚本
SDK 使用后端的 AWS 读取身份,不会发送 FlashData 查询。
S3 版本控制提供版本 ID 时,URL 指向已验证的对象版本。没有版本 ID 时,验证后不要覆盖输出 Key。读取指定版本的身份需要 s3:GetObjectVersion;无版本读取需要 s3:GetObject。
在 pipeline.py 旁创建 create-download-link.py,复制下方完整代码并保存。使用开始时安装的 Boto3。
# create-download-link.py — run in your backend after authorizing the user.
import json
import os
import sys
import boto3
from botocore.config import Config
from pipeline import delivery_directory, read_json
def main():
os.umask(0o077)
# In an app, load this user's verified ready record from your own store.
directory = delivery_directory() / 'verified'
ready = read_json(directory / 'ready.json')
if any(not isinstance(ready.get(key), str) or not ready[key]
for key in ['bucket', 'region', 'objectKey', 'verifiedAt']):
raise ValueError('Use the ready.json produced by the verification step.')
destination = directory / 'access.json'
if destination.exists():
raise FileExistsError('Keep the existing access.json or choose a new output filename.')
params = {'Bucket': ready['bucket'], 'Key': ready['objectKey']}
version = ready.get('storageVersionId')
if version and version != 'null':
params['VersionId'] = version
s3 = boto3.client('s3', region_name=ready['region'], config=Config(signature_version='s3v4'))
url = s3.generate_presigned_url('get_object', Params=params, ExpiresIn=300, HttpMethod='GET')
with destination.open('x', encoding='utf-8') as output:
json.dump({'url': url, 'expiresInSeconds': 300}, output, indent=2)
# Do not log the signed URL; return it only to the authorized user.
print('Access record saved:', destination)
if __name__ == '__main__':
sys.stdout.reconfigure(encoding='utf-8')
sys.stderr.reconfigure(encoding='utf-8')
try:
main()
except Exception as error:
print(str(error), file=sys.stderr)
sys.exit(1)在任务目录安装签名包,再将下方源码保存为同目录中的 create-download-link.mjs。
test -f package.json || npm init -y
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner// create-download-link.mjs — run in your backend with its AWS read identity.
import { readFile } from 'node:fs/promises';
import { GetObjectCommand, S3Client } from '@aws-sdk/client-s3';
import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
try {
const path = process.argv[2];
if (!path) throw new Error('Pass the verified ready.json file.');
const ready = JSON.parse(await readFile(path, 'utf8'));
if (['bucket', 'region', 'objectKey', 'verifiedAt'].some(
key => typeof ready[key] !== 'string' || !ready[key])) {
throw new Error('Use the ready.json produced by the verification step.');
}
// Load this record from your own store after authorizing the requesting user.
const version = ready.storageVersionId;
const command = new GetObjectCommand({
Bucket: ready.bucket, Key: ready.objectKey,
...(version && version !== 'null' ? { VersionId: version } : {}),
});
const client = new S3Client({ region: ready.region });
const url = await getSignedUrl(client, command, { expiresIn: 300 });
console.log(JSON.stringify({ url, expiresInSeconds: 300 }, null, 2));
} catch (error) {
console.error(error instanceof Error ? error.message : 'Could not create a download link.');
process.exitCode = 1;
}将链接返回给有权访问的用户
预期生成私有 access.json,包含 url 及 expiresInSeconds: 300。后端仅向有权访问的用户返回 URL。持有者最多可在五分钟内下载文件;签名凭据提前过期时,链接也会提前失效。
用户下次请求访问时,从保存的 ready 记录创建新链接。在数据库保存对象 Key 及版本,不把会过期的 URL 当作永久资产地址。本地示例在 access.json 已存在时会停止。
python create-download-link.py(
set -euo pipefail
umask 077
: "${DELIVERY_DIR:?Set the directory printed by recovery}"
# Keep the signed link in a private file; do not log it or paste it into chat.
set -C
node create-download-link.mjs "$DELIVERY_DIR/verified/ready.json" \
> "$DELIVERY_DIR/verified/access.json"
)从最后确认的阶段重启
| 情况 | 后续操作 |
|---|---|
| 任务已接纳 / 本地等待超时 | 使用同一 runId 再次 runs recover,写入新的尝试目录。 |
| S3 拒绝访问或读取中断 | 修复 AWS 读取身份或连接。恢复到新目录后再验证,保留之前的不完整文件。 |
| 字节数 / MD5 / 目标不匹配 | 停止。使用媒体前,核对交付记录、对象版本及预期目标。 |
| verified/ 已存在 | 读取其中的 ready.json,或检查未完成验证。脚本不会覆盖现有目录。 |
| 应用下载 URL 已过期 | 再次确认访问权限,为已保存的对象及版本签名。无需再次提交 FlashData 下载。 |
| run_id_input_conflict | 请求已变化。恢复原运行;仅在明确创建新输出时使用新 runId。 |
| 401 / 403 / 402 / 429 | 修复访问、权限或余额;遵循 Retry-After。重试提交前检查已有工作。 |