Custom AI endpoint

Your server receives generation requests and returns the result for the requested operation. Configure the connection in AI providers.

Address securityCopy Anchor Link

Use HTTPS for every server and remote media address you pass to the editor. The only HTTP exceptions are http://localhost, http://127.0.0.1 and http://[::1], for development on your own computer. Relative paths and storage keys must resolve to an HTTPS address outside local development.

Use HTTPS for endpointUrl, captions endpoints, job status and cancellation addresses, and generated media returned by your server. The former proxyUrl name follows the same rule. A requestFn owns its network calls and must also use secure addresses.

Hosted requests that carry your license key, including hosted captions, are refused over HTTP to another computer. For other addresses, HTTPS is the integration rule; the SDK does not enforce it on every option.

Request formatCopy Anchor Link

endpointUrl receives JSON by POST, except for caption uploads. A requestFn receives the same request object with an AbortSignal. Neither transport receives your SDK license key.

A request without a models list uses its existing provider request format. The built-in picture request is:

{
  "type": "image", "prompt": "A red fox in snow",
  "input": { "operation": "image", "prompt": "A red fox in snow" },
  "options": { "action": "create", "provider": "…", "count": 1 }
}

provider is shown as … here. Its real value comes from the request builder. Your server owns provider selection and API mapping. Classic custom providers can build their own options.

OperationsCopy Anchor Link

Read input.operation to identify the requested work.

A source is { url } or { dataUrl }, never both. Picture edits from the editor send the picture in source.dataUrl. Read that field; do not assume source.url exists. The optional mask has white editable areas and black preserved areas.

Model listCopy Anchor Link

Classic video and voice controls can request a server model list. The models configuration does not fetch a catalog from your server; pass its list through ai.<operation>.models.

The classic list request is a sibling of your endpoint: https://api.example.com/ai/proxy uses https://api.example.com/ai/models?capability=video or capability=audio. A model-list request uses configured host headers with purpose models. Use an absolute endpoint address for this classic discovery route.

Requests with modelsCopy Anchor Link

A configured models list sends a checked selection in input.generation without options.

{
  "type": "image", "prompt": "A red fox in snow",
  "input": {
    "operation": "image", "prompt": "A red fox in snow",
    "generation": {
      "contract": 1, "catalogId": "fast-images",
      "modelId": "vendor-fast-image-1", "revision": "2026-10-11",
      "settings": { "size": "1024x1024" }
    }
  }
}

Unset and local-only fields are absent from settings. Bound settings can also appear on their corresponding input fields. The models route’s outer rewrite prompt is input.instruction; read input.text separately. A classic rewrite builder combines both into its outer prompt.

Route by modelCopy Anchor Link

Validate the model identity, revision and settings on your server before calling a provider. Client checks do not replace server validation, authorization, billing limits or idempotency storage.

TypesCopy Anchor Link

Derive the settings and request types from your configured model.

import { defineGenerationModel, type AIGenerationRequestOf, type AIGenerationSettingsOfModel } from '@layermetry/media-editor';

const model = defineGenerationModel({
  label: 'Images', operation: 'image',
  fields: [{ name: 'quality', label: 'Quality', options: ['low', 'high'] }],
});
export type Settings = AIGenerationSettingsOfModel<typeof model>;
export type Request = AIGenerationRequestOf<typeof model>;
export function readQuality(request: Request): Settings['quality'] {
  return request.input.generation.settings.quality;
}

Brand in requestsCopy Anchor Link

When the user leaves Use brand on, the request carries input.brand. It holds the id and name of the kit, the parts that were used, and addedToPrompt, the exact words that the editor added to the prompt, including the blank line before them. A server that words its own prompts can remove those words by an exact match, or read the named fields instead.

{
  "input": {
    "operation": "text",
    "prompt": "A headline\n\nWrite in a warm voice.",
    "brand": { "kitId": "northwind", "name": "Northwind Studio", "addedToPrompt": "\n\nWrite in a warm voice.", "used": ["voice"], "voice": "warm" }
  }
}

input.brand is missing when no brand is active, when ai.brand is false. A part that the person turned off is missing from the prompt and from input.brand. ai.brand.addToPrompt: false sends input.brand with addedToPrompt set to ''.

The name brand is reserved. A model field cannot be named brand, and a brand key in the model settings is removed before the editor adds its own. See Generate with brand for every field.

Response formatCopy Anchor Link

Return HTTP 200 with the operation result, or map your custom response in the model’s mapResponse.

items must contain at least one object with the fields required by the operation. Both routes refuse another operation’s result, an empty list or malformed SVG path data with invalid-output. An optional top-level receipt records the model and job details. Sticker results reject known JPEG addresses because JPEG cannot preserve transparency.

Both the built-in providers and the models route read { items: [{ text: "Suggested words" }] }. A custom parseResponse or mapResponse must return the same result format.

Serve media addresses with a valid content type. Your storage's CORS rules must allow the origin of the page that runs the editor, including its scheme and port: https://localhost:3000 and http://localhost:3000 are different origins. The editor loads generated pictures with crossOrigin="anonymous", so that the canvas can still be exported. If the origin is not allowed, the browser refuses the picture, and the editor cannot show it or place it. Generated media is also downloaded for storage and export.

A complete endpointCopy Anchor Link

This Node.js fixture accepts picture requests with and without a models selection.

import { createServer } from 'node:http';

const PORT = 8787;
const ALLOWED_ORIGIN = 'http://localhost:5173';
const IMAGE_MODELS = {
  'fast-images': { modelId: 'vendor-fast-image-1', revision: '2026-10-11', fields: ['size', 'quality'] },
  'detailed-images': { modelId: 'vendor-detailed-image-2', revision: '2026-10-11', fields: [] },
};
const DEFAULT_IMAGE_MODEL = 'fast-images';

// Replace this fixture with your server-side provider call.
async function makePictures(modelId, prompt, settings) {
  return ['https://cdn.example.com/pictures/sample.png'];
}

function send(response, status, body) {
  response.writeHead(status, {
    'Content-Type': 'application/json',
    'Access-Control-Allow-Origin': ALLOWED_ORIGIN,
    Vary: 'Origin',
  });
  response.end(JSON.stringify(body));
}
function refuse(response, status, code, message) {
  send(response, status, { ok: false, error: { code, message } });
}
async function readJson(request) {
  let length = 0;
  const chunks = [];
  for await (const chunk of request) {
    length += chunk.length;
    if (length > 2 * 1024 * 1024) throw new Error('Body exceeds 2 MiB');
    chunks.push(chunk);
  }
  return JSON.parse(Buffer.concat(chunks).toString('utf8'));
}
async function answer(request, response) {
  let body;
  try { body = await readJson(request); }
  catch { return refuse(response, 400, 'invalid-input', 'Expected JSON within 2 MiB.'); }
  const input = body?.input;
  if (input?.operation !== 'image' || typeof input.prompt !== 'string' || !input.prompt.trim()) {
    return refuse(response, 400, 'invalid-input', 'Expected an image prompt.');
  }
  const generation = input.generation;
  if (generation !== undefined && (generation === null || generation.contract !== 1)) {
    return refuse(response, 400, 'invalid-input', 'Expected generation contract 1.');
  }
  const name = generation?.catalogId ?? DEFAULT_IMAGE_MODEL;
  if (!Object.hasOwn(IMAGE_MODELS, name)) {
    return refuse(response, 400, 'unsupported-model', 'Image model not offered.');
  }
  const model = IMAGE_MODELS[name];
  if (generation && (generation.revision !== model.revision ||
    (generation.modelId !== undefined && generation.modelId !== model.modelId))) {
    return refuse(response, 400, 'invalid-input', 'Model settings have changed.');
  }
  const settings = generation?.settings ?? {};
  if (typeof settings !== 'object' || settings === null || Array.isArray(settings) ||
    Object.keys(settings).some((key) => !model.fields.includes(key)) ||
    (settings.size !== undefined && !['1024x1024', '1536x1024'].includes(settings.size)) ||
    (settings.quality !== undefined && !['low', 'medium', 'high'].includes(settings.quality))) {
    return refuse(response, 400, 'invalid-input', 'Invalid image settings.');
  }
  try {
    const imageUrls = await makePictures(model.modelId, input.prompt, settings);
    send(response, 200, { items: imageUrls.map((url) => ({ url })) });
  } catch {
    refuse(response, 502, 'transport-failed', 'Image service unavailable.');
  }
}
createServer(async (request, response) => {
  const path = new URL(request.url, `http://localhost:${PORT}`).pathname;
  if (request.method === 'OPTIONS' && path === '/ai') {
    response.writeHead(204, {
      'Access-Control-Allow-Origin': ALLOWED_ORIGIN,
      'Access-Control-Allow-Methods': 'POST',
      'Access-Control-Allow-Headers': 'Content-Type, Authorization',
      Vary: 'Origin',
    });
    return response.end();
  }
  if (request.method === 'POST' && path === '/ai') return answer(request, response);
  refuse(response, 404, 'invalid-input', 'Not found.');
}).listen(PORT, '127.0.0.1', () => console.log(`AI endpoint: http://localhost:${PORT}/ai`));

Save the file as server.mjs, then start it with node server.mjs. It listens at http://localhost:8787/ai. The sample model’s revision is 2026-10-11.

This fixture returns a placeholder address. Replace makePictures with your provider call and a readable media address. Configure your page’s origin in ALLOWED_ORIGIN. Add authentication and your billing policy before production use.

Caption uploadsCopy Anchor Link

Caption endpoints receive multipart/form-data.

Set ai.captions.transport to your upload endpoint or callback. Captions do not inherit the main transport. The Node picture fixture above handles JSON only. A captions callback receives the original audio Blob; create FormData inside that callback when sending it to your server.

Return { items: [{ segments: [{ text: "Hello", start: 0, end: 1.2 }] }] } for timed text. Segment start and end, plus each word’s startTime and endTime, use seconds. srt contains subtitle file text. text has approximate timing. An item can also contain srtPath, the address of an SRT file.

Long-running jobsCopy Anchor Link

A long request can return HTTP 202 with a job handle.

{ "ok": true, "job": {
  "id": "job-1", "statusUrl": "https://api.example.com/jobs/job-1",
  "cancelUrl": "https://api.example.com/jobs/job-1/cancel", "pollMs": 1500
} }

Custom transports require whole job addresses. Use HTTPS, except for the loopback HTTP addresses allowed for local development above. The editor polls statusUrl with GET. It sends POST to cancelUrl when stopping, if supplied. Host headers do not travel with either request in this release. This also applies when a callback returns a job handle.

{ "ok": true, "job": {
  "jobId": "job-1", "state": "ok", "percent": 100, "stage": "done",
  "receipt": { "asset": { "url": "https://cdn.example.com/result.mp4" } }
} }

Polling defaults to 1,500 ms and is bounded to 250–60,000 ms. It ends after 12 minutes or three consecutive failed status requests. Your submission timeout does not extend this limit. Cancellation does not promise a provider refund.

Music and sound effectsCopy Anchor Link

Music and sound effects return audio addresses with optional measured lengths.

{ "items": [{ "url": "https://cdn.example.com/track.mp3", "durationSeconds": 30.2 }] }

Put the duration in seconds in each item’s durationSeconds. If a duration is omitted, the editor reads it from the file. The returned address must permit cross-origin reads.

Model list for musicCopy Anchor Link

The classic server list is the endpoint’s sibling models route with capability=music or capability=soundEffect.

{ "ok": true, "capability": "music", "default": "music-1", "models": [
  { "id": "music-1", "label": "Music", "limits": { "minSeconds": 5, "maxSeconds": 60 } }
] }

An empty list with default: null hides that operation. With requestFn or host-provided models, no server list is required. Host duration bounds take precedence over server list bounds.

Refusals from your serverCopy Anchor Link

Return a non-2xx status with the error shape below when generation is refused.

TypesCopy Anchor Link

AISoundAnswer and AIModelListAnswer describe classic sound responses. Models use EditorResult and the typed generation selection.

Error codesCopy Anchor Link

Return a non-2xx response with a known SDK code to identify a failure.

{ "ok": false, "error": {
  "code": "rate-limited", "message": "Too many requests. Try again later.",
  "requestId": "request-123"
} }

An unknown provider code stays in error.providerError.code. The SDK maps HTTP 401 and 403 to auth-refused, HTTP 429 to rate-limited, and other unknown failures to transport-failed. See AI errors for every code and retry guidance.

providerError contains optional code, message, status and requestId. Request IDs can come from x-request-id, x-correlation-id or the body. Keep vendor details here; compare error.code for application logic.

Configure your own endpointUrl or requestFn. hosted: true is coming soon and available only with an enterprise licence from Layermetry.