Workers AI

Workers AI lets your app run open models, such as Llama and Mistral for text and BGE for embeddings, on GPUs near your users. There is nothing to create and no API key: turn AI on for the app, then name a model on each call. Use it for summaries, classification, chat, and creating embeddings for vector search.

Turn on AI#

1. In your app, open Overview.
2. Choose Add resource, then AI.
3. Deploy the app.

AI is added as soon as you choose it; there is no form. It starts working after the next deploy. An SSR or hybrid app gets it as env.AI. A container app calls a private host shown on the AI card.

Call a model#

SSR and hybrid apps (Workers)#

const { response } = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', {
  prompt: 'Say hello',
});

const { data } = await env.AI.run('@cf/baai/bge-base-en-v1.5', {
  text: ['a sentence to embed'],
});

Container apps#

POST to http://<host>/run with {"model", "input"}. The reply is the model's JSON.

$answer = Http::post('http://dply.<app>.ai.internal/run', [
    'model' => '@cf/meta/llama-3.1-8b-instruct',
    'input' => ['prompt' => 'Say hello'],
])->json('response');

const res = await fetch('http://dply.<app>.ai.internal/run', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({
    model: '@cf/meta/llama-3.1-8b-instruct',
    input: { prompt: 'Say hello' },
  }),
});
const { response } = await res.json();

Use the exact host shown at the top of the AI card and on its Connect tab.

Inputs and outputs#

Any Workers AI model id works. The Try it tab lists a few examples and runs one prompt from the dashboard:

Pricing#

Workers AI is billed in neurons, Cloudflare's unit for GPU work, at the invoice line AI, browser rendering and vector search.

Each model has its own rate in neurons per million input and output tokens (the Workers AI pricing table). dply counts the tokens the model reports for each call and applies that model's rate. When a model reports no token counts, dply estimates them from the length of the input and the answer (about four characters per token). A model that is not in the table is billed at a high text-model rate with a minimum of 100 neurons per call.

Monthly limit#

AI, browser rendering and vector search share one monthly limit per organization, $25 unless an owner changes it on the Billing page. It counts every app in the organization and the demos in the dashboard. Owners are emailed at 80% and 100%. At the limit, calls are refused until the next billing period or until an owner raises it:

• A Worker app's call rejects with an error whose message says the limit is reached (error.status is 429).
• A container app gets HTTP 429 with {"error": "..."}.

Setting the limit to 0 removes it, up to a platform maximum. dply can also turn one of these services off for everyone for a while; calls then fail with status 503 and say so. The AI, browser and vector search cards show this period's usage and the limit.

If the organization leaves its paid plan, AI is left out of the app's next deploy and calls to it fail.

Turn off AI#

Open the AI card and choose Remove from this app, then Remove. AI is removed on the next deploy, and code that calls it gets errors. Nothing is stored, so nothing is lost.

Related#

• Vector search
• Browser rendering
• Plans & pricing