AI gateway

The AI gateway lets your app call language and embedding models through OrbitNest. You save your own provider key once in Studio. Your app code never sees it: requests go to OrbitNest, which adds the key on the server, enforces your spend caps and records every call with its token count and cost.

Before you start

• Bring your own key. Each project uses its own provider account. OrbitNest does not add a markup to these calls.
• Who can call it. A request needs a signed-in end user (the anon key plus the user's access token) or the service-role key. The anon key alone is refused with 401, because it is public and anyone holding it could spend your tokens.
• Supported providers: Google Gemini, OpenAI, Anthropic, and any OpenAI-compatible API (Groq, Together, OpenRouter, a vLLM server). Anthropic has no embeddings API, so use Gemini or OpenAI for embed.

Add a provider key

1. Open the project and go to Settings → AI. Saving keys and caps needs the workspace admin role.
2. Paste the key for a provider. Optionally set a default model: requests that name no model use it.
3. For OpenAI-compatible, also set the base URL, for example https://api.groq.com/openai/v1. It must be a public https address.
4. Click Save, then Test. The test sends one short request with the saved key.

The key is encrypted at rest. Studio only ever shows its last four characters.

Chat

import { createClient } from '@orbitneststudio/js';

const client = createClient({ apiKey: process.env.NEXT_PUBLIC_ORBITNEST_ANON_KEY! });
await client.auth.signIn({ email, password });

const { data, error } = await client.ai.chat({
  model: 'gemini-2.5-flash', // optional: falls back to the project default
  messages: [
    { role: 'system', content: 'You are a concise support assistant.' },
    { role: 'user', content: 'How do I reset my password?' },
  ],
  maxTokens: 512,
});

if (error) console.error(error.code, error.message);
else console.log(data.text, data.tokensIn, data.tokensOut, data.costUsd);

The Node.js SDK (@orbitneststudio/node) has the same client.ai methods. On a server, create the client with the service-role key. Those calls have no end user, so per-user caps do not apply.

model decides the provider: gemini-* goes to Gemini, gpt-* and o* to OpenAI, claude-* to Anthropic. For any other model name, pass provider: 'openai_compatible'.

Streaming

chatStream yields delta events as text arrives, then one done event with the model used, token counts and cost. Failures arrive as an error event instead of a thrown exception, so one loop handles everything.

let reply = '';
for await (const ev of client.ai.chatStream({ messages })) {
  if (ev.type === 'delta') reply += ev.text;
  if (ev.type === 'done') console.log(`${ev.model}: ${ev.tokensOut} tokens, $${ev.costUsd}`);
  if (ev.type === 'error') console.error(ev.code, ev.message);
}

Pass { signal } as the second argument to cancel a stream, for example when the user leaves the screen. Leaving the for await loop early also closes the request, so the server stops generating and you are not billed for the rest of the reply.

Tool calling

Describe functions your app can run; the model answers with toolCalls instead of text when it wants one. Run each call yourself, then send the results back and chat again.

const tools = [{
  name: 'get_order',
  description: 'Look up an order by its number',
  parameters: { type: 'object', properties: { number: { type: 'string' } }, required: ['number'] },
}];
const messages = [{ role: 'user', content: 'Where is order 1042?' }];

const first = await client.ai.chat({ messages, tools });
if (first.data?.toolCalls.length) {
  messages.push({ role: 'assistant', content: null, toolCalls: first.data.toolCalls });
  for (const call of first.data.toolCalls) {
    const order = await getOrder(call.arguments.number as string); // your code, under your rules
    messages.push({ role: 'tool', toolCallId: call.id, content: JSON.stringify(order) });
  }
  const answer = await client.ai.chat({ messages, tools });
}

• toolChoice is 'auto' (default), 'none', 'required', or { name } to force one tool.
• Send the assistant's toolCalls back unchanged: some Gemini models attach a signature they need to see again.
• A streamed reply delivers each call as a tool_call event before done.
• The gateway never runs tools. Your code decides what a call may do, so check the user's rights before acting on one.

Images

User messages can mix text and images:

await client.ai.chat({
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'What is in this photo?' },
      { type: 'image', url: 'data:image/jpeg;base64,/9j/4AAQ…' },
    ],
  }],
});

Send images as data: URLs (png, jpeg, webp or gif). OpenAI and Anthropic also accept an https URL, which their servers fetch; Gemini needs the data URL. A request body can be at most 2 MB, so resize photos before sending them.

Embeddings

Every vector has 768 dimensions, the size used for pgvector columns across OrbitNest. Send one string or up to 100.

const { data } = await client.ai.embed(['first document', 'second document']);
// data.embeddings: number[][], data.model: the model that produced them

To list the models your saved keys can reach, call client.ai.models().

Spend caps and limits

In Settings → AI → Monthly spend caps you can set:

• a project cap: the total spend of all app calls in a calendar month;
• a per-user cap: what one end user can spend in a month.

Once a cap is reached, calls fail with 402 AI_BUDGET_EXCEEDED until the next month. Caps are checked before each call, so the call that crosses the line still completes. Costs come from list prices that OrbitNest keeps, so they are close to your provider's bill but not exact. A model OrbitNest has no price for (a new release, or any OpenAI-compatible model) is charged at the highest listed price of that provider (of any provider, for OpenAI-compatible), so caps fire early rather than never. Set caps with that in mind if you use a cheap compatible model. A stream that stops part-way, because the user left or the provider failed, is charged an estimate of what was generated.

There is also a rate limit per end user (per project for service-role calls) that depends on your plan: 20 requests a minute on Free, 60 on Starter and 300 on Pro. Past it, calls return 429 AI_RATE_LIMITED with retryAfterSec.

The Usage panel on the same tab shows calls, errors, tokens and cost for the last 30 days: cost per day, by model, and your top end users. It counts calls made with your own keys.

HTTP API

The SDKs wrap these endpoints. Send the anon key in apikey and the user's access token in Authorization: Bearer, or the service-role key in both.

On the wire, messages use tool_calls (assistant) and tool_call_id (tool), and content is a string or a list of { type: "text", text } / { type: "image", url } parts. The reply has text, toolCalls, model, provider, tokensIn, tokensOut, costUsd and finishReason.

With stream: true, the response is text/event-stream. Each event is one JSON object on a data: line:

data: {"type":"delta","text":"Open Settings"}
data: {"type":"delta","text":" and choose Reset password."}
data: {"type":"tool_call","id":"call_0","name":"get_order","arguments":{"number":"1042"}}
data: {"type":"done","model":"gemini-2.5-flash","tokensIn":41,"tokensOut":12,"costUsd":0.000042,"finishReason":"STOP"}

Errors that happen before the first event (no key, cap reached, rate limit, a rejected key) come back as a normal JSON error response, not as a stream.

Errors

Flutter

The Flutter SDK has the same calls on client.ai. chatStream returns a Stream<AiStreamEvent>; cancelling the subscription stops the reply at the server.

final history = <AiMessage>[AiMessage.user('Where is order 1042?')];
final r = await client.ai.chat(messages: history, tools: tools);
history.add(AiMessage.fromResult(r));
for (final call in r.toolCalls) {
  history.add(AiMessage.toolResult(call, jsonEncode(await getOrder(call.arguments['number']))));
}

await for (final ev in client.ai.chatStream(messages: history, tools: tools)) {
  if (ev is AiTextDelta) setState(() => reply += ev.text);
}

Images go in parts: AiMessage(role: 'user', parts: [AiContentPart.text('What is this?'), AiContentPart.image(dataUrl)]). chat and embed throw AiException with code, statusCode and body.

Current limits

• Structured output (a JSON schema for the reply) is not available to apps yet.
• A streamed reply can run for up to 120 seconds.
• Request bodies are limited to 2 MB, images included.