Collections

A collection is a searchable set of documents. You add text, a URL or a file from project storage; NestCore extracts the text, splits it into chunks, embeds them with your project's AI provider key, and keeps them in your project database. Search combines vector similarity with keyword matching and returns the best passages with their source, ready to show or to pass to a model. Threads can use collections as cited sources.

Before you start

• Add a Gemini, OpenAI or OpenAI-compatible key in Project Settings → AI. Embedding uses your own key, like every AI gateway call.
• Creating collections and adding shared documents needs the service-role key (or Studio). Signed-in users can search, and can add private documents to collections that allow it.
• The data lives in your project database, in the ai_collections, ai_documents and ai_chunks tables, created on first use with row-level security.

Quick start

On your server, create a collection and add documents:

import { createClient } from '@nestcore/node';

const admin = createClient({ apiKey: process.env.NESTCORE_SERVICE_KEY! });

await admin.collections.create({ name: 'handbook' });

const docs = admin.collections.from('handbook');
await docs.ingest({ text: '# Refunds\n\nRefunds are paid within 14 days to the original card.' }, { title: 'Policies', metadata: { team: 'support' } });
await docs.ingest({ url: 'https://example.com/guide.pdf' });
await docs.ingest({ storage: { bucket: 'docs', path: 'manuals/setup.pdf' } }, { wait: true });

In your app, search as the signed-in user:

const { data, error } = await client.collections.from('handbook').search('How do refunds work?', { k: 5 });

for (const r of data?.results ?? []) {
  console.log(r.document.title, r.highlight, r.score);
}

client.collections is in the Node.js SDK from version 0.14.0 and the web SDK (@nestcore/js) from version 0.8.0.

Flutter:

final results = await nestcore.collections.from('handbook').search('How do refunds work?', k: 5);

Create a collection

A second collection with the same name returns 409. collections.update(name, patch) changes userIngest, metadata, model and provider. collections.delete(name) deletes the collection with all its documents and chunks.

Add documents

Give exactly one source:

Also optional: title (up to 300 characters), metadata (JSON you can filter on later) and, with the service key, ownerUserId to make the document private to one user.

ingest returns the document at once with status: "queued". A background worker extracts and embeds it, and the status moves to processing, then ready or failed (with error). Pass wait: true, or { timeoutMs, intervalMs }, to poll until it finishes (default 120 seconds, then a TIMEOUT error). waitFor(documentId) polls an existing document.

• Text is split on headings and blank lines first, then sentences, then words, so a chunk is a readable passage.
• Without a title, a web page's <title> or a storage file's name is used.
• For URL and storage sources only the chunks are kept, not the downloaded file.
• Images and Office files can't be read; they fail with "Cannot read text from a … file". A document with no text, or with more than 10,000 chunks, also fails.
• A temporary failure is retried, up to 3 attempts in total.

Manage documents with listDocuments({ status, limit, offset }), getDocument(id), remove(id) (deletes its chunks too) and stats() (chunks, chunks with the current model, documents and failed documents).

Search

const { data } = await client.collections.from('handbook').search('refund to original card', {
  k: 8,
  hybrid: true,
  filter: { team: 'support' },
  minSimilarity: 0.3,
});

Each result has:

• score: the fused rank score; results are sorted by it (by similarity when hybrid is false).
• similarity: cosine similarity, or null for a passage found only by keywords.
• keywordMatch: whether the keyword search found it.
• highlight: the best-matching words, wrapped in **.
• chunk: { id, ordinal, content, metadata }.
• document: { id, title, sourceType, sourceRef, metadata }.

Only ready documents are searched. The query is embedded on every search: it is an ordinary AI gateway call that counts toward your per-minute limit and spend caps. A collection that has no model yet (no document processed) returns no results.

Private documents (userIngest)

With userIngest: true, a signed-in user can add documents to the collection. Such a document belongs to that user:

• Only its owner finds it in search, and only its owner can delete it. Other users see the shared documents (those without an owner) and their own.
• A storage source must be one of the user's own files.
• The service key can add a private document for a user with ownerUserId.

await client.collections.from('notes').ingest({ text: 'My private note about the Q3 plan.' }, { wait: true });

Change the embedding model

collections.update('handbook', { model: 'text-embedding-3-small' }) switches the collection to a new model and re-embeds every chunk in the background. Until that finishes, search only finds chunks already re-embedded. stats() shows the progress as current_model_chunks of chunks.

Access rules

The anon key alone is refused with 401: every search spends tokens on your key.

Limits and quotas

Embedding documents counts toward the workspace's monthly embedding allowance, shared with automatic embeddings: 50,000 tokens on Free, 1,000,000 on Starter and 10,000,000 on Pro. It is checked before each batch of 64 chunks; when it is used up, the document fails and its error says the monthly embedding allowance is reached. Search queries don't count toward it. See Plans & billing.

In Studio

Project Settings → AI → Collections lists collections with their model and document counts. You can create a collection, add a document (text, URL or storage file), watch document status and errors, run a Test search, change whether users can add documents, and delete documents or the collection. Changes need the workspace admin role.

From a terminal, nestcore ai collections list lists collections; the MCP tool is nestcore_list_collections.

HTTP API

App endpoints, under /api/project/{slug}/ai/v1. Send the anon key in apikey and the user's access token in Authorization: Bearer, or the service-role key.

Rows come back as stored, in snake_case (embed_model, chunk_count, owner_user_id). The search response is { results, model, usage: { tokensIn, costUsd } }.

Studio uses the same operations under /api/projects/:projectId/ai/collections with a Studio access token.

Errors

Document problems found while processing (an unreadable file, a refused URL, the quota) don't fail the request; they set the document's status to failed and its error. Search can also fail with the AI gateway errors.