Batch-sets

A batch-set is Enscrive’s first-class record of a single ingest operation that was split into one or more provider batch jobs (OpenAI, Nebius, Voyage). It owns the full story of a multi-hour async embed — how many sub-batches it split into, which provider batch ids it submitted, how many documents succeeded or failed, which attempts have been made, and whether the results ever promoted to the live corpus.

If you are ingesting thousands of documents at once, every such ingest produces a batch-set. The state machine and the atomic-promote guarantees below are what makes “ingest a million documents overnight” safe — a failure two hours into the embed cannot leak partial results into your live search surface.

Why a first-class record

Previous iterations tracked batch jobs as rows on import_jobs only. That made three things hard:

1. Recovery semantics — when a provider batch failed two hours in, there was no single row describing what-was-submitted vs what-has-returned vs what-needs-retrying. Operators had to reconstruct state from logs.
2. Cross-provider correlation — a single logical ingest can split into multiple provider batches (OpenAI caps at 50k requests per batch; 100k docs = two batches). No single row owned the set.
3. All-or-nothing promote — with only per-job tracking, there was no point in time where you could say “every sub-batch succeeded, now we promote.” The new model moves the promote to the parent-set level so partial results cannot appear in your live corpus.

The state machine

ingesting ──► embedding ──► promoting ──► committed      (terminal success)
                │
                └─► failed_recoverable ──► abandoned       (terminal failure)

The critical property: no state between ingesting and committed makes embeddings visible in the live corpus. The promote is parent-level and atomic — either every sub-batch made it, or nothing did. If sub-batch 6 of 7 fails, the staging collections for sub-batches 1–5 are dropped and the live corpus is unchanged. This is enforced structurally inside embed-svc, not by convention.

What a batch-set row contains

{
  "id": "b7ca9a0…",
  "corpus_id": "787cfadf…",
  "tenant_id": "…",
  "environment_id": "…",
  "state": "committed",
  "document_count": 63926,
  "chunk_count": 63926,
  "total_tokens": 12400000,
  "sub_batch_count": 7,
  "attempts": [
    {
      "started_at": "2026-04-19T15:49:00Z",
      "ended_at": "2026-04-19T16:23:12Z",
      "outcome": "success",
      "failed_sub_batches": []
    }
  ],
  "committed_revision_id": "…",
  "primary_provider": "openai",
  "primary_model": "text-embedding-3-small",
  "provider_batch_ids": [
    "batch_69e572389…",
    "batch_69e572438…",
    …
  ],
  "backfill_incomplete": false,
  "created_at": "2026-04-19T15:49:00Z",
  "completed_at": "2026-04-19T16:23:12Z",
  "ttl_expires_at": null
}

Key fields:

Get one batch-set

GET /v1/batch-sets/{id}

Returns the full BatchSet row above. Tenant-scoped — rows belonging to another tenant return 404 Not Found (no existence leak).

List batch-sets for a corpus

GET /v1/corpora/{id}/batch-sets?limit=50&offset=0

Newest-first, paginated. Default limit = 50, maximum 200. Only batch-sets belonging to the caller’s tenant are returned.

enscrive batch-sets list --corpus enscrive-platform-docs
enscrive batch-sets get 0f9a01c5-…

Retry and abandon

Retry and abandon live under the Jobs endpoint group, not here — a batch-set is linked 1:1 to an import_jobs row via its batch_set_id FK, and operator-facing verbs attach to that job.

POST /v1/jobs/{id}/retry     # re-submit the failed sub-batches of the batch-set
POST /v1/jobs/{id}/abandon   # explicitly mark abandoned; drops staging collections immediately

A retry re-runs only the sub-batches whose outcome is failed — embedded sub-batches are not re-embedded. The retry appends a new entry to the attempts log so you keep the full history. Retries are bounded by provider error class: only transient and certain provider_5xx classes retry automatically; permanent and validation classes must be abandoned and re-started with corrected input.

An abandon moves the batch-set to abandoned, drops any remaining staging collections so their Qdrant capacity is released, and records the operator-initiated abandonment. Abandon is the right verb when you know the underlying ingest was wrong (bad metadata, wrong voice, wrong corpus) and retrying would just succeed at embedding bad data.

See Jobs — retry and abandon for the full semantics.

Why you might want to inspect a batch-set

• Operator dashboards — the batch-set row is the canonical “what is happening right now with my ingest?” summary. Provider-side dashboards show per-sub-batch progress, but only the batch-set ties them back to your corpus and your document set.
• Post-mortem a failed commit — attempts gives you the full failure timeline; failed_sub_batches gives you the specific error class and message per sub-batch; provider_batch_ids lets you cross-reference the provider dashboard for their side of the story.
• Cost accounting — chunk_count and total_tokens on the row are the authoritative billing surface for that ingest. Combined with the model’s price, this is the exact dollar figure the batch cost you.

Parallelism and throughput

Sub-batches within one batch-set dispatch in parallel under a concurrency governor (J018_MAX_CONCURRENT_SUB_BATCHES = 10). On a 7-sub-batch ingest you’ll typically see all seven provider batches submitted inside a few seconds of each other, running concurrently.

Promote, by contrast, is sequential and parent-level. The server waits until every sub-batch returns success, then walks the sub-batches in order applying each to the live corpus. A failure at any stage before the final promote leaves the live corpus unchanged.

What this endpoint group does not cover

• Retry / abandon verbs — Jobs.
• The ingest request itself — Ingest.
• Per-provider concerns (OpenAI rate limits, Nebius model-access pre-flight) — provider-specific behavior is surfaced through the Jobs row’s error_class and error_message, and summarized on Rate limits.