Rows Streaming

Process large result sets without loading everything into memory. SlintORM's .stream() method returns an AsyncIterator that yields batches of entities.

Basic streaming with .stream()

Call .stream(batchSize) on any query builder to get an async iterator. Each iteration yields an array of up to batchSize entities.

// .stream(batchSize?) — returns AsyncIterator<EntityWithUpdate<T>[]>

const iterator = User.query().stream(10);

for await (const batch of iterator) {
  // batch is User[] — at most 10 rows per iteration
  for (const user of batch) {
    console.log(user.id, user.name);
  }
}

// Default batch size is 100 if omitted
const allUsers = User.query().stream(); // batches of 100

Combining with filters and ordering

All query builder methods — where, orderBy, select, join — work before .stream().

// Combine with filters, ordering, and all builder methods

const iterator = User.query()
  .where('role', '=', 'admin')
  .where('active', '=', true)
  .orderBy('createdAt', 'ASC')
  .stream(50);

for await (const batch of iterator) {
  for (const user of batch) {
    await sendWelcomeBackEmail(user);
  }
}

Memory-efficient processing

.stream() uses LIMIT / OFFSET under the hood, fetching one batch at a time. This keeps memory usage constant regardless of table size.

// Memory-efficient processing of large datasets

// ❌ Loads EVERYTHING into memory — dangerous for large tables
const all = await User.getAll();
for (const user of all) {
  await processUser(user);
}

// ✅ Streams in batches — constant memory usage
for await (const batch of User.query().stream(100)) {
  for (const user of batch) {
    await processUser(user);
  }
}

// Useful for:
//   - Data exports (CSV, JSON)
//   - Migrations on large tables
//   - Back-filling computed columns
//   - Sending bulk notifications

Error handling

Wrap the for await...of loop in try/catch, or handle errors per-batch with Promise.allSettled for partial recovery.

// Error handling in streams

try {
  for await (const batch of User.query().stream(50)) {
    for (const user of batch) {
      await fragileOperation(user);
    }
  }
} catch (err) {
  console.error('Stream processing failed:', err);
  // The stream is aborted — partial results may have been processed
}

// Per-batch error handling with partial recovery
for await (const batch of User.query().stream(50)) {
  const results = await Promise.allSettled(
    batch.map(u => fragileOperation(u))
  );
  const failures = results.filter(r => r.status === 'rejected');
  if (failures.length > 0) {
    console.warn(`${failures.length} items failed in this batch`);
  }
}

Driver support

// Driver support

// SQLite    — ✅ supported (uses LIMIT / OFFSET)
// Postgres  — ✅ supported (uses LIMIT / OFFSET)
// MySQL     — ✅ supported (uses LIMIT / OFFSET)
// MongoDB   — ⚠️ supported (uses skip() / limit())
//             Batches are cursor-based, not snapshot-isolated.
//             Documents inserted mid-stream may be missed or duplicated.

// Note: Streams execute N+1 queries (one per batch). Each batch is
// a separate SQL query with increasing OFFSET. For true server-side
// cursors, use driver-native streaming (e.g. pg cursor, MySQL streaming).

PreviousRaw SQL & SqlExpr NextPlugins