Batch & Columnar Encoding

Batch encoding is where Twilic's advantage over MessagePack is largest. This guide explains row batches, columnar batches, batch sizing, and when each mode applies.

Row batch vs columnar batch

Mode	Wire tag	Best for
Row batch	row_batch	Nested objects, moderate field count, general API lists
Columnar batch	col_batch	Tabular data, many rows, numeric/time-series columns

Both share shape and string interning. Columnar mode additionally compresses each field as an independent column.

Row batch

import { encodeBatch } from "@twilic/core/advanced";

const users = page.map((u) => ({
  id: BigInt(u.id),
  name: u.name,
  email: u.email,
  role: u.role,
  active: u.active,
}));

const bytes = encodeBatch(users);

What happens on the wire:

1. Shape definition: field names registered once
2. Each row encoded with shape reference (no repeated keys)
3. Repeated strings ("admin", "us-east") interned across rows

Columnar batch

Ideal for telemetry, logs, and analytics events.

const events = [
  { timestamp: 1700000000100n, service: "api", level: "info", duration_ms: 12 },
  { timestamp: 1700000000200n, service: "api", level: "info", duration_ms: 9 },
  // ... up to 256+ events
];

const bytes = encodeBatch(events);
// Encoder selects col_batch when column regularity is high

Per-column codec selection:

Column type	Typical codec	Example
Monotonic timestamp	DELTA_BITPACK	Log event times
Low-cardinality string	Dictionary	service, level
Repeated value runs	RLE	Log levels (info, info, info, …)
Numeric cluster	FOR_BITPACK, SIMPLE8B	duration_ms, sensor readings
Homogeneous float array	XOR_FLOAT	Temperature samples

Size comparison (256 events, 6 fields)

Format	Size
JSON	~98 KB
MessagePack	~65 KB
Twilic row batch	~25–35 KB
Twilic col_batch	~8–12 KB

Batch size tuning

Priority	Batch size	Trade-off
Low latency	16–64 records	Smaller compression win
Balanced	128–256 records	Good default for APIs
Max compression	512–1024 records	Higher flush latency
Time series	256–1024+	Columnar codecs need volume

Rule of thumb: batch until compression gain plateaus on your payload — then measure P99 flush latency.

HTTP list endpoints

import { encodeBatch } from "@twilic/core/advanced";
import { twilicResponse } from "@twilic/hono";

app.get("/users", async (c) => {
  const users = await db.getUsers({ page, limit: 50 });
  const bytes = encodeBatch(users);
  return new Response(bytes, {
    headers: { "Content-Type": "application/vnd.twilic" },
  });
});

Or use a custom codec with createTwilicHono — see Hono integration.

Pipeline batching

[Agent: batch 64] → [Forwarder: batch 256] → [Kafka] → [Consumer]

Re-batch at each tier for cumulative compression. Do not re-encode individual events mid-pipeline unless necessary.

Anti-patterns

Pattern	Problem
Batch size = 1	No interning benefit
Mixed shapes in one batch	Shape table churn, poor compression
Batch on HTTP error responses	Overhead exceeds benefit for tiny payloads
Columnar on deeply nested objects	Row batch is better

Language examples

Rust

let bytes = twilic::encode_batch(&records)?;

Python

payload = twilic.encode_batch(events)

Go

bytes, err := twilic.EncodeBatch(records)

Related

• Cookbook — Telemetry
• Telemetry at Scale article
• Spec — Format Guide