xFormat: Universal Binary Share Envelope

Executive Summary

xFormat is the universal binary envelope for every XorIDA threshold share across the PRIVATE.ME platform. One format, 140 ACIs, zero custom routing logic.

Every PRIVATE.ME product produces threshold shares. Xail splits emails into K-of-N shares. Xecret splits seed phrases into QR codes. xLink splits M2M messages. xGit splits commits across geographic backends. Without a universal format, each routing layer needs product-specific parsing — an O(N²) integration problem.

xFormat solves this: one 26-byte header, 52 registered product types, instant detection via IDA5 magic bytes (0x49444135). Gateways read bytes 5-6 (productType) and route to the correct handler without understanding the payload. Zero-knowledge routing at scale.

Three core operations: serializeEnvelope() creates a binary envelope from metadata + payload. deserializeEnvelope() parses the header and validates all fields. hasXformatMagic() does instant rejection before any crypto operations.

Encoding options for every transport: binary (default, HTTPS/gRPC), Base45 (QR codes per RFC 9285), text wrapper (email shares). Peek operations extract routing metadata (productType, K/N/shareId) without full deserialization — O(1), zero allocation, perfect for high-throughput gateways.

Zero configuration out of the box. Zero npm runtime dependencies. Runs anywhere TypeScript runs — Node.js, Deno, Bun, Cloudflare Workers, browsers. Dual ESM and CJS builds ship in a single package.

Developer Experience

xFormat provides 14+ structured error codes and detailed validation messages to help developers build reliable share routing systems.

Structured Error Handling

xFormat uses a Result<T, E> pattern with detailed error structures. Every error includes a machine-readable code, human-readable message, actionable hint, field attribution, and documentation URL.

interface FormatError {
  code: string;         // e.g., 'INVALID_MAGIC'
  message: string;      // Human-readable description
  hint?: string;        // Actionable suggestion
  field?: string;       // Field that caused the error
  docs?: string;        // Documentation URL
}

Error Categories

xFormat organizes 14+ error codes across 4 categories:

const result = deserializeEnvelope(incomingData);
if (!result.ok) {
  const err = result.error;
  console.error(`xFormat error [${err.code}]: ${err.message}`);
  if (err.hint) console.log(`Hint: ${err.hint}`);
  if (err.field) console.log(`Field: ${err.field}`);
  return;
}

// Success path
const envelope = result.value;
console.log(`Product: ${envelope.productType}, Share ${envelope.shareId}/${envelope.n}`);

The Problem

Every PRIVATE.ME product produces XorIDA threshold shares. Without a universal binary format, each product must implement custom detection, routing, and validation logic. Cross-product interoperability becomes O(N²).

The platform now includes 140+ ACIs, each splitting data into shares. When a gateway receives an incoming share, it needs to answer three questions:

• Which product created this? Xail email? Xecret QR code? xLink M2M message? xGit commit?
• What are the threshold parameters? 2-of-2? 2-of-3? K-of-N?
• How do shares link together? Which share is this (index 1, 2, 3...)? What UUID groups them?

Without a universal envelope, every routing layer must understand every product's custom format. Adding a new product requires updating every gateway, proxy, and routing layer. This doesn't scale to 140 ACIs.

The Scaling Problem

Consider a typical enterprise deployment:

• xProxy routes shares between internal systems and external partners
• xWall inspects shares for compliance and threat detection
• xStore stores shares across multi-cloud backends
• Deaddrop temporarily holds shares for async delivery

Each of these systems needs product detection logic. Without xFormat, you'd write 4×140 = 560 product-specific routing handlers. With xFormat, you write 4 generic handlers + 140 product registry entries. The registry is maintained centrally — new products don't require code updates.

The Cross-Product Problem

Shares often traverse multiple products. Example flow:

1. xGit splits a commit (productType: xGIT, code 0x0012)
2. xProxy routes shares (must parse productType to route correctly)
3. xStore distributes shares geographically (must understand K/N to validate redundancy)
4. xVault-RAG indexes share metadata (must extract UUID for correlation)

Without a universal format, step 2 needs xGit-specific parsing. Step 3 needs xGit + xProxy dual parsing. Step 4 needs xGit + xProxy + xStore triple parsing. This compounds to O(N²) integration complexity.

The Old Way vs. The New Way

Use Cases

xFormat enables share routing across every PRIVATE.ME product without product-specific integration logic.

Architecture

xFormat is a 26-byte binary header followed by a variable-length payload. Magic number (IDA5) enables instant rejection. Product type (2 bytes) enables zero-knowledge routing. UUID (16 bytes) links shares from the same split.

Binary Header Layout

Total header: 26 bytes. All multi-byte integers are big-endian (network byte order).

Offset  Field           Size   Type    Description
──────  ──────────────  ─────  ──────  ────────────────────────────────
0       magic           4      u8[4]   IDA5 magic: 0x49444135
4       version         1      u8      Format version (always 1)
5       productType     2      u16     Product code (0x0001-0x0036, 0xFFFF reserved)
7       n               1      u8      Total shares (2-255)
8       k               1      u8      Threshold (2 ≤ k ≤ n)
9       shareId         1      u8      Share index (1-based, 1 ≤ id ≤ n)
10      uuid            16     u8[16]  UUID v4 (links shares from same split)
──────  ──────────────  ─────  ──────  ────────────────────────────────
26      payload         var    u8[]    Raw share data (1+ bytes)

Field Details

magic (4 bytes): Always 0x49444135 ("IDA5" in ASCII). This enables instant rejection of non-xFormat data before any crypto operations. Same principle as %PDF- for PDFs or PK\x03\x04 for ZIP files.

version (1 byte): Format version. Currently always 1. Future versions will increment for breaking changes. Parsers must reject unknown versions.

productType (2 bytes): 16-bit product code from the central registry. Currently 52 registered products (0x0001-0x0036). Code 0xFFFF is reserved. Code 0x0000 means unknown/unregistered. See Appendix A6 for the full registry.

n (1 byte): Total number of shares in the K-of-N split. Range: 2-255. Must be ≥ k.

k (1 byte): Threshold parameter — minimum shares required for reconstruction. Range: 2-255. Must be ≤ n and ≥ 2. (K=1 is not information-theoretically secure.)

shareId (1 byte): Share index, 1-based. Range: 1-n. Share numbering starts at 1, not 0, per XorIDA specification.

uuid (16 bytes): UUID v4 (RFC 4122). Generated once when splitting. All shares from the same split use the same UUID. This enables correlation across routing layers without payload access.

payload (variable): Raw share bytes. Minimum 1 byte (empty payloads rejected). No maximum — limited only by transport (e.g., HTTP request size limits).

Product Type Registry

52 registered products with sequential 16-bit codes (0x0001–0x0036). Code 0xFFFF is reserved for future use. Code 0x0000 means unknown/unregistered. Once assigned, codes are permanent — enabling forward compatibility. New products receive the next sequential code.

The registry is maintained in product-registry.ts as a TypeScript constant array. Each entry includes:

• code: 16-bit identifier (0x0001-0x0036)
• name: Product name in uppercase (e.g., "XLINK", "XECRET")
• description: One-line summary
• npm: Package name (e.g., "@private.me/agent-sdk", "@private.me/shareformat")

export const PRODUCT_REGISTRY: ProductTypeEntry[] = [
  { code: 0x0001, name: 'XAIL',    description: 'Secure email client', npm: '@private.me/win' },
  { code: 0x0004, name: 'XLINK',   description: 'M2M messaging', npm: '@private.me/agent-sdk' },
  { code: 0x0012, name: 'XGIT',    description: 'Version control shares', npm: '@private.me/xgit' },
  { code: 0x0033, name: 'XECRET',  description: 'Seed phrase QR custody', npm: '@private.me/xecret' },
  // ...48 more entries (see Appendix A6 for full registry)
];

Helper functions getProductByCode() and getProductByName() provide O(1) lookups (backed by internal Map caching).

Encoding Options

xFormat supports three encoding modes for different transports:

Zero-Knowledge Routing

xFormat enables infrastructure to route shares without cryptographic keys, without payload understanding, and without HMAC validation. Routing layers are untrusted — security comes from threshold splitting, not from trusting proxies.

The Routing Pattern

A gateway receives unknown binary data from the network. It must route to the correct backend without understanding the payload. xFormat makes this O(1):

1. Check magic bytes: hasXformatMagic(data) — instant rejection if first 4 bytes ≠ 0x49444135
2. Peek product type: peekProductType(data) — extract bytes 5-6 (uint16 BE) without full parse
3. Look up routing table: getProductByCode(code) — O(1) Map lookup returns handler name
4. Forward to handler: Backend-specific transport (HTTP POST, message queue, etc.)

The gateway never validates HMAC tags. Never decrypts the payload. Never touches the UUID. It's a pure routing layer — trust-minimized by design.

import { hasXformatMagic, peekProductType, getProductByCode } from '@private.me/shareformat';

async function routeShare(incomingBytes: Uint8Array) {
  // Step 1: Instant rejection (O(1), 4-byte check)
  if (!hasXformatMagic(incomingBytes)) {
    throw new Error('Not an xFormat share');
  }

  // Step 2: Peek product type (O(1), bytes 5-6)
  const code = peekProductType(incomingBytes);
  if (!code) {
    throw new Error('Buffer too short to peek productType');
  }

  // Step 3: Lookup handler (O(1), Map-backed)
  const product = getProductByCode(code);
  if (!product) {
    throw new Error(`Unknown product code: 0x${code.toString(16)}`);
  }

  // Step 4: Route to backend (no payload access)
  switch (product.name) {
    case 'XAIL':
      return await routeToEmailBackend(incomingBytes);
    case 'XLINK':
      return await routeToM2MBackend(incomingBytes);
    case 'XGIT':
      return await routeToGitBackend(incomingBytes);
    default:
      throw new Error(`No handler for ${product.name}`);
  }
}

// Backend handlers receive the full envelope, parse it, validate HMAC, reconstruct.
// The gateway never sees plaintext.

Performance Characteristics

Routing Flow Diagram

Complete Flow

End-to-end example: Xail splits an email into 2-of-3 shares, sends via Gmail/Outlook/Yahoo, and the recipient reconstructs.

Step 1: Sender Creates Shares

import { split } from '@private.me/crypto';
import { serializeEnvelope, getProductByName } from '@private.me/shareformat';
import { sendEmail } from '@private.me/transport';

// Original email message
const emailText = 'Meeting at 3pm tomorrow. Sensitive discussion.';
const emailBytes = new TextEncoder().encode(emailText);

// XorIDA threshold split (2-of-3)
const splitResult = await split(emailBytes, { k: 2, n: 3 });
if (!splitResult.ok) throw new Error('Split failed');

const shares = splitResult.value.shares; // 3 Uint8Arrays
const splitUuid = crypto.randomUUID(); // Links these 3 shares
const xailProduct = getProductByName('XAIL')!;

// Wrap each share in xFormat envelope
const envelopes = shares.map((shareData, i) => {
  const result = serializeEnvelope({
    productType: xailProduct.code,  // 0x0001
    n: 3,
    k: 2,
    shareId: i + 1,                 // 1-based indexing
    uuid: uuidToBytes(splitUuid),   // Convert UUID string to Uint8Array
    payload: shareData,
  });
  if (!result.ok) throw new Error(result.error.message);
  return result.value; // Uint8Array with 26-byte header + share payload
});

// Send envelope[0] via Gmail, envelope[1] via Outlook, envelope[2] via Yahoo
await sendEmail(envelopes[0], { provider: 'gmail', to: recipient });
await sendEmail(envelopes[1], { provider: 'outlook', to: recipient });
await sendEmail(envelopes[2], { provider: 'yahoo', to: recipient });

Step 2: Email Transport (Oblivious)

Gmail, Outlook, and Yahoo servers route the emails normally. They see base64-encoded binary blobs inside IDA5-branded text headers. The email providers cannot reconstruct because they only see 1 share each. The format looks like regular email to spam filters and content scanners.

Step 3: Recipient Receives and Reconstructs

import { deserializeEnvelope, hasXformatMagic } from '@private.me/shareformat';
import { reconstruct } from '@private.me/crypto';
import { fetchEmails } from '@private.me/transport';

// Fetch from all 3 email accounts
const gmailMessages = await fetchEmails('gmail');
const outlookMessages = await fetchEmails('outlook');
const yahooMessages = await fetchEmails('yahoo');

// Extract xFormat envelopes from email bodies (base64 decode + unwrap IDA5 headers)
const allEnvelopes = [
  ...extractEnvelopesFromEmail(gmailMessages),
  ...extractEnvelopesFromEmail(outlookMessages),
  ...extractEnvelopesFromEmail(yahooMessages),
];

// Filter for xFormat shares only
const xformatShares = allEnvelopes.filter(hasXformatMagic);

// Deserialize and group by UUID
const sharesByUuid = new Map<string, any[]>();
for (const envelope of xformatShares) {
  const result = deserializeEnvelope(envelope);
  if (!result.ok) continue; // Skip invalid envelopes
  
  const parsed = result.value;
  const uuidStr = bytesToUuid(parsed.uuid);
  
  if (!sharesByUuid.has(uuidStr)) {
    sharesByUuid.set(uuidStr, []);
  }
  sharesByUuid.get(uuidStr)!.push(parsed);
}

// Reconstruct each message (need K shares per UUID)
for (const [uuid, shares] of sharesByUuid) {
  if (shares.length < shares[0].k) {
    console.log(`Waiting for more shares: ${shares.length}/${shares[0].k} for UUID ${uuid}`);
    continue;
  }

  // XorIDA reconstruction (validates HMAC internally)
  const payloads = shares.map(s => s.payload);
  const indices = shares.map(s => s.shareId);
  const reconstructResult = await reconstruct(payloads, { indices, k: shares[0].k });
  
  if (!reconstructResult.ok) {
    console.error(`Reconstruction failed for ${uuid}: ${reconstructResult.error}`);
    continue;
  }

  const emailBytes = reconstructResult.value;
  const emailText = new TextDecoder().decode(emailBytes);
  console.log(`Reconstructed email: ${emailText}`);
}

Flow Diagram