@nextgensoftwares/folio-sync

Loading and saving: chunked progressive loading, step-based sync, offline outbox (IndexedDB), localStorage snapshots, server-verified checksums. Guides: Loading & saving, Real networks.

Protocol types

interface ChunkRef { hash: string; from: number; to: number; bytes: number; pageStart?: number; pages?: number }
interface Manifest {
  docId: string; version: number; attrs?: Attrs; blockCount: number; chunks: ChunkRef[];
  estimatedPages?: number; checksum: string;
  firstChunk?: Chunk; // only when requested with inlineFirst
}
interface RequestOptions { signal?: AbortSignal; reload?: boolean; inlineFirst?: boolean }
interface Chunk { hash: string; blocks: FolioNode[] }
interface PushOptions { confirm?: boolean; checksum?: string }
interface PushResult { ok: boolean; version: number; rejected?: 'mass-delete' | 'diverged' | 'unsupported'; error?: string }
interface PullResult { version: number; steps: unknown[]; clientIDs: string[] }
interface LayoutReport { version: number; totalPages: number; chunks: { hash: string; pageStart: number; pages: number }[] }

interface DocumentApi {
  manifest(docId: string, opts?: RequestOptions): Promise<Manifest>;
  chunk(docId: string, ref: ChunkRef, opts?: RequestOptions): Promise<Chunk>;
  push(docId: string, version: number, steps: unknown[], clientID: string, opts?: PushOptions): Promise<PushResult>;
  pull(docId: string, since: number): Promise<PullResult>;
  reportLayout?(docId: string, report: LayoutReport): Promise<void>;
  create?(docId: string, doc: FolioDocument, opts?: CreateOptions): Promise<{ docId: string; version: number }>;
  list?(opts?: RequestOptions): Promise<DocInfo[]>;
}
interface DocInfo { docId: string; version: number; title?: string }
interface CreateOptions { title?: string; signal?: AbortSignal }

Export	Description
class IntegrityError	loaded content doesn't match its content address
class OfflineError	thrown by transports when the network is unavailable (drives offline mode)
isOffline(e)	OfflineError, or a fetch/network TypeError
class HttpError	a non-success status other than the protocol's 409/422 (status)
class ChunkLoadError	a chunk could not be loaded after retries (index, ref, cause): the load stops, assembleDocument rejects
type DocInfo	one document a server lists (DocumentApi.list): docId, version, optional title
type CreateOptions	options of DocumentApi.create: title (display title, returned in manifests and listings) and signal (abort)
type RequestOptions	signal (abort), reload (bypass HTTP/CDN caches: refetch after a corrupt response), inlineFirst (manifest only: ask for chunk 0 inline)

Transports and reference server

Export	Signature	Description
createHttpApi	(opts: HttpApiOptions) => DocumentApi	REST client; 409 = behind, 422 = refused. Reads retry with backoff + jitter on network errors, timeouts, 408/429/5xx (honouring Retry-After); every request has a timeout and follows signal; GETs send no Content-Type (no CORS preflight per chunk)
type HttpApiOptions		baseUrl, headers?() (per request), fetch?, stats?: TransportStats, credentials?, plus RetryOptions
withRetry	(attempt: (signal) => Promise<T>, opts?: RetryOptions, signal?) => Promise<T>	per-attempt timeout + exponential backoff with full jitter; a final timeout becomes OfflineError
type RetryOptions		retries (3), baseDelayMs (300), maxDelayMs (8000), timeoutMs (30000)
isRetryable	(e) => boolean	network errors, timeouts, 408, 429, 5xx
class RetryAfterError		an HttpError carrying the server's Retry-After (retryAfterMs)
class MemoryDocumentServer	new (schema: PMSchema, docs?: Record<string, FolioDocument>, opts?: ServerOptions)	reference backend: put, has, version, manifest, chunkIndex, hasChunk, chunkBody (current or recently superseded chunks), push (throws StepError), pull(docId, since, limit?), reportLayout, document; subclasses persist in the protected committed hook (@nextgensoftwares/folio-server-node)
type ServerOptions		ChunkingOptions + GuardOptions
type StepEntry		{ json, clientID }: one accepted step in the log
class HistoryGoneError		pull(since) asked for steps older than the server keeps: reopen the document
validatePush	(schema, base: PMNode, steps: unknown, opts: { confirm?, checksum? }, guard?: GuardOptions) => GuardResult	the storage-agnostic push validation every backend shares: parse + apply steps to the server's copy, compare the result checksum, mass-delete guard; nothing is mutated
type GuardOptions		massDeleteGuardSize (2000, PM size), requireChecksum (true), maxStepsPerPush (5000)
type GuardResult		{ ok: true, doc } | { ok: false, rejected: 'mass-delete' | 'diverged' }
class StepError		steps that don't parse (kind: 'malformed', HTTP 400), don't apply ('unapplicable', treated as diverged) or use nodes/marks/attrs the server's schema lacks ('unsupported': push returns { rejected: 'unsupported', error }, HTTP 422; the message names them)
collectGaps, stepGaps, describeGaps	(schema, json) => SchemaGaps, (schema, stepJson, doc?) => SchemaGaps, (gaps) => string | null	what a document or step uses that a schema doesn't declare, and the readable refusal; PersistentDocumentServer.create uses them too
class ChunkIndex	new (opts?: ChunkingOptions)	incremental chunking: update(doc) walks the blocks once and re-serializes/re-hashes only chunks whose blocks changed (lastUpdate: { hashed, reused, ms }); has(hash), body(hash) for current and retained chunks
type ChunkingOptions		minChunkBlocks (100), maxChunkBlocks (1000), retainChunks (512 superseded chunks stay fetchable)
type ChunkSpan		{ hash, from, to, bytes }
memoryApi	(server, network: () => NetworkProfile, stats?) => DocumentApi & { stats }	the server behind a simulated network (latency, bandwidth, offline) with real JSON round-trips
NETWORKS	Record<'lan' | 'wifi' | '4g' | '3g', NetworkProfile>	presets
type NetworkProfile		{ latencyMs, bytesPerSecond, offline? }
type TransportStats		{ requests, bytesDown, bytesUp }
hashString	(s: string) => string	content hash used for chunk ids (two 32-bit FNV-style hashes, base 36)
compact	(node: PMNode) => Record<string, unknown>	node JSON without default-valued attrs (the wire format)
blockWire	(node: PMNode) => string	compact JSON string of a block, memoized by node identity

Chunks are cut at the next H1 once they hold minChunkBlocks blocks, and hard-cut at maxChunkBlocks. Chunk JSON is compact (default-valued attrs omitted).

Loading

Export	Signature	Description
openDocument	(api, docId, opts?: LoadOptions) => Promise<LoadSession>	manifest (cached when offline), then chunks cache-first, concurrency (6) in parallel, delivered in order, each verified against its hash
type LoadOptions		cache?, concurrency?, firstChunk?, manifest? (e.g. an outbox's base), signal? (aborts delivery and in-flight requests), staleWhileRevalidate? (open from the cached manifest, revalidate in the background), firstChunkAlone? (fetch chunk 0 before the others; for HTTP/2)
type LoadSession		{ manifest, offline, chunks: AsyncGenerator<LoadedChunk>, revalidated? }
type LoadedChunk		{ index, ref, chunk, fromCache }
chunkForPage	(manifest, page) => number | null	chunk containing a page, from the last layout report
assembleDocument	(session, opts: AssembleOptions) => { ready: Promise<FolioEditor>; complete: Promise<FolioEditor> }	editor after the first chunk; the rest appended in ≤100 ms batches; outbox restored at the end
type AssembleOptions		EditorOptions minus doc/plugins, plus clientID, outbox?, onProgress?(loaded, total, editor)
appendLoaded	(editor, blocks) => void	append blocks outside undo history and outside collab's unconfirmed steps
restoreOutbox	(editor, rec: OutboxRecord) => void	replay confirmed steps, then unconfirmed local ones
layoutReport	(manifest, layout) => LayoutReport	page each chunk starts on, for the server

Saving

type SyncState = 'synced' | 'saving' | 'offline' | 'error' | 'needs-confirmation' | 'diverged';
// status.reason === 'unsupported': state 'error', sync stopped for good (the server's schema lacks a plugin); edits stay in the outbox
interface SyncStatus { state: SyncState; version: number; pending: number; lastSavedAt?: number; error?: string }
interface SyncOptions {
  manifest: Manifest;                                   // what the editor was loaded from
  outbox?: Outbox;                                      // durable outbox (IndexedDB)
  debounceMs?: number;                                  // default 400
  pollMs?: number;                                      // default 2000
  confirmed?: { steps: unknown[]; clientIDs: string[] }; // when restored from an outbox
}

class SyncClient	Description
new SyncClient(editor, api, docId, clientID, opts)	the editor must have been created with the collab plugin (assembleDocument does this)
start(): this	warm the checksum memo, subscribe to the editor, start polling, sync once
stop()	unsubscribe and clear timers
subscribe(fn: (s: SyncStatus) => void)	status listener
sync(): Promise<void>	pull, then push pending steps with a result checksum; repeat until in sync (max 5 rounds)
confirmPending(): Promise<void>	resend a push the server refused as a mass delete, with confirm: true
status: SyncStatus	current status

Sync schedule

SyncClient takes ScheduleOptions (all optional): pollMs (first poll interval, default 2 s, growing ×1.5 per quiet poll; false = never poll, e.g. when a realtime socket carries steps), maxPollMs (30 s), retryBaseMs (1 s, doubling per failure with jitter) and maxRetryMs (60 s). Hidden tabs and offline browsers aren't polled; coming back online, showing the tab or client.retryNow() syncs at once. status.retryAt says when the next automatic retry runs. Saves cost one request (push); the client pulls only when the server says it's behind.

SyncSchedule is the scheduler itself (new SyncSchedule(run, options, env?) with start, stop, now, succeeded(remoteChanged), failed, backingOff, retryAt), exported for hosts building their own sync loop.

Integrity

Export	Signature	Description
docChecksum	(doc: PMNode) => string	order-sensitive checksum of the top-level blocks (+ doc attrs and child count); attrs at their schema default are skipped, so a plugin-extended client schema and the server's base schema agree; memoized per document node
nodeHash	(node: PMNode) => number	32-bit structural hash of a node, memoized by node identity

Stores

interface ChunkCache { getManifest(docId); putManifest(m); getChunk(hash); putChunk(chunk) }
interface Outbox { getOutbox(docId); putOutbox(rec); clearOutbox(docId) }
interface OutboxRecord {
  docId: string; clientID: string; manifest: Manifest; baseVersion: number;
  confirmed: unknown[]; confirmedClientIDs: string[]; unconfirmed: unknown[]; updatedAt: number;
}
interface Snapshot { version: number; doc: FolioDocument; savedAt: number }

Export	Description
class IndexedDbStore implements ChunkCache, Outbox	new IndexedDbStore(name = 'folio-sync', idb = indexedDB); stores manifests, chunks, outbox; clear()
class LocalStorageStore	whole-document snapshots for small docs: load, save(docId, doc, version?) (returns { ok, bytes }, keeps the previous snapshot), previous, remove
class IndexedDbSnapshotStore	whole-document snapshots in IndexedDB for books (structured clone, no JSON string, browser storage quota instead of ~5 MB): async load, save(docId, doc, version?) (resolves { ok: true } or { ok: false, error }, never rejects; keeps the previous snapshot), previous, remove. The playground's local mode uses localStorage for small documents and falls back to this automatically.