Why a Quiet Server Stays Stale Until Someone Types

What does your second server show when nobody on it is typing?

It shows the old text. That server learns about another server's edits only when its own next write trips over a hole in the log. Until then every reader on it sees a document that is behind. Nothing errors. No socket closes. The screen is wrong, and it stays wrong until somebody on that machine types.

The characters can still agree. They live in a CRDT, so the text is a function of the operations, not of the order the packets arrived. The sockets cannot agree that way. A reader who reconnects needs one sequence number, and that number is honest only if every frame extended a contiguous prefix of the Postgres log. collab-sphere got into trouble wherever we treated those two jobs as one.

Sequence 12 is already on the sockets. A hint arrives for 14 while this hub has delivered only through 12. The hub does not skip ahead. It reads 13 from the log, and only then releases 14.

Two typists and one character

Two people type after the same character. Their packets reach two machines in opposite order. Both screens have to end on the same string. A lock on the document works on one process. The second process has already applied the other edit by the time the lock matters.

Each character is a node. The parent is the character it was typed after. The root is a zero id. Children sort with the greater clock first, and with the greater client id when the clocks match. The text is a walk of that tree. A delete does not pull the node out. It records the delete on the node and leaves the node in place, so a later insert that named it still has a parent. The mergeable state is two sets, inserts and deletes. Adding to a set is commutative, associative, and idempotent. The same edit can arrive twice, and the text does not double.

Ids are a client and a Lamport clock. A local edit takes the next clock after the largest one that replica has applied. That puts the new character first among its siblings, which is the place the typist meant. Apply rejects an insert whose clock is not greater than its parent's. We did not use a linked list in document order. That list converges only when every client obeys the clock rule. The tree converges for any set of valid operations. A buggy clock puts one character in the wrong place. It does not fork the document.

The library will wait for a parent that has not arrived. The server still has to decide when other people are allowed to see a character. The tempting answer is to show it first and store it second.

The edit the database never kept

There is a faster path. Apply the edit, send it to every socket, and write it to Postgres after the round trip. People have already seen a character that a restart will not load.

The hub we shipped refuses that order. One goroutine owns each open document. It alone holds the tree, the connected clients, and the last cursors. The WebSocket pumps do not touch the tree. They send joins, leaves, batches, and cursors down a channel. For each batch the goroutine checks the operations, applies the ones that pass, writes them through document-service, and only then broadcasts. The sender is on that broadcast. That frame is the acknowledgement. A second ack message would be a second order to keep honest, so there is no second ack.

If the write fails, memory is ahead of the database. The tree cannot be un-applied in any way we trusted. The hub tells every client, closes them, and stops. The next join loads from Postgres. We call that fail-stop. Early clients had no offline buffer, so the edits in the failed batch were lost to the sender. Later clients keep an unacknowledged buffer and resend. The hub still does not reconcile a private copy with the log. It throws the private copy away.

A slow reader follows the same rule. Each socket has a send buffer of 4,096 frames. When the buffer is full, the hub closes that peer as a slow consumer rather than stall everyone else. One goroutine per document was sized for 100 editors on one process. Past that, we add hub processes. We do not add a second writer inside the first one.

A second process is not a second writer. It is a server that will not look up the log until someone connected to it types.

The hub that never opens a gap

With two hubs, one process can hear about sequence 14 before it has heard 13. A reconnecting client sends the last sequence it holds. The hub reads the log until the sequence it has delivered reaches that number, and only then accepts new operations from that client. If the log ends below the sequence the client claims, the hub refuses the join. The client is asserting operations Postgres never stored. A set of frames that arrived in whatever order the network chose does not give anyone a number to resume from.

The quiet hub is worse than a hole. A hub that only serves readers learns nothing until it appends something itself. The other hub's edits sit in Postgres. The quiet hub does not look, because nothing local opened a gap. Its reader waits until someone at that hub types. The reader did nothing wrong. The server had no reason to ask.

Postgres stays the only order. Each hub keeps D, the highest sequence it has delivered to its sockets. A sequencer sits between the log and the sockets. It holds D and any ranges that arrived above D. It releases a range only when that range starts at D + 1.

14 waits until 13 is in hand

func (s *Sequencer) Offer(b Batch) []Batch {
	var fresh []Batch
	lo := s.delivered + 1
	for _, p := range s.pending {
		fresh = appendNonEmpty(fresh, b.clip(lo, p.FirstSeq))
		lo = max(lo, p.Next())
	}
	fresh = appendNonEmpty(fresh, b.clip(lo, math.MaxUint64))
	s.pending = mergeAdjacent(append(s.pending, fresh...))
 
	var ready []Batch
	for len(s.pending) > 0 && s.pending[0].FirstSeq == s.delivered+1 {
		ready = append(ready, s.pending[0])
		s.delivered = s.pending[0].Next() - 1
		s.pending = s.pending[1:]
	}
	return ready
}

Offer keeps the part of a batch that is not already delivered and not already buffered. A batch that starts at 14, while D is 12, stays pending. Nothing in that call moves D. Offer does not go and fetch 13. A later batch that covers 13, whether it came from a hint or from a log read, lets both ranges out in order. The hub applies them and broadcasts to every local peer, the sender included. The sender's copy is a duplicate on the tree, and that duplicate is the point. The echo and everyone else's delivery are the same frame, carrying the sequence the log assigned.

Between events, the hub's tree is the log through D, plus the operations this hub appended that still wait above D. A joining client must not build on the waiting part. Another hub may not have delivered it. Before a full sync goes out, the hub fills until nothing waits. The snapshot is then a prefix every other hub can also reach.

The sockets never see 14 until 13 is in hand. Offer will wait if 13 never arrives. Something outside Offer has to go and read it.

When the hint never arrives

Redis is how a quiet hub hears about 13 without typing. After the append returns, the hub publishes the new sequences on doc:{id}. The message holds only the operations that append newly stored. The publish goes onto a queue of 1,024, drained off the hub loop. A full queue drops the batch. A failed publish does not stop the hub.

Subscribers offer the batch to their own sequencer, which trims anything they already hold. Each hub subscribes before it loads, and waits up to two seconds for Redis to confirm. A publish that happens after the snapshot still reaches it. If the confirmation is late, the hub starts anyway and reads the log when the confirmation arrives.

Pub/Sub delivers at most once. A subscriber that was disconnected misses messages, and Redis will not replay them. Every loss is repaired from Postgres.

A later batch exposes the hole, and a 30 millisecond timer reads the missing sequences. That is the common case while people are typing. An empty read backs off and tries again. It does not treat the hole as closed. Three failed reads in a row fail-stop the hub. A hole the log will not fill is not a hole the sockets should keep.

Quiet documents need a second path. If the hub has clients and D has not moved for a second, it reads the log past D. That catches a lost last message. It also catches a hub that crashed after its commit and before its publish, on a document nobody else is editing. A failed idle read waits for the next tick. It does not count toward the three failures. Finding nothing is the normal result.

The third path is a reconnect. go-redis resubscribes on its own, and any message from the gap is gone. The bus reads with PubSub.Receive rather than Channel, so it sees each subscribe confirmation. A confirmation for a channel that was already live means messages may have been lost, and every hub on that document reads the log at once.

Redis going down is a latency event. Hubs keep serving their own clients. They learn the other hubs' edits through the gap timer and the idle read, one second behind on a quiet document. Stopping the hub would turn a latency dependency into an availability dependency. Every client would drop during a cache restart, even though Postgres still has the log.

We did not build Redis Streams as a second log. Streams can replay, and they would need trimming, consumer groups, and a rule for which log wins. Postgres already makes a hole visible as a missing sequence. We also did not publish before the append. That saves a round trip and announces sequences that may never be stored. A message with no sequence cannot show that it was lost.

If you cannot name the sequence Postgres assigned, the edit has not been published yet.

The snapshot that sat on the keystroke

The slow part of the 100-editor load test did not show up in Postgres. The 99th percentile of the operation round trip ran from 36 to 284 milliseconds, against a target under 50. Postgres logged no statement over 20 milliseconds. Hub-side timing pointed at the snapshot written inside the append. It took 16 milliseconds at 1,000 operations and 37 at 4,000. The work grows with the document, because it parses the prior snapshot, replays the tail, and marshals the whole document again.

Snapshots keep long documents cheap to load. When op_seq - snapshot_seq reaches 100, we store one, as protojson in JSONB, and replay only the tail after snapshot_seq. LoadState reads the snapshot and the tail in one repeatable-read transaction, so an append cannot land between the two reads.

Compaction moved to after the commit. A first version rebuilt the snapshot inside a repeatable-read transaction, and every append that committed during the rebuild aborted it. Each retry redid the whole document. The version we shipped does not lock the document row for that rebuild. A failed compaction retries, and if it keeps failing, the next append schedules another attempt. Two compactors racing is harmless. The same sequence always rebuilds the same state, and the insert ignores a conflict. The keystroke path now locks the row, assigns sequences, and returns.

The snapshot was the right cache. It was inside the transaction a person was waiting on.

A client the local tree accepts and the log refuses

Once several hubs append, a check against the local tree is not a check against the log. The hub still stops at the first failure, in a fixed order.

The id on the operation must be this connection's client id. Anything else is CODE_CLIENT_MISMATCH, a socket writing as someone else. The clock must be at most the document clock plus one. A wider jump lets one operation push the clock forward for every replica. The parent, or the delete target, must already be in the tree. A missing one is CODE_MISSING_DEPENDENCY. The hub is not a peer that can wait forever, and a held operation would grow memory without a bound. Apply must accept the operation. An operation the tree already holds is skipped, so a resend does not double the character.

Encryption checks run before those. An insert is a rune or a ciphertext, matching the document, never both. On an encrypted document the ciphertext is 17, 18, 19, or 20 bytes, and the epoch is the open write epoch. A copy insert is refused from a client. Only the server writes copies, when it restores a character onto a new id.

The valid prefix of the batch is stored and broadcast. The failing operation closes its sender. CODE_UNAVAILABLE is the only code a client may retry. CODE_FORBIDDEN and CODE_KEY_ROTATION leave the socket open, so a viewer who tried to write, or a client in the middle of a key rotation, still receives later broadcasts. Every other code closes the connection. A cursor with a missing anchor is a different case. It races the sender's own in-flight edits. We drop that cursor update and leave the socket up.

We kept the clock bound after adding hubs. An honest client advances from the largest clock it has seen. After a resume, the hub has caught up to the client's sequence before it accepts anything new, so those clocks are already in the tree. A check that demanded the clock equal the client's own last clock plus one would reject a legal Lamport jump. Removing the bound would let one operation exhaust the clock for every replica.

The check that still hurts is a conflict the local tree cannot see. Another hub has already logged the same client and clock with different bytes. The local checks pass. Append returns AlreadyExists with reason OP_CONFLICT. We close that sender with CODE_INVALID_OPERATION. The group is already in this hub's tree and cannot be undone, so the hub fail-stops for everyone else with CODE_UNAVAILABLE. Honest clients reconnect and resend. Identical bytes dedupe.

The cost shows up when a client ignores a terminal error and repeats the conflict. Every peer on that hub reconnects each time. Closing that needs client ids a caller cannot mint at will. Sessions are Ed25519 tokens checked against AUTH_PUBLIC_KEY, and another system issues them. Roles are owner, editor, commenter, and viewer. A role decides whether a socket may append. It does not decide the sequence.

A local pass means the tree could take the bytes. Durability means the log agrees with those bytes. When the log disagrees, the tree is thrown away.

Text the server must not read

Some documents have to stay unreadable to the servers that order them. A document is plain or encrypted from its first insert, and the mode never changes. Plain bytes carry the magic CSS1. Encrypted bytes carry CSS2. Text, CanonicalBytes, and Diff fail on an encrypted document. They do not guess. Positions still resolve, with empty text.

Clients seal each rune with pkg/docrypt under a 32-byte document key. The nonce is the first 12 bytes of SHA-256 over the document id, the epoch, and the lineage root. It is never stored. A client must not seal two different runes under one replica and clock in one epoch. Reusing a GCM nonce leaks the XOR of the two runes and the authentication key to the server. That server is the party this mode is defending against. A client that loses its clock takes a fresh replica. Sealing the same rune under the same id again is harmless, because the bytes match.

Services never import pkg/docrypt. They store ciphertext in the log, in snapshots, and in version blobs. Restore copies an old character onto a new id, and the server cannot re-seal that copy. The copy stays sealed under the original lineage, so a client with the key can still open it. Export to Markdown, DOCX, and PDF goes through pkg/render, and an encrypted document is refused there.

The sequence number does not depend on knowing the rune. The calls that would invent one fail.

Deletes that a later edit can still name

A deleted character can still be somebody's parent. Suppose the root has two children, A at clock 2 and a tombstone T at clock 1, and T has a child B at clock 5. The text reads AB. Hang B on the root and the sort puts B before A. The text becomes BA, and nobody typed that. Moving a child is not compaction. It is a different document.

A tombstone is collected only when it is deleted, has no children, and a version vector every replica has reported covers the node and at least one of its deletes. The vector has to include replicas that are offline. Otherwise a late insert can name a node we already threw away. While a branch stays open, tombstones from clients that branch has not seen are kept. Storage grows. Losing a character a branch might still append under is worse.

Comments use the same ids. An anchor is a range of character ids, not two indexes, because an index moves when the log delivers someone else's insert. A comment created on a branch stays off the parent document until a merge brings that state in. Restore does not rewind the log. It appends copies under new ids. Versions older than RESTORE_HORIZON, 720 hours by default, stay readable. Restore and branch refuse them.

Presence sits outside this order on purpose. Cursors and names live in Redis and expire when a hub dies. A missing anchor drops a cursor. It does not close the socket, and it never takes a sequence. A presence miss must not become a document miss.

Half the laptop runs missed the target

The target is a 99th percentile echo and fan-out under 50 milliseconds at 10,000 open connections. make load-proof asserts it. The README records the runs, with the services, Redis, Postgres, and the generator on one machine. With Postgres in Docker, the 99th percentile ran from 53 to 114 milliseconds. Running Postgres natively, it ran from 42 to 162, and half the runs passed.

We do not quote the passing half as the latency of the system. A reliable number needs the generator on a different machine. What the laptop did was find the snapshot inside the append. It also justified keeping single-instance mode. With REDIS_URL unset, the hub uses an in-process bus, and the sequencer still refuses to step past a hole. Tests drop, delay, and reorder deliveries. The sockets still never see an out-of-order frame.

Decisions we would keep

The tree answers what the text is, given a set of operations. The log answers which of those operations a socket has been shown. A CRDT does not give you a reconnect cursor. A sequence number does.

Publish after the append, and fail-stop when memory gets ahead of the write. Showing the edit first is the crash window. Redis can be down. Postgres cannot be skipped. A hint without a sequence cannot be repaired, so we never send one.

Derived work stays off the keystroke. Tombstones stay until every branch that might name them has reported in. The cost is disk. The other choice is a document that quietly becomes a different document.

A quiet server is safe when it can name the number it is waiting for.