The Account We Deleted Three Times

At 09:14, the account deletion endpoint returned 204 No Content.

At 09:16, the customer disappeared from the admin console. Their profile query returned no rows, authentication failed, and the scheduled erasure report marked the request complete.

At 10:03, I opened an export link from an old support ticket and downloaded the same customer's name, address history, identity document, device records, and every support conversation attached to the account.

The link was eleven months old.

Nothing had bypassed authentication because the download did not require authentication. Nothing had defeated encryption because the storage service decrypted the object exactly as configured. Nothing had evaded our deletion code because the code had deleted every row it knew about.

That was the uncomfortable part. The deletion worked.

It worked on the account model we had drawn when the product was small: a user row, several child tables, and a storage prefix for uploaded documents. The system we were operating no longer resembled that model. It had exports assembled for customer access requests, copies attached to support cases, transformed records in analytics, search indexes, read replicas, CDN objects, and backups. Each subsystem had acquired its own retention behaviour. The endpoint deleted an account. It did not erase a person.

I had treated data deletion as a database operation. The incident forced me to treat it as a distributed security protocol.

This is the story of that correction. The code and identifiers here are purpose-built for the article, but the failure class is ordinary: if your system can copy personal data, it can outgrow the function that deletes it.

The object that survived

The URL in the ticket pointed to an object called customer-exports/2025/09/7f2a/complete-account.zip. The application had produced it for a customer access request. A worker queried five internal services, rendered the result into JSON and PDF files, compressed the directory, uploaded the archive, and generated a signed storage URL. Support pasted that URL into the ticket.

The export worker considered its job complete after delivery. It recorded the object path in a job table, but the path was not connected to the customer's retention identity. The deletion service knew the customer as account_id = 7f2a. Storage knew the archive as a string. Support knew it as text inside a message. Analytics knew the customer by a pseudonymous key. The CDN knew only a cache key derived from the URL.

We had identifiers everywhere and identity nowhere.

The first measurement was worse than the first example. The export bucket held 27,418 archives consuming 4.6 TB. The oldest was 397 days old. The written retention policy said exports expired after seven days. The storage lifecycle rule said 365 days. A failed infrastructure migration had left one prefix outside that lifecycle rule, which meant its objects had no automatic expiry at all.

The signed URL was configured for seven days, so at first I could not explain why an eleven-month-old link worked. The answer sat one layer above storage. Our download service accepted a stable export token, looked up the object, and minted a fresh signed URL on demand. The token had no expiry column. Anyone holding it could keep turning an old permission into a new seven-day permission.

The seven-day expiry was technically true and operationally irrelevant.

That distinction matters. Security reviews often inspect the duration on the final credential while ignoring the mechanism that can manufacture another credential. A hotel key card that expires tonight offers little protection if the unattended kiosk in the lobby will issue a replacement forever.

The token table looked harmless:

CREATE TABLE export_tokens (
    token_hash  BYTEA PRIMARY KEY,
    object_key  TEXT        NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now()
);

There was no subject_ref, no expires_at, no revocation state, no use count, and no foreign key to an erasure case. The application hashed the token, which protected it if the database leaked, but hashing did not impose a lifetime or answer who the archive belonged to.

I tried the same sequence against a test account. I requested an export, copied the link, deleted the account, waited for every deletion worker to report success, and downloaded the archive. Then I purged the object manually and tried again. The download service returned 404, but the CDN served its cached copy. After purging the CDN, the support platform still retained the message containing the token. Each correction exposed the next representation.

Deletion reached the database, not the copies

By the end of that afternoon I had deleted the same account three times: once from production, once from the export path, and once from delivery. Every pass had been necessary. None had been sufficient.

Reconstructing the afternoon

The ticket that started this was not exotic. A customer had asked months earlier for a copy of their data. Support had followed the documented path, waited for the export worker, and pasted the resulting link into the ticket thread. The customer later requested erasure. The erasure path reported success. Months after that, while reviewing an unrelated access pattern, I searched the support platform for the old account identifier and found the ticket still open in an archived queue. The link still worked.

I pulled request logs around the original export and the later deletion. The export job finished in four minutes. The deletion job finished in eleven seconds. Eleven seconds is a useful number. It told me the deletion path was touching a small, well-indexed set of operational tables and a handful of known storage prefixes. It was not walking support attachments, analytics transforms, CDN caches, backup catalogs, or token registries. Speed was evidence of incompleteness.

I then enumerated every system that could have received the customer's data during the export assembly. The worker called identity, billing history, document storage, device telemetry, and support history. Each of those services returned a payload. The export worker wrote those payloads into files. Those files became an archive. The archive became an object. The object became a URL. The URL became ticket text. Ticket text became searchable history in a third-party platform with its own retention settings. At no point did any of those steps record a shared retention identity that the erasure service could later query.

That inventory already explained why the deletion endpoint was honest and insufficient. It deleted what it owned. Ownership had been defined too early and never revisited.

The false comfort of encryption

Every archive was encrypted at rest. That line appeared in our control description and in more than one review response.

The statement was accurate but incomplete. Storage encryption protected disks and storage infrastructure. The same application role that created an object could read it, and the delivery service could ask storage to decrypt it whenever a token matched. Encryption had not reduced the set of people, services, or stale links able to retrieve the data. It had changed the format of the bytes on a disk we did not physically control.

Encryption at rest is a useful control. It is not deletion, revocation, purpose limitation, or retention.

I began writing the issue in terms that described the actual risk:

A completed erasure request does not revoke access to previously generated customer exports. Export grants are unbounded, archives are not linked to a retention identity, and delivery caches can continue serving copies after the operational record is deleted.

That wording gave us something testable. It also stopped us from closing the issue with another bucket policy.

How far the surviving export could reach

Before changing architecture I needed a threat model that matched the artefact we had found. The surviving export was not an abstract privacy issue. It was a concrete capability: a holder of an old link could reconstruct a deleted customer without authenticating as that customer and without asking any live product service for permission.

I treated three actors as relevant.

The first was an external holder of a leaked or forwarded export link. Support tickets get copied into email. Screenshots move. Browser histories persist. A token that can mint fresh signed URLs indefinitely is valuable to anyone who can obtain it once. That actor does not need to compromise production. They need only the token and network access to the download service.

The second was an insider with ordinary support or engineering access. Support staff could search historical tickets. Engineers could list objects in the export bucket. Neither action required the special privileges we associated with "accessing deleted accounts," because the accounts were deleted only in the operational database. The archives remained ordinary objects and ordinary ticket text.

The third was an attacker who compromised object storage or a backup of object storage. Under the original design, storage encryption used a service-managed key shared across the bucket. Anyone who could use the storage role, or restore a backup into an environment that still had that role, could open archives in bulk. The export format was a ZIP of JSON and PDF files. There was no per-archive key to destroy, and no grant check outside the download path.

I mapped the useful attacks against those actors:

The important finding was not that every actor was equally likely. It was that our deletion ceremony addressed none of them. Deleting the account row removed interactive product use. It did not remove reconstructability.

I also checked what the archive contained relative to what the live account had contained at deletion time. The export was a point-in-time package. In several cases that made it worse than the live store. A customer who had later corrected an address still had the old address in the export. A document the customer had replaced still existed in the ZIP. Deletion of the live account therefore removed the current profile while leaving an older, broader package available through a side channel. From a privacy standpoint, that is a reconstruction of the person at their most expansive recorded state.

One more detail mattered for the threat model. The download service logged successful fetches with the object key and a truncated token hash. Those logs were retained for security investigations. After account deletion, the logs still pointed at a reachable object. Log retention is often justified, and I do not argue against keeping access evidence. I do argue that an access log which still resolves to readable personal data is itself part of the retention graph. Either the referent must become unusable, or the log must stop being a retrieval map.

I spent part of the next morning quantifying residual exposure rather than arguing about intent. For a sample of recently erased subjects, I searched the export token table, object tags where present, support ticket text, and CDN cache keys derived from historical URLs. The hit rate was high enough that we stopped treating the eleven-month-old link as an anecdote. It was a property of the system: deletion removed interactive identity while leaving reconstructive artefacts in place for anyone who already knew where to look.

The threat model closed with a simple success criterion for remediation: after erasure verification, no supported path should be able to turn prior knowledge of an export token, ticket URL, object key, or backup generation into readable customer content, except where an explicit retention exception authorises a narrowly scoped residual record.

Building the retention graph

The first real fix was not code. It was an inventory that described how one person's data moved.

Most data inventories are organised by database, application, or team. That is convenient for ownership and poor for erasure. A deletion request begins with a person, so the useful question is not "what tables does the account service own?" It is "which representations can be reached from this subject?"

I introduced a stable retention identifier called subject_ref. It was not the raw account ID and it was not reversible:

func SubjectRef(secret, accountID []byte) string {
    mac := hmac.New(sha256.New, secret)
    _, _ = mac.Write(accountID)
    return hex.EncodeToString(mac.Sum(nil))
}

The HMAC matters. A plain hash of an enumerable account ID is easy to reverse by hashing the likely values. A keyed digest lets systems correlate records for retention without carrying the production identifier into every store. Rotating or destroying the correlation key later also gives us a way to sever the relationship between retained analytical records and the operational identity.

Every system that produced a durable representation had to record subject_ref as metadata. Export objects received it as an object tag. Support attachments received it in a private custom field. Search documents carried it outside the indexed body. Analytics transformations preserved it in a restricted column until their own retention window elapsed.

This did not mean every system had to delete on the same day. Legal and operational obligations differ. Financial records might require retention after the customer account is closed; an abuse investigation might be under legal hold; aggregated statistics might no longer be personal data at all. The graph carried both the representation and the reason it could remain.

The model was intentionally small:

CREATE TYPE retention_disposition AS ENUM (
    'PURGE',
    'ANONYMIZE',
    'RETAIN_LEGAL',
    'RETAIN_SECURITY'
);
 
CREATE TABLE retention_nodes (
    system_name     TEXT                  NOT NULL,
    representation TEXT                  NOT NULL,
    subject_ref    TEXT                  NOT NULL,
    disposition    retention_disposition NOT NULL,
    expires_at     TIMESTAMPTZ,
    authority      TEXT,
    PRIMARY KEY (system_name, representation, subject_ref)
);

authority is required whenever the disposition starts with RETAIN. "We might need it" is not an authority. It records the policy, contract, legal hold, or regulatory obligation that justifies the exception. That made the inventory useful to privacy counsel and security engineering at the same time.

The comparison below became our review shorthand:

Discovering nodes without trusting memory

The first graph was incomplete by construction. We built it from architecture diagrams, storage prefixes, vendor admin panels, and a week of asking teams what they wrote to disk. That process finds the copies people remember. It misses the copies created by convenience.

I therefore added discovery probes that started from a subject and asked each connected system a concrete question: do you hold any durable artefact tagged with this subject_ref, or any legacy artefact still keyed by the old account identifier? The probes returned counts and representation classes, not payloads. A non-zero count for a system absent from the graph became a failing check in the erasure case.

We also registered producers, not only stores. Any job that could emit a durable export, report, attachment, or derived index had to declare the retention node it wrote into. That declaration lived next to the producer configuration:

retention:
  writes:
    - system: object-storage.exports
      representation: complete-account-archive
      subject_ref_source: request.subject_ref
      default_disposition: PURGE
      default_ttl: 24h

A producer without a retention declaration could not deploy through the path that created durable customer artefacts. Soft guidance had failed us for years. The gate forced the question at the moment a new copy became possible.

Shadow copies still appeared. A data scientist materialised a one-off table for an investigation and forgot to drop it. A debugging session copied three export objects into a personal scratch bucket. A search reindex duplicated documents under a temporary alias that later became permanent. The graph did not prevent those mistakes. It made them visible when the erasure probes ran, because the probes looked for subject_ref tags and for residual account-id keys in known locations. Unknown locations remain a residual risk. Naming that residual risk is part of honest completion language.

The graph created a second problem: discovery could never be assumed complete. A new feature could produce another copy next week. We therefore added a design-review question for every new durable store: Which retention node owns this representation, and how does it prove disposal? A service could not enter production without an answer.

That was more effective than asking whether the service handled personal data. Teams interpret that phrase narrowly. A cache, queue, trace, generated report, or search document may all contain personal data without looking like a customer database.

A deletion system cannot remove a copy it cannot name. Inventory is not paperwork around the control. Inventory is the control's address space.

Revocation before removal

Deleting objects sounded like the obvious next step. It was not fast enough.

Object storage replicated across locations. Lifecycle tasks were asynchronous. Backups were immutable by design. CDN invalidation had its own delay and failure modes. If erasure depended on proving that every physical byte had vanished before access ended, we could not meet a useful response time and we would be tempted to lie about completion.

I separated two goals that had been tangled together:

1. Make the information inaccessible now.
2. Remove residual ciphertext according to the retention schedule.

Each export now receives a random 256-bit data encryption key. The worker encrypts the archive with that key, asks the key management service to wrap it, stores the ciphertext archive in object storage, and records only the wrapped data key in the grant registry. The plaintext key exists in memory for the duration of the stream and is then discarded.

func encryptExport(dst io.Writer, src io.Reader, dek []byte) error {
    block, err := aes.NewCipher(dek)
    if err != nil {
        return err
    }
 
    aead, err := cipher.NewGCM(block)
    if err != nil {
        return err
    }
 
    nonce := make([]byte, aead.NonceSize())
    if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
        return err
    }
 
    plaintext, err := io.ReadAll(io.LimitReader(src, maxExportBytes))
    if err != nil {
        return err
    }
 
    if _, err := dst.Write(nonce); err != nil {
        return err
    }
    _, err = dst.Write(aead.Seal(nil, nonce, plaintext, nil))
    return err
}

The example reads into memory to keep the cryptographic boundary visible. The production shape uses chunked authenticated encryption, unique nonces per chunk, and a signed manifest so large exports do not require a single allocation. The important property is unchanged: every archive has an independent key.

The grant registry now carries the lifetime and owner the old table lacked:

CREATE TABLE export_grants (
    grant_id         UUID PRIMARY KEY,
    token_hash       BYTEA       NOT NULL UNIQUE,
    subject_ref      TEXT        NOT NULL,
    object_key       TEXT        NOT NULL,
    wrapped_data_key BYTEA       NOT NULL,
    created_at       TIMESTAMPTZ NOT NULL,
    expires_at       TIMESTAMPTZ NOT NULL,
    revoked_at       TIMESTAMPTZ,
    downloads        INTEGER     NOT NULL DEFAULT 0,
    max_downloads    INTEGER     NOT NULL DEFAULT 1,
    CHECK (expires_at <= created_at + INTERVAL '24 hours')
);

A download no longer redirects to storage. The caller presents the token to our application, authenticates, and proves that the account subject matches the grant subject. The service checks expiry and revocation in the same statement that consumes the permitted download:

UPDATE export_grants
   SET downloads = downloads + 1
 WHERE token_hash = $1
   AND subject_ref = $2
   AND revoked_at IS NULL
   AND expires_at > now()
   AND downloads < max_downloads
RETURNING grant_id, object_key, wrapped_data_key;

Zero returned rows means no access. The conditional update prevents two concurrent requests from both consuming a single-use grant. Only after that update succeeds does the service ask KMS to unwrap the data key and stream decrypted content to the authenticated connection.

On erasure, we revoke the grant and destroy its wrapped data key before waiting for object deletion:

UPDATE export_grants
   SET revoked_at = COALESCE(revoked_at, now()),
       wrapped_data_key = '\x'
 WHERE subject_ref = $1
   AND revoked_at IS NULL;

Each archive has a revocable key envelope

The encrypted object may still exist in a replica, a backup, or a delayed lifecycle queue. It is no longer useful through our system because the only copy of the data key we retained has been destroyed. This is cryptographic erasure, and it gave us a fast security boundary while slower physical deletion converged.

It also changed the value of a bucket leak. An attacker who obtained object storage without the grant registry received independently encrypted archives, not a set of ZIP files the storage role could transparently open.

Key custody in practice

The envelope design only works if the wrapped key and the ciphertext travel through different failure domains. We stored ciphertext in object storage and wrapped keys in the grant registry. KMS held the wrapping key. The download task was the only role permitted to call unwrap for export keys. The export worker could wrap. It could not unwrap. Support tooling had neither permission.

The KMS key policy encoded that separation:

{
  "Effect": "Allow",
  "Principal": {"AWS": "arn:aws:iam::ACCOUNT:role/export-download-task"},
  "Action": ["kms:Decrypt"],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "kms:EncryptionContext:purpose": "customer-export",
      "kms:EncryptionContext:service": "export-delivery"
    }
  }
}

Every wrap included encryption context binding the key to purpose and service. An unwrap request missing that context failed. That stopped a well-intentioned engineer from decrypting an archive "just to debug" with a broad administrative role. Administrative roles could manage grants and trigger erasure. They could not read export contents without an authorised hold path that minted a temporary investigative grant under a different policy.

We also stopped putting export URLs into support tickets as durable text. The ticket received a reference to a grant ID and a short-lived customer notification channel. If support needed to resend access, they created a new grant with a new expiry. The previous design had treated the URL as a durable pointer. The corrected design treated access as an event that could be re-issued under policy, not as a string that outlived the account.

CDN behaviour changed with the download model. Because the application streamed decrypted bytes to an authenticated session, there was no long-lived public object URL for the CDN to cache as a customer-readable file. We still had to consider caching of error responses and range-request artefacts, but the previous failure mode (CDN serving a deleted archive) disappeared once the CDN was no longer the authority for plaintext delivery.

The key-destruction caveat

Cryptographic erasure is easy to overstate.

It works only if the key is genuinely unique to the data being erased and no recoverable copy survives. If a single data key encrypts ten thousand exports, destroying it removes all ten thousand or none. If the plaintext key appears in logs, traces, crash dumps, queues, or support tools, deleting the wrapped copy proves little. If the application can regenerate the same key from a stable secret, the key was never destroyed.

We therefore treated data keys as toxic values. They could not be logged, stored in job payloads, or returned through internal APIs. Unwrap permission belonged only to the streaming download task, not to the web application generally. KMS audit events alerted on unwraps outside the task role and on unusual volumes. Memory lifetime was short, though I do not pretend Go can guarantee that a garbage-collected byte slice is immediately overwritten everywhere it once existed.

The registry deletion itself needed the same honesty. Overwriting wrapped_data_key with an empty value ends application access at commit, but Postgres keeps the prior row version as a dead tuple, and a copy of the old value lives in the write-ahead log and any physical replica until vacuum reclaims the tuple and WAL segments recycle. So the wrapped key is unrecoverable through the application immediately, and physically gone once vacuum, WAL rotation, and registry-backup expiry have all passed. We treated that convergence window as part of the erasure schedule rather than assuming a single UPDATE scrubbed the bytes everywhere.

Backup interaction required explicit rules. Object-storage versioning and bucket backups could retain ciphertext after grant destruction. That is acceptable only while the wrapped key is gone and no backup of the grant registry restores it. We therefore backed up the grant registry with a shorter retention than the ciphertext bucket, and restore procedures refused to reintroduce revoked wrapped keys for subjects with verified erasure cases. Restoring ciphertext without restoring decryptability is an operational nuisance. Restoring decryptability after erasure is a compliance failure.

This is one of the trade-offs I kept explicit: cryptographic erasure gave us a strong operational boundary, not mathematical proof that no physical trace remained in volatile memory. The control depended on narrow key custody, auditability, and deletion of residual ciphertext in due course.

Encrypting data is common. Designing who can make it readable again is the security architecture.

Completion needed a state machine

Our original endpoint performed work synchronously and then returned success. That interface encouraged a lie. Distributed erasure can be initiated synchronously, but it cannot honestly be completed that way.

Some systems were temporarily unavailable. Replica lag varied. Storage deletion was asynchronous. A legal hold could block one representation while every other copy should still be purged. Retries had to be safe, and a partial run had to continue from known state after a deploy or outage.

I replaced the endpoint's Boolean idea of deletion with an erasure case:

CREATE TYPE erasure_state AS ENUM (
    'RECEIVED',
    'DISCOVERING',
    'REVOKING',
    'PURGING',
    'BLOCKED',
    'VERIFIED'
);
 
CREATE TABLE erasure_cases (
    case_id          UUID PRIMARY KEY,
    subject_ref      TEXT          NOT NULL,
    state            erasure_state NOT NULL,
    requested_at     TIMESTAMPTZ   NOT NULL,
    deadline_at      TIMESTAMPTZ   NOT NULL,
    verified_at      TIMESTAMPTZ,
    UNIQUE (subject_ref, requested_at)
);
 
CREATE TABLE erasure_evidence (
    case_id          UUID        NOT NULL REFERENCES erasure_cases(case_id),
    system_name      TEXT        NOT NULL,
    disposition      TEXT        NOT NULL,
    status           TEXT        NOT NULL,
    evidence_digest  TEXT        NOT NULL,
    observed_at      TIMESTAMPTZ NOT NULL,
    detail           JSONB       NOT NULL,
    PRIMARY KEY (case_id, system_name, disposition)
);

The API now returns 202 Accepted and a case ID. That is a small semantic change with a large honesty benefit. It says the request has entered a protocol, not that every downstream store changed before the HTTP connection closed.

Erasure is a state machine, not an endpoint

Each transition has an invariant.

DISCOVERING freezes the retention graph used by this case, so a changing inventory cannot make the denominator move while work is in progress. REVOKING cannot finish while any active export grant or unwrap-capable key remains. PURGING collects signed evidence from each required node. VERIFIED requires every node either to report a terminal disposition or to carry an approved retention exception.

BLOCKED is a first-class state, not an error log. A case enters it when a system cannot be reached, evidence conflicts, or an exception lacks authority. Blocked cases page the owner as their deadline approaches. They do not become verified because a retry budget expired.

Transition rules that refuse to flatter us

The worker uses an outbox so state changes and commands cannot separate:

func advanceToRevoking(ctx context.Context, tx pgx.Tx, c ErasureCase) error {
    tag, err := tx.Exec(ctx, `
        UPDATE erasure_cases
           SET state = 'REVOKING'
         WHERE case_id = $1
           AND state = 'DISCOVERING'`, c.ID)
    if err != nil {
        return err
    }
    if tag.RowsAffected() != 1 {
        return ErrInvalidTransition
    }
 
    _, err = tx.Exec(ctx, `
        INSERT INTO erasure_outbox (case_id, command, payload)
        VALUES ($1, 'REVOKE_EXPORT_ACCESS', $2)
        ON CONFLICT (case_id, command) DO NOTHING`,
        c.ID, c.SubjectRef)
    return err
}

The state update and outbox message commit together. A relay may publish the command more than once, so every consumer is idempotent. The export service can revoke an already revoked grant; the search service can delete an absent document; the support connector can return the same evidence digest for the same case.

This matters because exactly-once delivery is the wrong dependency for compliance work. The durable guarantee is at-least-once command delivery plus idempotent effects plus evidence that converges.

We added explicit guards against optimistic completion. A case could not enter VERIFIED if discovery found a node that later stopped responding. Silence after a positive discovery became BLOCKED, not success by omission. A case could not skip REVOKING because object deletion looked fast that day. Access revocation had to leave evidence before purge evidence was accepted. A case with a RETAIN_* disposition and a null authority field could not leave DISCOVERING.

Deadlines were treated as operational contracts. Each case carried deadline_at derived from policy, not from engineer optimism. Approaching deadlines escalated ownership. Missed deadlines remained visible as missed deadlines. The previous world had converted unanswered systems into unmarked success by simply never asking them.

Concurrency required care. Two erasure requests for the same subject could arrive close together, especially when support systems retried. The unique constraint on (subject_ref, requested_at) was insufficient alone. We added a rule that a new case for a subject already in flight joined the existing case unless counsel explicitly opened a separate matter. Joining prevented two workers from racing on the same grants and producing conflicting evidence digests.

func evidenceDigest(system, disposition string, result CanonicalResult) string {
    h := sha256.New()
    _, _ = io.WriteString(h, system)
    _, _ = io.WriteString(h, "\n")
    _, _ = io.WriteString(h, disposition)
    _, _ = io.WriteString(h, "\n")
    _, _ = h.Write(result.CanonicalJSON())
    return hex.EncodeToString(h.Sum(nil))
}

Canonical evidence mattered when retries returned equivalent outcomes with different JSON key ordering or timestamps. Without canonicalisation, the same purge looked like conflicting testimony. With it, repeated successful purges produced identical digests and the verifier could ignore duplicates.

Evidence without preserving the person

An erasure certificate can accidentally become another personal-data store. We needed to prove what was done without keeping the account payload we had just removed.

Evidence records contain the case ID, system, disposition, time, software version, count of affected records, and a digest of a canonical result. They do not contain names, email addresses, document numbers, message bodies, or object contents. The subject_ref remains restricted and can be rendered unlinkable by destroying its correlation key when no longer needed.

A verifier recomputes the case summary from the evidence rows and signs it with a key unavailable to deletion workers:

{
  "case_id": "ER-2026-0184",
  "policy_version": "retention-12",
  "inventory_version": "graph-2026-08-08.4",
  "requested_at": "2026-08-08T09:14:22Z",
  "verified_at": "2026-08-08T11:52:09Z",
  "nodes": {
    "operational": "PURGED",
    "exports": "KEY_DESTROYED_AND_PURGED",
    "support": "PURGED",
    "analytics": "ANONYMIZED",
    "backup": "RETAIN_UNTIL_2026-09-07"
  },
  "certificate_digest": "sha256:8c0b..."
}

The backup exception does not hide in prose. It is a machine-readable disposition with an expiry and authority. At that expiry, a second verification task confirms that the relevant backup generation has aged out. Until then, restoring that backup requires replaying completed erasure cases before the restored environment can serve traffic.

That restore rule was an important discovery. A backup can be compliant on the day of deletion and recreate deleted people months later during disaster recovery. Erasure therefore has to be part of the restore runbook, not only the live-system runbook.

The verifier key separation deserves emphasis. If the same workers that delete data can also mint the certificate that says deletion succeeded, the certificate measures process execution more than independent confirmation. Our verifier read evidence tables through a restricted role, recomputed digests, checked invariants, and signed with a key held outside the deletion task role. Compromising a deletion worker could still disrupt erasure. It could not quietly certify success for a case that lacked required evidence.

Completion is a claim that requires evidence from every place capable of contradicting it.

What we deliberately did not delete

The hardest discussions were not about technology. They were about records we had a reason to retain.

An indiscriminate purge can violate accounting, fraud-prevention, dispute, or legal-hold obligations. A system that promises to delete everything is often as poorly designed as one that deletes nothing. The useful target is purpose-bound retention: keep the minimum fields required for a named obligation, prevent their use for unrelated product activity, and remove them when that obligation expires.

We split operational identity from retained obligations. The customer profile, credentials, devices, support content, exports, and marketing history were purged. A narrow transaction record remained under a separate access role with no login credential, document image, free-text note, or reusable contact detail. The record kept the amount, currency, booking time, statutory category, and an internal case reference. Access required an audited compliance purpose.

The shape was closer to a sealed evidence box than a deactivated account. Product services could not query it. Support could not search it. Analytics did not receive it. The retention clock was attached to each class of record rather than to the account as a whole.

This was also where pseudonymization had to be described honestly. Replacing an account ID with HMAC(account_id) does not automatically make data anonymous. If we retain the key and can reconnect the record to the person, it remains personal data in practical terms. Pseudonymization lowers exposure and separates duties; anonymization requires that re-identification is no longer reasonably available.

We used both, for different purposes. Active legal obligations used pseudonymous references behind restricted keys. Product analytics used aggregation thresholds and removed the correlation key after the verification window, leaving statistics that could not be walked back to one account through our retained data.

Security logs followed a similar split. We retained authentication events and administrative actions needed for intrusion detection, but stripped free-text fields that had copied customer content into log lines. Several services had been logging entire request bodies "temporarily" for years. Temporary had become permanent. Erasure work forced a logging standard: identifiers and outcome codes were allowed; payloads were not. Where a security investigation needed content, it had to collect that content under an investigative hold before erasure, not mine application logs afterward.

Erasure is not maximal destruction. It is the removal of every use that no longer has a defensible purpose.

The tests that changed our confidence

Before the incident, our deletion tests asserted that selected database rows were gone. Those tests had passed for years.

The new suite starts from capabilities instead of tables. After erasure, can any supported path authenticate as the account, query its profile, search its documents, mint an export grant, unwrap an old export key, retrieve an archive, resolve a support attachment, or correlate an analytical event back to the subject?

One test preserves an export token before deletion and tries every delivery path afterward:

func TestErasureRevokesExistingExport(t *testing.T) {
    subject := fixtures.CreateSubject(t)
    token := exports.CreateAndAwait(t, subject)
    assertDownload(t, token, http.StatusOK)
 
    caseID := erasure.Request(t, subject)
    erasure.AwaitState(t, caseID, "VERIFIED")
 
    assertDownload(t, token, http.StatusGone)
    assertNoActiveGrant(t, subject)
    assertUnwrapDenied(t, token)
    assertObjectMissingOrUnusable(t, token)
}

Another restores a backup into an isolated environment, replays the erasure ledger, and confirms the subject remains absent before the environment is declared usable. A third deliberately makes one connector time out and checks that the case becomes BLOCKED, never VERIFIED. A fourth creates a legal hold and checks that only the authorised fields remain while every unrelated representation is removed.

We also added a canary export with a synthetic identity. A scheduled job deletes the canary, probes known access paths, and alerts if any representation survives beyond its objective. The canary catches drift that unit tests cannot: lifecycle rules changed in infrastructure, a CDN configuration replaced, a support connector losing permission, or a new prefix created without retention tags.

What the suite actually exercises

Capability tests forced us to keep a catalogue of access paths. Each path became a function that an erasure case had to defeat:

type AccessPath func(t *testing.T, subject Subject, prior PriorKnowledge) error
 
var postErasurePaths = []namedPath{
    {"password-login", tryPasswordLogin},
    {"session-resume", trySessionResume},
    {"profile-read", tryProfileRead},
    {"search-documents", trySearchDocuments},
    {"export-token-download", tryExportTokenDownload},
    {"kms-unwrap", tryDirectUnwrap},
    {"support-attachment", trySupportAttachment},
    {"analytics-join", tryAnalyticsJoin},
    {"backup-restore-without-replay", tryRawBackupRestore},
}

PriorKnowledge is intentional. The attacker after erasure may still know an old token, object key, email address, or ticket URL. Tests that start from a clean slate miss the incident we actually had. The suite therefore seeds prior knowledge before erasure and reuses it afterward.

The backup test deserves detail because it failed first in staging. Restoring a database backup reintroduced the account row. Restoring an object-storage backup reintroduced ciphertext. Replaying the erasure ledger revoked grants again and deleted operational rows again, but only if the restore runbook invoked the replay. Our first automation restored data and opened the environment to internal users before replay finished. The test caught that ordering bug by attempting product queries immediately after restore and expecting denial until replay evidence existed.

The blocked-connector test changed incident behaviour. Previously, a timed-out support purge would be retried quietly and, if retries exhausted, marked complete by a catch-all success handler. Now the case stays BLOCKED, the owner is paged near the deadline, and verification remains impossible until evidence arrives or an authorised exception is recorded. That feels worse in dashboards. It is better in reality.

We load-tested grant revocation for subjects with thousands of historical exports created under the old regime during migration. The envelope design helped here in a way worth stating precisely: because a single KMS wrapping key protected every per-archive data key, revocation never had to schedule thousands of individual KMS key deletions. It destroyed the locally stored wrapped data keys in the grant registry, which is what actually severs decryptability. That kept revocation independent of KMS availability and its per-key deletion rate limits. What still mattered was doing those local deletions in idempotent batches, so a subject with thousands of archives did not strand a case in REVOKING behind a long single-row loop.

The canary runs continuously. Every day it creates a synthetic subject, generates an export, places references in support and search fixtures, requests erasure, waits for VERIFIED, and then attacks itself with the path catalogue. Failures page the erasure owners, not a generic on-call rotation that lacks context. Continuous verification is what keeps the design honest after the incident memory fades.

We also kept a negative control: a subject that must remain readable because it sits under an active legal hold. The suite fails if that subject's authorised residual fields disappear, or if unauthorised fields survive. Erasure bugs include over-deletion. A control that only checks destruction will eventually train the system to destroy too much.

The result was not certainty. Distributed systems do not offer certainty cheaply, and data discovery is always vulnerable to an unknown copy. What changed was the nature of our confidence. We stopped proving that a function ran and started testing that the old capability no longer existed.

Putting the protocol into production

Design work is inexpensive compared with changing a live deletion path that customers, support, counsel, and auditors already rely on.

We did not flip the entire estate in one deploy. The rollout had four stages, each with an exit criterion that could fail without taking the old path offline prematurely.

Stage one instrumented the old deletion endpoint. For every successful 204, a shadow discovery job computed subject_ref, queried the emerging retention graph, and recorded how many additional representations still existed. The customer-facing behaviour did not change. Internally we learned the size of the lie. On a typical day, a "completed" deletion still had export grants, support references, or searchable documents for a substantial fraction of subjects. That measurement funded the rest of the work more effectively than architecture slides had.

Stage two introduced the grant registry and application-mediated downloads for new exports only. Old tokens continued to work under the previous model while we backfilled subject_ref tags and wrapped keys where we could. Where we could not wrap an old archive safely, we deleted it under the published export retention policy and notified support that historical links should be re-issued if still needed. Shrinking the set of immortal links was a security win even before the full state machine landed.

Stage three moved erasure requests onto the case API for staff-initiated deletions, while customer self-service continued to call a facade that created cases underneath. This let support learn the new statuses (DISCOVERING, REVOKING, BLOCKED, VERIFIED) with a smaller audience. We staffed a daily review of blocked cases. The first week surfaced missing connector permissions, incorrect object tags, and one analytics table that had never been declared as a retention node. Those were inventory defects, not exotic cryptographic failures.

Stage four made verification mandatory before any external completion signal. Customer notifications, regulator-facing timelines, and internal compliance dashboards all keyed off VERIFIED or an explicit authorised exception, never off HTTP acceptance. Only then did we retire the old synchronous success path.

During the facade period we kept a dual-write audit: every case creation also recorded what the old endpoint would have reported. The divergence was instructive. Cases that the old path would have called complete often spent hours in REVOKING or BLOCKED under the new path. Publishing both numbers side by side ended most arguments about whether the state machine was "making deletion slower" or making incomplete deletion visible.

Operational ownership had to change with the protocol. The account service team could no longer be the sole owner of "deletion" when deletion spanned storage, support, analytics, and backup. We assigned an erasure coordinator ownership model: a primary team for the state machine, named owners for each connector, and a privacy counsel contact for retention exceptions. Alerts routed to connector owners when their node blocked a case. That routing mattered. A generic page to platform on-call produced shrugs. A page that said "support attachment purge has not evidenced case ER-2026-0184 and deadline is in six hours" produced action.

We also changed how support talked about completion. The old macro said the account had been deleted. The new macro said the erasure request had been accepted and that verification would follow, with a case reference. Customers who needed a certificate received the verified summary after the state machine finished. Communicating uncertainty honestly reduced pressure to mark blocked work as done.

Runbooks had to grow with the code. The erasure on-call guide covered how to read a blocked case, how to distinguish a missing permission from a missing retention node, when counsel must approve a RETAIN_* authority, and how to abort a restore that skipped replay. Without those procedures, the state machine would have become another dashboard that engineers glanced at and ignored during incidents.

Migration of historical exports was the messiest stage. Some archives had no reliable account linkage in metadata and existed only as paths embedded in tickets. For those, we searched ticket text for export URL patterns, extracted object keys, tagged what we could, and deleted the remainder once the retention window justified it. Perfect archaeology was impossible. Reducing reconstructability for the population we could identify was still progress, and the canary thereafter guarded against regressing into immortal new exports.

The cost of an honest delete button

The new design was slower and more expensive.

Every export required a unique key operation and an application-mediated download rather than a cheap redirect to object storage. The retention graph added metadata to systems that had never shared an identity concept. Support integration needed custom fields and purge permissions. The erasure coordinator became another critical workflow to operate, with deadlines, retries, ownership, and alerts.

Short-lived, single-use grants frustrated customers who opened the email on one device and tried to download on another after the first browser had consumed the token. We changed the experience to allow a small number of resumable range requests bound to one authenticated session, rather than pretending one HTTP request always equals one human download.

Cryptographic erasure complicated incident response because destroying a key also removes our ability to inspect the archive later. That is the intended property, but it means security investigations must place an explicit, authorised hold before deletion, not discover after the fact that evidence vanished correctly.

The state machine introduced visible blocked cases. Our completion rate initially looked worse because the old metric counted accepted requests as completed requests. The system had not become less compliant. The dashboard had become less flattering and more truthful.

Latency budgets shifted. Customers who previously received an immediate "deleted" confirmation now waited for verification. Median verification settled into a range measured in tens of minutes when all connectors were healthy. Tail latency tracked the slowest dependency, usually support or analytics. We published that distribution internally so support could set expectations. Hiding the wait had been easy when the wait was a fiction.

Cost showed up in KMS requests, in streaming egress through the application tier, and in engineering time for connectors. The application-mediated download path also meant the delivery service became an availability dependency for legitimate exports. We accepted that coupling. A download path that can ignore erasure state is cheaper until the day it reconstructs a deleted person.

There is also a boundary I cannot eliminate with architecture. If someone downloaded an export to a personal device while authorised, our erasure workflow cannot reach that independent copy. We can restrict access, watermark archives, record downloads, train staff, and prohibit unauthorised local storage. We cannot honestly claim remote deletion of bytes outside systems we control. The retention graph marks that boundary rather than hiding it.

These costs are the price of making the promise precise. A delete button that only removes the easiest rows is cheap because someone else pays for the ambiguity later.

Boundaries the design still accepts

No retention protocol I trust claims total knowledge of every copy. The useful claim is narrower: within the systems we operate and the vendors we integrate, erasure follows a named graph, revokes readability quickly, collects evidence, and refuses to certify completion when evidence is missing.

Unknown shadow copies remain possible. An engineer can still copy a file to a laptop. A vendor can retain logs under their own policy. A law-enforcement process can compel a snapshot we must preserve. The design responds to those realities with controls and language, not with omnipotence. Endpoint protection, DLP rules, contractual deletion terms, and legal process workflows sit beside the erasure state machine. They are adjacent controls. Pretending the state machine replaces them would recreate the original mistake at a higher layer of abstraction.

Vendor platforms forced compromises. Some support tools offered purge APIs that removed customer-visible content while retaining internal indexes for a vendor-defined period. We recorded those as timed dispositions with authority and expiry, then verified again when the vendor window closed. Where a vendor could not evidence deletion to our standard, we stopped placing export tokens and document images into that vendor. Reducing what we send is sometimes the only honest remediation.

Replay after restore remains an operational hazard. Automation helps, but a panicked manual restore can still skip steps. We added a hard gate: restored environments present as read-only until the erasure replay reports completion or an incident commander records an explicit override with expiry. Overrides page privacy counsel. That gate has been unpopular during drills and correct during drills.

Cryptographic erasure also has a jurisdictional and durability boundary. If a wrapped key is destroyed and later a court orders production of the archive, we may be unable to comply because the plaintext is gone. That is a feature for customer erasure and a constraint for evidence preservation. The hold-before-delete rule exists for that reason. Teams that need investigative retention must act before revocation, through the sealed obligation store, not after.

Finally, the protocol cannot repair a promise made in product copy that exceeds what the systems can do. If marketing says "we delete everything immediately," engineering will be pressured to emit immediate success. We changed the external language to match the state machine: request accepted, verification follows, residual legal records may remain under stated authorities. Aligning speech with mechanism is part of the control. Divergent speech recreates the incentive to lie at the HTTP boundary.

Principles I kept

Delete a subject, not a row. The unit of erasure is every reachable representation tied to one person, including generated files, support tools, caches, replicas, and restore paths.

A short-lived child credential is not safe when its parent never expires. Inspect every mechanism capable of minting fresh access, not only the lifetime printed on the final URL.

Revoke readability before waiting for physical cleanup. A per-object data key lets access end while asynchronous deletion, replication, and backup expiry converge.

Keep correlation separate from identity. A keyed retention reference helps systems find related records without distributing the production account identifier everywhere.

Treat exceptions as data. Every retained representation needs a named authority, limited fields, restricted purpose, owner, and expiry. Free-text justification is where permanent retention hides.

Make blocked work visible. A timeout is not evidence of deletion. A state machine should refuse to call the case complete when one system cannot prove its terminal state.

Test the vanished capability. Row-count assertions prove an implementation detail. Try the old token, old URL, restored backup, support search, and analytical join.

Do not turn proof of erasure into a new identity archive. Evidence should describe actions and outcomes without preserving the payload that was removed.

Ship deletion as a staged protocol, not a flag flip. Shadow measurement, new grants, staff-facing cases, and mandatory verification each caught different classes of failure. The incident was caused by a narrow success signal. The remediation had to grow the signal before trusting it.

Write the external promise to match the weakest honest state. If verification can be blocked, customer language cannot imply instantaneous completion. Precision in speech protects the integrity of the control.

The old system answered a narrow question: did the account service delete its records?

The corrected system answers the question the customer was actually asking: can this organisation still use, disclose, or reconstruct me?

At 09:14, an endpoint can only begin that answer.

References

• GDPR Article 5: principles relating to processing of personal data
• GDPR Article 17: right to erasure
• NIST Privacy Framework
• NIST SP 800-88 Rev. 1: Guidelines for Media Sanitization
• OWASP Cryptographic Storage Cheat Sheet
• OWASP User Privacy Protection Cheat Sheet
• AWS Prescriptive Guidance: data perimeters and presigned URLs
• RFC 9457: Problem Details for HTTP APIs