How expired data is deleted

A retention declaration says how long a table keeps a record. This page explains what the engine does with that declaration each night, and what happens to the images the deleted records referred to.

Read Jobs and retention for the declaration itself.

The rule is a condition of the statement, not a script

The compiler turns every retention into a condition and puts it in the model. The engine puts that condition into the statement that deletes the records. No JavaScript runs, no hook is called, and no context is handed to anything.

A legal obligation must not depend on a script being correct. A script fails when it throws, when its time budget trips, and when somebody writes > where they meant <. It fails at 03:00, where nobody is looking.

One run a night, in batches of a thousand

The engine gives every organization one occurrence, mtd_retention, at 03:00 in the application’s time zone. It does two things in this order and never the other:

1. every retention rule of every table, 1,000 records per transaction.
2. the removal of images that no record refers to any more.

The order is the point. The records go first, so the images they were the last reference to are already unreferenced when the second step asks.

Batches exist so that your organization stays usable. An organization has one writer. One statement over 100,000 records would hold that writer past the statement timeout, and every form submitted during the run would fail. A hundred short transactions leave the writer free between them, and a submission that arrives mid-run is served.

A run has four seconds. It stops starting new batches a second before that, so the second step and the clock advance are not cut off. A run that reaches the limit stops between batches, records that it was incomplete, and continues the next night. That is not a failure: the condition does not match what has already gone, so an interrupted run has done a whole number of batches of work.

The day the run counts from is the occurrence’s own day, in the application’s time zone. Every batch of one run therefore agrees about the date, even when the run crosses midnight.

An image outlives the record that shared it

An image is stored once, by the digest of its content, in a directory beside the database. Two records that hold the same image hold the same stored file. Deleting one of them must not remove the bytes.

So the engine does not count references. It derives the answer from the model instead: exactly one kind of column holds a digest, so “which images are still referred to” is a query over every such column of every table. A query the model can write is a query no write path can forget.

Each night the engine lists the stored files, compares them with that set, and removes what is left over. A file goes only when no record of any table refers to it.

Nothing younger than 24 hours is removed

A stored file is protected for its first 24 hours, counted from the newest record about it.

The floor buys two things. An upload that has been staged but not committed cannot be removed under the request that is making it. And an operator who notices a wrong removal has a day in which the bytes are still there.

An image nobody refers to does not count against the quota

The storage quota counts only images that a record still holds. Replacing an image or clearing it frees its space at the save, not when the file is removed at night. So you can replace an image at a full quota with one of the same size or smaller.

A Markdown image does not keep an image alive. It shows the image of a record, and it shows nothing once that record holds a different image.

The disk still has a limit. Every stored file, including replaced images that have not yet been removed, may use at most twice the quota. An upload over that limit is refused with the sentence “Too many replaced images are waiting to be removed tonight. Try again tomorrow.” An organization reaches it only when it replaces more than its whole quota within about two days.

If an application of the organization did not load, the engine cannot read its image columns. The quota then counts every stored file, because an unreadable image must not look free.

Nothing is removed at all when an application did not open

If any application installed in the organization failed to load, the engine removes no images that night and reports why.

The reason is the one case in which this could destroy data. An application whose model did not open has no readable columns, so every image it owns looks unreferenced, and the next statement would unlink them. Image bytes are the one thing here that cannot be put back.

The record deletions still happen. Only the image removal is skipped.

A crash leaves a record without its file, never a file without its record

The removal of the bytes happens inside the transaction, before it commits. An organization has one writer, so while the removal holds it, nothing can attach a digest that was just decided unreferenced.

A process that dies halfway leaves a stored fact whose file is gone. That is the direction that can be repaired: the next run finds the digest still unreferenced and finishes the work. The other direction would leave bytes that nothing knows about.

What a run records

Each run writes one job.run audit event with the outcome, the duration, the number of records written and the number of images removed. A failed run writes one too, carrying its message, and carrying the number of records it managed before it stopped. “It deleted four thousand records and then failed” and “it did nothing” are different nights.

The administration page shows the same facts per occurrence. Read Read the job log.