Linear's TRUNCATE CASCADE Incident: A Postmortem Analysis
Per Linear's postmortem, a January 24, 2024 migration ran TRUNCATE TABLE with CASCADE on a new table, which also emptied every table with foreign keys to it. Caching hid the loss for about 30 minutes. Linear restored from a full backup, replayed changes from its sync log, and recovered over 99 percent of data within 36 hours.
What happened, per the postmortem
Linear's postmortem describes a migration intended to delete test data from a new table using TRUNCATE TABLE <new_table> CASCADE;. The CASCADE option truncates the entire contents of any table with foreign keys to the target. As a result, production data was deleted across issues, comments, notifications, favourites, and reactions.
The loss went undetected for about 30 minutes, because Linear stores most workspace data in a local client cache and backend caching masked the deletions. Errors surfaced when caches refreshed and users tried to make changes.
The team restored the database from a full backup taken at 04:47 UTC, about two hours before the incident, with around one hour of platform downtime. Point-in-time recovery was not used because the team had never tested it, while full-backup restore was a frequently tested procedure. Data changed after the backup was recovered by replaying user, integration, API, and automation actions from Linear's sync packets. According to the postmortem, 12 percent of workspaces had data unavailable, a further 7 percent had automated changes unavailable, and over 99 percent of data was restored within 36 hours.
The migration: small keyword, large blast radius
The statement looks harmless in review: truncate a new, empty-looking table. The danger is entirely in CASCADE, whose effect depends on the foreign key graph at run time, which is not visible in the diff.
This is the general problem with destructive SQL in migrations. Reviewers read what the statement says, not what the database will do given current schema and data. Defences that do not depend on reviewer attention are stronger:
- Remove the privilege. Linear's first remediation was removing TRUNCATE privileges from production database users. Migrations that need it can use a separate, audited path.
- Lint migrations for destructive operations, including TRUNCATE, DROP, CASCADE, and unbounded DELETE, and require explicit approval.
- Run migrations against a production-shaped staging database and compare row counts before and after. Linear listed automated staging testing among its actions.
Detection: caches hide data loss
Caching is designed to keep serving data when the backend is slow or unavailable. The same property hides a backend that has lost data. The 30-minute gap in Linear's case came from client and backend caches continuing to show data that no longer existed.
Data integrity monitoring closes that gap: row counts on core tables, sudden drops in record volumes, and checks that referenced records exist. Linear listed improved data integrity monitoring among its remediations.
Recovery: tested beats theoretically better
Point-in-time recovery would have narrowed the data gap, but the team chose the procedure it had practised. That is a sound call under pressure, and the lesson is not that they chose wrong. It is that an untested recovery path is not a recovery path. Linear's remediations include building point-in-time recovery tooling.
The near-complete recovery was possible because Linear keeps an application-level log of changes, its sync packets, which could be replayed onto the restored database. Many systems have something similar, such as event logs, audit trails, or message queues, and few have practised using them for recovery. Replay also has general limits worth planning for: a recorded change can fail to apply when it depends on a record that no longer exists or conflicts with the restored state, so recovery tooling needs a way to surface and resolve those cases rather than silently dropping them.
Questions for your own systems
- Which database users can TRUNCATE, DROP, or run CASCADE operations in production, and why?
- Would you notice a large drop in row counts on a core table within minutes?
- When did you last restore to a point in time, not just from a full backup, and how long did it take?
- Do you have an application-level change log that could replay recent writes after a restore?
- Can you put the application into a read-only mode during a data incident? Linear added this to its plans.
Frequently asked questions
- What caused Linear's January 2024 data loss?
- According to Linear's postmortem, a migration ran TRUNCATE TABLE with the CASCADE option on a new table to remove test data. CASCADE also truncated every table with foreign keys to that table, deleting production data including issues, comments, notifications, favourites, and reactions.
- What does TRUNCATE CASCADE do in PostgreSQL?
- TRUNCATE removes all rows from a table. With CASCADE, it also truncates all tables that have foreign key references to the target table, and tables referencing those, and so on. The effect depends on the current schema, so a statement that looks narrow can empty many tables.
- How did Linear recover the data?
- Linear restored its database from a full backup taken about two hours before the incident, then replayed user, integration, API, and automation actions recorded in its sync packets. According to the postmortem, over 99 percent of data was restored within 36 hours, with about one hour of platform downtime.
- How do you prevent destructive migrations?
- Remove destructive privileges such as TRUNCATE and DROP from routine production database users, lint migrations for destructive statements and CASCADE, require explicit approval for them, run migrations against a production-shaped staging database with row-count checks, and keep a tested restore path, including point-in-time recovery, ready for when prevention fails.