Database migrations are one of the highest-risk operations in software engineering. Done correctly, they are invisible — users notice nothing, data integrity is preserved, and the deployment is forgotten by the following morning. Done incorrectly, they produce the kind of incident that gets discussed in retrospectives for years.

We have been called in to help recover from database migrations gone wrong more times than we care to count. Every single time, the root cause traces back to the same pattern: someone skipped at least one step on this checklist.

Save this post. Share it with your engineering team. Run through it before every migration, regardless of how simple the change looks.

Why Database Migrations Go Wrong

The most dangerous migrations are not the complex ones. The most dangerous migrations are the ones that look simple.

A simple column rename. Adding a non-nullable field. Dropping a table that “nobody uses anymore.” Changing a data type from integer to bigint. These are the migrations that skip review, skip testing, and skip preparation — because they seem too straightforward to warrant it.

They are also the migrations most likely to cause extended downtime, data loss, or cascading failures that take hours to diagnose and recover from.

The checklist below applies to every migration. Simple or complex. Small table or large. Green field or legacy system. The steps do not change based on your confidence that this particular migration is “probably fine.”

The 8-Step Database Migration Checklist

1. Verified Backup — Not Assumed

Do not assume last night’s backup ran successfully. Verify it. This means confirming that the backup job completed, checking the backup file size against historical norms, and — critically — performing a test restore from the backup to a separate environment.

An untested backup is not a backup. It is hope.

This distinction matters because backup jobs fail silently. A misconfigured backup schedule, a full disk, a permission error, an expired credential — any of these can cause backup jobs to fail without triggering an alert. If your first test of a backup is during an active recovery from a failed migration, you have compounded one incident into two.

Before every migration: verify the backup exists, verify the restore works, and document when the verification was performed.

2. Dry Run on a Production Clone

Run the full migration against a copy of production data before running it against production itself. Not against your staging environment. Not against your development database. Against a clone of production.

This distinction is critical. Staging and development environments almost never contain representative production data. They lack the edge cases, the inconsistent historical records, the orphaned rows, the unexpected null values, and the sheer volume that production carries. A migration that completes in thirty seconds against fifty thousand staging rows can fail at four hours against fifty million production rows — for reasons that would have been immediately obvious on a production clone.

The dry run serves three purposes: it confirms the migration SQL is syntactically correct and logically sound, it surfaces data quality issues that will cause the migration to fail or produce incorrect results, and it gives you an accurate timing estimate for step four.

3. Written Rollback Plan

Before any migration runs, document exactly what you will do if it needs to be rolled back. This documentation must exist in written form — not in the head of the engineer running the migration, not as a verbal agreement, not as a plan you will figure out if something goes wrong.

The rollback plan should specify: the exact commands to execute, the order in which to execute them, who has the database access required to execute them, how long the rollback will take based on your dry run, and what the acceptance criteria are for a successful rollback.

If you discover during a failed migration that your rollback takes four hours, your team needs to have known that before the migration started — not when they are managing an active production incident at 2am.

4. Migration Timing Estimation

Every migration has a runtime. Know yours before the maintenance window begins.

Use the production clone from step two to time the migration accurately. Record the row count of affected tables, the total data volume, and the elapsed time. Extrapolate if your clone is a partial sample. Add a safety margin — production is never quite the same as a clone, and contention from live traffic will slow write operations.

Timing estimation matters for maintenance window planning, for customer communication, and for the rollback decision threshold. If your migration is expected to take forty-five minutes and it has been running for three hours, you need a defined threshold at which the team escalates to rollback rather than continuing to wait.

Define that threshold before you start. Not during.

5. Maintenance Window or Zero-Downtime Strategy

Every migration requires either a maintenance window or a zero-downtime strategy. There is no third option.

If your migration requires application downtime — because it involves a lock that blocks reads, or because it changes a schema that live application code cannot handle in its current state — plan the maintenance window explicitly. Define the start time, the expected end time, the rollback threshold, and who communicates status to affected users. Get approval from the appropriate stakeholders before the window begins.

If your migration cannot tolerate downtime, implement a zero-downtime strategy. Common patterns include the expand/contract approach (add new columns while keeping old ones, migrate data in batches, switch application code, then drop old columns), feature flags to gate new schema-dependent code paths, and shadow write patterns where data is written to both old and new schema simultaneously during the transition.

Choosing the wrong strategy — attempting a zero-downtime migration with code that requires a maintenance window, or taking unplanned downtime on a migration you thought would be online — is the single most common cause of extended production incidents during database migrations.

6. Monitoring and Alerting Live Before You Start

Have your monitoring dashboards open and your alerting configured before you execute the first migration command. Do not wait until you suspect something is wrong to open your monitoring tools.

The metrics to watch during a database migration include: error rates on application endpoints that touch the affected tables, database query execution times and lock wait times, replication lag if you are running read replicas, disk I/O and disk space consumption (large migrations generate significant write amplification), and application memory usage if your migration involves large result sets.

The difference between catching a problem at one minute versus ten minutes can be the difference between a brief blip and a multi-hour incident. Monitoring that you check after you suspect something is wrong is not monitoring. It is forensics.

7. Team Communication Plan

Before the migration starts, every team member involved should know the answers to these questions: Who is the primary engineer executing the migration? Who is the on-call escalation if the primary engineer needs support? Who has database access to execute a rollback? Who makes the decision to roll back, and at what threshold? Who communicates status to the business, and through what channel?

These questions feel obvious. They are not obvious at 2am during an active incident when the primary engineer is debugging a lock contention issue, the rollback decision is time-sensitive, and nobody is sure who has the database credentials to execute it.

Document the answers before the migration starts. Share the document with everyone in the migration channel. Confirm receipt.

8. Post-Migration Validation Queries

Write your validation queries before the migration runs, not after it completes.

Validation queries written after a successful migration are shaped by the assumption that the migration succeeded. Validation queries written before the migration are shaped by what you are actually trying to verify. These are not the same queries.

Your validation suite should include: row count comparisons between the state before migration and after, referential integrity checks on foreign key relationships, spot checks on specific records that represent critical business data, and functional checks that confirm the application behaves correctly against the migrated schema.

Run these queries immediately after migration completes. A migration that finishes without errors is not necessarily a successful migration. A migration where your validation suite passes is a successful migration.

The Pattern Behind Every Migration Incident

We have never been called to help recover from a database migration where the team ran through all eight steps and something still went catastrophically wrong. We have been called to recover from migrations where teams skipped one step — sometimes just one — and paid for it.

The conversation is always the same: “We thought it would be fine.”

It was not fine.

The migration checklist is not overhead. It is the difference between a migration that is boring and forgotten by the following morning, and a migration that becomes the case study in your next engineering retrospective.

“An untested backup is not a backup. It is hope. And hope is not an engineering strategy.”

How Onclick Innovations Handles Database Migrations

At Onclick Innovations, we treat every database migration as a production incident waiting to happen — until our checklist proves otherwise. This is not pessimism. It is the engineering discipline that has allowed us to migrate hundreds of production databases across 350+ projects without a single data loss incident.

Every migration we execute includes a verified backup, a dry run on a production clone, a written rollback plan with defined thresholds, an accurate timing estimate, a defined downtime or zero-downtime strategy, pre-configured monitoring, a team communication plan, and a pre-written validation suite.

For clients managing their own migrations, we provide migration review as part of our engagement — reviewing the migration plan, the rollback strategy, and the validation approach before the first command runs.

If you are planning a database migration and want a second set of experienced eyes on your approach, we are happy to help.

📩 Get in touch → www.onclickinnovations.com
📍 Based in Mohali, India · Serving clients globally across 10+ countries

Frequently Asked Questions

What is a database migration?

A database migration is any change to the structure or content of a production database — including adding or removing columns, changing data types, renaming tables, adding indexes, migrating data between tables or schemas, and upgrading database versions. Migrations range from trivial to extremely complex, but all carry risk when executed against live production data.

What is a production clone and why do I need one?

A production clone is an identical copy of your production database, created specifically for migration testing. It contains the same schema, the same data volume, and representative examples of the edge cases that exist in your live data. Running a migration dry run against a production clone gives you an accurate preview of how the migration will behave in production — including timing, any data quality issues, and potential failure modes that would never surface in staging or development.

What is the expand/contract pattern for zero-downtime migrations?

The expand/contract pattern is a technique for making breaking schema changes without application downtime. It involves three phases: expand (add the new schema element while keeping the old one, deploy application code that writes to both), contract phase one (migrate existing data to the new schema, verify, stop writing to the old schema), and contract phase two (remove the old schema element). Each phase is a separate deployment with validation between them, allowing the migration to proceed without any single deployment requiring downtime.

How long should I keep a backup before running a migration?

You should take a fresh backup immediately before running any production migration, regardless of your regular backup schedule. This backup should be verified by performing a test restore before the migration begins. Keep this pre-migration backup for at least 30 days after the migration completes, or longer if your business has regulatory requirements around data retention.

Can Onclick Innovations review our migration plan before we execute it?

Yes. We offer migration review as part of our engineering services — reviewing your migration SQL, rollback strategy, timing estimates, and validation approach before you touch production. Contact us at onclickinnovations.com to discuss your requirements.

Share

The post The Database Migration Checklist: 8 Things You Must Do Before Touching Production Data appeared first on Blog.