The reason usually isn’t incompetence. It’s ownership. Engineering optimises for shipping features and keeping things up. Finance sees a single monthly invoice with no visibility into what drives it. In between, nobody is specifically accountable for the number going up.

The good news is that most cloud waste comes from a small, predictable set of causes. This article walks through the six most common ones we see, what each typically costs, and roughly what fixing it is worth. It’s written for founders and CTOs — you shouldn’t need to be a DevOps engineer to follow it.

One caveat before we start: every percentage below is a typical range, not a guarantee. Real savings depend entirely on your architecture and workload. Treat these as places to look, not promises.

1. Servers Running at 12% CPU

This is the single most common source of waste, and it has an entirely human explanation.

Early in a project, someone has to pick an instance size. There’s no production traffic data yet, the cost of being under-provisioned (an outage) feels far worse than the cost of being over-provisioned (a slightly bigger bill), so they round up “to be safe.” That’s a reasonable decision at the time.

The problem is that nobody ever revisits it. Two years later, that instance is still running, still oversized, and still billing every hour.

How to check

In CloudWatch, look at average CPU utilisation across 30 days for your EC2 instances and RDS databases. Sustained utilisation below 20% is a strong signal you’re paying for roughly four times the machine you need. AWS Compute Optimizer will also generate right-sizing recommendations automatically, and it’s free.

A word of caution: CPU isn’t the whole picture. Some workloads are memory-bound or I/O-bound and will show low CPU while genuinely needing the instance. Check memory and network metrics before downsizing anything, and change one instance at a time.

Typical saving: 30–50% of EC2 spend.

2. Paying On-Demand Prices for Predictable Workloads

On-demand pricing exists so you can spin up capacity instantly without commitment. You pay a premium for that flexibility, and for genuinely unpredictable workloads, it’s worth it.

But your production database isn’t unpredictable. Neither is your main application server. These run continuously, every hour of every day, and they will still be running next year. Paying a flexibility premium for something completely predictable is pure waste.

AWS offers two main alternatives for committed usage:

• Savings Plans — you commit to a consistent dollar amount of compute usage per hour for one or three years, in exchange for significantly lower rates. Compute Savings Plans are the flexible option, applying across instance families, regions, and even Fargate and Lambda.
• Reserved Instances — a more specific commitment tied to instance attributes, still widely used for RDS and other services.

AWS advertises discounts of up to roughly 72% for the deepest three-year, all-upfront commitments. Most companies won’t hit that ceiling, but even a one-year, no-upfront Compute Savings Plan typically delivers meaningful double-digit savings with very little downside.

There’s also Spot Instances — spare AWS capacity at discounts that can reach around 90%, with the catch that AWS can reclaim the capacity with two minutes’ notice. Never use Spot for your primary database. It’s excellent for batch processing, CI/CD runners, data pipelines, and any fault-tolerant workload that can be interrupted and resumed.

Typical saving: 20–40% of compute spend.

3. Development Environments Running All Night

Your team works something like 45 to 50 hours a week. Your development and staging environments bill for all 168.

That means for roughly 70% of every week, you’re paying full price for environments with nobody logged into them. Unlike production, these environments have no uptime requirement whatsoever. If staging is down at 3am on a Sunday, nothing happens.

How to fix it

A scheduler that stops non-production instances outside working hours is genuinely a one-afternoon piece of work. AWS Instance Scheduler is a supported solution for this, but a simple Lambda function on an EventBridge schedule works just as well — stop instances at 8pm, start them at 8am, skip weekends.

The main objection is usually “but what if someone needs it at night?” In practice, giving the team a self-service way to start an environment on demand solves this completely, and the exceptions are rare enough that the savings hold.

Typical saving: 65–70% of non-production spend.

4. Storage for Servers That No Longer Exist

This is the most frustrating category, because you get absolutely nothing in return for the money.

When you terminate an EC2 instance, its attached EBS volume doesn’t always go with it — depending on how it was configured, the volume can survive, unattached to anything, billing every month indefinitely. The same happens with:

• Orphaned EBS snapshots. Teams take snapshots before risky changes and never clean them up. Years of them accumulate.
• Unattached Elastic IPs. AWS charges for Elastic IP addresses that aren’t associated with a running instance.
• Idle load balancers. A load balancer left behind after a service was decommissioned bills an hourly rate for doing nothing.
• Empty or unused NAT Gateways. These carry an hourly charge regardless of whether traffic flows through them.

None of this is serving a user. None of it is supporting a workload. It’s abandoned infrastructure that nobody remembered to delete.

AWS Trusted Advisor flags several of these categories directly. A quarterly cleanup review is usually enough to keep it under control. For most mid-sized companies this recovers a few hundred dollars a month; for larger or older accounts, it can be dramatically more.

Typical saving: highly variable, but always pure profit — nothing is lost by removing it.

5. Data Transfer Costs You Can’t See

This is the sneakiest category on the list, because the charges don’t appear next to the resources causing them.

Two things dominate here:

NAT Gateway processing charges

NAT Gateways charge both an hourly rate and a per-gigabyte data processing fee for everything passing through them. If your services in private subnets are pulling large amounts of data from the internet — container images on every deployment, package downloads, external API calls — that processing fee accumulates quietly.

The common fix is VPC Endpoints. Traffic to services like S3 and DynamoDB can route through an endpoint instead of the NAT Gateway, avoiding the processing charge entirely for that traffic.

Cross-availability-zone traffic

AWS charges for data moving between availability zones inside the same region — in both directions. This is the one that catches teams out.

If you’ve split a chatty set of microservices across multiple AZs for resilience, every internal call between them may now be a billable cross-AZ transfer. You did the architecturally responsible thing and got a surprise line item for it.

The answer isn’t to abandon multi-AZ redundancy — that’s there for good reason. It’s to be deliberate about which services genuinely need to talk across zones, and to keep high-volume chatty communication zone-local where availability requirements allow.

Typical impact: often 5–15% of the total bill, and almost always underestimated.

6. Logs and Backups You Will Never Read

By default, CloudWatch log groups retain data indefinitely. Not for 30 days, not for a year — forever, unless someone explicitly sets a retention policy.

That means many companies are paying premium storage rates to keep application logs from years ago that no human will ever open. The same applies to S3 buckets without lifecycle rules, where data that’s been untouched for years still sits in the most expensive storage class.

The two fixes

• Set CloudWatch retention policies. Decide how long logs are actually useful — 30, 60, or 90 days for most application logs — and configure it. Anything you need for compliance can be exported to cheaper storage first.
• Use S3 lifecycle policies and storage classes. S3 offers a range of tiers, from Standard down to Glacier Deep Archive, at dramatically different price points. S3 Intelligent-Tiering will move objects between access tiers automatically based on usage, which is a reasonable default when access patterns are unpredictable.

Both are configured once and keep saving money indefinitely, with no ongoing effort.

Typical saving: modest as a percentage, but permanent and effortless.

Where to Actually Start

If you do nothing else, do this: open AWS Cost Explorer, group your spend by service, and look at your top three lines.

Cloud bills follow a Pareto pattern almost universally. A small number of services account for the overwhelming majority of the cost. Optimising a service that represents 2% of your bill is a poor use of engineering time, no matter how inefficient it is. Start where the money actually is.

Three practices matter more than any individual optimisation:

• Tag everything. Without resource tags for environment, team, and project, you can’t attribute cost to anything. Cost allocation tags turn an opaque invoice into an actionable breakdown.
• Set budget alerts. AWS Budgets can notify you when spend crosses a threshold or is forecast to. Finding out about a cost spike on the 3rd rather than the 30th is the difference between a small problem and a large one.
• Give the bill an owner. Not a committee — a named person who reviews it monthly and is expected to explain changes. This single organisational change tends to outperform any technical fix.

Cloud spend isn’t a technical problem. It’s an ownership problem.

A Note on Over-Optimising

It’s worth saying the obvious counterpoint: cost optimisation has diminishing returns, and engineering time isn’t free.

If your monthly AWS bill is $800, spending three engineer-weeks to save 20% is a bad trade. If it’s $80,000, the same effort is obviously worth it. And some spending that looks wasteful is actually buying you something real — multi-AZ redundancy costs more and is usually correct; over-provisioned capacity ahead of a known traffic event is prudent, not careless.

The goal isn’t the lowest possible bill. It’s a bill where every line item is a decision someone made on purpose.

How We Approach This at Onclick Innovations

We build and maintain cloud infrastructure for clients across fintech, healthcare, e-commerce and SaaS, and cost efficiency is something we treat as an architectural concern from the start rather than a cleanup exercise later. Right-sizing, environment scheduling, storage lifecycle rules and sensible tagging are far cheaper to build in at the beginning than to retrofit onto a system that’s already running in production.

Frequently Asked Questions

How much are most companies overspending on AWS?
Industry analyses of cloud spending consistently estimate that roughly 30% of cloud spend is wasted across the market. The figure for any individual company varies widely depending on architecture, workload predictability, and whether anyone actively reviews the bill.

What’s the fastest way to reduce an AWS bill?
Usually two things: right-sizing over-provisioned instances, and scheduling non-production environments to shut down outside working hours. Both are relatively quick to implement and carry low risk compared to architectural changes.

What’s the difference between Savings Plans and Reserved Instances?
Savings Plans commit you to a consistent hourly dollar amount of compute usage and apply flexibly across instance families, regions, and services including Fargate and Lambda. Reserved Instances are tied more specifically to instance attributes. Savings Plans are generally the more flexible option for EC2 compute; Reserved Instances remain common for services like RDS.

Are Spot Instances safe to use?
For the right workloads, yes. AWS can reclaim Spot capacity with two minutes’ notice, so they should never run your primary database or anything that can’t tolerate interruption. They’re well suited to batch jobs, CI/CD runners, data processing, and other fault-tolerant workloads.

Why is AWS data transfer so expensive?
Data transfer charges come from several sources, most commonly NAT Gateway per-gigabyte processing fees and cross-availability-zone traffic within a region. They’re easy to miss because the charges don’t appear alongside the resources generating them. VPC Endpoints and keeping high-volume internal traffic zone-local are the usual mitigations.

What tools does AWS provide for cost management?
Cost Explorer for analysing spend, AWS Budgets for alerts and forecasting, Compute Optimizer for right-sizing recommendations, and Trusted Advisor for flagging idle and unused resources. All are available within the AWS console.

Share

The post 6 Things Silently Inflating Your AWS Bill — And What Each One Is Actually Worth appeared first on Blog.

Database migrations are one of the highest-risk operations in software engineering. Done correctly, they are invisible — users notice nothing, data integrity is preserved, and the deployment is forgotten by the following morning. Done incorrectly, they produce the kind of incident that gets discussed in retrospectives for years.

We have been called in to help recover from database migrations gone wrong more times than we care to count. Every single time, the root cause traces back to the same pattern: someone skipped at least one step on this checklist.

Save this post. Share it with your engineering team. Run through it before every migration, regardless of how simple the change looks.

Why Database Migrations Go Wrong

The most dangerous migrations are not the complex ones. The most dangerous migrations are the ones that look simple.

A simple column rename. Adding a non-nullable field. Dropping a table that “nobody uses anymore.” Changing a data type from integer to bigint. These are the migrations that skip review, skip testing, and skip preparation — because they seem too straightforward to warrant it.

They are also the migrations most likely to cause extended downtime, data loss, or cascading failures that take hours to diagnose and recover from.

The checklist below applies to every migration. Simple or complex. Small table or large. Green field or legacy system. The steps do not change based on your confidence that this particular migration is “probably fine.”

The 8-Step Database Migration Checklist

1. Verified Backup — Not Assumed

Do not assume last night’s backup ran successfully. Verify it. This means confirming that the backup job completed, checking the backup file size against historical norms, and — critically — performing a test restore from the backup to a separate environment.

An untested backup is not a backup. It is hope.

This distinction matters because backup jobs fail silently. A misconfigured backup schedule, a full disk, a permission error, an expired credential — any of these can cause backup jobs to fail without triggering an alert. If your first test of a backup is during an active recovery from a failed migration, you have compounded one incident into two.

Before every migration: verify the backup exists, verify the restore works, and document when the verification was performed.

2. Dry Run on a Production Clone

Run the full migration against a copy of production data before running it against production itself. Not against your staging environment. Not against your development database. Against a clone of production.

This distinction is critical. Staging and development environments almost never contain representative production data. They lack the edge cases, the inconsistent historical records, the orphaned rows, the unexpected null values, and the sheer volume that production carries. A migration that completes in thirty seconds against fifty thousand staging rows can fail at four hours against fifty million production rows — for reasons that would have been immediately obvious on a production clone.

The dry run serves three purposes: it confirms the migration SQL is syntactically correct and logically sound, it surfaces data quality issues that will cause the migration to fail or produce incorrect results, and it gives you an accurate timing estimate for step four.

3. Written Rollback Plan

Before any migration runs, document exactly what you will do if it needs to be rolled back. This documentation must exist in written form — not in the head of the engineer running the migration, not as a verbal agreement, not as a plan you will figure out if something goes wrong.

The rollback plan should specify: the exact commands to execute, the order in which to execute them, who has the database access required to execute them, how long the rollback will take based on your dry run, and what the acceptance criteria are for a successful rollback.

If you discover during a failed migration that your rollback takes four hours, your team needs to have known that before the migration started — not when they are managing an active production incident at 2am.

4. Migration Timing Estimation

Every migration has a runtime. Know yours before the maintenance window begins.

Use the production clone from step two to time the migration accurately. Record the row count of affected tables, the total data volume, and the elapsed time. Extrapolate if your clone is a partial sample. Add a safety margin — production is never quite the same as a clone, and contention from live traffic will slow write operations.

Timing estimation matters for maintenance window planning, for customer communication, and for the rollback decision threshold. If your migration is expected to take forty-five minutes and it has been running for three hours, you need a defined threshold at which the team escalates to rollback rather than continuing to wait.

Define that threshold before you start. Not during.

5. Maintenance Window or Zero-Downtime Strategy

Every migration requires either a maintenance window or a zero-downtime strategy. There is no third option.

If your migration requires application downtime — because it involves a lock that blocks reads, or because it changes a schema that live application code cannot handle in its current state — plan the maintenance window explicitly. Define the start time, the expected end time, the rollback threshold, and who communicates status to affected users. Get approval from the appropriate stakeholders before the window begins.

If your migration cannot tolerate downtime, implement a zero-downtime strategy. Common patterns include the expand/contract approach (add new columns while keeping old ones, migrate data in batches, switch application code, then drop old columns), feature flags to gate new schema-dependent code paths, and shadow write patterns where data is written to both old and new schema simultaneously during the transition.

Choosing the wrong strategy — attempting a zero-downtime migration with code that requires a maintenance window, or taking unplanned downtime on a migration you thought would be online — is the single most common cause of extended production incidents during database migrations.

6. Monitoring and Alerting Live Before You Start

Have your monitoring dashboards open and your alerting configured before you execute the first migration command. Do not wait until you suspect something is wrong to open your monitoring tools.

The metrics to watch during a database migration include: error rates on application endpoints that touch the affected tables, database query execution times and lock wait times, replication lag if you are running read replicas, disk I/O and disk space consumption (large migrations generate significant write amplification), and application memory usage if your migration involves large result sets.

The difference between catching a problem at one minute versus ten minutes can be the difference between a brief blip and a multi-hour incident. Monitoring that you check after you suspect something is wrong is not monitoring. It is forensics.

7. Team Communication Plan

Before the migration starts, every team member involved should know the answers to these questions: Who is the primary engineer executing the migration? Who is the on-call escalation if the primary engineer needs support? Who has database access to execute a rollback? Who makes the decision to roll back, and at what threshold? Who communicates status to the business, and through what channel?

These questions feel obvious. They are not obvious at 2am during an active incident when the primary engineer is debugging a lock contention issue, the rollback decision is time-sensitive, and nobody is sure who has the database credentials to execute it.

Document the answers before the migration starts. Share the document with everyone in the migration channel. Confirm receipt.

8. Post-Migration Validation Queries

Write your validation queries before the migration runs, not after it completes.

Validation queries written after a successful migration are shaped by the assumption that the migration succeeded. Validation queries written before the migration are shaped by what you are actually trying to verify. These are not the same queries.

Your validation suite should include: row count comparisons between the state before migration and after, referential integrity checks on foreign key relationships, spot checks on specific records that represent critical business data, and functional checks that confirm the application behaves correctly against the migrated schema.

Run these queries immediately after migration completes. A migration that finishes without errors is not necessarily a successful migration. A migration where your validation suite passes is a successful migration.

The Pattern Behind Every Migration Incident

We have never been called to help recover from a database migration where the team ran through all eight steps and something still went catastrophically wrong. We have been called to recover from migrations where teams skipped one step — sometimes just one — and paid for it.

The conversation is always the same: “We thought it would be fine.”

It was not fine.

The migration checklist is not overhead. It is the difference between a migration that is boring and forgotten by the following morning, and a migration that becomes the case study in your next engineering retrospective.

“An untested backup is not a backup. It is hope. And hope is not an engineering strategy.”

How Onclick Innovations Handles Database Migrations

At Onclick Innovations, we treat every database migration as a production incident waiting to happen — until our checklist proves otherwise. This is not pessimism. It is the engineering discipline that has allowed us to migrate hundreds of production databases across 350+ projects without a single data loss incident.

Every migration we execute includes a verified backup, a dry run on a production clone, a written rollback plan with defined thresholds, an accurate timing estimate, a defined downtime or zero-downtime strategy, pre-configured monitoring, a team communication plan, and a pre-written validation suite.

For clients managing their own migrations, we provide migration review as part of our engagement — reviewing the migration plan, the rollback strategy, and the validation approach before the first command runs.

If you are planning a database migration and want a second set of experienced eyes on your approach, we are happy to help.

📩 Get in touch → www.onclickinnovations.com
📍 Based in Mohali, India · Serving clients globally across 10+ countries

Frequently Asked Questions

What is a database migration?

A database migration is any change to the structure or content of a production database — including adding or removing columns, changing data types, renaming tables, adding indexes, migrating data between tables or schemas, and upgrading database versions. Migrations range from trivial to extremely complex, but all carry risk when executed against live production data.

What is a production clone and why do I need one?

A production clone is an identical copy of your production database, created specifically for migration testing. It contains the same schema, the same data volume, and representative examples of the edge cases that exist in your live data. Running a migration dry run against a production clone gives you an accurate preview of how the migration will behave in production — including timing, any data quality issues, and potential failure modes that would never surface in staging or development.

What is the expand/contract pattern for zero-downtime migrations?

The expand/contract pattern is a technique for making breaking schema changes without application downtime. It involves three phases: expand (add the new schema element while keeping the old one, deploy application code that writes to both), contract phase one (migrate existing data to the new schema, verify, stop writing to the old schema), and contract phase two (remove the old schema element). Each phase is a separate deployment with validation between them, allowing the migration to proceed without any single deployment requiring downtime.

How long should I keep a backup before running a migration?

You should take a fresh backup immediately before running any production migration, regardless of your regular backup schedule. This backup should be verified by performing a test restore before the migration begins. Keep this pre-migration backup for at least 30 days after the migration completes, or longer if your business has regulatory requirements around data retention.

Can Onclick Innovations review our migration plan before we execute it?

Yes. We offer migration review as part of our engineering services — reviewing your migration SQL, rollback strategy, timing estimates, and validation approach before you touch production. Contact us at onclickinnovations.com to discuss your requirements.

Share

The post The Database Migration Checklist: 8 Things You Must Do Before Touching Production Data appeared first on Blog.