The EBS bill nobody audits: gp3, sizing, and snapshots

Our storage line item had been climbing about 4% a month for a year and nobody could explain it. The obvious answer, unattached volumes, turned up eleven of them worth $63. Meanwhile EBS:SnapshotUsage alone was over $2,100 a month. I had been auditing the part of EBS that is easy to look at and ignoring the part that was actually growing.

EBS is the storage bill you inherit rather than choose. Volumes get provisioned once by a Terraform module written two years ago, snapshots get created nightly by a policy nobody has revisited, and neither shows up in a conversation until someone reads the invoice line by line.

Three places the money actually sits

• Volume type, gp2 volumes still running at a price gp3 beat years ago.
• Provisioned size, EBS bills what you allocated, not what you wrote.
• Snapshots, the only one of the three that grows on its own while you sleep.

Unattached volumes belong on the list too, but they are a one-time cleanup and every audit checklist already catches them. The three above compound.

gp2 to gp3 is still free money

gp3 is roughly 20% cheaper per GB than gp2 and decouples performance from size. gp2 gives you 3 IOPS per GB, so a 100 GB volume is capped at 300 IOPS and you buy capacity you do not need just to get throughput. gp3 ships 3,000 IOPS and 125 MB/s baseline on every volume regardless of size, then charges separately above that.

Cheaper and faster for anything under about 1 TB, which is most volumes. The conversion is online, no snapshot, no detach, no reboot:

# find every gp2 volume still running
aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[].[VolumeId,Size,Iops,Attachments[0].InstanceId]' \
  --output table

aws ec2 modify-volume --volume-id vol-0abc123 --volume-type gp3

One gotcha that bit us on a database volume: if you omit --iops, the volume lands on the gp3 default of 3,000. For any gp2 volume larger than 1 TB, that is a downgrade, since 3 IOPS per GB was already giving it more. Read the current Iops value first and pass it explicitly:

aws ec2 modify-volume --volume-id vol-0abc123 \
  --volume-type gp3 --iops 6000 --throughput 250

Even paying for those extra 3,000 IOPS at $0.005 each, the 2 TB volume above lands cheaper than gp2. You also cannot modify the same volume again for six hours, so get it right the first time.

Provisioned size is not used size

EBS charges for allocated GB from the moment the volume exists. A 500 GB volume holding 40 GB of logs bills at 500 GB forever. This happens because volumes grow easily and shrink not at all: expanding is one API call, shrinking means creating a smaller volume, copying the filesystem, and swapping it. So everyone over-allocates, once, permanently.

CloudWatch does not report filesystem utilization by default. You need the CloudWatch agent publishing disk_used_percent, or a one-off sweep via SSM:

aws ssm send-command \
  --document-name "AWS-RunShellScript" \
  --targets Key=tag:Environment,Values=production \
  --parameters 'commands=["df -h --output=source,size,used,pcent -x tmpfs"]'

Anything sitting under 30% used and over 200 GB is worth a resize conversation. For the rest, the fix is upstream: cap the default volume size in your Terraform module so the next hundred instances do not inherit the same padding.

Snapshots, and why deleting them frees nothing

This is the part that surprised everyone on my team. EBS snapshots are incremental: the first captures every written block, each one after that stores only blocks changed since the last. You are billed for blocks actually stored, around $0.05 per GB-month, not for the nominal volume size.

The consequence is unintuitive. When you delete a snapshot, AWS only frees blocks that no other snapshot still references. Blocks the next snapshot in the chain depends on get quietly re-parented to it. So deleting the oldest daily snapshot of a slow-changing volume typically frees close to zero, which is exactly why teams delete a few hundred snapshots, see no change on the invoice, and conclude snapshots were never the problem.

Snapshot spend only drops when the last snapshot referencing a block goes away. Pruning individual snapshots is theatre; pruning whole retention chains is the fix.

So audit by volume, not by snapshot. Group them and look for chains belonging to volumes that no longer exist, those are pure dead weight and deleting the whole set genuinely reclaims the space:

aws ec2 describe-snapshots --owner-ids self \
  --query 'Snapshots[].[VolumeId,SnapshotId,VolumeSize,StartTime]' \
  --output text | sort | awk '{print $1}' | uniq -c | sort -rn | head -20

Two things to know before you reach for Snapshot Archive as a cheaper tier. It costs about $0.0125 per GB-month, but it stores the full snapshot rather than the incremental delta, and it carries a 90-day minimum plus a 24 to 72 hour restore. Archiving a 4 GB incremental snapshot of a 500 GB volume bills you for 500 GB. It only pays off for large, standalone snapshots you must retain for compliance and will almost certainly never restore.

The durable fix is Data Lifecycle Manager with an actual retention policy, so the chain has a defined end instead of growing until someone notices the invoice.

Takeaways

• Convert gp2 to gp3 for roughly 20% off, but read the existing IOPS first and pass --iops explicitly on volumes over 1 TB.
• EBS bills provisioned GB, not used GB, and volumes only ever grow, so cap the default size in your IaC module rather than resizing one by one.
• Deleting individual snapshots usually frees nothing, because surviving snapshots inherit the shared blocks. Delete whole chains, especially for volumes that no longer exist.
• Snapshot Archive stores the full snapshot, not the delta, so it is cheaper only for large standalone snapshots with long retention.