Data engineering
Part 5 of 6 · Object StorageLifecycle, Storage Classes, CDN & Presigned URLs
Capacity is cheap and the wrong tier, request pattern, or egress path is not. Pair storage classes with lifecycle transitions and expiry, put a CDN in front of hot reads with a private origin, and hand browsers a short-lived presigned URL instead of a public bucket.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
How a browser uploads
Prefer
Your API mints a short presign
The client proves who it is to you. You sign one PUT or a multipart upload for one key, with a TTL of a few minutes. The bucket stays private.
- The URL expires. A leak in a log is a short window.
- The signature is method and key specific.
- You still verify the object and commit your metadata after the upload.
Alternative
Open the bucket for a minute
A public ACL or a public policy exposes list and get beyond the one client you meant. Closing it later does not pull back objects that were copied.
- Scanners find public buckets quickly.
- A CDN does not need a public origin. It needs origin access control.
- A long-lived URL in email is a credential.
Overview
Raw capacity is the cheap line on the bill. The expensive lines are the wrong storage class, retrieval fees, request volume, and egress. A senior answer ties four levers together: storage class, lifecycle (transition, expiration, noncurrent cleanup, abort incomplete multipart), a CDN for hot reads, and a presigned URL when a browser or a partner needs a temporary call.
SKU names differ across AWS, Google Cloud, and Azure. Interview the pattern.
Storage classes
| Family | Access pattern | Tradeoff |
|---|---|---|
| Standard or hot | Frequent GET and PUT | Highest storage price, lowest access friction |
| Infrequent or cool | Reads on the order of monthly | Lower storage; retrieval fees and a minimum duration |
| Archive or cold | Rare restores | Cheapest storage; restore latency and fees |
| Intelligent or autoclass | Mixed and unknown | Less policy to write; watch surprise transitions |
Do not quote a price sheet. Say what you pay for: storage, retrieval, early-delete minimums, and request class. Moving a 90-day hot dataset to archive on day 1 saves storage and makes every read a restore.
Lifecycle rules that matter
- Transition. After N days, change class. Scope by prefix or tag. S3 will not transition Standard to Standard-IA before 30 days. Other minimums exist. Know that transitions are not instant free moves.
- Expiration. Delete the current object after N days. On a versioned bucket this adds a delete marker rather than erasing history.
- Noncurrent version expiration. Required once versioning is on, or old generations bill forever. The data-model lesson defined the marker. This rule is how you bound it.
- Abort incomplete multipart. The cost backstop from the multipart lesson. A week is a common window. Pick one and alarm on what remains.
- Delete-marker cleanup. Expire expired-object delete markers so versioned buckets do not fill with markers that hide nothing.
Decisions
- 1
Object or upload lands
- nextBytes or parts
- ?
Bytes or parts
- bytesStandard then colder
- incomplete partsAbort after 7 days
- 3
Standard then colder
- nextExpire when retention ends
- 4
Abort after 7 days
- 5
Expire when retention ends
Lesson map
Lifecycle, Storage Classes, CDN & Presigned URLs
Capacity is cheap and the wrong tier, request pattern, or egress path is not. Pair storage classes with lifecycle transitions and expiry, put a CDN in front of hot reads with a private origin, and hand browsers a short-lived presigned URL instead of a public bucket.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB land["Object or upload lands"] kind["Bytes or parts"] hot["Standard then colder"] abort["Abort after 7 days"] land -->|Object or upload lands| kind kind -->|bytes| hot kind -->|incomplete parts| abort
A concrete lake policy, said in days not in SKUs: land in standard, infrequent around 30 days, archive around 90, expire when the compliance clock (years, not days) ends, and abort incomplete uploads in a week. Prefix or tag decides which rule applies. tenant/acme/ can have a different retention from tenant/other/ because the prefix is the filter.
Rule order on one object
Transition does not delete. Expiration does not change class. Noncurrent expiry is a third action.
- 1
Current bytes age in standard
Reads are cheap. You are paying the hot storage rate. - 2
Transition to infrequent, then archive
The key still resolves. GET may cost a retrieval, and archive may require a restore. - 3
Expire the current object
Unversioned: the object is gone. Versioned: a delete marker becomes current. - 4
Expire noncurrent versions
This is the action that actually drops the old bytes.
CDN in front of the bucket
| Concern | Practice |
|---|---|
| Origin access | Private bucket plus origin access control. Not a public ACL. Legacy origin access identity is the old CloudFront mechanism. |
| Cache key | Put a content hash or version in the URL for immutable assets. |
| Invalidation | Prefer the new URL over a purge storm. Purge when you must. |
| Headers | Set cache and security headers on purpose, at the CDN or on the object. |
| Cost | Edge egress is often cheaper and faster than origin egress worldwide. |
A cache hit never sees your lifecycle transition. A cache miss uses origin access control to read the private object. If you delete or overwrite the origin, the edge can keep serving the old bytes until TTL or until the URL changes. That is the consistency caveat from the previous lessons, applied here.
Decisions
- 1
1. GET a versioned asset URL
- next2. Edge cache
- ?
2. Edge cache
- hit3. Return cached bytes
- miss4. Origin fetch with OAC
- 3
3. Return cached bytes
- 4
4. Origin fetch with OAC
- next5. Cache and serve
- 5
5. Cache and serve
Do not make the bucket public so the CDN can fetch it. Origin access control grants the CDN, and only that distribution, permission to read. A public bucket lets the world skip the CDN, list keys, and ignore your cache policy.
Presigned URLs
A presigned URL lets a client PUT or GET one object for a short time without holding your cloud keys. Cloud Storage calls these signed URLs. Azure calls the similar idea a SAS token. The threat is the same.
| Do | Do not |
|---|---|
| TTL in minutes | Multi-day URLs in logs, tickets, or email |
| Exact bucket, key, and method | A signature that can write any key |
| HTTPS, and do not log the query string | Secrets or PII in the key name |
| A POST policy when the browser upload needs one | A public ACL "just for a minute" |
STS credentials cap the signature at the credential lifetime, which is hours at most and often less. IAM user keys can sign longer, up to seven days on S3, which is a reason not to use them for this. A teaching ceiling of 15 minutes (900 seconds) is stricter than the API maximum on purpose.
Partner upload of 5 GB from a browser:
- The partner authenticates to your API.
- Your API mints a short-lived presigned multipart upload or POST policy for one key.
- The browser uploads to the object store, not through your app servers.
- Your API checks the checksum and commits the metadata pointer.
The bucket never becomes public. The security lesson is what happens when that URL leaks anyway.
Age and TTL you can run
Press Run. Snippets must be self-contained — no network, files, or native modules.
Press Run. Snippets must be self-contained — no network, files, or native modules.
The class function is a policy sketch. Real transitions also have minimum durations and retrieval costs the function does not model. The presign helper refuses to be the place where someone passes 86,400 seconds, because that default lives in presign_ttl_ok, not in the mint function. Keep both checks if you implement this for real.
Interview Q&A
What is the difference between transition and expiration?
Answer
Transition changes the storage class. The key still names the object, with new retrieval costs and maybe a restore delay. Expiration deletes the current object, which on a versioned bucket means a delete marker. Noncurrent bytes remain until a separate rule expires them.
Why is a public bucket plus a CDN a bad default?
Answer
The CDN is not the only client once the bucket is public. The world can GET and often LIST, bypassing cache policy and access logs you thought lived at the edge. Keep the origin private and grant the CDN with origin access control.
A partner must upload 5 GB from a browser. What do you do?
Answer
Authenticate them at your API, mint a short-lived presigned multipart PUT or POST policy for that key, let the browser upload directly, then verify the checksum and commit metadata. Do not put cloud keys in the browser and do not open the bucket.
Why abort incomplete multipart in a lifecycle rule?
Answer
Clients crash after CreateMultipartUpload and lose the upload id. Parts remain billable. The rule is the backstop the client can no longer call. Seven days is a starting point, not a law.
What does versioning do to expiration?
Answer
Expiring the current object inserts a delete marker. You still need noncurrent-version expiration, and often delete-marker cleanup, or the bucket accumulates hidden bytes and markers.
How should cacheable static assets be named?
Answer
Put a content hash in the key or the URL and cache that URL for a long time. Overwrite-in-place plus a long TTL is how the edge serves the old file after you deployed the new one.
When is archive the wrong save?
Answer
When the object is still read on a human timeline. Restore delay and retrieval fees can erase the storage savings. Archive is for compliance and rare reads, after you have said the restore SLO out loud.
Presign versus a long-lived access key in the browser?
Answer
A presign is scoped to a method, a key, and a clock. An access key is a reusable credential. Prefer the presign, keep the TTL short, and do not log the signed query string.
What is origin access control?
Answer
The CDN identity that may read a private bucket. Viewers hit the CDN. They do not receive a policy that lets them call the bucket directly. It replaces the older origin access identity pattern on CloudFront.
Do lifecycle rules apply to every object the same way?
Answer
No. Filter by prefix or tag. A compliance prefix can expire in years. A scratch prefix can expire in days. A rule with an empty filter is how you delete a bucket you meant to tier.
Pitfalls
For a media bucket, write transition, expiration, noncurrent expiry, abort incomplete multipart, and the CDN origin. Put a number of days on each. Then say the presign TTL for a browser PUT in seconds, and why it is under 900.