Object Storage
Studies in this cluster, in series order. Each one keeps its own URL.
Data engineering
Pipelines, sketches, approximate aggregations, object storage, and stream processing with event time, windows, and exactly-once sinks.
Object Storage
6 studies- 1.Object Storage — Buckets, Keys, Consistency & ScaleObject storage is the default durable store for lakes, backups, media, and model artifacts: a flat bucket plus key over HTTP, not a POSIX mount. This hub maps block versus file versus object, then consistency, multipart, lifecycle, and the IAM threat model. Metadata partition keys and change streams stay on the sharding and CDC hubs.
- 2.Data Model — Buckets, Objects, Keys, Versioning & MetadataThe object data model is a bucket, bytes plus system metadata, a UTF-8 key, optional user metadata, and optional versions. Interviews fail when prefixes are treated as directories, ETags and version ids are ignored, or one hot prefix throttles PUTs and listings.
- 3.Consistency — Read-after-write, Listing & Conditional WritesModern S3 is strongly consistent for new objects, overwrites, deletes, and listings in-region. Interviews still expect the pre-2020 failure mode, what If-Match buys you, and how replication, CDNs, and client caches put staleness back in front of a strong API.
- 4.Multipart Uploads, Parallelism & ThroughputLarge objects should not ride one fragile HTTP PUT. Multipart splits bytes into parts, uploads them in parallel, and publishes one object at complete. Interviews expect the state machine, part size, checksums, and abort hygiene so incomplete uploads do not bill forever.
- 5.Lifecycle, Storage Classes, CDN & Presigned URLsCapacity is cheap and the wrong tier, request pattern, or egress path is not. Pair storage classes with lifecycle transitions and expiry, put a CDN in front of hot reads with a private origin, and hand browsers a short-lived presigned URL instead of a public bucket.
- 6.Security — IAM, Encryption, Public Buckets & Threat ModelThe famous object-storage incidents are public buckets, over-broad IAM, leaked long-lived keys, and confused-deputy access. This lesson is the threat model: roles instead of static keys, block public access, encryption that does not excuse a public policy, and short-lived sharing.