High-performing, scalable storage

Start from the access pattern: objects, shared files, or a disk for one instance. Then pick the class or volume type that meets the latency, IOPS, or throughput target, and prefer services that grow on their own.

Amazon S3 performance

• S3 scales automatically. Each prefix supports at least 3,500 write and 5,500 read requests per second, so spreading objects across prefixes raises the total.
• Multipart upload: Uploads parts in parallel and retries only failed parts. Recommended for large objects and required above 5 GB.
• Byte-range fetches: Download parts of an object in parallel, or only the bytes you need.
• S3 Transfer Acceleration: Routes uploads from distant clients through CloudFront edge locations and the AWS network to the bucket.
• S3 Express One Zone: A high-performance storage class in a single AZ with single-digit millisecond latency, for request-heavy jobs that sit next to their compute.
• For reads by users around the world, put CloudFront in front of the bucket.

Amazon EBS volume types

• gp3 (General Purpose SSD): The default. A baseline of 3,000 IOPS and 125 MiB/s at any size, and you can add IOPS and throughput without buying more capacity.
• io2 Block Express (Provisioned IOPS SSD): The highest IOPS and lowest, most consistent latency, for critical databases. Supports Multi-Attach to several instances in one AZ.
• st1 (Throughput Optimized HDD): Cheap, high throughput for large sequential reads such as log processing and big data. Can't be a boot volume.
• sc1 (Cold HDD): The lowest-cost EBS volume, for rarely accessed sequential data. Can't be a boot volume.
• Instance store: Disks physically attached to the host. Extremely fast, but data is lost when the instance stops or fails. Use for caches, buffers, and scratch data.
• Elastic Volumes change an EBS volume's size, type, or performance while it stays in use.

Shared file systems

• Amazon EFS: Serverless NFS for Linux that grows and shrinks automatically and is mounted by thousands of clients across AZs. Elastic throughput scales with the workload.
• Amazon FSx for Lustre: Parallel file system for HPC, machine learning, and media rendering, with sub-millisecond latency. Can present an S3 bucket as a file system.
• Amazon FSx for Windows File Server: SMB shares with Active Directory integration for Windows applications.
• Amazon FSx for NetApp ONTAP: NFS, SMB, and iSCSI from one file system, with ONTAP snapshots and replication. Good for moving existing NetApp workloads.
• Amazon FSx for OpenZFS: ZFS file system over NFS, for migrating Linux file servers that use ZFS.

Hybrid storage

• AWS Storage Gateway: S3 File Gateway: On-premises servers write files over NFS or SMB; the gateway stores them as S3 objects and keeps a local cache for low-latency access.
• AWS Storage Gateway: Volume Gateway: iSCSI block volumes backed by S3, with EBS snapshots. Cached mode keeps hot data local; stored mode keeps all data local and backs it up to AWS.
• AWS Storage Gateway: Tape Gateway: A virtual tape library for existing backup software, storing tapes in S3 and archiving them to S3 Glacier storage classes.
• Choose a gateway when on-premises applications need ongoing access to cloud storage. For a one-time or scheduled bulk copy, use DataSync (task 3.5).