Data ingestion and transformation

Match the ingestion service to how often and how fast data arrives, land it in a secure data lake, convert it to efficient formats, and choose the right engine to query and visualize it.

Streaming ingestion

• Amazon Kinesis Data Streams: Real-time streams that you read with your own consumers (Lambda, applications). Records are kept for a configurable retention period and can be replayed. On-demand or provisioned capacity.
• Amazon Data Firehose: Fully managed delivery of streaming data to S3, Redshift, OpenSearch Service, and HTTP endpoints. Buffers, compresses, can transform with Lambda, and can convert JSON to Parquet or ORC.
• Amazon MSK: Managed Apache Kafka for teams that already use Kafka APIs and tools.
• Amazon Kinesis Video Streams: Ingests video and other time-encoded data from devices for playback and analysis.
• Data Streams when you need custom real-time processing or replay; Firehose when you just need the data delivered somewhere with no code to run.

Batch ingestion and data transfer

• AWS DataSync: Online transfer from NFS, SMB, HDFS, or object storage into S3, EFS, or FSx, with scheduling, encryption, and integrity checks. Much faster than scripts.
• AWS Storage Gateway: Hybrid access for on-premises apps that keep writing locally while data lands in AWS.
• AWS Transfer Family: Partners drop files over SFTP, FTPS, or AS2 straight into S3.
• Amazon AppFlow and AWS Data Exchange: AppFlow pulls data from SaaS applications; Data Exchange subscribes to third-party data sets.
• Offline transfer: The guide still lists the AWS Snow Family, but Snowball Edge closed to new customers on November 7, 2025. AWS now points new customers to DataSync, AWS Data Transfer Terminal, or partner services.
• Size the pipe: work out data volume divided by available bandwidth. If the transfer would take too long, add Direct Connect capacity or a physical transfer option.

Data lakes and transformation

• Amazon S3 as the data lake: Cheap, durable storage for raw and processed data in open formats.
• AWS Glue: Crawlers discover schemas into the Glue Data Catalog; serverless Spark ETL jobs clean and transform data.
• AWS Lake Formation: Builds and secures a data lake with central permissions down to the table, column, row, and cell level.
• Columnar formats: Converting CSV or JSON to Parquet or ORC compresses the data and lets engines read only the needed columns: faster and cheaper queries. Partitioning (for example by date) skips data a query doesn't need.

Analytics and visualization

• Amazon Athena: Serverless SQL directly on data in S3, billed by data scanned.
• Amazon Redshift: Columnar data warehouse for complex analytics on large structured data; Redshift Spectrum also queries S3, and Redshift Serverless removes cluster management.
• Amazon EMR: Choose it for large-scale Spark, Hadoop, or Hive processing that needs framework control.
• Amazon OpenSearch Service: Search, log analytics, and dashboards over semi-structured data.
• Amazon Quick Sight: Business intelligence dashboards and natural-language questions over your data. It is the BI capability of Amazon Quick, the name the guide lists.

Securing ingestion points

• Producers authenticate with IAM roles; use interface endpoints so streams and delivery streams are reached privately.
• Encrypt streams and buckets with KMS, and use Lake Formation or bucket policies so each team sees only the data it should.