Intermediate LevelCloud Computing (AWS/GCP/Azure)
Compare Amazon Kinesis Data Streams vs Kinesis Data Firehose.
❓ Interview Prompt: Compare Amazon Kinesis Data Streams (shards, custom consumer code) vs Kinesis Data Firehose (managed delivery to S3/Redshift).
Amazon Kinesis ingests real-time streaming data at scale, offering two primary streaming engines:
1. Kinesis Data Streams (Custom Real-Time Processing):
•
Architecture: Partitioned into Shards (1 MB/sec write, 2 MB/sec read per shard).
•
Processing: Requires custom consumer applications (using KCL - Kinesis Client Library or Lambda) to read data streams.
•
Data Retention: 24 hours up to 365 days.
2. Kinesis Data Firehose (Managed Data Delivery):
•
Architecture: Fully-managed continuous delivery service with zero shard management.
•
Processing: Automatically transforms (via Lambda) and loads streaming data into S3, Redshift, OpenSearch, or Splunk.
•
Data Retention: Zero retention (buffer-and-dump stream delivery).
Code Implementation Example
typescript
function selectKinesisEngine(needsCustomRealtimeProcessing: boolean) {
if (needsCustomRealtimeProcessing) {
return 'Kinesis Data Streams (Shard-based custom consumer processing)';
}
return 'Kinesis Data Firehose (Managed buffer & load to S3/Redshift)';
}💡Selecting Kinesis Data Streams vs Kinesis Data Firehose.
🎯 Interview Key Takeaway
Kinesis Data Streams processes real-time data using custom consumers; Firehose automatically buffers and loads streams into S3/Redshift.
#Kinesis#Streaming#Data Streams#Firehose#Intermediate
Back to all Cloud Computing (AWS/GCP/Azure) Questions→