Reading data from a stream in the AWS CLI
Written by
Updated at May 13, 2024
CLI
To get data from a stream's first shard, run the command:
SHARD_ITERATOR=$(aws kinesis get-shard-iterator \
--endpoint <endpoint> \
--shard-id shard-000001 \
--shard-iterator-type LATEST \
--stream-name <stream_ID> \
--query 'ShardIterator'| tr -d \")
aws kinesis get-records \
--endpoint <endpoint> \
--shard-iterator $SHARD_ITERATOR
Where:
-
--endpoint: To get stream data over the AWS Kinesis Data Streams protocol, specifyhttps://yds.serverless.yandexcloud.net. -
--stream-name: Consists of the availability zone, folder ID, Yandex Managed Service for YDB database ID, and stream name.For example, your stream ID will appear as
/ru-central1/aoeu1kuk2dht********/cc8029jgtuab********/aws_streamif:aws_stream: Stream nameru-central1: Regionaoeu1kuk2dht********: Folder IDcc8029jgtuab********: YDB database ID
Command example:
SHARD_ITERATOR=$(aws kinesis get-shard-iterator \
--endpoint https://yds.serverless.yandexcloud.net \
--shard-id shard-000001 \
--shard-iterator-type LATEST \
--stream-name /ru-central1/aoeu1kuk2dht********/cc8029jgtuab********/aws_stream
--query 'ShardIterator'| tr -d \")
aws kinesis get-records \
--endpoint https://yds.serverless.yandexcloud.net \
--shard-iterator $SHARD_ITERATOR
Result example:
{
"Records": [
{
"SequenceNumber": "0",
"Data": "eyJ1c2VyX2lkIjoidXNlcjEiLCJzY29yZSI6MTAwfQ==",
"PartitionKey": "1"
},
{
"SequenceNumber": "1",
"Data": "eyJ1c2VyX2lkIjoidXNlcjEiLCJzY29yZSI6MTAwfQ==",
"PartitionKey": "1"
},
...
}
In the example above, the data output in the Data field are BASE64-encoded. If you decode it, the following string is returned:
{"user_id":"user1","score":100}