Connection module for Amazon Kinesis
New in version 2017.7.0.
This module accepts explicit Kinesis credentials but can also utilize IAM roles assigned to the instance trough Instance Profiles. Dynamic credentials are then automatically obtained from AWS API and no further configuration is necessary. More Information available at:
If IAM roles are not used you need to specify them either in a pillar or in the minion's config file:
kinesis.keyid: GKTADJGHEIQSXMKKRBJ08H kinesis.key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
A region may also be specified in the configuration:
If a region is not specified, the default is us-east-1.
It's also possible to specify key, keyid and region via a profile, either as a passed in dict, or as a string to pull from pillars or minion config:
myprofile: keyid: GKTADJGHEIQSXMKKRBJ08H key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs region: us-east-1
Create a stream with name stream_name and initial number of shards num_shards.
salt myminion boto_kinesis.create_stream my_stream N region=us-east-1
Decrease stream retention period to retention_hours
salt myminion boto_kinesis.decrease_stream_retention_period my_stream N region=us-east-1
Delete the stream with name stream_name. This cannot be undone! All data will be lost!!
salt myminion boto_kinesis.delete_stream my_stream region=us-east-1
Disable enhanced monitoring for the specified shard-level metrics on stream stream_name
salt myminion boto_kinesis.disable_enhanced_monitoring my_stream ["metrics", "to", "disable"] region=us-east-1
Enable enhanced monitoring for the specified shard-level metrics on stream stream_name
salt myminion boto_kinesis.enable_enhanced_monitoring my_stream ["metrics", "to", "enable"] region=us-east-1
Check if the stream exists. Returns False and the error if it does not.
salt myminion boto_kinesis.exists my_stream region=us-east-1
Collect some data: number of open shards, key range, etc. Modifies stream_details to add a sorted list of OpenShards. Returns (min_hash_key, max_hash_key, stream_details)
salt myminion boto_kinesis.get_info_for_reshard existing_stream_details
Get complete stream info from AWS, returning only when the stream is in the ACTIVE state. Continues to retry when stream is updating or creating. If the stream is deleted during retries, the loop will catch the error and break.
salt myminion boto_kinesis.get_stream_when_active my_stream region=us-east-1
Increase stream retention period to retention_hours
salt myminion boto_kinesis.increase_stream_retention_period my_stream N region=us-east-1
Return a list of all streams visible to the current account
salt myminion boto_kinesis.list_streams
The hash key is a 128-bit int, sent as a string. It's necessary to convert to int/long for comparison operations. This helper method handles python 2/3 incompatibility
salt myminion boto_kinesis.long_int some_MD5_hash_as_string
long object if python 2.X, int object if python 3.X
Reshard a kinesis stream. Each call to this function will wait until the stream is ACTIVE, then make a single split or merge operation. This function decides where to split or merge with the assumption that the ultimate goal is a balanced partition space.
For safety, user must past in force=True; otherwise, the function will dry run.
salt myminion boto_kinesis.reshard my_stream N True region=us-east-1
True if a split or merge was found/performed, False if nothing is needed