代码拉取完成，页面将自动刷新
This package provides an interface to the Amazon Kinesis Client Library (KCL) MultiLangDaemon, which is part of the Amazon KCL for Java. Developers can use the Amazon KCL to build distributed applications that process streaming data reliably at scale. The Amazon KCL takes care of many of the complex tasks associated with distributed computing, such as load-balancing across multiple instances, responding to instance failures, checkpointing processed records, and reacting to changes in stream volume. This interface manages the interaction with the MultiLangDaemon so that developers can focus on implementing their record processor executable. A record processor executable typically looks something like:
#!env python
from amazon_kclpy import kcl
import json, base64
class RecordProcessor(kcl.RecordProcessorBase):
def initialize(self, initialiation_input):
pass
def process_records(self, process_records_input):
pass
def lease_lost(self, lease_lost_input):
pass
def shard_ended(self, shard_ended_input):
pass
def shutdown_requested(self, shutdown_requested_input):
pass
if __name__ == "__main__":
kclprocess = kcl.KCLProcess(RecordProcessor())
kclprocess.run()
Before running the samples, you'll want to make sure that your environment is configured to allow the samples to use your AWS Security Credentials.
By default the samples use the DefaultCredentialsProvider so you'll want to make your credentials available to one of the credentials providers in that provider chain. There are several ways to do this such as providing a ~/.aws/credentials file, or if you're running on EC2, you can associate an IAM role with your instance with appropriate access.
For questions regarding Amazon Kinesis Service and the client libraries please visit the Amazon Kinesis Forums
Using the amazon_kclpy package requires the MultiLangDaemon which is provided
by the Amazon KCL for Java. These jars will be downloaded automatically
by the install command, but you can explicitly download them with the download_jars command.
From the root of this repo, run:
python setup.py download_jars
python setup.py install
If you'd like to override the default search location for the jars, you can set the KCL_MVN_REPO_SEARCH_URL
environment variable to the location of the maven repository you'd like to use.
export KCL_MVN_REPO_SEARCH_URL=https://path/to/maven/repo
Now the amazon_kclpy and boto (used by the sample putter script) and required
jars should be installed in your environment. To start the sample putter, run:
sample_kinesis_wordputter.py --stream words -w cat -w dog -w bird -w lobster
This will create an Amazon Kinesis stream called words and put the words specified by the -w options into the stream once each. Use -p SECONDS to indicate a period over which to repeatedly put these words.
Now we would like to run an Amazon KCL for Python application that reads records from the stream we just created, but first take a look in the samples directory, you'll find a file called sample.properties, cat that file:
cat samples/sample.properties
You'll see several properties defined there. executableName indicates the
executable for the MultiLangDaemon to run, streamName is the Kinesis stream
to read from, appName is the Amazon KCL application name to use which will be the
name of an Amazon DynamoDB table that gets created by the Amazon KCL,
initialPositionInStream tells the Amazon KCL how to start reading from shards upon
a fresh startup. To run the sample application you can use a helper script
included in this package. Note you must provide a path to java (version 1.7
or greater) to run the Amazon KCL.
amazon_kclpy_helper.py --print_command \
--java <path-to-java> --properties samples/sample.properties
This will print the command needed to run the sample which you can copy paste, or surround the command with back ticks to run it.
`amazon_kclpy_helper.py --print_command \
--java <path-to-java> --properties samples/sample.properties`
Alternatively, if you don't have the source on hand, but want to run the sample
app you can use the --sample argument to indicate you'd like to get the
sample.properties file from the installation location.
amazon_kclpy_helper.py --print_command --java <path-to-java> --sample
Running on EC2 is simple. Assuming you are already logged into an EC2 instance running
Amazon Linux, the following steps will prepare your environment for running the sample
app. Note the version of java that ships with Amazon Linux can be found at
/usr/bin/java and should be 1.7 or greater.
sudo yum install python-pip
sudo pip install virtualenv
virtualenv /tmp/kclpy-sample-env
source /tmp/kclpy-sample-env/bin/activate
pip install amazon_kclpy
Amazon KCL for Python uses Amazon KCL for Java internally. We have implemented a Java-based daemon, called the MultiLangDaemon that does all the heavy lifting. Our approach has the daemon spawn the user-defined record processor script/program as a sub-process. The MultiLangDaemon communicates with this sub-process over standard input/output using a simple protocol, and therefore the record processor script/program can be written in any language.
At runtime, there will always be a one-to-one correspondence between a record processor, a child process, and an Amazon Kinesis Shard. The MultiLangDaemon will make sure of that, without any need for the developer to intervene.
In this release, we have abstracted these implementation details away and exposed an interface that enables you to focus on writing record processing logic in Python. This approach enables Amazon KCL to be language agnostic, while providing identical features and similar parallel processing model across all languages.
shutdownRequested() method.isGracefulLeaseHandoffEnabled.idleTimeBetweenReadsInMillis (PollingConfig) now has a minimum default value of 200.
LeaseAssignmentManagerWorkerMetricStatsReporterLeaseDiscoveryWe found an issue with the release 3.0.0 regarding the build failure. Please use the release 3.0.1 to use KCL 3.0.
For 2.x and 1.x release notes, please see v2.x/README.md
This library is licensed under the Apache 2.0 License.
此处可能存在不合适展示的内容，页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容，可点击提交进行申诉，我们将尽快为您处理。
马建仓 AI 助手
![dependabot[bot]](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAADwAAAA8CAYAAAA6/NlyAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAKSSURBVGhD7Zi/TxRBFMe/i3cgIAl3DRhUTDwT0EaIQU2kIRAbCog2mqidxo5YSrw/gBhNLCzQjlBYqGBhooCRQizEO1vx14kU3g8SjjOs3N6dzozcQQSTIdnldmbflJuX7Hy+3zfz3jzj/vlnv+GhZRCw5m6Tw5obDHKYHNZMAUppzQzdhEMOk8OaKUAprZmhdGlRSlNKa6aAIyldWePDpQc9W0qVtwqwzByyKzlkkiYWY2kkPi1hPppEIVdwXN4dB/4f0epPC3PTC4g8+gjrV94xcNcAFwm56y/vRpH8nHYE2nFgM53F6LWp9c0bgH+3D1W1fgQP1KGxJYDDp5tQXV9ViuFuj4dnsPxjxXbonQfeAsHYZaC9P4RjfYdgVDBF2FqMLePJjdd6Ahepjp5pxqnLR0qQL269w3wkYSu0KxzeSNR78wQaW4Pi07fZOCZuR/QGbj7egJ7r7QKSn+WRK5N6A1fWsho+zGr436OMhwPTyCTsu7xcl9Ic8sK9LtSs3dpPw29EY2LXciXw2aFOBPbtEYzPh2bx/X3SLl5nBvEbW8tNdVhi6+cYcL2XgD2V0rwDuzjc7Z1L62BHA7oH1spShpWlq5qXpd7wSdFf8xV7G8fkHY0bj39bS7tvaC6iK8oSfzy09YXQ1r/+eEh9TWNscEbiTt9eSFmA/dX8eehDYH8d9rYEEepsKjUaoqVkZ1c8D+P2dVhFWRwH3p7+YG2kiSk2AEh9UXQAIAvMHwofXi0g+lizEQ8f1FlmHlnTEm6m2BAvPrck2kdlh3iyrpYjzpEzXA4Q2X8SsKxSqsaRw6o6J7tvclhWKVXjyGFVnZPdNzksq5SqceSwqs7J7pscllVK1ThyWFXnZPftOYf/AOGev+g28wRIAAAAAElFTkSuQmCC)
