Operator S3ObjectStorageSource

Primitive operator image not displayed. Problem loading file: ../../image/tk$com.teracloud.streams.objectstorage/op$com.teracloud.streams.objectstorage.s3$S3ObjectStorageSource.svg

Operator reads objects from S3 compliant object storage.

The operator opens an object on object storage and sends out its contents in tuple format on its output port.

If the optional input port is not specified, the operator reads the object that is specified in the objectName parameter and provides the object contents on the output port. If the optional input port is configured, the operator reads the objects that are named by the attribute in the tuples that arrive on its input port and places a punctuation marker between each object.

Behavior in a consistent region

The operator can participate in a consistent region. The operator can be at the start of a consistent region if there is no input port.

The operator supports periodic and operator-driven consistent region policies. If the consistent region policy is set as operator driven, the operator initiates a drain after a object is fully read. If the consistent region policy is set as periodic, the operator respects the period setting and establishes consistent states accordingly. This means that multiple consistent states can be established before a object is fully read.

At checkpoint, the operator saves the current object name and object cursor location. If the operator does not have an input port, upon application failures, the operator resets the object cursor back to the checkpointed location, and starts replaying tuples from the cursor location. If the operator has an input port and is in a consistent region, the operator relies on its upstream operators to properly reply the object names for it to re-read the objects from the beginning.

Examples

Summary

Ports
This operator has 1 input port and 1 output port.
Windowing
This operator does not accept any windowing configurations.
Parameters
This operator supports 11 parameters.

Required: accessKeyID, bucket, secretAccessKey

Optional: blockSize, encoding, endpoint, initDelay, maxAttempts, objectName, protocol, sslEnabled

Metrics
This operator reports 1 metric.

Properties

Implementation
Java

Input Ports

Ports (0)

The S3ObjectStorageSource operator has one optional input port. If an input port is specified, the operator expects an input tuple with a single attribute of type rstring. The input tuples contain the object names that the operator opens for reading. The input port is non-mutating.

Properties

Output Ports

Assignments
Java operators do not support output assignments.
Ports (0)

The S3ObjectStorageSource operator has one output port. The tuples on the output port contain the data that is read from the objects. The operator supports two modes of reading. To read an object line-by-line, the expected output schema of the output port is tuple<rstring line>. To read an object as binary, the expected output schema of the output port is tuple<blob data>. Use the blockSize parameter to control how much data to retrieve on each read. The operator includes a punctuation marker at the conclusion of each object. The operator forwards the object name from the input stream, if defined as second attribute of the output stream.

Properties

Parameters

This operator supports 11 parameters.

Required: accessKeyID, bucket, secretAccessKey

Optional: blockSize, encoding, endpoint, initDelay, maxAttempts, objectName, protocol, sslEnabled

accessKeyID

Specifies the Access Key ID for S3 account.

Properties
blockSize

Specifies the maximum number of bytes to be read at one time when reading an object into binary mode (ie, into a blob); thus, it is the maximum size of the blobs on the output stream. The parameter is optional, and defaults to 4096.

Properties
bucket

Specifies a bucket to use for reading objects. The bucket must exist. The operator does not create a bucket.

Properties
encoding

Specifies the encoding to use when reading files. The default value is UTF-8.

Properties
endpoint

Specifies endpoint for connection to object storage. For example, for S3 the endpoint might be 's3.amazonaws.com'. The default value is the IBM Cloud Object Storage (COS) public endpoint 's3.us.cloud-object-storage.appdomain.cloud'.

Properties
initDelay

Specifies the time to wait in seconds before the operator starts to read object. The default value is 0.

Properties
maxAttempts

Number of times we should retry errors. Default value is 20.

Properties
objectName

Specifies the name of the object that the operator opens and reads. This parameter must be specified when the optional input port is not configured. If the optional input port is used and the object name is specified, the operator generates an error.

Properties
protocol

Specifies the protocol to use for communication with object storage. Supported values are s3a and cos. The default value is s3a.

Properties
secretAccessKey

Specifies the Secret Access Key for S3 account.

Properties
sslEnabled

Enables or disables SSL connections to S3, default is true.

Properties

Metrics

nObjectsOpened - Counter

The number of opjects that are opened by the operator for reading data.

Libraries

Operator class library
Library Path: ../../impl/lib/com.teracloud.streams.objectstorage.jar, ../../opt/*, ../../opt/downloaded/*