Examples

These examples use the ObjectStorageScan operator.

a) Sample using bucket as submission parameter and cos application configuration with property cos.creds to specify the IBM COS credentials:

As endpoint is the public us-geo (CROSS REGION) the default value of the os-endpoint submission parameter.


composite Main {
    param
        expression<rstring> $bucket: getSubmissionTimeValue("os-bucket");
        expression<rstring> $endpoint: getSubmissionTimeValue("os-endpoint", "s3.us.cloud-object-storage.appdomain.cloud");
    graph
        // ObjectStorageScan operator with directory and pattern
        stream<rstring name> Scanned = com.teracloud.streams.objectstorage::ObjectStorageScan() {
            param

                objectStorageURI: com.teracloud.streams.objectstorage.s3::getObjectStorageURI($bucket);
                endpoint: $endpoint;
                directory: "/sample";
                pattern: ".*";
        }

        // use a ObjectStorageSource operator to process the object names
        stream<rstring line> Data = com.teracloud.streams.objectstorage::ObjectStorageSource(Scanned) {
            param
                objectStorageURI: com.teracloud.streams.objectstorage.s3::getObjectStorageURI($bucket);
                endpoint: $endpoint;
        }
}

b) Sample using the credentials parameter to specify the IBM COS credentials:

Set the objectStorageURI either in format "cos://<bucket-name>/" or "s3a://<bucket-name>/".


composite Main {
    param
        expression<rstring> $credentials: getSubmissionTimeValue("os-credentials");
        expression<rstring> $objectStorageURI: getSubmissionTimeValue("os-uri");
        expression<rstring> $endpoint: getSubmissionTimeValue("os-endpoint", "s3.us.cloud-object-storage.appdomain.cloud");
    graph
        // ObjectStorageScan operator with directory and pattern
        stream<rstring name> Scanned = com.teracloud.streams.objectstorage::ObjectStorageScan() {
            param

                credentials: $credentials;
                objectStorageURI: $objectStorageURI;
                endpoint: $endpoint;
                directory: "/sample";
                pattern: ".*";
        }

        // use a ObjectStorageSource operator to process the object names
        stream<rstring line> Data = com.teracloud.streams.objectstorage::ObjectStorageSource(Scanned) {
            param
                credentials: $credentials;
                objectStorageURI: $objectStorageURI;
                endpoint: $endpoint;
        }
}