Operator HTTPRequest

Primitive operator image not displayed. Problem loading file: ../../image/tk$com.teracloud.streams.inet/op$com.teracloud.streams.inet.http$HTTPRequest.svg
Issue an HTTP request of the specified method for each input tuple. For method NONE, the request is suppressed. The URL and method of the HTTP request either come from the input tuple using attributes specified by the url and method parameters or can be fixed using the fixedUrl and fixedMethod parameters. These parameters can be mixed, for example the URL can be fixed with fixedUrl while the method is set from each tuple using method. Entity enclosing requests (POST/PUT/PATCH) require a content type. The content type is specified with contenType or fixedContentType parameter. The default is application/json. The message body of an entity enclosing request can be defined in two ways:
  • Method POST: If the parameter requestBodyAttribute or requestBodyAttributeBin is defined and the value of the attribute is not empty, the value of the attribute is copied into the request body. Otherwise the names and values of the Request Attributes are used to generate the request form.
  • Other Methods: If the parameter requestBodyAttribute or requestBodyAttributeBin is defined, the value of the attribute is copied into the request body.
The operator can append Url Arguments to the request line. This happens if:
  • If parameter requestUrlArgumentsAttribute is specified and this attribute is not empty, the value of this attribute is copied as URL argument string and overwrites all other arguments. (all methods)
  • In method GET: If parameter requestAttributesAsUrlArguments is true, all Request Attribute names and values are converted to URL query parameters.
The content of the request is dependent on the method type.

GET

An HTTP GET request is made. If parameter requestAttributesAsUrlArguments is true, all request attributes are converted to URL query parameters. If parameter requestUrlArgumentsAttribute is specified and this attribute is not empty, this attribute is copied as URL argument string and overwrites all other arguments.

POST

An HTTP POST request is made, any request attributes are set as the body of the request message if parameter requestBodyAttribute and requestBodyAttributeBin is not present or the value of the attribute is empty. The encoding of the request body takes the content type into account. If content type is application/json, a json body is generated from request attributes. If content type is application/x-www-form-urlencoded, a url-encoded body is generated from request attributes. For all other content types, the content of all request attributes is concatenated into the message body. If requestBodyAttribute or requestBodyAttributeBin attribute is not empty, the body of the request is copied from one of these attribute instead. If content type equals application/octet-stream, attribute requestBodyAttributeBin is taken.

PUT

An HTTP PUT request is made, the body of the request message is copied from requestBodyAttribute or requestBodyAttributeBin attribute, depending on the content type.

PATCH

An HTTP PATCH request is made, the body of the request message is copied from requestBodyAttribute or requestBodyAttributeBin attribute, depending on the content type.

OPTIONS

No message body is generated.

HEAD

An HTTP HEAD request is made.

DELETE

No message body is generated.

TRACE

No message body is generated.

NONE

No http request is generated but an output tuple is submitted if the output port is present and attributes are passed from input port to output port.

Request Attributes

Attributes from the input tuple are request parameters except for:
  • Any attribute specified by parameters url, method, contentType, requestBodyAttribute, requestBodyAttributeBin or equestUrlArguments.
  • If parameter requestAttributes is set, all attributes of this parameter are considered a request attribute.
  • If parameter requestAttributes has one empty element, no attributes are considered a request attribute.

Http Authentication

The operator supports the following authentication methods: Basic, Digest, OAuth1a and OAuth2.0; see parameter authenticationType.

Behavior in a consistent region

This operator cannot be used inside a consistent region.

Summary

Ports
This operator has 1 input port and 1 output port.
Windowing
This operator does not accept any windowing configurations.
Parameters
This operator supports 43 parameters.

Optional: accessTokenAttribute, authenticationFile, authenticationProperties, authenticationType, connectionTimeout, contentType, disableAutomaticRetries, disableContentCompression, disableRedirectHandling, errorDiagnostics, extraHeaderAttribute, extraHeaders, fixedContentType, fixedMethod, fixedUrl, method, outputBody, outputBodyRaw, outputCharSet, outputContentEncoding, outputContentType, outputDataLine, outputHeader, outputStatus, outputStatusCode, proxy, proxyPort, redirectStrategy, requestAttributes, requestAttributesAsUrlArguments, requestBodyAttribute, requestBodyAttributeBin, requestUrlArgumentsAttribute, socketTimeout, sslAcceptAllCertificates, sslKeyPassword, sslKeyStoreFile, sslKeyStorePassword, sslTrustStoreFile, sslTrustStorePassword, tokenTypeAttribute, url, userAgent

Metrics
This operator reports 3 metrics.

Properties

Implementation
Java

Input Ports

Ports (0)

This stream contains the information sent in a http request. Each tuple with valid request data results in an HTTP request except if method NONE is specified.

Properties

Output Ports

Assignments
Java operators do not support output assignments.
Ports (0)

Data received in the http response be sent on this port. Other attributes are assigned from input stream.

Properties

Parameters

This operator supports 43 parameters.

Optional: accessTokenAttribute, authenticationFile, authenticationProperties, authenticationType, connectionTimeout, contentType, disableAutomaticRetries, disableContentCompression, disableRedirectHandling, errorDiagnostics, extraHeaderAttribute, extraHeaders, fixedContentType, fixedMethod, fixedUrl, method, outputBody, outputBodyRaw, outputCharSet, outputContentEncoding, outputContentType, outputDataLine, outputHeader, outputStatus, outputStatusCode, proxy, proxyPort, redirectStrategy, requestAttributes, requestAttributesAsUrlArguments, requestBodyAttribute, requestBodyAttributeBin, requestUrlArgumentsAttribute, socketTimeout, sslAcceptAllCertificates, sslKeyPassword, sslKeyStoreFile, sslKeyStorePassword, sslTrustStoreFile, sslTrustStorePassword, tokenTypeAttribute, url, userAgent

accessTokenAttribute

Attribute that specifies the access token if authenticationType is OAUTH2. Any value of this attribute overwrites the value of the accessToken property in authenticationFile or in authenticationProperties. This parameter is meaningless if the authenticationType is different from OAUTH2

Properties
authenticationFile
Path to the properties file containing authentication information. Authentication file is recommended to be stored in the application_dir/etc directory. Path of this file can be absolute or relative, if relative path is specified then it is relative to the application directory. The content of this file depends on the authenticationType.
  • If authenticationType is STANDARD: A valid line is composed from the authentication Scope (hostname or ANY_HOST, equal sign, user, colon, password. E.g.: ANY_HOST=user:passwd
  • If authenticationType is OAUTH1: The authentication file must contain key/value pairs for the keys: consumerKey, consumerSecret, accessToken and accessTokenSecret.
  • If authenticationType is OAUTH2: The authentication file must contain one key/value pair for key accessToken=myAccessToken.
The authentication file may contain one key/value pair for key authMethod. See http_request_auth.properties, http_request_oauth1.properties and http_request_oauth2.properties in the toolkits etc directory for a sample of authentication properties.
Properties
authenticationProperties

Properties to override those in the authentication file.

Properties
authenticationType

The type of used authentication method. Valid options are "STANDARD", "OAUTH1," and "OAUTH2". Default is "STANDARD". If "STANDARD" is selected, the authorization may be none, basic or digest authorization. If the server requires basic or digest authorization one of the parameters authenticationFile or authenticationProperties is required. If the "OAUTH1" option is selected, the requests will be singed using OAuth 1.0a If the "OAUTH2" option is selected, the requests will be singed using OAuth 2.0.

Properties
connectionTimeout

Set the connection timeout in milliseconds. If value is 0, the default connection timeout is used. (You cannot specify a connection timeout that increases the platform default (130sec.); you can only use it to decrease the platform default. Default is 0.

Properties
contentType

MIME content type of entity for POST and PUT requests. Only one of contentType and fixedContentType must be specified. Defaults to application/json.

Properties
disableAutomaticRetries

Disables automatic request recovery and re-execution. Default is false

Properties
disableContentCompression

Disables automatic content decompression. Default is false

Properties
disableRedirectHandling

Disables automatic redirect handling. Default is false. This parameter must not be used together with parameter redirectStrategy

Properties
errorDiagnostics

Name of the attribute to populate the error diagnostics with. This string contains the diagnostics information when the program execution of the http operation throws an exception. This string is empty when a http response was received. The status line of the http response is issued in the dataStatus attribute.

Properties
extraHeaderAttribute

One extra header to send with the request, the attribute must contain a string in the form Header-Name: value. If the attribute value is an empty string, no additional header is send. This parameter may be applied together with parameter extraHeaders.

Properties
extraHeaders

Extra headers to send with request, format is Header-Name: value.

Properties
fixedContentType

Fixed MIME content type of entity for POST and PUT requests. Only one of contentType and fixedContentType must be specified. Defaults to application/json.

Properties
fixedMethod

Fixed method for each HTTP request. Every HTTP request uses the method provided. One and only one of method and fixedMethod must be specified.

Properties
fixedUrl

Fixed URL to send HTTP requests to. Any tuple received on the input port results in a request to the URL provided (except for method NONE). One and only one of url and fixedUrl must be specified.

Properties
method

Attribute that specifies the method to be used in the HTTP request for a tuple. One and only one of method and fixedMethod must be specified.

Properties
outputBody

Name of the attribute to populate the response body with. If this parameter is set, the operators returns one tuple for each request. Only one of outputDataLine and outputBody must be specified. This parameter is not allowed if the operator has no output port.If this parameter is set and parameter outputBodyRaw is set, all responses with entitiy mime type not equal to application/octet-stream or default/binary generate output here.

Properties
outputBodyRaw

Name of the attribute to populate the raw response body with. The type of this attribute must be blob. Only one of outputDataLine and outputBodyRaw must be specified. This parameter is not allowed if the operator has no output port. You may use outputBodyRaw and outputBody.If this parameter is set and parameter outputBody is set, all responses with entitiy mime type equal to application/octet-stream or default/binary will generate output here.

Properties
outputCharSet

Name of the attribute to populate the response entity charset with. This parameter is not allowed if the operator has no output port.

Properties
outputContentEncoding

Name of the attribute to populate the response content encoding header with. This parameter is not allowed if the operator has no output port.

Properties
outputContentType

Name of the attribute to populate the response entity mime type with. This parameter is not allowed if the operator has no output port.

Properties
outputDataLine

Name of the attribute to populate one line of the response data with. If this parameter is set, the operators returns one tuple for each line in the response body but at least one tuple if the body is empty. Only one of outputDataLine and outputBody must be specified. This parameter is not allowed if the operator has no output port.

Properties
outputHeader

Name of the attribute to populate the response header information with. The type of this attribute must be string list. This parameter is not allowed if the operator has no output port.

Properties
outputStatus

Name of the attribute to populate the response status line with. This parameter is not allowed if the operator has no output port.

Properties
outputStatusCode

Name of the attribute to populate the response status code as integer with. The type of this attribute must be int32. This is the numerical value from the http response or -1 if no response was received. This parameter is not allowed if the operator has no output port.

Properties
proxy

Hostname of the http-proxy to be used. If this parameter is omitted no proxy is used.

Properties
proxyPort

The proxy-port to be used. Default value is 8080. This parameter is ignored if no proxy parameter is specified.

Properties
redirectStrategy
Set the redirection strategy. Possible values are:
  • DEFAULT: This enables the automatic redirect handling. This strategy honors the restrictions on automatic redirection of entity enclosing methods such as POST and PUT imposed by the HTTP specification. 302 Moved Temporarily, 301 Moved Permanently and 307 Temporary Redirect status codes will result in an automatic redirect of HEAD and GET methods only. POST and PUT methods will not be automatically redirected as requiring user confirmation.
  • LAX: This relaxes the default settings and enables the automatic redirection of all HEAD, GET, POST, and DELETE requests.
  • NONE: This disables the automatic redirection handling.
This parameter must not be used together with parameter disableRedirectHandling. Default is DEFAULT
Properties
requestAttributes

Names of the attributes which are part of the request body. The content of these attributes is sent as request body in method POST. If parameter requestAttributesAsUrlArguments is true, the request attributes are additionally appended as arguments to the url in method GET. If this parameter is missing, all attributes, excluding those that are used to specify the URL, method, content type, Request url arguments, extra header, access token, token type or request body, are used in the request body. One empty element defines an empty list which means no attributes are considered request attributes.

Properties
requestAttributesAsUrlArguments

If this parameter is true, the request attributes are appended as arguments to the url in method GET. If this parameter is false, the request attributes are not appended to the url. Default is false. These arguments are overwritten from a non-empty value in parameter requestUrlArgumentsAttribute.

Properties
requestBodyAttribute

Request body attribute for any method that accepts an entity (PUT / POST / PATCH). In method PUT and PATCH the body of request is taken from this attribute. In method POST, any non-empty value overwrites the request attributes.

Properties
requestBodyAttributeBin

Request body attribute for any method that accepts an entity (PUT / POST / PATCH). In method PUT and PATCH the body of request is taken from this attribute. In method POST, any non-empty value overwrites the request attributes. The content of this attribue is used when the content type equals application/octet-stream. In all other cases, the content of the requestBodyAttribute is used. Note: The input attribute must be of type blob.

Properties
requestUrlArgumentsAttribute

Request url arguments attribute. If this parameter is set and the value of this attribute is not empty, the content of this string is appended as arguments to the request url. This overwrites the arguments which are generated from the request attributes. The value is expected to be unescaped and may contain non-ASCII characters

Properties
socketTimeout

Set the socket timeout in milliseconds, which is the timeout for waiting for data (request-response cycle). A timeout value of zero is interpreted as an infinite timeout. A negative value is interpreted as system default. Default is -1.

Properties
sslAcceptAllCertificates

Accept all SSL certificates, this means the server certificate is not checked. If this parameter is set, parameter sslTrustStoreFile and setSslKeyStoreFile are not allowed. Setting this option will allow potentially insecure connections. Default is false.

Properties
sslKeyPassword

Password for the keys in the key store.

Properties
sslKeyStoreFile

Path to jks key store file used for client authentication. This store should contain one client key pair.If this parameter is set, parameter sslKeyStorePassword is optional.If this parameter is set, parameter sslKeyPassword is required.

Properties
sslKeyStorePassword

Password for the key store. If the password for the store differs from the key password, use this parameter.

Properties
sslTrustStoreFile

Path to jks trust store file used for server authentication. Self signed certificates will be accepted.If this parameter is set, parameter sslTrustStorePassword is optional.

Properties
sslTrustStorePassword

Password for the trust store.

Properties
tokenTypeAttribute

Attribute that specifies the access token type if authenticationType is OAUTH2. Any value of this attribute overwrites the value of the authMethod property in authenticationFile or in authenticationProperties. This parameter is meaningless if the authenticationType is different from OAUTH2

Properties
url

Attribute that specifies the URL to be used in the HTTP request for a tuple. One and only one of url and fixedUrl must be specified.

Properties
userAgent

Assigns the header User-Agent value. Default is "Apache-HttpClient/4.5.12 (Java/1.8.0_231)"

Properties

Metrics

nRequestTransmit - Counter

The number of request transmit attempts.

nResponseNoSuccess - Counter

The number of received responses with result codes other than success.

nResponseSuccess - Counter

The number of received responses with result code: success (2xx).

Libraries

Operator class library
Library Path: ../../impl/lib/com.teracloud.streams.inet.jar