Home
Developing
Learn about different Teracloud Streams tools and features to write, compile, run, and test streams applications.
Developing custom operators
Create custom operators if shipped toolkits do not provide the necessary logic or behavior needed for your stream applications.
Developing primitive operators
You develop new primitive operators by implementing them in the C++ or Java™ language. You must define an operator model that describes the syntax and semantics of the operator and implement the operator logic by creating the header and implementation code generator templates.
Developing Python primitive operators
To create a Python primitive operator inside an SPL toolkit, you must provide an operator model. The process is similar to how you provide operator models for C++ primitive operators.
Using operator metrics
The Python Operator API provides runtime access to metrics similar to the access C++ APIs provide. Operator metrics are accessed through the streams.OperatorMetrics interface by obtaining a reference using OperatorContext.getMetrics().

Welcome
Learn about the core capabilities of Teracloud® Streams, its architecture, and key concepts.
Installing
Use this information to install, upgrade, and uninstall the Teracloud® Streams product.
Configuring
Create a basic or an enterprise domain which is a single point for configuring and managing common resources, security, and instances.
Administering
Administer the product by using the Teracloud® Streams graphical user interface, APIs, or the streamtool command-line interface.
Developing
Learn about different Teracloud Streams tools and features to write, compile, run, and test streams applications.
- Development concepts
  Development of stream applications consists of several components such as operators, streams, tuples, Streams Processing Language, toolkits, and more.
- Developing stream applications
  Try out tutorials and explore details about stream application development, SPL features, and best practices.
- Developing native functions
  Extend SPL's computational capabilities by creating native functions written in C++ or Java.
- Developing custom operators
  Create custom operators if shipped toolkits do not provide the necessary logic or behavior needed for your stream applications.
  - Types of operators
    When developing new operators, developers need to choose the kind of operator to implement based on the task they want to accomplish, the programming language they want to use and the trade-offs they are willing to make between generality and ease of implementation.
  - Primitive operator artifacts
    Each primitive operator is stored in a directory whose name follows the format ([:alnum:]+\.)*[:alnum:]+ where the last part ([:alnum:]+) is the operator name.
  - Developing primitive operators
    You develop new primitive operators by implementing them in the C++ or Java™ language. You must define an operator model that describes the syntax and semantics of the operator and implement the operator logic by creating the header and implementation code generator templates.
    - Developing C++ primitive operators
      The SPL language supports writing two styles of primitive operators, namely: generic operators and non-generic operators.
    - Developing Java™ primitive operators
      To create a Java™ primitive operator inside an SPL toolkit, you must provide an operator model. The process is similar to how you provide operator models for C++ primitive operators.
    - Developing Python primitive operators
      To create a Python primitive operator inside an SPL toolkit, you must provide an operator model. The process is similar to how you provide operator models for C++ primitive operators.
      - Python Operator API overview
        You use the Teracloud® Streams Python Operator API to implement user-defined operators in Python.
      - Python operator lifecycle
        Python operators have a runtime lifecycle that starts when an instance of the operator is initialized and ends when the processing element (PE) that hosts the operator is shut down.
      - Understanding threading in a Python operator
        A Python operator must always be written to be thread safe as its methods might be called by different threads and might be called concurrently. Unlike typical Python applications, Python Operators utilize multiple threads.
      - Working with SPL types
        Most core SPL types are supported by mapping to a native Python type.
      - Implementing an operator using the Python Operator API
        An operator processes incoming tuples and submits outgoing tuples. The relationship between an input tuple and the submitted tuples varies according to the function of the operator.
      - Implementing a sink operator using the Python Operator API
        A sink operator transmits information or events to an external system, such as a dashboard, web-server, mail-server, or a database.
      - Implementing a source operator with the Python Operator API
        A source operator fetches information or events from an external system, such as a sensor, messaging system, or a database, and presents them as a stream.
      - Using operator metrics
        The Python Operator API provides runtime access to metrics similar to the access C++ APIs provide. Operator metrics are accessed through the streams.OperatorMetrics interface by obtaining a reference using OperatorContext.getMetrics().
      - Using the PythonOp operator
        The operator spl.utility.PythonOp invokes an operator that is implemented in Python by using the Python Operator API. It is a general-purpose callout to Python from SPL.
  - Creating C++ libraries for operators and native functions
    To enable operators and native functions, you must compile with the appropriate include file paths and options and link any shared libraries against the correct runtime libraries.
  - Best practices
    Use these tips to write operators that perform effectively in their application, and in other applications.
  - Performance recommendations
    These tips help you to improve performance of your primitive operators in C++.
- Developing custom toolkits
  Bundle and reuse custom functions and operators across several stream applications by creating custom toolkits.
- Working with files
  Applications and operators can read and write to files in several ways. If relative paths are used, the data directory is used as the root of relative paths. To enable logic to access the runtime application or toolkit directory hierarchies, language-specific functions can be used.
- Enabling Streams data exchange
  Teracloud® Streams provides a data exchange REST API for inserting and retrieving tuples within a job to easily integrate with other data services and external applications. Stream applications can enable the data exchange feature by using one or more Endpoint operators.
- Debugging stream applications
  Debug stream applications using the interactive, command line-based Streams Debugger (sdb).
Troubleshooting
Resolve problems with Teracloud® Streams using the troubleshooting tools provided with the product as well as the resources offered by Teracloud Support.
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
Glossary
Use this glossary to find terms and definitions for Teracloud® Streams.

Using operator metrics

The Python Operator API provides runtime access to metrics similar to the access C++ APIs provide. Operator metrics are accessed through the streams.OperatorMetrics interface by obtaining a reference using OperatorContext.getMetrics().

For detailed information about Operator metrics, see the doxygen documentation. OperatorMetrics provides access to the system operator metrics using OperatorMetrics.InputPortMetricName and OperatorMetrics.OutputPortMetricName enumerations for metric names and OperatorMetrics.getInputPortMetric and OperatorMetrics.getOutputPortMetric.

Custom metrics can be created and fetched (including metrics that are defined by the operator model) using OperatorMetrics.createCustomMetric() and OperatorMetrics.getCustomMetric(). PE-level system metrics are similarly obtained through the streams.PEMetrics interface by obtaining a reference using OperatorContext.getPE().getMetrics().

All metrics are represented as a streams.Metric instance, with the mapping of the SPL int64 type to a Python int. The enumeration Metric.Kind indicates the kind of the metric (COUNTER, GAUGE, TIME) solely as an indication to external monitoring tools as to what the metric represents and how to display its value. A metric's kind has no impact on its function.

Creating a metric

opMetrics = self.getContext().getMetrics()

discardsMetric = opMetrics.createCustomMetric("nDiscardedTuples",
				"Number of discarded tuples", streams.Metric.Kind.COUNTER);

Usage:

if (discardTuple)
  discardsMetric.incrementValue(1)
else
  self.submitTuple(tuple, port)