Home
Developing
Develop stream applications with the Streams Processing Language (SPL), Java, and Python.
Developing toolkits and operators
You can create custom toolkits and operators for your applications.
Developing operators
You can develop many stream applications with only the operators available in the SPL standard toolkit or in the specialized toolkits. If these toolkits do not provide all of the function you need to implement your application, you can develop new operators as required.
Developing primitive operators
You develop new primitive operators by implementing them in the C++ or Java™ language. You must define an operator model that describes the syntax and semantics of the operator and implement the operator logic by creating the header and implementation code generator templates.
Developing Python primitive operators
To create a Python primitive operator inside an SPL toolkit, you must provide an operator model. The process is similar to how you provide operator models for C++ primitive operators.
Python Operator API overview
You use the Teracloud® Streams Python Operator API to implement user-defined operators in Python.

Welcome
Learn about the core capabilities of Teracloud® Streams, its architecture, and key concepts.
Installing
Use this information to install, upgrade, and uninstall the Teracloud® Streams product.
Configuring
Create a basic or an enterprise domain which is a single point for configuring and managing common resources, security, and instances.
Administering
Administer the product by using the Teracloud® Streams graphical user interface, APIs, or the streamtool command-line interface.
Developing
Develop stream applications with the Streams Processing Language (SPL), Java, and Python.
- Development concepts
  Development of stream applications consists of several components such as operators, streams, tuples, Streams Processing Language, toolkits, and more.
- Writing stream applications
  Teracloud® Streams provides features and functions to help you write your applications to fulfill your business needs.
- Enabling Streams data exchange
  Teracloud® Streams provides a data exchange REST API for inserting and retrieving Streams job data. You can add one or more endpoint operators to your Streams application to enable Streams data exchange. Providing a standard REST interface for Streams data exchange eases the integration of Streams data with other data services and externally hosted applications.
- Compiling stream applications
  The Streams Processing Language (SPL) Compiler is called sc and is included with the Teracloud® Streams installation package. You compile SPL to create files to run on your Teracloud Streams instance.
- Performance improvements for stream applications
  You can improve the processing of your stream application by following these guidelines and tips.
- Debugging stream applications
  The Streams Processing Language (SPL) Streams Debugger (sdb) provides a command line-based interactive debugger for debugging stream applications.
- Developing toolkits and operators
  You can create custom toolkits and operators for your applications.
  - Developing toolkits
    Toolkits contain the building blocks for stream applications. In addition, you can add specialized toolkits or create your own toolkits.
  - Developing operators
    You can develop many stream applications with only the operators available in the SPL standard toolkit or in the specialized toolkits. If these toolkits do not provide all of the function you need to implement your application, you can develop new operators as required.
    - Types of operators
      When developing new operators, developers need to choose the kind of operator to implement based on the task they want to accomplish, the programming language they want to use and the trade-offs they are willing to make between generality and ease of implementation.
    - Developing primitive operators
      You develop new primitive operators by implementing them in the C++ or Java™ language. You must define an operator model that describes the syntax and semantics of the operator and implement the operator logic by creating the header and implementation code generator templates.
      - Developing C++ primitive operators
        The SPL language supports writing two styles of primitive operators, namely: generic operators and non-generic operators.
      - Developing Java™ primitive operators
        To create a Java™ primitive operator inside an SPL toolkit, you must provide an operator model. The process is similar to how you provide operator models for C++ primitive operators.
      - Developing Python primitive operators
        To create a Python primitive operator inside an SPL toolkit, you must provide an operator model. The process is similar to how you provide operator models for C++ primitive operators.
        Python Operator API overview
        You use the Teracloud® Streams Python Operator API to implement user-defined operators in Python.
        Python operator lifecycle
        Python operators have a runtime lifecycle that starts when an instance of the operator is initialized and ends when the processing element (PE) that hosts the operator is shut down.
        Understanding threading in a Python operator
        A Python operator must always be written to be thread safe as its methods might be called by different threads and might be called concurrently. Unlike typical Python applications, Python Operators utilize multiple threads.
        Working with SPL types
        Most core SPL types are supported by mapping to a native Python type.
        Implementing an operator using the Python Operator API
        An operator processes incoming tuples and submits outgoing tuples. The relationship between an input tuple and the submitted tuples varies according to the function of the operator.
        Implementing a sink operator using the Python Operator API
        A sink operator transmits information or events to an external system, such as a dashboard, web-server, mail-server, or a database.
        Implementing a source operator with the Python Operator API
        A source operator fetches information or events from an external system, such as a sensor, messaging system, or a database, and presents them as a stream.
        Using operator metrics
        The Python Operator API provides runtime access to metrics similar to the access C++ APIs provide. Operator metrics are accessed through the streams.OperatorMetrics interface by obtaining a reference using OperatorContext.getMetrics().
        Using the PythonOp operator
        The operator spl.utility.PythonOp invokes an operator that is implemented in Python by using the Python Operator API. It is a general-purpose callout to Python from SPL.
    - Creating C++ libraries for operators and native functions
      To enable operators and native functions, you must compile with the appropriate include file paths and options and link any shared libraries against the correct runtime libraries.
    - Fusing multiple Java™ operators into a single PE
      If multiple Java™ primitive operators and JavaOp operator invocations have compatible vmArg parameter lists, you can fuse them together into a single processing element (PE) or run them in stand-alone mode.
Troubleshooting
Resolve problems with Teracloud® Streams using the troubleshooting tools provided with the product as well as the resources offered by Teracloud Support.
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
Glossary
Use this glossary to find terms and definitions for Teracloud® Streams.

Python Operator API overview

You use the Teracloud® Streams Python Operator API to implement user-defined operators in Python.

The doxygen documentation for this API is provided with the Teracloud® Streams installation in the $STREAMS_INSTALL/doc/spl/operator/api/python/api directory. The package within this API is streams. Here is a brief description of its main classes.

streams.Operator: The base class and interface used to define an operator. The Teracloud Streams run time calls methods on this interface for initialization and then on for each incoming tuple and punctuation until a shutdown request is made. This is an alias to streams.Python.PythonOperator.
streams.OperatorContext: This interface provides access to the operator's runtime context, including descriptions of the streams and parameters.
streams.ProcessingElement: This interface provides access to the Processing Element, including information such as output and data directories.

There is also a python declaration module in the $STREAMS_INSTAL/doc/spl/operator/api/typings. In VSCode, you can add in settings.json the following entries to get auto completion (where you need to expand $STREAMS_INSTAL to the streams installation path)

    "python.autoComplete.extraPaths": [
        "$STREAMS_INSTAL/doc/spl/operator/api/typings"
    ],
    "python.analysis.extraPaths": [
        "$STREAMS_INSTAL/doc/spl/operator/api/typings"
    ],

Note: The python declaration module is only for editor assistance/documentation purposes only. During runtime, the streams api module is injected as a C module.

The Teracloud® Streams uses the Python version returned by python3-config. The python runtime that is linked to the application depends on the value that python3-config returns when the compiler (sc) is run.