Home
Developing
Develop stream applications with the Streams Processing Language (SPL), Java, and Python.
Developing toolkits and operators
You can create custom toolkits and operators for your applications.
Developing operators
You can develop many stream applications with only the operators available in the SPL standard toolkit or in the specialized toolkits. If these toolkits do not provide all of the function you need to implement your application, you can develop new operators as required.
Developing primitive operators
You develop new primitive operators by implementing them in the C++ or Java™ language. You must define an operator model that describes the syntax and semantics of the operator and implement the operator logic by creating the header and implementation code generator templates.
Developing C++ primitive operators
The SPL language supports writing two styles of primitive operators, namely: generic operators and non-generic operators.
Advanced operator implementation
Advanced operator implementation techniques provide additional runtime operator control.
Window handling
The SPL language run time provides APIs for building window-based operators.

Welcome
Learn about the core capabilities of Teracloud® Streams, its architecture, and key concepts.
Installing
Use this information to install, upgrade, and uninstall the Teracloud® Streams product.
Configuring
Create a basic or an enterprise domain which is a single point for configuring and managing common resources, security, and instances.
Administering
Administer the product by using the Teracloud® Streams graphical user interface, APIs, or the streamtool command-line interface.
Developing
Develop stream applications with the Streams Processing Language (SPL), Java, and Python.
- Development concepts
  Development of stream applications consists of several components such as operators, streams, tuples, Streams Processing Language, toolkits, and more.
- Writing stream applications
  Teracloud® Streams provides features and functions to help you write your applications to fulfill your business needs.
- Enabling Streams data exchange
  Teracloud® Streams provides a data exchange REST API for inserting and retrieving Streams job data. You can add one or more endpoint operators to your Streams application to enable Streams data exchange. Providing a standard REST interface for Streams data exchange eases the integration of Streams data with other data services and externally hosted applications.
- Compiling stream applications
  The Streams Processing Language (SPL) Compiler is called sc and is included with the Teracloud® Streams installation package. You compile SPL to create files to run on your Teracloud Streams instance.
- Performance improvements for stream applications
  You can improve the processing of your stream application by following these guidelines and tips.
- Debugging stream applications
  The Streams Processing Language (SPL) Streams Debugger (sdb) provides a command line-based interactive debugger for debugging stream applications.
- Developing toolkits and operators
  You can create custom toolkits and operators for your applications.
  - Developing toolkits
    Toolkits contain the building blocks for stream applications. In addition, you can add specialized toolkits or create your own toolkits.
  - Developing operators
    You can develop many stream applications with only the operators available in the SPL standard toolkit or in the specialized toolkits. If these toolkits do not provide all of the function you need to implement your application, you can develop new operators as required.
    - Types of operators
      When developing new operators, developers need to choose the kind of operator to implement based on the task they want to accomplish, the programming language they want to use and the trade-offs they are willing to make between generality and ease of implementation.
    - Developing primitive operators
      You develop new primitive operators by implementing them in the C++ or Java™ language. You must define an operator model that describes the syntax and semantics of the operator and implement the operator logic by creating the header and implementation code generator templates.
      - Developing C++ primitive operators
        The SPL language supports writing two styles of primitive operators, namely: generic operators and non-generic operators.
        Operator implementation
        You use skeleton template files that are generated by the spl-make-operator script as the basis for operator implementations.
        Advanced operator implementation
        Advanced operator implementation techniques provide additional runtime operator control.
        Blocking and shutdown handling
        SPL provides functions to block and shut down operator processing elements safely.
        Punctuation processing
        Punctuation markers in SPL create boundaries. Window punctuation create boundaries within a stream that can be used for creating user-defined windows. Final punctuation is used to mark the end of streams.
        Dynamic stream importing and exporting
        The Export operator exports a stream using a set of export properties, which can be modified at run time. Similarly, the Import operator is used to import a stream using an import subscription that is defined over the export properties. Import subscriptions can also be changed at run time.
        Logging and tracing
        When developing C++ primitive operators and native functions, use the SPLAPPLOG macro to write messages to the Teracloud® Streams product log location. Use the SPLAPPTRC macro to create a trace in the PE trace files.
        Checkpointing
        Checkpointing is the process of persisting operator state at run time to allow recovery from a failure. In case of failure, the operator can be restarted by resetting from the checkpointed state.
        Window handling
        The SPL language run time provides APIs for building window-based operators.
        Tumbling and sliding windows
        The SPL language supports tumbling and sliding windows to store tuples while they preserve the order of arrival, but differ in how they handle tuple evictions.
        TimeInterval windows
        The SPL language supports timeInterval windows which collect tuples into window panes specified by event-time intervals.
        Multi-threading and operator fusion considerations
        SPL operators can potentially run in a multi-threaded context, where the process functions of the operators are invoked concurrently. SPL provides a set of utility classes that handle threads, locks, and conditional variables to protect against concurrent access.
        Metrics access
        Metrics are simple counters that are maintained at run time, which can be read from outside of a running job to monitor statistics.
        Accessing paths at run time
        The SPL runtime APIs provide functions to access special paths at run time. These functions are provided as part of the SPL::ProcessingElement class.
        Using the reﬂective type system
        The types ValueHandle and Meta:BaseType provide support for reflection. The former type is a handle to a value of an SPL type. The latter type is a meta-type that represents an SPL type.
        Advanced code generation
        When you create generic operators, you can use advanced code generation techniques for more operator customization and control.
      - Developing Java™ primitive operators
        To create a Java™ primitive operator inside an SPL toolkit, you must provide an operator model. The process is similar to how you provide operator models for C++ primitive operators.
      - Developing Python primitive operators
        To create a Python primitive operator inside an SPL toolkit, you must provide an operator model. The process is similar to how you provide operator models for C++ primitive operators.
    - Creating C++ libraries for operators and native functions
      To enable operators and native functions, you must compile with the appropriate include file paths and options and link any shared libraries against the correct runtime libraries.
    - Fusing multiple Java™ operators into a single PE
      If multiple Java™ primitive operators and JavaOp operator invocations have compatible vmArg parameter lists, you can fuse them together into a single processing element (PE) or run them in stand-alone mode.
Troubleshooting
Resolve problems with Teracloud® Streams using the troubleshooting tools provided with the product as well as the resources offered by Teracloud Support.
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
Glossary
Use this glossary to find terms and definitions for Teracloud® Streams.

Window handling

The SPL language run time provides APIs for building window-based operators.

You can use the following window types:

Tumbling and sliding windows
TimeInterval windows