Home
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
Streams Processing Language (SPL)
Streams Processing Language (SPL) is a distributed data flow composition language that is used in Teracloud® Streams. SPL has primitive types, program structures, and definitions that are tailored for streaming data.
Operators
Operators process tuples in an incoming stream and produce an output stream as a result. SPL supports two kinds of operators: primitive operators and composite operators. A primitive operator is written in C++ or Java™. A composite operator is written in SPL and contains a reusable stream subgraph.
Operator invocations
The purpose of SPL is to allow users to create streams of data, which are connected and manipulated by operator invocations.
Window clause
A window is a logical container for tuples recently received by an input port of an operator.
TimeInterval windows
A timeInterval window collects tuples into fixed-duration intervals defined over event time.
Lifecycle of a timeInterval window pane
This topic describes the lifecycle of a timeInterval window pane.

Welcome
Learn about the core capabilities of Teracloud® Streams, its architecture, and key concepts.
Installing
Use this information to install, upgrade, and uninstall the Teracloud® Streams product.
Configuring
Create a basic or an enterprise domain which is a single point for configuring and managing common resources, security, and instances.
Administering
Administer the product by using the Teracloud® Streams graphical user interface, APIs, or the streamtool command-line interface.
Developing
Develop stream applications with the Streams Processing Language (SPL), Java, and Python.
Troubleshooting
Resolve problems with Teracloud® Streams using the troubleshooting tools provided with the product as well as the resources offered by Teracloud Support.
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
- APIs
  Application programming interfaces (APIs) provide functions that simplify applications development.
- Job configuration overlays
  You can specify or change configuration parameters when you use the submitjob or updateoperators operations. The settings control submission constraints, host constraints and locations, and operator status. Parameters are captured and updated in the configuration JSON file. When you change the values in the job configuration overlay file, the changes override any previously set values.
- Commands
  Teracloud® Streams provides a number of commands you use to perform tasks.
- Operator model
  An operator model is an XML document that describes the basic properties of a primitive operator.
- Streams Processing Language (SPL)
  Streams Processing Language (SPL) is a distributed data flow composition language that is used in Teracloud® Streams. SPL has primitive types, program structures, and definitions that are tailored for streaming data.
  - Language overview
    SPL contains types, expressions, statements, functions, clauses, and other elements of a programming language.
  - Grammar notation
    This language specification adopts a flavor of Backus-Naur Form (BNF) to describe the syntax of language features.
  - Lexical syntax
    SPL defines a set of rules to determine the syntax of a program.
  - Grammar overview
    Here is the SPL grammar syntax, with references to the sections that contain the semantics.
  - Expression language
    Though SPL is a streaming language, there are many places where it uses expressions that are found in traditional imperative or functional languages.
  - Annotations
    Operators can be annotated. Annotations influence the run time behavior of stream applications.
  - Types
    SPL has several primitive types that are tailored for streaming, and a few composite types that are inspired by scripting languages but statically checked. The following topics describe the types, how to define them, and how to convert values between different types.
  - Operators
    Operators process tuples in an incoming stream and produce an output stream as a result. SPL supports two kinds of operators: primitive operators and composite operators. A primitive operator is written in C++ or Java™. A composite operator is written in SPL and contains a reusable stream subgraph.
    - Operator invocations
      The purpose of SPL is to allow users to create streams of data, which are connected and manipulated by operator invocations.
      - Operator invocation head
        The head of an operator invocation lists the output and input streams and the name of the operator that processes the data.
      - Logic clause
        The logic clause consists of local operator state that persists across operator firings, and statements that execute when an input port receives a tuple or a punctuation and thus fires.
      - Window clause
        A window is a logical container for tuples recently received by an input port of an operator.
        Punctuation
        Punctuation is a control signal that appears interleaved with the tuples in a stream.
        Tumbling windows
        A tumbling window stores incoming tuples until the window is full, then executes the operator behavior, and finally flushes all stored tuples, before it starts over from scratch.
        Sliding windows
        A sliding window is specified by providing both an eviction policy and a trigger policy.
        TimeInterval windows
        A timeInterval window collects tuples into fixed-duration intervals defined over event time.
        Syntax for defining timeInterval windows
        The syntax for the timeInterval window clause.
        Lifecycle of a timeInterval window pane
        This topic describes the lifecycle of a timeInterval window pane.
        Partitioning
        The flag partitioned in the window clause specifies that instead of maintaining just one window per port, the operator invocation maintains one window per port per partition. A partition is a set of tuples that have the same values for the partitioning attributes.
      - Param clause
        The param clause contains code snippets that are supplied to the operator at invocation time.
      - Output clause
        The output clause assigns values to attributes in output tuples each time the operator submits data to an output stream.
      - Config clause
        The config clause specifies hints and directives that influence how the compiler builds this operator invocation, or how the runtime system executes it.
      - History
        Expressions in parameters or output assignments of operator invocations can refer directly to tuples received in the past.
    - Operator definitions
      A stream operator can be invoked to transform input streams to output streams. SPL supports two kinds of operators: primitive operators and composite operators.
  - Program structure
  - Operator config clause
    The config clause specifies hints and directives that influence how the compiler builds an operator invocation, or how the runtime system runs it.
- Toolkits
  A toolkit is a set of SPL artifacts, which are organized into a package. Toolkits make SPL or native functions and primitive or composite operators reusable across different applications.
- SPLDOC markup
  You can use SPLDOC markup in the descriptions associated with the SPL artifacts in a toolkit. The spl-make-doc command generates HTML documentation from the marked up artifacts.
Glossary
Use this glossary to find terms and definitions for Teracloud® Streams.

Lifecycle of a timeInterval window pane

This topic describes the lifecycle of a timeInterval window pane.

A timeInterval window is:

Created: When the first tuples are assigned to it.
Closed: When the current watermark is beyond the discard age. The discard age is defined as the end of the interval plus the tuple age beyond which tuples are ignored. In this case the window pane ignores further input and its contents are discarded.
Incomplete: If the current watermark has not yet reached the end of the pane interval.
Complete: If the current watermark is greater than or equal to the end of the pane interval.

When the current event time (specified by the watermark) becomes greater than or equal to the interval's end, the window pane is completed and triggers. If tuples with event times within the pane’s interval arrive after the pane has triggered (late tuples), the pane triggers again and the operator which handles the trigger event re-calculates output taking into account the late data.

A timeInterval window pane triggers when:

The latest watermark is larger than or equal to the interval end.
Late events are inserted, which arrive after the watermark passed its end.

As a result the event handler then calculates aggregate values based on the window content.

Example of a timeInterval window lifecycle

A timeInterval window can be configured to create non-overlapping window panes every hour with a discard age of 6 hours:

A new pane is created for the interval [00:00 … 01:00) when the first tuple with an event time, which falls into this interval, arrives.
It marks the window complete when the input watermark passes 01:00.
It closes (and removes) this window when the watermark passes 07:00.