Home
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
Streams Processing Language (SPL)
Streams Processing Language (SPL) is a distributed data flow composition language that is used in Teracloud® Streams. SPL has primitive types, program structures, and definitions that are tailored for streaming data.
Operators
Operators process tuples in an incoming stream and produce an output stream as a result. SPL supports two kinds of operators: primitive operators and composite operators. A primitive operator is written in C++ or Java™. A composite operator is written in SPL and contains a reusable stream subgraph.
Operator invocations
The purpose of SPL is to allow users to create streams of data, which are connected and manipulated by operator invocations.
Window clause
A window is a logical container for tuples recently received by an input port of an operator.
Punctuation
Punctuation is a control signal that appears interleaved with the tuples in a stream.

Welcome
Learn about the core capabilities of Teracloud® Streams, its architecture, and key concepts.
Installing
Use this information to install, upgrade, and uninstall the Teracloud® Streams product.
Configuring
Create a basic or an enterprise domain which is a single point for configuring and managing common resources, security, and instances.
Administering
Administer the product by using the Teracloud® Streams graphical user interface, APIs, or the streamtool command-line interface.
Developing
Develop stream applications with the Streams Processing Language (SPL), Java, and Python.
Troubleshooting
Resolve problems with Teracloud® Streams using the troubleshooting tools provided with the product as well as the resources offered by Teracloud Support.
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
- APIs
  Application programming interfaces (APIs) provide functions that simplify applications development.
- Job configuration overlays
  You can specify or change configuration parameters when you use the submitjob or updateoperators operations. The settings control submission constraints, host constraints and locations, and operator status. Parameters are captured and updated in the configuration JSON file. When you change the values in the job configuration overlay file, the changes override any previously set values.
- Commands
  Teracloud® Streams provides a number of commands you use to perform tasks.
- Operator model
  An operator model is an XML document that describes the basic properties of a primitive operator.
- Streams Processing Language (SPL)
  Streams Processing Language (SPL) is a distributed data flow composition language that is used in Teracloud® Streams. SPL has primitive types, program structures, and definitions that are tailored for streaming data.
  - Language overview
    SPL contains types, expressions, statements, functions, clauses, and other elements of a programming language.
  - Grammar notation
    This language specification adopts a flavor of Backus-Naur Form (BNF) to describe the syntax of language features.
  - Lexical syntax
    SPL defines a set of rules to determine the syntax of a program.
  - Grammar overview
    Here is the SPL grammar syntax, with references to the sections that contain the semantics.
  - Expression language
    Though SPL is a streaming language, there are many places where it uses expressions that are found in traditional imperative or functional languages.
  - Annotations
    Operators can be annotated. Annotations influence the run time behavior of stream applications.
  - Types
    SPL has several primitive types that are tailored for streaming, and a few composite types that are inspired by scripting languages but statically checked. The following topics describe the types, how to define them, and how to convert values between different types.
  - Operators
    Operators process tuples in an incoming stream and produce an output stream as a result. SPL supports two kinds of operators: primitive operators and composite operators. A primitive operator is written in C++ or Java™. A composite operator is written in SPL and contains a reusable stream subgraph.
    - Operator invocations
      The purpose of SPL is to allow users to create streams of data, which are connected and manipulated by operator invocations.
      - Operator invocation head
        The head of an operator invocation lists the output and input streams and the name of the operator that processes the data.
      - Logic clause
        The logic clause consists of local operator state that persists across operator firings, and statements that execute when an input port receives a tuple or a punctuation and thus fires.
      - Window clause
        A window is a logical container for tuples recently received by an input port of an operator.
        Punctuation
        Punctuation is a control signal that appears interleaved with the tuples in a stream.
        Tumbling windows
        A tumbling window stores incoming tuples until the window is full, then executes the operator behavior, and finally flushes all stored tuples, before it starts over from scratch.
        Sliding windows
        A sliding window is specified by providing both an eviction policy and a trigger policy.
        TimeInterval windows
        A timeInterval window collects tuples into fixed-duration intervals defined over event time.
        Partitioning
        The flag partitioned in the window clause specifies that instead of maintaining just one window per port, the operator invocation maintains one window per port per partition. A partition is a set of tuples that have the same values for the partitioning attributes.
      - Param clause
        The param clause contains code snippets that are supplied to the operator at invocation time.
      - Output clause
        The output clause assigns values to attributes in output tuples each time the operator submits data to an output stream.
      - Config clause
        The config clause specifies hints and directives that influence how the compiler builds this operator invocation, or how the runtime system executes it.
      - History
        Expressions in parameters or output assignments of operator invocations can refer directly to tuples received in the past.
    - Operator definitions
      A stream operator can be invoked to transform input streams to output streams. SPL supports two kinds of operators: primitive operators and composite operators.
  - Program structure
  - Operator config clause
    The config clause specifies hints and directives that influence how the compiler builds an operator invocation, or how the runtime system runs it.
- Toolkits
  A toolkit is a set of SPL artifacts, which are organized into a package. Toolkits make SPL or native functions and primitive or composite operators reusable across different applications.
- SPLDOC markup
  You can use SPLDOC markup in the descriptions associated with the SPL artifacts in a toolkit. The spl-make-doc command generates HTML documentation from the marked up artifacts.
Glossary
Use this glossary to find terms and definitions for Teracloud® Streams.

Punctuation

Punctuation is a control signal that appears interleaved with the tuples in a stream.

SPL supports the following kinds of punctuation:

Window punctuation creates boundaries within a stream that can be used for a user-defined window.
Final punctuation marks the end of a stream. Final punctuation indicates that no further tuples are received on an input port. The run time automatically forwards them to output ports as required.

Window punctuation processing

SPL supports two kinds of streams: a punctuation-free stream is a sequence of tuples, whereas a punctuated stream is a sequence of tuples that are interleaved with window punctuation. In both cases, the sequence of tuples continues indefinitely, unless the job ends, in which case the sequence is terminated by a final punctuation marker. A window punctuation separates groups of consecutive tuples on a stream to create window boundaries. For example, consider an invocation of an Aggregate operator with a punctuation-based window. Each time this operator invocation receives a window punctuation, it aggregates the accumulated tuples since the last window punctuation.

An output port of an operator invocation can either generate, remove, or preserve window punctuation markers. Examples for punctuation-generating ports include the output from the Join, Aggregate, and Punctor operators. Punctuation-free output ports ensure that their stream is not punctuated, and punctuation-preserving output ports forward window punctuation from an associated punctuated input port.

An input port of an operator invocation can either be punctuation-oblivious or punctuation-expecting. Ports are oblivious to window punctuation if they do not require a punctuated stream to function correctly. As a result, these kinds of ports work with both punctuated and punctuation-free streams. Ports are punctuation-expecting if they require a punctuated stream to function correctly. For example, ports with punctuation-based windows are punctuation-expecting. As a result, these kinds of ports work only with punctuated streams. Punctuation-expecting ports must be connected to exactly one input stream. The compiler enforces these rules of window punctuation by reporting errors when they are violated.