Home
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
Streams Processing Language (SPL)
Streams Processing Language (SPL) is a distributed data flow composition language that is used in Teracloud® Streams. SPL has primitive types, program structures, and definitions that are tailored for streaming data.
Operators
Operators process tuples in an incoming stream and produce an output stream as a result. SPL supports two kinds of operators: primitive operators and composite operators. A primitive operator is written in C++ or Java™. A composite operator is written in SPL and contains a reusable stream subgraph.
Operator invocations
The purpose of SPL is to allow users to create streams of data, which are connected and manipulated by operator invocations.
Window clause
A window is a logical container for tuples recently received by an input port of an operator.
Partitioning
The flag partitioned in the window clause specifies that instead of maintaining just one window per port, the operator invocation maintains one window per port per partition. A partition is a set of tuples that have the same values for the partitioning attributes.

Welcome
Learn about the core capabilities of Teracloud® Streams, its architecture, and key concepts.
Installing
Use this information to install, upgrade, and uninstall the Teracloud® Streams product.
Configuring
Create a basic or an enterprise domain which is a single point for configuring and managing common resources, security, and instances.
Administering
Administer the product by using the Teracloud® Streams graphical user interface, APIs, or the streamtool command-line interface.
Developing
Learn about different Teracloud Streams tools and features to write, compile, run, and test streams applications.
Troubleshooting
Resolve problems with Teracloud® Streams using the troubleshooting tools provided with the product as well as the resources offered by Teracloud Support.
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
- APIs
  Application programming interfaces (APIs) provide functions that simplify applications development.
- Job configuration overlays
  You can specify or change configuration parameters when you use the submitjob or updateoperators operations. The settings control submission constraints, host constraints and locations, and operator status. Parameters are captured and updated in the configuration JSON file. When you change the values in the job configuration overlay file, the changes override any previously set values.
- Commands
  Teracloud® Streams provides a number of commands you use to perform tasks.
- Metrics
  Teracloud® Streams provides metrics to help evaluate the health of Teracloud Streams services, to aid in diagnosing performance issues, and to analyze throughput of requests. You can use the streamtool checkinstancemetrics, streamtool checkdomainmetrics, streamtool getdomainmetrics, and streamtool checkresourcemetrics commands to view the metrics data.
- Operator model
  An operator model is an XML document that describes the basic properties of a primitive operator.
- Streams Processing Language (SPL)
  Streams Processing Language (SPL) is a distributed data flow composition language that is used in Teracloud® Streams. SPL has primitive types, program structures, and definitions that are tailored for streaming data.
  - Language overview
    SPL contains types, expressions, statements, functions, clauses, and other elements of a programming language.
  - Grammar notation
    This language specification adopts a flavor of Backus-Naur Form (BNF) to describe the syntax of language features.
  - Lexical syntax
    SPL defines a set of rules to determine the syntax of a program.
  - Grammar overview
    Here is the SPL grammar syntax, with references to the sections that contain the semantics.
  - Expression language
    Though SPL is a streaming language, there are many places where it uses expressions that are found in traditional imperative or functional languages.
  - Annotations
    Operators can be annotated. Annotations influence the run time behavior of stream applications.
  - Types
    SPL has several primitive types that are tailored for streaming, and a few composite types that are inspired by scripting languages but statically checked. The following topics describe the types, how to define them, and how to convert values between different types.
  - Operators
    Operators process tuples in an incoming stream and produce an output stream as a result. SPL supports two kinds of operators: primitive operators and composite operators. A primitive operator is written in C++ or Java™. A composite operator is written in SPL and contains a reusable stream subgraph.
    - Operator invocations
      The purpose of SPL is to allow users to create streams of data, which are connected and manipulated by operator invocations.
      - Operator invocation head
        The head of an operator invocation lists the output and input streams and the name of the operator that processes the data.
      - Logic clause
        The logic clause consists of local operator state that persists across operator firings, and statements that execute when an input port receives a tuple or a punctuation and thus fires.
      - Window clause
        A window is a logical container for tuples recently received by an input port of an operator.
        Punctuation
        Punctuation is a control signal that appears interleaved with the tuples in a stream.
        Tumbling windows
        A tumbling window stores incoming tuples until the window is full, then executes the operator behavior, and finally flushes all stored tuples, before it starts over from scratch.
        Sliding windows
        A sliding window is specified by providing both an eviction policy and a trigger policy.
        TimeInterval windows
        A timeInterval window collects tuples into fixed-duration intervals defined over event time.
        Partitioning
        The flag partitioned in the window clause specifies that instead of maintaining just one window per port, the operator invocation maintains one window per port per partition. A partition is a set of tuples that have the same values for the partitioning attributes.
      - Param clause
        The param clause contains code snippets that are supplied to the operator at invocation time.
      - Output clause
        The output clause assigns values to attributes in output tuples each time the operator submits data to an output stream.
      - Config clause
        The config clause specifies hints and directives that influence how the compiler builds this operator invocation, or how the runtime system executes it.
      - History
        Expressions in parameters or output assignments of operator invocations can refer directly to tuples received in the past.
    - Operator definitions
      A stream operator can be invoked to transform input streams to output streams. SPL supports two kinds of operators: primitive operators and composite operators.
  - Program structure
  - Operator config clause
    The config clause specifies hints and directives that influence how the compiler builds an operator invocation, or how the runtime system runs it.
  - SPL binary and character encoding
    SPL uses a platform independent binary and character encoding for serializing and deserializing data. This encoding is used for the FileSource, TCPSource, and UDPSource operators, as well as the FileSink, TCPSink, and UDPSink operators.
  - Mixed-mode SPL/Perl sources
    You often must parameterize application composition when you create parallel subgraphs, conditional segments, and variable length pipelines. The SPL compiler provides a Perl-based preprocessor for this purpose. SPL files that are authored with the preprocessor directives are named with the .splmm extension and are called mixed-mode SPL programs.
- SPL Compiler
  The SPL compiler is an incremental compiler that generates artifacts needed to run your stream application. These artifacts include, for example, C++ code for several entities in your SPL program and an application bundle file.
- Toolkits
  A toolkit is a set of SPL artifacts, which are organized into a package. Toolkits make SPL or native functions and primitive or composite operators reusable across different applications.
- SPLDOC markup
  You can use SPLDOC markup in the descriptions associated with the SPL artifacts in a toolkit. The spl-make-doc command generates HTML documentation from the marked up artifacts.
Glossary
Use this glossary to find terms and definitions for Teracloud® Streams.

Partitioning

The flag partitioned in the window clause specifies that instead of maintaining just one window per port, the operator invocation maintains one window per port per partition. A partition is a set of tuples that have the same values for the partitioning attributes.

While the partitioned flag in the window is a language feature, it is often supported by a partitionBy parameter, which is a feature of an operator, not of the language. But by convention, most operators implement it to follow the behavior described next. With the partitioned flag and parameter param partitionBy: x, y tuples belong to different partitions if they differ in either attribute x or y. When a new tuple arrives on a port, it is routed to the window for its own partition. Each partition window uses an eviction policy and a trigger policy as described for tumbling and sliding windows.

SPL supports the following partition eviction polices for partitioned windows:

partitionAge(float64 n): If a tuple is not inserted into a partition for n seconds, the partition is deleted.
partitionCount(uint32 c): If the number of partitions exceeds a count of c, partitions are deleted until the partition count is equal to c.
tupleCount(uint32 t): If the total number of tuples across all partitions exceeds a count of t, partitions are deleted until the tuple count is less than or equal to t.

Note: TimeInterval windows do not support partition eviction.