Home
Developing
Develop stream applications with the Streams Processing Language (SPL), Java, and Python.
Writing stream applications
Teracloud® Streams provides features and functions to help you write your applications to fulfill your business needs.
Best practices for developing applications
Use these tips to write operators that perform effectively in their application, and in other applications.
Stream punctuation
Punctuation marks are an in-band signals that are inserted between tuples in a stream. They indicate breaks in the data, which can be used by the operator logic.
Window punctuation
Window punctuation is signals that are inserted into a stream that are related to the semantics of the operator. Window punctuation, called Sys.WindowMarker in SPL and Punctuation::WindowMarker when implemented in C++ primitive operators, divide the stream in window boundaries that are defined by the user.
Configuring input ports
You have the following three choices when you configure the punctuation mode of input ports: Expecting, Oblivious, and WindowBound.

Welcome
Learn about the core capabilities of Teracloud® Streams, its architecture, and key concepts.
Installing
Use this information to install, upgrade, and uninstall the Teracloud® Streams product.
Configuring
Create a basic or an enterprise domain which is a single point for configuring and managing common resources, security, and instances.
Administering
Administer the product by using the Teracloud® Streams graphical user interface, APIs, or the streamtool command-line interface.
Developing
Develop stream applications with the Streams Processing Language (SPL), Java, and Python.
- Development concepts
  Development of stream applications consists of several components such as operators, streams, tuples, Streams Processing Language, toolkits, and more.
- Writing stream applications
  Teracloud® Streams provides features and functions to help you write your applications to fulfill your business needs.
  - Streams Processing Language
    Streams Processing Language (SPL) is a distributed data flow composition language. SPL has primitive types, program structures, and definitions that are tailored for streaming data.
  - Developing a simple application
    Companies build stream applications when they need to extract information from large data sets so that they can analyze only the information that is relevant to them.
  - Developing another simple application
    In addition to filtering large data sets, you can reduce the volume of data by reducing the size of the tuples from the input stream.
  - Best practices for developing applications
    Use these tips to write operators that perform effectively in their application, and in other applications.
    - Operator granularity
      To design effective stream applications, operators need to be sized appropriately. Effective operators perform a single task.
    - Stream graph granularity
      Design stream graphs with readability and reusability in mind, similar to the process you follow when you design operators.
    - Naming conventions
      Following standardized naming practices improves program readability.
    - Stream punctuation
      Punctuation marks are an in-band signals that are inserted between tuples in a stream. They indicate breaks in the data, which can be used by the operator logic.
      - Window punctuation
        Window punctuation is signals that are inserted into a stream that are related to the semantics of the operator. Window punctuation, called Sys.WindowMarker in SPL and Punctuation::WindowMarker when implemented in C++ primitive operators, divide the stream in window boundaries that are defined by the user.
        Configuring input ports
        You have the following three choices when you configure the punctuation mode of input ports: Expecting, Oblivious, and WindowBound.
        Configuring output ports
        You have the following three choices when you configure the punctuation mode of output ports: Generating, Free, and Preserving.
      - Final punctuation
        Final punctuation is a special marker that indicates the end of a stream. Final punctuation, called Sys.FinalMarker in Streams Processing Language (SPL) and Punctuation::FinalMarker when implemented in C++ primitive operators, indicates that the stream no longer carries new tuples.
    - Port mutability
      When you define operator models while you develop a primitive operator, you must configure the mutability of tuples that flow through each input port and output port in the operator model.
    - Error handling
      Operators might need to contend with errors in the form of erroneous input tuples, bad parameters, or unexpected values that are returned from functions. Recognizing errors is often easy. Error handling is more subtle and depends on where the error occurs.
    - Control ports
      Control ports are operator input ports that receive tuples that update the internal operator state. These ports do not generate output tuples through submit() calls. Control ports are useful to dynamically control the behavior of the operator by changing a filtering condition, for example.
    - Output assignments
      When you write a generic primitive operator, you can allow assignments to output tuple attributes through the output clause. Output assignments can be either plain assignments or assignments with custom output functions.
    - Default values for operator parameters
      Operators that accept optional parameters at their invocation site must provide default values or require the parameter to be provided. Choose default values that are safe, that do not adversely affect performance, and that follow the principle of least surprise.
    - Window support
      When you write a primitive operator, you might need to maintain a set of recently processed input tuples to compute the operator output. It might be sufficient to use Streams Processing Language (SPL) support for windows.
    - Documenting operators and toolkits
      When you develop an operator, a composite operator, or a toolkit for third-party use, you must document its functionality, expected inputs, and generated outputs.
    - Using application configuration objects to access secure data
      You can use application configuration objects to access securely stored configuration data from your Teracloud® Streams environment.
  - Developing applications with user-defined parallelism
    The @parallel annotation allows developers to easily take advantage of data-parallelism. In the streaming context, data-parallelism means replicating copies of operators, and splitting streams so that different tuples go to each set of replicas. The process of replicating operators, and creating all of the new streams to connect them, is called the parallel transformation. When developers add @parallel to a primitive or composite operator invocation, Teracloud Streams will perform the parallel transformation at submission time. The goal is to improve overall application throughput by executing the replicas in parallel.
  - Developing applications with consistent regions
    Because of business requirements, some applications require that all tuples in an application are processed at least once. You can use a consistent region in your stream applications to avoid data loss due to software or hardware failure and meet your requirements for at-least-once processing.
  - Developing applications with views
    To make streaming data available to external programs, your SPL application must create at least one view. A view defines the set of attributes that can appear in a specific viewable data stream.
  - Developing applications with event time
    Event time is a simple model which supports streams processing where time is not derived from the system time of the machine Teracloud® Streams is running on, but from a time value associated with each tuple. In a graph enabled for event time, tuples have an attribute which holds their time value. The time value for each tuple enables operations such as grouping tuples with a time value falling within specified time intervals, and running aggregate calculations on the group.
  - Logging and tracing options for SPL application developers
    SPL application developers can write messages to product log and trace files. Log and trace files help administrators, instance owners, application developers, and the support team to identify, diagnose, and resolve problems with the Teracloud® Streams and applications.
- Enabling Streams data exchange
  Teracloud® Streams provides a data exchange REST API for inserting and retrieving Streams job data. You can add one or more endpoint operators to your Streams application to enable Streams data exchange. Providing a standard REST interface for Streams data exchange eases the integration of Streams data with other data services and externally hosted applications.
- Compiling stream applications
  The Streams Processing Language (SPL) Compiler is called sc and is included with the Teracloud® Streams installation package. You compile SPL to create files to run on your Teracloud Streams instance.
- Performance improvements for stream applications
  You can improve the processing of your stream application by following these guidelines and tips.
- Debugging stream applications
  The Streams Processing Language (SPL) Streams Debugger (sdb) provides a command line-based interactive debugger for debugging stream applications.
- Developing toolkits and operators
  You can create custom toolkits and operators for your applications.
Troubleshooting
Resolve problems with Teracloud® Streams using the troubleshooting tools provided with the product as well as the resources offered by Teracloud Support.
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
Glossary
Use this glossary to find terms and definitions for Teracloud® Streams.

Configuring input ports

You have the following three choices when you configure the punctuation mode of input ports: Expecting, Oblivious, and WindowBound.

An operator input port is punctuation Expecting when all configurations of the operator require a punctuated stream to function correctly. One such example is a primitive operator that uses a punctuation-based window independent of how the operator invocation is configured at the Streams Processing Language (SPL) level. For punctuation Expecting input ports, the SPL compiler performs extra error checks to ensure that the port is connected to a punctuated stream.

The most common configuration for an input port is to be Oblivious to punctuation. Developers can use this configuration when the operator processing logic does not use window punctuation to change its internal state or to generate output tuples. An example of operators with Oblivious input port is the Filter operator from the SPL Standard Toolkit. The Filter operator function is to drop data items from the incoming stream that is based on a specified predicate and independently of window punctuation. The SPL compiler allows both punctuated and punctuation free streams to connect to Oblivious input ports.

Input ports have a WindowBound configuration when the port might be configured with the operator window clause and with a punctuation eviction policy (punct). If the window clause indicates a punctuation-based window, the port behaves as Expecting. If the window clause indicates other configurations, the port is considered Oblivious. The WindowBound configuration is especially useful when you develop primitive operators that can have different window eviction policies and one of them is the punctuation-based eviction. One such example is the Aggregate operator from the SPL Standard Toolkit, which allows count, delta, time, and punctuation-based (punct) eviction policies. In this case, the SPL compiler reports an error when Aggregate is invoked with a punctuation-based window and is connected to a punctuation free stream.