Home
Developing
Learn about different Teracloud Streams tools and features to write, compile, run, and test streams applications.
Developing stream applications
Try out tutorials and explore details about stream application development, SPL features, and best practices.
Tutorials
Learn how to create simple stream applications by completing the tutorials.
Filtering application
Stream applications can be used to extract information from large data sets so analysis on relevant information is performed.

Welcome
Learn about the core capabilities of Teracloud® Streams, its architecture, and key concepts.
Installing
Use this information to install, upgrade, and uninstall the Teracloud® Streams product.
Configuring
Create a basic or an enterprise domain which is a single point for configuring and managing common resources, security, and instances.
Administering
Administer the product by using the Teracloud® Streams graphical user interface, APIs, or the streamtool command-line interface.
Developing
Learn about different Teracloud Streams tools and features to write, compile, run, and test streams applications.
- Development concepts
  Development of stream applications consists of several components such as operators, streams, tuples, Streams Processing Language, toolkits, and more.
- Developing stream applications
  Try out tutorials and explore details about stream application development, SPL features, and best practices.
  - Tutorials
    Learn how to create simple stream applications by completing the tutorials.
    - SPL Tutorials
      Try out the following SPL tutorials to learn about the distributed data flow language including its program structures, types, functions, and other components tailored for processing streaming data.
    - Filtering application
      Stream applications can be used to extract information from large data sets so analysis on relevant information is performed.
      - Reading data from a source file
        Suppose that the stock transaction data for a day contains the following sample data in a file named StockTrades.csv.
      - Filtering large data sets
        In this example, the stream application needs to filter the stock transaction data for IBM transaction records. You use the Filter operator to extract relevant information from potentially large volumes of data. As shown, the input for the Filter operator is all the transactions; the output is only the IBM transactions.
      - Analyzing the subset of data
        To determine the highest and lowest trading price, you need to write the logic for determining those values. In addition, you need some local variables to track the highest and lowest price that is processed so far. You can use the Custom operator to accomplish these tasks. As shown, the input stream is only the IBM transactions; there is no output stream for the customized operator.
      - Combining the operators into a graph
        A stream application that is written in SPL code is a directed graph where streams of data (edges in the graph) flow into and out of operators (nodes in the graph). Operators perform analysis on the incoming data; and produce new streams of data which then flow into subsequent operators. You can use this basic pattern, input data --> operator --> output data, to create an assembly line of analytics where downstream operators use data from upstream operators to perform analytics and produce new data.
      - Testing the application
        Test the stream application to verify that the code generated from the graph is correct. You must build (compile) the application before you can run it. You can run the application in stand-alone mode or in a distributed environment..
    - Reducing tuple size application
      In addition to filtering large data sets, you can reduce the volume of data by reducing the size of the tuples from the input stream.
  - Application directory files and structure
    Learn about the different files and common directories that make up an SPL application. View the recommended structure so the SPL compiler and other Teracloud® Streams development tools work best.
  - Compiling stream applications
    Stream applications must be compiled before they can be run. Teracloud® Streams offers the SPL compiler (sc program) to compile SPL applications into a Streams application bundle file (SAB) that contains artifacts needed to run the application.
  - Application bundle files
    A Streams application bundle file (SAB) is a single, relocatable file that contains artifacts needed to run your application. The file can be invoked directly to run the application on a single host or submitted to a Streams instance.
  - SPL features
    SPL provides features and functions to help you write your applications to fulfill your business needs.
  - Best practices
    Use these tips to write operators that perform effectively in their application, and in other applications.
  - Performance recommendations
    You can use these performance options and recommendations to improve your stream application.
- Developing native functions
  Extend SPL's computational capabilities by creating native functions written in C++ or Java.
- Developing custom operators
  Create custom operators if shipped toolkits do not provide the necessary logic or behavior needed for your stream applications.
- Developing custom toolkits
  Bundle and reuse custom functions and operators across several stream applications by creating custom toolkits.
- Working with files
  Applications and operators can read and write to files in several ways. If relative paths are used, the data directory is used as the root of relative paths. To enable logic to access the runtime application or toolkit directory hierarchies, language-specific functions can be used.
- Enabling Streams data exchange
  Teracloud® Streams provides a data exchange REST API for inserting and retrieving tuples within a job to easily integrate with other data services and external applications. Stream applications can enable the data exchange feature by using one or more Endpoint operators.
- Debugging stream applications
  Debug stream applications using the interactive, command line-based Streams Debugger (sdb).
Troubleshooting
Resolve problems with Teracloud® Streams using the troubleshooting tools provided with the product as well as the resources offered by Teracloud Support.
Reference
Find details on the SPL language, toolkits, APIs, commands, and more.
Glossary
Use this glossary to find terms and definitions for Teracloud® Streams.

Tutorial: Filtering application

Stream applications can be used to extract information from large data sets so analysis on relevant information is performed.

About this task

Suppose you wanted to know the highest and lowest trading price for IBM® stocks for a specific day. To determine this, you can write a stream application to filter the stock transaction data to process only the IBM® transaction records, and then process the IBM® transaction records to find the highest and lowest trading price for that day.

Tutorial: Filtering application

About this task

Procedure