Compiling stream applications

Stream applications must be compiled before they can be run. Teracloud® Streams offers the SPL compiler (sc program) to compile SPL applications into a Streams application bundle file (SAB) that contains artifacts needed to run the application.

About this task

In order for the sc command to generate a SAB, it needs to be run at the top of the application directory and be able to access other toolkits that are listed as dependencies. An application gets its version and dependency information from the application directory that defines the main composite.
When instructed to compile a main composite operator, the compiler assumes the current working directory is an application directory and performs the following steps:
  1. Indexes the current working directory
  2. Looks up and loads toolkit artifacts
  3. Generates code and XML files
  4. Compiles native code
  5. Bundles artifacts
If an error is encountered, the compilation will fail and print messages about the issue. Common errors include:
  • Unable to find a toolkit the application depends on
  • Syntax errors
  • Missing required parameter arguments

Procedure

To compile an application, use the sc command and the -M, --main-composite option to specify the main composite of the application. For example, if the name of the main composite is 'WordCount' and it resides in the 'my.sample' namespace, then the following sc command should be used: 
sc -M  my.sample::WordCount

Results

After compilation, the sc program creates an output directory (called output by default), which contains all compiler-generated artifacts including the SAB (indicated by the .sab file extension). This file is specific to Teracloud® Streams and contains the necessary application artifacts to run on a single host or in a distributed Streams instance.
Consider the following application directory layout:
MyAppDir/                      # application directory
├── info.xml                   # name, version, dependency information
└── my.sample/                 # namespace directory
    ├── WordCount.spl          # .spl file with the WordCount main composite
    └── WordCountHelper.spl    # a second .spl file containing helpers
Compiling the 'WordCount' main composite by running sc -M my.sample::WordCount in the MyAppDir application directory results in the following layout:
MyAppDir/                          # application directory
├── ...                            # authored files, as before
├── toolkit.xml                    # generated toolkit index
└── output/                        # generated artifacts
    ├── my.sample.WordCount.adl    # generated application description file
    ├── my.sample.WordCount.sab    # generated SAB
    ├── bin/                       # generated PE libraries and standalone binary
    ├── build/                     # generated build artifacts
    ├── src/                       # generated source artifacts
    ├── etc/                       # configuration artifacts
    └── toolkits/                  # root of copied toolkits
Note: When compiling, the :: is used to separate the namespace qualifier from the main composite name, for example in my.sample::WordCount. The SAB uses . as a name separator in place of :: due to the restrictions on the list of characters that might appear in a file name in certain file systems.

What to do next

After compiling an application, the SAB can be run in stand-alone mode or submitted to a distributed Streams instance. See Running applications for more information.