Developing and running applications that use the HBase Toolkit

To create applications that use the HBase Toolkit, you must configure the SPL compiler to be aware of the location of the toolkit.

Before you begin

  • An Apache HBase server is required to use the operators in this toolkit. Apache HBase requires an Apache Hadoop environment.

About this task

When developing applications, you can fully qualify the operators by prefixing the namespace or include a use directive to bring the necessary namespace(s) into scope.

When compiling the application, the location of the toolkit must be communicated to the compiler.

Procedure

  1. For Apache HBase, you must configure your compile and runtime environment by setting the HBASE_HOME environment variable:
    • If your Streams application will be running on the same resource where HBase is installed, set HBASE_HOME to the directory that contains HBase.
    • Otherwise, copy the /hbase-install-dir/lib directory to the file system where Streams will be running, making sure to follow symbolic links: 
      
  cp -Lr path-to-hbase/lib /path-on-streams-host
    • Set HBASE_HOME to the directory you created, for example, to path-on-streams-host.
  2. Supply the operator with HBase configuration information. By default, HBASE_HOME/conf is searched for the hbase-site.xml. If Streams will not be running on the same resource as the HBase installation, you can do one of the following:
    • Create a conf directory in the same directory you created in step 1 and copy the hbase-site.xml from your HBase installation to that directory.
    • Copy the hbase-site.xml file from the conf directory in your HBase installation into a directory accessible by your application and use the hbaseSite parameter to specify the path to the copied file.
  3. Develop your application. To avoid the need to fully qualify the operators, add a use directive in your application.
    • For example, you can add the following clause in your SPL source file: 
      
use com.teracloud.streams.hbase::*;
      You can also specify a use clause for individual operators by replacing the asterisk (*) with the operator name. For example: 
      
use com.teracloud.streams.hbase::HBASEDelete;
  4. Configure the SPL compiler to find the toolkit root directory. Use one of the following methods:
    • Set the STREAMS_SPLPATH environment variable to the root directory of a toolkit or multiple toolkits (with : as a separator). For example: 
      
export STREAMS_SPLPATH=$STREAMS_INSTALL/toolkits/com.teracloud.streams.hbase
    • Specify the -t or --spl-path command parameter when you run the sc command. For example: 
      
sc -t $STREAMS_INSTALL/toolkits/com.teracloud.streams.hbase -M MyMain
      where MyMain is the name of the SPL main composite.

      Note: These command parameters override the STREAMS_SPLPATH environment variable.

  5. Build your application using the sc command.

  6. Remember to set HBASE_HOME and HADOOP_HOME if you did not use the hbaseSite parameter.
    • This can be done using the Streams Console or command-line with streamtool setproperty --application-ev.
  7. Run the application.
    • You can launch the application in stand-alone mode using the standalone program located in the output directory.
    • You can submit the application in a distributed environment, as a job, to a running Streams instance using the streamtool submitjob command or Streams Console.