Kerberos configuration

Kerberos authentication provides a more secure way of accessing HDFS by providing user authentication.

To use Kerberos authentication, you must configure the authPrincipal and authKeytab operator parameters at compile time.

The authPrincipal parameter specifies the Kerberos principal, which is typically the principal that is created for the Streams instance owner.

The authKeytab parameter specifies the keytab file that is created for the principal.

For Kerberos authentication it is required to create a Principal and a Keytab for each user.

If you use ambari to configure your hadoop server, you can create principals and keytabs via ambari (Enable Kerberos).

More details about Kerberos configuration:

https://developer.ibm.com/hadoop/2016/08/18/overview-of-kerberos-in-iop-4-2/

Copy the created keytab into local streams server for example in etc directory of your SPL application.

Before you start your SPL application, you can check the keytab with kinit tool

kinit -k -t KeytabPath Principal

KeytabPath is the full path to the keytab file

For example:

kinit -k -t /home/streamsadmin/workspace/myproject/etc/hdfs.headless.keytab hdfs-hdp2@HDP2.COM

In this case HDP2.com is the kerebors realm and the user is hdfs.

Here is an SPL example to write a file into hadoop server with kerberos configuration.

() as lineSink1 = HDFS2FileSink(LineIn) {
    param
        authKeytab     : "etc/hdfs.headless.keytab" ;
        authPrincipal  : "hdfs-hdp2@HDP2.COM" ;
        configPath     : "etc" ;
        file           : "LineInput.txt" ;
}

The HDSF configuration file core-site.xml has to be copied into local etc directory.