Examples

The following example scans an HDFS directory on IBM Cloud. The hdfsUser and hdfsPassword parameters are used to provide the username and password for authentication. 


stream<rstring filename> Files = HDFS2DirectoryScan() {
    param
        directory     : "/user/clsadmin/works";
        hdfsUri       : "webhdfs://hdfsServer:8443";
        hdfsPassword  : "password";
        hdfsUser      : "clsadmin";
        sleepTime     : 2.0;
}
The following example scans an HDFS directory on IBM Cloud. The hdfsUser and hdfsPassword parameters are now defined in credentials JSON string. 

param 
    expression<rstring> $credentials : getSubmissionTimeValue("credentials", "{
        \"user\"     : \"clsadmin\",
        \"password\" : \"IAE-password\",
        \"webhdfs\"  : \"webhdfs://ip-address:8443\"
    }"


graph 
    stream<rstring filename> Files = HDFS2DirectoryScan() {
        param
            directory     : "/user/clsadmin/works";
            credentials   : $credentials;
            sleepTime     : 2.0;
    }

The following example scans a HDFS directory every two seconds. The hdfsUri parameter in this case overrides the value that is specified by the fs.defaultFS option in the core-site.xml . 


stream<rstring filename> Files = HDFS2DirectoryScan() {
    param
        directory     : "/user/myuser/";
        hdfsUri       : "hdfs://hdfsServer:8020";
        sleepTime     : 2.0;
}