Replacing the duplicate attribute by the new bloomFilterResult attribute

The deduplication specifies new bloomFilterResult output attribute that provides the information about duplicate detection state. It replaces the old duplicate attribute of boolean type. The Result enum specifies type of the new attribute in the com.teracloud.streams.teda.utility::BloomFilterTypes composite.

If the application implements the Variant A, then you don't need to follow the described procedure. This procedure is valid for variant B and C only, if the deduplication feature is turned on.

If the application implements the duplicates detection, then you must replace the attribute in following custom composites:

  • <namespace>.context.custom::ContextDataProcessor
  • <namespace>.chainsink.custom::PostContextDataProcessor
  • other <namespace>.*.custom::*

Procedure

  1. Open the files that implement the listed composites.

    For example: 

    
gedit <YourProjectFolder>/<namespace>.chainsink.custom::PostContexDataProcessor.spl

  2. You must update following parts of code:

    • Include use com.teracloud.streams.teda.utility::BloomFilterTypes statement in the involved files.
    • Replace the code that is accessing the old duplicate attribute:
      • if(!duplicate) by if(BloomFilterTypes.unique == bloomFilterResult)
      • if(duplicate) by if(BloomFilterTypes.duplicate == bloomFilterResult)

    You find the sample in <YourProjectFolder>/<namespace>/chainsink/sample/PostContextDataProcessor.splmm file: 

    
use <APPLICATION_NAMESPACE>.streams::*;
use <APPLICATION_NAMESPACE>.functions::*;
use com.teracloud.streams.teda.utility::BloomFilterTypes;
...
public composite PostContextDataProcessor (
...
	) as DedupedRecord = Custom(InRec; InStat) {
		logic state: {
	 		mutable int64 detectedRecordDuplicates = 0l;
		}
		onTuple InRec: {
			// ------------------------------------------------
			// custom code begin
			// ------------------------------------------------
			<%if (0 == $dedupDisabled) {
			%>if (BloomFilterTypes.duplicate == bloomFilterResult) {
				// check if duplicate, to increase statistics counter
				detectedRecordDuplicates++;
			} else {<%}# end of 'if (0 == $dedupDisabled)'%>
				mutable OutRec otuple = {};
				assignFrom(otuple, InRec);
				submit(otuple,OutStream);<%
			if (0 == $dedupDisabled) {%>
			} // end of 'else bloomFilterResult
			<%}# end of 'else (0 == $dedupDisabled)'%>
			// ------------------------------------------------
			// custom code end
			// ------------------------------------------------
		}
...
}

  3. Save and close files that implement the listed composites.