Optimize the file processing throughput

If you require faster file processing per chain, then you can optimize the throughput by adding an thread to the processing chain. This change would increase the number of used CPU cores by a single chain. As a consequence the file reading and parsing runs in an own thread. The remaining chain operators for record conversion, validation, enrichment run in another thread.

About this task

Add a threadedPort configuration to input stream of the RecordConverter operator.

Procedure

  1. In the file <namespace>/demoapp.chainprocessor.reader.custom/RecordConverter.spl, find the operator with the composite input stream I as input port.
  2. To create a new thread on the RecordConverter operator input port, add the following config clause: 
    
  config threadedPort : queue(I, Sys.Wait, 1000);