Reader

The Chain Processor Reader extracts records from the input files, and validates them.

PreFileReader

This component can be used to optionally perform any preprocessing on the given input file. Usage of this component can be disabled via configuration parameters. You can implement similar functionality here, like in the File Ingestion’s FileTypeValidator. There are some differences. In this component you can enhance the Stream Schema with new attributes and the component runs in the Chain, so it is automatically parallelized and is less likely to become a bottleneck. You cannot reject the input file in this component, this is only possible in the FileTypeValidator. If your use case allows putting the required functionality in either of the two components, it is recommended to prefer the PreFileReader over the FileTypeValidator.

FileReader

This component hosts the FileSource operator that actually reads the content of the input file and the file parser you need to extract records from the file content. You can select one of the three parsers supplied in the toolkit, or you can implement your own custom parser. You can configure multiple FileReaders, each of them can be customized to a certain input file format. At runtime the framework chooses the correct reader based on a certain file type attribute you have to set either in the PreFileReader or in the FileTypeValidator component. The file type attribute serves as an index into the list of configured file readers. For each file reader you can configure aspects like the type of the parser (ASN.1, binary fixed-length, CSV, Custom), the output streams type of the reader, the mapping and structure documents of the parser, the block size of the FileSource, and many other parameters possibly specific to a certain parser type.

Converter

Use this component to convert the output of the parser to the stream type and format you need in the transformer or subsequent components of the application. You can implement simple conversions here like, for example, creating an SPL timestamp from a string containing date and time information, but you can also use this component to implement a common output schema for multiple input formats. For example if you have one parser extracting data usage records per subscriber and another one that extracts call related records per subscriber, you can use the converters in each of the file readers to produce a common record with a Streams type containing a superset of both parser outputs.

Validator

Use the Validator to implement checking the extracted records for logical errors, or format errors that are not captured during the parsing process. For example if the parser extracted an integer attribute and you want to apply a further check if the integer number is within a certain range of allowed values that check can be implemented here. More complex validations could involve checking sets of attributes for plausibility. In case the validator decides that a certain record is invalid, it should mark the record accordingly. Invalid records will be logged by the ITE application in certain files, so the user can inspect the erroneous records.