Transformer

The Chain Processor Transformer enriches the parsed records and applies custom transformation rules to the records.

Enrichment

Use the Transformer to implement the business logic for your use case. It can contain code to query the enrichment data provided by the Lookup Manager application. The ITE application provides a certain composite operator called the LookupCache, to support you with the data retrieval. You just need to set some parameters like the segment and store name, the key and the data type of the returned lookup entry. The retrieved data can be used to set or calculate attribute values in the record schema. You can also create new attributes in the record schema to hold any additional attributes. If the lookup operation fails you can reject the record.

Transformation

In addition to the enrichment and other business logic, you can also implement data transformation in the Transformer component. Transformation means to prepare your record data to meet the storage requirements of your use case. So far you have a stream of records, each record containing for example 20 attributes. Without further transformation, the ChainSink component would write these records into a CSV output file. The file would contain one line per record, the attributes would be separated with commas. A file like this can be loaded into a database table only, if the table columns match the order and number of fields in the CSV file. If the table layout does not match the record schema, you have to implement a transformation. To achieve this you can define a new Streams schema in the transformer and implement a mapping from your current record schema to this new schema. The new schema can exactly match the required layout for the target table in the database. This allows you to add and suppress attributes, and to reorder them.

This is only a simple transformation. More complex scenarios could require to store the record in multiple tables, or in different tables based on the value of some attributes. For example you could have a unified record schema containing CDRs for voice calls and SMS. Based on the type of the record you may want to write it either to the database table containing SMS records or to the table containing voice call records. Both target tables may have a different column layout. To support you with complex transformations like these, ITE introduces the concept of a TableStream. As opposed to normal records, which contain any number of SPL attributes, a tuple in a TableStream basically contains two string attributes. One holds the name of the target table in the database, the other holds a comma separated list of all attribute values to be written into the columns of the target table. Each input record from the so called RecordStream can result in multiple tuples in the TableStream, one for each target table the transformed record needs to be written to. The chain sink will create one output CSV file for each target table, containing only the relevant records. To actually implement the transformation from the record stream to a t able stream, ITE provides you a composite operator, the so called TableRowGenerator. The usage of this feature is optional.

Tuple distribution

There are some other things you have to implement in the transformer, depending on the features you use in the ITE application. If you configured the ITE application to use variant B, to implement record correlation, you need to make sure that a certain attribute in the record schema has the correct value. This attribute, named GroupID, is used to route the record to the correct Group or Context for correlation processing. If you configured the ITE application to use the built-in record deduplication, you need to make sure a certain attribute in the record schema contains a hash value, representing the record. To create this value you can concatenate other record attributes and use one of the provided hash functions on the concatenation. You should select a set of attributes that uniquely identifies the record. The creation of this hash value is not done automatically be the ITE application, because the set of attributes to choose depends on your use case.