Filename distribution

The File Ingestion component uses several mechanisms to distribute filenames to the chains.

Once the filenames have been processed by the functions described in the parent topic, they need to be distributed to the chains. The distribution may occur in two steps, depending on the configured variant. If the ITE application is configured as variant C, as shown in the following diagram, a logical distribution to the correct group is done first. Than a load distribution is performed within each group. For variants A and B, only a load distribution occurs. For more information about the File Group Split see the chapter regarding variants.

The File Ingestion component can be configured to use one of two different load distribution mechanisms. With the round-robin distribution filenames are distributed in a round robin fashion. This is simple and incurs no overhead. The disadvantage is that the chains may not be loaded equally if the sizes of the input files are very different. Use this mechanism if you expect input files of roughly the same size. In contrast, equal-load distribution keeps track of the processing state of the chains. Once a chain has finished a file it sends an acknowledgement to the load distribution, and immediately receives the next filename. This should lead to well load balanced chains, but has the disadvantage of incurring some overhead for the acknowledgment processing.. Choose this configuration if you expect files of significantly different sizes. In both cases, filename queues are kept in memory and are not limited in size, to cope with bursts of incoming data files.