Job configuration overlays
A job configuration overlay file is a JSON file that contains name-value pairs for job configuration parameters. You can use a job configuration overlay file to define, save, and distribute the submission-time configuration or to change the configuration of a running job.
If you want to modify a few settings, you can create a job configuration overlay file that includes only the relevant sections and parameters. Only the settings that are specified in the job configuration overlay file are changed. The settings in the job configuration overlay file overwrite any previously set values.
Structure of a job configuration overlay file
The following outline shows the order of the sections in a job configuration overlay file:
comment
jobConfigOverlays
jobConfig
jobName
jobGroup
submissionParameters
dataDirectory
preloadApplicationBundles
tracing
deploymentConfig
fusionScheme
fusionTargetPeCount
placementScheme
threadingModel
dynamicThreadingThreadCount
dynamicThreadingElastic
parallelRegionConfig
fusionType
jobResourceSharing
proposedOperatorsPerResource
operatorConfigs
operators
nameSpec
restartable
relocatable
partitionIsolation
partitionGroupIsolation
hostIsolation
declaredHost
hostColocationGroups
groupName
members
hostExlocationGroups
groupName
members
partitionColocationGroups
groupName
members
partitionExplocationGroups
groupName
members
poolPlacementGroups
name
OperatorsInPool
nameSpec
inPoolIndex
membershipMode
size
tags
declaredHosts
numberOfResourcesToRequest
configInstructions
ignoreBundleRelationalPlacements
ignoreBundleHostPlacements
convertTagSet
originalTagSet
targetTagSet
changeThreadedPortSettings
queueSize
adjustmentSection
targetParallelRegion
regionName
newWidth
numberOfResourcesToBeAdded
operationConfig
overrideResourceLoadProtection
forcePeStopped
enableAutomaticAugmentation
results
section. This content is generated when you preview how the job will be submitted to the
instance. For information about the parameters that you can specify in a job configuration overlay file, see Job configuration overlays reference.
Syntax of a job configuration overlay file
If you create or edit a job configuration overlay file, you must use the correct syntax. Specifically, ensure that you include the commas in the correct locations and that you include opening and closing braces, brackets, and double quotation marks.
The following code shows an example of a job configuration overlay file with the correct syntax:
{
"jobConfigOverlays": [
{
"jobConfig": {
"jobName": "myJob",
"jobGroup": "myJobGroup",
"dataDirectory": "myDataDir",
"submissionParameters": [
{ "name": "streams::pe.display", "value": "host_name:port_number" }
]
},
"deploymentConfig": {
"placementScheme": "balancedInstance",
"fusionScheme": "automatic"
},
"operatorConfigs": {
"operators": [
{ "nameSpec": "Op_1", "declaredHost": "hostB" },
{ "nameSpec": "Op_2", "restartable": "true" },
{ "nameSpec": "Op_3", "hostIsolation": "true" }
],
"poolPlacementGroups": [
{
"name": "myPool",
"size": 10,
"tags": [ "tag1", "tag2" ],
"operatorsInPool": [
{ "nameSpec": "Op_2" },
{ "nameSpec": "Op_3" }
]
}
]
}
}
]
}
Previewing or submitting a job with the job configuration overlay file
You can manually create a job configuration overlay file, or you can generate a job configuration overlay file when you preview how a job will be submitted to your instance. For more information about generating a job configuration overlay file from a preview, see Previewing a job before you deploy it.
The job configuration overlay file that is generated when you run a preview includes all of the settings that were configured for the job. For complex applications, the job configuration overlay file might be long and contain many entries.
A job configuration overlay file is one way that you can specify submission-time configuration
parameters for a job before you deploy a stream application to your environment. (Alternatively, you can specify submission-time configuration
parameters from the Teracloud® Streams
Console or by using the streamtool submitjob command.) However, unlike
manually specifying submission-time configuration parameters when you submit a job, a job
configuration overlay file enables you to reuse or share a configuration. It also reduces
the likelihood of entering information incorrectly if you want to specify submission-time
configuration parameters from the interactive streamtool
interface.
For information about how you can use submission-time configuration parameters to improve the runtime performance of the application without recompiling it, see Configuring submission-time parameters to optimize job deployment.
You can use a job configuration overlay file when you preview or submit a job in the following ways:
- From Streams Console, import the job configuration overlay file when you run or preview a job. You can import it by clicking the Import icon at the top of the page and selecting Import job configuration file.
- From the streamtool command-line interface, choose the appropriate option:
- To preview a job with a job configuration overlay file, run the
streamtool submitjob command with the --preview
option and specify the name of your job configuration overlay file, for
example:
streamtool submitjob --preview -g jobConfigOverlay.json myAppBundle.sab
Note: When you preview the job, a new job configuration overlay file in JSON format is automatically generated. The default file name is appName_config.json, where appName is the name of your application. You can change the file name by using the --out_jobConfig option. - To run a job with a job configuration overlay file, run the streamtool
submitjob command and specify the name of your job configuration overlay
file:
streamtool submitjob -g jobConfigOverlay.json myAppBundle.sab
- To preview a job with a job configuration overlay file, run the
streamtool submitjob command with the --preview
option and specify the name of your job configuration overlay file, for
example:
Modifying a running job with a job configuration overlay file
You can use the job configuration overlay file with the streamtool updateoperators command to change the width of a parallel region in a running job. For example:
streamtool updateoperators myjob -g jobConfigOverlay.json