Configuration settings

Possible values of Configuration Settings having Type “Log Level” are TRACE, DEBUG, INFO, WARN, ERROR, FATAL, or OFF.
Commonly used properties can be configured in the .env file. The other properties must be configured in the docker-compose.yaml under the respective service environment.
If you define property values in .env and docker-compose file both, then values from docker-compose will take precedence.
The dataset date format should be the same as the environment variable date format value.
In one dataset, all the inventories should use the same date format for all date formats.

The following table lists the Hyperscale Compliance properties with their default values.

Commonly used properties

Group	Property name	Type	Description	Default value
Controller Service	`API_KEY_CREATE`	Boolean	This property is by default uncommented to have the container create a new API key and print it in the logs when starting. Since the value is in the logs, this API key should only be used to bootstrap the creation of other - more secure - API keys and be discarded. Comment it once the bootstrap key is available.	true
	`LOG_LEVEL_CONTROLLER_SERVICE`	Log Level	Hyperscale logging level. This configuration controls the logging level of Hyperscale specific packages. This log level can be increased if Hyperscale service specific actions needs to be monitored closely. NOTE: It is recommended to keep this log level to INFO. Increasing the log level can impact application’s performance.	INFO
	`API_VERSION_COMPATIBILITY_STRICT_CHECK`	Boolean	These properties are used to check the version compatibility. Setting this as true will enable strict comparison of API versions of different services. In strict comparison, the complete version i.e x.y.z is compared while in other case when this property is set to false, only major version(x out of x.y.z) of APIs will be compared.	false
	`EXECUTION_STATUS_POLL_DURATION`	Milli-seconds	Time duration in which execution status is collected from different services	120000
	`LOAD_SERVICE_REQUIRE_POST_LOAD`	Boolean	Set if the Post Load step needs to be executed.	true
	`SKIP_UNLOAD_SPLIT_COUNT_VALIDATION`	Boolean	Skip ‘split count’ and ‘number of unload files generated’ validation, while determining execution status Note: For connector-specific values, see Data Source Support.	false
	`SKIP_LOAD_SPLIT_COUNT_VALIDATION`	Boolean	Skip ‘split count’ and ‘number of masked files loaded by loaded’ validation, while determining execution status Note: For connector-specific values, see Data Source Support.	false
	`CANCEL_STATUS_POLL_DURATION`	Milli-seconds	Time duration in which execution status is collected from different services for the CANCEL_INITIATED executions.	60000
	`VALIDATE_UNLOAD_ROW_COUNT_FOR_STATUS`	Boolean	Flag to control if row counts should be considered as a factor to decide the completion status of the unload step of an object.	true
	`VALIDATE_MASKED_ROW_COUNT_FOR_STATUS`	Boolean	Flag to control if row counts should be considered as a factor to decide the completion status of the masking step of an object.	true
	`VALIDATE_LOAD_ROW_COUNT_FOR_STATUS`	Boolean	Flag to control if row counts should be considered as a factor to decide the completion status of the load step of an object.	true
	`DISPLAY_BYTES_INFO_IN_STATUS`	Boolean	Flag to control the display of number of bytes in Execution Status response	false
	`DISPLAY_ROW_COUNT_IN_STATUS`	Boolean	Flag to control the display of number of rows in Execution Status response	true
Unload Service	`LOG_LEVEL_UNLOAD_SERVICE`	Log Level	Hyperscale logging level. This configuration controls the logging level of Hyperscale specific packages. This log level can be increased if Hyperscale service specific actions needs to be monitored closely. NOTE: It is recommended to keep this log level to INFO. Increasing the log level can impact application’s performance.	INFO
	`UNLOAD_FETCH_ROWS`	Number	Number of rows to be fetched from the database at a time.	10000
	`CONCURRENT_EXPORT_LIMIT`	Number	Number of concurrent mongoexport processes to be spawned.	10
Masking Service	`LOG_LEVEL_MASKING_SERVICE`	Log Level	Hyperscale logging level. This configuration controls the logging level of Hyperscale specific packages. This log level can be increased if Hyperscale service specific actions needs to be monitored closely. NOTE: It is recommended to keep this log level to INFO. Increasing the log level can impact application’s performance.	INFO
	`INTELLIGENT_LOADBALANCE_ENABLED`	Boolean	Set this to false if need to enable round robin load balancing in place of intelligent load balancing.	true
Load Service	`LOG_LEVEL_LOAD_SERVICE`	Log Level	Hyperscale logging level. This configuration controls the logging level of Hyperscale specific packages. This log level can be increased if Hyperscale service specific actions need to be monitored closely. NOTE: It is recommended to keep this log level to INFO. Increasing the log level can impact application’s performance.	INFO
	`SQLLDR_BLOB_CLOB_CHAR_LENGTH`	Number	SQLLDR properties	20000

Other properties

Group	Property name	Type	Description	Default value
Controller Service	`SOURCE_KEY_FIELD_NAMES`	String	Dataset configuration. These fields/columns are used to uniquely identify source data.	schema_name, table_name
	`LOGGING_LEVEL_ROOT`	Log Level	Logging configuration. This spring boot configuration controls the logging of all the packages/libraries getting used in application. NOTE: Increasing this Log Level will produce too many logs. It is recommended to keep this log level to WARN or below.	WARN
	`LOGGING_FILE_NAME`¹	String	Log file location & name	/opt/delphix/logs/hyperscale-controller.log
	`LOGGING_PATTERN_FILE`	String	Logging pattern for file	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_CONSOLE`	String	Logging pattern for console	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_ROLLINGFILENAME`¹	String	Archived file location & name	`/opt/delphix/logs/archived/hyperscale-controller-%d{yyyy-MM-dd}.%i.log`
	`LOGGING_FILE_MAXSIZE`	File Size (String)	Max individual file size	5MB
	`LOGGING_FILE_MAXHISTORY`	Number of Days	History in days (i.e. keep 15 days’ worth of history capped at 5GB total size)	15
	`LOGGING_FILE_TOTALCAPSIZE`	File Size (String)	Max limit the combined size of log archives	5GB
	`LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB_FILTER_COMMONSREQUESTLOGGINGFILTER`	Log Level	This configuration controls the logging information of the HTTP requests received by Hyperscale. This is by default set to DEBUG level for logging request URIs of the Incoming Requests.	DEBUG
	`API_VERSION_COMPATIBILITY_RETRY_COUNT`	Number	These properties are used to check the version compatibility. Number of times to retry the comparison if the services are not compatible.	3
	`API_VERSION_COMPATIBILITY_RETRY_WAIT_TIME`	Time in milli-seconds	These properties are used to check the version compatibility. Time to wait before next retry if the services are not compatible.	10000
	`SUPPORT_BUNDLE_AUTO_CLEANUP_INTERVAL`	Number	Poll duration in hours to check and clean the generated support bundle.	24
	`SUPPORT_BUNDLE_AUTO_CLEANUP_BUFFER`	Number	To determine when support bundle should be cleaned, use the buffer duration in days. For example, if the buffer duration is 5, then support bundles which ends later than the current time plus five days will be cleaned.	5
Unload Service	`LOGGING_LEVEL_ROOT`	Log Level	Logging configuration. This spring boot configuration controls the logging of all the packages/libraries getting used in application. NOTE: Increasing this Log Level will produce too many logs. It is recommended to keep this log level to WARN or below.	WARN
	`LOGGING_FILE_NAME`¹	String	Log file location & name	/opt/delphix/logs/hyperscale-unload.log
	`LOGGING_PATTERN_FILE`	String	Logging pattern for file	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_CONSOLE`	String	Logging pattern for console	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_ROLLINGFILENAME`¹	String	Archived file location & name	`/opt/delphix/logs/archived/hyperscale-unload-%d{yyyy-MM-dd}.%i.log`
	`LOGGING_FILE_MAXSIZE`	File Size in String	Max individual file size	5MB
	`LOGGING_FILE_MAXHISTORY`	Number of Days	History in days (i.e. keep 15 days’ worth of history capped at 5GB total size)	15
	`LOGGING_FILE_TOTALSIZECAP`	File Size in String	Max limit the combined size of log archives	5GB
	`LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB_FILTER\_COMMONSREQUESTLOGGINGFILTER`	Log Level	This configuration controls the logging information of the HTTP requests received by Hyperscale. This is by default set to DEBUG level for logging request URIs of the Incoming Requests.	DEBUG
	`SPARK_DATE_TIMESTAMP_FORMAT`	String	Spark date and timestamp format for unload. This property is only applicable for MS SQL unload image	yyyy-MM-dd HH:mm:ss.SSS
	`SPARK_SMALL_DATE_TIMESTAMP_FORMAT`	String	Spark small date and timestamp format for unload. This property is only applicable for MS SQL unload image	yyyy-MM-dd HH:mm
	`UNLOAD_CONF_DISABLE_REPLICATION`	Boolean	If set to true, disable database replication for source database. This property is only applicable for MS SQL unload image	true
	`UNLOAD_SPARK_DRIVER_MEMORY`	Integer	Spark driver memory to be consumed for unload. This property is only applicable for MS SQL unload image	90% of available memory
	`UNLOAD_SPARK_DRIVER_CORES`	Integer	Spark cores to be consumed for unload. This property is only applicable for MS SQL unload image	90% of available memory
	`UNLOAD_HIKARI_MAX_LIFE_TIME`	Integer	Value in Milliseconds, Please follow maxLifetime from Hikari Documentation.	1800000 (30 min)
	`UNLOAD_HIKARI_KEEP_ALIVE_TIME`	Integer	Value in Milliseconds, Please follow keepaliveTime from Hikari Documentation.	300000 (5 min)
	`FILE_DELIMITER`	Char	Column value delimiter character.This configuration for internal use to have data from DB tables into files. But can be updated in specific scenario Note: Use unicode char sequence for Quotation Marks characters (i.e. \u0022 unicode of double-quote) & Non-ASCII characters are NOT allowed	, (Single-Quote)
	`FILE_ENCLOSURE`	Char	This configuration for internal use to have data from DB tables into files. But can be updated in specific scenario (i.e. XML data masking) Note: Use unicode char sequence for Quotation Marks characters (i.e. \u0022 unicode of double-quote) & Non-ASCII characters are NOT allowed	“ (Double-Quote)
	`FILE_ESCAPE_ENCLOSURE`	Char	This configuration for internal use to have data from DB tables into files. But can be updated in specific scenario (i.e. XML data masking) Note: Use unicode char sequence for Quotation Marks characters (i.e. \u0022 unicode of double-quote) & Non-ASCII characters are NOT allowed	“ (Double-Quote)
Masking Service	`LOGGING_LEVEL_ROOT`	Log Level	Logging configuration. This spring boot configuration controls the logging of all the packages/libraries getting used in application. NOTE: Increasing this Log Level will produce too many logs. It is recommended to keep this log level to WARN or below.	WARN
	`LOGGING_FILE_NAME`¹	String	Log file location & name	/opt/delphix/logs/hyperscale.log
	`LOGGING_PATTERN_FILE`	String	Logging pattern for file	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_CONSOLE`	String	Logging pattern for console	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_ROLLINGFILENAME`¹	String	Archived file location & name	`/opt/delphix/logs/archived/hyperscale-%d{yyyy-MM-dd}.%i.log`
	`LOGGING_FILE_MAXSIZE`	File Size in String	Max individual file size	5MB
	`LOGGING_FILE_MAXHISTORY`	Number of Days	History in days (i.e. keep 15 days’ worth of history capped at 5GB total size)	15
	`LOGGING_FILE_TOTALSIZECAP`	File Size in String	Max limit the combined size of log archives	5GB
	`LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB_FILTER_COMMONSREQUESTLOGGINGFILTER`	Log Level	This configuration controls the logging information of the HTTP requests received by Hyperscale. This is by default set to DEBUG level for logging request URIs of the Incoming Requests.	DEBUG
	`MASKING_ILB_QUEUEINGFACTOR`	Float	This property, Queueing Factor(QF) can be used to increase the number of jobs being assigned to a Continuous Compliance Engine by the factor mentioned. (Net jobCapacity of CCE = CCE’s total jobCapacity * Queueing Factor). It can be used to increase the utilization of Continuous Compliance Engines. Increasing this property value can lead to jobs getting queued on Masking Engines. NOTE: This property is only applicable if INTELLIGENT_LOADBALANCE_ENABLED is set to true(default value).	1.0
	`MONITOR_POLL_DURATION`	Time in seconds	Used to set the frequency with which Masking Engines will be polled to fetch job Status	10s
	`ENGINE_FAILURE_POLL_DURATION`	Time in seconds	Incase of Connection/Handshake issues or unavailability of Masking Engines, the subjobs on the engine will be marked as failed after retrying for this duration.	60s
Load Service	`SQLLDR_SUCCESS_MESSAGE`	String	Message printed by sqlldr on successful loading of data.	‘successfully loaded.’
	`LOGGING_LEVEL_ROOT`	Log Level	Logging configuration. This spring boot configuration controls the logging of all the packages/libraries getting used in application. NOTE: Increasing this Log Level will produce too many logs. It is recommended to keep this log level to WARN or below.	WARN
	`LOGGING_LEVEL_COM_DELPHIX_MASKING`	Log Level	Log level for driver support. This configuration controls the logging level of the Masking Driver Support package. This Log Level can be increased when Driver Support Steps of Load Process need to be monitored closely.	INFO
	`LOGGING_FILE_NAME`¹	String	Log file location & name	/opt/delphix/logs/hyperscale-load.log
	`LOGGING_PATTERN_FILE`	String	Logging pattern for file	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_CONSOLE`	String	Logging pattern for console	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_ROLLINGFILENAME`¹	String	Archived file location & name	`/opt/delphix/logs/archived/hyperscale-load-%d{yyyy-MM-dd}.%i.log`
	`LOGGING_FILE_MAXSIZE`	File Size in String	Max individual file size	5MB
	`LOGGING_FILE_MAXHISTORY`	Number of Days	History in days (i.e. keep 15 days’ worth of history capped at 5GB total size)	15
	`LOGGING_FILE_TOTALSIZECAP`	File Size in String	Max limit the combined size of log archives	5GB
	`LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB_FILTER_COMMONSREQUESTLOGGINGFILTER`	Log Level	This configuration controls the logging information of the HTTP requests received by Hyperscale. This is by default set to DEBUG level for logging request URIs of the Incoming Requests.	DEBUG
	`SPARK_DATE_TIMESTAMP_FORMAT`	String	Spark date and timestamp format for load. This property is only applicable for MS SQL load image	yyyy-MM-dd HH:mm:ss.SSS
	`SPARK_SMALL_DATE_TIMESTAMP_FORMAT`	String	Spark small date and timestamp format for load. This property is only applicable for MS SQL load image	yyyy-MM-dd HH:mm
	`LOAD_SPARK_BATCH_SIZE`	Integer	Spark batch size for bulk load. This property is only applicable for MS SQL load image	10000
	`LOAD_SPARK_JOB_TABLE_LOCK`	Boolean	Spark job table lock for bulk load. This property is only applicable for MS SQL load image	true
	`LOAD_CONF_DISABLE_REPLICATION`	Boolean	If set to true, disable database replication for target database. This property is only applicable for MS SQL load image	true
	`LOAD_SPARK_DRIVER_MEMORY`	Integer	Spark driver memory to be consumed for load. This property is only applicable for MS SQL unload image.	90% of available memory
	`LOAD_SPARK_DRIVER_CORES`	Integer	Spark cores to be consumed for load. This property is only applicable for MS SQL unload image.	90% of available processors
	`LOAD_HIKARI_MAX_LIFE_TIME`	Integer	Value in Milliseconds, Please follow maxLifetime from Hikari Documentation.	1800000 (30 min)
	`LOAD_HIKARI_KEEP_ALIVE_TIME`	Integer	Value in Milliseconds, Please follow keepaliveTime from Hikari Documentation.	300000 (5 min)

spark_date_timestamp_format and spark_small_date_timestamp.format values should be the same for MS SQL load/unload services. Also if masking is applied on the date/timestamp datatype column, the applied inventory date format should be the same for MS SQL load/unload data format.

For each service, the file path(absolute) configured for logging.file.name and for logging.pattern.rolling-file-name has to be the same. This path is a path inside the respective container.
For each service, if the log files(configured through logging.file.nameand logging.pattern.rolling-file-name) need to be accessed outside the container, the respective log path has to be mounted by adding volume binding of that path in docker-compose.yaml for that service.