Configuration settings

Possible values of Configuration Settings having Type “Log Level” are TRACE, DEBUG, INFO, WARN, ERROR, FATAL, or OFF.
Commonly used properties can be configured in the .env file. The other properties must be configured in the docker-compose.yaml under the respective service environment.
If you define property values in .env and docker-compose file both, then values from docker-compose will take precedence.

The following table lists the Hyperscale Compliance properties with their default values.

Commonly used properties

Group	Property name	Type	Description	Default value
Controller Service	`API_KEY_CREATE`	Boolean	This property is by default uncommented to have the container create a new API key and print it in the logs when starting. Since the value is in the logs, this API key should only be used to bootstrap the creation of other - more secure - API keys and be discarded. Comment it once the bootstrap key is available.	true
	`LOG_LEVEL_CONTROLLER_SERVICE`	Log Level	Hyperscale logging level. This configuration controls the logging level of Hyperscale specific packages. This log level can be increased if Hyperscale service specific actions needs to be monitored closely. NOTE: It is recommended to keep this log level to INFO. Increasing the log level can impact application’s performance.	INFO
	`API_VERSION_COMPATIBILITY_STRICT_CHECK`	Boolean	These properties are used to check the version compatibility. Setting this as true will enable strict comparison of API versions of different services. In strict comparison, the complete version i.e x.y.z is compared while in other case when this property is set to false, only major version(x out of x.y.z) of APIs will be compared.	false
	`EXECUTION_STATUS_POLL_DURATION`	Milli-seconds	Time duration in which execution status is collected from different services	120000
	`LOAD_SERVICE_REQUIRE_POST_LOAD`	Boolean	Set if the Post Load step needs to be executed.	true
	`SKIP_UNLOAD_SPLIT_COUNT_VALIDATION`	Boolean	Skip ‘split count’ and ‘number of unload files generated’ validation, while determining execution status Note: For connector-specific values, see Data Source Support.	false
	`SKIP_LOAD_SPLIT_COUNT_VALIDATION`	Boolean	Skip ‘split count’ and ‘number of masked files loaded by loaded’ validation, while determining execution status Note: For connector-specific values, see Data Source Support.	false
	`CANCEL_STATUS_POLL_DURATION`	Milli-seconds	Time duration in which execution status is collected from different services for the CANCEL_INITIATED executions.	60000
	`VALIDATE_UNLOAD_ROW_COUNT_FOR_STATUS`	Boolean	Flag to control if row counts should be considered as a factor to decide the completion status of the unload step of an object.	true
	`VALIDATE_MASKED_ROW_COUNT_FOR_STATUS`	Boolean	Flag to control if row counts should be considered as a factor to decide the completion status of the masking step of an object.	true
	`VALIDATE_LOAD_ROW_COUNT_FOR_STATUS`	Boolean	Flag to control if row counts should be considered as a factor to decide the completion status of the load step of an object.	true
	`DISPLAY_BYTES_INFO_IN_STATUS`	Boolean	Flag to control the display of number of bytes in Execution Status response	false
	`DISPLAY_ROW_COUNT_IN_STATUS`	Boolean	Flag to control the display of number of rows in Execution Status response	true
	`NFS_STORAGE_HOST`	String	Host address of the NFS server.
	`NFS_STORAGE_EXPORT_PATH`	String	Path to the directory on the filesystem to mount.
	`NFS_STORAGE_MOUNT_TYPE`	String	The type of filesystem. Enum having values- CIFS, NFS3, NFS4.
	`NFS_STORAGE_MOUNT_OPTION`	String	The options for mount, e.g. “rw”	rw
	`APPLICATION_NAME`	String	Name of the Hyperscale application. For existing customers, this shall be analogous to the mount name used for staging area.	hs-staging-area
	`STAGING_STORAGE_TYPE`	`String`	Type of staging area. Supported values - MOUNT, AWS_S3	MOUNT
	`authMechanism`	`String`	Authentication mechanism to access AWS S3 bucket as staging area. Supported values - AWS_SECRET, AWS_ROLE Required if `STAGING_STORAGE_TYPE` is AWS_S3.
	`awsBucketName`	`String`	Name of AWS S3 bucket used as staging area. Required if `STAGING_STORAGE_TYPE` is AWS_S3 Required if `STAGING_STORAGE_TYPE` is AWS_S3
	`awsBucketRegion`	`String`	Region of AWS S3 bucket used as staging area (e.g. us-west-2). Required if `STAGING_STORAGE_TYPE` is AWS_S3
	`awsBucketPrefix`	`String`	Prefix of AWS S3 bucket used as staging area. Optional. e.g. hyperscale/oracle/
	`awsBucketDelimiter`	`String`	Delimiter for the prefix on AWS S3 bucket used as staging area.	`/`
	`stagingAwsS3SecretName`	`String`	Name of the Kubernetes secret that holds `awsAccessKey` and `awsSecretKey` information	`s3-staging-secre`t
	`awsAccessKey`	`String`	Access key ID of the AWS S3 bucket used as staging area. Not required if `stagingAwsS3SecretName` is provided.
	`awsSecretKey`	`String`	Secret Access key of the AWS S3 bucket used as staging area. Not required if `stagingAwsS3SecretName` is provided.
Unload Service	`LOG_LEVEL_UNLOAD_SERVICE`	Log Level	Hyperscale logging level. This configuration controls the logging level of Hyperscale specific packages. This log level can be increased if Hyperscale service specific actions needs to be monitored closely. NOTE: It is recommended to keep this log level to INFO. Increasing the log level can impact application’s performance.	INFO
	`UNLOAD_FETCH_ROWS`	Number	Number of rows to be fetched from the database at a time.	10000
	`CONCURRENT_EXPORT_LIMIT`	Number	Number of concurrent mongoexport processes to be spawned.	10
	`AUTO_CALCULATE_UNLOAD_SPLIT`	Boolean	Only for Oracle. if true, the unload service will unload data based ‘Automatic Calculation of Unload Split' approach by using max_records_per_split , max_parallel_connections_per_table, max_split_per_connection	false
	`MAX_RECORDS_PER_SPLIT`	long	Only for Oracle. Maximum records unloaded in an extracted split file	10000000
	`MAX_SPLIT_PER_CONNECTION`	Integer	Only for Oracle. This param indicates how many splits each database connection can take.	4
	`MAX_PARALLEL_CONNECTIONS_PER_TABLE`	Integer	Only for Oracle. This param indicates how many maximum parallel connections are allowed for a table during the unload process	10
Masking Service	`LOG_LEVEL_MASKING_SERVICE`	Log Level	Hyperscale logging level. This configuration controls the logging level of Hyperscale specific packages. This log level can be increased if Hyperscale service specific actions needs to be monitored closely. NOTE: It is recommended to keep this log level to INFO. Increasing the log level can impact application’s performance.	INFO
	`INTELLIGENT_LOADBALANCE_ENABLED`	Boolean	Set this to false if need to enable round robin load balancing in place of intelligent load balancing.	true
Load Service	`LOG_LEVEL_LOAD_SERVICE`	Log Level	Hyperscale logging level. This configuration controls the logging level of Hyperscale specific packages. This log level can be increased if Hyperscale service specific actions need to be monitored closely. NOTE: It is recommended to keep this log level to INFO. Increasing the log level can impact application’s performance.	INFO
	`SQLLDR_BLOB_CLOB_CHAR_LENGTH`	Number	This common property is used to define maximum allowed character length for BLOB,CLOB,RAW,LONG RAW and XMLTYPE data types in SQLLDR.	20000
	`LOAD_FEATURE_FLAG_REPLACE_NULL_LOB`	Boolean	Set if the automated load handling of CLOB/BLOB value (when null or empty) is needed.	true
Load Service (MS SQL)	`TARGET_DB_MAX_CONNECTION_POOL_SIZE`	Integer	This commands the size of target database connection pool for MS SQL Post Load process. This is only applicable for the post load process.	15

Other properties

Group	Property name	Type	Description	Default value
Controller Service	`SOURCE_KEY_FIELD_NAMES`	String	Dataset configuration. These fields/columns are used to uniquely identify source data.	schema_name, table_name
	`LOGGING_LEVEL_ROOT`	Log Level	Logging configuration. This spring boot configuration controls the logging of all the packages/libraries getting used in application. NOTE: Increasing this Log Level will produce too many logs. It is recommended to keep this log level to WARN or below.	WARN
	`LOGGING_FILE_NAME`¹	String	Log file location & name	/opt/delphix/logs/hyperscale-controller.log
	`LOGGING_PATTERN_FILE`	String	Logging pattern for file	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_CONSOLE`	String	Logging pattern for console	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_ROLLINGFILENAME`¹	String	Archived file location & name	`/opt/delphix/logs/archived/hyperscale-controller-%d{yyyy-MM-dd}.%i.log`
	`LOGGING_FILE_MAXSIZE`	File Size (String)	Max individual file size	5MB
	`LOGGING_FILE_MAXHISTORY`	Number of Days	History in days (i.e. keep 15 days’ worth of history capped at 5GB total size)	15
	`LOGGING_FILE_TOTALCAPSIZE`	File Size (String)	Max limit the combined size of log archives	5GB
	`LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB_FILTER_COMMONSREQUESTLOGGINGFILTER`	Log Level	This configuration controls the logging information of the HTTP requests received by Hyperscale. This is by default set to DEBUG level for logging request URIs of the Incoming Requests.	DEBUG
	`API_VERSION_COMPATIBILITY_RETRY_COUNT`	Number	These properties are used to check the version compatibility. Number of times to retry the comparison if the services are not compatible.	3
	`API_VERSION_COMPATIBILITY_RETRY_WAIT_TIME`	Time in milli-seconds	These properties are used to check the version compatibility. Time to wait before next retry if the services are not compatible.	10000
	`SUPPORT_BUNDLE_AUTO_CLEANUP_INTERVAL`	Number	Poll duration in hours to check and clean the generated support bundle.	24
	`SUPPORT_BUNDLE_AUTO_CLEANUP_BUFFER`	Number	To determine when support bundle should be cleaned, use the buffer duration in days. For example, if the buffer duration is 5, then support bundles which ends later than the current time plus five days will be cleaned.	5
	`MOUNT_ACCESS_RETRY_COUNT`	Integer	The `MOUNT_ACCESS_RETRY_COUNT` property allows the Hyperscale engine ten attempts to access the files or directory on the mount server.	10
	`MOUNT_ACCESS_TIMEOUT`	Number	The `MOUNT_ACCESS_TIMEOUT` property allows the Hyperscale engine 20 seconds to access a file or directory.	20
Unload Service	`LOGGING_LEVEL_ROOT`	Log Level	Logging configuration. This spring boot configuration controls the logging of all the packages/libraries getting used in application. NOTE: Increasing this Log Level will produce too many logs. It is recommended to keep this log level to WARN or below.	WARN
	`LOGGING_FILE_NAME`¹	String	Log file location & name	/opt/delphix/logs/hyperscale-unload.log
	`LOGGING_PATTERN_FILE`	String	Logging pattern for file	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_CONSOLE`	String	Logging pattern for console	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_ROLLINGFILENAME`¹	String	Archived file location & name	`/opt/delphix/logs/archived/hyperscale-unload-%d{yyyy-MM-dd}.%i.log`
	`LOGGING_FILE_MAXSIZE`	File Size in String	Max individual file size	5MB
	`LOGGING_FILE_MAXHISTORY`	Number of Days	History in days (i.e. keep 15 days’ worth of history capped at 5GB total size)	15
	`LOGGING_FILE_TOTALSIZECAP`	File Size in String	Max limit the combined size of log archives	5GB
	`LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB_FILTER\_COMMONSREQUESTLOGGINGFILTER`	Log Level	This configuration controls the logging information of the HTTP requests received by Hyperscale. This is by default set to DEBUG level for logging request URIs of the Incoming Requests.	DEBUG
	`UNLOAD_CONF_DISABLE_REPLICATION`	Boolean	If set to true, disable database replication for source database. This property is only applicable for MS SQL unload image	true
	`UNLOAD_SPARK_DRIVER_MEMORY`	Integer	Spark driver memory to be consumed for unload. This property is only applicable for MS SQL unload image	90% of available memory
	`UNLOAD_SPARK_DRIVER_CORES`	Integer	Spark cores to be consumed for unload. This property is only applicable for MS SQL unload image	90% of available memory
	`UNLOAD_HIKARI_MAX_LIFE_TIME`	Integer	Value in Milliseconds, Please follow maxLifetime from Hikari Documentation.	1800000 (30 min)
	`UNLOAD_HIKARI_KEEP_ALIVE_TIME`	Integer	Value in Milliseconds, Please follow keepaliveTime from Hikari Documentation.	300000 (5 min)
	`FILE_DELIMITER`	Char	Column value delimiter character.This configuration for internal use to have data from DB tables into files. But can be updated in specific scenario Note: Use unicode char sequence for Quotation Marks characters (i.e. \u0022 unicode of double-quote) & Non-ASCII characters are NOT allowed	, (Single-Quote)
	`FILE_ENCLOSURE`	Char	This configuration for internal use to have data from DB tables into files. But can be updated in specific scenario (i.e. XML data masking) Note: Use unicode char sequence for Quotation Marks characters (i.e. \u0022 unicode of double-quote) & Non-ASCII characters are NOT allowed	“ (Double-Quote)
	`FILE_ESCAPE_ENCLOSURE`	Char	This configuration for internal use to have data from DB tables into files. But can be updated in specific scenario (i.e. XML data masking) Note: Use unicode char sequence for Quotation Marks characters (i.e. \u0022 unicode of double-quote) & Non-ASCII characters are NOT allowed	“ (Double-Quote)
	`MOUNT_ACCESS_RETRY_COUNT`	Integer	The `MOUNT_ACCESS_RETRY_COUNT` property allows the Hyperscale engine ten attempts to access the files or directory on the mount server.	10
	`MOUNT_ACCESS_TIMEOUT`	Number	The `MOUNT_ACCESS_TIMEOUT` property allows the Hyperscale engine 20 seconds to access a file or directory.	20
Masking Service	`LOGGING_LEVEL_ROOT`	Log Level	Logging configuration. This spring boot configuration controls the logging of all the packages/libraries getting used in application. NOTE: Increasing this Log Level will produce too many logs. It is recommended to keep this log level to WARN or below.	WARN
	`LOGGING_FILE_NAME`¹	String	Log file location & name	/opt/delphix/logs/hyperscale.log
	`LOGGING_PATTERN_FILE`	String	Logging pattern for file	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_CONSOLE`	String	Logging pattern for console	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_ROLLINGFILENAME`¹	String	Archived file location & name	`/opt/delphix/logs/archived/hyperscale-%d{yyyy-MM-dd}.%i.log`
	`LOGGING_FILE_MAXSIZE`	File Size in String	Max individual file size	5MB
	`LOGGING_FILE_MAXHISTORY`	Number of Days	History in days (i.e. keep 15 days’ worth of history capped at 5GB total size)	15
	`LOGGING_FILE_TOTALSIZECAP`	File Size in String	Max limit the combined size of log archives	5GB
	`LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB_FILTER_COMMONSREQUESTLOGGINGFILTER`	Log Level	This configuration controls the logging information of the HTTP requests received by Hyperscale. This is by default set to DEBUG level for logging request URIs of the Incoming Requests.	DEBUG
	`MASKING_ILB_QUEUEINGFACTOR`	Float	This property, Queueing Factor(QF) can be used to increase the number of jobs being assigned to a Continuous Compliance Engine by the factor mentioned. (Net jobCapacity of CCE = CCE’s total jobCapacity * Queueing Factor). It can be used to increase the utilization of Continuous Compliance Engines. Increasing this property value can lead to jobs getting queued on Masking Engines. NOTE: This property is only applicable if INTELLIGENT_LOADBALANCE_ENABLED is set to true(default value).	1.0
	`MONITOR_POLL_DURATION`	Time in seconds	Used to set the frequency with which Masking Engines will be polled to fetch job Status	10s
	`ENGINE_FAILURE_POLL_DURATION`	Time in seconds	Incase of Connection/Handshake issues or unavailability of Masking Engines, the subjobs on the engine will be marked as failed after retrying for this duration.	60s
	`MOUNT_ACCESS_RETRY_COUNT`	Integer	The `MOUNT_ACCESS_RETRY_COUNT` property allows the Hyperscale engine ten attempts to access the files or directory on the mount server.	10
	`MOUNT_ACCESS_TIMEOUT`	Number	The `MOUNT_ACCESS_TIMEOUT` property allows the Hyperscale engine 20 seconds to access a file or directory.	20
Load Service	`SQLLDR_SUCCESS_MESSAGE`	String	Message printed by sqlldr on successful loading of data.	‘successfully loaded.’
	`LOGGING_LEVEL_ROOT`	Log Level	Logging configuration. This spring boot configuration controls the logging of all the packages/libraries getting used in application. NOTE: Increasing this Log Level will produce too many logs. It is recommended to keep this log level to WARN or below.	WARN
	`LOGGING_LEVEL_COM_DELPHIX_MASKING`	Log Level	Log level for driver support. This configuration controls the logging level of the Masking Driver Support package. This Log Level can be increased when Driver Support Steps of Load Process need to be monitored closely.	INFO
	`LOGGING_FILE_NAME`¹	String	Log file location & name	/opt/delphix/logs/hyperscale-load.log
	`LOGGING_PATTERN_FILE`	String	Logging pattern for file	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_CONSOLE`	String	Logging pattern for console	`%d{dd-MM-yyyy HH:mm:ss.SSS} \[%thread\] %-5level %logger{36}.%M - %msg%n`
	`LOGGING_PATTERN_ROLLINGFILENAME`¹	String	Archived file location & name	`/opt/delphix/logs/archived/hyperscale-load-%d{yyyy-MM-dd}.%i.log`
	`LOGGING_FILE_MAXSIZE`	File Size in String	Max individual file size	5MB
	`LOGGING_FILE_MAXHISTORY`	Number of Days	History in days (i.e. keep 15 days’ worth of history capped at 5GB total size)	15
	`LOGGING_FILE_TOTALSIZECAP`	File Size in String	Max limit the combined size of log archives	5GB
	`LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_WEB_FILTER_COMMONSREQUESTLOGGINGFILTER`	Log Level	This configuration controls the logging information of the HTTP requests received by Hyperscale. This is by default set to DEBUG level for logging request URIs of the Incoming Requests.	DEBUG
	`LOAD_SPARK_BATCH_SIZE`	Integer	Spark batch size for bulk load. This property is only applicable for MS SQL load image	10000
	`LOAD_SPARK_JOB_TABLE_LOCK`	Boolean	Spark job table lock for bulk load. This property is only applicable for MS SQL load image	true
	`LOAD_CONF_DISABLE_REPLICATION`	Boolean	If set to true, disable database replication for target database. This property is only applicable for MS SQL load image	true
	`LOAD_SPARK_DRIVER_MEMORY`	Integer	Spark driver memory to be consumed for load. This property is only applicable for MS SQL unload image.	90% of available memory
	`LOAD_SPARK_DRIVER_CORES`	Integer	Spark cores to be consumed for load. This property is only applicable for MS SQL unload image.	90% of available processors
	`LOAD_HIKARI_MAX_LIFE_TIME`	Integer	Value in Milliseconds, Please follow maxLifetime from Hikari Documentation.	1800000 (30 min)
	`LOAD_HIKARI_KEEP_ALIVE_TIME`	Integer	Value in Milliseconds, Please follow keepaliveTime from Hikari Documentation.	300000 (5 min)

For each service, the file path(absolute) configured for logging.file.name and for logging.pattern.rolling-file-name has to be the same. This path is a path inside the respective container.
For each service, if the log files(configured through logging.file.nameand logging.pattern.rolling-file-name) need to be accessed outside the container, the respective log path has to be mounted by adding volume binding of that path in docker-compose.yaml for that service.

Oracle Connector- From the 23.0.0 release onwards, you no longer need to configure any unload/load date format environment variables to mask date/timestamp columns as Hyperscale Compliance introduces a feature to handle multiple date formats automatically in a single Hyperscale job. Therefore, the following environment properties configured in the docker-compose.yaml/values.yaml file will no longer be used in the product. You can safely remove these properties:

JDBC_DATE_TIMESTAMP_FORMAT -- For Unload
SQLLDR_DATE_TIMESTAMP_FORMAT -- For Load

MSSQL Connector- From the 24.0.0 release onwards, a new Hyperscale feature can automatically handle multiple date formats in a single Hyperscale job. Therefore, you no longer need to configure any unload/load date format environment variables to mask date/timestamp columns.

Following the 24.0.0. release the below environment properties configured in the docker-compose.yaml/values.yaml file will not be used and you can safely remove these properties from unload and load services:

SPARK_TIMESTAMP_FORMAT
SPARK_DATE_FORMAT
SPARK_SMALL_DATE_TIMESTAMP_FORMAT