Fixed issues

ECO-10949

When using file connectors, you are no longer required to amend the unload container environment variable USER_PROVIDED_FORMAT_FILE to USER_PROVIDED_FORMAT_FILES in unload-deployment.yaml.

HM-3954

MSSQL: Fixed an issue where empty char and varchar were unloaded as a "" value.

HM-3956

MSSQL: Fixed an issue where leading and trailing whitespaces from a source table were not correctly loaded to the target table.

HM-3891

When masking in Hyperscale, if there were fewer split files (number of masking jobs on Continuous Compliance engines) than configured Continuous Compliance engines, this slowed down the masking process as the Hyperscale engine was unable to distribute jobs across multiple Continuous Compliance engines.

HM-3917

MSSQL: Fixed an issue where, for some users, column sequences were altered after recreating a clustered index.

HM-3934

MSSQL: Resolved an issue where the application expected all of the columns of related composite foreign keys to be present in the data set. This issue caused Hyperscale jobs to fail when composite primary or unique keys were partially masked. (Note, not all of the composite key's columns were masked).

HM-3939

MSSQL: When loading data to a target database, an error message was returned because the "null" string value was interpreted as a NULL value. This issue is now fixed.

ECO-569

Resolved an issue that prevented the file connector values.yaml unloadStorageType from using Hadoop and FS connectors together. You can now use the Hadoop and FS connectors without restarting the containers.

ECO-9105

Global Spark context is not allowing multiple jobs to maintain their session specific credentials.

HM-3685

MSSQL: The JDWP (Java debug wire protocol) debugging configuration caused the unload service to fail startup on hosts that did not support IPv6.

HM-3708

MSSQL: Unload having filter key of datetime type failed with Caused by: java.lang.ClassNotFoundException: org.apache.commons.text.StringSubstitutor

HM-3733

MSSQL: The masking date/timestamp column would only work when using the default date format.

HM-3740

MSSQL: Fixed an issue that caused MSSQL jobs to fail at load service if the target database table had a column with "Time" datatype.

HM-3756

MSSQL - Masking stuck in 'Running' for table with no unique/primary key.

HM-3775

Resolved an issue with Hyperscale Oracle Connector that caused jobs to fail in load service. This error: "Process terminated with exit code 2" occurred if data contained enclosure characters that got truncated during masking with Continuous Compliance versions 20.0.0 to 26.0.0.

HSC-740

Replace Oracle JDK-21 with openjdk17

HM-3554

[MSSQL]-FILLFACTOR attribute is not included while recreating index on post load

HM-3630

Unload failed with date format exception for "TIMESTAMP WITH TIME ZONE" and "TIMESTAMP WITH TIME ZONE" data types column

HM-3678

MSSQL- Unable to truncate table if having view references

HM-3530

Hyperscale Masking Hangs due to WARNING job status introduced in CC v22

HM-3549

Hyperscale Job failed during masking phase with "String index out of range" error with continuous compliance version between 20.0.0 to 23.0.0

HM-3557

HS Job execution stuck in RUNNING state if Masking Task has completely FAILED and number of rows masked > 0

HM-3560

MSSQL: Failed to load data in date type column if timestamp column is also present in table.

HSC-713

Parquet connector > failing to write data: For connector-type = FS, an intermittent issue exists where target files generated are not parquet files. For connector-type = AWS, source file name containing the substring "part-", result in empty target file

HSC-550

Parquet connector will be able to process decimal type of data as well

HSC-558

Delimited load service with “perform_join=false” fails for multiple source files in a single data set for S3 target location.

HM-3215

Masking service container json log grows too large and the huge log may cause controller service down because of local filesystem full

HSC-525

Delimited unload service for multiple source files for a single source key is being FAILED by the controller

HM-3100

Index creation on Local Partition table fails with ORA-14024

HSC-454

With pySpark we are removing the leading space for numbers

HM-1705

Improper error message in Hyperscale status response if CCE gets mount file system connection error

HM-2962

Incorrect gathering target table group list in case of the table refered from other tables.

HSC-176

Delimited connector: values with data type double and int64 will always be converted into a string

HSC-177

Delimited connector: Irrespective of user provided enclosure character, the output strings will be quoted using double quotes

HSC-279

Wrong counts in Unload: Using HS 14 delimited file connector, while masking two files in the dataset, job shows failed status, because the counts are all wrong

HSC-286

Load failed because the SQLite3 database locked up. Masking 15 files with the same data set. Unload has the wrong count. 

HSC-361

Hyperscale delimited connector should not add double quotes to output file

HM-2748

Unable to upload sync bundle of more than 20 MB

HM-2505

ServiceUnavailableException in Masking Service as numeric value out of range

HM-2554

Missing security context causes controller to fail with permission denied for JDBC connection to SQLite DB

HM-2609

Configuration added to load empty values of oracle CLOB/BLOB columns as null or empty string

HSC-182

400 Bad Request from POST http://unload-service:8080/api/unload

HM-2413

NTRS: Pre Load failed with errors: Error executing statement: Error disabling constraint

HM-2505

ServiceUnavailableException in Masking Service as numeric value out of range

HM-2554

Missing security context causes controller to fail with permission denied for JDBC connection to SQLite DB

HM-2568

Hyperscale API's working without auth key in k8s setup

HSC-178

Delimited load service will show the status as FAILED while it is waiting for all split files to be masked in order to perform join.

HSC-179

While executing masking against a 1.1 TB second time, the load job failed mid-execution.

HM-2399

MSSQL: The lower bound of partitioning column is larger than the upper bound.

HM-2443

Masking service is mapping the wrong file metadata when multiple data_info objects with varying delimiters are passed during data-sets creation.

HM-2268

Add support to configure FileMetadata related characters for structured (i.e. XML) data

HM-2344

sync-compliance-engine endpoint failing with "Invalid input" error

HM-2399

MSSQL: The lower bound of partitioning column is larger than the upper bound.

HM-1360

Job is created with default value of 'retain_execution_data' parameter if an invalid value for the same parameter is passed while creating the job object

HM-2041

Capture high-level processing timings for each process for each object

HM-2084

cancel script (cancel.sh ) does not read values from .env file

HM-2199

Masking service paginated GET call, skip the result of last page

HM-1845

java.sql.SQLException: Protocol violation While fetching metadata of the table

HM-1980

Post load status incorrectly reporting consolidated start and end timing.

HM-2017

The masking process is stuck in running when a newly registered engine is used in execution after the timeout time of the Engine

HM-1606

When tables have more than 100 columns on Oracle, the default API page at 100 causes an issue where we don't match the column name via the masking API

HM-1787

Oracle: Load service holds a lock on class level, causing parallel jobs stuck in the flow of preload

HM-1814

MSSQL: Load service holds the lock on class level, causing parallel jobs stuck in the flow of preload

HM-1617

Oracle - Hyperscale engine fails to mark partitions of the partitioned index as unusable in pre-load and fails to rebuild a partitioned index in post load leaving indexes in an unusable state

HM-1684

Running multiple Hyperscale jobs in parallel, using the same set of Masking Engines and the same value of env_name_prefix and app_name_prefix can cause job failure with the missing ruleset, any other masking objects, or execution error.

HM-1513

MSSQL - Slowness while performing load with large tables( more than 4M rows and 10 Columns)

HM-1521

MSSQL - Post load fails with 'transaction log is full due to 'ACTIVE_TRANSACTION' for large tables

HM-1608

SQLLDR doesn't load the data when the table has enabled triggers owned by a different user

HM-1645

Failed clean up of previous execution causes "\"File Format already exists with identifier: HYPERSCALE_16_9fd11fae45861fdb18fb658fb950ceab.fmt\" issue for next execution of same job

HM-1656

nginx configuration client max body size limits the size of the payload to post for the dataset

HM-1561

Oracle Load Failure: SQL loader control files doesn't contain character length when column size is less than 256 CHAR

HM-971

For HTTP protocol-type requests, trust store fields are accepted and displayed in response.

HM-1472

Oracle Unload service doesn't release the lock if fails to initialize the connection pool and the job stuck in a runnning state

HM-1520

Masking service: Starting execution throwing NPE after container restart

HM-1522

Update Oracle JDBC Driver to 21.3.0.0

HM-1155

Diagnosability: How do I tell which masking job on the masking engine relates to the failed message on the HS Jobs API status

HM-1168

The error text in Post Load failures is misleading/unclear

HM-1191

Optimization: We should execute Select Count (*) in parallel to the Oracle Unload process. It could take a significant amount of time to count data in large tables as well as 1000 objects.

HM-1201

Unable to import a ruleset with 5000 tables to HS 4.0, getting an error message.

HM-1210

While trying to process an HS job for 5000 table schema, ran out of Oracle cursor, and then the SQLite database locked up.

HM-1265

Oracle - Any index on a column having no constraint on it, is not getting dropped

HM-1334

Can't drop the index, because the index is not owned by the user we used to connect.

HM-1378

Oracle - Load service needs to include index owner name when attempting to modify/rebuild partition indexes owned by different db user

HM-177

Able to POST /hyperscale-masking/jobs with min job memory > max job memory

HM-530

POST/PUT request dataSet API error response received with empty/missing/invalid ‘source’/’target’ object/values can be improved

HM-754

POST/PUT connector jdbc_url, username, and password should be mandatory for Oracle load and unload service

HM-789

Error message upon not setting the ‘SSL’ field to False indicates ‘insecure_ssl’ property which no longer exists in the schema

HM-858

Status of sub-task coming wrong when overall execution failed

HM-932

Suppress the password message for the controller log

HM-1138

The description in the swagger doesn’t match the API call

HM-1140

The error needs more information to diagnose a connector issue.

HM-858

Status of sub-task coming wrong when overall execution failed

HM-873

Intermittently there is a mismatch in loaded_rows displayed in the load task vs the actual rows loaded in the target table

HM-915

Load: driver support plugin throws ORA-02297: cannot disable constraint - dependencies exist error for foreign key

HM-294

The updated file format is not POST’ed on the Continuous Compliance Engine if the file format name is the same