Data Source Support

Oracle Connector

Oracle Database (commonly referred to as Oracle RDBMS or simply as Oracle) is a multi-model database management system produced and marketed by Oracle Corporation. The following table lists the versions that have been tested in the lab setup:

Supported Data Types

The following are the different data types that are tested in our lab setup:

• VARCHAR

• VARCHAR2

• NUMBER

• FLOAT

• DATE

• TIMESTAMP(default)

• CLOB

• BLOB(with text)

• XMLTYPE

Property values

For default values, see Configuration settings.

MS SQL Connector

Supported versions

Microsoft SQL Server 2019

Supported data types

The following are the different data types that are tested in our lab setup:

• VARCHAR

• CHAR

• DATETIME

• INT

• TEXT

• XML (only unload/load))

• VARBINARY (only unload/load)

• SMALLINT

• SMALLMONEY

• MONEY

• BIGINT

• NVARCHAR

• TINYINT

• NUMERIC(X,Y)

• DECIMAL(X,Y)

• FLOAT

• NCHAR

• BIT

• NTEXT

• MONEY

Property Values

For default values, see Configuration settings .

Known Limitations

• If the applied algorithm's produced mask data exceeds the corresponding target table columns datatype's max value range, then job execution will fail in load service.

• Schemas, tables, and column names having special characters are not supported.

• Masking of columns with VARBINARY datatype is not supported.

• Hyperscale Compliance can mask up to a maximum 1000 tables in a single job.

Delimited Files Connector

The connector can be used to mask large delimited files. The delimited unload service splits the large files into smaller chunks and passes them onto the masking service. After the masking is completed, the files are sent to the load service which joins back the split files (the end user also has a choice to disable the join operation).  

Pre-requisites

• The source and target (NFS) locations have to be mounted onto the docker containers of unload and load service. Please note that the locations on the containers are what needs to be used when creating the connector-info’s using the controller. 

  # As an example
  unload-service:
       image: delphix-delimited-unload-service-app:<HYPERSCALE VERSION> 
       ...
       volumes:
            ...
       - /path/to/nfs/mounted/source1/files:/mnt/source1
       - /path/to/nfs/mounted/source2/files:/mnt/source2
  ...
  load-service:
       image: delphix-delimited-load-service-app:<HYPERSCALE VERSION> 
       ...
       volumes:
            ...
       - /path/to/nfs/mounted/target1/files:/mnt/target1
       - /path/to/nfs/mounted/target2/files:/mnt/target2
  

Property values

For default values, see Configuration settings.

Supported data types

The following are the supported data types for delimited files hyperscale connector:

• String/Text

• Double

  • Columns with double data types will be converted to strings.

  • e.g., 36377974237282886994505 will be converted to “36377974237282886994505".

• Int64

  • Columns with int64 data type will be converted to strings.

  • e.g., 00009435304391722556805 will be converted to “00009435304391722556805".

• Timestamp

Known Limitations

1. Supports only Single-character ASCII delimiters

2. The end-of-record character can only be \n, \r, or \r\n.

Output files will exclusively enclose all string types with double quotes (`”`).

MongoDB Connector

The connector can be used to mask large MongoDB files. The Mongo unload service splits the large collections into smaller chunks and passes them onto the masking service. After the masking is completed, the files are sent to the Mongo load service, which imports the masked files into the target collection.

Supported Versions

Pre-requisites

1. MongoDB user should have the following privileges:

   use admin 
   db.createUser({user:"backupadmin", pwd:"xxxxxx", roles:[{role:"backup", db: 
   "admin"}]})
   

2. Mongo Unload and Mongo Load service image names are to be used under unload-service and load-service. The NFS location has to be mounted onto the Docker containers for unload and load services. Example for mounting /mnt/hyperscale.

   # As an example docker-compose.yaml
   unload-service:
        image: delphix-mongo-unload-service-app:${VERSION}
   volumes:
      # Uncomment below lines to mount respective paths.
      - /mnt/hyperscale:/etc/hyperscale
   
   load-service:
        image: delphix-mongo-load-service-app:${VERSION}
   volumes:
      # Uncomment below lines to mount respective paths.
      - /mnt/hyperscale:/etc/hyperscale
   

2. Uncomment the below lines from docker-compose.yaml file under controller > environment:

# uncomment below for MongoDB connector
#- SOURCE_KEY_FIELD_NAMES=database_name,collection_name    
#- VALIDATE_UNLOAD_ROW_COUNT_FOR_STATUS=${VALIDATE_UNLOAD_ROW_COUNT_FOR_STATUS:-false}
#- VALIDATE_MASKED_ROW_COUNT_FOR_STATUS=${VALIDATE_MASKED_ROW_COUNT_FOR_STATUS:-false}
#- VALIDATE_LOAD_ROW_COUNT_FOR_STATUS=${VALIDATE_LOAD_ROW_COUNT_FOR_STATUS:-false}
#- DISPLAY_BYTES_INFO_IN_STATUS=${DISPLAY_BYTES_INFO_IN_STATUS:-true}
#- DISPLAY_ROW_COUNT_IN_STATUS=${DISPLAY_ROW_COUNT_IN_STATUS:-false}

3. Set the value of LOAD_SERVICE_REQUIRE_POST_LOAD=false inside the “.env” file.

   # Set LOAD_SERVICE_REQUIRE_POST_LOAD=false for MongoDB Connector
   LOAD_SERVICE_REQUIRE_POST_LOAD=false
   

4. Uncomment the below lines from “.env” file.

   # Uncomment below for MongoDB Connector
   #VALIDATE_UNLOAD_ROW_COUNT_FOR_STATUS=false
   #VALIDATE_MASKED_ROW_COUNT_FOR_STATUS=false
   #VALIDATE_LOAD_ROW_COUNT_FOR_STATUS=false
   #DISPLAY_BYTES_INFO_IN_STATUS=true
   #DISPLAY_ROW_COUNT_IN_STATUS=false
   

Property values

Mandatory changes are required for the MongoDB Connector in the docker-compose.yaml and .env files:

For default values, see Configuration settings.

Known Limitation:

• Sharded MongoDB Atlas is not supported.

• In-Place Masking is not supported.