Cleaning up execution data
As part of the Hyperscale execution run some data files and objects are created which should be cleaned on the execution completion. These files and objects are:
The system will create data files (unload service) and masked files (masking service) on the file server. As the data size can be large (2 times of source data) and include sensitive information, therefore, it is important to clean up this data.
The system will create multiple data objects like connectors, rulesets, file formats, jobs, etc on the respective Continuous Compliance Engines. Objects created by Hyperscale should be cleaned once Hyperscale execution is complete.
Additionally unload service, masking service, and load service will also store transient internal data for the execution while running it. This data is not required once execution is completed.
Following are the three ways this data will be/can be cleaned.
Using retain_execution_data
While setting up a Hyperscale Job (POST /jobs
), you can set the value for retain_execution_dataproperty to the intimate system when it should clean up data automatically based on the table below.
EXECUTION_STATUS | RETAIN_EXECUTION_DATA | CLEAN UP AUTOMATICALLY? |
---|---|---|
NA(SUCCESS/FAILED/CANCELED) | NO | YES |
SUCCESS | ON_ERROR | YES |
FAILED | ON_ERROR | NO |
CANCELED | ON_ERROR | NO |
NA(SUCCESS/FAILED/CANCELED) | ALWAYS | NO |
2. Manual clean up
Hyperscale exposes a delete API (DELETE /executions/{id}
) to manually clean up data for execution if it's not already cleaned.
3. Start a new execution
While starting a new execution, Hyperscale will first validate if the previous execution data is cleaned. If it’s not cleaned, then Hyperscale will trigger cleanup before starting new execution.
Limitations
As part of the clean-up process, applications created on the Continous Compliance Engines by Hyperscale job execution are not deleted. You must use the Continuous Compliance Engine APIs to delete the application.