- Release notes
- Before you begin
- Managing access
- Getting started
- Integrations
- Working with process apps
- Working with dashboards and charts
- Working with process graphs
- Working with Discover process models and Import BPMN models
- Showing or hiding the menu
- Context information
- Export
- Filters
- Sending automation ideas to UiPath® Automation Hub
- Tags
- Due dates
- Compare
- Conformance checking
- Root cause analysis
- Simulating automation potential
- Starting a Task Mining project from Process Mining
- Triggering an automation from a process app
- Viewing Process data
- Creating apps
- Loading data
- Transforming data
- Customizing process apps
- Publishing process apps
- App templates
- Notifications
- Additional resources
Process Mining
Incremental extraction
For most source systems, CData Sync has preconfigured how incremental extractions should be performed. If that's not the case, the settings have to be edited manually to perform incremental extractions. For more on how to do this, refer to the official CData Sync documentation.
For running an incremental extraction using CData Sync, you need to add a timestamp column in the extraction job for every table you want to incrementally load. The timestamp column will keep track of when the data was last extracted, enabling CData Sync to only extract new or updated data for the next iteration.
Follow these steps to configure incremental data extraction using CData Sync.
-
Edit the CData Sync extraction job created in Create a job.
-
Go to Task and edit the table you want to configure for incremental extraction.
-
Go to Columns and select Edit Mapping.
-
Select + (Add Column).
-
Enter the SQL command to convert the timestamp to the correct format. For example for Oracle source databases enter
to_char(CURRENT_TIMESTAMP, 'YYYY-MM-DD HH24:MI:SS'). Select Next. -
Enter
EXTRACTEDin the Column Name field. -
Select
TIMESTAMPas the Data Type and select Add Column. The following illustration shows an example. -
Go to Overview and edit the Source Information.
-
Select the applicable Incremental Column from the list.
Note:This should be a column that contains timestamp values for each change or on an integer column that holds the row count for each update.
-
Select Save.
-
Go to the Manage input data screen, and add the timestamp column to the Timestamp field to configure the incremental load for a table. Check out Managing input data for more information.
For incremental extraction, it is necessary to define a starting point (or 'incremental start date') from which new or updated data will be extracted for each table.
Follow these steps to define the incremental start date for a table.
-
Go back one level up to Overview in the Edit Job page.
-
Locate the Incremental Replication section and select Configure.
-
Select a desired Start Date.
-
Enter the desired Replication Interval, or accept the default of 180 days.
Note:A small replaction interval, may impoact performance.
-
Select Save.
Follow these steps to make sure incremental data extractions run properly.
- Start the CData Sync Admin console and login as and administrator using the credentials defined during installation of CData Sync.
- Go to Jobs and select the extraction job to display the job settings.
-
Go to the Advanced tab, and edit the Replication Option to deselect the Drop Table option. This enables incremental extraction.
The first time the CData Sync extraction job runs, a full extraction is performed and all source data is extracted. Every next time the CData Sync extraction job runs, only the data that has changed in the source is extracted and the delta is added to the already extracted data.
For more information on incremental data extraction with CData Sync, refer to the official CData Sync documentation.
