To meet the long-term business development strategy of the bank and improve the quality and efficiency of financial services, Huzhou Bank has launched a new core system construction project to achieve a dual enhancement of business and technological capabilities. In the process, DSG SuperSync, as a data synchronization tool, has helped Huzhou Bank to synchronize TDSQL data in real time to the Kafka cluster, ensuring the security and consistency of the data.

Synchronous architecture

• The DSG SuperSync, pgmd, and pgmcli programs are deployed on 2 32-core * 256GB physical machines to act as data collection agents, collecting TDSQL data and sending it to Kafka via TCP/IP.

• There are two modes of data collection: real-time and daily cut-off. The programs stop analyzing logs around midnight by setting up daily cut-off tables to meet the daily cut-off requirements. The real-time queue has a delay of less than 1 second during non-batch processing stages.

• Downstream systems can check the status of the queue and start or stop the daily cut-off queue using the SDK interface with DSG pgmd.

• The data collection agent is deployed on an intermediary server using a non-invasive production server deployment mode, with 32 cores*256GB physical machine resources for deployment.

• The data collection agent collects TDSQL data through TCP/IP and loads the obtained data into Kafka through TCP/IP.

• The pgmd and pgmcli are deployed to monitor queue status, and downstream applications can use SDK interfaces to view and control queue status.

Replace OGG and customize the daily cut requirement function on TDSQL

• Original situation: The customer's old core Oracle is synchronized with the downstream big data platform using OGG. Initially, there was a function to ensure that the downstream retains a complete daily snapshot of data. The logic was implemented by inserting a marker row into batchmid after the core business system determines that all transactions for a day have ended. OGG reads this marker and immediately triggers a script to stop the synchronization, ensuring that the downstream receives a complete daily snapshot. After the downstream performs the necessary operations, the synchronization is manually restarted, and OGG continues the data synchronization from the breakpoint.

• Current requirement: In the new core, the customer's database has been switched to TDSQL, but they still want to retain this function, which needs to be implemented on TDSQL.

• The implementation process is as follows: The synchronization task will automatically stop and record the synchronization time point (scn) based on a record in the scheduling table (generated by the business). When a new record is generated, and after completing actions such as reconciling the daily report, the synchronization task needs to be manually triggered to restart. At this point, the data synchronization needs to be resumed according to the time point of the stop (scn) from the previous step.

• The automatic stop analysis function has been implemented for the first time to meet the daily cutting requirements of the clients. Settlement of the previous day's data is required the next day, and after the settlement is completed, the queue will be brought into real-time status through the SDK interface, and this process will be repeated daily.
• It is necessary to ensure the continuity of data synchronization during TDSQL primary and standby switching.
• The delay during normal trading hours must not exceed 2 seconds.

During normal business hours, the delay is between 0-2 seconds.

During evening batch processing, the duration is approximately 1.5 hours, with an average TPS of 20,000 transactions and a delay of around 3 minutes.