In the past two years, a certain bank's clearing center has fully implemented domestic technology, marking a significant breakthrough in the financial industry's technological innovation in China. During this process, DSG data synchronization technology has played a key role in providing strong support for the full-stack localization process of the bank's clearing center.

Background and Current Situation of the Project

Challenges in project implementation

1. Strong complexity of business systems: The project involves multiple business scenarios, including single-row insert and multi-row update scenarios, as well as scenarios where the source-side primary key does not match the target-side primary key, and vice versa.

2. High requirements for data synchronization timeliness: In the online banking business scenario, it involves operations such as end-of-day processing, daily closing, and daily reconciliation. Data must be synchronized to the aggregation database at a fixed time every day to ensure the accuracy and timeliness of business reconciliation.

3. Customized development requirements: For TDSQL secondary partition tables, sharded tables, and broadcast tables, there are issues with updating data when only one primary key is provided. Customized development is needed to implement adaptive handling methods such as alerts, updating to insertion, and ignoring issues. Also, the source database needs to support automatic adaptation to cn and dn failures.

4. Diverse system characteristics: The online banking system has a two-site three-center, active-active architecture, and data sources need to be distinguished using specified identifiers. The foreign exchange system has a one-site to one-site architecture, with free-format data tables requiring special handling. Each system has diverse characteristics and flexible requirements.

5. High availability requirements: Disaster recovery switchover for 39 synchronization links needs to be achieved within 10 minutes.

The bank's online banking system adopts a two-site three-center architecture, using the DSG DataXone exchange integrated management platform, with GaussDB as the source database, and Oracle and TDSQL as the target databases. Data is synchronized in real-time from the two centers in Beijing and Shanghai to the three locations of Beijing, Shanghai, and Wuxi, ensuring data consistency and timeliness.

The foreign currency system of the bank adopts a two-site two-center architecture. It utilizes the DSG DataXone exchange integration management platform, with GaussDB as the source database and Oracle as the target database, to ensure real-time synchronization of data between the two centers in Beijing and Shanghai, guaranteeing data consistency and timeliness.

Project Collection Requirements and Product Description

1. DataXone exchange integration management platform permission management;
2. DataXone exchange integration management platform sorting function;
3. DataXone exchange management platform alarm classification;
4. DataXone exchange management platform home page automatic refresh;
5. DataXone exchange management platform monitoring display, including delay, data transmission volume, historical transmission volume curve, and historical performance data of DSG server nodes;
6. Ability to ensure the continuity of data collection in response to source database failures, including cn and dn failures;
7. Ensuring the continuity of data in response to various failures of the target database, including synchronization proxy failures;
8. High availability of synchronization tasks, automatic takeover by backup machine when the synchronization host is down, automatic startup, and breakpoint resume to ensure data consistency.

DSG DataXone Exchange Integration Management Platform is an independently developed data collection and sharing exchange platform with independent intellectual property rights by DSG. It integrates independently developed modules such as database synchronization, data transformation and desensitization, file synchronization, switch navigation, and differential comparison, enabling efficient real-time synchronization of multiple heterogeneous databases. It supports data collection, sharing, exchange, replication, migration, disaster recovery switching, and other scenario applications, compatible with more than a hundred heterogeneous source and target data sources, including domestic and foreign databases, big data platforms, cloud-based, on-premises, and intra-cloud data exchange and sharing. It is the foundational platform for enterprise group data exchange.

Performance indicators and deployment instructions

The data is synchronized in real-time at a minimum of 150 transactions per second (tps) and a maximum of 500 tps.

This time, the total center for liquidation of the bank was fully localized, adopting a middle machine deployment mode. The deployment architecture of DSG DataXone is highly available, with two DataXone main and standby servers, database master-master replication, main and standby managers, and a zookeeper cluster. The middle machine in the synchronization link consists of two machines and creates and associates a highly available host group to meet the high availability switching requirements of the synchronization link.