News Updates - DSG Updates
Global leading provider of data management software and unified application development solutions, dedicated to serving enterprise-level large data centers.
DSG assists Chongqing Bank in building real-time and scheduled data exchange and sharing platform.
发布时间：2024.03.18

To meet various data exchange needs within the bank, Chongqing Bank has introduced the DSG DataXone data exchange sharing platform to achieve real-time synchronization and application of data, enhancing the accuracy and efficiency of data processing.

Background and Issues of the Project

In the digital age, the applications for data analysis and mining in the financial sector have gradually transitioned from non-real-time modes to a hybrid mode of real-time and non-real-time. Systems such as the Banking Regulatory Commission, big data, and related transactions all have scenarios and demands for real-time data applications.

Established in 1996, Chongqing Bank is the earliest locally-owned state-owned joint-stock commercial bank in the western and upper reaches of the Yangtze River. It successfully listed on the Hong Kong Stock Exchange's main board on November 6, 2013, becoming the first mainland city commercial bank to do so. On February 5, 2021, it successfully listed on the main board of the Shanghai Stock Exchange, becoming the first city commercial bank in the western region to have an "A+H" listing.

In recent years, Chongqing Bank has gradually developed into a commercial bank with an optimal business structure, good asset quality, strong profitability, and great development potential.

With the rapid expansion of Chongqing Bank's business, the number of systems has continued to grow, and the data exchange between systems has become increasingly frequent and complex, placing higher demands on the specification and standardization of data exchange, the timeliness, stability, and controllability of data exchange and sharing.

The original system of Chongqing Bank used a traditional ETL+Flume model for quasi-real-time synchronization every 2 minutes, failing to achieve true real-time synchronization and application of data. This led to increased pressure on the source database, inability to synchronize large amounts of data, and limited applicability, with a lack of a real-time data synchronization tool.

The data acquisition and exchange for various downstream analysis systems of Chongqing Bank were all achieved through ETL, with three sets of ETL systems for data warehouse, big data, and ODS. Among them, the ETL for the data warehouse and ODS used the DataStage tool for data extraction and loading, but the architecture was outdated, with difficulties in adjusting data interfaces, excessively long synchronization times for large tables, and lack of support for big data platforms and MPP databases. The big data ETL was developed using Perl scripts, lacking stable ETL development tools and a complete interface operations and management interface.

For example, the entire ETL process for large tables in the credit system, such as contracts and loans, took approximately 7 hours (with extraction taking about 2 hours and loading taking about 5 hours), resulting in low efficiency and severely impacting the timeliness of data for downstream analysis systems such as the data warehouse and big data.

To address these issues, Chongqing Bank partnered with DSG to implement the DSG DataXone database comprehensive real-time collection and sharing exchange platform, meeting various data exchange needs within the bank, and improving the accuracy and efficiency of data processing.

Synchronization Flow and Deployment of the Project

The project is divided into two major blocks: real-time synchronization and scheduled batch synchronization. The DSG DataXone CDC module is responsible for implementing real-time synchronization from database to database, while the DSG DataXone ETL module is responsible for implementing scheduled batch synchronization from database to file, file to database, and file to file.



 

In ETL scheduled synchronization, data synchronization includes the mutual flow of data between Oracle, Gbase8a, DM, Hive, TDSQL, MySQL databases, as well as between files and files.

In CDC real-time synchronization, data flow includes Oracle-Kafka, TDSQL-Kafka, Oracle-File, MySQL-Kafka, DM-Kafka, etc., the synchronization scenes are complex, involving various types of databases.



Synchronize infrastructure on a regular basis.

 


 

ETL Timely Synchronization Basic Infrastructure Diagram for Data Exchange Platform

 

cheduled Synchronization Architecture Description
The scheduled synchronization of Chongqing Bank is completed by the third-party platform moia scheduling platform. Within the moia platform, scheduled tasks can be set for jobs.

  • Using the interface documentation provided by the moia platform, the moia platform interface is called to create a corresponding job information in the moia platform when building the queue, for the moia platform to call.

  • Providing external access to the DSG DataXone platform interface program, moia, when performing scheduled job tasks, calls the underlying script to execute the DSG DataXone external interface program, thus enabling the startup, operation, and shutdown of the DSG DataXone queue.

 

DSG DataXone  deployment architecture



 


DSG DataXone Deployment Architecture Diagram

 

Deployment Architecture Explanation

The DSG DataXone platform is deployed on 2 middle servers, with ETL collection and loading programs deployed on 4 middle servers (one master and three backups). The original database is deployed on a third-party TDSQL database, and random timed calls to the 4 middle servers are achieved through the third-party moia platform resources for high availability and load balancing of the process. Files are read and written through a NAS storage server.

The DSG DataXone CDC real-time synchronization is carried out in a high-availability mode, using the Manager program. When a server goes down, the Manager automatically detects it and initiates a failover to create queues and continue data synchronization on the backup server.

To achieve better results, the Chongqing Bank has also put forward corresponding synchronization requirements. After practical experience and evaluation, the performance indicators of DSG DataXone have met the collection requirements of Chongqing Bank, including high-speed data transfer, support for heterogeneous data from multiple sources, stable system operation, and powerful functional modules.

Synchronization Requirements

ETL scheduled synchronization

  • Exceed the DataStage platform ETL unloading speed of the original data warehouse (1G per minute)
  • Support for collection and loading of existing databases within the bank
  • Custom batch calls for views, complex ETL statements

CDC real-time synchronization

  • When not running batches, the analysis delay for CDC real-time synchronization does not exceed 3 seconds.

Conclusion

In general, DSG DataXone provides Chongqing Bank with an efficient, stable, and real-time data exchange and sharing solution, greatly improving the bank's data processing capabilities and business efficiency.

In the future, DSIJ will continue to focus on technological innovation, continuously optimize and upgrade products and services, and meet the needs of enterprise digitalization with flexible and intelligent solutions. We believe that through continuous efforts and innovation, DSIJ will become the best partner for many companies on the road to digital transformation!

About DSG DataXone.



                                 Visualization platform interface diagram

 

The DSG Database Comprehensive Real-time Collection and Sharing Exchange Platform (referred to as DataXone) is a data collection and sharing exchange platform independently developed by DSG with independent intellectual property rights. The underlying integrated independently developed modules include database synchronization, data transformation desensitization, file synchronization, switch navigation, and difference comparison, which can achieve efficient real-time synchronization of multiple heterogeneous databases. It supports data collection, sharing, exchange, replication, migration, and disaster recovery scenarios, and is compatible with more than a hundred heterogeneous source and target data sources, including domestic and foreign databases, big data platforms, cloud-based, and on-premises data exchange and sharing. It is the foundation platform for enterprise group data exchange.

DSG DataXone integrates high performance, high availability, multi-function, and multi-mode features. With its ultra-high real-time data synchronization efficiency, support for multiple heterogeneous databases, perfected data desensitization technology, and visual platform interface, it efficiently assists many customers in achieving data integration and sharing.

热门动态
警告弹窗