News Updates - DSG Updates
Global leading provider of data management software and unified application development solutions, dedicated to serving enterprise-level large data centers.
Small steps, phased approach, new heights. A certain operator's payment company successfully completed the migration of multiple Oracle systems to OB.
发布时间：2023.07.04




On June 20th, DSG successfully completed the first phase of the data migration project for a payment company's core system with the assistance of a certain operator.

In order to meet the customer's requirement for a 10-minute window for the payment system cutover time, and to ensure a smooth rollback in case of failure after the cutover, DSG used a real-time incremental differential comparison and reverse flow strategy. Through full + real-time incremental synchronization, full + real-time incremental differential comparison, and bidirectional synchronization technology, DSG successfully assisted the operator's payment company in migrating and cutting over the business core system from Oracle to the OceanBase database.

01


Replace Oracle with OceanBase at IOE.



The head office of a telecom operator group responded to the national call and issued a document requiring each company to "go IOE" step by step in an orderly manner to enhance data security. As a large payment platform under the head office of the group, the payment company of this operator initiated a practice action to "go IOE".

The first-phase project mainly focused on its payment link, migrating and cutting over the core system data to achieve the replacement and business takeover of the old system with the new one.

The original database of the core system was an Oracle database, with high maintenance costs and unable to support a comprehensive upgrade to the new generation of technology architecture, as well as to meet the "capacity" elastic scalability and "disaster recovery" multi-active requirements. OceanBase, as a domestically developed distributed database fully controlled through independent research and development, became the protagonist in replacing the Oracle database. Based on common servers, it can achieve enterprise-level, financial-level high availability and data consistency, bringing better performance, security, and sustainable development opportunities for customer systems.

Advantages of OceanBase

Capacity: From traditional centralized, to single IDC distributed, and finally leapfrogged to multi-city, multi-IDC parallel computing;
Disaster recovery: Cross-region multi-active, achieving the highest level 6 disaster recovery level of the China Banking Regulatory Commission: remote (>1000 kilometers) RPO=0, RTO<10 minutes;
Security: Provides data redundancy backup and automatic fault recovery functions, supports data encryption and access control security features, and strengthens system data security protection.


02


Migration + comparison, the DSI escort system is cut in online.



During the project execution, the complete and efficient migration of a large amount of existing and incremental data from the original Oracle database to the new OceanBase database, while ensuring the normal operation of the cut-over business, is a major focus of the project. DSG has become the guardian of this data migration.

 

The project revolves around the payment chain, with the overall characteristics of large data migration, short cut-off time, and high comparison requirements.


项目特点

Large amount of data: mostly tens of terabytes of data and frequent system cut-offs (2 sets per month, 4 sets per month), need to migrate Oracle data to ob_oracle in 2-3 days, and complete full verification in about 2-3 days;

Short cut-off time: the cut-off window is only 10 minutes, full comparison ends + incremental data behind needs real-time incremental comparison (to ensure data consistency). The cut-off operation cannot exceed 3 minutes (incremental data verification completed - stop forward synchronization - start reverse synchronization);

High comparison requirement: there are large tables of 4-6 terabytes, the comparison tool needs to be optimized, full comparison is automatically split by id or partition, incremental joint primary key comparison, specific primary key, to ensure data consistency.

 

This migration mainly includes 6 sets of systems, namely withholding, acquiring, payment, transaction, CIF, and accounting systems. Through DSG's self-developed DSG SuperSync large-scale database high-performance replication tool and DSG XCMP data comparison platform, the core systems are migrated and cut-over from Oracle to OceanBase.

During the data migration and synchronization process, the method of full + differential + real-time incremental comparison is used to ensure the accuracy and integrity of the data. At the same time, in order to ensure the security and reliability of the data, backflow link protection measures are synchronously implemented. This means that in the event of an abnormal situation, data can be rolled back or restored in a timely manner to ensure the stability of the system and the continuity of the business.




Migration architecture diagram


The project started on September 26th of last year with entry testing, and completed on-site testing verification of the withholding system on October 20th. Subsequently, the first system was put online on November 17, 2022. A total of 9 systems were successively put online, and finally, the online work of all payment link systems was completed on June 20, 2023.





Milestone in project phase



03


Great strides, advantages and breakthroughs concurrently.





Full synchronization performance: 560G/H; Comparison performance: 860G/H.


 Advantages of the product

  1. High efficiency in data migration and verification: able to complete migration and data verification of large amounts of data within a short period of time. For example, approximately 10-20T of data can be migrated within 2-3 days, and the first round of full data verification can be completed within 2-3 days.
  2. Real-time data verification: capable of real-time data verification, allowing for full data verification and real-time incremental comparison shortly after completing the full migration, ensuring data consistency and the smooth launch of the system.
  3. High efficiency in incremental comparison: for systems that are already online, incremental comparison can be completed within 1 minute during the cutover window. In addition, reverse synchronization can be done at a specified time to reduce downtime and minimize the impact on customers.
  4. Flexible migration tool: for customers using non-stop cutover, source-side read-only tables, and gray verification mode, reverse synchronization can be done during the cutover without loading data.
  5. Rapid detection and repair of lost data: able to quickly detect and repair data loss caused by parameter configuration and on-site errors using DSG XCMP comparison tools.
  6. Fast addition of incremental data in synchronization tools: in busy business environments with large amounts of data, adding one day's worth of incremental data only takes an hour.


This project is a full-scale comparison of TB-level data for DSG, and it is also DSG's first project to use incremental comparison, which is a challenge for DSG's comparison products!


Product optimization



Full comparison


Automatic partition splitting comparison: For the situation where the partition table names of the on-site source Oracle and ob_oracle are inconsistent, modify the way to take partitions from both databases to achieve automatic partition splitting comparison;
High availability: If there is an error or query timeout during the data retrieval process, it will reconnect and retrieve the data again.



Incremental Comparison


Modify the logic of extracting incremental data from the source and target ends: In the previous version, the source and target ends separately extracted their monitored change data, and synchronization inevitably caused delays, leading to the risk of data omission in special cases. By optimizing the data acquisition mode, the current version can ensure that data will not be missed;
Add comparison types to improve comparison efficiency: The SQL query for the source oracle dbink has been optimized, and support for comparing multiple types of primary keys has been added, including numbers, character strings, and timestamps. For on-site development, the time for incremental comparison of several terabytes of partitioned tables from the source has been optimized from over 2000 seconds to the second level;
Incremental comparison when the primary keys on the source and target ends are different: For the ob_oracle database partitioned table, the primary key must include the partition key, which means that the primary key on the source end is a single primary key, while on the target end it is a composite primary key consisting of the primary key and the partition key. Through on-site development, incremental comparison is now possible even when the primary keys on the source and target ends are different;
Add support for monitoring ddl operations on the source end: The original version only supported dml monitoring, but on-site, the source oracle partitioned table automatically creates a new partition every day, and the monitoring does not support ddl operations, leading to monitoring errors. After modification, ddl operations are now supported.


04


A brief summary: the spark between engineering and research and development.



This project is a complex and phased migration across multiple systems, with high difficulty and large data volume, and a very short migration window. The key to the success of this migration lies in engineering capabilities and research and development capabilities! In the face of on-site issues, DSG has continuously promoted the project through engineering and adapted development, ensuring the smooth completion of the project. Throughout the project, DSG and various teams have made tremendous efforts to overcome various challenges.



Project Cutover Site Memorial


For the client, the successful completion of this project has improved the stability and reliability of the system, and also provided greater development space and potential for the future of the enterprise. For DSG itself, participation in this project has not only enhanced its technical strength, but also established a good industry reputation, marking a significant milestone in the practice of independent and controllable innovation in the domestic information and communication technology industry, leaving more possibilities for future development!


热门动态
警告弹窗