Health is a topic of eternal concern for people, and the world of data is like an endless stream of movies.

As a forward-thinking health insurance company, we have faced challenges and opportunities on the road to technological advancement, achieving major leaps forward time and time again!

I. Real-time synchronization of the core commercial insurance system

With the trend of domestication of information technology and the promotion of innovative policies, a certain health insurance company has launched a new core system for commercial insurance, using distributed Huawei GaussDB for the underlying database.

In order to pursue higher security and stability for the system, the company proposed a strategic coexistence of the new and old core systems.

The most critical technological challenge during the implementation of the strategy was how to efficiently synchronize data between the new and old systems.

As a professional data synchronization solution provider, DesiJie took on the important task of building a data synchronization bridge between the new and old systems, utilizing its deep technical expertise and experience.

The "first gap" is essentially how to achieve real-time synchronization of new data in the core system, seamlessly synchronizing this data from Huawei GaussDB to the DB2 database in the old core system. Although it sounds simple, in reality, this process involves complex technical details and strict performance requirements.

The primary challenge is that the incremental data from the new core system needs to be accurately and rapidly synchronized to the old core system for downstream queries by the business.

Customers have extremely high requirements for the accuracy and latency of the data, aiming for a latency of 0-3 seconds.

DesiJie's data real-time synchronization tool can achieve end-to-end real-time synchronization from Huawei GaussDB to DB2, but before the data reaches DB2, customers need to process it according to business requirements, so Kafka was introduced. Kafka serves as a middleware for temporarily storing data, making it easier for customers to process and eventually store the data in the database.

The synchronization task from end-to-end Huawei GaussDB to DB2 has now been changed to Huawei GaussDB to Kafka synchronization. DSI still relies on its professional technical expertise to overcome the difficulties of synchronizing data from Huawei GaussDB to Kafka, which has a high-risk and special flow direction:

1. Analyzing GaussDB database transactions using a slot mode. Once the logs in the slot are analyzed, they will immediately disappear, making it impossible to specify and analyze forward logs, leading to synchronization interruption.

2. Kafka also cannot perform full repairs on individual tables, so once data loss occurs due to problems, it cannot be remedied.

DSI not only needs to overcome these technical barriers but also needs to adapt to the Kafka message format of Huawei DRS, to ensure that data is written into the Kafka cluster according to the customized message format required by the customer, for easy subsequent processing and ultimately landing in the old core DB2 database.

So, how did DSI do it?

In short, through continuous development and optimization.

DSG uses its self-developed alp mode to pull slot log and convert it into a unique xdt format file, saving it in the cache directory of the intermediate machine. This ensures that even after the data in the slot is consumed and automatically pushed forward, the backup logs can still be found in the DSG cache directory and can be reanalyzed at a specified time within the remaining log range, maximizing data security.

DSI adopts an intermediate machine deployment mode, using DSG DataXone to develop and adapt data message formats, achieve real-time synchronization of incremental data to Kafka, and deploy monitoring programs to track the synchronization status in real-time.

The project was successfully completed, meeting all customer requirements for real-time synchronization, and maintaining a stable delay time of 0-3 seconds.

Furthermore, since its launch in November 2021, the system has been running stably for over a year. After completing the business requirements, it was successfully taken offline and received high recognition from customers!

II. Migration of Core Commercial Insurance System + ecif System to the Cloud

Following the successful data synchronization project, in order to facilitate the unified management of servers and databases, the customer decided to migrate all existing cloud-based Huawei GaussDB database data to the cloud-based Huawei GaussDB database, gradually replacing the on-premises environment.

This migration needs to be completed within a short downtime, including the migration and data verification of all data, and after the migration, the business will be switched to the new cloud database.

DigiJet once again takes on the heavy task. This cloud migration involves 12 databases and a data volume of 500G.

DigiJet used DSG DataXone to deploy a non-intrusive intermediate machine mode to statically migrate the data from the Huawei GaussDB database under the cloud to the Huawei GaussDB database in the cloud, and after the migration, a comprehensive data verification was carried out. After all data verification was correct, the business system was connected to the new cloud database for use.

The implementation results were satisfactory. Within a 3-hour downtime window, the cloud migration and data comparison of 500G data were completed, with an overall data migration speed of 62.90MB/s.

After the completion of the subsequent migration, the new commercial insurance core system and ecif system and other business systems were successfully connected to the cloud database and operated normally, without any data differences. With this, the data cloud migration work has come to a successful conclusion.

With that said, this sharing comes to an end, but the endless new chapter of data streaming for this health insurance continues to be written!