In 2023, East Asia Bank (China) is collaborating with DeCiJe to build a real-time big data processing platform to meet the demand for real-time data analysis and drive digital transformation. Currently, all tasks for the construction project are progressing smoothly.

What is a real-time data processing platform?

A real-time data processing platform is a system that is capable of collecting and processing on-site data in real time as it occurs. This process includes the entire process of data generation -> real-time collection -> real-time caching storage -> real-time computation -> real-time landing -> real-time display -> real-time analysis.

In order to ensure the integrity and accuracy of the data, a real-time data processing platform needs to have high reliability, meaning the system cannot have issues and data cannot be lost. At the same time, in order to improve processing efficiency, the platform needs to have high performance and low latency characteristics.

一、Real-time synchronization feature

Support full + incremental data import from Oracle, MySQL, SQL Server, and other databases to the Kafka cluster, while ensuring the real-time and complete data.

Specific requirements for data synchronization are as follows:

• Data synchronization: full synchronization, real-time incremental synchronization, heterogeneous synchronization;

• Data disaster recovery: active-active disaster recovery, disaster recovery incremental switch;

• Data transformation: data ETL, data desensitization, data filtering, data processing;

• Data comparison: full comparison, incremental difference comparison, scheduled comparison, heterogeneous comparison;

• Content collection: support structured, semi-structured data collection;

• Collection requirements: second-level delay, encrypted collection, meet real-time query requirements and real-time data sharing.

二、Secondary development functions

It can provide the ability for secondary development, with flexible configuration and support for real-time application scenarios towards the future.

三、Scalability requirements

Metadata collection and data security requirements to meet data ETL transformation and ensure data security.

一、Platform Construction

The installation and deployment of real-time data synchronization software and the Flink real-time stream processing platform form the unified real-time data collection and real-time processing platform capability for East Asia Bank.

二、Real-time data collection implementation

三、Development of application scenarios

Based on the requirements of private, public, and retail business scenarios, we will define indicators, design models, develop real-time streaming tasks, schedule tasks, and perform other data development work as requested by users.

四、Maintenance and optimization

System maintenance; daily maintenance, software upgrades; customized maintenance; system optimization.

1. Data Synchronization Requirement

Meet the full and incremental real-time synchronization requirements in heterogeneous scenarios, with an overall latency control in the seconds. At the same time, in the real-time collection process, it meets the functions of data cleansing, transformation, and desensitization.

2. Business Scenario Requirement

• Retail Banking Business Scenario: Real-time notification of deposit account changes, real-time notification of account balance transfers, real-time notification of wealth management changes.

• Corporate Banking Business Scenario: Real-time query of deposit balances for various institutions, real-time query of loan balances for various institutions, real-time deposit query for specific customers, anti-money laundering business functions, real-time change notification, real-time data forecasting reports.

• Retail Business Scenario: Real-time change notification (inflow and outflow), real-time change notification (wealth management product purchase, redemption).

- DSG adopts a streaming and batch integrated architecture, fully utilizing the capabilities of streaming and batch integration to redesign the offline indicator calculation logic, making it suitable for real-time stream processing.

• Through the application of state management and fault tolerance mechanisms, the efficient and reliable operation of the system is ensured under large-scale data and exceptional circumstances.

• Adjustments to the real-time output and feedback module ensure the immediate delivery of results and business feedback.

• Configuring comprehensive monitoring and debugging mechanisms provides assurance for the stable operation of the system.

The Dsigeo Flow Integration Platform is an end-to-end data service pipeline based on the Hadoop ecosystem, aiming to create a one-stop, standardized, visualized, and transparent intelligent big data full lifecycle development management platform with autonomous development and full-stack data research and development capabilities for enterprises internally.

D-Sight Stream-Batch Integrated Platform Functional Architecture Diagram

The D-Sight Stream-Batch Integrated Platform supports both real-time streaming data and batch data processing, as well as real-time and historical data processing. It provides a consistent API and development model, supports multiple data sources and data formats, and includes visualization tools and monitoring capabilities. It also features functions such as data transformation, aggregation, filtering, and joining.

It can meet various complex application scenarios, such as data transmission, data processing, data governance, and data sharing, to help users analyze data and develop applications more efficiently, and to improve data processing efficiency, truly realizing data business and business data.