ENTERPRISE DATA WAREHOUSE FOR BUSINESS ANALYTICS


INSURANCE

Use Case: Data as Product using Data Mesh

What is a Data Mesh?

Data Mesh is a modern data architecture that decentralizes data ownership and creates a network of data products owned by the teams who know the data best. These are not silos, as they remain governed and interconnected. The core principle of Data Mesh is the realization of "Data as a Product."

How is "Data as a Product" achieved?

One effective way is through in-motion data classification. "Data as a Product" is created by different Lines of Business (LoBs) from their respective domains. For systems spanning multiple industries, domain-oriented teams create data products from the transactional database by capturing "Data in Motion."

Why creating “Data as Product”?

There are two main reasons. First, security is more easily achieved on a subset of data by applying domain-specific security rules. Second, it enables data monetization, as seen in applications that leverage social media data for various purposes.

This Data Mesh reference architecture leverages Oracle GoldenGate's ability to call out to REST APIs, so that based on domain query parameters, it creates a JSON store that can be consumed by applications, as shown here.


Another technique is to stream data to Kafka. For external subscribers, secure REST API exposes data to users as shown here.


The software licenses in the reference architecture are: Oracle Database, Oracle GoldenGate for Oracle, Oracle GoldenGate for Big Data and Oracle Stream Analytics.












TELECOMMUNICATONS

Use Case: In-Memory Business Analytics

User experience is critical when performing business analysis. For enterprises running analytical queries against terabytes of data, performance is often suboptimal. As the dataset grows, performance degrades linearly and can impact other applications running on the same system.

What is the solution?

A real-time, in-memory data warehouse, fed either from an active standby database, if available, or directly from the transactional database.

How do we achieve this in-memory data warehouse?

Oracle GoldenGate is the ideal tool for this. It provides real-time data capture, transformation, and delivery. In most cases, the mapping is handled automatically by default. For complex transformations, leveraging Oracle PL/SQL delivers the best performance.

As shown in the reference architecture, the transactional database is running on a 2-node Real Application Cluster (RAC). This is referred to by the source system. This Database is configured for heavy transitional applications. Typically, with a database block size of 8KB.

The Enterprise Database on a 2-node Real Application Cluster (RAC) with In-Memory option enabled. This is referred to by the target Operational Data Store Repository. This Database is configured for Data Warehouse query applications. Typically, with a database block size of 16KB or 32KB, the later has hardware-dependent.

An Oracle GoldenGate running in Integrated mode referred to by the Downstream system. Its independent system deployed to capture real-time data from the source system and create the trail files that are used to update the target system – the Operational Data Store Repository.

In the Operational Data Store Repository system, it empowers a Database feature called Materialized views, which stores the result of the queries. So, we begin by identifying the queries used by the business analytics applications and create the materialized view using these queries. The next step is to force the Database Least-Recently used algorithm to maintain the materialized views in SGA, so they remain in specific area in memory, as shown below.


The below shows the moment of truth. It's a comparassion between the two systems. With and without the real-time in memory Data Warehouse.


The queries time showns 2.6199999 seconds


The queries time showns 0 seconds

Download the reference architecture implementation and let me know your thoughts.






BANKING

Use Case: Credit Card Data encryption

Now days most bank issue credit cards online through multiple channels, it could be issued by the bank directly, by the customer online, or by partner online. On all cases, the credit card details must be encrypted when sent across the different networks.

After the card is issued it must be send to different systems, such as Enterprise Data Warehouse, Loyalty, data must be sent encrypted, leaving no single point of unencrypted data transition. As shown the reference architecture, the entire encryption is handled within GoldenGate.

Once the credit card transaction committed (card is issued) Oracle GoldenGate capture and encrypt data, leaving no gap of unencrypted data. The encrypted data is stored in a TOKEN and send to the next systems, this highly an optimized process where the single capture is sent to participated systems.

As shown in the reference architecture, one of the participated system is Kafka. GoldenGate for Big Data (Kafka connector) receive the data and the Replicat stream to Kafka topics in Kafka cluster. This PoV demonstrated Kafka, but it can be any supported Big Data connector such as Snowflake, Databricks, etc.

Oracle Database apply another layer of security through Access Control List (ACL). Only database schemas granted access are able to perform the encryption/capture operations as shown below.


Phase one of the project was rolled out for two participated systems, but can scale by specifying the destination and creating the Replicat process.






Return to Homepage