Near Real-time Data Transfer using AWS Database Migration Service (DMS)

This blog explains about real time data transfer from source system to the downstream application using different AWS services, the main intention here is not to burden source system. Direct database query can negatively impact source system performance. Following solutions tries solve this problem by using various AWS services

Note : there are different ways to implement above mentioned solution, it all depends on multiple different factors.

Proposed Solution Architecture

Solution Architecture

Services Used as Part of Solution Architecture

1. Database Migration Service (DMS)

   • Enable Change Data Capture (CDC) on the source database.

   • DMS can be configured for continuous data replication with near real-time data transfer.

   • There are multiple third-party products available, such as Qlik Replicate, which can be used as alternatives to DMS.

2. Kinesis Data Stream

   • A fully managed, scalable, real-time data stream application that allows you to process data.

   • Depending on requirements, services such as Managed Streaming for Kafka (MSK) can also be used.

3. AWS Lambda

   • AWS Lambda can be used to read data from Kinesis Data Stream and transform the data (if required).

4. Destination Database

• The destination database can be any compatible database such as DynamoDB, RDS, etc.

Reference

• Send Relational Database CDC Information to Kinesis Data Streams | Amazon Web Services

Conclusion

This solution provides a simple approach for transferring near real-time data from on-premise to the AWS cloud. However, the actual implementation depends on multiple factors such as the volume of data, data transformation requirements, etc.