Orchestration of SAS Data Integration Processes on AWS

This paper talks about the orchestration of SAS® data integration processes based on the arrival of the SAS data integration input files in Amazon Web Services (AWS) S3. Our client runs a daily process where they generate credit statements for their customers. Each customer receives their statement once in a month. Every day, around 200K customers are processed, eventually reaching out to their entire customer base of roughly 6 million in a month. The process starts in their on-premises datacenter, followed by the APR calculations in SAS data integration in AWS, and finally culminates with the generation of the statements in the on-premises datacenter. The entire architecture is built using microservices; small independent and highly decoupled components such as AWS Lambda, Simple Notification Service (SNS), Amazon Simple Storage Service (Amazon S3), and Amazon Elastic Compute Cloud (EC2). This makes it easier to troubleshoot any issues in the data pipeline. The choice of using a Lambda function for the orchestration adds a certain complexity to the process. However, it also provides the most stability, security, flexibility, and reliability for an enterprise architecture. There are simpler alternatives like S3Fs and CloudWatch SSM, but they do not fit well for an enterprise architecture.


Share content on email