This extends on the core Claim Check design pattern, by incorporating a streaming message platform like Apache Kafka on Heroku or AWS Kinesis. Incorporating a streaming message platform like Apache Kafka on Heroku is important to consider for three reasons:
Organization-to-organization replication with transformations of the records.
Here a monolithic transformation app could read a message in from Kafka, transform the data, and then write back out to Postgres. Transformation apps from multiple external services could subscribe to this message stream and handle data in whatever ways may be needed.
Multiple Kafka topics can be used to enable multistep transformations.
In this example, the logic for handling messages is segmented into four components. The first routes the message, determining what transformation needs to be applied and what the final destination of this message should be. From there, multiple message transformers can be implemented to apply different layers of changes from simple field mapping to more complex translations like processing parent/child relationships. The last agent then writes out the final message once modified.
In this pattern, multiple Kafka topics provide delivery for each segment of the pipeline. In this way, not only does the code remain more modular, but it also becomes possible to scale each component independently. This allows for message prioritization, but also helps optimize the cost of processing against the true demand.
Steve Stearns is a Regional Success Architect Director at Salesforce on the Scale and Availability team. Scale and Availability architects apply the learnings from our large customer engagements to help influence product scale evolution and innovation.