Pattern Overview

This extends on the core Claim Check design pattern, by incorporating a streaming message platform like Apache Kafka on Heroku or AWS Kinesis. Incorporating a streaming message platform like Apache Kafka on Heroku is important to consider for three reasons:

  1. There’s a potential for spikes in volume where the data flowing into the integration can temporarily exceed the capacity of the synchronization and transformation processing.
  2. There may be multiple services that need to respond to the data changes.
  3. Data transformation can be complex and may need to be segmented to better support scaling and maintainability of the pipeline.

Organization to Organization Replication

Organization to organization replication with transformation of records

Organization-to-organization replication with transformations of the records.

Here a monolithic transformation app could read a message in from Kafka, transform the data, and then write back out to Postgres. Transformation apps from multiple external services could subscribe to this message stream and handle data in whatever ways may be needed.

Multi-Step Transformation with Kafka Topics

Multiple Kafka topics enabling multistep transformation

Multiple Kafka topics can be used to enable multistep transformations.

In this example, the logic for handling messages is segmented into four components. The first routes the message, determining what transformation needs to be applied and what the final destination of this message should be. From there, multiple message transformers can be implemented to apply different layers of changes from simple field mapping to more complex translations like processing parent/child relationships. The last agent then writes out the final message once modified.

In this pattern, multiple Kafka topics provide delivery for each segment of the pipeline. In this way, not only does the code remain more modular, but it also becomes possible to scale each component independently. This allows for message prioritization, but also helps optimize the cost of processing against the true demand.

About the Contributors

Steve Stearns is a Regional Success Architect Director at Salesforce on the Scale and Availability team. Scale and Availability architects apply the learnings from our large customer engagements to help influence product scale evolution and innovation.