Provided as a free resource by DataStax AstraDB
In this episode, Scott interviews Kurt Gardiner, Engineering Manager of Data Engineering at Australian Insurance company nib Group.
Kurt shared some insights into nib’s journey so far, including the search for something like data mesh before Zhamak published, tool choices (Snowflake, dbt, Fivetran, EventBridge, Kinesis), the slow-role approach to replacing legacy implementation (the “application strangler” pattern mentioned), how they got started, and much more.
Much of nib’s approach is the small-scale tactical while building incrementally for the bigger strategic focus. E.g. helping teams to design their data products somewhat manually while building the reusable tooling to be far less manual going forward.
Along their journey, there was some internal pushback from data consumers, especially those used to consuming from the data warehouse. To do data mesh right, Kurt and Scott both emphasized the need to set things up so they can evolve. That will frustrate or scare some people and it’s important to work with them to see why that matters. There also needs to be a high tolerance for failure – you will NOT get everything right on your first go.
Kurt also waxed poetic (said nice things) about event streaming patterns, especially CQRS – see link below for more info -, for a useful and scalable pattern that is good for both application development and creating a scalable and useful domain data model. But it requires a complete redesign so it is probably something to slowly introduce where it makes sense, if at all.
Some pithy nuggets of wisdom from Kurt that are highly applicable to data mesh:
“The single biggest problem in communication is the illusion that it has taken place”
“Nobody cares what you know until they know that you care”
Application Strangler pattern (recently renamed Strangler Fig Application pattern): https://martinfowler.com/bliki/StranglerFigApplication.html
Kurt’s LinkedIn: https://www.linkedin.com/in/kugardiner/
nib Group careers page: https://nib.wd3.myworkdayjobs.com/careers
Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/
If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
All music used this episode created by Lesfm (intro includes slight edits by Scott Hirleman): https://pixabay.com/users/lesfm-22579021/
Data Mesh Radio is brought to you as a community resource by DataStax. Check out their high-scale, multi-region database offering (w/ lots of great APIs) and use code DAAP500 for a free $500 credit (apply under “add payment”): AstraDB