Provided as a free resource by DataStax AstraDB
In this episode, Scott interviews Andrew Jones, Tech Lead of the Data Infrastructure Team at GoCardless.
Andrew shares the story of how operational system changes kept breaking downstream data consumption (sound familiar?), especially using CDC. The software engineers couldn’t easily use the CDC tooling and the data engineers could easily use the data from it either as CDC didn’t structure the data for easy consumption.
Andrew wasn’t really sure how other people were handling taking the API contract concept and leveraging it for data but started building out some generic simple tooling to let consumers and producers feel somewhat comfortable with their data contracts.
A big revelation was in helping data consumers make better asks for data. The data consumers weren’t used to asking the producers for data, especially in a reliable and scalable way (sound familiar?). GoCardless now has an actual standard form for data consumers to use to request data and that is working quite well.
GoCardless plans to completely remove their CDC architecture by 3Q of this year to replace with data contracts. They are focusing on providing tooling to give domains the autonomy to serve data to consumers in their own way. While it isn’t data mesh, especially with the lack of interoperability between data products and lack of source/producer-aligned data products, it seems to be working for GoCardless thus far.
Andrew’s Medium post called Improving Data Quality with Data Contracts: https://medium.com/gocardless-tech/improving-data-quality-with-data-contracts-238041e35698
Twitter: @andrewrjones / https://twitter.com/andrewrjones
Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/
If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
All music used this episode created by Lesfm (intro includes slight edits by Scott Hirleman): https://pixabay.com/users/lesfm-22579021/
Data Mesh Radio is brought to you as a community resource by DataStax. Check out their high-scale, multi-region database offering (w/ lots of great APIs) and use code DAAP500 for a free $500 credit (apply under “add payment”): AstraDB