#68 The Build vs. Buy Dilemma of Data Platforms – Interview w/ Doron Porat

{{top-bit}}

Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here

In this episode, Scott interviewed Doron Porat, a Data Infrastructure Leader at the SaaS company Yotpo.

Some crucial points Doron made:

1) Be kind to yourself when you make mistakes – it’s worse to stagnate so don’t be afraid of change and making choices

2) Build versus buy is always tough but don’t let your ego get in the way and push you towards building everything

3) If you do buy, build a close relationship with your vendors to help influence the roadmap and have an outlet if you are having issues

4) A data platform team’s job is to drive usage as usage means creating value – drive towards that and set your KPIs around platform usage

5) There will likely be many different types of consumers of your data platform – work to improve / optimize the user experience for most folks

Doron is a technologist at heart so for each decision she instinctually wants to build instead of buy. And at the start of building out the data platform for Yotpo, that was typically her decision. But as the demands for more and more capabilities from the platform, the increasing ubiquity and quality/scalability of as-a-service offerings, and the growing need to drive usage and developer happiness instead of manage cool tech, she started to consume more and more managed services.

When you are building out the platform, vendors, in the long run, can often better serve your needs because they have a whole lot of people focused specifically on making what you use better. You need to make bets on the vendors getting to where you need them to be and sometimes they don’t pay off either. To up the chance of making those bets pay off, Doron recommends building relationships with your vendors to influence their roadmaps and get help when necessary.

Doron strongly recommends putting together a framework for evaluating build/buy decisions. Some of the factors she considers are how extensible is the offering, is this taking on too many challenges in one solution, cost, the cost of later migration to or from a managed service, open source compatibility, etc. One thing Doron talked about that many teams seem to struggle with is the ego hit of saying someone else managing a service we use will drive more value. That’s always tough but needs to be addressed.

Doron talked about the strong need to drive your platform forward, not just be responsive. Provide a roadmap, set time aside for innovation, etc.

Doron made the point there is a difference between learning to leverage a tool and operate it. You want to make sure you build out the knowledge around leveraging a tool whether you are operating it yourself or not. It’s an interesting balance when you build that you don’t focus too much on one or the other.

Doron talked about the general job of a data platform team is to drive usage because that usage means creating value. Serving critical needs for users is crucial to driving adoption and focus the user experience on the business logic – no matter how cool the tech is, a data platform team’s job isn’t to expose that to users. And your team’s KPIs should reflect usage. However, there are all kinds of users – on both the consumer and producer side – that you need to serve so don’t create a one-size-fits-all experience.

If your data platform isn’t easy to adopt, no one will want to use it. Your data platform should make things fun and casual to do. There needs to be a big focus on enabling data testing including staging environments for data – that will drive usage.

Doron’s LinkedIn: https://www.linkedin.com/in/porat-doron/

Our journey towards an open data platform: https://medium.com/yotpoengineering/our-journey-towards-an-open-data-platform-8cfac98ef9f5

A simplified, lightweight ETL Framework based on Apache Spark by Yotpo: https://github.com/YotpoLtd/metorikku

The Data Swamp (in Hebrew): https://open.spotify.com/show/5YDdtRhp1RVw7r5fbYFtPQ?si=5x4HzOyhTX6n46HqY5kV6w&nd=1

{{bottom-bit}}

1 thought on “#68 The Build vs. Buy Dilemma of Data Platforms – Interview w/ Doron Porat”

  1. Really valuable perspectives on many things, including long-term considerations on Build or Buy linked with “What business are we in?”. However, for Data Mesh to become truly a hit beyond hype, companies of all types and shapes will be dependent on platform capabilities. And not just any data platform but platform of Data Mesh kind. That is, platform with data product and mesh “experience planes”. This episode focused on “infrastructure utility” plane which is necessary but far from sufficient. Hope this can be addressed later – if not as podcast episode with single guest then at least as Data Mesh learning community task.

Leave a Reply

Your email address will not be published.