Please Rate and Review us on your podcast app of choice!
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
Episode list and links to all available episode transcripts here.
Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.
Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here. You can download their Data Mesh for Dummies e-book (info gated) here.
Srinivas’ LinkedIn: https://www.linkedin.com/in/spaluri/
In this episode, Scott interviewed Srinivas Paluri, CTO at Rentbase. Srinivas was previously part of a data mesh implementation as the Senior Director of Data Engineering at Zillow.
Some key takeaways/thoughts from Srinivas’ point of view:
- Data mesh advice to former self #1: Ambiguity is inevitable. Don’t be afraid of ambiguity – it’s often better than binary thinking – but also be as clear as possible on responsibilities even if the target outcome is ambiguous. Clear responsibilities at least drive things forward.
- Data mesh advice to former self #2: Involve product management way earlier. Every product owner needs to understand product ownership, prioritization, and what is the value to the business. And then communicate the value of the work to the business. If that value’s not clear, why are you doing the work?
- Data mesh advice to former self #3: Create a small team, maybe 5-6 people, focused on enabling new domains to learn how to own their own data and create data products. Scott note: see episode 48 for how ITV is implementing this pattern
- Prioritization – and communication around prioritization – is probably one of your most useful tools in data mesh. If you get good at that, teams will often buy-in more quickly. Data producers see changing priorities, not more work. Consumers have a clear understanding of what work is done when and why instead of silence or a link to a Confluence or JIRA page.
- Good data mesh product management isn’t only focused at the data product level or the platform. You need to look at the bigger picture of how everything fits together to drive even more value. Make sure you have thought coverage of mesh-level product management.
- Make sure everyone is aligned on how data engineering work supports business value and the organization’s goals. The exec team should understand it of course but don’t skimp on informing the data engineers – if they know more, it can help with better prioritization.
- Data ownership – and buy-in on that ownership – is crucial and can have immense value. You can drive that buy-in by showing people the consequences of centralized data ownership creating bottlenecks. What more value could have been created if things weren’t stuck in the JIRA logjam?
- To communicate the cost of data issues, it’s very hard to try to put dollar amounts on them but you can show people the business impact and they can start to calculate it themselves – ‘this report takes 5 people 3 days to clean up at the end of the quarter’ – that has a big impact on business people who can calculate costs far better than the data team.
- Work with teams to assess and document wins with data. You might not get to exact numbers but you can show value with a little bit of partnering. Then share those wins and attract others that want some of that data-enabled value creation 🙂
- Look to business partners for setting prioritization around data work. See if you can get them to seek the budget to do the work with you. What are the company’s priorities and where should you be investing to support those? How does data work play into that?
- Set yourself up to learn and iterate when it comes to architecture. Prepare yourself to fail, and fail fast. It’s way too hard to try to get everything right up front, you’ll expend too much energy. Embrace that and get moving.
- Engineering teams need to be part of key business decisions. Not necessarily as a decision maker but engineering finding out about changes once systems break – this is a pattern that has to change. If you want to be data-informed, data-driven you have to consider the impact to data as part of the product decisions/changes.
- You should make sure you support new data owners with technology and capability building. You need to make them capable of owning data, not just give them the responsibility. And that won’t be free.
- In data mesh, seek out teams that will give you blunt and honest feedback and are willing to work with you to improve your data mesh implementation. You want people who will push you to do better and share in the pain and the success.
- Focus on communicating what data mesh changes for your organization, what value it unleashes. Really talk with your central data team and explain what it means for them and that it’s not a threat to their jobs, they get to focus on cooler things than building pipelines.
- ?Controversial?: Don’t look to rebuild your entire tech stack to do something like data mesh. See how far you can get with a lot of what you have already.
Srinivas started by saying how important architecture is, yes, but how it’s too hard to try to get everything right up front. Instead, set yourself up to be able to change your architecture as you learn more. Fail fast is crucial, you need to be able to get moving sooner rather than later. Otherwise, there is too much risk in building something that doesn’t fit needs or even more likely, never building anything because you’re always waiting for the perfect technologies.
To do engineering – especially data engineering – well, Srinivas believes engineering needs to be part of the decision making process. If not providing input, at least then being aware of strategic shifts and working with key stakeholders to shift systems to better align to new changes. You can’t change your business model and not expect it to not require a shift in what data you need and how you work with it! And you could have started collecting data sooner for the business model shift too 🙂
As data mesh aficionados know, centralized data engineering can create bottlenecks – and almost certainly will at scale. Srinivas recommends you show people the impact of these bottlenecks through specific examples. You can use that to drive better buy-in for decentralized, domain-based data ownership. It can be hard to quantify the exact impact but it’s important to try to at least communicate the outcome of those bottlenecks, how did it impact day-to-day business operations and capabilities.
When trying to push data ownership to domains, it’s crucial to make sure you give them the support to actually own the data per Srinivas. That’s technology and that’s capabilities/understanding. If you are saying data ownership has value and you can show the value, then the organization should support that value, it’s not free, it’s worth investing in.
Srinivas believes you should focus on making gradual changes rather than sudden shifts in a data mesh or other large-scale data implementation. There needs to be commitment to making change to do it right. And as part of that, look to create and foster close collaboration with users. You need teams to be blunt and honest to help you get to where you need to be, to get to a valuable outcome. That feedback will help you improve your processes and platforms.
If Srinivas could give 3 pieces of advice to his former self about data mesh:
#1 On ambiguity: it can be helpful rather than binary right or wrong type thinking. So get comfortable with ambiguity, the world is rarely black and white. But ambiguity should be more about the ways of achieving or the expected end result – make sure to set responsibilities as clearly as possible because ambiguous ownership rarely works out well.
#2 Get product management involved earlier. They should be there from the start. Understanding product ownership is so crucial to getting data mesh right. You need people who are focused on prioritization and focusing on things that are valuable to the business. If it doesn’t have a clear value, should you be doing it? And then when you establish value, communicate it!
#3 Look to create a data mesh/domain enablement function. A team that is specific to helping additional domains figure this out. It’s not easy and you will find great advocates that can really help teams get moving quickly. A team of 5-6 people would probably be a good enough size. Scott note: see episode 48 with Scott Hawkins for a great example of this internal consultant team in a box approach
Another thing Srinivas learned looking back on his time at Zillow is to focus on communicating to people what does data mesh change for the organization and especially for them. There is a vague sense of data mesh changing the way we work but what is the actual value we expect to drive. Not a specific dollar amount but what’s the vision of the organization of once you reach a relatively data-informed, data-driven state? Faster reactions to market changes? Better identification of new opportunities? Significant cost savings? And again, talk to people about what changes for them, driving value and also responsibility/role wise.
If you want a somewhat visceral approach to showing people why the central data engineering team has become a bottleneck, Srinivas recommends asking how long would it take to clear your current backlog with your current team if no new tickets came in. What about how big of a team would you need to actually clear your backlog based on the number of tickets coming in – is anyone’s backlog actually decreasing? What could your organization be doing if you didn’t have that bottleneck? What more value could you be creating by really enabling teams to understand and leverage the organization’s data? Yes, there will be a cost but we have to invest to create value 🙂
When driving data mesh buy-in, Srinivas again went back to the need for product management in general. Talking with the producers about why this matters; how you are shifting their prioritization, not just adding additional work; how they can actually achieve this and what are meaningful milestones / incremental value deliveries; etc. Prioritization – and communication around prioritization – are likely to lead to your easiest, happiest paths.
Data mesh product management should not only focus at the micro level – the data products and the platform – but also at the mesh level according to Srinivas. To really drive value at scale, you need interoperability and things working in harmony. Product management is not just about managing the product but the entire suite of products. Think about your data products as an overall suite of information to serve many use cases and drive a huge amount of value.
For Srinivas, when looking at setting your priorities for data work, start with what are the overall organization’s priorities, what are the priorities of your business partners? If you want to do data work that isn’t valuable to them, will it be valued even if it drives the expected value? Look to find data work that are both valuable and valued and that support the overall organization’s priorities. Scott note: this is especially key as you are getting going on a data mesh journey.
Everyone on the data engineering team should understand how their work supports the company’s priorities and drives value for Srinivas. And the exec team should understand how the data engineering work maps to value creation too. Sometimes, that can be harder with platform work and the like, but it’s important for everyone involved to understand how the data engineering work drives value. That helps map to prioritization decisions as well.
Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/
If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf