Decoding the data dilemma: How to strike a balance between metrics and instinct

Chris LoSacco: Social media is a little bit of a barren hellscape right now.

Gina Trapani: Yeah.

Chris: And if LinkedIn can be the glimmer of sunshine that we need, let's take it.

(CATALYST INTRO MUSIC)

Chris: Welcome to Catalyst, the Launch by NTT Data Podcast. I am Chris LoSacco, VP of product at Launch, and I am joined, as always, by my co-host Gina Trapani. How are you, Gina?

Gina: Hey, Chris! I'm doing great.

Chris: Great.

‍Gina: I've been quoting LinkedIn to you. I've been spending a lot of time on LinkedIn. Here's the thing. Longtime listeners of this podcast know that we, and our co-host before us, made fun of LinkedIn ruthlessly.

Chris: We had a couple episodes where we just...

Gina: Mocked.

Chris: ...Like, cut it apart, you know.

Gina: We did. Just cut it apart. And yet, it is where I spend most of my time right now, because literally everything else is a garbage fire.

Chris: (Laughs) Right. When was the last time you looked at Twitter, X, whatever?

Gina: Ugh. I... I mean, it's been... I try not to.

‍Chris: Months? Yeah.

Gina: Months, months. And, look. LinkedIn is a professional network. It's where you, you know, announce your career changes, where you list jobs and find jobs. And this is my argument with LinkedIn, is like, it's just like... It feels so fake. And it's people talking about careers. 

Chris: There's a bit of it. Sure.

Gina: I mean, that's just real. And it depends on who you follow. Right? It is a totally different world. So I have to, like... even out my dose of, like, earnest real LinkedIn with the Reddit, you know, commentary on what happens on LinkedIn.

Chris: I think that that is a healthy thing that you're doing to just balance the scales a little bit.

Gina: Yes. (Laughs) I need a little cynicism. I'm too much of a New Yorker.

Chris: 100% cynicism is not good, but a little bit of cynicism is good.

Gina: Keeps you honest.

Chris: Like, you need, you know... Exactly. You need to have a little bit of like, you know, look at that from the corner of your eye and be like, wait, wait a second. What? Like, that is an okay thing.

Gina: Yes.

Chris: But, I mean, I find that the signal to noise ratio on LinkedIn is pretty good.

Gina: It is pretty good. Yeah.

Chris: Like, it's more signal than noise. There's noise. Like, don't get me wrong.

‍Gina: For sure. 

Chris: But it's a lot of, like, relevant, good stuff. And I actually do like seeing, oh, this person wound up at this other place. Like, good for them.

Gina: Yes. Yes.

Chris: Oh, this person launched their startup finally. Cool. Like, I, you know, I'm gonna take a look at that. So, there's really good stuff. And I do think, you know, there are some, like, interesting soundbites about product and business and leadership and it's like, okay. Again, you can drink the Kool-Aid a little too much, and then it's like, not good. But in small doses I think it could be really valuable.

Gina: It's pretty good. And we're at a moment, I think, especially in our sector and our industry, there's a lot of transition. People are leaving jobs and starting jobs and transitioning from W-2 to freelance, or going from freelance to W-2.

Chris: Yep.

Gina: And it is really interesting to see, especially, like, the good writers and the thoughtful folks and the folks who are willing to be, like, a little real and a little vulnerable, to see how people kind of tell themselves the stories of their careers and, like, the things that they learned, and the things that were hard and the things that were good. You know, I think for sure there's pressure on LinkedIn to put on this, like, professional facade.

Chris: Oh, without a doubt. 

Gina: And come across as this amazing leader, and with... You know. And I'm actually, I'm writing more on LinkedIn. I'm going for real. I don't always succeed. But... And I've learned a lot. Like, there's... Sometimes there's some really good content out there. And there are definitely some, and I'm doing air quotes, "thought leaders," who, you know, are crafting their PDF slideshows with inspirational quotes from various leaders. And I got taken in by one of these. Actually, this person posts videos.

Chris: It was a video. 

Gina: Video clips of, like, interviews with leaders. And what I appreciate is when people do those, like, short video snippets, because I don't have time to, like, watch a big, long, you know, interview. But then, like, put the captions, so even if you're scrolling and you have your volume off, you can kind of, you can read what the person is saying.

Chris: Yes.

Gina: So I got sucked into this particular video, and I, and it started just live rent-free in my head. And then I started talking to you about it, and you're like, you should, we should do an episode about this. So this was... And we'll link this in the show notes. This was an excerpt of a video interview with Jeff Bezos, talking about his  time at Amazon. And it's real short, like a minute and 30. And he was making a point that got me, because it was antithetical, like it was the opposite of what, you know, it was a little bit of a contrarianism view, right?

Chris: Yeah. 

Gina: And the point that he was making is, not to rely on data. Entirely. Like, that if you have a hunch about something that's happening in your business, in your product, in your customer experience, and the data says there's not a problem, but you have a hunch that something's off, trust your hunch. Which really surprised me.

Chris: Yeah. 

Gina: Because Amazon was huge. (Laughs) And to have a hunch about something at that scale. First of all, it feels impossible, right? Like, a human brain can't actually... (Laughs) You know, get itself around the experience of, you know, millions of customers. But this was his message. Trust your hunch. And he tells... Ugh. Great story.

Chris: It's an amazing story.

Gina: You tell the story. What was the story that he told? I love this. I wanted to be in this conference room so bad when this happened.

Chris: Me too. Can you imagine if there was, like, a documentary-style recording of this? It would be just...

Gina: I would just love to see.

Chris: Must-see TV.

Gina: Must-see TV.

Chris: So it's an executive meeting at Amazon, right? Where you've got the senior leadership in the same room, sitting around a conference table. And Bezos is in the room with his team. And they're talking about their customer service team. And they're reviewing the metrics, and the metrics are great. Like, the reports say, you can get a response from a customer service rep in under a minute, and people are happy and the customer service team is killing it. And the, you know, head of customer service is in the room, and saying, like, I'm really proud of what these reports are showing. My team is doing an amazing job. And Bezos is like, I don't buy it.

Gina: (Laughs)

Chris: I have this hunch that we're not doing as well as these reports say we're doing. I've heard a lot of anecdotal evidence, or stories, that are like... 

Gina: And complaints, right? From customers. 

Chris: ...That our customer service is not great. And I just, I'm not feeling it. And the head of customer service pushes back, and is like, no, no, no. The reports say we're doing well. Our average response time is under a minute.

‍Gina: Under a minute. Right.

‍Chris: There's nothing to do here. And Bezos stops the meeting, and says, why don't we call?

Gina: Let's call customer service right now. And see if it's actually under a minute. (Laughs)

Chris: And he pulls out his phone, and he dials the 800 number. And the whole music starts playing, and he's like, we're gonna test this live. And they sit there.

Gina: Around the conference table.

Chris: And sure enough...

Gina: Listen to the hold music.

Chris: Like ten, 15, 20 minutes.

Gina: It's multiple minutes. 

Chris: Before they get a response. (Laughs)

Chris: And it just cuts through all the BS. Where it's like, reports be damned. 

Gina: Right. 

Chris: I just called, and I didn't get the response, you know? So there's clearly a problem. And then it kicked off a bunch of work to go figure out what was going on. But it's such an amazing story. Because it's like, you know, you can imagine an executive saying, well, I'm looking at my report.

Gina: Data says under 60-second response time when you call. 

Chris: Thumbs up. Green light. Yeah. 

Gina: Customer service. We're measuring call times, and under 60 seconds. But also, you can see Bezos, like, surfing around Amazon, and seeing reviews from people saying, you know, I sat on hold waiting, and being like, hm. And he said, I called customer service. It wasn't never under a minute. And I thought, well, maybe, you know, it's easy to think, well, maybe I'm the outlier, right? Like, they're dealing with a huge volume of calls, right? So there's a possibility that, on average, it's under 60. Maybe it's just me. But I love that he was like, let's just call now. (Laughs)

Chris: Let's just call now. It's verified. You gotta verify what you're seeing. And... 

Gina: That's right. Trust but verify. 

Chris: Again, you can't argue with it. It's like, oh. We all just had that experience.

Gina: Yeah.

Chris: So, I don't care what I'm looking at on this report. Like, we gotta investigate. We gotta dig deeper.

Gina: Right.

Chris: And data... Data can lie to you. Now, well... That's not the right way to say it. It just... There's a lot of nuance, you know, when you are measuring something, especially at a large scale.

Gina: Yes.

Chris: To try to interpret and judge, kind of, what you're seeing, you know? And sometimes the data doesn't tell the full story. Sometimes you do have to say, I need to dig a little deeper here. I need to do some pressure testing myself to realize, like, yes, we're good, all systems go, or, we actually do have a problem here, despite what I see on the dashboard.

Gina: Yeah, I mean, this... I love this point that he made, which is like, the data's not lying, but sometimes you're measuring the wrong thing. Sometimes there's a problem with your methodology.

Chris: Of course.

Gina: Just because the data says one, you shouldn't just accept it, right? Like you gotta interrogate a little bit. Like, trust that hunch. This really spoke to me, because, you know, as someone who, you know, I spent my career in small companies. And the thing about small companies is that you don't have the scale, you don't actually have data.

Chris: Right. 

Gina: You don't have a lot of data to tell you. So a lot of times, you really are kind of going with your gut. And I  used to kind of feel like, oh, that's sort of rookie. And you know, you've heard these stories of founders who were like, believed that they were doing the right thing even though their customers were telling them they needed something different. And, you know, these like, crash and burn stories. You gotta listen to your customers. You gotta look at your data. Like, and I do... I think that data is really important. The conclusion that I've come to in my career is that I don't want to be data driven, I want to be data informed. I want to have the data. I want to look at the data. I want to consider the data. But there are times when you have to use your brain and go, like, this doesn't feel right. I have a different hunch. This has been my, this hasn't been my experience. This isn't what I'm seeing. And so, I want to push on this a little bit. Like, I'm not just going to accept this as, we have thousands of data points over a certain amount of time and they show us this. Like, you just got to ask. And maybe you're wrong. Maybe your hunch is wrong. 

Chris: And then you say okay. Yeah. 

Gina: But it's important to interrogate. And then you say, okay, just checking.

Chris: Yes.

Gina: Data, for me, is like a lot like process. I think that...

Chris: What do you mean?

Gina: So, I think that people rely on data so much to tell them what is right and what is happening, and then make decisions based on it. And I think that we, sometimes, I think on, especially on bigger teams, you know, you build this data team and you're just working a whole lot on getting the data right, because you feel like if, once you have the data right, you'll just know what to do. And process is similar. I think that people start to lean on the process, and take the thinking out of it, you know? (Laughs) And just feel like, if we just get our process right, and if we follow the process, then we'll do the right thing. But it's a balance, right? Like, data is a support. Process is a support. You'd have just enough to make the right decisions. But ultimately, like, as leaders, as executives, as product people, you have to make a decision that is based on, you know, the factors that you know, but also take into account those hunches and your strategy, you know?

Chris: Yeah. Yeah. I think you're making an amazing point. Both data and process, to some extent, they are in abstraction. They're one level removed, right?

‍Gina: Yes. Yes. 

Chris: You're not looking at the product itself. You're looking at, in the data case, you're looking at, you know, metrics you're gathering about the product. In the process case you're looking at the how, of how you're building the product. And for large platforms, there is work that you need to put in to say, how do we make sure that our data is good? How do we make sure we're gathering the right analytics? How do we make sure we're asking the right questions? Like, there's value there for sure. And I think neither of us are saying, like, you know, ignore all the data. That's not the point.

Gina: Right.

Chris: Same thing with process. You do need to have some level of process, right? Especially... Maybe you can, you know, if you're a very small team and you're just, you know, focused on talking to each other constantly, then maybe you can get by with, like, little to no process.

Gina: Yes.

Chris: But once you reach a certain size, you need to establish a concrete way of working, or the thing starts to fall apart. It doesn't feel good. Like, to the people on the team. But the problem is, when you go too far in the other direction and you only care about the abstraction. 

Gina: Right.

Chris: You don't care about what it's enabling, right? You're only thinking about the data, and how you make those numbers move 5% up or 2% down or whatever it is, and you're not thinking about the product itself. You're not thinking about what you're building.

Gina: Yes.

Chris: The example I love to use when I think about this kind of thing is, like, getting people to sign up to a newsletter. And... 

Gina: Yes. 

Chris: If you're just looking at the metrics, you're going to do...

Gina: Sign-ups. Yes. 

Chris: Everything you can, right? Sign-ups is what's on the dashboard. You need to increase that number. You're going to put pop-ups in people's faces. You're going to send them push notifications. You're going to email them all the time. Because you want those sign-ups to go up. And sign-ups will go up.

Gina: Right.

Chris: But the reason that sign-ups are going up is not because you're getting more interested customers. It's because you're slamming things in people's faces. And they're like, how do I get this pop-up to go away? Oh, I've gotta put in my email address.

Gina: ...thing out of the way? Fine, I'll just put this fake email address or my email address. Yep, that's right.

Chris: Exactly. And what you've optimized for is okay, yeah, the number's going up, but you're actually severely harming the product that you're putting out into the world.

Gina: Yes.

Chris: And it's very easy for product teams. If they lose sight of the, you know, the core thing that they're building, to get a little, like, misguided, if they're only thinking about driving up a metric. There's also this, um... Have you ever heard of Goodhart's law?

Gina: No. What's that?

Chris: I was looking around a little bit about this, and there's this British economist who wrote about this phenomenon in, back in the '70s. And, and it's often stated as, when a measure becomes a target, it ceases to be a good measure.

Gina: Mm. Oh, that's so good.

Chris: It's so good, right?

Gina: It's so good. Yes.

Chris: If you're just thinking about, how do I optimize newsletter sign ups?

Gina: Right.

Chris: ...Then you're going to make a whole bunch of changes that are maybe actually counter...

Gina: In service of that KPI. Right.

Chris: Yes. It's only about the KPI, and it's maybe counter to the larger organization's goals. And then the KPI itself becomes, like, problematic.

Gina: Yes. Right. You cannibalize the user experience and the spirit... This is, it's... You know, I have a long history in online media. Like, the big example for me is, you know, there was a time when, you know, A/B testing, like, headlines, you know, like on a big news site, like a BuzzFeed or, you know, a big news site that's putting out a ton of content and news stories, you know, and they'd have editors write like 3 or 4 versions of a particular headline. And then, you know, I think BuzzFeed was actually pretty famous for this. This is years ago now.

Chris: Yeah. 

Gina: But they'd run the different headlines on the site, and the ones that got the most clicks, that would be the headline. And it's  kind of brilliant, right? Because you're like, oh, I'm, you know, I want to engage readers. I want to give readers what they want. This can go really wrong, though. This happens in marketing too, right? Like you're incentivized... Your targets are about clicks and views, right?

Chris: Yes.

Gina: Because you're selling ads. But that's a race, that can be a race to the bottom, right? You can be like, well, it turns out that the most sensationalist and binary and ridiculous, absurd headlines... You know, eventually you become a tabloid, right? (Laughs) 

Chris: Yeah. Exactly. Exactly. 

Gina: Because, like, it's like... And it's like, okay, like, if that's what you want to be, but also, like, is there an editorial strategy? Is there an overarching... Like, what is your whole, you know, news organization's goal? If it's to inform, the more sensationalist headlines, like, yeah, they're clickbait, but they might be not the quality and in the spirit of what you're actually trying to do for readers, right?

Chris: Exactly.

Gina: And also, your audience changes. This happens in marketing too, right? Like, I've seen... 

Chris: Of course. 

Gina: You know, I've heard of marketing teams, A/B test or just test certain kinds of copy or imagery. And it's like, oh, well, this one tested the best. It's like, okay, well that tested the best, but it's just not as good as the other ones. Like, sometimes you just have to make a quality choice.

Chris: Right. A quality choice. 

Gina: And be like actually, quality choice. Let's pick out the higher-quality thing. 

Chris: Or it doesn't... this doesn't sound like us. This doesn't feel like us. Like, there has to be room for that.

Gina: That's right, that's right. Yeah, absolutely.

Chris: There's this idea... I mean, it's hard to quantify, right? Because it's subjective. The beauty and the curse of data is it's objective, right?

Gina: Yes.

Chris: You measure it, and then either it's going up, or it's going down, or it's staying the same, and you can make decisions based off of that. That's very reassuring, I think, to people, where it's like, well, we don't have to argue about this because it's just numbers. And we're just measuring stuff, and we can see. But the subjective stuff is really important, right? This idea that, like, there's taste. 

Gina: Yes. Yes.

Chris: You have to introduce someone who has a vision, or a particular style, or voice, or tone, in mind for their company, and what they are putting out into the world, right? And that's just as applicable for digital products as it is for content. And, I mean, frankly, as it is for, like, physical products, too.

Gina: Yes. For sure.

Chris: And what you are producing is a representation of your company, and it is how your customers or your readers or your visitors, it is how they experience your brand, and it's going to inform whether or not they want to give you money for your product, you know?

Gina: That's right.

Chris: And if you are only thinking about, I need to make sure that my data is up into the right, then, you know...

Gina: (Laughs) Right. Do I present it to the board or my boss or whatever... Yeah.

Chris: You can very easily lose sight of, but what are the things that we're putting in front of people? You know?

Gina: Yeah. What are these metrics in support of? Yeah.

Chris: Right. And then before you know it, you know, your company or your team is, like, off in a different area of the world. And you're like, oh, this is... You know, this is not what we signed up for. This is not what we started to build, like, three years ago. Like, we're just in a very different place. And it's hard to come back from that, frankly.

Gina: Yeah. It is. Right. I mean, the Amazon example, right? Like, they set a 60-second hold time, KPI-like metric. No call should go over that. But that measure, right? Is in service of, they want to be a customer-obsessed... 

Chris: Exactly.

Customers will get the support they need in a timely manner and they won't, you know, they're not a company that wastes people's time, right? It's like, well, I see a bunch of complaints about how our customer service, you know, takes forever. And there are all these hold times. Like, so it doesn't, like, I'm, you know, I'm focusing on great customer experience. We're not providing it. So these numbers... Like, this is what I love about the story, right? Is that Bezos was thinking about, you know, I want happy customers, and I'm not seeing happy customers.

Chris: Right. Right.

Gina: And the customer service person was like, my team is killing it. Under 60 seconds, here, look, look here. Here's what the data's showing. And so, they had to align on that.

Chris: Yeah. 

Gina: I talked to a leader once, really interesting conversation. You know, he was saying to me, you know, I'm hearing from my team that, you know, morale is down and there's, you know, all this attrition and people are just leaving and our business is in danger. Everyone's going to leave. No one's happy. And, you know, we've got to make huge changes to our employee experience and our policies and our perks. He's like, you know, and I thought, like, oh, maybe I'm missing this, you know? So I went to HR and I said, can you pull our attrition data? Like, are we bleeding talent?

Chris: Yeah.

Gina: And HR came back, and he's like, and it turns out, you know, our attrition levels are at a record low. And so, here's what I loved about this, this leader... I thought this was really smart. He was like, either our data is wrong or the perception is off. And I'm not sure which it is. Like, I love that he was willing to question the data.

Chris: Yeah. Yup.

Gina: Like, yeah, it could be that this data is wrong. You know, because I want to believe my team member. Or it could be that this team member's perception, maybe they had a couple of high-profile, you know, exits...

Chris: Departures. Right. 

Gina: Where, you know, made a lot of noise on the way out. Maybe it was people just near this person versus another group. Like, you know, perception is informed so much by where you are and what you hear and the story that you're telling yourself about what's going on. And so, what I appreciated about this conversation is that he didn't immediately dismiss the employee's concern. He was like, maybe this data... Like, maybe I shouldn't trust this data. Maybe this is actually happening. And I remember him saying to me, like, I'm just not sure.

Chris: Yeah.

Gina: I'm not sure what to believe here. As leaders, it's so important to not just blindly trust perception and emotion and hunches, but also not just blindly go like, well, HR tells me we're at a record low, so this is just not a problem, right? They're all, sort of, information, you know? About what's going on.

Chris: Data informed versus data driven.

Gina: Yes.

Chris: What's interesting about that story is, it could cut, you could cut it the other way too, right? Like, the person approaching that leader and saying, we have a huge problem. If he didn't go to the data, he may have overreacted, right? Like, oh my God, we have...

Gina: Oh my God, bleeding talent. Yes. Yes. 

Chris: Right? This is, like, a huge problem for us, and we need to make all these changes, and that may have not been necessary.

Gina: Right.

Chris: It's really important, you know, to make sure that you do have good data so that you can have that be a component of your decision making, right?

Gina: Yes.

Chris: If you're going totally on instinct, you may overcompensate or overreact to something that is...

Gina:  A problem that maybe doesn't exist. (Laughs)

‍Chris: Right. Or something that's louder, you know, in the moment, than it might otherwise have been.

Gina: Yes.

Chris: You know, you can sort of properly calibrate it, when you're like, let me also back this up with what we're seeing in the data.

Gina: It's all signals, right? As leaders, we're getting different signals, right? Like, you see the data set, you see what you're hearing from customers, your own experience, right? And I think there's a train of thought which is like, just, you know... Especially when you're building product, right? You're like... Releasing something in MVP. You're building the next thing. You're listening to customers. You're getting user feedback. You're constantly iterating, right? You're constantly releasing, getting feedback, releasing, right? And I think that, in that process, it can be easy to just start to rely... (Laughs) On more, on the data. Right?

Chris: Yeah.

Gina: But it's so important that, this is the point you made earlier, right? Like, the metrics are in service of a bigger vision, right?

Chris: Right.

Gina: Like, you can't just drive to the metrics, even though they might look great on that slide in your boss presentation. You know, the weekly business review that Bezos was sitting in. So this is important, right? Like, I think you gotta think about the whole holistic picture.

Chris: Yes.

Gina: And not just say, this is the data. You know, these are what the numbers telling me about this one particular metric.

Chris: That's right. When you're thinking about digital products or digital platforms, you also have to think about where you are in the life cycle of that product. Because, if you're further along, or if you're in a phase where you are optimizing, you're going to lean, I think, a little more on the data. And you're going to look a little more at, like, how are people actually using this piece of software? How can we do a little more A/B testing, see what feels right, and make, like, these... Not micro-adjustments, but these, like, smaller tweaks, and these, like, refinements, let's say. That can be really helpful when you drive them by data. But if you are thinking about, like, the next major release and you have to... I mean, you and I, you know, we have, uh, our own views on the word "innovation." But if you are innovating, right?

Gina: Yes. 

Chris: If you are coming up with something new... 

Gina: Yeah. 

Chris: Data's not going to get you there.

Gina: Right. The signals are very different. You don't have millions of users' data points, you know. (Laughs)

Chris: Right. And often the data will... You know, if you try to use data to figure out where should your product go, it might send you off in a completely wrong direction.

Gina: Yes.

Chris: Because, you know, we've had, we've done experiments where we put up, like, you know, landing pages or nav items that don't really have anything there, to try to start to gather data about, like, user intent. And again, that can mislead you. Because yes, you might gather that, like, oh, people want this feature to be available, but it could also mean like, oh, I'm clicking on this thing because I just... it's a new button and I want to see what's going on. So, you need to have a level of separation from the data when you are looking at a longer-view effort on your digital platform. You need to allow for some subjective decision making. And take some chances, and take some risks. And say, we're going to go off in this new direction, and it's informed by things that are not just quantitative user metrics. You know what I mean?

Gina: Yeah. So you're saying, early in a product development, there's less giant sets of qualitative data and more quantitative qualitative.

Chris: Correct. Quantitative. Right.

Gina: Like, you're... More hunches and less data, kind of. And when I say hunches, your hunches also should be informed. Right? Like, when you're starting to build something new, you should be talking to users.

Chris: Yes.

Gina: In depth. 

Chris: Of course.

Gina: Like, hearing their whole story, understanding where their pain points are, what they... Sometimes, you know, users will tell you, I wish I had this or that. That's a different kind of data than, like, a giant data set, right? But then you're going to try some things. But in the beginning, you've got a very small user base, right?

Chris: Yeah.

Gina: So... Data set's just not going to be as big, so it's not going to be as predictive. So, more data and more, kind of, tests and hunches early on. But later on, you've got a big giant platform at scale. You have, you know, and you've got a good data implementation, which is a whole other art and science and practice. Right? You're going to have these big data sets, which, as I imagine, you know, Amazon had, when they had this conversation about the calls.

Chris: Right.

Gina: And that's going to be... It's a different thing. So, at different stages you're just going to have different kinds of signals.

Chris: That's exactly right. And at every stage of the process you need to be using your judgment about the signals, right?

Gina: Yes. Yeah.

Chris: Both the quantitative signals and the qualitative signals. The things you're hearing anecdotally, and the dashboards or reports or whatever that you're getting from your team. You need to say, I'm looking at both of these things with a critical eye, and I'm going to apply some critical decision making to some of these things.

Gina: Right. Yes.

Chris: And I'm going to decide, in this instance, we do want to follow the data, and we want to run a... Give an A/B test or, you know, there's a lot you can do. Or we're developing a major new feature and we're going to go off of, you know, some really in-depth customer conversations we had. And we really feel like this is a bet we want to make. And both are necessary.

Gina: Yes.

Chris: Like, you need to be able to choose between those paths.

Gina: That's right. Right. I mean, if you do a user survey of 100 users of a very early product, you know, one person represents 1%. (Laughs) You know what I mean?

Chris: (Laughs) Exactly.

Gina: If you've got 100,000 users, one person is a smaller, right... You're getting that aggregate, you really, it's a little bit more meaningful, right?

Chris: Yes. Yeah. That's right.

Gina: ...When 20,000 people say something versus 20. For sure. Yeah. So, I mean, point really well taken. And, about the stage of the work, and the kind of data that you're going to have, I mean, certainly we believe that products and platforms should be based on customer needs and listening to customers, right? Maybe they're not exactly what the customer asked for, right? Like, that's where some of the innovation, some of the testing things out, right?

Chris: One of my favorite, one of my favorite quotes, I'm sorry to interrupt you...

Gina: No, no, go ahead. 

Chris: One of my favorite quotes is, give me what I want, not what I asked for.

Gina: Yeah. Give me what I want, not what I asked for. That is...

Chris: I love that.

Gina: That is... Yes. This is, I think, really, truly a guiding principle, just in product, in great product management and great product strategy and great customer service. Yes.

Chris: Right. Exactly. Customer service, client services. Right?

Gina: Yes. 

Chris: This is how we treat many of our clients.

Gina: Oh my gosh, yes.

Chris: Because sometimes our clients come to us and they're like, I know exactly where I'm going. I need, ba ba ba ba, and I... 

Gina: Yes. Yep.

Chris: And it's so funny, because sometimes we'll propose, let's do a workshop, right? Let's do a... You know, let's take an afternoon and do a discovery session. And they're like, we don't need to do this. And we say, we'll do it on us.

Gina: Right. Yes.

Chris: This is free of charge. We just want to have the conversation just to make sure we're aligned. And guess what? A lot of times, we come out of that room with a different direction than when they walked in with.

Gina: Yes, yes. Absolutely. And... I mean, not... Now I'm really tooting our horn, but like, having that outside perspective and just... 

Chris: Oh, it's so helpful.

Gina: Just the gentle interrogation of, how did you get to this conclusion? What is your business actually trying to do? That is... I mean, it's so important. And it's so important to ask those questions up front, so you don't... 

Chris: Yes. 

‍Gina: ...Sink a bunch of money and time and effort into something that's guided by a particular metric, or without the general, you know, the whole strategy in mind. Yeah. And we, I mean, obviously, we love when there are clients who are willing to have that conversation with us. (Laughs)

Chris: Yeah.

Gina: Instead of just, come in and sort of order number three off the menu.

Chris: Exactly.

Gina: We don't really have a menu, but yeah. Well, as usual, you and I are in violent agreement. I love these conversations. (Laughs)

Chris: We should... We need to... We need to pick topics where we disagree more.

Gina: We disagree... We need to debate... 

Chris: You know, it's a healthy tension on the show.

Gina: Right? No, seriously.

Chris: Yeah.

Gina: I just, I felt... Bezos really spoke to me. Thanks, Jeff. You really spoke to me. (Laughs)

Chris: I'm sure he's a listener.

Gina: So don't just rely on the data. And, yeah, metrics are just things to measure. What was that quote again?

Chris: Goodhart's law?

Gina: Yeah. Goodhart's law.

Chris: Charles Goodhart. When a measure becomes a target, it ceases to be a good measure.

Gina: I mean, the guy's name is Goodhart, so. It's gotta be good.

Both: (Laughter)

Chris: I feel the need to clarify for the listeners, it's Goodhart, G-o-o-d-h-a-r-t, not Goodheart. 

Gina: Okay. Well...

Chris: But interpret as you will.

Gina: Interpret as you... (Laughs)

Chris: Here's what I'm picturing. I'm picturing people listening to this and saying, this makes sense, but man, I struggle with it sometimes. I am just... My organization is too focused on the data. Or the other way around. My organization is not really using data at all, and we could use a little bit more of it. And I need help just figuring out how to do it. You are a perfect candidate to reach out to us.

Gina: Yes. Wow.

Chris: We would love to talk to you at Launch by NTT Data. We have teams of product people, designers, engineers, analytics specialists. Right? We have instrumented entire platforms to be able to gather good data and ask the right questions, so that you're not just looking at, you know, the standard output from Google Analytics. You're getting something that is much more thoughtful and targeted, and can be a more helpful input as you think about where your product roadmap should go. So if this is you, please reach out to us. Catalyst@NTTData.com. C-a-t-a-l-y-s-t at NTT Data.com.

Gina: Yeah, please do. We'd love to hear from you.

Chris: Alright. Back to work, Gina.

Gina: This was really fun. Back to work. Have a great week, everybody.

Chris: Alright.

Gina: Bye. 

Chris: Bye y'all.

(CATALYST OUTRO MUSIC)