Tuesday, September 06, 2016

Red Hat's Jen Krieger on DevOps feedback loops

The focus on metrics and feedback loops in software development tends to be around technical measurements like uptime. Or, if we're being really clever, the business outcomes associated with those numbers. In this episode, Red Hat Chief Agile Architect Jen Krieger argues that the human side of feedback loops may be the most important.

Show notes:


MP3 audio (14:55)
OGG audio (14:55)

Transcript

Gordon:  Hi, everyone. I'm here with Jen Krieger, the Chief Agile Architect with Red Hat again. In the last podcast, we were talking about distributed teams and in general how to make distributed teams work effectively and presumably leading to get better outcomes in terms of delivering software, because that's the name of the game after all.
I'd like to follow up with Jen with something else that we were discussing came out of the Agile 2016 Conference. That was the idea of feedback loops. I've been talking alone with my colleague William Henry at some events about this idea of metrics. How do you know if your DevOps is working?
The focus tends to be measuring the throughput per second or the latency or the number of successful deploys or the down time. Those are important as are the business outcomes that stem from them.
For all the talk we have about the culture in DevOps though, we don't talk an awful lot about how you monitor how the team is doing, both from a day to day basis and in a longer term. What are some of your thoughts on that, Jen?
Jen:  Yes, it is absolutely true that when we think about the words DevOps, and we think about feedback loops, we immediately go to the space in which we're thinking about some sort of computer somewhere giving us some sort of data that will help us make a decision about some sort of thing that we're stumbling over, but we rarely, if ever, think about the fact that every single thing that we do with other people is a feedback loop, having visual confirmation of your idea.
Say you're in a meeting and you're sitting with a bunch of people. You are saying, "What do you think about my idea?" Somebody sitting across the table leans in and maybe crosses their arms and raises an eyebrow.
You might take this as a visual form of feedback and not really know what to do with it. There are all these other methods of feedback that we're getting on a daily basis that we have zero idea of what to do with.
I recently wrote an article for opensource.com where I was talking about feedback loops. I was talking about the idea that for everybody in the DevOps space ‑‑ and this might be a very salacious thing ‑‑ you just need to stop thinking about the feedback loops that you're worried about at work.
All the computer feedback loops, and all those things that come out of a machine, you just need to stop worrying about them right now. You need to start focusing on the human feedback that you're getting and the human feedback loops that you're trying to cultivate, because, I guarantee you, those feedback loops are about 90 times harder to resolve than whatever's going on with your computers.
If your production server is down, don't ignore that. I assure you, you can resolve [laughs] that problem.
If you are having a fight or a negative interaction with somebody that you're working with, I guarantee you that, if you cannot figure out how to deal with that feedback loop, you will continue to not be able to deal with that feedback loop, and it probably will also impact other relationships that you have around that particular interaction.
One of the things that I've been kind of thinking about lately that might help folks processing feedback is to also recognize that we live in a digital age where there is so much feedback coming in that sometimes it's hard to identify what feedback we actually need to listen to.
We were talking last podcast about distributed teams, and the nature of IRC, and the amount of information that comes in across just one form of communication. If you are a person who likes to participate in open source communities, email will be a problem. Trying to keep up with email from a popular open source project like kubernetes is impossible.
It's just impossible. You can't do it. So at some point you have to start identifying what feedback is the right feedback to focus on. That is going to also be critical to your addressing the entire loop.
Gordon:  What you say there in terms of having all these numbers, metrics, what have you, the fact is it's true from a technical standpoint, too. I was at the DevOpsDays and I think it was London earlier in the year, and we were talking about metrics. We were having an OpenSpace about metrics. What do you really want to measure?
One of the participants, he made the comment that his company, they used to get this monthly report that tracked, I think it was 2,000 something metrics associated with their IT systems, and nobody ever looked at this thing.
Even if we're just talking about the technical aspect of things, much less the entire system, it includes people. It's really important to think about what matters. Maybe you won't log all that other stuff, and use Splunk, and maybe do predictive analytics, but it's not something you should be focusing your attention on, on a day‑to‑day basis.
Jen:  The win here is not the number of data points that you're logging. The win is, "If you're logging the data, is your company actually using the data that you are logging?" Because, frankly, logging data costs you money, and it costs you a lot of money. It's not for free. If you are using Splunk, you pay them based on the amount of data that you're storing.
Yeah, [laughs] just to produce a report that no one is doing anything with is not the ideal feedback loop.
Gordon:  It may have been the same event. Somebody from ‑‑ I think it was Google ‑‑ made the comment that data you don't use has a negative ROI.
Jen:  A couple years ago we were talking about that. It was something like black data or dark data, or something like that. [laughs] It was ridiculous buzzword. In any case, the most critical part about feedback loops is not just receiving the data, it's understanding what to do with it, which is what I was alluding to before.
You can be getting a bunch of information coming in, so data from somewhere is the first step. You're actually getting some sort of feedback from something, whether it be human or machine. Identifying which data you want to respond to is important. Figuring out what it means when something happens is important.
Then, somehow, emotionally tying value, and that's an individual thing. The example I used in my blog article was the fact that my husband...he's a video gamer, and he plays a game called "Dark Souls." He continues to run around in this game and die over and over and over again.
We were having a simple conversation in which he told me that he had a lot of souls, which is the monetary system in the game. I said to him, "If you die, you're going to lose all your money."
He's like, "Oh, no, no, no. That won't happen." I was like, "Sure," but I knew for sure that he was going to die again. Sure enough, a day later I heard him cursing in the basement about this. I said, "Sure enough." He died and he lost all his souls.
I looked at that situation and I thought, "Well, gosh. I'm thinking about feedback loops, and I should assign that whole thought process that I've been having on feedback loops and figure out how I might have compelled him to change his behavior in order to not be in the situation right now where he's angry in the basement."
I thought, "Well, instead of me just saying, 'you're gonna die,' I could have said something to the tune of, 'You've got a lot of money, and there's all these things that you can buy. You could upgrade your character. You could do all these things with it. What are the things that you want, that you haven't done yet? You don't have to spend it all.'"
I know he's frugal, which is probably why he's not spending it. I thought, "You can spend half of it, and just go ahead. It's OK." He could have actually spent half of it and gotten a brand new sword or something and been happier for it, but he didn't.
It's the lack of assigning some sort of emotional value to it, or emotional feeling to it that prevents people from understanding feedback, or really participating in a loop.
If you go all the way back to that early conversation about stand up and where it becomes just a status meeting for everybody, it's because no one is assigning, or at least buying into, the idea that it’s actually going to improve the situation for the team. No one really understands how it does that. No one really sees the value of participating in the activity, and, therefore, they don't.
That is the fundamental part in the feedback loop when things kind of go south. You can monitor the heck out of your computer systems, but, if you fundamentally don't care whether they're up or down, it doesn't matter. It just doesn't. That's when you can...sure, monitoring what data you should be doing on your systems, throughput, all that good stuff is really critical.
"What is the most valuable data you should be capturing?" means absolutely nothing if you don't care about the data. Caring about the data is a human quality. You can't make somebody care about it, they just have to want to.
Gordon:  Talking about teams and for the help of teams, what are some of the things as Agile Architect, Agile Coach that you really think about, that you really keep your eye on?
Jen:  The engagement of teams. I tend to pop in and out of a lot of team meetings. I keep my eye on a lot of different people. We, as an industry, have an engagement problem. We have a bunch of people who are incredibly highly paid thought workers who are probably taken advantage of a little bit.
I don't mean that in a way that we should have higher egos, but we are also ‑‑ especially in the United States ‑‑ in a situation where we'll just work ourselves to death, and we're just going to keep working. Not a lot of people pay a whole lot of attention to the boundaries that one would hope to expect from work.
Some of the simple things I look for from teams is to see whether they're going to fall down is whether or not they're aggressively answering email over the weekend. Whether or not I get into a meeting with them and they immediately dive into a heated argument or a stilted conversation,
Or do they spend maybe the first couple minutes just chatting about whatever. It could be chatting about something that is completely unrelated to what we came to talk about, or they could be excitedly talking about some technology. That's fine, too, but they're having an interaction where most of the people on the phone are actually participating. I'm basically looking for engagement. How engaged are people?
For video conferencing, are people smiling? Do they look like they're going to fall out of their chairs because they're so tired? I'm looking for that kind of stuff. Also, too, a lot of my job is just making friendships with people so that I can notice when things aren't going well.
It's not always like a science, and that's part of the problem in this, because the monitoring of computers is really easy. You can set parameters and universally use them. Regardless of what company you work at or what server you're using, it's pretty much universal, but humans are so different.
If I've got one engineer who I know is never going to smile because it's just his personality, I can tell when he's angry about things because I know him, and I know for other things to look at, but if I got an engineer that I know is smiling all the time and suddenly she's not smiling anymore, then I know there's something going on there and I can try to dig in a little bit and see if I can help her with something. It's really just about making human connection.
Gordon:  To close out this podcast, what advice would you give to our listeners to use feedback effectively in the context of people and teams?
Jen:  One of the most important things is do not participate in feedback loops just because somebody told you to. Hands down, the worst thing that you could possibly do. If you are being asked to do, maybe, your annual performance review, chances are I'm not going to tell you to stop participating in that.
That's probably something you need to do, but the other thing that you probably should consider is that, if you are already in a space where you're not going to listen to something that your boss or the person doing your performance review is going say to you, chances are you're not going to get any value from that at all.
There has to be some sort of connection you make to the information that you're getting. If you're not really participating in a, say, daily stand up because your team is already talking nonstop in other communication streams during the day, maybe you want to talk about, "Do we actually need to sync? Do we need to sync about work?"
Maybe it's not work that we have to sync about during that 15 minutes. Maybe it's just to get on the phone and say, "Good morning," to everybody, or see people's faces for a few minutes a day. Maybe it's not an enforced thing that you do just because you're told to.

Honestly, just don't do it if it's not providing value for you, but, on the other hand, make sure you understand the value that it's supposed to provide before you make any sudden decisions about it.