Episode #3 - C Chris Crowley - 2020 Security Operations Center Survey Results
Ron Gula interviews Chris Crowley about the results of the 2020 Security Operations Survey. In this hour long interview, we discuss how there are two types of SOCs and how this affects budget, staffing, technology effectiveness and overall cybersecurity.
Episode 3 Transcript
Ron Gula: [00:00:00] Hi there. It's Ron Gula with the Gula Tech Cyber Fiction show. Today we have a guest Chris Crowley. We're going to be talking about his SOC survey. Chris, how's it going? I'm doing really well. Thanks for having me. Thanks for being here. Before we get into that and want to do a couple shout outs. We had to Gula Tech.
It ventures portfolio companies get acquired. So congratulations to the team at a white ops, they were acquired by a conglomerate of Goldman Sachs Night Dragon and clear skies. So clearly they're going to be prepping for greater things in the future. And then we also had, StackRox get acquired at least announced by red hat, which is also IBM.
So congratulations to those teams. And of course those companies are going to go on to do great things. So how that, yeah. Also just want to make a quick comment on the News. We're still in the middle of solar winds and the investigations that are going on. So if you're not investing in hunting and hygiene, you should be doing that.
And I'm sure we're going to get into that today. Chris, how are you? Yeah,
Chris Crowley: [00:00:55] things are going well, been busy. Fortunately, or this is the, this is the time where there's a lot of cycles coming back around in the new year and all this stuff. And so this is usually a slow time for me, but it hasn't been so it's nice while it's
Ron Gula: [00:01:10] it's we have a lot of Career insurance in this industry.
Yeah. So before we get into the SOC survey, let's talk about your background a little bit, right? So you started out as a radio DJ, or you worked in radio stations.
Chris Crowley: [00:01:22] Started out. In computers when I was 15 years old doing Ultrix and VMs systems as an operator after high school.
So my dad worked for digital equipment corporation, that's like my, my far background, the radio thing was when I was actually in college. I DJ'ed. And then ended up on being the basically the general manager for radio station in New Orleans, at Tulane University, which was a non-commercial radio station.
And so 24 hours a day, all that stuff, jazz disco. So it's interesting. The format of WTO was unusual. Most of what they played was progressive, which was more like alternative rock has probably an easy category for that, but there are a lot of specializations WT has an amazing jazz. And on a project, something that I wanted to work on at one point, I actually fixed the catalog of the records by manually inspecting everything that was there.
One record at a time to basically verify the stuff that was in place.
Ron Gula: [00:02:25] That's excellent. I, there's definitely a trend where people who go into computer security have a music background of some sort to do. Do you do music in
Chris Crowley: [00:02:32] general? It's funny. I First instrument I'm not a good musician.
I don't play well, but like first instrument when I was a kid was piano. I played the Viola and then drums. When I was a little older, I played guitar, sing a little bit, just all sorts of things of playing around, never done it in terms of this is what I do, but I love music. And does it as a DJ for roughly 10 years.
Rough numbers. I got to listen to a lot of music and a lot of music in very broad range of scope and so W2 well with its format the DJ chose what they wanted to play very unusual for most places. So I did world music shows, jazz music shows lots of progressive, more of the outside rock stuff.
I also, a couple of times did something, they call cheese music was the modern sort of, airy Ambien kind of stuff. So lots of exposure to music and it's something that, suits my brain very
Ron Gula: [00:03:27] Actually. So how do you go from that to getting into computer security?
Chris Crowley: [00:03:30] No, it's interesting. Had always worked with computers. Yeah. The music stuff was something that I never wanted to do for work. This is like we were talking about food. I love food. I've worked in restaurants my whole life. I would never want to work in a restaurant because it's really hard work.
It's hot, it's difficult. There's not a lot of opportunity for, for making money. This is a real challenge. So I had always worked with computers. And I actually did molecular biology as an undergraduate degree and English as an undergraduate degree when I was at Tulane. And then I basically didn't want to do either of those for work.
And so I went back and did a computer information systems degree of I've always worked with computers, my whole life. It's something I've done. So let me actually formalize that and then go into that for work. And so what happened with getting into computer security was basically okay. When I was working at Tulane after I graduated from their program, 98. I started working there 2000 and the FBI showed up and named, and I'm sorry, I'm blaster and Nachi and all these other worms SQL slammer, we were just getting overrun by stuff. And so I got into computer security out of necessity because I was managing computer systems. I was managing computer networks and it was just, this is what we have to do now. Like we built an anti-spam system from nothing because we didn't need it. And so this is what happened. I had a good, strong background in that capability. I've been trained in computers just by using them. And then I was responsible for basically managing this for a lot of people.
Ron Gula: [00:05:13] That's excellent. So how do you go from that? To working at the department of energy?
Chris Crowley: [00:05:17] Yeah I was in new Orleans. And I don't know if you heard about this, but in August of 2005, August 28th, I'm not sure why I can remember that day. Hurricane Katrina hit. So I participated in a major disaster recovery operation for Tulane university.
When we moved my girlfriend and I moved around a bed. She actually is a federal government employee and I specifically, but we decided we don't really want to go back to new Orleans to go live there. At least not now. Because things are really bad. So we ended up moving to Washington DC. Now I continued to work remotely and this was in 2005, I continued to work remotely for two lane for an extended period of time, but the draw of sort of opportunities in the DC area, as there's a lot of employment opportunity here, especially in the cybersecurity space. So I actually went first to national Institute of health and I was at NCI.
And I was doing bioinformatics support. So still all in the computer side, but for bioinformatic systems. And I was just looking at the at the options that were out there. I ended up working for a group called TDI, Tetra digital integrity for on a contract that brought me to DOE. So DOE I was at office of science for a while, and then I actually went over to the OCR. OCIO, which is their office, the chief information officer running their, nationwide network is what the OCIO is responsibility is. And so basically I brought, I was brought over into that team in order to do incident response. That's phenomenal.
Ron Gula: [00:06:56] And I liked going into these kinds of background stories cause for a lot of our readers who don't under or don't readers and viewers that is, who don't know, cyber security.
It's really good to talk about these origin stories and where you went. And so you do incident response. And, but you also did a bit of vulnerability management, right? And we overlapped
Chris Crowley: [00:07:14] a little bit at one point, a lot of different stuff in the space. One of the things that I always say about myself is I am a generalist.
I have done operations. I've done incident response. I've done. I've developed vulnerability management programs. In terms of implementing for a specific organization. So we actually, this is where you and I first met was when I was at office of science. So this is something that I. In talking about my origin story, I've always embraced what needs to be done and done what was there and needed to happen. So that's basically the way that I've moved is more like a leaf in the wind. This is what's happening. Let me go in and make that work. Okay. When I went to I am 60 DOE the I actually went over there to do incident response for a little while. I actually was the program manager for the cyber program there because the person who brought me in got sick and had a serious health issue. So I was acting program manager for an extended period of time across the cyber program for that group.
Ron Gula: [00:08:19] So excellent. Now you're giving back with doing all sorts of trainings. So how did you get involved with sands? How do you get involved being and how do you transfer this knowledge and become an instructor?
Chris Crowley: [00:08:28] Also from a background perspective, I've always been good at articulating. Complex concepts, right? Just anecdotally, a friend of mine from, back when I was in university, he said to me one time when I saw him, about 10 years later, he said, you're the reason I got into medical school.
I was like, what are you talking about? And he said, I was basically failing physics and you. And a group of us, we would sit and study, but you would explain it to me. And he said, I would understand it for long enough to go take the test. He was like, but you could make me understand it. So this is just something from a conversational perspective.
It's always been something that's come natural to me in terms of. I, if I want to understand something, I work to be able to articulate it to other people.
Ron Gula: [00:09:17] So it's clear. I, there's that, there's a really horrible quote that says that those that can't, teach, but the reality is if you don't know something, you can't teach it.
So I think that's really good. So let's talk a little bit about just the current state of cybersecurity training. So how, w what do you think of things like, there's just the huge volume of courses that are out there. You can do a lot of things on, we didn't have this when we started in cybersecurity.
So how do you think it is for new people today?
Chris Crowley: [00:09:39] Yeah, honestly, I think it's great. I think that there's a phenomenal thing about the massive amount of stuff that's out there. However, I think that there's another problem that comes with that abundance. That in some cases, people will like binge eat on the abundance of cyber security stuff.
And now all of a sudden it's you don't, you can't do anything. All that you've done is read at a hundred classes and you can't actually tactically deploy anything. So that's one part of the problem. The other part of the problem is that true excellence. Expertise really comes from an individual being self-motivated in order to basically get to some position some, and it's not even positioned like a social position, but it's a position of knowledge that requires self motivation. I actually I don't want to. Go down the too many different references of other stuff that I've written, but I have an EDUCAUSE (short talk link) talk that I did, and I did that recently and I have it on YouTube. I have a very short version and I have a very long version in terms of how to create an internal team. Training program, a lot of which is self motivation. And you as a group, basically taking your capabilities and enhancing everybody else, but then also selectively pulling in information and programs and trainings from outside. And part of what this internal training is supposed to do is to get. People on the team selecting the right things from outside and, the free books and the free articles and all that stuff as part of that portfolio. But also the very specific tailored training is also part of that part.
Ron Gula: [00:11:16] That's phenomenal. We'll definitely put that talk in the show notes. The when you were talking about your background, being like a leaf in the wind and going from one thing to another that's okay. That's basically the ANSYS of cybersecurity, we really don't know where our technology is going and what policies we're going to have to be implementing here. And I think a lot of people who say, I want to learn and be a cybersecurity expert, it's, there's no real clear roadmap. So I really like your backup around there. Yeah. How do you feel?
Chris Crowley: [00:11:40] Okay. I just want to say one thing about that from a security operations perspective. I think that we've gotten to a bit more maturity and what we can do now is say we have a flexible framework. Yeah, no, that's I think where we are now. And then before we
Ron Gula: [00:11:56] move off to a different topic, let's just talk about diversity and just trying to get more people into this career field. So what kinds of things have you seen that have worked to get more women, more African-Americans more just more people.
Chris Crowley: [00:12:07] Yeah. So first of all, everybody with an individual commitment to make sense of where other people are coming from. If you can't do that personally, figure out how to do that.
That is what is going to make it, so that inclusion actually works. Okay. The other part of it is develop a system and I talk about analysis or programs of analysis related to this develop a system where people have a way to contribute on equal footing. Now, some people are going to be really good at doing a technical analysis.
Some people are not going to be very good at doing technical analysis. If you put those people in a room. And you don't have any structure where there's equal contribution that people who are strong, who want to have their voice heard, are going to crush everybody else in the room. And I talk about this as there being an alpha analyst on every team and it doesn't matter, race, gender, religion, whatever, if somebody wants to be heard and they insist on overpowering everybody else in the room, they will do it because they've developed that.
Capability personally, if you require that all those people show their work individually contribute their work into basically a system where there's waiting. And I, as an example of this, I use analysis of competing hypothesis by Howers as a really good example of how this might work. If I can actually construct structure where you need to show me this and everybody on the team does this. And then we actually look at that in aggregate. And that's how we do analysis as a team. Then all of a sudden it matters a lot less, what color hair you have or whatever, in terms of making that be a thing for the analysis. Now, all of a sudden it's the work and this is, I don't want to segue to it too soon, but this is what I tried to do with this SOC survey. The SOC survey the reports. Easy to read. I'll put that in air quotes. If you understand that the concepts it's easy to read. The work that I did to get there is very complicated on the backend. If you go read the Jupiter notebook that I shared as part of this people who understand Python are going to be like, what is this dude do? So even being able to actually develop some way of sharing and communication, communicating the work is really hard. It's a really hard thing to do and doing it individually is one thing, doing it amongst a group of people with different backgrounds who don't really share a strategy for analysis is near impossible.
Unless those individuals, a grade to basically conform to some system, some.
Ron Gula: [00:14:51] Method of that's well said, so two comments, then we'll get into incident response in the SOC survey here. The process that you're describing about having people being giving their thoughts on unhinged and not letting that alpha person, alpha Malika person there, there's a company we invested in called retrim that does this for the agile software development process. And I've been arguing that they really need to get into these other processes because there's a lot of different decision-making that goes on like this. And then the second thing is just, what do you think of our data care concept, where we're trying to just really make it such that you don't have to be an expert to start in cybersecurity. You can really be, entry-level and empowered.
Chris Crowley: [00:15:28] I think that's important. I think that the only way that as a community, when I say that I'm really thinking globally, Like the global community, the only way that we will be able to use data effectively in a way that both accomplishes the objective of using that data and in a way that doesn't have it end up being something that's loaded over of us, lorded over us by governments and by companies, because that's a big risk that's privacy.
The is a huge risk. And then that's also extraction of information from individuals that is then used against them for whatever nefarious purposes some very powerful organization wants to have. So it's that depends on. A very broad scale of people understanding how these things work and how they can work well and how they can be used to abuse people. And that's a very scary thing to me, honestly.
Ron Gula: [00:16:30] Got it. Got it. All right. Let's transition to talk about actually detecting these bad guys. And so what do you think the current state of incident response intrusion detection. That, that whole incident management industries what how are we doing as an industry there?
Chris Crowley: [00:16:44] We're getting a lot better in the last. 20 years, which I actually have a perspective on it for the last 20 years. In the last 20 years as an industry, as a individuals, we have gotten incredibly better. You probably remember blaster and SQL slammer and like all of those sorts of worms, just destroying networks.
There weren't firewalls, there wasn't compartmentalization, things were open. There were default passwords, all that stuff was easy. Okay. So that's what I would cite as an example of how we've gotten better. There's better information sharing. It's not perfect, actually. It's not even good yet, but there's better information sharing.
And so case in point recent ones, solar winds the way that most people learn that they were actually affected by the solar winds is third-party notification. Third-party notification, there was one company, right? FireEye who found it? How did fire? I find it, they were watching their network. If there were a hundred companies who were watching their network with the competence of FireEye, the rest of the internet, the rest of the internet would be much safer. And think about cybersecurity in the way that I think about material science. We have very advanced material science because we've been doing it for thousands of years. We've been doing cybersecurity for decades. So millennium scale versus decade scale. The fortunate thing that we have for cybersecurity is that we're actually able to make advances in cycles much faster than we're able to make advances in materials.
Like I can't rebuild this house very easily, but I can rebuild that computer system with a completely new operating system and applications. Relatively easily.
Ron Gula: [00:18:34] How about things like the MITRE attack framework? Is that giving some rigor to defenses?
Chris Crowley: [00:18:39] Yeah, that's an amazing thing. So I actually I love the MITRE attack framework.
I think it's an elegant display of a matrix thinking example. Yeah. And I really like all the stuff that people are doing and I applaud MITRE for doing that work and releasing it. MITRE is a non-profit entity and they basically said we are, and they've done so much amazing work, CVS, CWA, all the background stuff, they basically said, we think this is what people need. And shared it and then allowed derivative works to basically cascade from that. And it's great.
Ron Gula: [00:19:15] So as part of this information sharing and getting more rigor, we have the SOC survey. So what is the SOC survey? And what's been your history with
Chris Crowley: [00:19:22] great. So I did a Sans Institute SOC survey in 2017, 2018, 2019.
The, this, the surveys that sands does are vendor sponsored surveys. Okay. Business for everybody has been a little weird, right? 2020. Basically what the analyst program, which is the portion of Sans, which deals with this what their take on it was, things don't really seem like they change very much 17, 18, 19, not a lot of change.
So we don't think we're going to do the 2020 survey. And I said that kind of sucks. I. I like this is a thing that I do. I think it's a very useful and powerful thing. So I'm going to do it anyway by myself. And I say stuff like this pretty frequently. I just say, Hey, I'm going to do this.
And then I go do so what I didn't want to have in terms of trending was we've got 17 data, 18 data, 19 data. 21 days. So I didn't want to have that gap. So I filled the gap personally. I saw, I think
Ron Gula: [00:20:25] he deserved a lot of credit and leadership, not only for doing it, but then the perseverance of getting it done.
Chris Crowley: [00:20:31] Right . So yeah, not easy. It was not easy. And so it started in June maybe a little before that and from a thought execution in June in terms of getting the survey out there, getting people to take it only a hundred people roughly took it this year. Past years it's been 500, 400 kind of a scale.
I basically attributed that to the fact that I just don't have the marketing reach that sands Institute has. I really don't. And so it's interesting that. In spite of the lower numbers. I still don't think that the numbers are anywhere relative to the scale of security operations centers in the world.
Actually don't have that number. I'd be interested if anybody has that number.
Ron Gula: [00:21:09] That's it interesting. So let's have our operation center go and put the 20, 20 SOC survey up here on the monitor. Excellent. So it's called a tale of two SOCs, right? So that's very poetic and very nice. W how'd you come up with that title?
What are some of the basics?
Chris Crowley: [00:21:24] 2020 has been a very tough year for a lot of people. Okay. This a tale of two cities is basically a A book. I was forced to memorize the first paragraph and then regurgitate it when I was in high school. Oh, you're recording
Ron Gula: [00:21:38] Star Trek 2
Chris Crowley: [00:21:39] right. The car, Charles Dickens. So this is a, this is basically a thing where I personally feel like I'm in a position where I'm very much insulated from the damage that 2020 has done to a lot of people. And a lot of us are lucky. I see that. And
Ron Gula: [00:21:58] we're also been very proactive as an industry.
Chris Crowley: [00:22:00] So I've been very happy about that.
Yeah. And I've worked hard my entire life. I've worked hard my entire life. There's no doubt about that. And the reality is though that, like I personally see I've got stuff, I've got food, I've got a house to live in. There are people who don't have that. And not just in America, but all over the world, there are people who don't have that and they were living in house to work in.
Yeah, exactly. And so this is what I'm seeing in 2020. It's wow, there is this massive dichotomy. And then part of why I came up with this also is in this survey responses, the very first thing, if you go to that very first chart, there is this there is this. Thing that popped out at me almost equal numbers.
33 31 to management pays close attention to the needs of SOC. But right down the middle management does not pay any mind to this. Now, these people saying management does not pay any mind to this. They could actually be completely wrong. However, this is what they perceive. This is what we get in the survey response.
The perception from the people who are answering on behalf of the security operation centers for their organization. And they're not representing that. This is just, Chris, Ron saying, this is what I say. And this person says this and that person says that, and it's a dichotomous opposite.
And it's wow, that is astounding. And so I wanted to see if that actually played out in the rest of the responses. Cause there are a lot of ways that you can pivot through. Any sort of response data. And so that's what I then did from this. And I actually talk about this in a couple of later sections where I say, okay, here's a bunch of boring stuff, right?
Here's a bunch of demographic stuff, but also if I take that and say this is what Chris sees in this, does that actually play out still? If I pivot from that to other responses based on that split, because I have in the spreadsheet, this data row says this person responded management cares, and this person says management doesn't care.
So then I can go and pivot to see what those responses look like for other answers and a couple minor comments.
Ron Gula: [00:24:17] So one. When you download this. So it's at SOC-survey.com. You can click and download it. I w one of the first tweets about this after it came out was, Hey, at least you're not collecting my personal data.
So that was good. Second thing is you have the Jupiter notebooks and that's a huge thing. It's very popular and incident response builds, but it's also getting more and more popular for research. So I'm really happy that you're
Chris Crowley: [00:24:38] sharing. Yeah. Let me mention one thing about the Jupiter notebook specifically.
I have programmed a lot in my career. I don't really program in Python. Now I want to program in Python. I, and by the way, like I've taught programming classes before, so I'm a very strong programmer, but Python was something where I wanted a project to learn Python. And so this is my, the Jupiter notebook is me sharing.
And if you look at it, if you understand Python and you look at it, you will see in the early parts, the early entries that I have, I left all my sloppy work in there. Those were all mistakes that I made personally. And yeah. You asked me about training, take any problem. It doesn't matter how small it is.
Take a problem. And so solve it in a programming language. And so Jupiter notebook is my way of sharing them. Cool. All right.
Ron Gula: [00:25:32] So the first big finding was that not everybody worked from home, it was almost a foregone conclusion that 20, 20, we all worked from home, but you had a small group of people that didn't work from home.
Chris Crowley: [00:25:41] What's up with that. I. I find that fascinating. I don't, I didn't do the follow on conversation of Hey, you answered this. What were you doing? I actually suspect that some people have a mature enough facility that they can actually continue to send people there and they already had planning.
In place for dealing with this sort of pandemic circumstance. And again, going back to my disaster recovery, 2005, hurricane Katrina, we didn't have that in place. I'm looking at, I was in Japan in March of 2020. So timeframe of March of 2020, I was in Japan. I went to Japan knowing that the COVID was an issue there at that time.
It wasn't yet an issue in America, least not a big issue. And when that happened, the capability for the facility that I was in to actually maintain distance was actually very good now. And I don't want a good filtration, good clean room. Yeah, exactly. So I don't want to digress too far into this, but it cultural capability for management of environment within within buildings.
And so some people would want to ask the question how does Japan do that? In Tokyo, Tokyo specifically, it is a very dense. Densely populated in with this for years, they've been dealing with this for a long time, since the SARS outbreak. And so in the SARS outbreak, when they dealt with it, then they basically have the capability in place to deal with it.
Now what they didn't have was. And this is anecdotally from me talking to people. They actually didn't have a work from home strategy. So some of the people that I know that work in Tokyo had a very difficult time with their companies adjusting, because it was very much of a you come to the office, the idea of you not being in the office was a big problem.
So they had difficulty with that. But other companies who basically had decided, what we can work from anywhere. That's fine, but the companies who decided it really matters, you must come to the office, maybe data sensitivity. If we're doing classified network operational monitoring, I can't send that.
Data to somebody's home. So if the organization understands its needs of having onsite staff and understands the reality of what that might look like in the circumstances of a pandemic, then that's where you get. That is my suspicion.
Ron Gula: [00:28:17] Yeah. That's very good. So then what constitutes having a SOC? Is it the vulnerability people I can remember more than once go into like the SOC and then being escorted to an office for the full management team.
I've seen places where it's. Together, I've seen boards Birdsville. And if you go to these commercial, MSSP is you have the smoke glass, electronic showrooms. Beautiful.
Chris Crowley: [00:28:42] counseling. So I think of a security operations center. The way that I actually think of it is in terms of functions. Can you do these things? So my functions. The stuff that I defined for this, and this is not necessarily what is responded in the survey, but the way that I think of this is you have some sort of a command center, some sort of customer interface layer, illusion of control.
Yeah, exactly. And also illusion of control to present to your constituents and keeping your internal SOC capability on track and basically providing Pia mater that you know, that barrier. Then you have monitoring capability, threat intelligence capability. You have responsive capability. You have the ability to basically self-assess to see the status of the systems.
And you also have a capability to do forensics. Now, some of these things might be outsourced and that's fine. But these functions are basically what I think of as a SOC. And so I talked about this in the survey by basically asking the question of what capabilities do you have. And so the thing that I would suggest is it doesn't matter if it's insource or outsource, but the things that people actually talk about them doing, if they're responding to a SOC survey, then that becomes what the SOC capabilities are.
Ron Gula: [00:30:01] And almost everybody I've dealt with has some sort of outsource. And even if they're hearing, a feed from a vendor or a lot of products these days are really SAS products and you're outsourcing function in that space. So what type of outsourcing did you
Chris Crowley: [00:30:15] So pen testing and all of its various Formulations red teaming, whatever, and then threat intelligence and digital forensics.
These are really the ones that are most commonly outsourced. And my opinion on that is that it's it's a capability that's so expensive and so specialized that it's very difficult to actually develop this in-house if you choose to. And then it's especially difficult to maintain the staff with high caliber capability on this.
Because these tend to be experts
Ron Gula: [00:30:48] any the red team being outsourced versus managed firewalls and stuff like that. How, why do you think that was the most popular
Chris Crowley: [00:30:55] one? I think that's a popular one because the maturity level of the vendors in that space is actually high.
And so it's an easy thing to see. That there's a, that there's basically a circumstance of being able to to do them.
Ron Gula: [00:31:13] Excellent. Excellent. So I think the next section we talk about I wasn't sure if we wanted to go through the whole doc, is this a good pace? All right. So let's talk about funding then.
A lot of the folks. Said that they didn't know what their funding was. And I was expecting that to correlate with my management. Doesn't know what I'm doing.
Chris Crowley: [00:31:28] What was your, what were your thoughts on that? It's interesting. I'm not sure that's actually the case for that. I didn't do that specific.
Pivot necessarily. My take on this was, if you didn't know the budget, you probably weren't the manager. What was interesting. And I've actually reached out to a lot of people around the survey and said, Hey, here's what I'm working on. This is what I'm doing. And. One of the things that actually came from that was, I had somebody say, I actually am the overall SOC manager and I do not know the SOC budget.
And here's why. And it was interesting from that perspective, in terms of in terms of it being a I don't know it because that's not what I see and my management. Knows that even though I am the overall SOC manager.
Ron Gula: [00:32:15] Excellent. Excellent. So then I thought this the metrics section was interesting because most modern business processes, if you care about it, you actually have metrics to measure it.
So the fact that some of your respondents said, nobody cares about our metrics,
Chris Crowley: [00:32:31] yeah. It's concerning to me, but here's another thing is that I think. That the metrics which actually get reported oftentimes are little more than a, Hey, we reported something. And if you look at what the most common metric that was reported is it's account of how many we handled it.
Ron Gula: [00:32:49] 25 million events today. Yeah, exactly. For meaningless.
Chris Crowley: [00:32:53] Yeah. It is a show of something which has done, but it's not really a good driver of. We're providing value. And so I think that I think that another thing is that there's a lot of difficulty in actually aggregating these metrics. And one of the other answers was how do you make metrics?
Are you able to automate this? And is it just clear and straightforward? And in many cases it isn't for people automated, so they just don't report it.
Ron Gula: [00:33:22] So we get a lot of pitches from companies in this space. And they want to bring metrics. And one is like the visibility it's just because you have 25 million events a day, doesn't mean that half of your CrowdStrike agents are misconfigured and not reporting.
Or get all these Palo Alto alerts. What do you think of things like Dimitri's, one 10 60, where you have a minute to really identify an endpoint 10 minutes to verify, and then 60 minutes to get rid of true compromisers in
Chris Crowley: [00:33:46] implants. Time-based objectives. I think of that in terms of service level objectives.
I think that's a good, I think that's a good strategy if we have a time-based objective, but what's your corresponding quality measure that assures that you're actually doing that well. And so anytime that I see a time-based metric, I say, can I actually also measure quality because I would rather go slow.
On a small number of things and do really good quality on that and be able to have confidence in my quality level before I try to compel my team to go fast.
Ron Gula: [00:34:22] Yeah. So this was one of the reasons we invested in the site. So it really does the amazing, breach, adversarial, simulation.
What do you think of tools like that? And, is it just more work for red teams or is it more. More ways to test these
Chris Crowley: [00:34:35] time-based metrics. Yeah. So here's what I think about, sorry, a simulation adversary emulation and the implementation of that is I would rather have a test case at a smaller level for my use cases for the detections, because I see adversary emulation.
In the active network as an advanced capability, but adversary emulation is what we want to get to in terms of confidence for performance of our detections. So long as I'm actually building that throughout my system. Then I think that actually works really well from the perspective of, should we be doing this?
Yes. Is that where you start now?
Ron Gula: [00:35:20] Let's talk about staff, which is the next section. So the stereotype is I hire a bunch of low level. Entry-level whatever you want to call out. I try not to use a negative term here, but yeah. Entry-level folks to handle the huge amount of alerts and maybe not as many level, two level three folks.
So what did the survey show along? Those,
Chris Crowley: [00:35:39] those lines? Yeah, so it's interesting. The size tends to be two to 10, so it, And this is an overwhelming volume of of people who responded this. You tend to have two to 10 people as such, which are probably gonna end up with is people who are more generally capable and not necessarily specialized.
Again, some of the some of the respondents said, our organization thinks that it's okay to just hire people, to stare at alerts and that's somehow going to solve the problem. So that's a problematic stance in my opinion. I think that if I only have two to 10 people, I would prefer to be able to have.
Maybe three to five, highly specialized people who are then managing our outsource capabilities, but usually it's budget reality that limits that ability. And so if I can't afford five super high powered expert level people where everybody is a level three, and I actually don't even talk about it in terms of level one, two, three, I just say junior, senior.
And the way that I differentiate junior from senior is a junior level person could not independently arrive at a solution. I would not be able to, I would not expect that person to independently weigh all the various factors of organizational requirements and technical details, a senior level person.
I expect that individual to be able to weigh. Of the environmental and organizational factors that would have them come to a decision and I would empower them to make those decisions.
Ron Gula: [00:37:16] Political. I've seen a lot of SOC operators. Of course, we don't want to report
Chris Crowley: [00:37:20] this metric organizational. It makes her look bad, or this is why I get fired from every job that I ever worked is I'm happy to say what I think to people about the metrics.
Actually I literally almost got fired from a job. The very first meeting I was in. Because somebody said to me now that you're here, I suspect this is going to happen. And I said, honestly, that's actually not at all. The thing that you just told me is completely contrary to what we're trying to do.
And it was specifically around the count of incidents. Specifically around the count of incidents. It was, we expect the number of incidents to go to zero. And I said, look, I can make the number of incidents, security incidents go to zero. That's not what you want there really isn't. You want to have the right.
Number of incidents. That's not zero necessarily, unless you truly have no incidents. And if you truly have no incidents than our SIM based detections and our threat hunting would corroborate the fact that there are no security incidents in place. So if that's what we have, then let's do it.
But if that's not what we have, then zero, isn't the right number, then talking
Ron Gula: [00:38:33] about metrics and how to measure security for a long time. And we
Chris Crowley: [00:38:36] still have a ways still the way to go. I've talked a lot. So
Ron Gula: [00:38:40] one thing I skipped over here was this retention, so the stereotype is that the SOC operators are overworked.
They've got bad about tools they're constantly, doing and yet there's. All these new companies doing soar. And we'll talk about technologies here in a little bit, we got sore, you've got AI, you've got like the things that can do enrichment. Like our, one of our investments isn't in polarity.
It doesn't matter what you use. You can overlap whatever data you want, whether it's from your threat intelligence, providers, assets, stuff like that. So do you see a role in technology making the SOC operators lives easier?
Chris Crowley: [00:39:13] I see an opportunity for technology to make. People running SOCs far more effective.
I really do. And it is truly the blending of procedures, people in technology that is going to drive competence and then eventually drive Exxon. Excellent. Excellent.
Ron Gula: [00:39:33] All right. So I think we talked about. We talked about this a little bit here. Any other insights on this particular slide here
Chris Crowley: [00:39:42] in terms of, yeah.
In terms of the challenges that best describe the hindrance to your SOC. I actually, one of the things in the hypothesis that I use and that sort of a sort of tale of two SOCs analogy that I'm trying to use is this lack of skilled staff, actually in the people who say that they have management support.
That is no longer number one, in terms of the biggest challenge, implementing soar is the biggest challenge for the people who say that they have management support who care about having skilled staff. So that means that they've, in my opinion, that they've very much moved on from the look, we're just trying to get people who know what they're doing in here.
They've moved on to the, okay, let's move to another part of that triangle. It's let's make the technology work.
Ron Gula: [00:40:32] The, hats off to the SOAR vendors who have to integrate with the thousands of cyber products out there. Yeah. But hats off to the people who actually have to install it, manage it, do the rules.
That's an ongoing, it's an ongoing thing. But
Chris Crowley: [00:40:44] Talk about this in terms of glue. What is the glue that holds everything together in your SOC. And this is just a term that I use, but the soar technology does not need to be the glue. There are other approaches to that glue, but what they're looking at from a soar technology is a thing which is agnostic, right?
Which is able to interact across a large number of different technologies and take the data and basically do that data transformation to stick stitch this stuff
Ron Gula: [00:41:14] together. Awesome. So now we're getting into part of the survey where you really go into your hypothesis is for why we've got these two different, your first one, you touched on this a little bit before that management support equates to more funding and that's not necessarily true,
Chris Crowley: [00:41:31] right?
Okay. That is not what the responses say. I don't know. I don't want to say whether it's true or not, because I don't want to extrapolate from these responses to the larger population cause, and this is something also that I feel is a very important thing. When looking at survey responses, does that extrapolate to the larger population?
Yes. In this case? I don't think so. I'm the person that wrote this. I don't think that this extrapolates to, to the larger population, because I can't show that extrapolation. And it's not
Ron Gula: [00:42:01] exactly a leading question either. And so well,
Chris Crowley: [00:42:04] and so this is what I was trying to do is say does this make sense?
And when I looked at this and there are a couple of these charts, there are like eight or nine plots. So it's hard to see this. And I would be interested if someone sees something different than what I actually show in this. But from what I see, I can't see any clear indication that you get more money.
At any size. And I separated this based on employee count size filtration so that I'm comparing similar sized organizations and in similar sized organizations, it looks like. The budget doesn't correlate strongly to. Yeah. My, my management cares about skilled people. And so one sort of takeaway that I have for this is that a management team, which actually does truly care about this.
I see this as a management team, that's looking to extract maximum value. That's a way that you can think about this. They're going to spend about the same as their peers, but they're going to make sure that spend really counts.
Ron Gula: [00:43:14] That's excellent. So when I was a CEO of Tenable, one of the things we would use to qualify vulnerability management teams was, do they use credentialed scans or uncredentialed scans because needing a password audit a potential security issue required a lot of management approval, right?
We have another hypothesis where management support. At least the technology that the SOC users that there's some high satisfaction and this wasn't necessarily the case either.
Chris Crowley: [00:43:40] This one, actually it was, I think it was more from the from the responses, it seems to me, so I had two different two different charts.
Basically this is a chart of how many. A's were given. So the way that I tr I tried to look at this as how many A's were given by the people who self assessed that they were in this skilled category versus how many A's were given by the people who self assessed that they were in the unskilled category.
And it's interesting, the people who self assessed as being in the skilled category gave a lot more A's for the top five technologies. I'm skilled. And
Ron Gula: [00:44:16] I use stuff that I like. Yeah which is something I personally see a lot of 10 to skill people who buy a technology, they come vetted with that technology.
Yeah. You're not going to replace Splunk by somebody who loves Splunk and selected Splunk and deployed Splunk. Even if you gave them a free thing. That was a hundred times more powerful.
Chris Crowley: [00:44:33] So I'm not sure. I'm not sure if I gave somebody a free thing that was a hundred times more powerful and they were able to see it.
I think that they dropped what they had. And I think that somebody who's skillful. Could actually show the organization. This thing is a hundred times more powerful and a hundred
Ron Gula: [00:44:53] times, I suppose I misspoke. Yeah. Weiss's powerful. Powerful.
Chris Crowley: [00:44:56] No, I
Ron Gula: [00:44:57] don't think certainly 10% better. 10% cheaper. Not enough to get, I agree with you.
Chris Crowley: [00:45:02] I agree with that. Yeah. Yeah. And so it's a magnitude, it's a magnitude at that at that case, but I have said multiple times previously, I would rather a team. You of people who are motivated to work, then a whole bunch of expensive tools. And to me, this sense of we're skilled, we're capable of the management supports us.
We do what we want to do. We have the tools that we need. I think that empowerment, that notion also goes over into the technology. These folks are going to make it work, they're going to make it work and they're not going to, and to be in a position to blame the technology. They're not going to be in a position to blame the management.
And for the
Ron Gula: [00:45:42] folks who are unskilled and unsatisfied, I really wonder maybe in a future survey you could ask. What percentage of the features of the products you have, are you using? And then when new features come out, are you using them? We would ship new capabilities all the time, a ton of, and a lot of our portfolio shipped new features, but they don't get adopted because they were bought and deployed for one use case.
And one use case
Chris Crowley: [00:46:04] only. Yes. And that is a flexibility issue on behalf of your customer. I think it's also probably a bit of a training and deployment issue on behalf of the vendors or the vendors hear from a customer here. Here's the change we'd like to make, but then they don't actually see how that works into the ecosystem, across all their customers.
Ron Gula: [00:46:25] And then lastly, you talk about, is this. Real or perception. I think there's some perception in here.
Chris Crowley: [00:46:31] What do you think? Yeah. So this is something that for me, I see this and something that I'm challenging people to do is say, do you see that too? Because from a, from an analyst perspective, one of the things that I always look for is peer review.
Do you also see this right? Does the stuff that I saw in the analysis that I performed actually. Show up when you look at that analysis too. And this is something that say I leave this as an exercise for the reader. It would be really hard for somebody to just jump into my Jupiter notebook and verify that.
And this is one of the problems that I'm actually identifying with surveys generally, is people love the charts. People will take the chart without actually going and saying, is this person crazy? Does this person actually know what they're talking about? Does the. Response set, which is present. Actually show that or did somebody just mess with the numbers to make it show that, okay, excellent.
Ron Gula: [00:47:27] One. I'm stepping through a lot of chapters, that are “boring”, stuck here, but I want to encourage people to download this and reference it, download the other there. The other ones are at sands, right?
Chris Crowley: [00:47:36] The other, yeah. Yeah. So 17, 18 and 19. Our sense something that's awesome. And I actually have this, I've discussed this with the analyst program.
We're going to release all the old data. Oh, as well. That's really good. In 2021, my plan is to go and do some trending based correlation, 17, 18, 1920. The problem with that for for me for doing that work is the questions aren't always exactly the same. And this is something that I was screaming and yelling about, in 18.
Again, I'm happy to share my opinion about stuff and sometimes people don't like to hear that. And one of the things I basically said an 18, yeah. Was we have to freeze the questions. If you want annual trending, you have to freeze the questions. Because I can't do trending year over year. If I have different questions, like I'm not even a survey design, but if you go look at medical surveys that actually do this, they study populations for decades. This is if you want to have medical science level investigation of how to run security operations, that's what you're signing on. For five, 10, 15 years. And it's really tough in a cycle environment where the operating system changes every six months where the vendors come and go every couple of years.
Ron Gula: [00:48:51] So one of the boring things I found really interesting, the technologies that got the course. So let's just talk about each of these really briefly. So full P cap. Why do you think people are giving apps to full peak cap as like the worst thing on the list of technologies?
Chris Crowley: [00:49:02] Yeah I think that.
And again, this is my opinion because it doesn't show it in the responses. My opinion, first of all, full peak app is easy to deploy and then you can't really do anything with it. So it's very expensive. It's very easy to deploy. And then all of a sudden the analyst is like, how do I use this? Okay.
Ron Gula: [00:49:21] I have to be an expert in every protocol I've collected.
Chris Crowley: [00:49:23] Right, so then all that full P cap does is it provides a video recording type equivalent of the cameras are there. Now I have to do the analysis. So people don't like tools that say here's all the stuff you do. The analysis,
Ron Gula: [00:49:42] because one of the first sort of forensics features we put into the dragon IDs was when you had an event.
You could collect any number of packets after that event and you could still get a 10 cap out of
Chris Crowley: [00:49:53] that. Yeah, that's good stuff. It's good. It's great. Let me add one other thing is full P cap also has a lot of visibility in visibility problems and I'll just cite TLS intercept as an issue. Now, all of a sudden, like I have all this stuff and I have the data.
But I actually can't do anything with all this data. So another problem. Absolutely. So
Ron Gula: [00:50:13] then network deception. So honeypots honey tokens honey accounts, all that kind of stuff.
Chris Crowley: [00:50:18] Nobody likes it. Yeah. People, some people like it. But a lot of people don't like it. And so what I think about that is.
People looking for something to drop into their network and fix problems. And they haven't fixed all the other problems first because network-based deception. Any deception stuff is really something that you deploy effectively. You need good threat intelligence to actually deploy deception.
Ron Gula: [00:50:46] We should talk about that in another time.
So application control, this is the white listing. These are things are hard to do, right? Super hard to make a list of the apps that I only want to open certain data, you
Chris Crowley: [00:50:56] know? Interesting. It'll take you 10 minutes on a windows 10 enterprise thing to build that it doesn't work. Once you build it though.
So again, this is the fastest I know anybody has done it six months. More, more like a year, one year. Are you willing to sign on with your team for a one year management and cultivation project? I am. Like I can make a team do that, but most people don't understand. That's what you're signing on for the other thing that you're signing on for when you do that is now you need to integrate with all of the rest of the it operations in order to have the appropriate sort of hygiene across all of their processes.
You ever go and try to change somebody else's work process. That's what you're signing on for. It's exciting. I personally, I'm actually excited by that. That's something that I actually really but I don't change it. I don't even provide recommendations for changing it until I've studied it.
Ron Gula: [00:51:57] Yeah. One of the reasons we invested in secure circle, which is a desktop encryption is really a alternative to CASBY in many ways, but you can do process to data. So you can say only these processes can open up PDFs or word documents and really break that down. And, but it's a little bit more flexible than a traditional, app application white listing thing.
All so AI, and then we'll, we're going to talk about the doozy of a TLS intercept. So AI is big disappointment, right?
Chris Crowley: [00:52:21] Big disappointment small deployment. The people. So if you later, a different chart shows you relative to the other things. So not a lot of people have actually deployed it and people don't like it.
Okay. AI is intended to be the expert it's intended to be the analyst replacement. It's. Studied expert systems a couple of decades ago I had a class on it and it talked about decision support. This is good in a very small circumstance, and you have to realize that for your AI system, You need a tremendous amount of training.
I've been trained for 47 years. I still we'll make a lot of mistakes. So training for your AI systems, it can learn faster. It can actually respond faster than the human brain. It cannot synthesize as well yet. If we keep deploying this, eventually it will get better. And that's the promise. And frankly, absolutely frightening thing about AI is if we keep deploying this, it will just keep getting better.
So I don't
Ron Gula: [00:53:37] disagree. We've been pitched a lot of AI cybersecurity solutions. And they are typically, machine learning to look for anomalies. And the assumption is that change is an anomaly and that all anomalies are somehow bad or that somehow all threat actors are going to cause an anomaly.
And I have an issue with that. I'm starting to see people do AI for like SOC augmentation. And my issue with that is you can't prove. What your procedures are, so it's it gets interesting.
Chris Crowley: [00:54:03] I don't think it needs a proof of procedures. What I think it needs is development.
And. Again, this is my opinion. This is the way that I tell people to develop their security operation centers. What you need to embrace is constant change, flexibility and the idea that you are going to encounter things that you're not ready to encounter, right? And you want to go with the star Trek analogy all day you're sending a ship out.
In advance and you're equipping that ship with a multitude of capabilities. And you are interesting that ship will do the best that it can do in order to accomplish your objectives. That's what you're doing with the SOC, least a SOC. And I basically break SOCs into two big categories.
One is the, you're just a compliance SOC and the other is you actually want to try to do security. And if you're just a compliance SOC, don't necessarily agree with that, but that may be what you are, do that like your way back from that, your way back from the point, just do your compliance and just do that.
Cause that's what you're funded for. Don't strive beyond that, but then compel the organization to change that. Sorry. All right. That's great.
Ron Gula: [00:55:14] So last technology, the Galata app. So the net TLS intercept. So I got https://SOC-survey.com up there. You're talking about getting in between that and seeing the actual data that's moving by and people aren't happy with it, even though what we got Gigamon PA Palo Alto is a whole list of applications that they can decrypt.
And on the fly, like
Chris Crowley: [00:55:32] before there was a lot of opportunity for it. And I think the simply comes down to a deployment issue and acknowledging the fact that your network perimeter doesn't exist like that anymore. And I think that the network TLS offer is basically when you have a network perimeter where you can assert that you control the entirety of the end points on that network.
Then you can do network based TLS.
Ron Gula: [00:56:02] I definitely think there's opportunities with Zscaler and Netskope deployments to do this. So it's interesting. All right, let's go back to the widescreen and we'll wrap up. So you have your own consultancy. How do people get ahold of you? What are you hoping for next year?
What can you leave with our
Chris Crowley: [00:56:16] our viewers? Yeah, so it's interesting. I have my own consultancy and it's just me. And so it's I basically do everything. Why I'm, because that's the way that I've set it up. And it's the way that I want to do it because I have a very serious problem of scaling.
Like I can't scale with just one person. I basically have like three things that I offer. I've got a very cheap $35. Cheap. No, it's cheap. It's cheap because I'm selling it for less than it's worth $35 Gantt chart, which is basically the entirety of my methodology of building a SOC start to finish.
So you want it, you want
Ron Gula: [00:56:52] to engage Sox
Chris Crowley: [00:56:52] as a business. Yeah, exactly. And so it's I also do maturity assessments and then I have my soccer class that I talked to him. People want to talk to me. LinkedIn. Christopher Crowley, pretty easy to find them connected to you. I've got red hair, every photo you'll see of me is different.
So at least the public ones. So it's every profile photos, a little different. So sometimes people are like expecting the same one. It's never the same. So there's that. And then Twitter, it's @CCrowMontance. M O N T A N C E and Montace is not a word. I made it up. I wanted something that was eight characters.
Ron Gula: [00:57:33] Sounds like a place in California. It
Chris Crowley: [00:57:34] has so many different things that it sounds like I literally, I spent days doing research to find something that sounded like a word, but did not exist in the English language yet. M O N T A N C E . And you have the domain
Ron Gula: [00:57:51] for it too. Of course I do. That's
Chris Crowley: [00:57:53] awesome. I wouldn't make something up and then not have the domain for it.
Chris, thank you very much for coming on. I really appreciate you inviting me and assessment.
Ron Gula: [00:58:02] Getting the next stock survey for 2021 Dawn. What happens? Good luck getting this out, and I'm sure you're gonna get a lot of feedback from this. So thanks. I
Chris Crowley: [00:58:10] really want to engage people on this. And so if people have something to say about the SOC survey, please tell me good or bad.
We're just, Hey, I don't even know what you're talking about. Can you share some more information on that?
Ron Gula: [00:58:20] Awesome. Thank you for everybody who's watching. And this is episode three of the Google Tech Cyber Fiction show. For more information, you can visit us at Gula dot T E C H.