HS136 – Sponsored Bluecat

[John Burke] (0:00 – 0:05)

Hi, I’m John Burke, CTO of Nemertes, here with my co-host.

[Scott Robohn] (0:05 – 0:14)

 I’m Scott Robohn, CEO of Solutional and host of Total Network Operations, a sibling show here on Packet Pushers.

[John Burke] (0:16 – 0:28)

And you’re listening to Heavy Strategy, the show that tries to ask the right questions, not give the right answers. Joining us today is Andrew Wertkin, Chief Strategy Officer at BlueCat Networks. Andrew, thanks for joining us.

[Andrew Wertkin] (0:28 – 0:30)

Thank you for having me. I’m looking forward to it.

[John Burke] (0:30 – 0:52)

And on today’s show, we’re going to be talking about just what a network management company is going to talk about, the challenges of using AI for enterprise network automation, because we all know IT folks are bending AI to the task of speeding up network automation efforts. Why isn’t that as simple as it sounds? Why can’t we just take Claude’s word for it 99% of the time?

[Andrew Wertkin] (0:54 – 2:08)

Yeah, just put it in a permissive mode and just click go. Done. Yeah, you know, it’s interesting because, you know, I’ve been in, well, I’ve been developing software since I can remember.  You know, there’s a reason I can’t write. I’ve been typing my entire life. And I’ve enjoyed the craft very, very much. But you know, it’s been interesting on the network side because the teams have been doing a really good job. And by the teams, I just mean network engineers out there driving automation, like, you know, pre-code generation, pre-LM, and really trying to sort of, you know, mature the methodology of developing software akin to how software development companies develop software. And with AI and with LM-driven or generated code, without those best practices, it’s kind of scary to think about what might happen on the network. So, you know, it comes down to probably a further transformation in those teams to sort of more sophisticated processes around software development, because the faster stuff can be created, the more stuff can break, probably.

[Scott Robohn] (2:09 – 3:24)

You have a thesis that really resonates with me, right? I would say, you know, one of the other hats I wear is a co-founder of an organization called the Network Automation Forum. And as of recording today, I’m coming off our last meeting, AutoCon 5 in Munich, Germany.

And, you know, we started that discussion on the attenuation against the adoption of network automation before AI entered the room, right? And so there’s always been a certain level of resistance to letting automation go full throttle. And now, you know, the bots and the agents are walking into the room and, you know, the eye rolls are happening and the skepticism is strong. I would love to use that as part of our conversation here to say, I agree with you. We need to see great best practices develop in software engineering and software development, because a lot of what we do in network operations is downstream from that. So I know I’m throwing a little maybe taking a little too much time here up front, but just that’s my perspective. And I’m really interested to have this dialogue with you.

[John Burke] (3:24 – 3:42)

Yes, super. And I think you’re spot on. The distrust issue of automation in the network shop runs 25 years deep. I mean, we have been hurt. We have been hurt badly by trusting automation too far, too fast, many times.

[Andrew Wertkin] (3:43 – 6:00)

Yeah, what I always hear is trust, trust more so for repeatable, templated actions. Yep. If it’s outside of sort of simple, repeatable, templated, then, yeah, that’s where that sort of lack of trust starts, starts coming in pretty heavily. And, you know, as we know, our brethren, a vendor coming by saying just point and click, simple, easy automation is probably the worst way to gain trust with end customer. You know, the more sort of opaque it is, the more concern there usually is, which, yeah, is, you know, drives, it just sort of accelerates potential that distrust in the world of AI, you know, and I don’t know, well, we can talk about ways to sort of approach it and drive more trust, but, you know, it’s not just in networking and network, you know, it’s obviously across the board. You know, many of us have lived in environments where source code control wasn’t being used, where, you know, reviews weren’t done appropriately, where code was copy and pasted from somewhere else without being scrutinized for current environment, where I could just go on and on and on.

And you can trip like, you know, 10 of those major issues and like within a day if you’re if you’ve only accelerated it, but it does bring a tremendous amount of promise. And but it’s not just it’s not just software development, right? It’s debugging what’s going on and trying to figure out what’s going on in live environments and why it’s not working in some of those areas. As long as you stay, you have somebody with the right expertise and you stay critical of what you’re being told, the amount of work that be done in a short amount of time is just it’s it’s amazing, just absolutely amazing. As long as you’re not just like, oh, you said it’s that it’s that, OK, I should do this, I should do this. That’s where the concern is, because we know the LM’s, you know, tend to as hard as you prompt them otherwise tend to get pretty black and white about, you know, they think they found the root cause.

[John Burke] (6:01 – 6:43)

I like with the idea of accelerating automation, the idea that you’re going to be able to help people accelerate better practices if they’re using better practices with AI, they’ll get to the end of their automation project more frequently. They won’t leave it half finished or three quarters finished, never fully debugged, never fully documented. Right.

And then move on to the next emergency. There’s more chance that they’ll be able to in a single day or two days or a week or two weeks get all the way through and leave something behind that they can actually use again confidently as opposed to, you know, with some fear and trembling.

[Andrew Wertkin] (6:44 – 6:58) 

Yeah, yeah. It’s a lot of it’s just about architecture. A lot of it’s about. Well, one is that, you know, the appropriate developer mindset, which is I’m writing this for the people that will maintain in the future, so let me keep that in mind.

[Scott Robohn] (6:58 – 7:02)

Which is automatic, right? Every developer thinks about that from day one.

[Andrew Wertkin] (7:02 – 9:15)

Yeah, yeah, yeah, yeah, yeah. Especially when it started with, oh, you know, we need a quick script to try to solve this issue. And and, you know, and then somebody finds that six months later or. So a lot of it comes down to architecture, you know, like a lot of what is going to be in that automation will be repeated, right? Like how do we authenticate or what are standards around logging or how do we want to log or, you know, just how are we going to interface with with our devices? How are password faulted and how do we get those out of the vaults and, you know, how do we want to test or how should we think about backing out changes or whatever the case might be? Almost everything I just listed off is something that is repeatable and in this appropriate methodology. Well, now you just need to go sort of add your business logic, not necessarily all these other pieces. And and therefore all the code stays fresh and it’s being tested and on and on. But, you know, it’s it’s it’s really easy to build something quickly. And everything I just listed off also sounds like it’s going to take longer than just having eight people reel off a different, fully functional, you know, A to Z scripts, applications, you know, whatever we’re building. And so it’s just there’s there’s, you know, it’s probably more important than ever to sort of start with the right foot forward and start thinking about how we’re going to make this sustainable and and, you know, where humans will be in the loop and where they, you know, it’s critical they’re in the loop.

And, you know, what what sort of practices are we going to bring forward to do this safely? All of these things are just, you know, it’s many of these are the same things we’ve always considered or we’re concerned about as building software over, you know, the last several decades. Right. Some of it’s just how rapidly it can be done now. And and it’s not like just people copying and pasting stuff off of Stack Exchange.

[Scott Robohn] (9:16 – 10:33)

Like anybody can have code generated that works in that environment, you know, like that’s easy to do, it’s, you know, and that it was not the case two years ago, like I think like we you were in the middle of this fast changing and experimental time, right, where if you go back just a couple of years, you know, people had their their initial experiences with chat based tools and got widely varying results. You know, the whole nondeterminism of LLMs thing that was that was our first experiences. And then, you know, we see we see the advent of vibe coding and what an enabler it is for prototyping and but what a spaghetti mess it can produce for an enterprise code base to now emergent methodologies like spec driven development and test driven development. Like, I think we’re seeing the emergence of, OK, here’s how I use the tools, you know, and here’s how I can actually build trust in the technologies versus my first, you know, chat GPT three dot X experience. Right. What are you seeing on those fronts, you know, in the emergence of these strategies and repeatable tools that are coming up?

[Andrew Wertkin] (10:34 – 11:59)

Yeah, no, we’re we’re we’re seeing them for sure and have been experimenting and adopting a couple. I think we’re we’re. We my company and also me personally have had the most success is interestingly, like the part I spent probably the least time about in my career, not the company, but me personally, which is that upfront spec development and working with the LM on trade offs, and that’s where I can bring in my experience and how should we approach this and spending that time up front, breaking it down into reasonable chunks and and then, you know, putting the appropriate guardrails in place and, you know, that that sort of stuff has been transformative and versus, you know, trying to type out a prompt describing exactly what you want, just the sort of iterative spec and test development that really starts with, OK, I want to create a high speed TCEBF logger. Right. I’ve never done that before. Right. And and so I don’t even know what the trade offs are. And so, you know, I’m not I’m not, you know, so how much can I trust, you know, whatever? But I can start at a very high level. I want to do this. What don’t I know?

[John Burke] (11:59 – 12:47)

Intent. You got it. Yeah. I think there’s a really important point, sort of one level underneath that, which is that stating your intent and need very clearly is crucial because when you’re having software twiddle the settings on hardware out in your network, twiddling the wrong settings shuts down the traffic flow or overburdens the processors so much that you get, you know, terrible performance drops as it as it maybe devotes all of its time to generating logs and sending them. So, you know, you’re in a space where infrastructural mistakes can happen and can happen at automation speed and have really huge consequences.

[Andrew Wertkin] (12:47 – 14:57)

Yeah. Yeah. I mean, in our world of DNS, I mean, just on the public side, if you look at the last several major outages that were announced by huge public corporations, I think, you know, 80 percent of them were automation driven change to DNS that took everything down, you know, and and now that I’m not going to remotely blame an LLM for that. That’s just but but you just, you’re well more likely the faster you can come up with the solutions and the more especially the more certain your your coding partner or your debugging partner, your performance partner, whatever the use case might be, the more certain they are, the more likely you’re going to be like, OK, you know, and then and then, you know, they’re everybody suffers from permission fatigue and. You just your fingers over enter and you just keep pressing it, you know, and so, you know, that comes down to also like, how do we how do we appropriately supervise the behavior and make sure those things don’t happen?

But, yeah, and, you know, so the more successful that sort of more successful express intent includes those things. You know, what do you know could potentially go wrong? What are you concerned about? What do you make sure is actually tested? What does load actually mean here? How should it behave? Now, you don’t need that stuff up front, but having the right experience to know what sort of the categories of things that you should probably be considering are good, you know, but what what you don’t need to do, and I think this is really matured in both tooling, but also process over the last six months, insanely small period of time, is just that you don’t need to start by writing a four page prompt on exactly what to deliver, right? You you do if you’re using like a, you know, less expensive, more run rate type model, like not a frontier model, but but just have a frontier model, write the spec for the run rate model. You know, you don’t need you’ll never be as good as an LLM in generating prompts.

[Scott Robohn] (14:58 – 15:55)

Right. Well, I I so I’ve layered in a technique where I basically ask the LLM to interview me where I’ll bring, you know, a page, page and a half of intent that’s reasonably detailed. And if you’re using it, using the process to develop all the all the all the right markdown files for SDD, number one, it’s it’s cheap. I know there’s token cost here, but, you know, I’m creating human readable documents that I can validate and verify after I’ve gone through an interview process. And I heard a great comment last week. You know, the code is free in a sense where, you know, we’re used to thinking about us, you know, laboring to get the code just right. But now we’re asking agents to write the code and we can iterate and we can throw away stuff that looks like garbage. And you still have to have enough domain knowledge to know that it’s actually garbage, right?

[Andrew Wertkin] (15:56 – 16:32)

Exactly. And that last point is, well, on the personal side, sort of my fear, what everything looks like in 10 years, but but also the most critical thing now without the domain knowledge and the sort of experience and expertise of how things can fail and how things should work and how they should gracefully fail and on and on, then then you’re you’re asking for trouble. But but yeah, I like that sort of and I do that to probably less regiment than you do. But that sort of like, you know, what what am I not thinking about here? Exactly. What else could go wrong here?

[John Burke] (16:33 – 16:53)

And when the LLMs are just as happy to bring you the, you know, deadly poisonous mushroom as the tasty and edible mushroom, and they look the same to a casual inspection. Yeah, that kind of domain expertise becomes irreplaceable and absolutely necessary.

[Andrew Wertkin] (16:54 – 20:08)

Yeah. And here’s what happens and why process is so important here is people sit down and they think they’re on the same page with the LLM. And, you know, over a couple of weeks, it seems to be true, like it’s seemingly remember stuff. And then, you know, on day seven, you start a new session and seemingly it forgot everything it ever knew. And it’s been sort of degrading over the week because you haven’t been documenting this. You haven’t been having, you know, saving the appropriate memories and making sure you’re updating your prompts and everything else. And then it makes a decision that is unbelievably naive. And you, because we humanize these things, have this thought of like, you know, you’re you’re working with a doppelganger or who is this person I’m working with? Like, how is it possible they forgot this? And so there’s a lot of errors that occur in those moments when operator, developer, network operator, whoever’s using it, is unclear of the state of the memory of the session and assumes some not necessarily something deterministic, but something at least informed by things we’ve done in the past is about to happen. And then, you know, surprise, you know, and and what do we call that? Like, you know, part of it is rediscovery tax. So, you know, I’m going to spend a lot of money as in tokens or credits or whatever in rediscovering the same thing over and over and over again because it’s not appropriately documented. And and these are things that the LMs in general, but, you know, depending on the coding tool. But, you know, I tend to use Claude a lot, but but regardless of you, several. But, you know, they’re really good at, you know, every time I end a session, like, what did we do? What did you have to rediscover and what that you had to rediscover? Should we should we appropriately document so you have to rediscover it again? And that’s not everything, because, you know, experimentation works. Sometimes you learn a ton when you make mistakes, you know, and maybe going back to the personal side is still professional side of sort of hardened software, like software that works, as we say, like works in anger, doesn’t just work in the lab. It doesn’t just work in the happy cases like it works, put pressure on it and still works. And, you know, getting getting to that point is, you know, it takes takes experience of having done it, but specifically having failed at doing it like it’s like, you know, something smells bad because you once made that error before, you know, or you around when that happened. And that kind of learning, which is, you know, very specific to the very large context sizes of our brains, you know, like is like you don’t even have to go review your notes. You’ll spot that pattern immediately if you’ve had the experience of failing, it just it’s wired in and those are things I just don’t know how they’re going to happen, you know, without purpose, without purpose.

[John Burke] (20:09 – 21:10) And I sometimes hear people saying it won’t be too long now when, you know, we won’t even be having AI develop these scripts or these programs for us. We’ll just tell it what we want to have happen. It’ll spin up a program in the background, send it off and get the thing done. And the next time we need it done, it’ll generate a program and send it off to get executed and we’ll be done. So it’s like it will be creating automation for us. It’ll be the automation. Sure. And for me, I think what you’re saying right now is the pushback on that. It’s like, OK, but the second time it sees this problem and it’s time to generate a program to solve it, is it going to remember how it solved it the last time?

Is it going to remember what went wrong and how it got fixed? Maybe, maybe not. I think until that problem is resolved and resolved by stuff that works in anger, as you say, like no matter how mad I am when I type the prompt to the AI, it’s still going to give me the right answer and the good answer. Then it’s not solved.

[Andrew Wertkin] (21:11 – 24:34)

And yeah, then it’s not solved. And the only thing I was going to add to that is you run into the opposite problem of, of, um, you know, cause the elements are matching patterns basically. Uh, and so something looks like it fits this pattern. It doesn’t mean that’s the answer. And ironically, like if you only have a couple of cases of this sort of like, you know, something failed, here’s why I know the pattern, like then the more likely something’s going to get sort of, you know, force fit into that, like, you know, pound the, the, the round peg into the square hole if it almost fits, you know? And, uh, and it’s just, I think there’s a, still a bit of a chasm between, um, especially in the world of Brownfield, but, but, uh, between, right, I’m just going to state my intent and the scripts will be written, stuff will be done. I’ll never have to look at this twice. Um, for that actually being done in an enterprise network where, um, you know, you don’t have all the tools today to get that done. Um, you don’t necessarily have the budget today to get that done. Um, and you have history. And that history includes things that are not good in the world of AI, which are things like tribal knowledge and one-offs. And, you know, this was done for a very specific reason, but there’s like yellow tape around it. Nobody knows why, but we just know if we change it, everything breaks. So, so we’re not changing anymore. You know, it just, this was never documented that wasn’t documented. So, um, you know, I, I just go back to, um, one of the sort of, um, really well-known, um, foundation books in software development that, that, you know, tools and tactics have changed, but this book is as true as it ever been was, uh, Martin Fowler’s refactoring, which, which I think in the forward, like the first, in the first couple of paragraphs, it says, if you can’t test, put this book back on the shelf, um, paraphrasing something, something close to that, like you can’t change it unless you can test it. And in this world of like, um, you know, tribal knowledge and one-offs and not everything following a pattern, um, how, how are you going to, how are you going to approach that, um, especially if there’s nothing to test against your lab, doesn’t have those one-offs, your lab, you know, it’s how do we actually test this appropriately? And, and, and that’s what causes all of this, you know, longer change management process. They know the organization’s learned if we touch that, it’s going to break. So that’s only going to happen on like this type of change window where we have three and a half hours and nothing else can be scheduled during that change window and like they’ve, they’ve never fixed the underlying problem, but they’ve, they’ve wrapped it around process. And so, you know, companies also enterprise is going to have to look at, you know, where to start with this stuff. Um, and, and stating the obvious, um, uh, net new is, uh, you know, build this way to begin with great, um, changing something exists, whether that’s software or a network architecture, um, that’s way harder for humans and LLMs, uh, but on the L at least the humans hopefully have some of the tribal knowledge.

[Scott Robohn] (24:35 – 26:08)

One of the, so another exciting thing that I think is in front of us here is, you know, to your point of everything you’ve just laid out. Um, you know, if I have an LLM, everything looks like a prompt, right. And we’re, I think many of us are figuring out that, you know, not everything is a problem for an LLM. And, and we have, we have this really interesting and potentially useful set of new computational techniques in front of us, and we haven’t gotten rid of all the others. Like machine learning is still really useful for other things in the, in the AI space, statistical regression last I checked doesn’t really cost me any tokens. Right.

Right. So I’m adding to the tools in my toolbox and the really fun puzzle that’s going to be in front of us all is like using the right tools for the right job here. LLMs where they make sense, ML where it makes sense, regression where it makes sense, um, maybe for the, the, the problem that you called out of, you know, reinventing the wheel every time the question’s asked, maybe part of my constraints are, okay, I’ve solved a problem and I’ve put it into my IT service catalog. And if there’s something in my service catalog that is a 90% or better match for this prompt, use what’s in the service catalog and don’t spend tokens on creating another solution for it. And I’m, I’m riffing here, right? But I think these are all like approaches we need to put on the table as we collectively all figure this stuff out.

And I’m, I’m much more positive about it than I am cynical about it. Um, check in with me in a year and see if I say I feel the same way.

[Andrew Wertkin] (26:08 – 30:11)

So, no, it’s interesting on that sort of service catalog side. I mean, it’s in the good news is just have the LM check to see if you solve the problem before, you know, and, uh, and they’re again, super good at matching patterns, they’ll find something. And that’s actually like of all the sort of techniques I’ve developed, uh, over the last year, let’s say, um, that sort of tight link between, uh, documentation of what I did and what I actually did is, um, is, is, is super both from the perspective of, you know, it’s, it’s, it’s quick story. I years and years ago, I was CTO of a company in the application lifecycle management space. So we built software to help build software and we specifically, um, like part of our, like ideal customer was in, uh, embedded software that was safety relevant where, um, a defect could cause harm to the operator, for instance, um, like automotive or, you know, you name it. And so, um, there in, in that world, there’s all sorts of standards like ISO 26000 and, and, uh, automotive SPICE and on and on. And part of the standards around, like there must be traceability between requirements and code. Right. Um, and if the requirements change or the code changes, that trace becomes, um, you know, potentially problematic. And I’m going to go in and look for any of those impacts and I’m going to have to say in this change, by the way, I changed the code, the requirement still, you know, is correct. The test case is still correct. The architecture is still correct.I’m going to have to go do that for all these suspect traces. Um, and that like, you know, you’re not going to convince somebody building, um, B2B software that they need that level of traceability. Right. But like I do that when I’m doing LLM based development, cause it’s amazing, you know, one and mostly, well, I enjoy it, but, but also because the LLM then goes and traces. Um, and the LLM can quickly figure out why we did something the way we did it, because it can trace back to the documentation and, and, and the LLMs writing that documentation and the LLM knows to write the documentation for itself, which is something that, you know, like, um, uh, even if I, sometimes I will say, write this for a human, sometimes I’ll write it like, you know, write this document for a customer that may or may not be aware of software development, how software development works, or may never have installed Python before, whatever the case might be. Um, but oftentimes with those docs, I’ll be like, um, either include an addendum just for an LLM or just write this for an LLM and, um, and one, it’s more terse, but two, like, you know, LLMs are pretty good at writing documents for them to read and, uh, and, um, you know, both from an efficiency and cost standpoint, reading it, right. You know, like, um, the human version is, is more expensive, but regardless of that, um, that also like goes to this, um, okay, now I’ve included in the right software for the next person to maintain to also write software for the next LLM to maintain. Um, and given it, you know, the capability obviously to pull in and parse a lot of text rapidly, um, yeah, I’ve never had better documentation in my code or outside of my code. Um, would never have done this much documentation as a human. Um, ever, uh, you know, with the assumption, like most humans have, like while you’re writing it, uh, yeah, I’ll remember this. Yeah, this is pretty easy to read. Like I’ll just read the code, like it’s obvious or anybody would understand what I meant by that. Yeah, exactly. Yeah. I said, sorting, I’m sorting here, you know, and, um, and that, that those things lead to a tremendous amount of defects, but so does, um, uh, that, you know, even worse sometimes, I mean, what’s the old adage? Like the only thing worse than, um, no internet connectivity is crappy internet connectivity. That’s right.

Yes. Yeah. It same thing with documentation.

[John Burke] (30:11 – 30:48)

So in a way, um, you’re both mentoring your AI to improve its domain expertise. You’re teaching it more about what it means to run a physical network of physical devices and what the risks are and, uh, what the limits of experimentation are and so forth. Uh, and at the same time, you’re mentoring it on being a good team member by doing things like documenting its code and explaining itself for sure.

[Andrew Wertkin] (30:48 – 32:30)

No, it’s a good, it’s a good analogy because, um, that’s what I usually, uh, advise people is like, um, yeah, especially if they’re, they’re senior, like, um, yeah, just imagine you’ve got four junior to intermediate software developers working with you and, um, and that are pretty self assured, you know, like that, that are convinced they’re correct, you know? And, and so if one of them said to you, no, no, this is the best way to do it. You would say, why, what evidence do you have? Do we have enough evidence? You know, like what alternatives did you pursue before you came to the solution? And if you think about your interactions like that, then, then you’ll always think about what the right question, even though from a ability to generate code that works rapidly, you know, this is not a junior or intermediate software developer, but in terms of thought process, um, yeah, in many cases, intermediate software developers is, is, is, you know, is, is better thought process, you know, in other words, cause it’s thought process, it’s not, it’s not pattern matching and, and, you know, the other magic of LMs. Um, and so like that, I think that’s the biggest mistake, uh, people, you know, whether it’s, it’s bias, you know, confirmation bias or otherwise, like, um, uh, people are looking for, um, that sort of positive, this is how we’re going to do it. And, uh, and if they say, Hey, I was thinking about this, this is a good idea. We all know. Um, that’s the worst way to interact with an LLM because it’s going to happily come back and tell you brilliant. Wow.

[Scott Robohn] (32:31 – 32:39)

Yeah. You are so smart. That’s the best way I’ve ever heard. Yeah. Just, just include no sycophancy in every one of your problems.

[Andrew Wertkin] (32:39 – 33:53)

Yeah, no, a hundred percent do that. Cause I, I, I don’t need that from my peers, my employees and my parents. I definitely don’t need from an LLM, but, um, I do like it for my wife every once in a while. Um, but, but regardless, and my kids definitely, but regardless of that, yeah, no, it’s, but that idea that you’re like, you are the expert here and you’re working with, um, uh, something with less knowledge than you do of what you want to create and how it should work and what success looks like and how it’s failed in the past. And just keep that mindset, keep that mindset, keep that mindset. And then at some point you feel like, this is like, I’ve got the four best interns in the world working for me because that to me with the LLM, maybe five, sometimes seven, but like, it’s, you know, um, uh, the, the acceleration of being able to get this done tonight, please launch 10 parallel agents.

I wouldn’t say please, by the way, launch 10 parallel agents and run through our blitz testing protocol. And then I wake up in the morning and a whole team has been doing work all night and, uh, I, I don’t feel guilty. I don’t feel, you know, I, I have my coffee and read through the results and very powerful.

[John Burke] (33:53 – 34:32)

And you’re sort of, I think by implication saying what vendors like you can do to support this kind of work in enterprise network shops, and that is, you know, basically building the power tools that those AIs can wield with appropriate safety features. So, you know, the, the, uh, circular saw has the automatic shutoff, so you can’t cut your own fingers off as easily as you used to be able to things like that. Uh, you want to help them not crash the network by failing to do the obvious things with your power tools.

[Andrew Wertkin] (34:32 – 39:51)

Yeah. No, it’s interesting. Cause like, you know, like, um, most, uh, software vendors today, one of the things that we, we produce our, our MCP servers for our backend products.

And we have several backend products and, and, um, and I think like a lot of vendors out there, like initially that was just, okay, let’s wrap up our open API tools and this new protocol. And we’ll just expose that and, you know, uh, LM will figure it out. And you realize pretty quickly that, um, that’s not the right approach at all. You’re just going to waste a ton of tokens. It can be a ton of guesswork. You just sit there watching the LM guess and guess and guess. And, um, uh, you know, you’re yes, no, I’m can pretty quickly figure out how to use a rest API. And if it’s wrapped in, in, uh, MCP tools even faster, but that doesn’t mean they understand your domain. And yes, especially the frontier models have been trained on not just your domain, but probably understand more about your product than you might realize. Um, but, but none of them, um, like part of, part of what we deliver, it’s, it’s not just, um, you know, uh, more, uh, domain specific tools than an open API. Like, you know, have more conversation between the MCP server and our backend server than the LM has to see. Don’t don’t send the, uh, there’s a packet capture example. Like there’s no reason to send the LM a 20 megabyte packet capture that that’s, that ain’t going to do anything or send it a whole time series that you should be sending to, uh, ML, not, not in AI, you know, like, and so how do I get the data needs summarized data, maybe even opinionated data, because we know our systems well, so, you know, with, with sort of opinions built in. What does the LM need to answer the user’s question, which might’ve been like, you know, why doesn’t this work and think through that stuff as opposed to just, it’s a bit, I’m going to use the word lazy, but it’s a naive. I think it’s the better word. It’s naive just to assume the LMs will figure this stuff out well. And especially if you want this stuff to be remotely repeatable. Um, so yeah, so we do that, but it’s way more than that. Like, so now, you know, we just went through software development stuff. And, and so like, yeah, our, our customers can now pump out tons of automation scripts and otherwise against our products, super. How do we encode the best practices for doing that? So our customers don’t, you know, automate themselves into, um, downtime. So that, uh, so that they know, and that comes out in things like, you know, uh, network advisors and other things that, that, um, whether through prompts or, uh, specific code or capabilities or training material, like, uh, help our customers succeed at automation, because in our industry, like on the DDI side of our industry, on the, in the world of DNS and IPAM, especially, um, the tailwind of this industry has always been automation. You have to change the stuff faster. And so, um, it, we want our customers to be successful. It, it pains us. Um, we had a customer a few years ago, well before LMs, and it just going back to an example of like, um, the horrors that can happen if there’s no processes around the way you develop software was sort of teaching himself how to script against our product. And he had an administrative user and he wrote a script and went to test it and deleted the entire DNS environment inside this company, which immediately locked him out of systems and active director was down and they’re using that for LDAP and yada, yada, yada glass had to break. And it was, and he’s sitting there, um, like I can imagine I’ve, I’ve had that experience before, you know, like, um, I think once in university, I wrote an endless loop in an application running on a like old IBM PC where there was no, like you couldn’t control seed. You couldn’t like, you know, three hours of work. The only thing I can do is reboot the computer. You know, I know that sort of, that’s one of several examples, some being professional, that sort of sinking feeling of, oh crap. Yeah, we, we got, and I can imagine what was going through that guy’s head, you know? And, um, and so like, yeah, obviously that’s an extreme example, but my point is, um, the more we like, um, we always get asked in software development, like, uh, about metrics and measurements and somebody always goes, well, why don’t you guys just use like lines of code or something? And that’s like, yeah, like the idea for anybody in software engineering that the number of lines of code they wrote, the more, the better is a appropriate metric is, is insanity, right? So we don’t do that and nor do we suggest it. But my point is like, when you start thinking about the right metrics, well, certainly in the world of SAS, but now also in the world of like these scripts, it’s just not about like, the question is, did it work the way we expected it to? Was this a successful automation? Um, and, and what did we learn that didn’t succeed? And, and how do we sort of bring that back into a feedback loop? And like, it doesn’t, um, you know, I mean, I was like, uh, the, the least number of lines of code to solve a problem, the better, but, but mostly did it actually solve the problem?

[Scott Robohn] (39:52 – 40:36)

Well, in going to, you know, least lines of code as, as the other, um, could cause other problems, right? And like, again, one of the things that really jazzed about this whole environment and conversation is like the art of specifying our constraints is becoming much more important, um, probably than it ever has in our careers, right? You know, we’ve got power tools, where are we going to put the right blade guards on? And there were grounding plug. Um, my grandfather made a power tools for Porter cable. And I’ve actually have some of his old prototypes that I would never use to build a deck because there are pieces missing that keep losing digits. Right. So the system’s thinking, um, and there’s tribal knowledge.

[Andrew Wertkin] (40:36 – 40:39)

Cause yeah. Somebody else can, uh, that was right.

[Scott Robohn] (40:39 – 40:44)

For sure, John, this could go so many ways. Like, where do you want to take this?

[John Burke] (40:44 – 41:20)

Well, I think just to, to round it out, I want to follow through on this idea that, uh, folks in the business now need to be thinking about their software portfolio as power tools that will ultimately be wielded by robot hands, not human hands. And, um, how that’s going to change the enterprise software landscape. You know, we’ve talked a bit about how it changes process inside the enterprise, uh, for using it, but what about the rest of it?

Like, does it change licensing? Does it change, uh, you know, how else does things for sure.

[Andrew Wertkin] (41:20 – 44:13)

And I think, I mean, you know, part of it is sort of what you’ve seen in what the, you in, in the private markets in general with, um, with the devaluation of some software companies, which, uh, like, you know, we, well, we all should know the stock market doesn’t is, is it’s own thing. It’s his own thing. Yeah. Uh, but, but outside of that, um, it, for sure, it’s going to change licensing models. Uh, like, you know, if you’re only, if your licensing model is just based on per user and I’m not saying cause everybody’s in a fire, everybody, I’m just going to say, okay, what’s an agent is an agent user. And, and, um, uh, you know, it, it, it, yeah.

Companies are going to have to figure out, um, you know, what, what the right metric is going to be. And, you know, a lot of companies are switching to, okay, it’s going to be credits. And that’s the real value is, you know, how, how much you’ve pushed through this LM or how much, and, and their, you know, enterprises don’t like surprising bills every month.

Um, they, they like consistency, you know? And so, um, I, I think, uh, you know, early companies, especially the, the LM companies themselves, or the AI companies themselves, obviously are very focused on a sort of an opaque credit type, um, model where we all know the price is going to keep going up and up. We won’t be able to do the networking vendors.

Won’t be able to go in there and go up. Sorry, your bill this month is, um, yeah, it’s twice as much this month because yeah, we’ve changed the ratio between, um, credit and token. And, uh, and by the way, uh, we improved our stuff. So it’s better. So you’re going to pay more. Like it doesn’t work that way in our world. And so, yeah, I think licensing models are going to have to change. And, um, uh, uh, but so much, so much more than that, I think there’s going to be, I think we now have the opportunity, you know, how, how we always sort of go back and forth between platform and best of breed and platform and best of breed. I think we’re going to be heading back towards best of breed because sort of interoperability between these vent a lot between different products has become, um, much simpler to solve. Um, especially with things like MCP, like I don’t necessarily think I need a specific integration to, uh, ITSM systems anymore because all the ITSM systems, uh, have the ability to speak MCP. And so, um, that becomes easier, uh, picking a platform versus best of breed, but, but so, so in certain areas, it’s going to be disruptive to businesses where, you know, if their only goal was put as much stuff in our platform as possible, uh, and nobody will be able to compete with us, uh, that’s gonna change. Um, you know, and, uh, and I think should I launch in this? Yeah, I’ll do the 60 second, 30 second version.

[Scott Robohn] (44:13 – 44:13)

30 second.

[Andrew Wertkin] (44:14 – 44:29)

Yeah. Yeah. It used to be like in the world of big, ugly central systems that nobody can change, like the big ERP systems and stuff like that, like where companies would have to, like in their earnings report would be, we missed it this quarter because of a bad upgrade to this massive internal system.

[Scott Robohn] (44:29 – 44:31)

That wasn’t that long ago. 

[Andrew Wertkin] (44:31 – 45:58)

Yeah, it wasn’t that long ago. Yeah. And that was like, that was like the whole Salesforce marketing campaign initially with the no software. And we got that for you and all this greatness about, uh, SAS, um, uh, in that world, how they started getting disrupted was all these like new companies were bringing SAS based systems that were easy to use. They’re still the system of record, but here’s this new system engagement for HR, for travel management, for purchasing, for whatever, any part of that system. And then of course, those large companies in that space would just go buy those companies one after the next, after the next, after the next. And so to some extent, I, outside of the, I’m not quite sure acquisition is going to happen at, at, uh, there can be way too much to acquire.

Um, you’re going to see a lot of companies that, um, are coming up with system of engagement, like value propositions for existing platforms out there to slowly start taking user engagement or system engagement, uh, away. And, um, and that’s going to happen way faster than, than, uh, travel management and, and, uh, you know, goals on the HR side or purchasing or whatever these system engagements were that sort of started doing this with a large internal system. So in that world where it’s simpler to disrupt or at least do better a small part of an existing platform with integration almost laid out for you, I think that’s going to sort of drive this, you know, us back to best of breed.

[John Burke] (45:59 – 46:42)

And I can even, you know, envision taking it, uh, really far down that path. Uh, only the AI’s that work for us really understand what the software portfolio currently is because a new service pops up, they see that if they can do half of what they need to do in functional area X better, faster, cheaper. So now I’ve got two things in that area and it’s the one managing what work goes to which one. And, you know, it’s like a redundant array of software vendors kind of an approach. Uh, I’ll just arbitrage wherever it works best against the criteria that my AI, uh, understands for me matter the most saving time, saving money, better performance, whatever. Yeah. Yeah.

[Andrew Wertkin] (46:43 – 50:10)

Yeah. Cause I, I see like go to some expo floor in tech these days, especially in networking, but, um, you know, pretty much any, any tech related area. And you’re going to see company after company after company, uh, selling your platform for AI, you know, no matter where they came from, they’re now have a platform for AI that’s going to be multi-vendor. Why not? Everybody’s got MCP servers. And, and so what are you going to have as an enterprise, multiple platforms for agentic operations, uh, a single platform, um, uh, you know, one meta agent that manages a bunch of other agent platforms. What if I put the wrong one here? And, uh, and, and how quickly can I change? And, uh, um, and, and we seemingly, and I don’t understand this part cause it’s not the way I work, but people do have easy bias as well.

Oh. Oh yeah. They said they can handle this all for me. So I’m just going to buy that, you know? And, uh, and it comes down to like this theme. It’s funny, you know, earlier I said, I gave a real example. I had never written a EBPF, like, uh, you know, uh, internal space, high-speed logger. And, uh, and now I have, I still couldn’t write one. Um, right.

I use that as an experiment of what if I follow this process in an area that yeah, I’m well aware of how this stuff works, but I’ve never actually written software there. Like what will I learn? What I learned was how rapidly I can do something that I didn’t know how to do ahead of time. Um, and just with establishing the things we’re talking about before, like, you know, criteria and exit criteria and things I’m concerned about, I’m worried about, I have something that works. I don’t know if it’s the best way to solve the problem I started with. I don’t know if like, uh, you know, all I know is it’s maintainable code that, that works.

I mean, you know, I know more than that. That’s really well documented. So, so I guess my point is, um, uh, just like people still need to learn this stuff, you know, to, to sort of bring that experience forward. And that, I mean, like as much as I appreciate this technology, that’s my biggest fear is just, um, the worst mistakes always start when somebody believes they’re right. And people are either too intimidated or too starstruck or too odd or just too lazy to question person who for sure knows how to get something done. And the end result are things that could have been prevented that cause, um, problems. And we, we see that across like, you know, everything from tech to, uh, social environments to everywhere else, you know, and, uh, groupthink or, but, but, you know, it’s, um, that’s what I rely on my experts for, uh, you know, even if they don’t know, even if it sounds right In fact, the more right it sounds to them, the less likely they’ll take that at face value, uh, because there, there must be something wrong with this. What is it? It’s a puzzle. These are, you know, engineers in many cases. And, and, um, um, so as much as I appreciate the technology, use the technology, pushing forward with the tech, like creating value with the tech, uh, there’s part of it that just makes me uncomfortable.

[John Burke] (50:12 – 50:35)

Uh, a solidly cautionary note on which to end the conversation. Yeah. Thank you, Andrew, for joining us today.  This has been fantastically interesting and lots for the enterprise strategist to chew on as they think about how things go next, uh, in their it shops. Uh, Scott, thank you so much for, for joining us today as well. And as always, thanks to everyone watching.