IT Pros Debate: Guilty Networks, Speaking Up The Stack, and Essential Career Skills

WARNING: The transcription was generated by a transcription service. Sometimes they confuse words – especially technical terms – and speakers. Read at your own risk. Oh, and use common sense to interpret where a mistake may have been made.

Dana:                          Okay, so this is us today. There’s seven of us, which is ambitious. We just want to go by alphabetical order and do a quick intro. What we do, how we ended up here. Go for it. Amar, not to put you on the spot.

Amar:                          Hey guys. My name is Amar Vishidu. You can also call me Amar because I think that’s easier. I’ve been with BlueCat for seven years now. I’ve done, we use different roles within Bluecat while most of my time was spent with supporting our enterprise customers, large customers. I’ve also managed to lead a team of our talented individuals, support engineers today.

Dana:                          Thank you. Oh that’s me. Okay, fine. My name is Dana. I think I’ve met, yep, we’ve all met. I run corporate communications here at BlueCat. I’m looking at this conversation from the perspective of what knowledge right now is hidden in everybody’s heads that would be better shared. I’m also looking at it from the perspective of the IT community is a really special thing. The more that we can share I think knowledge and information and also be known as the people who are leading conversations like this, I think that there’s a reason that all of you are part of the tech field day family for example, is because we go out of our ways and do this kind of stuff. So that’s what I’m looking at this conversation at or the lens I’m looking at it through today.

Joep :                          All right. My, my name is Yupishka. I no longer have a daytime job. I work for my own company as a doer of many things. So I organize conferences, both independent community events as well as commercial events as well as help out vendors in this space organize their events. I blog, I write. I create content together with some other industry friends. And so basically I am in the marketing space of IT, but I have a technical background, which makes me look at the IT industry and especially the Dev ops movements that are happening with fresh eyes, with a pair of different eyes. Instead of looking at the technical part of dev ops, I tend to look at the more cultural and organizational aspects of dev ops and try to bring value to, to the community that way.

Dana:                          Okay. And you in terms of backgrounds, so you said you no longer have a day job. What’s the experience been in?

Joep :                          So I kind of did it all but the last, I don’t know, last 10 years, I’ve done my VMR or VCDX. So I’ve done a lot of VMR implementation and design. I’ve been a CTO for about five years for a big service provider back here in the Netherlands. So I’m thinking about, technology decisions and how they impact the company I worked, for training people, mentoring people, creating IF-platforms and most recently, working for a supermarket where I helped start them on their transition into the dev ops role.

Dana:                          Awesome.

Jean:                           How am I going to compete with that? So I do have a daytime job as a senior consultant at Open Line, I focus mostly at storage hypervisors backup, so Open Line’s a cloud provider all the way in the South of the Netherlands. We do a lot of projects at the customer site or we provide cloud platforms for them. So then I’ll manage the insourcing or the migration projects. Yeah, that’s it.

Dana:                          Can I interject and just ask the context of these migrations and cloud projects? Like how they apply?

Jean:                           Yeah, how big are they or what? So it ranges. So we serve smaller providers, smaller customers all the way up till, well, the bigger ones in the Netherlands, at least ranging to couple of thousands of VMs. So it’s mostly lift and shift where we provide the platform. We basically pick up the customer environment and host it for them. Yeah. We’ll do anything all the way up till the technical domain. So do everything from dev center up till networking virtual machines. We’ll hand it off when it gets to the application site.

Dana:                          Okay. Good to know.

Kevin:                          Hi everyone. My name is Kevin Wilson. I currently am the manager of solution architects for the Bluecat networks, reporting to Nell Reynolds, who you’ll meet here shortly. My background, I’ve been in the IT realm, will say for a good 25 years. Now starting off in the military, in the US Navy right out of high school, doing your basic, LAN-WAN management, desktop support type stuff. Stuck around in the networking world so to speak for the first 5 or 10 years and then moved over into service management, service delivery and getting into more of operations, and then got into the InfoSec and info protection world. Did a stint at Symantec for several years before joining BlueCat and kind of moving back into the networking world with a bit of a play on security stone.

Dana:                          You’re on mute, Ned.

Ned:                            Okay. So I’m Ned Bellevance. Like Yup, I don’t have a technical day job. I work for myself. I’m founder of Ned in the Cloud LLC and apparently also a cartoon character, which is fine by me, try to look at that than anything else. I’ve been in the industry for, I don’t know, almost 20 years now doing all sorts of roles from help desk all the way up to CIS admin and then consultant on cloud and Dev ops projects. These days, I write for my blog. I create courses for plural sites. I do fun stuff like this when I’m invited and I host a podcast called Day 2 Cloud.

Dana:                          Which rocks.

Ned:                            Thank you. Thank you. We do our best.

Noel Reynolds:          So I’m Noel Reynolds, I’m the director of solution architects for BlueCat today. But to kind of like all of you guys, I’ve been in it for about 25 years. I started really doing consulting and then I moved into a role where I was a support engineer for a large data networking company and I did that. I was at that company where I kind of evolved from like a solution or sorry, enterprise support to more of a professional services role.

Noel Reynolds:          I was a resident engineer onsite at one of the biggest universities in the state of California. Then I became what we call a sales engineer at the time where I supported Navy and Marine Corps for a number of years and then switched into the commercial space. I joke, I did a two and a half year sabbatical at Symantec where I learned security and then I went back into telecommunications and then shortly after that, I came over to BlueCat where I’ve been for about four and a half years now. I fill my spare time with listening to Ned’s podcast and being jealous now of his cartoon avatar. Yeah, it’s good times.

Dana:                          So for contacts, I love that they come on the little cards and they’re like playing cards and you collect them at the trade shows. I think, was it Cohecity that does those? Or who was it?

Ned:                            I thought it was Rubric.

Dana:                          Move a lot it is, you’re right.

Ned:                            They’ll get very angry if you mix them up, you know.

Noel Reynolds:          So true.

Dana:                          Well, they did invest a lot of time into making them. So I get all of that.

Ned:                            Yes.

Dana:                          Yep. Okay. Awesome. So from here, I think that it’s probably just easiest to go into the context of how some of us know networking. I think probably the easiest way is to start with the people whose job or whose day job it is not, just to see what kind of interaction you guys have had with the network team.

Ned:                            I can volunteer. So for a long time, mine was the network team because there was three of us, the company I worked for. By virtue of the last in, they were like, “You do the networking now.” So that was my interaction for the first six years of networking. Then I went to a bigger organization that actually had a network team.

Ned:                            It was an interesting process because I had to sort of show my credentials to them before they’d let me make any suggestions about the network, they didn’t necessarily trust me to have good ideas and suggest different things. One of the very early things, I needed them to add some VLANs. I was using Cisco terminology. They were using all extreme switches. So they were like, “You don’t know what you’re talking about.” I said, “You don’t know what you’re talking about.” So yeah, a little fun friction there. But we all worked it out. We made friends, we brewed beer together and life was good.

Amar:                          It’s funny, Ned, that you mentioned that. I can remember as an end user supporting our customers when I had a desk job with an organization, they seem like they would never trust you. You were never the expert, even though you knew the environment inside and out They bring in a consultant, right, or they’d hire a team to come in and all of a sudden, anything they wanted to do they would trust explicitly. Right? So kind of made me laugh that I could never get anything done, but I’d bring in a contractor, quick and easy.

Kevin:                          Yeah. That resonates to me. So about 2 years ago I started this whole journey of creating an iOS platform for my then employer. That was based on totally new technology. So one of the things we implemented was NIC RATS, so back before it even was NSX by VMware. We just had discussions all night long for weeks, for months about the networking stack because that had to change significantly, right? The design of an underlay overlay type network is just completely different than what they were used to back then. I remember in the discussions. At some point, I was so done with it that I did in fact hire a contractor who did not know anything about NSX at all. I just whispered to him, “Say this” and then he’d say that. That worked. So in the end, we did have a pretty awesome network created for a multi-site iOS platform which was for that time, a significant effort.

Dana:                          What’s the reason for that? Like coming from a very non technical background, what is the reason for this defensiveness?

Ned:                            I think it’s blast radius, right? If you break a server, you just broke the server. Like whatever, you can fix that. If you screw up a storage array, maybe you take down someone’s one, but not too bad. If you screw up the network, nobody can do anything. So I think that has made network administrators, and I know me personally, when I was a network admin, I was very protective of changes. I was very cautious about making those changes that had to happen during the maintenance window. Because I knew that if something was screwed up then boom, nobody can get anything done. Then I’ve got all these high level execs breathing down my neck asking me why they can’t get on the internet. I think that’s definitely one source of the defensiveness.

Amar:                          I think the other thing too and I would see this, when I used to do support or when I was onsite at a customer environment, but there’s the concept of meantime to innocence, which is a joke, but it’s true. It is a real thing because it’s always the network until you prove that it’s not. I mean if the server is running slow or is not provisioned correctly or what have you, it doesn’t matter. You have to prove that all of the traffic is being delivered and it’s being delivered in a timely manner before you can say, “No server team, go look at the way you have your application scaled on this box.” Because again, it’s always the network at first.

Jean:                           That’s it. I can definitely concur to that too, that most of the time that our team spends solving customer issues, it’s 5 or 10 times every one out of two it’s usually something on the network that causes some of the internet somewhere that we have to start pressing down. Then sometimes they have to go down to the basics. You start from the client, you go hub by hub, use space back at the packet. That’s how you prove. That is a mountainous task to prove. Until you’ve proven, you’re always at fault.

Ned:                            Yeah. So I had a very similar sort of situation. I was doing consulting for an active directory forced trust that wasn’t working properly. If anybody’s worked with active directory, they know DNS is the backbone of active directory. If anything is wrong with DNS, anything at all, AD does some funky stuff. Well, we couldn’t get this trust to authenticate properly going one of the ways. We ended up having to do Wireshark packet capture across multiple nodes. Then we finally figured out that somehow all the authentication was getting sent down to a particular domain controller in a child domain that had been turned off. I don’t know, we don’t know why the traffic was going there. But it was something in DNS. We eventually honed it down to that and removed it. But in that case, the network actually was the problem. But the real problem was just misconfiguration of DNS and active directory. But you had to prove it, right?

Jean:                           Yeah. The funny thing is it’s not easy to say it’s DNS because it’s definitely the last thing that comes to anybody’s mind. Wait, how about DNS? What is that pointing to you though? It’s always everything else that you check first. DNS is definitely the last thing. Sometimes it turns out to be more expensive, especially when you’re working with large trading companies that have billions of dollars on the line by the minute per hours. DNS is going to be people look at last. Just working with many of our companies that are abroad, one thing I tell them every time is that if you were the manager or managing team off this infrastructure, no matter if it’s DNS fault or AT fault, just call us. Just get us on the phone because it may not be as at the end of day, but we want to be there as you troubleshoot this because it will save us time. It will save you time.

Ned:                            Yeah. Always check your host files too.

Jean:                           Yeah, yeah.

Kevin:                          Well everyone, everyone has slightly experienced in infrastructure and especially in the VMware world will check DNS as one of the main culprits first. I mean, it’s going to be DNS or MTP, right?

Jean:                           Yeah. I still see a lot of installs that actually don’t configure DNS properly as well. They just installed some equipment or install some version machines. They’ll have some IP addresses. But if you resolve them, they still don’t have any DNS. It’s often forgotten and change processes as well, I think.

Kevin:                          Yeah. Especially in infrastructure, people just don’t care about DNS.

Jean:                           Yeah, yeah.

Dana:                          Why?

Kevin:                          Which is still the founding, it’s the lower level of your infrastructure.

Jean:                           Yeah. It needs to be in there. It makes your life easier.

Ned:                            It’s really taken for granted and the best analogy that I’ve come up with and when I’m interviewing for Solution Architects, for my team or whatever, is to talk about it in the context of the oxygen in the rooms that we’re all in. If the oxygen in the rooms that we’re all in is just there, and we’re breathing and whatever, it’s fine. If it’s not there or if there’s a problem with it, we’re in real serious trouble but no one is going to talk about it or notice it unless someone brings it up. That’s kind of how DNS is and that’s one of the challenges.

Dana:                          My question is, okay, I mean first of all this conversation came right to DNS and I get it like …

Noel Reynolds:          Random.

Dana:                          Why do some teams not care? Like certain infrastructure teams or whatever it is, they don’t set it up. Right? How are they able to do that in the first place? How does it not impact them or how are they able to think that it doesn’t impact them?

Ned:                            Nope. I guess some just use the IP addresses and they just rely on memorizing the IP address of … I still remember the IP address of the two domain controllers in the first network that I’ve ever worked in. There’s no good reason for saving that like, right? So I think part of it’s like IP address and the other thing, and this is kind of like going back a while, is Winds actually does work sometimes. So they may have been getting by by using Winds and not even knowing it, which that was the Windows version of DNS that never worked that well, and please don’t use that.

Amar:                          This will turn it by default.

Noel Reynolds:          I was going to joke that I was too young to know what that is, but, but anyone who knows me will know that that’s not true.

Ned:                            Oh, we could all pretend though. But yeah.

Noel Reynolds:          Exactly.

Jean:                           Sorry. What is Winds again?

Noel Reynolds:          I almost forgot about it, Ned, and then you zone bring it out.

Ned:                            I think it was the Windows internet name system?

Noel Reynolds:          It was, it resolved net-bios names to IP addresses.

Ned:                            Yep.

Noel Reynolds:          Yeah, yeah. It used to be one of the three things that you had to configure on a Windows. So when you were setting it up, you had to provision. You had to actually define the Wind server just like you do a DNS server or just like you do. DNS came later. But around the time that Windows was competing with … What the hell is it called? IPX.

Ned:                            Nobel? IPX.

Noel Reynolds:          Nobel, thank you. Yes. Yeah. About that time where it was making that transition, you saw it kind of fade away in favor of IP. We’re all better off for it.

Ned:                            But if you ever worked on an older network, you might come across it sometimes. Setting up proper replication, they had like pull and push servers. There was ways to clear out and rebuild the Winds database and it was, what a mess. So I’m so glad that DNS came in and sort of saved the day. But I think for some older curmudgeonly Windows admins, they just want to use their IP addresses and not have to worry about this crazy DNS thing.

Noel Reynolds:          Yeah-

Dana:                          Does that-

Noel Reynolds:          Yupa was kind of onto something earlier too though. He was talking about proving that the network wasn’t at fault and sort of the complexity. It’s kind of funny because in my career, I started off the networking company I was working at had … We supported every physical topology. So token ring, ethernet, Fitty, the ATM, etc. The thing that was interesting was there was a period of time where everything was Ethernet and IT, and it was relatively simple compared to today, right? When you get into you’re deploying an overlay, multiple overlays on your underlay and you have to have the knowledge of all of those things, right? Then also you’re going to connect it to the cloud, if you’re coming up today in networking, you’re not really able to grow with it from that simplistic way where it was just IP and just ethernet on prem.

Noel Reynolds:          You have to really be knowledgeable on a variety of things. That adds to the complexity, adds to the difficulty of deploying it. There’s a lot to learn. I mean it’s sort of balanced by the fact that now we can go to YouTube or go to a podcast and learn for a period of time. But really there is a high level of complexity and detail and multiple environments that we have to be knowledgeable about and we have to support. That can be a real challenge for someone who’s kind of coming up and learning. It puts a lot of pressure on the networking team to stay to pace with all the things that are changing.

Dana:                          Not just from the networking app, right? There’s all these other teams that seed us from different lines I’m assuming.

Ned:                            Exactly. Yeah.

Dana:                          When they’re non-respond to this.

Ned:                            One of the thing that made me really have to learn and adapt to DNS was the adoption of cloud and well before that VMware, but then also the adoption of cloud where I no longer could rely on a particular IP address being assigned to my system. I was tearing down environments and building them pretty dynamically. So everything had to be C names. I had to rely on the DNS resolution that was provided by whatever cloud provider I was consuming. So that made me have to go back and be like, “Okay, what was the C name? What’s the difference between that and A Record? What are all these TXT records that I suddenly have to add every time I have to verify a new domain?” I had to gain a new appreciation for DNS, which the only time I knew it any better was when I was studying for MCSC and you had to know what a stub zone was. Dana, we want to explain the stub zone?

Kevin:                          It’s going to be that kind of call, huh?

Noel Reynolds:          Good seeing.

Kevin:                          So it’s kind of funny though that back into in the old VMware data center days, stuff was pretty static. There was an IP scheme. Everyone knew that IP scheme. Maybe there wasn’t enough of a reason to properly do DNS. Now with cloud, with Kubernetes, with dynamic services, ephemeral containers, there is no other way than to rely on a naming scheme. There is no way IP addresses aren’t going to work out. I have never used IP addresses, BEC since I’ve stopped doing VMWare installs. There’s no other way, I think that in the cloud or with Kubernetes or with containers, there is just no other way than to rely on something as dynamic as DNS to kind of save you from that complexity. Just as Noel was talking about. It’s just too complex to do by arm. Yeah.

Noel Reynolds:          Yeah. And if IPV six ever actually takes off, I mean, I always joke with people that I think I’ve learned IPV6 four times. I remember the DOD issued a, DOD is a US department of defense, but they issued a mandate that said that by 2008, everything in the DOD will be single stack IPV6. This was 2001 so I had plenty of time start studying. I swear I’ve learned it three or four times, but it’s not, in my view, it’s really not that complicated once the hardware and software has kind of caught up with the ability to deploy IPV6. But to your point, no one’s going to memorize IPV6 addresses. You, it’s going to have to be DNS and so that’s also where we see as really critical.

Kevin:                          Yeah. I mean especially if you go large scale or you go something that is really dynamic skills up and down, you have to have something more reliable than dynamic IP addresses.

Noel Reynolds:          Yeah.

Dana:                          You guys are agreeing way too much. Very friendly conversation.

Kevin:                          Gives us a difficult topic.

Dana:                          Okay, go.

Ned:                            The programming language?

Dana:                          Actually I would definitely learn something in that case. I’m more just curious if you guys want, because it sounds like you’re pretty much on the same page though about networking in general. Is there any sort of like … Are you just very educated people who happen to come from different teams or is this the general consensus among the people that you work with on dev work?

Noel Reynolds:          I mean I’ll say something controversial. I mean I think and I say this not to pick on anyone or whatever. But I think a CCNA course should be required training for everyone who works in IT, period. Networking is the foundation of absolutely everything that we’re doing from an IT perspective. It will help you in cloud. It will help you if you’re a storage guy, it will help you. Far too often when I’m interviewing people who have come from a company where they’re supporting some kind of a security point solution, a SIM or pick a product, right? Networking to them is they get the IP address of their endpoint. They get the IP address of the gateway and they’re done. That’s not enough.

Noel Reynolds:          The other thing I would say is you know, people need to know how to open up Wireshark. You should do that just for fun because you’ll learn so much about what kind of traffic are you’re putting on the wire, what kind of traffic is coming to you, what protocols exist on your network. It’s I think that I don’t want to hire people who don’t have at least a little bit of a networking background who don’t understand that these, all these devices have to communicate and how they communicate in different ways. Because like I said, you learned so much just from having that lens or that view. If you don’t have that, you’re probably not in the right field to be totally honest with you.

Ned:                            I’ll take issue with that a little bit because I think it really depends on what you do for work. Does it make sense for a software developer who’s working on like serverless to understand how to take a Wireshark capture? I don’t think it necessarily does, but I’m curious if you want to make the case that developers should also be able to pop up in Wireshark and watch the line.

Noel Reynolds:          I think at some point in time that serverless container device connects to a network. When you’re troubleshooting the bigger issue down the road, right, which you’re going to have at some point, you want to have an understanding, at least an understanding, right? I mean, this goes to the whole challenge of all the different groups that work within the company and what their knowledge is and how can they talk and communicate with each other.

Noel Reynolds:          I think networking is sort of the glue that holds everything else together and having that, again, it doesn’t have to be super complicated. When I said CCNA, I competed against Cisco forever. Cisco is not at that time, like it wouldn’t blast me to say that. I don’t have a CCNA, it’s not the route that I took, but I think it’s a really good basic certification to kind of get people to think about how devices communicate with each other.

Noel Reynolds:          Like I said, at some point in time if you are that server guy and you’re having slow performance. You say, “Hey, I think it’s the network,” it gives you the ability to talk to the networking team in a way where you’ve done a little bit of troubleshooting. It may not be the serverless guy, but the server guy. But you’ve done a little bit of troubleshooting to kind of get yourself to a point where you can talk intelligently. You can give them useful information and your meantime data center or your meantime to resolution is going to be a lot faster than it would’ve been otherwise. So I think you can take issue with it. I’m fine with that, but I think networking is the glue that holds all this stuff together.

Amar:                          Well, I like the concept you share. There’s like a common language in IT and that networking has to deal with that common language. So you have to as a serverless guy, a serverless developer versus the networking admin, you have to have some kind of common understanding to be able to communicate. I think there is merit to the concept of the network being that shared language. I do see that changing.

Joep:                           I do see that instead of the layer two, layer three, actual networking concepts, it is becoming much more on a layer seven plane where API’s are the thing that glues us together. So we should be able to talk APIs. How does this API communicate to that endpoint, etc., etc. So I do see that changing. I do see those conversations kind of moving up in the stack a little bit. But still I do agree that by having some kind of understanding of networking will help your discussions, definitely.

Noel Reynolds:          The thing I’ve sort of been wondering about, and I think this is the first time where I’ve ever said it out loud. By the way, I just on the prior point, Dana did ask me to say something controversial. So I was trying to come up with something. But the thing I’ve sort of been wondering for a while is when we do talk about DNS, a lot of times the DNS guy at a company is this gray beard who’s been doing it for 25 years or longer. Right? He’s really kind of grown up on the technology. I agree with you, I think everything now is going to kind of an API centric model. Everything’s going to be automated, everything’s going to be orchestrated or what have you. I wonder if we’re going to lose though some of the networking graybeards if you will, who have that knowledge of how to provision redundant services. And you know, like the, there’s this idea that we won’t have to think about that anymore.

Noel Reynolds:          We don’t have to be concerned about it, this is all just going to work. My question is, are we going to lose something as those guys who feel like they’re being kind of pushed out, they know the CLI really well. They know networking really well but they haven’t gotten into the API mode. Is there still a place for them today because of their knowledge of how these systems interconnect and how you do things redundantly and what have you? Again, I’ve been thinking this out loud or out loud for the first time, but what do you guys think about that in terms of just the way things are changing?

Amar:                          Well, I’d like to compare that to the physical server admin or the virtualization admin from just a couple of years ago. There’s been this debate within those groups as well. Is our role still going to be there in a couple of years? Do we still add value to the company? I dare argue that in a lot of cases, no, you don’t actually add any value anymore, but you still need to be there. Right? So that the cloud service provider still needs remote hands to fix, replace a drive, fix a server, replace cooling components or whatever.

Amar:                          Then within certain companies, the VM admin still has a place. But is it novel? Is it innovative, does it actually solve business problems? No. Right? So it’s become so much of a commodity that at some point, you can argue is it better to just consume this as a service from a third party service provider instead of doing it ourselves and having that knowledge in house. For a lot of networking components, I’d argue just get it as a service, make sure that you have a reputable partner that does it for you. They might employ the people. They definitely need the knowledge. It’s a matter of, is this relevant for us to build that capability and answer about?

Ned:                            Yeah. I’ve been hearing similar debates from the networking field on you have traditional CLI. I like to copy things out of notepad and drop it in the config. Then the newer or the people who have updated their skills a little bit are leaning more on that API declarative model of configuration. The push from them is, if you’re not learning these APIs, if you’re not at least learning a little bit of programming, then you’re not keeping your skills up to date. If you wanted to get into an industry that didn’t require you to keep your skills up to date, this is probably not the industry for you. I think that’s been fairly obvious from the get go. It’s not like technology didn’t move quickly 20 years ago.

Ned:                            It may have moved slightly less quickly, but it’s not like it wasn’t constantly changing and evolving. So I think if you are that graybeard and you just want to rest on your laurels, you’re like, “I did my time. I went through the learning process. I’m now a[n expert] DNS person.” That’s not going to work out for you. It’s not like becoming a blacksmith.

Noel Reynolds:          Yeah. To be totally clear, like I wasn’t challenging that guys need to learn APIs today. I was just wondering what about the other knowledge that they have, how things are connected, how you want to connect disparate systems, how do you measure network properly, things like that. I agree, if you’re doing everything within a public cloud environment, it’s kind of all taken care of for you, right? You don’t have to worry too much about … I mean you have some concern about availability zones, and where things are and whatnot, and how you get access to services and whether things are properly load balance, but you’re sort of in a walled garden where everything takes care of itself.

Ned:                            I think the abstraction of the concept is extremely useful from those people. So maybe the specifics of configuring a leaf and spine network is not necessarily useful when you’re designing the cloud, but the abstraction of that concept saying the way that I’m only three hops away from any other node because of the way I’ve designed this, that translates well to when you’re designing at your networking or you’re trying to do a multi-cloud kind of situation where you need to understand the number of hops between things, and latency, and bandwidth. So I think if they’re willing to take the abstract concepts and apply those, then the skills do transfer to a certain degree. I have all this arcana about Winds apparently that I didn’t even know I had. That’s not useful. But understanding the idea of a push pole replication, that’s still useful and relevant for other things that use that same replication model.

Kevin:                          I think that’s a good point.

Noel Reynolds:          Yeah. I think it goes back to, you know who said it earlier, right? The people in those roles, they have to be flexible and willing to adapt with the change, right? I mean that’s the only constant we have in this industry is change. Again, if they’re not flexible and adaptable to that, they’re just not going to succeed.

Noel Reynolds:          I can remember years and years ago when I got into some heavy. the new world of automation, and this goes back to home appraisers, right? I was working for a bank who was doing obviously a lot of home appraisal type work. The way they were doing it at the time, this was back in the kind of early 2000s, it was a very manual process, right? They would go out there and they would take pictures on a camera and go get those pictures developed. This was before digital cameras. They would get faxes of the work orders. They would put all this stuff together. They would literally buy atlases and then tear out the pages for the maps and then reuse whatever they could until their Atlas was completely worthless and buy a new one.

Noel Reynolds:          The team I was working with and who I was supporting from the IT side, they were working on this whole new way to do appraisals. That’s what we’re all familiar with now. The work orders go out via email, they go out with digital cameras or just use Zillow or Google images to get all these pictures of the neighborhood.

Noel Reynolds:          They use whatever street tool that they want to use to download all the maps and get all the comps and do all this stuff. So the long story short is they went from like a 30-day process. If you ever bought or sold a house back in the early 2000s, It was like a 30 day process just to get your appraisal. Whereas we turned it into like a 24-hour turnaround to just be able to quickly, the appraiser comes out and knocks it out in less than a day. But the people at this bank that were doing it the previous way and there was people just printing these reports and sending them out. This was like a full time job for a lot of people. They were really worried about automation and what it was going to do to their job.

Noel Reynolds:          We’re all going to lose our job. We’re not going to have work here before too long. So we were getting a lot of pushback from those teams. again, on a daily basis, it was almost like the scabs crossing the picket line as we’d go into work. These people would be very angry at us and you would just have to sit down with them and explain if there’s so many new opportunities here, yes, that door is closing, but look at all the doors we’re opening up now, right?

Noel Reynolds:          You could become any one of these new support teams handling all this stuff, right? Email is going to be big. You’re going to have to start doing this stuff. Look at that aspect of it, all the digital camera stuff. Maybe you can get into that world or maybe you can get into the I can’t remember what tool we were using at the time. But it was some street tool for pulling up the maps that we were OEM and stuff. But again, I was like there’s all these different avenues that you could go down and can take your career versus being the person that was sorting print out jobs or something. So it was just, again, you got to be flexible and willing to adapt with the change.

Amar:                          Just to switch gears a little bit. In my experience, we can work with lot of different teams, a lot of different companies. But fundamentally, the people that we’ve talked to, regardless of who, which industries, which sector somebody, some customer or some partner is from, it doesn’t change who we are talking to. It’s always has been the networking teams, the automation teams. That’s the extent of teams that we have talked to from a support team perspective. I have never seen issues being raised from somebody like storage background or a cloud background coming directly into DNS. I want to know why that is the case. Maybe something that Jean can help me understand a little bit.

Jean:                           I think the storage folks have usually kept to themselves maybe, maybe that’s the stigma I think. Because they had their own networks, they have their fiber tone networks. They had their own race that nobody understood because you had to calculate IOPS and stuff, and terabytes and play with that. That’s now integrating a lot with the networking, with the shared networks. So I think there’s a big mentality change in that area definitely. Yeah. But if you look at the traditional OnPrem storage systems, they are mostly static. So like I said, in the previous briefing, we got a couple of management IP addresses, and that’s it. Those are static for five years. Then we replaced the box, we got a couple of new IP addresses, couple of new management addresses and names and that’s it. That is definitely changing now as well.

Jean:                           With all the scale out systems and with all the containers, which referenced this storage in a different way. So yeah, lots of changes. But I think going back to Noel’s comment, I think it would be good if other teams had a small basic introduction to storage as well. Because if we talk with application owners, sequel database owners, they have no idea what has changed over the years with storage resulting in exchange requests that are still dated in the early 2000s. They gave me a written volume. Oh wait, that’s old school. So yeah.

Dana:                          People recommend that people start with to learn about storage? This is actually the conversation to have is what should those common languages be? What should everyone know about what you do and vice versa?

Jean:                           I don’t have a specific course in mind, different vendors have different introduction, associate courses, talking about the different types of storage and what the advantages and disadvantages are. So that would probably be a good thing. Read a lot of blog posts as well, maybe? To get a bit of rest.

Ned:                            Could be … Go ahead, Kevin.

Kevin:                          I was just going to say, I mean, one of the things we see quite often and one of the things that we actually kind of tout as a future when we’re dealing with APIs and specifically automation and more specifically, self-service is kind of the argument against these guys needing to understand these guys. Meaning the customers needing to understand what’s changed in storage or what’s DNS or how to administer DNS or what’s an A record versus a C or any of that stuff. We’re trying to tear down those walls and say, “Hey, via the use of a self service portal, for example, let’s simplify this down.” I’m trying to choose my words rightly, correctly, but let’s make this so simple that somebody that doesn’t understand what storage is or they’re still stuck in the raid 10 days, right?

Kevin:                          Give them the ability to answer a simple questionnaire about here’s what you need. Ask those three to five questions. Or maybe it’s 10 or 20. I’m not a storage guy either, right? How much information do you need? But again, fill out this simple form with basic questions that they would understand that relate to their job and why they’re requesting, quote-unquote “storage.” Then based on the automation and the API has let that automatically do that. So again, we don’t need to train the masses on these things and let the technology do the heavy lifting for us. I think that’s one of the things at least from a DNS perspective, we try to focus on. Again, dev ops in the teams out there that are trying to leverage the tools that we all bring to the table, we can’t train these guys.

Kevin:                          I mean, look at all of us. It’s collectively, there’s hundreds of years of experience here. So to train these individuals for example in marketing, everything there is to know about DNS and all the different record types that they need to launch off a campaign, which is never going to happen, right? So to get out of the way, we used to do things historically that used to take days or weeks or months to get through change management, let’s just give them a self service portal where they can go out there and type in a few simple answers to some basic questions and it’s spin everything up to where they need it.

Dana:                          So then all the marketing example network team at some point then would have to learn a little bit more about what marketing is trying to do to be able to build out that self service portal in a way that logically fits, what are ex-marketer for example? Would be looking?

Ned:                            I mean, yes, yes. But I mean, so I think there’s two things, right? On your list, it’s silos and collaboration. I think that means within the different IT teams, but if we do look at the marketing team for just a second, if you’ve got a marketing team or sales team or whatever, the IT team necessarily supports the business and whatever the business objectives are, right? Having them understand what the marketing message is that the business is trying to deliver will help them better understand that the overall goals of the organization so that they can be more strategic in what they do from an IT perspective. So there’s no question that that’s important, but I think specifically on the context of you’ve got a large organization, you’ve got an automation team, a virtualization team, security, etc., right?

Ned:                            They do all need to work together to make sure. So I think a couple of things. One, it has to be a top down approach so that from the top, culturally you have to push those teams to collaborate and cooperate with each other. I think that you can do that a variety of ways. If someone from another group within IT is willing to do a lunch and learn where they go through the basics of the storage system that we’re using. Because I’ll tell you one of my challenges from just storage specifically has always been, from a networking perspective I think you can learn basic networking and bounce around to your bunch of different vendors and everything’s relatively fine. Storage feels like from the little I’ve delved into it, every time I have, it feels like it’s like, “Oh this is really Net Appy or this is really Convolt or this is very semantic. It’s always kind of felt a sort of one off-y after you get off of the wire.

Ned:                            And so it gets confused. I mean there are some general concepts that are definitely across the vendors. But I think that you have to be sort of be committed to that. I think, again, it’s one of those places where the collaboration between the different teams, knowing what capability the storage team has that they can deliver to you that you might not have even known about, helps you solve problems, right?

Ned:                            If you as an organization are committed to that, and you work toward it, and you try to create that kind of culture of sharing, it can only help. We see organizations we work with that want to do everything via API, to Kevin’s point, it has to be really simple. It’s supposed to break down. There’s sort of a double edged sword because if once you give someone API access to something, they’re going to use it. So there has to be some constraints, but at the same time, I mean that’s what’s going to tie it all together and make you be able to deliver things at the speed of business is the API for all these systems. Necessarily you have to understand what the capabilities are so you can tie it all together in a way that’s meaningful.

Kevin:                          Well, and that’s the hard part is making sure that everybody understands each other. Let’s say your point, do a lunch and learn. Make sure the teams understand each other, that they are familiar with the work other teams do. So at my last job, I actually spent a significant amount of time in redesigning how the organization was structured. So instead of having isolated virtualization teams or networking teams or whatever teams, we basically turned that whole model upside down and said, “Okay, this is a team that’s responsible for this little part in the customer journey.” So be it checkout or be it promotions or there were hundreds of teams. For a lot of those teams, we actually made a decision to include a storage engineer, a networking engineer, a cloud engineer without them actually having to take responsibility for a networking system or a storage system.

Kevin:                          But they were the ones that helped these teams to consume those resources. Right? So one of the things I realized in the last year or two is that in order to be successful with transitions like this towards dev ops, towards clouds, microservices, containers, all these things, you need to step away from actually doing the thing yourself. So don’t do networking yourself anymore. Don’t do storage, don’t do virtualization, don’t do Kubernetes platforms, but instead be the consumer of those services.

Kevin:                          Then there can still be a very detailed, very specific teams that actually managed a network underneath as long as they’re able to surface it as a surface to other teams. So the interface between teams has to become so clear that there was no longer a real need to do that lunch and learn to help other teams understand your thing, because you’ve abstracted it in such a way that that’s no longer needed. Right? Just as cloud abstracts it in a way that showed that it’s easy to consume. Similarly within enterprises, I see a movement more and more that networking, storage, virtualization, you name it as being abstracted away to a point where people don’t care anymore. If it’s in house or cloud or 10 years old or five years old or brand spanking new, it doesn’t matter anymore.

Ned:                            Yeah, I agree with the idea of consuming and getting that perspective, stepping away from being like neck deep in the technology.

Kevin:                          Yeah. Getting away from the nitty gritty.

Ned:                            It does help to give you some perspective in the same way that like when software development teams are working on a new product, end user feedback is super useful because you might assume a whole bunch of things about the steps that a user might take or how they’re going to interact with it, but once you actually put that software or that interface in front of a user, you suddenly realize all the assumptions that you made about that technology that that user is coming with a totally different set of assumptions. It’s sort of like the first time I ever put together a like lunch and learn that was lab oriented and had people try to follow along a lab.

Ned:                            It was a disaster. It was not because the lab was bad or not because I was bad at presenting. I mean I wasn’t the greatest presenter in the world, but the main problem was I had made assumptions about how people were going to interact with the lab, that they were going to read the steps. They were going to follow the directions, that I was being clear about how I was explaining things. After sitting down with like four or five people in a row who misunderstood different portions of the lab and how to do it, I was like, “Oh, I really need to user test this kind of thing before I assume that a class of 20 is just going to look at my lab guy and be like, this is perfect. I have no problems”.

Ned:                            I think when it comes to these advanced technologies and breaking down silos, it’s sort of have the storage team talk to the networking team and understand how the networking team sees what the storage team does and how the two are interfacing with each other. It doesn’t have to necessarily be super formal.

Ned:                            Just having two network guys and two storage guys or gals going out to lunch and shooting the breeze, they’re going to learn a lot even if it’s not highly structured, maybe even more if it’s not highly structured. They don’t feel like it’s an assignment that they have to do. It’s just let’s be somewhat social. But you do get back to the problem that a lot of people in technology, ourselves excluded, because we’re all in this. But a lot of people are very introverted and don’t like to have conversations with people they don’t know. So you do have that kind of working against you, if you do go talk to that storage person and be like, “Hey let’s go grab lunch.” They’re like, “No, I sit at my desk and I eat cup noodle and don’t talk to me.”

Noel Reynolds:          That’s right.

Ned:                            Okay, you’re only going to get so far, right? But hopefully there’s some synergy across the different silos. If you do reach out that they’ll reciprocate.

Noel Reynolds:          Yeah. You’re reminding me of two things that Kevin and I have joked for quite a while. So one, we joke that we can’t wait to work for a vendor because, or sorry, we can’t wait to not work for a vendor and to work for a customer again because we’re going to make the vendors lives hell. One of the things that occurred to me is if you do have a vendor for virtualization or storage or whatever, you can leverage that vendor to come in so that you don’t have to do it and have them do a basic presentation to your customers. Vendors will do a lot more, I think in some cases than we realize. You have a lot of leverage, especially if there is a sale coming up. You have a lot of leverage to get vendors to do more for you and for your team.

Noel Reynolds:          So we also joke, you know as necessarily like we work in sales, right? So we work in sales for a vendor we have for a long time. We have to be professionally extroverted, right? Personally he and I are the same way. We try, we’d rather just hang out and not necessarily eat the cup of noodles but sit at our desk or whatever and read. But you sort of have to push yourself to be out there. I actually think it’s a career skill for anyone in our field, right?

Noel Reynolds:          The communication part of it is a big part of what we do. It’s not good enough for you to just sit at your desk and eat cup of noodles. You should be pushing yourself, even if it’s uncomfortable to go out there and talk to other people. I think that that’s, yeah, there’s going to be people who, it’s just really difficult for them and they’re not able to do it. But your career will be better off if you can represent your ideas up the chain of command to your leadership or and even to your subordinates or to people at your level. I mean, the more that you’re able to communicate and share the better off you’re going to be.

Ned:                            I think your life just in general might be a little bit better.

Noel Reynolds:          Yeah.

Ned:                            When you need to go ask for something, having that preexisting relationship with the networking team means it’s going to be that much easier to get the thing you need done. Next time you need to garner support from a bunch of teams to get a project off the ground or you want to prove your innocence, people a little more likely to believe you if they know who you are versus you’re some stranger who like, sits at their desk never talks to anyone, is kind of squirrely who’ll be like, “I don’t trust that guy. It’s probably something he did.”

Ned:                            Yeah. I agree with the fact that like a lot of the times I would rather just sit in my basement, well I am sitting in my basemen. But I’d rather just sit in my basement and write posts or code or anything else, but interact with another human being, especially after like a week of a conference or something. But I acknowledge the fact that if I don’t get out, if I don’t talk to people, it stagnates my learning and makes my life slightly more difficult. So I guess suck it up, and people aren’t that scary. They’re really not, mostly nice.

Kevin:                          No, I’m not. Most of them. Most of them.

Ned:                            Most of them.

Noel Reynolds:          That’s the other thing I always say, I’m like, “This is only work, guys.” I mean luckily, most of us, I’m assuming don’t work in a life or death kind of situation on a daily basis. I always just have to tell myself, I get stressed out or I get overwhelmed. I’m like, “Man, this is just work, right?” It’s not the end of the world if this doesn’t get done or if this happens or that happens. It’s just work. Now granted, you got to take that with a grain of salt, right? Yeah. You’re working in customer support and maybe a little bit different story, right?

Ned:                            Yes. Way, way true.

Noel Reynolds:          But again, maybe it’s just a matter of reminding your customers that, “Hey buddy, this is just work, right?” I mean there’s things like-

Ned:                            You’d like to have.

Noel Reynolds:          Yeah, exactly.

Joep :                          If I could say that to our healthcare customers, I think they will shoot us back.

Ned:                            That’s what I’m saying. Obviously there’s different situations and scenarios, but by and far for the most part, what we’re doing here is extremely important in a lot of cases. But the level of stress and the animosity we get in these silos and the walls that we build and the conflicts that are created, it’s almost comical when we take a step back and look at it.

Dana:                          Well, and also if people’s lives really are on the line for this kind of work and this kind of collaboration, then definitely get up and go talk to somebody.

Ned:                            Yeah. Even more the reason, right? Yeah.

Noel Reynolds:          Totally.

Dana:                          So I think we covered that middle bullet pretty well. Like if there’s no more tips or anything, any pieces of advice that we can give around that, talked about automation and Dev ops, but I’m wondering, Ned, is there else on the outside that you want to cover or are we good?

Ned:                            Well, I think we’re good. In the lead up to this, the preparation, I did start picking Noel’s brain a little bit about the world of DNSSEC. So do you want me to get into that a little bit? But kind of what’s going on with DNSSEC, what’s going on with DOH and DOT? What’s the intersection there? I’m just curious about other people’s reactions and opinions because I don’t think I have enough information to make an informed opinion about any of those things.

Noel Reynolds:          Sure. So why don’t we tackle DOH and DOT first. Again, I’m going to stay reasonably high level. You can go online and find a video of me saying that I think DOH and DOT or DOH specifically is a bad idea. DOH seems like it might win the war between DOH and DOT in terms of what’s going. So DOH is DNS over HTTPS. DOT is DNS over TLS. They work differently, but they do something similar, which is they encrypt the DNS quarries and query responses from endpoints to DNS servers. DNSSEC is a little bit different in that DNSSEC is a protocol that allows you, so it’s an extension to a DNS server that allows you to establish a chain of trust so that the end point when it makes a DNS query can ensure that the response that it’s given is, and it does it, DNS, a fully qualified domain name is made up of if we look at www.TechfieldDay.com, the dot-com is the top level domain.

Noel Reynolds:          Then you have the Techfield Day, which is the domain. And then WWW is either normally an alias or it could be a host name, but it’s kind of a chain of records there. If you are working in an environment where you don’t have that record cached somewhere, you have to go through the chain. You have to go through the root server, which is the invisible dot at the very end of that and ask him, “Hey, where do I go to find the dot com server?” Then you have to find the dot com server. DNSSEC allows you to trust that at every point along the way, a server that is responding to you and giving you an answer back is a trusted server and is authoritative for those records and is the real server for those records. Not someone who’s a man in the middle who’s pretending to be there.

Noel Reynolds:          The two technologies are a little bit different and DNSSEC is one that’s been around for 25 years. I want to say, something like that. If you were to go and grab DNS, and bind, and crack it open, and figure out how to, how to configure DNSSEC, I think it can be very complicated. It seems complicated depending on what server platform you’re using, what vendor you might be using to do DNS. I had that experience where I actually went to configure it by scratch, from scratch. It was really, really difficult. I think that that put a lot of people off from using it for the most part.

Amar:                          So I concur.

Noel Reynolds:          Throw them off of that.

Amar:                          I think with DNSSEC and configuration wise and what all you need, I think people find it is often complicated because they have to manage DNSSEC data. But if you are one of those DNS service providers that don’t host any of the data yourself and the whole function of your servers is to go and fetch it from somewhere else, the contribution wise you strip down landing and percent of what’s needed from that bind book to turn on the DNSSEC. It will do the same thing that Noel just talked about. They start from the DOT, go to dot com, go to Techfield Day.

Amar:                          It keeps on validating all the way down and until find the response and sends back. But like Noel said, the thing that puts people off is hosting being a sec sign data on the external server side, which is often complicated. Then there’s a lot of vendors in space, in the space that can simplify things for you. But even after that, you have to then own the yearly, the monthly rotations of keys and things of that sort, which is still a manual work that’s required by the system admins.

Noel Reynolds:          There’s a lot of I would say nomenclature and components, right? I’m not going to explain what all these are, but when you start, KSK, RRC, signed zones, like signed records, what happens when you add a new record to a zone. All of that stuff is taken care of. But honestly, if you go watch a video on YouTube, it seems significantly more complicated than it has to be from a configuration perspective.

Ned:                            It was that just built into the spec that way or is that the sort of thing that it could have been made easier and more simple, and still just as secure? It just never really reached the adoption point where it would become simple?

Amar:                          I think adoption wise, at some point, the US government mandated that everything, every agency that falls into the US government umbrella has to have the external DNS data signed by DNSSEC, there’s no other way. But DNSSEC, if you go and read history on it, it’s primarily made to solve two-use cases. One which was the bigger one is it needs to be backward compatible. Hence why there’s a lot of stuff that they have to do around DNS to make it not break DNS because a lot of the legacy systems don’t understand the DNSSEC today.

Amar:                          So when you introduce DNSSEC, it adds additional headers, adds additional records. We tried to make it simple for the systems, but we made it complicated for assisted names. I think at that time, when DNSSEC came out, the intention was never doing encrypt the path between the servers and the clients. It was more so they can validate the data itself. I don’t think cybersecurity was at its prime when this came out. So it was never a thought I would say that that data encryption is a thing that it needs to be done on DNS as well.

Ned:                            Yeah. That seems to have plagued many internet thing in its time. It was never designed with security in mind back in the day. That all had to kind of be bolted on later.

Amar:                          Yeah.

Ned:                            Is there anything different about, and I don’t know if anybody knows this, but I know when they were designing IPV6 that was much more, okay, security was in mind because it’s a little more modern and so there’s some security mechanisms just built into IPV6 that IPV4 never had is there anything similar with IPV6 for DNSSEC or is DNSSEC just doesn’t care about which of the two is being used?

Amar:                          DNSSEC generally doesn’t care, but what DNSSEC does is it will sign the like type records with one hashing of signature that all your A records, all your quality of records will use as key. So DNS works above the IP stack, so it doesn’t care if it’s IP before initiative IPV6 initiated. It does work the same on IPV4 as IPV6.

Ned:                            Okay, cool. Yeah, so I guess that probably answers most of my questions about DNSSEC, which means I probably won’t have to learn a whole lot more about it unless for some reason, it gains mass adoption out of nowhere.

Noel Reynolds:          I would say, you know one other thing about DNSSEC and Amar spoke to kind of the backward compatibility aspect of it. You’re unlikely to have a problem with it. It’s unlikely to cause you a problem. I think the biggest problem is when you want to do what’s called DNSSEC validation. You want to make sure that DNSSEC is happening to a record that you’re querying and it’s not.

Noel Reynolds:          That’s where you can … If you’re a user of DNS, that’s where you can have a problem. If you’re an administrator of DNS, it’s a different story. Then you have to do some troubleshooting and figure out where the trust chain is broken or something like that. But that’s the other kind of reason why DNSSEC is not maybe as thought about because you don’t really necessarily have to worry about it unless you really care that you’re validating those records.

Ned:                            Right. That kind of reminded me of something else that I think happened maybe this year or last year. I remember just reading a Blurb about it, DNS day where the day when DNS might break or something.

Noel Reynolds:          DNS flag day.

Ned:                            Flag day, that’s what it was. What was that all about? Because you mentioned backwards compatibility. I feel like that had something to do with like forcing everybody to a newer or a slightly newer standard of DNS.

Noel Reynolds:          So that covered that really well on one of the Packet Pusher’s episodes. I would have to find it for you. But yes, flag day. So DNS is a 35-year-old protocol. I think it’s 35. I’d have to go look, but I think it was 1984, 1985 was when DNS was first kind of put into play and different models of DNS had existed before that as well, primarily host files or distributed host files, things like that.

Noel Reynolds:          But because of the age of the protocol, it necessarily has had layers and layers and layers of additional things put on it including something called extensions to DNS or EDNS0. There are many, many servers out on the internet that don’t support all of the latest functionality from a DNS perspective. So what DNS flag day was a day to say, I can’t remember if it was to specifically test or to test and say that, yes, your DNS server support all of the latest extensions that can respond to queries in the appropriate way for a modern client. Then I think Amar is like dying to speak. I’m going to shut up for a second.

Amar:                          So I think this ties back to DNSSEC a little bit, that since the DNSSEC was introduced into DNS, I think we spoke at the last, the briefing. That it causes more stress on the network, that now your network devices that now that DNS is 512 bytes, DNS is not 512 bytes anymore, it’s much bigger than that. One of the reasons why this is important is that with introduction of DNSSEC, EDNS which is extensions to DNS mechanism, allowed for DNS to still operate on a UDP, which is TCP because that would have cost much more problems on not just now the networking but also on the servers themselves that are now operating on PCP versus UDP. Imagine if your DNS server is a 50,000, 20 per second server, now you’re causing, looking at trouble.

Amar:                          So they introduced EDNS to allow for DNSSEC, and the flags to propagate easily. So I think the flag day was there to talk about the EDNS has helped us get this far, but we now need to make it more stricter to allow for better compliance. So that’s when they introduced the version for EDNS1. Most of the software providers were now releasing patches to comply with EDSN1. But it was also going to break EDNS0, which is what’s in use in legacy and current systems today.

Ned:                            Okay. Yeah. So breaking can be bad.

Amar:                          Yeah.

Ned:                            Yeah. So I’ve got to imagine like this push for, well, DOH seems to be the thing that’s going to probably win out longterm is that’s going to force another upgrade if ISP use and these different providers want to support that protocol.

Noel Reynolds:          Absolutely. That’s probably sort of my biggest concern with DOH right now and why it’s … So DOH, and I’m going to call it that, but DNS over HTTPS is, the idea is that your DNS queries will now be encrypted like all the other traffic that you have. If you’ve ever been at a coffee shop or something like that, I think we’re all aware that when I’m going to my bank’s website, I’m using HTTPS so it’s encrypted. So it’s not that big of a deal if I’m not using a VPN or something like that which encrypts sort of all my traffic off of my end point. But the DNS queries are in clear text. If you were to look at any person’s DNS queries over the course of a full day, let’s say for all their devices, you could learn a lot about that individual, right?

Noel Reynolds:          When you start to, if you’re an ISP or you’re someone who’s hosting DNS servers on the internet and you’re looking at an individual’s queries in aggregate, you’re seeing a whole bunch of data that can be useful to advertisers. It can be useful to lots, lots of people. It can be useful to some kind of a foreign actor who’s trying to do some kind of espionage. I mean there’s just lots of uses for it. The challenge is is that DNS has always been kind of playing text. Individual DNS query, you probably don’t care about, it’s not a big deal. But again, in aggregate, it’s a challenge. And so the idea is, “Hey, let’s encrypt this traffic.” Everything else is encrypted now. Again, that’s a good idea. I like that idea. That’s fine. The problem that I have is right now, the number of DOH servers that are available on the internet.

Noel Reynolds:          So when I have an endpoint that and I say, “I’m going to use DNS over HTTPS on this endpoint,” first of all, almost definitely none of my internal servers are going to be DOH enabled. So there’s going to have to be some configuration done there. Microsoft just recently announced that they’re going to make all of their endpoints use DNS over HTTPS in the future. Yet their DNS server doesn’t support DOH. ISC’s bind does not support DOH right now. So as a consumer, as an end user, right? Because again, it’s not going to work in an internal corporate environment today or in the next five years based on the way that some organizations upgrade. So I only have a handful of choices about who I want to point DNS to.

Noel Reynolds:          In addition to that, specifically DNS over HTTPS, I only have a handful of choices. I can use CloudFlare, I can use Google, whatever. So if the advertisement to me as a consumer is, “Hey, this is going to help your privacy” yeah, I’m not going to point to my ISP who maybe I trust, maybe I don’t, whatever. But I’m going to point to Google or I’m going to point the CloudFlare or I’m going to point to some of the organization. So the problem that I have with it is I do think this is what’s going to happen. It’s for sure DNS is going to be encrypted like every other protocol. There’s no question. But what’s frustrating for me right now is they’re selling it as a privacy solution. Yet there just aren’t enough choices for me to feel like I really have privacy.

Noel Reynolds:          I would almost rather, and I actually I would rather just get a VPN client, encrypt all of my traffic and point to again, so I want to do something. I’m going to point to one of my VPN points of presence. Then I’m going to disconnect point to a different one and do something different. I mean if I’m really concerned about my privacy, I’m going to do something like that rather than continue. Again, that’s only assuming that I’ve done some research and trust my VPN provider, right? Because that’s a whole other discussion. But that’s my frustration right now is I just feel like it’s sold as privacy and yet you have a very, very limited choice.. It’s not like you can host your own DNS over HTTPS server at your house today.

Amar:                          To me, I take it on that same kind of thought process, but I take it a little bit even higher than that. Right? So again, Noel, going back to your point about what you can see in, in what information over the course of the day, your DNS records can divulge, right? The way this is being introduced is a user-based privacy thing, right? It’s based off of our browser, right?

Amar:                          What about the 70 other applications running on my device that are all making DNS queries that now are not encrypted, right? I’m all for encrypting the traffic. To your point, VPN, let’s go, right? That’s the way to kind of look at this because now I’m getting whole box type encryption for lack of a better term, right? All the applications, everything running on my device is now under the same level of protection versus just my browser. Tat browser, I bet if you go back and look at my DNS queries throughout the day, very few of them, I would say a very small percentage of them are actually coming from my browser.

Amar:                          I’m not making a ton of DNS requests through a browser only. If you look at say the thousand that come through in an hour from my one MacBook, a fraction of them are from my browser. The rest of them are from all the other applications and system things that are making DNS calls. So that’s where I kind of look at it like I’m good with the idea like you said of encrypting everything, right? Everything else is already encrypted. Why not? But it’s just the approach seems to be very consumer-centric right now based on user browser type usage versus just the true security aspect of it.

Noel Reynolds:          And the other way where it’s consumer centric and maybe a little bit concerning for enterprises is the idea, and again, this is one of the things that’s going to get worked out in the detail. But for right now, Ned, I was listening to, I forget which one of your podcasts. But you had a gentleman on who was talking about using DNS in the cloud environment to get actual data on what the endpoints are doing because everything else was encrypted. But DNS was something that they could see so they could track what was going on.

Ned:                            Mm-hmm (affirmative).

Noel Reynolds:          We see corporate organizations mining DNS data because it’s in plain text. I mean they’re going to have some, need to have necessarily some other view into the DNS data because they’re using that DNS data to understand what the endpoints are doing, to understand whether there’s a potential security issue to understand.

Noel Reynolds:          I mean Cisco’s purchased open DNS based on the fact that they could use the DNS intelligence to block malware and lock threats and things like that. The concern that I have is if everything is an HTTPS, if you’re looking at an endpoint and you’d have no way to decrypt it or to know which HTTPS traffic is browser based to a website versus DNS-based to a DNS server. I mean, I think as employees, sometimes you hear concern about the fact that the organization has put a proxy server in place that does a man in the middle before data is going out to the internet because now you don’t have any privacy in your corporate environment. Look, in a lot of environments, that makes a lot of sense. But now you’re going to have to have bigger proxy servers or bigger devices that can decrypt, reencrypt, right? Which is going to add latency to what you’re doing just so that you can look at that traffic.

Noel Reynolds:          So I think that there are some challenges that we need to solve and maybe. Again, I can think of a lot of ways we could solve those problems. Maybe internally it’s plain text DNS until you get to your internal DOH server that then those DOH resolution out to the internet or something like that, so you’re encrypting only one time, and it’s a little bit safer. I mean, again, that’s to be determined, right? There are different organizations that are working to push these standards forward. But I just think right now, there’s this massive push for privacy, by the way, from a bunch of people who are still using Instagram, and Facebook, and Twitter, and whatever. But there’s this massive push for privacy. But at the same time, there are implications to doing this that I don’t think have been completely resolved. So …

Ned:                            It reminds me a lot of what happened when iPhone started moving from the personal into the corporate, right? The corporate world was not ready to deal with iPhones, and mobile devices, and being able to properly administrate them, lock them down. A whole bunch of solutions came out to handle mobile device management and mobile application management because there was nothing in place. Just people unaware but when your CEO walked in with his brand new iPhone and iPad and was like, “These need to work on the corporate network.” You’re like, “Okay, I guess that’s what I’m doing.” So this is kind of a similar thing. It’s being pushed by Google, by adding that in Chrome by default. I don’t know if it’s enabled by default but it’s definitely there by default. Then a Firefox has it there and enabled by default, I think with the latest version.

Ned:                            So it’s definitely being pushed from the consumer space into the corporate space and now once again, somebody is going to have to figure out how to deal with that. It’s going to be a company that’s somewhat nimble or maybe a brand new company that sees an opportunity in the gap there. So a startup idea. Anybody? Anybody want to start stuff?

Joep :                          The funny thing that you reminded me, Ned, about the iPhone working in the DNS space currently, I know how many CIS admins that manage DNS KPI forms. Every single one of them, they come with the same names, Iphone.org and you end up with the iPhone record, the links are now 60 IPs. You cannot tell who’s who.

Ned:                            Yeah, that is something up to the, if you have at least have it under MDM, you can like force a rename over the phone I think.

Noel Reynolds:          Right. Yeah, I was just going to say that that’s that Mac dream that that has never been realized I think. Right? Because that would solve that problem.

Ned:                            I’ve seen like one organization actually successfully get knack set up and working, but it was very fragile, like don’t sneeze at it because …

Noel Reynolds:          Yeah, right. It’s very much like DLP. Very, very much like DLP, right? Once you get it set up, then the company can’t culture, the stomach’s just not there for it. So it quickly gets, the tabage.

Ned:                            Well, yeah. People would just find ways around it. If you make any security control to onerous, people will be like, “Well, why can’t you go around this way?” What can you do?

Amar:                          One of the things, I think you all touched on the DOF and the different implications of putting it out now. We’re not ready for it. The other one that I was reading upon just a couple of days ago was a lot of the malware detection today by the cyber security teams that are inside a corporate network, they do a lot of their work through DNS. The moment you turn encryption on on DNS, you can save by view, finding out who, which machine is playing which malware because everything is unencrypted. It’s hard to decode and put back. Now, there you need to respond with the minutes. Somehow RASOLA that are way more stricter than the others. Now you’re avoiding all those RASOLA because you need to … Your system was slowing you down now versus making you faster.

Dana:                          I think there’s this other additional point of like centralization of the internet if only certain actors are able to encrypt those queries, and one point of failure if one of them goes down.

Noel Reynolds:          Right, right, right. That’s where its kind of funny. CloudFlare’s is known as an organization that values privacy, that deletes records, that doesn’t keep records, things like that. It reminds me of a conversation I was having with one of my good friends about my iOS device, right? I love my iPhone. You’ll take it from my cold, dead hands. ButI trust Apple necessarily, right? Everything that I have is in iCloud from like all my email, et cetera, et cetera. Right now, Apple, when they have like a security blip where they’re recording conversations, whatever, I don’t worry so much because I don’t think that they care.

Noel Reynolds:          I actually had conversations with folks at Apple. I know they don’t care about my personal data or about selling my data or whatever. That’s fine right now. But what happens when Tim Cook is no longer the CEO and they get a different board? The board wants to do more mining of customer data or something like that. So it’s like all of these organizations that we’re dealing with are publicly traded for profit companies. At some point in time, if CloudFlare, and again, I’m not saying anything bad about CloudFlare, I like what they do, but maybe they change in the future. As a consumer, where if all my data of some kind is going toward one organization, what’s the impact of that on me if their ethics change?

Ned:                            That’s true. I mean, I guess to a certain point, a lot of us are just kind of married to the device we have. It’s hard to get away from it, for better or for worse. But yeah, I guess the solution is if the stance of Apple changes and you’re an Apple consumer, you could go somewhere else. But do you necessarily trust Android anymore or a well or Android, well, or Android or wait. It’s just the two of them.

Noel Reynolds:          Right. Exactly. Exactly. Yeah. And The other thing too is what’s the … I mean, for you and I, that’s not hard, right? For you and I, that would be easy. I mean and when I say easy, relatively I wouldn’t want to move all my digital life out of the services that I get from Apple, right? I’ve got photos there and whatever or my mom or for your mom or for grandma or your friend who works in marketing, right? Hi Dana.

Noel Reynolds:          It might be a lot more difficult because they don’t have that kind of technical knowledge, that technical background. So our digital privacy is something that really concerns me. Our kids are growing up in an environment where their photos and images have been shared since day one. I know that a lot of the, I’m not going to use names, but I know that social media companies, regardless of whether you have an account or whether you don’t or whatever, they are tracking you as an individual with a serial number the first time that your photo appears on one of their services. They do that so that when you finally register an account, because you get a certain age, you immediately have all your friends that are there. You can be friends with them. So it is something that I get concerned about. Sorry. I know it’s not like necessarily core to what we were talking about, but it’s related for sure.

Ned:                            Yeah. No, I agree.

Dana:                          These are almost the costs that we accept as part of living life where we do, and the way that we like to, and the way that the people we love do.

Ned:                            Yep.

Dana:                          We kind of wait to see how it plays out. So that was really interesting. I found personally that I’m curious in the last five minutes, A, is there anything that you feel like we should have brought up that we didn’t bring up? Is there a question that you have that we can either answer right now or just going forward to think about? For one, we’re talking about different people with different skill sets and different ways that they see their organization’s technology.

Ned:                            I had a question, right? Yup seems to really like, and I’m putting words in your mouth right now, so like feel free to tell me I misheard you or whatever. But you seem to really love managed service providers.

Dana:                          Yeah.

Joep :                          I love IT as a service. So having clearly defined boundaries between different teams and instead of just doing a technology thing, thinking about how technology is consumed wherever you are, if you are at an MSP, a cloud provider somewhere within your organization. But that whole thinking around a service boundary is really interesting to me. Yeah, definitely.

Ned:                            So you just want the IT to be consumed as a service. You don’t care if it’s an outside organization. Because I can tell you that the organizations that I’ve spoken to have had managed service provider relationships in general. I mean, I would love to find one, but I have yet to find one who’s real happy with the service that they’re getting ever. There’s been a lot of different organizations with, and I’m not going to mention names, but …

Noel Reynolds:          That’s the thing that surprises me. I agree. IT needs to be consumed and offered as a service for sure. But it’s just outside organizations tend to … It’s like your neighbor watching your garden and they water it. But they don’t ever weed and they don’t ever put any care into it. That’s been a challenge that I’ve seen with every customer that I’ve ever seen in that kind of an environment.

Joep :                          Yeah. I’m not necessarily talking about a managed service. I’m really talking about the service resource, whatever you need.

Ned:                            Okay. Cool.

Joep :                          Yep. Because I come from a MSP background. I used to be CTO of a company like that. Yes, definitely we took care of other people’s loans, and they were horrible.

Ned:                            Yeah. By the way, I have one of my best friends works at Atos. He’s told me that there’s a customer side of it too, right? You try to hammer out the right T’s and C’s with them to make sure, and the right SLAs to make sure that you’re doing what needs to be done. A lot of times, the customer either doesn’t want to pay for it or it doesn’t want to … It can be a real challenge. So that’s, yeah. It is what it is.

Kevin:                          Well, and that’s again why standardizing that interface between whoever it is. If it’s two different companies, if it’s an MSP and a client or within the company, I mean one thing that public cloud has given us is that a very standardized way of consuming services. There’s just no contract to be negotiated. It’s the same thing, I’m consuming whatever it is, a database, a web server, a whatever as a service. It’s just the ability to either find someone who’ll manage it for you all or you do it yourself. But public cloud has given us that very strict interface separating the two.

Ned:                            I think you can make a strong argument that public cloud has done absolutely amazing things for just the on premises IT teams in terms of how they think about and how they offer services.

Kevin:                          Yep. Yeah. Again that’s why the whole dev ops transformation for a lot of companies is so hard because it requires them to shift this thinking. This is the major bottleneck companies go up against because they have tens or hundreds of people that are not in this mindset but are in a role that will change significantly because of this.

Dana:                          Okay. If there’s nothing else, then I feel like we’re literally right on time.

Noel Reynolds:          Well, that was fun.

Ned:                            Yeah? That’s good to know.

Dana:                          Yeah.

Noel Reynolds:          Yeah, thank you, guys.

Kevin:                          Yeah, I appreciate it.