John Capobianco • 00:00

From Insight to Impact, closing the loop on Network Ops with Selector and Itential with John Cabobianco, head of DevRel at Selector, and my really good friend William Collins from Itential. So let me get William up on the screen here. Thank you so much for joining us, everyone. I know there’s a lot of interest in this. This has really been beyond hype. We did want to announce this and make noise, so we partnered up at Cisco Live. You’ve seen lots of social media posts. We even had foam Legos of different colors with our logos that fit together to really drive home this idea of closing the loop, but also our partnership and friendship with each other at Selector and Itential that we see an opportunity for us to solve some real challenges together with our solutions and that, you know, the sum is greater than the parts and that holistically we can really help drive you know, fully autonomous networks, and you’re gonna see that today. William, why don’t you say hello and introduce yourself?

William Collins • 01:01

Hey, everyone. So, yeah, my name’s William, and I work as Director of Tech Evangelism at Itential, and I’m so excited about this. We’re just off the excitement of Cisco Live, and, you know, this really, I think, it was back at the end of May, where Itential and Selector made the announcement about this strategic, you know, partnership, and the partnership takes, it does so many cool things, and I’ll try not to get ahead of myself before we get there, but yeah, super excited to be here and talk through this and really get into the nuts and bolts and, you know, demonstrate it, you know, not just slideware, but actual doing.

John Capobianco • 01:43

Yeah, before we even started at Cisco Live, I was literally there before anything opened. Our booths weren’t even set up yet and someone said, you know, would you mind taking a minute to show me this, I tend to close the loop automation. So we’re not going to kill you with slides where, but we do want to set the stage and kind of get into some ideas here first. So what are the challenges? And we have shared customers, William and I together probably have close to 50 years of experience in the field, operating networks, designing clouds, doing security, doing operations and event analysis and remediation, and it really, you know, resolving P1 incidents right at being an escalation point for our enterprise. So we’re stuck, reactive firefighting, as the slide says. We’re chasing alerts, we’re creating tickets, we’re manually executing remediations that feel like Band-Aids and are Band-Aids, and those one-off unicorn treating the network like pets instead of cattle, right?

John Capobianco • 02:47

It doesn’t scale, and it’s in the legacy mentality. We’ve gone through a phase of network automation, and now we’re into AIOps. So together, we can, you know, these manual processes have become a closed loop event-driven workflow. William, what kind of excites you about this coming from the field?

William Collins • 03:07

Yeah, so just going back to my days is just a network engineers. If you think about, You know, something that, um, as you become like a senior network engineer, you become like more senior in the technical staff, like you always contend with, Hey, you know, especially if you work in a highly regulated industry like healthcare and, you know, then services, but for the, for the longest time, the, the it industry place, the cost of an outage, you can find blogs galore probably, but it was somewhere around like 5,000 per minute is like the cost of an outage. Now, I remember seeing that for years and years and years, super long lived statistic, but in reality, this number is much higher. So there’s been recent findings from like EMA research and others, um, you know, back in 2022 that placed the average cost at, I don’t remember the exact number, but somewhere North of, uh, 12,000 for, for every minute of unplanned downtime, you know, with that number really rising to somewhere North of 14,000 per minute. And even as high as like, if you’re a large enterprise, like the big, big ones, I mean, I imagine it’s 25 to 30 per minute for like these, you know, unplanned outages and incidents. And you know what that means for like a large organization out there is every minute of troubleshooting and triage really counts.

William Collins • 04:34

Um, you know, moreover, there’s this. I think there’s a big divide in actually, okay, like we’ve got to find the problem. We have to really understand what it is before we can begin fixing it. Then executing the fix is a completely different thing altogether. Basically, combining these two areas in a timely manner is extremely challenging. My excitement stems from- I don’t think- No, I want to hear why you’re excited about this.

John Capobianco • 05:08

You were just breaching a pinnacle and I jumped in and cut you off.

William Collins • 05:12

I was going to say the thing that excites me is like the, I know that when we think of like all these AI things, we, I mean, all the stuff that you do on a day-to-day basis, I see that and I’m like, wow, that’s as far as like AI on the planet, that is the coolest, like sexiest stuff in the world when it comes to infrastructure and AI, it’s awesome. One of the things that excites me along with that is also the mundane things that like we’ve just done year after year after year, where like there’s all these troubleshooting steps that you have to take. And it’s like simple, like unsexy things that we’ve been dealing with for years and years and years, like port flapping, neighbor flapping, things like that. So having a closed loop approach as far as like reacting to these incidents is just super exciting to me, make these easier for the operators.

John Capobianco • 06:00

Well, yeah, and we can’t scale, we can’t just keep throwing humans at these problems. These are complex problems, complex systems. It takes months, weeks, months, years. for operators to really understand the infrastructure if they’re a new hire or if they’re a junior and working their way up. You know, not all of us are the senior network architect and know every IP address and every host name and every ACL and every little nook and cranny of a network. There are some people that are like that. Those people can also take advantage of AI, right?

John Capobianco • 06:28

It’s a democratic sliding scale. It’s going to help everyone from juniors to seniors all the way through the spectrum of experience and, you know, expertise. You know, I’ve had pressure, speaking of that downtime per minute. These could be life-saving services. These could be political or government apparatuses in the public sector, right? With tens of thousands of constituents as we used to call them in Canada, the constituents wanting to consume federal government services, right? And if the infrastructure is down per minute, right, there’s more than just a cost sometime associated with this, the societal impact, right?

John Capobianco • 07:06

But in terms of cost, it might take you two minutes to find the host name or the IP address or the credentials. There’s a lot of friction involved, William, and when we’re talking about tens of thousands of dollars a minute, every minute counts, every second counts. Quality is hard to maintain during those pressure situations. Identifying the true root cause and maybe not a symptom is difficult when we have cascading failures, where’s the true root of this problem, right? I don’t want to spend an hour troubleshooting an OSPF log if the problem is a BGP problem upstream, right? Now, as humans, that’s tough to do. We are using artificial intelligence, we as in selector and attentional, as tools, as actual problem-solving tools, as a value-add, as the promise of this technology.

John Capobianco • 07:59

We didn’t do it just for the sake of doing it. We didn’t just bolt on, you know, a chat interface to an existing solution. This is from the ground up, and our teams, our developer teams have been collaborating over the past few weeks. Microsoft Mechanics www.microsoft.com to work on this idea of insight to action. And individually, our platforms are extremely good, best in breed at the insight and the action. So why not bring them together, right, William?

John Capobianco • 08:28

Does it not make sense for us to just collectively and collaborate and bring the platforms capabilities together to drive this closed loop automation?

William Collins • 08:40

Yeah, it’s a really natural fit for sure. And we’ve seen that validated both by the interest of just industry folks out there, but also our customers are super interested. And I think it would be pertinent, I guess, to define. So I’ve had a few different questions that I’ve gotten as far as our partnership and what we’re doing. And I think the most asked or maybe the second most asked question that I’ve gotten anyway is like, what do you mean by closed loop automation? Can you define that? What does that mean to you?

William Collins • 09:14

And I’m gonna take a stab here and then get your thoughts on if I hit the mark or what you would change. So, If I had to define closed loop automation, I would describe it as a process where a system automatically does, it monitors, it analyzes, and it adjusts operations with zero or minimal human intervention using feedback, I guess, to optimize performance and success. So maybe that system collects data from sensors or software, and software in our case. It processes it to detect, looking for the right word, deviations. So detect deviations, anomalies in behavior or inefficiencies or what have you, and then executes corrective action based on predefined rules or algorithms, kind of like a self-regulating loop of sorts that continuously improves outcomes over time.

John Capobianco • 10:24

Well, I think that’s a pretty good stab at it, William. I like that definition. I have a similar take. I like to think of it as maybe a more sophisticated version of self-healing networks. We’ve heard a lot about software-defined networks and the promise of software-defined was we could programmatically interface with these. You know, the evolution is now adding artificial intelligence and generative AI and other components of machine learning into this mix of not only self-healing, William, but like you said, validating and notifying and creating ServiceNow tickets or ITSM incidents, updating CMDBs, referencing a source of truth. There’s more to this than just this idea of a self-healing network.

John Capobianco • 11:04

I kind of like the idea. It’s a bit of a paradox here, but we’re going to kick off the first domino. We all understand a greenfield network. We’ve put all of our configs, we’ve plugged in all our wires, and we have more or less a human expectation of the services and quality. And based on, like you said, sensors and metrics and logs and information and events, we can judge the health of this network. I like this idea of hands-free, of hands-off. Once I kick the first domino off and apply my configs and plug things in, the system can maintain itself and work with me with this new AI approach in natural language.

John Capobianco • 11:45

It can tell me through words I can understand, not some syntax or some CLI or some language. that I need to study to understand. It will literally say, right, there is an interface shut that shouldn’t be shut. Would you like me to address this? So this is really exciting in that it’s a closed loop with the feedback, right? It’s not something that we have to log into and peer into. It will naturally create those artifacts to give us a level of trust.

John Capobianco • 12:18

I think when we’re gonna hand over this type of power and responsibility to AI agents and an agentic system, we definitely have to have trust and we have to have artifacts and timestamps and all of those wonderful things, right?

William Collins • 12:35

Absolutely. The guardrails become so important because it’s like, Hey, we, you know, just like in the pre MCP days, you know, really, if you were building out or trying to build out integrations to LLMs, it was like on a very piecemeal basis, like this, not only this LLM provider or this model, but individual things to individual resources. So you really, you know, got to this integration craziness where it was like very hard to integrate because everything was so custom. And then, you know, throwing guardrails around it, guardrails around it became even more challenging, especially doing the predefined guardrails that you already had built out based on your governance framework of whatever industry or whatever it is that you were doing in your business.

John Capobianco • 13:24

Yeah, so let’s not, you know, bury the lead too much here. Let’s get into the demo. Now, I just want to mention something for people to keep in mind for the demo. Every human interaction you see, for example, you’re going to see me right-clicking and clicking a button to send and kick off the workflow, kick that first domino off. That is there for this demo’s purposes. Similarly, William, you’re going to click start workflow a few different times at a few different breakpoints. Those are human breakpoints, right?

John Capobianco • 13:54

None of this needs human intervention. This is purely for demo purposes, right?

William Collins • 13:59

Yes, sir.

John Capobianco • 14:00

Okay, wonderful. So, let me go to my dashboard, and I’m going to give it the stage. And hopefully, the resolution will clear up a little bit. I don’t know why it’s doing this. It will look better in a second. But what we have is we have some device inventory information. This is a Catalyst 8K.

John Capobianco • 14:21

We have the interface information and some inventory information. And we have a timeline that it’s unhealthy. And this is GI 1.112. And down below, we don’t have any itential events. And this is in the last minute. So what I’m going to do is I’m going to bring up the menu here. And what this says is itential remediation service.

John Capobianco • 14:46

So I’ve sent this payload as an API call over to itential. So I’m just going to leave my screen here as it is. And let me remove this from the stage. And now, William, if you want to go ahead, let me put your stage on. Here’s what William is going to see in his itential platform.

William Collins • 15:21

Okay, so, let’s see. So pulling this up, sorry, I’m switching my windows a little bit. Yeah, here it is. Here we go. That’s okay. So basically what we have in itential, basically, we have workflows and we have jobs with workflows. So the moment that John triggered that out, you can see it’s going through.

William Collins • 15:45

I’m going to set this to auto work. So you’re going to see some pop-ups come up. So as soon as he clicked that, it basically triggered, you know, we’re basically going to start out this top task. We’re going to create an incident and, you know, service now. So, yeah, you triggered everything. And the first thing we want to do is we want to begin documenting everything that we do, task by task as we go along. So…

William Collins • 16:10

That was quick, so I’m going to go ahead and click success. We’re ready to move on, and basically this…

John Capobianco • 16:17

Actually, William, if you click on the response there, so everyone, you can see the dropdown here. This is the payload, right? So it is picking up that it’s down, and we violated this rule, and Selector is actually going ahead and giving this to Itential as context to set the stage for the AI to remediate.

William Collins • 16:37

Yes, and kind of one point here is right now, I am a human in the loop, so I’m not saying, hey, you have complete control, do whatever you want, and this is natural for trust with certain things. So this is popping up, and I have to come here, and I have to click success to keep things going. Okay, and now I can approve or reject this change, and basically what’s happened here is we’re using chat GPT in one of these jobs, so GPT received the incident details and generated a structured remediation plan in JSON format. So this is kind of like our… If you think of your senior network engineer who lives on coffee and really doesn’t sleep at all, this job will analyze the incident and create a detailed remediation plan from everything prior. I’m just going to click provision, and then over here to the left, you can see all the output in JSON as well.

John Capobianco • 17:49

Yeah, that was one of my, one thing I liked is all of the cookie crumbs left behind in JSON throughout every step, even the selector payload that came in. So now, right, we can see that it’s up.

William Collins • 18:02

Yeah. And this is a form, you know, this is, you know, again, post checks, pre checks, post checks are really important. So when we receive the payload from selector, of course, we’re going to look and we’re actually going to, you know, check the validity of that, like, yes, this interface is down, we’ve confirmed it. We’re taking these actions. And now, you know, we can see that the interface is back up. So I’m going to click on success.

John Capobianco • 18:27

And I just want to point out back over in my dashboard, it’s back to green and you can see the timeline here from red to green. So it did actually come back online based on this itential remediation.

William Collins • 18:42

Okay, so I’m going to do a proceed and basically what’s going to happen now is, you know, everything that we’ve been doing has been documented in a service now request. Step-by-step, task-by-task, and let’s see this last job’s running. This is going to do basically a root cause analysis of everything that we’ve done and do the final remediation paperwork. And that paperwork is going to be added to the service now incident, but also we’re going to send it back in a payload to selector. So we can see key issues, JSON analysis, root cause, recommendations, etc. So I’m going to finish.

John Capobianco • 19:28

Can we just take a moment to appreciate, imagine coming in Monday and your coffee and you’re checking emails and in your emails is this report that says we had this problem, we fixed it all by ourselves for you, here’s everything we did, here’s all the hyperlinks, here’s the timestamps, the documentation. Like that is what we mean by closing the loop here, right?

William Collins • 19:52

Absolutely. And do you want to show your screen so you can, you should see the output of that change incident as well.

John Capobianco • 20:01

Yeah, I’m just going to let me just refresh my page here and we should have the summary incident at the bottom. So I just have to change this from 30 minutes to the last minute. Everything is green and I should have an intentional summary here. There we go. So there’s the intentional summary from this incident. And we can see, right? Chat GPT troubleshoots the issue and generates a detailed remediation plan, plan generated.

John Capobianco • 20:29

Gemini validates steps, assesses risk and recommends the optimal sequence, validated and optimized. Execute changes with pre post checks, configuration applied. Here’s the root cause analysis documented. This tracks the ticket. So this could take me to service now. And this takes me back to the job over in itential. I just have to authenticate.

John Capobianco • 20:56

So we have a full trace and it takes me to that job from the selector dashboard. And I apologize if this is not looking very good. I’m trying, I’m not sure why that’s not displaying properly. However, a couple of things to point out. Again, this could have been done without any hands. We just fixed the problem and document the problem and validate the problem has been fixed.

John Capobianco • 21:24

The other thing is, this is one example, right, William? We picked a shut interface that shouldn’t be shut as something that is tangible for most network engineers to appreciate and understand. But quite literally, and I don’t, you know, I’m not trying to be hyperbolic. Selector can detect the incident, the root cause of the problem, whatever that may be, BGP, OSPF, ISIS, route reflection, physical interface problem, fiber cuts, et cetera. And Itential platform can document, remediate and validate and document it back into selector. So it’s all tied into either dashboard has all of the information required. It truly is a collaboration between us.

William Collins • 22:11

Yeah, and you saw that I used a few different LLMs. I had like, you know, chat GPT and Gemini and a little bit of Claude in my side of the house. And I just wanted just to say, you know, just out loud as I’m thinking it, you know, it doesn’t have to be one of the public, you know, the hyperscaler LLMs, you know, you could be hosting your own private LLMs in your data center. It doesn’t matter where or what we can integrate with anything that has an API at the end of the day. So this gives you really broad and really flexible ability to do end to end.

John Capobianco • 22:46

Well, we lost William by accident, I hope he rejoins, but to William’s point, the mix of experts that Itential has is totally up to you. If you have a singular LLM provider that you want to use on-prem or in the cloud, you can do that. If you wanted to incorporate multi-AI and lean on what they’re good at, code generation or code validation or text generation for the ServiceNow ticket, you can do that as well. So, I’m hoping William jumps back in here. If not, I want to thank you for joining us and if you have any questions or want a demo of this solution or other solutions from Itential or Selector, just reach out and let William or I know and we will make sure that we get you connected to the right people. Thank you again. Thank you for everyone who supported us at Cisco Live, who came out to our booths and said hello, and stay tuned for more.

John Capobianco • 23:43

We’re going to have more of this in a series of demos. We have some other use cases. that we can provide. There’s William. He is back to say goodbye. I’m just wrapping things up, William. So William, we’re going to have a series of these.

John Capobianco • 23:56

This is this month’s episode. We want to keep this nice and tight, 30 minutes. But we have an NTP drift example. We have some other configuration examples. We have some source of truth examples. So William, hopefully I can invite you back and next month we can show off some more use cases. Do you have any final thoughts before we go?

William Collins • 24:16

Yeah, I guess the only thing is any use case that’s a consistent challenge that you run into operationally that, you know, Selector gets that context around and has that data and does that triage, anything that is happening, you know, can be basically thrown into this closed loop automation machine, if you will. So just think about all those things that take up so much time that are so tedious that operations, these things that you deal with again and again and again. And, you know, when I was in the network engineering space, I always thought, you know, gotta be an easier way. There has to be a better way, come on. And this is really that catalyst of really a better way of doing ops. So really excited, really excited.

John Capobianco • 25:04

I’m also excited about seeing artificial intelligence applied to solve problems in such an elegant way. I know a lot of you are thinking probably as you watch this 5 to 10 or 15 use cases off the top of your head that you as a human never want to deal with again, that it’s either so trivial or such low risk or such high frequency of an occurrence or even a low frequency occurrence that happens four times a year and you just want to dealt with automatically. This is the solution. This is the promise of artificial intelligence. So again for William and John, thank you again for joining us. We’ll be back next month or in a few weeks with some more use cases and we really do appreciate your feedback. So as you’re watching this, ask questions, right? Share this around, reach out to William for a demo, reach out to me for a demo.

John Capobianco • 25:56

We really are excited about this future together. Thanks again.