Robert Rice, COO of Health Lyrics joins us to discuss Cloud Infrastructure for Healthcare. We explore 8 years of experience migrating to the cloud with lessons learned and best practices.
Please follow the show on Itunes, Google Play and Stitcher
Follow This Week in Health IT on Twitter @ThisWeekinHIT
Take a look at the website www.thisweekinhealthit.com
Follow Bill Russell on LinkedIn and Twitter
Reach out to [email protected] if you want to discuss your next project.
AI Powered Transcription.
Bill Russell: 00:04 Welcome to this week and help it where we discussed the news information emerging thought with leaders from across the healthcare industry. This is episode number 29. Today we do a deep dive this, uh, this one is in the cloud infrastructure and migration with our sponsor health lyrics. Find out how to leverage the cloud to new levels of efficiency and productivity. Visit [inaudible] dot com to schedule your free consultation. My name is Bill Russell, recovering healthcare cio, right, or an advisor with the previously mentioned healthy Eric’s a. before I get to our guests and updated our listener, Dr. uh, I recorded a brief segment yesterday was Shawna Smith, the executive director for hope builders. And uh, I’m going to play that right now for you as we’re coming to the close of our campaign and support and hope builders and organization that gives disadvantaged youth the job training and life skills that they need in order to succeed. I wanted to share a little bit more about the organization with everyone today. I’m joined by Sean Smith, the executive director of hope builders. A good morning Shauna, and thanks for joining me today.
Shawna Smith: 00:59 Hi Bill, it’s really a pleasure to be here. Thank you so much.
Bill Russell: 01:02 Well, I’m, I’m really excited to have you. Here are our great community. As I shared with you, have listeners, has responded and raised $2,000 for hope builders over the last couple of weeks. We still have a couple more weeks in our campaign or our sponsor has agreed to give a thousand dollars for every hundred new podcast listeners or a youtube subscribers and I just thought it was this great opportunity to connect with you and give the audience a little bit more idea of what you guys do. So I’m just going to throw it to you with a tell us a little bit about hope builders. Where’d you come from? Who Do you serve and what? What are you doing in the community that we live in?
Shawna Smith: 01:39 Well, I think I’ll start with the where we came from question because it’s a really, I think, compelling one and a unique one. Actually, I’m almost 25 years ago, a group of four Catholic nuns, the sisters of Saint Joseph of orange or living in a really tough neighborhood in a city of Santa Ana and Orange County, California. Not always a place thought of as a place of high need, but there are definitely pockets of that and that’s especially true in the early nineties and youth violence were really prevalent at that time and the sisters are really caught in the middle of that and realize that for many young people there weren’t a lot of options outside of the neighborhood. And so they were compelled into action and as is our tradition, they engage the community in 23 years later, we’re serving close to 600 young people a year providing workforce training, really seeing jobs as a great vehicle for helping young people who are really disenfranchised and move out of poverty and build lives of self sufficiency for themselves. So that’s really what we focus on.
Bill Russell: 02:44 You find these, these, these youth that are struggling and you give him a second chance. So how do you identify them and, and how do you, um, you know, and what is your program do for them,
Shawna Smith: 02:55 you know, so we really try to seek out those young people who aren’t in school. They’re at work, we’re particularly focused on young adults so that 18 to 25, 26 year old who is struggling and stuck maybe not in work or in a minimum wage job, going nowhere, you know, and, and, and where we are in southern California, like many parts of the country, it’s very high cost of living. So we want to ensure these young people get into the workforce and we’re particularly focused on four industries construction, healthcare it in business and we provide training and then we partner with local employers to revise them, employment opportunities which really kind of help enrich and grow their businesses. So it’s a win win for both the youth and our local employer community.
Bill Russell: 03:40 I want to thank you again for the opportunity. I want to thank you for the work that you’re doing and then it’s such a great opportunity for us to support and I really appreciate our listeners who have subscribed to the podcast and our sponsor for, uh, for giving that money. We have another four weeks of the, um, of the, uh, a fundraiser going so I would just encourage people to share with their friends and uh, let’s see if we can drive the number higher and thanks again for, for coming on, Sean. I really appreciate it.
Shawna Smith: 04:08 Well, thanks to you and your listeners for your generosity and the opportunity to share about our good work. Really appreciate it.
Bill Russell: 04:14 I appreciate Shauna coming and joining me and, uh, and I also really appreciate our listeners that have spread the word about the show and we really appreciate it. As I said, we’re a, we’ve raised $2,000 but we helped raise more in the next couple of weeks and we appreciate everything you guys do with regard to that. So, uh, today’s guest is Robert Rice, who is the chief everything, chief architect chief technology or chief chief operating officer for health lyrics. Robin, I have worked together for a while. A good morning, robin, welcome to the show.
Robert Rice: 04:48 Good morning. Thank you.
Bill Russell: 04:50 So you have a, you have an interesting background. You’ve worked in a lot of, a lot of different instruments besides healthcare. Uh, what are some of the other industries you’ve, uh, you’ve worked in?
Robert Rice: 05:04 I started in finance and then prior to healthcare I worked for the entertainment industry. So The Walt Disney Company, for example,
Bill Russell: 05:11 Walt Disney Company. So you worked for Disney. What’s, what’s it like to work for Disney? Is it, is it as magical as we would think?
Robert Rice: 05:18 It’s a lot more intense than magical. Yeah,
Bill Russell: 05:21 it’s, it’s intense for the people that make it magical for the people who are visiting, I guess. Uh, what’s, what’s the, what’s the craziest it projects you had ever had to pull off at Disney, just out of curiosity.
Robert Rice: 05:34 It’s not that there’s one intensity over the other. So for example, in identity project in a normal large institute might have 100,000 identity accounts. At Disney it was about $2 million identity counselor. Just sheer volume alone caused you to rethink and relook at the way you’re doing things
Bill Russell: 05:53 was. Was there anything about the characteristics in terms of downtime? I would, I would assume this. He had like a zero tolerance for downtime kind of situation.
Robert Rice: 06:04 Yeah. We worked when most people were, so we had to make sure that we didn’t impact that. The mantra that we only have one time to get right for the user experience with a family experience dependent measure. Thrice cut once.
Bill Russell: 06:22 Yeah. So that was a good, good training for you to, uh, to come into healthcare. So then you came into healthcare. You actually joined me at, at St Joe’s in Orange County and um, and that’s really going to be the, just, we’re going to talk a lot about that time there. Uh, and then when the providence merger happened, you and I went off and full disclosure, we are a partners in health lyrics and we’ve been doing cloud migrations now really since about 2010, uh, not only with a St Joe’s but with other health systems. And that’s the justice. So this is a deep dive episode, which means we take one topic and really explored, we’ve, we’ve done this before with ai and now we’re going to do this around cloud computing, but more specifically we’re going to focus it on cloud infrastructure. We can talk about cloud data models and some other things, but this one’s going to be about cloud infrastructure.
Bill Russell: 07:15 Uh, I’ll do the setup and then we can move deeper into the process. Um, and just, you know, as I said, I, I, you know, Robin, I have a lot of the same shared experience with this. A plus. Some of my talking more on this episode is just going to be just sympathy for rob because his plane got into lax after midnight last night. So, um, we really appreciate it. I really appreciate you coming on the show. Maybe not at the top of your game right now. I guess. So. So let’s, uh, let’s start off with a quote. This is pretty recent. This is happening this morning. So, uh, Dale Sanders, president of technology with health catalyst, uh, had this to say on Linkedin this morning at roughly 35 minutes ago, if it went, I ever returned to the cio role in healthcare outraced as fast as possible to move my data centers to the public cloud.
Bill Russell: 08:07 There’s no way I can provide the same capabilities on any dimension with my data center. Even the dentist said data centers like viawest can come close to matching azure, aws, et cetera. Um, and I think it’s interesting. I think you’re seeing a lot of the John [inaudible] had on the show also have similar comments about, uh, the capabilities and things that are available in the cloud. So, um, so let me set this up. So let me put a little definition around it. Uh, I came in as cio in 2010. I looked at the organization, looked at rob and said, you know, we’re going to, here are the characteristics of cloud that we’re looking for. And this was back in 2010. Not a perfect definition, but it sets the groundwork. We said it has five characteristics. First, a self service provisioning from anywhere, so a web based really from, from your phone, from Alexa, from self service provisioning, from anywhere, uh, that was number one.
Bill Russell: 09:02 Number two, rapid elasticity, which meant you could scale up and scale down. Now scaling up was not that big of a deal back then, but uh, it, but it was somewhat, sometimes you had to add a crack units and air conditioning and power to your data center, in which case it was millions of dollars. Uh, other times you had to add servers, took a lot of time. But one of the things that we were looking at it when I came in in 2010 was the ability to scale down the org. We kept asking the Organization for tens of millions of dollars for new data centers and whatnot. And they were saying, okay, hey, we’re, we’re actually selling off portions of our business. Can we have some of that money back in? The answer is no. So cloud gives you the ability for self service provisioning and rapid elasticity. So scale up, scale down a meter service, pay for what you use.
Bill Russell: 09:47 So, um, one of the benefits of the cloud is if you have a thousand users today, you pay for a thousand users. If you have 800, you’re paying for 800, but even more than that, the, uh, the services are much more granular so you’re only paying for the storage you need and the compute uni and now it goes all the way down to a, you know, much, you know, Ai and other things as well. Uh, the fourth thing was programmatic access to resources. This is where we get into the Dev ops. This is where a lot of your efficiency comes in. And then the fifth thing was granular accounting. I’m to the point where aws now you can do compute by the minute. So, uh, for those of you who are wondering what you would use that for, sometimes you have a processing that’s done at night. You could scale up 100 servers, use them for 10 minutes, do all your processing scale and back down, and only pay for using those servers for 10 minutes.
Bill Russell: 10:39 So, so granular accounting, so self service provisioning, rapid elasticity, metered service, programmatic access to resources and grain, lower accounting, and like I said that, that not a perfect definition, but it’s sort of set up what we were looking to accomplish. We were looking to get agility, efficiency, and automation within our health system. So we thought the cloud was going to bring us really two important things first in 2010 when I came in, we, we interviewed a lot of the executives and said, where’s healthcare going? And the most common response was we don’t really know. We know that there’s a lot of change coming pophealth value based care, a mergers and acquisitions, new competition. Silicon Valley was coming in. Um, and uh, you know, one of the things is when you don’t know what’s next in your industry, you need to create something that is, that you’re able to make quick moves, you’re able to adapt very quickly and so cloud gives you that ability.
Bill Russell: 11:35 Uh, the second thing we knew is that healthcare is moving outside the walls of the traditional health system. So we used to have data centers which serves our 23,000 staff, uh, but that wasn’t going to be the case moving forward. We are going to start putting devices all over the place. We also had clinics, critical access facilities, long term acute care facilities, more and new types of partnerships so that not only were we looking at 23,000 end points, we were now looking at literally millions of endpoints accessing resources from our data center and the cloud lend itself pretty well for that. So, uh, one last piece of background, rob, before we get going. So I’m just to give people a little idea of where we started our journey. Um, you know, our, our scenario is pretty challenging. We had a year some of the statistics that 50 percent of our data center equipment was end of life when we started our project.
Bill Russell: 12:26 Sixty percent of the workstations were also end of life, uh, 1800 instances of 900 distinct applications, which is not the worst we’ve seen in the industry. I know I’ve seen organizations, when we say 1800 instances, they just go, oh, well that’s, that’s nothing. We have 10,000 of you just sort of shake your head like that’s a almost impossible to manage 10,000 instances of distinct applications. But we had 1800, 900 a distinct applications. Uh, we had $15, million in data center upgrades that needed to be done. And I’m the data center. I mean this is the real thing here. The data center just gone down eight times in six weeks prior to my arrival as cio. Uh, it’s actually one of the things that precipitated my coming in as cio. So, uh, and the other thing I will give people just because it gives a picture of our relationship, I wish the unreasonable cio, you were the vp of infrastructure and operations and I looked at you and said, hey, get our data centers to the cloud.
Bill Russell: 13:28 And Oh, by the way, here’s one last thing. You can only have two racks per hospital and they have to be self contained racks and not need data centers. And I remember actually I told the team that before, before I hired you and they looked at me and said, that’s not possible, and then when I brought you in I said, hey, here’s the requirement, I thought you were going to tell me the same thing, and you looked at me and said, oh yeah, piece of cake. Yeah, we could do that. So we’ll, uh, we’ll shed some light on why that was a piece of cake and actually how you exceeded that expectation, which was pretty interesting. All right, are you ready? Great. So we’ve set it up. It’s a, obviously it’s a picture perfect environment. Most of the stuff’s data centers going down. You come in as a VP of infrastructure and operations and uh, we have this cloud strategy, so you take the cloud strategy. But one of the things that we talked a lot about is we’re not moving to the cloud for cloud’s sake. We’re looking to really derive benefits for healthcare organization. Um, let’s start out there. So what are some of the benefits that a healthcare organization will get from moving to the cloud?
Robert Rice: 14:40 First and foremost, it gets you out of the business. You’re not an IP business, you shouldn’t be there and we don’t do it. Well. We don’t attract the level of talent that we need because honestly I know so that little talent wants ugly boy wants a Disney wants that Amazon on their resume.
Bill Russell: 15:00 In fairness, we were in southern California, so we were competing with startups with Disney and others, so it was, it was a tough labor market.
Robert Rice: 15:10 Right, right, exactly. But the beauty is that applications back is also not very complex in the grand scheme of things that lend itself easily to get yourself out. Once you get over that emotional hurdle is, you know, I need to own the servers.
Bill Russell: 15:26 We did have people that wondered if we move to the cloud, how are we going to walk into the data center? How did you address some of those, some of those concerns and, and I mean the benefits far outweigh that, you know, being able to walk into the data center.
Robert Rice: 15:42 Yeah. Well first and foremost, reading, it’s understanding improving. Do we know what we need to know and what do we don’t know? And a kind of funny speak that the basic is for is what is our, what do we actually management and what do we think we know versus all that shadow stuff that doesn’t make sense. Once you start getting your hands around that or say it becomes an easier picture to paint that you probably don’t manage enough of that already. So it’s easier to start kind of boxing up. I’m shipping out.
Bill Russell: 16:16 Yeah. Between you and David Baker do Baker who we had on the show earlier between you and David. You guys kept reminding me that um, you know, it’s not compute storage and uh, those things that people buy, they buy a, what they want from it is uptime, uh, ease of use access. Um, they’re, they’re buying services and, and so much of our, a challenge up front was to get it to think different. You could get the culture to think different of what’s the best way to deliver service to the organization, not what’s the best way to get storage, compute and all that stuff to the, to the business.
Bill Russell: 16:57 It was a, it was an interesting cultural change. So we, we, we saw a financial benefits. What are, once you get to Dev ops and everything’s programmatic. We went from 35 down to five, uptime went up the, uh, the, uh, overall costs were more tied to the level of the business. Uh, and then I think the other thing that people are surprised to hear is that, um, our, our availability actually goes up in the cloud and we’re going to talk about that a little bit, uh, later, but, uh, you know, you have disaster recovery capabilities that aren’t available in traditional models that you were able to leverage on a regional basis and on a, on a system wide basis which were, uh, were pretty interesting. So, um, all. Alright. So let’s, let’s just, we’ll step into the process here. So, uh, and again, this is the process that you gave me way back when.
Bill Russell: 17:55 Um, there’s really three movies together, get us to the cloud. One is to establish business case, second to architect a plan, and then the third to migrate and operate. Then under each one of those there’s a, there’s a handful of steps and movements and we’re going to touch on this. So the first thing under established a business case, you have a readiness assessment and um, I think the thing that people will be interested to hear is not every health system is ready to go to the cloud. So why is that? Why do you think, what makes one health system more ready than an other to, to, to move to the cloud?
Robert Rice: 18:37 Simplicity. I think the context is organic growth versus growth. When that most health systems that I’ve seen have grown organically, like I’m gonna buy this system, that system or that system and just keep adding onto the bandaid ball as opposed to know logical and methodical growth, you know, we know we’re going to grow by a factor of x year over year. And I think that’s kind of really at the epicenter of why, from what I’ve seen coming from other industries, we’ve done that into healthcare. I’ve yet to see that.
Bill Russell: 19:11 Yeah. I think the other thing we noticed is that somehow systems still don’t have a handle on the basics, the basics of blocking, tackling. So when we go in there, like for example, um, you know, when I came in as cio, I asked, okay, I would like a list of all the organizations that we’re connected to and what data we’re sharing. That took almost six months to a year for me to get that list. So organizations that have trouble with the basics of, hey, can we have access to this data, this information, do you know where your contracts are? Those organizations need to, not that it’s an insurmountable hurdle because we got, we got through it and um, but you know, being able to have access to that data, even though it’s, you know, a lot of times we can just take raw data and process it. Um, just knowing where it is and having access is one of the things. So, you know, what are some other things you found in terms of organizations being ready?
Robert Rice: 20:14 I think process maturity, so standardization, and this isn’t a process for process sake, is it, do you have standard processes that a large portion of your staff fall and then a lot of cases that doesn’t exist and that’s the baseline for building blocks of automation. So asset management is Wonky, but do I have standardized procedures for and for change or things like that? So, so if it’s measured in applied consistently across a larger of the organization, a much more mature and ready to take those building blocks in place first
Bill Russell: 20:53 they’ve done those kinds of things, you’re, you’re going to be in good shape, I would assume. Right? Alright. So the next step is to establish the point of truth. And uh, you know, I wish I found this interesting because, uh, you know, for me, you came in and started dumping all this, uh, information on me and it didn’t match with what the organization had told me. Um, you know, prior to you doing this deep dive into our, uh, information and uh, and, and to be honest with you, I was a little defensive because I’m looking at, you know, we had just done a Microsoft agreement based on information that Microsoft has given us, so they come in and done an audit and said, hey, here’s all the stuff you have, here’s what you should license. So that’s what we license. And then you came in, did this deep dive and came in and said, hey, over licensed by about a million and half dollars. And that was, and I got defensive and I’m like, how can you say that, you know, more than Microsoft, so it has all these tools on my environment, but give us an idea of what it means to establish the point of truth and how you get to, um, really that, that starting point for the cloud migration,
Robert Rice: 22:04 your, we’ll say your source logical systems have the data of how you actually use it from a licensing perspective or a consumption perspective. So active directory, for example, every user that’s gonna log in and every account that’s going to do it, but not every account is active. So when you start looking at it like how fast or kind of terminate my account, that was one of the problems that we had is, you know, we had approximately 50,000 accounts but they have been fallow for multiple years, meaning that we don’t use them at all, so we have licensing for the wrong amount. The same thing in most other areas, right. So we never do a good job of rationalizing down that we can buy, we could acquire with the build, but what’s our lifecycle management process? And that’s kind of one of those, you know, easy sacred cows to go after first and foremost say, how much have we been deconditioning over the years?
Bill Russell: 23:02 Yeah, I think it’s interesting that you, you literally are just saying, Hey, I just want the raw data out of active directory. I want a, I want to look at the physical contracts. You, you, you really, this process, you go back and one of the things you, you sort of drove home to me is that every report I’m seeing has a bias or a purpose, a purpose or a bias, right? Has a bias to make something look a certain way or has a purpose that it was designed for and I might be using that report for a different purpose. Um, so talk about why you go back to the source data and, and how and why that doesn’t take forever. I would think if you go to the source data, it would just take forever to get that done.
Robert Rice: 23:47 Was part of our journey together. Generic performance asset reports are just that. They’re generic, they don’t apply to the, we actually use it. They applied to the way the tool is done. So when you look at the problem from a tool perspective, you’re solving a problem with a hammer. When you actually need to build a house, when you go to the law of the data, you’re looking at actually how you’re using it and which is really the key, which takes a little bit more time upfront, but it moves a lot faster once you lay out all the cards. So when they’re solving a puzzle, for example, you typically look at the edges rather than starting in the center, you know, and that’s kind of the approach where you have to Kinda look at it or am I actually using it and what am I actually doing with it first, which allows us to move faster once you get the edges of the puzzle figured out.
Bill Russell: 24:31 Yup. And then we have this point of truth and they establish the business case itself, which closes out this first step. And in the business case, you have things like financial models. You have usage, uh, you have skills, you have, um, uh, you know, I’m, I’m just, I’m, I’m actually shortening this up for, for purposes of the show, but, uh, there’s a lot of elements of this business case. One of the things that the combination of you and I really works as people can tell, you’re very analytical and I like to tell stories and so, uh, I really strongly encourage people to tell the whole story of how health care transitions in this process and then you come back with just all the supporting data in terms of, uh, how, how agility is going to play out in the organization, how, uh, there are cost savings even though people need to be aware of the fact it’s, it’s operational dollars versus capital dollars and know a lot of cases and some organizations struggle with that transition. So we now move into architect and plan which is the second step that you have here. And the first part of that I think is probably one of the more fascinating, which is decomposed the applications and the services to core elements of what are you actually doing there?
Robert Rice: 25:51 You’re just looking for the commonality across your application portfolio, right? Every application has the same components. A wall, you know, where the data exists, how does it process what the client side looked like. So all we’re doing is we’re taking that and establishing a commonality across the largest portion of the application portfolio that we can start standardizing and really focusing on because you’re not going to solve the problem individually. If you have hundreds or thousands of applications, you’re going to look for the least common denominator across the group which allows you to do that. So a database, for example, most applications have a database and an on premise legacy world. You have your production database plus your backup, maybe a a high availability one, so now, but from a core conceptual, it’s all the same thing. So that allows us to start with losing down and focusing on solving the database problem rather than the thousand application problem.
Bill Russell: 26:46 Yup, exactly. It’s interesting because I think, I forget the number of layers, but it was a, you know, it’s everything you would think about. You have the fiscal hardware, the networking, the virtualization layer. I’m going to get some of these wrongs storage layer ostp and whatnot. And when you got done that, one of the, one of the big findings I found was you came over and said, hey, we have a. It was a couple hundred images, I forget the number. Do you remember the number of images? Server images we had?
Robert Rice: 27:16 It was a little over 200.
Bill Russell: 27:20 Yeah. So it’s. So that’s one of the things when you look at it through a different Lens, you’re able to see this, wow, we have a couple hundred images. Well that’s very hard for an organization to maintain that level. And then when you broke it down even further, I think what your team came back and said is immediately we can get this down into the teens because of, you know, if we just do some work around the images and eventually I think the goal was to get it down to six is that, is that pretty close?
Robert Rice: 27:49 So about $200 down to you and then you can start leveraging efficiencies because now I’m not spending time chasing after 200 different types of like a manage the exact same.
Bill Russell: 28:06 And it was really interesting. You had, before you came in, we were chasing the CMDB, a nirvana where we were going to have the CMDB, which would always tell us what was going on in the servers and you just, you sort of minimize that. You’re just sort of like, hey, you know what, if we get down to five images, then you know, it’s, it’s the CMDB becomes not irrelevant but becomes, first of all, a lot easier to manage. But second of all, I mean, it’s one of five images. You don’t have to figure out all the things that you would normally have to dive deep into a CMDB to figure out. So, um, it was, I found that to be interesting. So the next thing, and this really highlights the mistake that I made prior to you coming in is the skills Matrix and training plan. Um, so I skip this step and one of the things people need to know is that the biggest obstacle they’re going to get to moving to the cloud is not from the organization.
Bill Russell: 28:57 The organization’s going to say, Hey, I use the cloud at home and I got 25 Gig of storage for my email and I use dropbox or box at home and I, you know, I have all these capabilities and they understand the cloud gives them better services. So they’re not, they’re not pushing back to you when you say, Hey, we’re moving to the cloud, but it does. And one of the things that I skipped was a skills matrix part and because I didn’t, people didn’t see where they fit in the organization when all this was done. And uh, and so you, you made a step back. You created, um, you know, a skills Matrix for where we’re at today and what skills they would need to develop to get to where they were going. Uh, so talk about the process of not only developing the skills matrix, but also having the conversations with the employees and the, and the staff around. I’m moving to the cloud.
Robert Rice: 29:48 Well, first and foremost, right, that you’ll set shifts or elevate up in the cloud because again, you’re not emphasizing the petty engineering skills. You don’t need a lot of really core storage depth and knowledge because all you’re doing is splitting up gigabytes and terabytes of data at a time here. So as you’re building out the skills matrix, what we started realizing was we never really defined what success was from a scale perspective, so we be emphasized planning, design, reporting, those types of skills that you actually really need more so because you need to map out more intensive skills matrix. We rated all the skills that we’re gonna need in different types of roles on a one to five basis and we added in more soft skills and those design skills and those reporting skills that are often neglected that we need to emphasize. So we kind of reset what success was for each individual team and group to say right now your engineering heavy boat. We need you to get to this level to be able to emphasize things like that because that’s what really kind of helped us and not everybody wants to do that. Or it takes that lead to that as well. So we gave him a different
Bill Russell: 31:00 and that was the fascinating thing to me. I mean, you sat down with a significant staff and started mapping that out and some of them just self opted out and looked at him and said, hey, here’s where it needs to get to. And they said, you know, I’m fine, I’m fine working with Kobalt and you’re like, okay, that’s fine because we’re going to have this legacy environment for at least a year, maybe a year and a half while we do the migration and we need somebody to take care of it. So, you know, but at the end of the year and a half, we’re not going to need cobol programmers. I’m using that as an example to cobalt programmer. So you’ve essentially self opted out over the next 18 months. But, you know, but people appreciate it knowing that, of just, okay, so what you’re saying is my job going to go away in 18 months, but you need me for the next 18 months to get there. And they, they were really appreciative. Um, and I, I would just strongly encourage people to do it seems so obvious to see those, but for some reason I skipped it. A, you have to show people where they’re going to fit in the future state. Otherwise they, uh, they will fight, fight the process. So, um, yeah,
Robert Rice: 32:05 soft skills and those kinds of intangibles. In a Dev ops world, there’s no step by step guide. So like how do you identify your innovative thinkers, your testers are tinklers because part of the process is learning self learning and if you need a, you know, cbt or a training module to that, you’re probably not the right fit for that role. And people we found came out of the most unlikely places when we started doing that.
Bill Russell: 32:31 Yeah, it always surprised me where you were, you, you were yesterday, that person was doing that and now you have them doing that. Are you kidding me? And you’re like, no, let me tell you how these things match up. This is, you know, this is the new world. And um, I think the other thing was we just had a conversation with a, a client with a potential client. Actually, I should be honest there. So with the potential client and they were saying, are you saying in the Dev ops world, I’m going to have to hire all these programmers because that’s going to be really hard for me and I, I think you, you, oh, well why don’t you, why don’t you talk about that a little bit? Because I think that’s a common misconception
Robert Rice: 33:11 once you lose them because they are going to the bigger names, they’re building their references and their resumes based on that. So you have to kind of create more than the grooming and training on how do you invest in the people that need the time to learn and train and develop their skills to do because you can’t compete with those big guns in the industry that will kind of give them a growth path. So pure that you have the ability to really kind of build that skillset.
Bill Russell: 33:42 And then the other thing we’re finding is that the tools are becoming more configuration based and less, you know, hardcore coding based, right? So it’s, it really is going into an interface and clicking on a couple of things and then boom, you’ve just, you’ve just stood up a whole bunch, not that you don’t need a couple of those hardcore programmers who really understand how the underlying framework works, but a lot of the, uh, tools or are moving higher up in the, in the, uh, in really the, into the presentation layer. So, uh, so people who just understand how that entire stack works can make this stuff work.
Robert Rice: 34:21 Yeah. So you had to have core development skills to do this, but now like 50 to 75 percent of the work is configuration based. There’s still an element of development and Dev ops, you know, that now it’s kind of funneling higher and higher.
Bill Russell: 34:43 Yep. Absolutely. So the next thing, and uh, you know, probably the sexiest thing here is establish a digital architecture, security models and aligned work loads with the appropriate deployment models. Um, and what, what we talked about a lot here is that each cloud will look a little bit different from client to client based on what they’re trying to accomplish. But you do have five primary environments that you consider today. Um, can you, can you walk us through the five environments again? I mean you gave us. So there’s a, there’s a legacy of environment anyway. I mean, you know what list I’m talking about?
Robert Rice: 35:18 Yeah. So the legacy environment is acknowledging the reality that not all your systems are going to lift out on that. So you have to maintain it.
Bill Russell: 35:31 Why is that? Is that just, is that just because because of the way they’re written and way they function today, they, they’re just not going to go to a cloud environment,
Robert Rice: 35:45 direct physical connection. So in the health industry, biomed devices, you’re just not going to get out of them anytime soon. The larger ones that are critical systems that support different process, monolithic in nature, but so you may as well just acknowledged them, put them in the corner and then come back to them and a couple of years time and then focus on the other. You’re scattered.
Bill Russell: 36:12 So the first one we have is a legacy environment. Walk us through the rest.
Robert Rice: 36:17 Yeah, so the next level up is you have applications that are still server based, but virtualized. A virtualized server can exist anywhere. It doesn’t have to exist in your data center for the most part. Those are the target to move out into infrastructure and hosting your aws instances, but they’re still building server instances and then when you take the next step beyond that, now you’re focusing higher up which is more workflow based. So you’re going after a softwares and service platforms, which everybody has right now. Whether you’re using salesforce or dynamics or other software based platform, you’re focused on workflow. You’re not focused on anything below the workflow for the most part. And that also kind of lends itself to platform as a service to those investments and the sharepoint online into salesforce, into other crm platforms that you have more platforms that I could kind of build different workloads and different processes, different solutions in a similar platform instance rather than different softwares. And then ultimately the utopic vision when people get to that level is native development called native development models where you’re designing micro services, web services and the like. But that’s typically the end state of a journey.
Bill Russell: 37:45 So the distinction when we go. So every, every health system will have those five components, legacy or a private cloud infrastructure as a service, software as a service platform. And then cloud native. So what’s the distinctive is how much of each of those you’re going to use. So some somehow systems might have 20 percent legacy, uh, you know, 30 percent infrastructure as a service and so forth and so on. Um, but the goal really is whatever, whatever that, that core layer of legacy, we’re shrinking that over the next couple of years. We’re moving to infrastructure as a service. Even that layer, we’re moving to software platform and cloud native, and the reason we’re doing that as the levels of efficiency, agility, cost savings, and access to innovation that organizations are gonna have I go into the cloud are just significantly more then a traditional traditional models. All right? So we create these really cool diagrams.
Bill Russell: 38:40 We map out the entire, uh, digital architecture of the security model and, uh, we aligned the workloads to the various layers of, of cloud infrastructure that we’re going to use. Now we’re ready to migrate and operates. And uh, this is the easy part, right? So we, uh, we just have to migrate $1,800 application, 800 instances of applications. Pretty straightforward. Um, and uh, I’m being a little facetious here. So here’s where we, here’s where you start this process. You start with a negotiating agreements and Kpis with the cloud vendors. What are some of the things that healthcare cio or healthcare organization should be thinking about when they’re negotiating these cloud based contracts?
Robert Rice: 39:25 Oh, environment, any virtual servers or instances I’m going to need to maintain it because they just can’t get rid of them, but they’re still server based. You could lock in certain pricing models, so when you’re locking in reserve pricing for a use azure for example, say I know I’m going to maintain $500 the ends, you’re gonna get a very different price point than if you say, you know, I’m going to spin up instances waiting and it’s a bachelor of magnitude to. So knowing what were reserved pricing and forecast will be, let’s say I know I’m going to release three years shaped very differently than just kind of winging it and going into it. So again, back to that very first step, which is knowing your world and what you’re going to maintain is going to be key for you. And then from there we’ll spend appealing back to your usage case in models.
Bill Russell: 40:21 Yeah. And some of the things that we’ve talked about, you can’t outsource security because you have a background in security. So you can’t ask our security. So you need to negotiate that. And, uh, so understand the HIPPA requirements, understand any PCI requirements, you have a and you have to be able to provide the reports and, and pass through all the way, all the way down so you can’t outsource it. You’re still responsible. You still have to be able to supply that. Um, the other thing, the other Gotcha that think people don’t think about is, um, especially with cloud applications, like, uh, we’re, you know, you’re moving vast amounts of data into these cloud environments. Um, make sure you can get it out. Um, you know, when you’re negotiating these contracts, uh, not only make sure you can get it out tactically, make sure you can get it out financially.
Bill Russell: 41:10 Uh, I know that some of these clouds, it’s interesting, they will, they’ll literally send trucks out to pick up your data, bring it to their data center and pump it in. Had some of those, some of the services almost free, but to get it out, they hit you with bandwidth costs and all sorts of other stuff. And it costs a fortune to get it back out. So be careful of those kinds of things. And, you know, there’s, is a, we only have a couple more minutes, but those are, those are just some of the things off the top of my head that I would, uh, I would say just keep those things, keep those things in mind. So this is the farmer,
Robert Rice: 41:47 what type of data. So data that I need fast response on is very different than data. I probably only need one for long periods of imaging for example.
Bill Russell: 42:02 Yeah. And when we’re looking at, when we’re looking at the architecture, we’re looking at those bandwidth requirements and we’re looking at those access requirements because the, the, the more access you need it in the cloud, you can pay for very high availability, very high access, very low bandwidth or latency, low latency, low latency servers. But if you don’t have to, you really shouldn’t. So, uh, so really knowing your workloads and aligning correctly to the, to what you’re buying is really important. So here’s where I want to spend a bunch of time because this is the, you spent so much time here and this is where the big payoff is. So, um, you’ve, you’ve done all this, you’ve done this deep dive, if you’ve done all this planning and now we’re getting ready to migrate workloads and uh, you know, at first migrating the workload goes a little slow, but then it really picks up pace because you’re just doing the same thing over and over again. So give us an idea of some of the things you think about going into a migrating to the cloud and some of the processes that you, uh, you take a look at.
Robert Rice: 43:05 Well first and foremost, if we’re doing a lift and shift migrations that you to do a disaster recovery, fail over. So we create the deal disaster instance in the cloud, which allows us to full production snapshot everything we need to go to scale, build the competence where the expertise for it and then go through a few iterations of fail over without impacting the business in those cycles. Now we’ve built everything that we need to at a low price point, at a low impact to the production environment and got the business to buy in and sign off on it and build our total competence. And that’s kind of the, probably the primary way that we start the process because it allows us to kick the tires and lost the building confidence and building on migrations.
Bill Russell: 43:52 Yeah, you really encouraged people to start with the easy stuff first because your team is learning a new skill, so you, you, you, you don’t want to start with, hey, let’s start with packs. You really want to start with something that’s a pretty, pretty small, pretty self contained. I’m not a complex architecture and your teams just getting that, building up those skills. And then as you, as you sort of moved down the road, then you start doing some of the more challenging, more difficult, uh, migrations. But I want to go back to something you said. So you start by standing up a Dr Environment or greenfield environment that you can fail over to that. That really is one of the, one of the secrets that you brought to the, to the process that kept our, uh, kept our downtime pretty minimal in this migration process. I mean, because when you could fail over from one to the other and just sort of switched those users over, now you’re not looking at extended downtime’s a and plus in the process you’re building out dr capabilities for each of these applications as you move forward. Um, so does that work for all the applications or just a certain set of those applications?
Robert Rice: 45:13 Qualify that to say a large portion of them is transparent for the user experience is barely a blip on the radar for them. There are some that struggle with that, but architecturally, you know, you identify them way before.
Bill Russell: 45:26 And so I think the other thing was she drove home and David both drove home to me was a, this migration process is more of a people and culture a challenge than a technical challenge. Um, obviously we’re, we’re bringing our it staff along, we’re training them as we go. Uh, but also you guys built up a pretty solid team, uh, with Rachel and the rest of the team at Rachel who we fired at health lyrics now, um, who, who put together great processes for engaging the end user community. So they uh, engage them early in the process, walk them through it, talked about the benefits. Um, so that team became really good after, you know, after you do a couple hundred applications, they, they actually ended up with an extensive database of everyone in the organization. They knew who the key decision makers were. They were able to engage them at every point within that and it became a trusted source for that. So, uh, is that, I mean, do you still see that as a pretty critical team to have in place for this migration?
Robert Rice: 46:28 It’s not worth doing if you don’t understand your business and if you’re not talking right because they’re going to define their tolerances are going to define their appetite. They’re going to tell you, don’t bother me these days, but I’m okay these days, which gives you your operating model to build their competence and there are so many other problems that you find that exist outside, warm it that you went a lot of good will just by solving problems too. So building that relationship is absolutely critical.
Bill Russell: 46:58 I’m not necessarily seen benefits for us or A. I think we got rid of 10 percent of the applications just by talking to people just by going out there and saying, hey, we’re going to be migrating this application. You’re listed as the owner. And they looked at you like, uh, you know, we don’t use that application anymore. You’re like, great. Um, and so we, we didn’t know we were doing a, uh, an application rationalization projects at apparat project, but we ended up, um, you know, archiving a whole bunch of data and a whole bunch of applications just in the process because as you pointed out earlier, a lot of healthcare just grew over time. Have we added this application, this application, we acquired this company. We never really cleaned all this stuff up and this is a great opportunity, uh, to do that. So we get 24 months later, here are some of the things, you know, we, we, we never did spend that $50 million dollar on data center, uh, improvements because we didn’t need to.
Bill Russell: 48:00 Um, we went from a server provisioning from and people find this hard to believe, but really from a two to three weeks to provision a new server with the processes that we had to almost two to five minutes, uh, to provision a new server in. And sometimes our clients find that hard to believe, but in a, to be honest with you, with a credit card right now, I could go online and, and provision 100 servers in the next 10 minutes on aws and I don’t know, I don’t even have the requisite stuff lined up yet. So, um, that’s the kind of stuff that’s available in the, in the cloud. Um, and then the other thing is we, we got access to a whole bunch of new capabilities. Uh, you created a, a, a, the ability to fail over regionally, uh, and, and some other things.
Bill Russell: 48:49 But I want to get back to the one thing because you private cloud, you went into hyper converged route, which I found interesting. And I came to you and said, hey, no more than two racks and every hospital. And I know when, when I said that to the team, they sort of were incredulous and I think some so of our listeners from health systems are saying, we, you know, we, there’s no way our health system, uh, in that hospital could get down to that, that number. Um, but talk to a little, talk a little bit about what hyperconvergence was able to do a at the hospital level, a forest,
Robert Rice: 49:24 really condensed compute and storage memory power, right? So most applications that exist nowadays are nowhere near the level that you can provide in basically what I could fit under a desk right now. So when we started getting into the applications, I think, you know, these are my toys element to the local staff to. So being able to come in and say what I’ve got to take your toys out of there, but we’re going to do is just put them in a new case. But that new case now is a fraction of the convergence of data and compute. So we found almost immediately that about six years, which is roughly six to 12 inches of space, could hold hundreds of servers of what you currently have. So getting to that point was extremely easy and it also kind of want us a lot of different competencies to by saying you’re not moving it out. And once it was in the hyper converged, our ability to migrate out of the Vr stuff was almost push button easy.
Bill Russell: 50:24 Yeah, absolutely. And then you can go to third parties and be able to migrate out and migrate in as needed. Uh, and uh, so yeah, you were able to get down to, and a lot of our hospitals we’re able to get down to a half rack. I think that the largest was maybe two racks. I, I know you always argued with me on that, but um, but essentially self contained racks, you didn’t have to have a specialty room. You can move that stuff around. And I think that has some Dr Dr. benefits for it. Uh, the other thing is people, people always ask me, you know, if you’re moving this stuff into the cloud, what happens in the case of disaster? So if your connection to the cloud goes, what a,
Bill Russell: 51:05 what, you know, what do you do in that case? And so one of the more ingenious things I think that your, you and your team came up with was a almost reversed the process, which is a, it was our primary was in the cloud, but we were actually a trickling data back down into the local data center so that if, if the, if something went down, I’m on the network. Let’s say there was an earthquake in southern California network went down, those hospitals would still be able to run a certain core systems because we had trickled the data back. So it was almost a reverse a Dr kind of thing. Um,
Bill Russell: 51:41 hey rob, I want to thank you for, I know you got in late last night. I want to thank you for coming on the show. Um, and uh, you know, it’s, it’s, it’s always fun to, uh, to talk through this stuff with you. And to, to be a part of helping now other health systems, uh, to go in this direction. So thanks again. Um, let’s see. All right, so let’s close out so you can follow, you can follow the show on twitter at a this week in health it. And you can check out our website at [inaudible] dot com. Uh, catch the videos on the youtube channel this week, uh, in health it dotcom slash video. Please come back every Friday for more news, information and commentary from industry influencers. We have a great lineup of people over the next four, four or five weeks. And we’re starting to set our fall schedule as well. So you’ll want to check us out every Friday. That’s all for now.