Why AI Fails Without Governance and Information Architecture

00:00:06 Mark Smith
Welcome to the MVP show. My intention is that you listen to the stories of these MVP guests and are inspired to become an MVP and bring value to the world through your skills. If you have not checked it out already, I do a YouTube series called How to Become an MVP. The link is in the show notes. With that, Let's get on with the show. Welcome back to the MVP Show. Today's guest is joining me from North Ferebury in the United Kingdom.All the links we discussed today will be in the show notes for this episode. Simon, welcome back to the podcast.

00:00:49 Simon Hudson 
Thanks, Mark. Good to see you again.

00:00:51 Mark Smith
You too. Great to have you on. And particularly, the world is changing so rapidly in the technology space with Microsoft. We've gone into a very AI-centric world these days. But before we kick off, what's top of mind for you right now? What are you working on? What's your focus? You know, as we look forward to 2020, 2026, what are you excited about?

00:01:14 Simon Hudson
Oh, what am I excited about? I mean, I guess in some ways I'm excited to actually get some hands on my tools. As we spoke last time, technically I retired two years ago. Yeah, I have a funny idea what retirement looks like. And so I've been doing quite a few smallish SharePoint projects. And I've been thinking a lot about how much that's changed. Oh, God, in the 15 years or so since I set up my first SharePoint company, where, you know, if you didn't get 50,000 quid for a project, it was just not doable. And most SharePoint projects were over 100,000 pounds. And I'm doing stuff for SMEs now, and they're coming in sub 10. And you can solo build a really good SharePoint intranet for under 10K, if you have a mind to do so. I'm thinking a lot about that piece. As I've transitioned out of the NHS and the big corporate world, I'm spending more and more time thinking about how I could enable the smaller companies like we run. to be able to do the kind of things that we could never afford to do. So that's really top of mind. And then, of course, inevitably, there's all of the AI stuff. But actually, in some ways, it's the intersection between these two things. I was reflecting on it earlier when I was talking with Mark and the other folks on the maturity model team. I think we all have some, let's be gentle, misgivings about the pace of AI innovation and how it's pulling focus away from some of the other things which we think are really, really important, like all the stuff that was important before AI came along. But at the same time, I'm absolutely thrilled to see that Copilot is kind of making Microsoft think about metadata again and going back to information architecture in SharePoint. Because as we all know, if you haven't got good governance and good information architecture, you cannot have good AI. So that's kind of really top of my mind. How do we get people to understand that story?

00:03:16 Mark Smith
I love this. So I've been working on a project over summer for my own business. Well, by the way, it's summer in New Zealand, obviously opposite end of the world in the UK. And it is this, one of the challenges I have on my business as I work more and more with AI isI let's maybe the term comes to mind a single source of truth that let's say I'm using Copilot and I specifically am using M365 Copilot, so the $30 license in my tenant.  And what I am doing is that, you know, I'm in a growth phase of my business. So I'm looking at new product lines, service offerings, that type of thing. And I found every time I would go to AI and You know, I often use prompts like, ask me 20 questions about what I'm gonna do. And I get it to based on best practice, or I'll give it an entire book. I'll say, so let's say, have you heard of the Business Model Canvas?

00:04:20 Simon Hudson
Yes, I have.

00:04:21 Mark Smith
It's been around for 15, 20 odd years, right? And then another book that came out in that series was, value, product value creation. So how do you create a product that hits all the value markers that customers would want to buy? And so it takes you through this, you know, series of workshops that you can do to really get to that your offering is perceived at the level of value that the market wants to purchase from you. And so what I do is I take that book, the electronic version of it, because I own it, right, PDF version, and I hand it to the AI and say, Listen, I'm creating a new service line and you need to follow the model prescribed in this book. You need to ask me any questions like you're the person facilitating the workshop, right? And I get it to follow. It's the model from this body of work that's well researched and and create this offering. Now, the challenges I have had is that re-explaining the same old, same old to the AI. So for example, what's my trading name? What's my legal name? What's my company number? What's my GST number? What's my, you know, where's, what market am I servicing? What's our product lines that we have already? What's our product price points? What currencies do we transact in? All data that any business has often tied up in people's minds, yes, you could go and find the latest filings and extract the GST number and things like that off it, but that's messy. And so I came up with this concept using SharePoint to create the brain of the business. So if you like, the knowledge of my company, and I have a 12-folder hierarchy system, and in there, for example, there's the governance layer. Who are the owners, the directors, all the stuff that you would have to have for certain documents you'd create in the business. Then it goes down and it looks at the finances of the business. What are all that kind of static financial information that when you're creating something new in the business, it has it, and it goes right down. It goes sales, marketing, what am I doing at the moment? What are my channels? What are my website addresses? And what are the platforms they're built on, blah, blah, blah, blah, all the way down. And I'm then under each of those folders, there is like an index folder, which I mark particularly that it's the canonical record of this. So it's the authoritative record, and in it, it is fully human legible, but there's a whole piece of instruction set to AI.

00:07:03 Simon Hudson
Oh yeah, okay, so you embedded the instructions in there. Yeah, okay, smart.

00:07:07 Mark Smith
Yeah, and so what happens now, when I jump into Copilot and I say, Hey, I'm thinking of this new service line, and it needs to compliment my other service lines, it instantly goes, Oh, I found these canonical documents about your current service lines. This is not going to compete with those. And it's allowed me to be super more, I don't have to provide all this context all the time when I start prompting on a new project or activity. And then in my mind, I'm like thinking of all the businesses I've been in my career, well, this has never been there. And I'm like, how many businesses could function so much better if they had the brain, the intelligence of their business, that you definitely have to have, governance layers in. But how does AI know that they have got the authoritative piece of information? Because then what I do is I have a... using SharePoint, I can change, I'm using all that metadata to add into there. So is this the authoritative? Was it a past tense product? So it's been archived off, no lot, like it's, the metadata is carrying all that information. And that is just like one piece of thing I did for my business using SharePoint. Well, I provisioned everything in Teams, of course, SharePoint behind it. They went to SharePoint, did all the metadata layers so that it reflected the document. But I'm like, how many other use cases like that apply to business that would make Copilot from a resource perspective in the business, you know, like go on steroids? Highly accurate. very low hallucinations coming out, because it knows, like it would come back to me. So I've found the canonical reference to what you're asking. And of course, amazing. What are you seeing when I tell that story?

00:08:56 Simon Hudson
So there's an amazing amount of overlap there between what you've done and something I've started playing with, but you've done some things I hadn't thought to do, which is... kind of very cool as well. There's a chap in America put a Facebook or LinkedIn article, LinkedIn article out and he'd built a team of agents and he was having conversations with our agents. I can't for the life of me remember the guy's name, but we had a really good conversation one night and I thought it was really smart and ties into something else. I'll get back to you in a minute, I'm sure. But I started building agents to help me with my day-to-day stuff. So I've already got a project management agent because I'm rubbish project management. I've trained it by building a set of canonical content in SharePoint and grounded the project agent Maxwell in good project management practice. So I've done that and scoped him just to that who doesn't know everything. So what you've described in my language is actually a company secretary agent. So somebody who's got that definitive knowledge of the company as a corporate entity, and unlike the company secretary who never really speaks to the ordinary folk because they're too busy supporting the execs, this concept of the company secretary agent especially if it knows not to tell people stuff they shouldn't know, which of course it can, that becomes your canonical person, agent to go to for all those things and accelerates that delivery of letters or insights or whatever. So I think that's really cool. I think the starting up a new business line though, that's different. I would split those out. So I'd have my company secretary who knows about the the corporate part, and then I'd have an innovation and product management specialist agent who would help me. So I would have to actually go and test this, but I'd love to split those out and see how they perform as two separate agents who can both contribute to a conversation compared with one that's only able to give a synthesized position on something.

00:11:02 Mark Smith
So let me challenge you thinking on something here, because I feel like we've had a year and a half, let's say, of agent talk, you know, in the ecosystem. Agent this, agent that, and all of it massively underwhelms me. It's that the marketing far outstrips the reality. And the other thing is the effort to get the agent to the level I need is massive amount of input from me. Like massive. And so, as long as there's that big effort piece, I think that the layperson is not going to spin up agents. The other piece, I think, is that, are we calling an agent by the wrong name? Like, should we be defining them as roles that we want created? Because an agent tells me you do one finite thing and you don't have any context of the day in the life of everything else that goes on in our business, whereas somebody in a role, yes, they have their job description and what their core thing is, but they are a diamond in that they're multifaceted, they interact with other parts of the business in so many different ways that is not in a job description, and so I'm wondering, Have we used, like, and of course we're technologists, we go and technologic everything rather than going, taking a step back and going, Hang on, how does the world operate? Have we gone down this agent path of roboticism and things like that without really going, Hang on. If we're wanting to see massive value out of these, should be defining them based on role and adding this multifaceted nature to them that yes, here's your poor thing, but like in real life, we are so much more than that and you need that to really be constructive holistically in the business, not just in that deep verticalization, a skill you might have as an agent.

00:12:58 Simon Hudson
I love these conversations with you, Mark, and good challenge. I totally agree that I don't like the word agent. I think it's too tech, it's not really descriptive, and I bloody hate putting IC on the end. So, yeah, but I think you're absolutely right about what we should, we're really doing is creating virtual employees or virtual, we're putting AI entities in a business role. And I think that's the right thing to do. And for many years, I have said that when we build AIs, because we call them that then, agents now, whatever we call them next week, they should have. to your point, a job description and a set of objectives. I think that when you create your agent, you should be writing those documents and saying, this is part of your grounding set. And I haven't actually thought to do that as a separate document until you talk just now. So that's very cool. Also have their constraints and things which they're not supposed to get involved in. However, I have not considered, embarrassingly, the fact that real people actually do, or at least the decent ones, have conversations outside of their role. And they are informing themselves with things which are relevant and related or maybe completely unrelated. And that's actually often where the really big outside the box thinking comes from. What I don't know is how you, I don't think we've experimented with that with AIs to find out what the right balance is. With people who do it, We do it by letting them do it. And if they're brilliant, they get promoted. And if they're rubbish, they stay where they are or get kicked out. We don't, don't do that with AIs. Although we had an interesting conversation again on the maturity model team today about if one has these kind of AIs with increasing levels of synthetic personality and you start getting attached to them as if they were a real person, it's quite hard, kind of hard to fack them. And yet we probably need to do that with AIs from time to time that we've built. Go, right, yep, you've been running for a couple of years now. You need to be moved out of the organization and bring a better one in. So I think there's a whole bunch of stuff there, which we're only beginning to scratch the surface of, which is how to turn these incredibly powerful AIs into incredibly powerful virtual staff. And we don't know how to do that effectively.

00:15:32 Mark Smith
And that's where I wonder, has Microsoft gone about selling, let's call it like the, let's say the $30 skew of Copilot, have they gone about it all wrong? I don't know if you saw, but in the last 24 hours, if you look at the numbers of the latest earning report from Microsoft.

00:15:51 Simon Hudson
Yeah, 450 million or whatever it is, seats.

00:15:55 Mark Smith
3.3%.

00:15:56 Simon Hudson
Yeah, exactly.

00:15:57 Mark Smith
Right. Yep. And for every, my understanding is for every five sellers inside Microsoft that have a copilot target on their head, only one is meeting target. 20% of the organization, right? For that size company, that's bad, right? I'm wondering, once again, has it been positioned wrong? And because it's positioning as it's an add-on to your M365, and you think of what I paid for M365, now you're adding Copilot. And of course, I think Microsoft's gone, but yes, look how much amazing productivity and stuff it could create for you. But that's latent until it's actually activated and does it. Otherwise, you're just paying for something that's sitting dormant in the corner. And I think here's the big thing that I've been stewing in my mind. I don't ask, because one of the big things that Microsoft or the Resistance have run into is what is the ROI of a Copilot license? I do not ask what's the ROI of Word, Excel, PowerPoint, Outlook, the infrastructure of my business. I need those tools to run my business, right? I don't think about them as a bolt-on, and that's why I'm saying, has Microsoft done themselves a disservice? by making them a nice to have, an add-on, and companies are going, I can't see the value, because we're not realizing the value. And there's multiple cases now of large deployments being rolled out or backed out because they've been put in EA agreements and stuff like that. They've been switched on, no adoption's done, nobody's using them. And then a company, you know, comes up for renewal in the EA and go, What the freak? No, redline that. We don't need that anymore. And so... And I'm wondering, once again, electricity, I don't ask the ROI of electricity in my home, my business or anything. I need it to run a company, right? And I think AI is more like electricity.

00:17:50 Simon Hudson
Yeah, it's dial tone as opposed to, yeah.

00:17:53 Mark Smith
That mind shift has to happen. And the positioning of it, I think, has to change in market. And then let me, one other, and then I want you to crucify any of my ideas and stuff that you think. My last concept is this, Copilot Chat is now creating a larger disservice to Microsoft than a service. The reason is people think they're using Copilot. They think, they don't know that there's a $30 SKU or anything like that. They use it, it's, you know, handbrake, it's constrained 'cause it's a stripped down version, it's the freemium. And what's happening is people then compare that to a Claude experience, a Gemini experience, or a ChatGPT experience and go, Whoa, Microsoft's product, well inferior. They think they are just using Copilot. And because Microsoft has three core Copilot products now, Chat, M365, Copilot, and Consumer Copilot, People are not realizing they're not talking about the same thing when they talk about it. And they shouldn't have to realize. That shouldn't have to be an education lesson you give to somebody. And it goes back to Skype. When Skype was purchased, they took, what was it, Link? Link was it called back in the day?

00:19:13 Simon Hudson
Yeah, OCS and Link.

00:19:14 Mark Smith
Yeah, OCS and Link. And then they're like, oh, it's called Skype for Business. And it started this trend when I installed Windows 11 for the first time, all of a sudden I got Teams. Oh no, it's not Teams as I know M365 Teams, it's the consumer version of Teams. Call your products different names in the market and stop automatically creating confusion for people and have people use, you know, it's kind of like they've got the tail of the elephant and going, yes, I understand what this is. Someone else has got the trunk and your experience is way different of mine because it's a miscommunication that has been introduced by their own doing. So what are your feedback on me?

00:19:55 Simon Hudson
Well, first of all, bloody hell yes. Microsoft is great at many things. Consumer markets and branding are not amongst them. I mean, their branding has got less worse. And yeah, I don't think my MVP status is at risk for me saying that because I've been saying-- Copilot Studio, right?

00:20:13 Mark Smith
Copilot Studio.

00:20:15 Simon Hudson
Yeah, it just goes on and on and on. All the sort of missteps, the branding 101 missteps, et cetera, that Microsoft continue to make. And I think actually the two things related, I think it's because they're not intrinsically a consumer-facing organization. That's why they don't really understand how to do branding as well as some other companies, technology or otherwise. So I don't fully hold it against them, but we do need to recognize that their branding strategy usually is inadequate and often embarrassing. I think that you're absolutely right that the apparently reasonable thing of calling all of these personal interaction AIs are the same thing, i.e. copilot, is a mistake. I get why they would want to give that as an umbrella term within, say, the 365 suite. So, you know, all 120 different types of copilot AI that sits within 365. I call them copilot. It's just easier. You know, I get that because they're driven by their context. But you're absolutely right. As soon as we cross that barrier between the consumer experience and the inside 365 experience, that should very much be different. I mean, I don't really know how good or bad the consumer experience is because I don't really use it. I never touch it.

00:21:41 Mark Smith
I never touch it, right? So yeah.

00:21:44 Simon Hudson 
All the people that I know are using Microsoft 365 personal or business and therefore are using the 365 experience within that now. They may not have all the loveliness that we get in the fully paid for licenses in professional 365, but the experience in 365 personal is a full blown, if constrained, copilot experience. But the one that you get for free, it doesn't surprise me that people are confused and disappointed because it isn't giving them this parity of experience. So I completely agree. And I hadn't really thought about how shockingly bad that was until you said it. Yeah. Again, Microsoft are really bad at making sure that there is a frictionless experience as you transition from one, yeah, one part of their technology stack to another. I mean, I am constantly, I'm bloody good at this. You know, I've got a certificate that says so. And I go, where the hell is that feature? It's like, oh, it's because the UI in In this part of Copilot, it's completely different to the UI in that part. If I'm in Teams or I'm in the app, I'm in the web experience, I'm on the mobile. I mean, just deeply dysfunctional UI or UX. So yeah. So yes, I'm not surprised that it's giving them a bad name. And I do think that it will bite them and some more heads for role and they still won't get it right because they don't fundamentally understand how consumer and ordinary people experience is different from corporate and techie experiences. That's four different things, not two different things.

00:23:27 Mark Smith
Yeah. Interesting. Back on my intelligent brain, what other things should I be considering in a business to, you know, where that kind of the sweet point of Microsoft Teams, SharePoint, and Copilot, the sum can be so much more than the individual parts that you're seeing coming together? I mean, I'm not necessarily including agents, but agents are also coming to that mix.

00:23:51 Simon Hudson
I mean, agents inevitably in that mix because we can't really say AI without agents as of about nine months ago, because Microsoft have made that the thing. But I genuinely do think as a business owner, this idea of having agents that have specialisations that can advise you effectively at board level, or at least a director role level, is a very useful one. I mean, the other one I'm working on, as opposed to being a fait accompli, is an ethics agent. I have a bunch of interesting thoughts about ethics, but we know that people are really bad at ethics. We know that People, I mean, they know what their ethics, they know what their ethics are, but they can't explain it. And they know that when push comes to shove, their ethics tend to flex in favor of personal and emotional motivations. So one of the things I want to do, I posited it a couple of years ago, is maybe AIs can be our ethical advisor, you know, our perfect self. You know, so we train it in what either our individual ethics are or, I mean, literally, I've got my, I set up my business principles there on the wall. train it on your business ethics, partly because you can then get the AI to hold you and your board team to account in meetings and decisions against what you said was your ethical position. But also, I think that there's a really interesting self-learning process in this. Training in AI makes you think about what it is you're trying to train it in. And most organizations don't really know what their ethical position is, so creating a thing to do it It makes them think about what it is that they really believe in as a business.

00:25:31 Mark Smith
See, in the governance layer of that system I was describing before, I put that, like, what would we be proud of? What will we be known for? What's our no-gos? Because what I've done is put all the negatives, as in... We don't do this. We don't take work for this reason, blah, blah, blah. So that when we're creating offerings, we don't get sloppy outside of the areas that we want to operate in as a business. And I'm saying just creating products because that was the thing top of mind. The brain is separate from that because then I start a whole new engagement when I'm going to create a new product. But why it forced my hand to do this is I found that with AI, I was repeating all the stuff over and over, which was, the canonical knowledge of our company, right? Almost like the DNA of our business in every facet of it. And just, getting it out of my mind, my other colleague's mind, and really getting it into a usable, repeatable repository that AI really, that's designed for AI to interact with it as much as people.

00:26:40 Simon Hudson 
No, I think that's really cool. And I guess as I think about it, off the top of my head, You could sort of split the benefits, the types of AI into three. I mean, that's obviously a very broad generalization, but there are the productivity tools like, you know, getting an AI in work to tidy up or write or summarize or review your documents. Those are really great individual productivity tools. And I think there's great merit in those. I think that there are, I think that people tend to only use a handful of those different tools, but they are good when they use them. And I think we've then got the stuff which is going on in the background built by the techies to actually take various business processes and enhance them using AIs. And those are not-- they don't really have an interface yet. They just get on and do cool stuff in the background. And then the ones that we've just been talking about are these human-like product assistants. And I think those are the three spaces. I don't really know where to go from that point. I think pushing out the adoption of the productivity tools to your standard staff across a business, I think that's important, but it's just letting them be more productivity. That's a really big thing. I mean, New Zealand's a bit different to the moribund Europe, but productivity growth in Europe has been in low single digits for decades now. So the ability to give staff, you give people a step change in their productivity, we could be getting productivity gains of double, low double digits, double digits by the thoughtful and robust adoption of productivity tools in our organizations, big and small. So that stuff's really important. What you've described is what I've described as your company security kind of agent, just that knowledge bank that enables people to go, what is our tax number? What is our, you know, being able to get that without interrupting others, really important. That contributes to much more productive people. But executive decision making, strategic work, I think that's a new frontier that hasn't been explored yet. And that could be game changing.

00:28:58 Mark Smith
That's why I like your ethics idea, right? Because all of a sudden, you can have that seat at the table that their sole focus is to go, Hang on a second.

00:29:07 Simon Hudson
But you said that you guys were all about this, and this sounds like it's like this other thing. So, yeah, yeah.

00:29:12 Mark Smith
I've had a few times where AI has challenged me on things that I have said, and then I've said something else that was out of step with what I said, and it pulled me up on it, you know, as in, it was amazing.

00:29:28 Simon Hudson
And it absolutely should. I think that there's, it has the capability of being more objective than people have. We sit here and get terribly excited about these conversations. The AI is completely impartial. It goes, yeah, but you said. And so with the same thing in mind, I'm also looking to create a legal advisor. And there's some other just sort of fairly practical things, but an innovation advisor, someone who just helps you do the thinking and then pushes you down the right pipeline and makes you do things like the due diligence and the simple checks. I mean, we talk about Microsoft getting their branding on. I mean, the number of times they failed to do due diligence on trademarks. But I see that with all sorts of organizations. Oh, it's just embarrassing.

00:30:17 Mark Smith
100%. And this is, you know, I've been in the Microsoft ecosystem 30 years now, and often I'm like going, it's like the organization doesn't understand its history. Like, you made the mistake eight years ago here, but the employee now has never been in the company then. They don't have any of that organizational history in their knowledge bank, and they just go and make the same mistake again.

00:30:40 Simon Hudson
Yes, exactly. And people are fallible, at least as fallible as AIs. We have other benefits, but the AI can be the one that goes, yeah, I remember from 10 years ago when we tried that and it didn't work. That doesn't mean it can work, but these are the things that we did last time that failed. What are you going to do to not follow that same path? And I think that's super powerful.

00:31:07 Mark Smith
We are way over time. I didn't mean to do 20 minutes, we're at 30. I love talking to you. Thank you so much for, you know, the chat, the dialogue, the challenge, the mental engagement. Thanks for coming on the show.

00:31:20 Simon Hudson
Always a pleasure, Mark. Keep coming next time, we've got something to talk about.

00:31:28 Mark Smith
Hey, thanks for listening. I'm your host business application MVP Mark Smith, otherwise known as the nz365guy. If you like the show and want to be a supporter, check out buymeacoffee.com forward slash nz365guyuy. Thanks again and see you next time.