Moving Beyond Chatbots and Automation

Mark Smith: 0:14
Welcome to the Copilot Show, where I interview Microsoft staff innovating with AI. I hope you will find this podcast educational and inspire you to do more with this great technology. Now let's get on with the show. In this episode, we'll focus on AI trends and, of course, agents will also be discussed. My guest is from Massachusetts in the US. He works at Microsoft as a principal product manager. He's a technologist, foodie and archer. Now that's something new I did not realize. His experience is across energy sustainability, retail manufacturing, healthcare and finance, and he sharpened his ability to design AI-powered solutions for complex challenges. You can find links to his bio and socials in the show notes for this episode. Welcome back to the show, Ashish.

Ashish Bhatia: 1:03
Thank you so much, Mark. It's always a pleasure.

Mark Smith: 1:05
It's good to have you back on. It's been over 100 episodes of past. You were on last time, episode 530, and we're chasing down 700-odd episodes at the moment, so it's a long time ago.

Ashish Bhatia: 1:18
Yeah, phenomenal growth.

Mark Smith: 1:19
I love it. It's good. It's good. I tell you what, as in the world is changing rapidly for me, in that my whole world seems to be evolving around AI nowadays more than ever, and so I'm starting to drift slightly from my business applications roots, as my universe is absorbed by AI, and so I always love discussing AI with you. I find your posts insightful that you often post on LinkedIn. I highly recommend people follow Ashish on LinkedIn to get those insights, but before we go there, tell me about Archery and this classification of foodie what's your latest cuisine that you're into? What's happening in your world?

Ashish Bhatia: 2:00
Yeah, world is good to me so far. Again, who is not kind of riding the wave of AI, I would say. And again I'm enjoying them, having a lot of fun. As you mentioned, I've been with Microsoft for about 13 years. All those 13 years have been as an AI product manager in different kind of teams, different roles, latest with BizApps for the last three years. But I've seen a bit of the transition moving away from machine learning, ML kind of now called as more traditional ML, machine learning, data science to the generative AI and agentech world.

Ashish Bhatia: 2:36
It's super interesting, super fascinating to kind of follow that trend Outside of work. I kind of enjoy kind of more than food. I like to cook stuff.

Mark Smith: 2:46
Nice.

Ashish Bhatia: 2:47
And that kind of is a relaxing habit for me. Over the weekends I love to kind of listen to podcasts or follow a video or whatever and just multitask cook stuff Archery is kind of meditational. Or follow video, whatever and just multitask book stuff actually is kind of meditational for me. It's super relaxing. I do competitive shoot and that's kind of nerve-wracking, but other than that, whenever I have four in my hand, I'm smiling and enjoying myself.

Mark Smith: 3:12
That is so cool. It's so cool. You mentioned podcasts. What are your top three podcasts focusing around ai that you're kind of you know and I don't know whether they would stay consistent but flow in and out?

Mark Smith: 3:27
You know, last week I was in seattle, of course, with the mvp summit and one of the the guys that was speaking from the ai platform, that workshop is using in the mvp space and one of the areas that covers his trust with the ai. But he was just talking, it was a panel and and he was like I was listening to a podcast on the way here and it was covering all the latest stuff that open ai had released in there for a model with image generation right and like it only been announced at morning. I'm in the session, you know, passing it photos and and giving it a go, but I'm really interested because there's a plethora of ai related content, like when you think of shows that you kind of you you've got on your listen from an ai perspective and of course I assume this is much broader than just your Microsoft world you're wanting to get. You know what, what's the temperature and what's happening in the market. What are your go-tos?

Ashish Bhatia: 4:24
for AIs. I mean, there are a bunch of podcasts that I follow, but for AI specifically, vez Roth does a podcast. He's more sensational and kind of stuff like that. Like chronography is very interesting sometimes, but for me if I need to get the latest stuff in the day of ai, so he's a go-to because he's always. He always has the latest content was that viz viz rock?

Ashish Bhatia: 4:48
roth roth r-o-t-h yeah and v-e-s is that you're saying w-e-s awesome, okay, I'll check that out. Yeah, he posts interesting stuff. The other one that is for a deeper dive for me, which is ai explained yep, that I kind of follow. I am also a subscriber to philip's patreon, which gives a little bit more content, uh, in there. So he does very critical analysis and in-depth of new AI trends and not just trends, but new model rolls out and he has his own kind of benchmarking data set that he kind of validates stuff.

Ashish Bhatia: 5:27
So he brings in very insightful stuff, non-hypey, again down to the ground, in-depth. And there's another one called Training Data. They bring in a bunch of leading air labs and again go in depth into new stuff. So they were interviewing how, for example, operator works, how deep research works and stuff like that. So that that's interesting to understand as well very cool, very cool it's.

Mark Smith: 5:52
Uh, I'll go take a listen to those. What are you seeing, or what is your current sentiment on AI, the changes in the market? It seems that even in the first part of this year we're three months down now and we've still had a lot of releases of new models, new tooling and even from your perspective inside microsoft and what microsoft is doing, I'm sensing there's more of a stabilization coming to the ai and everything, and now we're starting to see it really stabilize. I suppose is my observation inside microsoft from all, from in the first days, of just jamming coilot into everything. What are you seeing in the market, in Microsoft? What's your observations of the AI world at the moment?

Ashish Bhatia: 6:42
I see there's multiple kind of layers to things. The top layer is kind of where the industry is going and, if you see that a lot of questions being asked of all the capital investment, we're kind of in that phase too where we pass kind of the hype, hype. People are putting a lot of money and investors are asking. That phase too, where we passed kind of the hype, hype, people are putting a lot of money and investors are asking customers are asking return on investment, return on value, and we're starting to see a bit of that right that we're now getting grounded in hey, what's the value? What is it doing differently from everybody else? So we're chasing value right now. I feel the model will keep improving. This kind of parity, I feel and that I mentioned as well right, commoditization of ai models. You look across top three, four kind of model companies, ai labs, the models are at par, almost like give or take a few capabilities. So that trend will continue to be the case and again, I think it'll come down. I'll always quote this that'll come down to like cloud right. You take top three or four cloud companies, yeah, pretty much in parity on capabilities, right, even at price point. In that sense. Yes, it comes down to where is your data, where your kind of team skills lie. Who do you trust most? Where do you have the most relationship? Where are you more more comfortable buildings your things? So it would come down to that. I feel that's where people will gravitate, naturally. I mean folks who are aligned with google tech, they'll use a lot more of gemini. Who are aligned with microsoft, they'll use a lot more open ai and azure serves, and then same for aws as well.

Ashish Bhatia: 8:14
On the capability side spectrum, I feel you have started to see kind of first party agents from a lot of companies. Microsoft announced a couple last week. We have seen a few from OpenAI now and we'll see more of these, and so we're getting into the stage of agentification of business logic. Right, we're moving away. Or gradually we're gonna move away from apps to agentification of business logic. Right, we're moving away, or gradually we're going to move away from apps to agentification of things. And you were there last week right At MVP Summit. You saw a lot of kind of that trend in our own announcements and things that we are starting to work on.

Ashish Bhatia: 8:52
So we're moving away from the chatbots of the world to more of assistance of the world right, is this assistants, which can take action? The reason I'm kind of differentiating between the two is chatbots were reactive you ask a question, it responds and probably brings in some insights and value. All of that. With assistants much like human assistants, you can get them to do some tasks, but again on your behest, you're asking them to do something and they they're acting on that. So we'll see a lot of that where connecting value and sometimes me weeks or days worth of value is created in very short amount of time.

Ashish Bhatia: 9:28
Right, deep research is a very good example of that, but we're still not in that state we can totally trust the outcome.

Ashish Bhatia: 9:35
Yet and I've seen this in my own experimentation deep search you have to go next level. So it kind of we're still not in that stage of like the next step where you can offload things to agent, trusting and believing they will do stuff on your behalf the way you would do to a human assistant, human intern and things like that. I think we're not there yet. From tech perspective. One of the key things that's missing for me is just capability of these agents to retain context memory. You've seen a lot of kind of hey like short-term memory examples all around, but just long-term. An assistant is not an assistant if you have to kind of deal with everyday diminutia of the models. If you go to a new session, you have to explain yourself and you don't carry forth a lot of that context. That is the biggest kind of next jump, at least for me, for my own expectations for these things to become kind of so Satya, twice this year, I think, in podcasts.

Mark Smith: 10:33
That he's done and I'm loving that he's getting on to being guest in more guest in more podcasts. But a couple of observations of the podcast he's been on is that one he's a master of handling the interview or the interviewer. When they try to press him to go down a path he's not interested in going. He just, you know, without being rude, he just handles it. But he also he's dropped some clangers, you know, and I assume the biz app team, particularly from his podcast in january, were left reeling when he basically said basically crm, zrps, all these type of, we didn't say that, but these big apps are kind of they will disappear, right and and we won't need them. And I've been thinking for a while about things like power apps, whether it be Canvas or Monk Driven Power, automate, that these are all incredible tools that I don't think will go away. But people interacting with them will go away. In other words, people building solutions with them will go away. They'll be building blocks underneath the surface and the concepts of menus, the concepts of lists, et cetera, I think will disappear within the next five years because you'll just be displayed the context of whatever you're wanting to work on and things like that.

Mark Smith: 11:51
One of the things he's also brought up, which is going back to your what you were saying there, is that he believes that 2025 will solve the mem, will solve memory.

Mark Smith: 12:01
Right, this concept of you know, I shouldn't have to keep explaining myself, you know, like I've got prompts that are multi-paged in length, right, if I'm wanting to go, I want to write a piece and I want to go to a specific target audience. I've already done a whole piece of work about a year ago on who my target audience is, so I pre-prompt with that to kind of give context if I'm creating a piece of content. Now, what's your thoughts on memory? Because I have noticed in 4.5 model that it's definitely bringing in a bit of memory about stuff that I've discussed with it before. I brought that context into something I just yesterday was working on. When do you think we get to the point? Do you think it's in 2025 in your observations or further, before we kind of the memory issue disappears and it's just that's our expectation yeah, I mean the little bit of memory that you mentioned.

Ashish Bhatia: 12:56
I think it's. It depends on how you configure your chat gpt, remember. There's a config where you can go and specify who you are, right, yeah, and I feel it adheres to that pretty much. I mean, every time I ask it something to do creative and say, oh, as an air product manager, you might be interested in this, right? Yes, yes, yes, but that's not the context, that's something that you've handcrafted, right, yeah? Yeah, where is the conversation about? Or the memory that I'm interested in is like you and I have this conversation. We spoke almost a year back or more. Right, we still might have some context of that. We can bring that in. Yes, this is just organically in our context of our one-on-one conversations, like that kind of memory. Yeah, and I feel again, not in-depth into that space, but I feel that could easily be a year plus after right, we might do a little bit more on short-term memory in this year, but kind of real agentic memory.

Ashish Bhatia: 13:56
The way you would want human assistant to have is probably a year or so away let's talk about agentic and agents and the marketing as opposed to the reality.

Mark Smith: 14:09
Particularly when I hear this term autonomous agent and I'm just like it kind of rings my bells and that I don't believe anyone's built an autonomous agent as yet and I don't see that autonomous agents are, I can't see, in my field of view I suppose I'm saying at the moment that scenario in that all the agents I'm seeing created today involves a person being very prescriptive, adding all the if then type statements into the you know configuration.

Mark Smith: 14:44
I'm like, well, hang on a second, I'm just building automation here and I was with donna sakar in van the week before last and you know she talked about the autonomous region of France where you can buy not sparkling wine but champagne, right, and she was saying that these automations or this agent we're really talking about sparkling automation at the moment. Right, we're not there on the champagne and having that real authentic, where I feel that and I keep using a travel agency as an example if I go to them and ask to arrange me a trip somewhere, I don't have to tell them every single little detail, they know already. I don't have to tell them how to plan it, how to book it how to go to galileo or armadillos or whatever system they're using and and go through that. They know how to do it because that's their job. Where do you think we are on this spectrum from your perspective, and are we getting our definitions correct, or are we just at the behest of marketers at the moment?

Ashish Bhatia: 15:53
I'm with you on that right. I feel we have reincarnated a lot of automation of the past process automation into agentic automation. Yes, and that's like we're in step one or step zero or step minus, whatever you want to call of automated agents and think of. If you think of automated agents, like what kind of automated agents would we imagine something that can work on your behalf without you having to explicitly explain? You tell the task and they go off and do stuff. Right, you still want them to work on your behalf and again, not you triggering them, but you kind of initiating hey, I want this to be done, can you take care of catering? Can you take care of drinks? Right, and somebody you know and trust will make the right decision.

Ashish Bhatia: 16:44
And if they run into a conundrum of hey, what I was told or initially asked for, I have a couple of options. Maybe I should ask right, yes, that's when you come back and ask for hey, a or B or ABC, which one? But then you get that succinct view of what ABC looks like and as a decision maker you can make that decision, but otherwise the heavy lifting is done for you. Yeah, the automation that we are automated agents or agents that we are building are very much still in. Do this action and somebody has curated that action, like yes or that path should be in the if then else of that or side cases, for that, right, we're not that in there yet.

Ashish Bhatia: 17:27
I feel in terms of, uh, true, automated which can apply reasoning Again, I think the building blocks are coming in place. To be honest, right, if you look at deep reasoning models, they are falling in place and they will be the ingredient for the automated agents of the world, because, remember, agents started to fall in place on top of, I would say, if I use open AI language, on 4.0 generation of models, the 0.3 generation of agents are to be built right.

Ashish Bhatia: 17:59
And that is where I feel we're in, the kind of build phase on top of those models, so even the deep reasoning kind of models or the operator model, right.

Mark Smith: 18:11
Yeah.

Ashish Bhatia: 18:12
Operator probably is a good example, but that process needs to get so much more efficient. I feel I can't sit through that painful process and watch every single click.

Mark Smith: 18:21
Yeah, exactly.

Ashish Bhatia: 18:22
So that again, it is step one of that thing. It will become better. But I would want my automation to be like that right when I can say hey, hey, go, look for three options for travel to this destination, give me assistant reports of all of these and let me make the choice, and then you're done at a personal level, what do you think your first agent will be?

Mark Smith: 18:45
and I'm talking about when it's so in my mind. I think of health. So I want an agent that knows you know, I wear two different type of electronic a ring and a watch that's giving health data that it can monitor. It can I have supplements?

Mark Smith: 19:04
I work out there's a range of things that I do and I have this idea that at some point I want a health agent which is always factoring in all the you know, my sleep patterns, environmental variables, et cetera, and totally optimizing to me, not to a healthcare provider or anything else, but it is just my dedicated to keep my body in absolute optimal state.

Mark Smith: 19:25
And then I think I wouldn't mind a mental agent and what I mean by that, that someone that says hey, you know what, based on your stress levels or whatever we this is the actual exercise I'm going to take you through and we're going to do some meditation and you're going to go out, you're going to go for a walk or whatever, because of the mental clarity that you need to develop at this point. And so I want that one you know that agent to be really tuned in on me. And I've got about five because I I think that you know. The other one is finance. I want somebody you know, an agent, that's across all my financial scenarios, investments etc. And is always optimizing, you know, for the best case. Do you think about this type of thing and rather than the business of developing this for others, but how you think about it in your own life?

Ashish Bhatia: 20:16
So that's great. I mean, I've not thought deeply about it, to be honest, but my first agent or the most use of these bots that I do today, and I have a huge appetite for learning and embracing new content, new stuff.

Mark Smith: 20:29
Yes.

Ashish Bhatia: 20:29
Always kind of on a daily basis I look out for, hey, what's the latest ai news of the day and stuff like that. So I've set up a few of these scheduled tasks, if you will, to kind of look out. So that is my first kind of agent of choice. If I would write, hey, figure out what happened today and what should I know today, and then give me all of that in like some audio form factor, yes, in like half an hour or an hour, right. So an hour right, when I'm either cooking or doing dishes or just through the normal chore, I can kind of listen and embrace all of that.

Ashish Bhatia: 21:06
That is just such a time saver for me. Yeah, yeah. And then much like you said, right, finance, right, in my daily kind of routine, all of that stuff. I just don't get enough time to think about it deeply and I feel if I can offload that to some trusted friend, that is what you want from an agent, right? Some trusted agent who can work on your behalf and only your goals are important to them and nobody else's.

Ashish Bhatia: 21:29
Yeah, and that would be really kind of a very good use of agentic capabilities.

Mark Smith: 21:36

Yeah, I totally agree, and, by the way, I didn't invent those three for me, darius Amadei. He wrote this piece about a year ago now around the five areas that we really need to be thinking about how we apply AI to ourselves and our future. But at the moment I'm not seeing anything that's glimmering to provide that for me yet. And another interesting metaphor I saw recently was that if you apply for a job, right, you get a job description and the job description is, let's say, 10 or 15 things. It's your job. But after two weeks in the job, you know that job description wasn't complete, right, because you absorb tacit knowledge transfer. You know there's so much more. Nobody says, oh, by the way, your job involves sifting through 200 emails a day to look for the signal to noise ratio in that email and this is what you know.

Mark Smith: 22:24
I think that an agent in the future needs this ability to self-learn and adapt right to our situations to really provide the type of value needed. One of the things that I'm seeing is the word productivity as a sales tool for, let's say, co-pilot, and I heard somebody exclaim the other day that they are a very productive person. And I'm not looking for more productivity, I'm looking for more time back, right. I don't want to work 60, 70-hour weeks. I want to work 40-hour week and give me back some of my time. I want to see my kids more. I want to go home on time. I want to see my kids more. You know, I want to go home on time, and that's their hope. Is that AI is going to provide that world for them where they'll be able to just get stuff done in a timely fashion so they can live their life. What do you see?

Ashish Bhatia: 23:50
Yeah, I see. On the contrary, to be honest, it depends on personalities as well. Right, how much work you want to do, how much time you want to spend on stuff On a daily basis. I feel, even all the AI systems that we have, I have. I still leave work on the table, yes, to be caught up next day. Yes, yes, yes.

Ashish Bhatia: 24:09
And it kind of piles right. Step one for me would be to not have that stage. I still have to review if I'm passing, hey, let's rewrite this document. If I get a rewrite of something, I still have to review all of that. So I'm constantly in this state of curating my thoughts somehow, augmenting it with AI and then still spending time in review. I might not be spending less time in the back and forth. Maybe I'm getting better outcome, better kind of draft or the doc. That definitely is the case, right, because I'm using it as a bouncing pad, right that let me throw something, get something back. Each day I'm actually spending more time, to be honest, right, I feel, when I'm writing something that way. So I'm not getting time back for sure. So, again, that for me would be the next level of productivity where I can kind of not leave stuff on the table to be caught up later in the night or whatnot, and having to spend less time overseeing the work of AI.

Ashish Bhatia: 25:13
Yes to spend less time overseeing the work of AI. Yes, and that's why I mentioned right. We are kind of in this phase two of we're getting assistance but we're not giving off agency yet. The agency is still with us and the rework and the reviews and the overviews are still with us. The next big step for me step up would be where we can let go some of that, but for that, hallucinations have to come down. Ai should be able to review its own work. Right. It should be able to deeply understand your context, where you're coming from, whether it's business context or personal context, and then give you something that is easy to digest and say yes, no.

Mark Smith: 25:51
Yeah, exactly, filter out the signal to noise ratio, right of work. We're a quarter of the way through this year calendar year. If you were to be standing talking to me in December in your mind, what do you think our achievements will be In Copilot Studio? What do you think our achievements potentially could be in the future state?

Ashish Bhatia: 26:15
that could be achieved only in the next nine months yeah, copilot studio as a product has come a long way from its pva pedigree. Right, and remember, I mentioned that we're building these agents on the next set of models, the reasoning models now, yes, so I'm super excited about looking forward towards the end of the year as to seeing the fruits of that right Now, bringing the deep reasoning capability into the orchestrators of these models, into the brain of these agents. How better the outcomes are, those are yet to be seen. Right. Agents how better the outcomes are, those are yet to be seen. Right, that is super interesting because we made a generational leap towards the end of the last year in terms of model tech. And will that actually cause a generational leap in agentic capabilities?

Ashish Bhatia: 27:06
Is yet to be seen, because agents are a more complex system than an LLM call right, yes, yes, you need to stitch up tools that it has access to knowledge, that it has access to context that it should have. There is a lot of plumbing beyond the model itself or the orchestrator itself. That needs to come together and then would people trust them? Right? Is there observability around it? Can you monitor it when it goes wrong? Right, it would not have a way of self-correcting but at least identifying coming, so you can observe it right. As a builder, you can kind of tweak things right. We're not in the state of self-correction yet, but that's what I would hope towards the end of the year. Where we are, we're seeing far more intelligent agents operating.

Mark Smith: 27:54
Do you think we'll get beyond having to choose models? In other words, you know, at the moment it seems like the list just keeps growing longer and longer and it's like, do I use that one? Scott hansman said something interesting last week, which is you should choose the smallest model closest to you from an efficiency point of view. Yes, and it's once again just sound bites like that, that a paradigm shifts and going hey, my laptop now is a co-pilot laptop, a PC, so it's got a high spec neural hardware on it and I've got one terabyte of storage just sitting for running my own LLMs, which I've done on my desktop computer. But do you think we'll get to the point where we don't have to think about which llm we're using, potentially even maybe which provider of the llm, but we're going to get the correct output based on our engagement?

Ashish Bhatia: 28:48
that is appropriate yeah, I mean, you hit so many things that I have so many thoughts of. So, interestingly right. So first of all, I mean this came up in our in my MVP session as well. Right, we'll get model routing very quickly across the board.

Ashish Bhatia: 29:03
So as a consumer of some bot right Copilot, chat, gpt, cloud, whatever right we will get model routing very quickly. So as a user, you don't have to decide, hey, which model should I hit? You will be served the right model for the intent which is probably cost effective for the providers as well.

Ashish Bhatia: 29:21
Right, because if I'm for whatever summarization task hitting 4.5, it's not a good use of 4.5 in that sense. Right? So that for their own CapEx, opex perspective providers will fix. So that is the small uplift. We'll see that in months, I would say, if it's not further away. The other thing how do you choose a model? That came up as a conversation last week. A lot and my pet peeve on that is start with the beast of the model that you can get Ensure that your task is accomplished, because if your task is not accomplished you have not created the value that you were looking for.

Ashish Bhatia: 30:00
So go for the outcome, optimize for the outcome. So take the beastiest model that you can and go solve it so you know that your scenario is solvable. Then come down the model generation side. So, if I just took an example, there was start with 4.5, come down to 4.0, mini, maybe 5. Keep going down to the smallest model you can until your scenario breaks. And then on the right is the best model that you can work with.

Ashish Bhatia: 30:29
We are optimizing for outcome, which is great at demo stage. Right when you're demoing, you're showcasing. It's good because you have something to production. You need to watch out for price to performance yes and that price performance will only come down to have you done the due diligence of what's the right model for the task, because otherwise you might be using the beastiest task a model and would not help your price ratio.

Ashish Bhatia: 30:54

Yes, so that's another one. The other thing was we were talking at the local college here, Babson College, at the AI and sustainability. From a sustainability standpoint as well, there's a huge kind of number to watch out for, right. What is the carbon footprint of that one single query that you have done? And if you're using far more superior model than you need, you are actually kind of putting carbon credit. You're burning more carbon than you need to. So that's another kind of responsible thing to do as a builder and it'll help your pockets.

Mark Smith: 31:27
Yeah, totally agree, we're running out of time. We're already 10 minutes over. It's so interesting talking to you. Any final words actions for me to go do before I let you go.

Ashish Bhatia: 31:39
Final thoughts would be again this year. People said this is the year of agents. We'll kind of see a lot of that come real. What I would love to see is what do people want from these tools that are being served to them? Right is, what do people want from these tools that are being served to them right? From our vantage point, we have some point of view.

Ashish Bhatia: 31:57
I feel a lot of folks are still building with wipe checks, right. It kind of looks okay. I feel it gets the response right. We need to move beyond kind of wipe check to concrete validation. We need to start thinking of these things as software systems, right. Yeah, so what do people need there, right? Where are we not thinking in terms of observability, monitoring and then really building these as, like, robust systems? That's been where I'm spending most of my time these days, kind of deeply thinking about and I would love to again your feedback on that stuff to see, hey, as I'm building these things, what do I need? Right, if I really need to put into production, put my name tag to it and say, hey, I'll build this, what am I missing right now?

Mark Smith: 32:40
yeah, it's always a pleasure to talk to you. Thank you for coming on the show absolutely pleasure is mine, thank you hey, thanks for listening. I'm your host, mark smith, otherwise known as the nz365 guy. Is there a guest you would like to see on the show from Microsoft? Please message me on LinkedIn and I'll see what I can do. Final question for you how will you create with Copilot today, ka kite?