5 Model Context Protocol Mistakes Killing Your AI Projects

[00:00:16] Ghazenfer Mansoor: Hello and welcome to Lessons from the Leap. I’m your host, Ghazenfer Mansoor. On this show, I sit down with entrepreneurs, founders, and business leaders to talk about the bold decisions, pivotal moments, and innovative ideas that shape their journeys. This episode is brought to you by Technology Rivers. At Technology Rivers, we bring innovation through technology and AI to solve real-world industry problems.

[00:00:38] Ghazenfer Mansoor: Most of our work is in the healthcare space, where we work with health tech entrepreneurs and health tech companies building innovative, HIPAA-compliant software products. If you’d like to learn more about us, head over to technologyrivers.com and tell us more about your project.

[00:00:53] Ghazenfer Mansoor: Today we are joined by Anuj Tyagi, a site reliability engineering leader specializing in AI infrastructure, with a decade of experience building and scaling cloud-native platforms. I’ll let Anuj introduce himself. Anuj, welcome to the show. Tell our audience how you got to where you are today, and then we’ll continue afterward.

[00:01:19] Anuj Tyagi: I have a background in computer science from my studies, and then I got into systems and networks early on. After that, when I moved to the US for my grad studies, I did my internship in the content delivery space.

[00:01:43] Anuj Tyagi: From there, I moved to an organization where I worked more with open source, and then I moved further into the SRE domain specifically. I was working with large-scale systems, since CDN is also part of that, and around 2021 I started working with the organization where I’m currently working.

[00:02:13] Anuj Tyagi: It’s more in the AI space. The team’s stack was mostly working with Kubernetes and providing AI services to different products. That’s where I gained my interest in AI, and I’ve been working specifically with AI infrastructure for four to five years now.

[00:02:40] Anuj Tyagi: And for the past year, it’s been more in the agentic tech space, following how the trends are moving. Everyone goes with the flow. So yeah, that’s how it is.

[00:02:56] Ghazenfer Mansoor: You just said everyone is going with the flow, so I have an interesting question. Obviously, everybody’s building something with AI, AI POCs, whatever, but not many are getting to production. Why do so few make it to production?

[00:03:17] Anuj Tyagi: I think one of the reasons is that in the past you needed coding knowledge to write an application, to build a POC. Now you can just give prompts and build a small experimental app. For a POC, it’s just to test if the idea works or not.

[00:03:49] Anuj Tyagi: But after that, if we think about applications running in production, meaning we’re expecting real customer traffic and the application needs to handle it, that takes more experience, and prompts alone won’t help with that, because you need a reliable system that your traffic

[00:04:17] Anuj Tyagi: The customer experience will be consistent even when traffic is distributed from different locations, and customer queries could be different from your own test queries. You need to make sure your application can handle traffic spikes. So, in the end, it comes down to reliability, and that’s where the site reliability engineer comes in.

[00:04:45] Anuj Tyagi: Are we using caching where we need to? Are we able to handle API timeouts if they happen? For example, there was downtime with Anthropic’s Claude APIs recently, I’m not sure if it’s fully resolved, but the application should be able to handle that. And if you need high availability, that’s another factor.

[00:05:12] Anuj Tyagi: There’s a lot in the system architecture you need to handle. Production isn’t just about sending a query to the LLM and getting everything done. So, if one person tries to bring an application to production at a small scale, it can work

[00:05:36] Anuj Tyagi: But if you want to move it to a good amount of traffic, you’ll need more experienced, specialized people, and there’s a budget involved when you’re building larger systems.

[00:05:52] Ghazenfer Mansoor: Yeah, absolutely. This is something we notice in our business too, a lot of misunderstanding. People still assume it’s like ChatGPT or Claude, you just load a document, query it, and it gives you results. They’re using it more like a traditional search rather than going deeper.

[00:06:17] Ghazenfer Mansoor: The foundation is the most important part. If you don’t understand how the AI works, your expectations shift. You give prompts, generate output, and expect the same result every time, but it won’t always be the same. Understanding how these things actually work is more important. It took people a long time to understand how the product really works, and a lot of people still don’t.

[00:06:50] Ghazenfer Mansoor: So, knowing that is most important.

[00:06:53] Anuj Tyagi: Yeah, earlier, machine learning existed, and now with LLMs, responses are based on probabilistic behavior, but we need deterministic responses. That’s the whole point of things like agentic tech or

[00:07:17] Anuj Tyagi: RAG: We want the response to be more deterministic than probabilistic. We don’t want random responses; we want precision. That’s why a lot of systems are involved in the pipeline: training, testing on training sets, validation metrics, and then it goes to production after being verified against test datasets.

[00:07:47] Anuj Tyagi: That’s why things get complicated.

[00:07:51] Ghazenfer Mansoor: While we’re on agentic AI, what makes it different from a simple chatbot or RAG application from an infrastructure standpoint?

[00:08:00] Anuj Tyagi: Early on, we were just using LLMs. For example, when ChatGPT came out, it was trained on a dataset as of a certain point, say a year back, so it only knew data up to that point. But what if we want

[00:08:23] Anuj Tyagi: But what if we want to do more than just ask something in the chat, like get recent information it wasn’t trained on? It won’t have that data, and in some cases, it can give hallucinated answers when it doesn’t know something. In other cases, it won’t even attempt a response, it’ll say something like, I’m only trained on data up to a certain year, and I don’t have anything more recent

[00:09:00] Anuj Tyagi: With agentic AI, what changed is that it allows you to use external tools. For example, a web tool lets it search outside, pull data from external sources, schedule calendar appointments, check your calendar, or take any action you need.

[00:09:26] Anuj Tyagi: And with that, you get MCP, Model Context Protocol. Each organization or use case can have its own MCP servers, and we can build a client and connect to them. Every MCP gives you a standard way to connect to agents.

[00:09:47] Anuj Tyagi: That lets us access multiple tools through each MCP. But we need to limit that to some extent, because having too many tools or too many MCP servers in an agent can sometimes hurt application performance, from what I’ve seen.

[00:10:11] Ghazenfer Mansoor: True. And as you mentioned MCP, for those who aren’t familiar, MCP is Model Context Protocol. It’s similar to what Zapier or an API is for regular systems, but for AI. You can connect any system using MCP and query it, rather than just loading data into

[00:10:39] Ghazenfer Mansoor: Claude or ChatGPT. You can use MCP to connect to any system and run AI searches through it.

[00:10:54] Anuj Tyagi: Right, and one advantage is that if you want to use an agent for something like Confluence or Jira, instead of connecting to multiple separate agents, you can just connect to an MCP, and the MCP has different tools that act as agents in the background.

[00:11:15] Anuj Tyagi: So, from our application’s perspective, as engineers building this, we don’t need to create separate connections for each service. We can just connect to one MCP, and it decides which tool or agent is useful for a specific task.

[00:11:40] Ghazenfer Mansoor: While we’re on this, what do you recommend? In what scenarios should teams self-host a model versus using hosted LLMs?

[00:11:55] Anuj Tyagi: In cases where a general-purpose LLM is used, it has data collected and trained from public sources, so it’s like a smarter version of Wikipedia, in a way.

[00:12:16] Anuj Tyagi: And it can now take actions too, with more agentic capabilities on top. But if you want to build a product-specific chatbot for customers, a lot of organizations are doing that now.

[00:12:34] Anuj Tyagi: For example, if you want to take orders for a restaurant and show its menu, a customer might ask something specific, or an e-commerce customer might ask about the return policy for a product.

[00:12:55] Anuj Tyagi: Using a general LLM for that may not be very helpful, so it’s better to use an internal RAG application or a small language model trained mostly on your internal data, since you don’t need external data for that kind of application. So yeah, that’s one of the main use cases, ideally.

[00:13:19] Ghazenfer Mansoor: So how do you keep costs from exploding?

[00:13:24] Anuj Tyagi: There are multiple ways. It’s similar to what happened around 2014-2015, when the cloud boom started and everyone moved from on-prem to cloud, or at least some of them did, so you didn’t need your own hardware.

[00:13:43] Anuj Tyagi: Then, after some time, we had to manage cloud costs and optimize them. The same thing is happening now with AI, we started moving to AI, and now we have to manage AI costs too. Cost management is where reliable systems come in. For example, if we cache responses that are more frequent, we can keep some retention

[00:14:16] Anuj Tyagi: so, we don’t need to send a request to the LLM every time. Another approach is adding guardrails. In the past, I’ve seen examples, even externally, where a chatbot built specifically for a restaurant gets random, unrelated questions.

[00:14:45] Anuj Tyagi: If the model responds to those anyway, we’re burning tokens unnecessarily. With guardrails in place, we can block those kinds of queries and save cost. Another use case is protecting against attacks, which also happens in some cases.

[00:15:08] Anuj Tyagi: That’s not purely about cost, it’s also partly a security concern, and the defense mechanism for that is essentially guardrails. If someone tries something like prompt injection to pull a lot of data, that can burn a lot of tokens too, and it can be avoided.

[00:15:37] Anuj Tyagi: So yeah, those are some of the main use cases.

[00:15:42] Ghazenfer Mansoor: Thank you. Can you talk about hallucinations? Most people load data and don’t get the right results, and the results keep coming out different. What are the reasons behind that?

[00:15:58] Anuj Tyagi: I have more experience in this area, particularly with RAG applications. It comes down to a combination of machine learning engineering and general engineering skills, because when we’re building an application,

[00:16:19] Anuj Tyagi: We initially give it relevant prompts or AI messages and get responses back that match what we expect, and everything looks fine at that point. But when it goes to real users, who may ask random questions or use improper sentences, the responses don’t always match expectations.

[00:16:46] Anuj Tyagi: When we notice responses aren’t as expected, the next question is how do we detect that. In a RAG-based system, we have a lot of metrics that help us see how the application is performing. For example,

[00:17:11] Anuj Tyagi: Some metrics help us understand how hallucination is happening. First, if the model isn’t trained on certain data and a user asks about it, in some cases it can still give a response, and that response is actually incorrect, it doesn’t even exist in the data.

[00:17:38] Anuj Tyagi: This happened once in my experience: a user asked a question, and the LLM gave a fully confident response, complete with a confidence score, a URL, and steps to follow. But the URL didn’t exist, and neither did the steps. It just made something up based on other data it was trained on.

[00:18:09] Anuj Tyagi: So how do we catch that? We can’t manually check every user query and response, that would be difficult. Some metrics help here, especially around the data retrieval step, like context precision: how relevant or precise the retrieved data is.

[00:18:36] Anuj Tyagi: There’s also context recall and noise sensitivity, general metrics for RAG applications specific to the retrieval step. The second part is the generation step, where we look at context recall, response relevancy, how relevant the generated response is, and faithfulness, another important metric that measures whether the response is actually grounded in fact or just made up.

[00:19:19] Anuj Tyagi: Faithfulness, in my view, gives a good estimate of whether the application is hallucinating in its responses to user queries. And another factor is whether a query is burning more tokens than others, we can evaluate that too, and in some cases

[00:19:44] Anuj Tyagi: for example, if you’re using MCP and ask for some action, but the MCP server can’t figure out which tool is correct, it might loop trying to work it out, and if it gives a hallucinated response, or eventually says it doesn’t know the answer.

[00:20:10] Anuj Tyagi: That’s also a case where hallucination can occur. So, checking response time or token usage for responses matters too.

[00:20:22] Ghazenfer Mansoor: Yeah, you’re right. LLMs are the most confident ones, they always give you an answer even when they don’t know it, and they sound so confident you assume it’s true. There’s always an answer for something, especially with ChatGPT, and it will make up a response if it doesn’t know one. That’s the dangerous part, because people start believing it.

[00:20:53] Anuj Tyagi: Yeah. I recently contributed a Grafana dashboard with a Grafonnet config in its description, which lets people monitor common metrics and gives ideas about what can be instrumented and how. I used Open Telemetry for that, which is a common tool for collecting metrics, logs, and traces.

[00:21:26] Anuj Tyagi: That can be useful for anyone who wants to use it. On saving tokens, I’d add that to increase reliability and save tokens at the same time, we discussed caching frequent query responses and blocking harmful queries.

[00:21:50] Anuj Tyagi: In some of those cases, LLM gateways or agent gateways are very helpful too, since they let you do all of that: caching, adding reliability by load-balancing queries across multiple models, or adding a fallback mechanism.

[00:22:14] Anuj Tyagi: So, if one LLM fails, you can switch to another. If OpenAI is down, you can use Anthropic’s models, or even within Anthropic switch between higher and lower tiers, which also helps manage cost. For queries that require more mathematical work or more computation, you can use a higher-tier model, and for smaller tasks you can use a cheaper model, which also saves costs.

[00:22:53] Ghazenfer Mansoor: A couple more questions. You mentor a lot of engineers and work with many others, so based on your experience, what separates the people who successfully transition into AI from those who struggle to make that leap?

[00:23:10] Ghazenfer Mansoor: AI is obviously changing everything, so engineers now have to work differently than before, changing how they code and how they use AI tools. How easy is it for engineers to make that leap? What’s the difference between those who can’t adapt and those who quickly adapt to AI and become far more productive?

[00:23:39] Ghazenfer Mansoor: Do you see any gaps? Any differences? What are they?

[00:23:44] Anuj Tyagi: I’d say, given how quickly technology is moving, it’s difficult in one way, you either need to stay in the same domain or keep following how the trends are moving.

[00:24:10] Anuj Tyagi: There’s a lot of information overload nowadays, and it’s not easy to keep up with how AI is progressing. I’d say, like with any other skill, if someone is trying to learn, it helps to be more disciplined and learn steadily, rather than rushing to learn everything in a short time. Try to get good at one part first. If you’re just starting out, try to learn

[00:24:48] Anuj Tyagi: If you’re just starting out, try to learn about agents, starting with the foundations, and seeing how this works in production, or at a larger scale, how the application can be scaled. In some cases, a machine learning foundation is helpful too. In other cases, from a developer’s perspective, when we’re building applications, other foundations, like caching, using async communication tools like Kafka or something for queuing, still matter.

[00:25:34] Anuj Tyagi: Machine learning is really just a feature or subset within that. For building an application, you still need a front end, like TypeScript or React, those are still needed. Some people can build parts of it with AI, but to make a reliable application that can handle thousands of customer requests

[00:26:09] Anuj Tyagi: And to stay at risk from security attacks, you need real experience with professional application development. So, the foundation still matters. On the SRE side, it’s not just about sending logs, it’s about how you trigger alerts, decide on them, set error budgets so you’re not overspending, and decide on guardrails and evaluation metrics for the dev environment before production.

[00:27:00] Anuj Tyagi: Different test cases still need to run through the application: integration tests, stress tests, and more.

[00:27:10] Anuj Tyagi: I should probably use these more.

[00:27:13] Ghazenfer Mansoor: Yeah, good point, really good insights. In my view, it’s more of a mindset shift. AI is changing everything, so if we keep thinking the traditional way, the leap becomes difficult. Engineers need to change how they work. This ties back to something we talked about at the beginning of this podcast:

[00:27:42] Ghazenfer Mansoor: How do you go deeper? It’s not just about giving a broad prompt and expecting it to build the whole application. If that worked, people would have finished applications in five minutes. But it takes iteration, going deeper, it’s really about problem-solving. You start with one thing and gradually build from there. That’s the most important part.

[00:28:11] Anuj Tyagi: Yes, exactly.

[00:28:11] Ghazenfer Mansoor: So, thanks for sharing this, Anuj. Really good insights. Just a couple more small questions. What’s one AI tool or platform you can’t imagine working without?

[00:28:24] Anuj Tyagi: I’d say

[00:28:26] Anuj Tyagi: AI-based IDEs used for development. They’re very helpful, they can help you review code quickly and answer questions you have about it. With a large codebase, that can save a lot of time, especially in my domain, where at any point there can be

[00:28:53] Anuj Tyagi: a production incident. If I need to narrow down which microservice has the problem, and there could be different error codes involved, there’s usually documentation, but it may only give one reason for an error. If I want to dig deeper and ask more questions,

[00:29:20] Anuj Tyagi: An AI-based IDE acts like an assistant, and they’re very helpful. I won’t name just one, since I use a few different ones depending on the situation.

[00:29:35] Ghazenfer Mansoor: They won’t say anything bad about you, don’t worry. One last piece of advice for our audience: anything on the future of AI, or anything people should be doing? Any nuggets before we close off?

[00:29:53] Anuj Tyagi: Yeah, what I like right now is how the industry is moving toward this broader idea of an AI harness. It’s no longer limited to agents or LLMs alone, we’re now looking at the bigger picture: context memory, building reliable applications, adding observability, and all of that. When we talk about that, we’re really talking about the whole ecosystem around an application

[00:30:29] Anuj Tyagi: Adding guardrails, feedback loops toward a goal, and making the whole thing more deterministic. With that, I think it’s already becoming part of our applications.

[00:30:47] Anuj Tyagi: But now, in production, we’re not just focused on building an AI-powered application, we’re focused on the whole set of features. Context memory was a hot topic a few months back, and now it’s becoming normalized, and I hope it’ll be able to help humanity in some cases.

[00:31:20] Anuj Tyagi: Like in healthcare, as you mentioned, and in other use cases too. That can reduce some of the fear-mongering happening out there, but AI will do its work in a positive direction for us. That’s my expectation for the future.

[00:31:46] Ghazenfer Mansoor: Thank you, Anuj. We’ve been talking with Anuj Tyagi, who has shared such valuable insights with us today. Anuj, this has been an incredible discussion. You’ve helped us see that the future of AI won’t be defined by the smartest models alone, but by the infrastructural ability and engineering discipline that bring them to life. Where can people learn more about you and your work? Is there a website, an email, a LinkedIn?

[00:32:21] Ghazenfer Mansoor: We’ll add it to our podcast notes.

[00:32:24] Anuj Tyagi: Yeah, I have a dev.to blog, and I write sometimes as an Alibaba Cloud Community Builder. I’m also planning to start an Instagram page for sharing knowledge, but that’s moving a bit slowly. I also write sometimes on LinkedIn, sharing updates, and I have a LinkedIn page for

[00:32:51] Anuj Tyagi: AI Techna, my nonprofit for mentoring people in AI, SRE, and cloud engineering. That LinkedIn page is something people can follow, and it has a GitHub repo under the same name, where I sometimes post open-source work I’ve been doing.

[00:33:11] Ghazenfer Mansoor: Thanks, Anuj, thanks for being a valuable guest on Lessons from the Leap. And again, to our audience, this episode is brought to you by Technology Rivers. At Technology Rivers, we bring innovation through AI and technology to solve real-world problems. We do this in two ways: first, by helping businesses streamline and automate their operations through AI and technology, and second, by partnering with startup founders, entrepreneurs, and product owners to create innovative software products, SaaS platforms, and web and mobile apps.

[00:33:42] Ghazenfer Mansoor: Most of our work is in healthcare, where we work with health tech companies to build secure, HIPAA-compliant applications. If you’d like to learn more, head over to technologyrivers.com and tell us more about your project. Thanks everyone for listening to this episode with Anuj Tyagi. Anuj, thanks.

[00:34:00] Anuj Tyagi: Thank you very much for inviting me.