Hello my friends and welcome back to the channel. This is Noah with Learn Meta Analysis and today I need to just rant a little bit, but I'm going to teach you something as as we go. So, I keep hearing from people, just rag it. Why didn't you try rag? And very clearly they don't understand what rag is. So, what we're going to do is we're going to talk a little bit about rag today and why I generally don't recommend academic rag anymore for trying to understand what's going on in a field. So, this is going to be a really quick video. I know it's going to sound a little bit ranty, but bear with me and probably learn something about rag. And when you decide to disagree with me, leave notes and comments cuz I know you guys are out there and you're going to like rag is the greatest. So, I'm going to start off by saying there's like 25, 30 types of of rag that can happen. I'm going to be talking in very high-level generalities, okay? And my overall message, it's not the end-all be-all answer to all life's problems with AI, okay? It's just one tool in the toolbox. It's very good at what it does, but it's not always the answer. So, let's take a look. All right, guys, here we go. Why am I ranting today? Well, I'm really, really, really sick of people saying, just rag it. I hear this all the time. I see it when I get reviews back on my papers, I'll see people saying, you didn't try rag. Well, the reason I didn't try rag is cuz it has nothing to do with what I'm trying to do. So, most of the time that I hear people say, just rag it, and they use that kind of terminology, it lets me know that they don't know what they're talking about. And it's kind of embarrassing because I am not a computer scientist, I'm just a nerd who likes AI. And if rag was the right answer, I would have tried rag, right? So, this video I hope will be helpful for all of the people out there who are like, oh, just rag it, but they don't really know what they're talking about. I'm going to teach you the basics of rag right now. Okay, so what is rag? Rag is retrieval augmented generation. It's a lot more fun to just say rag cuz retrieval augmented generation kind of long. So, what does it do? Basically, we are going to find chunks of semantically relevant text and use that as context to help an LM create an answer. That sounds kind of convoluted, but essentially we are going to help an LM find the right answer based on information we gave it. That sounds awesome in theory, right? Sounds so cool. Well, here's the thing. I know this is a a big graphic. It was created with AI. That's why there's this little ugly thing in the bottom right cuz it said something about cost-efficiency that was wrong. But bear with me here. So without rag, if you're just like chatting with an AI, it's really only going on based on the data it knows, right? But with rag, we can provide some documentation first and help ground the LM in that information. So in theory, we have some pretty cool benefits of rag, which is why you always hear just rag it, right? So we can potentially reduce hallucinations. Not always, but sometimes. We can make sure that there's fresher data in there. We're no longer limited by the training cutoff data, which is often 2023, 2024, 2025. We can get cited sources depending upon the the actual software that we're using, where it can tell us where it got that information from, and this can be really helpful, especially in academic settings if you want to know like what paper said something. That can be really cool. Last but not least, rag's usually free or cheap to run on your own machine, so it can be very cost-efficient. So here's the thing. Here's how rag works at a really high level, and then I'm going to break it down because this sounds easy, but there's a lot of things that can influence it, okay? So first, you're going to ask a question to your LM. Your LM is going to go search your database, which is called a vector database or a vector store, to try and find that information. That information is going to come back to the LM, and it's going to get added to the original prompt as context to help the LM generate an answer. That sounds awesome. That sounds easy. Yes, for all you guys out there, I know this is a really high-level overview. There's a ton of different types of rag. Like I said, there's like 25 or 30. I'm talking in very big generalities here, so bear with me. Now let's break that down, okay? So let's look at a really This is a pretty basic rag architecture, okay? This is This is not a complicated one. This is an easy one. And here's what I want to show you. Number one, it's not as easy as just those four steps. And number two, there's a lot of factors that can influence each of those four steps. So, let's take a look at the beginning when we are first setting up that vector database, okay? So, first we're going to have our input documents. In academic rag, this is our existing papers or studies a lot of times, right? So, there's a lot of things that can influence this like how well did your OCR work? How well can we actually get text out of whatever that document is? And how are we actually going to parse it? Those are big factors. Those all come before chunking. And chunking is one of the most important things. I'm going to have a slide here that shows you about chunking a little bit more to help it make a little bit more sense. But chunking is not straightforward and easy. There's a lot of different strategies you can use to chunk something. So, imagine an entire paper and you have like an introduction, methods, results, discussion, conclusion, right? Well, typically when we set up chunking in a really basic structure of rag, we're not actually looking for those sections. Instead, we're going to just pick a relatively random character or token count. Let's say 1,000 and then have them overlap. I'm going to show you why that's a problem on the next slide. Then we have to choose our embedding model. A lot of different embedding models and they're good at different things. Then we have to choose a vector size for those embedding models. That can also influence things. Then it gets put into our vector store, okay? That's all of the information to get into our vector store. So, we can see we have factors that can influence the effectiveness of this at all three steps. Your accuracy of your OCR, for example, that's going to influence the input documentation. The chunking strategy you use is going to impact everything else downstream. The embedding model you use is going to depend on how effective we are at actually retrieving things, right? So, all of these factors matter. So far, we're at three major factors, each of which have lots of smaller things within them. And I just put two on here or I should say I just had the the image model put two on here because it was easier to understand. So, let's look at what happens next when you actually ask a question, right? So, So, going to ask your question. It might be written well, it might be written really crappy. If you use LLMs like me, you probably wrote a really crappy question for the LLM to understand. I don't usually take a lot of time uh to ask really nicely clean questions. I just type in whatever I want. So, you have that. Then, we need to actually retrieve that information from the vector store, right? So, it's going to go, it's going to retrieve what it finds to be most similar. Now, what influences this? Uh all sorts of things. How about that? All sorts of things can influence this, such as your top K, uh if you're using hybrid search or not. There's lots of things that can influence the retrieval step. Then, we have an optional step that a lot of people do called reranking. There's a lot of factors that can influence this, such as, you know, what model are you using as your reranker? And what factors are influencing that that reranking? Then, the information is going to get augmented and put into your LLM as additional context, right? Because as we talked about before, when we retrieve that information, we are simply adding it to the existing context that the LLM has. And there's things that can influence here. Context window, uh with modern models, it may not be as much of a factor because we have such large context windows. And then also the order of the context. So, like, what is the information actually in there that the LLM is seeing? Then, it's going to actually generate its response, and we have all of our normal LLM-related factors here, such as our system prompt, our temperature, top P, all of those things that would normally influence your generation, are also influencing here. And then, last but not least, it spits out an answer, right? Which will have things like citations in it. And hopefully, you have some sort of way to check for hallucinations. So, what's our takeaway from this slide? Well, we got eight steps, roughly, and all of them have, at least on the screen, at least two things that influence them. That's a lot of squish factor of where things can go wrong, right? So, let's take a look at chunking because this is the piece that I find a lot of people don't understand and has a huge impact on what we're doing. So, this is just a very simple couple sentences. So, I'm going to read all these together, and then we're going to talk about how it chunks. Retrieval augmented generation relies on breaking down large documents into smaller pieces called chunks. These chunks are converted into vectors so an AI can find the relevant information quickly. If a chunk is too small, it might lose the context of the surrounding sentences. To prevent losing information, blah blah blah. We taught and then it goes on to talk about other things. So, if we look here and we say our chunk size is 100 characters. Now, in real life you're probably not going to have 100 characters as your chunk size, but this is to make a point, okay? Imagine your entire paper is let's say 8,000 words and your chunk size is 1,000 words with 250 words of overlap or tokens, you know, exchange the word the word words for tokens if you want if you like to think about it that way. What's going to happen here? Essentially, we're going to get probably or likely end up with chunks of paragraphs that are split between chunks. We'll end up with some chunks having sections of like the literature review while other ones have sections that are based on the introduction. The methods might get overlapped with the results, etc. So, we can end up with all of these chunks that like yes, they do match a certain number of characters, but they don't necessarily contain fully coherent information. And so, there are other sorts of chunking that you can do like you can do markdown-based chunking and all of these other things. But, the point that I wanted to highlight here is your chunking strategy matters a lot and if you're just setting up a really basic rag pipeline, your chunking might be making it so that your rag pipeline isn't working very well. Okay, so let's check out this next one. What can we do instead, right? Like if I'm saying that academic rag isn't generally a great idea for actually chatting with papers, what can we do? Well, here's the thing. Most modern models have a really big context window. Like 128,000 is kind of the standard that I know of now and that is a lot of papers. How many papers is that? That's like 12 8,000 word papers, roughly. And that is a relatively small context uh for modern models. A lot of modern models, you know, go all the way up to a million context. So, imagine what you could do here. Yes, we will see some some potential for what they call context rather as the context gets longer and longer. But, we can fit all these things into context where it's going to make more sense to the model. So, why on earth would we want to break it up into random chunks and then put it in a database and then use a bunch of different models to go try and retrieve chunks that may or may not be similar to what we actually want and then get a partial answer. Right? So, here's the thing. When RAG first came out, it was a couple years ago, right? 2022, 2023, and models only had a context window of like 4,000 tokens. That's so different than today. And models, a lot of the modern models perform really well at high context. So, if I was wanting to chat with a paper, personally, I would probably put the entire paper directly into it and not mess around with the RAG pipeline. It's easier. We can put the whole paper into context and we can get answers that are probably a lot more sound than what it would be if it was trying to take the results from a RAG database. At least that's been my experience so far. So, the real thing that I need you to consider here is what is it that you are ultimately trying to do? Because the question that I have is why bother trying to chat with a really large corpus of papers? Like, what is the point of that? Let's say, like previously in a video I showed having like my entire Zotero database uh synced into a vector store and then using RAG with Open Web UI as a knowledge base to like actually go and query it. It was kind of cool, but I got a lot of wrong answers, right? Like, you get you I just ended up with a lot of wrong answers coming out. And I know there are definitely different ways you can do RAG and there's different chunking strategies, different embedding models, some are better than others for different things. You can use re-rankers. The LLM that you choose is going to matter in how well it can do these things. That's so many factors. So, the question that I have for you is why are you trying to do RAG? Because if you just want to chat with a couple papers, you may not need to you do RAG. You can just put them all into the context window and get the information that you need. That might actually give you better results. What I'm saying is you don't necessarily want to have your default as rag. Do rag. Don't Like, in my opinion, rag is often not the correct situation. Now, I'm not talking about corporate environments, I'm not talking about customer service agents, I'm not talking about any of those things where rag makes great sense, right? There are plenty of situations where rag makes great sense and it's a great tool. I do not think that chatting with papers in academia is that particular case and I also don't think that all the all the research that I'm doing right now where people say just rag it or you didn't try rag. That's also definitely not the case, but I'm not going to rant about that here because I wanted to talk specifically about chatting with papers. So, if you're going to be chatting with papers, I think that the general idea you'd want to do narrow down to the papers that you really want to chat with and focus on those. They'll probably all fit in the context window if you have a reasonable amount of papers depending upon the model that you're using. So, love to hear you guys' thoughts on this. I know that a lot of people are probably going to say, "What about learn LM?" Learn LM's pretty cool, right? I mean, I haven't used it in a while. It's probably been, I don't know, a year, maybe 2 years. I used it when it first came out because it it was based on rag and it would give us sites and sources and things like that. It was really great. But, ultimately, I don't find myself using any of these tools anymore. Like, honest truth, I'm not doing rag anymore. I'm not using learn LM anymore. Uh I'm not really using any of those things because ultimately it's still an LLM and you can't really trust it, right? Because it can still hallucinate. Even though you're using rag, it can still hallucinate. And that's the main thing I want you to keep in mind. It can still be an issue. So, I'd really just encourage you to take a deep dive and think about what is your goal and is there really a point to trying to chat with like a huge corpus of papers? Is there some reason that you're doing that as compared to narrowing it down and chatting with a smaller subset or, you know, maybe just taking a completely different approach. So, that said, I'm done ranting for now. Thanks for listening. I hope you learned a little bit about rag. If you have any questions, feel free to drop them down in the comments. If you think I'm a complete idiot because you think RAG is the best thing ever, you're welcome to drop those in the comments as well, but you can also just keep it to yourself. You know, you can you can also just keep it to yourself. I know this was a very high-level overview. I know that there are definitely use cases where RAG makes a ton of sense. I just don't think chatting with academic papers is one of them because we need perfect accuracy. So, anyway, hope you all have a wonderful day. Please like and subscribe if you like this video, and I will see you all in the next video.