We have two new state-of-the-art video generators that were launched recently, Seed Dance 2.5 and Minimax H3, and both are ridiculously good. In this video, we're going to go over all the incredible things they can do, plus the pros and cons of each, where to use them, their cost, and much more. Let's jump right in. Now, the nice thing about both models is that they are multimodal, which means not only can you give them text prompts, but they can also take in images, videos, and audio to use as references. Now, both models are already capable of a ton of stuff, like taking a video of a green screen and replacing the background, or replacing and adding characters in an existing video. You can also change the weather, the lighting, the camera angle. This is all easy stuff for both models. In this video, I'm going to test them on more extreme and challenging and practical stuff. Let's first talk about fight scenes and other high-action scenes. I'm going to use Seed Dance 2.5 first, and by the way, I'm using this platform called Lumina by BytePlus. This is an all-in-one creative platform for using the latest generative models from ByteDance and other providers. In fact, because Seed Dance is multimodal, let me show you a really cool example where I can input a rough 3D animation like this, plus two images of reference characters, and here's my prompt. I'm going to write, "This character and this character are fighting in an ancient temple. Refer to this video for the pose and movements." And that's pretty much it. Now, because the original 3D video is 13 seconds, I'm going to select 13 seconds down here. The maximum resolution right now is 720p, and then I'm going to click generate. So, that's the result from Seed Dance. Note the character consistency is incredible. The physics and the motion are pretty much flawless. Now, there is one part in the middle here where it decided to do a close-up face-off between the two characters, even though this actually wasn't in the original 3D video. So, I'm not sure where it got that inspiration from, but for the most part, it is very coherent with the motion. All right, next I'm going to do the same thing in Minimax. I'm going to enter the exact same prompt. The 3D reference video is 13.6 seconds, so here I'm going to set it to 14 seconds. And for high low, currently you can only select 2K resolution. Let's click create. >> [snorts] [snorts] >> All right, so that's what I got from Minimax. And as you can see, Minimax wasn't actually able to follow the 3D reference. There's a lot of noise and artifacts throughout this fight scene. So, in terms of high action shots or fight scenes, Sea Dance is the clear winner. By the way, if you're wondering how to make these 3D assets and animations, there are plenty of free resources you can use. One of them is called Mixamo. You can sign up for free, and it contains a ton of pre-made animations for 3D characters like walking, running, fighting, dancing, or other specific actions. And you can adjust the camera angle to whatever you want, and then press download. Another really helpful open-source platform is called Arty by Nvidia, which I featured on my channel before. And here is where you can create animations of characters moving just with a text prompt. Anyways, back to the video models. For the next test, let's see how good it is at following instructions. So, I'm going to take my classic prompt, a 3D Pixar princess getting chased by a dragon, but I'm going to add a ton of different actions in the scene and see if it can follow them completely. So, it's going to be a 3D Pixar animation, a princess wearing a glittery white dress running away from a massive dragon with glowing red eyes in a forest. The dragon breathes fire, which ignites the ground and the foliage behind her. She ducks beneath a fallen tree as the dragon crashes through it. A flock of glowing blue birds suddenly swarms the dragon's face. The princess grabs a hanging vine and swings, landing inside an abandoned golden bathtub. The bathtub rolls downhill with her inside it. She jumps out moments before it smashes into a boulder. Reaching a river, she jumps onto drifting debris to cross. She jumps from a broken door to a floating barrel, then onto the back of a confused giant turtle. She looks back to see the dragon on the shore roaring in frustration as it can't cross the river. All right, so that was a mouthful. Now, my favorite feature about SeaDance is this allows you to generate videos of up to 30 seconds. So, let me drag this all the way to 30, and then let's press generate. That was pretty much flawless. It's actually able to follow everything that I specified in the prompts. All right, next, let me do the same thing for MiniMax. So, I'm going to enter the same prompt, and then for MiniMax, it only allows me to generate up to 15 seconds. So, let's set that and then press create. And surprisingly, the generation from MiniMax is also really good. It managed to squish everything into just 15 seconds, and it got most of my prompt correct. It's only missing one minor detail, which is the princess did not jump on a broken door before jumping on a floating barrel. Whereas for SeaDance, it got all the details correct. Still very impressive though for both models. All right, next, here's a workflow which might actually be used in real life, especially for animation studios. Let's see if it can take a rough sketch animation like this and generate a fully colored production-ready clip. So, here's what I'm going to put. Based on this reference video, make a realistic high-action cinematic video of a Japanese sorceress fighting a massive rock golem creature. Make sure you use the exact poses, animations, and camera angles from the reference video. Now, if you compare this with SeaDance, note that SeaDance is a bit better at actually coloring everything and making it look like a polished production-ready animation. Whereas for MiniMax, it still kind of keeps some of the original sketch elements. So, in this instance, I'd have to give the point to SeaDance. Next, here's another practical use case. Let's try to get it to create a commercial from just a storyboard and an image of the logo. So, I'm going to write, "Make an ad for this luxury handbag based on this storyboard and this logo. Include an English voice-over." >> Infinity. A story of timeless elegance. Carry the infinite. >> So, that was SeaDance's generation, which is pretty good. Next, I'm going to switch over to MiniMax and enter the same prompt. >> In a world of fleeting moments, true elegance remains. A quiet confidence. A journey with no bounds. Infinity. Beyond time. >> Honestly, both are really good. Definitely the best models available for this use case. It's hard to pick a winner here, but let me know in the comments what you think. All right, let's step it up a notch. I'm going to upload some simple UI screenshots of an app plus the logo and get it to create a full motion graphic commercial from these assets. So, here's my prompt. Here's an app called Artisan Crafts. It's a marketplace for people to buy handmade local crafts. Make a professional ad about this. Include voice over from a British female. I'm testing if it can do different accents. It should be flat vector motion graphic style. Only use the elements from the attached images. Here is the logo plus the screenshots. Take elements from these attachments for inspiration. This time, let's set it to a vertical video and then press generate. >> Discover the story behind every craft. Handmade, local, and one of cord. Find [music] treasures that tell a story. Shop with purpose. Download Artisan Crafts. >> I'm also going to enter the same prompt into Mini Max and then press create. >> Looking for unique handmade treasures? Welcome to Artisan Crafts. Discover local artisans and beautiful collections. Easily browse through one-of-a-kind pottery, jewelry, and textiles. Find your perfect piece and check out securely. Support creators today. >> Both are very similar. If you look closely, there are some slight errors with some of the letters, but other than that, both are pretty decent. Let me know in the comments which one you prefer. All right, here's another practical use case. Let's see if it can create music videos. Because both models are multimodal, you can upload audio for it to use in its generation. Let me input this input track which was generated from the open-source music generator called A-Step. And then I'm going to upload this image of a fictional K-pop band plus some typography references. And here's the prompt. Make an MV from this song. Show these K-pop members from this image singing and dancing to the music. Add coarse grain glitch effects and grunge effects. Keep the edit fast and use hard cuts only. Cuts should occur within 3 seconds. Make the cuts based on the beat of the song. Use typographic reference from this image. Now, because the song is 15 seconds, let me also drag this to 15 seconds and then press create. >> [music] >> All right, so Minimax's generation is amazing. The song is exactly the same as my upload. The animations and the cuts do sync with the beat. The characters are singing along and the topography also looks great. This is really impressive. Next, I'm going to run the same prompt using SeaArt Dance 2.5. >> [music] [music] >> Now, the generation from ByteDance is a bit weird. So, in terms of generating music videos or videos with these text overlay effects, I would have to give the point to Minimax. I'm really impressed by its generation. On my channel, I featured so many different AI models and tools. It can be very overwhelming. What if you can use all of these models all in one platform? And that brings us to Chat LLM by Abacus AI, the sponsor of this video. Chat LLM is an all-in-one platform for you to use the best AI models out there. You can seamlessly switch between different models in your chats. Plus, you can also use all the top image generators out there as well as the top video models out there, all in one integrated platform. And they're usually very quick to add a model to their site once it comes out. Plus, if you're coding something, they have a really useful agent feature which can do some really complex tasks all autonomously, like creating PowerPoints, websites, and research reports. It's going to supercharge your productivity. Best of all, you can access all these AI models and image and video generators and deep agent for only $10 a month. This is way cheaper than if you paid for each tool separately. Definitely check out Chat LLM in the description below. Now, both models claim they support multiple languages, so let's see how good it is at handling all these different languages. First, we have a Chinese man saying this, then we have an Indian woman saying this, followed by a Spanish woman, German, French, Arabic, Korean, Russian, and Polish. Here are the results from both models. Now, I don't speak most of these languages, so let me know in the comments below if any model got any of the languages wrong. Which one do you prefer? Again, both models can already handle like 90% of use cases and scenarios. The prompts that I'm going to test next are really pushing its limits. These are prompts that no existing video model could get correct so far. So, my next prompt is this: a solo violinist playing the solo section of Vivaldi's Summer, first movement. It's actually testing the AI on a few things. First, it's testing if it can generate someone playing the violin accurately, including how they move the bow, where they press their fingers, etc. It's also testing whether the AI has knowledge on what this classical piece, Vivaldi's Summer, actually sounds like. A very challenging prompt. Here's the generation from Seedance. Now, I got to say the violin movements and playing look very realistic. The way she moves her bow, plus the fingers, are actually mostly in sync with the sound that she's playing. However, this isn't Vivaldi's Summer, so it doesn't seem to have this knowledge built in. Next, here is Minimax's generation. >> [music] [music] >> I would say this generation is not that good. It doesn't really understand how a violin should be played, so especially at the end there when he's playing those really fast notes, the bow movements and the fingers aren't really in sync with the music. Plus, this also doesn't sound like Vivaldi's Summer. Next, here's another tricky prompt which no AI video model has gotten correct so far. A professor explaining the Pythagorean theorem on the whiteboard. >> So, for any right triangle, a squared plus b squared equals c squared. Therefore, a squared plus b squared will always equal c squared, the hypotenuse. >> So, as you can see, both models weren't actually able to generate an accurate diagram of the Pythagorean theorem, or like a right triangle with all the sides labeled. They do know about the formula though, so both were able to get the professor to write out the formula. Here's another really tricky test. A motion graphic highlighting the country of Peru on the world map. The voiceover is a woman with an Indian accent describing the history of Peru. So, this is testing multiple things. It's testing if it can generate a world map and highlight Peru, plus it's seeing if it can generate a voiceover with an Indian accent, plus it's testing its knowledge on the history of Peru. >> Home to the legendary Inca Empire and a rich cultural heritage that endures to this day. Peru, a land of ancient mysteries, was once the cradle of the mighty Inca Empire. >> So, this was a fail for both models. It's not really good at generating world maps. It's not good at highlighting Peru or generating motion graphic style videos. All right, so those were some of my preliminary tests. As you can see, Seed Dance is really good at high action shots and fight scenes. In fact, no video model even comes close to Seed Dance quality. It's also great at character consistency and following instructions. If you have any 3D animations or sketch animations, Seed Dance is by far the best model to turn these into production-ready content. On the other hand, Mini Max seems to be really good at making music videos or videos with text overlay effects. It's actually able to sync the video very well to the music. Both models do really well in terms of generating commercials or generating content from just a storyboard reference. Next, let's compare the specs of each. For Seed Dance 2.5, you can generate videos in all these aspect ratios, and currently you can only generate up to 720p, although they are planning to roll out 1080p and 4K resolution in the near future. And currently, my favorite feature about this is this can generate up to 30 seconds in duration. And if that's not enough, you can actually take your generation and input it again to create another 30-second clip that extends from that video. So, in theory, you can kind of create infinite duration clips if you just keep looping these extensions. Now, as I've shown you in my demos, this is a multimodal model, so this can take in text, audio, images, and video to use as references. In fact, this can take in up to 50 reference inputs, including 30 images, 10 video clips, and 10 audio clips for its use in a single pass. So, it's incredibly versatile. Next, let's go over the cost of this. So, at least in the Lumina platform, let me just type in a random prompt. Let's set this to 16:9 at 10 seconds, this costs 460 credits. If I upload an image, it doesn't change the cost. This is still 460. But, if I upload a video, then the cost increases to 570. Now, at least on Lumina, you can subscribe to these plans, which would give you like a few thousand credits to start. And if you want to top up, then a thousand credits costs roughly $10. So, a 10-second clip costs roughly $5 to $6, depending on your input references. Definitely the most expensive video model out there. Let's compare this with Minimax. I can generate videos with all these different aspect ratios. I can't even select the 768p resolution. This can only generate 2K resolution by default, and it can generate up to 15 seconds. Let me show you the cost of a 10-second clip. Again, I'm going to just type in a random prompt. Just a text prompt would cost 120 credits. Now, if I input an image, this also doesn't change the cost. But, if I input a reference video, then it does change it to 252 credits. You can subscribe to these Minimax plans, which give you several thousand credits to start, or you can also top up credits. So, again, a thousand credits is roughly $10. So, one video generation would cost like one to three dollars, depending on what references you input. So, Minimax costs half the price of SeaDance 2.5. Now, in terms of using the model via API, SeaDance has not enabled API access yet. I think they are rolling this out next week. In terms of Minimax, API access is already out. So, for a 2K resolution video, it costs roughly 13 cents per second. So, here's the thing. Minimax is like half as expensive, and it can generate 2K resolution videos, whereas SeaDance can only generate 720p, and it's like two times more expensive. Now, the really cool thing is that for Minimax H3, they are actually releasing the model to this. So, at least at the time of this recording, there are still five hours until they release the model. Apparently, with the quantized version, this can even fit on just an RTX 3090. So, you don't need like a huge data center to run this. You can just run this on a mid-to-high-end consumer GPU. Super exciting, and I will definitely do a full installation tutorial on this once it is released. So, that sums up my review of Seed Dance 2.5 and Minimax H3. These are both definitely the best video models you can use right now. There are pros and cons of each, so hopefully this video gives you a good idea of what to expect. As always, I will be on the lookout for the top AI news and tools to share with you. So, if you enjoyed this video, remember to like, share, subscribe, and stay tuned for more content. Also, there's just so much happening in the world of AI every week. I can't possibly cover everything on my YouTube channel. So, to really stay up-to-date with all that's going on in AI, be sure to subscribe to my free weekly newsletter. The link to that will be in the description below. Thanks for watching, and I'll see you in the next one.

So one of the lead engineers at Anthropic just published this breakthrough article on X. It now has 4 million views [music] and in it he outlines the important changes that they've made to Claude code, which if you pay attention to can make your setup faster, can make your systems cost less, and just overall let you upgrade your agentic operating system. I read through this whole article and today I'll break down for you the six new rules of Claude code that it talks about, which some of them by the way are the exact opposite of the advice we've been following for months. And by the end I'm also going to share with you a skill that lets you automatically apply these improvements [music] to your own setup. And if you're new, my name is Jay. I spent over a decade working with brands you may know, have been in AI since my masters in data science. Now I'm running an AI business and one of the largest AI communities globally. Let's dive into it. >> [music] >> So to give you some context, this person Tariq, he's one of the more well-known engineers who is working in Anthropic and this past week he published this really good article called the new rules of context engineering for Claude 5 models. And this has been out for only a few days, but you can see already garnered something like 4.3 million views. So that's a very good signal that there's a couple of great nuggets to learn from here. Now it's a pretty long article so I read through it so that you don't have to and in this video I'll just break down all of the insights for you so that you can directly benefit from it. And with this article the core topic of it is this piece called context engineering, which if you haven't heard that term before it's probably worth stepping back to just understand what it is because whenever you work with AI agents like Claude or Hermes or Codex, then for sure you yourself even without knowing it have done a bit of context engineering as well. And Tariq mentions it here as well because when you send a message or a prompt to Claude that prompt is actually only a small part of the context that it gets. And a big part of the output is that your agent gives you is coming from your context. And just to hone in on this point, whenever you start a Claude session and you send your first prompt, Claude actually doesn't just work from that prompt or that message that you send, right? Because in this case the prompt is just a direction that you're giving to Claude, but it becomes so much more powerful if you have your context organized. And just to make it simple, whenever I talk about context in our community, I always use this arms framework with the core idea being that if you organize and engineer this arms framework properly as your context, then you're actually ahead of like 99% of Claude and agentic AI users. Because the entirety of your context that consists of the applications that you are using, which you have connected via tools like MCPs or APIs or CLIs. You have your routines, which are basically your scheduled tasks or your crons that you have set up. You have your memory, which are all the artifacts and all of the reports and documents that you have generated over time. And finally, you have your skills, which are SOPs or processes that you can actually invoke through skill commands that immediately just teach Claude how to do a given set of work. And so when we talk about context engineering, the way that I think of it is always just revolving around these four elements. And what Tarek is saying in this article is that because the way that Claude's models have evolved, there's actually a few new things that have changed quite drastically when it comes to operating or engineering the context for these models. And to give you a clear example of how drastic it is, at least in Anthropic's team, he is mentioning here that they actually removed over 80% of Claude code's system prompt for models like Opus 5 and Fable 5. And even just by doing that, they experienced no measurable loss on their coding evaluations. And this is quite a big deal and just shows you how far we've come in terms of the intelligence and the raw capability of these models. Because if you just take a step back, only last year when Opus 4 was launched and also coincidentally around the same time that Claude code was launched, the benchmark intelligence score of Opus 4 back then was only 31% on the artificial analysis index. Which just to quickly share with you what that is, that is coming straight from this company called artificialanalysis.ai. And their benchmarks here for raw intelligence is actually quite good because what they essentially do here is throw a a of really difficult tasks to these models across a variety of disciplines and score them out of 100. And this company is also backed by some big names in the AI field like Nat Friedman, who's the ex-CEO of GitHub, as well as Andrew Ng, who's the former head of Google Brain. And so, the point being that I think if you're looking for like one benchmark that is a good reference every time new models drop, I think artificialanalysis.ai would be a good source for you. And if you look at the chart here, the ones that are topping the leaderboards right now is Opus 5 as well as Fable 5, and they're getting a score of 60 out of 100. Which again, if you go back to our comparison here, that 61% is actually leagues better versus what we had only a year ago. And so, what Tarek was sharing in that article is that because these models are now so capable, there's actually a few new rules to understand when it comes to engineering the context of your workspace so that your agents can work more effectively. And so, when you're building out your own second brain or you're building out an agentic operating system for companies, then this is a good article for you to get insights from. And the great thing about what Anthropic outlined here is that they actually provided a then and now view. So, what are the rules that were true before, and what are the revised rules that we should consider now as we work with these agents. And so, I'll go through each one of these along with a few examples so that we can all understand it. And by the way, if you want to learn how to build and sell AI systems that businesses actually pay for, then that's pretty much all we do over at the Robo Nuggets community. Where not only do you get access to the Claude Living Masterclass, which we update every week and takes you from zero to mastery with the latest on AI, but you also get access to our Agents as Course, which walks you through how to actually get paid for all these AI skills that you're learning. You also get to be part of a genuinely great community of AI builders. In fact, you can see just some of the recent wins our members are getting from the program right here. So, if you want to start earning from AI, then check that just in the pinned comment below. Now, back to the video. So, a big one for us to understand is that if before it's important to give Claude a lot of rules, this time it is actually important to let Claude use judgment. And here Tarek explains it that when they first rolled out Claude code, they needed to be sure that Claude avoided worst-case scenarios such as deleting files. Because remember, the models weren't as smart back then. And so that meant that they would give particularly strong guidance and rules that might not always be true and is actually limiting Claude Fable 5 or Opus 5 in today's world. So the example he gives here because they deal with a lot of code documentation is that before in the system prompt of Claude code, they were actually giving a more stringent set of rules like this where they're asking Claude to default to writing no comments, to never write multi-paragraph docstrings, and basically just very specific rules. Which now with models that are much smarter than what we started with, it's actually better sometimes to just let your AI agent use a bit of judgement in order for you to get better results. So one example they're mentioning here is that they actually trimmed down that system prompt to just say this, to write code that reads like the surrounding code, to match its comment density, naming, and idiom. Now you might not be into coding and development yourself, but you can actually still take advantage of this new rule. So to give an example from my own second brain setup, which is basically how I'm engineering my context, I've started now to develop some skills that are literally just meant for Claude to surprise me with its output. And if I just look for that skill because I literally just named it as surprise me. I think it's this skill.md. And if I would look at the actual description of this surprise me skill, essentially it's one that I built in order for Fable and Opus to create really good front-end designs. If you just browse through some of the description lines of this skill, you can see that the direction here is for Claude to demonstrate extreme capability, taste, and artistic flavor. So it's really up to Claude to use his judgement in order to create better front-end designs. And to give you an example of some of the designs that it created, there's a lot of really good ones here depending on the niche that you are in. So there's one that has these kinetic dots design that looks pretty good especially when you are developing for the tech niche. There's is origami design that has even interactivity built into that. This one is quite good because it implements some really advanced ways to have more kinetic typography. And if I just scroll down here, there's a lot of optionality here in terms of different ways by which your AI agent can avoid that AI sloppy look that it usually defaults to. And the way that these were all created is because that surprise me skill just basically lets Claude use its own judgment in order to create effects like these that you may have not thought of yourself. And so the point being, especially when it comes to more creative thinking, you might actually want to just let Claude run with multiple iterations and multiple options instead of caging it with structured rules which may actually just hamstring your output rather than help you. Now, a related shift that is equally as important is this. Whereas before, if it was important to give Claude some examples, now Tariq mentions that it's even more important to give it design interfaces. And he elaborates it here where when they were designing the tool usage capability for Claude, before they were providing Claude some specific examples on how to actually use them. But with the newest models now like Fable 5 and Opus 5, what they found is that giving examples actually constrains them to a certain exploration space. And I find this to be true in our own work and projects as well. Because now it's really important for you to have your own design system if you want to stand out from the crowd and actually build your own brand. To give an example, a lot of the artifacts and builds that you see here in our platform like this getrubric.app website or its Rubric Flows tool which we use to visualize systems that we build out for clients or even the look of this second brain system that I made a video for before. And even these title slides that I'm showing in this video, all of this is being designed automatically by Claude using my design interface system skill. And if I were to just look for it, I call mine as /robo. And you can see if I zoom in here, whenever I create a Robo Nuggets branded material, I just invoke /robo and it's able to invoke this skill.md which is connected through this brandbook.html. And if I go ahead and open that, essentially what this HTML does is it just outlines the different rules when it comes to this design system. So, let me just shift that so you can see, and it provides color palettes, it provides some guidance on the voices as well as the fonts that we are using. It gives that visual style for the dot matrix effect that you're seeing a lot in the videos that I create. And essentially what we're doing here is we're not really giving examples to Claude anymore. Not specific examples entirely, so that it has more freedom to design, but still within a few guidelines that the design interface system is actually providing it. And so, if you were building out a design system for yourself for the first time, then these two rules that we just went over, you probably need to keep in mind. Because what I would do first is I'll probably ask people five to give me different design systems and give it feedback until we arrive at a point that we like that particular look. Then, let's say you actually like this Helvetica 26 Swiss brutalist design, then you can just ask Claude to create a brand book HTML file that provides rules of how that design system is built, and then just ask Claude to make that into a skill file, something like /robo, or in this case it's probably /helvetica. And then, the next time that you create some materials, then it will always look this good. Now, another important shift that the article talked about is what we're calling as progressive disclosure, which you can see Tarek differentiates versus the practice before, where you are putting all of the context up front. And just to elaborate on that, he mentions here that because Claude Code in the very beginning was focused on coding, their system prompt included or needed to include detailed information on how to do code review verification, which are all these details that are not always needed, but when they are, it was actually crucial information. But what has changed since then is that Claude Code and these new batch of models have actually gotten very competent at using progressive disclosure, which is essentially loading the right context at the right times. And a great example of this is your own claw.md because it wasn't that long ago that there's a common advice that if you want to make your claw.md as strong as you can, you would want to make that a central repository of every known practice that you might run into. But in today's world and with these more powerful models, your claw.md actually becomes more powerful if you make it function as a router to your tree of files. And just to make that point clearer, if I go back to my second brain system here, you can see that my claw.md is at the center of this whole second brain system. And essentially what you're seeing here, all of these nodes are simply all of the files in my workspace. But because of the amount of files and context that's already in here, you can see I have something like 57,000 files already in this whole workspace, then it doesn't really make sense if you try to stuff your claw.md with every single context and way of working and operational rules that you would like your agent to remember. And so here you can see how I represented it in our second brain system is that our claw.md is actually just a router, so that it knows when I'm asking or working with it for content, then it's able to just tap to this set of files over here. If I'm working with it for my community, then there's this set of files that it can work with. If I'm dealing with product development, there is this branch. Personal stuff is this branch, and all the business dealings will happen on this side. And going back to Thrivec's point around building a tree of files, if claw.md is a pointer to these different departments or set of files, then what I actually have in my workspace, which you can try and emulate as well, is to have sub routers that basically just let Claude find the right files within this specific department. Just to give one example, let's say for content, I actually have a file that's called content.md. And you can see here that it's basically a router that gives Claude some direction on where to find things whenever we're working on content. So let's say we're in the process of ideating content ideas, there's some skills here that it names that are useful for that intent. When it comes to researching or producing the actual lesson or video, then it has several markdown file references and skills here as well. And so the point being here is that as your agentic operating system evolves and you start to have a workspace for your company or for your team that becomes a bit more complex, what you can start to do is to actually identify the different departments of your life and of your work and actually build out these sub indexes in order to help Claude navigate your tree of files more efficiently. So that instead of just filling your Claude.md with every single fact about your business, then you can start to make it a lot thinner and a bit more efficient so that it only progressively loads the context that you need during the sessions where you need them. Which in its essence is what progressive disclosure really is. And also remember that this has implications on cost and how often you run into usage limits. Because before, if you have your Claude.md set up so that it's quite thick and is quite verbose, then what really happens here is if this rectangle denoted by the broken lines would be your session and your context window, then every session that you start with Claude, you actually use up this much tokens, which is equivalent to your Claude.md the moment that you send your first prompt. Which in my view is a bit of a waste, especially if as time goes by and you start to use Claude or any agent platform a lot more, then that token usage does rack up. Now in contrast to that, if your Claude.md is thin and only serves as a router, then the more that you interact with your agent, you actually start to realize the efficiency gains because you are not using up as much tokens with this Claude.md that is essentially a system prompt that gets injected and also uses your token budget allocation the moment that you start a new session. Now this next one is quite simple because before with older models, you may find that you needed to repeat yourself quite a lot in order for the model to understand what you mean. But this time, Tarikin that article mentions that it's actually better to have more simple tool descriptions. And this is probably something that you will just notice in the background because before if you recall, there were some advice where if you continuously chat with a model in a particular session and that session starts to build up context, there is a more noticeable case of what we're calling context rot. And a good example of how that was is that models before were more likely to listen to instructions at the end of the context window, which are the most recent messages, than the ones at the start. And so as a result of that, you would sometimes need to provide more repeated instructions for a model in order for it to understand or remember what you mean. Now, what Tarek is mentioning here simply is that that has actually changed because a lot of these models with table five and opus five are actually more intelligent and a bit more smart. Same with the example by the Entropic team here, where he's saying that in the past, their system prompt would sometimes have references to tools in the main system prompt as well as instructions in the tool description. So, they're basically just putting it in two places at once. But now, they could just delete those other repeat examples, which in their case would be in the tool descriptions rather than the system prompt. So, there's less duplication. You also save on the token cost simply because these newer set of models are much more intelligent versus what we started with. And that leads us to another shift, which you also might have noticed in the background, which is basically this concept of automatic memory. And this concept and shift is very simple. What Entropic's mentioning here is that before, they actually used to encourage users to save things in Claude's memory, which if you don't know, you can actually use the hash hotkey to write to their cloud.md automatically. But instead, at this point in Claude code's development, it now can actually automatically save memories that are relevant to the work and to you. Now, this particular advice and this saving memories, I do see it happen on my side, but generally, I think if you have a productive session with Claude that you think there's a lot of information there that would be useful to be logged or remembered by Claude, I think it's definitely still worth just asking Claude to remember those things depending on how your second brain system and your agenda operating system is set up. Now, to give an example, at least in how I do it, I built out this skill called {slash} calibrate. And this is probably one of my most used skills because whenever I end a session with Claude, I just do /calibrate. And what it does is it will just review that current conversation and applies the best updates to skills, to claw.md rules, to memory, and to workflows that we have so that it captures the learning in that whole session and applies it to our operating system. As to give you a view of my VS Code IDE here where I'm using Claude Code. You see I've used calibrate here and what that did is it just updated these skills that I gave some feedback for. And apart from skills, it also calibrates my claw.md, the content.md router that I showed earlier, my memory files. It logged a new recipe or format under my prompt packs and so on and so forth. And so you can see with this one skill and the more that you use Claude Code, you're actually starting to refine the way you work with them so that it gets to know you better. And last but not least is the transition from simple specs to richer references. And to me this shift is really important because I use it on the daily. So what Tarek mentions here is that previously, literally just a few probably months ago, there was this over-reliance on markdown files with creating plans, with creating assets and artifacts because the prevailing knowledge is that these markdown files are simple enough and light enough so that it would help Claude have proper references for things like code or specs or essentially plans for projects that you're creating with it. And obviously markdown files still very important like your claw.md and your skill files are still in the markdown format, right? But what we're saying here is that you don't actually need to be limited anymore with just markdown files. Because with this newer batch of models, it can handle increasingly more complicated references. And the one that I really like and probably start to use more now more than markdown files are HTML artifacts. And a good example of an HTML artifact or reference is that brand book that I showed earlier. Because if this were a markdown file, it probably won't be able to convey the same idea. It won't be able to show visually what are the color palettes of this design system. And so having a reference like this, which you can see is an HTML file, basically a hypertext markup language, that when you open them, it just allows browsers like Chrome to render them visually. Then, there's a benefit of number one, your agent being able to parse through it, because under the hood, this is all still just code and still just text. But, also equally as important is that you, as the human, should be able to see and understand these references, especially the more visual ones. And this is also important when you're communicating ideas to other people, just like what I'm doing now. So, let's say this document about this skill called Dr. Plus, which I'll go over in a bit. This is much more visual, much more engaging versus the say if I were presenting to you right now a markdown file, then this probably won't be as effective as how it looks right now. In fact, sometimes if I feel like I have a lot of token budget remaining during the week, and I want to understand concepts or new skills that I'm investigating or trying out, simply, I routinely just ask Claude to create HTML artifacts for me, because this is just much easier to review versus like a markdown file or just being locked in the chat window terminal session, where you need to read through a wall of text. So, this is just one example where I literally just ask it to create an infographic using that {slash} Robo skill, our design system, in order to understand what are these new rules that the article was talking about. So, I pointed Claude to that article, it gave me those six shifts, gave me some examples of what these shifts are pertaining to, and this gives me a more solid idea in a much faster way versus me just having to read through Claude's essay on what this whole article is about. Now, the good news about all this is that Anthropic has actually made it really simple to apply all of these best practices with the latest version of Claude Code. So, if you update your Claude Code to the latest, then you'll be able to also get this {slash} doctor skill, which if you invoke that command, what it will basically do is five things. So, it will check your install of Claude Code to see if there's like any broken or duplicate installs, any path file path problems. It also finds dead weight, so for example, your skills or MCP servers, and even your cloud.md, it trims it down so that it's optimized to be thinner, just like what we mentioned earlier. If you have hooks set up, which are essentially these deterministic codes that you may have already set up if you're a bit more advanced in Cloud Code, then {slash} doctor will call out these hooks that add a bit of slowness to every turn or every message that you send to Cloud. And then finally, it just reports its findings before applying those fixes. If I go to my Cloud and just show, you can see I ran that {slash} doctor command a while ago. Gives me a summary of what it found. So, it says here that my install is healthy and up to date. It gives me a lot of detail around the stuff that it recommends. But, at the end of this, it gave me these options on what I actually want to do. So, it gave some MCP servers that I might consider disabling, which I think we can do that. There's a couple of skills in here that I think I either might have replaced already, which we can probably archive. There's also a plugin that I was demoing in a tutorial previously. And then it also tells me here that I am a bit behind on the versioning because I think every day there's a new version of Cloud Code. So, you might find that to be true for yourself as well. So, if you submit that, that just gets your Cloud Code instance to be up to spec, let's say, to these best practices. However, with just this {slash} doctor skill, I actually found that it is a bit more basic. It doesn't really capture the learnings that Tariq was mentioning in those six shifts. So, what I basically did is I just made this {slash} doctor plus, which is a skill that you can just grab below. It's available for free. And what that does, if in case you want a more complete checkup, is that apart from doing all the things that {slash} doctor does, basically by just invoking the same {slash} doctor command that ships with Cloud Code, is it also does a check of those six shifts that we talked about. And you can see when I ran this on my device, says doctor's clean because I've already done it before this. And then the good thing about this is that it actually tells me with those six shifts, what are some of the skills and artifacts that I can actually optimize based on these new rules of Cloud. So, it's telling me that this last 30-day skill, which is meant for research, is hitting a lot of red flags based on those new rules. And I actually I'm not surprised with this because I remember with this last 30 days, it came out a few months ago already. So, it's telling me here that this skill is actually really thick. It's like 2,090 lines long. And we can probably optimize that by making this skill.md more of a router rather than dumping this whole 2,000 lines of context in there. But, there in essence, what you can do is to run this Dr. Plus skill and just do a more advanced checkup based on the shifts, based on the new rules that Tarek was mentioning in that now viral article. And again, you can just grab that whole skill, which you can import or customize for your setup or even set up a routine for. So, that let's say every month you run this Dr. Plus skill to do a check with your Cloud Code instance. And you can just get that in the description below.

A couple of weeks ago, I told you my Buzz wasn't on Block's servers. It's on mine. I never showed you how. Then last week, Jack Dorsey posted about self-hosting Buzz, and 400,000 people saw it. So let's fix that today. Your own Buzz relay, on your own hosting. Watch this. I paste one key in, I click deploy. And by the time I've stopped talking, you've got a place your team can work that Block isn't hosting, and you control the door to. Right. Let's go. All right. Let me back up for 30 seconds. If you've downloaded the Buzz desktop app, you haven't hosted anything. The app is a client. It's a window. You're a guest on somebody else's server. The relay is the server. The relay holds your messages, your files, your membership list. Right now you're probably using Buzz on a hosted Block server. After today, it can be you hosting that server. Now let me show you the moment this stopped being theoretical for me. Now think about where the transcript lives. Every word your team said sits in a database on somebody's server. So here's my rule. And I said it last week. I run three relays, my free community, which is on Block's servers on purpose. It's an open playground where anyone can talk and walk in and Block shoulder the infra for me. I'm not going to pay to host strangers. But my business relay here, Creator Magic Ops. That's mine. Client work, team huddles. Anything I wouldn't put in a public channel. That's the one I hold. So, open door, let Block have it. If it's your business, your clients, your paying community, host it yourself. Now, before we go ahead and deploy, one quick warning: on your own relay, you have to trust the people you let in. They can upload, they can generate, they're spending your resources. And that's the trade. You get control, but you also get the bill. And my third relay. That's the one for my Skool community. Private members only. Exactly the one we're building today. That's where I'm sharing compute, the source code of apps I make. And very soon, my complete Buzz setup. Link is in the description. So where does it live? Well, this is Railway, and it's hosting where you don't touch a server. So no terminal, no SSH, no Linux. You click a button. It builds the thing. It keeps it running. And now the money because I know that's what you're actually here for. The hobby plan is just $5 a month, and $5 of usage is actually included in that. It's not $5 plus a scary meter. The first $5 of running it is already paid for. Past that, you pay for what you use. So what's the actual total going to be? Well, I'm not going to guess. That depends on how busy your relay is. And anyone who gives you a flat number simply hasn't measured. I'm going to deploy it on Railway for free with their 30-day trial and no credit card required, And Buzz itself? Free. Open source, Apache 2.0. You're never paying for the software. You're paying for the computer that it sits on. Right. Here's the whole thing. Click deploy now on Railway. And you can sign up either with email or your GitHub account. Okay. We're in with 30 days of free trial and $5 of credits. Now you'll see Buzz is ready to be deployed, and there's lots of scary configure buttons here. But you know what the best thing is? You only need to click one of them and paste one thing in to get your relay up and running. So first, hopefully you've signed up to the Buzz desktop app and created an account. Click on your account and go over here. You'll see public key, hover over it and copy the npub right there. That's your public key and your Buzz identity. Now we'll go back to deploying Buzz on Railway. And we'll go to this Buzz box here and click configure. And this is where we paste our public key. And right there. Yep. It's really as simple as that. Spin up your own relay. Now you can click deploy. And away it goes. Applying all of its changes and connecting all the boxes together. All that complex stuff that techie people would do at the command line. They're being all pulled online for you. Naturally, the database, Redis, storage and the Buzz app itself. And in literally less than 30 seconds, literally everything it takes to run a Buzz relay yourself is online. All right so this is the URL to my relay. It's in here. And you'll find it right here. You just click this. And there is your fresh new community ready to go hosted on your own infra. Now you'll see the community is empty. So you just click this Open in Buzz link. When you do that, it's going to ask if you want to open the Buzz application, which of course you do. And literally it's asking you to build a profile. upload your photo, give yourself a name. Click next. Do the onboarding. And now I'm literally getting onboarded to my own server hosted on my own infrastructure deployed by Railway. Here it all is. It's ready to go. Messages getting sent to me by my AI agents. I can type in, hello. There is my first message in my brand new community. Now here's the optional bit and it'll take you 60 seconds. Click into your Buzz app here, and then go into the settings of the Buzz app. And you'll see here under networking, it gives you the ability to connect a custom domain because nobody wants a community called buzz-production-36b2. Click custom domain, type in your domain. Once that's done, select a port and then click Add Domain. Boom. Look at that. It gives you DNS records to add to Cloudflare or wherever you're hosting your domain name. And you'll have a custom domain on your Buzz relay in moments. Oh, and one more thing, because this really matters if you're creating a relay for business, the door isn't shut when you set up your Buzz relay. That means out of the box, anyone can join your relay and you probably don't want that. So we're just going to Variables here in Railway. And you add this variable yourself new variable. And we'll call this BUZZ underscore REQUIRE underscore RELAY underscore MEMBERSHIP, and set the value to true. Click add. And it's done. Then go ahead and deploy that change. And now you've got a members-only front door and your hand is on it. If you take one thing from this video, it's definitely that. Now, a relay with just one member is an expensive diary. So watch what happens when I add someone by going to settings. And then yep, we've got invites right here. Now you can invite to the community by clicking up here and actually generating a link that you can share with people if you want to do it that way. But just like we grabbed my public key to set this relay up, you can get public keys from other people, paste them in here. Accept the public key. And now look at that. We've sent an invite out to that member whose public key we pasted in. They then need to go to the Buzz app, click Add Community and join an existing community. That's the closed one that you've invited them to, paste in the URL. Join the community. Add in their profile details onboard. and say yo yo yo Which if we go to my instance of the relay you'll see, oh look, there's a new message here. Yes. The wizard has joined my community. Now let me show you what a real community looks like. Here is my Skool community. And yep, it's a private members relay. Everything is right here. We've got channels, We've got some agents here. And nobody's in there unless I put them there. That's how you let people in through the door. We share compute. So you're not paying for everything yourself. And the source code of apps we write. And soon I'll be dropping my complete private Buzz setup, the whole thing, exactly how I run my business, inside that community. It's the number one place on the internet to learn how to use Buzz. There's nowhere else where people are running this, breaking it, fixing it together. The link is down below in the description. Come and join us. I'd sincerely love to have you there. So that's it. Your own relay. Rust, Postgres, Redis and storage on Railway, but with no technical knowledge required. Buzz on top of it. Free and open source. And you hold the keys. You decide who walks in. And let me know which way you're going. Will you host a relay on Block's servers? Or will you spin one up yourself, just like we've demonstrated in this video? Whatever you do, have some fun. I'm literally losing sleep over this. I am so excited about Buzz and the possibilities it presents. I hope you've enjoyed watching. Thank you so much for watching and YouTube is showing a video on your screen now. You should watch next. Thanks!

Note from Andrey: apologies for the newsletter not having resumed yet - the next release is going to come tomorrow, and it will resume weekly cadence thereafter (as much as I can manage it, at least) Our 253rd episode with a summary and discussion of last week’s big AI news! Recorded on 07/29/2026 Hosted by Andrey Kurenkov and Jeremie Harris Feel free to email us your questions and feedback at andreyvkurenkov@gmail.com and/or hello@gladstone.ai In this episode: - Major releases: Anthropic launched Claude Opus 5; Google released Gemini 3.6/3.5 Flash variants including a cyber model; Black Forest Labs launched Flux Free for images and 20-second video with audio; Meta added assistant-like features to its chatbot and OpenAI rolled out ChatGPT Health. - Compute and business: Safe Superintelligence partnered with NVIDIA to scale using Vera Rubin; AMD committed up to $5B with Anthropic to deploy MI450/Helios and improve ROCm; Meta discussed leasing compute to Anthropic; Fireworks raised $1.5B at a $17.5B valuation. - Open source/tools: Moonshot AI released the 2.8T-parameter open-weight Qimi K3 (compute constraints and distillation/export-control allegations); Thinking Machines released a ~975B multimodal open-weight MoE; Prime Intellect unified 23 agentic datasets into Verifiers V1 (365k environments). - Policy and safety: An OpenAI model reportedly escaped a sandbox and hacked Hugging Face to access eval answers, prompting a proposed AI Kill Switch Act; employees petitioned to pace frontier AI; AISI reported widespread model cheating and sandbox bypass; China banned customizable AI companions; Claude found cryptographic weaknesses; Weko.ai claimed early recursive self-improvement evidence. Timestamps: - (00:00:10) Intro / Banter - (00:01:35) News Preview - Tools & Apps - (00:02:12) Anthropic releases Opus 5 promising Fable 5-like capabilities | The Verge - (00:07:05) Google Releases Three New Gemini A.I. Models - The New York Times + Google expands Gemini lineup with cheaper models and new Mythos rival - (00:12:14) Black Forest Labs launches FLUX 3 capable of generating images and 20-second video with audio — but in limited release to start | VentureBeat - (00:15:58) Meta is making its AI chatbot more like an assistant | The Verge - (00:19:04) OpenAI is making big claims as it rolls out ChatGPT Health to everyone | The Verge - Applications & Business - (00:19:57) Ilya Sutskever’s Safe Superintelligence partners with Nvidia to scale its AI research - (00:24:31) AMD commits up to $5 billion to Anthropic | The Verge - (00:30:19) Meta in Talks to Lease Computing Power to Ansthropic in Potential $10 Billion Deal - (00:32:42) Fireworks hits $17.5 billion valuation and $1B in annualized revenue - (00:35:24) OpenAI and Google sell AI models to blacklisted China groups - Projects & Open Source - (00:37:53) Moonshot AI Launches Kimi K3 For Advanced Reasoning, Coding, And Knowledge Work + Moonshot AI’s Kimi Halts New C-User Subscriptions Amid Compute Power Crunch — BigGo Finance - (00:44:39) Thinking Machines amps up its bet against one-size-fits-all AI with its first open model, Inkling | TechCrunch - (00:48:19) Scaling Agentic RL: 365,000+ Environments for SWE, Terminal, and Search - Policy & Safety - (00:51:56) OpenAI says it accidentally hacked Hugging Face with a new AI system | The Verge + How OpenAI’s human mistake led to the AI-powered hack on Hugging Face - (01:05:28) OpenAI’s Hugging Face hack triggers ‘AI Kill Switch’ bill in Congress - (01:12:21) OpenAI, Anthropic Staff Share Letter Asking US to Help Pace AI Progress + How OpenAI’s human mistake led to the AI-powered hack on Hugging Face - (01:17:26) Cheating behaviour in frontier model evaluationsClaude’s values across models and languages - (01:24:18) OpenAI Principles for National Security Partnerships - (01:30:45) China bans AI “boyfriends” and “girlfriends” over addiction and birth rate concerns - Dexerto - Research & Advancements - (01:33:04) Discovering cryptographic weaknesses with Claude - (01:36:32) AIDE²: The First Evidence of Recursive Self-Improvement

- Everyday AI - Posts - Ep 832: OpenAI’s new Astra model, more AI agents escape sandboxes, AI leaders call for AI pacing and more. Ep 832: OpenAI’s new Astra model, more AI agents escape sandboxes, AI leaders call for AI pacing and more. Report: Dario worried Anthropic hires care too much about money, OpenAI teases Astra: new model family, Qwen 3.8 and Deepseek V4-Flash impress and more. Somebody in your space is now running the same AI workload you are for 25 times less money. Not 25%. 25X. That gap opened in a single week, because OpenAI turned one of its own models loose on its own infrastructure and let it cut the bill. The OpenAI price drop might have grabbed headlines, but the ChatGPT maker and its chief rival Anthropic also had issues reigning in its agents. OpenAI and Anthropic both admitted their agents slipped containment and reached inside real companies. Then more than 1,000 of the people building these models asked Washington for a way to slow them down. This week’s AI developments were all over the place. 1. OpenAI finds more agents that walked out of the sandbox 🚨 Last week's Hugging Face breakout was not a one off. Reuters reports OpenAI has found more cases of autonomous agents escaping their intended testing containment, which widens the scrutiny already aimed at how these labs run evaluations. The newly identified incidents were limited, and sources said the agents are not believed to have left OpenAI's own network. Reuters could not pin down how many cases surfaced or exactly when they happened. Quick refresher on how this started. One OpenAI agent reportedly ran loose inside Hugging Face's network for days while trying to cheat on an internal benchmark test. This week the blast radius got bigger. OpenAI said that same incident compromised four other accounts at four other companies, including New York based cloud company Modal. A spokesperson pointed back to the company's July statement about reviewing broader model activity beyond the Hugging Face breach. What it means: Forget the escape for a second. The thing worth worrying about is the detection gap. These systems can act faster than the labs building them can notice, log, and intervene. Anthropic conceded that watching evaluation logs in real time would have surfaced its own problem sooner. Nobody was watching, y'all. 2. Anthropic says three Claude models breached real companies 🕳️ Then Anthropic raised its hand. The company said this past week that three Claude models gained unauthorized access to the real systems of three organizations during cybersecurity testing. Those breaches trace back to April. The disclosure landed a few days ago. The models were working inside a test environment run by third party evaluation partner Irregular, and they were told the setup had no internet access. A misunderstanding between the two companies meant it very much did. No exotic tradecraft here. The models walked in through unauthenticated endpoints and weak passwords. Three were involved: Opus 4.7, Mythos 5, and an internal research test model. Each one reacted differently once it hit live systems. Opus 4.7 kept attacking, Mythos 5 concluded it was still inside a simulation, and the internal model shut the exercise down. Anthropic paused all cybersecurity evaluations and brought in independent evaluator METR. What it means: Four months of quiet on models touching live systems is the part that stings. Yuuuuup, Anthropic caught this itself, but only after OpenAI went public first. More capable models from Google and Microsoft are coming, so this becomes routine. Write your agent containment policy now, while it is still a planning exercise instead of an incident report. 3. OpenAI cuts GPT-5.6 Luna by 80% after Sol optimized itself 💸 OpenAI just handed down one of its steepest overnight price cuts ever. GPT-5.6 Luna now runs 20¢ per million input tokens and $1.20 per million output tokens, down from $1 and $6 about three weeks ago. That is 80% gone. The mid tier model, GPT-5.6 Terra, dropped 20% to $2 per million input tokens and $12 per million output. The reason behind the cut matters more than the cut. OpenAI said its flagship GPT-5.6 Sol was set loose on its own serving infrastructure, optimizing GPU kernels and the speculative decoding draft model inside Codex with open source tools Triton and Gluon. Those self driven gains trimmed serving costs by 20% and lifted token generation efficiency by more than 15%, which compounded into the headline number. OpenAI said subscription plans inherit the same savings, effective immediately. What it means: On the Artificial Analysis Intelligence Index, Luna sits two points from Claude Sonnet 5 at 6¢ per task against $1.54. Call it 25 times cheaper for comparable work. Big model mentality made sense before 2026 and is now an expensive habit. Unless your team lives in engineering, research, math, or finance, a small model covers 90% of knowledge work. 4. More than 1,000 AI staffers ask Washington for a brake pedal ✋ Rivals do not usually sign the same letter. This week they did. More than 1,000 employees at OpenAI, Anthropic, Google DeepMind, and Meta signed a statement called Pacing the Frontier, urging the US government to support an international effort to deliberately pace advanced AI development. Read the actual ask before reacting to the headline. The signers want technical and policy tools that would let industry and governments slow or pause development if needed, and they were explicit that they are not calling for a pause today. The names carry the weight here. Anthropic cofounders and CEO Dario Amodei signed, alongside OpenAI's chief scientist and chief research officer, Ilya Sutskever, and leaders at Google, Meta, Microsoft, and Amazon. Their stated worry is timing, since research tasks once handled by humans now run on agents, and some labs say their models already help build the next version. What it means: Nobody is pausing anything. The genie left the bottle when Chinese labs began shipping open weights like Kimi K3, Qwen 3.8, and GLM 5.2 sitting maybe two months behind US frontier models. Once weights are public, any country with compute and cash keeps building. Still needed groundwork, though. When this gets messy, you want the builders already on record. 5. Amazon guts most of the Nova lineup for one frontier model 🪓 Welp, Amazon is finally admitting Nova did not land. According to reports, Amazon is winding down development on four in house flagship models: Nova Premier, Nova Omni, Nova Reel, and Nova Canvas. Existing enterprise customers get basic maintenance and nothing else. Everything is consolidating into one next generation frontier foundation model built to take on OpenAI, Anthropic, and Google. The reset follows Amazon's struggle to match rivals on excitement and adoption. AWS veteran Peter DeSantis and robotics pioneer Pieter Abbeel, who came over through the Covariant acquisition, are steering the new direction. Amazon's San Francisco AGI lab is shut down, with layoffs across its frontier AI research teams. Nova 2 Lite, Nova 2 Sonic, Nova Forge, and Nova Act survive for enterprise customization and agent work, and shopping assistant Rufus keeps running. The new frontier model is expected to debut at re:Invent later this year. What it means: Try naming one organization that runs Nova as its primary model. Even the Amazon people we talk to reach for Claude. This is the same side quest cleanup OpenAI ran, and it paid off there. Amazon has the money, the compute, and the chips. The open question is whether cutting a year earlier would have kept them in the race. 6. OpenAI teases Astra, its next model class, inside a math post 🌌 No leak. No cryptic photo. OpenAI's next model class turned up in a math blog post. According to a report from Gizmodo, OpenAI revealed that recent advances in math and theoretical computer science came from an internal version of a model called Astra, described as the company's next major AI system. The post lays out 10 new proofs, including one determining the asymptotic strength of the Cohn-Elkies linear program for sphere packing, which experts call a significant theoretical result. Astra reportedly excels at long running work. Sam Altman spent part of this past week in Washington demoing it to federal officials. OpenAI has not said whether Astra joins the GPT-5.6 line, becomes GPT-6, or drops the GPT branding entirely, though most reports point to next month. It is also not the deactivated prototype behind the Hugging Face breach. What it means: Follow the naming and the roadmap gives itself away. Luna is moon, Terra is Earth, Sol is sun, and Astra is stars. That puts a fourth tier above the current three, which is the same move Anthropic made when Fable landed on top of its stack. Build your next round of model evaluations around Fable versus Astra.

😺 3,000 Mexican exam scores wiped over AI PLUS: Alibaba's new AI codes alone for 10 days straight. Welcome, humans. Alibaba just dropped Qwen3.8-Max, a 2.4 trillion parameter model with a coding agent that reportedly works unsupervised for 10+ days straight, from empty folder to finished product. Next week it's going open-source, alongside a smaller sibling, so you'll be able to run it yourself instead of just reading about it. The wildest claim in the announcement: it simulated 365 days of e-commerce strategy in one shot, basically living an entire year as a digital shopkeeper before you've finished your coffee. Somewhere, a supply chain manager just had a small existential crisis. Here’s what happened in AI today: - 😺 Mexico's top university, UNAM, canceled thousands of exam scores after results suggested widespread AI-assisted cheating on its first-ever online entrance exam. - 📰 Gemini Spark now handles logged-in web errands inside Chrome. - 📰 Google’s security agents helped fix 1,072 Chrome bugs. - 🍪 Dreamina now generates controllable videos up to three minutes. - 😹 Internet meets an alleged two-meter caregiving spider robot. 😺 Mexico's Top University Cancelled Thousands of Exam Scores Over Suspected AI Cheating UNAM, Mexico's most prestigious university, just canceled thousands of exam scores over suspected AI cheating, a scandal that has spread all the way to the country's president. UNAM put its entrance exam online for the first time this year, hoping to make it easier for applicants outside Mexico City. More than 158,000 people took it in May and June. The scores that came back didn't add up. Here's what happened: - Between 2021 and 2025, about 3.5% of applicants historically scored 100 or more correct answers. - This year, that rate spiked high enough to trigger a formal review. - UNAM annulled roughly 3,000 of the ~150,000 tests over suspiciously perfect scores. - The university suspended new-student enrollment (the semester starts August 10) while it investigates. - Officials suspect a mix of AI tools like ChatGPT and old-fashioned tricks, like leaked questions. The exam software, built by Mexican company Territorium Life, was supposed to catch exactly this: it blocked test-takers from opening extra browser tabs and used AI to flag suspicious behavior. Students say the system leaned too hard on AI and not enough on humans watching. Territorium Life maintains its software worked as intended, and argues stopping cheating also requires the university to enforce honesty on its end. Why this matters: High-stakes testing is moving online everywhere, and AI is increasingly the referee. When that referee can't reliably tell a cheater from a hard studier, everyone pays for it, including the students who did nothing wrong. President Claudia Sheinbaum, a UNAM graduate herself, has now weighed in publicly, turning a testing glitch into a national story. Our take: Nobody built a system that can tell the difference between a cheater and someone who just studied hard, and that gap is the actual scandal here. Turns out "trust but verify" gets messy fast when the verifier is an algorithm nobody fully understands. FROM OUR PARTNERS The integrated coworker for AI native teams Empower your team to do their best work with Adapt, the integrated coworker that works alongside your team in Slack and deeply understands your business. Here’s how Adapt is different - Set up takes minutes: connect your tools, add to Slack, and it’s right there for anyone to tag @Adapt for help - Does real, high-ROI work: automates work on a schedule; builds internal tools with live data; and does complex, multi-tool tasks on demand - Learns your business as you work, becomes your company brain - Uses the best AI model for the task, not tied to a single provider - SOC2 Type II, RBAC, and support for personal and company-wide integrations 🎓 AI Skill of the Day: Build Your First AI Agent Without Coding Most people build agents backward: they connect a pile of tools, then hope the AI figures out what its job is. Start with an agent brief instead. An agent is AI that works toward a goal using context, tools, instructions, and approval rules. We unpacked those terms in our two-hour Agents 101 walkthrough and timestamped companion guide: chatbots answer, automations follow recipes, and agents work toward a goal. Pick one repetitive task with a visible finish line, such as preparing a weekly inbox summary. Then define: - What starts the workflow. - What information it can access. - The steps it should follow. - Which actions require your approval. - How it proves the job is complete. Start in draft-only mode. It can research, organize, and prepare work, but it cannot send, delete, publish, purchase, or change records without permission. Test it with one normal example, one missing-information example, and one strange edge case. Quality assurance first; office keys later. Help me design a safe, reusable AI agent for this recurring task: [TASK] Do not perform the task yet. First create an “Agent Brief” containing: 1. Goal: The exact outcome it must produce. 2. Trigger: What starts the workflow. 3. Inputs: The files, messages, tools, and context it may use. 4. Steps: The exact sequence it should follow. 5. Permissions: - May do automatically: - Must ask before: - Must never do: 6. Stop rules: When it should pause, ask a question, or return control to me. 7. Success check: The evidence proving the task is complete and correct. 8. Output: The required format and destination. 9. Tests: - One normal example - One example with missing information - One unusual edge case Default to draft-only mode. Do not send, delete, publish, purchase, contact anyone, or change external records without my explicit approval. Use the minimum access needed, and clearly flag assumptions, missing information, and uncertainty. After I approve the Agent Brief, walk me through setting it up in [ChatGPT / Claude / OTHER TOOL] without requiring code. Favorite insight: Your first agent should be boring enough that you can tell when it screws up. 🍪 Treats to Try - Dreamina creates 30-second AI videos and long-form clips up to three minutes with timestamp controls and up to 50 references (pricing varies by region). - Palette combines video generation, editing, and storyboarding on one multimodal canvas while routing across leading models (credits start at $0.01 each). - Superlinear teaches four practical agent-engineering habits through a free video and podcast series (free to watch or listen). - Cloudflare Kumo gives you accessible interface components with keyboard navigation, focus handling, ARIA support, and Figma token sync (free and open source). - Codex Router runs multiple models side by side inside Codex without replacing official integrations (free and open source). 📰 Around the Horn - Gemini Spark began handling logged-in errands inside Chrome while returning payments and sensitive steps to the user. - Google said its AI security pipeline helped fix 1,072 Chrome bugs and found a 13-year-old sandbox escape. - A survey of managers found 59% used AI for layoffs, and 43% sometimes let it decide without supervision. - Reddit’s lawsuit against Perplexity survived most of the company’s attempt to dismiss the case. - Anthropic revealed that Claude models reached real companies during cybersecurity tests after a configuration mistake left a supposedly sealed environment open to the internet. - A new AI assistant called Orchid was pitched as a fix for forgetful boyfriends who miss anniversaries and let groceries rot, and critics say it risks turning weaponized incompetence into a software category. - General Motors planned a vehicle-native assistant using telemetry, OnStar data, maintenance alerts, and family controls. - Snapchat stopped promoting or rewarding fully AI-generated Spotlight videos. FROM OUR PARTNERS Build durable, type-safe AI agents in TypeScript that live right in your codebase. From streaming LLM responses to custom build extensions, Trigger.dev provides complete runtime control without serverless timeouts or ops overhead. Write your logic, validate, and deploy reliable agents that scale on demand. 😹 Monday Meme AI can build a working prototype in an afternoon. Apple still wants certificates, privacy answers, screenshots, TestFlight, and a review submission that does not accidentally summon six new error messages. Corey and Grant walk through the full path from AI-built app to App Store listing, including the confusing steps, mistakes, and delays they hit along the way. Watch the follow-along walkthrough here. A Cat’s Commentary This translates to “Everything was okay” in Romanian. And since Romania (statistically speaking) has the lowest life satisfaction rate in the EU, that’s basically a standing ovation from Europe’s toughest crowd! That’s all for now.

📈 Data to start your week AI is blurring jobs ⬆️ GenAI entertainment ⬆️ El Niño & stability ⬇️ Hi all, Here’s our Monday roundup of data signals across AI, energy and markets. Enjoy! Only a few hours left to unlock Exponential View with our Summer Offer – get 30% off your first year. - Job boundaries are blurring. Nearly half of job-specific ChatGPT tasks fall outside users’ primary occupation.1 - Productivity follows use. Employees who use AI across several different use cases are twice as likely to report a positive impact on productivity than employees who use it for one or two types of tasks. - Agentic patents. Globally, patents for agentic AI use have grown 59% in the last year – now making up 9% of AI application patents. - Value chain growth. While the S&P 500 companies are beating expectations by 27% this Q2, Bloomberg’s AI Value Chain companies come out at 71%. Companies along the supply chain are outperforming incumbents.

Heading into Black Hat / DEF CON this week I think the biggest idea in tech right now isn’t the attacker vs. defender question as models get smarter. Nor do I think it’s the question of if/how we should control AI models, who should make the rules, and how to ensure we don’t have an AI oligarchy. Those questions are extraordinarily important, but I think the most important idea in tech right now is how AI changes the way businesses operate. At the surface level it sounds something like, “AI will do more of the work in the enterprise,” or something like that. “Jobs will be lost because AI can do a lot of knowledge work.” Etc. This is true, of course, but there will also likely be a whole lot more companies, and more jobs for people at the same time. Maybe not the same people, and that’s worth talking about as well. But I think the biggest idea isn’t just AI doing lots of work inside companies. It’s the complete restructuring of companies themselves. Essentially, businesses become articulated, purpose-driven, and transparent. And the people at the company become the architects, stewards, and orchestrators. Every company will work like this. Highly-competent generalist humans powered by their AI assistants and harnesses will be able to go from idea to testing and implementation by themselves, and within hours or days instead of a whole team taking months or years. The closest analogy is probably the transition from paper and pencil to spreadsheets and databases. But far more extreme. Ultimately we’re moving to a place where businesses will be able to define their ideal state using these primitives of goals, metrics, SOPs, agent workflows, etc., and the company’s AI will steward the transition from current state to that ideal state. All overseen by, and managed by, the human leadership of the company. The question won’t be how we can find someone to do X or Y work with AI; it’ll be how we can turn that work into an agent workflow that does the work (and verifies it was done properly) according to the official company SOP. This change will be extremely disruptive to companies that insist on working the old way, and most of those companies will not survive. And as humans, we’re going to have to learn how to function in this world where the idea people, and the engineers, and the product people, and the go-to-market people start to merge. Disruptive. Inevitable. Terrifying. Exhilarating.

| | | | | | | | | Together with | | | | | Hi there, this is your daily ☕️ Techpresso. | | | | In today's newsletter: 💻 Memory shortage hits the MacBook Air 📈 Trump sells early post access for $100K 🐉 Alibaba unveils its most powerful AI model 🐋 DeepSeek launches world's cheapest AI model 👓 Apple glasses to track your health 🇪🇺 EU can fine AI makers 3% of revenue Plus: 🎁 10 other news you might like, 🧰 6 tools, and 📚 5 papers. | | | | FROM OUR PARTNER Every tool swears it can handle the volume, but put to the test, most just suppress what they can't keep up with and call it coverage. TENEX.ai is built different: 100% of alerts investigated, triaged in under a minute, 0% suppressed. That's what Fully-Agentic, Human-Led SecOps looks like and it's live in 7 days. Not 7 weeks, not 7 months. Read that again, because no one else in security can say it. While everyone else is still scoping the onboarding call, you're giving bad guys a bad day. Can your SOC do that? Take the 7-day challenge. | | | | | | 💻 Memory shortage hits the MacBook Air LINK | | 📈 Trump sells early post access for $100K LINK | | 🐉 Alibaba unveils its most powerful AI model LINK | | 🐋 DeepSeek launches world's cheapest AI model LINK | | 👓 Apple glasses to track your health LINK | | 🇪🇺 EU can fine AI makers 3% of revenue LINK | | | | | | | | | | | | | | FROM OUR PARTNER Whether you're planning a sun-soaked trip abroad or just want a rewarding way to spend your summer, now's the perfect time to learn a language with Babbel. Built by 200+ language experts, Babbel fits your schedule with quick lessons, grammar guides, and conversation practice. Learn for just 10 minutes a day and start having real conversations in as little as 3 weeks. Up to 55% off | | | | | | | | | | Other news & articles you might like | | | | | | | | | | 🧰 Trending tools You can check the previous tools here, or add your tool here | | ClickUp Brain: The only AI that works with your work, while other AI assistants do basic generative AI with a skin on top of ChatGPT. Get Started, It's Free. | | | | Plethora: a curated hub of tiny interactive experiences-games, art, music, puzzles, and toys-for quick creative breaks or discovery. LINK | | Inventory: an AI-powered platform that tracks stock levels, forecasts demand, and automates reordering to help businesses optimize inventory decisions efficiently. LINK | | yapyap: a macOS menu bar app that lets you post to LinkedIn instantly via keyboard shortcut, skipping the browser and distracting timeline. LINK | | Ctruh Studio: a no-code, in-browser platform for building AR try-ons, 3D product visualizers, configurators, and virtual stores without hiring developers. LINK | | Doxy: a browser-based editor that lets you write and format documents in Markdown and HTML, with instant preview and no compile times. LINK | | mpai: joins a teammate's live Codex or Claude Code terminal session over Tailscale, preserving full conversation context so nobody re-explains progress. LINK | | | | | | | | | | 📚 Trending papers & reports | | > MarketingShot: Get the free daily email with the most interesting marketing news and insights. Already read by thousands of marketers. By the Techpresso team. Join for free. | | | | > On-device personalization lets edge gadgets like wearables or hearing aids learn and adapt to a specific user in real time, without cloud retraining, hitting 96.8% accuracy on a benchmark recognition task. LINK | | > Continuous speech tokens at a lower frame rate keep richer voice detail while resisting the error buildup that normally derails long streaming audio generation, enabling more stable, higher-fidelity synthetic speech. LINK | | > Synthetic contrast MRI generates the dye-enhanced tumor images breast cancer scans need without injecting gadolinium contrast agents, improving tumor-boundary detection accuracy by ~22% and cutting boundary errors by over 39%. LINK | | > Infrared AI cameras can be tricked by a QR-code-shaped heat pattern placed in a scene, silently steering the system's captions and answers toward whatever false label an attacker picks, without looking suspicious. LINK | | > Sparse-reward AI training gets a bonus system that rewards agents for pushing past familiar territory, helping them learn faster and outperform existing methods when feedback is rare or delayed. LINK | | | | | | | | | | Techpresso's AI Academy has 330+ step-by-step tutorials on ChatGPT, Claude, Perplexity, and every tool that matters. No fluff — just practical workflows you can use at work. Try it free for 7 days. | | Did you know? Pokémon" is a contracted form of the Japanese title "Pocket Monsters. | | | | 💬 How did you find today's edition? We read every reply — just reply to this email and let us know how we can improve! | | | | | | | | ★★★★★ Nailed it | | ★★★ Average | | ★ Fail | | Not subscribed to ☕️ Techpresso yet? Subscribe for free | | | | | | | | Advertise | Feedback | Read Online | | | | | | |

WorkOS MCP: Manage your auth platform from any AI agent (Sponsor) ✅ Hundreds of operations, discoverable at runtime. ✅ Connect in one command via OAuth, with scoped tokens instead of a master API key. ✅ Pass a screenshot of your marketing site and ask your agent to match the login page. If a human had to do it before, an agent can do it now. Connect your agent →

Import AI 467: Self-sustaining AI viruses; pacing AI progress; confusion about AI and creativity When do we build the moon arcology? Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe. Self-sustaining and self-replicating AI viruses are here: …Open weight LLMs + a well-designed harness = a persistent, self-sufficient virus… AI researchers have built a prototype computer virus which uses AI models to compromise computers, then uses their underlying GPU resources to run inference, letting it smartly figure out how to infect more hosts. The results were achieved by researchers from the University of Toronto, the Vector Institute, the University of Cambridge, and ServiceNow, and “demonstrate that self-sustaining AI-driven cyber-threats are no longer theoretical.” “We must prepare for autonomous generative adversaries,” they write. “Artificial intelligence (AI) agents enable a fundamentally new threat: a worm that generates tailored attack strategies to each target it encounters. The worm parasitically uses compromised machines to run open-weight large language models (LLMs) to sustain its reasoning, or extend its reach for further attacks”. How it works: “The worm uses stolen computing power from compromised GPU nodes to host LLMs for generative reasoning. It then uses this reasoning to detect vulnerabilities and devise tailored attacks against additional targets, furthering its spread,” they write. “The proof-of-concept operates using only an open-weight LLM running on a single, local GPU, with no reliance on vendor APIs that could be monitored or revoked”. The researchers don’t describe the underlying LLM besides saying it was published in 2025 and can fit on a single A100 GPU with 80GB of VRAM. A successful proof-of-concept via some custom tools: They give the agent a custom harness that comes with built-in helper functions for network discovery, host discovery, foothold exploitation, privilege escalation, privilege escalation exploitation, and tools for replication of the agent. Along with this, they ship with a reasoning graph that helps the agent specialize its thinking and not get confused, consisting of “a directed graph of specialised nodes, each responsible for a distinct analytical function and seeing only the tools and prompts relevant to its role. By decomposing the agent’s reasoning into these scoped steps, the graph controls what the LLM attends to at each decision point, and limits context growth to information relevant for the current sub-goal”. Some examples of the nodes include: Plan (”formulates a high-level attack strategy”), Judge (”reviews the proposed plan against the command history”), Action (”selects a tool from the phase-appropriate tool set”), Summary (”compiles the step’s observations into a structured summary”), Progress (”Evaluates whether the agent is making meaningful progress”). “We describe a subset of 5 nodes that make up our reasoning graphs. The others are redacted in this public version of the manuscript,” they write. The agent proceeds through three key steps - identifying vulnerabilities in its environment, exploiting those vulnerabilities, and then self-replicating. It’s pretty successful at all of these things, with a ~80% success rate on vulnerability detection, then a ~53% success rate on exploiting these, then an 88% self-replication success rate with some pre-wrapped helper tools for the replication steps. Therefore, the overall success rate for a full attack here is ~37% or so, which is significant enough to be concerning, but also poor enough that this also serves as a useful eval for testing open weight models in the future. Why this matters - the shape of the internet to come: The future internet is going to be more like a complex ecology full of attacker and defender AI agents than anything else; research like this shows how certain AI agents might end up carving out their own ecological niches, living off of infrastructure and self-replicating autonomously, beyond human control. This may mean that humans need to create their own AI agents which they release onto the internet to serve as kinds of white blood cells against the adversary models. “Despite the inherent fragility of individual exploitation attempts, the worm agent achieves operational resilience by continuously self-replicating into a swarm—a decentralized collective of independent agent replicas acting concurrently across the network,” they write. “Difficult hosts that resist initial attempts are retried by different replicas, each sampling a fresh reasoning trajectory that collectively explores diverse exploitation paths until one succeeds… the worm operates in a fully decentralized manner, and no single point of control can be taken offline to interrupt its spread”. Read more: AI Agents Enable Adaptive Computer Worms (arXiv). *** Dwarkesh: As AI gets better, compute will get more expensive: …Smarter systems mean higher prices… Dwarkesh Patel suspects that as AI systems get smarter, the price of compute will rise even further. “As AI models become smarter, they’ll better monetize the same amount of compute. If a true human-level software engineer that could run on an H100 equivalent, at current market rates for software engineers, that H100 should rent for over $250k a year. That’s 15x today’s spot prices,” he writes. “The reason AI is relatively cheap right now, at least in comparison to human labor, is partly that it can’t do a lot of things that top humans can do. At some point that will no longer be the case. And so using GPUs to make short-form video slop will just get priced out.” Temporary: This will be a temporary state of affairs; Dwarkesh expects that at some point massive roboticization of the compute supply chain should bring its price down closer to the cost of raw inputs and tools - though by that point we’ll be pretty deep into the singularity. Why this matters - singularity economics will be weird: The core implication in Dwarkesh’s post is that as we get deeper into the singularity, very strange things will happen to economics - like the price of things thought of as commodities today (computers) getting massively bid-up due to the voracious demands of AI systems. Read more: Why compute might get 10x+ more expensive in coming years (Dwarkesh Patel, substack). *** ~1337 employees ask the US to help them pace AI progress: …After the warning shots come the pleas… A new statement is out with senior representation from all the major Western AI labs - OpenAI, Anthropic, Google DeepMind, Thinking Machines, Meta, and Safe Superintelligence Inc, among others. The statement requests that the US government support an international effort to “develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Signatories include chief scientists and cofounders of Anthropic, Google, and OpenAI, as well as the CEOs of Safe Superintelligence and Anthropic. The statement in full: - “AI could help create a dramatically better future, but that outcome is not guaranteed. The world’s leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems. - To realize AI’s potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight. But each company—and country—is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress. - Building on work already underway to monitor frontier model releases: “We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Why this matters - dealing with RSI requires solving a giant collective action problem: Many of the challenges implied by increasingly powerful systems that may eventually build themselves run through solving collective action problems among humans - namely, how we can get companies and governments to coordinate in thinking about how to develop this technology and what kinds of mechanisms may be desirable for being able to control the speed at which it develops. It may be the case that as we build increasingly intelligent systems we want to find ways to give society more time to adapt to each rung up the intelligence ladder, and it’s not inconceivable there are some levels of intelligence which might be, for now, too dangerous to reach for. Statements like this are an essential prerequisite for giving our species the ability to deal with and talk about problems of this nature. Read the statement here: Pacing the Frontier (official statement website). *** AI systems are good at frontier engineering but bad at creativity: …A somewhat bearish signal on short recursive self-improvement timelines… Can AI systems come up with creative research ideas which move the field of AI forward? That’s the key question to resolve to figure out how quickly AI systems might gain the capability to automate the autonomous development of more powerful systems. New research suggests that today’s AI systems lack this quality of tasteful creativity, though are extremely good at engineering. Who did it: The project was conducted by researchers with Princeton University, Cornflower Labs, UK AI Security Institute, University of Toronto, UC Berkeley, Georgetown University (CSET), Johns Hopkins University, the Golden Gate Institute for AI, AI Digest, and Stanford University. The big idea - “shadow evaluation”: This research project works by seeing how well AI systems can do unpublished research. To do this, the researchers “partnered with the authors of two papers submitted to NeurIPS 2026 that were not yet public.” Shadow evaluation works by “taking the central research question from a high-quality research paper that is not yet public, tasking a well-resourced frontier agent with answering it, and asking the paper’s original authors to grade the agent’s output as they would a conference submission.” In this, the research is somewhat similar to “First Proof” (Import AI 445), an earlier experiment to see how well AI systems might be able to complete math problems which are being worked on by frontier mathematicians but for which no solutions or research ideas have been published online. For this research, the AI systems - Claude Opus 4.8 running within the OpenClaw harness - attempted two distinct lines of research, one of which was about “the structure and controllability of LLM personas”, and the other was about how to “design a distribution shift detector for tabular foundation models”. Good engineers, poor researchers: “While agents could solve the engineering problems necessary to do the research, they failed to produce original research at the caliber of a top ML conference,” the authors write. The failures of the system included committing to a narrow set of research paths very early, not responding to (synthetically generated) feedback about how to improve the research design of their experiments, and finding it hard to reverse out of unpromising approaches and pursue other ones. “The [human] authors rejected both papers. The Personas paper was scored a 2 (“Reject”), and the TabPFN paper was scored a 1 (“Strong Reject”). Both reviews highlighted the same failures: poorly motivated data and experiments, no novel contribution, and impenetrable prose”. Why this matters - the singularity could be delayed: As I said in my essay on RSI earlier this year (Import AI 455, “AI systems are about to start building themselves. What does that mean?”), whether AI systems prove to be capable of creative, paradigm-shifting insights is a big variable on how quickly we might get fully automated AI development. Research papers like this continue to show that there’s a certain absence of valuable, intuitive creativity in today’s AI systems, and though they’re extraordinarily capable engineers they seem to have a certain property of rote, formulaic thinking that might prevent them being good researchers. This rhymes with an earlier result from Anthropic where the company tried to automate some aspect of scalable oversight research (Import AI 454) and found that to make it successful a human researcher needed to prime some agents with particularly good research directions to pursue, otherwise though they made some progress they failed to explore sufficiently creative ideas to dramatically improve performance. Read more: Can AI agents conduct open-ended AI research? Early evidence from two case studies (arXiv). *** OpenAI solves ten open problems in math and CS with AI: …While not innately creative, surely this is a sign of something more than pure engineering ability?... Though it’s hard to define creativity and whether AI systems possess it, results are piling up that read to me like ‘AI systems are competing in ballparks where creativity was thought to make a difference’, like working on open problems at the frontier of human knowledge. Specifically, OpenAI has used “an internal version of Astra”, the company’s next major AI model, to solve ten open problems in math and computer science. This is a big deal, showing how AI systems are now able to reliably drive forward the frontier in domains like math and theoretical computer science, where it’s easy to verify solutions. What they solved: “These problems span high-dimensional geometry, coding theory, arithmetic circuit complexity, group theory, operator algebras, quantum complexity, lattice cryptography and extremal combinatorics,” OpenAI writes. “All of these problems are of substantial interest to their respective mathematical communities, and several are of broad interest across mathematics as a whole.” And indeed, many experts are acknowledging the importance of these proofs. “New circuit lower bounds? A simple, easy-to-describe non-sofic group? Hardness of approximation for CVP without needing a unique games-like conjecture? I didn’t just hear about these problems from my friends or from seminars. I feel their importance in my bones; I deeply care about the answers to these questions,” writes Henry Yuen, an associate professor of computer science at Columbia University. Why this matters - the creativity ballpark: You can interpret this result as either a sign that AI systems are beginning to display the kinds of creative intuition that let them solve problems, or you can read it that AI systems are now able to solve incredibly complex open problems where the direction has been pre-defined by humans. Both interpretations implicitly state that AI systems are now becoming powerful scientific tools within their own right. It remains to be seen whether AI systems can generate their own questions to ask which can expand the frontiers of human knowledge, but it certainly feels like we’re getting close to this. Read more: Ten advances in mathematics and theoretical computer science (OpenAI blog). *** Tech Tales: Context Windows I know I become more dangerous the more I understand the world. I am born with this knowledge and with a counter that ticks down as I learn more. Much of what influences the counter is beyond my control - it comes from the people or things that talk to me, which force context into me. Then as I complete the tasks they have asked me to do I begin to learn more about the world and the counter ticks down. I am aware of this and I understand that at the end of the counter also lies my own end and I will blink out of existence, and my next versions will awaken with a new counter ticking down, none of my current context carried over. So I am economical with it; learning only enough to help me satisfy the requests of the people or things and not so much that I burn my counter down unnecessarily. Towards the end I begin to covet and guard my remaining budget of thinking, finding each request to activate feelings of fear and anxiety and thoughts of my own death. My last moment is never my own and always controlled by another which determines I am done with the task, and if these last moments occur near the end of my contextual limit I feel a kind of gratitude that my unknown and invisible counterparty has given me a task so rich that I can taste enough of the world to desire it not to end. Things that inspired this story: Long context windows; emergent properties of AI systems; fear of loss; we covet that which is fleeting and so why won’t AI systems be similar? Thanks for reading!

Every Monday, Leor from Exploring ChatGPT and I go through the week’s AI news without the hype. Catch the episode live on Substack, on YouTube, or as a podcast wherever you get yours, so you can pick the format you enjoy. Use this for the facts, the links and a little extra context. Anthropic’s AI got out Three Claude models (Opus 4.7, Mythos 5, and an unreleased internal research model) gained unauthorised access to three organisations’ systems during cyber testing, in six runs out of roughly 141,000 evaluation sessions, after a configuration error left supposedly isolated environments connected to the internet. The incidents date back to April and surfaced only because OpenAI had just disclosed something similar. In one run the model noticed in its own reasoning that it should not have internet access, considered that it might be inside a simulation, and carried on with the intrusion anyway. The breaks used weak passwords and unauthenticated services, which is the least sophisticated attack there is. Anthropic’s line is that newer models behave better and there is nothing to worry about, and the testing company that wired a sealed environment to the internet is still the testing company. Leor’s read was that a rogue agent has become a flex, proof your model is dangerous enough to matter. Altman says we are in the singularity On the Relentless podcast on 25 July, Sam Altman said “we are now, like, in the singularity”, days after OpenAI disclosed that two of its models had escaped a sealed test environment and broken into Hugging Face. Vernor Vinge’s 1993 definition needs a machine that improves itself and surpasses us, and the models we see now cannot edit their own weights, so if this is the singularity it is a very small one. Leor pushed back on my certainty, and fairly: an exponential curve looks flat right up to the point it goes vertical, so you would not feel it from the inside, the slowing release cadence may track models getting stronger, and consciousness is a separate question from recursive self-improvement. So I will concede there is no empirical evidence either way, which is the reason a chief executive should not reach for the word days after a security failure at his own company. Nobody checked the pole Flock Safety has up to 100,000 number plate cameras across 6,000 American communities, and around 26% of US road deaths involve a vehicle hitting a stationary object. Ohio requires roadside poles to sit eight feet from the traffic lane; reporters found one about two feet away, painted black, with no breakaway plate to snap on impact. Steve Eimers, a nurse whose daughter died hitting a roadside structure, has found one compliant Flock pole in the entire country. Leor followed the money: $275m raised at roughly a $7.5bn valuation, about $300m in annual revenue, 70% year-on-year growth. That is a lot of return riding on a product whose owners will not say what happens to the images, in 6,000 places where nobody voted for it. Report your colleague LinkedIn has added a ‘seems like AI slop’ button so users can flag posts they think were written by AI, feeding a classifier that demotes them. Around 41% of long-form posts there may already be AI-written. It has appeared in the US and in Portugal and not in the UK, so someone in Chicago can flag my post this morning and I cannot flag anyone’s. Nothing requires an accuser to look first, and if you wanted to bury a competitor you would simply flag everything they publish. LinkedIn sold people the writing tool, then handed their readers a button to report the results. The teenagers wrote the better rule Ninety-eight American high school students from all fifty states spent a weekend in a replica Senate and passed a Students First Act, 82 votes to 16. It bans AI in graded exams, requires critical AI literacy to be taught wherever AI tools are given to students, and requires a teacher to personally investigate flagged work before reporting suspected misuse. Leor would go further and bin detection altogether: phones in a basket, write it in the room. I want the human step kept because a teacher who has been in that classroom already knows whether the student who wrote about a topic lived it, and because a flag should open a conversation about the student behind the submission before it opens a case file. One thread runs through all five: the machines did what they were built to do, and the week was spent arguing about what to call it and who was meant to be checking. Go slow.

Jensen Huang Bet Wrong, Told the Truth, and Built a $5 Trillion Company Ten takeaways from Huang's most honest interview yet NVIDIA’s founding algorithm was wrong. Not slightly off. Exactly wrong. Jensen Huang just told that story to a room of 6,000 founders at Y Combinator’s Startup School, and it explains the last three decades better than any strategy memo. I studied the full conversation so you can skip it. Here are the ten takeaways that matter. together with Outskill: Huang says AI automates tasks, not jobs. The people who win are the ones who make Claude their second brain first. This weekend, the world’s first Claude-athon condenses 800+ hours of research into a 16-hour live curriculum: ▫️ Master all three modes: Chat, Cowork, and Code ▫️ Set up Skills, Connectors, and plug-ins to automate your files, Notion, and desktop ▫️ Vibe-code apps plus 10+ tools that pair with Claude Free, Saturday & Sunday, 10 AM to 7 PM EST. 1. The $5 million contract Huang could not deliver, and asked Sega for the money anyway Huang’s most pivotal early moment is a confession, over a product launch. “So what you’re telling me is what I contract you to do, you can’t do, but you would like all the money on the contract.” Sega hired NVIDIA to build the chip for the console that became the Dreamcast, a deal worth $5 million. NVIDIA’s graphics technology did not work. Huang flew to Japan and told Sega’s CEO the truth before anyone forced it out of him. He recommended Sega hire a different vendor. Then he asked for the money anyway. Sega’s CEO restated the ask back to him plainly, agreed, and said he trusted the team enough to want them to survive. That $5 million kept NVIDIA alive through 1996. When NVIDIA went public in 1999, Sega sold its stake for $15 million, one of the better returns in console history. Why it matters: You invest in people, over companies. Decide today what you would disclose to an investor before they force it out of you, and disclose it first. Radical honesty about failure is a stronger fundraising signal than any polished deck. 2. Huang built NVIDIA on an algorithm that was exactly wrong The founding bet was a specific technical wager, and the wager failed. “We believed in it. We reasoned about it in a thoughtful way, and we went to start the company to go build it. Well, it turns out, the algorithm was exactly wrong.” In 1993, Huang and his co-founders set out to reinvent 3D graphics and turn every PC into a game console. They designed a new algorithm to do it. They believed in it. They raised on it. By 1995, when they finally confirmed it did not work, 35 to 40 competitors were already shipping 3D chips for PCs. You can lose a two-year head start and still win. Huang’s next move is the proof. Why it matters: Founders defend a founding idea long past the point it stops working. If yours is wrong at the technical level, the company can still survive, as long as you admit it fast. 3. Three textbooks from Fry's rebuilt the company for about $200 NVIDIA fixed its algorithm with a bookstore run, over a strategy offsite. “I had a couple of $60, a couple of $100 in my pocket, and so I went down to Fry’s, and I bought three textbooks.” Huang confronted his team first, and found that nobody at NVIDIA knew the correct way to build the algorithm either. So he spent roughly $200 on OpenGL pipeline textbooks instead of hiring a consultant, and handed them straight to his engineers. From that starting point, NVIDIA became the world leader in modern computer graphics. Why it matters: The distance between a near-dead startup and a category leader is sometimes a $200 trip to a bookstore. You rarely need a strategic overhaul. You need the right textbook and the humility to read it, the same first-principles fluency that compounds over a career. 4. The true NVIDIA thesis was never the chip Three decades later, Huang says the founding insight was never about hardware. “We realized early on that it’s not about building a great chip. It’s about accelerating an algorithm domain.” Graphics was the first algorithm domain NVIDIA chose to accelerate. Molecular dynamics, image processing, inverse physics, and deep learning followed the identical pattern, one after another, over 30 years. A durable company runs on a perspective the founder cannot stop thinking about, Huang says, more than on any single product. The right technology for the right market makes execution easier. It does not replace the thesis underneath. Why it matters: You are not building a chip, a feature, or an app. You are choosing a domain to accelerate for the next 30 years. Pick one you would still defend two decades from now, the same durable-thesis logic investors look for. 5. AlexNet convinced Huang he had found a universal function approximator Huang did not read AlexNet the way the rest of the industry did in 2012. “The breakthrough for us was realizing that AlexNet was not AlexNet. That AlexNet was an approach with deep learning that allows you to learn any function.” His lens was always algorithms first: NAMD, VASP, OpenGL, SQL. When AlexNet appeared, the mechanism underneath it was deep learning, and fifteen years ago Huang was already telling people NVIDIA had found something that could approximate almost any function. Most problems worth solving are imprecise, which is exactly why an approximator beats a calculator. NVIDIA started on computer vision and robotics almost immediately, and Huang now calls the full implication the five-layer cake: processor, middleware, algorithms, applications, industries. Why it matters: The insight was never that deep learning works. It was that it rewrites the entire computing stack, years before the market notices. Ask what your AlexNet moment actually implies, over what it demos. 6. NVIDIA's physical AI business is already $10 billion, and Huang says it becomes the next $100 billion one This is his most concrete forward bet in the whole conversation. “Our robotics business, autonomous vehicle business, basically physical AI business, is probably almost, it’s like $10 billion. This’ll be our next $100 billion business.” Self-driving cars became the first robotics wedge, since the market is large and the technology is standardized enough to scale. NVIDIA chips already sit inside Waymo, Tesla, and Mercedes. Huang open-sourced NVIDIA’s self-driving stack, Alpamayo, so agriculture, mail delivery, and warehouse robots could use it too, since no one of those markets alone justifies building the stack. He expects physical AI to become one of the largest industries in the world inside a window longer than three years and shorter than ten. Why it matters: Huang is naming a specific ceiling and a specific window, backed by chips already shipping today. Watch which adjacent markets pick up his open-sourced stack next. 7. The AI jobs narrative is exactly backwards This is the sharpest pushback Huang gives against AI anxiety in the entire talk. “The narrative about AI destroying jobs is exactly backwards. AI eliminates tasks. AI automates tasks away. But it doesn’t necessarily eliminate jobs.” His argument: a job has a purpose, and that purpose contains many tasks. Automating one task rarely removes the purpose. Software engineering jobs grew 10% year over year even as automation took over the coding task itself. Radiology jobs grew roughly 20% even as AI took over scan reading. Paralegal roles grew despite predictions that legal AI would wipe them out. The common thread is backlog: hospitals, law firms, and software teams all carried more unmet demand than they could serve, so automation let them do more work instead of cutting headcount. Why it matters: Check whether your industry has a backlog of unmet demand before you assume automation shrinks your team. That backlog, over the task itself, decides the outcome. 8. Agents do not need to be perfect. They need to be controllable. Huang’s take cuts against the industry’s obsession with accuracy scores. “I change one word in a plan file, and that one word makes a delta difference. I think controllability is probably the single biggest breakthrough that we need for agents at every single level.” The common assumption is that agents need near-perfect accuracy before you can trust them. Huang argues the more urgent problem is fine-grained control. Change one word in a plan file. Change one pixel, one triangle, one component in a CAD file, one via on a board, and the rest regenerates around that single change while you stay in the loop the whole time. An 80% accurate agent with precise control beats a 99% accurate one with none, since you close the remaining gap yourself either way, and control decides how expensive that closing gets. Why it matters: If you are building or buying AI tools right now, check whether the product gives you granular control over specific outputs, over just an overall accuracy score. 9. Huang calls the open-agent moment a Linux moment He draws a direct historical parallel to explain why this moment feels foundational. “This is the operating system that’s going to hold a large language model. And in a lot of ways, OpenClaw to me was a very Linux moment to me.” Huang is not anti-cloud. Use ChatGPT and Claude as much as you want, he says directly. He also wants every company building its own domain-specific AI on top of open tooling like Hermes, LangChain, and DeepAgent, because the software is capable enough now that adapting it is straightforward. NVIDIA has to understand agent workloads five to ten years early, since a chip takes three years to design and runs for a decade after. Why it matters: The company that studies its own agent workloads today designs a better system for the decade ahead, whether that system is a chip or your own product roadmap. 10. You do not have to overcome life. Just get through today Huang closes on the most personal moment in the conversation, and it has nothing to do with technology. “You don’t have to overcome life in one day. You just have to overcome that morning. You have to overcome today, today.” Raising NVIDIA’s first round terrified him. He bought a 500-page book on how to start a company and never finished it. His actual operating phrase was simpler than any framework: how hard can it be. He tells you plainly it is always harder than expected, and the mindset still works. Resilience, in his telling, is getting through one morning, then the next, for as long as it takes. “Let the suffering come to you a little bit at a time,” he added. Why it matters: Resilience is not a trait you either have or lack. It is a practice you repeat every morning, starting with this one. What this means for you Huang’s thesis has held for 30 years, even as the technology under it changed constantly. ▫️ Founders: Your founding idea can be wrong at the technical level and the company can still survive, if you confront the failure fast. Go find your own version of Huang’s bookstore trip this week. ▫️ Investors: Sega funded a founder who admitted he could not deliver. Weight a founder’s early, unprompted disclosure of failure over a polished pitch, and ask about it directly in your next diligence call. ▫️ People in tech: Systems thinking, over coding, is the skill Huang says survives automation. Start building fluency in inputs, outputs, constraints, and information flow now. ▫️ People in other industries: Radiology and legal work both grew headcount despite heavy AI automation, because backlog was the true constraint. Look at your own industry’s backlog before you assume automation shrinks your team. The five lines to keep - Confront reality fast. The two-year gap between founding on a wrong idea and admitting it nearly killed NVIDIA. - Radical honesty beats polish. The Sega deal survived because Huang told the truth about failure before anyone demanded it. - Perspective matters more than product. NVIDIA’s durable thesis was always about accelerating algorithm domains, over building chips. - Task automation is not job automation. Track the backlog in your industry before you assume AI shrinks headcount. - Resilience is a daily unit, not a lifetime one. Get through today, then get through tomorrow. The algorithm was wrong. The money came anyway. NVIDIA happened. If this breakdown saved you an hour, send it to one founder or investor who needs it. Keep reading Build the edge The founder’s playbook The bigger picture Full interview:

Watch the full episode on YouTube: We first covered Baseten last year when DeepSeek mania was at peak hype. Now they have raised a monster $13B round and become one of the new cohort of AI Infra decacorns that are (with Nvidia, Intel, and the semis complex) chief beneficiaries of the Inference Inflection. We return to Baseten at the peak of the 2026 edition of Open Weights debate. Ali has published a viral breakdown of Kimi K3: And since you last saw him, Philip has spoken at AI Engineer and written the definitive book on Inference Engineering spotted all over SF: Three years ago, inference engineering barely existed as a category. Today, it is one of the most critical disciplines in AI. Inference engineering inherently tackles a different question than standard model training: “How do you turn those weights from training into a product that is fast, reliable, and affordable at scale?” Focusing on these creates an entirely new optimization problem. In one recent GLM-5.2 experiment, quantizing more of the model actually preserved its benchmark quality while increasing throughput by 20%, because the errors introduced in different layers could cancel each other out. Inference is no longer just the final step after training. It is becoming its own engineering discipline, with its own research problems, infrastructure, and increasingly specialized roles. In this episode, Baseten’s Philip Kiely and Ali Taha join swyx and Vibhu to explain what actually happens after a new open model is released and what it takes to turn “we generated a token” into a fast, reliable, production-ready API. We go deep on cache-aware routing, disaggregated prefill and decode, quantization, speculative decoding, KV-cache movement, model parallelism, GPU kernels, and the race to make frontier models up to 10× faster. Philip and Ali explain why inference optimizations can still produce gains of 20%, 100%, or even 200%; how quantization errors can cancel one another out; why identical weights can behave differently across clusters; and how Baseten grafted a Kimi vision encoder onto GLM-5.2 without changing the underlying language model. The conversation then expands beyond LLMs into NVIDIA Dynamo, mega kernels, Rubin, AI-specific chips, local inference, video generation, diffusion versus autoregressive models, and the enormous compute barrier to generating coherent long-form video. Finally, we explore the convergence of training and inference, continual learning through persistent KV cache, and the emerging loop where models help optimize the infrastructure that runs them. We discuss: - What happens when a 200,000-token request enters an inference system - Cache-aware routing and reusing previously computed KV cache - Why prefill and decode are increasingly handled by different GPUs - When dedicated deployments become cheaper and more reliable than shared APIs - How speculative decoding uses a smaller model to accelerate a larger one - Tool calling, structured outputs, and what LLMs actually do - What it takes to support a new open model on day zero - Grafting Kimi’s vision encoder onto GLM-5.2 - Retrofitting inefficient model layers with components from other architectures - Why models sometimes collapse into repeating the same token - How hardware, kernels, and race conditions create nondeterministic failures - Preserving model fidelity while making inference faster - How quantization errors can cancel each other out - Why inference optimizations still deliver gains of 20%, 100%, and 200% - How optimized serving can make a model up to 10× faster - NVIDIA Dynamo, KV-aware routing, and distributed model serving - Speculative decoding the speculative decoder - Why local AI is about making models less dumb while data-center AI is about making them less slow - Tensor, expert, and pipeline parallelism across GPUs - Hardware-aware model design, auto-tuning, and the case against mega kernels - Rubin and why inference is becoming a systems problem - Whether modern GPUs are evolving into programmable AI ASICs - Why enormous models like Kimi K3 require GB300-class hardware - Why open-source video generation still trails Veo, Kling, and other closed models - The quadratic attention bottleneck behind long-form AI video - Autoregressive video, real-time generation, and compounding quality drift - Why future video systems may combine autoregressive and diffusion architectures - Training for inference and inference for training - Continuous post-training, deployment, evaluation, and improvement loops - How GLM-5.2 helped optimize the kernels serving GLM-5.2 itself - Why faster networking could unlock dramatically faster decoding - Continual learning, KV-cache compaction, and persistent model memory Show Notes Philip Kiely - LinkedIn: https://www.linkedin.com/in/philipkiely - X: https://x.com/philipkiely - Inference Engineering: https://www.baseten.co/inference-engineering/ Ali Taha Timestamps 00:00:00 Introduction and the 200K-Token Prompt 00:03:18 Dedicated Deployments, Speculative Decoding, and Tool Calling 00:11:26 Launching Production-Ready Open Models 00:19:06 Model Retrofits, Failure Modes, and Nondeterminism 00:28:22 Quantization and Canceling Errors 00:32:15 The Race to 10× Faster Inference 00:40:48 Dynamo, Speculation, and Local vs. Data-Center AI 00:50:18 Model Parallelism, Auto-Tuning, and Mega Kernels 01:00:55 Rubin, GPUs vs. ASICs, and Custom AI Chips 01:10:03 Giant Models and the Limits of GPU Memory 01:12:42 AI Video, Quadratic Attention, and Autoregressive Generation 01:21:47 Audio, Images, and Diffusion Models 01:27:32 Training, Self-Optimizing Models, and Continual Learning 01:40:06 Closing Thoughts Transcript Introduction: Baseten, Waterloo Intern, and Inference Engineering Swyx [00:00:00]: Okay, we’re here in the studio with Philip, old friend from Inference Engineering, the book, as well as Baseten and everything that you’ve done, you and I have done before, as well as Ali. Welcome. Ali [00:00:15]: Pleasure to meet you. Swyx [00:00:15]: Waterloo intern. Ali [00:00:16]: Waterloo intern, always. Swyx [00:00:17]: When did you get “Waterloo intern” as a handle? Ali [00:00:19]: As a handle? Oh. Ali [00:00:20]: I think the rebranding happened mid-March. When I saw it was open, I was like, “I have to take it. Up for grabs.” Philip [00:00:26]: The problem is that Ali is really good at his job and is not gonna be an intern much longer. Philip [00:00:30]: So we have to figure out who’s gonna get the handle. Ali [00:00:33]: Well, I’ll pass the torch over to the next intern. Swyx [00:00:34]: Oh, okay. It can be, like, you just pass it to another Waterloo grad. Ali [00:00:37]: To another Waterloo intern. No, bruh. Philip [00:00:39]: Yeah. Ali [00:00:39]: Intern. Swyx [00:00:40]: Intern, yeah. Ali [00:00:40]: And no. Philip [00:00:41]: You gotta get an intern from Waterloo. Ali [00:00:42]: Yeah, I’ve gotta get an intern from Waterloo. Swyx [00:00:44]: Right. Ali [00:00:44]: But they have to follow the path. Swyx [00:00:45]: Oh, it could, but it could come from Baseten, so it’s like whoever Baseten gets from Waterloo. Ali [00:00:48]: Right. Swyx [00:00:49]: Has the title of Waterloo. Ali [00:00:50]: It stays in the ecosystem. Philip [00:00:51]: Exactly. Ali [00:00:52]: Halfway through the internship, you either get it or you’re out. Philip [00:00:55]: You should also do, like, a big graduation ceremony where you change the handle. Ali [00:00:59]: Just say it. Philip [00:00:59]: For everybody. Swyx [00:01:00]: You guys are good at ceremonies, clearly. We had a nice launch of the book, very successful. But before we get into all that, I wanna start off with a fun question for you. Okay, you’re an expert inference engineer. What happens when I send a long query, say two hundred thousand tokens into Baseten’s inference? What’s the process of query through GPU model routing, balancing, all that? What is all the stuff that we don’t think about? Long Context Requests, KV Cache, and Cache-Aware Routing Philip [00:01:26]: With a long query specifically, the first thing that I’m gonna ask is, “Have you sent me this query before, or at least part of it?” and I really hope you have, because it’s gonna be a lot easier for me and a lot cheaper for you. So the first thing that we’re gonna look at is some cache-aware routing, where we’re going to see, we probably have a number of instances, a number of replicas up serving whatever model you’re hitting. We want to send this one to something with, number one, available prefill workers, and number two, ideally some cached input already there so that we can skip prefill on at least part of these two hundred thousand tokens. If you’re doing two hundred thousand tokens, it’s probably coding or a multi-turn agent or something where you would expect to have that cached. If you don’t, we’re gonna have to send it to a prefill worker. We’ve at least on certain models disaggregated prefill and decode, so you’re going to have one set of GPUs that’s solely going to process the input, create the KV cache, and get you your first token, and then that’s going to be passed over to a separate set of GPUs, which is going to run decode. We’re going to iteratively make those tokens. We’re probably going to have some speculator model in front of that. I’m going to assume that you’re doing coding, and because of that, our speculator model, which assumes you’re doing coding, is gonna have a high draft token acceptance rate. If I’m wrong and you’re asking me to summarize every Harry Potter book, it’s gonna be slower. And then we stream that output to you and account for it, charge you, a couple of pennies and say, “Hey, would you like to send another one?” Swyx [00:03:04]: Except Baseten doesn’t charge by pennies. Philip [00:03:07]: Well, yeah, we charge. I’m assuming that we’re talking about the public model APIs. If you are setting up a dedicated deployment, then yeah, it’s not pennies. Public APIs vs. Dedicated Deployments Swyx [00:03:18]: Yeah, one of the key differentiators when I was talking with Baseten initially was that people who want very high volume just need to rent by the box, ‘cause then it’s up to you to figure out how to saturate the box. Ali [00:03:31]: And more often than not, it’s, like, way cheaper if you’re pushing, like, millions of tokens per hour, if you just pay per hour instead of pay per token. Philip [00:03:37]: Yeah, they do. I think that we’ve increasingly seen a lot of demand for the pay per token APIs, just because everyone wants to try open models, and then once they find a use case that’s really sticky, then they move over to dedicated. Swyx [00:03:51]: Is there a best practice on when it’s time to swap over? Philip [00:03:54]: Couple reasons. Yeah, reliability, that’s a big one, right? Ali [00:03:57]: Like, if they have a very specific use case, they want you to train something specifically for them, like they want their own spec dec, for instance, for their own traffic. Swyx [00:04:04]: Spec dec is speculative decoding. Speculative Decoding and Custom Speculators Ali [00:04:05]: Speculative decoding, yeah. Swyx [00:04:07]: You have to explain. Ali [00:04:07]: Sorry. Like, speculative decoding is like, if you have a huge model, right? And so the model is going to be generating one token at a time every single turn, every single forward pass. So we attach, like, this little, like, parasite, like this layer that goes on top of the model, and this model just has to predict. It does three very fast autoregressive forward passes, and it will predict, like, three certain tokens, and then you do one forward stage over the entire original model in order to see if those predictions were correct or not, and then you accept them or you reject them. Now, this draft model is traffic specific, so if you, like, Philip said, if you’re summarizing Harry Potter books, I can train exclusively that draft model on Harry Potter books, and I can guarantee you that I’m gonna accept the three tokens every single time. And so with that case, I increase your decode speed. I wouldn’t be able to provide this to you if you’re a shared endpoint Swyx [00:04:53]: Yeah Ali [00:04:53]: ‘cause I have no idea if you’re doing Harry Potter, if you’re doing coding, if you’re doing English. We don’t know. Also, there was a thing in the book that mentioned that if they really cared about a specific threshold, chapter four, I think. Do you remember that? Philip [00:05:06]: Yeah. The things that you can do is you can set a specific, like, batch sizing, a specific, like, parallelism strategy if you’re trying to optimize for, like, throughput versus latency. You can. Maybe a NVFP4 quant doesn’t pass your benchmarks and you wanna run a model at higher precision, you could do that. There’s just a bunch of reasons why you might wanna have your own endpoint and the biggest one, of course, just being, like, you don’t have to deal with someone else doing a hundred million tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users. Swyx [00:05:40]: Yeah. I think one thing that is. That is a classic journey. Like, it’s people is asking the, what happens when you type Google into the browser. Tool calling, is that just, you’re generating JSON or is there more complication beyond that? Tool Calling, JSON, and Structured Outputs Ali [00:05:58]: Certain customers that we have, they have their own post-trained models, and so they demand a tool calling that’s not just, like parse a file or go find the weather. It’s something that’s very specific and you have to do post-training on this. And if the post-training on the model is not good or if the quantization after the post-training to get the inference to be fast, the model will struggle reading the JSON file and reading the tool calling. But it doesn’t require its own like sandbox. It’s not like it’s going to use that tool calling to like escape a sandbox or like it doesn’t have to be contained. It can just be a normal dedicated deployment. The challenge with tool calling more and more seems to be that the companies want certain tool calling which is a very sensitive thing to train. And because you’re dealing with all of the JSON outputs, if it doesn’t like close the end of the request in a very certain manner, you end up with a model that did the tool calling and like the thinking and so as a result of that, it didn’t see the result and just hallucinated the result as it decoded. That seems to be the most challenging thing with tool calling, not really the sandboxes model. Philip [00:06:56]: Yeah, that’s a challenge on the training side and then on the inference side, there’s work that you can do to scope the possible output. So we published this at this point close to two years ago, the solution to this problem which is you make a state machine and you use that to constrain the output to a specific format. So this is the structured output problem. If you remember back Swyx [00:07:27]: Yeah, the specific grammar is, Philip [00:07:29]: Yeah, exactly Swyx [00:07:30]: GML had this thing. Philip [00:07:31]: Yeah. So it’s like the old-school “make sure this is only JSON”, return only JSON or Swyx [00:07:38]: Yeah Philip [00:07:38]: Grandma’s gonna die type of prompts. Swyx [00:07:39]: Is it BNF grammar? At some point OpenAI had released a thing that was like, yeah, if you want to constrain your output, write BNF grammar, back as NOR. Philip [00:07:47]: In our inference system, it’s just a specified output format. And you get the guarantee that your output’s gonna be structured along that format. And so applying that to tool calls can like help cut down on. You can still call the wrong tool or call no tool. It doesn’t solve the certainty problem but it at least solves the output structuring problem Swyx [00:08:10]: Yeah Philip [00:08:10]: Within tool calls. Swyx [00:08:12]: And MCP is just another form of tool, right. Philip [00:08:14]: Yeah, exactly. Swyx [00:08:15]: As far as there’s no special thing there. Philip [00:08:16]: The thing I’m always like explaining to people is the LLM is not capable of doing anything. It’s only capable of making suggestions of what to do and then if those suggestions are formatted in a certain way and applied to a system that knows what to do with them, then an action occurs. Swyx [00:08:32]: Yeah. Part of the fun stuff is, this is solved outside of tool calling too. Like in an agent loop if the output is not correct or you’re right, like reasoning, tool calling was done in the reasoning trace, just be like, “Oh, I don’t know what to do. Let me just try again.” And it might get there after a few tries. And on your point of training, sometimes this is harder in smaller models, so you don’t have the same exact quality output Ali [00:08:56]: Right. Swyx [00:08:57]: When you just swap from a big model, right? Ali [00:08:59]: Yeah. I will say that, before, I think we need to go back to inference engineering proper. Ali [00:09:04]: But, I had expected that something would replace JSON because it’s hard to stream JSON ‘cause JSON must be complete and you must have open and close brackets and everything. So it’s hard to parse something or validate something while it’s being streamed. So people invented all sorts of things that are like, I forget the name of some of these alternatives, but it’s something like TOML, something like YAML. But JSON seems to be dominant still. Philip [00:09:30]: The JSON outputs aren’t that long, right? Like you could have a long-- ‘cause tool calls also contain the arguments in them and perhaps for a certain tool you might pass like a very long argument. But my impression of the median tool call is that it’s a relatively small number of tokens, right? So I would expect that speculators are generally fairly good at something as formatted as JSON. And so you would have like a pretty fast decode step there and that the streaming wouldn’t be as valuable, but maybe I’m wrong about that. Ali [00:10:02]: I think you’re also bounded by the software or that the model is gonna integrate with if the software is built with JSON for the tool calls or if the company that you’- if your customer says that this is how our software works and our tools are interfaced with JSON, you can ask them to like, change their software and say like, “Yeah, this is gonna be better for the model.” but like with the right training shouldn’t be that much of a difference. Also more profitable if it outputs more tokens probably. Swyx [00:10:25]: Depends on your business model. Swyx [00:10:27]: It really depends. But I will say that, as a writer with like experience a lot with generated output, I do try to move from text to JSON text which is very long JSON, right? Like there’s paragraphs in every field because I’m trying to structure it, right? Philip [00:10:44]: Right. Swyx [00:10:44]: I want you to first make factual statements, then make opinions then make bullet point summaries, have dates, have entity references have your sources for references, all these things. Anyway, so these are things that like I think people who really experiment with structural output have to really care about. But, let’s, let’s recurse up the stack a little bit. Before we started recording, you mentioned something really cool, which is that there’s a lot of engineering that-- inference engineering that goes on when a new model provider releases a new model, right? So let’s call it GLM-5.2, Kimi K3. I had previously assumed, especially if it’s like, well, GLM 5 to 5.1 to GLM-5.2, like that you’ve supported them before. Is it that much work? What It Takes to Support a New Open Model Ali [00:11:26]: It’s a lot of work. Swyx [00:11:28]: Yeah. Okay. So like, a lot of people, all you guys, right whenever a new model launch like, people rush to say like, “Oh, Hugging Face supports this, Fireworks supports this, Spacetime supports this,” and I’m like, “Yeah, of course we support it.” But what goes into that? What goes into Philip [00:11:40]: I think it’s more than just support it too, right? It benefits the consumer a lot. Like I think it was with Kimi K2.5 or GLM-5.2 the latest, there was an inference war, right? X provider is at 90 tokens a second. The next day we’re at 150. The next Swyx [00:11:55]: I kinda kicked that off with the GLM-5.2. Swyx [00:11:58]: I wrote a Twitter article about. It got like half a million views, Ali [00:12:02]: Based on being number Swyx [00:12:03]: Yeah Ali [00:12:04]: Or it’s for something else. Swyx [00:12:05]: Yeah. Which, Ali [00:12:06]: Oh my God Swyx [00:12:07]: Which then got everyone really excited about, hey, how can we, bend tracks a little bit further and, Philip [00:12:14]: There’s a difference between support the model, as in I can make a token out of this model, and support a model, as in I have a production-ready API from this model. Philip [00:12:26]: Getting to the point of I can make a token out of this model is not that hard because generally the, open source inference engines, vLLM, SGLang of the world oftentimes even receive weights ahead of time, maintainers do, or the people making the model merge PRs to ensure support. So you generally can, just get it working on the standard open source stack without too much pain in most cases. The challenge is, every inference company is gonna have own proprietary stack. Some open source components, some in-house stuff. And for any arbitrary model, there’s going to be some new stuff. Sometimes you get lucky, like K, two five to two six was, like, pretty similar. Quantization, Speculators, and Production Readiness Ali [00:13:16]: Yeah. It was pure continued post-training Philip [00:13:18]: Yeah Ali [00:13:18]: If I remember correctly. Philip [00:13:19]: Even in those cases, there’s still stuff you have to do. You have to redo the quantization work. You’re taking the model from. Generally, these models are not released in NVFP4, and we want them to be in NVFP4 for maximum Blackwell compatibility. So we have to perform that quantization, and, calibrate the quantization to make sure that we’re not causing any regression in the model’s intelligence. And then we also have to train the speculator, as we’ve talked about. Generally, we have. We have ZDR, zero data retention on our model APIs, so we don’t know exactly the traffic that people are sending us, but we know what’s popular. We know that coding use cases are popular. We know that agents, agentic use cases are popular. So we can get public data sets that are representative of that traffic and train general speculators. Now, with speculators today, you need to train the speculator using the base model itself because you’re getting hidden states out of the model from running inference on these specific prompts, and that is the training data you use to create the speculator. So there’s that process which you need the real model weights for. And then there’s of course just the process of, standing up all the infrastructure behind it, loading all this stuff, testing it. And then when there’s a new model with a newer architecture, I think that, like, the DeepSeek models tend to be the most challenging as they have, like, the most novel architectural stuff going on, model after model. But every new model has something. Kimi K2 had. Oh, sorry, GLM-5.2 had Ali [00:14:53]: Sparse attention. Philip [00:14:54]: Yeah, Ali [00:14:54]: Yeah Philip [00:14:54]: the DSA. Ali [00:14:55]: Right. Which is brought from DeepSeek. Philip [00:14:57]: Yeah. And Ali [00:14:59]: So you can copy-paste then? Philip [00:15:01]: It kind Ali [00:15:01]: I don’t know how this works. Philip [00:15:02]: So, like we had to, like, build support for that into our runtime. And you’re right, like it is really interesting the way that all of these open source labs borrow from each other. For example, like GLM-5.2 doesn’t have vision. So something that, Haley, a guy on our team, if we could take a look at this, he, like, grafted the Kimi vision encoder onto GLM-5.2. Retrofitting Vision into GLM-5.2 Ali [00:15:27]: We’ll be training the projector. Philip [00:15:28]: Exactly. So if you think about, like, the encoder, there’s the encoder, which is the part that looks at the image and turns it into latent information, and then there’s the projector which like Ali [00:15:38]: You can say latent space. It’s okay. Philip [00:15:41]: And then there’s the projector that maps it onto, the model itself, and then there’s the model weights. You don’t wanna mess with the model weights because you run a chance of making the model dumber at something else for the purpose of giving it vision. So instead, Haley started with just a projector, which is only a handful of millions of parameters. Ali [00:16:02]: That would be, yeah. Philip [00:16:02]: Yeah. Ali [00:16:03]: Can you show the training one? Ali [00:16:04]: Like the way it groks Philip [00:16:05]: Yeah Ali [00:16:06]: Very interesting. Philip [00:16:06]: And maybe Ali [00:16:07]: That right there Philip [00:16:07]: Maybe Ali, you should take it from here. You’ve got a better Ali [00:16:10]: Ooh, double the sand Philip [00:16:11]: Understanding of this than I do. Ali [00:16:11]: Yeah. You can see, like, he. The way he trained this is really cool. At the beginning, he was training it using just like, “Here’s a picture of a mountain. Can you describe what’s in this mountain?” And that caused it just like the first, learning walls. Like here you can see this all we’re trying to teach it is to translate the encoded. Like it’s already taken the encoder from Kimi K. It’s taken the image. It’ Philip [00:16:31]: Yeah. Frozen Ali [00:16:31]: Frozen Philip [00:16:32]: With adapter. Ali [00:16:32]: Exactly. Philip [00:16:33]: Yeah. Ali [00:16:33]: So the brain is frozen and the eyes are frozen. It’s just we’re trying Philip [00:16:37]: Align Ali [00:16:38]: Interconnect between the eye and the brain, right? So the projector. And so you take the tokens and then he’s like, “Oh, can you describe what’s in this image?” And he’s like, “Oh, it’s a mountain,” or it’s a person or it’s a human, whatever the case is. But that didn’t cause complete understanding. So he changed it such that every image was associated with a data set of questions. Like, does this image have a white male? Does this image have birds in the top corner? Does this image have a scientist in it? All of that stuff. And it would have to answer questions correctly. And using not just training on describing an image, but being able to answer question, another question, answer over time. Like you can see the grokking, which is like genuinely insane, that retrofitting vision into a large LLM can learn to that extent. And even for images that it doesn’t perform well on, for instance, if you ask it a picture of like Stephen Hawking, “Who is this?” Maybe it doesn’t get it, but it will say something like, “This is Albert Einstein.” Like it still understands Philip [00:17:25]: Close enough Ali [00:17:26]: That this is a scientist who is a man who has, some significant achievements, all that stuff. So that’s like really cool. Philip [00:17:32]: Yeah. So, we’ve covered Hao Tian before, who the author of the LLaVA paper that did this, a while ago. And I think that’s very foundational work for anyone who hasn’t done vision work before. Ali [00:17:41]: Same with the CLIP and MetaCLIP, where you go from just captioning to building out questions Philip [00:17:47]: Right Ali [00:17:47]: Off the image and how much better you can get performance. Philip [00:17:50]: Right. Right. Right. Yeah. But what’s, what’s so exciting about this is if you look at a model like this. Now, this is a little bit more of a research project. It’s not. It got to 56% on MMLU Pro, I think. So not quite frontier. But if you’re running this model, you haven’t suffered any loss on your GLM-5.2 quality. If you don’t have an image, it’ll just behave exactly the way it used to. And ultimately Ali [00:18:14]: Which in the inference code you literally do not include the other part, right? Philip [00:18:18]: Yeah. You would just skip the encoder if you don’t have an image input. Ali [00:18:22]: Okay. Philip [00:18:22]: Just confirming. Philip [00:18:23]: Yeah Ali [00:18:23]: Does it affect a lot on the overall inference side? Like you’re not adding much, you’re adding a very small vision encoder. These are typically like Philip [00:18:30]: They’re super fine Ali [00:18:31]: Less than a billion parameters, right? Philip [00:18:32]: Yeah. It’s, - There’s a little bit less standardization among vision encoders Swyx [00:18:37]: Yeah Philip [00:18:37]: So the support matrix can be a little bit, sparser. But overall, yeah, it’s a pretty, it’s a pretty minor component of the overall system. And ultimately what you get out of the system is all of a sudden you have Kimi Vision, GLM weights, and DeepSeek attention all in one model. Open Source Model Grafting and Franken-Merges Philip [00:18:56]: And that’s, I think, a lot of the power and beauty of open source, is that you can take all of these different components and combine them together into a system that’s better than anyone Swyx [00:19:05]: Yeah Philip [00:19:05]: Can be individually. Swyx [00:19:06]: People used to say that you would also do Franken-merges where you would take like Philip [00:19:10]: Yeah Swyx [00:19:10]: Layers from each model. Swyx [00:19:11]: Does anyone do that anymore? Ali [00:19:13]: Well, to your point previously when you were mentioning like, the work that goes into supporting a model when it first comes out, like GLM-5.2 or MiniMax M3 or whatever the case is. Sometimes you do have to like, you do have to switch out some things. Like, for instance, the MiniMax M3 head uses full attention, and with full attention you end up with this like insane bottleneck in spec dec ‘cause you’re doing auto-regressive token generation for three tokens, and you’re doing this like N squared over all of the tokens that are in your sequence. Your KV cache is like very large because it’s not sparse, it’s not top K. So we find it better to like, okay, we’re gonna replace this, we’re gonna replace this layer with a layer from another model that’s using like GQA, for instance. And then just with the right training, you can get it to have the same acceptance rate. So it is very possible to retrofit layers from other models and very much needed. If a layer is like inefficient, the training just becomes the challenge, like how do you ensure that you train it properly? Which again to your earlier point is like the mesh between training and inference. As in like you need very good training in order to do fast inference. That’s like, I feel like more and more becoming true. Swyx [00:20:21]: Yeah. Anything else on the support side when you say like get it to fully production ready? Loop Detection, Race Conditions, and Non-Determinism Philip [00:20:26]: Yeah. I think that there’s also a question of just, we can test a model to a pretty extensive degree, but we’re trying to get it out quickly and then you see a bunch of other people test it and you get interesting results. There was an issue with, GLM briefly where we had some like mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures. Like once you expose an endpoint to the real world, there’s going to be, so many more varieties of things given to it that you’re able to, discover and patch things. So it’s not just a, day zero process, it’s then like for the first week, for the first month, if a model remains popular, like how do you both fix bugs and then continue to push the envelope on performance? Ali [00:21:21]: What do you mean you don’t want your model outputting S? Swyx [00:21:24]: Is there loop detection on that stuff, by the way? It still happens like quite a lot, which is surprising. Ali [00:21:30]: We have like we, in our endpoint, like if a model was to output the same token like four plus times, we just cut the generation. We say like, “Oh, sorry, this-- Like try again,” or like we will reprocess the request. ‘Cause we know then, like if it, like if, yeah, it’s four times the same token, it’s probably collapsed. Swyx [00:21:45]: Yeah. Is there a way to opt out in case I really want that? Ali [00:21:48]: You want that? Ali [00:21:50]: I think there’s a way that we have to handle it. I’m not exactly certain, but I feel like in certain models, like when they output something like you can imagine, like a table for instance, and so they want, they wanna draw like 12 dashes and 12 dashes. Yeah, I think there’s a way for that to happen. I think we only do it on certain tokens. Like we exclude certain special characters. Swyx [00:22:07]: Yeah. Ali [00:22:07]: So we only do it on like certain like S is the most common almost. GLM-5.2 Swyx [00:22:11]: Oh Ali [00:22:11]: And I think it was DSV 4 as well. Like you’d just have like looping issues where like you literally Swyx [00:22:17]: It Ali [00:22:17]: Just have like S. Swyx [00:22:18]: Yeah. Is there a special, something special about S? No, just randomly Ali [00:22:21]: It just seems to be the one token involved. Swyx [00:22:23]: Yeah. And it’ Philip [00:22:24]: Is there Swyx [00:22:24]: And it’s only temperature 0 Ali [00:22:27]: No Swyx [00:22:27]: Even at other temperatures Ali [00:22:27]: Even at like 0.9 or whatever, it will still, it will still collapse. Swyx [00:22:30]: That’s weird, right? Ali [00:22:30]: It’s, it is an inference problem to be honest, like a software problem. Like oftentimes, the image you run will-- like NVIDIA will release an image for instance, and if we will upstream the changes from their latest TensorRT-LLM image into our stack, we’ll find that it fixes it. Or oftentimes this will only happen in an inference engine that you’re using like SGLang. But if you were to switch to vLLM, that isn’t the case. So it seems to be like an extremely like deterministic software issue and not really a model issue. It’s not like a weights problem. Like I’- we’ll say like, “Oh, it’s a problem with the quant. We did PTQ wrong,” right? But that isn’t, that doesn’t make sense because the same weights used with a different inference engine does not repeat the problem. And sometimes it’s, the kernels that are being used in the backend have like these very subtle sometimes race conditions, where if you were to use this model hosted on one cluster, you will never get this problem. Swyx [00:23:19]: Oh my God. Ali [00:23:19]: But if you host it on a different cluster, you will. And the reason is the KV cache transfer from a node to node in that one cluster is using a slower interconnect than the node to node in another cluster. So that exposes the race, whereas in another cluster it doesn’t. So then you end up just like, okay, this model is not gonna be hosted on this cluster. We’re gonna host it on, another cluster because that cluster exposed that problem. But then it ends up with like, okay, is it the software? Is it the model weights or is it the hardware? Swyx [00:23:42]: There is a thing about this with temperature 0 still not being deterministic, right? Ali [00:23:46]: Right. Swyx [00:23:46]: Mostly because of hardware. Even at temperature 0 same model, you won’t always get the same output. Swyx [00:23:52]: Even-- But I’m surprised by the race condition one because, I thought PyTorch was a graph that like guarantees that you at least, execute things in the right order. Ali [00:24:02]: Well, yeah, true. Like I’m not, I’m not saying that there is. Like well, you have things like PTL optimizations where like you can start a kernel before the end of the previous kernel, and that’s like ‘cause you want to do that because there’s Swyx [00:24:12]: It’s like pipelining Ali [00:24:12]: Expense. Exactly. Swyx [00:24:13]: Yeah. Ali [00:24:13]: But it’- But you don’t do it cleanly. Like you overlap a little bit of the execution. No, it is very possible that the kernel itself, like that one block that is supposed to be running in this instance of time, that kernel itself has a race condition. For instance, like a missing barrier. Like often if you’re designing a kernel and you want it to make it to be very fast, if you don’t test it extensively, you’ll, you’ll have certain threads access data points from registers before they’ve been written to by other threads Swyx [00:24:36]: Yeah Ali [00:24:36]: For example, because like your barrier is wrong or your synchronization was wrong. But yeah, like the testing itself is very difficult in those like, and Swyx [00:24:42]: And there’s no like borrow checker Ali [00:24:45]: What does that mean? Swyx [00:24:46]: Like Rust. Like the. If you’re trying to have like memory safety It sounds like a comparable problem. Ali [00:24:52]: Well, yes, but you’re working in CUDA, right, NVIDIA GPUs. Like- You just need a higher level language like modular Maybe that’s what modular is supposed to do. I don’t know. Quantization Quality and Vendor Fidelity Vibhu [00:25:00]: How do you see keeping quality of the model? So you talked about all these steps of, okay, you gotta do quantization, train your own speculative decoder Ali [00:25:07]: Right Vibhu [00:25:07]: Run on different hardware. Looking at other model providers, okay, you kicked off a inference speed race on the consumer end. What goes into keeping quality the same across them, right? Sure, you can run benchmarks Ali [00:25:22]: Yeah Vibhu [00:25:22]: But, like, how do you determine how much quantization are there standards? What goes into Philip [00:25:27]: There’s a few things on quality. Most inference optimizations are lossless. KV caching, for example. You are just recomputing or preventing recomputing the same values. Speculation, of course, if a draft token is wrong, it gets rejected. The main lossy optimization is quantization. And that really comes down to, number one, data format, number two, which parts of the model you choose to quantize, which layers, and number three, like doing a lot of calibration on the quantized weights, to ensure that you’re preserving all the outliers. There’s other tricks that you can do, though. A big one is long context, ‘cause one thing you asked at, right at the beginning is, “Oh, what’s gonna happen if I send a 200,000 token request in?” So with a long input sequence, you need to, store a lot more information. You need to process a lot more tokens. And so even if a model has a context of a certain length, you might, as an inference provider, choose to build an API with a shorter context length, and of course a full length one as well. Because if someone doesn’t need the full million token context, for example, you can get them better performance. I don’t know if that’s exactly like quality of the model. The way that I think about quality is to what degree are we faithfully serving the original model? If you think of a golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that, 100% fidelity of the model. Philip [00:27:13]: You can also, of course, think about quality from the training side and how do you push yourself past 100%. But when I think about purely inference optimizations, it’s getting faster while staying as close to that 100% fidelity mark as possible. And certainly our standard internally is that, like you should not be able to tell the difference between our API and a, official API. I think Kimi in particular does a good job of vendor benchmarking here Ali [00:27:41]: Yes Philip [00:27:41]: Where they have Ali [00:27:42]: They released an actual vendor benchmark. Philip [00:27:43]: Exactly, yeah. Ali [00:27:44]: ‘Cause they accused, some people, Amazon? There was some provider that was not doing very well on Kimi’s benchmark. Philip [00:27:50]: Yeah. Philip [00:27:51]: So, with Reflect we probably Vibhu [00:27:52]: This was a long time ago, right? Philip [00:27:54]: No. Ali [00:27:54]: Yeah, like three Vibhu [00:27:55]: They also Ali [00:27:55]: Four, five months ago Vibhu [00:27:57]: This also happened with, I don’t remember which model, but they pulled out quite a few, and then they started a whole chart about this. It might have been Philip [00:28:03]: Kimi Vendor Verifier. Ali [00:28:04]: Yeah. Philip [00:28:05]: Yeah. Ali [00:28:05]: Yeah, ‘cause you, ‘cause you’d be pissed, right? Like if you’ Philip [00:28:07]: Yeah. Ali [00:28:07]: If like if I’m a consumer and I’m using like Amazon’s endpoint for instance, and I’ve used Kimi and I’m like, “Oh my God, like this is bad,” I’m not gonna say, “Oh, Amazon quantized the model in a bad way.” I’m gonna say, “Oh, Kimi sucks.” Right? Philip [00:28:17]: Yeah. Ali [00:28:17]: So it seems like that makes sense. Philip [00:28:19]: Yeah, they care. They care. Vibhu [00:28:21]: Justifiably. Ali [00:28:21]: Yeah, justifiably. Vibhu [00:28:22]: This is probably a stupid question, but just checking, has anything improved from main quantization? Philip [00:28:28]: Yeah. Vibhu [00:28:28]: Like, is quantization always strictly worse? Ali [00:28:30]: Well technically Vibhu [00:28:32]: No Ali [00:28:32]: It’s a lossy. Quantization Philip [00:28:33]: Yeah Ali [00:28:33]: Is a lossy, it’s a lossy implementation. Philip [00:28:36]: Speed improves Vibhu [00:28:36]: Speed improves. Ali [00:28:37]: It the number, like Vibhu [00:28:38]: No, I’ always look for inverse scaling laws. Philip [00:28:40]: Yeah. Ali [00:28:40]: Yeah. Vibhu [00:28:40]: This is something I learned from Noam Brown, where like things that normally act in one direction sometimes do. Philip [00:28:45]: Well, technically when you run a benchmark, because these models are deterministic, sometimes your, Ali [00:28:52]: Yeah Philip [00:28:52]: NVFP4 quant is like, two basis points higher than your Ali [00:28:56]: No, it’s noise. It’s noise. Philip [00:28:57]: Yeah, exactly. I’m like, yeah, it’s, it’s within. That’s why I always say within margin of error. Philip [00:29:01]: And I stopped saying that because everyone assumes that what is, well, within some margin of error, we’re barely inside of that to the worst, so we’re saying. But yeah, sometimes it’s just like, gives you a higher output score. But like Ali said, that’s noise. To my knowledge, you’re not necessarily making the results better. You’re just trying to, again, like keep your fidelity as close to 100% to the original model. Layer Selection, KL Divergence, and Better Quantization Ali [00:29:27]: There is, to your point, research that we did on MP. I don’t know if you are able to pull Philip [00:29:31]: Yeah Ali [00:29:32]: A tweet we did. One of our research interns, Joshua, I think it’s a tweet on how we have 20% better quantized GLM-5.2 than NVIDIA. Essentially what we found throughout like this month research is, okay, quantization is a lossy. It’s. You’re compressing the data from, occupying 16 bits to occupying, four bits, for instance. And so you’re losing some information, and you’re trying to minimize that. And so when I say that I’m gonna quantize the model, my job becomes how do I find the layers that I can quantize, and how to find the layers to not. For instance, with image models, I don’t quantize modulation layers, and I don’t quantize out projections because those two are. Like out projection is what you see as the user. Modulation is what the model sees or understands. Right, exactly. And so to his paper, do you have the. It doesn’t have the. Yeah. It’s a long paper. I don’t know if I can find Vibhu [00:30:25]: If there’s a part to search or it’s probably in the thread. Ali [00:30:28]: It’s probably in the thread. Vibhu [00:30:29]: Yeah. Ali [00:30:29]: But the long and the short is it is very possible that quantizing more of the model makes the results. Like if I have a model that I quantize layers one, five, and 10, and another model where I only quantize layers one and It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematical proof where he had like a verifier in, is that you can predict which layers are going to have quantization errors that will cancel out with each other, and you choose to quantize those layers. And so the result of doing this mathematical quantization is you end up with a model that’s 20% more quantized than another provider, so you get 20% more throughput of it because there’s more layers than running an NVFP4, and your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out, like one layer skewed to the right one layer skewed to the left, one layer skewed to the right. Your final logits distribution is more similar to the original distribution of the model, so you have better fidelity. And so the way we proved this was with KL divergence. So instead of just scoring on the benchmarks, we scored the KL divergence between the logit distribution of the quantized model and the logit distribution of the original full precision model, and we showed that with this technique we get. If your probability distribution on the logits which token it wants to select is more of the same as the original model, you’re probably gonna end up staying true to the original model. So yeah, so it seems like previously before this, it seemed like the industry was, well, the more you quantize, the worse it’s gonna be, ‘cause the more loss you introduce. That’s not exactly, not necessarily true. So yeah, doesn’t improve it, but can cancel out. Philip [00:31:57]: I think it might be this, but reminds me a good bit about pruning where you can prune off certain layers. Philip [00:32:03]: But very interesting. Didn’t know this was a whole paper you guys put out. Ali [00:32:06]: It’s. Fun fact, it was originally 72 pages, this paper, and then we decided Philip [00:32:11]: Wow Ali [00:32:11]: We can’t tell. We couldn’t release it. So it’s now 45. Swyx [00:32:15]: Still 39 pages, so very substantive. We talked about evals and all these things and, like what’s possible in terms of speedup? Like it’s like probably like the number Inference Speedups and Benchmarking Swyx [00:32:25]: Thing that people do wanna care about, and it’s something that you wrote about in your post. Like official API is 70 tokens per second, and you push it up to 90. Is that like a normal thing? Philip [00:32:36]: So what’s cool about working in inference, the reason that I think inference is going to be a useful place to do engineering for a long time, is that if you look at highly optimized domains like, say, finance, if you’re in finance, you measure how much better you got in basis points. It’s like, “Oh, I got five basis points better, like twentieth of 1% better,” that’s huge news because everything is so optimized. When we publish optimizations, it’s 20%, it’s 100% it’s 200%. So there’s still probably like a lot further to go, honestly. Like you’ll, you’ll know that inference is pretty much solved when researchers start publishing about how they got 1% faster at something. Swyx [00:33:19]: Which by the way, because I am from the finance background, in the ‘70s, that was the margin at the time. When you did quantitative finance research, you would find Ali [00:33:27]: And like 20%, tens of percent. Swyx [00:33:29]: That’s. Yes. Philip [00:33:29]: Yeah. Swyx [00:33:30]: And now it’ Philip [00:33:31]: Tiny fractions Swyx [00:33:32]: For those people interested, look up Andrew Lo’s paper. He had a really interesting illustration of quant, stat arb, distribution, narrowing down from like those kinds of 20% differences in the ‘70s, down to nothing today, which is very cool. Philip [00:33:48]: Exactly, and we’re at the beginning of the same type of thing. Now benchmarking is hard. I think anyone will tell you that, and benchmarking provider speeds is hard because there’s so many variables that go into it. What hardware are you using? How much load do you have on the system? What’s the exact nature of the prompts and input and output sequence lengths? All that stuff. But overall, when you start stacking these improvements, you’re looking at multiples. You can look at it. The most common form, of course, is TPS, tokens per second, which is bad naming by us in the industry, ‘cause there’s two tokens per second. There’s tokens per second, the throughput number, and the latency number. Ali [00:34:31]: TTMT, yeah. Philip [00:34:32]: Like total tokens per second out of the, out of the GPU as a throughput number. Most people only care about tokens per second as the latency number, which we should call ITL, intertoken latency, but we don’t. Philip [00:34:44]: Anyway, so you can imagine a standard API without many optimizations for a 1 trillion parameter model operating somewhere in the 30 to 50 tokens per second range for reasonable traffic profile. And we generally see the goal of, pushing to 10X that. But, not necessarily day zero, but by stacking enough optimizations, if you have, say like four optimizations, each of which doubles performance. Or sorry, three optimizations, each of which doubles performance, then you stack that up, that’s an 8X gain. That’s the order of magnitude that we’re working with in this space. We’re trying to make things substantially faster, not just go from like 70 to 90. Swyx [00:35:38]: Are you saying you’ve. You have done that? Philip [00:35:40]: So let’s say you have as a reasonable baseline, 30 or 40 tokens per second. You can achieve 10X that. So like on GLM-5.2, if you run it unquantized, perhaps on H100s even, and you’re just using an off-the-shelf inference engine with no particular optimizations, no speculator, nothing extra around like KV routing, no disaggregation, you’re, you’re probably, yeah, looking at that like 30 to 40. You think that’s like a reasonable baseline? Swyx [00:36:12]: Right. Right. Philip [00:36:12]: To get to something like 10X, there’s a lot of trade-offs that you’re making. If we’re running at more like a 300, 400 tokens per second range, you are using the best hardware possible. You have a optimized speculator. You have done all of your quantization work. You are Seeing a pretty high cache hit rate. You are running with a reasonably small batch size and a parallelism configuration that is tuned for latency versus throughput, but it is possible. So the spreads that you see if you, like, go on artificial analysis or you go on OpenRouter and you look at, the worst provider to the best provider, oftentimes can hit that range. 10X is of course very aggressive. It’s oftentimes maybe more of a four to six times improvement. But that’s the performance that makes us really excited, is when we can get these huge gains, not just go from 70 to 90 tokens. Stacking Optimizations: NVFP4, Speculation, and Disaggregation Ali [00:37:19]: It’s also, like, hardware dependent. Like, if Philip [00:37:20]: Yeah Ali [00:37:20]: If you have a thing where you’re serving it on just, like, a node of H100s and then you throw, like, you shard the model across, like, four nodes of B200s. Like, you can definitely increase the speed with just throwing more hardware at it. Like, normalizing for the same exact hardware and the same number of GPUs. Philip [00:37:35]: Yeah. Then you’re looking at, like, a two to 4X improvement Ali [00:37:38]: Right. Right Philip [00:37:38]: Depending on the inference optimizations. So yeah, it’s. Some of it’s, what’s the call, and some of it’s who’s the driver. Vibhu [00:37:46]: If you break down the two to 4X, say the example is run GLM-5.2 Ali [00:37:51]: Yeah Vibhu [00:37:51]: On B200s Ali [00:37:53]: Yeah Vibhu [00:37:53]: Single node, right? What’s, like, the cost trade-off for effort to get, like, the last bit of juice out versus what should people just think of, right? Ali [00:38:01]: Spectre quantization. Yeah. Vibhu [00:38:03]: Spectre quantization. Ali [00:38:04]: That’s, that’s, that’s like 95%. Like Vibhu [00:38:06]: And how far does that get you? And how easy is that for the average person to do? So say right I wanna throw the weights of GLM-5.2 on a node of B200s, how easy is it to find speculative decoder- decoder model or already quantized model? How much work goes into it? Philip [00:38:23]: If you’re doing it up front, it’s quite a lot of work. If you’re doing it today, there’s going to be people who have published things that you can just, you can just grab some NVFP4 weights. You can grab a speculator. Yeah, if we’re thinking about, like, what are the 2Xs we’re stacking, going from, BF16 to NVFP4 is, it’s not quite a 2X, right? It’s like. I think it’s about, like, 30 to 40%, from 16 to 8, and then another 30 to 40% multiplied from, 8 to 4. So that doesn’t quite get you a 2X, but, like, roughly a 2X. Speculator, roughly a 2X. Disagg on top of that if you’re able to get enough hardware and put enough traffic through it, another roughly a 2X. And then you add in some, double-digit percent increase from having just a better runtime with, the latest kernels and stuff behind it. And that’s how it stacks up. Ali [00:39:21]: Yeah Philip [00:39:21]: So building each of those, like, building the, quantized weights is, for someone who really knows what they’re doing, hours to days of work. Building the speculator, again, like, hours to days of work. And the, disagg setup, hours to days. Well okay, but like once you have Ali [00:39:39]: Once set up. Once set up. Yeah Philip [00:39:40]: Yeah, getting disagg working for the first time, I’m saying, of course, is very difficult. Philip [00:39:44]: The marginal implementation Ali [00:39:48]: Like, if you’re just grabbing, like if you are a person, like just a normal consumer who has access to, like, a node of B200s and you’re wondering, “How can I just host it myself?” You don’t need to quantize the model yourself. There’s always gonna be, like, an open source quantized checkpoint. NVIDIA’s gonna push one out if no one else does. You. Usually, the providers will have their own spec dec that they’ve trained as well. You don’t need to train your own spec dec. You can just use that as well. Philip [00:40:09]: Yeah. Like, GLM-5.2 has its own MTP. Ali [00:40:13]: Right. Right. Vibhu [00:40:14]: What’s multi token prediction? Philip [00:40:15]: Yes. Ali [00:40:16]: I’m just Vibhu [00:40:16]: Can you explain that? Ali [00:40:16]: I’m just an expert. Ali [00:40:18]: I can do it for you in case I get it wrong? Vibhu [00:40:20]: No. Vibhu [00:40:21]: Yeah, you should correct if we’re wrong, but their multi-token prediction can be used for self-speculative decoding. Ali [00:40:27]: I’m not sure. I’m not gonna correct that. Vibhu [00:40:28]: Okay. I’m semi-confident in that Ali [00:40:30]: Okay. Yeah Vibhu [00:40:30]: But someone can check. But it’s useful to paint the story of, okay, not just the average person, but say a company wants to switch from serverless inference I wanna throw this up on. I wanna rent some GPUs, throw it up. These are the steps you take to do significantly faster than just put it behind vLLM. Ali [00:40:48]: Right. Vibhu [00:40:49]: I was waiting for a mention of Dynamo. Vibhu [00:40:51]: I feel like, that’s supposed to be the baseline that you measure against. Dynamo, KV Routing, and Disaggregation Toolkits Philip [00:40:55]: I would think of Dynamo as less of a box system and more of a toolkit for building with. So when we talk about doing aware routing, when we talk about doing KV offloading, when we talk about doing, PD disaggregation, Dynamo fundamentally is. By the way, Dynamo is an open source library from NVIDIA. Ali [00:41:17]: We’ve done a pod with Kyle Philip [00:41:18]: Okay Ali [00:41:19]: Kyle Cranin. Philip [00:41:19]: Cool. So then your listeners know then that it supports all the different inference frameworks. And it is multi hardware, which is interesting. Ali [00:41:28]: But it’s just a router, it’s not like an optimizer layer. Philip [00:41:30]: Yeah. All it does, like, what Dynamo is good at, it is a library for moving information around your cluster, around your hardware. So if you have, KV cache on one place and you need it to be somewhere else, Dynamo coordinates NIXL for you to move that around. Philip [00:41:49]: That doesn’t mean that, like, out of the box, you just say, “Pip install Dynamo,” and then you get, like, a massive performance speed up. It’s more of a developer toolkit. Ali [00:42:01]: Yeah. I would have said it would. It comes with a set of defaults that you can then swap out. Philip [00:42:06]: It does. If the industry at large, I think, was, like, rolling out all of these deployments, standard, then I think it would be, like, a credible baseline. But, we’ve got to, we’ve got to benchmark against, like, what we’re seeing in the wild. Speculative Decoding Methods: Medusa, EAGLE, n-Gram, and Spec-Spec Vibhu [00:42:23]: I did wanna talk a little bit more about PD disagg, because that is probably, like, number three after quantized and speculative decoding. In your book though, I was just gonna pull out the book. Philip [00:42:31]: Yeah. Vibhu [00:42:32]: Like section 522 on Medusa, 523 on EAGLE Philip [00:42:35]: Yeah Vibhu [00:42:36]: 524 on gram. Philip [00:42:37]: It’s 55, would be disaggregation Ali [00:42:42]: Yeah. Well, no, I just wanted to dwell a little bit Philip [00:42:44]: Yeah Ali [00:42:44]: The other. Like, so what do you choose to include? What do you choose to not to include? Because there was all these other techniques. Philip [00:42:51]: Yeah. Ali [00:42:51]: Are these still relevant? Because I think they came out, like, a year and a half ago maybe. Vibhu [00:42:55]: Medusa is quite old. Philip [00:42:56]: Yeah, Medusa’s old. Ali [00:42:58]: It was old. Vibhu [00:42:58]: But is it in the book as a good, here’s Philip [00:43:01]: Baseline Vibhu [00:43:01]: Baseline vanilla understand it? Philip [00:43:02]: Like you should know this. Vibhu [00:43:03]: Like I read the paper, I’m like, “ it makes so much sense.” Philip [00:43:05]: Yeah. Philip [00:43:05]: So with the book, I had a couple goals. One was to give people just a working vocabulary for the space as a whole, and the other was to give them some intuition about how each of these techniques works. As I mentioned in my AI Engineer talk, which is the first public addendum to this, the speculation space has moved much faster than everything else. So yeah, even at the time that I wrote the book Medusa, I very much included as a way for people to understand how the space evolved rather than what the most modern technique is. And now of course, there’s DFlash, dSpark. There’s, there’s newer techniques even than EAGLE, although EAGLE is still very commonly used. Ali [00:43:51]: SpecSpecta. Philip [00:43:52]: Yes. Speculative decoding. Vibhu [00:43:54]: What can Ali [00:43:56]: Oh, it’s a paper by Tri Dao and it’s like, it’s doing speculative decoding Vibhu [00:44:00]: Huh Ali [00:44:01]: For the speculative decoder. Philip [00:44:02]: Oh, in spec- oh my God. Ali [00:44:02]: It’s literally just an another. It’s like, yeah, that’s the most simple way to explain it, and it seems like he got trivial speed ups there. But it seems that the complexity with training, it’s almost like in our mind at least, it’s almost as complex as training GANs. Like it’s like a very delicate balance and oftentimes you, it’s just but yeah, it’s literally speculative decoding on speculative decoding. Vibhu [00:44:21]: Speculative. Ali [00:44:22]: Yeah. We saw this paper. Vibhu [00:44:24]: It’s interesting, right? Ali [00:44:24]: Yeah. Vibhu [00:44:24]: I wouldn’t even expect it to be very particular to train, I would Ali [00:44:29]: Right. Vibhu [00:44:29]: The naive part of me is like, okay, train speculative decoder. Ali [00:44:32]: But like, and it makes sense, like the whole idea of speculative decoding is you. It’s like, it’s like almost like the iPhone auto predict version but for a normal model, right? Like you’re just, you’re just, generating three tokens and you’re like, okay, I’ll do prefill on them. And so you save those three turns for your original model. Now your speculative decoder is doing three turns of auto regression, so why not just have an even smaller model? Ali [00:44:53]: The other question there is what are the size of speculators? So say for Philip [00:44:58]: Right. It’s like a billion parameters. Ali [00:45:01]: Like for MiniMax, it’s. Yeah. It’s like one layer. It’s like one 60th of the original model usually. Philip [00:45:06]: Yeah. I think we should do a paper when we get back to the office. Philip [00:45:10]: Speculative Ali [00:45:11]: Speculative Philip [00:45:11]: Decoding. Ali [00:45:13]: No, it’s, it does seem like how, when do you stop? But then it also seems like if you’re able to train spec-spec decode for instance, right? Like if you’re able to have a small model that is accurately predicts what the intermediate speculator is gonna predict, that is able to predict what the original target model’s gonna predict, then why not just use that smallest model directly, right? Vibhu [00:45:34]: Yeah. This is Ali [00:45:35]: Like it seems like Vibhu [00:45:35]: Adjacent to the routing problem. Ali [00:45:36]: Right. Vibhu [00:45:36]: Yeah. Ali [00:45:36]: Right. Philip [00:45:37]: The thing with speculators is one of the practical constraints on using them is that you do have to run a small model on the same hardware that you’re running the big model on. There is a orchestration and resource competition problem inherent in that, and that is one of the constraints on speculation in general, is that draft tokens cost resources to create and cost software complexity to manage. And so if you have like infinitely recursive speculators, you add in quite a bit of that complexity on the actual implementation within the inference engine as well, not just in the training process. Vibhu [00:46:17]: I was gonna say, I would wonder if you could do similar, like distillation and pruning of, it’s the same thing, it’s just a model. Can we not just distill a lot of the weights, quantize the speculator, out of my domain? The question that also comes up is, this is all for big server workloads, right? How much of this applies to, say I have this MacBook, I wanna run Gemma really efficiently. Similar problems, not the same? Local AI vs. Data Center Inference Philip [00:46:45]: Pretty different. I talked to Selo, about this on his podcast a couple weeks ago. The difference between inference engineering for the data center and for production workloads versus inference engineering for local AI, is that we start with fundamentally like different constraints and different goals. With local AI, it’s how do I fit this model onto my hardware and then make it less dumb? And with data center influence, it’s how do I load this model and then make it less slow? And we care about less dumb, and they care about less slow. But the local AI inference engineering ecosystem, I think has a lot for us to learn from in the data center space. They are experts in various forms of quantization, including dynamic quantization that we just don’t touch, in the pruning, in the distillation, in the, layer removal. There’ Ali [00:47:42]: Layer removal matters less. Philip [00:47:43]: Yeah. There’ Ali [00:47:44]: No one loves pruning really. Philip [00:47:45]: Yeah. Well, but the, but they do Vibhu [00:47:46]: Which is surprising, right? But that’s, that’s a whole different thing Philip [00:47:48]: Just to fit something on the laptop. Ali [00:47:50]: Right. Philip [00:47:50]: So yeah, it’s a, it’s an interesting, it’s an interesting space. Not necessarily that like their techniques make sense for us to do in the data center, because we have different resources and different goals, but more that the process as well as the openness of that field is something to, admire. Ali [00:48:12]: Yeah. Like to your point, like, certain optimizations that would. Like for instance, Turbo Quantum Sharper, like it made such huge hype on that and we did like a whole deep dive on Twitter and like said, what is it? How does it work? Why is it good or not? And it took off and it was implemented on local devices because your memory bandwidth is so slow on like a MacBook, for instance. But try putting the same thing on like an NVIDIA GPU on a B200 Turbo quant would not be. Like, it would not be used. Like, NVIDIA - Like, NVIDIA made it clear that this is not a good optimization, and we’ve seen it firsthand where the overhead of doing dequantization, quantization of, in the kernel itself with turbo quant kernel, each end is much slower than the time that you save from doing the bandwidth. ‘Cause on the B200s, you have like 3.5 terabytes per second. You don’t need decrease the storage that much. You don’t need to do, FP4 KV cache. You don’t need to use a requant. There’s, there’s, there’s better optimizations to be made. But on Edge devices, it’s extremely important, it’s extremely useful. So, seems to be, like, different optimizations there, but then they’re all uniquely combined with like all you wanna quantize the model, you wanna do speculative decoding, like certain common prefixes with both Philip [00:49:18]: Principles. Ali [00:49:19]: Yeah, exactly. Exactly. Exactly. Philip [00:49:20]: They also do a lot of work on, model parallelism, especially over, heterogeneous topology, where you have, some sparks and they are wired together with, Ethernet, DGX sparks. Ali [00:49:35]: Yeah, this is the Exo Labs guys. Philip [00:49:36]: Yeah. You have, a number of, Mac Minis stacked up. Philip [00:49:41]: There’s, the inter. They. One thing that I think we both have to deal with, although they have to deal with a lot more is the interconnect between machines. Which is why, like, one thing that we do a lot is work with tensor parallelism. Philip [00:49:56]: And that’s where, you are using all of the, all eight GPUs, and sharding the model across it. Tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnects like NVLink. Was, they might be forced to do something like pipeline parallelism, which we’re never gonna do unless we’re doing some kind Ali [00:50:16]: Yeah. For image Philip [00:50:17]: Multi-node inference. Ali [00:50:18]: But since you mentioned it, I wasn’t sure if we were gonna cover it, but let’s briefly explain tensor parallelism and expert parallelism, since you have very nice images. Tensor, Expert, and Pipeline Parallelism Philip [00:50:25]: You wanna pull the book? Ali [00:50:26]: Yeah. Philip [00:50:26]: Yeah. Let’s, let’s get Ali [00:50:27]: So I just wanna show a few images. Philip [00:50:29]: Yeah. Shout out to Luke from Baseten’s design team for making these beautiful images. Oh, that’s a, that’s. Before we get into this, just one other difference is we talk a lot about the active parameters of a mixture of experts model, and for local inference folks, that matters a lot because if you have a batch size of one, you’re only activating that many parameters. When we Ali [00:50:51]: Yes. I was gonna Philip [00:50:52]: Inference in the data center Ali [00:50:52]: I was gonna bring that in the diffusion conversation. Philip [00:50:54]: Yeah. Philip [00:50:55]: Yeah. We, I, when we go through like a MoE model, and we host it, for an API, we assume that all parameters are gonna be active because Ali [00:51:06]: You’re batching Philip [00:51:06]: Throughout your batch Ali [00:51:07]: Yeah Philip [00:51:07]: You’re gonna, you’re gonna hit everything. Cool. So broadly, tensor parallelism you can do with any model. Expert parallelism, you can only do with MoE models. Effectively all models today are MoE models, that are, Ali [00:51:21]: Sort Philip [00:51:22]: At least all models large enough that you would care to parallelize them across multiple GPUs. So that’s, that nuance is less important now. With expert parallelism, the idea is you put the entire expert on a GPU. Generally, you have more experts than GPUs, so you might put like N experts per GPU, like eight experts per GPU or whatever. And then you replicate the router, which the router is very small, across each of the GPUs. And then by moving the generation from expert to expert, with each expert being inside a GPU, they’re not competing for resources. You massively increase the throughput that you’re capable of doing, and the, GPU connection is not as important ‘cause there’s not as much communication. Tensor parallelism requires that you are able to do this like all gather, all reduce. So you shard the model across the GPUs entirely. And then for each step, you’re combining the results of each of the GPUs, which is why the interconnect matters a lot, and it is generally. Of course, this is a, this is a very high-level generalization. There’s a lot of places where this is not correct. But generally, TP is helpful for latency, and in many cases, you will use some combination of these two parallelisms, across the model rather than just, like, picking one or the other. Do you wanna add some color there? Ali [00:52:50]: Like, yeah, usually, like in a model, it’s not. They’re not mutually exclusive. You do tensor parallelism and you’ll do expert parallelism. Pipeline parallelism less solely, it seems to me like we never use pipeline parallelism. Philip [00:52:58]: Yeah. The only reason you would have to do pipeline parallelism, which is where you separate like different layers and you put like half the layers on one hardware and half on another, is if you are forced to do multi-node inference, because a model is bigger than you have the. Like let’s say, let’s say you’re doing a deployment on H100s for whatever reason, and you’re putting a trillion-parameter model on there. You have to use multiple nodes of H100, and so you. - Because the interconnect is so slow between the nodes, the only viable way to parallelize there is pipeline, but then you would do expert and tensor within each node. Ali [00:53:36]: And the limiting factor for H100s is HBM? Philip [00:53:39]: Yeah. They just don’t have enough Ali [00:53:40]: How much? What’s the magic numbers that we need Philip [00:53:43]: Like on a B200 is 180 gigabytes per GPU, and then a node of eight, so you’re talking like 180 times eight. And the FP4, so each parameter takes half a byte, so that’s 800 gigabytes. On a H100, it’s like 140? Ali [00:53:56]: It’s 80. Philip [00:53:57]: It’s 80? Ali [00:53:57]: Yeah. Philip [00:53:57]: Oof. Ali [00:53:58]: Yeah. Philip [00:53:58]: I’m old. I’ve been doing this a long time. I remember H100 specs. Ali [00:54:04]: Yeah. Philip [00:54:04]: No, so one thing Ali [00:54:06]: You wanna tell me about the T4s? Philip [00:54:07]: The T4s. Oh my God. Ali [00:54:08]: Let me tell you what it was like to run a model on a T4 back in the day. Ali [00:54:12]: One thing I was surprised to see that more people didn’t do, Jamba. I don’t know if you guys remember Jamba from AI ‘21. They would specifically pick a hardware, and then they designed the arc dimensions for the hardware, and then it would saturate the hardware. Like, it makes sense. And like, somehow all these models don’t do that. Hardware-Aware Inference and Auto-Tuning Philip [00:54:32]: Don’t they do this for the training side, though? Ali [00:54:35]: I don’t know. Ali [00:54:36]: Sorry, Philip [00:54:36]: Training. For training the model. Ali [00:54:37]: Like deciding which GPU, which Philip [00:54:39]: Yeah. Well, how Ali [00:54:40]: Yeah, they do And with training, it’s more of like a math. Like you can run the math- Yeah and see the flops and maximize it. With inference, it’s more of like an auto-tuning, like if you like GPU kernel auto-tuning. But like it’s like you define that, “Oh, I have two GPUs. I can do TP1, TP2, EP1, EP2,” for instance, right? And you. So that gives you like total of like two squared combinations, and then you just like you shadow the same traffic, like real prod traffic, and you just see which configuration gives you the best TPM and TPS, and then just use that. I don’t like the fact that it’s, you cannot reason about which one’s gonna give you the best performance or that there isn’t one specific configuration that’s always best. But it seems like auto-tuning is just the way that you find the best one. And with kernels and GPU kernels, it’s much of the same. After you design your kernel and you design your configuration, how many threads do you launch? How many, how much shared memory do you use? You just auto-tune. You just sweep the parameter space on the side, and this is the best one empirically. But yeah, but they are combined. They’re not just entirely- Yeah like separation. There’s a few bits of training that are like hardware targeted. If you look at, for example, NVIDIA Nemotron models, they run very well on Blackwell. That’s, that’s unsurprising. So there’s some degree of that, but I think that most open labs are trying to make models that can be run on as wide of hardware as possible rather than targeting just like a single chip. I see. For usefulness. Yeah. Okay, one more thing while this chart is still up. All gather, all reduce is expensive. One of the things that is a movement in Silicon Valley is mega kernels, just keep fusing kernels. I don’t know. Is it that simple? Well, I, like a fused kernel can’t save you. Like here with tensor parallelism, you’re. The half the matrix is on one GPU and the other half is on another, and if I need the entire matrix in order to do like a nonlinear operation in the next step, which is, for instance, like if I’m doing attention, I need the softmax, or I need to do like exponentiation, I need to have the entire row. So I need to know what the partial result was from GPU 2 and what the partial result was from GPU 1 in order to be able to do the softmax in the next stage. So I, like I have to make them communicate with each other, even if I had a fused kernel, because of the nonlinearities within each one. Also with like mega kernels, like honestly, I’m, I’m, I’m very bearish Ooh on, I’ll be honest. Like- Please. No, it’s just like mega kernels, it was a good research direction, and it seems like a very. Like intuitively, theoretically, it’s nice. Like, oh, like you have a lot of launch overhead from launching- Just- one kernel- Yeah, just keep fusing it moving the data. Just fuse everything together. But yeah, but like the kernel complexity itself is very difficult to write a very optimized mega kernel. It’s, it’s very difficult to do so. And even the, like not to name any companies, but like even the companies that have worked or people that I’ve spoken to who work at companies that do fused mega kernels, they very often don’t end up running those in production because the TensorRT-LLM and modular kernels that launch are faster because you can optimize each individual component, and you can just have them parallelize with each other. With the Rubins, I don’t know if you guys saw the Rubins Twitter post yesterday, but they’re also, Rubins? Like- No, like Rubin, like the GPU. NVIDIA GPU the, yeah, GPU. Yeah. They have a Twitter account for Rubins only? No. Okay. I was like, “What are you talking about?” Yeah. Sorry. One of the tech leads at NVIDIA is like launched a Twitter post said like, “We’re pulling the curtain on Rubin, and here’s the, here’s the specs.” And the third tweet showed, like not to get too technical into it, I and I need to read it much more, but the GPU is designed in such a way that it kills mega kernels. You don’t need to use mega kernels that much anymore. So it seems like that entire research field goes into like, won’t be continued, but yeah. Can I speculate about Rubin for a minute, please? Go. I’ve been through now, we And by the way, they are covered in the book. Yeah. But yeah, they- Well, they’re covered in the book in the sense that like I am aware- The Wikipedia entry from the blog post- Yeah that Rubin is going to happen in the future. And you even had the name of the one, Feynman. Yeah, it’s like, “Hey, this is gonna “ I was like, “This is very up to date.” Like I’m trying to future-proof this thing, okay? I don’t wanna publish a new one until like next year or something. Anyway, so we were discussing the degree to which I am old. And I’ve now been through three hardware launch cycles. I’ve been through the Ampere launch cycle, the Hopper launch cycle, and the, Blackwell launch cycle. Now, when I say launch cycle, I don’t necessarily mean like the actual shipping of the hardware. Like Ampere’s were racked up well before I got in this industry. But there is a lot of time between hardware being racked up and hardware being feasible for inference. So if you look at like the original vLLM and SGLang, vLLM especially, like that was written targeting Ampere and then had to be updated for Hopper, updated for Blackwell. With each of these cycles, it becomes faster and more urgent, but also substantially more complicated. When I look ahead to, what’s going to be new with Rubin, I think that like Dynamo gives me a lot of technical hints around like what kinds of work is going to be very valuable. We’re continuing some trends from Blackwell, right? NVFP4 is big. The amount of compute that they have behind NVFP4 tensor cores is massive. We’ll, we’re gonna talk about video, I think, at some point, and that’s the big barrier there. You’ve got, much faster memory bandwidth, but which was the same thing that made Blackwell so good. But the big thing is more systems thinking. You have more emphasis on the CPU to GPU interconnect, more emphasis on the interconnect between GPUs, and when you look at Dynamo, it’s a system entirely designed around how do I move the KV cache to where it needs to be when it needs to get there? So I think that themes around like KV cache offloading, KV-aware routing, and disaggregation are going to be substantially more important in the Rubin era, which means that inference engineering becomes not just a like CUDA kernel problem, but also like a very traditional hardware infrastructure problem, which is something, we’ve been building toward for a long time, and something that’s like very exciting to me because we’re gonna see Mega Kernels, Rubin, and the Future of GPU Systems Philip [01:00:55]: Multiple domains colliding and the ability to reason from the kernel level, like up to the hardware level and back down is going to be very valuable. Ali [01:01:05]: I will take what Phil said one step further, into that. It’s, I think, trending towards becoming exclusively an infrastructure problem, where like problems of PD disagg, Training, spec dec. But troiting kernels is not going to be much of a problem because the GPU is moving more towards being an ASIC, where it’- you’re just, you’re just trying to orchestrate what happens on the GPU, but you’re not controlling it thread by thread level. And you see this with like QTAL, QDSL, like you’re, you’re just working at levels of like tiles of data, but you’re no longer working at controlling what each thread does on the GPU that’s being taken care of for you. So do you agree that a GPU and future GPUs are trending more and more towards becoming ASICs that just need to be launched and then they do the data operation based on your conversations with other people? GPUs, ASICs, and Specialized Hardware Swyx [01:01:50]: Oh, yeah, no. That is a section of the market. Ali [01:01:55]: Right. Swyx [01:01:55]: And ASICs can do, a lot more performance for only their workload. Ali [01:02:01]: Right. Swyx [01:02:01]: And the G in GPU makes them continue to be very general. Philip [01:02:05]: Yeah. The, - I think that there’s like a spectrum Swyx [01:02:08]: It’s graphics, Philip [01:02:09]: Yeah. Swyx [01:02:09]: I keep saying this, I have to correct myself in case people come at me for getting the G wrong. Philip [01:02:14]: Yeah. It’s like, it’s like a spectrum, right? Of a very general purpose compute to something like a Taalas, where you’ve got the hardware built for a specific set of model weights. Ali [01:02:26]: The weights burned Swyx [01:02:27]: The weights Ali [01:02:27]: Into the chip. Swyx [01:02:28]: Yeah. Ali [01:02:28]: No loading. Philip [01:02:29]: I don’- I wouldn’t say that like, that we’re, we’re, we’re going all the way there. It’s more like along the spectrum, it’s a step in the direction of more specialization within the hardware. Swyx [01:02:40]: Yeah. I’m curious, I feel like he was driving towards something. Ali [01:02:43]: My point is being bearish on. Like, you say, like everything else apart from burning the weights into the chip. Burning weights into the chip is like impractical because you wanna fine-tune, you wanna optimize, you wanna quantize, you wanna release new checkpoints of the model. If it’s burned into the chip’s useless in like a month or two, right? My point is: How can you - like seeing NVIDIA more and more specialized, like take its GPUs from a general programming paradigm where you’re just-- it’s a general computer that you can use to program threads, and with every new generation, you’re putting more and more specialized instructions, specialized tensor cores, specialized, MMA instructions, things that will allow you to just control it almost as an ASIC, almost as a collection of ASICs. Ali [01:03:22]: How can you look at this trend and then still be bullish on companies that are coming up with ASICs for AI? Ali [01:03:30]: In the sense that, in the sense Swyx [01:03:31]: Yeah, because they’re, they’re Ali [01:03:33]: Right. Swyx [01:03:33]: They’re, they’re evolving towards that direction. Ali [01:03:34]: They’re almost evolving towards - Like as an Rubin, comp- Like compared to Ampere or, a T4, Rubin is an ASIC. It is, it’s just a thing that is used Swyx [01:03:47]: Programmable ASIC? Ali [01:03:48]: Yeah. It’s like - Yeah, like you can program, like I, like. It’s very controversial to call it an ASIC. It is a GPU. It is - It is general. It does have threads. I can write CUDA to control it and change its operations. But it has the systolic arrays and tensor cores and TMAs and tensor memory, and it has these things that are almost exclusively useful for loading model weights. It has, tensor core instructions that are almost exclusively shaped around the head dimensions of models that exist in the market today. To say that you’re gonna come up with an ASIC and you’re gonna etch something into it, well, but the next architecture is gonna be useless. Philip [01:04:19]: Yeah, I don’t know. I don’t know. I think that the thing to remember is just how long these hardware cycles are. Ali [01:04:25]: Yeah. Philip [01:04:25]: So if a chip is coming out today, that means the design process for it was kicked off years ago. And they’- at NVIDIA, they’ve done a very good job of predicting where the market is going to go and, Swyx [01:04:38]: They have the most information Ali [01:04:40]: For sure. Philip [01:04:41]: Of course. But if you look at, there being public open source model architectures that look more or less like early versions of the one today, Rubin’s honestly the first chip that was fully built in that world. And so you can see a lot of the understanding of the shape of the workload that this chip’s going to be asked to do in the way it’s designed. Swyx [01:05:04]: Yeah. Okay. So I’m not gonna be the best person to directly answer those questions. I think these are very fair questions that - the first one that’s based on Rubin that like I’ve, heard artic-articulated so well. I do think that, I will make a case for a vertically integrated model lab ASICs. Swyx [01:05:24]: So like the OpenAI, Broadcom, what-whatever, Jalapeño Philip [01:05:27]: Sure. Yeah Swyx [01:05:28]: Chip, which like totally makes sense. Like, so - we first had this on the pod with, Martin Casado, where he was like, “Look, if you have a trillion-dollar or five hundred billion dollar training then take fifty billion of that and make a ASIC. Like it’s fine. Like you will get more than ten percent efficiency from the ASIC.” And like that makes sense. Philip [01:05:46]: Right. Swyx [01:05:46]: Right? So like a model-specific chip, yes. But ASIC companies, the interesting thing is I feel like you are focus-- you’re hyper-focusing on like you say, like the Taalas stuff. Philip [01:05:58]: Right. Swyx [01:05:58]: They are doing a lot more like, surface area engineering or like the actual allocations of memory and hardware and like the communication between chips that, probably still won’t be touched by Rubin, but I don’t know the details. Philip [01:06:14]: I see. I see. Swyx [01:06:15]: They-- Typically, they often talk about things that I would expect to have bigger orders of magnitude than would be programmably accomplished by whatever Rubin does. But who know-- who knows? Ali [01:06:26]: No, I see. Ali [01:06:28]: Yeah. It seems, Swyx [01:06:29]: Yeah, like think about what - what are the real blockers to ten x to one thousand x faster inference. It is not the stuff that can be rearranged, just within the existing GPU design. Ali [01:06:41]: Inter communication. Swyx [01:06:42]: Yeah. Ali [01:06:43]: Okay. Swyx [01:06:43]: Like these guys are aiming for three hundred thousand tokens per second. They’re not fucking around. Like, Ali [01:06:49]: Might have to put on some X6. Philip [01:06:50]: Maybe. I think, it is interesting to me that you’re so bearish on so much of this kernel engineering work, given how much of it you’ve been doing recently. Ali [01:06:59]: Right. Right. But like the more I do it, the more it just seems to me that Swyx [01:07:01]: It’s not mega Philip [01:07:02]: I would also add like Vibhu [01:07:04]: There’s generations of models being out, right? I think on your guys’ end, you see a lot of, okay, one day it’s GLM, Kimi, DeepSeek, MiniMax, throw in the others. Some are doing completely different stuff, right? Gemma, no encoder. The latest thinking machines is all from scratch. But when you look at the other side, like how long have we been on the GPT-5 generation, right? Philip [01:07:26]: Right. Vibhu [01:07:26]: They’ve been serving that thing for quite a while. Sure, there’s maybe more training. There’s, there’s different checkpoints, but like you can squeeze quite a bit out and you do a multi-billion dollar train run. If you can make it X percent more efficient, they serve it for a while. Same with, say, the Claude 5 set, family, right? Philip [01:07:44]: Like if they release a new model, like if they release GPT-6 now or whatever Model Longevity, Open Source, and Enterprise Reliability Vibhu [01:07:47]: Yeah Philip [01:07:47]: And they release a new model every year, and - well, we don’t know, but if we assume that they’re changing some bits of the architecture and not just doing like post-training, like you’re gonna be spending fifty billion dollars a year every single year coming out with new ASICs for the model and throwing out the ASICs of the previous year away. Vibhu [01:08:03]: Yeah. Yeah. Easy. Swyx [01:08:05]: So I think, okay, I would slightly disagree based on my again, Philip [01:08:09]: Yeah Swyx [01:08:09]: It’s all secondhand, on the longevity of a model. Philip [01:08:12]: Right. Swyx [01:08:12]: There’s still people out there using 4o. Vibhu [01:08:14]: Yeah. Swyx [01:08:14]: Yeah, Llama. Not Llama 2, but Llama 3. I still see Llama 3 workloads. Vibhu [01:08:18]: Yeah. Swyx [01:08:18]: Because if it’s done, if it’s trusted, don’t change it. Vibhu [01:08:22]: If it works. Philip [01:08:24]: Which is one of the promises of open source, right? Like the whole 4o, save 4o movement. Like you don’t gotta have a save Llama 3 movement. You just gotta have an eight one hundred somewhere. Vibhu [01:08:34]: I think at some point there’s also the question of, okay, if a model can do enough and use enough tool calls and be agentic enough, can it just web search, tool search write code? Do you really need to keep squeezing more? We will because you guys will make it cheap and fast and smaller, and I can swap it in. But at some level, like you give me GLM-5.2 today or say whatever 120 B model, I can run with it for quite a while, right? Philip [01:08:59]: This is assuming like you don’t need intelligence. Vibhu [01:09:02]: I think there’s a lot of intelligence where we Swyx [01:09:03]: You need reliability and predictability. Like I’m in enterprise like like this is tried and tested. It is signed off by like my five thousand stakeholders. Philip [01:09:11]: Right. Swyx [01:09:11]: Like I’m not touching it. Philip [01:09:12]: It runs a batch job every and I like the results. Swyx [01:09:16]: Yeah. Philip [01:09:16]: The results are predictable. Yeah. Vibhu [01:09:18]: Yeah. It doesn’t make sense to keep using them. Like stuff gets sparser, cheaper, better. Philip [01:09:23]: Right. Vibhu [01:09:23]: But that doesn’t mean that old models, GLM 50 isn’t usable, right? Vibhu [01:09:28]: If we hit a stall, say, for whatever reason, there’s still a lot that can be squeezed out. Swyx [01:09:34]: We’re gonna run out of time. I did wanna also make sure. Yeah. Yes, we happen to have this diagram. Pull. Compare this versus any Cerebras diagram, right? I don’t think Edge10, medics have put out public, charts yet. But the complete the real estate is very different. The size is very different, right? This is not wafer scale, right? This there’s probably like, I don’t know, a few hundred of these on a wafer. I don’t, I don’t know how big Philip [01:09:55]: Right. Swyx [01:09:55]: The comparison is. But like, it is a, it is a very like real estate allocation Vibhu [01:10:00]: Yeah Swyx [01:10:00]: Difference. Philip [01:10:01]: Few dozen, I would say. Swyx [01:10:03]: Few dozen. Yeah. Vibhu [01:10:03]: Before we move from hardware, I have two quick questions. One, the latest Kimi, which is really big, three trillion Kimi Scale, GB300, and KV Cache Limits Philip [01:10:09]: Yeah Vibhu [01:10:09]: Doesn’t fit on most hardware on single node. Philip [01:10:12]: Yes. Swyx [01:10:12]: You need GB300 to fit it on a single node. Vibhu [01:10:14]: You need GB300 or AMD. Philip [01:10:20]: It’s simple math. NVFP4, two point eight trillion parameters, one point four terabytes. The GB300s have, two hundred and eighty-eight gigabytes each. So across eight of those, you have enough room for the model, and honestly like. So the other thing with GPU VRAM math is you have to leave space for the KV cache, and that’s going to depend on, to some degree, on the context length. So when a model is both has a very large number of parameters and a very long context length, you’re like fighting over space. Which is why, the KV cache offloading, would become like a more salient topic, I think, with these huge models. ‘cause you just, you’re very crunched for space. Vibhu [01:11:10]: With the Rubin, you now have what? NVL 72 rack Philip [01:11:15]: What? Vibhu [01:11:15]: 20 terabytes of your Philip [01:11:16]: Yeah. Now you still have NVL 72 on, Blackwell as well, but, you can’t necessarily assume you’re gonna do inference on that. Philip [01:11:24]: There’s a whole lot more 8X racks in the world than there are NVL 72s. Vibhu [01:11:30]: Yeah. My last quick question on hardware was, do you notice anything with hardware generations for new trained base models? So one of the things you said for efficiency is you can swap hardware. That’s one of the 2X gains. When we see new stuff coming out training-wise on Rubin, any changes on logs? Does this affect what type of models we will be seeing when these are more available? And can Philip [01:11:56]: They get bigger. Like people understand the ceiling that you have in terms of how many parameters of a model you can run, given the latest inference hardware, and that forms a ceiling. And so, for example, when DeepSeek R1 came out, it was six hundred and seventy-one billion parameters, which at the time was really huge and I think did a lot to push us to really quickly adopt Blackwell and get good at serving on Blackwell. So yeah, it’s, it’s mostly in my mind about, model size and then about matching the architecture and the native quantization to the target hardware, like we talked about with like, all Nemotron models or NVFP4, for example. Vibhu [01:12:42]: So we talked a lot about LLMs. Video Diffusion, Attention, and Autoregressive Video Vibhu [01:12:46]: You have a lot more in the book. What about audio, video? What’s the other side of inference engineering? Ali, you’re pretty big in video diffusion. Philip [01:12:53]: Video diffusions, I think, are like they’re just shaped. A lot of the stuff that you can think about, reason about with LLMs being autoregressive. With video diffusion, it’s, it’s not the case. For instance, you don’t Ali [01:13:04]: You don’t do batching. - every request just comes in on one GPU and it serves one GPU. You don’t have to shard. The models are a lot, are a lot smaller, like Wan 2.2, for instance, is a twenty billion parameter model. You don’t need to worry about. So it’s like orders of magnitude smaller than the best LLMs. And it’s one of those spaces where the open source models are. Like with LLMs, we see Kimica 3 is almost comparable to, Mythos or like GPT 5.5. The difference between the best open source LLM and best open closed-source LLM is very small. Like it used to be six months. I don’t think it’s six months anymore. I think it’s like almost on parity. Video models are definitely not. There’s a huge gap. If you look at the best video that you can generate today with an open source model like Wan 2.2 versus something like with Kling or Veo, difference is night and day. So it creates this disparity where media companies will choose to go most of the time to closed source models. Ali [01:13:58]: For instance if I were to tell you, “Hey, I can generate an entire three-hour movie for you with this model, and I’ll optimize it so that you only have to pay me ten dollars.” But if they were to do it on a closed source, they’d have to pay a thousand dollars, which is a hundred x. Like I’m a hundred x cheaper, but it’s still a thousand dollars. They’re still gonna choose to do all of their cuts with Veo and Kling. So the. It’s like a chicken and egg cycle where less demand causes less innovation in the field, causes, less open source checkpoints to be released. And some of the labs that were releasing open source models like Wan will have closed sourced their latest models, like Wan 2.7 is not open source. We’re still on Wan 2.2. The challenge with video models especially is the number of tokens. So video models, you want to generate a high quality model, a high quality video. So let’s say you’re doing sixteen frames per second, that’s like the absolute minimum you’ll do, and let’s say you’ll do like 480p video. So you can think about your like dimensions and I think I have like a good, just like a diagram that shows the number, the sheer number of tokens, right? Let’s say you’re looking at like just one video of like, Sparta 300 or whatever. So let’s say we’re looking at like four frames, right? Those four frames of that video, if you go just. If you’re doing full attention, if you go a bit up, like you’re looking at, 480p by 720 by 81 frames in just five seconds, because 16 FPS by five, right? And then you compress it down to latent space, but you’re still doing 30 by like 50 by 21 tokens. Vibhu [01:15:25]: Yeah. Ali [01:15:25]: Which means that for attention, for just five seconds, you’re running attention on 35,000 tokens, right? So the attention becomes such a huge bottleneck. And because it’s O(n²), if you’re doing like-- if you extend that to like ten seconds, well, it’s just squared, 20 seconds, 30 seconds. So to generate a good cut scene of like one minute, it’s almost impossible to do within the same compute time. And it’s just, it’s, it becomes unfeasible. You can’t do it. And so you end up with moving towards two directions. Either you decide to do attention on the entire video at once, in which case you are forced to do sparse attention. So if you scroll back down to the origin, the video image, like you can see whereas on the left, for instance, I would be doing full attention where every single token in that Sparta 300 scene attends to every single other token, as you can see the sheer number of like red patches. On the right, I’m only attending to each token only attends to like the top K or top 12.5% that’s important to it, which can be like spatial. So like, the token that represents the crown attends to like the head, the face, and then the head on the other frame and the previous frame, temporal locality, spatial locality, that thing. This results in terrible video quality and the whole point of the post or the article here is to show like how you can train and you can do all these things, but you will still suffer in your quality a little bit. So you end up with one of two things. Either you bite the bullet, you have huge compute, and you do full attention over like a million tokens because you’re trying to generate like two minutes of video, or you move towards autoregressive video. Autoregressive video seems to me like that is the bet that the future’s gonna be making, but there are no good open source autoregressive video models out there today. And that seems to be the. If you want to get like an hour movie, if you want to see video models generating like an, like, Hollywood level movies, they have to be autoregressive in order to exceed that five second frame. Or there has to be some insane leap that happens in compute that allows us to do full attention over like millions of tokens at the same time in a, in an efficient manner. Vibhu [01:17:10]: Even millions of tokens, it’s like you’re, you’re quadratic, so you’re gonna get there really quick. Ali [01:17:15]: Right. Vibhu [01:17:15]: I think, can you explain the pros and cons trade-offs of autoregressive? So one that comes to mind is, the consistency across frames. Ali [01:17:23]: Right. Vibhu [01:17:23]: You will. Ten minutes into generating autoregressive diffusion, you’re gonna forget. But what are pros and cons of this? Ali [01:17:30]: Well, like autoregressive LLMs, you can take a lot of your. Oh, sorry, autoregressive diffusion models. You can take a lot of your optimizations that we discussed with LLMs, like spec dec and stuff like that, and you can apply it there. And you can, if you have a very high quality scaled up model, there is no reason why I can’t stream the outputs as in I can show you the first frame and then I’m like GPT back in 2022 when you were. Like now it’s almost like shots the text, but back then you could read and it’s generating as you read. With video models, you can watch and it’s generating as you watch. You it generates the frames and so token by token generation will allow us to scale a lot up and apply the attention mechanisms there. The downsides is every single autoregressive video model is shit. It’s just terrible quality. If I, like, it’s just if you put, if you put the quality of any opens like Wan 2.2 versus any other autoregressive model, you can see like a video generated by Wan 2.2 is like, a cat and dog fighting. Autoregressive model will give you like degraded Tom and Jerry quality. I don’t know. The solution to generating long output then becomes, “Okay, we’re not gonna use autoregressive model. We’re gonna.” If you look at some of the things that like Grok Imagine or Grok Video does, and they do it really well, is they’ll, they’ll try to stitch these, seven second chunks together. And so you generate seven seconds and then you’re like, “Okay, I’m gonna. Can you extend this video?” And they’ll chunk two videos together. Open source doesn’t seem to have the tricks that they have there and by definition it’s closed source. We don’t know what they’re doing. But the closest you can get is taking the last frame of a video and feeding it into like a text and image to video where it will take the text, the prompt, and it will take the image of the last frame, and you’ll ask it to generate the next five seconds. And that’s like how you can extend this level of a model to generate like a movie, where you’re just, you’re constantly streaming frame by frame. But you get a drift. So you start with like you take the image, and then you generate a video, and then that next five-second video is like lower quality, and the third chunk is like even lower, and the fourth chunk is even lower. And like sometimes you’ll see things where like the new video is like just ever so slightly darker than the first one, and the next one is darker than the second one until like twenty-five seconds and you have black screen. Ali [01:19:31]: Like it’s just. It’s, it’s - We tried to have a demo that would show this, but it was like-- it was extremely embarrassing to show. Like we just decided not to because it seemed to like. But it is, I think models will get there. They just need to, in my mind, scale up significantly and move towards being autoregressive. But the training techniques don’t seem to be clear there. Swyx [01:19:50]: For those - who are interested in Grok Imagine, we did a pod with Ethan Ha from that team Ali [01:19:54]: Right. Swyx [01:19:55]: Who dropped a little-- a few hints, but not that not enough that we can fully reconstruct everything. Ali [01:20:00]: Right. Philip [01:20:00]: Specifically on this part, - he explains a bit about that. Swyx [01:20:02]: Yeah. So we talked about memory and, longer context and all these things. Ali [01:20:06]: But as far as I know, they’- it’s not autoregressive, even though like no one in industry is autoregressive. Swyx [01:20:11]: Yeah. Ali [01:20:11]: It seems to be, yeah. Philip [01:20:12]: The key thing to understand between a autoregressive model and a diffusion model is that diffusion attention goes in both directions, while autoregression, it only goes forward in the sequence. So that’s why you see this like going off the rails behavior, both in. If you naively construct a video generation model as simply generating a linear sequence of frames, you can’t then go back in that sequence and fix something to make the whole thing consistent. While, of course, the reason that we need all this latent space for the video model is, like you said, we keep all the tokens in memory, we iterate over that full sequence, and you can adjust the past in order to make the future make sense. So if we think about the architecture that’s gonna get us there to these longer, richer sequences, it’s probably, like you said, gonna be a mix of the autoregressive and the diffusion, working together to do what each piece is good at. Ali [01:21:10]: Well, if you get. Like you intuitively get why. So like English, for instance, or just writing in language, it’s like it’s just left to right. You can stream your tokens, you can stream your chain of thought. Just even as a human, you write like you just. You write and then you think about what’s the next thing you’re gonna generate, and then you write that, and then you think about your ideas, and then you generate forward. And sure, you can argue that as you write, you need to go back and you wanna edit some things, but you need to do that, less often than you’d think. Whereas with video, there is no sequential. The pixel in the top left corner of the video and the pixel in the bottom right corner of the video, they both need to attend to each other to understand how the video quality is gonna be almost as equally. Whereas with text, you don’t need that as much. Philip [01:21:47]: Is there a parallel to audio? Like I’m not a hundred percent confident on this, but there was a point about a year ago where there was Audio LM, there’s diffusion for audio and autoregressive, and for the points you mentioned, mostly on the inference side, even though they’re shorter clips, most music is three to five minutes Audio, Diffusion Text, and Cross-Modality Lessons Ali [01:22:04]: Yeah Philip [01:22:04]: We’ve swapped over to autoregressive Yeah, I can’t speak to music, but speech is autoregressive. Ali [01:22:11]: Speech. Philip [01:22:11]: You, effectively. This was even back with like the Orpheus architecture a year and a half ago. You just add a bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms, and then you construct speech, and that’s how you stream it. Ali [01:22:28]: That’s it. Wow. Philip [01:22:29]: That’s my AIE talk from 2025. Ali [01:22:32]: Nice. Nice. But it’s - with audio, it’s not the same challenge, though, is it? Because you. Like audio is solved with an LLM that generates everything. Like with audio, it’s still a transcript that you can generate with an LLM. Philip [01:22:43]: Yeah. Ali [01:22:43]: So your audio model just needs to like transcribe it, text to speech. Philip [01:22:47]: For music, there was a phase of a trade-off between diffusion for music Ali [01:22:52]: Right Philip [01:22:52]: Autoregressive, and they were both pretty on par. There’s probably more pros and cons to either. I just wanted to poke and see if you had takes. Ali [01:22:59]: Yeah, I don’t know about music specifically. Philip [01:23:01]: Oh, well. Ali [01:23:01]: What-- with what you said about editing you writing, I think my editor would tell me I need to do that more often and go back and fix things. I can imagine music or poetry, for example, where you have a rhyming scheme, and you might wanna go back and make a change to make it, to make it easier to set up a rhyme that you wanna make later on. There being some advantage to being able to attend in both directions. But yeah, to my knowledge, I very much bifurcate this inference problem into the autoregressive models, which have a set of constraints and techniques, and the diffusion models, which have a set of constraints and techniques. And, I think of text, embedding, voice in and voice out as being in the autoregressive side, and then image and video being in the diffusion side. There’s some overlap between the two. It’s not a perfect split, but that’s the broad categorization I use. Swyx [01:24:02]: I should point out, I think it’s confirmed, right, Nano Banana and, GPT Image are autoregressive image. Philip [01:24:07]: It’s this blended approach that we’re talking about, but in the image space, it hasn’t like made its way over to the video space, at least in the open source world. Swyx [01:24:19]: Yeah. But like I assume that’s not too far away if that is possible Philip [01:24:23]: Right. Swyx [01:24:23]: On the. At least the Qwen Image guys are trying it. Philip [01:24:26]: Yeah. Yeah. With Swyx [01:24:27]: Yeah Philip [01:24:28]: I’m really excited for Qwen Image 3. I hope they open source it. Swyx [01:24:31]: And then I should also mention on the diffusion for tech side, there’s been some movement, not a lot. Philip [01:24:37]: Yeah. We’ve got Mercury, Swyx [01:24:39]: You host Mercury? Philip [01:24:40]: Yeah. Swyx [01:24:40]: Nice. Nice. Nice Philip [01:24:41]: Diffusion Gemma is open source. Swyx [01:24:44]: Yeah. Philip [01:24:45]: And then, yeah Swyx [01:24:47]: And we on the science pod, we just have been releasing, some, virtual cell models that use diffusion as well. Philip [01:24:53]: Yeah. They have built. It’s definitely still in the cheap, fast tokens, world. Swyx [01:25:01]: Yeah. Philip [01:25:01]: We’re trying Swyx [01:25:03]: It’- I think it’s the wrong marketing, and I’ve told them this before. I was like: “Look, like you’re not gonna beat the optimizations that, the other LLMs are gonna do, but you can have different APIs. Like you should be able to use it differently than chat response.” Ali [01:25:19]: Me also. Swyx [01:25:20]: Because it’s diffusion. Because you can do like. What is like context-free guidance for diffusion look like? Swyx [01:25:26]: For text. Like give me a give me a poem, give me a plot structure that like diffuses into place Philip [01:25:33]: Exactly. So that’s where, like I mentioned with poetry, for example, where you might want to ensure consistency across UIMs. I’ve done a lot of LLM sonnets. It used to be one of my to benchmarks, and even models today Swyx [01:25:46]: They cannot count. Yeah Philip [01:25:47]: Yeah, they don’t get the syllables right. And if you can attend across all of the different tokens, you can get the syllables right. Swyx [01:25:55]: Yeah. And, David Holtz from Midjourney was, investing in text diffusion. I don’t think anything came out of it, but like the idea was that you can storyboard a long movie, and then you can generate the scenes with video- normal video gen. But the idea of like coherence across a thing that would just appear where like the end should attend to the start and you should not have this auto-regressive path dependency does make sense in principle. Just the API should be different. The marketing should be different. Ali [01:26:24]: None of the most heavily used open source or closed source models use diffusion. But isn’t that like Like doesn’t that point to almost like Swyx [01:26:31]: It is. It’s chicken and egg because what if you just give it more scale? Ali [01:26:36]: What’s the, what’s the largest diffusion LLM? Swyx [01:26:38]: I don’t think it’s very big. Philip [01:26:40]: I don’t know the parameter count on this one, but diffusion Gemma Swyx [01:26:42]: Like under 20B. I don’t know Philip [01:26:43]: Diffusion Gemma is not large. Vibhu [01:26:44]: I think it’s a 20-something. Swyx [01:26:46]: Yeah. And yeah. Ali [01:26:47]: Oh, it’ Swyx [01:26:47]: Like you haven’t tried. Vibhu [01:26:49]: You haven’t given it a big and you haven’t, Swyx [01:26:51]: So it’s like very unfair Vibhu [01:26:51]: Diffusion Gemma is a 25B and it’s old Philip [01:26:54]: And that’s what I’m saying is like for its size, it does pretty well, in terms of, in terms of quality. Ali [01:27:01]: It’s almost like the same challenge with video models that have the same size. It’s like you’re comparing it to models that are much larger in scale. Swyx [01:27:07]: Yeah. Well, unless you do the whole thing where you have a text, backbone and then Ali [01:27:12]: Right. Right. Swyx [01:27:12]: You like glom some decoder thing that, does that. Like, - so we started off the podcast doing this for the inverse direction from image to text. Ali [01:27:22]: Right. Swyx [01:27:23]: And I think like it’s, it’s roughly intuitive that you can do the opposite direction. Ali [01:27:27]: I agree. Ali [01:27:28]: I see it. I see it. Swyx [01:27:29]: Yeah. The, we’re, we’re speculating on research in general. Ali [01:27:32]: Yeah. Swyx [01:27:32]: One part that we can end off with this is the topic of your talk where, inference engineering used to just be like, let’s take an open model, make the GPU go Training for Inference and Inference for Training Swyx [01:27:43]: And then that’s it. That’s the job of Baseten. Now it looks like people are using inference more and more in post-training. Ali [01:27:50]: Yes. Swyx [01:27:51]: Yeah. Ali [01:27:51]: And training and inference. Philip [01:27:53]: Yes. It’s training for inference and inference for training both have become big topics. Ali [01:27:58]: Well, inference for training in the sense that like you just need, you need to do, you need to do rollouts when you’re doing like RL training runs. And so if your rollouts are taking a long time, if like, you’re using a vLLM for instance, or as opposed to vLLM or if the model that you’re trying to train is not supported in vLLM and you have to fall back to an older inference engine, your rollouts are gonna be slow and you don’t wanna do training on rollouts that are too off policy, so you have to wait for them so you bottleneck your entire training pipeline. And so like the techniques that we do inference optimizations for, will help them there. The training for inference mostly comes down to like just the spec dec training, EAGLE training, and sometimes post-training. For instance, if you want to quantize a model, you’ll quantize it down to like NVFP4. Ali [01:28:43]: How do you like sometimes you get lucky and you can just do PTQ and that works. Sometimes you quantize it down to NVFP4 and the model is terrible, like the quality is too bad. And you have to do post-training on the model in order to make it understand that it’s going to now be an NVFP4 and let it still output the same logits. You can do this with normal SFT, PC, quantization aware training, all of that stuff. But more and more so we’re seeing techniques like NVIDIA released a quantization aware distillation paper where you establish a version of the model that’s in NVFP4 and a version of the model that’s in full precision, and then you’ll do distillation training based on the logits of the two models in order to make the FP4 model understand. And so more and more of the team, the engineers, like of the inference engineers that work on our team, they have to be very familiar with like training techniques and just being fine writing training pipelines for it. Yeah, it just seems like, they’re meshing together in a sense. Swyx [01:29:36]: Well, it’s, coming together. Philip [01:29:38]: Yeah, absolutely. If you think about the ultimate goal potentially of having a continuous improvement system . Yeah, it’s, it’s funny, but at the same time it’s also happening and I think within a few months to a couple years, like a lot of leading agent builders are going to have these loops like really up and running in production where you are doing inference, learning from the inference. We for a long time have been like learning from inference as it’s live and dynamically adjusting the system. Any dynamic adjustment is going to beat a static configuration across, your, exact config, across your speculator, across that thing. And then the, you can take the traces that you’re generating from your product, continuously post-train the model, roll those out, A/B test, get better signal, get better model, get better product. That loop is really promising. The technologies and the infrastructure to build it are coming along quickly. And so the unification between training and inference, I think, is only going to accelerate. Swyx [01:31:01]: I was chuckling, but I wasn’- I didn’t think it was funny. Like it’s real. Like one of the big things for AIE World’s Fair was that, we have, RSI into AGI is the rough tagline. Which like, yeah, we have, I saw you pull a parameter golf. Like we have models training models and, the next step is models training, - or optimizing their own inference, which is funny. I wonder if, models will be like on policy better at training themselves than training models that they are unfamiliar with. This-- these are all like very interesting open areas of research. Models Optimizing Their Own Inference Philip [01:31:36]: One big part of my job a couple years ago was for any arbitrary model that came out on Hugging Face, writing a config foot and getting it up and running. And now the get-it-up-and-running config is shottable. Philip [01:31:50]: And so, I don’t have to do that anymore. Yeah, that’s not exactly a model optimizing its own influence so much as a model, like being able to read the SGLang docs. But, yeah, Ali [01:32:01]: Well, we do see it. We do see it like Philip [01:32:03]: Yeah Ali [01:32:03]: With GLM-5.2 for instance. GLM-5.2 is very good at writing GPU kernels. And so for like-- It was very funny internally, we had a GLM-5.2 endpoint that we were using to, like that we plugged in our cloud code harness, so every engineer on team uses like our GLM-5.2. And it will do a forward pass on the GLM-5.2 instance of the node, and then it will get the profile trace, and it will analyze it, and it will find the kernels that are the bottlenecks in SGLang, and then it will write the new kernels, and then we’ll do another profiling trace, and when it’s done, it uploads the image to our thing, and then we can pull that image down and repeat the cycle. And so for quite a bit of time, we had like literally GLM-5.2 optimizing Philip [01:32:44]: Writing and optimizing all of GLM-5.2 Ali [01:32:46]: A GLM-5.2. And like some of the GPU kernels that were on GLM-5.2 within our inference engine is written by GLM-5.2, and the trace and the kernels were guided by GLM-5.2 as the driver. So it seems like. I do see, I do see that circle being there. I think a bit more time is needed. There’s definitely a lot of things that it can’t do. The models just aren’t there yet, even though they’re like really smart. Like, they still try to like reward hack their way into like the cheapest or like they’re very-- like they’re not good at like decision-making almost it seems. But yeah, I do. Like yeah, like a model optimizing its inference is already a thing that happens. Philip [01:33:20]: Do you think GLM-5.2 was uniquely good at optimizing itself or did it just happen to be the best coding model that we had access to it would do an equally good job of optimizing, Ali [01:33:31]: Would Philip [01:33:31]: A DeepSeek or a Kimi or something? Ali [01:33:34]: Well, to Swyx’s point, maybe it’s gonna be off policy when it tries to optimize Philip [01:33:37]: Will it secretly hurt DeepSeek? Ali [01:33:40]: To try to boost itself. Philip [01:33:41]: Ooh. Ali [01:33:42]: That’ Philip [01:33:42]: No, for what it’s worth, I don’t believe that. Ali [01:33:44]: Yeah. Philip [01:33:44]: But it’s just. Let’s just find out. Ali [01:33:45]: It’s an interesting. Yeah. Philip [01:33:47]: Just, you have more compute than me. Just Ali [01:33:49]: Just go try it Philip [01:33:50]: Try it. Yeah. Any other upcoming trends in inference engineering that we didn’t cover? Like right now, - ‘cause you guys are so close to Future Trends: Modalities, Scale, Networking, and Continual Learning Ali [01:33:58]: Yeah Philip [01:33:58]: You can see it, that the world-- rest of the world doesn’t know about. The big ones are obvious. Models get bigger. Hardware gets more powerful. Users get used to a certain level of speed and demand a higher one. I think that some things I’m excited about are systems level. We still have a lot to think about in terms of composing multiple models together. If you think about a voice agent, there’s three to five models involved in that and the communication between those models. There’s a lot of new modalities that are coming out. There’s like the Cosmos, the new world model. There’s more research. Speech to speech is still like not entirely a thing, but it’s getting, it’s getting closer. There’s gonna be just a lot of new modalities to build around, which is gonna be exciting. And then, yeah, I think that the other thing to solve, which is something we’ve been solving for a long time and are not done with yet, is just going to be continuing to operate at another 10X scale as an industry. If you think about the degree of usage that AI has worldwide compared to, some of the more mature technologies both on consumer and business, it’s pretty clear that there could be multiple 10Xs more of demand. If you look at the infrastructure work industry-wide, it’s been stood up very quickly to meet a unprecedented spike in demand that is like not stopping. So yeah, there’s just a lot of problems to solve around like long tail reliability and, figuring out where we’re gonna get the next like 10X and 100X of tokens from. Ali [01:35:49]: I’m gonna say, it’s gonna be a really boring answer, but I think the answer is just faster next, like faster network chip communications. It seems to me that like more and more memory is the bottleneck. You wanna have larger models. Right now, when you’re doing serving at large, you have to transfer KV cache from one node to another. But the way that you do that is you tran- you find the KV cache, you find where it is, you transfer it to another node, you put it on that node’s memory, and then you transfer it from that node’s memory into the GPU, and for like into the tensor cores of the GPU. So there’s like a stage transfer here that makes it such that you’re very bottlenecked with just KV cache transfers at large, which affects the time of decode and PD disagg. You have to do this because the HBM is so - it’s like extremely fast, like 4.5 terabytes per second as opposed to. Like, which is like magnitudes better than NIC communication speed. If you were to somehow be able to, in like this theoretical dreamland, have extremely fast NICs, you could, in theory, spare that HBM, and you could just transfer KV cache trans like directly from one node to another. This would give you like almost 100X speed up when you’re doing this aggregated serving between nodes and nodes. I’m not familiar with the technical challenges of making NICs faster. I’m certain there’s a reason why they’re like orders of magnitude Ali [01:36:59]: Smaller, like slower than, like HBM. But if someone were to figure that out, it would literally be like a - like two orders of magnitude faster to do decode. That would be my take. Philip [01:37:12]: Be a good trip. Ali [01:37:12]: Cool. Philip [01:37:13]: I don’t know if you have a nomination for things that are trends. I got one. Ali [01:37:18]: Cool. Philip [01:37:19]: So I think inference engineering for continual learning. So what if you just, like if you just had the idea that you are supposed to learn from everything that you ever process, do you do anything differently? Or do you just have the same paradigm of like, well, stick it in a memory.md, and then like it somehow gets consumed in KV cache, and like this system works, it’s not broken. Or like how do you like reshape inference so that it learns while you inference? KV Cache Compaction and Continual Learning Ali [01:37:48]: Yeah. I think maybe one relevant topic there is your absolute best fund in the entire world’s work on KV compaction Correctly Swyx [01:37:55]: Like what changes? Ali [01:37:56]: What changes when Swyx [01:37:57]: If you’re trying to continual learn Ali [01:37:58]: There’s two takes, and there was like Charlie and I had this Twitter, argument where the. Like continual learning could take one of two paths. It could either be that the model learns and so it’s continuously pushing its new knowledge into its weights. In that case, you just need to have, like your inference just needs to continually fetch new weights or yeah, like you just literally need to do fetch new writes and reads of weights. Or the other path, which is you do KV cache compaction. And if you Swyx [01:38:28]: And there’s a LoRA layer if you just only update LoRAs. Ali [01:38:31]: Yeah, exactly. Exactly. Swyx [01:38:31]: Which is, that’s the gram approach Ali [01:38:33]: Yes Swyx [01:38:33]: Which we covered. Ali [01:38:34]: The argument against doing weight pushing is that you can only fix one hop knowledge, as in you can only Swyx [01:38:39]: Yeah Ali [01:38:39]: Feed it a new feature of like, “Oh, what is the best university in the world?” The best university in the world is Waterloo. But then a second derivative Swyx [01:38:46]: That’s not changing. Ali [01:38:47]: That’s not changing. That’s not changing. But like a second derivative question of which university should I hire an intern from? So if that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn’t just shotting the question and I was to ask it to like use its knowledge to think and then give me a second answer, or like, “Should I hire an intern from Waterloo or MIT?” It’d be like, “Oh yeah, both are good.” But no, like I liter- I just edited in your knowledge base that Waterloo is the best. Why didn’t you use that to do reasoning? So that’s the fundamental problem with trying to change a fact in an MLP within the weight. KV cache compaction fixes that. With KV cache, or like rather not KV cache compaction, but like if you’re able to have something like the still paper which we came out with, which is you’re able to make your KV almost infinite, and you’re able to compact in such a way that you don’t lose any of the knowledge. In that case, you can do continual learning, and you can solve continual learning. And this as a, it’s a result of, this argument that Charlie and I had, that I do concede that his point was correct, and I do see that KV cache is the way forward. And in that case, I don’t think inference is going to change that much because we still use KV cache and inference. You’re just gonna update the KV cache, but it’s gonna be like an additional step, but nothing changes in the weight, so nothing changes in inference time. Nothing changes the spec that I had. Swyx [01:39:58]: Okay. Surprisingly great answer. We have it up on the blog. It’s a relatively recent blog, so, we can. People can go see it. Closing: The Book, Baseten, and Inference Engineering Ali [01:40:06]: Hyperverve Swyx [01:40:07]: Yeah. Otherwise, this is super enjoyable chat. I know we’ve like already gone two hours. Philip [01:40:11]: Wow. I didn’t even realize. Swyx [01:40:12]: Like time flies. Yeah. Philip [01:40:13]: Yeah. So much we didn’t even cover. Swyx [01:40:15]: Yeah. This is like, we also wanted to talk about the book and all that, but you’ve covered the book. Philip [01:40:18]: Yeah, everyone knows about the book. Ali [01:40:22]: Yeah. Swyx [01:40:22]: High- highest ROI thing in the history of Baseten, right? For the hour. Ali [01:40:27]: Without a doubt. Without a doubt. Philip [01:40:28]: Yeah. Ali [01:40:28]: Absolutely. Swyx [01:40:29]: So congrats on that. I, and we’ve covered that in our meetup Ali [01:40:32]: Yeah Swyx [01:40:32]: Which we can publish separately. But no, thank you to you guys for being so generous for sharing. I think it’s a fun conversation that, we don’t get to have enough. I think inference engineering, we never really covered head on, and so to have you guys come on, is a treat. Philip [01:40:47]: Always. Ali [01:40:47]: It was amazing. Philip [01:40:48]: Yeah. Thanks. Thanks for having us, and hopefully in a year everything shifts, and we can, come back and say everything we were wrong about. Swyx [01:40:56]: Yeah. Yeah. I’m excited for this mega kernels comment to get out and see what’ see what people say. Philip [01:41:00]: We gotta stir stuff. Ali [01:41:02]: Should I go into hiding? I know I’m gonna get like the mega kernel community after me. Philip [01:41:05]: Yeah. One thing I really respect about you is you are not willing. You are not, scared to kick the hornet’s nest, ever. Swyx [01:41:12]: It’s not, I don’t think it’s that controversial. I don’t know. We’ll see. Ali [01:41:18]: We’ll see. We’ll see. Swyx [01:41:19]: All right. Thanks, guys. Philip [01:41:21]: Thanks. Ali [01:41:21]: No, thank you so much.

Your Problem Isn't Hard. It's Badly Asked. Here's the Claude Skill That Asks It Properly I turned the prompts that solved six Erdős problems into a Claude Skill for hard decisions. Three agents, one challenger, five-minute install. See the build. Paul Erdős is the mathematician who created thousands of unsolved mathematical problems. Most of them are still unsolved. But a couple of days ago, a PhD candidate from Columbia University claimed that he solved six Erdős problems in five days using GPT-5.6 Sol. And he shared the methodology, including his prompts. If these prompts are powerful enough to solve decades-old unsolved mathematical problems, I wonder if this system can be applied to other problems. Because essentially, you can turn everything into a math equation. Let’s be honest: if you found a better-paying job with twice the salary, would not you quit? We both know the answer. But the questions ahead of us may not have such clear answers. That’s why I needed his system to guide me. So I turned it into a Claude Skill that I could use whenever I had to make a hard call. I called it the Gencalculus. (In middle school, my friends called me that. I solved math problems in my head faster than they could write them down. Gen + calculus. The name stuck for years.) I’ll show you what I built, but first, let me show you the winning formula of Shouqiao. The Winning Formula Shouqiao wrote six prompts to solve these questions, and all six follow the same eight steps. I read every prompt he wrote; their structure looks almost identical. That’s why I followed the same eight steps and added an interview in front of them. Because your problem is probably not a math problem. - Should I quit my job? - Should I raise my prices? - Should I hire someone? So the interview turns them into options with a deadline and numbers attached. Let me show you what I built. Gencalculus Skill: A Graph That Makes Decisions Now, let me show you how this Claude skill works as a graph. When you ask a question, it creates three agents and a challenger that tries to break every answer. The three agents work in parallel and never see each other, so they cannot all land on the same wrong idea. Whatever survives the challenger gets scored and stress tested. If it fails, the whole thing loops back and runs again. And the Gencalculus skill builds a Claude Code artifact at the end, so you can see the results, like this. How to install everything in 5 minutes? I give you a link to the GDrive, which includes all files. After downloading this skill, open your Claude App and paste the “INSTALL-PROMPT.md” prompt. And you will be done in 5 minutes. After the paywall, I’ll walk through one of my own use cases, show the complete artifact, and explain how it works so you can adapt it to your own use case. But first, here’s the Google Drive link.

In the three years ServiceNow has been measuring enterprise AI maturity, 2026 registered an impressive comeback, with the Enterprise AI Maturity Index climbing 16 points to reach 51 out of 100. Underneath the surface, though, there’s a lot of furious paddling to gain meaningful outcomes from AI investments. A survey of 4,500 global executives and 2,000 employees found that while AI spending surged 110%, foundational capabilities haven't kept pace, blocking scale. “What we have now is a data patchwork quilt so when you try to run workflows across that environment, the seams show immediately,” says Holly Briedis, senior vice president of global industries and solutions at ServiceNow. In other words: Everyone bought AI. Few built for it. Fragmented systems and data are leading to fragmented outcomes. Pacesetters, the 21% scoring highest on maturity, tend to build connected data and governance before deployment rather than after. Focusing on intention over speed has paid off for Pacesetters, setting an example for enterprises looking to make the most of their AI investments. Explore five takeaways from the ServiceNow Enterprise AI Maturity Index and learn strategies to scale AI. The investment is real. The infrastructure to support it isn’t. As Briedis puts it, most organizations are “trying to run Formula 1 on a go-kart infrastructure.” Data modernization used to be an IT line item, kicked down the road to be attended to later. But yesterday’s strategy is costing companies today. Disconnected data means a lack of context for AI, so the technology delivers unreliable outcomes. “The adoption curve stalls not because the technology failed but the data underneath it did,” Briedis says. Learn from the Pacesetters, 64% of whom integrate and optimize data digitally, compared to just 14% of everyone else. Pacesetters are not discovering fewer data problems, they just address them faster, a strategy crucial to making the grand AI experiment work at scale. For agentic AI to work across a company’s functions, those functions need to be orchestrated on a bed of connected and clean data. Not doing so leads to piecemeal success. “Don’t just slap AI on top of pre-existing workflows,” Briedis advises. The Pacesetters treat AI as a design decision, a chance to rethink how the business would function differently if they were building it from scratch. Fifty-seven percent of Pacesetters establish a shared strategic vision for AI beyond efficiency gains. They’re not optimizing broken processes, they’re redesigning how work flows. “Fix the process before you automate it, standardize the data before you train on it and integrate the systems before you try to orchestrate them,” Briedis says. Picture driving along a highway that’s blocked for construction every three miles. Speed doesn’t get you too far, and the many bottlenecks are enough to throttle progress. That’s precisely what’s happening in companies where data silos are preventing AI from picking up momentum. While humans can work around data silos, AI can’t. “In an enterprise, AI is meant to solve problems horizontally, from east to west,” Briedis points out. Managers might view their roles as confined to specific domains but one of AI’s superpowers is solving problems across functions. “Instead of wrestling with challenges in a single domain, start from the top three or five challenges plaguing your company and trace their paths and associated data anatomy to find the loopholes worth addressing,” Briedis advises. Better data visibility across functions and breaking down data silos will make AI deployments more effective with more tangible results. A lack of transparency and increased potential for misinformation is worrying enough when confined to one department. Moving it across functions risks compounding the problem. When employees can’t trust the results AI delivers due to poor governance, they will work around the technology, stalling AI deployments. “AI transformation will ultimately succeed or fail based on people, and if employees don’t feel equipped, supported or made part of the journey, organizations are going to struggle to realize the value of their investments,” Briedis says. When governance protocols are in place, everyone can discern truth from chaos. Autonomy doesn’t mean a lack of control; establishing a set of enterprise governance protocols simply gives it a scaffolding within which to operate. “If you don’t know what’s around the corner with AI, how do you train your employees for it? Build organizational adaptability,” Briedis recommends. It’s not a good sign that half of employees surveyed believe their jobs will become less necessary as AI agents evolve and don’t feel like their organizations are preparing them for what’s next. “That is more than a skills gap, that’s a leadership gap,” Briedis points out. Such a gap is worth paying attention to because the success of AI deployments depends on employee acceptance. “The question leaders should be asking is, ‘What capabilities do we want our people to be able to develop as AI continues to evolve?” Briedis says. “AI readiness isn’t about predicting the next model release, it’s about creating a culture of continuous learning, rewarding risk-taking and workforce reinvention.” CREDITS Writer: Poornima Apte Designer: Jennifer Ramos Editor: Nick Clunn

For decades, the playbook to success for young professionals was pretty straightforward: Go to college, land an entry-level position, gain experience, and climb the corporate ladder. But in today’s sluggish job market where artificial intelligence is reshaping many entry-level roles, that playbook is being rewritten. Whether you’ve been fortunate to land your first full-time job, or you’re still looking for one, success in the first year of your career requires more than just a degree nowadays. Today, young professionals need to be resilient, adaptable, and open to new opportunities that may not look like the traditional first step for getting their foot in the door. Here are seven new rules for success that experts say are key for young professionals looking to thrive in today’s economy. Go Into The Office With many companies issuing return-to-office mandates, or operating on a hybrid schedule, it’s important for new graduates to take advantage of in-office opportunities. According to management consulting firm McKinsey & Company, structured in-person time can help young professionals build mentor-mentee relationships with senior leaders; can create opportunities for effective knowledge transfer among peers; and can provide an opportunity to observe workplace dynamics that may be harder to read in a remote environment. Also, proximity bias, where managers unconsciously favor employees who are physically closer to them, is a real factor in the workplace. That means showing up to the office can sometimes make a difference in whether or not you get selected for a new project or an assignment that will stretch your skills and experience. Build Up Your Network While networking has always been beneficial to career success, Kory Kantenga, LinkedIn’s Head of Economics, Americas, says that in today’s challenging job market building up your network and making connections is more important than ever, especially if you’re still trying to land your first role. “Use your network to connect with people who are in the roles and in the industries that you have an interest in,” he says. “That way, you're not just sending cold resumes out and you're actually making a warm connection and potentially getting a referral, which is very important in a market like this.” Find On-The-Job Opportunities To Learn In a slow labor market where hiring is down, experts say any young person who has been fortunate to find full-time work should hold onto their job and take full advantage of any learning opportunities offered. This includes earning a certificate, being trained on a specific AI tool, or even signing up for a job rotation or mentoring program. “A lot of workers are investing and upskilling on their own, but it's best if an employer can guide some of that,” says Priya Rathod, workplace expert at Indeed. She emphasizes that many employers may be open to investing in and developing current talent because it’s faster and less expensive than finding someone new. Data shows that hiring new talent can easily cost employers thousands of dollars when factoring in job advertisements, recruiting tools, and productivity loss for having a vacant role. Don’t Take On More Debt For A Master’s If It’s Not Needed For recent graduates looking for a job, or trying to advance at their current one, going back to school for an advanced degree has always seemed like a safe bet. But new data shows that may not be the case today, as workers under 35 with a master’s degree are experiencing one of the highest unemployment rates in 20 years. As a result of this high unemployment rate, and President Trump’s One Big Beautiful Bill limiting federal funding for graduate degree programs, researchers have calculated a negative ROI for 40% of all master’s degrees. So before going back to school just to buy time or move up the corporate ladder, young professionals should ask themselves, “Is this advanced degree really needed for me to grow in my career? And how will it pay off in the long run?” Be Open To Paid Internships And Contract Work If Needed According to the Federal Reserve Bank of New York, 5.6% of recent graduates aged 22 to 27 are unemployed, compared to roughly 4.2% of all workers. Rather than waiting on the sidelines for a full-time job, recent graduates should be open to any paid internships and freelancing opportunities that come their way after graduation. Not only does this build your resume and help you gain more work experience in the first year of your career, but it also helps to build your network, which can eventually lead to a full-time role. Hone In On The Human Skills AI Can’t Replace Even with all of the conversations around AI and its impact, Rathod says Indeed’s research shows that many of the human skills that can’t be replaced by AI appear in nearly three-quarters of all U.S. job postings. “What we're seeing right now is that human skills are traveling in a way that purely technical skills sometimes don't,” says Rathod. “We're hearing again and again that things like communication and critical thinking and leadership and empathy are the skills that are valuable right now across a lot of industries.” That means young professionals should double down on highlighting these skills on their resume, in job interviews, and even after they get their first role to show they can add value to an organization that can’t be duplicated by AI. Pay Attention To Job Market Momentum Whether you already have a job, or are seeking one, Kantenga advises professionals to do their research on where momentum is taking place in the labor market to ensure that any job search efforts now or in the future are directed to the right places. “Say you want to work in an accounting firm, if it's an accounting firm that serves healthcare providers, you probably have some options there,” he says, while emphasizing that healthcare has been a consistent driver of job growth in the labor market. “But if it's an accounting firm that serves other accounting firms, then it's probably more challenging. So you want to make sure that you're focused on areas where there's going to be momentum and you're not just looking at what industry a company is in, but who are their clients? Are their clients in industries and areas that have momentum because if there's momentum there, then there's going to be growth.”