# Recursive Language Models w: Alex Zhang

- **Channel**: alphaXiv
- **Date**: 2025-10-31
- **Duration**: 74m 57s
- **Views**: 4,128
- **URL**: https://www.youtube.com/watch?v=_TaIZLKhfLc

## Description
AI4Science on alphaXiv
Friday October 31st 2025 · 11AM PT
Featuring Alex Zhang
Casual Talk + Open Discussion

## Transcript
Hey, welcome everyone to our latest talk Halloween special in our AI for science community. Today we're very lucky to have Alex Zang joining us um to talk talk to us about recursive language models. Um, so for those of you who don't know, AlphaKive started approximately a year ago kind of out of this shared experience that so many people are reading AI papers these days, but it just felt like there's not much community and it's very hard to discover relevant literature and be able to understand it um, efficiently. And so we started Alphacive to better be able to engage with the latest AI research and also bring community together. And um yeah, our AI for science community is a fantastic place to kind of talk about some of this stuff. So with that, I'll I'll let Alex take it from here. >> Cool. Yeah, thank you. Um yeah, so I guess I I had promised a costume, but it's not here yet. So I I will be in my my normal attire for now. Um but yes, so uh I'm Alex. uh I will be talking about uh a blog post that I came out with maybe two weeks ago uh on this idea of recursive language models. Um and so very generally I think this talk uh will just kind of gloss over like what the blog post is about. Um so some of the people here might already know or might have already read the blog post. Um, so I think the most useful uh use of this time is just like if you have any questions or anything cuz I I'm sure there's a lot of questions about applications um and implementation details and things like this uh that weren't fully shared and so I'm happy to answer those things. Uh as well as questions about like the full paper that um kind of I have planned coming out with a bunch of experiments that I didn't share uh in this blog post. So um I'll try to share as much as hopefully I can. Um, and yeah, so I think uh the structure is going to be like a relatively short talk um where you can kind of interrupt me whenever you want. Uh, and then like I'll leave a lot of room for questions or uh any kind of I don't know like criticisms or anything that you might have. Um, so yeah, I I figured maybe um it might be useful to tell a little bit about me uh and like how I kind of got here and where this idea sort of um came from. So, uh, I I guess did my undergrad at Princeton. I graduated like a year ago. Um, and like during this time I was working a lot, uh, on like RL and text related stuff. So like RL for text games and things like this. Um, I got involved in uh, with like the SweetBench team on like Sweetbench multimodal. Uh, and I I started doing a bunch of benchmarking work. Um, so I I guess like generally maybe the two things that people um know me by are either like some benchmarking paper that I've put out um or some like GPU mode like GPU programming thing that I've done. Um so these are like two loosely like directions that I I've been kind of focused on for the past year. Um, a lot of the benchmarks tend to be tasks that uh I guess language models are are particularly bad at, but maybe we want them to be good at these things for like XYZ reasons. Uh, in the case of like kernel writing or SWEBench stuff, uh, a lot of it is just about um getting models that are more capable of doing these like highly productive tasks. Uh, for video games, it's it's more about um seeing if we can have very general agents that can solve these like interactive environments. um and some of the other like GPU mode leaderboard stuff is it's kind of a similar like flavor of of research. Uh and the thing that I'm getting at here is that um one of the common strategies that I've seen in the past year uh is some form of like scaling reasoning or like scaling iterations or debugger loops or or things like this. Um and this is kind of like a motivating thing for what I wanted to do at the start of my PhD. uh which I guess I I started at MIT uh like 2 months ago. Um and kind of the idea was that a lot of these really really long like multi-turn conversations or like like these prompts um or like input contexts that are really really long um you generally have to build some kind of like really odd uh or like really task specific agent scaffold to do this. Um and this is kind of an awkward thing. I I I think like we we tend to design our agents around a specific task and then we run into these issues where like some things don't fit into the context of our language models. Uh or maybe we like observe a more nasty form of like performance degradation and things like this. And this is kind of something that I I was like looking to uh kind of figure out. Um so I also want to preface this by saying there are a bunch of other like works nowadays that talk about recursion in the context of like deep learning stuff. Um, all of these papers or like all of these ideas are really cool. Uh, some of them are a bit older, some of them like came out like very very recently. I think uh maybe TRM is the most recent one. Um, they're all very cool. Uh, they're all entirely unrelated to this idea. So, I I I should clarify that like they they have they perform recursion in very different ways than what I'm about to talk about. Um, and yeah, like so there there is almost no relationship between uh our idea and and a lot of these ideas. Um, but I do think recursion is kind of a theme that is popping up now because we're we're reaching this kind of regime where um a lot of uh kind of the like the ideas revolving around like single model calls um or like using a model to do like as a function or something. uh there are a lot of limitations in doing this and like we might want to think about how we act on like a a higher abstraction here. Um but anyway, so I I think the the way that uh this blog post kind of started and with like a timely Halloween themed uh joke is that uh kind of language model research uh for long context tasks is kind of unsatisfactory. So we've had um a few kind of uh really important papers uh like ruler and some other ones that talk about like how language models deal with long context stuff. Uh and in general I think the conclusion is like you know weirdly they're kind of bad at a lot of these things. Um but like over time like maybe GPT5 or like claude 4 are able to do or like Gemini are able to do these like million token plus problems. um where you like feed in some kind of relevant information um in embedded in some like really really noisy prompt and you ask the model to fetch this information. Uh and I think the conclusion that people have kind of come to is well um even though these models are very good at doing these tasks uh they for whatever reason are not actually very good at dealing with long context things. Um, and I think the other kind of important thing here, which is like related to a lot of the work I was doing in the past, is like I don't really think there's as big of a distinction between long context tasks and regular tasks as uh people maybe have in their minds. Uh, and the reason I say this is like I think any task in general can be framed as a long context task in some way. So like you for example can solve a coding problem um but you can treat like all of your extra reasoning tokens or any extra tools or any like all this like extra stuff that you feed in as like quote unquote long context. Uh and in some sense like I think these two problems are related. Uh we just have been dealing with it in a very like oddly specific like needle in the hay stack type of way. Uh, and this is something that we we also kind of didn't maybe address uh super deeply in our blog post, but it is something that we are addressing, I guess, in our our full paper as kind of a sneak peek. Um, but uh to kind of give some intuition on like why we would want to do this like recursive language model idea. Um, which I guess I I haven't uh I I didn't explain what this is. Um very generally a recursive language model um as we put it is just a like inference strategy for having a language model have control over calling other language models. Um and like the idea here is is generally if you treat a language model as a function um you want to have a little bit more flexibility over like what this function does. And like right now what we do is we just have like a blackbox model and we like query it and we get something out of it. Um and we know that there are some limitations for doing this. So one of like the big ones is we have like systems level limitations. Um like GPT5 can only support like 400k tokens for example. Um Gemini can only support uh like 1 million or some something in the millions of tokens. Um and you know another kind of part natural part of this problem is like language models are are fully built on like the distributions that they're trained on. So in some sense like long sequences of data is kind of like a longtailed thing like this or this this doesn't really appear very naturally um in a lot of the data that we're given. Uh and so like explicitly training models to handle longer and longer sequences of data is very very hard. Uh and it's also super expensive. Um so the thing that we kind of want with this inference paradigm is sort of like we want the illusion of a language model call that can handle near infinite context uh without degradation. Uh and like this design is also not meant to be like a replacement for every agent available. So I think I think there are a lot of really useful things you can do with agents. Um and in some sense like we want to replace like the base language model function call uh with this like new inference strategy and so we don't really have to deal with like a lot of the problems that current language models have regarding context rot or performance degradation and things like this. Um I see something in the chat like this is like virtual memory. Um like yes in some sense uh this is there there is actually another another term that I forgot um that my my adviser Omar uh had sent me about uh related to like database execution stuff but in some sense yes like you are uh effectively like offloading a lot of things um from the like main language model call and like this is a form uh of memory in some sense. Um, but I also think that like there is a a very strong like still use case for you know more traditional agents and and things like that. Um, so uh kind of the other part of this which I also for whatever reason didn't highlight very well in in the blog post is I I like to think of recursive language models as uh kind of a strong bet uh on the bitter lesson. Um, and like kind of what I mean by this is the way that we treat agent scaffolds, uh, and agents in general nowadays is we build these like very complex human-engineered scaffolds around our models. Uh, and when we want to kind of use this on a new task, we have to rewrite our scaffold or like modify our scaffold in some meaningful way. Uh and the reason why this is this is not great is generally like language models as we know from instruction tuning and and things you know research of this flavor like we want language models to answer things uh or answer prompts in a form that they've roughly seen before. Um, and if we have some kind of like more fixed common uh quote unquote scaffold, like there is a way where we can basically train our models to be better at using this specific scaffold. I think usually we don't do this because it's quite an expensive thing to go for. Um, but in a lot of ways like this is kind of what reasoning uh models have been doing uh maybe in the past half a year or two a year. Uh and so like the thing that we kind of are thinking here is well with recursive language models the idea is sort of we want to defer the management of context uh calling like subm models uh dealing with like tool calling and all this stuff. Um like we want to defer this all to the language model itself. Uh and in some sense like we can optimize for this functionality directly uh with something like RL or or even like SFT or or things like this. Um and so the view that we take with this is like an RLM is effectively just a language model or like for for for from the perspective of the programmer or the user uh they treat it like a language model. Um but from the perspective of someone training something or or like that wants to use this system uh they can actually take advantage of like how this is used um to you know uh better optimize it. Um but yes uh sorry I'm seeing lots of things in the chat. Maybe I can address some of these things afterwards. Um, so this is kind of like what I was saying before. Um, I think the the very abstract view of a recursive language model is basically this system where um the language model doesn't treat or or doesn't directly deal with the context. Uh, in some sense like it deals with the context as if it's some kind of game like as if it's some kind of memory that it's poking at. Um and it in some users specified environment it pokes around at in this environment and can recursively call language models uh over like different transformations of this input uh to do some meaningful things. So I think the like the really intuitive natural kind of way to think about this or like example is like if I have some summarization task like I have a giant book uh and I want my language model to answer a bunch of questions about this book but I know that it's going to have like context rot when it deals with this giant book. So what a human might do is you might split up the book into chunks and then like like call language models over each chunk, summarize everything, feed it back to like the base model and then have the base model look at the summaries and do something with it. Um and like a recursive language model is sort of a generalization of this idea um to like more flexible environments and things like this. Uh, one thing I I also don't particularly like about this diagram that I made is that I think the distinction between a query and a context is not necessary. Like I think you can treat actually just all context as the prompt. So this can include like a codebase um or like many many instructions embedded inside of like a multi-turn conversation. Like I think the in the end like a recursive language model is functionally just taking in a prompt and giving you an output. Um but like under the hood there are maybe some more interpretable kind of components uh of this. Um and yeah so the experiments that we ran in our paper and I think maybe the most interesting instantiation of this idea um is implementing this recursive structure with a ripple environment. Um, and the reason I say this is um I think like if you kind of think of like um I'm I'm not a very like AGI pill or like AGI type person, but like if you were to imagine roughly like what is a a a more powerful flexible system going to do? Like is it going to be this system that you treat like a blackbox function that like magically gives you answers to things? Probably not. And like you can probably prove things about like why such a system has some constraints over it. Um in terms of like uh what is its uh like representation class or like things like this like there's there's all sorts of things you can say about like learning a function that actually does this kind of thing. Um, and I think like the Kodak paper is a is a really really good idea in that like what they were kind of saying was, you know, we we should ground these models or these like more basic agents in some kind of environment where they can do stuff. Um, and and kind of the the the proposal of like RLMs with the ripple environment is like one step like further which is is sort of to say like even the management of what the language model sees should be handled in this like ripple- like environment. Uh and I think this is one of the bigger distinctions between RLMs uh with ripple uh and Kodak um in that like the treatment of the context or like the treatment of the prompt is really really important. Um because you I I I sort of the pitch here is that like we should never really have limitations regarding context length or like context rot. Um what this kind of means is like we we actually should deal with this in like a slightly different abstraction. uh because you don't really need a model to take in like you don't need an individual model call to take in all of the context. Um but at the same time like we sort of want to be we want to think in this lazy way like we we want to think in a way where all we care about is feeding in the full context like I I we don't have to like deal with it or partition it in any way. We just feed it to a model and like it has a way of dealing with it itself. Um, and uh, this is another thing that I didn't really share, but like these are some like older kind of diagrams I had when I was pitching uh, some stuff to Omar uh, like early on. I mean, I guess I'm still early on in my PhD, but um, like very very early on. Um, and like the point I'm trying to make here is sort of that uh RLMs are are they're they're not um they're not completely novel in the sense that like there are various instantiations of this idea that we already see in practice uh in things like codecs or claude code and how they deal with like large code bases as context um or even like summarization techniques and tricks like these all of these ideas in in a lot of ways are a very task specific way of dealing with the context. Uh and kind of the pitch here is that like hey like this this framing uh can be made in a more general way and it can be made in like a task agnostic way uh where in some sense like we treat the context as an object that you want to peer or transform or like perform some kind of sub calls over. Um, but yeah, I guess before I I gloss over the experiments, because I I don't think it's that important that I I go into that much detail about them. Um, are there any questions that uh I can answer before I continue? >> Um, there there's uh a couple questions in the Q&A. Um, I can just invite uh Nathan Parker to unmute and and ask. >> Cool. Yeah. >> Nathan, are you with us? >> Yeah. Hey, thanks. Um, hey Alex. Uh, yeah, I was just I, you know, fascinated by this approach. Um, I do a lot of building with uh with models and agents for a living and come up against this this problem. Uh, I'm curious if you could elaborate on your choice of ripple environments as a context container versus any of the other ways you could have solved that. >> Sure. Yeah. So I think the reason we ended up choosing the ripple environment is like if you think about how a like as a researcher or even like yourself um like when you do some kind of like if if you're given some task like some some project that you want to do and you're given a bunch of data sets and like things that you need to like look at and like figure out like how you're going to build your method to like deal with this thing. Uh generally I think we like to deal with things in code. Um and I think so if if I'll go back to this this version that I was thinking of um is kind of at odds with the like bitter lesson thing I pitched before. And the reason I say this is like we want these models to have as much flexibility as possible in how they deal with the context. Uh and I think in my mind like there is a balance between how abstract you can get here and how like practical you can get. Um and like the most practical thing you can do is basically define a bunch of tools that the language model like that can steer it in a certain direction. So like in this example, one of the tools that we gave it was like explicitly a function for partitioning and recursively calling a language model over every partition. Um, and this is a kind of strategy that I think is very like human imposed. Like I think it's a good idea and I think it will help a model like do this task. Um, but I also think in a lot of ways like you know we deal with like data sets and and and things like this in Jupyter notebooks. Um, and like we have ways of like how we peer into things and view things in a reasonable amount of time. Uh and like if we were to fix or like pick an environment that would be useful like now like kind of like the 20 end of 2025 be beginning of 2026. Um in my mind I think it's like equipping language models with code makes the most sense. Uh and I think like that is also something I I I almost think like codeact was sort of ahead of its time a little bit um in that like I think it's kind of an underrated idea or like an underrated work. Um, and I think language models like dealing or thinking or using code in some meaningful way should be a more like general strategy like even outside of code domains. >> No, that makes sense. >> Cool. Thank you. I uh really really briefly one followup and then I'll open the floor. I was just curious if you have uh actually done experiments about like you know the the quality of outputs when they're using code as their primary medium of expression versus text or something else. Like is this just sort of your intuition as a researcher or is there like comparative data? >> Um there is comparative data. I think they're not super clean experiments. Um which is why we we didn't put them up. Um it also I guess is is a bit of like intuition uh as to like or like my own belief into what I think is is useful. Um I think in the full paper we we do have like some alternate environments that we're planning on running experiments over. Uh what I will say is just gener like intuitively kind of what happens when you don't give it a code environment um is it sometimes cannot express what it wants to do correctly. Uh, and I think a lot of this has to do with like just language models right now are just trained heavily on code or like Python specifically. So they're quite good at like dealing with objects in memory um in a like ripple like way. But if you give it tools to do this um there's a bit of a mismatch between what it actually wants to do or like what it says it's trying to do and what it actually ends up doing in practice. >> That makes sense. >> Yeah. >> Thanks. Awesome. Um, yeah, feel free to keep going, Alex. >> Sure. Yeah. Um, I'm I'm happy to answer anything at at any point. Um, I did see something in the chat like we view language models as RLMs. Um, yes, I I think that's exactly the way that like I I have been trying to like make this idea clear. And I think also like I'm very excited to see if other people like come up with other examples of environments like maybe maybe an improved version of the ripple, maybe a safe version of the ripple, maybe just not a ripple at all and like see if it actually works better. Um because there are some problems with the ripple environment which I I will go over kind of in the end. Um but I think like I I'm I'm very excited to kind of see how this idea matures um or like manifests over time. Um, you know, beyond just like an academic researchy type of thing. Um, yeah. So, I I think the experiments are maybe what what were the more exciting things about this blog post. Um, I will go over this in a kind of a TL TLDDR kind of way because um there are a lot more experiments I've been thinking about recently and and things that are running um that maybe I I can't fully share now, but um these experiments are very exciting. I think those experiments are also maybe even more exciting. So I you know I'm I'm I'm very like excited to share all this with you guys. Um but generally with these experiments at least in the blog post the thing that we wanted to show um was like okay in this like long context domain right there's generally two problems that we deal with. Uh the first one is like I actually I flipped the order on these slides but I'll start with the second one which is like you know we can't feed in infinite context into the language model like it it will throw an error if we do this. Um, and like in some sense as a lazy programmer or like a lazy researcher, I just want to feed in as much stuff as I can. Um, and I don't necessarily want to build an agent that can deal with it. I just want this like language model function to deal with it. Um, so we wanted to show that like, you know, I could just jam in lots of stuff. Um, and it will be able to deal with this effectively. Um, and the other point is like even if it can fit everything in context, um, it doesn't necessarily actually deal with it correctly. And this is kind of this context rot phenomena or like I actually don't know where this phrase originated from. Uh, my first the first time I saw it was in the anthropic blog post. Um, but I think like generally we want a way of of of dealing with this. Um so the experiment we showed for context rot was this ulong benchmark um which as of today still is not public uh unfortunately it will be public very soon. Um we are also not affiliated with this paper or these authors at all. Um they were just very kind and generous in letting us evaluate on this data set. Um this data set is basically just like a bunch of very difficult questions. um where the questions require you to look very densely at the input and answer something about it. Um and so it's like very different than the like kind of retrieval style tasks, but all of these tasks fit fully in context for GBT5. So I think the largest um problem is like 260k tokens. Uh and GPT5 is like 272 I think for like input tokens. Um, but basically this was like the really cool plot that we showed which is like hey like if you have an RLM using GPT5 mini uh not only can you like effectively double the performance of GBT5 um the actual like API cost of doing this is roughly as expensive or cheaper. Uh which is kind of crazy. Um and we even tried this with just GPT5 as well. Uh generally the cost is like kind of comparable. Uh it's a it's a bit more expensive uh but the performance is like comparable to the GBD5 mini. Uh so like this was maybe the more exciting and interesting results. Um we also tried it against like a react loop uh with like a BM25 retriever. In this setting a retriever is really awkward and so like you know there there might be some criticisms about this kind of thing. Um but yeah like I I I think like this is kind of to show also that uh one of the things that that popped up in um my discussions with my lab is like is every long context task retrieval and I think the answer is like obviously no uh but this is kind of an example of like retrieval doesn't really make sense here. Um yeah uh there are some trajectories also as well uh that are in the blog. Not actually not full trajectories um but like you can kind of see roughly what the model is doing or like what the like root level language model is doing um and like how it does the recursive sub calls uh which is kind of um an interesting thing. Um but yeah uh and kind of the second benchmark uh was this browse comp plus thing uh which is basically so browse comp I think is a benchmark from openai correct me if I'm wrong I I think it's Jason we's paper um but the idea is like it's like a deep research type of benchmark which deep research just means multihop retrieval um over the web um here what they did was they took browse comp they like downloaded this giant corpus and they like guaranteed that the solution is somewhere in this corpus. Um but it's like an offline multihop retrieval problem. So kind of the idea uh that we wanted to see was like uh if you scale um the number of documents that you feed in context and you also you know do the same thing for retriever. Uh we actually just use the same setup that they used in browse comp plus for the retriever uh to kind of give like a fair comparison here. Um but uh yeah the idea is like you know RLM scale very well and like this is not that surprising right uh and and the reason why this isn't that surprising is like generally when you're doing this kind of retrieval type thing um you can do a lot of these things in code as well. Uh you can do this by querying like subquerying models over your corpus or like you know reject grepping for certain things or like certain articles and then sub calling a model over the article to check that it's like correct. Um and I think this is one of those things where like it's not that surprising but also like it's very very useful to know. Uh, and the other maybe more interesting thing is like we didn't do this, but you can also equip an RLM with a retriever uh, if you want as well. Uh, it's it's kind of like it is a very interesting thing regardless. Um, but yeah, I I also see a question in the chat I think maybe is a good time to address. Uh, how do you know you made a fair case for the other retrieval methods? Would a retrieval system with optimized recall precision outperform RLMs? So, um, this is a good question. I think on the ulong benchmark there is no retrieval system like retrieval doesn't make any sense for like a dense prompt like a dense set of data. Um like and and the reason why it doesn't really make sense is generally for retrieval what you're going to do is you are going to do some kind of pre-indexing over your data. Uh like you you cannot just like arbitrarily retrieve things right. Uh and and in a lot of settings like in the oolong setting for example uh like if I were to give you some like random prompt and like I maybe I have like hundreds of thousands of prompts uh that are each like different data that are like uh you know 200,000 lines of things. um you would generally not use a retriever here because the the cost runtime of doing this either like an embedding based like rag approach or a rag actually you would never do in this setting but if you use like BM25 or or some other like kind of heristic based retriever um you generally wouldn't pre-index this like you you would want to do this like on the fly so this is kind of like an we kind of call this like an on the-ly kind of setting where like an RLM is actually very useful here because you just want to dump things like you don't necessarily want in indices for this. Uh but browse comp plus is a separate example uh where yes you can so the BM25 plus react we took directly from browse comp plus that that's like one of the baselines that they ran on this benchmark um they also have a quen like embedding rag based thing uh which we did not run and the reason is like the results aren't very good on in their original paper uh and we can run this and actually we might run this anyways on the full paper just for the sake of like a good comparison person. Um, but you know, at at the end of the day, like a well-tuned system over like a a problem where you pre-process the inputs, uh, most likely will do better than like a very naive RLM. Like I I would be very surprised if that was not the case. Um, but I guess like also kind of here it's like um for in the broader sense like this isn't that important. Um, I think we still, you know, will and should include these these baselines anyways. Um, but in most settings where you would care about using an RLM, uh, you generally can't really afford to do this kind of like hyper optimized, uh, retrieval strategy. So yeah I don't know if that that answers it. Let me know too, by the way, also like uh because we're still interested in running some stuff. If there are any like methods or benchmarks that you think like are not included here and should be included here, I'm very very happy to like test them and and put them in the full paper too because the the funny thing is like in doing this project, the thing that I've struggled with most is finding comparable methods and benchmarks to evaluate on because in the long context setting a lot of benchmarks in my opinion have been kind of off the mark a little bit. I think browse comp plus is actually pretty good, but there there aren't a lot that I would say kind of hit the mark of what we want. Um, and in terms of like baseline methods, they're kind of all over the place, right? So like React plus BM25 can mean just about anything. Um, we just kind of like took the code that was in their their codebase, but I think like in general like you could hyperoptimize a React loop too and you could probably squeeze out some performance here as well. Um, but there is kind of this question of like how much do you really want to do that? Like even for the RLM for example, we didn't really optimize it very well. Like I I I kind of just threw in like a prompt. I did not run DSP or anything on this prompt. Although maybe we should. Uh, and I kind of just let it run. Uh, and so like you know that's why we have these results so quickly or like so uh soon I guess. But um yeah. Anyways, um so kind of maybe the last thing uh before I just turn this uh to a Q&A is like one question that we got a lot on Twitter um was like what is the relationship between RLMs and agents like claude codecs uh things like this um and like why are RLMs not the same thing as this? Um, so there's like two answers I I I would give to this. Uh, number one, obviously both of these systems are designed for code. Uh, now some people don't like this answer because like kind of the the response I've been given is like you can use these systems for things other than code and people have done this and I fully agree. I mean I I would argue you can even use sweet agent or some other like you can use agentless if you really want uh for non-code tasks. um like would you do this? Probably not, but like you could uh because language models are very general task solvers. Um but I think like the the one of the bigger differences is that like so these agents are are basically designed for like navigating the type of context that relates to code. Uh and like I mean I guess if you pitch that like all problems um can be viewed in this way then like in some sense yes these systems are a very specialized instantiation of the RLM abstraction. Um I I don't really want to say that because I also think that these systems are very special in their own ways and this is a good thing or it's kind of like a pro in their favor. Like these systems do a lot of really cool things that are really specific to software engineering. Um, and I I don't really want to say like I don't want this to be like a like a Schvid Huber type thing where I'm like RLM like CodeEx and Cloud Code are just RLMs. Like I I don't really think this is true. I I think there is an aspect of these systems that um are similar or like fall under the RLM abstraction of doing these like sub aent sub call things. Um but I I also think that like if we want to think about like how we design systems in the future uh and and how we want to design like even like APIs in the future for people to you know call language models um I think like we want to think about this very differently like we don't want to apply the like clawed code hammer to very general long context tasks like this is extremely bloated uh and I think in this way like This is why I think RLMs are a particularly interesting way of of of thinking about this. Um, but I'm I'm happy to answer more questions about this by the way like especially if if people um you know have some concerns or or like thoughts about this because it is also something like I I have been you know thinking about more deeply uh because I I also want to see like you know maybe there are some applications of RLMs that make a lot more sense u than than these kinds of things. >> Um >> hey Alex, we've got some questions line lined up in the Q&A section. Um before we jump into that um one very high level thing um I think most people um probably didn't get a chance to read your blog post. So for the benefit of those who um are playing catchup um can you just spell out what the thing is? >> Yes. >> I like what is it? >> Let me let me go back to here. Um, okay. And I hope this diagram isn't too confusing because if it is, it means I need to redo the diagram. Um, so a recursive language model or like the the the entire idea uh behind this blog post uh and and this method is it's basically you can think of it like a wrapper on top of a language model. Um it maybe it is a description of the a wrapper on top of a a language model. Uh and the idea is like what this is doing is it calls the language model inside of some kind of interactive environment. Um now in this interactive environment what the language model can do is basically call itself or like call another language model. um to do anything in this environment. Uh and and the key kind of component here is when you query this this like wrapper like this thin recursive language model which we treat exactly like a language model. So in your code uh instead of doing gpt5 completion you would do rlm.comp completion. What ends up happening is the prompt that you feed in doesn't directly go to the language model. Um what what happens is this prompt goes into the environment and the language model basically has to figure out a way to address this prompt entirely. So it gives the illusion of a language model that is just directly answering the prompt. Um, but it uses this environment to kind of look at or like maybe chunk up the prompt um, and answer like different parts of the prompt uh, and things like this. Uh, and so the choice of environment that we use here is like a Python notebook. So I can kind of walk through a really really simple example here. Um, I think actually maybe this diagram is a little bit better. So if I have this like prompt, right? It's like a huge context and some question about that context. Maybe the question is like um I am looking for this uh article that describes um I don't know like uh game five of the World Series uh and like I I want to find this thing in this context. So normally I would feed this to a language model but maybe the language model can't actually deal with this like maybe this is like a billion tokens or something. So in the recursive language model, what ends up happening is the programmer views this as like just feeding feeding it into the language model. Uh in the RLM abstraction, what happens is this context basically gets thrown into this like ripple python. You can think of it like a Jupyter notebook. It gets fed in as like a variable. Um and basically the root language model knows like I'm trying to answer this question and I need to look at this context. Um and I have access to run code in this environment. Uh and in addition to running code, I can also call a language model over like anything in this code. So like for example what I can do or what the language model can do is it can split up this giant context into like a hundred different um chunks and then in these individual chunks it can recursively call a language model to look at each of the chunks give me an answer. So that's what this like LLM query thing is give me an answer. This answer is again stored in memory. So so this top level language model doesn't see any of this. the only thing it sees is this left side. Um, and it knows that like the answer that some other language model gave is sitting in some variable in memory. Uh, and it basically is sort of a way of dealing with the context without the language models themselves having to look at the context. Um, and so this idea actually extends like beyond like you can keep recursively doing this. So like a recursive language model can also call a recursive language model and like spin up another environment over this like subcontext uh and do effectively the same thing. Um but at the very end like given this whole process um it can produce a final answer uh based on either stuff that sits in memory or based on like if it's confident in its answer. Uh and so all of this is kind of hidden from the user in some sense. Um and it's fully like all the user sees is just like the question like the prompt uh and the answer that is given by uh this recursive language model system. Uh but it is very generally just an inference strategy over a language model. It is not >> is it is it fair to say this is just tool calling but you're only allowed one tool which is a generic >> language model call. >> I I I wouldn't really call it tool calling. uh I would say it's more similar to code act uh which is just you give it access to a execution environment like I think when you think of tool calling you you usually think of it as like a stateless thing like you're just calling like a calculator or something to do to do and and in some sense you can give a recursive language model tools >> um but it's also like this I think the key part here is that the language model doesn't see the context this This is maybe one of the bigger differences between this strategy and like just generally any other like code environment tool calling whatever even like sub aent type strategy um which is that it just doesn't see like it doesn't have to look directly at the context um and neither do its sub agents either. Eventually one of the calls do have to look at the context um but yeah >> but but it'll be a manageable chunk of the context. >> Exactly. Um, and >> okay, >> another reason actually why uh actually one of the questions uh I I also want to address is like why is this cheaper? Like I I think this is not obvious because what I described before like the thing I just described makes it seem like it's really expensive because you are going to keep like you you almost expect the model to look at everything. Um and this is what a base language model would do anyways. Um, but the reason why the code environment is so powerful is it's effectively a form of filtering. Like it's able to filter things in code without having to necessarily use the language model to do it. Um, and this >> so for for example, it it might not actually run an LM inference on every chunk of a million token >> piece of thing. It might actually decide, well, I'm going to run a a search first. >> Yes. Exactly. >> Like it's it's allowed to do whatever it can in code. So it'll find a trick. >> Yes, I think code is the most flexible way to do this. >> Brilliant. That that helps a lot. >> Yes. I And I think this is another difference between this and the the goal of codeact was sort of to say like um uh where is it? That's fine. Kodact for context by the way is just a lang you give a language model a like Python environment. That that was like their their kind of idea. I I think that the the one of the huge differences is how they frame it and and like their the purpose of their paper is to say like we should give language models code because then they can solve like multiplication and they can solve math problems and like they can do these like kind of reasoning over like using code. Uh, and I I agree that this is really really powerful. But I think one of the key things here is like we're not like I we're we're not treating code as like like a calculator type of thing like oh language models aren't really trained to do math so we have the the code environment so it can like do these things. Uh it's more so like I think code is the most flexible way at least currently um to basically give the language model options as to like how it deals with context. Um and like that is kind of the beauty of using code. Um it's I don't know maybe that doesn't fully make sense. In my mind it makes a lot of sense. Uh but yes. Yeah. >> Thank you. Thank you so much. And um if you click on the Q&A um on the bottom of your Zoom interface, Alex. >> Okay. >> Take um I'd like to point you to Shangian Tan's question. It feels like mine was a good seg to that one. Um and Shangen, if you are able to ask yourself, you you provided some citations that I wanted him to see. >> Oh, I see. I did not I didn't know this was a thing. Okay. Um >> yeah. >> And I'm gonna see if I can unmute. I think I think Shen Yun is unmute now. >> Okay, I see the question now. >> Okay. >> What team? Oh, yeah, you can ask him. Yeah. >> Yeah. Um, so hi. Uh, nice talk by the way. Um so yeah so my question is basically like what do you think the relationship between ROM and other work that where um they try to like train a model to basically uh let the model like um like spawn um a parallel like LM or like a async LM to reason about the same problem and then that that like that that parallel LM or async can join the main thread LM later. >> Yeah. Uh there is actually another paper called thread. I don't know if if this was included here. Um that came out like a year ago. It's kind of doing what you're describing. Um I I think the big difference for RLMs is this like context ccentric view. Uh and and I I tweeted about this a little bit and and what I mean is I think people like to view things in terms of tasks or like problem decomposition. So when they think of like parallel reasoning like what they're thinking of is like okay I am do I'm doing an Amy problem uh uh sorry I'm doing a math problem um and I like the way I'm going to go about solving this is like I'm going to try to uh like decompose the problem down into like five different sub problems and then I'm going to call sub agents to solve those problems and then we're going to like kind of come back and like get some final answers. answer and then I'm going to verify it and then I'm going to give you an answer. Um, and I think this is fine. Like I think this is also a very interesting line of research. Um, but it doesn't really solve the problem that we were originally trying to solve, which is like in in a context ccentric view, the the idea is like I don't care at all what this context is. Actually, maybe most of the context is not even a task. Like it's just information. Um, and the thing that I that I kind of want to do here is like I want the language model to deal with this context. I I actually don't care if it calls recursive calls to deal with it. It just sort of is like that makes sense as like a necessity for what it needs to do. Um, but it it's sort of like you're trying to do this in a lossless way is is the way I would say this. Um because things like summarization or things like task decomposition and and stuff like this are like focused on I have some smaller like query task that I want to do and I want to like reason about it. Uh in this case it's kind of like I have this giant thing that I need to navigate um and I need to effectively deal with it. And so like it's not enough and this is why like the ripple stuff was kind of important. It's not necessarily enough to do parallel reasoning here. Uh like you you you kind of like there are different strategies depending on the information or like the the task prompt or whatever that you're given. Um so I I do think a lot of the ideas like async style stuff is super relevant to this too. Like we should I have not been implementing this. I I should there are some reasons why it's it's quite tricky to do. Um but like you know for reducing runtime and things like this like you know doing parallel reasoning or doing async style stuff is definitely very meaningful. Um but I I think it's in my mind like it's sort of a distraction from like what the actual goal of of doing this strategy is. >> Alex, we've got another question coming in. Um is from Louise Frias and I think Louise had to had to actually jump off. Um, I can't find Louise in the list anymore. Uh, so I'll just ask it on uh their on their behalf. You mentioned that the method um involves the concept of a game. Um, what kind of game did you have in mind when experimenting and what outcome? And just to add to that, like um it I can see where you're headed with this. You're basically saying wherever h we human engineers are are clearly making handgenineered decisions about how you should think, step back. Let the thing just have access to a code environment and go. But then of course you've got a new problem. Now you have to be the environment engineer and and do reward shaping. So um any thoughts about what this adventure is going to look like? >> Yeah. So I'll answer these I'll answer the the two separate things which is number one I mentioned a concept of a game. Um so I I don't this could be referring to two things. The first thing this could be referring to is uh all the way back in the beginning I mentioned that uh I worked or like I I had this benchmark called video game bench. Uh and this was one of the motivations for this idea. The reason this is a motivation is in video game scaffolds like like for for vision language models which effectively are language models um it is also a long context problem which I think people don't fully see because in video game settings like if you want to turn a like an actual video game like I I mean like a like Counterstrike or like uh like Doom or like League of Legends or something like these kinds of games um the state is uh there are the time components. So when you feed this into a language model, you can't feed still frames like this is not an MDP anymore. Uh so you have to feed in like sequences of frames. Um but frames are very expensive uh in terms of like token space. So this ends up actually becoming a long context problem. Uh and the funny thing about RLMs is that there is no assumption of text here. actually any context that fits in computer memory is technically allowed in this setup. Um, and this is actually a very unique and nice thing about this. Uh, and I actually ultimately want to apply this to games. I don't think it's a reasonable thing now, but it is something I'm very excited about. Um, now on to the other thing about environments. So um I think in the current like ripple like environment um if we want to do some kind of like RL training over this it's actually quite simple. Uh we basically just optimize for correct answers in some sense. There is other components to this like reward function that you define and it's things like oh like how many iterations do I want the language model to do? uh can I constrain the cost in some meaningful way? This is very like like lrangee multiplier kind of um way of thinking of things. But um I I I think like we there there are many interesting ways to go about like how you actually train a model to be better at using an RLM or sorry as being an RLM. Um but yeah, hopefully that that kind of answers uh that part. But that is actually something I'm I'm very very interested in doing as like a follow-up of this work, which is a lot of like training of models to see um if they can more effectively like use these strategies. >> Alex, we've got a question from Tim Woo. Um let me see if I can >> um >> get him to the stage. Tim, you still here? He had a question about um it was a follow-up to Nathan's question on the choice of environments and he was wondering if um instead of a ripple an environment uh the environment was an ephemeral pix compliant sandbox or something. >> Yeah. Hi. Um sorry not sure if you can hear me but >> yes first of all like very very sick paper. I really liked like the recursive approach with ripple and the action space is like pretty cool. I've tinkered around with the GitHub implementation and um you know made some modifications to handle larger memory context. So I've been using it to reason over exports from agentic chats. Um so agentic chats the agentic chats I export they're typically in the same format. So I was just wondering whether a pausix compliance sandbox um where we develop you know repeatable scripts to u more effectively navigate the context would be an interesting approach. Yeah. Um, so I think if you want to use an RLM with a ripple environment in a production setting, you actually probably need to do something like that. Um, like the way that I've done it currently is is very sus. Uh so I I would not like it if the the way if you look at the the the minimal uh example that I I put up I I basically take like an exec environment and I remove some like global built-ins uh for like safety purposes but that is not like there's a lot of concerns about doing that. Um so I I think that's a very good idea in terms of like pre um like pre-populating it with certain scripts and and options to do things also a very good idea. Um I think that the only thing to be careful of is like like I was saying before and and and like uh John had also mentioned like we generally want to give the model as much flexibility as possible in like what it does. And so like the thing you have to be careful of when you predefine stuff is like you you do sort of limit what it can do. Uh that's not always a bad thing though. Um, but the other part of it is like the other part to think here. I I think this is maybe something that a lot of people don't think about very well. Uh, and it's this idea that like language models are very sensitive to conditioning. And what I mean by that is like I I mentioned this before, they're very sensitive to the way they're prompted. So I if you're kind of in the business of like building agents or like building very general scaffolds like that you want people to use very generally like I I want to people to use my RLM with ripple scaffold. Um the thing that you want to consider here is like you want whatever prompts are fed into any part of this RLM stack like the the outer top root model any of the recursive models you kind of want the prompts that they're being given to be somewhat controllable and and the way that it's controlled is like your system is designed around what they're good at or the opposite is like the system is so interesting that like OpenAI or Enthropic start post-training their models to handle these kinds of inputs. Uh, and like this is kind of a way like I I think this is why I I've kind of been pushing for I I hope in the end like when I when this kind of becomes a more mature software thing, it is like a fixed kind of library or like people use it as like a fixed kind of scaffold um and they can like kind of build things on top of that. But the idea is that like future frontier models hopefully should be tuned to use these scaffolds very well. this is like effectively what instruction tuning is. Um and I think we like I do kind of want to think in this way like maybe it won't actually work out in that way but that would be the ideal case. >> Thank you >> Alex. I just want to be respectful of your time. I realize we are past the hour but um obviously there's a lot of amazing um questions. Um so whenever you need to >> Sure. Yeah. Yeah. Yeah. I I'm I'm happy to to stay on for for a bit longer. Um if there are other I think there are a bunch of other questions in this Q&A. I don't know if I I also could just go through the list and just go through them. >> Yeah, absolutely. Alex, just uh take a browse and anything that seems like it's something you want to jump right on, go for it. We we have so many good questions. Thank you everyone for putting them in there. Yeah. So Paul asked, basically what he's asking is there are some cases where an RLM just feeding it into the language model is better than an RLM. Yes, this is a very good point and uh as a little bit of a sneak peek for our actual paper, one of the things that we're studying right now is if you we can a can we characterize long context problems? I I kind of alluded to this because not all long context problems are equal. uh and there is a way well I am proposing a way to characterize long context problems uh with some kind of like uh actually maybe I shouldn't say this uh the paper will come out pretty soon but um there we have a way of characterizing long context problems uh and and the reason why this is cool is in our experiments what we want to show is like if you scale the input sizes uh given like some characterization of this problem can we sort sort of understand when is appropriate to use a regular language model and when is appropriate to use an RLM. Um, and the reason you might want to do this is like one perfectly plausible way of doing using an RLM is just like an if statement. It's just like if you if I can characterize my prompt very quickly like I can classify it in some way uh and I know the length of the context I can choose to just call GPD5 or I can choose to use an RLM. Um, and like I think when we think about future systems that just slot in RLMs, this is actually what we want to do. Um, we very much like I think RLMs are are specifically designed for dealing with complicated contexts. um and like avoiding kind of I I I I really don't like the way that um that we've been dealing with long context from the model architecture perspective which is basically like we add all these tricks like sliding window and and and things like this and hopefully we build models that can handle longer and longer context um and maybe we scale our synthetic trading data to handle this too. Um, but I think this is a super costly way of doing things. Uh, and like we've seen maybe in the past like three or four years like you don't get that much of a benefit from doing this. Uh, and I I think like we're at a point now where these models are so good and like they are large enough or like they can fit a large enough context where we now want to consider like you know uh using an RLM instead. But yeah, >> but Alex, uh, it sounds like you're saying there's something upstream of the RLM that needs to do routing to decide whether it goes to an RLM or something else. But couldn't an RLM just make that decision? >> Yes, that that's also a very good point. Uh, in the end, an RLM could perfectly make that decision. And that is we've actually, funnily enough, there are some trajectories where this is literally all it does. So it will load the context into the ripple. it will just call a recursive sub call over that context and return that that answer. Uh and like the overhead of doing this is basically negligible. So it's not like at least compared to the runtime of the actual language model. Um runtimes are a separate thing though. So this is my third bullet point. Um I'm very I I you know used to work on a lot of systemsy things or like systems for ML GPU stuff like serving engines and things. So, like I I probably will not work on this myself, honestly. Uh it's not really in the direction of of what I've been interested in recently, but I am like excited to see if people figure out kind of like how you hyper optimize um RLMs uh from a latency perspective. But yeah, um we already characterize long context tasks, needle in the haststack, risk queries. Um, yes. So, we do, but the thing is like there is a more rigorous way to classify these things is kind of the the the teaser I will give. I I I genuinely think this like I think there is a more mathematically rigorous way of describing the differences between two different problems. Um and it's it's not just like oh retrieval looks for one thing uh and like ulong is like you have to list like count things like I think there is a way to describe this uh where we there's like a spectrum that you kind of can observe and how this affects your scaling plots. But this is still something I'm I'm I'm working on. So yeah, just as like a sneak peek um Oh yeah, sorry. Uh, let me continue. So uh >> and Alex, totally up to you. We're over time so >> yeah, you know, I I >> I I uh I have more if people want to ask more questions, like I'm I'm happy to to ask anything else. >> Cool. Thank you so much. >> Um, Alec asks, "LMs can also decide to use named entity resolution, co reference resolution, etc. Does it work or is additional fine-tuning needed?" Um I maybe I don't fully understand this question. Uh I think like I think in general like if if the question is is just generally like can you implement XY feature into an RLM the answer is yes. And and that's kind of why the description of the framework is so abstract because you can implement any tools you want. You can implement any environment you want. It's completely centered around the LLM itself. Um, but any kinds of like specialized tools that you think are useful for your task are perfectly fine to use with an RLM. The warning I would give you, of course, is that like the more stuff you add, the far the further you kind of deviate from like what is a more stable input distribution uh for the model to see. And I actually think this is a very important consideration in general like um you do kind of have to think a little bit about how models were trained uh and like what they are are are are used to seeing which is kind of a weird way of of framing things because I think we tend to think of LLM as just like this like super intelligent function. Um but there are definitely certain input distributions that uh when you condition on them they they work much better. Um >> I mean ju just to pull on that thread, imagine uh the the correct way to deal with a very large context is that you actually need to train a language model to do something. Could the RLM just figure out, oh, hold my beer, I need to actually like do a whole ML adventure in order to train this model, in order to do this processing, in order to answer that question. >> Yeah. Uh so this is also kind of the like the if you think about what a a more powerful language model system would be doing it would be doing exactly that and and this is why I think code environments are are very interesting. Uh there was a question have you tried no cha um for context this is a benchmark it's a long context book benchmark with like it's it's particularly challenging like there there's like true false claims uh things like this uh the answer the short answer is yes I did try it the problem is that this benchmark is not public um so they have a leaderboard and so it makes sense like this benchmark what they do is they evaluate like just base language models on their like website they have like four public novels or something. Um, and I did try it. The RLMs do like marginally better than the base language models. I didn't investigate this any further, but it's so it's such a noisy thing. It's it's very few examples. So, like it's not really anything to go off of. Um, and it's it's why we we we didn't end up including those results. Maybe I'll put them in the appendix or something uh in the full paper, but yeah. Um, another question is, are there interesting applications that aren't based on long context quality? I suspect managing intermediate results would in a ripple would allow for better decomposition of results and tasks. Yes. So, this kind of goes back to my point about every task is a long context task in some way. Uh, and like you know there are long reasoning chains and there are long outputs and there are long inputs. Uh, and all of these are derived from like maybe you need more tokens to answer a short question or something. Um, I think the problem is that there aren't a lot of benchmarks for what I'm describing very well. If there are, I would love to evaluate on them. Um, there is also kind of, I guess, the uh the like academic budget thing going on here. um where like I don't want to spend too much of of Omar's budget, but um yeah, I I am very excited to try these things on like just about everything. So, if people have suggestions, like I I' I'd love to try even if it's like your own benchmark, like I I would love to to try. Um, >> speaking of budget, Alex, um, don't you think that, uh, implementing an RLM system, um, in production and hence with some kind of sandboxing or at least limitations, um, you're going to have to provide the model with the context on its limitations in some way. At the very least, you should be injecting sort of what the current state of the budget is or how much because if you want it to be completely flexible, it needs to have uh self, you know, um introspection and you probably don't want to solve that with a language model. You just have some code running in the background, right? But that has to be part of its training environment. >> Yeah. So, >> thoughts on that? Um, in the code snippet I shared on GitHub, I removed all of this. Actually, in our experiments, I have a version of this implemented. So, there there are >> cool. So, it does see for example how long its context currently is or like how long each of the recursive calls knows how long um the thing it's looking at is uh but not in tokens. And the reason is that tokenizers are really slow even like the rust based ones which is really frustrating. Um so it knows the character length. Um it also knows roughly the cost but we don't use this information currently. So in this experiment and I think this is what makes it even cooler. I do not tell it at all to optimize for cost. Like I don't tell it like be as cheap as possible or something like that. Um it just is like this and I the reason why I think this is exciting is like you can totally push these numbers down. Um and that is a super super exciting thing I think for like in in general for people interested in in iterating on these these systems. It's it's it's very exciting. >> Um >> Raj just Raj just DM'd me a reminder that um Shang Yin 10 actually um shared a link to something that's really maybe relevant and cool for you. It's this uh Zork game benchmark. >> Yes. Um, funnily enough, >> you know about that? >> Yes. My senior year of college, which I guess is two years ago now. Um, I was working on this benchmark uh with >> Oh, cool. >> GPT4. Um, I was uh in Caric Nar Simhan's lab. Uh, I was working on like >> basically like a memory based agent uh like a for like search and stuff over text games. I have no idea how good GBT5 and stuff is at these games. I remember around two years ago they were absolutely terrible. Like they were just not able to track any information. Um yes, this would be very interesting to to see uh because long context was a big part of why these systems failed. So um >> Exactly. Exactly. And also the results are so unambiguous if you could like win this game better than any other approach like >> Yeah. >> just I don't currently know what the state of the art is. I know uh RL methods have been very good at this game. I don't know if they solved it completely uh because the the game itself is like it's kind of tricky I guess like it's it's one of those things where uh I think the state action space is very large. Um so it's like nasty in that way but yeah I I that is a that is a good suggestion. Thank you. >> Any other questions catch your eye? >> Yes, there is one. What do you think about entropy triggering your actions in your model? I don't know exactly what this means. I I'm assuming you you mean like the entropy of the language models themselves. Um and the fact that like a recursive language model trajectory has high entropy. Uh this is a good thing. So this is very much related to how like reasoning stuff works and like why things like star and like quiet star work. Um in some sense we want the model to have or we want the RLM to to have high entropy because we want it to produce many many different uh trajectories and solution paths to the same prompt. Um this makes training them also easier and it also makes like RL style training more interesting. Uh I can go on about this but I I think this is a maybe a separate discussion about like post-training stuff. Uh but yeah. Yeah. >> Alex, maybe this is a a good point to bring it to a close. >> Yes. >> Um I think these are >> do you want to do you want to round up your thoughts at all and send us off with any final reflection? >> Sure. Yeah. Let me let me go to this slide. Um, so I think you know the paper is coming out and the paper is of an academic flavor and what I mean by this is you this idea is already out there and like there may there actually there most definitely is a better version of this idea that does not exist currently um or some implementation of this idea that is extremely effective. uh and I'm I'm very very excited for this. Um I think this is not the focus of my paper. Uh so what I'm trying to say is like you know I would encourage anyone to go ahead and try and like follow your intuition and follow your nose on how to build up a system around RLMs that is really effective for whatever task you want to do. Um, I think the most interesting task in my head is these code like these like cursor codecs like multi-turn huge multi-turn conversation things where I'm sure all of you or most of you have noticed like when you use cursor for a long time like it starts to just like produce gibberish. uh and I think like this is easily solvable with RLMs and I think there are many other settings uh both in academia and in production where this idea is very useful. Uh and I think like I'm very very excited to see a like any you know open-source kind of like very robust libraries that implement this idea that we can kind of use as like a common thing. Um, I probably will do that if nobody does that, but you know, obviously I I have other things I I I need to do as a PhD student. Um, and then two, like, you know, I'm very excited to see where this stuff goes like in in production, like genuinely in production settings. I think there are situations where you would want to do this, um, and where it would be very useful and and you know, I'm very excited to see all sorts of tasks where this kind of stuff is applied. So, um, yeah, feel free also to message me or email me. You can DM me on Twitter. Um, or if that doesn't I don't really know how Twitter works, but if you can't for whatever reason, my email is on my website, uh, my MIT email. So, feel free to ask me for anything. I'm always happy to chat. Um, but yeah, uh, happy Halloween. Um, you know, uh, I hope this talk was was informative and I I hope I answered everything, uh, that people were wondering. Um, but yeah, thank you for coming.
