[0.00s -> 8.00s] Hi everyone, I'm Lauren Tan. I guess not many people know my last name. I am Potato on Twitter, [9.36s -> 18.64s] Potato with spelled with an E, and I have been at Cursor for about five months. Previously, [18.64s -> 24.32s] I was at Meta where I worked on the React team, specifically working on the React compiler, [24.96s -> 30.64s] which was a whole lot of fun. I'm still on the core team and contributing to open source here and [30.64s -> 38.16s] there, so that's really nice that they still let me do that. And before Meta, I was at Netflix [38.16s -> 48.40s] where I was both a tech lead and I transitioned to be an engineering manager for about two years. [49.36s -> 55.76s] So I've had a lot of experience going between engineering management and being an individual [55.76s -> 61.60s] contributor. And I think something I've noticed actually, which is quite interesting, is that [61.60s -> 68.56s] there are so many parallels with management skills and how to manage agents. And that's actually a [68.56s -> 75.68s] big part about what I wanted to chat with you and everybody else about today. But yeah, that's me. [76.40s -> 82.48s] I do have some very light slides, but it's not going to be just rambling, [82.48s -> 87.20s] so let me just share my screen and hope that I don't leak anything. [93.42s -> 95.42s] Oh no, I need to allow permissions. [96.30s -> 98.06s] Not always. Oh good, take your time. [98.06s -> 102.14s] There's always tech trouble. Give me one second to rejoin. [102.78s -> 109.73s] Yeah, go for it. I see many of you already know Lauren from the looks of the chat here. [110.69s -> 116.37s] So yeah, it's exciting to get a chance to chat with her and go through some of her recent work. [116.37s -> 122.45s] As you guys heard, you know, a lot of recent experience from Netflix to Meta and then now [122.45s -> 128.29s] over at Cursor. We're going to chat a little bit about Grokbot as well. So that'll be exciting. [128.29s -> 131.65s] I don't know if you guys saw that was a recent release. I think literally maybe yesterday or [131.65s -> 137.17s] the day before from the Cursor team, which is kind of like, let's call it like agents for everyone. [137.65s -> 140.53s] You can go check it out if you want and learn a little bit more about the product, [141.09s -> 144.05s] but yeah, we'll explore that a little bit today as well. [146.82s -> 163.07s] Alrighty, welcome back. And Lauren, you're just on mute there if you want to hop off [163.07s -> 165.31s] of me to do chatting. Yeah, sorry. [165.31s -> 169.31s] No worries. It's 2026 and I still don't know how to use Zoom. [171.79s -> 175.15s] Okay, so I assume you can see my screen. Yes, yeah, we're good. [175.87s -> 182.19s] So yeah, today, yeah, I think the big theme for me as I've been using agents to write code, [182.19s -> 187.31s] and I'm sure a lot of you have had the same experience as well, is how do you trust it? [187.31s -> 192.51s] You know, especially if you are an engineer that's been writing code for a very long time, [192.51s -> 198.27s] you have a lot of opinions and lessons that you've learned about doing good engineering. [198.99s -> 203.87s] And when you see agents just, you know, winging it and, you know, guessing, hallucinating, [204.83s -> 210.35s] you know, confidently stating that they found the smoking gun for the hundredth time, [210.99s -> 214.75s] but it's actually not the real problem, you lose a lot of trust. [214.75s -> 217.79s] And when you lose, when you don't have much trust in your agents, [218.91s -> 221.39s] I feel like you really can't get the most out of them. [222.19s -> 224.99s] And for me, the parallel is like with management. [225.79s -> 231.31s] So if I'm in a manager, an engineering manager for a team, and I have a bunch of, [231.31s -> 236.43s] you know, I have a team of engineers on my team, and I don't trust them, [236.43s -> 240.91s] then the mode of operation I'm going to be in is going to be like micromanagement, right? [240.91s -> 247.07s] I'll have to spend a lot of time looking over my reports shoulders and checking that they're [247.07s -> 250.67s] doing their work well, you know, that they're not shipping bugs to production. [252.85s -> 258.61s] And so I drew this chart because it's not, it's not a very scientific chart, but like, [258.61s -> 263.65s] this is how I imagine myself and my journey through using agents. [264.69s -> 274.37s] So, you know, like fast forward or back forward, or fast backwards, like a year or so when, [274.37s -> 277.57s] you know, nobody was, or not many people were using agents to code. [278.85s -> 283.17s] I think you, you know, get into this mode where you are [283.89s -> 289.41s] in, very heavily in the loop with one or several, like a handful of agents, [289.97s -> 294.77s] and you find yourself just constantly trying to understand what your agents are doing. [295.57s -> 297.17s] And you're very, very in the loop. [297.17s -> 299.17s] You're watching every single output. [299.17s -> 305.89s] You are sitting there prompting, and you really can't parallelize beyond that because [306.45s -> 308.45s] again, you don't have that trust, right? [308.45s -> 312.13s] You can't go to a hundred agents, like spawn a hundred agents, [313.09s -> 315.57s] when you don't even trust the output of one agent. [317.25s -> 323.49s] So over the past five months, I feel like I've really been able to like ascend this [324.37s -> 325.73s] trust curve. [325.73s -> 331.17s] And now I'm at the point where I actually have, this sounds kind of scary to say this, [331.17s -> 334.29s] and it makes me sound like a slop artist, but I promise I'm not. [335.01s -> 341.49s] But I actually have my agents now auto-merging PRs for me, which is like a wild thing to say. [342.13s -> 347.49s] But like I woke up today, and there were like 20 PRs landed, and I just reviewed them [348.13s -> 351.17s] on main, like they were already landed, and they were good. [352.37s -> 354.37s] So how did I get to that point? [354.37s -> 357.09s] It's basically what I wanted to talk about today. [358.93s -> 361.97s] And again, like, yeah, feel free to jump in if you have questions, Colin. [363.65s -> 369.57s] But oh, yeah, of course, I got to show this chart where, [370.05s -> 378.93s] uh, do not trust, someone requested to control my computer, probably won't do that. [379.81s -> 385.25s] But yeah, so this chart, I think, I'm sharing this chart not to kind of like flex, [385.81s -> 388.45s] but to kind of show like the journey. [388.45s -> 392.37s] Like, so you can see like the curve, like it sort of like inversely matches [392.37s -> 395.33s] the contributions I've been able to land at Cursor. [396.21s -> 400.37s] So I joined five months ago, and five months ago, like I said, you know, my first month, [400.37s -> 404.77s] I was like, not very productive, because I was, you know, I was learning the code base, [404.77s -> 406.37s] didn't know what the heck was going on. [407.57s -> 413.65s] And as I got more confident in my Asians, I've really been able to kind of ramp up [413.65s -> 414.61s] my productivity. [415.41s -> 421.09s] And again, like, yeah, like last month, I shipped 1000 PRs, which is ridiculous. [422.05s -> 428.05s] And then this month, we're only on the 12th, I'm already at like almost 800 PRs landed. [428.85s -> 432.21s] So the velocity is definitely high. [432.21s -> 436.93s] And I'm sure a lot of you will definitely be questioning like how much of this code [436.93s -> 437.57s] is actually good. [438.77s -> 441.49s] And I think, yeah, that's definitely fair to question. [443.65s -> 453.25s] But yeah, I think if you set up your agents well, you can definitely get to a very similar level. [454.61s -> 456.21s] And so I'm going to talk about how we do that. [459.68s -> 464.08s] So for me, I think I'm curious, like, I guess calling your experience as well. [464.08s -> 472.48s] But for me, I think the most important skill that you should have in your toolbox, when [472.48s -> 474.48s] you work with agents is verification. [475.52s -> 482.80s] And by verification, I mean the ability for an agent to actually run the code, or take [483.44s -> 491.12s] CPU traces or heap snapshots or, you know, open an iOS simulator, whatever, you know, [491.12s -> 499.04s] however your application is exposed to your users, it can do the same thing and run it [499.04s -> 501.84s] for real and actually test and verify it don't work. [502.80s -> 505.20s] Because that's the thing that really closes the loop. [506.08s -> 511.60s] It doesn't guarantee your agent writes good code, but it allows them to at least write [511.60s -> 517.60s] correct code, which is a really big step forward for being able to trust your agent. [521.04s -> 526.24s] I can share one example that we have within Cursor. [528.72s -> 532.48s] Oops, where let me open this. [543.10s -> 543.60s] There you go. [543.60s -> 552.00s] So for Cursor's agent window, so this is actually an interesting story. [552.00s -> 558.96s] But when I joined Cursor five months ago, they're actually, well, I was supposed to [558.96s -> 559.84s] join a different team. [559.84s -> 562.00s] I was supposed to join like the cloud agents team. [562.80s -> 568.16s] But then since I have a lot of experience working on React and agents window is a React [568.16s -> 577.12s] application, I was asked to basically help out with the agent window work. [578.24s -> 584.16s] But there wasn't really a lot of like skills to help me. [584.16s -> 588.16s] So I just found myself like, okay, agents window is going to launch in like a week, [588.16s -> 588.40s] right? [588.40s -> 594.96s] We have a really tight deadline and there was, you know, I was just sitting there like, [594.96s -> 600.64s] okay, I'm going to open up the Chrome DevTools and just like take a trace, look at it myself [600.64s -> 602.96s] and try to make sense of this flame graph. [602.96s -> 605.68s] And keep in mind, I was just like in my first week. [605.68s -> 609.84s] So I had no idea what I was looking at, no idea what, you know, I mean, I had some idea, [609.84s -> 612.64s] but you know, the code base was completely fresh to me. [614.00s -> 617.44s] And I realized like my agent had no idea either. [617.44s -> 620.40s] You know, like I would take a screenshot of the trail, download trace or send it to [620.40s -> 624.80s] it and it'd be like, yeah, it kind of looks like this, you know, and it would like [624.80s -> 626.96s] confidently state like it's this thing. [626.96s -> 630.16s] And then I try to fix that and turns out that's not the actual thing. [631.68s -> 634.16s] So this was a very, very slow process. [634.16s -> 638.96s] And if you've ever done any like performance work yourself or, you know, just even development [638.96s -> 644.16s] with an agent where you don't have a verification skill, you are the verifier, right? [644.16s -> 645.44s] You're the bottleneck. [646.32s -> 649.92s] You tell your agent to do something and then it goes off and write some code. [649.92s -> 653.92s] Then you open up your, you know, local dev build and then you start to say, oh, you know, [653.92s -> 654.48s] it doesn't work. [654.48s -> 659.60s] Then you have to copy paste screen screenshots or console errors or whatever. [660.40s -> 665.84s] And then your agent like slowly kind of like, you know, works with that and then tries to [665.84s -> 669.76s] understand it and fix the thing. [669.76s -> 672.96s] But then you're constantly just in the loop and being a bottleneck. [672.96s -> 674.56s] So there's really no way to parallelize. [675.36s -> 681.44s] So the control glass skills are like one of the first skills I built for cursor and glass, [681.44s -> 686.08s] by the way, is the code name for agents window that we use internally, but it's just [687.04s -> 687.84s] cursor, I guess. [689.60s -> 694.80s] And so this skill is, I guess that the code itself is not super interesting. [694.80s -> 700.56s] Your agent can very easily make one for you where if you're building an electron app or [700.56s -> 707.84s] a web app or even iOS applications, you can teach your agent how to use like the Chrome [707.84s -> 715.12s] DevTools protocol or through Apple has some utilities as well for running the simulator [715.12s -> 719.20s] and taking traces and controlling programmatic control as well. [720.40s -> 722.56s] So that's really useful. [723.60s -> 729.28s] But one thing I actually want to talk about is the this thing. [731.12s -> 732.00s] You have to read me. [732.96s -> 739.60s] So this skill comes with this very unique feature called or not featured, a unique file [739.60s -> 740.56s] called a feature map. [741.52s -> 744.96s] And so the story then is like I built this skill. [744.96s -> 751.52s] And so now the agent was able to actually run the agent window and take traces and whatnot. [752.96s -> 756.40s] But it had no idea what the agent's window was. [757.04s -> 764.88s] So someone would say, oh, the left side bar is like laggy or something like that. [764.88s -> 768.08s] Or the right side, the PR tab is not working. [768.64s -> 770.88s] And the agent would just be flailing around. [770.88s -> 774.72s] It would spend a lot of time trying to look up the code and where is this feature? [774.72s -> 779.52s] How do I actually get to it on the UI, which made it basically completely useless. [780.48s -> 787.04s] You know, like we would I would run the skill locally and it would spawn a dev build. [787.92s -> 790.00s] But then it just be churning. [790.00s -> 791.68s] I just try to click here. [791.68s -> 793.28s] It wouldn't know how to get to things. [794.48s -> 796.16s] And it was just an awful experience. [797.20s -> 800.16s] So who is putting arrows on my screen? [801.44s -> 805.92s] So yeah, this this feature map has been really useful [806.72s -> 810.40s] because it teaches the agent how to get to all of the features that you have. [811.68s -> 819.52s] And in PSAC, the plugin that I've made, if you search for PSAC cursor on Google, [819.52s -> 820.08s] you'll find it. [821.04s -> 824.72s] But there is a create verification skill in that plugin [824.72s -> 827.60s] where it actually helps you set up something like this for yourself, [828.88s -> 830.08s] including the feature map. [830.08s -> 834.00s] So it will actually explore the code and build up this initial feature map [834.00s -> 837.92s] that tells your agent how to get to all of the different features that you have. [838.88s -> 843.92s] And this is extremely powerful because now that you have these user reports that come in, [845.20s -> 849.28s] you can actually map even like a vague report or even a screenshot. [850.08s -> 856.00s] So we have this internally at cursor where we have a Slack channel with, you know, [856.00s -> 860.88s] lots of people giving us feedback on the agent's window and rock bar and whatnot. [861.60s -> 867.28s] And oftentimes the report is very bad, like very low quality, like someone will just put, [868.16s -> 871.44s] very often we get like a screenshot and then someone just says question mark, [871.44s -> 873.04s] question mark, question mark, like, what is this? [873.84s -> 878.40s] And, you know, like without this, your agent is like, I have no clue, right? [878.40s -> 883.68s] But with a feature map like this, it has a lot more context and understanding [884.32s -> 888.00s] of how to actually navigate, how to get to all of the different features. [888.88s -> 892.64s] So like, you know, example, like, I guess like the sidebar, like, what is the sidebar? [893.84s -> 896.88s] You know, like all the different sub features that are present in it. [898.40s -> 902.08s] Like from the user point of view, here's how to get to it. [902.08s -> 903.68s] All the different keyboard shortcuts. [905.20s -> 907.68s] Even like the, what do you call it? [907.68s -> 912.72s] The DOM elements or yeah, like the attributes that you use for selecting things [913.36s -> 916.08s] through the CDP are all there. [916.72s -> 922.64s] So again, yeah, this is like really, really powerful for agents. [923.44s -> 929.04s] And P-Stack ships that create verification skill, but also a maintain verification skill. [930.00s -> 931.76s] So you can keep this up to date. [932.72s -> 933.04s] Cool. [933.04s -> 933.20s] Yeah. [933.20s -> 934.32s] I was just going to ask how you created that. [934.32s -> 939.04s] So do you mind sharing a little bit more about that process in the context of P-Stack [939.04s -> 941.28s] and maybe just what P-Stack is for the folks who aren't familiar? [941.52s -> 942.08s] Yeah. [942.08s -> 947.04s] So P-Stack is pretty interesting because, well, first of all, the name is kind of goofy. [947.52s -> 957.52s] Like the P, the P in P-Stack is like potato, potato snack, because I, so there's a pretty [958.56s -> 962.64s] famous person, Gary Tan, who is the CEO of Y Combinator. [963.20s -> 967.68s] And he's come up with this plugin called G-Stack, Gary Stack. [967.76s -> 972.08s] And funnily enough, we share the last name. [972.08s -> 978.00s] We have no relations, but I thought it'd be funny to kind of poke fun at Gary and make [978.00s -> 987.52s] P-Stack, my version of his plugin, but kind of tailor it to my own set of engineering practices. [989.20s -> 993.68s] But I honestly actually never set out to build P-Stack, but it just started with a bunch of [993.68s -> 994.64s] skills, right? [994.64s -> 998.96s] Like I started with that control glass skill, and then I started with another skill like [998.96s -> 1003.84s] called how, which I also noticed through like observing agents. [1005.36s -> 1008.64s] So like, you know, in the early days of me, you know, trying to climb this ladder, [1008.64s -> 1014.40s] I was like super in the loop and I was basically nitpicking my agents to an extreme degree. [1014.40s -> 1019.60s] I was like, I would tell it, you know, this feature has stopped working. [1019.60s -> 1020.96s] Here's a bug report. [1020.96s -> 1022.16s] Like, why isn't it working? [1024.24s -> 1029.20s] And very often the agent would just like confidently state like, oh, it has to be this, [1029.20s -> 1029.44s] right? [1029.44s -> 1030.40s] It has to be this thing. [1031.12s -> 1036.00s] And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually reading [1036.00s -> 1039.36s] the code that I thought should be affected. [1039.92s -> 1042.08s] And that made me just extremely suspicious. [1042.08s -> 1046.16s] And at that point else is that I'm not going to, I can't trust any, this agent anymore [1046.16s -> 1048.40s] because it's just, it's just completely hallucinating. [1049.12s -> 1055.52s] And I think, I think it's very easy to just, you know, like build up that distrust and [1055.52s -> 1057.92s] not, and kind of feel helpless. [1057.92s -> 1060.96s] Like, you know, you don't know how to help your agents succeed. [1061.68s -> 1065.84s] But like, again, I think the, the, the management analogy is super helpful because like, [1065.84s -> 1071.36s] imagine if you were a manager of an engineering team and you had an engineer in your team [1071.36s -> 1074.56s] who was a really good coder, no business context whatsoever. [1074.64s -> 1079.52s] You know, they, they just, you just hired them and they onboarded, you know, like five [1079.52s -> 1080.16s] seconds ago. [1081.60s -> 1084.72s] And so how do you actually teach that person to be effective? [1085.76s -> 1090.16s] So how you do that is through a skill, a skill being just, you know, it's just markdown, [1090.16s -> 1090.56s] right? [1090.56s -> 1097.60s] But, you know, it encodes a lot of information, instructions, a lot of, you can really draw [1097.60s -> 1103.44s] out a lot of intelligence from an agent by, well, some people on Twitter call it like, [1103.44s -> 1107.92s] you know, pull the agent to a different latent space, which is kind of like a fancy way of [1107.92s -> 1112.80s] just saying, like, since, you know, LLMs are sort of like, they predict the next token. [1114.16s -> 1119.04s] When you give it some high quality tokens to begin with, then, you know, it, it can kind [1119.04s -> 1123.36s] of pattern match on like a higher space that's, you know, smarter. [1125.44s -> 1128.88s] So that's like a very interesting model there. [1128.88s -> 1131.76s] But yeah, I built P-Stack very, very incrementally. [1132.64s -> 1137.92s] So I started with just really observing how agents, you know, all the different failure [1137.92s -> 1140.80s] modes of, of that agents were having. [1140.80s -> 1144.16s] And every time I saw that, I just, okay, I'm just going to make that a skill, right? [1144.16s -> 1149.44s] Like, stop hallucinating, actually go and search up, look up the code, use a lot of [1149.44s -> 1150.08s] subagents. [1151.12s -> 1153.28s] And yeah, stop guessing. [1156.52s -> 1157.32s] Yeah, that makes sense. [1157.32s -> 1160.20s] One, one kind of follow up question here, both from myself and from a bunch of people [1160.20s -> 1160.68s] in the chat. [1160.68s -> 1163.24s] So I guess it's two, two parts. [1163.24s -> 1165.24s] So one is like, how do you maintain these skills? [1165.24s -> 1167.00s] So like the product changes over time. [1167.00s -> 1169.56s] Obviously there's a lot of people are shipping against the code base. [1169.56s -> 1171.40s] So how do these skills get maintained? [1172.04s -> 1176.20s] And then second to that is like, how do you know when your verification is good enough? [1177.08s -> 1180.92s] Like in, you know, you can trust that the verification loops that you've built are [1180.92s -> 1184.20s] going to, I guess you trust the outputs when they're done. [1187.38s -> 1191.46s] Yeah, maybe I'll talk about, I think I was somewhat related. [1191.46s -> 1192.82s] Maybe I'll start with this one first. [1193.62s -> 1196.98s] So like, how do I maintain these skills? [1198.82s -> 1204.42s] So if you're not familiar with this concept and eval is essentially like a way to, [1205.78s -> 1209.22s] well, I think the mental model I have is like, it's like a unit test for an agent. [1210.66s -> 1214.10s] And you can actually make your own evals. [1214.10s -> 1216.50s] You don't need like a special framework for them. [1216.50s -> 1221.62s] You can build, you can, you can build one depending on like, you know, how scientific [1221.62s -> 1223.06s] and how rigorous you want to be. [1224.34s -> 1225.38s] My screen is red. [1226.26s -> 1226.42s] Yeah. [1226.42s -> 1227.38s] There's a little button. [1228.34s -> 1228.66s] Sorry. [1229.86s -> 1231.70s] Like disabling the drawing or something. [1231.70s -> 1232.74s] I can't see my screen. [1233.30s -> 1233.62s] Yeah. [1233.62s -> 1234.02s] Sorry. [1234.02s -> 1235.78s] If you guys could not draw on the screen, that'd be great. [1235.78s -> 1239.22s] But there's a little button in the troll. [1239.22s -> 1239.46s] Yeah. [1239.46s -> 1240.82s] The little drop down. [1242.34s -> 1243.14s] How do I clear? [1244.50s -> 1244.74s] Yeah. [1245.94s -> 1246.42s] You got it. [1246.42s -> 1246.74s] Perfect. [1247.54s -> 1254.50s] Um, yeah, so evals are a way to unit test your skills basically. [1255.14s -> 1258.82s] And actually in P stack, we ship under potato moon. [1258.82s -> 1259.70s] There's a playbook. [1259.70s -> 1261.46s] If you search for it called eval playbook. [1262.66s -> 1269.14s] And it's, uh, uh, it's like not, it's actually pretty, pretty rigorous the way it's done. [1270.10s -> 1274.90s] But, um, essentially what I do is I spawn a lot of different subagents. [1274.90s -> 1282.26s] I have like my main coordinator agent, uh, come up with a rubric for, uh, what I want [1282.26s -> 1283.62s] the skill to do. [1283.62s -> 1289.86s] Um, and then it spawns all these subagents and it, it creates individual directories [1289.86s -> 1295.94s] for them, uh, which are cleverly named to not let the sub agent know that it's being [1295.94s -> 1299.94s] evaluated because, uh, agents can actually tell. [1299.94s -> 1301.86s] And when they do, they change their behavior. [1302.50s -> 1304.74s] Uh, but it does a bunch of stuff like that. [1304.82s -> 1312.10s] To, um, essentially, yeah, like test whether or not the skill I'm making or changing is [1312.10s -> 1314.50s] actually doing what I think it does. [1314.50s -> 1319.38s] Um, and one of the really nice things about cursor is that we are, we support so many [1319.38s -> 1320.02s] different models. [1320.66s -> 1325.30s] So you can actually eval your skill across all sorts of different models. [1325.30s -> 1331.70s] Um, and you know, get a sense of how well it performs across that different matrix, [1331.70s -> 1333.78s] um, especially for the models that you use. [1335.30s -> 1336.74s] Uh, so I do this a lot. [1336.74s -> 1342.26s] Every time I modify a skill, I will run one of these, uh, like the eval playbook, uh, [1342.26s -> 1346.34s] and make sure that, you know, it's actually leading to a result I want. [1347.06s -> 1352.42s] Uh, but I will say like maintaining skills is actually pretty hard. [1352.42s -> 1355.94s] Uh, it requires, I think, a lot of taste and observation. [1356.74s -> 1361.54s] So you kind of need to be very good at being a backseat driver. [1361.54s -> 1362.10s] You know what I mean? [1362.10s -> 1366.18s] Like if you do pair, if you've ever done pair programming, for example, uh, and you [1366.18s -> 1370.18s] watch a coworker code and you're just like, you could probably do this better. [1370.18s -> 1372.50s] You know, you could do it, you know, like, why did you not do this? [1372.50s -> 1372.74s] Right. [1372.74s -> 1376.74s] You, you ask a lot of questions to your coworker and it's kind of a similar thing here. [1376.74s -> 1379.94s] You like, you don't want to just be a passive observer of the agent. [1379.94s -> 1384.34s] You want to be very in the driver seat in the initial stages when you're building up [1384.34s -> 1385.54s] your own set of skills. [1386.26s -> 1389.94s] Uh, you know, obviously you can use something like PSAC, but if you're building your own [1389.94s -> 1395.06s] set of skills, it's very, I think, you know, opening up the, all the tool calls and like [1395.06s -> 1401.30s] reading the code and reading all the, uh, the agent behavior and their thinking blocks [1401.30s -> 1405.62s] is a really great way to see where they, they fail, right? [1405.62s -> 1408.98s] Like what, what, you know, where are they being done? [1408.98s -> 1410.66s] And then you can go and build a skill for that. [1411.54s -> 1416.42s] And then with verification, how you trust it is it's, I think it's also a very similar [1416.42s -> 1422.02s] iteration loop, uh, where, you know, like I actually did the same process for verifying [1422.02s -> 1428.98s] the verification skill where I actually get, um, so one thing that's interesting about evals [1428.98s -> 1435.22s] is that you can sort of hill climb them, meaning that, uh, your eval can produce a score, [1435.22s -> 1435.54s] right? [1435.54s -> 1441.22s] Uh, a score that you can get your coordinator to produce, uh, but also, uh, you can have [1441.22s -> 1447.38s] a judge agent of a different model to, uh, kind of cross-reference and make sure that [1447.38s -> 1450.42s] the first model is not being biased, right? [1450.42s -> 1454.66s] The model that's judging all of the subagents that are running the thing, uh, but you can [1454.66s -> 1456.02s] also like hill climb. [1456.02s -> 1461.46s] So meaning that you can, you can use like slash loop in cursor and you can say, okay, [1461.46s -> 1464.10s] keep looping on this eval, right? [1464.10s -> 1467.06s] Until everything is 10 out of 10 as an example. [1467.54s -> 1471.22s] Uh, and I did the same, the basically the same approach with the control skill. [1471.22s -> 1474.42s] And so I kind of, it was very, it was very hands off actually. [1474.42s -> 1479.94s] Uh, so, you know, I, uh, I kind of built, I built that skill that way, like the CLI [1479.94s -> 1481.14s] in that skill. [1481.14s -> 1485.06s] Um, and over time it's gotten really good. [1485.06s -> 1488.50s] Uh, but yeah, it was definitely not super smooth at the beginning. [1488.50s -> 1495.38s] It required a lot of iteration and I think there's an analogy here for me, which is, [1495.38s -> 1500.66s] um, well, I make this analogy later in a different slide on my drawing here. [1500.66s -> 1507.78s] Uh, but I think of it like, uh, you know, as a, as a engineer now, you're sort of more [1507.78s -> 1514.74s] like, uh, like maybe a manager or the analogy I like is like, you're like a chef in a restaurant. [1515.30s -> 1517.06s] Uh, you know, you're the head chef. [1517.06s -> 1519.70s] Uh, you're not cooking all the food yourself anymore. [1519.70s -> 1521.14s] You have a team of cooks, right? [1521.14s -> 1525.22s] You have a line cooks, you have a sous chef, you know, you know, all these different stations. [1526.18s -> 1529.62s] Um, and it's your job to really design the environment. [1529.62s -> 1532.10s] You know, you, you, you're in charge of setting up the kitchen. [1532.10s -> 1536.26s] You're in charge of, you know, like giving tasks to different people. [1536.98s -> 1542.98s] So, um, yeah, it's a very interesting way of working. [1543.70s -> 1548.18s] Uh, but yeah, that's, that's how I've basically built, uh, these verification skills. [1548.74s -> 1549.06s] Yeah. [1549.06s -> 1550.18s] Just, just one fault there. [1550.18s -> 1552.58s] I'm like, I had to go try to go one layer deeper. [1552.58s -> 1559.06s] So are you, let's say we wanted to build, um, uh, an eval or a skill for, for something [1559.70s -> 1563.62s] and we wanted to kind of get better on its own, which is, is what I think you're suggesting. [1564.34s -> 1569.22s] Are you doing that in like a work tree kind of isolated with like the subagents and then [1569.22s -> 1574.58s] the reviewer agent and all that, is it happening like in some type of cloud hosted environment? [1574.58s -> 1576.50s] Like what's the, the more, the practical steps. [1576.50s -> 1580.26s] If I wanted to go do this, uh, and like set up a verification system for something, [1580.26s -> 1581.86s] what would I, what would I do or where would I start? [1584.39s -> 1589.75s] Um, I think that, uh, the best place to start is local because you can observe, [1590.95s -> 1592.87s] you can definitely observe what your agents are doing. [1592.87s -> 1598.23s] So, uh, if you're building a verification skill for yourself, uh, I would definitely start local [1598.23s -> 1604.87s] and just have your agent bring up the application, whether it's like a CLI or a desktop app or [1604.87s -> 1605.27s] whatever. [1605.83s -> 1607.35s] And so you can actually observe, right? [1607.35s -> 1612.47s] You can see how the agent is interacting with the, the application. [1612.55s -> 1616.15s] You can see it, you know, how it calls like the different [1616.15s -> 1620.55s] APIs that allow it to interact with the, uh, the application. [1622.15s -> 1628.15s] Um, but, uh, for me personally, uh, I have basically been kind of all in, [1628.15s -> 1631.35s] mostly all in on cloud agents because they're extremely powerful. [1632.07s -> 1636.87s] Uh, and the really powerful thing about cursor is the, the cloud agents actually, [1637.43s -> 1639.83s] where if you spend a little bit of time setting up your environment, [1639.83s -> 1645.19s] then these control skills, these verification skills pay a huge amount of dividends [1645.91s -> 1651.27s] because it's not just something that makes you as a single engineer better. [1651.27s -> 1656.23s] It actually levels up your whole team, uh, and even your whole company, because, uh, [1656.23s -> 1660.23s] you can actually start thinking about cloud agents and start thinking about automations [1660.23s -> 1662.79s] that automatically do things. [1662.79s -> 1667.99s] Like, uh, I'll, I get, I kind of talk about this a bit later, but I'll just kind of [1668.95s -> 1673.11s] get into it, uh, where, where, you know, for example, like I talk a lot about this [1673.75s -> 1678.31s] agent we have called Benny, right? Who, who, uh, you know, takes all of the [1679.11s -> 1684.47s] bug reports that we get and it automatically goes off in the cloud, opens up a cloud, uh, [1684.47s -> 1690.79s] it's, you know, it's desktop, it runs cursor in its own computer and it uses the same [1690.79s -> 1696.39s] control skills to interact with the application and try to reproduce the bug, uh, or the user [1696.39s -> 1702.39s] report. And this is so, so powerful because at once I can immediately, I get so much [1702.39s -> 1707.11s] information from this automatically. Like here in this example, you can see that, uh, the [1707.11s -> 1712.23s] Benny actually reproduced the bug, uh, but it's already fixed on main. [1713.11s -> 1717.43s] So it actually confirms that we fixed this problem already and all I need to do is just [1717.43s -> 1723.19s] release another build of, of cursor. Uh, so that's like huge information there that I [1723.27s -> 1726.95s] didn't have to go off and sit with an agent, you know, and spend an hour trying to fit, [1726.95s -> 1732.23s] like, is this fixed? Is this not fixed? So you, you, you gain back so much time. Uh, [1732.23s -> 1736.47s] but you know, everybody on my team benefits from this, everybody in the company benefits [1736.47s -> 1743.91s] from this. Uh, so definitely think that, uh, you know, keeping these, uh, using cloud agents [1743.91s -> 1749.27s] is super powerful. Uh, but yeah, it's like a journey. You have to trust it first, right? [1749.27s -> 1753.83s] Before you, you get to this point and that's, it goes back to what I was saying here where, [1753.83s -> 1758.95s] you know, it's very hard. It's almost impossible. And I would definitely encourage you not to try [1758.95s -> 1765.19s] to jump from, you know, like if you're still in this zone, you don't want to jump to like, [1765.19s -> 1770.55s] I'm going to spawn a hundred or thousands of cloud agents right now because you're just going to [1770.55s -> 1776.15s] waste a lot of tokens. Um, and it's going to be extremely expensive. Yeah. So just to kind of [1776.15s -> 1780.55s] recap so far, basically the, if we wanted to go on the journey that you've kind of gone on, it [1780.55s -> 1786.79s] would be to start with verification, uh, building some, some skills and some, some ways of determining [1786.79s -> 1790.71s] that the agents are producing at least like correct code, whether, like you said, whether [1790.71s -> 1793.91s] it's good code or not, it's maybe a separate question, but like it's, it's technically solving [1793.91s -> 1798.95s] the problem by looking at, you know, stack traces, looking at, you know, the actual behavior in the [1798.95s -> 1803.91s] app and so on. Um, and then once we trust it locally, then we can start to think about scaling [1803.91s -> 1808.39s] into the cloud and running more agents that are picking up signals, I guess, on their own, [1808.39s -> 1811.91s] right? So whether that's like a bug report that comes in or something, they can go and pick it up [1811.91s -> 1817.43s] and solve the problem and give us back a PR. I think maybe the last step is like auto-merging [1817.43s -> 1822.95s] the PRs, which is where you're at and maybe not where everyone is at and then reviewing them on [1822.95s -> 1827.99s] main. But, um, is that, is that about right? Yeah, exactly. I think, yeah, that's why I drew this, [1828.31s -> 1833.67s] uh, this, this curve, right? Because that, this, this basically describes my journey of, you know, [1833.67s -> 1838.71s] when I started, barely could use a couple of agents and I was just observing every single thing. [1839.35s -> 1844.15s] I think there's really no shortcut for going from here to there because this is really about [1844.15s -> 1849.75s] your personal level of trust in agents, right? Um, obviously, you know, as an engineer, you don't [1849.75s -> 1856.15s] want to just slop code into production. So how do you actually build up that trust takes, um, a lot [1856.15s -> 1863.35s] of, uh, I guess, taste and judgment. Um, but, uh, you know, like, I think plugins like P-Stack [1863.35s -> 1870.23s] definitely kind of help you get, uh, get up to speed much quicker. Uh, and so I guess it's like, [1870.23s -> 1876.15s] if you trust me and you trust P-Stack, then in, by extension, you can maybe trust your agents. [1876.15s -> 1880.71s] But if you don't trust me, and I definitely would not encourage people to blindly trust me, [1881.67s -> 1886.95s] uh, uh, you know, if you build up your own set of skills that you can obviously, you know, [1886.95s -> 1891.99s] take a look at P-Stack and kind of fork it, make it your own, improve the skills, definitely [1891.99s -> 1897.75s] encourage that. Uh, but for me, it's really all about, it just keeps coming back to trust. [1897.75s -> 1903.19s] You know, every one of us here in this chat have a different standard for engineering. Uh, and there [1903.19s -> 1910.15s] are different things that are important for us in our code base. And, uh, when you are able to encode [1910.23s -> 1915.03s] all of that into skills and you can verify that your agent is actually doing them, that allows [1915.03s -> 1922.63s] you to really kind of ascend this curve and, um, uh, you know, start automating things. Uh, there's [1922.63s -> 1927.83s] another piece I wanted to talk about, um, if there's... Yeah, go for it. I'll pick up more [1927.83s -> 1931.75s] questions as I go, but yeah. I think there's a third part to this, which I haven't talked about [1931.75s -> 1938.39s] yet, which is kind of an interesting one, which is like refactoring and rewriting, like one of the, [1938.39s -> 1943.99s] uh, I guess most controversial, one of the most controversial topics in the industry, I think is [1943.99s -> 1951.11s] like, should you rewrite your app or not? Um, because I think engineers are very prone to this [1951.11s -> 1955.59s] where, especially when you join a company, you come in and you see like the code base and you're like, [1955.59s -> 1960.63s] man, this is shit. Like who wrote this code? You know, it's terrible. I want to rewrite the whole [1960.63s -> 1967.43s] thing. There is a very common inclination and I think a lot of, you know, before agents, um, [1967.43s -> 1972.47s] and I guess arguably even now people will definitely discourage you from rewriting stuff, [1973.27s -> 1978.71s] but I'm actually here to make a case for why you might want to consider it. Um, because [1980.71s -> 1986.15s] I think it really depends. Uh, you know, uh, Brownfield applications, I think are actually [1986.15s -> 1991.83s] in a pretty good spot, especially if they're set up well already. Uh, and like recently I've been [1991.83s -> 1999.35s] talking to some people, but, uh, you know, I was just observing, I just noticed this parallel, [1999.35s -> 2005.99s] which is that a lot of big tech company problems are now everybody's problems. Um, and the big tech [2005.99s -> 2010.87s] company problem, you know, like when I was working at Meta, like we had this giant monorepo, we had [2010.87s -> 2016.31s] like, I don't know, tens of thousands of engineers just, you know, like banging on their keyboards [2016.31s -> 2024.71s] and shipping code. And a lot of really great engineers at Meta. Uh, but, uh, I'll say like, [2024.71s -> 2029.51s] you know, you'll be surprised that the code quality is actually not that good. Um, and so [2030.23s -> 2035.91s] I often joke that like, you know, before AI sloth, we had human sloth. Um, and so, uh, you know, [2035.91s -> 2041.75s] I think a lot of big tech info, like, uh, like what Meta has or Google, you know, you know, [2041.83s -> 2046.63s] really big tech companies are actually designed for that, where you're sort of like, you're [2046.63s -> 2051.83s] catering to the, the, you know, like, uh, that sounds so bad to say, but like the, the least [2051.83s -> 2057.43s] capable engineer on your team, right? You build, you build frameworks, you build conventions, [2057.43s -> 2062.07s] you build guardrails, you know, you restrict credentials so that, you know, your intern [2062.07s -> 2069.27s] doesn't wipe your production database. Um, there's, uh, you know, if you have that level of [2069.27s -> 2074.79s] info already, I think your agents can actually already do a very solid job, right? Because they [2074.79s -> 2081.83s] have the, the guardrails are already in place for agents to not cause havoc or not cause too much [2081.83s -> 2088.39s] havoc, uh, in your code base. Um, and you can always add more, you know, guardrails. Uh, but I [2088.39s -> 2092.71s] think like greenfield applications, especially are, you know, like the brand new applications [2092.71s -> 2099.19s] are like the biggest risk in my opinion, uh, and also the greatest opportunity because, you know, [2099.19s -> 2105.27s] if you vibe code a project, uh, a prototype, um, like we did for Grokbot, you know, Grokbot [2105.27s -> 2110.47s] was spun up very, very, very quickly. Um, and if you, uh, if you haven't heard of Grokbot, it's [2110.47s -> 2115.67s] like our new application we just launched yesterday. Uh, it's, it's really cool. Uh, [2115.67s -> 2121.35s] lets you orchestrate your, create like individual agents that have their own identity and you can [2121.35s -> 2125.75s] kind of orchestrate them. It's super cool. Definitely check it out. Um, but yeah, that was, [2125.75s -> 2130.71s] it's like a very, it was a very greenfield application, like most prototypes are. So [2130.71s -> 2137.11s] it was like vibe coded very quickly. Humans were not reading the code at all. And, uh, [2137.11s -> 2143.83s] I had this tweet recently, uh, where I said something about organic architecture. Um, [2145.27s -> 2153.43s] maybe I'll find it. Uh, but the idea is that, uh, when you have a completely vibe coded application, [2153.43s -> 2157.67s] you essentially have no guardrails whatsoever. So, uh, your agents, [2159.51s -> 2164.31s] when you give them a task, they will just solve it in whatever method is the most convenient. [2165.11s -> 2171.19s] And over time you get into this, uh, situation where you have a code base that is spiraling out [2171.19s -> 2177.19s] of control because you don't understand it. Uh, your agents understand it, I guess in a way, [2177.19s -> 2182.95s] but like they've built something that is, you know, optimized for short, for shortcuts. Uh, [2182.95s -> 2188.47s] and, uh, you know, it will, you will suffer, you have a lot of issues with that application. [2190.31s -> 2195.91s] Uh, so I think starting your code base with, uh, like very strong constraints is [2197.75s -> 2202.95s] very much needed. Uh, because like when you have a code base that you can trust, [2202.95s -> 2209.59s] right? When you have guardrails that actually help you, uh, uh, help your agents write good code, [2209.59s -> 2215.19s] you can get into the, you know, like into this part of the curve where I, I, where I said, [2215.19s -> 2221.43s] you know, I woke up today and I had like 20 PRs merged, uh, by my agents. And that's because I [2221.43s -> 2229.91s] invested a lot, a lot of time, uh, over 600 PRs. I, I calculated yesterday, uh, when I refactored [2229.91s -> 2237.03s] all of Grokbot to this new architecture that I've been building. Um, and yeah, I've gotten to a [2237.03s -> 2243.99s] point where I, I don't really look, I really don't look at the code anymore. And, um, I say that not [2243.99s -> 2249.91s] just, you know, to sell you tokens, but because I, you know, it took a lot of work to get to that [2249.91s -> 2254.95s] point. I spent a lot of tokens to get the code base to this point where I no longer have to look [2255.03s -> 2261.35s] at it. Uh, but I'm very excited because, you know, the potential where, you know, it's not just, [2261.35s -> 2267.03s] it, this doesn't just benefit me. It benefits everyone contributing to Grokbot. And it also [2267.03s -> 2273.51s] empowers, you know, designers and product managers and, you know, people, even GTM people to add [2273.51s -> 2278.71s] features to Grokbot. And I don't have to worry, you know, I don't have to wake up at night in, [2278.71s -> 2282.71s] in the middle of the night and worry like, oh shit, someone's just merged a perf regression. [2282.71s -> 2288.23s] Right. I have a ton of constraints and CI. I was like, it's actually very annoying to write [2288.23s -> 2295.03s] code in, in Grokbot, but like agents absorb all of that annoyance. Um, but yeah, I'm happy to talk [2295.03s -> 2303.03s] about what exactly that is. Um, yeah, I think one question, um, before we get into the, this part [2303.03s -> 2307.99s] here is just around that element of like what your, your, your CI looks like, or maybe some [2307.99s -> 2312.31s] of the constraints and then also like the average PR size. I saw a question about that earlier, [2312.31s -> 2316.15s] just to give people a, you know, kind of a glance. It doesn't have to be like mathematically [2316.15s -> 2322.23s] average, but just a, you know, like what generally the size of the PR is. Um, if it's only a couple [2322.23s -> 2332.55s] lines of code or, you know, um, yeah. Um, I think it depends on me. I'm trying to do this in a way [2332.55s -> 2336.87s] where I'm not gonna like, yeah, you don't have to share the actual number of you'd like an actual [2337.59s -> 2345.35s] this is fine, but like we have, so, okay. This is not that interesting, but, uh, well, fun fact [2345.35s -> 2353.83s] is that virtualization in Grokbot and in, uh, cursor is actually powered by a pretext, uh, [2353.83s -> 2359.83s] which is a sort of new library that someone's built. Um, that's really interesting. You should, [2359.83s -> 2365.35s] you should check it out, but that's not really that important. Uh, I think the average PR size, [2365.35s -> 2371.03s] I actually don't know. I probably, I don't know if I want to click on these. Uh, I probably can, [2371.03s -> 2377.35s] but I would say like they can range anywhere from a few hundred lines or 50 lines to like a thousand, [2377.35s -> 2383.03s] depending on what the thing is doing. Uh, so like here I'm actually like deleting a bunch of files. [2383.03s -> 2388.47s] So I expect that it's just, it's like mostly deletion. Uh, but yeah, it, it kind of varies. [2389.67s -> 2393.27s] There's no, like, yeah, there's only hard cap or are they, they're all like 50 line. [2394.15s -> 2397.83s] Yeah, there's definitely no hard cap, but I do encourage my agents to split up their work [2397.83s -> 2406.95s] into multiple PRs. Uh, I do that mostly because, uh, I like, I like the idea of the, I guess maybe [2406.95s -> 2411.27s] this is much harder to do now as in the world of agents and you have like so many commits, [2411.83s -> 2415.51s] but I like the idea that, you know, the Git history is a very rich source of context. [2416.15s -> 2423.99s] Uh, and I like the, I like each PR to sort of atomically describe what that small piece of [2423.99s -> 2428.95s] thing is doing, which also makes it easier for me to revert changes and like figure out, you know, [2428.95s -> 2435.91s] oh, I shipped a bug and it's just, it's here. It's not in this 40,000 line PR where who knows what [2435.91s -> 2441.75s] landed in there. Uh, but yeah, I don't have a hard cap on PR size. [2442.71s -> 2447.19s] Cool. And then, um, yeah, also quick question on like CI. So again, you don't have to go into [2447.19s -> 2451.75s] like, uh, the screen share of like your, your CI does, but just generally would you describe [2451.75s -> 2460.07s] what the CI kind of looks like, uh, or how strict it is? Uh, yeah. So, uh, well specifically for [2460.07s -> 2466.15s] Rockbot. So Dune is a, it's the sort of cheeky code code name for the architecture that we've [2466.15s -> 2474.07s] built for Rockbot. Um, the CI looks pretty annoying because there's checks for everything. [2474.71s -> 2480.23s] So like literally I have, um, uh, well, if you've written any React, for example, you know, [2480.23s -> 2486.79s] you know that one of the biggest foot guns in React is a use effect. Uh, so in, uh, [2486.79s -> 2493.59s] Dune and in Rockbot, we've banned use effect. So Dune is just the mental model of what Dune is. [2494.23s -> 2499.99s] Uh, you can kind of think of it as like Next.js for, uh, Electron apps and it's designed for [2499.99s -> 2506.71s] agents to write, uh, and it's like custom for, you know, our agent powered applications. Um, [2507.35s -> 2511.99s] so the CI checks are very like specific to that, like, you know, don't use use effect. It's, it's, [2511.99s -> 2518.07s] it's banned, like CI will fail, uh, and yell at you. We have like some of the more interesting [2518.07s -> 2522.55s] ones that people might raise eyebrows is like, I actually banned code comments as well. [2523.27s -> 2531.03s] Uh, which is very interesting. Um, but I've noticed that 99% of the time agents just write [2531.03s -> 2536.15s] code comments that kind of describe some historical thing that is actually totally irrelevant to the [2536.15s -> 2541.59s] code. Um, like it will often say like, you know, Oh, Lauren said you should never do this. And it's [2541.59s -> 2546.63s] now in the, in a code comment. Like what, like why, what that was, I didn't say that as like a [2546.71s -> 2552.23s] durable, you know, global rule. I just met like your, this PR sucks and you should change that [2552.23s -> 2559.99s] part. Uh, agents don't really understand us that well, surprisingly. Uh, and, or they kind of assume [2559.99s -> 2566.39s] too much and you kind of do things in like very stupid ways. So like, yeah, we just ban everything, [2566.39s -> 2573.19s] everything you can imagine, like the agents are bad at, we ban. Uh, so one example that we actually [2573.19s -> 2578.95s] suffer a lot in the agent's window is we have, uh, you know, if you've used agent's window, [2578.95s -> 2582.71s] you've definitely seen performance issues and, you know, we're constantly trying to fix them. [2583.27s -> 2589.19s] Uh, but it's like a, it's a never ending struggle because there's so many pull requests that get [2589.19s -> 2595.43s] merged. Every, any one of them could just regress performance or stability or reliability. Uh, you [2595.43s -> 2599.59s] know, the agent's window doesn't have this architecture yet. I plan to do, bring this [2599.59s -> 2606.31s] learning back there and kind of refactor everything there. Uh, but, uh, it just regresses [2606.31s -> 2613.83s] super often, uh, because, uh, there's just one example is like, we have very poor, um, isolation [2613.83s -> 2618.71s] between processes. So like on, you know, on, on electron, you have a renderer thread that renders [2618.71s -> 2623.59s] your UI, but you also have like a main thread that you can run other code that, you know, doesn't need [2623.59s -> 2630.95s] to block the renderer. Um, but we do a poor job of separating those things. And so oftentimes [2630.95s -> 2636.39s] you just accidentally have code that gets pulled into running on the render thread. And then all [2636.39s -> 2641.75s] of a sudden you're competing with the renderer that, you know, that has a very, if you want like [2641.75s -> 2648.55s] 60 FPS, you have to, every frame that gets drawn has to be done in 16 milliseconds. So very, very [2648.55s -> 2654.47s] small, you know, deadline per frame, uh, if you want, you know, a very smooth product. Uh, and when [2654.47s -> 2659.27s] you start building, bringing in, accidentally bringing in, you know, things that are like very [2659.27s -> 2665.51s] computationally heavy or they have a lot of IO, uh, then you just get into like a lot of jank [2665.51s -> 2671.35s] and your, your FPS really drops. You start, uh, you know, losing frames, you get long tasks that [2671.99s -> 2675.43s] take more than 16 milliseconds and you just get this really choppy experience. [2675.59s -> 2680.23s] So all of those patterns that we've learned, basically building Electron apps, we've encoded [2680.23s -> 2685.91s] into this framework and it becomes like a hard failure. So I literally, in, in GROCKBOT, we [2685.91s -> 2693.91s] literally have a directory called Electron main, Electron renderer, and we have, uh, import, uh, [2693.91s -> 2699.19s] CI, I guess, where we actually check the dependency graph to make sure they are not accidentally [2700.15s -> 2708.07s] importing code from one directory to another. Uh, so that's enforced by CI, um, as well as [2708.07s -> 2715.75s] BugBots, uh, which is our, which cursors, um, like code review tool that runs on CI, uh, you know, [2715.75s -> 2722.95s] in our agents MD, it's everywhere. Like, so I, I, I, um, I have this thing here where I talk about, [2722.95s -> 2728.71s] like, um, you know, like there are multiple layers, I think, for building a good code base. [2729.51s -> 2734.23s] Obviously the code base is one where, uh, if you have an architecture like this, [2734.23s -> 2740.31s] where it's extremely strict, uh, you know, the, the, the way to build features is very conventional. [2740.31s -> 2745.83s] That's like the strongest, strongest level of enforcement because agents just love to copy [2745.83s -> 2751.91s] existing patterns. So, uh, one example of this in GROCKBOT is like, we have this, these concepts [2751.91s -> 2756.87s] called like a feature and we have entry points and transcript cards, like, oh, you know, the cards [2756.87s -> 2763.51s] that you see in the chat, these are all like, like nouns, I guess, in, in the framework. And so [2763.51s -> 2770.39s] there's a very conventional way of creating them. And so like a feature is all in a single directory [2770.39s -> 2775.75s] as an example. And so all of the code that contributes to that feature lives in one directory. [2775.75s -> 2780.47s] So it's all co-located in one place, makes it super easy. You know, agents don't have to like, [2780.47s -> 2785.51s] uh, grab around and try to figure out like where all the things are. It just looks at the feature [2785.51s -> 2791.43s] and like, oh, okay, I'm working on the onboarding feature in GROCKBOT. Uh, I'm just going to work in [2791.43s -> 2798.39s] this directory. And for 80% of the work, it's mostly just very encapsulated there. But, uh, [2798.39s -> 2804.23s] like it's like very, it's like designed again for, you know, like the dumbest agent, like you don't [2804.23s -> 2810.23s] have to think, right? The, the, the, one of the key principles I have for this framework is like [2810.87s -> 2818.15s] the shortest, the shortest path is the best path. So, uh, because that plays exactly to how agents [2818.15s -> 2822.39s] love to write code is that they like to take shortcuts really, you know, they'll find the [2822.39s -> 2830.15s] quickest way to solve the problem. So why not make that the best way to solve the problem? Uh, so I, [2830.15s -> 2836.55s] I, I probably won't get into all the specific details. Um, and, uh, the, the, this framework's [2836.55s -> 2840.23s] really more of a collection of ideas and principles rather than something that will [2840.23s -> 2844.95s] open source. Uh, you can, you can, you know, screenshot this, I guess, if you want and, [2844.95s -> 2850.15s] uh, tell your agent to, uh, do something, build, build something like this for you too. [2851.51s -> 2856.79s] Um, yeah, but it's really all about the layers. Uh, you know, like the, the code base is one part [2856.79s -> 2863.35s] with features, uh, and directories and, you know, import or blocking import dependencies [2863.99s -> 2869.67s] that shouldn't be imported. Uh, but, and, and it all enforces that and static analysis. So like, [2870.23s -> 2877.19s] uh, there's CI checks. We have a lot of links for bad patterns that we observe, uh, compiler [2877.19s -> 2884.31s] diagnostics. Uh, there's also rules and bug bot, which are, um, I think like three, four, five are [2884.31s -> 2891.99s] more soft, right? These two actually make, make CI read, right? So that, you know, there's a hard [2891.99s -> 2900.63s] constraint where the agent can just write crappy code for rules and skills and bug bot. Your agents [2900.63s -> 2908.55s] can still forget, right? You can still, or it may not always consistently apply them. So I like to [2908.55s -> 2915.43s] layer them, but I don't, I don't like to rely on them as the only source of enforcement because [2915.43s -> 2919.75s] it's very, very soft, right? And if you, if you only have rules and bug bot and skills and the [2919.75s -> 2925.19s] style guide for your code, you will, it's only a matter of time before your code base looks like [2925.19s -> 2930.95s] complete trash. Um, sorry to say that, but, uh, I definitely recommend, yeah, like, you know, investing [2931.83s -> 2938.15s] in, you know, things that can be hard and forced, right? And this is why, you know, maybe the choice [2938.15s -> 2944.63s] of tech stack that you use is also very important. Um, like I think for example, rust is sort of [2944.63s -> 2950.95s] making, you know, it's like getting super popular again, uh, because the compiler is so strict, [2950.95s -> 2955.19s] right? The compiler enforces so many different things, you know, there's a borrow checker that [2955.19s -> 2961.03s] you have to appease. And as long as you make sure your agents don't write unsafe code blocks, uh, [2961.03s -> 2966.47s] you can more or less feel somewhat confident that if the code compiles, it probably works and it's [2966.47s -> 2973.67s] good. Uh, but you'll see, it gives you that level of trust and confidence that you, as a human [2973.67s -> 2980.07s] engineer, no longer need to go and check it yourself. You know, you, you rely on code and [2980.71s -> 2988.79s] static analysis to actually make that, uh, a lot smoother. Um, and I guess the worst part, [2988.79s -> 2995.43s] the worst place to be in is if you are stuck in code review land where you actually enforce all [2995.43s -> 3002.15s] of the constraints, the invariance in your code base by literally the human person saying, you know, [3002.23s -> 3006.87s] reading the code and like, okay, you should not do this, right? Every time you have to do that, [3006.87s -> 3011.83s] you should consider that as a code smell, like a, uh, anti pattern. And you should say, okay, [3011.83s -> 3018.31s] instead of me commenting on the PR, how do I turn this into a hard rule, right? How do I turn this [3018.31s -> 3024.31s] into a lit rule? How do I turn this into a CI failure or how do I even categorically eliminate [3024.31s -> 3031.03s] this problem, uh, entirely? Uh, I can talk about another migration I've done, but I'll probably [3031.03s -> 3036.15s] pause here. Sure. Yeah. I feel like that's, that's where I am to be honest is, is it what [3036.15s -> 3040.15s] you're describing right now, which is that like, I don't have all of these rules. Uh, so I have [3040.15s -> 3044.47s] some things to go do after this session, uh, in terms of being able to scale my agents. I'm, [3044.47s -> 3049.83s] I'm definitely on like the, uh, you know, maybe a couple of parallel ones locally stage, like two [3049.83s -> 3054.79s] to three locally. And I'm sure most people here are on the same. So, uh, yeah, I know we only have [3054.79s -> 3058.47s] a couple of minutes left. Uh, Lauren, was there anything else that you wanted to, to highlight? [3058.79s -> 3062.47s] Obviously there's lots of questions, so I can grab more, but I want to give you a few minutes, [3062.47s -> 3065.27s] if there's anything else you want to talk about. Well, I think I've been yapping for quite a lot. [3065.27s -> 3071.59s] So maybe let's just do questions. Okay, cool. Uh, one question that had a couple of, uh, came up [3071.59s -> 3077.35s] a couple of times was just around like token usage. So the question is like, is, is what you're [3077.35s -> 3083.99s] describing a realistic thing for people who are on, you know, uh, a normal set of token usage? They [3083.99s -> 3089.99s] don't have, you know, obviously unlimited tokens, uh, to work with. Well, I think that's a really [3089.99s -> 3095.19s] good point. I mean, like, obviously, you know, I work at an AI lab where we have unlimited tokens. [3095.19s -> 3103.19s] So, uh, I definitely cannot say that, you know, this is something everyone should do in the exact [3103.19s -> 3107.35s] same way that I did it. I think it's possible to get to this point without, you know, breaking the [3107.35s -> 3113.03s] bank. But, you know, if you're like an engineering leader or, you know, you're, you're, you have a [3113.03s -> 3121.35s] startup that you lead. Um, I think to me it's a question of ROI. Um, and it's like, uh, yes, [3121.35s -> 3127.43s] you spend a lot of money on tokens in the upfront stage. You know, like refactoring your code base [3127.43s -> 3131.75s] is going to take a lot of tokens. Uh, adding all these things, uh, is going to take a bunch of [3131.75s -> 3138.71s] tokens. But if we're heading into a world where agents are writing all the code and, you know, [3138.71s -> 3143.27s] you want to be very lean, right? You don't want to have to hire, you don't want to be, you don't [3143.27s -> 3147.99s] want to become like meta, right? Like, I mean, like in terms of, you don't want to become a 10,000 [3147.99s -> 3154.07s] person engineering org because, I mean, that's a cool problem to have, but also, you know, you have [3154.07s -> 3160.71s] so much overhead. There's like planning, you know, like, it's, it's, uh, personally, I, I wouldn't, [3160.71s -> 3166.71s] uh, it's not super fun, but, um, I think you want to stay very nimble, right? And you want to, [3166.71s -> 3172.39s] you want to be like agents are all about allowing you to do things that you couldn't do before. [3172.39s -> 3177.51s] That's really, to me, like the value of agents, you know, it's not just storing tokens on every [3177.59s -> 3181.91s] single little thing. But, um, to me, like the thing I couldn't do before is like, [3182.71s -> 3189.59s] enforce this level of constraints in a code base by myself, right? Like I'm just a single person, [3189.59s -> 3196.71s] you know, uh, it would have taken me years to build this framework, uh, and do all the refactoring [3196.71s -> 3202.71s] and test everything myself and verify, you know, like run, imagine if there, it was just me right [3202.71s -> 3209.27s] now in, in pre-agent era, just like running, you know, it would take me so long, right? And my [3209.27s -> 3215.19s] salary is pretty high, right? Like, so, you know, the, the question I think an engineering leader [3215.19s -> 3221.35s] might have is just then, you know, like, what is, there's a trade-off of, do you hire someone to do [3221.35s -> 3228.39s] this or do you spend the tokens to set up a code base so that even the, the most naive, right, [3228.39s -> 3234.47s] the dumbest agents can do a good job. And when you actually get to this point, like even agents [3234.47s -> 3240.63s] that are not, you know, fable size do an excellent job of writing code. And this pays a lot of [3240.63s -> 3248.07s] dividends as well for me personally, where I have empowered not just myself, but again, like PMs, [3248.07s -> 3253.51s] designers, engineers who are not familiar with RockBot to just contribute in a way that [3254.31s -> 3260.23s] is sustainable. So I think, yeah, it's definitely like a trade-off for sure. You know, like nothing [3260.23s -> 3267.27s] is like free for sure. And tokens are pretty expensive. But oh, actually, I don't know how [3267.27s -> 3272.79s] many of you have seen this, but we actually announced Grok 4.6 today. So very exciting, [3272.79s -> 3279.43s] finally out. So yeah, Grok 4.6 would be like a great, it was very, very smart. It's really [3279.43s -> 3285.83s] good on the benchmarks. And it's the same, the tokens, well, I hopefully I'm not saying [3285.83s -> 3292.87s] this incorrectly, but I believe the cost per token is the same as 4.5. So you're actually getting [3292.87s -> 3300.47s] more intelligence for the same cost. I think this is an area that Cursor tries to, Cursor and SpaceX [3300.47s -> 3307.75s] AI try to really optimize for like that Pareto frontier of, you know, cost versus intelligence. [3308.55s -> 3313.03s] You know, we don't necessarily want to build the biggest model ever because that is extremely [3313.03s -> 3317.91s] expensive to run. It's really about like, how do you find that sweet spot, right? You don't need [3317.91s -> 3322.63s] a giant model, but it's just super smart, right? And it's not very expensive for inference. [3324.71s -> 3331.11s] But yeah, I think to kind of round it up, I think it's like, there's a, [3331.67s -> 3337.43s] if you do your own analysis, I feel like it's pretty positive. It'll be pretty positive that [3337.43s -> 3343.59s] the ROI you get from investing and stuff like this just empowers not just yourself, [3343.59s -> 3348.79s] but your whole team to be so much more productive, right? Like imagine if you have an army of [3348.79s -> 3355.91s] engineers like me who are shipping so much improvements and bug fixes, you know, every day, [3355.91s -> 3362.39s] right? Like that is pretty exciting. Cool. One last question before we wrap up. [3362.39s -> 3367.03s] This one is for the people in product on the call. So let's say we do have an army of [3367.03s -> 3372.07s] engineers who are shipping like Lauren. I'm just curious, like how is the product team or other [3372.07s -> 3376.47s] functions of your company keeping up given that like, if you're shipping so quickly, [3377.51s -> 3381.59s] are they using AI more to do their jobs? Like as much as you can speak to that, I know obviously, [3381.59s -> 3384.47s] you know, like you're not in that role, but just curious about how that works. [3386.15s -> 3393.11s] I think this is where Grokbot has been actually exceedingly powerful where, so before Grokbot, [3393.91s -> 3399.83s] obviously Cursor only had Cursor, like we only had Agent's Window, we had a CLI, we had an IDE, [3400.55s -> 3406.63s] and these are really like power user tools, right? They're designed for developers, so it's very, [3406.63s -> 3412.47s] very developer-centric. You can do knowledge work in them, but the UI is not really optimized for [3412.47s -> 3418.55s] that. So we actually didn't really have, well, I think like a lot of people in like, you know, [3418.55s -> 3424.79s] GTM product, like they might have used Cursor to do their work, but it definitely wasn't like a [3424.79s -> 3433.67s] delightful experience for them. I think now with Grokbot, it's become, Grokbot is basically like [3433.67s -> 3439.67s] their Cursor moment for people who are not in tech, in my opinion. Like it's like a very, [3439.67s -> 3445.19s] very accessible way to use agents in a very comfortable, very familiar interface. It looks [3445.19s -> 3450.79s] like iMessage, and it's very fun too, you know, you can give your agent a fun name, [3451.75s -> 3457.51s] you can have, you can kind of do orchestration in a very like natural way, where you can sort of, [3457.51s -> 3461.03s] you know, each agent is like a person, and now you've got a team of agents like working on, [3461.03s -> 3465.99s] you have one agent per account that you manage, as an example. Or if you're a PM, you have, [3465.99s -> 3469.83s] you know, you can have an agent that summarizes all the work that Lauren did last night, [3469.83s -> 3474.71s] and then now you know what I did, right? So I think our PMs are leveraging that a lot, [3475.27s -> 3480.23s] and they're shipping code too, so you know, like oftentimes they will just say, oh here's a bug, [3480.23s -> 3484.63s] I fixed it, can you look at it, and then I'll go review it, and actually it's just perfect, [3484.63s -> 3490.95s] I'm like, okay, stamp. So I think that shows that, you know, the Dune architecture is holding up, [3490.95s -> 3497.83s] right? The, all the really strict constraints allow people who are not experts in engineering [3497.83s -> 3503.35s] to contribute at a high level, so I feel like I'm already seeing that pay off a lot, where, [3504.23s -> 3511.91s] you know, designers and PMs are just able to ship features directly, and that just makes [3512.63s -> 3520.47s] the Grokbot team super fast, right? Where we can ship so quickly, and we have a lot planned, [3520.47s -> 3526.79s] so I'm very excited to, you know, to ship more, ship more stuff. [3526.79s -> 3532.39s] Yeah, that's awesome. We are at time, so I guess Lauren, if folks want to support you, [3532.39s -> 3538.79s] maybe go try out Grokbot, try out 4.6, and you know, provide some feedback, but yeah, [3538.79s -> 3542.55s] this was awesome, really appreciate you taking the time. Thanks everyone for all the messages [3542.55s -> 3545.99s] in the chat, lots of good questions, I know we didn't get through everything, but as I kind of [3545.99s -> 3550.55s] said at the top, way more questions than we could get through, but yeah, really, really thanks, [3550.55s -> 3554.39s] thanks for joining, thanks everyone for joining, and hopefully you enjoyed the session. [3555.59s -> 3560.71s] Yep, yeah, I see, thanks for having me, and if you have any more questions, yeah, just DM me on [3560.71s -> 3565.51s] Twitter, I'll open them up, I guess. You're gonna get a lot of DMs. [3567.51s -> 3572.31s] So yeah, DM me, maybe I'll do like a Twitter space at some point, that's all, for more questions, [3572.31s -> 3577.19s] but really appreciate everyone for showing up, you know, taking an hour out of your day. [3577.19s -> 3579.11s] Yeah, alright, thanks all, I'll see you the next one. [3579.11s -> 3580.39s] Okay, thanks everyone, bye.