[0.00s -> 8.00s]  Hi everyone, I'm Lauren Tan. I guess not many people know my last name. I am Potato on Twitter,
[9.36s -> 18.64s]  Potato with spelled with an E, and I have been at Cursor for about five months. Previously,
[18.64s -> 24.32s]  I was at Meta where I worked on the React team, specifically working on the React compiler,
[24.96s -> 30.64s]  which was a whole lot of fun. I'm still on the core team and contributing to open source here and
[30.64s -> 38.16s]  there, so that's really nice that they still let me do that. And before Meta, I was at Netflix
[38.16s -> 48.40s]  where I was both a tech lead and I transitioned to be an engineering manager for about two years.
[49.36s -> 55.76s]  So I've had a lot of experience going between engineering management and being an individual
[55.76s -> 61.60s]  contributor. And I think something I've noticed actually, which is quite interesting, is that
[61.60s -> 68.56s]  there are so many parallels with management skills and how to manage agents. And that's actually a
[68.56s -> 75.68s]  big part about what I wanted to chat with you and everybody else about today. But yeah, that's me.
[76.40s -> 82.48s]  I do have some very light slides, but it's not going to be just rambling,
[82.48s -> 87.20s]  so let me just share my screen and hope that I don't leak anything.
[93.42s -> 95.42s]  Oh no, I need to allow permissions.
[96.30s -> 98.06s]  Not always. Oh good, take your time.
[98.06s -> 102.14s]  There's always tech trouble. Give me one second to rejoin.
[102.78s -> 109.73s]  Yeah, go for it. I see many of you already know Lauren from the looks of the chat here.
[110.69s -> 116.37s]  So yeah, it's exciting to get a chance to chat with her and go through some of her recent work.
[116.37s -> 122.45s]  As you guys heard, you know, a lot of recent experience from Netflix to Meta and then now
[122.45s -> 128.29s]  over at Cursor. We're going to chat a little bit about Grokbot as well. So that'll be exciting.
[128.29s -> 131.65s]  I don't know if you guys saw that was a recent release. I think literally maybe yesterday or
[131.65s -> 137.17s]  the day before from the Cursor team, which is kind of like, let's call it like agents for everyone.
[137.65s -> 140.53s]  You can go check it out if you want and learn a little bit more about the product,
[141.09s -> 144.05s]  but yeah, we'll explore that a little bit today as well.
[146.82s -> 163.07s]  Alrighty, welcome back. And Lauren, you're just on mute there if you want to hop off
[163.07s -> 165.31s]  of me to do chatting. Yeah, sorry.
[165.31s -> 169.31s]  No worries. It's 2026 and I still don't know how to use Zoom.
[171.79s -> 175.15s]  Okay, so I assume you can see my screen. Yes, yeah, we're good.
[175.87s -> 182.19s]  So yeah, today, yeah, I think the big theme for me as I've been using agents to write code,
[182.19s -> 187.31s]  and I'm sure a lot of you have had the same experience as well, is how do you trust it?
[187.31s -> 192.51s]  You know, especially if you are an engineer that's been writing code for a very long time,
[192.51s -> 198.27s]  you have a lot of opinions and lessons that you've learned about doing good engineering.
[198.99s -> 203.87s]  And when you see agents just, you know, winging it and, you know, guessing, hallucinating,
[204.83s -> 210.35s]  you know, confidently stating that they found the smoking gun for the hundredth time,
[210.99s -> 214.75s]  but it's actually not the real problem, you lose a lot of trust.
[214.75s -> 217.79s]  And when you lose, when you don't have much trust in your agents,
[218.91s -> 221.39s]  I feel like you really can't get the most out of them.
[222.19s -> 224.99s]  And for me, the parallel is like with management.
[225.79s -> 231.31s]  So if I'm in a manager, an engineering manager for a team, and I have a bunch of,
[231.31s -> 236.43s]  you know, I have a team of engineers on my team, and I don't trust them,
[236.43s -> 240.91s]  then the mode of operation I'm going to be in is going to be like micromanagement, right?
[240.91s -> 247.07s]  I'll have to spend a lot of time looking over my reports shoulders and checking that they're
[247.07s -> 250.67s]  doing their work well, you know, that they're not shipping bugs to production.
[252.85s -> 258.61s]  And so I drew this chart because it's not, it's not a very scientific chart, but like,
[258.61s -> 263.65s]  this is how I imagine myself and my journey through using agents.
[264.69s -> 274.37s]  So, you know, like fast forward or back forward, or fast backwards, like a year or so when,
[274.37s -> 277.57s]  you know, nobody was, or not many people were using agents to code.
[278.85s -> 283.17s]  I think you, you know, get into this mode where you are
[283.89s -> 289.41s]  in, very heavily in the loop with one or several, like a handful of agents,
[289.97s -> 294.77s]  and you find yourself just constantly trying to understand what your agents are doing.
[295.57s -> 297.17s]  And you're very, very in the loop.
[297.17s -> 299.17s]  You're watching every single output.
[299.17s -> 305.89s]  You are sitting there prompting, and you really can't parallelize beyond that because
[306.45s -> 308.45s]  again, you don't have that trust, right?
[308.45s -> 312.13s]  You can't go to a hundred agents, like spawn a hundred agents,
[313.09s -> 315.57s]  when you don't even trust the output of one agent.
[317.25s -> 323.49s]  So over the past five months, I feel like I've really been able to like ascend this
[324.37s -> 325.73s]  trust curve.
[325.73s -> 331.17s]  And now I'm at the point where I actually have, this sounds kind of scary to say this,
[331.17s -> 334.29s]  and it makes me sound like a slop artist, but I promise I'm not.
[335.01s -> 341.49s]  But I actually have my agents now auto-merging PRs for me, which is like a wild thing to say.
[342.13s -> 347.49s]  But like I woke up today, and there were like 20 PRs landed, and I just reviewed them
[348.13s -> 351.17s]  on main, like they were already landed, and they were good.
[352.37s -> 354.37s]  So how did I get to that point?
[354.37s -> 357.09s]  It's basically what I wanted to talk about today.
[358.93s -> 361.97s]  And again, like, yeah, feel free to jump in if you have questions, Colin.
[363.65s -> 369.57s]  But oh, yeah, of course, I got to show this chart where,
[370.05s -> 378.93s]  uh, do not trust, someone requested to control my computer, probably won't do that.
[379.81s -> 385.25s]  But yeah, so this chart, I think, I'm sharing this chart not to kind of like flex,
[385.81s -> 388.45s]  but to kind of show like the journey.
[388.45s -> 392.37s]  Like, so you can see like the curve, like it sort of like inversely matches
[392.37s -> 395.33s]  the contributions I've been able to land at Cursor.
[396.21s -> 400.37s]  So I joined five months ago, and five months ago, like I said, you know, my first month,
[400.37s -> 404.77s]  I was like, not very productive, because I was, you know, I was learning the code base,
[404.77s -> 406.37s]  didn't know what the heck was going on.
[407.57s -> 413.65s]  And as I got more confident in my Asians, I've really been able to kind of ramp up
[413.65s -> 414.61s]  my productivity.
[415.41s -> 421.09s]  And again, like, yeah, like last month, I shipped 1000 PRs, which is ridiculous.
[422.05s -> 428.05s]  And then this month, we're only on the 12th, I'm already at like almost 800 PRs landed.
[428.85s -> 432.21s]  So the velocity is definitely high.
[432.21s -> 436.93s]  And I'm sure a lot of you will definitely be questioning like how much of this code
[436.93s -> 437.57s]  is actually good.
[438.77s -> 441.49s]  And I think, yeah, that's definitely fair to question.
[443.65s -> 453.25s]  But yeah, I think if you set up your agents well, you can definitely get to a very similar level.
[454.61s -> 456.21s]  And so I'm going to talk about how we do that.
[459.68s -> 464.08s]  So for me, I think I'm curious, like, I guess calling your experience as well.
[464.08s -> 472.48s]  But for me, I think the most important skill that you should have in your toolbox, when
[472.48s -> 474.48s]  you work with agents is verification.
[475.52s -> 482.80s]  And by verification, I mean the ability for an agent to actually run the code, or take
[483.44s -> 491.12s]  CPU traces or heap snapshots or, you know, open an iOS simulator, whatever, you know,
[491.12s -> 499.04s]  however your application is exposed to your users, it can do the same thing and run it
[499.04s -> 501.84s]  for real and actually test and verify it don't work.
[502.80s -> 505.20s]  Because that's the thing that really closes the loop.
[506.08s -> 511.60s]  It doesn't guarantee your agent writes good code, but it allows them to at least write
[511.60s -> 517.60s]  correct code, which is a really big step forward for being able to trust your agent.
[521.04s -> 526.24s]  I can share one example that we have within Cursor.
[528.72s -> 532.48s]  Oops, where let me open this.
[543.10s -> 543.60s]  There you go.
[543.60s -> 552.00s]  So for Cursor's agent window, so this is actually an interesting story.
[552.00s -> 558.96s]  But when I joined Cursor five months ago, they're actually, well, I was supposed to
[558.96s -> 559.84s]  join a different team.
[559.84s -> 562.00s]  I was supposed to join like the cloud agents team.
[562.80s -> 568.16s]  But then since I have a lot of experience working on React and agents window is a React
[568.16s -> 577.12s]  application, I was asked to basically help out with the agent window work.
[578.24s -> 584.16s]  But there wasn't really a lot of like skills to help me.
[584.16s -> 588.16s]  So I just found myself like, okay, agents window is going to launch in like a week,
[588.16s -> 588.40s]  right?
[588.40s -> 594.96s]  We have a really tight deadline and there was, you know, I was just sitting there like,
[594.96s -> 600.64s]  okay, I'm going to open up the Chrome DevTools and just like take a trace, look at it myself
[600.64s -> 602.96s]  and try to make sense of this flame graph.
[602.96s -> 605.68s]  And keep in mind, I was just like in my first week.
[605.68s -> 609.84s]  So I had no idea what I was looking at, no idea what, you know, I mean, I had some idea,
[609.84s -> 612.64s]  but you know, the code base was completely fresh to me.
[614.00s -> 617.44s]  And I realized like my agent had no idea either.
[617.44s -> 620.40s]  You know, like I would take a screenshot of the trail, download trace or send it to
[620.40s -> 624.80s]  it and it'd be like, yeah, it kind of looks like this, you know, and it would like
[624.80s -> 626.96s]  confidently state like it's this thing.
[626.96s -> 630.16s]  And then I try to fix that and turns out that's not the actual thing.
[631.68s -> 634.16s]  So this was a very, very slow process.
[634.16s -> 638.96s]  And if you've ever done any like performance work yourself or, you know, just even development
[638.96s -> 644.16s]  with an agent where you don't have a verification skill, you are the verifier, right?
[644.16s -> 645.44s]  You're the bottleneck.
[646.32s -> 649.92s]  You tell your agent to do something and then it goes off and write some code.
[649.92s -> 653.92s]  Then you open up your, you know, local dev build and then you start to say, oh, you know,
[653.92s -> 654.48s]  it doesn't work.
[654.48s -> 659.60s]  Then you have to copy paste screen screenshots or console errors or whatever.
[660.40s -> 665.84s]  And then your agent like slowly kind of like, you know, works with that and then tries to
[665.84s -> 669.76s]  understand it and fix the thing.
[669.76s -> 672.96s]  But then you're constantly just in the loop and being a bottleneck.
[672.96s -> 674.56s]  So there's really no way to parallelize.
[675.36s -> 681.44s]  So the control glass skills are like one of the first skills I built for cursor and glass,
[681.44s -> 686.08s]  by the way, is the code name for agents window that we use internally, but it's just
[687.04s -> 687.84s]  cursor, I guess.
[689.60s -> 694.80s]  And so this skill is, I guess that the code itself is not super interesting.
[694.80s -> 700.56s]  Your agent can very easily make one for you where if you're building an electron app or
[700.56s -> 707.84s]  a web app or even iOS applications, you can teach your agent how to use like the Chrome
[707.84s -> 715.12s]  DevTools protocol or through Apple has some utilities as well for running the simulator
[715.12s -> 719.20s]  and taking traces and controlling programmatic control as well.
[720.40s -> 722.56s]  So that's really useful.
[723.60s -> 729.28s]  But one thing I actually want to talk about is the this thing.
[731.12s -> 732.00s]  You have to read me.
[732.96s -> 739.60s]  So this skill comes with this very unique feature called or not featured, a unique file
[739.60s -> 740.56s]  called a feature map.
[741.52s -> 744.96s]  And so the story then is like I built this skill.
[744.96s -> 751.52s]  And so now the agent was able to actually run the agent window and take traces and whatnot.
[752.96s -> 756.40s]  But it had no idea what the agent's window was.
[757.04s -> 764.88s]  So someone would say, oh, the left side bar is like laggy or something like that.
[764.88s -> 768.08s]  Or the right side, the PR tab is not working.
[768.64s -> 770.88s]  And the agent would just be flailing around.
[770.88s -> 774.72s]  It would spend a lot of time trying to look up the code and where is this feature?
[774.72s -> 779.52s]  How do I actually get to it on the UI, which made it basically completely useless.
[780.48s -> 787.04s]  You know, like we would I would run the skill locally and it would spawn a dev build.
[787.92s -> 790.00s]  But then it just be churning.
[790.00s -> 791.68s]  I just try to click here.
[791.68s -> 793.28s]  It wouldn't know how to get to things.
[794.48s -> 796.16s]  And it was just an awful experience.
[797.20s -> 800.16s]  So who is putting arrows on my screen?
[801.44s -> 805.92s]  So yeah, this this feature map has been really useful
[806.72s -> 810.40s]  because it teaches the agent how to get to all of the features that you have.
[811.68s -> 819.52s]  And in PSAC, the plugin that I've made, if you search for PSAC cursor on Google,
[819.52s -> 820.08s]  you'll find it.
[821.04s -> 824.72s]  But there is a create verification skill in that plugin
[824.72s -> 827.60s]  where it actually helps you set up something like this for yourself,
[828.88s -> 830.08s]  including the feature map.
[830.08s -> 834.00s]  So it will actually explore the code and build up this initial feature map
[834.00s -> 837.92s]  that tells your agent how to get to all of the different features that you have.
[838.88s -> 843.92s]  And this is extremely powerful because now that you have these user reports that come in,
[845.20s -> 849.28s]  you can actually map even like a vague report or even a screenshot.
[850.08s -> 856.00s]  So we have this internally at cursor where we have a Slack channel with, you know,
[856.00s -> 860.88s]  lots of people giving us feedback on the agent's window and rock bar and whatnot.
[861.60s -> 867.28s]  And oftentimes the report is very bad, like very low quality, like someone will just put,
[868.16s -> 871.44s]  very often we get like a screenshot and then someone just says question mark,
[871.44s -> 873.04s]  question mark, question mark, like, what is this?
[873.84s -> 878.40s]  And, you know, like without this, your agent is like, I have no clue, right?
[878.40s -> 883.68s]  But with a feature map like this, it has a lot more context and understanding
[884.32s -> 888.00s]  of how to actually navigate, how to get to all of the different features.
[888.88s -> 892.64s]  So like, you know, example, like, I guess like the sidebar, like, what is the sidebar?
[893.84s -> 896.88s]  You know, like all the different sub features that are present in it.
[898.40s -> 902.08s]  Like from the user point of view, here's how to get to it.
[902.08s -> 903.68s]  All the different keyboard shortcuts.
[905.20s -> 907.68s]  Even like the, what do you call it?
[907.68s -> 912.72s]  The DOM elements or yeah, like the attributes that you use for selecting things
[913.36s -> 916.08s]  through the CDP are all there.
[916.72s -> 922.64s]  So again, yeah, this is like really, really powerful for agents.
[923.44s -> 929.04s]  And P-Stack ships that create verification skill, but also a maintain verification skill.
[930.00s -> 931.76s]  So you can keep this up to date.
[932.72s -> 933.04s]  Cool.
[933.04s -> 933.20s]  Yeah.
[933.20s -> 934.32s]  I was just going to ask how you created that.
[934.32s -> 939.04s]  So do you mind sharing a little bit more about that process in the context of P-Stack
[939.04s -> 941.28s]  and maybe just what P-Stack is for the folks who aren't familiar?
[941.52s -> 942.08s]  Yeah.
[942.08s -> 947.04s]  So P-Stack is pretty interesting because, well, first of all, the name is kind of goofy.
[947.52s -> 957.52s]  Like the P, the P in P-Stack is like potato, potato snack, because I, so there's a pretty
[958.56s -> 962.64s]  famous person, Gary Tan, who is the CEO of Y Combinator.
[963.20s -> 967.68s]  And he's come up with this plugin called G-Stack, Gary Stack.
[967.76s -> 972.08s]  And funnily enough, we share the last name.
[972.08s -> 978.00s]  We have no relations, but I thought it'd be funny to kind of poke fun at Gary and make
[978.00s -> 987.52s]  P-Stack, my version of his plugin, but kind of tailor it to my own set of engineering practices.
[989.20s -> 993.68s]  But I honestly actually never set out to build P-Stack, but it just started with a bunch of
[993.68s -> 994.64s]  skills, right?
[994.64s -> 998.96s]  Like I started with that control glass skill, and then I started with another skill like
[998.96s -> 1003.84s]  called how, which I also noticed through like observing agents.
[1005.36s -> 1008.64s]  So like, you know, in the early days of me, you know, trying to climb this ladder,
[1008.64s -> 1014.40s]  I was like super in the loop and I was basically nitpicking my agents to an extreme degree.
[1014.40s -> 1019.60s]  I was like, I would tell it, you know, this feature has stopped working.
[1019.60s -> 1020.96s]  Here's a bug report.
[1020.96s -> 1022.16s]  Like, why isn't it working?
[1024.24s -> 1029.20s]  And very often the agent would just like confidently state like, oh, it has to be this,
[1029.20s -> 1029.44s]  right?
[1029.44s -> 1030.40s]  It has to be this thing.
[1031.12s -> 1036.00s]  And I noticed like when I looked at the actual tool calls, I noticed it wasn't actually reading
[1036.00s -> 1039.36s]  the code that I thought should be affected.
[1039.92s -> 1042.08s]  And that made me just extremely suspicious.
[1042.08s -> 1046.16s]  And at that point else is that I'm not going to, I can't trust any, this agent anymore
[1046.16s -> 1048.40s]  because it's just, it's just completely hallucinating.
[1049.12s -> 1055.52s]  And I think, I think it's very easy to just, you know, like build up that distrust and
[1055.52s -> 1057.92s]  not, and kind of feel helpless.
[1057.92s -> 1060.96s]  Like, you know, you don't know how to help your agents succeed.
[1061.68s -> 1065.84s]  But like, again, I think the, the, the management analogy is super helpful because like,
[1065.84s -> 1071.36s]  imagine if you were a manager of an engineering team and you had an engineer in your team
[1071.36s -> 1074.56s]  who was a really good coder, no business context whatsoever.
[1074.64s -> 1079.52s]  You know, they, they just, you just hired them and they onboarded, you know, like five
[1079.52s -> 1080.16s]  seconds ago.
[1081.60s -> 1084.72s]  And so how do you actually teach that person to be effective?
[1085.76s -> 1090.16s]  So how you do that is through a skill, a skill being just, you know, it's just markdown,
[1090.16s -> 1090.56s]  right?
[1090.56s -> 1097.60s]  But, you know, it encodes a lot of information, instructions, a lot of, you can really draw
[1097.60s -> 1103.44s]  out a lot of intelligence from an agent by, well, some people on Twitter call it like,
[1103.44s -> 1107.92s]  you know, pull the agent to a different latent space, which is kind of like a fancy way of
[1107.92s -> 1112.80s]  just saying, like, since, you know, LLMs are sort of like, they predict the next token.
[1114.16s -> 1119.04s]  When you give it some high quality tokens to begin with, then, you know, it, it can kind
[1119.04s -> 1123.36s]  of pattern match on like a higher space that's, you know, smarter.
[1125.44s -> 1128.88s]  So that's like a very interesting model there.
[1128.88s -> 1131.76s]  But yeah, I built P-Stack very, very incrementally.
[1132.64s -> 1137.92s]  So I started with just really observing how agents, you know, all the different failure
[1137.92s -> 1140.80s]  modes of, of that agents were having.
[1140.80s -> 1144.16s]  And every time I saw that, I just, okay, I'm just going to make that a skill, right?
[1144.16s -> 1149.44s]  Like, stop hallucinating, actually go and search up, look up the code, use a lot of
[1149.44s -> 1150.08s]  subagents.
[1151.12s -> 1153.28s]  And yeah, stop guessing.
[1156.52s -> 1157.32s]  Yeah, that makes sense.
[1157.32s -> 1160.20s]  One, one kind of follow up question here, both from myself and from a bunch of people
[1160.20s -> 1160.68s]  in the chat.
[1160.68s -> 1163.24s]  So I guess it's two, two parts.
[1163.24s -> 1165.24s]  So one is like, how do you maintain these skills?
[1165.24s -> 1167.00s]  So like the product changes over time.
[1167.00s -> 1169.56s]  Obviously there's a lot of people are shipping against the code base.
[1169.56s -> 1171.40s]  So how do these skills get maintained?
[1172.04s -> 1176.20s]  And then second to that is like, how do you know when your verification is good enough?
[1177.08s -> 1180.92s]  Like in, you know, you can trust that the verification loops that you've built are
[1180.92s -> 1184.20s]  going to, I guess you trust the outputs when they're done.
[1187.38s -> 1191.46s]  Yeah, maybe I'll talk about, I think I was somewhat related.
[1191.46s -> 1192.82s]  Maybe I'll start with this one first.
[1193.62s -> 1196.98s]  So like, how do I maintain these skills?
[1198.82s -> 1204.42s]  So if you're not familiar with this concept and eval is essentially like a way to,
[1205.78s -> 1209.22s]  well, I think the mental model I have is like, it's like a unit test for an agent.
[1210.66s -> 1214.10s]  And you can actually make your own evals.
[1214.10s -> 1216.50s]  You don't need like a special framework for them.
[1216.50s -> 1221.62s]  You can build, you can, you can build one depending on like, you know, how scientific
[1221.62s -> 1223.06s]  and how rigorous you want to be.
[1224.34s -> 1225.38s]  My screen is red.
[1226.26s -> 1226.42s]  Yeah.
[1226.42s -> 1227.38s]  There's a little button.
[1228.34s -> 1228.66s]  Sorry.
[1229.86s -> 1231.70s]  Like disabling the drawing or something.
[1231.70s -> 1232.74s]  I can't see my screen.
[1233.30s -> 1233.62s]  Yeah.
[1233.62s -> 1234.02s]  Sorry.
[1234.02s -> 1235.78s]  If you guys could not draw on the screen, that'd be great.
[1235.78s -> 1239.22s]  But there's a little button in the troll.
[1239.22s -> 1239.46s]  Yeah.
[1239.46s -> 1240.82s]  The little drop down.
[1242.34s -> 1243.14s]  How do I clear?
[1244.50s -> 1244.74s]  Yeah.
[1245.94s -> 1246.42s]  You got it.
[1246.42s -> 1246.74s]  Perfect.
[1247.54s -> 1254.50s]  Um, yeah, so evals are a way to unit test your skills basically.
[1255.14s -> 1258.82s]  And actually in P stack, we ship under potato moon.
[1258.82s -> 1259.70s]  There's a playbook.
[1259.70s -> 1261.46s]  If you search for it called eval playbook.
[1262.66s -> 1269.14s]  And it's, uh, uh, it's like not, it's actually pretty, pretty rigorous the way it's done.
[1270.10s -> 1274.90s]  But, um, essentially what I do is I spawn a lot of different subagents.
[1274.90s -> 1282.26s]  I have like my main coordinator agent, uh, come up with a rubric for, uh, what I want
[1282.26s -> 1283.62s]  the skill to do.
[1283.62s -> 1289.86s]  Um, and then it spawns all these subagents and it, it creates individual directories
[1289.86s -> 1295.94s]  for them, uh, which are cleverly named to not let the sub agent know that it's being
[1295.94s -> 1299.94s]  evaluated because, uh, agents can actually tell.
[1299.94s -> 1301.86s]  And when they do, they change their behavior.
[1302.50s -> 1304.74s]  Uh, but it does a bunch of stuff like that.
[1304.82s -> 1312.10s]  To, um, essentially, yeah, like test whether or not the skill I'm making or changing is
[1312.10s -> 1314.50s]  actually doing what I think it does.
[1314.50s -> 1319.38s]  Um, and one of the really nice things about cursor is that we are, we support so many
[1319.38s -> 1320.02s]  different models.
[1320.66s -> 1325.30s]  So you can actually eval your skill across all sorts of different models.
[1325.30s -> 1331.70s]  Um, and you know, get a sense of how well it performs across that different matrix,
[1331.70s -> 1333.78s]  um, especially for the models that you use.
[1335.30s -> 1336.74s]  Uh, so I do this a lot.
[1336.74s -> 1342.26s]  Every time I modify a skill, I will run one of these, uh, like the eval playbook, uh,
[1342.26s -> 1346.34s]  and make sure that, you know, it's actually leading to a result I want.
[1347.06s -> 1352.42s]  Uh, but I will say like maintaining skills is actually pretty hard.
[1352.42s -> 1355.94s]  Uh, it requires, I think, a lot of taste and observation.
[1356.74s -> 1361.54s]  So you kind of need to be very good at being a backseat driver.
[1361.54s -> 1362.10s]  You know what I mean?
[1362.10s -> 1366.18s]  Like if you do pair, if you've ever done pair programming, for example, uh, and you
[1366.18s -> 1370.18s]  watch a coworker code and you're just like, you could probably do this better.
[1370.18s -> 1372.50s]  You know, you could do it, you know, like, why did you not do this?
[1372.50s -> 1372.74s]  Right.
[1372.74s -> 1376.74s]  You, you ask a lot of questions to your coworker and it's kind of a similar thing here.
[1376.74s -> 1379.94s]  You like, you don't want to just be a passive observer of the agent.
[1379.94s -> 1384.34s]  You want to be very in the driver seat in the initial stages when you're building up
[1384.34s -> 1385.54s]  your own set of skills.
[1386.26s -> 1389.94s]  Uh, you know, obviously you can use something like PSAC, but if you're building your own
[1389.94s -> 1395.06s]  set of skills, it's very, I think, you know, opening up the, all the tool calls and like
[1395.06s -> 1401.30s]  reading the code and reading all the, uh, the agent behavior and their thinking blocks
[1401.30s -> 1405.62s]  is a really great way to see where they, they fail, right?
[1405.62s -> 1408.98s]  Like what, what, you know, where are they being done?
[1408.98s -> 1410.66s]  And then you can go and build a skill for that.
[1411.54s -> 1416.42s]  And then with verification, how you trust it is it's, I think it's also a very similar
[1416.42s -> 1422.02s]  iteration loop, uh, where, you know, like I actually did the same process for verifying
[1422.02s -> 1428.98s]  the verification skill where I actually get, um, so one thing that's interesting about evals
[1428.98s -> 1435.22s]  is that you can sort of hill climb them, meaning that, uh, your eval can produce a score,
[1435.22s -> 1435.54s]  right?
[1435.54s -> 1441.22s]  Uh, a score that you can get your coordinator to produce, uh, but also, uh, you can have
[1441.22s -> 1447.38s]  a judge agent of a different model to, uh, kind of cross-reference and make sure that
[1447.38s -> 1450.42s]  the first model is not being biased, right?
[1450.42s -> 1454.66s]  The model that's judging all of the subagents that are running the thing, uh, but you can
[1454.66s -> 1456.02s]  also like hill climb.
[1456.02s -> 1461.46s]  So meaning that you can, you can use like slash loop in cursor and you can say, okay,
[1461.46s -> 1464.10s]  keep looping on this eval, right?
[1464.10s -> 1467.06s]  Until everything is 10 out of 10 as an example.
[1467.54s -> 1471.22s]  Uh, and I did the same, the basically the same approach with the control skill.
[1471.22s -> 1474.42s]  And so I kind of, it was very, it was very hands off actually.
[1474.42s -> 1479.94s]  Uh, so, you know, I, uh, I kind of built, I built that skill that way, like the CLI
[1479.94s -> 1481.14s]  in that skill.
[1481.14s -> 1485.06s]  Um, and over time it's gotten really good.
[1485.06s -> 1488.50s]  Uh, but yeah, it was definitely not super smooth at the beginning.
[1488.50s -> 1495.38s]  It required a lot of iteration and I think there's an analogy here for me, which is,
[1495.38s -> 1500.66s]  um, well, I make this analogy later in a different slide on my drawing here.
[1500.66s -> 1507.78s]  Uh, but I think of it like, uh, you know, as a, as a engineer now, you're sort of more
[1507.78s -> 1514.74s]  like, uh, like maybe a manager or the analogy I like is like, you're like a chef in a restaurant.
[1515.30s -> 1517.06s]  Uh, you know, you're the head chef.
[1517.06s -> 1519.70s]  Uh, you're not cooking all the food yourself anymore.
[1519.70s -> 1521.14s]  You have a team of cooks, right?
[1521.14s -> 1525.22s]  You have a line cooks, you have a sous chef, you know, you know, all these different stations.
[1526.18s -> 1529.62s]  Um, and it's your job to really design the environment.
[1529.62s -> 1532.10s]  You know, you, you, you're in charge of setting up the kitchen.
[1532.10s -> 1536.26s]  You're in charge of, you know, like giving tasks to different people.
[1536.98s -> 1542.98s]  So, um, yeah, it's a very interesting way of working.
[1543.70s -> 1548.18s]  Uh, but yeah, that's, that's how I've basically built, uh, these verification skills.
[1548.74s -> 1549.06s]  Yeah.
[1549.06s -> 1550.18s]  Just, just one fault there.
[1550.18s -> 1552.58s]  I'm like, I had to go try to go one layer deeper.
[1552.58s -> 1559.06s]  So are you, let's say we wanted to build, um, uh, an eval or a skill for, for something
[1559.70s -> 1563.62s]  and we wanted to kind of get better on its own, which is, is what I think you're suggesting.
[1564.34s -> 1569.22s]  Are you doing that in like a work tree kind of isolated with like the subagents and then
[1569.22s -> 1574.58s]  the reviewer agent and all that, is it happening like in some type of cloud hosted environment?
[1574.58s -> 1576.50s]  Like what's the, the more, the practical steps.
[1576.50s -> 1580.26s]  If I wanted to go do this, uh, and like set up a verification system for something,
[1580.26s -> 1581.86s]  what would I, what would I do or where would I start?
[1584.39s -> 1589.75s]  Um, I think that, uh, the best place to start is local because you can observe,
[1590.95s -> 1592.87s]  you can definitely observe what your agents are doing.
[1592.87s -> 1598.23s]  So, uh, if you're building a verification skill for yourself, uh, I would definitely start local
[1598.23s -> 1604.87s]  and just have your agent bring up the application, whether it's like a CLI or a desktop app or
[1604.87s -> 1605.27s]  whatever.
[1605.83s -> 1607.35s]  And so you can actually observe, right?
[1607.35s -> 1612.47s]  You can see how the agent is interacting with the, the application.
[1612.55s -> 1616.15s]  You can see it, you know, how it calls like the different
[1616.15s -> 1620.55s]  APIs that allow it to interact with the, uh, the application.
[1622.15s -> 1628.15s]  Um, but, uh, for me personally, uh, I have basically been kind of all in,
[1628.15s -> 1631.35s]  mostly all in on cloud agents because they're extremely powerful.
[1632.07s -> 1636.87s]  Uh, and the really powerful thing about cursor is the, the cloud agents actually,
[1637.43s -> 1639.83s]  where if you spend a little bit of time setting up your environment,
[1639.83s -> 1645.19s]  then these control skills, these verification skills pay a huge amount of dividends
[1645.91s -> 1651.27s]  because it's not just something that makes you as a single engineer better.
[1651.27s -> 1656.23s]  It actually levels up your whole team, uh, and even your whole company, because, uh,
[1656.23s -> 1660.23s]  you can actually start thinking about cloud agents and start thinking about automations
[1660.23s -> 1662.79s]  that automatically do things.
[1662.79s -> 1667.99s]  Like, uh, I'll, I get, I kind of talk about this a bit later, but I'll just kind of
[1668.95s -> 1673.11s]  get into it, uh, where, where, you know, for example, like I talk a lot about this
[1673.75s -> 1678.31s]  agent we have called Benny, right? Who, who, uh, you know, takes all of the
[1679.11s -> 1684.47s]  bug reports that we get and it automatically goes off in the cloud, opens up a cloud, uh,
[1684.47s -> 1690.79s]  it's, you know, it's desktop, it runs cursor in its own computer and it uses the same
[1690.79s -> 1696.39s]  control skills to interact with the application and try to reproduce the bug, uh, or the user
[1696.39s -> 1702.39s]  report. And this is so, so powerful because at once I can immediately, I get so much
[1702.39s -> 1707.11s]  information from this automatically. Like here in this example, you can see that, uh, the
[1707.11s -> 1712.23s]  Benny actually reproduced the bug, uh, but it's already fixed on main.
[1713.11s -> 1717.43s]  So it actually confirms that we fixed this problem already and all I need to do is just
[1717.43s -> 1723.19s]  release another build of, of cursor. Uh, so that's like huge information there that I
[1723.27s -> 1726.95s]  didn't have to go off and sit with an agent, you know, and spend an hour trying to fit,
[1726.95s -> 1732.23s]  like, is this fixed? Is this not fixed? So you, you, you gain back so much time. Uh,
[1732.23s -> 1736.47s]  but you know, everybody on my team benefits from this, everybody in the company benefits
[1736.47s -> 1743.91s]  from this. Uh, so definitely think that, uh, you know, keeping these, uh, using cloud agents
[1743.91s -> 1749.27s]  is super powerful. Uh, but yeah, it's like a journey. You have to trust it first, right?
[1749.27s -> 1753.83s]  Before you, you get to this point and that's, it goes back to what I was saying here where,
[1753.83s -> 1758.95s]  you know, it's very hard. It's almost impossible. And I would definitely encourage you not to try
[1758.95s -> 1765.19s]  to jump from, you know, like if you're still in this zone, you don't want to jump to like,
[1765.19s -> 1770.55s]  I'm going to spawn a hundred or thousands of cloud agents right now because you're just going to
[1770.55s -> 1776.15s]  waste a lot of tokens. Um, and it's going to be extremely expensive. Yeah. So just to kind of
[1776.15s -> 1780.55s]  recap so far, basically the, if we wanted to go on the journey that you've kind of gone on, it
[1780.55s -> 1786.79s]  would be to start with verification, uh, building some, some skills and some, some ways of determining
[1786.79s -> 1790.71s]  that the agents are producing at least like correct code, whether, like you said, whether
[1790.71s -> 1793.91s]  it's good code or not, it's maybe a separate question, but like it's, it's technically solving
[1793.91s -> 1798.95s]  the problem by looking at, you know, stack traces, looking at, you know, the actual behavior in the
[1798.95s -> 1803.91s]  app and so on. Um, and then once we trust it locally, then we can start to think about scaling
[1803.91s -> 1808.39s]  into the cloud and running more agents that are picking up signals, I guess, on their own,
[1808.39s -> 1811.91s]  right? So whether that's like a bug report that comes in or something, they can go and pick it up
[1811.91s -> 1817.43s]  and solve the problem and give us back a PR. I think maybe the last step is like auto-merging
[1817.43s -> 1822.95s]  the PRs, which is where you're at and maybe not where everyone is at and then reviewing them on
[1822.95s -> 1827.99s]  main. But, um, is that, is that about right? Yeah, exactly. I think, yeah, that's why I drew this,
[1828.31s -> 1833.67s]  uh, this, this curve, right? Because that, this, this basically describes my journey of, you know,
[1833.67s -> 1838.71s]  when I started, barely could use a couple of agents and I was just observing every single thing.
[1839.35s -> 1844.15s]  I think there's really no shortcut for going from here to there because this is really about
[1844.15s -> 1849.75s]  your personal level of trust in agents, right? Um, obviously, you know, as an engineer, you don't
[1849.75s -> 1856.15s]  want to just slop code into production. So how do you actually build up that trust takes, um, a lot
[1856.15s -> 1863.35s]  of, uh, I guess, taste and judgment. Um, but, uh, you know, like, I think plugins like P-Stack
[1863.35s -> 1870.23s]  definitely kind of help you get, uh, get up to speed much quicker. Uh, and so I guess it's like,
[1870.23s -> 1876.15s]  if you trust me and you trust P-Stack, then in, by extension, you can maybe trust your agents.
[1876.15s -> 1880.71s]  But if you don't trust me, and I definitely would not encourage people to blindly trust me,
[1881.67s -> 1886.95s]  uh, uh, you know, if you build up your own set of skills that you can obviously, you know,
[1886.95s -> 1891.99s]  take a look at P-Stack and kind of fork it, make it your own, improve the skills, definitely
[1891.99s -> 1897.75s]  encourage that. Uh, but for me, it's really all about, it just keeps coming back to trust.
[1897.75s -> 1903.19s]  You know, every one of us here in this chat have a different standard for engineering. Uh, and there
[1903.19s -> 1910.15s]  are different things that are important for us in our code base. And, uh, when you are able to encode
[1910.23s -> 1915.03s]  all of that into skills and you can verify that your agent is actually doing them, that allows
[1915.03s -> 1922.63s]  you to really kind of ascend this curve and, um, uh, you know, start automating things. Uh, there's
[1922.63s -> 1927.83s]  another piece I wanted to talk about, um, if there's... Yeah, go for it. I'll pick up more
[1927.83s -> 1931.75s]  questions as I go, but yeah. I think there's a third part to this, which I haven't talked about
[1931.75s -> 1938.39s]  yet, which is kind of an interesting one, which is like refactoring and rewriting, like one of the,
[1938.39s -> 1943.99s]  uh, I guess most controversial, one of the most controversial topics in the industry, I think is
[1943.99s -> 1951.11s]  like, should you rewrite your app or not? Um, because I think engineers are very prone to this
[1951.11s -> 1955.59s]  where, especially when you join a company, you come in and you see like the code base and you're like,
[1955.59s -> 1960.63s]  man, this is shit. Like who wrote this code? You know, it's terrible. I want to rewrite the whole
[1960.63s -> 1967.43s]  thing. There is a very common inclination and I think a lot of, you know, before agents, um,
[1967.43s -> 1972.47s]  and I guess arguably even now people will definitely discourage you from rewriting stuff,
[1973.27s -> 1978.71s]  but I'm actually here to make a case for why you might want to consider it. Um, because
[1980.71s -> 1986.15s]  I think it really depends. Uh, you know, uh, Brownfield applications, I think are actually
[1986.15s -> 1991.83s]  in a pretty good spot, especially if they're set up well already. Uh, and like recently I've been
[1991.83s -> 1999.35s]  talking to some people, but, uh, you know, I was just observing, I just noticed this parallel,
[1999.35s -> 2005.99s]  which is that a lot of big tech company problems are now everybody's problems. Um, and the big tech
[2005.99s -> 2010.87s]  company problem, you know, like when I was working at Meta, like we had this giant monorepo, we had
[2010.87s -> 2016.31s]  like, I don't know, tens of thousands of engineers just, you know, like banging on their keyboards
[2016.31s -> 2024.71s]  and shipping code. And a lot of really great engineers at Meta. Uh, but, uh, I'll say like,
[2024.71s -> 2029.51s]  you know, you'll be surprised that the code quality is actually not that good. Um, and so
[2030.23s -> 2035.91s]  I often joke that like, you know, before AI sloth, we had human sloth. Um, and so, uh, you know,
[2035.91s -> 2041.75s]  I think a lot of big tech info, like, uh, like what Meta has or Google, you know, you know,
[2041.83s -> 2046.63s]  really big tech companies are actually designed for that, where you're sort of like, you're
[2046.63s -> 2051.83s]  catering to the, the, you know, like, uh, that sounds so bad to say, but like the, the least
[2051.83s -> 2057.43s]  capable engineer on your team, right? You build, you build frameworks, you build conventions,
[2057.43s -> 2062.07s]  you build guardrails, you know, you restrict credentials so that, you know, your intern
[2062.07s -> 2069.27s]  doesn't wipe your production database. Um, there's, uh, you know, if you have that level of
[2069.27s -> 2074.79s]  info already, I think your agents can actually already do a very solid job, right? Because they
[2074.79s -> 2081.83s]  have the, the guardrails are already in place for agents to not cause havoc or not cause too much
[2081.83s -> 2088.39s]  havoc, uh, in your code base. Um, and you can always add more, you know, guardrails. Uh, but I
[2088.39s -> 2092.71s]  think like greenfield applications, especially are, you know, like the brand new applications
[2092.71s -> 2099.19s]  are like the biggest risk in my opinion, uh, and also the greatest opportunity because, you know,
[2099.19s -> 2105.27s]  if you vibe code a project, uh, a prototype, um, like we did for Grokbot, you know, Grokbot
[2105.27s -> 2110.47s]  was spun up very, very, very quickly. Um, and if you, uh, if you haven't heard of Grokbot, it's
[2110.47s -> 2115.67s]  like our new application we just launched yesterday. Uh, it's, it's really cool. Uh,
[2115.67s -> 2121.35s]  lets you orchestrate your, create like individual agents that have their own identity and you can
[2121.35s -> 2125.75s]  kind of orchestrate them. It's super cool. Definitely check it out. Um, but yeah, that was,
[2125.75s -> 2130.71s]  it's like a very, it was a very greenfield application, like most prototypes are. So
[2130.71s -> 2137.11s]  it was like vibe coded very quickly. Humans were not reading the code at all. And, uh,
[2137.11s -> 2143.83s]  I had this tweet recently, uh, where I said something about organic architecture. Um,
[2145.27s -> 2153.43s]  maybe I'll find it. Uh, but the idea is that, uh, when you have a completely vibe coded application,
[2153.43s -> 2157.67s]  you essentially have no guardrails whatsoever. So, uh, your agents,
[2159.51s -> 2164.31s]  when you give them a task, they will just solve it in whatever method is the most convenient.
[2165.11s -> 2171.19s]  And over time you get into this, uh, situation where you have a code base that is spiraling out
[2171.19s -> 2177.19s]  of control because you don't understand it. Uh, your agents understand it, I guess in a way,
[2177.19s -> 2182.95s]  but like they've built something that is, you know, optimized for short, for shortcuts. Uh,
[2182.95s -> 2188.47s]  and, uh, you know, it will, you will suffer, you have a lot of issues with that application.
[2190.31s -> 2195.91s]  Uh, so I think starting your code base with, uh, like very strong constraints is
[2197.75s -> 2202.95s]  very much needed. Uh, because like when you have a code base that you can trust,
[2202.95s -> 2209.59s]  right? When you have guardrails that actually help you, uh, uh, help your agents write good code,
[2209.59s -> 2215.19s]  you can get into the, you know, like into this part of the curve where I, I, where I said,
[2215.19s -> 2221.43s]  you know, I woke up today and I had like 20 PRs merged, uh, by my agents. And that's because I
[2221.43s -> 2229.91s]  invested a lot, a lot of time, uh, over 600 PRs. I, I calculated yesterday, uh, when I refactored
[2229.91s -> 2237.03s]  all of Grokbot to this new architecture that I've been building. Um, and yeah, I've gotten to a
[2237.03s -> 2243.99s]  point where I, I don't really look, I really don't look at the code anymore. And, um, I say that not
[2243.99s -> 2249.91s]  just, you know, to sell you tokens, but because I, you know, it took a lot of work to get to that
[2249.91s -> 2254.95s]  point. I spent a lot of tokens to get the code base to this point where I no longer have to look
[2255.03s -> 2261.35s]  at it. Uh, but I'm very excited because, you know, the potential where, you know, it's not just,
[2261.35s -> 2267.03s]  it, this doesn't just benefit me. It benefits everyone contributing to Grokbot. And it also
[2267.03s -> 2273.51s]  empowers, you know, designers and product managers and, you know, people, even GTM people to add
[2273.51s -> 2278.71s]  features to Grokbot. And I don't have to worry, you know, I don't have to wake up at night in,
[2278.71s -> 2282.71s]  in the middle of the night and worry like, oh shit, someone's just merged a perf regression.
[2282.71s -> 2288.23s]  Right. I have a ton of constraints and CI. I was like, it's actually very annoying to write
[2288.23s -> 2295.03s]  code in, in Grokbot, but like agents absorb all of that annoyance. Um, but yeah, I'm happy to talk
[2295.03s -> 2303.03s]  about what exactly that is. Um, yeah, I think one question, um, before we get into the, this part
[2303.03s -> 2307.99s]  here is just around that element of like what your, your, your CI looks like, or maybe some
[2307.99s -> 2312.31s]  of the constraints and then also like the average PR size. I saw a question about that earlier,
[2312.31s -> 2316.15s]  just to give people a, you know, kind of a glance. It doesn't have to be like mathematically
[2316.15s -> 2322.23s]  average, but just a, you know, like what generally the size of the PR is. Um, if it's only a couple
[2322.23s -> 2332.55s]  lines of code or, you know, um, yeah. Um, I think it depends on me. I'm trying to do this in a way
[2332.55s -> 2336.87s]  where I'm not gonna like, yeah, you don't have to share the actual number of you'd like an actual
[2337.59s -> 2345.35s]  this is fine, but like we have, so, okay. This is not that interesting, but, uh, well, fun fact
[2345.35s -> 2353.83s]  is that virtualization in Grokbot and in, uh, cursor is actually powered by a pretext, uh,
[2353.83s -> 2359.83s]  which is a sort of new library that someone's built. Um, that's really interesting. You should,
[2359.83s -> 2365.35s]  you should check it out, but that's not really that important. Uh, I think the average PR size,
[2365.35s -> 2371.03s]  I actually don't know. I probably, I don't know if I want to click on these. Uh, I probably can,
[2371.03s -> 2377.35s]  but I would say like they can range anywhere from a few hundred lines or 50 lines to like a thousand,
[2377.35s -> 2383.03s]  depending on what the thing is doing. Uh, so like here I'm actually like deleting a bunch of files.
[2383.03s -> 2388.47s]  So I expect that it's just, it's like mostly deletion. Uh, but yeah, it, it kind of varies.
[2389.67s -> 2393.27s]  There's no, like, yeah, there's only hard cap or are they, they're all like 50 line.
[2394.15s -> 2397.83s]  Yeah, there's definitely no hard cap, but I do encourage my agents to split up their work
[2397.83s -> 2406.95s]  into multiple PRs. Uh, I do that mostly because, uh, I like, I like the idea of the, I guess maybe
[2406.95s -> 2411.27s]  this is much harder to do now as in the world of agents and you have like so many commits,
[2411.83s -> 2415.51s]  but I like the idea that, you know, the Git history is a very rich source of context.
[2416.15s -> 2423.99s]  Uh, and I like the, I like each PR to sort of atomically describe what that small piece of
[2423.99s -> 2428.95s]  thing is doing, which also makes it easier for me to revert changes and like figure out, you know,
[2428.95s -> 2435.91s]  oh, I shipped a bug and it's just, it's here. It's not in this 40,000 line PR where who knows what
[2435.91s -> 2441.75s]  landed in there. Uh, but yeah, I don't have a hard cap on PR size.
[2442.71s -> 2447.19s]  Cool. And then, um, yeah, also quick question on like CI. So again, you don't have to go into
[2447.19s -> 2451.75s]  like, uh, the screen share of like your, your CI does, but just generally would you describe
[2451.75s -> 2460.07s]  what the CI kind of looks like, uh, or how strict it is? Uh, yeah. So, uh, well specifically for
[2460.07s -> 2466.15s]  Rockbot. So Dune is a, it's the sort of cheeky code code name for the architecture that we've
[2466.15s -> 2474.07s]  built for Rockbot. Um, the CI looks pretty annoying because there's checks for everything.
[2474.71s -> 2480.23s]  So like literally I have, um, uh, well, if you've written any React, for example, you know,
[2480.23s -> 2486.79s]  you know that one of the biggest foot guns in React is a use effect. Uh, so in, uh,
[2486.79s -> 2493.59s]  Dune and in Rockbot, we've banned use effect. So Dune is just the mental model of what Dune is.
[2494.23s -> 2499.99s]  Uh, you can kind of think of it as like Next.js for, uh, Electron apps and it's designed for
[2499.99s -> 2506.71s]  agents to write, uh, and it's like custom for, you know, our agent powered applications. Um,
[2507.35s -> 2511.99s]  so the CI checks are very like specific to that, like, you know, don't use use effect. It's, it's,
[2511.99s -> 2518.07s]  it's banned, like CI will fail, uh, and yell at you. We have like some of the more interesting
[2518.07s -> 2522.55s]  ones that people might raise eyebrows is like, I actually banned code comments as well.
[2523.27s -> 2531.03s]  Uh, which is very interesting. Um, but I've noticed that 99% of the time agents just write
[2531.03s -> 2536.15s]  code comments that kind of describe some historical thing that is actually totally irrelevant to the
[2536.15s -> 2541.59s]  code. Um, like it will often say like, you know, Oh, Lauren said you should never do this. And it's
[2541.59s -> 2546.63s]  now in the, in a code comment. Like what, like why, what that was, I didn't say that as like a
[2546.71s -> 2552.23s]  durable, you know, global rule. I just met like your, this PR sucks and you should change that
[2552.23s -> 2559.99s]  part. Uh, agents don't really understand us that well, surprisingly. Uh, and, or they kind of assume
[2559.99s -> 2566.39s]  too much and you kind of do things in like very stupid ways. So like, yeah, we just ban everything,
[2566.39s -> 2573.19s]  everything you can imagine, like the agents are bad at, we ban. Uh, so one example that we actually
[2573.19s -> 2578.95s]  suffer a lot in the agent's window is we have, uh, you know, if you've used agent's window,
[2578.95s -> 2582.71s]  you've definitely seen performance issues and, you know, we're constantly trying to fix them.
[2583.27s -> 2589.19s]  Uh, but it's like a, it's a never ending struggle because there's so many pull requests that get
[2589.19s -> 2595.43s]  merged. Every, any one of them could just regress performance or stability or reliability. Uh, you
[2595.43s -> 2599.59s]  know, the agent's window doesn't have this architecture yet. I plan to do, bring this
[2599.59s -> 2606.31s]  learning back there and kind of refactor everything there. Uh, but, uh, it just regresses
[2606.31s -> 2613.83s]  super often, uh, because, uh, there's just one example is like, we have very poor, um, isolation
[2613.83s -> 2618.71s]  between processes. So like on, you know, on, on electron, you have a renderer thread that renders
[2618.71s -> 2623.59s]  your UI, but you also have like a main thread that you can run other code that, you know, doesn't need
[2623.59s -> 2630.95s]  to block the renderer. Um, but we do a poor job of separating those things. And so oftentimes
[2630.95s -> 2636.39s]  you just accidentally have code that gets pulled into running on the render thread. And then all
[2636.39s -> 2641.75s]  of a sudden you're competing with the renderer that, you know, that has a very, if you want like
[2641.75s -> 2648.55s]  60 FPS, you have to, every frame that gets drawn has to be done in 16 milliseconds. So very, very
[2648.55s -> 2654.47s]  small, you know, deadline per frame, uh, if you want, you know, a very smooth product. Uh, and when
[2654.47s -> 2659.27s]  you start building, bringing in, accidentally bringing in, you know, things that are like very
[2659.27s -> 2665.51s]  computationally heavy or they have a lot of IO, uh, then you just get into like a lot of jank
[2665.51s -> 2671.35s]  and your, your FPS really drops. You start, uh, you know, losing frames, you get long tasks that
[2671.99s -> 2675.43s]  take more than 16 milliseconds and you just get this really choppy experience.
[2675.59s -> 2680.23s]  So all of those patterns that we've learned, basically building Electron apps, we've encoded
[2680.23s -> 2685.91s]  into this framework and it becomes like a hard failure. So I literally, in, in GROCKBOT, we
[2685.91s -> 2693.91s]  literally have a directory called Electron main, Electron renderer, and we have, uh, import, uh,
[2693.91s -> 2699.19s]  CI, I guess, where we actually check the dependency graph to make sure they are not accidentally
[2700.15s -> 2708.07s]  importing code from one directory to another. Uh, so that's enforced by CI, um, as well as
[2708.07s -> 2715.75s]  BugBots, uh, which is our, which cursors, um, like code review tool that runs on CI, uh, you know,
[2715.75s -> 2722.95s]  in our agents MD, it's everywhere. Like, so I, I, I, um, I have this thing here where I talk about,
[2722.95s -> 2728.71s]  like, um, you know, like there are multiple layers, I think, for building a good code base.
[2729.51s -> 2734.23s]  Obviously the code base is one where, uh, if you have an architecture like this,
[2734.23s -> 2740.31s]  where it's extremely strict, uh, you know, the, the, the way to build features is very conventional.
[2740.31s -> 2745.83s]  That's like the strongest, strongest level of enforcement because agents just love to copy
[2745.83s -> 2751.91s]  existing patterns. So, uh, one example of this in GROCKBOT is like, we have this, these concepts
[2751.91s -> 2756.87s]  called like a feature and we have entry points and transcript cards, like, oh, you know, the cards
[2756.87s -> 2763.51s]  that you see in the chat, these are all like, like nouns, I guess, in, in the framework. And so
[2763.51s -> 2770.39s]  there's a very conventional way of creating them. And so like a feature is all in a single directory
[2770.39s -> 2775.75s]  as an example. And so all of the code that contributes to that feature lives in one directory.
[2775.75s -> 2780.47s]  So it's all co-located in one place, makes it super easy. You know, agents don't have to like,
[2780.47s -> 2785.51s]  uh, grab around and try to figure out like where all the things are. It just looks at the feature
[2785.51s -> 2791.43s]  and like, oh, okay, I'm working on the onboarding feature in GROCKBOT. Uh, I'm just going to work in
[2791.43s -> 2798.39s]  this directory. And for 80% of the work, it's mostly just very encapsulated there. But, uh,
[2798.39s -> 2804.23s]  like it's like very, it's like designed again for, you know, like the dumbest agent, like you don't
[2804.23s -> 2810.23s]  have to think, right? The, the, the, one of the key principles I have for this framework is like
[2810.87s -> 2818.15s]  the shortest, the shortest path is the best path. So, uh, because that plays exactly to how agents
[2818.15s -> 2822.39s]  love to write code is that they like to take shortcuts really, you know, they'll find the
[2822.39s -> 2830.15s]  quickest way to solve the problem. So why not make that the best way to solve the problem? Uh, so I,
[2830.15s -> 2836.55s]  I, I probably won't get into all the specific details. Um, and, uh, the, the, this framework's
[2836.55s -> 2840.23s]  really more of a collection of ideas and principles rather than something that will
[2840.23s -> 2844.95s]  open source. Uh, you can, you can, you know, screenshot this, I guess, if you want and,
[2844.95s -> 2850.15s]  uh, tell your agent to, uh, do something, build, build something like this for you too.
[2851.51s -> 2856.79s]  Um, yeah, but it's really all about the layers. Uh, you know, like the, the code base is one part
[2856.79s -> 2863.35s]  with features, uh, and directories and, you know, import or blocking import dependencies
[2863.99s -> 2869.67s]  that shouldn't be imported. Uh, but, and, and it all enforces that and static analysis. So like,
[2870.23s -> 2877.19s]  uh, there's CI checks. We have a lot of links for bad patterns that we observe, uh, compiler
[2877.19s -> 2884.31s]  diagnostics. Uh, there's also rules and bug bot, which are, um, I think like three, four, five are
[2884.31s -> 2891.99s]  more soft, right? These two actually make, make CI read, right? So that, you know, there's a hard
[2891.99s -> 2900.63s]  constraint where the agent can just write crappy code for rules and skills and bug bot. Your agents
[2900.63s -> 2908.55s]  can still forget, right? You can still, or it may not always consistently apply them. So I like to
[2908.55s -> 2915.43s]  layer them, but I don't, I don't like to rely on them as the only source of enforcement because
[2915.43s -> 2919.75s]  it's very, very soft, right? And if you, if you only have rules and bug bot and skills and the
[2919.75s -> 2925.19s]  style guide for your code, you will, it's only a matter of time before your code base looks like
[2925.19s -> 2930.95s]  complete trash. Um, sorry to say that, but, uh, I definitely recommend, yeah, like, you know, investing
[2931.83s -> 2938.15s]  in, you know, things that can be hard and forced, right? And this is why, you know, maybe the choice
[2938.15s -> 2944.63s]  of tech stack that you use is also very important. Um, like I think for example, rust is sort of
[2944.63s -> 2950.95s]  making, you know, it's like getting super popular again, uh, because the compiler is so strict,
[2950.95s -> 2955.19s]  right? The compiler enforces so many different things, you know, there's a borrow checker that
[2955.19s -> 2961.03s]  you have to appease. And as long as you make sure your agents don't write unsafe code blocks, uh,
[2961.03s -> 2966.47s]  you can more or less feel somewhat confident that if the code compiles, it probably works and it's
[2966.47s -> 2973.67s]  good. Uh, but you'll see, it gives you that level of trust and confidence that you, as a human
[2973.67s -> 2980.07s]  engineer, no longer need to go and check it yourself. You know, you, you rely on code and
[2980.71s -> 2988.79s]  static analysis to actually make that, uh, a lot smoother. Um, and I guess the worst part,
[2988.79s -> 2995.43s]  the worst place to be in is if you are stuck in code review land where you actually enforce all
[2995.43s -> 3002.15s]  of the constraints, the invariance in your code base by literally the human person saying, you know,
[3002.23s -> 3006.87s]  reading the code and like, okay, you should not do this, right? Every time you have to do that,
[3006.87s -> 3011.83s]  you should consider that as a code smell, like a, uh, anti pattern. And you should say, okay,
[3011.83s -> 3018.31s]  instead of me commenting on the PR, how do I turn this into a hard rule, right? How do I turn this
[3018.31s -> 3024.31s]  into a lit rule? How do I turn this into a CI failure or how do I even categorically eliminate
[3024.31s -> 3031.03s]  this problem, uh, entirely? Uh, I can talk about another migration I've done, but I'll probably
[3031.03s -> 3036.15s]  pause here. Sure. Yeah. I feel like that's, that's where I am to be honest is, is it what
[3036.15s -> 3040.15s]  you're describing right now, which is that like, I don't have all of these rules. Uh, so I have
[3040.15s -> 3044.47s]  some things to go do after this session, uh, in terms of being able to scale my agents. I'm,
[3044.47s -> 3049.83s]  I'm definitely on like the, uh, you know, maybe a couple of parallel ones locally stage, like two
[3049.83s -> 3054.79s]  to three locally. And I'm sure most people here are on the same. So, uh, yeah, I know we only have
[3054.79s -> 3058.47s]  a couple of minutes left. Uh, Lauren, was there anything else that you wanted to, to highlight?
[3058.79s -> 3062.47s]  Obviously there's lots of questions, so I can grab more, but I want to give you a few minutes,
[3062.47s -> 3065.27s]  if there's anything else you want to talk about. Well, I think I've been yapping for quite a lot.
[3065.27s -> 3071.59s]  So maybe let's just do questions. Okay, cool. Uh, one question that had a couple of, uh, came up
[3071.59s -> 3077.35s]  a couple of times was just around like token usage. So the question is like, is, is what you're
[3077.35s -> 3083.99s]  describing a realistic thing for people who are on, you know, uh, a normal set of token usage? They
[3083.99s -> 3089.99s]  don't have, you know, obviously unlimited tokens, uh, to work with. Well, I think that's a really
[3089.99s -> 3095.19s]  good point. I mean, like, obviously, you know, I work at an AI lab where we have unlimited tokens.
[3095.19s -> 3103.19s]  So, uh, I definitely cannot say that, you know, this is something everyone should do in the exact
[3103.19s -> 3107.35s]  same way that I did it. I think it's possible to get to this point without, you know, breaking the
[3107.35s -> 3113.03s]  bank. But, you know, if you're like an engineering leader or, you know, you're, you're, you have a
[3113.03s -> 3121.35s]  startup that you lead. Um, I think to me it's a question of ROI. Um, and it's like, uh, yes,
[3121.35s -> 3127.43s]  you spend a lot of money on tokens in the upfront stage. You know, like refactoring your code base
[3127.43s -> 3131.75s]  is going to take a lot of tokens. Uh, adding all these things, uh, is going to take a bunch of
[3131.75s -> 3138.71s]  tokens. But if we're heading into a world where agents are writing all the code and, you know,
[3138.71s -> 3143.27s]  you want to be very lean, right? You don't want to have to hire, you don't want to be, you don't
[3143.27s -> 3147.99s]  want to become like meta, right? Like, I mean, like in terms of, you don't want to become a 10,000
[3147.99s -> 3154.07s]  person engineering org because, I mean, that's a cool problem to have, but also, you know, you have
[3154.07s -> 3160.71s]  so much overhead. There's like planning, you know, like, it's, it's, uh, personally, I, I wouldn't,
[3160.71s -> 3166.71s]  uh, it's not super fun, but, um, I think you want to stay very nimble, right? And you want to,
[3166.71s -> 3172.39s]  you want to be like agents are all about allowing you to do things that you couldn't do before.
[3172.39s -> 3177.51s]  That's really, to me, like the value of agents, you know, it's not just storing tokens on every
[3177.59s -> 3181.91s]  single little thing. But, um, to me, like the thing I couldn't do before is like,
[3182.71s -> 3189.59s]  enforce this level of constraints in a code base by myself, right? Like I'm just a single person,
[3189.59s -> 3196.71s]  you know, uh, it would have taken me years to build this framework, uh, and do all the refactoring
[3196.71s -> 3202.71s]  and test everything myself and verify, you know, like run, imagine if there, it was just me right
[3202.71s -> 3209.27s]  now in, in pre-agent era, just like running, you know, it would take me so long, right? And my
[3209.27s -> 3215.19s]  salary is pretty high, right? Like, so, you know, the, the question I think an engineering leader
[3215.19s -> 3221.35s]  might have is just then, you know, like, what is, there's a trade-off of, do you hire someone to do
[3221.35s -> 3228.39s]  this or do you spend the tokens to set up a code base so that even the, the most naive, right,
[3228.39s -> 3234.47s]  the dumbest agents can do a good job. And when you actually get to this point, like even agents
[3234.47s -> 3240.63s]  that are not, you know, fable size do an excellent job of writing code. And this pays a lot of
[3240.63s -> 3248.07s]  dividends as well for me personally, where I have empowered not just myself, but again, like PMs,
[3248.07s -> 3253.51s]  designers, engineers who are not familiar with RockBot to just contribute in a way that
[3254.31s -> 3260.23s]  is sustainable. So I think, yeah, it's definitely like a trade-off for sure. You know, like nothing
[3260.23s -> 3267.27s]  is like free for sure. And tokens are pretty expensive. But oh, actually, I don't know how
[3267.27s -> 3272.79s]  many of you have seen this, but we actually announced Grok 4.6 today. So very exciting,
[3272.79s -> 3279.43s]  finally out. So yeah, Grok 4.6 would be like a great, it was very, very smart. It's really
[3279.43s -> 3285.83s]  good on the benchmarks. And it's the same, the tokens, well, I hopefully I'm not saying
[3285.83s -> 3292.87s]  this incorrectly, but I believe the cost per token is the same as 4.5. So you're actually getting
[3292.87s -> 3300.47s]  more intelligence for the same cost. I think this is an area that Cursor tries to, Cursor and SpaceX
[3300.47s -> 3307.75s]  AI try to really optimize for like that Pareto frontier of, you know, cost versus intelligence.
[3308.55s -> 3313.03s]  You know, we don't necessarily want to build the biggest model ever because that is extremely
[3313.03s -> 3317.91s]  expensive to run. It's really about like, how do you find that sweet spot, right? You don't need
[3317.91s -> 3322.63s]  a giant model, but it's just super smart, right? And it's not very expensive for inference.
[3324.71s -> 3331.11s]  But yeah, I think to kind of round it up, I think it's like, there's a,
[3331.67s -> 3337.43s]  if you do your own analysis, I feel like it's pretty positive. It'll be pretty positive that
[3337.43s -> 3343.59s]  the ROI you get from investing and stuff like this just empowers not just yourself,
[3343.59s -> 3348.79s]  but your whole team to be so much more productive, right? Like imagine if you have an army of
[3348.79s -> 3355.91s]  engineers like me who are shipping so much improvements and bug fixes, you know, every day,
[3355.91s -> 3362.39s]  right? Like that is pretty exciting. Cool. One last question before we wrap up.
[3362.39s -> 3367.03s]  This one is for the people in product on the call. So let's say we do have an army of
[3367.03s -> 3372.07s]  engineers who are shipping like Lauren. I'm just curious, like how is the product team or other
[3372.07s -> 3376.47s]  functions of your company keeping up given that like, if you're shipping so quickly,
[3377.51s -> 3381.59s]  are they using AI more to do their jobs? Like as much as you can speak to that, I know obviously,
[3381.59s -> 3384.47s]  you know, like you're not in that role, but just curious about how that works.
[3386.15s -> 3393.11s]  I think this is where Grokbot has been actually exceedingly powerful where, so before Grokbot,
[3393.91s -> 3399.83s]  obviously Cursor only had Cursor, like we only had Agent's Window, we had a CLI, we had an IDE,
[3400.55s -> 3406.63s]  and these are really like power user tools, right? They're designed for developers, so it's very,
[3406.63s -> 3412.47s]  very developer-centric. You can do knowledge work in them, but the UI is not really optimized for
[3412.47s -> 3418.55s]  that. So we actually didn't really have, well, I think like a lot of people in like, you know,
[3418.55s -> 3424.79s]  GTM product, like they might have used Cursor to do their work, but it definitely wasn't like a
[3424.79s -> 3433.67s]  delightful experience for them. I think now with Grokbot, it's become, Grokbot is basically like
[3433.67s -> 3439.67s]  their Cursor moment for people who are not in tech, in my opinion. Like it's like a very,
[3439.67s -> 3445.19s]  very accessible way to use agents in a very comfortable, very familiar interface. It looks
[3445.19s -> 3450.79s]  like iMessage, and it's very fun too, you know, you can give your agent a fun name,
[3451.75s -> 3457.51s]  you can have, you can kind of do orchestration in a very like natural way, where you can sort of,
[3457.51s -> 3461.03s]  you know, each agent is like a person, and now you've got a team of agents like working on,
[3461.03s -> 3465.99s]  you have one agent per account that you manage, as an example. Or if you're a PM, you have,
[3465.99s -> 3469.83s]  you know, you can have an agent that summarizes all the work that Lauren did last night,
[3469.83s -> 3474.71s]  and then now you know what I did, right? So I think our PMs are leveraging that a lot,
[3475.27s -> 3480.23s]  and they're shipping code too, so you know, like oftentimes they will just say, oh here's a bug,
[3480.23s -> 3484.63s]  I fixed it, can you look at it, and then I'll go review it, and actually it's just perfect,
[3484.63s -> 3490.95s]  I'm like, okay, stamp. So I think that shows that, you know, the Dune architecture is holding up,
[3490.95s -> 3497.83s]  right? The, all the really strict constraints allow people who are not experts in engineering
[3497.83s -> 3503.35s]  to contribute at a high level, so I feel like I'm already seeing that pay off a lot, where,
[3504.23s -> 3511.91s]  you know, designers and PMs are just able to ship features directly, and that just makes
[3512.63s -> 3520.47s]  the Grokbot team super fast, right? Where we can ship so quickly, and we have a lot planned,
[3520.47s -> 3526.79s]  so I'm very excited to, you know, to ship more, ship more stuff.
[3526.79s -> 3532.39s]  Yeah, that's awesome. We are at time, so I guess Lauren, if folks want to support you,
[3532.39s -> 3538.79s]  maybe go try out Grokbot, try out 4.6, and you know, provide some feedback, but yeah,
[3538.79s -> 3542.55s]  this was awesome, really appreciate you taking the time. Thanks everyone for all the messages
[3542.55s -> 3545.99s]  in the chat, lots of good questions, I know we didn't get through everything, but as I kind of
[3545.99s -> 3550.55s]  said at the top, way more questions than we could get through, but yeah, really, really thanks,
[3550.55s -> 3554.39s]  thanks for joining, thanks everyone for joining, and hopefully you enjoyed the session.
[3555.59s -> 3560.71s]  Yep, yeah, I see, thanks for having me, and if you have any more questions, yeah, just DM me on
[3560.71s -> 3565.51s]  Twitter, I'll open them up, I guess. You're gonna get a lot of DMs.
[3567.51s -> 3572.31s]  So yeah, DM me, maybe I'll do like a Twitter space at some point, that's all, for more questions,
[3572.31s -> 3577.19s]  but really appreciate everyone for showing up, you know, taking an hour out of your day.
[3577.19s -> 3579.11s]  Yeah, alright, thanks all, I'll see you the next one.
[3579.11s -> 3580.39s]  Okay, thanks everyone, bye.
