00:06how did you end up working in tech for
00:10Tech was earlier I started you know
00:12really young and I got really into Linux
00:15when I was 14 or so and um yeah it was
00:18right around the time when the web was
00:19like a new thing and you had to work to
00:21kind of get on the web and then Linux is
00:23I just sort of found it it popped into
00:26my life and I really liked the I read
00:28the GPL really early at 15 and
00:33is the idea of helping in the open and
00:38making it freely available
00:40so other people could learn from things
00:43like I was learning from things seemed
00:45so I went and spent many years just
00:48working on open source stuff and
00:50I don't know I guess I didn't realize
00:51there was like there was kind of a
00:54there was like Legacy technology right
00:57like Windows was popular at that time so
00:59people ran their you know data centers
01:02on Windows and it works on Windows and
01:05um this Linux thing was like very
01:07strange and people didn't really know
01:09spent many hours compiling kernels and
01:11hacking on stuff and uh yeah and what
01:15was the thought process behind starting
01:18oh hack that uh yeah so I had just
01:22finished like four or five years at
01:24VMware and uh I wanted to get into
01:27startups I knew I knew that and then uh
01:31so I left VMware and I started working
01:32on an education startup like many of us
01:35do uh I don't know if you know this but
01:37like many many Founders start with like
01:40the idea of an education startup
01:42it's like a rite of passage yeah so I
01:44spent I don't know nine months working
01:45on on on on that most of us have done
01:48either that or consumer social sure yeah
01:51and uh it's uh turns out education is
01:55um yeah you nod your heads knowingly
01:56yeah and uh so after nine months I was
01:59like all right this isn't going anywhere
02:00uh I I know I don't know if there's a
02:03value prop here I mean that's that was
02:05the values that I learned that like you
02:06have to make something that is
02:10both achievable and that people want to
02:12pay for or spend their time on so yeah
02:14then I was just kind of fishing around
02:17um I was living in like a warehouse in
02:19San Francisco with a bunch of Burning
02:21Man people and uh we were having trouble
02:23organizing burning large-scale Burning
02:26Man projects and so uh I've forked ether
02:29pad and started hacking on it
02:32uh recruited a friend to
02:35in the community to start working out
02:36with me and yeah just grew from there we
02:39um you know before we did YC we had most
02:42many of the large burning man camps
02:44using it to organize their their builds
02:46can you describe the product experience
02:48oh yeah it was a you know it was a
02:50real-time text editor uh kind of like
02:54um Google Docs is the only other one
02:56that did that at that time and uh it was
02:59kind of nice because uh it would
03:01highlight who said what so you could go
03:03track down if somebody had a
03:05contribution be like oh what did you
03:06mean or I heard you say something about
03:09this so that's very useful in like um
03:12large-scale anonymous pseudo-anonymous
03:14groups where you don't really know who
03:17has ownership over what like burning man
03:21let's see and then we did YC and then we
03:23got a bunch almost many of those people
03:25did it at my like my hack was to go do
03:29YC and try and get all the companies
03:31doing my CDs my product
03:33which many of them did I also took like
03:35very extensive notes of all the ycee
03:37presentations using the product that
03:38everyone would then like look at and uh
03:41we were able to get stripe and stripe
03:43used this for many years actually they
03:45built for for the I think we were their
03:47first knowledge base they used it for a
03:49long time a bunch of other big companies
03:52um yeah we're very lucky
03:57Dropbox acquisition he worked on what
04:00became paper how do you think about like
04:02what you want to go work on next after
04:06I kind of did you know I spent the next
04:07few years kind of poking around at stuff
04:11I knew that I wanted to make a robot
04:14that does stuff for you
04:15yeah so there's a company called Magic
04:17so I worked at this chemical magic they
04:19were doing this like uh text based
04:21personal assistants uh do you remember
04:23this one I I think uh just like
04:26everybody starts the education startup
04:27magic is one of those names of cycling
04:28and there's a really cool AI company
04:30right now called Magic as well so I feel
04:32like there's also these names that kind
04:34of persist from generation to generation
04:35which I think is really cool I'm sure
04:37you know it's code gen yeah yeah yeah no
04:39yeah uh I haven't seen the demo yet but
04:41it sounds like they're doing the right
04:43um there's a few people doing the kind
04:45of whole repository uh uh changes so it
04:49seems like a great Direction
04:51uh what What's like one learning from um
04:55was like a company early in trying to do
04:58yeah machine learning in general uh no
05:02no it wasn't trying to do machine
05:03learning it was all app space it was
05:04super heavy op space so uh you know they
05:07had teams of people and it was 24 hours
05:09and they would cycle in and you know
05:10they'd lose context and like they were
05:12all busy because they're trying to
05:13they're trying to like deal with lots of
05:15people with lots of requests all the
05:16time and so really it was like a crash
05:19course in like human behavior right like
05:21what do people do under stress how do
05:24they act what do they say what can you
05:25train what can't you train like
05:28um uh can you bucket stuff and the
05:30answer is no like humans are complicated
05:33especially in Texas one so the beauty of
05:36the web in like traditional uis right is
05:39that you fill them in and like if it if
05:42you can't if it doesn't do what you want
05:43then you make the decision to either go
05:45forward or not go forward right text has
05:47this uh annoying property of you can be
05:50all the way at the 99 Mark and then
05:54change the goal entirely right and so
05:57yeah so it's complicated it's hard to
05:59it's hard to understand what uh what
06:01people want it's hard to understand the
06:03complexity involved especially when
06:04you're dealing with the real world
06:06um flights get delayed passwords get
06:09lost do you think we're the last
06:10generation to deal with that in other
06:12words it feels like we're about to hit
06:13transition point in which agents can
06:14actually start doing some of these
06:16things for us for real when before I
06:18think all these products really started
06:19with these operations heavy approaches I
06:21remember you know there was a really
06:23early sort of um precise search engine
06:26called artbark for similarly if you kind
06:28of look behind the hood it was a lot of
06:29Ops people and there was a little bit of
06:30algorithm right yeah that was really
06:32with like um you would like sort of
06:34describe what you're good at or
06:36something and then they would try and
06:37send questions to you yeah exactly they
06:39kind of Route things but I think there's
06:40actually people doing some of the
06:42routing at least or I can't exactly
06:43remember right but I think it was you
06:45know I think a lot of people wanted to
06:46build these really complex sort of bots
06:49or agents that were doing really rich
06:50things and the technology just wasn't
06:51there it feels like for a period of time
06:53there's all system startups I remember
06:55one company that I got involved with
06:57that was um trying to do like virtual
07:00scheduling and assistance you know six
07:02seven eight years ago and again it felt
07:04like it was a little bit early cold
07:21yeah yeah it's like um
07:27yeah what we were able to do with magic
07:28as a like an aside there you know I
07:30started working there because I wanted
07:31to work on I wanted to work on the AI
07:34part uh I think somewhere in there
07:36Facebook M started as well uh and it was
07:41just it was a fun place to try and learn
07:43everything I could about solving those
07:46before Transformers there was a sequence
07:48of sequence was kind of the previous
07:51and um we were able to like take all the
07:54all the histories of the chats between
07:56assistants and and people and train a
07:59little model on it and uh
08:01a little by today's metrics uh uh and
08:05run it and it would like it would show
08:07some gray text in the little text bar
08:09and people could edit it and hit enter
08:11and if the operators yeah yeah the The
08:14Operators and um yeah we would measure
08:16how much time they sense they spent uh
08:18typing with it and without it and so it
08:20would save like half an hour or across
08:22100 people per day across an eight hour
08:25shift uh so yeah that was like my first
08:28shipping AI product I guess and then
08:30how'd you end up going to um Microsoft
08:33like that's oh there was a bunch of
08:35other yeah there's a bunch of other
08:36stuff along with there was a so after
08:39uh I got into crypto
08:41uh my friend was doing age captcha which
08:45uh was a sort of a capture Marketplace
08:48which is now like something like the
08:51number one or number two capture service
08:53in the world which is crazy yeah so kind
08:56of launched that that was fun uh you
08:59know annoyed people the world over for
09:02many man hours in Aggregate and then
09:04worked on uh left that to work with uh
09:07Moxie on on a cryptocurrency for signal
09:12um so that was really fun
09:15um complicated and it all worked in a
09:19um so you know we were shooting for
09:21venmo quality which I think pretty much
09:25so when you think about crypto in the
09:26context of AI because people talk about
09:28it in a few different contexts right one
09:30is uh you have programmatic
09:33um sort of money as code running and so
09:35that could create all sorts of really
09:36interesting things from an agent-driven
09:39perspective but then the other piece of
09:40it is identity and some people think I
09:43mean world point would be one example
09:44but there's other examples of
09:45effectively trying to secure identity
09:46cryptographically on the blockchain in
09:48an open way and then using that identity
09:50in the future to differentiate between
09:51AI driven agents and people do you think
09:54that's going to be important or does
09:55that stuff not really matter in terms of
09:56the identity portion of not only crypto
09:59but just like how we think about the
10:02the honest answer is I think we're going
10:05to go through a many year period of
10:07extreme discomfort where uh AIS pretend
10:12to be things or or confuse people or
10:15extract money from your grandparents or
10:19um you know drain people's life savings
10:22in ways that are uh that are scary and
10:26you know open AI is trying to do their
10:28best but for some reason the focus has
10:30been on open AI doing everything and
10:32instead of like we should go build the
10:34systems that prevent that we should go
10:36pass the legislation that drops the
10:39hammer on people doing that stuff we
10:40should go all this kind of stuff that is
10:43um unfortunately it seems like we're
10:45gonna need some really bad things to
10:48um we align correctly I'm not really
10:51scared about uh ai's killing us although
10:54I'm very grateful that there are people
10:56that are thinking about it
10:58um I'm more worried about bad people
11:00using new technology to hurt us yeah
11:02Elia Vermeer has some really interesting
11:04thoughts on this because he was one of
11:05the main authors or he was the last
11:06author on the Transformer paper before
11:08he started near and he's brought up
11:10these concepts of like how do you stress
11:11test Society relative to the coming wave
11:14of AI which I think is an interesting
11:15concept yeah it's a great it's a great
11:16way to look at it like uh it's not as
11:19bad as it could be right if you think
11:22um most of the things that you want to
11:27uh either have a Spam blocker or are
11:30somewhat difficult to create an account
11:34um you know doing a better job of uh of
11:37a sock puppet account filtering is going
11:39to be really important going forward
11:41um you know I like what cloudflare is
11:44doing with their kind of fingerprinting
11:47um instead of visual captions which are
11:49not good enough anymore
11:51you know one thing that is like kind of
11:52a Saving Grace Grace here is is that uh
11:55you know many of the things that uh you
11:59would want to do cost money so calling
12:02everybody costs money
12:04um uh texting everyone should be
12:06hopefully illegal soon but also costs
12:08money maybe not enough money to to
12:11prevent these things but probably not
12:13enough those agents can make money right
12:16and just look at the trade-offs of costs
12:18yeah I think it's interesting yeah for
12:20sure you know the other thing the other
12:22nice thing there is that like Grandma's
12:24don't have crypto right they have bank
12:26accounts and bank accounts can be traced
12:29um I I guess I would say it's not you
12:32know somewhere in the middle it's like
12:33uh you know it's imagine there's an you
12:37North Korea has been trying to do this
12:38to us for a long time right now there's
12:40a North Korea that has more resources or
12:42is more distributed or whatever it's you
12:45know we have some mitigations we need
12:47more we need to be thinking about it a
12:49lot more how did you end up at GitHub
12:52and how did you end up working on
12:54oh yeah okay uh let's see so while I was
12:58working on mobile coin I uh my dad's
13:01kidneys failed and I tried to donate
13:03them and um a kidney and uh they found a
13:07lump in my chest as part of the the
13:11and uh I had to have my right most of my
13:14right lung removed in 2018
13:29yeah healing it's weird healing from
13:31internal injuries takes a lot longer
13:33than you than you think anyway happy
13:36story is that uh I don't cancer now for
13:39uh over four years and
13:42uh in my dad's kidneys got he got a
13:44transplant also so things are good and
13:47so after uh I don't know
13:50I guess I was recovering for for quite a
13:53while and then I went and uh begged my
13:56friend for a job I figured I should
13:59start working again and so I worked on
14:01some random stuff at first uh
14:04uh a converted GitHub to using their own
14:06product to build GitHub which was kind
14:08of fun they weren't yeah so I think
14:10people still use code spaces now to
14:12build GitHub which is pretty cool
14:15um but yeah then this kind of
14:16opportunity to work with openai came up
14:20uh yeah because I had been tracking AI
14:25um and was pretty aware of what was
14:27going on I jumped on it
14:28was that proposed by open AI or by
14:30GitHub or who kind of initiate it at all
14:32so I don't know the exact settings I
14:35know that uh openai and Microsoft were
14:37working on a deal for super computers
14:40um so they wanted to build a big cluster
14:41for training and there was a big deal
14:43that was being worked out and there was
14:45some software kind of Provisions thrown
14:46in I think office and Bing probably
14:49and GitHub was like oh okay well maybe
14:51we can let's like uh
14:55maybe there's something GitHub can do
14:57here uh I think openai threw a small
15:00through a small fine tune over then was
15:01like here's a here's a small model
15:03trained on um on some code see if this
15:06is interesting you know
15:07so we played around with it and
15:11I don't know this was uh I have to
15:13remember now I think this is before I
15:16knew very much so it was definitely not
15:17a da Vinci size model that's for sure
15:19um I don't know I don't know what says
15:23yeah I don't know what size was
15:25um probably less than 10 but I can't
15:27I learned later this was basically a
15:29training artifact so they had wanted to
15:31see what introducing code into their uh
15:34into their base models would do
15:37um I think they had positive effects on
15:38Chain of Thought reasoning
15:40code is kind of linear so
15:43um you can imagine that you know you
15:45kind of do stuff one after another and
15:50and yeah it was not that good it was
15:52very bad uh you know they it was I think
15:55just like I said just an artifact and
15:57just a small sample of uh GitHub data
16:00that they had crawled and
16:03um yeah we played around with it this is
16:05before I actually I joined I was sort of
16:07uh me and this guy Albert uh Ziegler
16:11were the first two after uh uge who um
16:16who got a hold of this model and started
16:18playing with it he was able to say like
16:21well you know like it doesn't work most
16:22of the time but here it is doing
16:23something you know here's here's it was
16:25only python at that time here it is you
16:27know generating something useful
16:30we didn't really understand anything you
16:31know it's just sort of lobbed over at us
16:33so that was enough to like okay well you
16:35know go fetch a couple of people and
16:36start working see if there's anything
16:38there we didn't really know what we had
16:40so the first you know task was to go
16:42test it out see what it did
16:44um so we crowdsourced a bunch of uh
16:47python problems in-house stuff that we
16:49knew wouldn't be in the training set and
16:51then we we started work on
16:54um fetching repositories and finding the
16:56tests in them so that we could um
16:58basically generate functions that were
17:00being tested and see if the tests still
17:02there had been like a brand new uh Pi
17:04test feature introduced like recently
17:06that that allowed you to see which
17:08functions were called by the pi test so
17:11you'll find that function zero its body
17:13ask the model to generate it and then
17:15rerun the test and see if it passed
17:18and uh I think it was
17:20less I don't know 10 something like that
17:23of of those guys and the dimensions are
17:25kind of like how many chances do you
17:29um how many students do you give it to
17:30to solve something and
17:33um and then how do you test whether it's
17:38Standalone tests that was we had people
17:42write test functions and then we would
17:43try to generate the body and uh if the
17:46tests passed then you know it works and
17:48for the in the wild in the wild test
17:50harness we would download a repository
17:52run all the tests look at the ones that
17:54passed find the functions that they
17:55called make sure that they weren't
17:57trivial generate the bodies for them
17:59rerun the test see what it passes to it
18:00and then you get your percentage
18:04yeah I mean it was something like some
18:06very very low percentage up front but we
18:09knew that there was kind of a lot more
18:15um all of github's code into the model
18:19um and then a bunch of other tricks that
18:21we hadn't you know we hadn't even
18:23thought of at that time
18:24and eventually you know it went from you
18:27know less than 10 on in the wild test to
18:31so that's like one and two tests it can
18:33just generate code for which is insane
18:35right somewhere along the way you know
18:37there was like 10 to 20 to 35 to 45 you
18:42know it's kind of like improvements
18:44along the way somewhere along the way we
18:46did more prompting work so that the
18:49um somewhere along the way they used you
18:51know all the versions of the code as
18:53opposed to just the most recent version
18:55they used diffs so that it could
18:57understand small changes like
18:59yes it got better and so but when we
19:02first started we were just winning all
19:03the ad we were just trying to figure it
19:06and uh at the time they were thinking in
19:08terms of like maybe you can
19:11replace that overflow or something you
19:13know do a stack of workflow competitor
19:15um was that the first like product idea
19:18I think you guys had for it I I don't
19:20know that we had that idea I think that
19:21was kind of like uh Beyond height yeah
19:24yeah yeah there was more like a
19:26um it'd be nice you know be nice if you
19:28could make something that computers like
19:30overflow because we have all this code
19:31wouldn't it be nice to leverage it
19:35yeah and so we made some uis but like
19:37early on you know it was like early on
19:39it was bad so it would be like you'd
19:41watch it and it would run and most of
19:43them would be bad and be like there'd be
19:45like one success and be like oh sweet I
19:47got a success but I had to wait you know
19:49some number of seconds for Success User
19:52Group just like the six of you like some
19:55larger group we made a little yeah the
19:57first iteration was just like an eternal
19:59tool that people would help people write
20:01these tests and then
20:03um we wanted to see if maybe we could
20:04turn that into some UI that people would
20:07use could where if there's some way to
20:09cover up the fact that one in ten things
20:10pass right so you try to we've tried a
20:13few UI things there and then uh it was
20:16actually open AI it was like it would be
20:17nice if we could we're testing these
20:19model fine tunes it'd be nice if we
20:20could like test them more quickly can
20:22what about doing like a vs code
20:25um just do autocomplete so we did
20:27autocomplete at first and that was kind
20:29of that was a big jump you know because
20:30they they were still thinking in terms
20:33of like stack overflow you know but this
20:36is it's like you know I didn't have any
20:38ideas basically I didn't know how to
20:40beat stack overflow with this thing but
20:42we could come up with we could play with
20:43some stuff in in vs code that was maybe
20:46closer to the code you know
20:49um and so we tried uh first we did
20:51autocomplete and that was kind of fun it
20:53was useful you know um it would it would
20:56it would show this little pop-up box
20:57like autocomplete does and you could
20:59pick some strings and so that format
21:01actually you know the the usage was you
21:03know it's fine uh it wasn't the right
21:06metaphor exactly right but you've got
21:08like this code generated mixed in with
21:12um specific terms that are in the code
21:14and it's a little it's not exactly the
21:18um we tried things like uh uh adding a
21:21little button over top of empty
21:23functions so you would go generate them
21:24or you could like hit a control key and
21:27it would create a big list on the side
21:29that you can choose from or there's a
21:32little pop-up thing so basically tried
21:34every single UI we could think of in vs
21:37and multiple Generations like the list
21:39that didn't work yeah none of them like
21:42really works like I think lists were
21:44like you know maybe you'd get one
21:46generation per person per day and this
21:49is just a small sample it's just like a
21:50few people that were interested in at
21:52GitHub language nerds or
21:55um people that have written tests for us
21:56uh and open-air people
21:59so so then you know I had some very
22:02early on I had this idea that I should
22:06uh gray text autocomplete
22:09which was I was like enamored with that
22:11product it's like it was the first
22:15you know quote unquote large language uh
22:17model deployment in the wild
22:20um it was fast it was cool like the
22:23paper's great like they give you all
22:25sorts of details on how they do it all
22:26the all the workarounds they had to do
22:27so that was always in the back of my
22:29head you know and it it was bad also it
22:31was like you know those completions are
22:36it seemed like the right the right thing
22:39um anyways somewhere along the way after
22:41after we tried all the UI you know sort
22:43of come up with some idea for
22:47PS code didn't support this so I tried
22:48to hack it in I finally came up with a
22:50way to hack it in and uh enough to make
22:55like a little demo video and uh was
22:58their support to like build real support
23:00for it within the organization
23:03it's a little complicated I I guess you
23:05know we were pretty much a Skunk Works
23:06project so no one really knew
23:09you know no one knew about us so we
23:10would go to like the if you go to vs
23:12code too and be like hey we need you to
23:13go implement this very complex feature
23:15like I don't even know who you are like
23:17what are you talking about
23:19um and uh yeah there was there was
23:22definitely some politicking that
23:23happened to to get the vs code people to
23:26um to dedicate some resources to that on
23:28a short short time frame like we were
23:30moving really fast you know it was less
23:32than a year before from beginning to
23:34ship Public public launch
23:38um was there a certain
23:40um metric where you're like this is good
23:42enough like we need to actually put it
23:44in the public product
23:45yeah there was I mean we had a long we
23:48had a nice long window of Public Access
23:50before GA so where it was free and you
23:55um and we did a bunch of optimizing for
23:56different groups of people that would be
23:58you know okay well you know do we want
24:00more experienced people do we want more
24:05I want people from this area or this
24:09um that gave us a bunch of really good
24:10stats so we were able to learn that for
24:14uh speed is the only thing that matters
24:16um so uh yeah there's something crazy
24:20thing like every 10 milliseconds is one
24:21percent fewer completions that people
24:23would do that adds up and also gets this
24:26pretty fast uh we learned that because
24:31somewhere in our first few months of
24:34public release we noticed that Indian
24:36completions were really low like the for
24:39whatever reason they were just
24:40significantly lower than than Europe
24:42so Network latency to India yeah and it
24:45turns out because opening I only had one
24:47data center so it was all in Texas and
24:50if you can imagine you're like typing
24:52and so that goes from India through
24:54Europe over the water down to Texas and
24:57back and back and back and now if you've
24:58typed something that doesn't match the
25:00thing that you requested then that's
25:02useless and so you don't get a
25:04completion but by the way that I know
25:06it's obvious but that's happened on
25:07every single product I've ever worked on
25:08you know like when I was at Google like
25:11um I worked on a variety of mobile
25:12products same thing you know Paisley
25:13Times Obviously search in general 100
25:15millisecond difference like so they do
25:18makes a big shift in market share so
25:19yeah it's kind of that's how much speed
25:20matters yeah and so that once we figured
25:23that out then we were able to we knew
25:25maybe this is too far ahead but we knew
25:27that we had something awesome right like
25:29people that were close to Texas were
25:31like this is freaking great like they
25:33were like you know we had a slack
25:35Channel and people were posting in it
25:39yeah and the fun the most fun stuff was
25:42like these people that would pop up and
25:44be like I don't program but I just like
25:45learned how to write this 100 line of
25:47underlined script that like does this
25:48thing that I need it's like oh my God I
25:51definitely feel like I now speak
25:52different languages that I don't
25:53actually know the syntax of it's very
25:56exciting yeah it turns out these models
25:58are really great at uh mid-like finding
26:01so like once after we figured out that
26:03you know once we knew that so that once
26:06the UI mechanism that worked which was
26:09this like great text you had tab it
26:13once we knew and then also like a bunch
26:15of work to get the multi-line working
26:16because we knew that the problem with
26:18the autocomplete was that it's only one
26:20line you get this one line to like tell
26:22people what what you wanted but we could
26:24renew the model could do more than that
26:25it could do you know a few like the 10
26:28lines realistically now I can do way
26:30more but and so we needed something that
26:32would show that off and uh so yeah we
26:35tried this ghost text thing and we just
26:37the key Insight was realizing that you
26:39programmers kind of think in blocks so
26:42if you can just auto complete the rest
26:44of the current block
26:46then that's what people want people can
26:48they're kind of like judging what this
26:51block is going to do and if it does the
26:52right thing they know quickly
26:55so yeah so if you can complete an if
26:57statement and it does the right thing
26:58then it's easy to judge
27:00so we do when you're on something like
27:02that and then it was just this like that
27:05squeezing as much performance as we can
27:07get out like we basically never found
27:09the bottom of like you know we hit we
27:12made it as fast as we could and it was
27:16you know completions
27:18yeah so that was like and then that
27:19perpetuated like okay okay uh I know
27:23there's this plan for like Azure to run
27:26open AI in six months
27:29we need you to do that in the next month
27:30so like can we let's figure out how to
27:33because we wanted to run
27:35a bunch of gpus in Europe so so we could
27:39um hit Asia you know there was no there
27:42was no other place that we could we
27:43could run them in Europe we could run
27:45them on West Coast we could run them in
27:47um at the time so yeah so that was and
27:49that and Microsoft stepped up there we
27:53um yeah and then pretty much after that
27:55we launched and yeah were you surprised
27:59by the uptake press launch
28:02no yeah no I mean our retention rate was
28:0550 like it never went like months later
28:09it was still above 50 percent
28:11by like weekly cohort which is like
28:16um and we didn't we didn't know if
28:18people would pay for it that was one
28:21um I lobbied pretty hard for
28:23uh for going cheap and capturing the
28:26market how did you guys think about
28:28inference costs for this thing at the
28:30beginning oh yeah we were our estimates
28:33were wildly off wildly off yeah so we
28:36got estimates so we're like
28:38you know it'll be 30 bucks a month for
28:41Ev for your on average right
28:44and then um we were able to like once we
28:48once we were able to Once Microsoft was
28:51able to like do some kind of do their
28:52Azure infrastructure rather than like
28:54Fork off little bits so we could do more
28:56accurate uh projections
29:00and uh yeah there's a bunch of like
29:02there's a bunch of like moments where
29:03like how much is it gonna cost you'd
29:05like wait for these results
29:07and uh the first big one was 10 bucks a
29:10month it'll cost 10 bucks a month
29:12I was oh my God it's so much cheaper
29:16um and then and then we could optimize
29:18it and that was like we hadn't even
29:19optimized on price yet right
29:21and then we optimize them price a bunch
29:23and yeah now let's listen that so like
29:27it was very fortuitous right like we
29:29were thinking like okay well maybe it's
29:30maybe it's Enterprise only because
29:32that's the only people who are going to
29:33be willing to pay for this like
29:35um you know 30 bucks a month is not
29:37that's a it's a lot and that's like with
29:40no margins right but yeah so 40 of your
29:43code it's not a lot yeah that's the
29:45thing we know that we know that now and
29:46not only that it's like a crazy thing is
29:51um there's some whales out there like
29:52there's people who it rides 80 of the
29:56um which is insane uh where do you
29:58extrapolate all this like three to five
30:02are there basically going to be just
30:03like agents writing code for us in
30:05certain ways is it 95 of code is written
30:11copilots and you know humans are kind of
30:13directing it like what do you kind of
30:15view the the world evolving into in the
30:17next couple of years and next couple I
30:18mean like three to five not twenty
30:20yeah it's hard I don't know
30:23I think it's hard for me to you know
30:26it's hard for me to imagine
30:29what that world looks like because it's
30:31such a shift from like I have my hands
30:33on something and I know that it's right
30:36um where we are now which is I mostly
30:38know what's right or I have a sense that
30:39it's right but I have to I have to test
30:43um to know that it's right to then
30:46just write this and I'll I'll trust that
30:48it's going to work like those are pretty
30:49crazy transitions right
30:52um they exist like sure they could also
30:55Imagine like certain ways to do like a
30:56code review post some chunk or some some
30:58other sort of quality check to yeah I
31:01think every if that's if that's the goal
31:03we want to get to is the people like
31:06uh I think every barrier to that is
31:10um so we can code review only those the
31:12dangerous parts or only the scary are
31:14the confusing parts or
31:17uh we could do things like train a model
31:20on functions before and after changes to
31:22say like okay this looks like you're
31:25looks like a more polished version of
31:27this function would be this or
31:30yeah we can do things like you know just
31:32start at the very basic you know very
31:34basic main Loop and then add everything
31:38um with tests so that it's you know what
31:40what works and what doesn't and then
31:41just have the uh that I keep generating
31:44what's the logical next feature like
31:47um all these things will get figured out
31:49so if that's what we want to do that's
31:51what's going to happen so what's the
31:52what's the idea behind in
31:55yeah I think I mentioned uh making Bots
31:58that do stuff for you it's a
32:04um I think that's where we see uh going
32:06you know the next few years are are
32:09in AI is taking action
32:13not just answering questions or
32:21um helping us in our daily lives
32:26organizing my schedule or booking
32:29flights or finding a trip for me to take
32:37telling me which contacts I haven't
32:39talked to in a long time I should reach
32:43there's a lot of stuff that we can do by
32:48ai's access to information and letting
32:52them act on that information in a
32:56um checks to make sure that we're
33:01and yeah I think that'll be a really fun
33:03feature I think you know
33:06almost like you can imagine copilot
33:08applied to Everyday activities right
33:10like copilot gives you a little bit of
33:12help so I want minion to give you a
33:14little bit of help outside of your code
33:17how do you decide to work on minions
33:21minions specifically so basically this I
33:24stopped working at Magic I quit because
33:26I I couldn't figure out how to hook up
33:28the AI to data I was like if in order to
33:31in order to improve the quality I need a
33:32PhD in math like I don't know what to do
33:34now and it just sort of the models got
33:39better enough for that specific problem
33:41seems solvable yeah so the the tech
33:43operator and so that specific problem is
33:46where interacting with the real world
33:48broke down in the past
33:50you know like I said floods get delayed
33:54um not just like a little bit all the
34:00you know they might not have the seat
34:01you on at the at the concert that you
34:05um and so you know AIS are this kind of
34:08compression of everything that you in
34:12but they're not a real-time mechanism
34:16anyway so that was the idea I was like
34:18okay well uh I think we can you can work
34:21on this old problem of how to make a
34:24bot do stuff for people
34:28that's what that's what we want
34:30let's go make it I don't know it's like
34:33maybe I can use the excitement from
34:35copilot to launch into something which
34:39um and but which I believe the
34:41technology is around for
34:43um if we can figure it out
34:45so yeah that's it that's the only reason
34:47I've ever gotten to startups actually is
34:48like uh okay I want to do a startup so
34:51that I can do a harder startup yeah you
34:53know or I want to do a project so I can
34:56um this is one sufficiently hard this
34:58one's hard yeah it's good and hard
35:00there are these fun things you kind of
35:02learn along the way that keep you that
35:03keep you like engaged right
35:05um like if the thing with code with
35:07copilot is like it turns out code is
35:09pretty special right like you can run it
35:11so if an AI generates some code and it
35:14runs and you know something about that
35:16code that you wouldn't know necessarily
35:19um and and accomplish like doing
35:22agent-like work on the internet uh or on
35:25apps is uh has a lot of these similar
35:27kind of properties where
35:29um it's like oh yeah you can you can
35:31learn something here we can learn what
35:33we can learn from what people do or we
35:36um we can see if it's a success or a
35:38failure and then learn from that or
35:40um we can see what we can optimize what
35:42works based on what doesn't work or what
35:44uh you know figure out what's annoying
35:47and and try to improve it so I think
35:49these things all compound in some really
35:51interesting way the goals right is is
35:54you know straight out of sci-fi right
35:55you want to make a thing where
35:57you say hey computer file my taxes and
36:00it does the right thing you know
36:02I think we can get there
36:04it's certainly in the next few years
36:09but it's also fun to like think about
36:11how to break these things down and um it
36:14turns out breaking down tasks in the
36:17same way that humans do take a complex
36:19task you figure out you know you write a
36:21list of things to go through it same
36:22kind of thing works for AIS as well so
36:24you break them you break down complex
36:26tasks figure out if there's any
36:28information you need maybe you have to
36:30write some code maybe you have to ask
36:31some questions maybe you have to query
36:35um same thing a human do
36:37humans don't usually write code to do
36:38their taxes but sometimes it's kind of
36:41the same thing you go through a list of
36:42your uh a list of your pay stubs
36:46um you know that's executing a for Loop
36:51I like I like data sets that don't exist
36:54so uh people clicking on stuff on the
36:59gigantic data set which is currently
37:02uh and so I think that's pretty exciting
37:04I think there's a lot of stuff to learn
37:07so you're hiring and one thing we've
37:09talked about is like it's kind of a
37:11funny thing to try to hire for building
37:14products with this new set of
37:16Technologies and not like working in
37:19machine learning for the last decade may
37:21not help you that much
37:24um how do you how do you think about the
37:27yeah it's been strange it's been it's
37:30been uh it's been odd actually you know
37:33what one thing I don't know if other
37:34people have noticed this but it seems
37:37very senior people get it
37:40uh and very young people get it and so I
37:44don't know that the middle has really
37:45caught up yet or realizes that that um
37:49are we on the other side are we in the
37:52no I just made the middle in terms of
37:54like yeah I just mean the middle in
37:56terms of like uh it's like I'm very
37:59young it's like super no it's not an age
38:01thing it's not interesting it's more
38:02like there's like there's like um
38:04it's kind of like a experiential Midway
38:06curve you know the Midwood IQ sure sure
38:09it's kind of like that for experience a
38:11little bit yeah it's like it's almost
38:14it's like you tell someone who's very
38:15naive and you tell someone who's very
38:19um I can make magic here's how I can do
38:22it and then the very naive person is
38:24like that's awesome and then the very
38:26experienced person is like ah maybe yeah
38:30um but the response from almost
38:32everybody else is that's not possible or
38:35I don't see it a number of companies I
38:37know are trying to develop that
38:38intuition internally now and often this
38:40is the first time in a really long time
38:42that I've seen certain Founders come
38:44back and start coding again you know
38:45they'll have a multi-hundred person
38:46company and they'll get so enamored by
38:48what's happening that they'll just dive
38:49in and it's almost uh a number of them
38:51have told me that they feel like the
38:53some of their team members just don't
38:57for what this can actually do and so
38:59they don't even know what to build or
39:01where to start or how interesting or
39:02important it is and so it's almost like
39:05this founder mindset is needed to your
39:07this is a really interesting new set of
39:09capabilities and how to actually learn
39:11what these are and then how do I apply
39:12them and I think people lack natural
39:14intuition for what what this does right
39:16now especially not intuitive yeah I mean
39:19I I would say I have some intuition now
39:22and people that are again on that
39:23experience Spectrum has the magician
39:27um but it's not intuitive you know it's
39:31there's many kinds of different Learners
39:34and there's many kinds of different
39:37um it takes a certain kind of programmer
39:38to be comfortable with this idea of like
39:40I don't know what's going to work let me
39:45it turns out it's a similar similar
39:46attribute to the web or product product
39:49um you know being able to test stuff
39:51look at the right metrics find the right
39:55um yeah those are those are useful
39:57skills I think that are reusable
40:00um but the the intuition and the
40:02tenacity that it takes to like you know
40:04this doesn't work at all what do I do
40:07that's still rare I think in that field
40:10because there's so much uncertainty that
40:11it's it's easy to go like I tried 10
40:14things and it didn't work so like this
40:15is impossible like okay
40:17part of the natural reaction of somebody
40:19looking at something that has 10
40:21performance at the beginning is like why
40:24even bother we're not going to get there
40:28the crazy thing is these things work at
40:33not only that but they scale and they
40:35improve the scale right most things
40:38break at scale almost everything breaks
40:40at scale and that's what you know that's
40:42what you hire people to deal with you
40:44know every doubling you usually someone
40:46breaks you gotta fix it
40:51it's a pretty crazy thing to think of
40:54what kind of emergent properties might
40:58um you know if we can get these things
41:0010 times bigger so I'm always I'm always
41:02in favor of people pushing those limits
41:04you know like it doesn't make sense like
41:07even even the best people can't explain
41:09what's going to happen when you you know
41:11go bigger but like that's we built the
41:14Large Hadron Collider for that reason
41:15too you know no I think maybe we'll find
41:19um these Endeavors are valuable
41:22and uh yeah I'm grateful to be left or
41:25anything this time for sure
41:27it's a great note to end on thank you
41:29for joining us for the podcast yeah
