00:05we've talked to many practitioners who
00:07are pushing the state of the art this
00:08week on the podcast we're exploring the
00:10dominant ml developer tool weights and
00:12a lot and I are sitting down with CEO
00:14and co-founder Lucas bewald he has a
00:17knack for creating companies that
00:18support pain points and ml development
00:20his first company figurine addressed the
00:22problem of data collection for model
00:23training and his second company weights
00:26and biases has created an
00:27experimentation platform that supports
00:29AI practitioners at companies including
00:31Nvidia openai Microsoft and many more
00:34Lucas thanks for doing this welcome to
00:36no priors thank you great to be here
00:38Lucas you studied at Stanford where I
00:41assume you discovered your interest in
00:43machine learning and under one of our
00:45previous no priors guests Daphne Kohler
00:47can you talk about when you start
00:49working in Ai and learning from Daphne
00:51yeah totally as a kid I was obsessed
00:54with playing games and I got really into
00:56go and I was super into the idea of or
01:00thinking about how would computers win
01:01at these games and so I actually sent
01:04Daphne an email maybe as a freshman
01:06being like hey can I can I work with you
01:07like I'm really interested in games I
01:09want to learn how to like beat go and
01:11Daphne wrote me actually a pretty polite
01:12email being like that's not what I do go
01:15away a few years later I I took her
01:17course and I was actually I studied math
01:19at Stanford and I have to say Daphne
01:21cared about a thousand times more about
01:23teaching than even the best professor in
01:26the math department and so it was really
01:27just eye-opening like I just loved how
01:29much she actually cared about teaching
01:31and it got me really excited about the
01:33AI that was working there and I went on
01:36to be a research assistant for her and
01:39it's a funny thing at that time was like
01:41nothing really worked like it was just
01:43before kind of you know Google was
01:45thought to be really like page rank at
01:47the time was the thing that was making
01:48them work and I think later you know
01:50became clear that machine learning was a
01:51very a big part of that but really when
01:54I was doing ml it was like searching for
01:56applications that were working and
01:58Daphne was actually really obsessed at
02:00the time with a thing called Bayes Nets
02:02which you don't hear about too much
02:03anymore because I don't think they ever
02:05um you know worked for many applications
02:07I hope I'm not offending anyone but
02:08that's my my understanding I actually
02:10think you know the thing that I really
02:12took away from Daphne that that really
02:15um I mean she was one of the smartest
02:17people I've ever encountered and she had
02:19this incredible Clarity of thought and
02:23an intolerance for sloppy thinking that
02:26that's just like really served me well I
02:28think that's sort of separate from
02:29machine learning you'd see like other
02:31professors would come and give like
02:33guest talks and you know they would say
02:35something's kind of lazy and like we'd
02:37all just be sitting there just like
02:39waiting for Daft to like eviscerate them
02:41and I think her personality is mellowed
02:43a little bit over time but I kind of
02:45miss I just missed that sort of like
02:48aggressive clear thinking
02:50um and I I really admire it I don't
02:52think we got a taste of that but we did
02:54talk about whether or not probabilistic
02:56graphs are are coming back a little bit
02:58how did you how did you go from you know
03:00Stanford to founding figure eight
03:03yeah you know it's funny I actually
03:04really struggled doing research with
03:07with Daphne basically the things that I
03:10tried just barely barely worked like you
03:13know I published a couple papers that I
03:15feel kind of ashamed of where it was
03:16sort of like go from like 68 accuracy to
03:1970 accuracy in a task nobody cares about
03:22by throwing like a thousand X to compute
03:24and by the way like kind of guessing the
03:26most likely answer is probably like 64
03:30um you know it just it felt honestly
03:33kind of pointless and sad like I love
03:34the idea of like computers learning to
03:38do things but it's hard to sort of
03:40sustain the enthusiasm for that when
03:41everything you try just completely you
03:44know doesn't work and even the things
03:46that do work you kind of wonder if
03:47you're like p-value hacking like okay I
03:48tried a thousand things you know so I
03:51guess something's gonna be like a little
03:52bit more accurate than a baseline what
03:55tasks were you working on did you did
03:56you end up working on go or games or
03:58anything no Daphne was Daphne is not
04:00interested in games let me tell you and
04:02it's actually another I kind of admire
04:04that that perspective too as much as I
04:06love games I'm a go nerd so I'm curious
04:08oh you are oh me too I I yeah I I love
04:11go yeah Daphne was very not interested
04:13she really was practical and so I worked
04:17on a task that you really don't do now
04:19called um Word Sense disambiguation
04:21where you're trying to find out like
04:23okay I have the the word plant actually
04:25if you look in most corpuses because
04:27they're government generated often at
04:29the time plant typically will mean like
04:32the power plant sense of plant our
04:34cabinet often means the sort of
04:35president's cabinet sense of cabinet and
04:37so you're kind of trying to figure out
04:38like what is the meaning here of these
04:40words and then applied it to um to
04:42translation it's a cool task I mean and
04:45actually it turns out I think that these
04:47again nobody kill me but my general
04:49sense is that these sort of like
04:50linguistic oriented strategies really
04:53don't work that well it's kind of like
04:55by feeding more data in and sort of like
04:57working on outcomes you can figure these
04:59things out much better so
05:00um a little bit of a dead end and
05:01actually in you know I was so frustrated
05:04by that that I I just really wanted to
05:06work on something that people cared
05:09about I actually turned down an offer
05:11from Google because they didn't tell me
05:13what I would be working on to go to
05:15Yahoo because they they were like okay
05:17you can work on you know search rank
05:19ranking in different languages and but
05:21that actually turned out to be
05:22incredibly fun right because it was
05:24super applied it's actually a task that
05:26works really well and Yahoo is kind of
05:28in the infancy of switching from hand
05:29toon weights to machine learned way so
05:32they really had no one not many people
05:33actually like working on deploying this
05:35stuff so I was like writing code to
05:38translate machine learning algorithms
05:39into C code and then check it like we
05:41would check it into our little code base
05:43and run this kind of like semi-hand
05:45generated C code in in production so
05:48that was that was super fun but you know
05:50the thing I learned there actually which
05:51I think I'm not the only one that
05:52learned this but I just felt it I would
05:54go from like country to Country trying
05:56to switch from hand tune weights to an
05:58ml model and like I was sort of the
06:01messenger here so like sometimes it
06:02would work and sometimes it wouldn't and
06:03so like people are either really happy
06:04with me when it did work or they'd be
06:06really pissed at me when it when it
06:08didn't work but I kind of realized
06:10actually the model that I'm building is
06:12like the same for each country it's the
06:14the training data though is different so
06:16some countries would take the training
06:18data collection process really seriously
06:19and they'd get a great model and some
06:21would just like really half-ass it or
06:23like you know have these crazy like
06:24issues in the data collection and then
06:27the model wouldn't work and so I just
06:29really kind of viscerally felt how much
06:32the the training data process mattered
06:35and I kind of felt like you know why
06:37don't they let me get involved in the
06:39training data process like that would be
06:40a better use of my time than building
06:42these models and so I wanted to make a
06:45company where the people doing the ml
06:48could actually have control over the
06:50training data collection process and
06:53really get like visibility into it
06:54because you know at the time I think the
06:56thinking was like oh this is sort of
06:57like a manual task that's like my like
06:59an operations team should deal with this
07:01and and they would like they would do
07:03this thing where you'd like make this
07:05giant requirements document
07:07it's gonna sell like waterfall like it
07:08would be like yeah it wasn't iterative
07:10oh was iterative and it'd be like you'd
07:12make like a 50 page document and like
07:14you know that the people doing the
07:15labeling are not like reading that
07:16document but you kind of need that's
07:18like cover your ass if they did like
07:19labeled something you know not the way
07:21you want and it would have been so much
07:22better to be like look we're trying to
07:24make search results like put yourself in
07:26the mindset of like someone you know
07:28who's like looking at this like is it
07:30good or bad versus trying to lay out in
07:32like excruciating detail what makes
07:34something relevant or or not relevant I
07:36think also at this time like when when
07:38you first started um I think originally
07:40was called Dolores labs and then crowd
07:41flower and then eventually figure eight
07:43like I think I met you in your Dolores
07:44Labs days or something I know I remember
07:46yeah yeah yeah and at the time there
07:48weren't really um solutions for data
07:50labeling externally right some people
07:51are using mechanical twerk from Amazon
07:53to sort of run jobs on untrained workers
07:55there wasn't like scale there wasn't you
07:57know there was none of these Services
07:58yeah and so you got really early to this
08:01idea of starting like a data labeling
08:02company and that that was actually very
08:04useful for machine learning and so it'd
08:06be great to hear like you know what were
08:07the early days of of that like and what
08:10was it industry like and how did you get
08:11all that running yeah I mean it was
08:12funny right because back then I was
08:15coached actually quite a lot by You Know
08:17Travis kalanick who's you know famous
08:18now for for doing Uber and other things
08:20but he was like don't tell anyone that
08:23it's like AI like VCS like don't want to
08:25hear Hey I was actually good advice
08:27um at the time and it's good advice in
08:28the early days of the company and
08:30started interrupt I think one
08:31interesting side note on that just from
08:33a Silicon Valley history perspective is
08:35Travis used to have these effectively
08:37like hackathons or meetups at his house
08:39called the hack pad and you know I think
08:42you used to go to those you know a bunch
08:43of friends of mine used to and so a lot
08:45of startups actually had some impact or
08:48influence from Travis in those days like
08:50due to his fact of like you know being
08:52another founder in the scene and kind of
08:53getting everybody together and so it's
08:54kind of an interesting
08:56moment in timer and history and to your
08:57point back then like AI wasn't really as
08:59popular as it as it became later so it's
09:02kind of an interesting like side note
09:04well I mean Not only was a AI not
09:06popular but like startups weren't
09:08popular right like my family didn't you
09:10know understand about startups and I had
09:11graduated Stanford you think I'd have
09:13all these great like connections but it
09:15didn't feel like that like I had no one
09:17who knew how to like raise money from
09:18feces I didn't know any you know VCS or
09:21I didn't really know any like
09:22entrepreneurs honestly and we had this
09:25website for Dolores labs and early days
09:28just trying to get customers and it put
09:29my my personal phone number actually
09:31remember I was like the first user of
09:32twilio because I needed to make a phone
09:34tree and so I used twilio software and
09:37then like all three of the founders came
09:38to my house to like help me like make
09:40that phone she like work better which is
09:42kind of amazing it was like you know
09:44like you know one of those like you know
09:47um you know grunge apartments in the
09:49mission and then uh and then Travis
09:51called in but you know it's funny
09:52because the phone treat we were just
09:54trying to pretend like we were a big
09:56and Travis called in
09:58because in the phone numbers on the
09:59website not because he wants to buy
10:00anything but he just like thought it was
10:01like awesome and so I'm just like you
10:03know I pick up my phone and then there's
10:05just like this guy and then just be like
10:06oh man like this is so cool you know I'm
10:08like okay like who are you you know it's
10:10like it's like hey do you want to like a
10:12coffee and uh and that actually turned
10:14out to be incredibly like helpful but
10:18the thing that was so different back
10:21then is that the people doing ml there
10:24just weren't that many like there were
10:25people like heavily investing in ml but
10:27they're but it wasn't that many and so
10:29what happened was you know we got like
10:30eBay as a customer which has really
10:33mattered at the time and we got like you
10:35know Google as a customer and Bloomberg
10:38and then there just like wasn't
10:40anywhere else to go so like you know my
10:42board was always like recommending like
10:43read crossing the chasm and and we tried
10:46like a million different ways to like
10:47you know grow the company and you know I
10:50don't know I hope this doesn't sound
10:51defensive I mean maybe I'm just a bad
10:52CEO but we had like years of like
10:54struggle because there was no Chasm to
10:57cross right there was like nowhere else
10:58to go so we tried all these different
10:59things to like you know build more
11:01complete solutions for our customers and
11:04it just didn't work and then kind of all
11:08um you know autonomous vehicles got
11:10popular and that really actually
11:11suddenly caused our Revenue to um you
11:15know start to to grow really fast again
11:17but it was like an eight-year lull of
11:19like you know really no growth right so
11:21it's hard because we started a fast got
11:22everyone really excited you know kind of
11:24got like whomped for just like years and
11:26years and years actually we had all
11:27these competitors they all went away so
11:29at some point we had like no competitors
11:31left right because like everyone had uh
11:33had gone out of business and then it was
11:35a funny experience because like scale
11:36came along and totally ate our lunch on
11:38in the self-driving mark which is a
11:40market like I knew and loved and so you
11:42know I I was so excited to sell the
11:44company after you know so many years of
11:46struggle you know but then like right
11:48after that we see like scale just like
11:49skyrocketing and revenue it's like oh
11:51man like I wish we had just like you
11:53know maybe held on a little bit longer
11:55but then you know it gave me the space
11:56to start weights and biases so you know
11:58who knows I I want to be like Daphne
12:00color and evaluate my decisions like
12:02accurately and and critically but it
12:05also does seem like you know I've had
12:06some good luck along the way yeah I know
12:08the market shifted so dramatically and I
12:10think to your point self-driving was the
12:11first time that you suddenly had a bunch
12:13of systems at scale that people needed
12:14data labeling for and then of course now
12:16we have this llm way but it's all very
12:18very recent and I think a lot of people
12:20basically view ml as this sort of
12:22continuity and everything has always
12:23been kind of rising in a sort of almost
12:25linear way and in reality it's this very
12:28bumpy set of discontinuities in terms of
12:30the set of Technologies and markets that
12:31people are adopting it in and so it's
12:33not continuous it's a discontinuous
12:34thing and nobody thinks about it that
12:36way when you started weights and biases
12:38you said something along the lines of
12:40you can't paint well with the crappy
12:41paintbrush you can't write code well in
12:44a crappy IDE and you can't build and
12:46deploy great learning models with the
12:47tools we have now I can't think of a
12:49more any important more important goal
12:53and that's I think like when you
12:55announced that you were starting with
12:55some biases and so I was just curious
12:57like what lapses and capability really
12:59got you going on um 1B and can you also
13:03just you know many of our listeners um
13:05know what it does but for those who
13:06don't could you explain what the product
13:07does and how it works sure yeah so it's
13:09kind of constantly evolving right
13:10because we're saying it's like a set of
13:12tools for for people doing machine
13:13learning we're best known for our first
13:15thing that does experiment tracking
13:17which keeps track of like how your
13:19models like perform over time as they
13:21learn and train oh we also have a lot of
13:23stuff around like kind of data
13:24versioning data lineage you know
13:26production monitoring model registry
13:28kind of the the sort of end-to-end stuff
13:29that you need to do machine learning
13:33and I think the thing that happened to
13:37I had been running cauliflower for years
13:38and I always loved machine learning but
13:41I was like really starting to get out of
13:43date like deep learning came along and
13:44at first I was kind of skeptical of it
13:45because people are always saying I have
13:47a better model that's like magically
13:48better and they're like wrong wrong
13:50wrong wrong wrong it's just like really
13:51like data and then but then they were
13:53right right so there actually was a sort
13:55of a better modeling approach that
13:56worked and I kind of realized you know
13:58when I was in my early 20s I was really
14:01judgmental of you know the people in
14:03their late 30s that hadn't like adapted
14:04to machine learning at the time because
14:07like rule-based systems were kind of all
14:08the rage when a different generation was
14:10was growing up and I was like wow you
14:12know I am actually getting out of date
14:14myself like I'm saying these kind of
14:15wrong things that were true 10 years ago
14:17and are not true now and I honestly felt
14:19like really bad about myself and so I
14:22did a couple projects to try to you know
14:24get up to speed I started teaching free
14:26machine learning classes and and deep
14:28learning classes to kind of force myself
14:30to to learn the material and actually
14:32like interned briefly at um openai where
14:35I was just like look I will just do
14:37whatever you know work you want just I
14:39want to be like I need I know that I
14:41need like an accountability partner
14:42essentially to force me to learn stuff
14:44even though I love to learn stuff like
14:46my favorite thing but I always need
14:47accountability practice for anything I
14:48do so I sort of use the students as an
14:50accountability partner and open Ai and
14:53then what was happening was I was
14:54showing my old co-founder Chris like all
14:56the the cool stuff and he's like a
14:59really good engineer and I'm like
15:00actually really like bad engineer like
15:01I'm like really lazy and I try to write
15:03the like you know I'm just like like
15:05people my co-founders make fun of me all
15:07the time for like you don't really know
15:08how get works and I just openly I have
15:09no idea how git works I just sort of
15:11mash the keyboard until like I kind of
15:14like you know get an event date and then
15:16I like call Chris and beg him to like
15:23I don't understand it and it's like my
15:26co-founders just find it like baffling
15:27that I wouldn't understand it but I
15:29think it's like um for them
15:32you know it's like they're like wow this
15:34guy like needs some basic tools you know
15:36like because you know they're like okay
15:37like reproducibility like why don't you
15:39just use Docker I think that's sort of
15:40the Ops mindset but I'm like man I don't
15:43understand Docker guys I feel like I
15:45installed on my like laptop and then
15:46it's always like taking up memory and
15:48stuff I like I don't like don't really
15:50know what it's doing and I'm like kind
15:52of scared of it and like I don't know so
15:53it's like I just feel like it's adding
15:54weird complexity to understand and so I
15:56think the tools kind of exist in a way
15:58but they just weren't made in a way that
16:01like ml people could really use them
16:03because like you know if you're like me
16:04you kind of come from a mathy background
16:05or like a research background you kind
16:08of didn't really learn to do like
16:11industrial style coding and so you know
16:14I think companies have this idea that
16:16like the researchers are just gonna like
16:17throw the thing over the fence and then
16:18it's going to be in production but it
16:20doesn't really work actually like I
16:23pattern that people sort of like imagine
16:26they're gonna do and they don't ever
16:27really do that you end up like with
16:28research is always the research code
16:30bleeds into production and every company
16:32and so I think a better way is to give
16:35you know researchers and like ml people
16:38tools to just make their stuff more
16:41reliable and it has to be simpler maybe
16:44or it's just a slightly different
16:45audience like you can't just give
16:47someone like Docker you can't just like
16:49you could I mean a lot of people like
16:50hey why don't you use like the get large
16:52file system stuff to to version your
16:55and like there actually are some reasons
16:57like it doesn't work well with like
16:58object stores so there's some like
16:59ergonomics reasons but it's also just
17:02git is like complicated I'm like willing
17:05to use it for code but if you start
17:07making me like version my data with Git
17:09like I just want to like cry you know
17:11what I mean so like give me something
17:12like simple you know what I mean where I
17:14don't have to like think about it or I'm
17:15just gonna start like renaming my data
17:17sets like latest latest really latest
17:19session relief for sure June 27th so I I
17:23just need my stuff to be simple that's
17:25kind of the mindset you know behind the
17:26companies like let's like make these
17:28like kind of simple clear things that
17:30actually help people we were talking
17:32about how much you wanted to like you
17:35were thinking through how much LMS were
17:37gonna change like experimentation and ml
17:40tooling when we last saw each other in
17:42person not at the zoo but before that
17:44yeah and you you guys launched this
17:45prompt Suite in April like can you talk
17:48us through the sort of you know thought
17:49process of hey like you know I I really
17:52admire this as a leader and as a
17:54technical person you're like trying to
17:55stay really plastic about what is
17:57actually changing in machine learning
17:59how do you think through this change
18:00well it's really hard right I mean so
18:03what happened was we have a great
18:04business that you know makes like an ml
18:07uh a set of ml tools for training models
18:09and we actually helped most of the llms
18:12out there were built using weights and
18:14biases and then we started to see like
18:17wait a second some of these ml tasks you
18:19could just ask the llm right so instead
18:22of doing like a sentiment analysis model
18:24you could just be like hey like is this
18:26document positive or negative sentiment
18:27like for structuring documents you can
18:29just be like hey find all the names like
18:31in this document and it actually works
18:33super well and a little piece of me is a
18:36little bit sad about that because we
18:37have this like great simple relaxing
18:40business that grows Revenue every every
18:42month that I always dreamed of right so
18:44you know part of me is like [Â __Â ] this is
18:46actually our kind of first real
18:49existential threat I think you know and
18:51and um and you know I went to my like
18:53leadership team and I went to my board
18:54and I was like I think there's like a
18:56real existential threat here and I think
18:58they were like hey you know we don't
18:59like see it in the data like are you
19:01sure like maybe you're being paranoid
19:02and I guess I do feel sure and I don't
19:05want to say I'm like the only one or
19:06like pay myself as the hero like you
19:08know my co-founder is also seeing this
19:09and you know people talking about it but
19:11it's sort of like you know this threat
19:13is like now right and we have to
19:15actually like get the whole company to
19:17to do this thing because it doesn't show
19:18up in any of our like metrics yet but I
19:22just really believe that you know our
19:24customers are rational and they're gonna
19:26do a thing that like makes sense for
19:28them and so I see a lot of my colleagues
19:30being like Oh there's going to be like
19:31lots of different models it's like nice
19:34if it were true but like what I see
19:35everyone doing right now on July 27th is
19:38using GPT like I didn't see like 95 of
19:42the people out there you know using GPD
19:44for these ml tests and so it's like look
19:46we gotta support that and so
19:48we've really rallied the whole company
19:49behind it and uh we pushed out prompts
19:53we'd also this is really my my
19:55co-founders my co-founder Sean had
19:58really put a lot of effort into making
20:00our stuff really flexible because he's
20:01like you know what Lucas like there's
20:03gonna be like changes you know coming we
20:06don't know exactly what they are but
20:07like you know kind of from the beginning
20:09we really tried to build very flexible
20:12infrastructure so this was kind of a
20:13moment we could really sort of like Flex
20:15that and get out of um you know a
20:18product for for monitoring stuff and you
20:20know now it's like you know kind of it's
20:22our number one priority is getting out
20:23more tools for this new this new
20:25workflow out of curiosity because you
20:28know there's a lot of debate right now
20:29in terms of proprietary models versus
20:30open source models and um I think
20:33there's a really great quote I think
20:34it's from Harrison from link chain which
20:36is you know no GPU until product Market
20:39fit right you should first like figure
20:41out if the thing works at all or if
20:42there's a customer need and that means
20:44using GPT and then once you prove it out
20:46you know you may use gpt4 or something
20:48for very Advanced use cases and then you
20:50kind of fall back to 3.5 or you start
20:52training your own model for things where
20:54you just want cheap sort of high
20:55throughput things happening and it
20:57increasingly feels to me like people the
20:59most sophisticated people who are at the
21:01farthest sort of Cutting Edge on this
21:02stuff are kind of doing both right they
21:05they use GPT to prototype and then in
21:07some cases they're they're training
21:08their own incidence of llama 2 or
21:10whatever they're using do you think
21:12that's where the world is heading or do
21:13you really think things kind of collapse
21:15onto some of these proprietary models
21:16like over time like it's six months from
21:18now it's a year from now it's two years
21:19from now I'm just sort of curious about
21:21how you think about adoption of Open
21:22Source it was funny I feel like lately
21:24what I've been telling people is like
21:26I'm just trying to see the world clearly
21:28as it is today I can't predict the
21:30future and I can barely keep track of
21:32you know what people are doing today
21:33when I consider it like my my full-time
21:36job so I'm like scared to
21:39prognosticate like what you know might
21:41be coming but I think you're right that
21:43that's what's happening now I think like
21:46there are like a bunch of things that
21:48could change right like I think like you
21:49know GPT is way far out ahead and it's
21:52hard to fine tune it not even possible
21:54with with gpt4 and I think that that is
21:57like a little that's not like a
21:59technical limitation I guess sort of
22:00like a business model
22:02um you know limitation so that might
22:04change I think that there's a lot of
22:06hidden costs to running your own model I
22:09think people are really enamored with
22:10the idea of running their own model and
22:13I've kind of seen this before where I
22:16think at the end people do rational
22:17things but it kind of takes them a while
22:19so I'd rather sort of support what looks
22:22like the rational workflow I mean I
22:24think the insane thing must be crazier
22:26to be an investor in this world is like
22:28very very few people have llms in
22:32production like there's probably more
22:34companies that have raised money as like
22:36llm tools than companies that have LMS
22:39in production which is like
22:41insane it's just like an insanely
22:43saturated tools Market with very few
22:46people getting things out but it's
22:49because when you Lucas when you say when
22:52you say Ellen's in production you mean
22:54my own that I have fine-tuned about it I
22:58serve myself no sorry I mean like G like
23:00GPT like using GPT in production oh
23:03really okay look I mean you you may be
23:06quite closer to this to me but it's a
23:07small handful yeah I'm like desperately
23:09trying to find them because like these
23:11are our customers like we you know our
23:13stuff is just like our ethos is like we
23:14want to help people do things in
23:16production so it's like if you're not in
23:17production we're not relevant to you so
23:19I like I mean back in January February
23:21this year we were looking for design
23:23partners that had stuff in production
23:25and boy was it hard to find right like
23:28you know now there are more but even
23:31when you you know you find people that
23:33are sort of like claiming to have this
23:34things in production it's sort of like
23:35well it's like you know it's coming like
23:37you know we have like all these like
23:38sort of like prototypes you know running
23:40and so I think it'll change I think it's
23:42changing quickly but I think it's a it's
23:44a funny moment where I mean I think if
23:46you actually looked at the Tam today of
23:48like tooling for like oh I was like I
23:50don't know I bet you it's um it's small
23:53and I think also I think VCS Maybe
23:55sometimes have this this funny window
23:57where you see like all the companies
23:59that are using LMS but the Enterprise
24:01adoption has been slower I mean despite
24:03the fact they talk about it like
24:04constantly like constantly like
24:06everyone's talking about it but in
24:10boy I don't know if I've like used the
24:12product of like any Enterprise that
24:15actually like was backed by a um an LM
24:17and there's a bunch of things that make
24:19it hard it's like you know it's kind of
24:21unfair because this stuff has only been
24:22out for like six months or so but it is
24:25like I think the adoption maybe maybe
24:27take a little longer the short term that
24:28people think I think that's a really key
24:30point because ultimately you know Chachi
24:32PT came out eight months ago and that
24:34was kind of the starting gun for all the
24:35stuff in my opinion and then gpt4 came
24:38out in March or something right which is
24:39three four months ago and if you look at
24:41Enterprise planning cycles for large
24:43Enterprises it takes them six months to
24:44plan something right and so people often
24:47ping me and ask about adoption of these
24:49sorts of things and it's like well
24:49notion is seeing you know has adopted it
24:52in addition ways already a zapier is
24:54adopted in interesting ways but it's
24:55basically these technical founder-led
24:58companies that jumped on it really early
25:00relative to everybody else and the big
25:02Enterprises are going to take another
25:03year or two because it's they're just in
25:05their planning cycle still around this
25:06stuff they just started really thinking
25:07about it and how to incorporate it and
25:09what to use it for and then they're
25:11gonna have to prototype and experiment
25:12for a while and then they'll push it
25:13into production and so that's why I was
25:15kind of asking a little bit about the
25:16future I just feel like it's so early
25:18yeah and we all talk about it again as
25:21if it's this continuous industry cycle
25:22but it's really not it's a disruptive
25:24new technology and so you know I think a
25:26lot of it's still to come in really
25:28interesting ways oh totally and there's
25:29tons of product issues too right like
25:31you know like notion and zapier both
25:32have these really compelling demos and
25:35they're both products that I use but
25:36then I actually don't use the LM like
25:39piece of them myself and I wonder I have
25:41no Insider knowledge of the level of
25:43adoption but I think they're I think
25:45they haven't gotten it like perfectly
25:46right yet despite like a lot of thinking
25:49and really smart people working on it
25:51sure for the core 1B product you know
25:53you folks are being used for a wide
25:55variety of areas around autonomous
25:56vehicles Financial Services scientific
25:58research media and entertainment
26:00is there any industry in particular that
26:02you think you're either surprised by
26:03adoption of the product or you're really
26:06excited to see sort of how people are
26:08yeah I mean the one that stands out for
26:10me because this is the one that's really
26:11different than you know my figure eight
26:13days is Pharma so I actually think this
26:17is kind of flying under the radar a
26:19little bit but every Pharma company
26:22is making major investments in in ml and
26:25not just on this sort of like I mean
26:27they do have these operations to sort of
26:28like sell more you know drugs to to
26:30doctors that uses sort of like light ml
26:33but I think the thing that's really
26:34exciting is like the actual testing of
26:38drugs you know before they they have to
26:40test them the physical world and that's
26:41like obviously working you know super
26:43well and I think I I see this before too
26:46it's like autonomous vehicles and stuff
26:47it's like there's a big lag there right
26:49before you get something through like
26:51all the clinical trials
26:52so no drug developed by ml has gone
26:55through clinical trials but if you look
26:56at the behavior of all of the big Pharma
27:00I can tell that it's working because
27:03they're hiring hundreds of people right
27:05like you know like companies will hire
27:06like a few people for like an experiment
27:09but they're all gearing up to like
27:11operationalize this stuff and that just
27:13gets me really excited I mean they could
27:14all be wrong I suppose and I don't
27:16really have any Insider knowledge except
27:17for the seats that get bought on you
27:19know wasted biases but when I see that I
27:21get pumped because I I just like you
27:23know the drugs that they're working on
27:24you know the diseases that they're
27:26curing it's like the ones that like you
27:28know like our relatives have right like
27:30you know Alzheimer's and Parkinson's and
27:32these kind of horrible things and I
27:34think there's just a huge promise in
27:35being able to do physics like inside a
27:38computer versus in the world yeah I
27:40think there's a I think that this is a
27:42really important Point too it's actually
27:44commonly said like no no machine
27:46learning developed drug has actually
27:48come to market today but it's a
27:51backwards looking metric in a very slow
27:53industry right like the clinical trial
27:55cycle is very long and and so um I'm
27:58actually like quite quite optimistic on
28:01this yeah and I don't think that's a
28:02that stands out in Pharma because it's
28:04very under discussed but there's certain
28:06Venture funds that have done incredibly
28:07well financially and Pharma where
28:09there's one in particular I can think of
28:11that never ship the drug
28:12until the covered era and they were in
28:14business for 20 years wow and they made
28:16all this money and they found out all
28:17these companies and none of their
28:19biotechs ever launched anything in the
28:21market wow so I think that's a that's a
28:24broader sort of issue with Pharma and we
28:25can talk about that I think some other
28:26time but it's kind of interesting how
28:28how little biotech has actually
28:30delivered and there's been amazing
28:32deliveries right in terms of different
28:33drugs and things but it's actually more
28:35common than just the ml side I think
28:37yeah Lucas you okay so Pharma is
28:41something you're excited about and you
28:43think has promise and and growth in um
28:46at least seats of 1B figure eight like
28:48you talked about you know Yahoo eBay
28:50like it's a very small set of people who
28:53else do you see in the weights and
28:55biases like customer base now like how
28:57has that changed since it's it's
28:58actually incredible to me that you've
29:00been you know working on this from the
29:01entrepreneurial side since 2007 because
29:04it's like you know pre pre-even deep
29:07learning Revolution right and so I
29:09imagine you know you've got a much
29:11broader user set now oh yeah it's so
29:14cool I mean the coolest thing about
29:15running weights and biases is the
29:17customer set is everyone I I really
29:19think every Fortune 500 company is doing
29:23something with ML that they like
29:24actually really care about and it's
29:27always surprising right like we work
29:28with you know most of the big game
29:30companies like I'm not a big gamer so
29:31like I you know like I'm vaguely aware
29:33of like Riot games and like unity and
29:34stuff but you know but they do all this
29:37cool stuff with ml to like you know make
29:39the games more fun to make like you know
29:41models in the games and this is like big
29:43Investments they really really care
29:45about because you know again like we're
29:46sort of the last step in your journey is
29:48to want good tooling for your ml team
29:50you kind of need something to work so
29:51you hire an ml team you get into
29:53production then you like run to problems
29:54then you come to waste devices so like
29:56we see stuff you know after it works and
29:58like you know like AG Tech like you know
30:01big agricultural companies I like never
30:02heard of some of them when they showed
30:03up and then they're like these huge you
30:05know businesses that are actually using
30:06ml to find ways to do like cleaner
30:08farming like a lot of the reasons you
30:10know you you spray a whole field with
30:12with pesticides it's just cause it's
30:13like so expensive to do something
30:15smarter and so you know I think I think
30:18that like crop yields and the you know
30:20the the cleanness of the the farming
30:22practice is about to like dramatically
30:25um improve like you know we worked with
30:26John Deere for years back from a figure
30:27eight days to you know weights and
30:29biases and they're they've deployed
30:30sprayers that only target the weeds in
30:33in fields it's deployed it's like you
30:35know I remember like for years seeing
30:36pictures on the wall and then showing me
30:38like prototypes and then one day they're
30:40like yeah you can like buy this you know
30:41and it's it's cool because like this
30:45it's like software right so it's like
30:47it's not like a machine you just like
30:48press copy and then you have you know
30:50more of it and so so yeah we see that we
30:52see like a lot of um
30:54you know I mean fintech probably obvious
30:55to you guys but like they're kind of I
30:57think always on the Forefront you know
30:59the stuff for Lots I mean like there's
31:01like consumer oriented stuff that you'd
31:03recognize like you know making chatbots
31:05not annoying right and then there's like
31:07you know kind of more you know Financial
31:09forecasting and and things like that but
31:11yeah I mean it's funny we don't do any
31:13vertical based marketing because there's
31:16not one vertical that's like dominant
31:18enough to to Warrant it and our
31:20customers bounce around between
31:22verticals so much that I think the
31:24Common Thread here is people doing like
31:26ML and data science versus any
31:28particular application which I just do
31:30is super cool that means it's sort of
31:31like table Stakes you know for everyone
31:33you you know made jokes I think jokes
31:36about like not being a terribly good
31:38engineer and now the weights and biases
31:41messaging is very much about developer
31:43first right can you talk a little bit
31:45about how you think about like you know
31:48and it actually it is like yeah as far
31:51as I understand it's like one of the
31:52most broadly adopted tools tools by
31:54developers work at nml how do you think
31:56about like developer adoption versus
31:58like researcher adoption and what did
32:02yeah I mean it's like developers and
32:04researchers they kind of blend together
32:06but I think that I think that what
32:09happened in the sort of ml app space
32:11is that you got a lot of well the early
32:14companies had to sell to Executives
32:16which I totally understand like that's
32:17what crowdflower had to do and the the
32:20problem there is you kind of get stuck
32:22in these like multi-million dollar deals
32:23and like you just can't get out of that
32:25like you can't switch to like a plg
32:27motion and so the early companies I
32:29think are kind of stuck right with like
32:30these products that like cios love and
32:32the you know Engineers hate and that's
32:34just like I just didn't want to do that
32:36with weights and biases no matter how
32:37big the market is or how like juicy that
32:40is and the good news is it's like not a
32:42good market like a developer oriented
32:44sales better when you when you look at
32:46like developers versus ml researchers
32:49that line is really blurred in the time
32:51that we've been doing it and and I think
32:54there's sort of like subtle differences
32:56but you know when Nvidia came along and
32:58these chips worked for deep learning it
33:01just like broke the entire stack like it
33:02was like a first time that in in my
33:05career where I'm like running into like
33:07like Linker errors and what the [Â __Â ] is
33:09a Linker like I vaguely like remember
33:11this you know from you know like a CS
33:14class I took you know like and um and so
33:16it's like I think that ml research has
33:19really had to be kind of become software
33:21developers and then at the same time you
33:23know the AI class is the most popular
33:25class so like all these software
33:26developers the Smart Ones kind of become
33:27ml researchers so I think that line has
33:30weirdly blurred but then I I think
33:32there's a funny thing that also has been
33:35like every devops person on the planet
33:37rebranded themselves as like an ml apps
33:39person all of a sudden
33:41and so you get all these companies that
33:43come out of like every ml Ops Team then
33:46realizes they could raise like a
33:47shitload of funding you know and so like
33:49you got like every every major company
33:51their ml apps team like went off and
33:52like raised money to like make a new
33:54product in the market which I think from
33:56an investor that's logical right it's
33:58probably they have a good thing but
33:59they're just like not good at connecting
34:02with actual developers right because
34:05they're actually like devops is is like
34:06a little bit of a different discipline
34:08where you're sort of obsessed with
34:09reliability kubernetes seems like simple
34:12to you and that's just not like the
34:15experience of like an ordinary yeah
34:17developer like you know like like my
34:18co-founders or or me and so I think the
34:21joy of weights and biases is we're kind
34:23of making software for like ourselves
34:25and I think it turned out that like
34:27maybe in the median of my three
34:28co-founders was actually the the target
34:31audience for us here I think I skew more
34:34towards you know an ml researcher barely
34:36but you know if I had to like pick one
34:38end of that spectrum and you know my
34:40co-founder Chris probably excuse more
34:41thread software developer and Sean's
34:43probably somewhere in between one of the
34:45things that's common to people or to
34:47developers is that they love to write
34:48their own tools and they tend to really
34:51enjoy using open source over closed
34:53Source Solutions how did you think about
34:55the open versus closed Source approach
34:56and how did you think about you know
34:58making something that's valuable enough
35:00and good enough to overcome that natural
35:01inclination to just do it yourself
35:03what's funny like I think the tools
35:05thing I've always felt like I've always
35:08felt like kind of proud of making tools
35:09for developers like that's always felt
35:11like really good because I think
35:12developers sort of know what quality is
35:15like I mean it's like I kind of like
35:16making a tool for someone that could
35:18make the tool themselves because it kind
35:20of raises the bar and this definitely my
35:22grandfather was like a pattern maker
35:24which is like a sort of you know like
35:26the person makes a pattern for other
35:27machinists and he had the same attitude
35:28of like look I'm making this stuff for
35:30like other engineers and like there's
35:33like an honor in that so I definitely
35:35feel that pressure and love it the open
35:38source was closer thing
35:40was really just like we didn't know how
35:44to make an open source business so so
35:47like we kind of started off close to us
35:49because we just we actually wanted to
35:52business and it's had like a pro there's
35:58been a major probe which is that all our
36:00competitors are open source
36:02and what that means is that they don't
36:04get to see how users actually use their
36:07software and so I think our software is
36:09a lot more ergonomic because we have
36:11like metrics on what people actually
36:12click on if people aren't click on a
36:14button we remove it if people like you
36:16know pick an option all the time then we
36:18know to like make that the standard
36:19option as we've grown and you kind of
36:21can't just like rely on anecdotal user
36:23feedback that I think has made our
36:25product like a lot better like people
36:27find it like nicer to use at the same
36:31time I understand why people want to go
36:32to open source stuff but
36:34honestly I feel like it's a little bit
36:36of a devops mindset also like I mean
36:38devops people like they're obsessed with
36:40like you know open source and usually
36:42like the ml Ops people we talk to in
36:43companies really want like an open
36:45source piece which is why our client is
36:47open source everything actually runs in
36:48your servers is open source but like I
36:50don't know like ml researchers aren't so
36:52precious in my experience generally they
36:55kind of want to get a job done and I
36:57think they're kind of happy to like
36:59that we have like a stable like business
37:02that generates money in like a normal
37:03way and isn't going anywhere
37:06or at least that's what I tell myself I
37:08think this is like the part about like
37:10the need for like ongoing Telemetry and
37:13application feedback like there are a
37:16you know zero to Marginal number of Open
37:19Source applications that have actually
37:21succeeded I think part of it is like
37:23this from you know hierarchy of honor of
37:26like the deeper in the stack you go like
37:28do people really want to work on like
37:30web UI in the open source or just like
37:33random business Logic on a relational
37:35database like yeah it's not as sexy and
37:37exciting to like go put your like GitHub
37:40badge on but I think the piece that you
37:42describe is actually really important
37:43where you know you work on complex
37:45workflows and if it's something that
37:48like somebody can just run in
37:49infrastructure and it's like you know
37:51you you get data back on like config
37:53files or yaml or whatever like that
37:55might that might work in terms of like
37:58one person's architectural point of view
38:00or some framework but I really don't
38:02think it works at the application layer
38:03for for these two reasons right like one
38:06total lack of feedback and to sort of
38:08the lack of interest in the I don't know
38:09technical brownie points you get for it
38:11yeah do you still pay attention I'm sure
38:13you do actually to like annotation like
38:15what do you what do you think happens to
38:17the data data annotation space and like
38:19you know the land of LMS and ra CHF and
38:24you know I'll be like honest actually I
38:26guess I'll be like totally honest I find
38:27it like incredibly stressful because I
38:29still feel bad that we lost the scale
38:30like it's still like it's just like
38:32lingered with me and I I admire skill
38:34actually I know hard that that
38:36businesses so I have just like deep
38:38admiration for their like execution but
38:41as a competitive guy I kind of can't get
38:43over it so I'm like always inundated
38:44with questions from VCS like whenever
38:46any annotation company's raising I know
38:48about it because everyone like calls me
38:49but I I honestly try
38:54I know I should be closer to it but I
38:56try to stay away from it just because it
38:57caused me so much anxiety to look at
38:59what's going on that I uh I just can't
39:02what were some of the things you did
39:04differently with the second company I
39:05feel like you know I've started two
39:06companies in with a second one there's
39:07all sorts of lessons I applied
39:08immediately where there are two or three
39:10key takeaways that when you start awaits
39:12and biases made the second time around
39:14easier or was it harder how did you
39:16think about you know key key learnings
39:18or how to apply new things
39:20yeah I mean I think like one thing was
39:23like extreme clarity about who we were
39:27serving so I'm surprised I don't hear
39:29this more because like the the wasted
39:31biases started with a with a customer
39:35and I think it's actually a nice way to
39:37start a company because you know
39:39especially as like a Founder you have to
39:40spend so much time with your customers
39:42you have to seek them out like picking a
39:44customer that you love I think is a
39:46really good thing for your like mental
39:47health you know and so that was like a
39:50big thing and then I think like
39:52I think I've just been a more confident
39:54person in myself like any time I start
39:57thinking like okay like long term or
40:00it's just like you always want to think
40:01long term like everybody wants you to
40:03think short term like everyone's going
40:04to push you to think short-term they
40:06wouldn't say it like that but it's like
40:07you know it's like people can see like
40:10ARR growth they can see like user growth
40:12think it's harder to see like product
40:14quality right and so I think like I
40:17think I'm a competitive guy who likes
40:19you know metrics and likes
40:22but I actually think that can get
40:23counterproductive for me where you know
40:26you start like sacrificing short-term
40:28things to grow these external facing
40:30metrics and I just really try to fight
40:32that myself I know everybody like chases
40:35every entrepreneur chases like
40:36short-term like ARR numbers like in
40:38quarter but then it like hurts your
40:40growth rate the next quarter it's like
40:41it would actually be better always to
40:43like push out deals but like nobody
40:45thinks like that right you can't think
40:47like that but it's I don't think it's
40:48totally rational is there any advice
40:50that you would give to Founders who are
40:51running their first AI company or just
40:53getting up and running
40:55yeah you know the advice I always give
40:58it's like the generic advice that
40:59everyone says it's like even truer than
41:02you think it's even truer than like I
41:03know even though I like deeply believe
41:05it so it's like caring about like if
41:08you're making something people want like
41:09everybody knows it but like no one cares
41:11about enough right like people just they
41:13get distracted they do other weird stuff
41:14even I do it I understand but like you
41:17should care more than you think no
41:18matter how much you think I've never met
41:19anyone that cared too much about that
41:21and then spending time with customers
41:23it's like it's so critical everyone says
41:25they do it but I don't really believe it
41:28like I feel like I'm obsessed with this
41:29I mean like getting like when you're an
41:32early company getting like three
41:33customer calls in a week that's like
41:36tough man I mean you gotta like scrape
41:38and Claw and like beg to get those
41:40meetings and you know like two of them
41:41are gonna like cancel so I know people
41:43tell me oh I met with like 30 customers
41:45this week or something it's like
41:47really did she like I don't know I I
41:49enjoy that really hard to get customers
41:51attention like so I don't know I have a
41:54feeling that nobody does enough of that
41:55but I don't really know I think people
41:56are lying to each other but how much
41:58like actual kind of customer meetings
42:00they're doing and then it's like you
42:01know when you get to a customer it's so
42:03precious it's just like man like show up
42:06prepared and like ask the tough
42:08questions like I think like
42:10I feel like one thing about me is like I
42:11always like default to like wanting
42:13people to like me and it's a terrible
42:14trait in a in a CEO you know it's like I
42:17feel like I have all these like coping
42:18mechanisms for myself to like not just
42:20like kind of flip into that mode but I
42:22think it's good for customer Discovery
42:23because I'm always like so afraid that
42:24they secretly like hate my product you
42:27know that I get like really insecure I'm
42:29just like okay like you know tell me
42:30like more you know like like are you
42:32sure this is really like working for you
42:34actually because it does actually help
42:35in that one important like
42:37entrepreneurial process to lean into
42:40your insecurities with your with your
42:41early customers Lucas this has been
42:43great is there anything you want to talk
42:45about that we didn't cover no this has
42:47been fun I mean I just I think the
42:48message that I'm trying to tell the
42:50world is that we're really trying to
42:52make tools for this new llm workflow
42:54that people are calling lmops and some
42:57of my my advertisement for weights and
42:59biases is like hey if you knew us and
43:00liked us for our ml op stuff try our llm
43:03up stuff called prompt I think it's I
43:06think it's not amazing yet but I think
43:08it's kind of ahead of the market and
43:09it's about to get a lot better because
43:11we are like investing every every
43:13resource that we have into making as
43:15good as possible and we're really
43:16listening to feedback and iterating so
43:18if people want to you know email me
43:19directly and tell me some issue they had
43:21with prompts I I really want to hear it
43:22is it is it Lucas juanb.com come yeah
43:26Lucas are the K yeah at 1b.com okay
43:29you're gonna get a flood
43:31um well I'm I'm optimistic you're such a
43:32Pioneer here thanks so much for doing
43:33this okay that's great thanks yeah
