00:00 on company updates please be honest
00:05 there for you and not for us and if you
00:10 make them clearly crazy like you know
00:14 we're never ever launching we're
00:17 launching in four million years
00:20 we'll get the hint so don't do that
00:27 there have been a lot of questions about
00:30 the graduation requirement which was
00:32 that you had to submit nine out of ten
00:35 weekly updates it this course was
00:39 originally a ten week course and things
00:41 changed around a little bit and it's
00:42 actually more of a nine-week course now
00:45 and so and it was a little bit difficult
00:47 for some groups to get going because of
00:49 our little snafu in the beginning of the
00:51 course so so here's the deal
00:54 eight out of nine updates is just fine
00:57 but we're also going to extend the class
01:00 oh actually really two weeks beyond the
01:03 last lecture so that there will be
01:04 plenty of time to do ten updates if you
01:07 so choose and you choose to do nine out
01:09 of ten but eight out of nine will be the
01:12 absolute requirement for consideration
01:16 for the four that the $10,000 grant
01:20 so we're will be flexible but make them
01:24 good updates make them real updates
01:29 another reminder those weekly updates
01:31 are due 11:59 p.m. every Sunday
01:37 Pacific Standard Time there's been some
01:40 confusing confusion about that as we've
01:42 written it a number of times but please
01:44 recall when that is a questions again
01:50 about anything about about your
01:52 moderators about your groups about your
01:54 updates please send to start-up school
01:56 at Y Combinator comm and my very last
02:01 note is a reminder to launch if you
02:07 haven't already the only way to find out
02:11 if you've made something that people
02:14 is to have people really use it so get
02:16 your products out there if you haven't
02:18 now I'm going to introduce our first
02:20 speaker who is uh my esteemed partner at
02:26 Y Combinator Gustov astronomer who in
02:30 2012 was part of the group that built
02:33 the growth team at Airbnb and if you
02:37 know anything about Airbnb you know from
02:39 2-12 2012 for the next five years they
02:42 grew an enormous amount this is
02:44 something that Gustav knows better or at
02:48 least as good as anyone in the entire
02:50 world so I'm thrilled to have him talk
02:52 today about growth thank you very much
03:02 before I begin thank you so much to
03:05 Steven Jeff adora bill for putting all
03:08 this together so we can be her and do
03:10 this today I'm going to talk about
03:13 growth my background before joining YC
03:16 last summer was being one of the first
03:19 person on the growth team at Airbnb and
03:21 then growing that team from I think we
03:24 were three people in the beginning this
03:27 is how many people were in 2015 we then
03:29 grew two hundred and hundred and twenty
03:31 people or something like that
03:33 almost everything I'm talking talking
03:36 about today here I owe this group that's
03:39 sort of what I learned all this stuff
03:40 some of the best ways to learn to work
03:43 on user growth is to actually work on a
03:45 growth team and I was fortunate to do
03:46 that for for almost five years ago
03:50 so before I start one of the questions
03:56 that might be obvious the answer might
03:58 be obvious for some people but not for
04:00 everyone is sort of like why should we
04:03 why is growth important and I I'm
04:07 surprised how often I actually get this
04:10 questioned but it might be worth
04:11 thinking about it for a second so when
04:15 you make a product some people think
04:19 that if you just put that product into
04:21 the market people will come and people
04:22 will start using the product in my
04:24 experience that's not really how to work
04:27 if you start a startup or a company
04:30 that's aim to be a start-up growth is
04:35 important because that's actually what a
04:37 start-up is if you make something that
04:40 has the potential to be really big then
04:43 making that really big is sort of what
04:46 starts all about there's actually post
04:49 that Paul Graham wrote that it's called
04:51 startup equals growth where he talks
04:53 about specifically why growth is so
04:55 important for startups and why it's not
04:58 important why not every company's to
05:00 startup and why growth isn't important
05:02 for every company but for startups it's
05:05 it's really really important and I'll
05:09 talk in the second sort of like what it
05:12 means to be intentional by growth but
05:14 but the first question is you kind of
05:16 have to answer yes to is sort of like I
05:18 have a startup I want to grow now how do
05:21 we go about doing that so who's this
05:25 talk for in my experience those kind of
05:29 things you work on one and work on user
05:30 growth started with consumer companies
05:33 those were the companies that started
05:37 embracing a lot of the tactics and
05:38 strategies and technologies that growth
05:41 teams were doing now I would argue that
05:43 almost every company that sells anything
05:46 online or get users online should be or
05:51 could be using a lot of the skills and
05:53 knowledge that I'm gonna talk about
05:55 today so you don't need to be just a
05:58 consumer company you don't need to be
06:00 just a social network to apply these
06:01 things these are applicable to a wide
06:05 range of companies if you're a b2b
06:08 company and are excellent at growth you
06:10 have a massive advantage to other
06:12 companies in your space because they are
06:13 probably not going to spend as much time
06:15 doing growth as you are now
06:19 there's one type of companies who this
06:23 is relevant for but too early and that
06:27 actually might apply to most of you
06:28 and this is what this talk is dangerous
06:31 because working on growth the wrong time
06:35 in your company's history can be
06:41 for you it's sort of like when you are
06:44 kind of off to the races a little bit
06:46 too ahead of time so the most important
06:49 thing which is was a lot of lectures
06:50 here start school is about is how you
06:53 make something people want and how you
06:55 actually find product market fit if you
06:58 apply a lot of the things I'm going to
07:00 talk about here to a company that hasn't
07:02 built something people want or haven't
07:05 actually found product market fit really
07:09 bad things happen you will very often
07:12 then this is a very common graph grow
07:14 like crazy in the beginning but then
07:16 when you realize that sort of like that
07:17 fuel they had isn't fueling anything
07:20 it's just fueled kind of evaporates you
07:22 come straight down so that's why it's
07:24 dangerous and now I'll I can go into
07:27 more details about that in a second of
07:29 applying some of these things before you
07:32 actually have product market fit so the
07:34 first thing I'm talking about today is
07:35 going to be around measurement around
07:37 proc market fit because that's actually
07:38 the most essential part for for
07:40 continuing growth now do growth teams
07:45 have impact or are growth team sort of
07:46 like like how do you know that growth
07:50 seems actually matter how do you know
07:52 that that these things man I'm gonna
07:54 talk about that as well like my
07:55 experimentation but before I get there I
07:57 want to tell your story from Facebook
07:59 and this is not a story I came up with
08:01 myself I found it on one of the talks
08:03 from Facebook and I think it's so
08:04 essential for describing the level of
08:08 impact you can have if you are applying
08:12 these kind of skill sets to a company so
08:16 this is the timeline of Facebook from
08:19 2004 to 2015 is sort of their growth
08:22 story some of this is public some of
08:24 some of this wasn't the one thing
08:27 Facebook had they had an excellent data
08:28 science team from the early days they
08:31 were really good at measuring sort of
08:33 like forecasting how big face was going
08:35 to be so back in 2006 2007 when Facebook
08:39 had just started the girth team they had
08:41 a bunch of data scientists doing this
08:44 sort of forecasting how big will
08:46 Facebook eventually be and they looked
08:48 at all the type of metrics at
08:49 facebook.com/ and and they came to
08:51 conclusion that Facebook will be roughly
08:54 by 2015 that was their forecast by using
08:58 all the data they had now we know that
09:00 wasn't really true they actually started
09:02 growing really fast in 2008-2009 and
09:05 this actually as a result of something
09:08 that the growth team came up with and no
09:10 one here had an idea what make Facebook
09:12 growth so fast in 2000 2008 roughly
09:16 she's in 7g a opening to high school
09:20 students yes they did but that wasn't as
09:23 a big of a growth newsfeed was 2005
09:26 sorry recession recession did not know
09:31 10 friends and yeah that's good actually
09:34 it's sort of like the girls story all
09:35 across a Michigan could have been that
09:40 translations is the answer so this is
09:44 surprising for many for many of you
09:46 might maybe but there's actually core
09:48 part of many growth teams is that you
09:51 want to make your product available for
09:52 as many people in the world and Facebook
09:54 star is sort of hitting the limit on
09:55 people who spoke English and then assume
09:57 that people just because Facebook was in
10:00 English and so that the content were in
10:02 local language and Facebook were in
10:03 English it would continue to scale that
10:06 wasn't the case so they implement this
10:08 new platform that automatically
10:09 translated Facebook into hundreds of
10:12 languages and and the growth to be
10:14 now the same thing started happen again
10:18 I think they call this lockdown boom
10:19 they were doing very important things in
10:21 2010 2011 something very big was
10:24 happening at Facebook how to make some
10:26 really large changes to some extent also
10:28 driven by sort of like using data to
10:30 figure out these things any idea what
10:32 what am what happened in 2010 it was a
10:35 really big acceleration mobile I heard
10:38 mobile so so actually what's happening
10:40 is that the data science is a Facebook
10:43 we're forecasting Facebook to be about
10:45 700 million users or something like that
10:47 and because most of those people using
10:51 Facebook on computers they hadn't
10:53 intended this massive shift that was
10:55 happening in 2010 2011 which is people
10:58 getting our smart phones and Facebook
10:59 switching the entire team they I even
11:01 have large training classes for
11:03 engineers just are learning mobile so so
11:07 change to happen then and then actually
11:09 one more thing happen sort of like in
11:11 2013-2014 where the Maeda forecast their
11:14 Academy kind of not hitting the the
11:17 ceiling of growth again but then
11:18 something again happened any idea what
11:20 happened so 2013-2014 Instagram and
11:26 whatsapp could be good answers that's
11:27 not what I'm looking for here messenger
11:30 is a good answer is not what I'm looking
11:32 for so Facebook actually running into
11:34 the ceiling of the Internet the number
11:37 of people actually the one line so this
11:38 started this thing called org which was
11:41 intended to get more people in line and
11:43 they went to carriers and they gave
11:46 basically work with carriers to get
11:47 people on line to get free facebook and
11:49 this actually were very important for
11:51 sort of the continuous growth of
11:52 Facebook now what can we learn from this
11:55 well we can learn that the initial
11:57 forecast of Facebook's growth was about
12:00 400 million people but Facebook today is
12:02 a 2.1 2.2 billion user platform so it's
12:07 a very large platform and the forecasts
12:08 were wrong now the forecast wasn't wrong
12:11 it just didn't intend and taken account
12:13 everything great that the company was
12:15 going to do so if there's anything we
12:18 can learn from this is that if you're
12:19 intentional about growth and you're
12:21 really trying to sort of break through
12:23 these forecasts in these ceilings you
12:24 can grow really really fast so this
12:28 applies to every company and I showed
12:29 this graph to everyone that you own air
12:30 B&B like Aaron leak join the growth you
12:33 midden be and try to get them to think
12:34 in the same way so natural adoption
12:37 which is sort of like that initial
12:40 forecast of how fast your part is
12:42 growing without any of that work is
12:43 always going to eventually slow down but
12:46 if you do the things that you do as a
12:48 growth team you can't sort of like
12:50 continue to push push the growth of your
12:52 company now I mentioned this before
12:56 having a growth teams before you found
12:58 parking market fit isn't really useful
13:00 so let's talk about product market fit
13:03 so the important term but it's hard to
13:05 hard to so exactly define what it is so
13:11 the one way that I think about measuring
13:13 prime market fit is is these two things
13:16 first you find the metric you
13:19 can do this for yourself after this
13:20 talking find the metric that represents
13:22 the value that your customers get from
13:25 your product and then you measure the
13:28 repeat usage of that metric it sounds
13:30 pretty simple and we'll see see if it
13:32 works you can make this kind of a table
13:36 if you want where you have the company
13:37 yeah the metric that represents sort of
13:40 the the value you should get from your
13:41 product and then you have the ideal
13:42 frequency so let's take let's take care
13:45 of MB so you get value from a MB when
13:50 you are booking and staying so if when
13:51 you when you're actually traveling MB
13:53 that's when you find that MB is really
13:55 valuable it's an amazing amazing
13:57 experience unfortunately people don't do
13:59 that more than annually so if you would
14:01 want to figure out the product market
14:02 fit of it air B&B you can always ask
14:04 people after the first day but you
14:06 wouldn't really know until they booked
14:08 again when I come back and book again
14:10 so because the booking cycles for travel
14:13 is so slow annal will be a good metric
14:14 here let's take facebook so facebook if
14:18 you come back to facebook being
14:20 involuntarily come back to facebook as
14:22 an active user you're doing that because
14:24 you find value there's something that
14:26 makes you come back to Facebook or
14:28 Instagram or anything like that and the
14:31 question then is how often should I come
14:33 back well probably daily or hourly or
14:35 something like that they start this they
14:37 started looking monthly and then they
14:40 went down to daily I think there might
14:42 be even going more than that let's talk
14:45 about Gus though what's the product
14:47 value as a customer of Gus though
14:49 whether you get out of using gussto well
14:51 when you run your payroll which is sort
14:54 of what Gus to do for your employees
14:56 that's sort of the one of the best ways
14:58 to sort of measure the value you get out
15:02 of Gus though I know often you do that
15:04 well you run pay low every other week or
15:06 every month I'm not gonna go into
15:08 details of all of this but for lyft it
15:10 might be rides for checker it might be
15:12 background checks for striping might be
15:14 transactions now for your company you
15:16 should figure out what is the one metric
15:18 that I can measure easily every day or
15:21 every time it happens and what's the
15:25 ideal frequency through which that
15:27 should happen if you can answer those
15:29 two questions you can make a cohort
15:31 analysis of your own company right now
15:33 and you can start trying to figure out
15:34 if you have product market fit so then
15:38 you want to measure these things so you
15:39 take a graph on one axis you have time
15:41 and the other one you have have the
15:43 metric that we just decided and from
15:46 that you can try to figure out if you
15:47 have a product market fit so this
15:51 company is measuring this on a weekly
15:55 basis the ideal use case of this company
15:58 is using this product on a weekly basis
16:00 so every week here there's a dot which
16:02 is the percent of people that use this
16:05 product on week zero the first time they
16:07 used it they used it again so let's take
16:11 startup school example so I joined
16:14 startup school and in week one only 60
16:18 percent would come back to school and
16:21 then in week four only thirty percent
16:23 will come back well that would suggest
16:25 the start of school we wouldn't be a
16:26 very good product but that would be a
16:27 good way to measure if start school is
16:29 actually a good product people come back
16:30 to all the content they were creating
16:34 I'll take another product this is a
16:37 great product this is kind of like a
16:39 yeah a normal product would look like
16:40 because almost always you have a little
16:42 bit too many people in the beginning of
16:43 the product and then he will go down
16:45 over time but it will sort of flatten
16:47 out where you keep measuring these these
16:50 events of your metrics this might sound
16:52 technical but it's actually just a
16:54 representation of how many people each
16:56 dawn is how many people does using your
16:58 product for that metric that you decided
17:00 on that on that time window most good
17:03 products flatten out let's look at some
17:07 examples here so these are examples that
17:10 based on payment retention so this is a
17:13 company that you all heard of that
17:16 retains 10% after one month and then 12%
17:21 after 12 months it's sort of like
17:22 unusual who can that be stripe is this
17:27 too hard this is Shopify so shop if I
17:30 have this product where a lot of people
17:32 sign up not that many people continue in
17:34 month two but then they kind of stick on
17:36 forever so what do we know from from
17:39 from from this well if if this curve
17:41 would go down to zero or sort of keep
17:43 going down then shop at five would be a
17:46 it wouldn't be something that actually
17:48 found product market fit and should have
17:49 been working in growth in this case they
17:52 have good good product market fit and
17:53 they should be working on growth they
17:54 should do all they can to continue work
17:56 on growth they should do something about
17:58 sort of like the initial onboarding here
17:59 where they lose a lot of people in the
18:01 beginning but they should be working on
18:04 here's another company 50% retention
18:06 after one month and then 10% after 24
18:09 months so similar to stripe but like or
18:12 similar to Shopify but they continue to
18:14 lose users all the way down to even 24
18:18 months maybe there's some flattening out
18:20 at the very end here but even 12 months
18:22 after people start paying for this
18:24 product there's they every month fewer
18:26 and fewer people pay for the product is
18:29 this product market fit hard to say but
18:32 but it's not an obvious case that they
18:34 have found product market fit so this is
18:37 blue apron here's one that's pretty good
18:40 so 70 percent retention after 12 months
18:45 and then 30 percent after 7 years this
18:49 company definitely had product market
18:51 fit any ideas of which company this
18:52 might be you're all using it Amazon that
18:57 hurt Amazon play wasn't the right one
18:59 Apple no this is Netflix so 70% of
19:05 people that start paying for Netflix pay
19:07 for Netflix 7 years down the line that's
19:11 definitely a company with market fit
19:13 let's just spend every time everything
19:14 they can work in growth so raise your
19:18 hand if you're if you're measuring your
19:19 retention right now not that many people
19:22 well there are other ways to figure out
19:24 retention if you're really small you
19:26 should go out and talk to users you
19:28 should ask them questions such as how
19:31 would you feel if you can no longer use
19:32 my product and you should sort of like
19:35 say as close as you can to your users
19:36 this is by measuring retention like this
19:38 it's very hard when you have 10 users
19:41 when you have a thousand it's easier but
19:43 when you have 10 users is very this is
19:45 not the way to do it you can actually go
19:46 and just talk to all any person so there
19:49 are other ways to measure it but you
19:50 sort of have to know that you have a
19:53 product that's retaining otherwise you
19:57 shouldn't be working on growth because
19:58 you're going to be end up
19:59 burning cycles and things it doesn't
20:04 so a lot of people might be wondering
20:08 how does growth and marketing relate to
20:11 each other isn't that is it the same
20:13 thing or how should I think about it
20:16 historically the way you had a company
20:18 20 years ago is you had a product team
20:21 that made the product and then you had a
20:23 marketing team or a product marketing
20:25 team the marketed the product that's
20:27 nothing's used to work and a lot of the
20:29 sort of like hierarchies and companies
20:31 or sort of organizations are still based
20:33 on this idea that you have a product
20:35 team that's separate from the marketing
20:36 team and these are different teams and
20:38 different skill sets and and that
20:40 engineers work over in the product team
20:42 and then the marketing people work over
20:44 the marketing team actually it's not how
20:46 things work anymore so the way you
20:51 should think about this is that I'm
20:53 gonna come to this in this in a second
20:55 is that there is three times three
20:58 different types of people that can
20:59 organizations that can drive growth
21:01 there is what I call the product growth
21:04 a growth engineering this has different
21:05 names this is effectively product
21:08 managers engineers data scientists and
21:11 sometimes marketers that work on growth
21:13 but they work on growth using technology
21:16 so they're actively changing the product
21:19 to drive more growth and much of this
21:22 work here is about people that already
21:24 arrived here product but haven't really
21:26 found the value of your product yet and
21:28 they're changing the product to make you
21:30 grow faster commercial optimizations
21:33 fall into this this group some of the
21:36 growth channels actually fall into this
21:37 group now there's this big other group
21:39 here performance marketing which is
21:41 effectively Google and Facebook
21:42 marketing which is also super technical
21:44 and super data-driven I would argue that
21:47 these things are very very very similar
21:49 so five or ten years ago you would go to
21:53 a company you see these being very
21:55 different groups maybe different
21:56 different floors in the office now they
21:59 in my opinion should sit together so
22:01 engineers should sit together with
22:02 performance marketers and vice versa
22:05 because they actually are doing very
22:07 similar things now there's this fifth
22:09 button fifth thing should be number
22:13 which is brand marketing brand marketing
22:16 is sort of like the hardest to measure
22:20 the the hardest to measure a type of
22:23 marketing where you're not really having
22:25 a direct response you're not having
22:27 someone directly giving you feedback on
22:29 how that marketing perceived you kinda
22:30 it can't even measure that very easily
22:32 so this is not something that started
22:34 should be doing in the early days my
22:36 opinion start should not be getting my
22:37 brand marketing entail way down the line
22:39 when sort of like they hit the limits on
22:41 these two things so startups should be
22:43 doing these two things and there's some
22:44 qualifications to which you can do this
22:46 but almost everyone can do this if
22:47 you're a startup you have engineers you
22:49 can do product growth or growth
22:51 engineering or or things like that
22:54 all right let's talk about that first
22:56 Proc growth so your product is a funnel
23:01 what does that mean your product has
23:03 many many many steps between the first
23:05 user and the person sort of like
23:08 completing what they're trying to do on
23:09 your product let's say I'm on an
23:11 ecommerce store and I'm trying to buy
23:12 something there are many many steps in
23:14 that funnel and what the product growth
23:17 team is working on is funnel
23:19 optimization or conversion with
23:20 optimization so if you look at what we
23:23 did at Airbnb on the growth team many of
23:26 those growth teams were we're conversion
23:28 rate rate optimisation teams they're
23:31 working in a specific part of the funnel
23:33 so the funnel could start with SEO we
23:37 could start with performance marketing
23:38 it could start with referrals of
23:40 morality but it would jump through a
23:42 number of different steps say sign up
23:45 would be a very common step or if you're
23:47 an e-commerce site payment conversion or
23:50 buying conversion all of these steps are
23:52 things that as a conversion rate
23:54 optimising part of the growth team
23:56 should be working on and these are the
23:58 some of the easiest things to get
23:59 working out and I'm going to be a little
24:03 bit high level here because to go into
24:05 depth about every single area here is
24:07 going to be we can go on for hours so
24:11 some of the sort of like good ideas of
24:15 areas to start working on for conversion
24:17 realizations one of them is translation
24:19 so if you have an international product
24:21 it's not translated you should be
24:23 thinking about that
24:23 that immediately drives
24:26 and drives more people to sell using a
24:29 product the second thing is that
24:31 occation so most of your products
24:34 probably have some idea of user account
24:36 so you can sign up to your product and
24:38 you can come back and login to your
24:41 product you'd be surprised how much how
24:44 hard this is and from Airbnb I know we
24:46 spent many years working on just
24:49 authentication signing up and logging in
24:51 it sounds so simple but it turns out
24:54 that's a very fragile moment of users
24:56 sort of like flowing through your
24:57 product that you can always continue to
24:59 make optimizations if you go to a B&B
25:02 and you go to Pinterest today assume
25:06 that whatever is there in terms of
25:07 authentication is the most optimized
25:10 version these companies spend enormous
25:11 amount of time optimizing sign up
25:15 conversion another big area for
25:17 commercial optimization is onboarding so
25:20 when I come to your product what's the
25:23 first thing I experience and sort of
25:25 like how do I work what can the product
25:26 do to bring me towards the value of the
25:29 product is quickly as possible those are
25:31 things that you're working when you work
25:32 on converting and then another big area
25:36 here is purchase conversion sort of like
25:38 when I'm about to buy something a
25:40 product and actually can take the final
25:42 around what you do there's so many
25:44 things you can do here alright so then
25:49 there is something called growth
25:50 channels so what is it growth channels
25:52 your chance is sort of how people
25:53 discover your product now there is when
25:56 you're a small company and I don't know
25:58 how how big you guys are but let's say
26:00 you have less than 50 users you
26:03 shouldn't really be thinking about
26:03 growth channels even if you're less than
26:06 500 users you probably shouldn't be
26:08 thinking about this because it's too
26:09 early but the things that you do when
26:11 you're small that don't scale have the
26:14 word don't scale in it for a reason they
26:16 don't actually scale and there are very
26:19 few companies who kind of grow big
26:21 without growing on one of these scalable
26:24 growth channels they're not them any
26:26 platforms and the channel slash platform
26:28 can be used in the same way they're not
26:31 that many things that are really really
26:32 big in the world on which you can build
26:35 a large company so let's talk about what
26:39 those those channels are
26:40 so the first thing here is basically you
26:42 think through the behavior of your
26:45 product and let's talk about the first
26:46 one here which is if the way I discover
26:50 your product is a rare behavior where
26:53 people use Google to find a solution and
26:54 this is actually how vary a lot of
26:57 products in the world are being
26:58 discovered if you have something through
27:00 which to answer the question around what
27:02 you do once a year you go to Google
27:05 probably if you're building a company
27:07 trying to answer that question you
27:08 should be an easy B in Google so a good
27:10 example here might be buying a house you
27:13 aren't buying house more than once or
27:15 twice in your entire life which means
27:17 when you go and buy a house you're
27:19 probably going to go to Google which is
27:21 not surprisingly there's something like
27:23 Zillow or red fin or all of those
27:25 different sites that allow you to buy
27:27 homes online are entirely optimized for
27:29 SEO and and sometimes paid search if you
27:34 are using something every day you're not
27:36 going to go back to Google every day
27:37 you're gonna go straight to that product
27:38 you're gonna open the app on your phone
27:40 you can go to the straight to the
27:41 website whatever it might be you're not
27:42 gonna go to Google anymore just kind of
27:44 figure out which of the different things
27:46 I'm going to use and now their Google is
27:48 not the only search and then there are
27:50 other search engines sort of like that
27:52 you can optimize for us well but Google
27:54 is the one that still matters
27:55 next behavior do people of my product
28:00 already share the product using
28:02 word-of-mouth so if that's the case then
28:07 there are lots of things you can do
28:08 around morality and referrals so you can
28:10 grow your product by kind of
28:12 accelerating that behavior from your
28:14 existing users and you can either
28:18 incentivize them with referrals we were
28:19 you getting paid or you can do it for
28:21 free and with with with using virality
28:23 techniques that's having more users on
28:27 my product actually improve my
28:28 experience so what I mean by that well
28:30 if I am building the next LinkedIn it
28:33 makes sense the product is not as
28:35 valuable when it's just all of us on
28:37 LinkedIn but when there's a bunch of
28:39 more people and companies on LinkedIn it
28:42 makes sense that the value kind of
28:43 increases well in that sense you should
28:45 absolutely continue to do virality
28:47 because every single new user is sort of
28:49 opportunity to bring in more users so
28:51 there's another way to think about
28:54 and then this is that common question I
28:59 asked companies in sort of the early
29:00 days of YC can you make a list literally
29:03 a spreadsheet of all the people in the
29:05 world that would use my product let's
29:07 say I sell to buyers who decides to
29:17 doctors offices they sell to doctors
29:20 offices well I can probably make a list
29:23 of all the doctors offices in the United
29:25 States or in California it wouldn't be
29:27 that hard it's totally possible so I
29:30 would find a way to make that list and
29:31 then would go out and do sales and this
29:33 is surprisingly something often
29:35 something that you should start with if
29:36 you can make the list if you know those
29:38 people are you should go on new sales
29:40 and the last one here is does my users
29:43 have high LTV that's me use a hat that's
29:46 does it but do I charge enough for my
29:48 product for it to be valuable
29:51 well then I should definitely go and do
29:53 paid acquisition for example Google and
29:56 Facebook I shouldn't do acquisitions or
29:58 paid ads unless I actually and charging
30:01 for my product all right I'm gonna go
30:03 through each one of these sort of like
30:05 it's not going to be possible to go into
30:07 super deep detail for all of them but
30:09 I'm going to go into a little bit deeper
30:10 into some of them and not the others all
30:13 right let's talk about referrals so I
30:15 worked on a referral program at MB for
30:17 very long time the way to think about
30:20 referrals is sort of an engineered
30:22 word-of-mouth so if people are already
30:24 talking about your product referrals is
30:27 a way through which you can engineer
30:29 more people talking about your product
30:31 one way could be just making it easier
30:32 another way could be by using financial
30:37 incentives so in the mb referral program
30:39 we had a financial incentive where as a
30:43 referre I would make $20 for every users
30:46 that would sign up in travel credit and
30:47 the servery I would I would get $40 as a
30:54 new user of MB in travel credit and we
30:57 would we would kind of like start with
30:59 that principle and then try to get as
31:01 many people what we call through our
31:03 first phones as possible so you meant
31:06 you notice I'm I use the word
31:07 funnel here again so every product is a
31:10 funnel and even there a full product
31:12 which is a sort of a product of itself
31:14 inside the RMB had its own funnel so I'm
31:17 not going to go through all of these
31:19 details but the way you think about
31:22 something that's kind of engineering and
31:23 product led is you break it down into
31:26 into different steps and then you
31:28 measure every single one of these steps
31:30 and then you kind of measure the
31:32 conversion rate to the very end the
31:34 first step here is weekly active users
31:36 that saw the referral program so how
31:39 many uses saw their full program if I
31:41 wasn't measuring this step I wouldn't
31:43 know how many people did that when we
31:45 started measuring this step and I've
31:47 seen this with many other companies when
31:48 they who have a referral program is that
31:50 a small percentage of your active users
31:52 see their full program well how could
31:54 you be expected to use it if you can't
31:57 see it so we started measuring this in
32:00 the early days of a B&B turns out that
32:02 there's a lot of opportunity I'm just
32:04 telling people about you having your
32:05 full program but then there's all these
32:07 opportunities to optimize to through
32:10 that funnel so there's sort of people
32:13 sending invites how many they sent to
32:15 the conversion rate to new users into
32:17 new guests and finally to them booking
32:19 their first-time nights here's another
32:21 new display and this slides going to be
32:23 available online afterwards you don't
32:25 have to I take notes and photos we kind
32:28 of separate the funnel into even more
32:30 detail and we would continue to optimize
32:32 this for years like this all matters a
32:35 lot and we would continue to optimize
32:37 for years I'm going to go through one
32:40 example of sort of like one of those
32:42 conversion rate optimization where did
32:44 for therefore all program so here's the
32:46 referral invite email so if if someone
32:49 else if I invited someone one of you
32:51 guys you get one of these emails and the
32:55 email would say Gustav ahlstrom MIT of
32:57 Jerry B&B Gustav sent you 40 dollars on
33:00 your first trip on the urban bee you can
33:02 book rooms Beauvoir just sign up at 25th
33:04 on may 2018 and then there's a button to
33:07 call accept the invitation and then
33:09 there's a photo of me and my name looks
33:13 this email is a result of dozens of
33:16 experiments nothing here is random
33:18 everything here's for a reason
33:20 let's talk about that the first one is
33:25 the subject line the subject line has my
33:29 name if it's sent to any of my friends
33:30 that make them more likely to open it
33:33 the sort of like a headline of the email
33:37 have a clear value it's very very clear
33:40 what this email is about you'd be
33:42 surprised how many emails that don't
33:44 have a clear headline and the clear
33:45 values are what the email most about
33:46 these email is about that I sent you $40
33:49 on your first trip on this website
33:51 called Airbnb this which is sign up by
33:56 May 25th 2018 is urgency you should sign
33:59 up by May 25th 2018
34:02 that's the urgency that makes you
34:04 actually go that increases their chance
34:06 of people who see this email actually go
34:07 and do it the name here except imitation
34:12 it's the sense of exclusivity it's not
34:14 it doesn't say sign up for reason
34:16 because sign up anyone can sign up by
34:18 accepting an invitation sends an idea of
34:21 exclusivity and finally it has my name
34:24 where I live and how long I've been a
34:26 user of Airbnb with my photo that's
34:29 really strong social proof that I
34:31 endorse this website I endorsed this
34:33 product so the way you think about this
34:36 is sort of how you think about all a
34:37 conversion rate optimization it's a set
34:39 of optimizations pay growth this is one
34:45 of the areas I'm not going to go too
34:46 deep too deep into the the options here
34:49 the things that matter for pay growth is
34:51 that you shouldn't do it unless you have
34:54 revenue too many companies are buying
34:56 ads when they don't have any revenue or
34:59 know how they're gonna make any revenue
35:00 that's the big mistake you shouldn't be
35:02 doing that if you have revenue there's a
35:06 couple concepts that really matter the
35:08 first one how much money am i paying for
35:11 each new users that arm acquiring it's
35:13 called custom acquisition cost or CAC
35:16 what's the lifetime value or payback
35:18 time of the users that I've acquired
35:22 through paid growth so what that means
35:24 is what's the longest that I can with
35:27 some level of accuracy forecast how much
35:29 these users will be worth if a users
35:34 and then we can sort of have this cohort
35:36 analysis that we had in beginning with
35:37 the attention we can figure out exactly
35:39 how much our use is going to be worth
35:41 and if my customer acquisition cost is
35:45 lower than my lifetime value then that's
35:48 good that means you have a great payback
35:51 if if you only pay $20 for a user that
35:55 means if you make $10 per month you have
35:58 probably around $20 at two months
36:01 payback time that's more complicated in
36:03 that but it's sort of like a simplified
36:05 version these are the most important
36:07 concepts to keep in mind for for pay
36:09 growth attribution so this matters if
36:13 thinks it complicated if using both
36:15 Google and Facebook if you're getting to
36:18 some some level of scale you need to
36:19 understand what it means to activate a
36:21 new paid user to a dollar that you spent
36:25 I'm not gonna go too much into detail of
36:27 what this means but but this is
36:29 something that you should learn if
36:30 you're paying time on paper oath and
36:32 finally in my opinion there are only
36:34 four channels to sort of matters at
36:35 scale Facebook Instagram Google and
36:38 YouTube these are the very very large
36:41 paid growth channels through which many
36:44 companies are actually built on these
36:45 days sort of like one of the sort
36:50 unspoken truth in my opinion is that a
36:52 lot of the free channels of growth is
36:54 going down and the and the pay channels
36:56 are going up a good example of that is
36:59 that this hard to grow on the Facebook
37:00 newsfeed unless you're paying Facebook
37:02 money the same is true for Instagram
37:03 it's sort of changing always it when
37:05 there's a new platform they often kind
37:08 of promote free usage in beginning and
37:09 then they start to monetize that and
37:11 that sort of was happening for both
37:13 Facebook Google and Instagram today all
37:18 right search engine optimization so
37:20 search engine optimization or SEO some
37:23 people might say well this is something
37:25 of the past in my experience it is not
37:27 it still matters because as long as we
37:29 go back to Google to make our decisions
37:31 about what we want to do in the future
37:33 this matters Google is probably one of
37:37 the largest websites in the world it's a
37:39 big deal the one thing you should know
37:42 about SEO is that there's a difference
37:45 between what you see
37:47 on a website and what Google see so
37:50 Google can't view images Google can't
37:55 view JavaScript very easily so if you go
37:59 to Airbnb and you see all of this just
38:02 remember that Google can't see this so
38:04 if you're trying to communicate to
38:05 someone who's searching for Stockholm on
38:08 it on Google or apartments in Stockholm
38:11 this is not what going to be delivered
38:13 back to Google Google is going to see a
38:16 bunch of lines of text through which
38:18 you've marked up in your code and the
38:21 Google have hopefully indexed if you
38:23 done the right thing and if you haven't
38:26 then it's kind of like your fault
38:29 so they're a bunch of basics food that
38:31 you can do with SEO you just have to get
38:33 right the easier thing you can do is to
38:36 run your website through an SEO browser
38:38 and try to figure out if you only saw
38:40 this would you understand what your
38:42 practice is about so using clear
38:45 language not in JavaScript so you
38:49 describe what your product does is the
38:51 most basic things you can do for for SEO
38:56 there's two different areas of
38:59 optimization for SEO and and this is as
39:02 high level as it gets so at MMB there's
39:05 a team of 20 people working on this of
39:08 which 12 or 13 are engineers this is a
39:10 really really big deal but but but this
39:13 is sort of the high level there's two
39:15 types of after my stations for SEO there
39:17 are things that you can change on your
39:18 website and there are other people in
39:21 the world of linking to your web sites
39:23 this is the two main levers that you
39:24 have an SEO for the on-page optimization
39:29 the right way to start is not to start
39:32 with like a list make some changes the
39:34 right ways to start with the strategy
39:35 what am I trying to rank for what
39:38 keywords exist on Google today that I
39:40 might want to be the number one result
39:42 for now to do that you have to do what's
39:45 called a keyword analysis or keyword
39:46 research you have to try to figure out
39:48 what are all the things that people are
39:50 searching for that might relate to my
39:52 product and then hopefully they have
39:55 some amount of large volume that they're
39:57 searching for and then I can try to say
39:59 well these keywords that have
40:01 sort of a medium to high volume and
40:02 they're not that much competition around
40:04 these keywords those are the ones I want
40:06 to rank for when you decide that that
40:08 the other areas of impatient of my
40:10 stations now are easier because now you
40:11 know we're trying to to rank for at
40:14 scale but I've seen this as small scale
40:17 to SEO at this point is about SEO
40:21 experimentation which means you can do
40:23 the set of best practices in SEO it
40:29 might take you to be a decent size
40:31 company but if you want to become a
40:32 massive company a really really large
40:34 company that's growing through SEO you
40:36 have to use experimentation to make
40:38 those decisions the other thing that
40:41 matter here is off page optimization
40:42 who's linking to you so there are lots
40:45 of tools you can use to figure out who
40:47 were all the websites online the links
40:48 to you what's their sort of domain
40:51 Authority are they actually
40:52 authoritative on the web to have
40:54 something there's a huge difference if I
40:56 will link to your website and if you
40:57 have x link to your website it matters a
40:59 lot in sort of how in google's
41:03 perception of your of your of your site
41:04 so one surprising way there may be is
41:09 that we had a lot of press a lot of
41:12 people in the world from media would
41:14 come and write about Airbnb and that was
41:17 surprisingly important for off page
41:19 optimization a few people riding my you
41:21 either impressed or in other areas
41:23 that's actually a great thing if you
41:24 don't have anyone writing about you well
41:27 you're not gonna have them any links
41:28 because the web has changed the aren't
41:30 that many links anymore kind of it's not
41:32 like everyone have a a website these
41:35 days they'll link in to each other so
41:36 they said things have changed you have
41:37 to be strategic about who are you
41:39 getting links from easiest way you can
41:41 do is whenever you get written out up by
41:43 anyone presser or whatever just ask them
41:46 to link to you it matters to you last
41:51 thing I'm talking about here is on
41:52 growth teams so a growth team today is
41:57 typically engineers data scientists
42:03 designers product managers and user
42:06 researchers these are not the all the
42:09 people you might have in your company
42:10 today but these are the company
42:12 companies that you will have when you
42:13 start making decisions to run growth
42:14 the way you organize the growth team is
42:17 there's sort of two options you either
42:20 have the growth team as his own team and
42:22 the rest of the practice is the product
42:23 team and that's actually sort of my the
42:30 the challenge here is that when you when
42:32 your small company and and you're not
42:34 kind of like somewhat ignorant about
42:36 growth what you eat often say is like
42:38 I've hired a growth engineer I hired a
42:40 growth spike manager that person is
42:41 going to be doing all the growth in the
42:43 company that's sort of not really a
42:45 recipe for success so there's kind of a
42:48 fine balance here with between saying
42:51 everyone is responsible for growth which
42:53 doesn't work and having a small team
42:55 that are responsible for all the growth
42:57 you kind of have to find a balance here
42:58 and the right way to find that balance
43:00 it's a set very clear goal and goals and
43:02 very clear dividing lines in your
43:04 product so a good example here might be
43:07 everyone that works on the core product
43:09 which is sort of the value of your
43:11 product let's say I am gusto the the
43:18 running payroll for employees is the
43:20 core product now everyone trying to get
43:23 to that experience that's sort of like
43:25 the growth area of gusto so there'll be
43:27 one easy way to kind of make distinction
43:29 between growth and product so how do you
43:32 decide what to work on you make a simple
43:37 calculation of what's the effort what's
43:39 the minimal viable thing I want to try
43:40 try here and sort of how big of an
43:43 outcome can that be so I always try to
43:45 forecast out the outcomes before you
43:47 actually start doing the work because if
43:49 you're forecasting something and a
43:50 best-case scenario it's small you
43:52 shouldn't be doing it even though it's a
43:53 low effort all right I have two small
43:58 short sections left of this talk the
44:01 first one is called making decisions if
44:05 you ask any product manager MB today
44:08 what is the most important tool or
44:11 learning that you learn to ever be that
44:13 you'll apply to your next thing they
44:15 would say experiment or experiment
44:17 framework or some way of making a B test
44:20 inside the company and I'm so happy that
44:23 so highl is talking later because I
44:25 stole like a quote from his investor
44:27 on line this is the quote most of their
44:31 world will make decisions by either
44:33 guessing or using the gun they will
44:35 either be lucky or wrong if you keep
44:37 making decisions without using data and
44:39 experimentation you will be lucky or
44:41 wrong and this is a huge problem so if
44:46 you if you kind of get to a scale of a
44:49 and B or something even smaller than
44:50 that then every decision you're gonna
44:52 make that you don't use a/b testing for
44:54 you won't actually know what the full
44:56 true outcome of that decision might be
44:57 so whatever BMB we use experimentation
45:00 and a be testing for every single major
45:02 decision in an tighten the entire
45:04 company so this is how that tool work a
45:07 and B but my talks are sort of like more
45:09 about how that look like let's say in
45:12 your company you decide to ship a
45:14 feature and this is how many kind of
45:17 measurements of Madaya metric per day
45:19 and and the Wednesday here is when a
45:21 ship that new feature and then I look at
45:24 it two weeks later and looks like that
45:26 the metric of that feature went up so
45:28 that was a good thing right well it's
45:31 not that easy because there's so many
45:33 more different factors who knows what
45:36 happened between here if you're a soccer
45:38 app and the World Cup you started well
45:40 that and you wouldn't know if the
45:41 features have made a difference if this
45:45 was just peak season or whatever you're
45:47 doing this is a school app education app
45:49 and this is September then you wouldn't
45:50 know either so the only way you would
45:53 know is what hack what's called a
45:55 counterfactual basically an a/b test by
45:58 running two different versions of the
46:00 same feature the same site the same time
46:03 you would be able to know what the true
46:06 difference between making these
46:08 decisions and not make me it was this
46:11 might sound a bit technical but it's
46:13 very very important to internalize
46:14 because you will get to a point where
46:16 you were successful up until that point
46:18 and you think that you're so good at
46:20 making these decisions and then they get
46:22 harder to make and you have to have a
46:24 framework to make these decisions so
46:28 what we and then because this was so
46:30 important MB we built something we call
46:33 experiment experiment review experiment
46:35 of you is when the whole growth team
46:37 would meet in a room like this would go
46:39 through all the features that we had
46:41 and before we told you which actually
46:44 should have the different features in
46:46 the experiment they actually won with
46:48 asked the audience so I'm gonna do that
46:49 with you guys here so here's a photo our
46:53 experiment review we would do this every
46:55 two or three weeks it's really fun but
46:58 he drove home this one thing which is
47:00 that making practices are really hard
47:02 alright let's get started so the goal of
47:06 this project was to increase the number
47:09 of shares from the FMB mobile app
47:12 specifically the number of shares of
47:14 listings and back at that time we had
47:17 two options we had the native share
47:19 shoot that you'll know about and then we
47:21 have this experiment share sheet which
47:23 takes up the entire screen has more
47:25 colors the same type of buttons sort of
47:28 you can see the difference here now
47:30 before I show the answer how many here
47:32 think that the native share sheet led to
47:35 more shares raise your hand it's behalf
47:39 the room how many here thought they'd
47:41 the new share sheet that one of our
47:42 engineers built drove more ships raise
47:44 your hand so the most many hands would
47:47 do not race I'm assuming you guys think
47:49 there was no difference because this is
47:51 so hard you have to make a decision you
47:53 have to make a decision you have to have
47:55 an opinion if you don't have an opinion
47:56 here you're basically saying I can't
47:59 so this share sheet led to 40% more
48:04 shares so very important in this case
48:06 that we'd use experimentation because if
48:08 we just gone by a gut we would have been
48:09 wrong half of the room more than half of
48:11 the room would have been wrong
48:14 let's do one more so we send out this
48:17 email to existing users of everyone b22
48:21 sort of when I would book make a booking
48:24 ebon be listen up this email to the
48:26 people that I listed as my code
48:28 travelers and the email will look in the
48:31 control like this where it would have
48:33 the itinerary the address to that
48:35 listing and the button on that email
48:38 would say join Gustav's trip now we
48:43 tried a new version where the email look
48:45 a little bit different a less content
48:48 and then we have a different button here
48:49 called accept Gustav's trip imitation
48:52 how many here the goal here was to
48:55 sign up so people have to click on this
48:57 button and then sign up how many here
48:59 think that Joan Gustav's trip and the
49:02 control led to more signups to raise
49:04 your hand some people how many Herot
49:07 think that except good stuff stripper
49:09 mutation raise your hand that's good
49:11 how many here think that there was no
49:13 difference so this is a 14% increase of
49:19 just changing the basically a name of
49:21 the button let's try one more for
49:23 simplicity so in this case now they're
49:26 sharing experiments I'm kind of giving
49:29 you the very easy to understand ones so
49:32 the control here was sort of like a
49:35 bunch of sharing options on the MMB
49:37 listing with had some Twitter icons and
49:39 facebook icons demon icons we got
49:41 another version of that which the icons
49:43 were round and then we have one that was
49:45 called sort of square buttons and it was
49:47 just displaying email and Facebook and
49:49 then you can click on more how many here
49:51 thought if the goal was sharing that the
49:54 control with the icons one raise your
49:56 hand some people how many a thought that
50:01 the round buttons were better raise your
50:02 hand some people I leave every thought
50:07 the square buttons wow you guys are
50:09 how many thought there was no difference
50:13 well both of these were winners this was
50:15 by far the biggest winner and these are
50:19 the kind of things that we debated and
50:20 because we run experiments you don't
50:22 have to debate it anymore so proc
50:25 decisions are really hard at scale
50:27 you'll want to use experimentation a be
50:29 testing to make these decisions because
50:32 otherwise it'll be the loudest room it
50:34 allows us voice in the room they'll make
50:35 the decision so you don't want that to
50:38 summary sort of my talk if you're
50:41 running startup you should be thinking
50:42 about growth before you start working on
50:45 current act exceed be measuring your
50:48 attention and knowing if people are
50:50 using repeatedly using your product you
50:53 should pick a metric and then pick a
50:55 goal for that metric and drive that
50:57 metric towards that goal
50:59 very simple and then eventually but not
51:03 you should start running a/b tests for
51:05 the decisions that are hard to make in
51:07 the early days decisions
51:08 that hard to make as there might be
51:09 obvious but the moment that not easy
51:12 anymore you should be building doing
51:14 this with experimentation thank you I
51:24 think we have time for a couple
51:26 questions yes do you have any frameworks
51:34 or ways you think about trying to be
51:36 honest so the question was around
51:41 experimentation in SEO how do you go
51:45 so what you're trying to test when you
51:49 run an on-page experiment what you're
51:52 trying to test is sort of the amount of
51:54 organic traffic you get from Google and
51:56 the sort of how your ranking shifts so
51:59 Google actually are changing the
52:01 rankings all the time so if you make a
52:03 big change on your website on say say
52:05 half a say let's call you have an for
52:08 every B's case listing pages sorry
52:11 searched search results pages so
52:13 Stockholm London San Francisco we buy at
52:15 one group and then you have New York
52:17 Paris and Barcelona B another group you
52:19 would change the content on the first
52:21 group of pages but not the others within
52:24 not too long of a time you would see
52:27 more or less traffic going to either of
52:29 these groups from Google sometimes
52:31 there's no difference but if there is a
52:33 difference you'll see more traffic give
52:35 you an idea of a very small change you
52:37 can make let's change let's let's for
52:39 example change the title tag from air
52:43 and be listings to the twenty top best
52:46 listings in Barcelona one of them is
52:49 going to rank better because you can be
52:50 more inviting to click on and then you
52:53 run that experiment towards the search
52:55 engine and then nutrition will kind of
52:57 tell you which one is better based on
52:58 the amount of traffic and the search or
52:59 anything you get yes
53:21 the question was around a B testing how
53:24 do I determine if it's right for me when
53:26 I'm really early so a B testing is a
53:30 function of change so you can have a
53:33 medium to small size audience if the
53:35 change is large enough it actually can
53:38 be since the significant like the
53:40 numbers can be significant now what you
53:44 can do is you go do you can go to Google
53:47 and typing a/b testing calculator and
53:49 there's a kind of a form will pop up and
53:51 you can type in sort of the the metric
53:54 that you have and then they change and
53:56 you can see how big the change have to
53:58 be for you to be able to see a
53:59 difference now if you're really really
54:01 small it probably shouldn't shouldn't be
54:03 doing maybe testing at all so I would do
54:07 a B testing when you have large like
54:10 enough traffic that you can see a medium
54:13 to small size change within say two or
54:15 three weeks so let's say small change
54:17 would be a couple of percent if you can
54:20 see that in a couple of Ko weeks
54:22 kind of like that sort of that level you
54:23 could be at that's probably sort of my
54:28 general recommendation I've certainly
54:31 seen a lot of companies and their early
54:33 embracing a/b testing relatively early
54:35 and then have that guide saw that their
54:38 decision making and I don't think that's
54:40 a bad idea because I think that's much
54:42 it's better to start a little bit too
54:43 early then to start way too late if
54:46 those are the only two options
54:48 of course you start at the right time
54:49 yep how do you apply growth to high
54:57 barrier entry market like like health
54:59 insurance so I think you should separate
55:05 sort of the acceleration of your growth
55:08 in that market and the market itself so
55:11 if it's hard to reach people that means
55:15 you have to probably try to reach to
55:18 reach a lot more people before you
55:22 if if I hi tell me why we mean by bytes
55:27 are like a high barrier yeah yeah so if
55:44 you have a risk in regulation involved I
55:47 don't think that the principle of growth
55:49 changes but if you're purely growing
55:52 through sales for example you can apply
55:55 a lot of automation and technology to
55:57 how to do sales so if I'm a company and
56:00 I'm selling to insurance buyers at
56:05 startups I might be starting by emailing
56:09 10 people and in fact that can email 100
56:12 people or a thousand people with the
56:14 same type of level of the email itself
56:20 is kind of a field as personal and as
56:23 direct to that to that receiver as if I
56:25 send one email that is remove myself so
56:27 there's certainly kind of things you can
56:29 apply to your growth that allows you to
56:32 reach a lot more people now the growth
56:34 doesn't solve any of the risk challenges
56:36 at all it doesn't solve any kind of
56:38 major market issues and and be certainly
56:41 had a lot of issues with legal like
56:43 legal challenges in different markets
56:45 and sort of growth was disconnected from
56:47 that that wasn't sort of our our goal if
56:51 you're in a start-up you have those
56:53 those problems I don't think you should
56:55 solve them solving them and solving
56:58 growth are two different things and
56:59 they're very separate from each other
57:00 and and growth is a way to accelerate
57:03 you getting to more health insurance
57:05 buyers it doesn't actually say anything
57:07 whether that's that's a good thing or a
57:20 do I have any wisdom amusing
57:22 non-sustainable tactics a new market
57:24 like uber so if you're doing things that
57:28 don't scale you sort of have two options
57:32 one is they don't scale so you should
57:34 stop doing them eventually or you build
57:36 sort of a playbook where you take those
57:38 skills and you kind of try them in a
57:40 different City now they might not
57:41 eventually scale so let's say I am out
57:45 manually to recruit uber drivers now
57:48 that might be an unsustainable growth
57:50 tactic it might be working for the first
57:52 twenty drivers but eventually it's not
57:55 going to have high ROI in comparison to
57:57 say running ads on Facebook to recruit
58:00 drivers so I think you kind of have to
58:05 determine this manual thing that I'm
58:07 doing in the very beginning which is
58:08 doing things that don't scale which is
58:09 sort of recruiting every single user to
58:11 become a user of your product eventually
58:13 it won't work anymore and you kind of
58:14 have to find another channel or you
58:16 won't be high ROI in comparison of the
58:19 other channels and you'll most companies
58:20 will go through this transition period
58:21 where you go from say writing content
58:25 manually to do engineering changing your
58:30 website so that we get more search
58:32 traffic for example what about doing
58:37 subsidies or incentives for rides well I
58:40 don't I actually think the incentives
58:42 are super scalable like at MMB we still
58:45 have the referral program they're
58:46 sending up on hundreds of thousands of
58:48 users every day almost which and large
58:52 portion of them are sending up through
58:53 referrals so that is the example with
58:55 subsidies are actually scalable now me
58:59 handing you a coupon in your hand it's
59:01 not scalable but doing that through an
59:03 email system or through whatsapp or
59:04 through messenger or some other channel
59:06 is actually scalable as long as you're
59:08 not losing losing the money so you have
59:11 to have a very good ROI calculation on
59:13 the money that you're handing out and
59:14 knowing that that's incremental money
59:16 that if you didn't get get that money
59:18 out those users wouldn't actually start
59:19 using your product so as long as you
59:22 have a good handle on that that is
01:00:00 so question what's around they have two
01:00:01 groups of users so the users and the
01:00:03 paying users and some of them are very
01:00:05 active and some of them are not not
01:00:07 exactly but at least I don't want to pay
01:00:08 how would you use growth there so one of
01:00:11 the things that we're trying to figure
01:00:12 out is what is the retention rate of
01:00:14 those paying users are the actions
01:00:16 taking around and using the product that
01:00:18 they're paying for and what's the
01:00:20 retention rate for the free users do
01:00:22 that change once it starts paying and
01:00:25 then it will look at the conversions or
01:00:26 like how what percent of the freezes and
01:00:27 am I converting to paid and there's many
01:00:29 different ways to do that there's one
01:00:30 which is having the freemium concept you
01:00:33 can also have a free trial which means
01:00:34 you actually don't have freemium you can
01:00:36 just have a very limited set of free
01:00:38 users and then they kind of have to go
01:00:39 to paid but ultimately to the extent
01:00:42 that you should be growing all of this
01:00:44 comes down to how much usage you have
01:00:47 from these the one you they want the
01:00:49 users that your value so if the paid
01:00:51 users are the one you value the most if
01:00:52 if they stop using a product after three
01:00:54 months then you have a more fundamental
01:00:56 problem in the conversion between these
01:00:58 two groups so I would still go back to a
01:01:00 retention and look at usage for those
01:01:02 people and see are they actually
01:01:03 sticking around and using the product
01:01:05 last question from over here yes in the
01:01:30 so the question is sometimes
01:01:33 experimentation can be really hard to
01:01:36 execute is there an ideal sort of
01:01:37 frequency or which which experiment
01:01:40 should I be actually be running for
01:01:43 something like a marketplace
01:01:44 experimentation is really hard that is
01:01:48 so in that case it's easier it's harder
01:01:54 than just setting up a simple tool
01:01:56 online or using Mixpanel to set up your
01:01:57 simple a/b test it's more difficult than
01:01:59 that if you are it's kind of having a
01:02:03 simple product funnel it's generally not
01:02:05 that hard to set up experimentation if
01:02:06 you have an engineer and if you have
01:02:09 decent amount of traffic it's not that
01:02:11 hard to set it up now there's different
01:02:14 kind of products that make
01:02:15 experimentation harder I think that if
01:02:20 you're at this at this stage where you
01:02:23 have a lot of traffic you in my opinion
01:02:26 you don't have really an option you sort
01:02:27 of have to invest in some level of
01:02:29 infrastructure to use data to make
01:02:31 decisions if you don't do that
01:02:33 the alternative cost is you end up
01:02:35 making a lot of bad decision or they're
01:02:37 either long or they're lucky you're
01:02:39 lucky because they're right or you're
01:02:40 wrong so you don't really have an option
01:02:42 but to run experiments and it is true
01:02:45 that there's some type of products
01:02:47 through which experimentation is easy if
01:02:48 you're running a mobile app
01:02:49 experimentation is not that difficult
01:02:51 there are lots of tools you can use to
01:02:53 run experimentations if you're running
01:02:55 out of ideas for experiments well then
01:02:57 you should go back to user research and
01:02:59 you should look at other products that
01:03:01 are sort of look at something like
01:03:04 Pinterest or Airbnb or Facebook they're
01:03:06 probably highly optimized so a lot of
01:03:08 the things in their funnels is there for
01:03:11 very specific reasons because of a/b
01:03:13 testing so I don't have any better
01:03:17 answer then if you don't do it you'll
01:03:20 have a lot of other problems and in
01:03:23 terms of the frequency there is a cost
01:03:26 to setting up an experiment and that
01:03:29 cost should be minimized as much as
01:03:32 but it's sort of like you should just
01:03:35 try to get better at it and and pick the
01:03:38 easiest smallest thing that you can test
01:03:39 and just test that if you have a big new
01:03:41 features you want to test we'll just
01:03:42 test the first part of that feature
01:03:44 don't test all of it and see how people
01:03:45 react to the kind of the first part of
01:03:47 the feature I can talk for hours about
01:03:49 this but thank you very much
![How To Succeed As A NEW & YOUNG Realtor [Deals Every Month + Luxury Listings]](/_next/image?url=https%3A%2F%2Fi.ytimg.com%2Fvi%2FrtrRghgTayc%2Fhqdefault.jpg&w=3840&q=75)
