00:00how would you set the goals for youtube
00:07hey everyone i'm here today to do
00:09another product management mock
00:11selena we're super excited to have you
00:14back on the show to do another interview
00:15selena but before we jump into it
00:17could you tell us a little bit more
00:18about yourself absolutely
00:20nice to meet you viewers my name is
00:22selena i'm a product manager at facebook
00:25and actually knows steven from back
00:26during our time together at stanford
00:28and i also previously did a mock
00:31interview with exponent about
00:33facebook movies which is really popular
00:35stephen mentioned so happy to be back
00:37yeah um your video just reached a
00:39hundred thousand views by the time
00:40people are watching this one it might be
00:42much more than that but we're super
00:44excited to have you back uh today and at
00:46the request a lot of people
00:48to talk about an execution type question
00:51these are more analytical metrics driven
00:53questions that are asked
00:54by top-tier tech companies um and so
00:58i'll jump into asking you the question
01:00and there'll be a couple follow-on
01:01questions that we'll do to investigate
01:04um and so first off we're gonna be
01:05talking about youtube today um
01:07which is the platform that we're gonna
01:09be showing this video on
01:10um and an awesome like big high growth
01:14right now especially during the
01:15chronovirus pandemic um
01:17so the question that i have for you
01:19selena is how would you set
01:21the goals for youtube yeah absolutely
01:24just give me one second to collect my
01:26thoughts take your time
01:33all right so i think the way that i'd
01:35like to go about this is
01:37first we'll talk a little bit about
01:38youtube youtube's mission
01:40and what their overarching goals would
01:42be and like a high-level metric bucket
01:44that we would want to set as success and
01:46then we'll dive into
01:48user actions that could best support
01:50that overall metric so we can hone in a
01:52little bit closer on
01:53what we'd actually like to track and
01:55then from there i'll propose some
01:57tactical metrics as well as a top line
01:59goal metric and then probably some
02:01counter metrics that we want to watch
02:03and then i'll close it up with an
02:04evaluation of any trade-offs
02:07sounds great thanks for laying it out
02:10yeah so youtube's mission overall i
02:13is to first of all entertain its users
02:17billions of people all over the world
02:18love to go to youtube to watch videos
02:21but also to inform i think a lot of
02:22people also use youtube with that use
02:26educate themselves and learn something
02:29so in terms of an overall metric i think
02:32we could either go for
02:34you know an engagement bucket metric or
02:37retention or monetization um so youtube
02:42ads and they actually also have a
02:44subscription product a couple
02:45subscription products um
02:46youtube premium youtube tv so i think we
02:50could consider a monetization as well
02:53i really think that it's important to
02:55focus first on engagement and then
02:57monetization will follow so
02:59if you have users who are loving your
03:00product who are coming back to your
03:01product and spending time in your
03:03that's how you're able to monetize and
03:05you know either insert ad breaks or
03:06upsell people to the subscription
03:08so i think i would really like to focus
03:10on an overall goal of monetization we'll
03:12we'll drill i mean sorry
03:14of uh engagement and we'll drill into
03:18further as we go in that sounds great um
03:21before we jump into that perspective i'm
03:23curious selena if you would have a
03:24perspective on when we wouldn't
03:26prioritize it that way like would we
03:27um is there a case you can imagine where
03:29we might prioritize monetization over
03:33um yeah i think you know it youtube
03:36is a mature product so we we could focus
03:40at this stage i think you just want to
03:42be careful to not focus only on
03:45and forget why uh you're building the
03:47product in the first place which is to
03:49i think once you have a really stable
03:51sturdy base like definitely
03:53you can add teams that really focus on
03:55monetizing and really growing your
03:58your books but i think you know overall
04:00engagement is the way to win
04:02and you know there's competitors
04:04encroaching in the space like uh
04:06uh videos for example on facebook watch
04:08so i think engagement is the way to go
04:10in this particular case
04:12totally cool yeah go ahead with the
04:14engagement and and we'd love to drill
04:17okay awesome so i think next is
04:20um thinking about user actions that
04:22would help to lead to
04:23engagement so youtube is all about
04:28i would say that's really the number one
04:30action that we want to drive is you know
04:32how many videos users are watching per
04:34day how long they're staying to watch
04:36videos um you could probably also look
04:38at the diversity of videos they're
04:41deep diving into one particular area or
04:44you know exploring the full variety of
04:46content from different content producers
04:50so i think that's on the watch time side
04:53on the other engagement activities that
04:56you can do i know you can
04:57like you can comment you can subscribe
05:00so i think those are all really
05:02important actions that help fuel this
05:04you know youtube it's not just viewers
05:06there's also creators
05:07which is really important for the
05:09overall amount of content that's on the
05:11so i would want to make sure that there
05:13was um you know engagement in terms of
05:15users uh following them and commenting
05:18so i think those are the two buckets
05:20that i'm going to start with for now so
05:25which is you know number of videos
05:30watched overall you know length of
05:38and we can also look at the average per
05:43as well as the overall top line numbers
05:47kind of other engagement actions
05:51which was liking commenting
06:01i'm trying to think are there other
06:02actions that i want to highlight
06:06in terms of engagement
06:10you know i think you could also consider
06:14uploads might be a way to measure
06:22i think i do want to focus on the viewer
06:24in this particular case
06:25so i'll probably focus more on there but
06:27i did want to highlight that since this
06:28is a two-sided market you'd want to
06:29think about the creator side as well
06:31but i think the overall you know north
06:33star is really about people watching on
06:42okay so now that we have this like large
06:45metrics that we could look at i'm going
06:47to prioritize a couple so
06:49i'm going to pick some tactical metrics
06:51that we would want to you know it's more
06:53of a gut check to make sure we're on the
06:54track and then a top line metric which
06:57is our overall north star this is the
06:58one we want to be driving
06:59to accomplish our overall mission of um
07:02entertaining and informing people
07:04um and then i'll also suggest uh
07:08um things to watch out for to make sure
07:11running into too many trade-offs along
07:16okay sounds great so i'm just gonna take
07:24okay so i have my three categories
07:27i have my tracking metrics i have my
07:29north star metric and i have my counter
07:32so i'll start first with tracking so
07:35this is really what's telling me if this
07:36was early on in the launch of the
07:38product that things are going in the
07:40these are early indicators leading
07:42indicators that things are going well
07:44so i would want to track average watch
07:48and this is letting me know for a given
07:50user who's watching how long
07:52are they watching i'd also want to track
07:55average number of videos watched per
07:58this is giving me a good understanding
08:00of you know the number of videos that
08:04as well as potentially some distribution
08:08and then the last one is a likes
08:13shares i'm sorry not shares um
08:16videos viewed so i'd want to see that's
08:18almost like a like through rate
08:20um a kind of blended metric so i'd want
08:22to do a weight to figure out you know
08:24what particular weight of these leads to
08:26users being more likely to come back
08:28and retain a week from now and to engage
08:30more with videos and i would want to see
08:32a healthy score there
08:34got it and on that last one do you have
08:36a hypothesis on what blend or how you'd
08:38prioritize or wait some of those
08:40likes comments and shares or subs i
08:44yeah i think here i would actually be
08:46inclined towards subscribers because i
08:48think that's what really powers
08:50the ecosystem that then would lead
08:52creators to want to upload more videos
08:54so i think that would be really
08:56important but also this is also a social
08:58product so i would want to make sure
09:00you know discussion going on in the
09:02threads as well got it
09:06um okay so then this brings me to my
09:08north star which is really watch time
09:10at the end of the day that's how you
09:12know you're gonna get your monetization
09:14down the line and how you're also going
09:16users are enjoying the content that's
09:18there um and this can vary with length
09:20of video so there's different lengths of
09:22content on youtube like this video will
09:24be a certain length so
09:26i think we also want to pay attention to
09:30watch time is distributed as the length
09:33right because maybe you'd have someone
09:36one hour video and bailing out and they
09:38only watched one video versus people
09:39watching a lot and there's a lot of
09:41content format links that are coming out
09:43so i think that could help to inform
09:45the strategy moving forward
09:48totally um that's something a great
09:50north star metric like
09:51um any concerns with it as we've come to
09:54this or kind of circling around this
09:57yeah so i think that's a great uh
09:59transition to my counter metrics
10:01and one of them would be you know uh
10:04wanting to make sure that people spend
10:06it's like time well spent um i know
10:08that's the facebook metric but basically
10:10we don't want people spending so much
10:12on youtube that they forget to live so i
10:14would want to make sure that people are
10:17healthy and enjoying the internet in a
10:20i think there's also a distribution
10:21concern right if people are just
10:23really rabbit holing on one particular
10:25length you know they're not
10:27expanding to the full ecosystem of of
10:29creators so i'd want to make sure that
10:31you know all the different creators had
10:33at least some share of that distribution
10:35um and then i i guess there's other
10:39counter metrics that aren't related to
10:40watch time do you do you want to talk
10:42inverse of watch time metrics or am i am
10:45i good to cover the other counter
10:47um you can cover the other kind of
10:50yeah so the other two that i wanted to
10:54churn so this is all about engagement um
10:57and retention's not quite the same but
10:59you know a drop-off in users who weren't
11:02this would show me that the quality of
11:04the videos we were showing wasn't
11:06matching what they expected
11:07so i would be looking out for that and
11:09another one i would look out for
11:11is the amount of reported videos so
11:13maybe watch time is up that is really
11:15these videos are low integrity which is
11:19you want to make sure that you know the
11:21content is up to a certain standard
11:22especially there's you know a lot of
11:23kids that watch youtube
11:24so i'd want to make sure that you know
11:28a positive and safe place for all of our
11:29users to be totally that makes a lot of
11:32um any other comments about the metrics
11:35and sort of the ones that you've the
11:37kind of metrics as well before
11:39i ask a hypothetical question about the
11:43yeah i think you could maybe also
11:45consider the counter metric of
11:47creator distribution you know i kind of
11:49mentioned that as something you'd want
11:50to track along watch time
11:51but i could see that also being a
11:53counter metric to look out for if you
11:54just saw a really high concentration on
11:57um i know this is also something that
12:00youtube changed the way that videos
12:01could go viral to help ensure
12:03kind of more distribution for all
12:05different types of videos so
12:07you wouldn't want too much watch time
12:09concentrating on any one particular
12:11producer or video type so you might want
12:13to track that not just as a north star
12:14but as a counter metric as well
12:16totally um okay awesome well thanks for
12:19the thoughtful discussion
12:20around watch time i want to transition
12:22this interview into a different sort of
12:25we're going to imagine that that key
12:27north star metric that you picked of
12:30is unfortunately suffering a three to
12:33four percent decline
12:34over the past couple of weeks and so or
12:38weeks two months maybe uh let's
12:40approximate but um the time range isn't
12:43as insofar as what i want you to do is
12:45to imagine that you're a pm
12:47at this team um and you know you've done
12:49this thoughtful consideration
12:50around watch time but
12:53how would you investigate what's going
12:57causing this decline in that north star
13:01yeah absolutely so i think the most
13:04important thing whenever
13:05this happens in an interview situation
13:07or in real life on the job as a product
13:09manager you want to start by asking
13:11you never want to immediately just run
13:13off and start diving
13:14into the data by yourself like make sure
13:16that you're asking some key questions
13:17about this new trend
13:19so i think the first thing i would want
13:22long has this been happening did this
13:24happen yesterday has this been going on
13:26maybe a month yeah let's say that it's
13:30weeks to a month so maybe like four
13:34a month or two ago like we've seen this
13:37um within the range of like two to three
13:39months or one to three months or so
13:42got it so sounds like it's been going on
13:45does that mean it's been a gradual
13:47change or was it acute
13:48like three months ago and we're just now
13:51getting around to dealing with it
13:52yeah we're seeing the gradual decline to
13:54the state where it is today over the
14:02so i think my next question would be are
14:06decline across the board
14:09in terms of geography or is there a
14:11particular geography where we're
14:15we're seeing the decline actually mostly
14:18domestic market so united states um
14:21we're not seeing the decline as much in
14:22emerging markets or foreign markets
14:25got it that's good to know okay let me
14:32you know normally i would ask is it just
14:34this product or a broader product but
14:36since it's youtube that's kind of
14:37the whole thing i guess i could ask um
14:41and youtube premium is this focused on
14:44watch time of a particular segment
14:47um that's a great question it's not
14:49focused on a particular segment so we're
14:50seeing this decline kind of across the
14:52on youtube over the past couple of
15:00all right this is going to be my last
15:02question before i'll put into buckets my
15:05issue triage identification which is is
15:07there a particular device type that this
15:09is focused on great question um there
15:13a device type associated with this
15:14decline so the devices seem to be
15:16uniform across iphone android
15:19as well as the different device types
15:21within those categories so like
15:23the different iphones or the different
15:27or um like uh pixel phones
15:31got it okay so at this point i'm gonna
15:34pause my general questions and then i'll
15:36focus in three different buckets of
15:39the issue could be arising and i like to
15:42internal to my team um so i should
15:45probably ask you what team i'm on
15:46um an internal to the company and then
15:50are the three general buckets i look
15:52into so i i know i'm about youtube is
15:54there any particular team that i'm on
15:57um let's say that you're on the um
16:00general youtube team you're not on
16:02premium or youtube tv
16:04you're on the youtube product that
16:06people generally see
16:08when they come to youtube.com as a
16:11got it okay so given
16:15what i'm hearing that this happened
16:17gradually from weeks to months
16:19focused in the u.s and it's happening
16:21across the board this would lead me to
16:23think it was either due
16:24to something that happened externally
16:26potentially with a competitor
16:28or that something that happened
16:29internally that has happened over time
16:32um that's either due to an experiment or
16:35but since it's gradual this is telling
16:37me that it's a user behavior shift
16:40normally more than it would be um an
16:42infrastructure issue or a system that
16:45which would typically demonstrate a more
16:49acute sudden decline so uh i'll run
16:52through a couple questions
16:54i think first i'll ask the the burning
16:56question which is you know
16:57uh did our competitors launch something
17:00month period that's drawing um share
17:03away from our products because that's
17:04what i'm feeling is maybe most at play
17:08um definitely we've seen a lot of
17:10competitive products in this space in
17:11the past couple of months
17:13um and so there is tough competition and
17:15we do know that from just like a
17:17marketing or partnerships perspective
17:19but it's a little unclear to us right
17:20now if that's the reason that we're
17:22experiencing the decline
17:23but that definitely could be something
17:24we want to investigate further
17:29um all right let me think um
17:33maybe i'm also thinking externally i
17:36uh partners and third parties so is
17:38there anything that happened potentially
17:40with a partner or someone that we work
17:42with that would have gradually
17:44caused this decline maybe they were
17:48um like i know we offer youtube on tvs
17:50maybe some older tvs no longer
17:52support our youtube app or or something
17:54like that that could be causing the
17:56yeah that's a great question and you
17:58know a lot of for example
18:00um android devices might partner with
18:03oems to make sure certain apps appear
18:05there and there could be a case where
18:07the youtube app did not appear on a
18:10but in this case um we're not seeing any
18:12changes in partnerships in fact the
18:15let's just say is in two months so the
18:18renewals of all those contracts would
18:20then and it wouldn't have happened
18:21within the past few months
18:29so i guess this is my last external
18:31question before i go into
18:32internal to the company problems was
18:34there any bad press that happened about
18:36company um maybe a couple months ago
18:38that would cause users to slowly but
18:43it's a great question again i mean uh
18:45cedric himself has been
18:46testifying in front of the u.s
18:48government um so there could be
18:51for now probably for purposes we're not
18:52seeing something so damaging or so
18:55shocking that would have caused a huge
18:57change in the measures
18:59got it okay so now i'll ask some
19:01questions that probably are arising from
19:03uh internal to the company issues um but
19:06i'm still holding on to that it could
19:07have been something with the competition
19:13my second leading theory uh second from
19:16competition would actually be
19:20did we launch something new that's
19:23watch time um that would maybe be
19:25drawing users slowly away
19:27um or maybe something we're
19:28experimenting with that's being rolled
19:30out gradually that could be
19:31impeding watch time yes so we've
19:34launched um a feature on youtube
19:38where you can see it's kind of like
19:39instagram reels or tik tok
19:41um where we've created a lot of like
19:43short clips of videos that users can
19:45browse through really quickly
19:48youtube shorts yes um
19:52yeah well i would say that that would be
19:56since it's shorter length content to
19:58lead to a decrease in
20:00watch time three to four percent seems
20:03consistent where i wouldn't expect it to
20:05be an enormous sea change but
20:08as users are adopting this product more
20:11if that length of videos is shorter than
20:14youtube video length then you would see
20:16a corresponding drop in watch time
20:19um like that's not necessarily a bad
20:22you know um given all of the engagement
20:25especially among younger audiences with
20:27tick tock this could be a strategic
20:30um and it is a strategic choice to
20:32launch this product so
20:33i wouldn't say that it's necessarily a
20:35cause for alarm in the near term but we
20:37should definitely be thinking about
20:38you know how much watch time are we
20:40willing to give up um to support this
20:43and you know maybe maybe watch time
20:45isn't the right north star maybe it's
20:48enjoyment users are getting out of the
20:50videos and since there's a general trend
20:53videos maybe you know a total max watch
20:55time isn't necessarily the
20:57best demonstrator of people enjoying
21:00totally um and yeah like you know in
21:02this particular case that we're talking
21:05um we can imagine that youtube shorts
21:08um and so in that case the algorithm
21:11maybe let's say to ensure long-term
21:14engagement or longer term watch
21:16time and so that might have cannibalized
21:19traffic that we normally expect with
21:22short form content that is not as
21:25at finding the algorithm to get users to
21:27continue to watch the experience
21:29which could definitely happen when a
21:31company launches a new product but
21:32hopefully in the future cases youtube
21:35wouldn't see that decline
21:36as much yeah absolutely i think since
21:40algorithms learn from user behavior it's
21:42almost expected in the near term
21:44if you're moving away from a longer
21:46content form to a shorter content form
21:48that it's not going to immediately get
21:50totally um so congrats to getting the
21:51right answer i know it's stressful and i
21:53thought you did a great job kind of
21:54discussing a lot of the different
21:55criteria i'm curious also for you if you
21:59were going to just dig any further is
22:01there anything else you would have
22:03had you had the opportunity to to
22:05explore some of the other
22:06avenues or answers yeah so um i think a
22:09similar side of the coin i would ask you
22:11know are we de-prioritizing our
22:13regular product in favor of another one
22:16the exact same question but focusing
22:20relative prioritization of the two
22:22products which i think is absolutely
22:24um sometimes i look at internal to the
22:28which i think with a product like
22:29youtube maybe is a little less valid but
22:32sometimes you can think okay did my
22:33particular team launch an experiment
22:35that's impacting another team or vice
22:37um i think here it's easier to think
22:39about it as a total platform effect
22:41the way youtube is as a product but
22:43that's always an area i like to explore
22:45totally um well selena you can take off
22:48the interview e-hat you did an awesome
22:50job i want a debrief
22:52yeah i want a deep brief um like how
22:54this was for you and
22:55um i wrote down some thoughts on what i
22:57thought you did so awesome with this
22:58interview and i thought it was a really
23:00conversation that's really educative
23:02also so i'm really grateful
23:04you were able to do this but before i
23:06jump into my thoughts did you have any
23:09self-feedback on how that went for you
23:12yeah i think overall it went really well
23:14i mean full disclosure i did intern at
23:16uh about a year and a half ago now feels
23:20so i did have a little bit of
23:21familiarity with with the company
23:23um and yeah i think the the one area
23:27that i felt like i could have done a
23:28little bit better was defending
23:30why not to go for monetization um you
23:33know i think it's always
23:35a little bit of a snake chasing its own
23:38engagement and monetization but you
23:40can't run a company without having money
23:42at the end of the day
23:43so i think that's something that's
23:45important to understand
23:46when is the right time to focus on that
23:48and i think that's something a lot of
23:49companies struggle with
23:50um but otherwise i thought that that
23:53i think maybe you were looking for a
23:55different trade-off that i didn't
23:57during the the metric identification so
24:00i guess i'll i'll listen to you on that
24:02um and then uh in the second half
24:05around the issue identification i think
24:08there's two kind of flavors for this
24:10sometimes interviewers are looking for
24:13um the case and sometimes they just want
24:15to hear how you would talk through that
24:18both in my my interviewing experience
24:21you know either way you want to just be
24:23reasoned with your thinking and
24:25if you don't always get to the right
24:26answer even if that's what they were
24:27looking for and you talk through it in a
24:29structured way i think you can still do
24:31totally that's awesome advice um yeah
24:34let me run through i mean i agree with
24:36some things that i thought you did
24:37really really effectively celina were
24:39one you you did such a great job
24:41i felt like i was always on the same
24:42page with you throughout that interview
24:45you know you even did stuff where you
24:46would say like you know i'm gonna move
24:47on to this section and you really
24:49communicated with me
24:50frequently throughout the interview
24:52especially with execution questions it
24:54really easy to kind of slip into this
24:56pattern where you're gonna say
24:57hey like is it this is it that is it
25:00and then um i as an interviewer don't
25:03really have a sense of what you're
25:04thinking or how you're thinking about
25:06and so that's not an effective tool you
25:08know for the interviewer to assess your
25:10but um one line in particular that stood
25:12out to me was you know you said i'm
25:13going to keep competition in the back of
25:15over here and you know i'm going to move
25:17on to this section and i thought that
25:18was really effective because you
25:19you sort of acknowledged the interviewer
25:21you know i haven't fully figured that
25:23and i know it's still an open issue but
25:24i'm going to investigate these other
25:26and i thought that communication style
25:28was really really valuable
25:29i also thought you did a great job of
25:31breaking down the problem so you'd often
25:33or um split up the problem space into
25:36different components for example
25:38external to the company external or
25:40sorry internal to the team internal to
25:42or external that was a really awesome
25:45way to break down the problem
25:46and to know where your head was
25:47throughout each of those steps so i
25:49thought that was really effective as
25:51you did that a couple times um i think
25:54i may have jumped uh ahead of you on
25:56this one but mentioning countermetrics
25:58you know making sure you slip those in
26:00is really important um i actually didn't
26:02counter metric that i was looking for so
26:04it's funny that you thought that maybe i
26:05was just putting on my tough interview
26:06or face or something but
26:08um it was more more honestly like uh
26:11the counter-metric discussion that i
26:12think you did was a really thoughtful
26:14um if i had more time i would have asked
26:15you more about you know like time well
26:17spent what does that look like in that
26:19um i do agree the monetization angle we
26:21could have had maybe a richer
26:22conversation around that
26:24but i didn't think that that was so um
26:26important to the essential
26:27interview question that i was asking it
26:29was kind of a side tangent that i took
26:31um so honestly overall and the last
26:33thing i wanted to mention is that you
26:35really did a great job of thinking about
26:36the goals of the company first
26:38these questions often seem tricky but
26:41really make a lot of sense when you
26:42think about the goals and so i love that
26:44you thought about the goals first and
26:45then kind of boiled it down so
26:46i don't have too much constructive
26:48feedback i thought this was like a
26:49really awesome answer
26:50it was a really fun conversation also um
26:53and then yeah maybe maybe one last point
26:55that i would say is just the execution
26:57um you know being specific about how
26:59you'd identify those issues so you know
27:01you asked if they're competitors
27:02um but there might be more specific ways
27:04to identify competitive
27:06competitive uh companies in the space um
27:09so perhaps you could have asked about
27:11um you know are there other analogous
27:12products within google's ecosystem
27:14or partner network that have seen a
27:17are users of youtube's api seeing a
27:20decline or you know embedded videos on
27:22youtube seeing a client that might
27:24start to hint if it's a competitive
27:27that was just one constructive piece of
27:30feedback but overall
27:31really awesome interview do you have any
27:34thoughts reactions comments or last tips
27:38no i think that all sounds great and i
27:40think that's a really awesome suggestion
27:42you know think about other other ways to
27:45plug into is this an internal or
27:46external issue outside of the the core
27:49apis was a great suggestion if that's
27:52type of product you're focusing around
27:54to get at that same question from a
27:57um thanks again selena for being on the
27:59show thanks it was really great to see
28:02and for those of you watching good luck
28:03on your upcoming interview
28:06thanks so much for watching don't forget
28:08to hit the like and subscribe buttons
28:10to let us know that this video is
28:11valuable for you and of course
28:14check out hundreds more videos just like
28:16this at tryexponent.com
28:19thanks for watching and good luck on
28:20your upcoming interview
