00:00I didn't do anything yet so you don't
00:09need a clap but yes we're not going to
00:12talk about the nature thing you can
00:14probably look online about it but yes
00:17we're gonna talk about this other stuff
00:18okay good so a lot of you guys are doing
00:22sprints already I'm not going to talk
00:23about you know any details about the
00:26sprint methodology I'm more going to
00:28teach some related valuable concepts
00:31which I think helped to pull the whole
00:33thing together like the sprints in the
00:35larger context of getting effective
00:38outcomes and also organizational change
00:40and as mentioned the intro I was one of
00:43the founding team members of Google X
00:45which is the organization made the
00:47self-driving car glasses project wing
00:49this is project loon some geothermal
00:51 nobody even heard about that anyway
00:55this is a new approach to artificial
00:57intelligence as a contact lens that
00:59measures your glucose levels and I think
01:02that the when people hear about these
01:04projects it's like well it's because
01:06Google has like so much money like they
01:09have so much money they can explore
01:11stuff like this they have so many great
01:12engineers they can throw a hundred
01:14people onto any project and make it work
01:17you know it you know they had all the
01:19time in the world to go do this but just
01:21to reset your expectations a little bit
01:23Project loon which is has the you know
01:26small ambition of trying to provide
01:27Internet access to every square inch of
01:29the earth the first 100 Hardware
01:31prototypes of Project loon were produced
01:33in in four months by just five people so
01:36100 Hardware prototypes four months five
01:38people and by the end of 100 Hardware
01:40prototypes we had multiple balloons up
01:42at 25 kilometers doing communications
01:45with each other in the ground and the
01:47total amount of costs like the
01:49prototyping material cost for that
01:51hundred prototypes was $70,000 so that
01:54is not a lot of people it's not a lot of
01:56time that is not a lot of money like the
01:58difference was how we were working and
02:00I'm going to tell you a little bit about
02:02some of the ideas of how are we working
02:05all righty so when I joined Google it
02:08was right at the beginning of Google X
02:09it was so early that the division didn't
02:12yet and you couldn't talk about it so I
02:14guess I was like entering a group that
02:16didn't exist but luckily there was like
02:20eight other people in the group that
02:21didn't exist and that included folks
02:23like Sergey Brin Sebastian throng yeah
02:26the best probabilistic roboticist in the
02:28world for my money not that anybody bets
02:30on that but Bach Pervez Astro teller
02:34these are people that I call nerd famous
02:36like if you're super nerd like yes if
02:39you're not you're like that's a bunch of
02:40people's names and for whatever reason
02:43I'm in the room too and we start and we
02:45sit down on my first day of work and
02:47this is near the beginning of Google X
02:48so we self-driving car doesn't exist yet
02:51glass doesn't exist yet loon doesn't
02:53exist yet that sort of thing we have
02:55this single sheet of paper and it comes
02:57from Larry and Sergey the founders of
02:59Google and if anybody has got in a
03:01project spec before you're used to like
03:03kind of a big document this is a single
03:05page it only you know it's only eight
03:07sentences long it covers a third of the
03:10page and the very first sentence in that
03:12and that piece of paper said imagine
03:15that Google was in your brain now this
03:17is my first meeting on my first day of
03:19work and I'm reading that the first
03:22sentence on the first page and I was
03:24like whoo that's a terrible idea
03:26but luckily there were seven more
03:28sentences and basically the seven more
03:30sentences said something like this well
03:32right now Google is kind of a web
03:34service that you access through a
03:35browser and you to bring that browser up
03:37in your device and usually to get onto
03:39your device you need to login or unlock
03:41your phone that kind of thing so it
03:43means that between the time that you
03:45have an idea for something you want to
03:46know and being able to use Google to go
03:48and you know get the answer to that
03:50thing that you want to know you got
03:52these four or five steps in the way so
03:54what we kind of imagine is that there'll
03:55be some point in the future that as soon
03:57as somebody you know thinks that they
03:59want to know something they just
04:00instantly know it so that's the seven
04:02sentences and the last sentence says
04:04realize this in less than five years
04:05done so luckily we had a good long
04:10meeting we had a 90 minute meeting to
04:12kick this off and I was like oh that's
04:14plenty of time and we jumped right into
04:15the topic and it's in that meeting that
04:17we start talking about hey what if we
04:19could build a heads up to slay what if
04:20we could build you know that would allow
04:22you to you know at least get on the
04:24trajectory toward being able to
04:26know things as soon as you want to know
04:27them without having to pull a device out
04:29what if we built a contact lens well
04:31then you don't need to pull anything out
04:32at all actually yeah you know you don't
04:34need to wear anything visible because
04:36the contact lens no it just being your
04:39eyes so it's kind of invisible what if
04:41we improved a voice assistance
04:43technology AI such that we were able to
04:45go process queries and in a lot of other
04:48ways other than just typing them in all
04:50these things would get us on trajectory
04:51to a world where you could instantly
04:54know things just by as soon as you
04:56wanted to know them and this was
04:58basically the first half of the meeting
04:59and I thought the first half of the
05:00meeting was one of the best meetings I
05:03had ever experienced in my life because
05:05right before I was at Google I was at
05:07Yahoo as an executive helping to run a
05:09multi-billion dollar business in 44
05:10countries and it sounds kind of
05:12impressive but my entire life was just
05:15meetings and politics so many meetings I
05:17made a new unit of measure which I
05:19called meetings per linear foot because
05:21I had like meetings blocked it back to
05:23back all day and even when I was walking
05:25between meetings like people would run
05:26up trying to get my approval on a thing
05:28I was like oh no I had three meetings in
05:30the 800 feet between these two meetings
05:33so I was exhausting who was comes from a
05:38formal training in engineering and
05:41design like that's the sort of life that
05:43will destroy your soul so just to talk
05:46about things all the time and never
05:48build things and have it all be very
05:49political so when I got to Google and I
05:52was in this meeting on my first day of
05:53work I was like this is a breath of
05:55fresh air we were talking about making
05:57this stuff there's equations that are
05:58appearing on the board these are my
06:00people and that was the first half that
06:03was the first 45 minutes of the 90
06:05minute meeting but right around the
06:08midpoint it turned from being the best
06:09meeting of my life to one of the worst
06:11meetings in my life because we spent the
06:14next 45 minutes just arguing about what
06:17color the heads-up display was going to
06:18be and not the paint job but like the
06:21photons that would go through the device
06:23and it sounds kind of trivial but it's
06:25not completely engineering trivial
06:27because different colors of light have
06:29different indices of refraction which
06:30means you design your optics slightly
06:31differently you need a different light
06:33engine to go power it a bunch of things
06:35so it wasn't super completely stupid but
06:38what was stupid was the way we were
06:40stuff I was instantly transported back
06:42to Yahoo where we would have these
06:44meetings that I would label guess a
06:46Thon's everybody we're just guessing the
06:48guy think it's gotta be this color let's
06:50go out be this color well I think it has
06:51to be this color and everybody would
06:53just guess at each other guests at each
06:56and the way that guests of Thon's get
06:58handled is either the person that is
07:01best at arguing or the person with the
07:02highest title in the room ends up
07:04deciding and that you know meeting after
07:06meeting was extremely exhausting so here
07:08I was we went from one of the best
07:11meetings of my life in the first half to
07:12one of the worst and by the end of that
07:14meeting you know the normal guest Athan
07:18thing happened remember the person who's
07:20best at arguing the person with the
07:21highest title in the room on that
07:23particular day it was the same human
07:25being sergey brin and he basically told
07:27us hey we talked about this for 45
07:30that's enough I've made the decision the
07:33color in the heads-up display is gonna
07:35have to be red it has to be red for two
07:38important reasons number one red photons
07:40are the least energetic they're going to
07:42impart the least energy onto your retina
07:44and given that we're trying to make
07:45something that's comfortable enough to
07:46wear all day has to be red least energy
07:48most comfortable wear all day has to be
07:51red reason number two it's always red in
07:56now everybody in the room their boss is
07:59Sergey and I think they're sitting there
08:00nodding and it's all good but I had just
08:03come from this corporate environment in
08:04Yahoo and I was like no this is how we
08:06made decisions in Yahoo and look at how
08:08that how that turned out and I remember
08:12like you know even as I was walking out
08:14the door of that meeting I was like no
08:18this is my first day of work that we're
08:20at the beginning of an entirely new
08:22group it's not like we have all this
08:24ingrained bureaucracy there's no reason
08:26that we need to work like this like we
08:29can make this work any way that we want
08:32we're just starting you know we're not
08:34going to do that like and and by the
08:37time I had walked out the door I vowed
08:39well what can we do instead what we're
08:41gonna do is we're just going to build
08:42the heads up display we have all these
08:44great like engineers physicists
08:45designers in the room we're just going
08:47to build it we're gonna answer the
08:49question based on what we've built so
08:52for anybody that doesn't know the answer
08:53to this how long do you think it took to
08:55make the first working version of the
08:57device so that you could go answer the
08:59question from that morning you can say
09:03something now okay good this is good you
09:09guys are good so that the the answer is
09:11later that day and this is the second
09:14one it's not the first one the first one
09:15was done before lunch this is made
09:17slightly after lunch and basically what
09:20it is is there's a coat hanger that
09:21rests behind the neck actually right
09:23behind it you can see my welcome
09:24balloons anyway there's a coat hanger
09:27that RESPA that hangs behind the neck
09:29the other end of the coat hanger is bent
09:31in a spiral it holds up a clear piece of
09:33plexiglass so you can easily see through
09:35that and at the end of the piece of
09:37plexiglass there's a sheet protector
09:39which is a material that's like really
09:41easy to see through but if you project
09:43an image onto the material also holds
09:44the image really clearly and at the base
09:46of the plexiglass there's a pico
09:49projector so basically anything that you
09:50can put on to a little netbook a little
09:53computer you can basically have floating
09:55in front of you as you walk around and
09:57and interact with it and there he's
09:58working with he's wearing a glove that
10:00he can also interact with it but that's
10:02a separate thing so very first thing
10:04given the guest on in the morning was
10:07let's go tape out several paragraphs of
10:09put them in every color of the rainbow
10:10and make it so that we can just try to
10:12read in this color and then you know
10:15flip to the next color try to read in
10:16this color flip to the next color
10:17and once this prototype was built i
10:19brought the people from the meeting you
10:21know over to experience it and one by
10:23one they put it on and remember these
10:25are the same human beings that argued
10:26about this for 45 minutes and you know
10:28finally through the gesso thaad process
10:31decided that red was going to be the
10:34color within 90 seconds it didn't take
10:37them more than 90 seconds go only takes
10:38you about 10 seconds to read a paragraph
10:39and then you flip to the next color 10
10:41seconds Siri paragraph you flip the next
10:43color within 90 seconds every single
10:45person that tried this agreed that red
10:47was the absolute worst color for
10:49heads-up display and it was not just the
10:51worst it was the worst by an enormous
10:52margin but and after red purple was the
10:56second worst much better and the best
10:58ones were light blue white and yellow
11:00now and total agreement across all the
11:03people I used this and ironically it
11:06turns out that red is the absolute worst
11:08color for exactly the reason that Sergey
11:10said it would be the best color red
11:12photons are in fact the least energetic
11:14they do impart the least energy on your
11:15retina and it means that when you're
11:17walking around in a little Armant every
11:19other color of photon outcompetes red
11:21you know if you're using night-vision
11:23goggles absolutely red is fine there's
11:25not that many competing photons in every
11:27other environment in the world walking
11:29in office being outside anywhere in the
11:31world red is the absolute worst color to
11:33be able to read in now
11:36I bet though when I shared those two
11:37reasons that Sergey gave to and the
11:39guests at thon at least that first one
11:41sounded a little smarter right like
11:44there's not that many dumb people that
11:45can use the word photon and retina in a
11:48sentence that is true to physics because
11:50all that is true and if you took away
11:53nothing else from this talk this would
11:55be a great thing to take away which is
11:56that smart people will always come up
11:58with smart reasons for their guesses but
12:02it does not mean that their guesses are
12:03not guesses and and what we spend most
12:06of our time at work doing is sitting
12:08across from people and trying to
12:10evaluate whether what they're saying
12:11sounds smart or dumb or something in
12:13between whether we agree or we don't and
12:15I'm just going to tell you that the 90%
12:17of the cognition that you spend at work
12:19doing that process is wasted cognition
12:23it actually doesn't matter how much you
12:25agree or disagree if it sounds smart
12:27it doesn't sound smart the only thing to
12:29heat listen for is is what is when I
12:33hear something is it a guess or is it a
12:35direct experience if it's a guess it
12:38needs to be treated a certain way and it
12:41doesn't it's not problematic there's
12:42great things that you can do with
12:44guesses but if it's a direct experience
12:47then that's the stuff we want to make
12:49decisions off of when everybody has the
12:51direct experience of all the colors and
12:53we we know that red is the hardest one
12:55to read when rendered as text it's easy
12:58to make that decision right because the
13:00direct experience derives very clean
13:02decisions in the absence of the direct
13:04experience you can have all the same
13:05people you can have really smart people
13:07on your team you can even have a great
13:09working dynamic but if the if the
13:10material that you're working with is
13:12guesses not only can you be wrong you
13:14can be wildly off and you can be certain
13:16you know about the status of the
13:18rightness of your thing which is wildly
13:20off right so think about the material of
13:23cognition is the material of cognition
13:24that is coming in guesses or the
13:26material of cognition that you're
13:27working with direct experiences and the
13:30reason that we have the sprint process
13:31the reason that sprint pasta's is so
13:33essential is we start in a world where
13:35most people are working in the medium of
13:37guesses right and the sprint process
13:40says well let's all get together at
13:41least be having the same sort of
13:42conversation and at the end of the
13:43sprint process let's start making some
13:45prototypes so that direct experiences
13:47can start happening and in that process
13:49you guys kind of go through that
13:51transition from working with a very poor
13:53material just you know arguing about
13:55each other's guesses to working with a
13:57highly enriched quality material which
14:00is direct experience and this is my
14:02mantra for this oh I forgot to say one
14:04if you tell a smart person that they're
14:06guessing they would tend to be offended
14:08if you tell them that they have a
14:10conjecture they're like oh yeah I have a
14:14now of course they mean the same thing
14:16but and I said this before but this is
14:19this is the mantra this is the way to go
14:21parse this out when you are sitting
14:23across from that person in your hearing
14:25is this a gas or this is a direct
14:26experience like an actual experience
14:28that they've had well if it is a gas or
14:31a conjecture it doesn't mean that you
14:32need to be like hey well that's just a
14:34guess or that's just the conjecture
14:35that's that's not that productive but
14:37what you do in your mind is you pose the
14:40question to yourself well that's a
14:41really interesting conjecture how could
14:43that become an experiment what's the
14:45thing that we could try that would allow
14:46us to have a direct experience in the
14:48process of moving a conjecture to an
14:51experiment we allow for a direct actual
14:53experience to happen now once we do
14:55those experiments we will have a lot of
14:56actual experiences and I just call them
14:59actuals for short and actuals are great
15:01for driving decisions if nobody can read
15:03the color red it's really really easy to
15:05decide that the color red is not the
15:07color for the text in that system all
15:10right I got to go fast because there's a
15:12bunch of things to cover so that's the
15:14thing from Google I also work with a
15:15bunch of organizations like helping them
15:18speed up their rate of innovation and or
15:21work on really difficult technical you
15:23know innovative projects but in this
15:26case I got called in at the last minute
15:27it was actually my birthday too is a
15:28weird day but I got called in the last
15:30minute to work with this company
15:31Schneider Electric and they're like a
15:33big electrical utility provider in in
15:37European and internationally not as big
15:40in the US but just to give you a sense
15:42they made 23 billion top-line revenue in
15:44the year that I was working with them so
15:46yeah they now make some money but they
15:48came to me and they basically said
15:52because remember I usually work on how
15:54to make companies more innovative or
15:56work on like oh how do I bring
15:57artificial intelligence to this this new
15:59area like a really difficult technical
16:01problem they came to me they're like Tom
16:03we got something really difficult for
16:05you it's an HR problem I was like oh did
16:09you read my bio because I like to make
16:10self-driving cars and stuff but ok HR
16:14problem what is the HR problem and they
16:16basically described you know some of the
16:18background to me but it came out to
16:20something like they were losing 70% of
16:22their top executives every three years
16:24and I was like whoa that is not
16:27little HR retention problem that sounds
16:29like a Geneva Convention problem and I
16:32was like well what's happening and they
16:34basically gave me the background there's
16:36two things that they were
16:37organizationally very proud of that when
16:39you add them together it led to them
16:41losing all these top executives number
16:42one is like I said they're there in all
16:45these countries but instead of like
16:47working as one big centralized org they
16:49broke it all up so that all the
16:50different countries are basically their
16:52own org their own P&L and that P&L has
16:54got a CEO at the top of it that gets to
16:56make those local decisions and the
16:58benefit of that business-wise is great
17:00they're able to make these very nimble
17:01decisions so that they can gain market
17:03share in that specific country and it
17:05was working for them so great now the
17:07other thing was that you know if you
17:11guys have ever you know spent any time
17:13around executives there's this thing
17:14called succession planning which is if
17:16you get hit by a bus or like you need to
17:18leave the org you need to have other
17:19quality people that can come in and be
17:21the next executives after you but not
17:23only were they fans of succession
17:25planning they notice that innovation
17:28wise when a person first gets into the
17:30role they have they make a lot of
17:32changes in the first couple years but by
17:34year three or four they've basically
17:35gotten done you know the the agenda they
17:38were going to get done in terms of the
17:39big changes and then it's a little bit
17:41more steady state so they thought
17:42themselves well let's take succession
17:44planning to the next level let's have it
17:46so that you know you know even as you're
17:48two years in you begin thinking about
17:50who your successor is and you start
17:52looking at the candidates and by the
17:53three to four year mark you select a new
17:56CEO that's going to replace you and then
17:58you kind of ushered them in and and they
18:00did that and they got what they wanted
18:01they got a second boost in innovation
18:03because that next person even if they
18:04were selected by the previous CEO was
18:06excited went hard at it but when you add
18:10those two things together it led to them
18:12losing all their top executives because
18:14let's say you know that a new CEO is
18:16going to get selected in six months or
18:18twelve months and you're one of the top
18:20people in the org and you want to be the
18:21CEO well you're not the only one there's
18:24like four or five other people at your
18:25level of skill but the CEO can only
18:28choose one of them and when the CEO does
18:30choose one of them suddenly you're
18:32reporting to somebody that used to be
18:33your peer so if you had a weird
18:35relationship with them you probably want
18:36to leave but even if you didn't and you
18:38were an ambitious person by them being
18:41it basically tells you immediately that
18:42you could not get that promotion for the
18:44next three to four years until the next
18:46succession cycle comes through so a lot
18:47of ambitious people left anyhow so with
18:50all this I was like shoot you guys kind
18:52of understand the problem pretty well
18:53what's happened and that's when they
18:55trotted out all the slides so many
18:57slides like and they walked me through
19:00like five different proposals those five
19:02different retention proposals for like
19:04an hour and even though for
19:05confidentiality reasons these are not
19:07the actual slides there was definitely a
19:09bow tie in there and there's definitely
19:10a circle in a circle there's definitely
19:12a three-dimensional pyramid for some
19:13reason and at the end of listening to
19:16this for over an hour and I'm charging
19:18for the whole thing by the hour is like
19:21like I was like yeah so it seems like
19:25you worked on it pretty hard
19:26so what happened when you tried these
19:29out like oh we didn't try these out yet
19:31it was like hmm okay it's a lot of work
19:34to not try did you just did you just
19:37invent you know did you just come up
19:38with these you just not have time to do
19:39it yet like oh no these five proposals
19:41the oldest ones like 22 months old and
19:43the youngest one is like 10 months old I
19:46was like oh oh well how could you know
19:49about the problem for a long time have
19:50all these potential solutions that
19:52you've thought through and then have
19:53years go by without trying it and then
19:56their tone got super serious they're
19:57like Tom I don't think you understand
19:59this is really serious business if we
20:01execute the wrong retention program we
20:03might lose our top executives and I'm
20:06doing the math I'm losing 70 percent
20:08every three years I was like oh you're
20:10losing them right now so I basically
20:11told them you know what the way that you
20:14actually get things done is you need to
20:16move out of ideas space things need to
20:18happen in real time in space if any of
20:20these colorful polygons mean anything
20:23it's because something in the real world
20:25is going to change so let's just zoom
20:27into that moment where it would change
20:29and let's make it happen right now and
20:31of course they're still staring at me
20:32but I was like whatever we're just going
20:34to do it right now so this is a room of
20:36all the CEOs of all the countries in
20:38Snyder and I was like pick your paper
20:40favorite proposal and they had won their
20:42favorite proposal was something like a
20:43you guys know what a student exchange
20:45program is it's kind of like an
20:46executive exchange program like let's
20:48say you were one of these top executives
20:49you didn't get the CEO role but you're
20:51great you were the CEO of Brazil you're
20:54they have all these operations issues in
20:56Egypt what you do is you do the
20:58executive exchange program you pop over
20:59to Egypt you solve a bunch of their
21:01problems for a couple years you come
21:02back a hero Schneider is healthier you
21:05know you're excited because your careers
21:06better Beaubois but I was like cool well
21:09if that's your favorite program we're
21:10gonna make it happen right now and I
21:11just pointed to somebody in the crowds
21:12like who are you and they're like oh I'm
21:15the CEO of Brazil is like come over here
21:17he came over here and I pull the chair
21:19around sat him in the chair and I was
21:21like okay you're the CEO of Brazil I
21:22pull the table over and this is the desk
21:25in your office and he's like okay I was
21:27like who are you I'm person like stood
21:29up he's the CEO of a different place and
21:31in Schneider and I just like come over
21:32and when you walked over I was like okay
21:35for the purpose of this exercise you're
21:38the CEO of Brazil and we are about your
21:41coming into this office to find out
21:43whether you got the promotion or not and
21:45just act like this is actually happening
21:46to you and luckily these guys are all
21:48the CEOs they've been in this position
21:49before they walked into this office at
21:51some point and got told they got it so
21:53they remember this situation so I was
21:55like just pretend like this is actually
21:57I'm gonna have you knock on the door and
21:58make this happen and I leaned over the
22:00CEO of Brazil and I said well what's
22:02gonna happen is he's not gonna get the
22:04job but you're gonna tell him about this
22:06amazing executive exchange program cool
22:08so knocks on the door opens it they
22:13shake hands they both sit down and the
22:15CEO of Brazil begins talking it's like I
22:17know you've been super eager to find out
22:20the the results of our new CEO selection
22:22process and I I want to let it be known
22:26that you are absolutely one of the most
22:28talented and valuable members of our
22:29team if you ever hear this very bad news
22:34and then he's like but I need to tell
22:37you that she didn't get the promotion
22:38but we have this amazing program and he
22:40starts talking about the executive
22:42exchange program now remember I told
22:43this person to pretend like this is
22:45actually happening to them imagine
22:47you're in that office again
22:48you don't know whether you got it or not
22:49and you're sitting down and somebody's
22:51saying these words to you and like
22:53initially you know face is kind of
22:55he's just nodding and then there's like
22:57some concern on the face like some
22:59furrowing of the brow and then there's
23:01like some tension and the shoulders and
23:03back and like within two minutes of him
23:06talking he like pushes back from the
23:08table stands up and he's like that's
23:11enough I've heard enough look I've been
23:14a professional for over 20 years I've
23:16been an executive for over 12 years I
23:19understand sometimes you get the
23:21sometimes you don't I didn't get the
23:24promotion that's fine no issue but now
23:27what you're telling me is in order to
23:29continue to have a great career tonight
23:30er I need to pull my kids out of school
23:33convince my wife to move to a country we
23:35don't even speak the language work with
23:37the team I've never even met once and
23:39abandon the team that to your credit is
23:41one of the best operational teams in all
23:43of Schneider you are crazy I quit
23:48now remember this is their favorite
23:50retention program like two minutes into
23:54their favorite retention program they
23:55lose the person it's because a lot of
23:57times we're working in this type of
23:59currency right symbolic currency just
24:02filled with conjecture is filled with
24:04guesses it doesn't land as anything real
24:06so part of why we do sketches in the
24:09sprint process part of the reason that
24:10we move toward prototyping toward the
24:13end of the sprint process is we take
24:15these things that are kind of floaty
24:16gessie type things and like concretize
24:19them so they can be directly experienced
24:21when it becomes a sketch I can directly
24:23experience it when it becomes a
24:25prototype and I can sit in that room or
24:26sit in front of that mobile you know
24:28like the mobile app screens or or
24:31interact with a person in a phone
24:33meeting it becomes something that can be
24:35directly experienced and in that process
24:37of directly directly experiencing it
24:39then we can very very quickly go sort
24:41out issues and you see how many people
24:44are in the room right and there was five
24:45of these proposals so I basically just
24:47after demoing that for people I broke
24:49them up into a lot of little sub groups
24:51and each each sub group had one of those
24:54five proposals to go prototype and
24:56within 30 minutes every single one of
24:58the fibra proposals had been tried you
25:01know half a dozen times because there's
25:02enough people in the room to try this
25:03pretty fast and at the end of 30 minutes
25:07people had quit in every proposal now
25:10not in every room of every proposal but
25:12the quit rate was very very high so
25:16understandably on just throw this in
25:17because this is important you know they
25:21were kind of down after all of that
25:23right it's like you're workshopping you
25:25like expect to have all this excitement
25:26afterwards but every single one of the
25:29things like every proposal made people
25:31quit even if you succeeded and not
25:32quitting in your room right next to you
25:34somebody quit using that same proposal
25:36and they were kind of like yeah and as a
25:38facilitator is like how was that
25:39everybody and they're like no no I can
25:44enough can it never happen again that's
25:46like okay I was like okay it seems like
25:49it didn't go the way that you wanted but
25:51was there any energy in the room was
25:53there any energy that was coming back
25:54from from the you know the person you
25:57were testing with and these people you
25:59know snyder is like a european-based con
26:02company so you know they looked at me as
26:04like energy is this like some California
26:05thing it's like no no I'm talking about
26:08something very specific like if you see
26:10somebody and there's having like an
26:12average whatever day and they're kind of
26:13neutral energy you know what that looks
26:15like right they're like oh that's having
26:16a normal day no problem you can see how
26:18different and I'll call that zero energy
26:20and you can see how different that would
26:22be then a day where they have like
26:24really positive energy and the really
26:25enthusiastic engage and they're excited
26:27right like you can tell that difference
26:28right like oh sure sure so I was like
26:31okay so this is zero then this is a lot
26:33of energy but it's positive and you can
26:35see how this might be also different
26:36than a lot of negative energy because
26:38negative energy be like I'm frustrated
26:39I'm set I like you know I feel like not
26:42being heard like that sort of thing like
26:45you can you can tell the difference
26:46between that and neutral - right and
26:48like yeah so this is zero then this is a
26:51lot of energy positive this is a lot of
26:53energy negative so was there with that
26:56definition was there any energy in the
26:58room and then they were like oh yeah
27:01there was a lot of energy a lot negative
27:03energy and I said cool that means you
27:07are this close to the breakthrough right
27:10because nobody changes their behavior
27:11for a neutral outcome right let's say
27:15you were a big fan of whatsapp and then
27:16one day your friend comes to you is like
27:18oh I got this new message gap you have
27:19to install it it does exactly the same
27:21things as whatsapp you'd be like what's
27:24wrong with you right because like who
27:27would change their behavior to get
27:28exactly the same outcome and this is
27:30really important because when we do
27:31these design sprints when we're trying
27:33to go advance our product a lot of times
27:35we think well we put something in front
27:37of the user and they were able to get to
27:38the end they successfully completed the
27:40task this is good but no no that's not
27:43what you that mean you're not close if
27:46you have neutral energy and they
27:48successfully complete the task you
27:49basically end up with that situation
27:51where I built this sophisticated thing
27:53which is equally good to whatsapp nobody
27:55wants to use it right what's way more
27:57important than like whether they got to
27:59the end of the prototypes or whether the
28:00sketches were voted upon and greed upon
28:02by all the people as the greatest sketch
28:04in the world is actually when a customer
28:06experiences it does energy come out and
28:09does the energy get satisfied in an
28:11interesting way so actually having a lot
28:13of positive energy or a lot of
28:15creative energy means you're much closer
28:16to the innovative breakthrough and
28:18here's the reason if you have a lot of
28:20positives you'd say well positive it's
28:22very different than negative no actually
28:23they're the same because the reason is
28:25the following if you have a lot of
28:27positive energy what it means more
28:29fundamentally is that there is something
28:32that you really really care about but
28:34and you are perceiving that that thing
28:35that you care about is improving or the
28:37options are getting wider or like you're
28:39making progress on it right and then if
28:42you have a lot of negative energy then
28:44actually once again at the core it means
28:46that there's something you really care
28:48about is being exposed but in this case
28:50you feel like it's threatened or
28:51diminished or the options are being
28:52reduced but in either case by seeing
28:55that energy come forward it means that
28:57you've discovered something that they
28:58really care about now you know that may
29:01happen but we never even asked about it
29:03so given that they were still sitting
29:04across from these people that were so
29:06disappointed in the proposals I
29:07basically said great recall a moment in
29:12the in the running of the prototype
29:15where you felt like there's a lot a lot
29:16of negative energy and then just ask the
29:18person what was it that you really cared
29:20about that you felt was that threat what
29:22was it that she really cared about that
29:23made you so upset in that moment and you
29:26know this is a workshop moment so they
29:27had a couple minutes they would talk
29:28with folks and they shared back the
29:29answers and it turned out that the
29:31things that these executives cared about
29:33that made them upset were all the same
29:35things you know it was like two or three
29:37things and it was things like oh I want
29:39to feel like I continue to I can
29:41continue to have a really productive and
29:43exciting and creative career Sneijder
29:45number two I want to do right by my team
29:48you know I've spent all this time
29:50investing my tunes I want to do right by
29:51my team whatever I do next I want to do
29:53right by my team it's like those are the
29:54things that they really cared about and
29:56even though they did all these different
29:57greater iterations they had all these
29:58sad points and different prototypes the
30:01underlying things that actually people
30:03cared about were the same across all
30:05these people and I said well now that
30:07you've figured out what that is I want
30:09you to throw away all these prototype
30:11sketches and I want you to just do
30:12something that addresses one or two of
30:15the things that they deeply care about
30:16and they did so and they iterated for
30:19about 45 minutes and then after that
30:21much time they found a program that got
30:24everybody excited to stay at nighter all
30:26right I'm not going to tell you until we
30:29because we may not have enough time to
30:30go do it but I will tell you that it
30:32worked very well now why did we do it
30:35here's the way to explain why to go do
30:37this type of methodology imagine there's
30:39a game and all of you guys have a single
30:42six-sided dice and you can choose the
30:43dice there's no you know GT dice in here
30:45and the game works like this everybody
30:49in the room gets to roll their dice
30:50exactly once and if it comes up a six
30:53you win and I pay you a million dollars
30:55it's a pretty good game so far
30:57nice here's the rest of the rules if it
31:00comes up anything about a six then you
31:02lose and you pay me a million dollars
31:06okay now as people don't like this game
31:08anymore and actually if you were forced
31:12you'd be stressing right oh my gosh we
31:15might win a million but I'm probably
31:16gonna lose a million Dothraki gonna lose
31:18a million dollars and unwittingly this
31:20is the game that we're playing all the
31:21time when we're trying to do new and
31:23interesting things in Oregon in an
31:25organization because really what the the
31:27guys you're about to roll will probably
31:29cost a million dollars because let's say
31:31you put four or five staff on it and
31:33they're on it for six months we'll think
31:34about the salary cost think about the
31:35opportunity cost you add up the actual
31:37numbers it is like a million dollar bet
31:39and if you roll it wrong but we blossom
31:41million bucks if if probably won't work
31:43we're probably gonna lose the million
31:44bucks and this is why people stress so
31:46much about these types of changes that's
31:48the game they're playing the first game
31:49now we're going to improve this game the
31:51update to this game is instead of
31:53getting to roll once you now get to roll
31:5520 times and if it ever comes up a six
31:58you win a million bucks
31:58if it never comes up a six and 20 rolls
32:01you lose and you pay me a million bucks
32:03now that's already a better game right
32:05and like you know people I play this
32:07game and actually that game you're like
32:08oh let's start rolling I'm probably
32:09gonna win a million bucks and then you
32:12don't worry at all until it's roll 17
32:13and you haven't rolled six yeah yeah
32:15good there's only 30 rolls left well are
32:19we gonna make it so okay game 2 is a big
32:23improvement because you're not afraid of
32:24starting anymore now let's talk about
32:27game 3 same roles but now you get to
32:30roll a hundred times well you're just
32:32gonna win a million dollars right in
32:34fact you don't even need to stress about
32:35that much you can run that experiment
32:37thousands of times you'll win a million
32:39dollars every time right like the
32:41likelihood that you will not roll a sick
32:43100 is so close to zero that is
32:46basically it's basically zero for all
32:49practical purposes in business and life
32:51at zero and what I encourage teams to do
32:55and the sprint process definitely helps
32:57you get into that space is move from
32:59playing game 1 to playing game 3 right
33:02in game 1 you're scared to even roll
33:04you're scared to even give it a try
33:06in game 3 it's like we're rolling we're
33:08rolling we're rolling and we were
33:10definitely going to win because it's a
33:11hundred now remember I mentioned at the
33:14very beginning we did a hundred
33:16prototypes of Project loon in four
33:18months with five people and you're like
33:21wow it's so so random it ended up on
33:23such a round number after that months
33:25it's like no that was the actual goal
33:28because the way that we do it is we
33:30basically defined what we're going to do
33:34as a rate of experimentation and we said
33:36there's a prototype number that if we
33:38hit we will have near perfect certainty
33:40that we would have found an outcome that
33:42is interesting even if that outcome says
33:44let's not do the project that's fine
33:46that's near perfect certainty and for us
33:48it was a hundred prototypes after a
33:50hundred prototypes of Project loon we
33:52would know with near perfect certainty
33:53whether this is completely uninteresting
33:56and Google should do no more on it or
33:57whether this is interesting and maybe we
33:59can make some awesome progress and we
34:01should fund it more and you want to
34:02divide that by the available time in
34:04this case we gave ourselves four months
34:06to go do it because it seemed like a
34:08reasonable amount of time to do that
34:09type of experiment and then it gives you
34:11your rate goal so 100 prototypes over
34:13four months is twenty five prototypes
34:16per month we had five people on team so
34:18there's roughly one prototype per person
34:20per week over the course of those four
34:22months and doing that it was actually
34:24very straightforward to play game three
34:26right game three is not some like
34:29complicated you know like witchcraft to
34:32be able to go do you just saw decide you
34:34know at a hundred we're going to know
34:36the answer for sure and how much time do
34:38we need to know the answer within well
34:40now we know the rate and that we know
34:41our staff and we tells you how long you
34:43have for each experiment and of course
34:45if it gets so small where it's like
34:46three minutes for each experiment then
34:48that's a little too tight but like you
34:50know one person doing one experiment per
34:52week seemed totally reasonable and we
34:54basically made that the goal and at the
34:56week we said how's your experiment this
34:57week how's your experiment this week
34:59how's your experiment two people wanted
35:00to team up they need to team up and do
35:02two experiments together right it's
35:03relatively easy management but at the
35:06end of that we were able to get two near
35:07perfect certainty and and you know for
35:10$70,000 a prototype community areas were
35:12get two near perfect certainty and all
35:14that's happening there is we're playing
35:16a game where we make the dice roll so
35:18cheap that you know by the end of a
35:20hundred dice rolls we had only spent
35:22$70,000 of protein my prototyping
35:24materials then it was easy to say let's
35:27go spend more money on this project
35:28let's go do more and this is the sort of
35:31world that can open up as you guys are
35:33able to do more sprints you know get
35:36people into this mindset get into faster
35:38experimentation and depends on how close
35:41you are to the leadership in this
35:44particular case I was the boss of this
35:46org so it was just like oh I've
35:47convinced the leadership because I was
35:49like it's me and I was like this is how
35:52we're gonna do it so that was a very
35:53easy convince but but if you were not
35:56the actual boss of the org then you
35:58basically can within the scope of the
36:01world that you contain like maybe you're
36:02a designer and there's a certain number
36:04of prototypes you're gonna go hold
36:06yourself to you say well yeah I'm the
36:08designer I'll do three per week and I
36:10know that by the time I do 20 doing
36:12three per week I'm gonna be near
36:13perfectly confident that I've done what
36:15I can on this right so even if you're an
36:18individual contributor you can use this
36:19sort of scale but the you know as you do
36:22that it actually becomes easier and
36:23easier to convince folks higher up in
36:25the organization to come along okay
36:27we're gonna do one more thing and then
36:29close this out so yes so every
36:33organization has got a thing that's
36:34called a learning loop and this is a
36:35Fast Company where we in Forbes The Fast
36:38Company I don't know I think it's fast
36:39number yeah Fast Company article about
36:41some work I did with this company that
36:43does like a one-for-one model for social
36:45good they basically sell a water filter
36:47and if you buy their filter plan then it
36:50also gives water to somebody in
36:52developing world separate thing but they
36:55came to me and they basically said hey
36:57we did version 1.0 of our device and it
36:59really took like 15 months to make and
37:01we heard that you're really fast at
37:03prototyping and teaching organizations
37:05to be faster how can we be faster can
37:07could we like do it in like five
37:10and I was like hmm we could do better
37:12than that and and I basically OH before
37:15we get there I was like tell me a little
37:17bit about how people end up buying your
37:21product and you know he kind of walked
37:24through there were some direct online
37:25sales but more than 70% of their sales
37:27were coming through retail chains like
37:29Target and I was like okay well tell me
37:31what happens and Target that it's like
37:32well person's you know rolling through
37:34the aisle and they'll walk by the Brita
37:36filters and purify errors and they're in
37:38the market for a water filter and we'll
37:40be there as well and they might pick up
37:41a couple boxes and look at the sides and
37:44they'll look at ours and you know if
37:46they if our thing is compelling though
37:47you know put it in the put it in their
37:49cart and keep on rolling and I was like
37:51how long does that take like you've seen
37:53this happen right and he's like yeah
37:54yeah it's like 30 seconds you know
37:57people don't sit there and spend 10
37:58minutes deliberating what water filters
38:01you get I was like okay then I have a
38:05way to go prototype the new one that
38:06you're trying to go create and the way
38:09that we did it is we basically analyzed
38:11because there is a something called a
38:14learning loop which dictates the speed
38:15at which you can go learn about your
38:17current product or a new product and it
38:19basically involves four different pieces
38:21which is how fast can you build a new
38:24variation which is different than what
38:25you're trying today how fast can you
38:27test it with customers how fast can you
38:29observe the actuals that come from
38:30customer interaction and then how
38:32quickly can you adjust from those
38:33actuals because sometimes you do a test
38:35you write a report you send it to the
38:37executives they don't read it you know
38:38nothing actually a you know adjusts so
38:41even you can even do all the great stuff
38:43and then it doesn't connect so you need
38:45that so that the loop can close you can
38:47keep going and we quickly analyze their
38:4915-month version of it and everything
38:51was super slow this was super slow and
38:53they were very nervous about this and
38:55once they got the actuals like people
38:57didn't even look at the actuals very
38:58clearly because they had so many
38:59opinions about how you know the customer
39:01should have responded but a lot so kind
39:04of normal stuff but I said well we're
39:06going to do this one differently and
39:08what we did is I basically brought the
39:10designers and a product person all in a
39:13room and this is caught this was called
39:15the home team room and we all put
39:18together like a first version of what
39:19would appear on the side of the box like
39:21if you were to pick up the box at Target
39:23what would people see on the side of
39:24since this is the thing that they're
39:25interacting with for 30 seconds before
39:26they put it in a cart or not right and
39:28so we just like basically mocked up the
39:30design of the side of the box block and
39:32that's the home team and then we had
39:34five away teams and the away teams were
39:36two people each and each one of them had
39:38a tablet and on the tablet it basically
39:40had you know the side of the box and you
39:42could swipe and see the rest of the
39:43stuff on the side of the box right easy
39:45stuff and then they also had a version
39:47of the current water filter with them so
39:49they could show a were the company that
39:50makes this here's a new product that
39:52we're making so we sent the five away
39:55teams to places where that had consumers
39:58similar to the consumers that bought
39:59this product which were like Whole Foods
40:00Apple Store those sorts of places so
40:03five away teams and luckily we have a
40:04bunch of Whole Foods and Apple stores in
40:06the city so we we we sent the five away
40:09teams to those places and as people were
40:10walking out of the store they uh hey do
40:12you have a minute left to show you this
40:13thing and they would begin to give this
40:15feedback and because we had five away
40:18teams and it took about ten minutes on
40:20average for them to go talk to a person
40:21it meant that on average we're getting
40:23new feedback about the thing every two
40:26minutes and the feedback wasn't going
40:28into the ether a lot of times we were
40:29putting the home team on speakerphone or
40:31somebody that was in the away group was
40:33actually texting the home team all
40:35throughout the session so that we were
40:36getting the input and we all had it on a
40:38group messaging thing it was like bloop
40:40bloop bloop or over speakerphone and we
40:42were hearing it and remember the
40:44designers and product people are in this
40:45room so we start modifying it super
40:47quickly oh they don't understand this
40:49that's on the side okay oh this call-out
40:50means nothing to them oh it's getting a
40:52little too messy over here can you make
40:53a version with fewer things and we edit
40:55edit edit and that would happen over the
40:57course of about thirty minutes and
40:58remember if we're getting feedback on
41:00average every two minutes that means 30
41:02minutes of integrating is about fifteen
41:04pieces of feedback that we've done and
41:06by the end of that that 30 minutes we
41:09sent one word to the group text which
41:11said refresh and everybody out in the
41:14fields like blue all the images changed
41:16and then they kept going and we went
41:19from in the first version people were
41:20like what is this product I don't know
41:22if I need this like you know I was happy
41:25to give you some answers but I don't
41:26want it to like the middle where they're
41:28like well this is pretty desirable like
41:30when is this going to be available to
41:31the end where we actually had to go make
41:33a PayPal page to accept pre-orders
41:35because people are like I want to be the
41:37to get this and that happened just over
41:40the course of about three and a half
41:41hours now all that was happening here is
41:44we were optimizing the heck out of the
41:45learning loop because building a new
41:47variation was happening you know we're
41:49hearing this feedback who were do making
41:51these adjustments the text the design to
41:53testing with customers I was happening
41:55every two minutes on average right being
41:58able to observe the actuals it was
41:59basically real-time because it was
42:01common over speakerphone it was like
42:02immediately texted right after the
42:04person said it and then the adjustment
42:06you know like I said was happening at
42:07you know shortly after we heard them and
42:10instead of having a 15-month learning
42:12loop we brought it down to you know 30
42:14minutes roughly depending on how you
42:16want to count it you know five minutes
42:18and in that process you cannot help if
42:20you think about how easy it is to play
42:22game three if rolling a dice takes you
42:24five minutes right well then rolling a
42:27dice a hundred times doesn't take you
42:29very long and by the end of the day
42:30you're selling your product and this is
42:32what I mean by like getting to near
42:33perfect certainty in a short amount of
42:35time you can design the stuff into this
42:37by optimizing your learning loop by
42:39focusing on things like this and this is
42:41like a classic kind of exercise I do
42:44with organizations like we diagram the
42:46current loop and all the guest thongs
42:48and conjectures that they have and the
42:50time when they're actually building and
42:51testing and does it there might be 18
42:53weeks long we remove like a lot of the
42:55conjecture stuff we we improve their
42:57tool chain so they're able to do it
42:59faster and this is a this is going to be
43:01a whirlwind tour because we need to get
43:02to the end I need to be done soon um
43:04because I'm well I'm not over yet I'm
43:07very close to it but this is like a
43:09normal work week for a lot of teams that
43:11I would run blue is all the areas where
43:13you can actually do your work your
43:14design work your coding work that sort
43:16of thing but you'll notice on Tuesday
43:18and Thursday there was three hours each
43:20to be in front of customers and people
43:22hear this and they're like that's crazy
43:24I have such a busy week where the hell
43:26does six hours come from to be with
43:28customers and I was like that six hours
43:31saves us 15 hours of arguing because
43:34what would normally happen is like I
43:36think this well I think that and then
43:38the whole meeting goes like a whole hour
43:39goes and you didn't decide anything but
43:42when you have this sort of format
43:43there's either a test later today or
43:46there is a test tomorrow that is one
43:48feature of this this schedule and what
43:51conversations or people start arguing we
43:53could resolve it very simply which is
43:55like could you prep it for the test
43:57and they're like sure as like could you
43:59prep yours for the test tomorrow it's
44:00like sure well let's use the rest of the
44:02meeting to prep our tests and then we
44:04would just do them and then we do it
44:05right so so like all the stuff that
44:07expands it to these conjectural
44:09arguments these Gasset ons hope you say
44:12those 15 hours you if we actually would
44:14end some of the weeks like what do we do
44:15now it's like will it just go back out
44:17and get some beers now and this is
44:20actually what it looked like remember in
44:22the three hours we would just take a you
44:25know we'd take an uber and we'd we throw
44:27the the dev team and some designers into
44:29the uber and we would over over to the
44:30mall because the mall has the if you do
44:33consumer stuff if you have a b2b thing I
44:35would have a separate thing to go talk
44:37to you about but it's fine like if you
44:39have b2c you stuff the nice thing about
44:41the mall is every type of person is in
44:43the mall and they're broken up by
44:44demographic based on store so if you
44:47want to go meet you know women over 40
44:49then there's like Ann Taylor and if you
44:51want to go meet you know like teenagers
44:54then there's Hot Topic whatever there's
44:56like a store that has that type of
44:58demographic streaming out of it so
45:00whoever you're trying to reach there are
45:01like places in the mall that you can
45:02just go stand and this will work fine
45:04and in this case this is one of our
45:06developers and here's two people that we
45:07met in the food court and they're
45:09basically using the product right there
45:10and we use this so that the user you
45:14know recruiting is very straightforward
45:16and fast and what we're looking for is
45:18not just tasks complete because I think
45:20a lot of times people go in there and
45:22their target is usability which is I put
45:24something together people generally
45:26understood it they got to the end of the
45:27flow they didn't dislike it we're good
45:30and I was like no no that remember like
45:33if you replaced whatsapp with a totally
45:35perfectly functioning but exactly the
45:37same whatsapp nobody's on it right
45:39so like usability and task complete is
45:42not the bar the bar needs to be their
45:45eyes light up there's a moment in the
45:47process that when they're using it
45:48they're like amazed they're excited that
45:50the thing can do something like this you
45:52know because that's you once again when
45:55the eyes are lighting up you're
45:56addressing that energy so remember
45:58neutral versus positive versus negative
45:59energy when their eyes are lighting up
46:01it's because you've addressed that
46:03energy you've taken something that they
46:05and you've given them you know like you
46:08know it's really working for them really
46:09excited about that or you took something
46:11that they're really concerned about and
46:13and you've alleviated that concern those
46:15things will make people's eyes light up
46:16and that's actually what we're looking
46:18for the reason that we do all these
46:19iterations is not to get to just some
46:21usability task complete it's because we
46:23want to do enough you know have a high
46:26enough prototype number that we find
46:27that magic moment and once you go find
46:29that magic moment everything becomes
46:31simpler right the reason that we do like
46:34this fast adaptable sketching
46:36prototyping it enables us finding the
46:39magic expediently and another common
46:43thing remember conjectures versus actual
46:44is another common mixing that really
46:47messes up teams is they start talking
46:49about scalability before you find any
46:50magic all right what's the point of
46:53having something that's usable you know
46:55that people get to the end of the task
46:57every time and scaling it out to a
46:58hundred million people when it provides
47:00no magic in their life when it's just
47:02like the version of whatsapp that does
47:04all the same things now you can have it
47:06reach everybody it's like oh these
47:08servers are ready for infinite scale and
47:10then ten people show up ok well the
47:14difficulty here too is when you start
47:15talking about scalability really early
47:16it actually actively impedes the finding
47:19of magic because you just say well this
47:20is close enough people seem to get to
47:22the end of the task that's a little ship
47:23and it's like well that's not how you
47:26make things that are really powerful in
47:28their lives that's not how you make
47:29things that are memorable and and the
47:30way that I summarize this is you know in
47:32if this is the timeline of experience
47:34with a particular product your product
47:36is really the magic moment and there's
47:38really only two types of features once
47:42you've discovered your magic moment
47:43there's features that get you to the
47:45magic moment faster and amplify the
47:47magic moment when it happens and there's
47:49features that don't and anything that
47:51doesn't get you to the magic moment
47:54faster or amplify the magic moment when
47:56it happens you should cut even though
47:58you feel like oh it's more functionality
47:59and like people might enjoy this
48:02functionality this is the way that you
48:04make terrible products basically because
48:06like actually what it is doing it is
48:08detracting and delaying the point where
48:11they actually experience that magic
48:13right even though I added more maybe I
48:15got my you know my bonus for shipping
48:18these are things that
48:19to make products worst generation after
48:21generation like I don't know who thinks
48:23that iOS 12 is like way better than iOS
48:256 right it's like there was some point
48:28where's like there's not this is not
48:29magic that's getting added now this is
48:31just stuff like and you feel it anyway
48:39so is there time for questions or okay
48:41so I'm going to end then and we're going
48:44to take just a couple questions if you
48:45need to reach me this my social media
48:47thing and also my my online course is
48:50starting next week which is like a
48:51six-week deep dive into applying this
48:54type of stuff to your work okay awesome
48:57any questions in the back perhaps yeah
49:15absolutely so you don't already
49:18predefined what all hundred experiments
49:19will be you just say well hey we're
49:21going to do 100 so we know the rate
49:23which is one experiment per week per
49:26person and there's five of us so there's
49:27five this week so what's your five this
49:29week and the next week we would have
49:31learned some stuff and we adjust next
49:33but we stay to the rate so it's not
49:35totally wild because there is a rate
49:37that grounds you but it's also not
49:39totally predetermined because how can
49:40you predict the outcome of a hundred
49:42experiments and just like a quick
49:43clarification the reason we picked 100
49:45is we were trying to build a global
49:47internet with balloons that's something
49:49that never been done before if you're
49:51doing something that's a little easier
49:52like making a webpage you might pick 20
49:54or 10 that's actually probably fine for
49:57making a webpage so so like you know
49:59whatever it is you pick the number where
50:01you have near perfect certainty that you
50:03would have looked at the options that
50:05you needed to look at and maybe to make
50:07an informational webpage you know 15 is
50:10actually fine like well we looked in 15
50:12different directions we got that
50:13feedback that's fine so you can pick
50:15your prototype number to be anything
50:16it's basically if you were to look at
50:18the additional reward per number of
50:20prototype there's a point where it
50:21asymptotes and as that gets to be pretty
50:24horizontal that's where your prototype
50:25number is so for building a worldwide
50:28internet using balloons it's about a
50:29hundred and then for making a web page
50:31or something it's probably too
50:33team yeah over here in the front right
50:41so okay this is really good and this is
50:45very important for effective sessions a
50:46thing that we fall into all the time is
50:48like the guided tour and the guided tour
50:50is like Oh first you're gonna get to
50:51this page and you're gonna notice this
50:53new thing that we have over here so
50:55click that like okay right and if you do
50:58that you're basically you learn nothing
50:59what you want to do instead is you just
51:01want to put them in a situation it's
51:02like hey you saw so-and-so on the side
51:05of the bus you typed in the URL go
51:07alright and they're like I'm lost and
51:10that's more real right so you will get
51:12much better feedback if you just allow
51:14them to have the situation including all
51:15the fails and the guided tour we do that
51:18because maybe we worked on these seven
51:20screens here we want to make sure they
51:21get to them but it's actually way more
51:23important to make it realistic and find
51:25out that nobody gets to those seven
51:26screens because they're totally confused
51:28about this language or your product
51:30really seems about this and not that so
51:31they don't even think that such screens
51:34would exist like that is actually way
51:36more real because remember the Schneider
51:37Electric example they believe that was
51:39going to drive retention right and they
51:42believed for a long time years they
51:44believed it was going to drive retention
51:45when you put people in the real
51:47situation though made people quit so by
51:49being able to like actually keep people
51:51fully situated and just watching what
51:53unfolds then you're able to get the real
51:55stuff and once they're in the pathway
51:57the real stuff when their eyes light up
51:58it's it must know I'm mistakable and
52:01you're like that is what we were looking
52:03for we ran fifty experiments because we
52:05wanted to find that and now that we
52:07found it then great let's go build a
52:08whole world around it actually there was
52:10one slide that wasn't in there but I'll
52:11just tell you about it like you know you
52:15guys know uber and I remember the first
52:17time that I was using uber I was like
52:19damn this is amazing this is amazing I
52:21think had to tell everybody about it but
52:23actually when you think about let's say
52:25your uber ride is twenty five minutes
52:26long almost every minute about 25
52:29minutes is exactly like being in a cab
52:31the majority of it is like you're
52:32sitting in the back of a car and
52:33somebody's up front driving somewhere
52:35it's like what the hell but actually
52:37uber is really only 15 seconds different
52:40there's ten seconds the beginning where
52:42you pull out your phone instead of like
52:44being in the street like hey hey hey
52:45you're like blue pick me
52:47and there's five seconds at the end
52:48instead of like rustling with your
52:50wallet and being like oh you want 20%
52:52jeez I mean you just drove me somewhere
52:53okay right like arguing with the person
52:56about tip you're like yeah thanks and
52:57you just pop out the door that 10
53:00seconds plus that 5 seconds about 15
53:02seconds built a 50 billion dollar
53:04company because both of those are magic
53:07moments right and every single employee
53:10at uber has something to do with those
53:1115 seconds all the people that write
53:13that routing algorithm so it's more
53:15efficient it's so that when you press
53:16this button you know people are show up
53:17at that first ten seconds all the people
53:19that are working on all the different
53:20payment systems and currencies and so
53:23that when it's time to leave you just
53:24like hey and you pop out the door right
53:27so once you find your your magic moments
53:30and it's typically not hours and hours
53:33is typically not even a minute long
53:34usually your magic moment really is a
53:36moment it's like five seconds long 10
53:38seconds long 30 seconds would be a very
53:40long magic moment but once you find it
53:42that is the main thing everything about
53:46how you're creating your organization
53:48your product needs to be about that
53:50right if uber went into the rest of it's
53:53like well the 23 minutes so the ride is
53:55sitting in this car what if we start
53:56serving uber mints and say no this is
53:59not the thing it's like you don't have
54:01magical mints right that's not the thing
54:04that I'm gonna be okay you gotta you got
54:05to get it here because they got Altoids
54:07now you're like no that's not the reason
54:09that I'm gonna get to the end of the
54:11thing and be like you have to use this
54:12service that's crazy town so you keep on
54:15doing these experiments these cheap
54:16experiments and you guys are from the
54:18area you probably know the original
54:19thing in uber they didn't even have any
54:21cars they just like rented a black car
54:23service for ten weeks and that was you
54:25know the supply side of their
54:26marketplace and the demand side they
54:28like they made a really like lightweight
54:30web app that wasn't even a real app yet
54:31but anyway that was enough to be able
54:33for them to find the magic and once they
54:35found the magic they're like let's build
54:37a whole whole big company around it okay
54:39another question yes talking about how
54:44your experience Yahoo was like meetings
54:46after meetings after meeting yes can you
54:48talk about your top tip for changing
54:49that culture that enabled your teams to
54:52go after and build a hundred prototypes
54:54versus sitting in meetings telling
54:56executives why they should be doing this
54:58or justifying what they're
55:00that's a great question so here's the
55:02fun because remember my DNA is as a
55:05engineer scientist you know designer
55:08like I have those different types of
55:10careers in my history even coming up
55:12into Yahoo and I was so frustrated by
55:15this meeting meeting meeting thing I
55:16created a meeting on Thursday at 2:00
55:20p.m. and the rules of the meeting were
55:22the following anybody can arrive you
55:25know anybody can come to the meeting
55:26there is no like fixed list there's
55:27nobody that's excluded but the only
55:30criteria is if you come you need to have
55:32created something you know recently that
55:36you want to go show everybody that's in
55:38the meeting so only people that have
55:39created things and they're like oh they
55:41need to be in your divisions like nope
55:43just anybody now like I just started
55:45doing this with my own team and it was
55:46like a super fun kind of Thursday thing
55:48and because like creators were showing
55:50things to creators and like oh that's so
55:52cool I like you know yeah you you did
55:54something super cool with that API maybe
55:55I could help you skin that oh that's a
55:57pretty cool design idea maybe I could
55:58like you know beef that up a little bit
56:00and right after that meeting they would
56:01chat about it and I was the boss of some
56:03of the people so I could be like hey
56:04well you guys really like this on
56:06Thursday afternoon why don't you take
56:07Friday off and just work on it so it was
56:09cool but there was a point in time where
56:11some of the folks were like hey I know
56:13these people over in Yahoo TV they're
56:15doing something cool like dthe could
56:17they come and show stuff and I was like
56:18well did they build it in the last
56:20couple weeks and like yeah I was like
56:21they meet the criteria bring them on so
56:24they did and like little by little we
56:26basically created an underground network
56:27of all the people to actually do the
56:29stuff and who are like constantly
56:31experimenting and and trying and and
56:33like we were really supporting each
56:35other in that meeting and me as like the
56:37semi executive at the time because I
56:38like went to the SVP meeting I was on
56:40that staff so I could like tell some
56:42people do stuff I would just when people
56:44had that energy on the Thursday
56:46afternoon I'd be like hey watched the
56:48three of you just like take Friday off
56:49and work on this sometimes they weren't
56:51even in my group but it was fine but
56:54here's the funky thing Monday meet a
56:56Monday morning I had the executive
56:59meeting where I was one of the
57:00executives and the senior vice president
57:01was up there and there's all these other
57:02executives around the table and we're
57:04doing all these like little staff
57:07and told these people oh yes you can
57:09experiment on the Friday a lot of times
57:10they like send me a link
57:11oh the prototypes done you know Sunday
57:14morning or you know whatever and I come
57:15to you know the end of my status update
57:17in going hey and I have this new cool
57:19prototype to show and I showed it to
57:21this executive staff and they're like
57:22what in the world are you doing like
57:24every single week there's new stuff
57:27you're me do you have like thousands of
57:28engineers I was like no and and then
57:31they would get to like this is a funny
57:33one they got to un was like that is so
57:35applicable to our group and I was like
57:36cuz your people built it Hey
57:40I mean with the help of one of my people
57:42but still it doesn't matter it's like
57:44it's like they were creating an
57:46environment that was so constrained
57:48about like whose guest has got to be at
57:50the top that they didn't allow the main
57:53currency which is this experimental
57:55experiential currency to come to the
57:57fore so even if you have limited control
58:00you can make a meeting like that and you
58:03can like be interacting with other
58:05people that have that type of energy and
58:06and even if you don't have the ability
58:08to bring it back to the boardroom and be
58:09like hey here's the prototypes in last
58:11week you will have a richer set of
58:12things to go work with that you when you
58:15go back to your own work people be like
58:17hey how come you're so much more
58:18productive it's like you got new ideas
58:20every week now and they're like real I
58:22can experience them that's super cool
58:23and just like the sprint process will be
58:26like hey let's sketch let's prototype
58:27these have amazing convening and
58:29clarifying power right like when you can
58:32actually produce a thing that can be
58:33experienced as opposed to you know you
58:35should just listen to because I have the
58:36authority to talk because I have the
58:38title then this has clarifying convening
58:41power that talking does not have and the
58:44more that you can like look at your
58:45schedule remember like the optimizing
58:46the learning loop is the same sort of
58:48concept the more that you can optimize
58:49your week so that more of the hours of
58:52the week you can produce things like
58:54that that have that clarifying convening
58:55power then the more you can kind of
58:57shift the organizational dynamic toward
59:00a lot of the stuff that people want to
59:01do in the first place I mean the
59:03executives are usually not mad after
59:04this change happens they're like well
59:06more of the stuff we wanted to do is
59:07happening now how did that happen
59:09and I was like long your people on
59:11Thursday um without any of your guidance
59:14were amazing okay yes
59:21oh you can do it for anything like so
59:25this is the last story basically when I
59:27was in Yahoo I was an executive in the
59:30600 person organization and I i reported
59:32up to this guy who had 9600 people
59:34reporting to him it was a little crazy
59:36and you know we're in this like 90
59:41minute strategy review and about like
59:44the future of our 600 person
59:45organization and what we're going to do
59:46on product is product strategy and I was
59:48like a product design type Lee executive
59:51and 15-minute mark into the meeting he
59:55was like hey well that's really
59:56fascinating like god you couldn't you
59:58guys try something like this and he like
01:00:00you know shares this idea and and I'm
01:00:02over at my laptop and I'm like typing it
01:00:04in and it just looks like I'm taking
01:00:06notes well it's really happening as I'm
01:00:08opening up three messaging windows to
01:00:10like to designers and a prototyper and
01:00:12I'm basically writing the idea in and
01:00:13then 30 minutes later a little less than
01:00:1630 minutes later let's say around the 42
01:00:18minute mark in the meeting I'm like hey
01:00:20Ari remember that idea that she had
01:00:22about half an hour ago he's like yeah of
01:00:23course the blah blah blah blah and I was
01:00:25like cool you want to try it out and I
01:00:28just handed the laptop to him and he and
01:00:30he sits down he's like oh my god when
01:00:33did you make this I was like I didn't
01:00:35know I guess it's been about 28 minutes
01:00:36and and and then it's like he goes in
01:00:39there more and you say this is real data
01:00:42it's like you know my team and then like
01:00:45because there wasn't just a mock-up it
01:00:46was like actual you know funky
01:00:48functioning prototyping code real data
01:00:50and then he like finishes experiencing
01:00:52it and he like scoots back from the
01:00:54laptop he's like I have 9600 people
01:00:57reporting to me and I have no idea how I
01:01:00would do this in three months like how
01:01:02did you do this in 28 minutes and like
01:01:05the and the reason that it relates to
01:01:07like what problems can you prototype for
01:01:09is because I prototyped everything I
01:01:11like change the layout of how I sat like
01:01:13normally executives sit around all the
01:01:15other powerful people but I was like hey
01:01:17why would I sit around all these
01:01:18directors if a director should be a
01:01:21talented person who's talented enough to
01:01:22basically take a little bit of input and
01:01:24go work for a month independently right
01:01:26like why would I sit next to them and
01:01:27talk with them every day
01:01:29there the people are supposed to be more
01:01:30independent so I like change it so that
01:01:32all like my you know executive staff
01:01:34ders and senior managers or whatever
01:01:36they were like in a corridor that I
01:01:38would pass once in the morning and once
01:01:39in the afternoon he didn't need more
01:01:41contact than that because there's senior
01:01:42enough people for that the people I
01:01:44actually sat around I did have a corner
01:01:45office or whatever it's fine
01:01:47so a corner office but right around me
01:01:49was a ring of prototypers and beyond
01:01:51that ring was a ring of designers and
01:01:52researchers and what I would do is I
01:01:54would you know in physical life I would
01:01:57stand up and be like hey we got a quick
01:01:59prototype who's available and then like
01:02:01little groundhog some people would pop
01:02:03up have you okay Olivia Jeremy so and so
01:02:07and we like super quickly form the team
01:02:09and then they'd all pop back down and do
01:02:11it and what I was doing there was
01:02:12virtually doing the same thing in these
01:02:14instant messaging windows now that
01:02:17doesn't happen accidentally each one of
01:02:18those things like those optimizations of
01:02:20physical space of organizational design
01:02:23of the fact that they even had free time
01:02:25because I implemented you know in
01:02:27parallel to Google doing 20% time I made
01:02:30up a thing called 15% time which is
01:02:31actually a little bit more focused which
01:02:32is you can use 15% of your time to go
01:02:35and study anything that you feel like
01:02:36would be relevant for your work and you
01:02:38know you have a little bit of budget
01:02:39like learning budget for it but the only
01:02:41other rule about 15% time is if there's
01:02:43something really critical that I need
01:02:45your time on to prototype then you'll
01:02:47drop your 15% time for that week and
01:02:49give it to me to do that and I didn't
01:02:50abuse it right so like people were like
01:02:52oh this is real real and because of it
01:02:54I'd be like you have 15% times yes yes
01:02:56yes and it's like boom the team would
01:02:58form in like 14 seconds they get the
01:03:01brief we would do it it would get done
01:03:03and we kept inter iterating on a
01:03:05prototype systems - it's like how do we
01:03:07make it so it's really fast to access
01:03:08real data in a way that you could still
01:03:10you know have privacy on like an
01:03:12internal cloud type thing you know how
01:03:14do you make it so that we have lots of
01:03:16quick ways to iterate design oh we we
01:03:17use this greasemonkey
01:03:18injection thing and we created a little
01:03:20framework around the types of stuff that
01:03:22we could overlay dadada and like each
01:03:24one of these things was basically a
01:03:26target of multiple prototypes that led
01:03:29to faster you know a reduction of the
01:03:31learning loop and whether that is
01:03:32physical space or the digital like
01:03:34prototyping space or just the ability to
01:03:37go grab people's time
01:03:38within seconds those were actually the
01:03:40culmination of a series of like 15 20
01:03:43prototypes each and when you add all
01:03:45those together that leads to oh your
01:03:47idea is this okay
01:03:48okay okay and then it's like try it out
01:03:51and that's what I mean it's like it can
01:03:53actually be applied to anything that you
01:03:55would like to do faster and better and
01:03:58then you can you know you can point it
01:03:59at self-driving cars and balloons and
01:04:01stuff - yeah all right I think thank you
