00:00this is a real privilege for me we are
00:02here today with dr. Verner bogles he is
00:05the CTO of Amazon and of course has a
00:09lot of really exciting experience with
00:12that and so we're gonna be talking to
00:14him today about his experience with
00:15Amazon about his experience with
00:17startups and about lots of technical
00:20topics as well that will be relevant to
00:22many of us so thank you all and let's
00:24give it up for dr. Boles thank you okay
00:32so we're gonna be talking of course
00:34about Amazon today and about your role
00:37there but I'd like to start with a
00:38little bit of background would you mind
00:40telling us about your career before you
00:43started at Amazon and sort of what
00:44brought you to that point how much time
00:49so I'm sorry I was an academic before I
00:53joined Amazon so I'd been research
00:56scientist at Cornell for 10 years
00:58building very large scale distributed
01:00systems and NS as in common in in
01:04American academia you to be motivated to
01:08do startups on the side so we did two
01:11startups on the side one of them that
01:14already existed when I joined them and
01:16they got sold off and were successful to
01:18a company called Stratos I don't know if
01:20anybody remembers that this before your
01:22time yeah and an another company that
01:26actually failed so we had both
01:29experiences that was great kind of so
01:33before that so I'm not the typical sort
01:37of computer scientist there wasn't until
01:4028 when I decided to actually to go back
01:42to school I worked in hospitals before
01:44that a radiotherapy in the Dutch Cancer
01:47Research Institute doing radiotherapy in
01:49cancer patients and I don't know what I
01:53realized that I really hated all these
01:55people dying around me and so it's I
01:58decided to go do something that had no
02:00humans involved whatsoever so computer
02:03science seemed like a really good thing
02:04to go into and this is me mid mid 80s so
02:08you know the computer scientists know
02:10where where it is now - - today but
02:13turn out I had a humans back then I had
02:18put it like that yeah so turn out I had
02:23a gift for it and I didn't know that up
02:25front so but and from there I wanted to
02:29research because I worked as for the
02:30kind set kind of things that I really
02:32was interested in comma pitch the
02:34working poor to go for a number of years
02:36in a Research Institute and then was
02:39invited to come to Cornell and then at
02:41one moment it so what I did do around
02:44that time also when I was still at
02:45Cornell was actually either consult for
02:47large companies like Nike HP's and the
02:50sons and whatever of this world and also
02:53often gave talks and so at one moment
02:55Amazon invites me to come give talk
02:57about sort of some of the material I was
02:58working on and I think like really
03:01really have to go what is this this is a
03:03book shop you know it's a web server in
03:06a database how hard can it be
03:07yeah one glimpse in that kitchen however
03:11and I realized that this is a a massive
03:13technology operation is not a retailer
03:15it's a technology company and and
03:18operating at the scale that I'd never
03:21definitely not at all the companies that
03:23are consulted for and so the challenges
03:26that they were faced with from from a
03:28distributed systems researcher
03:29perspective or amazing and so I didn't
03:33need to think very hard when they
03:34offered the new job well that's
03:37incredible so that's interesting
03:39do you feel like that was a change like
03:41today sort of like the interesting
03:44distributed systems problems are kind of
03:45like huge companies and maybe before
03:47with academia or or was that just that
03:50it's the way yeah I'm gonna go I think
03:52that still is the case I think I think
03:54most distribution researchers have
03:57become more aware sort of the kind of
03:59scale that these very large companies
04:01need to operate or not even these very
04:02large companies I think if you think
04:04about you know any successful internet
04:07company or digital-only company needs to
04:10operate at the scale that is
04:12you know 2004 and I joined Amazon I mean
04:15many of you if you're gonna be
04:17successful will be easily operating at
04:18that particular scale that Amazon was at
04:20in 2004 but there was no body of work
04:24that you could really be relying on
04:27the chatty with a lot of effort went
04:30into basically keeping the lights on
04:32long things that iCloud and other
04:33technologies now gives you and so Amazon
04:40that gotten it 2004 to scale purely by
04:44being practical and it's not that you
04:47there was a book or so that you could
04:49read how to build a scalable
04:50organization or a scalable company you
04:53and so everything I was almost sort of
04:55Amaya is five to ten years ahead of the
04:58curve both in terms of the usage of
05:00technology of the development of
05:01technology but also at the operation of
05:04scale and especially if you want to be a
05:07fast-moving company it's not like you
05:08could look at traditional enterprises
05:11because they are all sort of a might
05:12suffer form innovators dilemma and all
05:15the things are really slow at one moment
05:17once they become sixty success or how to
05:19build a company that needs to continue
05:20to move fast it's a whole different
05:23story and and so they may be you may be
05:26doing things that you may be making
05:30business trade-offs for example the
05:33creation of technical debt or the
05:35allowing duplication to happen and
05:37things like that that would be out of
05:38the question in the traditional
05:40enterprise because efficiency is their
05:42their main goal at Amazon moving fast
05:45and innovating fast and having them very
05:48long pipeline of experiments that's the
05:51more and most important things and
05:52you're willing to allow duplication to
05:55happen you allow to create technical
05:57depth as long as you know that you have
05:58to pay it off so the many trade-offs
06:00that I must one has been willing to make
06:02over time that you can't find in the
06:05traditional MBA book so most of these
06:09things I'm also had to develop for
06:10themselves and whether it's technology
06:11or whether it's processes or business
06:14processes that helps of course that with
06:17Jeff basis at the helm you have someone
06:19who's a true visionary and truly
06:22understands sort of how the next modern
06:25world will look like or should look like
06:28Thanks so a couple of years after you
06:31joined you were appointed the CTO of
06:37someone asked you do you want to be the
06:39CTO guys he's not gonna ask next year is
06:41he no so alpha mutant who was that
06:46company by the way didn't really he
06:49really wanted to be developer and they
06:51had been looking for a green replacement
06:52for him and as I said early sometimes
06:55you have gifts that you don't know about
06:58until you start doing them and so I
07:01think I joined in September and freaking
07:03January I became the CTO well so tell me
07:07what that was like at Amazon and then
07:10sort of what that role has shifted into
07:12as we get told today yeah so if you if
07:15enactment so and one of the things that
07:17Amazon sort of one of the reasons for
07:20hiring me and actually a few of my
07:22abroad actually was it about five of my
07:25students former students with me and I
07:27came there was really two in straight
07:29more sort of academic rigor into the way
07:32that we wanted to approach scaling where
07:36Amazon had become really good at scale
07:38but the challenge was that we wanted to
07:40do orders of magnitude more in in
07:43essence we go from order of magnitude
07:45growth you have to revisit almost
07:46everything that you do whether it's your
07:48processes but also your technologies and
07:50so to really be on a solid footing to do
07:55sort of the next one or two orders of
07:57magnitude of growth that were coming we
07:59needed to sort of insert way more rigor
08:01into our thinking and it was for example
08:04around performance how do you measure
08:05what kind of infrastructure do we need
08:07for measurements how do you if you want
08:09to be a truly data driven company that
08:11sort of makes data different decisions
08:14you need to first of all faster data but
08:16you also need to have a culture around
08:18how do you measure or and how do you
08:20interpret those those measurements and I
08:22mean a medium latency of let's say one
08:24point two seconds of your webpage to
08:26your customers doesn't say anything just
08:29says that 50% of your customs have a
08:31worse experience you need to know how
08:33much worse and then from an engineering
08:36it so for example the 99th percentile or
08:38the 99.9% of it's way more important
08:40often in those cases and then how do you
08:43create an engineering discipline that
08:44has control over that 99 percentile so
08:47cheeky can actually start pulling it in
08:48if that's what you want to do and then
08:50be able to associate that with business
08:53decisions for example I mean the
08:58commonality is that if you improve
09:00latencies of your pages conversion goes
09:02up and so the question then is how much
09:06are you willing to spend to drive that
09:08conversion of you need to have more
09:09capacity you need to do more engineering
09:11and so when when you have a diminishing
09:14return on sort of the investments that
09:17you have to make that Icahn control so
09:19all year we focused really on
09:21performance and measurement the things
09:23that also bit in progress already when I
09:25joined live the whole year on focusing
09:28on single points of failure really I
09:31think in 2004 we were pretty good in
09:32terms of reliability there were rules
09:36around so we had to use data centers you
09:39have to replicate whatever he did over
09:40these see data centers you be able to
09:42lose a data center customer she not be
09:44impacted she could lose two data centers
09:46customer can be impacted terms of
09:47latency but not in functionality all of
09:50those rules were there and I think we
09:52were pretty good at it until at one
09:54moment we decided so why don't we pull
09:55the plug out of one of these data
09:57centers and see what happens and the
10:00first time I redid this was not like we
10:04made it as a surprise now we gave
10:05everybody a heads up that this is what
10:07we're going to do we're not really
10:08propylene to plug let me pull you pull
10:10the network basically yeah so data
10:12center gets gets isolated turns out all
10:15these things that look great on paper
10:16didn't really look that great in
10:18practice although the first time when
10:21you do it now lots of manual processes
10:23steel manual database failover swings
10:26like that this is all pre AWS and so
10:29that's sort of a whole year focusing on
10:32that and so by the time you do the third
10:34or the fourth of these what we call game
10:35days you actually get to a point where
10:38these things will come really well
10:39automated and that you can actually do
10:40them without human intervention one
10:43biggest surprise when we did the first
10:44time was and then you bring the other
10:46data center back online and then
10:49certainly that data center has to sync
10:50with the other data centers get man
10:52that's a nightmare that was a scenario
10:53that nobody had thought about and so but
10:56until you do these things you don't know
10:58them and especially at scale so
11:02and then we did here on efficiency with
11:03it's a lot of things so mostly the world
11:05of a CTO is to drive really big programs
11:07I would think about sort of what does
11:09the technology that we need for the
11:10future to be able to be on solid footing
11:12as a business or in the case of Amazon
11:14what are the kind of unique technologies
11:16that we've developed that may be able to
11:19turn into a business then things change
11:22at all moment because in early 2000 we
11:26had opened up the cathode we put an API
11:28on the catalog that was popular in those
11:31days you just put an API on something
11:32see what innovation happens and it turns
11:35out lots of great companies were being
11:37built so you would have access to the
11:39catalog search shopping cart and if you
11:42want more from snails anyway so lots of
11:45new e-commerce completely being built
11:47based on the Amazon catalog they would
11:48do comparison shopping and have
11:50wonderful new you eyes towards the whole
11:53site really great stuff being built but
11:55as soon as one of those companies became
11:57successful they all started to stutter
11:59in the execution they all needed to get
12:02investment not because they needed to
12:04operate do the business operations the
12:06only to start buying hardware and they
12:08need to hire IT people and so all these
12:10investment was in in essence in indirect
12:12funding of the American hardware
12:14industry yeah because now and that was
12:16where all them so all these companies
12:18had hard time doing that they had hard
12:20time getting the investment at the hard
12:21time getting the hardware because you
12:23know you might order 50 service it's not
12:28like driving up to Costco and actually
12:29getting them in your shopping cart know
12:31it's and then hire IT people and things
12:34so most of those companies that were
12:36initially successful around the Amazon
12:38Catalog all failed and they didn't fail
12:41because they didn't have great ideas
12:43they fail because they couldn't get your
12:45IT stuff ready or get the money to
12:48actually do to do to that and I think
12:50one of the things that has shifted with
12:52with AWS and cloud is that the
12:55investment that you get no longer goes
12:57in large numbers to hardware companies
13:00it goes into hiring more people that are
13:03relevant for your company and actually
13:04build your products better so so then we
13:10become Toby we started eight AWS and we
13:12could talk more about that later
13:14but when you become a technology
13:16provider your world assisity all changes
13:18I think there's sort of a voting bloc
13:21postpones about that is in my eyes is
13:23sort of four different types of safest
13:25folks ask any last come every last
13:27company of office it will be different
13:29but I think there's sort of four large
13:30categories one ISM in enterprises
13:32they're often sort of the the
13:35infrastructure manager yeah they will
13:37report to the CIO managers large amount
13:40of infrastructure and then there is sort
13:43of the second world or think is that of
13:46what you often see in younger businesses
13:48where the sitios did the technical
13:50co-founder the technical did one with
13:52the technical vision I do think that
13:55that the world is a dangerous one
13:57because there's so many other things
13:59that actually fall into that bucket in
14:00terms of VP of engineering teams and
14:03things like that the teachers may not
14:05necessarily be really good at but to
14:06talk about that later then there is the
14:09role of a big finger that's really where
14:12you're sort of driving innovation if you
14:14look at companies like 18t and loosened
14:17and others the all that ceos has had an
14:19office of the CTO that will be really
14:22building next-generation technologies or
14:24experimentation there and then it's the
14:26role of the external facing
14:27technologists if you're a technology
14:28provider to other companies there's a
14:31role of foreign executives to actually
14:33interact at a tech deep technical level
14:35with your customers to understand how
14:38are you using my products yeah and and
14:41looking for debating bigger patterns
14:42among your customers what are the bigger
14:44pain points that they still have not
14:47only with our technology but just in
14:48general you know if often jokingly said
14:51so we're in the we're in the business of
14:54pain management AWS yeah so what kind of
14:57pain do customers still have that they
14:59have to do work on that actually don't
15:01contribute to the products that they
15:03want to build and so that's sort of on
15:06one hand helping customers understand
15:08things sort of being a bit of the
15:11evangelist for the whole notion of of
15:15cloud which definitely they say 10 years
15:17ago was much more important that it is
15:19now but then most importantly take the
15:22feedback from your customers back and
15:23start thinking about sort of what new
15:25features or products or you have to do
15:28or what kind of processes we need to
15:29change to make sure your we serve our
15:31customers better and so it's a much more
15:34customer focused position than a
15:37technology focused one great um I'm
15:41curious you were talking earlier about
15:43you know especially in the early days
15:44there were lots of tasks like you know
15:47the game day you were talking about to
15:49make sure that the infrastructure was up
15:50for scale I'm curious if you've noticed
15:54in your time and Amazon as it as it's
15:56grown in the engineering culture has
15:57grown so much are there any like tips or
16:00tricks you've seen on how to grow an
16:02engineering culture and keep it moving
16:04quickly you know keep it effective well
16:09given that Amazon is in reality my first
16:12real job for a long time I thought that
16:14the rest of the world was just like
16:16Amazon and it's not there's a very
16:18unique culture at Amazon I think that
16:20works really well for fast-moving
16:25companies and that is to have to build
16:27your teams as independent as possible to
16:30remove as much hierarchy and structure
16:32from your organization as that is both
16:33hierarchy in my eyes is totally
16:36unnatural I mean you need to have some
16:38form of IQ maybe for reporting or things
16:40like that or for some management pieces
16:42but you know if you look at nature
16:45there's a head monkey and all the other
16:47monkey's there are no lieutenant monkeys
16:50yeah and so no I mean you may have
16:54multiple groups of this we've always
16:56there has monkey but there is and it
16:58scales really well I mean if you look at
16:59ants and also to all the patterns in
17:01nature if you're able to build the self
17:04or self-organizing organization which
17:06means basically that you hire people
17:09that really want to be independent not
17:11want to be followers now that actually
17:13want to own a piece of the product and
17:16actually take ownership on that I think
17:19that's extremely important when you're a
17:21younger business yeah where you don't
17:23need followers you don't need just
17:25coders you need people that want to have
17:27ownership over the piece that you want
17:29and I think look for that there is
17:32something at Amazon that we call the
17:33leadership principles there's 14 of them
17:35and basically that's all around culture
17:37and so it's a custom obsession ownership
17:40dive deep and so this
17:4114 of them check them out if you type
17:45them in your favorite search engine
17:46you'll find them if a deep explanation
17:48of all of them but that's really dries
17:52what the Amazon culture is and so when
17:54we hire let's say we bring in someone
17:56for an interview you probably really
17:58figured out whether he or she is
18:00including of engineer through the phone
18:02interviews and things like that you
18:04interview all about culture it's really
18:07do you have a real good culture fit
18:08because there's nothing worse than a bad
18:10hire when it comes to sort of the
18:11culture kind of a van because that sort
18:13of disrupt your small teams tremendously
18:16so at Amazon we've some believe him into
18:19two in fairly small team so 10 to 12
18:22people if you because we tend to tell
18:25people you're still everybody knows of
18:28each other what you're doing yeah and I
18:30mean by the way if you won't squirm
18:31if you want to stand in the corridor in
18:33the morning with 25 people that doesn't
18:35work yeah and so you're really focusing
18:37on small teams and making sure that they
18:40have strong ownership over the pieces
18:42that they need to do and so yeah one of
18:46the building teams that are have you
18:49know our independent thinkers have
18:51control want to have control over their
18:54I think it's important I think one of
18:56the challenges is when when you grow
18:58with a young business often I mean the
19:01role of a city or region is probably
19:03that it does everything that has to do
19:05with tech let's basically think about
19:07the technology but also manage the teams
19:09now I think managing teams is a
19:12completely different discipline now
19:15there's a major difference between what
19:17I would call VP of engineering and the
19:18CTO VP of Engineering every morning
19:21wakes up thinking do I have the absolute
19:22best team is my team in the best
19:25position to deliver what they need to
19:28it's a people person where I'm really
19:31thinking about making sure that your
19:33engineers in the right position to
19:35deliver the technology that in the
19:36products that they need to deliver CTO
19:38things about technology are we building
19:40the right tech detect technologies are
19:42we do we have the right tools do we have
19:44all of these kind of things so two very
19:47different jobs in my eyes and that of a
19:49VP of engineering so it I think it's a
19:54you guys know my calab rants I think you
19:57see well-known as sort of the bloggers
19:58right these pieces if you're a manager
20:01if you think about managing think in
20:03terms of technology management he's
20:04probably sort of the most well known
20:07model writer that that really were
20:09things deeply about how do you make
20:11teams effective and that's very
20:14different and he but he's not a CTO he's
20:16a VP of engineering and so I think that
20:19sort of really embodies that can evoke
20:21box about it as well so really check out
20:24his stuff Michael lob rants is his his
20:27rants and responds we suppose I think is
20:30the blog so really check this example
20:33great so you mentioned earlier that in
20:37sort of the early days of Amazon you had
20:38those those companies integrating with
20:40you and they were failing for the
20:41infrastructure reasons and then it's
20:44always that where AWS came from or tell
20:47me how did the idea for a do us happen
20:49how did that get get popular internally
20:51well so if we so internally we're gems
20:55on the retailer and and AWS as well now
20:58I'm more or less organized in such
21:00independent teams that most of those
21:01teams look like startups as well yeah
21:04and all right there they have complete
21:06control over their own destination and
21:07believe me the recommendation engine for
21:10boots is very different than the
21:12recommendation engine for shoes yeah and
21:14so there's a different team that
21:16different goals have their own
21:18innovation agenda all developed by the
21:21team themselves by the by the by the way
21:23so what we'd gone through a number of
21:26architectural changes so end of the 90s
21:29and so Amazon's goal was to get big fast
21:32and you were basically violating all
21:35sort of architectural principles to get
21:38to that point that meant it had ended up
21:41with some sort of monolith and a massive
21:42database infrastructure in the backend
21:44that was very brittle and basically
21:47yeah from an architectural point of view
21:50so but remember that nobody had done
21:53this before so they can't be blamed it's
21:55not like you're violating the textbook
21:56there was no textbook and so then Amazon
22:00moves to what's called a
22:01service-oriented architecture that was
22:03not a word that they existed around that
22:05time either so carving
22:07of pieces of the monolith bringing
22:08together the data that they operated on
22:10putting an API on it and then ever
22:12networked infrastructure that can call
22:13these public health services and and
22:16basically that took two to three years
22:19to get that done mostly because there
22:21was again now history around how do you
22:23run services and so basically by what we
22:26were doing we were swinging the pendulum
22:27all the way to the other end so in this
22:30one with the monolith the whole set of
22:32databases were a shared resource and
22:34basically the constraint
22:36yeah because shared resources are the
22:38thing that sort of make you move slower
22:40what we saw what are the reasons for
22:43changing architecture was that we saw
22:45that the the effectiveness of engineers
22:47was dropping off basically deployments
22:49were coming slower new new features were
22:51getting slower all these kind of things
22:53on why because the backend databases
22:56were managed by a group of DBAs yeah and
22:58so database administrators the DBA cabal
23:00and to make any any schema changes she
23:04had to go through them and these guys
23:06were responsible for these databases so
23:08they were as conservative as that you
23:10can imagine yeah because the reliability
23:14of the site dependent on them so that's
23:16so all that we sort of our effectiveness
23:19in innovation was really dropping off
23:20and so moving to this new architecture
23:23we traded all these themes that weren't
23:25going to be independent at the code they
23:27would have their own data stores there's
23:29no longer a centralized data store is no
23:31longer single and centralized resources
23:33so all great except for that we've made
23:37some mistakes there so what one of them
23:39gets to see years later we carved off
23:42all these pieces the monolith is largely
23:43gone all services since I mean that's a
23:47side story we've made a mistake in the
23:50three very large data sets customers
23:53that's the catalog and orders and we
23:56basically taken all the code that
23:58operated on that particular data as
24:00being one service within two years that
24:03was as big as that the monolith was yeah
24:05so we and so we had to decompose into
24:07much smaller functional building blocks
24:09for example if you talk about the item
24:12master so the customer master service
24:15the basic consisted of ten different
24:17smaller services each with their own
24:20reliability capabilities and so for
24:23example if there would be a login
24:24service in there that basically is
24:25called on every webpage but in that same
24:28piece of software also said the address
24:30book service that only is needed really
24:32checkout and so but that whole thing
24:34needed to scale at the scale of the
24:36login service because that was the one
24:37that requires most scale so decomposing
24:40that into different building blocks that
24:42each after unique scaling and
24:44reliability or maybe security
24:45requirements made that we went to what
24:48now is called the micro services
24:50architecture okay so move forward all
24:53these teams have small their own their
24:55piece and then we see the effectiveness
24:59of these teams tapering off again
25:01because what happens that each of those
25:04services now becomes I needs to scale
25:06they all need to work on the same thing
25:08you only to manage heart they only to
25:10manage the load balancers Hardware
25:11databases they're not each of the teams
25:13now is responsible for actually
25:15replicating the database of a seed event
25:17of a three different day datacenter so
25:19actually all these teams you see the
25:21communication between these teams and a
25:23networking infrastructure increasing was
25:26always new tick is being created all
25:27this work being done but they're not
25:29working on innovation they're not
25:30working on keeping the experimental
25:32pipeline going so it was another red
25:34flag and what I realize was that all
25:36these teams are doing the same thing
25:37because it's wrong dependent to that
25:39side that now he had to manage your own
25:41database so dropping actually databases
25:44into a shared services platform dropping
25:46storage dropping networking in the end
25:48and for example also dropping making it
25:51an environment where he would where
25:54servers no longer a servers but became
25:56virtual and you could actually manage
25:58them so we I think we were really good
26:01in those days in terms of hardware
26:03provisioning so as an engineer we come
26:06to a portal you would fill in what
26:07unique you need 10 more Linux servers of
26:09this particular size and now or two
26:11later you would have them and if you can
26:15imagine at the end of the year to
26:16exchange giving the spikes of spikes are
26:19four times as high so you would have a
26:20need teams need to know a lot more
26:22hardware if you then would go talk to me
26:24in January and tell them why ain't you
26:26releasing any capacity well you know
26:28there's this project coming up in March
26:30that we fought and we just hang on to it
26:33and apparently just an hour to get
26:35capacity is not enough is not a good
26:37enough incentive to sort of release
26:40stuff again so we needed to go to a
26:42mobile first of all where you could take
26:43engineers out of the loop and we could
26:46have business rules that decide how much
26:48capacity is applied to something so
26:49things needed together when the server's
26:52needed to become virtual they needed to
26:53get an API fix that so we built these
26:55things for ourselves internally what we
26:57look at these companies are failing on
26:59the outside that required scale we look
27:01like could we solve this for ourselves
27:03then starting to think to say some of
27:05those technologies or not the
27:07technologies themselves but think about
27:09how had we had built them for internal
27:11and then rebuild them on on the outside
27:13and so the first one we launched was I
27:16must want to see the simple storage
27:18service in in spring of 2006 storage for
27:23the internet that's what we called it
27:24and so really in the early days thinking
27:27that this would be targeted at but I now
27:29would call internet scale companies I
27:31don't like to be used to it startups
27:33because I think there's many other types
27:35of businesses that eventually he quiet
27:37internet scale and that this really sort
27:39of where we were pushing for storage
27:43became the first firm we launched ec2 in
27:44the fall of that that year same kind of
27:47mobile and interfaces that we used in
27:49the internally where suddenly compute
27:52capability became programmable within a
27:57few months enterprises to figured out
28:00that this was a way to go to deal for
28:01them as well and we know what the
28:04stories in since then so I am curious I
28:07mean at the time as you were developing
28:09this before it was released did Amazon
28:11have a sense I mean it's it's a dominant
28:13product now in the world it's changed
28:15the way developers work at small and
28:18large companies did you really have that
28:20belief coming in this is gonna change
28:22the landscape or was it just sort of
28:23like an iterate and every year more
28:25people are using it and it just slowly
28:26grew into what it is today I I think one
28:29of the things that that we were
28:31I won't take caught off-guard by is that
28:33how fast it grew that was definitely did
28:37we know was going to be really big yes
28:38that was the bet that we were making so
28:41an Amazon just two types of innovation
28:43that happens one of them is that each
28:46by themselves is in charge of the
28:47innovation world map for the coming year
28:48that so teams do that for themselves so
28:51as I said the recommendation engine for
28:52shoes who has as a metric to reduce the
28:56number of returns yes so can you
28:59recommend so if this this fallacy knows
29:02nine fitted you really well if you want
29:04to buy this Jimmy Choos maybe doing
29:06eight and a half yeah this seems silly
29:08but it's you know doing returns it's a
29:12it's a tax it's a customer unfriendly
29:14conferring so the more you can do that
29:16so those guys our in charge of building
29:19a world map for the coming year how to
29:20get new data sources how to maybe
29:22engaged if your customers differently
29:23and that's a that's something they make
29:26up themselves then there's no way
29:29there's no top-down saying and down
29:30shelf duties so that's one level of
29:32innovation the other level is the one
29:34that requires significant capital
29:35investment and that was clearly
29:37something that AWS falls into that
29:40particular category things like the
29:41Kindle Amazon Prime others are all
29:44things that to do that we needed to do
29:47significant capital investments then I'm
29:49also we have to rule that if you do that
29:51if it's going to be successful it needs
29:54to be successful in a way that has
29:56significant impact on the Amazon balance
29:58sheet you know it's not that we're
30:00interested in another let's say hard
30:02that sounds quite a bit another fifty
30:04million dollar opportunity now where if
30:06you make these capital investments you
30:08need to look for really big
30:09opportunities so we knew that if AWS was
30:12going to be successful there was going
30:13to be massively successful in a way that
30:17sort of we'd never seen before we knew
30:20that we had to develop lots of new
30:21processes and technologies and
30:23techniques and whatever for all of this
30:25if it was going to be successful now
30:28remember when we were developing SC we
30:31wrote a number on the board that before
30:34o and within six months were probably
30:36let's build this for this number of
30:38objects in the storage engine and then
30:41just for the heck of it we put two
30:43orders of magnitudes behind it that blew
30:46through that in the first three months
30:47yes oh now it suddenly it turns out that
30:52sort of that we some decisions we made
30:56from a technology point of view were
30:58really smart we knew that if you would
31:00with every order of magnitude of growth
31:02you probably need to revisit your
31:04architectures that you have so you need
31:06to build software that needs to be able
31:08to evolve over time evolved and for
31:11example take a storage engine if you go
31:12to the next release internally of your
31:14software you can't just copy the massive
31:17amount of petabytes that you have on the
31:19other storage disks to do this now so
31:21you have to live with multiple
31:22architectures at the same time multiple
31:24versions all these kind of things
31:26fortunately and with all the lessons
31:29from amazon.com we it's actually taking
31:31the right steps there there's been some
31:35challenges over time in that that sense
31:37and but I think also just like with any
31:41other company it's not only the tech
31:43that you have to to to scale when I
31:48think early days at AWS I believe we
31:52said something like we don't need any
31:53salespeople this stuff will sell itself
31:55yeah this is all self-service well turns
31:59out that's not the case yeah it turns
32:02out you need solution architect you need
32:04technical account managers you need a
32:06customer support you need all these kind
32:08of things to build around your product
32:10they have nothing to do with the tech
32:12itself but you cannot become successful
32:15company read without that I was looking
32:19just right before this interview
32:20actually and I was looking at sort of
32:22the AWS directory page and there's I
32:24don't know I think it was like a hundred
32:25and thirty services and probably that
32:28number will change you know by the time
32:29we finish this conversation I'm curious
32:33how do you as an organization determine
32:35like how do you decide to build
32:37something new is there like is it a
32:39top-down process is it just an
32:41individual team what does that look like
32:42to launch something new I think there's
32:44a workspace I think there is I mean we
32:48expect all of our teams as they are now
32:50to being very close contact with our
32:52customers so about 95% of features and
32:55services that we deliver are in response
32:57to direct requests from our customers
32:59and so that's and that's a massive
33:02influx of course the early early
33:04services that we built you
33:06could almost think about what they
33:07should be no I mean what is the basic IT
33:10infrastructure storage compute databases
33:13network security yeah I mean that that's
33:15you didn't need customers to actually
33:18sort of tell you that we knew that those
33:19were their basic pieces that we needed
33:21to build but pretty quickly customers
33:24came with also the photo requests I mean
33:26if I look at the sort of fun kind of
33:28things that a BBS was really good at at
33:30the fundamental level the scale
33:32performance security reliability and
33:36managing cost those are not profits on
33:38the outside but those are sort of core
33:40capabilities that come back in each and
33:42every one of these services and so
33:44customers then came to us they said well
33:46you know can't you want analytics for us
33:49and this this is the early days so
33:51everything about analytics around IOT
33:53about mobile development about you know
33:56that these days blockchain all these
33:59other technologies that customers
34:01actually want to use do not want to
34:03manage themselves yeah an asset you know
34:06will be helping them sort of putting up
34:08the the right features or tools and this
34:11is important and we also had a very
34:13strong culture around when we launch new
34:16products new services we will want him
34:19with a minimal feature set you could
34:21call it an MVP yeah but that sounds like
34:24that's that I mean this is technology
34:25where other people need to build their
34:27business on so you can't actually launch
34:30things that are flaky needs to be rock
34:32solid so in the launch rings rocks or
34:34believe a minimum feature set and then
34:36work with your customers on what should
34:37the other features be now in general we
34:40Lin have an inkling what the other
34:41features are going to be when we launch
34:42DynamoDB for example we really knew
34:44customers wanted secondary indices we
34:47didn't launch with them but that was
34:48obvious that that was what they wanted
34:50mostly it launched with with a minimum
34:52feature set because these are sometimes
34:55services nobody else has used before and
34:57nobody built and so you need to observe
34:59your customers how they're going to use
35:00your product because you don't know up
35:02front like see they probably use it in
35:04every possible way except for the one
35:05that you intended it to that is good
35:08because if it didn't launch if
35:11everything in the kitchen sink then sort
35:13of you can focus on how your customers
35:15are using your product and then slowly
35:17start to iterate and adding new features
35:20in the way that sort of their new modern
35:22way of development is we're working yes
35:25when we launched lamda which is our
35:27service environment so basically you
35:29just write code you dump it in in SC and
35:31you can just you don't need to think
35:34about servers and anything else you
35:36don't need to think about idle time you
35:38don't have to pay for what you use
35:39things that upgrade environment nobody
35:41done that before so how its development
35:44gonna change what's the kind of other
35:46support structures around it
35:48yeah or rittany maybe we launched it
35:50much more as an event-driven environment
35:52so the new file arrives in SC it
35:54triggers some codes you know a new
35:55message arrives it triggers some code
35:57API gateway things like that but it
36:00turns out some of the companies that
36:02actually immediately jumped on board a
36:04server for some of the largest
36:05enterprises why because you don't need
36:07to manage anything there and you don't
36:10need to pay if no execution happens I
36:13mean if you run a whole batch of ec2
36:15instances whether you use them or not
36:19you're being built for them and in this
36:21case the only being built for execution
36:23and so it changes the way the
36:25development happens so you don't launch
36:27everything and the kitchen sink I said
36:30earlier around servers you're going to
36:32see how your customers are using this
36:33and they quickly start iterating with we
36:35develop it x-ray as being the debugging
36:38deliver step functions to build more
36:40complex applications and there's just a
36:42lot of more things coming down in that
36:44pipeline mostly because you can observe
36:46how your customers are using your
36:48product so for example in dynamodb we
36:50new secondary indices turns out
36:52customers want to throw an item levels
36:54access management much more important
36:57for them than secondary indices so
36:59basically customers reordered our
37:02roadmap to be able to sort of and then
37:05we started delivering the things that
37:06mattered most to them which i think is
37:09an important part in this but again even
37:13though it looks like an MVP we can't
37:15treat it like an MVP because people will
37:17be building their business on it and
37:19will be depending on it so it comes with
37:21a very different culture structure on
37:23that so last year 1400 new features and
37:27services as the number of teams grow
37:31of course accelerates as well and and so
37:33we use within a DBS sort of the same
37:35structure the the Neptune team the graph
37:38database is supposed to be in contact
37:40with their most important customer most
37:42let's say demanding customers and
37:45understand what their needs are and so
37:47each of these teams has their own
37:48customer set and customer base and and
37:51all built a run web map the more
37:53services you get the more web maps you
37:54get but you know this is a is really a
37:58very fast-moving environment where the
38:01way that people are building software is
38:03changing dramatically changed
38:05dramatically as well if we would have
38:07sort of taken the step in deciding for
38:11our customers how they should develop
38:12software you probably would be having
38:14been developing software in let's say in
38:16the way that five years ago software ten
38:19years ago because that sort of the
38:20structure you have this time of it we
38:22need to decide developing software how
38:24we would like to develop software in
38:262020 or 2025 that's kind of the thinking
38:29that we have we can't do that by
38:32deciding for your customers you need to
38:34work closely with them and allow them to
38:35sort of drive your innovation engine
38:39about a decade ago you wrote a blog post
38:41called I think working backwards you
38:45remember what I'm talking about
38:46um I was cured I'm curious could you
38:48describe sort of the product development
38:51flow that you put in that post and then
38:54tell me what you learned and whether
38:56Amazon still uses that structure we use
38:58it everywhere and so it's it's a so the
39:03protocol working from the customer
39:04backwards so Amazon with a very strong
39:07focus on sort of developing only those
39:10things that really matter for your
39:12customer so even though we're technology
39:14company and I think if you're a heavy
39:15technology company lots of engineering
39:17there's a there's a risk that the
39:20engineers get in charge yeah if they get
39:23in charge you do not necessarily build
39:24products you build technology and so for
39:27us it's a drink that's agreed there's a
39:29big difference between the two I think
39:30what makes a be we ask for example
39:33successful is because we focus on
39:35products meaning what do we want to
39:38build for our customer much more like oh
39:40this is this very cool storage system it
39:43sits underneath there that we've never
39:45before or how do we do global tables in
39:47DynamoDB or so this is all this amazing
39:49tech that we building but that's not
39:52what's driving it is what do you want to
39:54build for your customer but it's the
39:55problem you want to solve for them so we
39:59want to make sure that emitter that is
40:00in AWS or whether that is in retail or
40:03whether it is opening a new office in a
40:06Menlo Park we use a process called
40:09working backwards why are we doing this
40:11so the first thing you do is you write a
40:12press release and it's not there's not a
40:15press release that will come out this is
40:17a press release you write for yourself
40:18in which we describe in in very clear
40:22and simple terms exactly what you're
40:24going to build then you write a document
40:27which that's a the 20 most frequent
40:29questions and then you have to answer
40:32those the very clean simple terms as
40:34well sometimes especially for more
40:36complex situations we iterate on those
40:39forward to first two documents maybe 10
40:41to 15 times until it's really absolutely
40:45exactly clear what you're going to build
40:47and not more than that then you write
40:50the UX document basically how are my
40:53customers going to use this or what is
40:55the interaction if the customer is going
40:56to come to B and then the fourth
40:58document is part of the user manual
40:59glossary and some other terms things
41:01like that at the end of that you have a
41:04set of four documents that describes
41:06exactly what you're going to do and then
41:08the ruling with the elements on is that
41:10and thou shalt not be even built more
41:12than that because as engineers and I'm
41:15going myself you have the tendency to
41:16sort of anything that should that makes
41:19it easy for you to build version two no
41:21matter what I start putting that in
41:22version one that's not an option
41:24yeah it's really focusing on building
41:27that and exactly only that it gives us a
41:31really strong structure around how to
41:34think about customers how to feel about
41:36think about products much more about
41:39them about tech technology so this is
41:41the product you want to build now what
41:43kind of technologies do we need to four
41:45or five or four for that so there's a
41:46very strong product thinking around that
41:52it combines with another process that we
41:56so meetings at Amazon we have a
42:00moratorium on using slides so no
42:03PowerPoint no key no nothing like that
42:05why because in meetings I think slides
42:07are deadly half of the room will be on
42:09their phones and the other half will be
42:11already complaining by slide number two
42:14that they don't understand what you're
42:15talking about that's obvious because you
42:18haven't seen the whole presentation yet
42:19so we we operate in Amazon with what's
42:23called six pages this is a six page
42:25document a narrative that you have to
42:28write and the first 30 minutes of a
42:31meeting will always be spent by reading
42:34this document in complete silence
42:36sometimes halfway through the reading
42:39you go like guys go back to the drawing
42:40board you know don't waste our time here
42:42because why because it is very important
42:45because it's very hard to write a clear
42:48document if you do not have clarity of
42:51mind writing and narrative is extremely
42:54hard and this might be this often a
42:56collaborative effort you why did you
42:57give it to some of your colleagues for
42:58feedback you put it in your drawer I
43:00week later you pick it up again and you
43:02revise it until you get to a point where
43:05you think you're really clear about
43:06describing what that's a feature or a
43:08product or an activity or new business
43:11that you want to go into and and after
43:14reading those six pages for 30 minutes
43:17everybody in the room is on the same
43:19page no pun intended battery for an
43:23intent that whatever Allah says so no
43:25and so you get a fairly high quality
43:27discussion after that because everybody
43:31is now is exactly what were you talking
43:33about so now often part of that process
43:35is then as an addendum to the sixth page
43:39it will be DPR and FAQ and you'll
43:41iterate for Londo so we have a very
43:44unique culture around that five years
43:48ago container sorry virtualization was
43:51still pretty new a lot of companies were
43:53just moving to that today everything is
43:57being containerized you have lots of big
43:59customers moving to lambda2 sort of
44:01totally server list platforms
44:04where do you think development is going
44:07to go five years from now what how are
44:09we going to be building our apps if I
44:12would have this crystal ball I will be
44:14sort of I do think I see quite a few
44:19companies skipping skipping the
44:21container step more and more not not
44:26that I think you know so I think there
44:27is a there's I think some part of the
44:29popular the the pile of I think what
44:33what come makes containers popular one
44:36thing there is everybody wanted to go to
44:39more micro services environment and I
44:41think that map's really well sort of if
44:43container tech technologies we can scale
44:45things up and down easily in these
44:48single components that you have and so I
44:50think that matches the micro services
44:52thinking really well I think most of the
44:54people that actually come into that
44:55phase often come out at the Molalla face
44:57or at the breaking autumn on live into
44:59sort of those container
45:01people that start from scratch more and
45:03more develop around sort of server the
45:05server environments and so you could
45:07even consider so for a brief an AWS to
45:12container products wellness is easy as
45:14the elastic container service it's all
45:16around docker and dr capabilities and
45:19things like that and deep integration of
45:21all the able services and then this eks
45:23which could be minutes could be neither
45:25service and so that's how we started off
45:28the thing with with the thing with
45:31containers is especially before we
45:34deliver the Fargate and I'll talk about
45:36is that basically it almost brings you
45:39back to the pre cloud days certainly you
45:42need to manage multiple containers over
45:44multiple availability zones your your
45:47there's almost so much they need to map
45:49them onto virtual machines because these
45:52things don't run by themselves
45:53you still need to manage all virtual
45:55machine environment underneath there so
45:56even though containers is a great sort
45:59of development expert abstraction it
46:02customers need to do a lot of work to
46:05Windows contain so part of that will be
46:07taken care of because it's a managed
46:09service we also delivered something is
46:11called fire gate so far gate basically
46:14takes away all the management of virtual
46:16machines and underneath there so
46:18just write a container you drop it in
46:20there Aaron's kind of that's how you
46:22that's the state where you want to be
46:24and yeah I think every time we you need
46:27to do things that actually have nothing
46:28to do with building your product or
46:30running your product in in most
46:32efficient way then that's sort of a
46:35waste of effort and so we continue to
46:37look at how can we take more and more of
46:39those pain points away because you know
46:40what nobody cared who were winning
46:42containers nobody cared about the future
46:44machines underneath there yes it's just
46:47the text that you had to pay and I think
46:50we were sort of bringing up especially I
46:53think the interaction in the kubernetes
46:55group is really feeding many things back
46:58into sort of the mainstream crew bonitas
47:00open source environment especially when
47:03it comes to things like security and
47:04things like that so how are we how I
47:07doing things differently five years or
47:08that was the question yeah I think
47:10there's a lot more service development
47:11because I think we see already an
47:13enormous pick up in the mountain I think
47:17we'll see more and more tools and sort
47:19of support platforms in infrastructure
47:21around the ability to build more complex
47:24service environments better integration
47:28I think with other services that's
47:29definitely things that we'll see but I'm
47:32gonna shift to something else I think
47:34one thing that we will be doing I hope
47:36five years from now is that everybody
47:39will be taking security as their
47:41number-one job whether you're the CEO
47:44whether the CTO whether you're an
47:46engineer we all need to become security
47:49conscious and we all need to become
47:50security engineers I think if you've
47:53looked at the past four or five years I
47:55mean there isn't a week that goes by
47:56where there is not some massive data
47:58breach it's embarrassing I think process
48:02technologists and as energy and as
48:04digital business leaders we should be
48:06embarrassed but we are not where's the
48:10outrage we almost seem to accept this as
48:13being sort of part of a base it's not
48:16you know without protecting your
48:18customers you have no business and I
48:22think forever and this is something that
48:23maybe as a young business as a startup
48:25you didn't think about ten years ago or
48:27five years ago where I think everybody
48:30needs to start with thinking about
48:32oh this is the kind of data I'm
48:33collecting for my customers how am i how
48:36am i protecting my customers you may say
48:38who is interested in when werner rented
48:42this bike here well you know what that
48:45data set plus to all the C's data sets
48:47that you may get from other places may
48:49have a very valuable position so I think
48:53we all need to become extremely security
48:56conscious conscious and make sure that
48:58we can continue to protect our customers
48:59it would be embarrassing I think as
49:01young businesses to actually start
49:03losing your data of your customers it
49:05will impact everybody else in your
49:08environment not just as customers but it
49:11will reflect on other young businesses
49:14as well and I think if you collect data
49:17of your customers if you have consumer
49:18data you have a great responsibility to
49:20keep that data secure and you know
49:24there's also tools for its secure
49:25there's this encryption and we give you
49:29dozens of different tools that all can
49:32help you would do to this but you need
49:34to keep it in your mind that that's your
49:36first and foremost job of course you're
49:38thinking yeah but we don't know we're
49:39building this new surface readability
49:42this new consumer service we don't fit
49:43now I think the only way if you actually
49:46build your your new business without
49:49security Mike it will be very hard to
49:51retrofit it into it and so you will have
49:54a nightmare let's say two three years
49:57down the road when you become successful
49:58and you grow you need to retrofit
50:00security you can't be in a major mess so
50:04that means also that your development
50:07processes because that's what you asked
50:09about need to change now security needs
50:12to become a default part of your for
50:14example your your continuous integration
50:16and continuous DD deployment pipeline
50:18pipeline yeah it needs the trig events
50:21whenever you're building something and
50:22someone adds a new open source library
50:24to it an alarm should be going off for
50:26someone to inspect why are we adding
50:28this why are we doing this do we check
50:31this out from a security point of view
50:32and so your your development pipeline
50:36itself needs to be secure there needs to
50:38be all sorts of alarms coming off in
50:39that whole section and security needs to
50:42become you first and foremost the con
50:44there's lots of automation tools around
50:46it yeah so you deploy you can use Amazon
50:49inspector to test about also to filner
50:51abilities but if you do continuous
50:53integration and contains the deployment
50:57especially if you for example in FinTech
50:59or in healthcare you're subject to also
51:01two regulatory requirements how do you
51:03know that these new next five lines of
51:05codes that you just vote don't violate
51:08your HIPAA yeah or that you're still in
51:12compliance with whatever the financial
51:13regulator wants from from for from you
51:15so if all sort of automation tools that
51:17can continuously test this for you but
51:19you do need to do it and it needs to be
51:21in your mind so development especially
51:23if you do continuous the deployment
51:25changes radically from how we've been
51:27approaching security in the past now I
51:29honestly believe continuous deployment
51:31actually is better from a security point
51:33of view because in the past he would
51:36write 50,000 lines of code security team
51:38would come in would review it would
51:39bless it they had no clue what they were
51:41doing that would bless it 50,000 lines
51:44of code you have no idea yeah and then
51:46you would DTD point changing five lines
51:49of codes it's something that you can
51:50actually test with automated processes
51:52are automatic reasoning and things like
51:53that that you actually are building the
51:55right right right right right things so
51:56I think it is great advances for
51:59security we do need to keep security in
52:01our mind so I hope that in five years
52:03time well not necessarily curity
52:06engineers were all super security
52:07conscious and protecting our customers
52:09will be our first concern it is at
52:12Amazon it will be forever a number one
52:14investment area both in terms of
52:16intellectual capital as well as sort of
52:18in in financial capital protect without
52:20protecting your customers you do not
52:22have a business beyond not taking
52:26security seriously enough are there any
52:27other common mistakes or errors you see
52:30startups doing as they use AWS
52:34well first and foremost I think there's
52:36a technical one so we've got quite a few
52:38people are the for the first time AWS
52:40users but that have already had
52:43experiences with say building services
52:45and using a traditional data center
52:46using AWS as a data center makes it you
52:49lose out I think yeah there are some
52:51advantages to it you have some
52:53elasticity and things are that it you
52:55can make use of but if you don't use
52:58services and security around data
53:01analytics around mobile all of these
53:04where we take away a lot of the
53:05development for love of the heart let's
53:09say the heavy lifting you're losing out
53:12if you just treat it as a data center
53:14with this on virtual machines a database
53:16and storage it's not that's not where
53:19your major productivity improvements are
53:22coming to come from but not the major
53:24ones next to that sort of a more of a
53:27meta level you have to figure out what
53:29kind of company you are and I think
53:32there's sort of two different companies
53:34to two different starts one of them is
53:36you know you you you go for the
53:38high-growth very far get big fast kind
53:41of thing you know acquire as many
53:43customers as possible as quickly as
53:45possible not necessarily thinking that
53:47much about revenue taking a lot of
53:49investors money and really get big
53:51really fast to become successful and
53:53then probably be acquired because that's
53:55what people interesting you
53:57that's that required that makes you your
54:00use of a degree a senior use of cloud
54:01very different from if you're a mere
54:04supporter really well because you can
54:05become a lot of worry that much about
54:07getting capacity or services whatever
54:09it's all there and it comes with clear
54:12cost picture as well that's different
54:14or if you want to become a sustainable
54:16company basically I want to build this
54:18company but still being business sweetie
54:20is from from for for now you know not
54:23necessarily focused on that position but
54:25just building a business I think if you
54:29guys follow a signal noise the guys from
54:33you know DHH and others taco and Jason
54:37free talked a lot about how they build a
54:39sustainable how they wanna build they
54:40wanna be in business still they want to
54:42still do this business 10 20 years from
54:45from now how do you do that and that's
54:47that's a very different that requires
54:49different architectures it requires much
54:52more control of a cost it requires a
54:54clear association between cost and
54:56customer acquisition so baby we are
54:59supported as well but it requires you to
55:02build fairly different architectures
55:04because you won't have much more control
55:06of constant scale then in the other one
55:09we another much concerned about cost
55:11you much more concerned about sort of if
55:13they say can be changed really faster we
55:15can address our customer concerns
55:17because we need to grow really fast
55:20Jeff often makes the distinction between
55:24mercenaries and missionaries yeah
55:28mercenaries are the startup founders
55:30that are in it for the money
55:31yeah and missionaries are the ones that
55:35are in it for the love of the product
55:38yeah I want to build this product or I
55:41want to build this business and you
55:42support both of them both fell at the
55:44bow fell at approaches to
55:46enterpreneurship it's just that the
55:49detect support and the tech that you
55:51built for each of each for these
55:53different groups it's very very
55:54different so figure out what you are
55:57alright well thank you so much
56:00this has been wonderful and we've
56:02learned so much and thank everyone a big
56:05thanks to dr. Verner Volvo
![How To Succeed As A NEW & YOUNG Realtor [Deals Every Month + Luxury Listings]](/_next/image?url=https%3A%2F%2Fi.ytimg.com%2Fvi%2FrtrRghgTayc%2Fhqdefault.jpg&w=3840&q=75)
