00:00 obviously the software industry depends
00:02 on the ability to create ever growing
00:04 amounts of software but the skills of
00:07 software development are difficult to
00:08 learn globally software development
00:11 skills are in great demand there's a lot
00:13 of talk about a shortage of software
00:16 development skills and this seems to be
00:18 a fact in most parts of the world this
00:21 shortage has been around for a long time
00:23 though and if anything seems to be
00:24 getting worse rather than better so why
00:27 is there a skill shortage now and what
00:29 should we be doing about
00:38 it hi I'm Dave Farley of continuous
00:40 delivery welcome to my channel if you
00:42 haven't been here before please do hit
00:44 subscribe and if you enjoy the content
00:46 today hit like as well according to
00:49 estimates the population of software
00:51 developers in the world is currently at
00:53 27.7 million this is up from 24.5
00:57 million in 2020 and it's is projected by
01:00 to increase by another million uh by the
01:03 end of next year projections predict
01:06 that this is to nearly double to 45
01:08 million by the end of the decade
01:10 although there have been many prominent
01:11 layoffs in Tech firms in recent years
01:14 despite this there remains a significant
01:16 shortage of people with the skills that
01:18 our employers are looking for us sources
01:20 claim a record of over a million
01:22 unfilled vacancies in technical jobs
01:24 this year so what are the companies that
01:27 employ People Like Us doing to cope with
01:30 this mismatch between supply and demand
01:32 well naturally they're looking for the
01:34 easy solutions first and going to common
01:37 sources of Solace like artificial
01:39 intelligence no code and seeking to
01:42 increase the use of Nearshore and
01:43 offshore workers but maybe more
01:46 significantly they're also adopting
01:48 remote working strategies in general as
01:51 the default hiring model which increases
01:54 the talent pool from which they can draw
01:56 making the pool of available developers
01:58 on which they can draw essent Al Global
02:01 this is a big change and was of course
02:04 initially driven by the covid pandemic
02:07 but is now widely seen as a tool to
02:09 provide access to more people with the
02:12 skills that the companies need
02:14 increasingly geographically distributed
02:16 teams are seen Less in the context of
02:18 saving money as they used to be and much
02:20 more now in gaining access to This
02:23 Global Talent pool this is presumably
02:25 good news for people with the skills
02:27 because wages will be higher this will
02:29 no doubt cause its own problems later
02:31 because Employers in high wage economies
02:34 will increasingly attract people from
02:35 lower wage economies to apply making the
02:38 developer shortage in lower wage
02:39 economies even worse than it is today
02:42 presumably if the economies of supply
02:44 and demand hold it will mean a global
02:46 pressure to increase wages for technical
02:49 people let me pause there and say thank
02:51 you to our sponsors we are extremely
02:53 fortunate to be sponsored by Eco experts
02:56 tricentis and transic all of these
02:59 compan companies offer products and
03:00 services that are well aligned with the
03:02 topics that we discuss on this channel
03:04 every week so if you're looking for
03:06 excellence in continuous delivery and
03:08 software engineering or maybe looking
03:10 for a new job click on the links in the
03:12 description below and check them out
03:14 longterm you'd expect all of this stuff
03:17 to balance out and as software
03:19 development becomes more appealing as a
03:21 job due to the increase in wages we'd
03:24 expect to see an increase in the numbers
03:26 of people training and joining the
03:27 profession which does seem to be
03:30 happening in many parts of the world
03:32 University admissions for software
03:34 courses are up software engineering
03:36 courses are the most popular engineering
03:38 courses of all in many geographies now
03:41 but there does seem to be a problem
03:43 still and it's not just that it takes a
03:46 while for this kind of change to play
03:48 out there's something more broken here
03:50 than only that in an article about the
03:53 shortage of software developers from a
03:54 company called hatchworks they list four
03:57 things that all contribute to the skills
04:00 shortage they also mentioned a fifth but
04:03 it didn't seem relevant to me so I'm
04:04 going to leave that out of my my take on
04:06 it the first that they list is industry
04:09 growth clearly as more software is
04:12 needed and we are to cope with this
04:14 increased demand we can address that by
04:17 either adding more programmers to write
04:19 the code making programmers more
04:21 efficient at writing it or finding
04:24 someone else or something else to write
04:26 it for us those are the only tools
04:29 really at our disposal which are likely
04:31 to have any effect and they're mostly
04:33 the theme for the rest of this today the
04:35 next barrier to growth in the list
04:37 though is I'm not quite so sure I'd
04:39 agree with that is the increasing
04:42 complexity of software
04:44 development I think that systems that we
04:46 build today are certainly often more
04:49 complex now but the tools and support
04:52 for programmers are now dramatically
04:54 better than they used to be so there's a
04:56 tradeoff there for sure I'm also not con
04:59 convinced that today's programming tasks
05:01 are really more difficult than
05:03 development was in the past it's
05:05 certainly easier to build a basic crud
05:08 system now than it was 20 or 30 years
05:10 ago because of the tooling and the
05:12 availability of examples I'd argue that
05:14 the level at which software is more
05:16 complex now than it was before is really
05:20 more at the limits of performance and
05:22 scalability and that's always been a
05:24 fairly esoteric game for genuine experts
05:27 building large scalable systems is
05:29 easier and cheaper now than it used to
05:31 be with the cloud and techniques like
05:33 servus where I think there is some truth
05:36 in this claim of increased complexity
05:38 though is is different to simply saying
05:41 that modern software is more complex and
05:43 this is this difference really has two
05:46 important parts to it one is that the
05:48 scale today genuinely can end up being
05:51 Global and the other is that we often
05:53 shoot ourselves in the foot by not
05:55 choosing sensible solutions that match
05:57 the scale of the problems that are in
05:59 front of us one way that systems were
06:01 simpler in the past is that at least
06:04 with the benefit of hindsight there was
06:06 a kind of relative golden age of
06:08 simplification we learned to be able to
06:11 build simple basic systems fairly easily
06:14 by following a very successful recipe
06:17 the relational database back system
06:20 relational databases introduced a
06:22 simplified programming model that most
06:24 people understood well enough to build
06:26 basic systems I'd argue that the key
06:29 here was in in dealing with concurrency
06:32 in a way that made systems scalable
06:34 enough but without relying on
06:36 programmers needing a deep understanding
06:39 of the intricacies of dealing with the
06:40 extremely complicated problems of
06:42 concurrency relational database
06:44 Management Systems allowed systems to
06:46 easily scale well enough to usefully
06:49 work for hundreds and low thousands of
06:51 users and through techniques like the
06:53 ability to roll back transa transactions
06:56 we effectively version controlled
06:58 changes to data so that it didn't end up
07:00 in a horrible un mergeable mess this
07:03 meant that changes were safer and that
07:05 you and I could be protected from the
07:07 consequence of changing the same value
07:09 in a database even though we were
07:11 working in parallel with each other and
07:13 changing the data in parallel so we
07:15 ducked one of the genuinely tough
07:17 problems in software the management of
07:20 concurrent change to data as long as
07:22 your system fit into this widely
07:24 understood programming model development
07:26 was reasonably easy at least compared to
07:30 Alternatives but if you did need to deal
07:32 with the real consequence of concurrency
07:34 for yourself rather than relying on the
07:36 Excellence of the system programmers
07:38 that built databases things were a lot
07:41 more complicated what happened next was
07:43 that we started to build systems that
07:45 needed to scale much further than the
07:47 few thousand concurrent users and the
07:50 database model then ran out of steam it
07:53 didn't work at those kinds of scales so
07:55 people with bigger problems
07:57 organizations like the huge web comp
07:59 companies started to innovate with other
08:02 more effective techniques that did work
08:04 at this new scale these techniques were
08:07 dramatically more scalable and became
08:09 fashionable and famous in the software
08:11 development World ideas like no SQL micr
08:14 services and event-based systems spring
08:16 to my mind the difficulty is that all of
08:19 these programming models exposed us more
08:22 directly to those really difficult
08:24 problems that are at root to do with
08:26 concurrency and scalability
08:29 the problems that the rdbms is hid from
08:32 us for a while software is easy as long
08:35 as it's local single threaded and
08:37 doesn't share data as soon as you
08:40 introduce distribution concurrency and
08:42 writable shared data though software is
08:44 worldclass complicated and needs
08:46 thoughtful deep expertise which is
08:49 indeed rare actually I think that may be
08:52 an interesting video for another day
08:54 have we got the programming models wrong
08:56 for the cloud let me know in the
08:57 comments if you think that's an
08:58 interesting topic so the two parts of
09:00 the problem are that this increased
09:02 scale of modern systems broke a model
09:04 that had worked pretty well for a couple
09:06 of decades and here's the really
09:08 annoying part teams that didn't need to
09:12 because they didn't have the problem of
09:13 massive scale still brought in the much
09:15 more complex programming model because
09:18 that was what was fashionable at the
09:19 time by choosing an approach that was
09:21 based on the old-fashioned relational
09:23 database architecture they may have
09:25 found a better fit for their real
09:27 problem rather than than trying to adopt
09:30 the problems of global scalability for a
09:31 system that wasn't if you're writing
09:34 software for an internal use in a
09:36 company of 10,000 people you don't need
09:38 it to be globally scalable and the
09:40 Technologies of global scalability come
09:42 at a a cost they are more complicated to
09:45 get right so yes modern software is
09:48 sometimes more complicated than it used
09:49 to be but often because we choose to
09:52 make it more complicated because of
09:54 fashion more than because it really
09:55 needs to be the skills to cope with the
09:57 complexities of leading scalability and
09:59 performance have always been esoteric
10:02 and rare writing a Flight Simulator game
10:04 in the 1980s that worked on the hardware
10:06 of the day or an operating system that
10:08 managed concurrent processes wasn't
10:11 easier then than building a global scale
10:13 website is today it was just difficult
10:15 in different ways partly because the
10:17 other factor that added to the
10:19 difficulty was that there was less help
10:21 then than there is today nowhere to look
10:23 up Solutions not really very much shared
10:25 code at all to look at no stack Overflow
10:28 no internet search developers knew what
10:30 they knew from reading books and they
10:32 had to remember it so let's not allow
10:35 our Rose tinted glasses to undervalue
10:37 how difficult software has always been
10:39 to create which brings us to the next
10:42 factor in the skills shortage a mismatch
10:45 between education and Industry this was
10:47 called out in the article knowing what
10:50 to teach people to get them started in
10:52 software development career has always
10:54 been a difficult problem to solve but
10:57 this is probably something that has got
10:59 worse my impression is that software
11:02 development has never been a very
11:04 strongly academic subject I don't know
11:06 if this is because of the nature of the
11:08 subject but at least partly it's because
11:11 of an accident in history I think and
11:13 this is EXA exacerbated by the
11:15 commercial imperatives surrounding
11:17 software development and the need to
11:19 create ever more software faster because
11:22 of the constant pressure on everyone to
11:24 deliver sooner what every employer is
11:27 seemingly looking for is developing with
11:29 proven experience of working on exactly
11:31 the same type of problems ideally in the
11:33 same industry using exactly the same
11:36 tools as them it's no wonder that it's
11:39 hard to find people so inevitably
11:41 education is never going to produce
11:43 graduates like that the result is that
11:45 even with a good degree it can be very
11:48 hard to find your first job and to gain
11:50 that vital industry experience that
11:52 everyone's looking for the step from
11:54 graduate to Junior developer is a huge
11:56 one I think that this is one thing that
11:58 employers could do more to improve on
12:00 the shortage of software developers that
12:02 is invest more time effort and money to
12:05 supporting people in their careers
12:07 particularly people who are just
12:08 starting out instead of trying to hire
12:11 clones of your existing Workforce hire
12:14 bright motivated graduates and invest in
12:16 their learning develop in-house
12:19 schemes this way you get to shape the
12:22 next generation of developers to
12:23 precisely fit your organization and
12:26 teams as a civil engineer you don't get
12:28 get to build a bridge alone on your
12:31 first day out of University why do we
12:33 expect the equivalent from software
12:35 Engineers when I suggest this to
12:37 organizations they often push back
12:39 though with yes but as soon as we have
12:41 trained them they leave for better jobs
12:43 we'll come back to that topic later when
12:46 we talk about competition for talented
12:48 people but for now let's get back to the
12:49 topic of education for software
12:51 professionals software development is a
12:54 difficult thing to do well and it takes
12:56 a long time to learn to be good at it
12:58 during in the early days of my career a
13:01 large fraction of software developers
13:02 were self-taught former hobbyists we cut
13:05 our teeth playing with software doing
13:07 all sorts of interesting sometimes silly
13:10 silly things and learning a lot by from
13:12 our mistakes if you'd like to see some
13:14 of the sillier things that I did check
13:16 out this very old video of
13:18 mine I don't much like the description
13:21 of software development as a craft but
13:23 often when people say it's a craft I
13:25 think what they really mean is that it's
13:29 and you learn to be good at creative
13:31 things through practice more than Theory
13:33 you don't learn to be good at writing or
13:35 playing music without writing or playing
13:38 music a lot this is true of other kinds
13:40 of things too though as well as craft
13:43 and I think that craft is only a rough
13:45 approximation for what we really mean
13:46 here this is just as true of engineering
13:49 for example engineering is a deeply
13:51 creative discipline too but it's
13:53 different to craft because engine in
13:55 engineering there are more effective
13:57 guidelines that help help us to rule out
13:59 some of the more dumb ideas along the
14:01 way and so to make progress more
14:03 efficiently and more effectively this is
14:06 another way for us to make ourselves
14:07 more efficient and so reduce some of the
14:10 pressure of the skills shortage at
14:12 present I don't think that university
14:14 courses do enough to give people the
14:16 core skills of our trade in this way
14:19 though this is not really their fault
14:20 though because as a trade we have not
14:22 clearly identified these core skills and
14:24 told the universities what it is that we
14:26 need but I think that they maybe
14:29 becoming clearer which is what my book
14:31 modern software engineering is all about
14:33 really there's a link in the description
14:36 below to the book if you're interested
14:39 more in universities I'd start by
14:42 teaching problem solving skills our job
14:45 is not to write programs it's to solve
14:48 problems with software it's been my
14:50 experience in employing graduates that
14:52 there's a difference between the ability
14:54 of graduates to get up to speed with
14:56 professional development based on their
14:58 course of study but it's not as you may
15:01 suppose the difference between computer
15:03 science and software engineering in my
15:05 experience the the bigger difference is
15:07 the difference between graduates with
15:09 more formal Science Background like
15:11 physics or chemistry who are better at
15:14 problem solving the stuff that computer
15:17 science and software engineering
15:18 graduates have learned is helpful but
15:22 it's the easier stuff to pick up on the
15:24 job than these core tools of problem
15:26 solving these are ideas like working
15:29 incrementally in small controlled steps
15:31 making progress by trying things out
15:33 small experiments keeping a record of
15:35 what you did so that you can easily step
15:37 back and try again planning your next
15:40 small step based on the data and
15:41 evidence not just your hunches and
15:43 working to control the variables so that
15:45 you can more easily see what's happening
15:47 when something changes and more broadly
15:51 how to research a problem fundamentally
15:53 how do you solve a problem when you
15:55 don't know and can't just look up an
15:57 answer I want I want to hire graduates
16:00 with those sorts of skills these are the
16:02 ideas that seem at the core of the
16:05 skills of the best developers to me and
16:07 these are the sorts of things that
16:08 universities seem to be pretty good at
16:10 teaching for other disciplines so while
16:13 it's helpful to learn supporting
16:14 theories and fashionable approaches that
16:17 is not enough without lots of practice
16:20 and the core skills of problem solving
16:22 are much more fundamental and
16:24 transferable not only from University
16:29 but from industry to to Industry in
16:31 software development so for our
16:33 profession to really grow we need to be
16:36 better at teaching how to be a good
16:38 software developer whether that is in
16:41 universities on the job or more likely
16:43 both the next in the list that the
16:45 article called out was in the increasing
16:48 competition for talented people when
16:50 organizations push back on my suggestion
16:53 of training inh house in reality they're
16:55 pushing back against Market forces in in
16:58 my experience good organizations
17:00 actively train the Juniors who join them
17:03 not just in their way of working but
17:05 also in the broader principles of what
17:07 it takes to be good at this stuff and
17:09 yes after such training some people may
17:12 choose to leave so find ways to help
17:14 them to stay make the organization a
17:16 good place to work support and help them
17:18 to Foster their Ambitions and their
17:20 career and even if you really must make
17:23 contract Clauses that tie people in to
17:25 stay for a time after their training is
17:27 complete other professions do this sort
17:30 of thing all of the time the most
17:32 effective response to people leaving
17:34 after training through though is to make
17:36 employment in your organization good
17:38 enough to appeal to and keep the people
17:41 there now you have a motivated engaged
17:44 Workforce so this is a double win I've
17:47 seen this approach work several times
17:49 and I've applied this approach myself
17:51 where I've been in a position to do so
17:53 to be honest though I don't really see
17:55 any practical alternative because this
17:57 is a Marketplace and talented people
17:59 will be attracted to good organizations
18:02 with interesting problems to solve so
18:04 look for ways to foster both of those
18:06 things the most common responses to the
18:09 issue of De developer shortage though
18:12 aren't much like any of these things and
18:15 I I am somewhat skeptical about their
18:18 Effectiveness it's obviously true that
18:21 if the world needs more software we can
18:23 either add more people with the skills
18:25 to produce it or find ways to make the
18:27 process of software production more
18:29 efficient and that means easier this is
18:32 really the corporate appeal of ideas
18:34 like artificial intelligence code
18:37 generation and low code Solutions but
18:40 while both of these things can help a
18:42 bit in some narrow fields of software
18:44 development I don't believe that they
18:46 are a general solution at least until AI
18:49 is better at humans at everything and
18:51 then they'll be running the businesses
18:52 as well as writing the software as I've
18:54 said before in both cases the current
18:56 approaches seem to be focused on the
18:58 problem being one of optimizing for Less
19:01 typing but that misses the central point
19:03 it is the problem solving that is the
19:05 bottleneck not the typing professional
19:08 software development is a process of
19:10 exploration and Discovery and for that
19:12 we need the mindset and the tools that
19:15 allow us to try out our ideas and
19:17 discard the bad ones not simply produce
19:20 software based on bad ideas
19:23 faster this is why I talk about the core
19:26 of our skills being about the problem
19:27 solving not the mechanics or even the
19:29 detail of writing the
19:31 code these problem solving skills are
19:34 the core of our discipline and the
19:36 sooner we recognize this and train
19:37 people in those skills the sooner we can
19:40 address both sides of the skill shortage
19:43 being able to add to the pool of talent
19:45 and make those talented people more
19:47 efficient in the application of their
19:49 skills thank you for watching and if you
19:52 have been watching do consider adding
19:54 your support to our team of patreon
19:56 supporters who keep this channel going
19:58 there's some great discussions currently
20:00 going on on our Discord channel so do
20:02 check those out if you join thank
