00:08my name is Sarah Wilson and I'm a
00:10designer here at Google and one of the
00:13creators of this series design is we
00:16started this series as a way to make
00:19space for the design community to talk
00:22about the intersection of design and
00:24technology and really crafting a future
00:28we all want to be a part of and so our
00:31topic tonight really sits squarely
00:33within our theme design is equitable I
00:37know when I started here at Google about
00:39five years ago I was amazing continued
00:42to be amazed at the number of people
00:43that use our products every day around
00:46the world and in all these different
00:48scenarios so really excited to hear from
00:50our two speakers tonight we've got Jen
00:55Jen leads the accessibility UX team at
00:59Google and works to make Google products
01:03universally accessible by developing
01:05standards and Consulting's consulting
01:07with teams across the organization
01:09nithya is a researcher leading research
01:13on underrepresented communities within
01:17the next billion and artificial
01:20intelligence space her her research has
01:23covered topics such as Internet access
01:25grid electrification crisis response
01:28civic engagement and more in the
01:32emerging markets so let's give it up for
01:34our speakers tonight hi I'm Victor I'm
01:38Jen and we are here today to talk about
01:42our experiences building equitable
01:44products in accessibility gender equity
01:47and AI and over the years we've been
01:50building equitable technologies for
01:52underrepresented communities and we'd
01:54like to share with you a practical
01:56framework for applying equitable design
01:59and research in product development so
02:02even though a billion people in the
02:03world today have some form of a
02:05disability according to the business
02:07disability forum 70% of uk-based
02:10websites do not pass base
02:13accessibility standards consider a
02:16common mechanism that we encounter
02:17online to distinguish humans from robots
02:20captures these are Turing tests to
02:23distinguish to make users identify
02:26obscure text however the task inherently
02:29excludes people with disabilities a
02:32capture can be difficult or even
02:34impossible for people with low vision
02:36blindness or dyslexia ProPublica repop
02:41reported on how a predictive algorithm
02:43to identify recidivism was particularly
02:46likely to flag falsely flag black
02:49defendants as future criminals at almost
02:52twice the rate as white defendants
02:54Britta Borden and her friend Shari Jones
02:58were two teenage girls who committed a
03:01petty crime despite being their first
03:03time they were classified as medium risk
03:05and it is argued that knowing the risk
03:07the judge raised the bond amount from
03:10zero to a thousand dollars women all
03:14over the world face online harassment
03:16misogyny and trolling in some form
03:18Ronna are you a senior Indian journalist
03:21discusses how she has been repeatedly
03:23attacked using online platforms through
03:25hate speech Photoshop tweets and deep
03:28fake porn videos made off her face one
03:32of the ways in which women respond is by
03:33simply reducing their online
03:35participation only 8% of Wikipedia
03:38contributors are reported to be women
03:40because of anticipated harassment our
03:44technology designers intentionally
03:48racist or misogynist why do these
03:51incidents occur there could be a number
03:54of reasons but you could argue that the
03:56people involved are ethical and yet this
03:59seems to happen today we can launch a
04:03new app that can be made instantly
04:04available all over the world the scale
04:08of technology is unprecedented in human
04:10civilization therefore when we miss an
04:13entire population or a context it can
04:15lead to unintended consequences at scale
04:18this can be a really large problem
04:20something we struggle to understand
04:23since both technology and society
04:26flex systems technology is not a value
04:29neutral medium it has inherent
04:31properties which can include or exclude
04:33communities depending on their social
04:36cultural or economic conditions
04:39unintentional exclusion has the same
04:41impact as intentional exclusion it does
04:44not matter what the intentions are its
04:46the outcomes that matter shift in the
04:51technology industry and media in the
04:53zeitgeist of what our responsibilities
04:55are equity has become much more
04:58important than ever before it's not a
05:01or an optional criterion anymore it is
05:03mandatory so our talk is about making
05:07equitable choices in design and research
05:10and product development and we are here
05:13to share with you a framework that we
05:14have found effective in driving our own
05:17inclusion based initiatives in gender
05:19equity and accessibility you could be
05:21single-handedly making a change in a
05:24product feature have an organizational
05:26mandate or have a fully funded and
05:29dedicated initiative for equity and our
05:31hope is that our talk provides best
05:33practices that could be useful for a
05:35variety of use cases Thanks so we're
05:40going to talk about three main topics we
05:43recognize that it's really hard
05:44sometimes to start these equity
05:46initiatives especially in large
05:48organizations so we aren't going to
05:50provide an explicit formula that you can
05:53follow to a tee instead it's a framework
05:55layer nathie a set of insights that
05:58we've gained over our time the framework
06:01is drawn from our work in gender equity
06:03online and accessibility for gender
06:07equity we draw our insights from having
06:09conducted qualitative and quantitative
06:11research in seven countries from Brazil
06:13to Indonesia and we spent the past two
06:17years working with various Google teams
06:18to make technologies more gender
06:21equitable as well as sharing our work in
06:23policy industry and academic venues and
06:26from accessibility side we're drawing
06:28from years of experience and really
06:30trying hard to develop a product design
06:33and development culture that really
06:35thinks puts accessibility first we're
06:39focus on shifting that mindset and we
06:42started with focusing on foundational
06:45research and understanding all of those
06:47unmet user needs that we hadn't
06:49discovered before in addition to that we
06:52were also looking at how our teams and
06:54organizations working what are their
06:56processes and where are their
06:57opportunities and gaps where we could
07:00actually bring accessibility in to make
07:02it a more holistic experience okay so
07:05with that let's start talking about how
07:07we can make a case for equity for each
07:12of these parts in the framework we're
07:13outlining some common challenges that
07:15we've seen so for making a case teams or
07:18organizations could be unclear on how
07:20this even fits with the business goals
07:22it could be difficult to get this type
07:25of work prioritized with competing
07:27efforts and finally it's something I
07:30think that is relatively unique to this
07:31this topic is some teams or individuals
07:34might feel a bit uncomfortable with the
07:37implications of iing being asked to work
07:39on this and we'll talk more about that
07:41in a second so let's start with how to
07:44demonstrate that this is good for the
07:45organization it's really common to kind
07:49of start with the ethical case here it's
07:51the it's the right thing to do and you
07:53can do this very effectively by going
07:55out and talking to users of these
07:57different diverse groups that maybe you
07:59hadn't talked to before understand
08:01observe what is their experience like
08:03using your product or service whatever
08:05it is bring this back to the teams and
08:08it's a really important critical step
08:10and can again gain a lot of initial
08:12traction now what we've seen though is
08:14that this gets you pretty far but
08:17sometimes kind of Peters out after
08:19people are like okay this is great and
08:21effective but I don't really know what
08:23else to do with this so this is where
08:25you can then move on to making a
08:27technical case and based on those
08:29insights you've learned from maybe your
08:31research and the unmet needs you've
08:33identified you can position these as
08:35challenges to team so how can we solve
08:37these using newer technologies such as
08:40ml or AI and really positioning it as a
08:43challenge that we need to get together
08:45and solve can be a really embedding
08:47to get more more involvement and then
08:51finally it's extremely useful to really
08:54focus on the business case creating more
08:57inclusive products can actually grow and
09:01dry product growth because you're
09:03starting to now tap into those untapped
09:06markets that you hadn't been focusing on
09:07before additionally you can actually you
09:13can actually position things and note
09:15noting that focusing on more inclusive
09:18products as well as having a more
09:20inclusive and inclusive design process
09:24can actually benefit you as an
09:26organization and differentiate your
09:27products and your organization from your
09:29competitors another tactic that can be
09:35really useful to make a strong case is
09:37to partner with teams that maybe have
09:39similar goals such as privacy or
09:42internet internationalization or
09:44localization the great thing here is
09:47that you can kind of package these these
09:51efforts to leadership as these are
09:53additional things we need to think about
09:55when we're device developing robust and
09:57holistic products they can't be think of
10:00thought of separately in addition to
10:03kind of pitching it that way you can
10:05work with these types of teams and
10:06understand what are we each of us are
10:09individual teams what are we asking of
10:11these product teams to do what kind of
10:13information do we need from the product
10:15teams what tasks are we asking them to
10:17do from maybe a localization perspective
10:20and are there opportunities to kind of a
10:22line and ask together and streamline the
10:25information you need from these product
10:27teams to make it just a more easier
10:30process for teams to adopt going forward
10:37another key thing is to ask if your
10:41product has considered all the
10:42stakeholders this is just getting at
10:44asking it may be obvious but important
10:47question what if your product is
10:50negatively impacting 5 percent of a
10:53market that you hadn't really focused on
10:55before is that ok is that what you
10:57really intended we really want to
11:00understand the experience obviously from
11:02different perspectives and really again
11:04focus on shifting that mindset to be
11:06open to hearing about ways people are
11:09using our products that we maybe never
11:10thought about before and thinking about
11:12how we can help those people framing is
11:17really important with this work and
11:19sometimes personal ethics can deeply
11:22conflict here with with professional
11:25ethics such as thinking about animal
11:28tests animal rights versus testing that
11:30might be necessary in some fields so
11:34therefore it's really important to
11:37understand that teams might feel like
11:39maybe they're being personally attacked
11:41here ethics are being personally
11:42attacked or some teams or individuals
11:44might just not feel comfortable about
11:46talking about this this topic overall in
11:49fear of offending others so when you're
11:51having these initial discussions it's
11:53important to have some ground rules of
11:56how to talk about this ensure that
11:57people that might be relatively quiet
11:59have a voice and can speak up and
12:01another great thing is to establish some
12:03common vocabulary so people can talk
12:05about this topic comfortably and talk
12:08about it in the same manner okay finally
12:12for making your case this is often
12:16unfamiliar territory for a lot of people
12:18and in our experience we've gotten a lot
12:22of received a lot of questions of ok
12:25understand this concept but what does it
12:27really mean to have an inclusive product
12:30or an ethical product what does that
12:32look like what does that feel like and
12:34this is where you really need to hone in
12:35on providing concrete examples there's a
12:38lot of great examples out there on the
12:40web from other companies and reports
12:42that you can take in fact on the screen
12:45up here we have one example here is
12:47called the Shirley car does anybody
12:50no hands okay so this was from Kodak one
12:53in the back this is from Kodak back in
12:56the day and they use this kind of as a
12:58template to understand skin tone and as
13:02you can see by this this picture it's
13:04all based on white skin and so the
13:07results of that is that people with
13:10darker skin tones when the shoulders
13:11were taking of them it was very washed
13:14out and dark which is not what they
13:15intended it or one in additionally we
13:19just wanted to highlight that you can
13:20actually create your own examples here
13:22as well and what we found useful is that
13:24if you you know in talking about this
13:26topic you can find people or teams that
13:29are particularly interested in gauged in
13:30this and you can use and ask them to be
13:32your pilots and say hey it looks like
13:34you're working on this new feature can
13:37we test out this more inclusive product
13:39development process with your team and
13:41through that obviously and hopefully
13:44you'll have a amazing case study at the
13:46end that you can then pitch and sell
13:47going forward and even if it doesn't go
13:50great it's an amazing learning
13:51opportunity to kind of work out the
13:53kinks as you continue how did we make
13:58the case for gender equity online a
14:00fairly non-traditional topic for the
14:03technology industry we started off by
14:05analyzing the access landscape how many
14:08women are online in various countries
14:10around the world we also built an
14:12ethical case around advancing human
14:14rights and creating an inclusive digital
14:16space as more women come online and
14:18finally we built a technical case around
14:21how our products can address computing
14:24challenges in privacy safety and machine
14:27learning we also got initial support
14:31from a few products which demonstrated
14:33the potential impact that we could
14:34possibly have and built confidence for
14:40accessibility as I mentioned before a
14:43lot of our initial efforts focused on
14:45the story of doing the right thing and
14:47again similarly as I said before we it
14:50took us pretty far but unfortunately it
14:52didn't get us as far as we'd liked so it
14:55was really good for us to understand
14:57what was that untapped market but
15:00and there's actually great data out
15:02there already and anyone can take this
15:03and from online and use it but just
15:07noting that there's a trillion dollars
15:10of collectible disposable income with
15:12people with disabilities and their
15:13friends and families worldwide or 21
15:17billion dollars of discretionary income
15:18of working-age people with disabilities
15:20in the United States this is really
15:23powerful and can help engage people at
15:25least talking more about this in
15:26understanding that oh there is this
15:28pretty significant portion of users were
15:30not talking about so this is good but
15:35believe it or not sometimes even that
15:36doesn't help and so this is because
15:39often teams might feel like okay that's
15:41that's great but we still don't feel
15:44that the of our total target user base
15:48the people with disabilities are still a
15:51small percentage and it's just not worth
15:52our time because it just takes too much
15:54time this is a common common story we
15:58get and pushback sometimes and so it's a
16:02great way to kind of reframe this is to
16:05help teams understand that actually
16:07disabilities kind of are on a spectrum
16:10and yes of course there are people with
16:12permanent disabilities that might be
16:14more extreme like users that are blind
16:16or people that are in wheelchairs
16:17permanently but there's also this notion
16:20of people with temporary and situational
16:22disabilities and this is where like in
16:25these pictures shown up here if you have
16:28a broken arm or an ear infection that's
16:30temporary and it prevents you from using
16:32your hands or hearing for a while we're
16:36in a situational side it's just consider
16:39it a very temporary disability like in
16:42this image up here if you're holding a
16:44bag of groceries or a child in one arm
16:46obviously you don't have full mobility
16:47as you would with you free hands so this
16:50is a great way to kind of again shift
16:52the way people think about this another
16:55way that we found is to kind of step
16:58away of thinking about this from a
17:00disability but think about from a
17:02usability perspective and say okay team
17:05XYZ what's your hands-free story here
17:08and or eyes-free and a lot of people can
17:11because they're already thinking about
17:12it from like a driving directions
17:14perspective or thinking about how people
17:16use their phones on the go or what if
17:18it's a partially audible scenario where
17:22you're in a loud environment but you
17:23have some sound or auditory information
17:26that's really important so this is
17:28another really useful tactic that we
17:31found so once we've made the case how
17:35can we create a knowledge foundation and
17:38principals through to act upon some of
17:41the resistance is that one might
17:42encounter in creating an equitable
17:44foundation our lack of clarity on what
17:47knowledge tools or resources would be
17:49valuable for teams not knowing when to
17:52stop or when the foundation is ready to
17:54work off off from the research side when
17:57do we know that our insights are
17:59evergreen and long-lasting and broadly
18:01applicable to various product teams and
18:03not having enough resources like people
18:06or money to build the foundation start
18:10by conducting foundational research on
18:12the problem space to get a baseline
18:14understanding of diverse people's
18:17practices and needs there is no
18:19substitute for actually talking to the
18:22communities that you're designing for
18:23use this knowledge to build empathy for
18:26various teams for example by videos
18:29reading transcripts which will help you
18:32build a lasting foundation that is
18:34internalized by the various teams and
18:36that lasts beyond the presentation
18:39however we may not always have enough
18:42resources to to carry out comprehensive
18:44research a highly scalable and efficient
18:47technique is to partner with other
18:48organizations that have deep commitments
18:51and expertise in working with the
18:52communities like NGOs research
18:54institutions and universities use what's
18:58already out there like Jen mentioned
19:00there's a decent amount of equitable
19:02design work that's already online and
19:04secondary literature that you can
19:06reference another way to gather diverse
19:09viewpoints is simply to make your
19:10existing efforts more inclusive for
19:13example in research can you start
19:15sampling for gender equity start
19:18including people with disabilities or
19:20LGBTQ community members so that existing
19:24processes are more diverse and
19:25James are more broadly valid to create a
19:29design foundation you could audit your
19:31products using existing checklists like
19:34the w3c guidelines or our emerging
19:36markets framework to assess the current
19:38state and evaluate problems
19:41so in our gender equity in an online
19:43initiative the challenge was that we
19:46were a small team one researcher one
19:48designer one program manager how could
19:51we build a lasting foundation of
19:53relevant insights for various products
19:55at Google and that could be applied more
19:58broadly to various technologies so
20:01firstly we constructed a research
20:03approach in seven countries to do
20:06comparative research across various
20:08contexts and in order to understand the
20:10gender equity space we use both
20:12qualitative and quantitative methods to
20:14understand the whys watts in the house
20:17we worked with several NGOs that work
20:19with communities like FGM survivors
20:23abuse survivors people with disabilities
20:25the LGBTQ community and technology
20:28communities like Google research
20:29scholars and llenan members we also
20:32attended several community events like
20:34pride marches in our countries of
20:36research sensitivity is key to
20:39conducting equitable research because
20:41many of the communities are marginal so
20:45we took three months to create a an
20:47ethical research protocol like
20:49conducting research in safe and neutral
20:51sites away from the home same gender
20:54moderation reduction of personally
20:56identifying information and so on in
20:59order to learn about women in various
21:01countries we established university
21:03partnerships in Pakistan let's Brazil in
21:06Mexico that helped us scale our research
21:08despite being a small team learn from
21:11regional expertise and establish
21:13academic ties a year and a half of
21:17research led to a mountain of data and
21:20findings so we structured this vast
21:22research into this one visual framework
21:25access content privacy in safety and
21:27this has been referenced over and over
21:30again in various meetings workshops
21:32presentations and has become sticky and
21:38for accessibility we learned pretty
21:40quickly that there's a relatively steep
21:42learning curve to understand truly how
21:44to design and develop fully accessible
21:47and good experiences and personally I'm
21:51still learning about this after being in
21:52it for about five years since our goal
21:56was to actually get other teams to do
21:58this work we didn't want to be the
21:59central team like Caniff EO we were a
22:01very small team we knew that we had to
22:04scale our efforts and so we we knew that
22:08teams would benefit from this solid
22:10foundation of resources tools and
22:13insights we also started focusing on
22:17foundational research so you see a theme
22:19here this is a great place to start go
22:21out and talk to your users we did it of
22:24users of many different disabilities
22:26doing many field studies for the most
22:29part and really just trying to
22:32understand what is a day in the life
22:34again what are their unmet needs their
22:36challenges how they work around issues
22:39and problems and from that foundational
22:42research developing a number of
22:45different resources that teens could use
22:47so some examples are videos you have
22:50these day-in-the-life videos from there
22:52we created personas that teams can use
22:54then and bring into design sprints or
22:58accessibility design guidelines this is
23:01an area where there were there are
23:04actually a lot of guidelines already out
23:06there on the web from agencies that
23:09focus on accessibility and we just
23:11tailored those a bit to MIT our material
23:13design guidelines so this was a really
23:15useful effort and again it just gave
23:18teams something to start with
23:20and then additionally like I mentioned
23:23in the beginning this the other thing we
23:25were really cognizant of is we didn't
23:28want to just throw these resources out
23:30to teams and let them fend for
23:33themselves and understand how to use
23:34them and additionally we didn't want to
23:36just create yet another process that
23:38teams had to to follow because they were
23:41already there's already enough things
23:42that people need to think about right
23:43when they're building products so this
23:45is where we did take time to understand
23:47okay what what is kind of the
23:49common product development process that
23:52most of the teams in the organization
23:55follow and then again identifying where
23:59can we integrate thinking about
24:01accessibility so for example for teams
24:03that had pretty well-established design
24:05reviews we went and talked to them and
24:08said hey how about you think about
24:10accessibility at this point and actually
24:12have a checklist or talk to the
24:14designers and ask how have you been
24:16considering contrast or your keyboard
24:19navigation experience similarly on the
24:22research side any time we understood
24:24that teams were going out and doing
24:25research be it foundational or
24:28evaluative are you including people with
24:30disabilities and there that's a great
24:32opportunity to at least get those
24:33conversations started and those insights
24:35brought into the team okay so question
24:41has anybody been in a similar situation
24:44where maybe you're not focusing on
24:46equity but you're you know kicking off a
24:48new effort project you've done all this
24:50legwork gotten some initial buy-in and
24:52then things start to fizzle out anyone
24:55okay you raise hands right and that's
24:59the last thing we would want to do here
25:01right we put a lot of hard work into
25:03building this foundation and getting
25:05this initial buy-in so we want to ensure
25:08that we can drive sustainable impact so
25:12here some of the common challenges we
25:14found is that probably pretty obvious
25:17that efforts do lose momentum right
25:19there's a lot of personnel changes new
25:21priorities come into play and things get
25:24prioritized another interesting thing we
25:27found in this space is that teams even
25:30though with all these great resources
25:31and stuff in general buy-in they still
25:34might not understand the real specifics
25:36of how to do this work and then finally
25:39teams might actually feel still a bit
25:41overwhelmed or even discouraged and this
25:43can come from the the feeling that it's
25:47just too big and broad like I can't go
25:49out and talk to all of these users with
25:51disabilities and different genders and
25:53stuff like it's just too much or they
25:56might feel like we're never going to
25:57make everyone happy so why do we even
26:00unfortunately we don't want that so to
26:05address this the first thing to do and
26:07this might be pretty obvious but it is
26:09important and we found this to be
26:10crucial is to establish top-level and
26:14formal goals with with your executives
26:18or leadership along the way while
26:21building the initial buy-in and
26:24establishing your foundation you may
26:26have had a lot of maybe peers and teams
26:30get excited and involved but you still
26:33really need that more formal buy-in from
26:35leadership that this is agreed-upon goal
26:37for the organization or team whatever
26:39level you're thinking about so whatever
26:42goal tracking system you have if it's
26:44okay ours or anything else it's really
26:47important to try to get that documented
26:50whatever that effort is or whatever your
26:52specific goal is around this this effort
26:56of equity getting those written down in
27:00order to hold teams accountable
27:03similarly as with any project it's
27:07really important to have clear success
27:09metrics and I think especially in is in
27:12this area where sometimes the teams that
27:14are developing the products and
27:16designing them they might not be the
27:18like direct receivers and understand
27:21that the impact that's actually being
27:23made by making these products more
27:25equitable so you want to be sure to
27:27communicate the the the impact more
27:32frequently throughout this throughout
27:35the experience and just demonstrating
27:38like yeah actually we are making an
27:40impact and just keep those lines of
27:42communication open to continue to
27:44encourage and motivate the teams some
27:47sometimes people ask well what kind of
27:49metrics are we talking about obviously I
27:51think a lot of it just depends on your
27:53products but the the main thing to think
27:55about here is whatever the metric is
27:58that you are have already established as
28:00a success metric maybe it's adoption or
28:03conversion just kind of think of that
28:06from through the lens of this these new
28:08these new diverse users or
28:11it's so what is the adoption rate of
28:14this diverse group XYZ some other ways
28:20you can think about this are are you
28:22receiving fewer complaints or more
28:24positive reviews from different
28:26different user groups even accumulating
28:30positive or maybe negative press stories
28:33can be useful just to continue and
28:35sharing that message of how the team is
28:37doing this this point here is about this
28:45notion that when you were building that
28:47foundation initially through all those
28:50amazing insights you were developing
28:53maybe a number of different types of
28:55resources again the personas the design
28:57guidelines etc all of those things are
29:00can be really useful but at this point
29:04you're interested in scale right you
29:06want the teens to start doing this work
29:08for you essentially and so you can't be
29:12there to handhold them and show them oh
29:14this is the exact resource you need and
29:16how to use it so one of the things that
29:19we found useful is to kind of bring
29:21these things together and put into a
29:23toolkit playbook whatever you want to
29:25call it but bringing some sense an
29:27organization to all of these resources
29:28so that teams are clear on exactly how
29:32and when to use these different
29:34resources the other thing that I wanted
29:38to highlight that we found effective is
29:41even it with like the best designed or
29:44organized toolkit team still feel
29:46overwhelmed right there's a lot a lot of
29:48great resources that often come out of
29:50this research so just highlight one or
29:55two things that teams can start with and
29:57just to get their feet wet and get
29:59involved in this type of work in this
30:01type of thinking can be really
30:02beneficial versus overwhelming them with
30:04a million choices now finally to kind of
30:09bring all of this work home and really
30:11drive it drive it home to teams we found
30:14that workshops can be really effective
30:16so you as kind of the expert in this
30:20area now can come to the workshops and
30:23then obviously the team members
30:25that are the product experts can come to
30:27these workshops and together you can
30:29kind of take a step back and say what is
30:31the experience for product XYZ for these
30:34different user groups and how can we
30:36make them more equitable and and
30:38actually brainstorm and work through
30:40those together so obviously this is a an
30:43amazing opportunity for training and
30:47it's a really rich and robust way and
30:49then as those team members go off to
30:51their their teens or even their future
30:53teams it kind of propagates this this
30:56learning in this way of thinking which
30:57can be really effective so Jen and I
31:02have diametrically opposite framing
31:04challenges with our research communities
31:08accessibility is something that many may
31:10not have closely encountered in their
31:12lives gender is something that is all
31:15too familiar and a subjective and a
31:18personal topic for most people so we've
31:21worked with various teams to conduct
31:24workshops and apply research insights
31:26into product features design or go to
31:28market plans teams usually bring metrics
31:32and product expertise to their table
31:33like Jen mentioned and we bring the
31:35gender expertise and together we
31:37co-create solutions unlike traditional
31:40sprints where research is shared in
31:42lightning talks and then we move on to
31:44user journeys and defining the product
31:46the research is the centerpiece of our
31:48workshops that we divide the workshops
31:50into our four themes of access content
31:52privacy and safety and dive deep into
31:57these topics we also had to help teams
32:00unlearn their assumptions around gender
32:03especially from the vantage point of
32:05product team members in Silicon Valley
32:08we we also had a translation challenge
32:11with converting gender equity insights
32:14into tactical product features so
32:18therefore we use a variety of activities
32:20in our workshops to apply the insights
32:23into the product first we introduce
32:25teams on to how to talk about gender and
32:29we introduce the concept of
32:30post-colonial feminism which provides a
32:33common vocabulary even though
32:35individuals bring their own lens
32:37we use scenarios for example on device
32:40sharing to identify how the products
32:43should respond to various use cases and
32:45to break up the monotony of a long
32:48workshop we also use quizzes like true
32:50identifying whether the content is fake
32:52or not which many women encounter
32:54especially in health and money domains
32:56surprisingly this is hard and our
33:00approach has been to share our work
33:02widely both inside of Google as well as
33:04outside into various communities we
33:06presented at the United Nations the
33:09hai soups and many internal fora at
33:13Google so to drive sustainable impact
33:17within accessibility we basically
33:19followed all the things we just talked
33:21about earlier such as providing a
33:24playbook which has been really effective
33:25for us in addition to that though we
33:28also wanted to find opportunities
33:30throughout the year within within the
33:34organization so kind of reinvigorates
33:37teams and and really highlight the work
33:41that's going on in accessibility so
33:42there is an accessibility week I think
33:45this this happens in a number of
33:46companies across the world and here's an
33:49opportunity for for teams to showcase
33:52the great work they've been doing do do
33:55research events or workshops and also
33:58highlight the unmet needs still that we
34:01haven't achieved and give teams some
34:03encouragement of some new design
34:06challenges that could that are out there
34:07and this doesn't have to be just one
34:09week obviously throughout the year this
34:10can be ongoing but critical events like
34:13this can be really critical in helping
34:15just resurface that because there is
34:17some times that that kind of low or
34:19people get a bit complacent and so it's
34:21it's a good reminder and refresher and
34:23then finally another thing that's been
34:25really effective and sustaining this as
34:27I mentioned before is we can't do this
34:30all ourselves right no matter how
34:32passionate we are we just can't scale to
34:34do this especially in large
34:35organizations so establishing some sort
34:37of championship program or something in
34:41that nature can be really effective
34:42where you you can take those people
34:44maybe you originally start with that
34:46show just some general or genuine
34:48in this and give them more training have
34:52have more kind of explicit goals for
34:54them such as okay you've been trained
34:57now you can be the trainer now go train
35:00your team have some workshops with your
35:01team see if you can influence processes
35:04within your team or organization and
35:06really kind of continue that work
35:08forward one thing I just wanted to
35:10highlight for this for any program like
35:12this and maybe some of you are already
35:14aware of it but we've found that to
35:17again keep that momentum going and keep
35:19people excited about this you really do
35:21need to think about how to reward and
35:23recognize them as well so that's that's
35:25really important to consider in these
35:27programs as well so to sum up we
35:33discussed a framework on how to make
35:37design and research more equitable
35:40starting with making the case through
35:41business technical and ethical cases
35:43creating a knowledge foundation with
35:45diverse people and understanding their
35:47practices and needs in achieving
35:50sustainable impact through clear goals
35:53metrics and workshops there are plenty
35:57of resources that are available online
35:58or canonical resources for accessibility
36:01or material design for gender equity
36:03look up designer google slash n bu and
36:06on Google Scholar and I we want to end
36:11the talk by saying that it's inherently
36:14challenging to do equitable design and
36:18we acknowledge that this is because
36:20there are no clear formula or templates
36:22that you can simply apply and while we
36:24have shared some of our experiences here
36:27it you may have to modify or improvise
36:29for your conditions and there are many
36:33ways to achieve and define equity it's a
36:35pluralistic enterprise especially when
36:38there are multiple members that are
36:40involved in a project if there's one
36:43thing you can take away from this talk
36:44ask the tough questions because if you
36:47don't who else will thank you write to
36:52us about your experiences with equitable
36:55design and research we'd love to hear
37:04okay I'm Jen that was awesome
37:07wouldn't open up to the room for
37:08questions myself and Nicole have mic so
37:13raise your hands if they're just one in
37:16the back hey thanks um so what at the
37:20outset I heard the examples given of the
37:23problem of email data sets being
37:26heterogeneous or misrepresentative
37:28or poorly representative a larger
37:30community and the other example of
37:32triple a compliance I'm less clear aside
37:36from the big data set you know research
37:39base example how how other communities
37:46might be affected by this kind of site
37:48you provide some problems that you see
37:51in designs and some examples like don'ts
37:53basically to help kind of make this make
37:57this problem more concrete so more more
38:02examples of how an equitable design can
38:05impact different types of users besides
38:07people with disabilities or yeah I mean
38:09I think triple A's is kind of a base
38:11standard we understand but I'm not clear
38:14maybe in some of the case of gender or
38:17of LGBTQ communities aside from the case
38:21where they might be misrepresented under
38:23or not represented in an ml data set
38:26like what what can we do or what's the
38:27problem that we might run into if we're
38:29not taking this into account yeah I
38:33think our framework is a good starting
38:35point here so let me start with the ml
38:40example when when women are so in in 2/3
38:44of the countries worldwide there are
38:46more male users than female users and
38:48when that happens there are the data
38:52sets of women the training data around
38:54women is just much smaller than the male
38:55majority data and so the the content
38:58that is presented to users it tends to
39:02be most queued towards majority user
39:04groups but in addition to that there are
39:06also considerations around simply
39:08starting with access what how can women
39:11get to the Internet online there are
39:12many assumptions that we make around if
39:14there is a network then there is
39:16connectivity then there is a user but
39:19that's simply not true there are many
39:21inherent socio-economic constraints that
39:24prevent women from going online like
39:26they simply have limited time because of
39:28housework or childcare responsibilities
39:30there's limited physical mobility which
39:33prevents them from going to say Wi-Fi
39:34hotspots and that that simply affects
39:38the the uptake of technologies say with
39:40privacy a common phenomenon that we've
39:43documented is their own device sharing
39:45and in how women are often in positions
39:48because of social expectations to share
39:50that their devices and that's something
39:53that we don't consider with one user one
39:55account one device paradigms and and it
40:00can go wrong when experiences are
40:02designed for one user and are used by
40:04multiple people and you know
40:06accidentally incriminates content that
40:09someone else has browsed take safety
40:12there are pressing concerns that women
40:16face with abuse that they face online
40:18and the consequences that can happen as
40:20a result like stalking and impersonation
40:23and so as technology becomes inevitably
40:28more provides more connective connected
40:30opportunities through network
40:32capabilities and ability to communicate
40:34with other users of the product we have
40:38to think about what are the implications
40:40when strangers can contact anyone online
40:43and what data can be taken from them and
40:47copied or abused or circulated virally
40:50online so I hope that gives you some
40:52sense of how things can go wrong
41:02hi I'm curious asou has this guy been
41:05applied to at an actual a Google product
41:08or feature and what was like someone to
41:10like a feedback or like whatever like
41:12more I guess specific results that you
41:14saw I said like how that how the guy was
41:17actually able to improve accessibility
41:18or equity like make the overall product
41:22I guess more equitable yeah we're
41:29working with a number of product teams
41:31there they're all in progress
41:33initiatives right now and the
41:36accessibility side we we have like
41:41through wit CAG and stuff a number of
41:42standards that we work with with teens
41:44on one of the key things that were
41:46working towards is moving away to from
41:49the conversation from just compliance
41:51and actually thinking about what is the
41:52actual user experience so a lot of our
41:56work is is actually taking teams and
42:00running through their user journeys with
42:03people with disabilities to really get
42:04that true sense of what's the actual
42:06user experience and then building off of
42:13I love your talk how would you define
42:16success for this equitable design
42:19initiative this upcoming year so so for
42:27accessibility I mean obviously a lot of
42:31our work I would say just in general
42:33right for measuring the success of
42:36products we're always looking at things
42:37like user satisfaction and just the
42:41overall usability of our products and
42:43that that goes without saying that we're
42:47looking at that from the same
42:48perspective of people with accessibility
42:49needs that's one of the things that
42:52we're really trying hard again is one of
42:55the key themes I think for all of us
42:57work is that we we we wanted again a lot
43:01of mind shifting here but we want to
43:03shift that mindset of thinking of this
43:05is like this extra other work you know
43:07we want this really to be integrated and
43:09so the way we just approach design in
43:11general so as far as metrics go and
43:14stuff like that you know we're using the
43:15same metrics it's just now we're trying
43:18to look at them from these different
43:20perspectives to see like are these
43:21experiences similar or are evaluated as
43:26good or as bad yeah adding to Jen on the
43:30gender side there are a few simple
43:32metrics that you can use and really it's
43:34not not only limited to gender you can
43:36apply this to any demographic
43:38understanding what's the market
43:44indicator for that particular
43:46demographic as in say you're building a
43:48mobile money application how many women
43:50have bank accounts that's sort of the
43:52baseline and then understanding from
43:54your product how many women have
43:57accounts how what percentage of them are
44:00engaged and so on so that you can look
44:01at the Delta between the possibility and
44:05where you are right now and that helps
44:07teams achieve understand where they need
44:10to go and where they are today
44:11I have one more question - and you
44:17sort of the three reasons why the you
44:21know the it's the right thing to do that
44:24technical reasoning in the business case
44:26which ones have you found most actually
44:29effective or does it kind of depend on
44:30the audience that you're that you're
44:33sort of trying to persuade in getting to
44:36buy in on doing this extra work or not
44:38next to work the work I personally have
44:42seen that you need all three again it's
44:46it not that it always falls sequentially
44:49in the way I kind of describe it but it
44:51often does because I think it is easy I
44:54mean it takes work but it is easier to
44:57just kind of show the experience and get
45:00that initial empathy buy-in but again
45:04when it comes down to I agree with this
45:07of course I don't want to you know
45:09exclude people intentionally there's
45:13still time and effort that needs to be
45:14put into that so you need to make that
45:16business case as well like why this is
45:18important related to that point I'm kind
45:22of I'm struggling with the idea of
45:25making a business case sort of before we
45:32have the impact measured right so like
45:36this is the potential business case and
45:39it might be slim on an aggregate level
45:43especially if you know at Google for
45:46example where there are billions of
45:48users across the world and so how when
45:52we're making like really specific
45:54tactical trade-offs on the day to day
45:56I'm curious if you've had conversations
45:59that aren't able to bring those things
46:02to light in a very tangible way where
46:04you can say okay this is the thing that
46:06we're actually trading off here yeah I
46:10think this is where the the idea of kind
46:15of creating your own case studies and
46:17doing pilots with certain teams to
46:19really prove out that the focusing on a
46:23more inclusive design product
46:25development process can pay off
46:28those like those initial numbers you
46:34know of potential income and stuff is a
46:36way to kind of get that conversation
46:37started and yeah I agree it's it's
46:40potential right so but that yeah what
46:43I've seen in my experience is you do
46:45have to kind of prove it out a little
46:47bit and that's why is one strategy is to
46:50kind of see if there's a an opportunity
46:53and I will say one thing about all this
46:55work is I learned to be so scrappy and
46:58opportunistic and just like identifying
47:00if there are teams working on something
47:03and just kind of latching onto that and
47:04saying like hey can we can we test this
47:06out with you to prove out that case I
47:08don't know if you had different yeah I
47:10know not everyone is the luxury of 1
47:12billion users and if there's only a
47:14hundred users let's say and it's a it's
47:16a startup that's just starting out to
47:20John's point have showing a qualitative
47:22improvement in how the experience can be
47:24improved for specific communities could
47:26be useful I think it's also worth
47:28thinking about positioning within the
47:30larger industry in terms of how being
47:34seen as equitable today is actually a
47:37differentiator and can help recruit
47:39better technical talent onto the
47:43organization and and can improve the
47:45brand value as well well thank you this
47:56that concludes our design is talk for
47:59the evening and come hang out with us
48:02next month when we have another talk
48:03well we'll be sending out details
48:04shortly but thanks everyone for for
48:07joining us and thanks to India and Jen
