00:00hi everyone welcome to the a six in Z
00:02podcast I'm sonal and I'm here today
00:04with Michael and we are talking to Karen
00:06Schneider who is the CEO and co-founder
00:08of text EO a company that analyzes job
00:11listings to predict how well they're
00:13going to perform and can help optimize
00:15them to get more qualified diverse
00:17candidates and interestingly they've
00:18been able to figure out besides what
00:20what doesn't work very well in job
00:22descriptions words like synergize
00:24they've been able to figure out what
00:26does work well language like intact
00:28people love to talk about hard problems
00:30and tough challenges but it's a lot
00:34bigger than just about jobs the ability
00:36to understand the words we use and how
00:38we use them is pretty important because
00:39even though we're completely immersed in
00:41a world of tech where a lot of the
00:43conversation is around big data as
00:45numbers a lot of the data that we
00:47produce of the output of our work is
00:48actually taking place in the form of
00:51words and those words matter sometimes
00:55how you say things is more influential
00:57than what you're actually saying right
01:00and it's counterintuitive to any of us
01:02who've built products before because you
01:03like to think you're leading with a
01:05strong vision clearly words matter and
01:08another place that that plays out is
01:10with hidden biases that are often
01:12revealed in words for example Karen
01:15examined a number of resumes to see the
01:18differences between how women and men
01:19describes themselves as well as in
01:22performance reviews to see the ways that
01:23women and men were described differently
01:25the word abrasive which has been talked
01:28about since then ended up you know being
01:31used in 17 out of a couple hundred
01:34women's reviews and 0 times and in men's
01:37the sort of stereotypical like
01:40aggressive was used in a man's review
01:43with an exhortation to be more of it and
01:45in women's reviews as a term of some
01:47judgement okay let's get started
01:50Karen welcome so the reason we actually
01:52invited you to the ACNC podcast today is
01:54because you've been writing a lot of
01:55interesting work based on the outcomes
01:57of your product where you've been
01:58analyzing people's use of language in
02:01certain contexts as a way to surface
02:03insights and I think that's really
02:04fascinating because I think we have a
02:06tendency in our world to focus on big
02:09just numbers and not other forms of data
02:12cuz you're really describing I mean what
02:14you describe your work is doing is
02:16applying machine learning to text and
02:18natural language so how does that how
02:20did you kind of how does that work and
02:22then we can talk a little bit more about
02:23how you got there yeah so how does it
02:26work language is just an encoding of
02:29concepts right and anything that can be
02:32encoded can be measured and so I was
02:34sharing the story the other day we were
02:36actually originally started out looking
02:38at Kickstarter projects right so we
02:41started out with this question could we
02:42just look at the text of a Kickstarter
02:46project and some of its you know
02:47metadata around the text and predict you
02:50know before it was ever published
02:51whether it was going to raise money and
02:53we didn't look at the quality of the
02:55idea we didn't look at whether a
02:57celebrity endorsed it turns out we got
03:00over 90% predictive on minute zero of a
03:03project as to whether it was going to
03:04hit its fundraising goal based solely on
03:07things like how long is the text and
03:10what kind of fonts are you using and how
03:12many headings do you have so a wait a
03:14minute just to clear up any little bit
03:16so before the project even went live on
03:19Kickstarter just looking at the those
03:23features of the tax you're able to
03:24predict whether it be successful or not
03:25exactly what were some of the high-level
03:27takeaways from that yeah so longer is
03:30better where Kickstarter interesting is
03:32concerned kind of counterintuitive one
03:35thing that broke our hearts because my
03:38co-founder Jensen Harris and I both have
03:40some design background you would think
03:42these cleanly designed projects with
03:44this beautiful use of single typography
03:46would do best not so you want to look
03:49like a ransom note so you want to be
03:51posh types you want lots and lots of
03:54headings visually wonderful images to be
03:57front-loaded kind of makes sense but a
04:00lot of what we found was not intuitive
04:04and so just demonstrated for us the
04:06value of actually measuring because the
04:08whole Kickstarter karp corpus is out
04:10there in the world right so you can
04:11actually see how great training data you
04:13can see how well prior projects have
04:15performed and we saw hey we're kind of
04:17on to something here just looking at the
04:20very painful as a product person the
04:22quality of your idea
04:23matter just looking at the the content
04:26aspects we could predict and how do you
04:29account then for all the other sort of
04:31outside variables you know whether it
04:33was at the beginning of the Kickstarter
04:34kind of like craze whether it was a
04:36certain time of year for that matter
04:38certain type of product even yeah or
04:40geography like how do you know that in
04:44fact your analysis was correct I mean
04:46you can look at some of those other
04:47factors right because you can see when
04:49projects are published turns out that
04:52doesn't make a big difference you can
04:53see the only things that really move the
04:55needle in a very short-term way are do
04:57you have a celebrity endorsing you
04:58because that can get you a lot of social
05:00media attention it doesn't make or break
05:04you but it can it can help quite a bit
05:07and generally how good you are at your
05:09social media strategy can can tip the
05:12balance a bit but none of those other
05:13factors turned out to be as significant
05:16as we expected the ability to really
05:19zero and fire just the text did that
05:22surprise you um I mean we started off
05:24with the hypothesis that it would be
05:26that way and that you know sometimes how
05:29you say things is more influential than
05:32what you're actually saying right and
05:34it's counterintuitive to any of us
05:36who've built products before because you
05:38like to think you're leading with a
05:39strong vision we weren't surprised we
05:43were curious as we started to apply the
05:47technology to some other verticals
05:48whether it would extend you know our
05:52first big area has really been in the
05:53area of job listings where we've looked
05:55to see the first real product
05:57application where we've looked at
05:58listings now from over 10,000 different
06:00companies we've measured who's applied
06:02to which listings and we do see the
06:05content matters we do see some tailoring
06:07by geography turns out what works in New
06:09York is different than what works in San
06:11Francisco we see a lot of tailoring by
06:13industry so what works to hire and tack
06:15is very different than what it looks
06:16like to hire a claims adjuster or
06:18someone in retail right so you see some
06:21differentiation but in all cases
06:22depending on how you're slicing and
06:24dicing the categories that text leads
06:26you know we've looked at real estate a
06:28little bit prior to launching our
06:31jobs application and we've seen the same
06:33principles apply so so far you've been
06:36talking about the form of the text like
06:38the length and the fonts and the design
06:40but like were there particular words
06:43that popped out as well in terms of what
06:45people said on those Kickstarter
06:46descriptions or anything like that I'm
06:48bringing this up because there's just
06:50this recent anecdote in the news that I
06:51read about someone saying that you can
06:53predict success or default of loan
06:56applications based on words people use
06:57like God or using God a lot will
07:00actually mean you'll you'll default
07:01you're more likely to fall on your loan
07:06in Kickstarter we didn't look at that we
07:09started looking at that for real estate
07:11listings and then jobs where we've
07:13looked at it quite a bit so we saw when
07:16we were prototyping out the real estate
07:17stuff that if you say off street parking
07:19on that really moves the needle for
07:23low-income homes but for high-income
07:25homes in terms of the number of people
07:26who will go to your open house and then
07:28of the eventual sale price of your home
07:29for higher priced homes it's actually a
07:32negative because why would you want to
07:34highlight that it has off street parking
07:36it's just sort of an expectation so we
07:38saw you know vocabulary matter quite a
07:42bit in jobs it matters hugely you know
07:45we've we've identified at this point
07:47over 25,000 unique phrases that move the
07:51needle on how many people will apply for
07:53your job what demographics I'm qualified
07:55they are could you share some of that
07:56insight with us because you know the
07:58reason that came across your work is
07:59because I read an article about how you
08:02analyzed performance appraisals and job
08:04descriptions for insights about what
08:06moves the needle and and the differences
08:08and how people communicate what are some
08:11of the things I mean just could we have
08:12a huge audience this that does job
08:14description yeah so there's sort of a
08:19set of language that works really well
08:21for everybody these are not surprising
08:24on the face of them but when you look
08:25you see lots of them so things like we'd
08:27love to hear from you be really
08:29encouraging and positive in your listing
08:31using the right balance of talking to
08:35the job seeker so your background is in
08:38science and you really enjoy
08:39rollerskating in your free time and
08:41talking about the company so we
08:43stand for this Center to the balance
08:45between you statements and we statements
08:47can matter you know language like intact
08:51people love to talk about hard problems
08:53and tough challenges curiously we see
08:56patterns change over time so my favorite
08:58example of this is the phrase big data
09:00so a year and a half ago if you use the
09:03phrase big data in a tech job listing it
09:05was positive you know it was seen as
09:08compelling and cutting edge in June of
09:102015 it's not negative but it's totally
09:12neutral it's interesting I wanted to ask
09:15because if everybody sort of gloms onto
09:18these best practices how then does the
09:20signal versus the noise shift exactly
09:23mark marketing content as with any
09:25marketing content the patterns that work
09:27change as they'd get popular and get
09:29adopted and so one of the reasons we
09:31believe software is so interesting is a
09:33solution here is that it can kind of
09:35keep track at broad scale of what's
09:38actually happening right now in the
09:40market so you may have published a job
09:42listing that worked really well a year
09:45ago and probably how a lot of your
09:47listeners write their job listings as
09:48they go back to that one and then they
09:50try to edit it and tweak it a little bit
09:51and face exactly what happens right but
09:53it actually doesn't necessarily work
09:55because the the market has changed and
09:58so so there's a lot there
10:00well you ever I mean I'm just curious
10:01about this were you ever able to find or
10:03study associations between people's
10:06intent and outcomes and job listings so
10:08for example and one of the things that
10:09we've seen happen a lot is that people
10:10only become real about what they
10:13actually want out of a job description
10:14when they actually put words to paper
10:16and words have that power to sort of
10:18help discipline what you're looking for
10:20you might not even know what you're
10:21looking for until you write it down have
10:23you ever looked at anything around that
10:25or found heard interesting anecdotes
10:27around that given your work we have seen
10:30that listings tend to perform better
10:32when they are originally authored so you
10:36can see some degradation over time when
10:39people patch you know I take a little
10:41bit from this listing and a little bit
10:42from this one and I sort of stitch them
10:43together and it's probably because when
10:46you're originally authoring it you bring
10:47that coherent point it was even
10:48interesting so a little bit we're pretty
10:53seen that we also identify phrases that
10:56torpedo your listing right you know
10:58there are you know corporate sort of
11:00cliches and jargon buzzwords Vegas you
11:03know the one of the very common we call
11:06it a gateway term that kind of torpedoes
11:08our listings the word synergy oh I
11:10should torpedo and keep the content I
11:12don't care what it is but it's a gateway
11:14term because when people include synergy
11:15they're also significantly more likely
11:17to include you know value add and make
11:21it pop right kind of silly but they're
11:24all over the place and and it turns out
11:26every candidate of every demographic
11:28group hates them yeah so there's a lot
11:32of opportunity to improve editorial
11:37world we would call that jargon and
11:39sounds like we also call it jargon
11:43totally actually it's interesting
11:45because with words like that they're
11:48obviously in use because they're useful
11:50words and it's kind of sad because I
11:52mean synergy at some point what's
11:53probably a useful word so it's kind of
11:55interesting because over time with your
11:56corpus of data you'll be able to sort of
11:58map how people's language changes and I
12:01think of dictionaries is like these
12:02static instruments for capturing text
12:04these days it is kind of fascinating how
12:07language is changing in a way that we're
12:09able to track differently now thanks to
12:10online and software and it changes
12:12lexicography like yeah as a whole
12:14discipline it changes lexicography for
12:16sure I don't know that you could do it
12:17in a static way anymore right I totally
12:19internet has just exploded that right
12:22exactly is there so if big data is kind
12:25of neutral now is there a kind of job
12:27type or job description that's the
12:29celebrity of the job search right now
12:32yeah what word is sort of popping out
12:34that's really moving the needle for you
12:35guys or that you've observed there are
12:37there are several most of your listeners
12:40are probably intact it varies a lot by
12:42industry so at scale right now out scale
12:45is a very popular phrase that's popular
12:47here - yeah you know it is you don't you
12:49don't want to do things in use methods
12:52that are perceived to be manual or
12:53perceived to be limited in some way so
12:55at scale is one that shines in and it
12:59started intact but it spread to other
13:01other industries which is common that we
13:04see that one of my favorite examples
13:06that we spend a lot of time talking to
13:08HR people is turns out workforce
13:12analytics is no longer a good phrase to
13:14use you want to use people analytics so
13:17you can get these highly specific you
13:20know deep in an industry changes that if
13:23you're in the industry and you're on the
13:25cutting edge you probably know but if
13:27you're just to start up trying to hire
13:29your first analytics person you probably
13:32have no idea you don't have a deep
13:33background in in the industry right yeah
13:35so you've described it for job listings
13:37in real estate and and so this approach
13:40you you think can extend and in
13:42different directions you start with
13:43Kickstarter but what is it that it's
13:45doing and how do you like it seems a
13:47little bit magical I have to say that
13:48like I know that this is a job listing
13:51so therefore it's gonna have to do this
13:53but a real estate listing has to do
13:55something kind of different that's great
13:57a really good question so I you know
14:00this is this approach is as powerful as
14:03the data set that you have so if you
14:05want to understand a document type the
14:08very first thing you need to do is
14:09collect a lot of examples of the
14:11document type and that means you need
14:13the documents and you'll also need some
14:15information about their outcomes so you
14:18are publishing a Kickstarter project we
14:21want to know did you make money or not
14:22that's the signal for us you're
14:24publishing a job listing we want to know
14:26did you attract a lot of good people did
14:28you attract only men did you attract no
14:30one so you know for each document type
14:33that we take on the first thing we do is
14:35we make sure we build out a great
14:36training data set and then we apply
14:39really classical natural language
14:40processing techniques so we look for
14:42patterns until we say okay these are the
14:44ones that were successful we're
14:45successful is defined as you know
14:48attracted more applicants than 80% of
14:52similar listings maybe and then we start
14:55looking for the linguistic patterns and
14:57the successes the ones that aren't as
15:00successful ones that skew in a certain
15:02way demographically and then we play
15:05that back so sort of a key thing for us
15:07is that you get that feedback in real
15:09time as you're typing so as you're
15:13working on your document before you ever
15:14publish it forever pay to publish it
15:17somewhere you can make it good and so
15:19that is the sort of core of all of that
15:22because without that outcomes data then
15:24it's just someone's opinion I think
15:26could you extend that to say like look I
15:27want to write a screenplay for a
15:29blockbuster I mean does it can could you
15:32I mean people probably tried in fact a
15:35very prominent Bay Area CEO proposed to
15:38us a couple months ago that we start
15:39applying this to screenplays uh-huh to
15:42start writing to actually start
15:43producing content or just analyzing them
15:45sell it to Hollywood oh wow yeah so I
15:48think anytime you were writing content
15:50to sell something this is really
15:52interesting technology and you could be
15:54selling your company you could be
15:56selling yourself you're a job seeker
15:57with a resume that you want to have
15:58optimized you could be selling your
16:00product in a you know an e-commerce
16:02setup you could be marketing yourselves
16:04you could be marketing blast emails
16:06anytime you're writing content to get
16:07people to take an action this is really
16:10useful technology let's talk about where
16:12this fits and let's actually go let's
16:14purposely use some jargon here and let's
16:16talk about where it fits in the tech
16:17trends like where that fits in that
16:20space so it sounds like you're
16:22describing big data techniques applied
16:25to natural language or machine learning
16:27techniques applied to natural language
16:28but natural language has been around for
16:30over three decades thirty years I mean
16:32and in the early days they didn't have
16:35this this kind of corpus to train the
16:37algorithms aren't on obviously so they
16:39had to use different kinds of techniques
16:40like where does your work fit and how do
16:42you see how it fits in the evolution of
16:43natural language like how is that how
16:45has it been and where we are where are
16:46we now kind of yeah I mean I think in
16:50core natural language processing
16:52empirical strategies have always been
16:54really important so when I was a grad
16:56student years ago writing a dissertation
16:59collecting data was just a lot more work
17:01right so I had to go and record people
17:04in the field and I had to transcribe
17:06things me feels like ancient now
17:08actually but I actually finished my PhD
17:09twelve years ago it wasn't that ancient
17:12the fact that the Internet has codified
17:15everything over the last 15 or 20 years
17:18at least in in English in most western
17:20languages means that you have this ready
17:23set of corpora available for you the
17:27tricky part is collecting the text and
17:31the outcomes are the part that's hard
17:33finding the content is easy so you're
17:37just grabbing the difference between
17:37just analyzing something to be able to
17:39predict something using that text
17:40exactly when you analyze something you
17:42could say oh cool this word is really
17:44popular now that's an interesting fact
17:46might be valuable to someone to know it
17:49but it's different than saying this word
17:51is actually helping your document in
17:54some way what are some other scenarios
17:55where you could use sort of this natural
17:58language text analysis to make more
18:00predict interesting things yeah so
18:03people are really starting to think
18:04broadly about this we saw a New York
18:08City based company helping people
18:10optimize the sale of their New York City
18:12apartments recently using the right
18:15phrases we've seen people do things in
18:18health care that I think are really
18:19interesting it's not a known vertical to
18:21me but looking at the kind of notes that
18:25doctors take about a patient and
18:27predicting the patient's likelihood of
18:29having a major insurance incident over
18:31the next you know 12 to 15 months is
18:33really interesting things in actuarial
18:34science like I think anytime people are
18:37producing text which by the way in
18:39businesses whatever your business's text
18:41is actually the thing you produce the
18:43most of I believe that which any any
18:46industry and so people produce a lot of
18:48text it's meant to describe often what
18:51they think is going to happen and so I
18:53mean the field of opportunity is pretty
18:55big the techniques you're describing is
18:57it the same underlying technique apply
18:59to all different domains but do you have
19:00to also train each corpus on a different
19:02domain like they're special lasers
19:05inside language in each industry or are
19:07they are there also universals across
19:09all of them um that's a really good
19:12you don't know until you train is the
19:14short answer to the question so we have
19:18a set of NLP libraries that look for
19:21common attributes of text and we always
19:23start out any new vertical by turning
19:26them on the on the documents and seeing
19:27what happened so things like sentence
19:30length almost always interesting things
19:32like the density of verbs and adjectives
19:35almost always interesting document
19:38length almost always interesting but the
19:40specific phrases that matter and what it
19:42means to write a job listing is very
19:45than what it means to predict whether a
19:46patient is going to become ill right so
19:49this the specifics matter the goals
19:52matter so if it's a document that's
19:54intended for broad consumption it really
19:57probably shouldn't be longer than 600
19:59700 words if it's a stock prospectus
20:03where you're giving a company some
20:05information about how their stocks are
20:07likely to perform it's gonna be pages
20:09and pages and so you know the the
20:10specific benchmarks that you're looking
20:12for are often very vertical by vertical
20:15but the principles of the kinds of
20:17things you look for are pretty similar
20:19in the past it seemed like only really
20:22big companies could do this because they
20:23had like the type of computing hardware
20:25and power processing power to pull this
20:27off like what's changed that
20:29AWS is what has changed things right I
20:32mean cloud compute at scale and you know
20:34Google Cloud a now sure there's a lot of
20:35competitors now but AWS did this for
20:37startups I think and I say that not
20:40because I worked at Amazon before but
20:43but it actually is like for up for our
20:45team to set up the server infrastructure
20:48that we need is trivial you know so I
20:51think that that's the thing and if just
20:52the fact that there's so much text data
20:55encoded on the internet Google has
20:57democratized a lot of access to data and
21:01so that has helped to that's great
21:05did you guys I have to ask did you kind
21:08of put any Kickstarter projects up there
21:10yourselves just to give it a whirl we
21:13were asked this a lot during our
21:14fundraising we did we'll get pitch decks
21:17by the way one of the things I would go
21:20back to your question one of the things
21:23has been fascinating about having the
21:24beta out there in the world is the ways
21:26people are using it so of course they're
21:27using it for job listings but people are
21:29using it for everything like just a
21:31couple days ago I had a material science
21:33professor write to me saying I put all
21:35my course syllabi through I was it
21:37really like how did that work for you I
21:39can't imagine that that was a good
21:40result he's like oh I threw out all of
21:41the job parts I just looked at gender
21:43wow that was a component that I needed
21:45or what I was shooting when you say put
21:47it through like what happens I
21:49understand like I in my head I have this
21:51idea that I'm typing along and you know
21:53suggestions come flying at me but that's
21:55exactly what happened so there's a
21:56website and you paste or type in your
21:58content and as you're typing it's
22:00getting annotated and marked up for you
22:02with patterns suggestions things you
22:05might want to change scores and you can
22:07in the case of the syllabi right you can
22:09dial it up or down depending on what you
22:11want the outcome to be so in his case
22:14look I'm sort of tracking for gender
22:16bias or he was looking for a specific
22:19aspect of what we provide and that of
22:23course the product isn't tuned for what
22:25he wants but he still found that aspect
22:27to be applicable to what he was doing
22:29we're seeing people put marketing
22:30content through pitch that content
22:31through so to your question about did we
22:34initiate any Kickstarter campaigns we
22:36didn't because we weren't making sure
22:38guys would be genius at it we might be
22:40yes we've given a lot of advice to
22:42people like it's Charter Project since
22:43then but we didn't because we were
22:47making an enterprise product right and
22:50we're if we hadn't followed through on a
22:51Kickstarter product and then it got
22:53funded then we'd have to build it right
22:54so would you but we helped friends for
22:57sure that's great so would you find out
22:58about the pitch decks actually I'm
23:00totally intrigued by that obviously
23:01given who listens to our podcast
23:03I mean pitch decks pitch decks are not
23:06always highly text oriented right so
23:08great pitch decks don't include just
23:10your your text attributes but there are
23:12certainly things like length of your
23:15deck that matters slide titles end up
23:17mattering quite a bit because people are
23:18looking to see a certain style of
23:22content and us let's beta we both in any
23:25kind of meeting where some one person
23:27gets hung up on one word in a headline
23:29yeah we think we didn't go deep on pitch
23:32decks but we looked at as many as we
23:34could find as we were building our own
23:36pitch deck in our last round of funding
23:39and found some patterns in the synergy
23:43line of questioning whether it was there
23:44were their words or phrases you should
23:46never include in your pitch deck you
23:48know I don't know I don't know I guess
23:51there might not even actually be hey I
23:52wonder if there's there's never I guess
23:54I bet I bet there are we didn't identify
23:56them synergy is probably actually let's
24:00talk a little bit more about some of and
24:01maybe we should wrap up on this note
24:02let's talk a little bit more about some
24:04of your findings around Gen
24:05differences he said the material science
24:07professor tested his own syllabus which
24:09again I'm not sure that made sense like
24:11you said good there wasn't a reference
24:12corpus to I guess I guess there wasn't
24:15but when you have you know about tens of
24:18thousands of phrases that are lighting
24:20up and he's writing for science stem
24:23student population odds are good that
24:26there's gonna be some lexical rely so on
24:29some things they're just so describe
24:31some of your findings around job
24:32descriptions because that's given what
24:33your product focuses on right now in
24:35terms of gender differences and how
24:36people get things you picked up on that
24:39yeah so prior to us doing this there was
24:44some really strong qualitative research
24:46right that National Coalition of women
24:48and Technology the Clayman Institute
24:50here at Stanford they've done some
24:52really interesting qualitative work but
24:54the number of phrases that they
24:56identified was on the order of a couple
24:58hundred avoid rock star avoid ninja you
25:02know we want to hire more women hugie
25:04the interesting things for us first of
25:06all we've talked to a lot of industries
25:08outside of tech and so well in
25:09technology we want to hire more women
25:11when I talk to people who are hiring ICU
25:14nurses or elementary school teachers
25:15bias goes the other way and so it's very
25:18important to ensure that we don't judge
25:19we just forecast and let you make the
25:22right choices for your business right
25:25whatever you're optimizing for given
25:27wherever there's an indifference or
25:28imbalance so I will say we have
25:31validated much of the qualitative
25:33research which is good that there's you
25:35know some alignment on those points we
25:38have found cases where things are it's
25:42pretty subtle right so the difference
25:44between fast-paced environment and
25:47rapidly moving environment it's almost
25:49head-scratching ly tiny but
25:51statistically one of them draws many
25:55fewer women to apply which one is it by
25:57the way are you allowed to fast paced
25:59interesting fast pace so you see
26:02sometimes these very fine distinctions
26:06between terms that you can only kind of
26:08play out statistically the other thing I
26:10would say is that most individual terms
26:12aren't that egregious one way or the
26:13other we put a lot of effort into making
26:15something visual so that you could see
26:18so if you have you know one sort of male
26:22bias term or female bias charm you're
26:24probably not going to shift your
26:25applicant mix that much but if you have
26:26ten then you're gonna see more
26:29substantial impact on your applicant set
26:32interesting I feel like this kind of
26:33validates the natural language approach
26:36because in the past I was betting in
26:39general people tend to put too much
26:40stock only on numbers and not on words
26:43like your the whole point of what your
26:44work is but the second part of it is
26:46that even you know think about when I
26:47was back in grad school there was a lot
26:49of debates between qualitative and
26:51quantitative data and what was more
26:52valuable and obviously at the end of the
26:53day they're both valuable exactly but
26:56it's interesting because for the first
26:57time you're really bringing in
26:59quantitative quantitative techniques to
27:02something that was traditionally in the
27:03qualitative domain yeah conversation
27:06analysis yes it's true I mean so you
27:08know I've looked at in some of my prior
27:10research I've looked at some other
27:11document types also as text EO is
27:15the piece that actually really brought
27:17us into jobs was some work I did on
27:19performance reviews so I collected
27:22hundreds of performance reviews from men
27:25and women who work in technology I they
27:28were all voluntarily given which meant
27:30they were all good reviews which I was
27:32betting on that I was going to be
27:33comparing strong performers regardless
27:36regardless cuz you don't give you a
27:37review unless it's a good one mostly and
27:40I found really striking demographic
27:43differences and how men and women who
27:45were getting good reviews were described
27:48in the language that was used Wow the
27:51word abrasive which has been talked
27:53about since then ended up you know being
27:56used in seventeen out of a couple
27:59hundred women's reviews and zero times
28:01and in men's reviews right the sort of
28:04stereotypical like aggressive was used
28:07in a man's review with an expectation to
28:10be more of it and in women's reviews as
28:12a term of some judgement and so that was
28:15really interesting I looked more
28:18recently at resumes so I collected 1100
28:21resumes from men and women in technology
28:23about half of each and found for men and
28:26women who have very similar backgrounds
28:28very systematic differences in how
28:30present themselves in a resume which is
28:32really interesting how what was the
28:34different ID that difference play out so
28:36men's resumes were shorter they were
28:39much deeper into detail about what they
28:43actually produced and worked on that's
28:45kind of counterintuitive because you
28:46would think that shorter means you'd be
28:48less detailed but you're saying that
28:50they were shorter but more detail about
28:52specific things versus a blings women's
28:54resumes tended to tell a story they were
28:57written in prose they didn't use bullets
29:00nearly as much they included executive
29:02summary summaries they included detailed
29:04statements of their personal interests
29:06that were twice as long as what men
29:08tended to include so the women's resumes
29:11were stronger on narrative much later on
29:13detail the men's resumes were generally
29:16stronger on detail and later on
29:18narrative but one of those kinds of
29:21resumes gets flagged as positive much
29:24more frequently right in tact especially
29:26we look really for what did they deliver
29:28how quickly and tersely can they
29:30communicate and so as we started looking
29:32at some of these documents we realized
29:34that there was just fascinating
29:36opportunities on the job listing front
29:39because in these other important
29:41business documents we were seeing
29:42demographic differences play out that's
29:44fascinating were there any other sort of
29:46takeaways you have for people who are
29:47job seekers out there who want to
29:48optimize their resume based on what you
29:50discovered I mean the length and the
29:53narrative is an interesting point I mean
29:54does it matter by the way for an
29:56industry I know you said you did I
29:57looked at tech and resumes okay I bet
30:01some of the same findings apply in other
30:04STEM fields - I bet finance you would
30:07see some similar patterns we do see tech
30:10and finance pattern together quite a bit
30:12in in other document types that's
30:14interesting by the way that those two
30:16it's the quant numbers we love data we
30:21love rigor those are things that you
30:23know right we share yeah I don't know I
30:27mean I'm always very reluctant to tell
30:29somebody to change the way they tell
30:31their story because I think both of
30:32those styles are needed for companies
30:35you need people who can tell a customer
30:36story and you need people who can track
30:38the fine detail and so I guess I would
30:42prefer to tell the story the way you
30:44our and find the company that values it
30:46rather than great point change the way
30:49you tell your story you're right because
30:51that is actually key that's a really
30:53my one question though is this whole
30:55idea of like optimizing language how far
30:59does that go because at some point I do
31:00think that an optimized becomes average
31:03or bland or you know okay not to point
31:06fingers but I'm thinking of demand media
31:08for example I collect all this data and
31:10what people like or want or and then
31:12spew out on the other side something
31:14that nobody likes or nobody wants I love
31:17that question so it turns out that if
31:20everybody sounds the same no one stands
31:22out and the good writers continue to
31:24find a way to stand out and then that
31:26changes what works so the beauty of a
31:28learning system we use the example of
31:30big data before is that if everybody
31:33tries to glom onto the same patterns
31:34they're no longer effective someone is
31:38gonna figure out as with any marketer
31:39someone is gonna figure out how to do it
31:41better and they're gonna introduce the
31:43next pattern for success so that's how
31:46we see it what are some other ways that
31:48people are using what's out there right
31:49now that's been surprising to you so it
31:52was initially surprising to us that
31:54people were using our tool for anything
31:56other than job listings because we
31:58trained on job listings where data quant
32:01oriented company that's the promise we
32:03made to say this is gonna help you with
32:05job listings as people started trying
32:07new things we realized oh there's
32:10nothing doing quite like what we're
32:12doing so people want to test its limits
32:14and see what kinds of content they can
32:15put through some of the crazier things
32:19we've seen so we've seen resumes which
32:22kind of makes sense we've seen lots and
32:24lots of marketing content we've seen
32:26people putting their product
32:27descriptions through a toy company that
32:32recently removed gender labels from
32:35their children's toys put toy
32:37descriptions through to see if they were
32:38still flagging any gender language just
32:42the ninja toys I expect just the ninja
32:44toys right and you know so it's we're
32:48not trained on those documents but
32:49people are continuing to use a system
32:51for it because I think there's a hunger
32:53for that kind of experience and there's
32:55nothing tailored to what they need
32:57and so for us it has offered some
32:59insight into what people need and what
33:02we might do to support that will we be
33:04able will we ever be able to use your
33:06technology to ask ask questions like say
33:08hey I want to know X Y or Z based on all
33:10the things you've trained on I don't
33:13know if you'll be able to ask questions
33:14but we do think that you'll be able to
33:16use it to generate simple content so I
33:20think there's something fascinating in
33:22the hey answer these ten questions about
33:25yourself tell me where you'd like to
33:26work and I will make the resume that is
33:28most likely to get you a good screening
33:31interview and from there you're on your
33:32own you've got to be good right we're
33:33not gonna lie for you we're gonna tell
33:35your story but we're gonna tell it in
33:37the way that is most likely to get you
33:38that job at Google that you want or I'll
33:40get that loan down the road right I
33:43think conference calls for participation
33:46coming through and grant proposals and
33:50you know again people are trying to
33:52write to get a result we're seeing quite
33:53a bit of variation it's not the majority
33:56of the usage but it's you know a few
33:57percent all right well that was Kieran
34:00Snyder of text EO and another episode
34:03the 86 and Z podcast thank you everyone
![How To Succeed As A NEW & YOUNG Realtor [Deals Every Month + Luxury Listings]](/_next/image?url=https%3A%2F%2Fi.ytimg.com%2Fvi%2FrtrRghgTayc%2Fhqdefault.jpg&w=3840&q=75)
