00:14 hey guys so I didn't know I was going to
00:16 be one of the keynote speakers so this
00:18 is probably going to be the most reduced
00:20 scope talk of today I'm talking about
00:23 hints and in particular I'm talking
00:26 about how pantic might be all you need
00:28 to build with language models in
00:30 particular I want to talk about
00:31 structured prompting which is the idea
00:33 that we can use object to Define what we
00:34 want back out rather than kind of
00:36 praying to the llm gods that the comma
00:38 is in the right place and the bracket
00:41 closed so everyone here basically kind
00:43 of knows or at least agrees that large
00:45 language models are kind of eating
00:48 software but what this really means in
00:50 production is 90% of the applications
00:52 you build are just ones when you're
00:53 asking a language model to Output Json
00:56 or some structured output that you're
00:57 parsing with a regular expression and
00:59 that experience is pretty
01:00 terrible and the reason this is the case
01:02 is because we really want language
01:04 models to be backwards compatible with
01:06 the existing software that we have you
01:08 know code gen works but a lot of the
01:10 systems we have today are systems that
01:13 change and so yeah the idea is that
01:15 although language models were introduced
01:17 to us to chat GPT most of us are
01:19 actually Building Systems and not chat
01:21 Bots we want to process input data
01:23 integrate with existing systems via apis
01:26 or schemas that we might not have
01:27 control over and so the goal for today
01:30 is effectively introduce open AI
01:32 function calling introduce pantic then
01:35 introduce instructor and Marvin as a
01:37 library to make you using pantic to
01:38 prompt language models are much easier
01:41 and what this gets us is uh you know
01:44 better validation makes your code a
01:46 little bit cleaner and then afterwards
01:47 I'll talk over some design patterns that
01:49 I've uncovered and some of the
01:50 applications that we
01:52 have um this is basically almost
01:55 everyone's experience here right like
01:57 you know Riley Goodside had a tweet
01:59 about asking to get Json out of Bard and
02:01 the only way you could do it was to
02:02 threaten to take a human life and that's
02:04 not code I really want to commit into my
02:06 repos and then when you do ask for Json
02:09 you know maybe it works today but maybe
02:10 tomorrow instead of getting Json you're
02:12 going to get like okay here you go
02:14 here's some Json and then again you kind
02:16 of pray that the Json parsed correctly
02:18 and I don't know if you noticed but here
02:20 user is a key for one query and username
02:22 is a key for another and you would not
02:25 really notice this unless you had like
02:26 good logging in place but really this
02:27 just not happening to begin with right
02:29 right like you shouldn't have to like
02:31 read the logs to figure out that the
02:33 passwords didn't match when you're
02:35 account and so what this means is our
02:37 prompts and our schemas and our outputs
02:39 are all strings we're kind of writing
02:41 code and text edit rather than an IDE
02:43 where you could you know get linting or
02:46 typechecking or syntax
02:49 highlighting and so open AI function
02:51 calls somewhat fix this right we get to
02:54 Define Json schema of the output that we
02:57 want and open AI will do a better job in
02:59 placing the Json somewhere that you can
03:03 out so instead of going from string to
03:06 string to string you get string to dict
03:08 to string and then you still have to
03:10 call Json loads and again you're kind of
03:12 praying that everything is in there and
03:13 a lot of this is kind of praying through
03:15 the LM Gods um on top of that like if
03:18 this code was committed to any repo I
03:20 was managing like I would be pissed
03:23 right complex data structures are
03:25 already difficult to Define and now
03:27 you're working with the dictionary of
03:29 Jon loads and that also feels very
03:31 unsafe cuz you get missing Keys missing
03:34 values and you get hallucinations and
03:36 maybe the keys are spelled wrong and
03:37 you're missing underscore and you get
03:39 all these issues and then you end up
03:41 writing code like this and this works
03:43 for like name and age and email then
03:45 you're checking if something is a bull
03:47 by parsing a string it gets really messy
03:49 and and what python has done to solve
03:53 pantic pantic is a library that do data
03:56 model validation very similar to data
03:57 classes it is powered by typ pints it is
04:01 has really great model and field
04:03 validation it has 70 million downloads a
04:06 month which means it's a library that
04:07 everyone can trust and use and know that
04:09 it's going to be maintained for a long
04:10 period of time and more importantly it
04:12 outputs Json schema which is how you
04:14 communicate with open AI function
04:16 calling and so the general idea is that
04:18 we can define an object like delivery
04:21 say that the time stamp is a date time
04:22 and the dimensions is a toule events and
04:25 even if you pass in a string as a
04:26 timestamp and a list of strings as tles
04:29 every everything is parsed out correctly
04:31 this is all the code we don't want it
04:32 right this is why there's 70 million
04:34 downloads more interestingly time stamp
04:37 and dimensions are now things that your
04:38 IDE is aware of they know the type of
04:40 that you get autocomplete and
04:42 spellchecking again just more bug-free
04:45 code and so this really want brings me
04:47 to the idea of structured prompting
04:49 because now your prompt isn't a you know
04:52 triple quoted string your prompt is
04:54 actual code that you can look at you can
04:58 review and every one has written a
05:00 function that returns a data structure
05:02 right everyone knows how to manage code
05:03 like this instead of doing the migration
05:05 of Json schemas in the onot examples you
05:08 know I've done database migrations I
05:10 know how some of these things work and
05:12 more importantly we can program this way
05:14 and so that's why I built a library
05:16 called instructor a while ago and the
05:17 idea here is just just to make open aai
05:19 function calling super
05:20 useful so the idea is you import
05:23 instructor you patch the completion API
05:26 uh debatable if this is the best idea
05:28 but ultimately Define your pantic object
05:31 you set that as the response model of
05:33 that create call and now you're
05:34 guaranteed that that response model is
05:37 the type of the entity that you extract
05:40 so again you get nice auto complete you
05:42 get type safety really
05:45 great I would also want to mention that
05:47 this only works for open AI function
05:49 calling if you want to use a more
05:51 comprehensive framework to do some of
05:52 this pantic work I think Marvin is a
05:54 really great uh library to try out uh
05:56 they they give you access to more uh
05:58 language models and more capabilities
06:03 response but the general idea here isn't
06:05 that this is going to make your Json
06:07 come out better right the idea is that
06:09 when you define objects you can Define
06:10 nested references you can Define methods
06:13 of the behavior of that object you can
06:15 return instances of that object instead
06:16 of dictionaries and you're going to
06:18 write cleaner code and code that's going
06:20 to be easier to maintain as they're
06:22 passed through different
06:24 systems and so here you have for example
06:26 a base model but you can add a method if
06:28 you want to you can Define the same
06:30 class but with an address key you can
06:32 then Define new classes like best friend
06:34 and friends which is a list of user
06:36 details like if I was to write this in
06:38 Json schema to make a post request it
06:40 would be very unmanageable but this
06:42 makes it a lot easier on top of that
06:44 when you have doc strings the doc
06:45 strings are now a part of that Json
06:47 scheme where that is sent to open
06:49 Ai and this is because the model now
06:51 represents both the prompt the data and
06:54 the behavior all in one right you want
06:56 good dock strings you good want you good
06:59 you want good field descriptors and it's
07:01 all part of the Json schema that you
07:02 send and now your code quality your
07:05 prompt quality your data quality are all
07:06 in sync there's this one thing you want
07:08 to manage and one thing you want to
07:09 review and what that really means is
07:11 that you need to have good variable
07:13 names good descriptions and good
07:14 documentation and this is something we
07:18 anyways you can also do some really cool
07:20 things with pantic without language
07:22 models for example you can define a
07:24 validator here I Define a function that
07:26 takes in a value I check that there's a
07:28 string that value and if it's not I
07:30 return a lowercase version of that CU
07:32 that just might be how I want to par
07:34 parse my data and when you construct
07:35 this object you get an error back out
07:37 right we're not going to fix it but we
07:39 get a validation error something where
07:41 we can catch reliably and understand but
07:43 then if you introduce language models
07:45 you can just import the llm validator
07:47 and now you can have something that says
07:49 like don't say mean things and then when
07:51 you construct an object that has
07:53 something that says that the meaning of
07:54 life is to evil and steal things you're
07:56 going to get an validation error and an
07:59 error message and this error message the
08:01 statement is objectable is actually
08:02 coming out of a language model API call
08:04 it's using instructor under the hood to
08:07 that but you know it's not enough to
08:09 actually just point out these errors you
08:10 also want to fix that and so the easy
08:12 way of doing that in instructor is to
08:16 retries right now what we do is we'll
08:18 append the the message that you had
08:20 before but then we can also capture all
08:22 the validations in one shot send it back
08:24 to the language model and try again
08:27 right but the idea here that this isn't
08:29 like prompt chain this is this isn't
08:30 constitutional AI here we just have
08:33 validation error handling and then
08:35 reasing and these are just separate
08:36 systems in code that we can manage if
08:39 you want something to be less than 10
08:40 characters there's a character count
08:42 validator if you want to make sure that
08:43 a name is in a database you can just add
08:45 a post request if you want to but this
08:47 is just classical code again this is the
08:49 backwards compatibility of language
08:51 models but we can also do a lot more
08:53 right uh structured prompts get you
08:55 structured outputs but ideally the
08:57 structure actually helps you structure
08:59 your thoughts so here's another example
09:02 uh it's really important for us to give
09:03 language models the ability to have an
09:05 escape hatch and say that it doesn't
09:07 know something or can't find something
09:09 and right now most people will say
09:10 something like return I don't know in
09:13 all caps check if I don't know all caps
09:16 in string right uh sometimes it doesn't
09:19 say that it's very difficult to manage
09:21 but here you see that I've defined user
09:23 details with an optional role that could
09:25 be none but the entity I want to extract
09:28 is just maybe user it has a result
09:30 that's maybe a user and then an error
09:32 and an error message and so I can write
09:34 code that looks like this I get this
09:36 object back out it's a little bit more
09:39 complicated but now I can kind of
09:41 program with language models in a way
09:42 that feels more like programming and
09:44 less like chaining for
09:48 right um we can also Define reusable
09:51 components here I've Define a work time
09:53 and a Leisure Time as both a Time range
09:56 and the time range has a start time and
09:58 an end time if I find that this is not
10:00 being parsed correctly what I could do
10:02 is actually add Chain of Thought
10:04 directly in the the time range component
10:07 now I have modularity in some in some of
10:10 how in some of these features and you
10:12 can imagine having a system where in
10:14 production you uh disable that Chain of
10:17 Thought Field and then in in in testing
10:19 you add that to figure out what's the
10:20 latency or performance
10:22 trade-offs you could also extract
10:24 arbitrary values right here I Define a
10:27 property called key and value and then I
10:28 want to extract list of properties right
10:31 you might want to add a prompt that says
10:32 make sure the keys are consistent over
10:33 those properties we can also add
10:35 validators to make sure that's the case
10:37 and then Reas when that's not the case
10:39 if I want you know only five properties
10:41 I could add an index to the property key
10:43 and just say well now count them out and
10:45 when you count to five stop and you're
10:47 going to get much more reliable
10:49 outputs uh some of the things that I
10:51 find really interesting with this kind
10:52 of method is prompting data structures
10:54 here I have user details age name as
10:57 before but now I Define an ID and a
10:59 friends array which is a list of IDs and
11:02 if you prompt it well enough you can
11:03 basically extract like a network out of
11:05 this data out of your
11:07 data so you know we've seen that
11:09 structured prompting kind of gives you
11:11 really useful components that you can
11:12 reuse and make modular um and the idea
11:15 again here is that we want to model both
11:17 the prompt the data and the behavior
11:19 here I haven't mentioned too many
11:20 methods that you could act on this
11:22 object but the idea is almost like you
11:24 know when we go from C to C++ the thing
11:26 we get is object oriented programming
11:28 and that makes a lot of things easier
11:29 and we've learned our lessons with
11:31 object oriented programming and so if we
11:32 do the right track uh I think we're
11:34 going to get a lot more productive
11:35 development out of these language models
11:37 and the second thing is that these
11:38 language models now can output data
11:40 structures right that you can like pull
11:42 up your old like lead code textbooks or
11:43 whatever and actually figure out how to
11:45 Traverse these graphs for example
11:47 process this data in a useful way and so
11:49 now they can represent you know
11:51 knowledge workflows and even plans that
11:53 you can just dispatch to a classical
11:55 computer uh computer system right you
11:57 can create the data that you want to
11:58 send to air flow rather than doing this
12:03 terminates and so now I think about 6
12:05 minutes so I'll go over some Advanced
12:07 applications um these are actually
12:09 fairly simple I have some more
12:10 documentation if you want to see that
12:11 later on but um let's go over some of
12:14 these examples so the first one is rag I
12:16 think when we first started out a lot of
12:18 these systems end up being systems where
12:19 we embed the user query make a vector
12:22 database search return the results and
12:24 then hope that those are good enough but
12:25 in in practice you might have multiple
12:27 backends to search from Maybe you want
12:29 to rewrite the user query maybe you want
12:30 to decompose that user query right if
12:33 you want to ask something like what what
12:34 was something that was recent you need
12:36 to have time filters and so you could
12:38 Define that as a data structure right
12:40 the search type is email or video search
12:42 has a title a query a before dat and a
12:45 type and then you can just implement the
12:47 execute method that says you know if
12:49 type his video do this if email do that
12:52 really simple and then what you want to
12:53 extract back out is multiple searches
12:55 that give me a list of search queries
12:57 and then you can write some like a in
12:59 coyota map across these
13:01 things and now because all that
13:03 prompting is embedded in the data
13:05 structure your prompt that you sent to
13:06 open AI is very simple your helpful
13:09 assistant segment the search queries and
13:11 then what you get back out is this
13:13 ability to just have an object that you
13:14 can program with in a way that you've
13:17 managed sort of like all your life right
13:21 straightforward but you can also do
13:22 something more interesting you can then
13:24 plan right before we talked about like
13:26 extracting a social network but you can
13:27 actually just produce the entire
13:29 dag here I had the same graph structure
13:32 right it's an ID a question and a list
13:34 of dependencies where I have a lot of
13:36 information in the description here and
13:38 that's basically the prompt and what I
13:40 want back out is a query
13:42 plan so now if you send it to a query
13:45 planner that says like you're a helpful
13:46 query planner like build out this query
13:48 you can ask something like what is the
13:49 difference in populations of Canada and
13:51 Jason's home country and then what you
13:52 can see is you know what like if I'm
13:55 good at Elite code I could query the
13:57 first two in parallel because there are
13:59 no dependencies and then wait for
14:01 dependency three to merge and then wait
14:03 for four to merge those two but this
14:05 requires one language model call and now
14:08 it's just traditional Rag and if you
14:09 have an IR system you get to skip this
14:14 queries you know an example that was
14:16 really popular on Twitter recently was
14:17 extracting knowledge graphs you know
14:19 same thing here here what I've done is
14:21 I've made sure that the data structure I
14:23 model is as close as possible to the
14:25 graph VI uh visualization API
14:29 what that gets me is really really
14:30 simple code that does basically the
14:34 creation and visualization of a graph I
14:36 just Define things one to one to the API
14:39 and now what I can do is if I ask for
14:40 something that's very simple like you
14:42 know give me the description of quantum
14:44 mechanics you can get a graph out right
14:47 that's basically in like 40 lines of
14:49 code because what you've done is you've
14:51 modeled the data structure graph is
14:52 needs to make the visualization and we
14:55 we're kind of try to couple that a lot
14:57 more this is a more advanced example so
15:00 don't feel bad if you can't follow this
15:01 one but here what I've done is I've done
15:04 a question answer is a question and an
15:06 answer and the answer is a list of facts
15:09 and what a fact is is it's a fact as a
15:11 statement and a substring quote from the
15:13 original text I want multiple quotes as
15:16 a substring of the original text and
15:18 then what my validators do is it says
15:21 you know what for every quote you give
15:22 me validate that it exists in the text
15:25 Chunk if it's not there throw out the
15:27 fact and then the validator for question
15:30 answer says only show me facts that have
15:32 at least one substring quote from the
15:33 original document so now I'm trying to
15:35 encapsulate some of the business logic
15:37 of not hallucinating not by asking it to
15:40 not hallucinate but actually trying to
15:42 figure out like what is the like par
15:44 like the paraphrasing detection
15:46 algorithms to to identify that what the
15:48 quotes were and what this means is
15:50 instead of being able to say that the
15:52 answer was in page seven you can say the
15:54 answer was this sentence that sentence
15:56 and something else and I know they exist
16:00 junks and so I think what we end up
16:02 finding is that uh as language models
16:06 get more interesting and more capable
16:08 we're only going to be limited in the
16:10 creativity that we can have to actually
16:11 prompt these things right like you can
16:13 have instructions uh per object you can
16:17 have like recursive structures right it
16:20 it it goes into domain modeling more
16:23 than it goes to prompt engineering and
16:24 again now we can use the code that we've
16:26 always used if you want more examples I
16:29 have a bunch of examples here on
16:30 different kinds of applications that
16:31 I've had with some of my Consulting
16:33 clients um yeah I think these are some
16:35 really useful ones and I'll go to the
16:39 is this doesn't have the p uh the QR
16:44 fine the updated slide has a QR code but
16:46 instead you can just visit jxl github.io
16:50 instructor I also want to call out that
16:52 uh we're also experimenting with a lot
16:53 of different uis to do this structured
16:55 evaluation right where um you might want
16:59 to figure out whether or not one
17:00 response was mean but you also want to
17:02 figure out what the distribution of
17:04 floats was for a different attribute and
17:06 be able to write evals against that and
17:08 I think there's a lot of really
17:09 interesting open work to be done right
17:11 like right now we're doing very simple
17:12 things around extracting graphs out of
17:14 documents you can imagine a world where
17:16 we have multimodal in which case you
17:18 could be extracting bounding boxes right
17:20 like one application I'm really excited
17:22 about is being able to say give an image
17:24 draw the bounding box for every image
17:26 and the search query I would need to go
17:28 on Amazon to buy this product and then
17:30 you can really instantly build a UI that
17:32 just says you know for every Bounty box
17:34 render a modal right you can have like
17:36 generative UI over images over audio I
17:39 think in general it's going to be a very
17:41 exciting space to play more with uh
17:42 structured outputs thank
