Generative UI as a Complement to Gen AI

The GEEK Conf 2024 / 27:06

Watch this talk on YouTube

Transcript

32 paragraphs

This is an automatic transcript of the recording above. It is published in full and unedited, apart from correcting names the recogniser reliably mishears. It will contain mistakes.

00:00Everyone believes put your hands together very quickly and very loudly for our next speaker Tejas. TEJAS. HEY, THANK YOU SO MUCH. Hey everyone. How we feeling today? Thank you Carmen. This Carmen waved over everybody. The only person awake here. Let's uh let's maybe try that again. Again, how is everybody today? Yeah, maybe better. Thanks so much for having me. I'm really excited to talk to you about things today. Um the the conference is React Native X Modern Web. I'm here to talk to you about the modern website because the previous one was React Native, but the thing we have in common is AI, specifically GenAI, okay? Um it it's a hyped topic. I'm trying to not deal with a lot of hype today and I want to give you a balanced take on AI um and specifically AI AI engineering and convince you that you could be an AI engineer. Um if I have if I try to see how confident you are saying I I probably could be an AI engineer, can you show me? Okay, like four of you. Um hopefully at the end of this there will be more. I my whole thing is eliminating imposter syndrome and actually helping you do your best work and live your best life, okay? So, with that little bit of an introduction, my name is Tejas.

01:17That's pronounced like contagious. Don't worry, I'm not. Um and I've over the years I've had um the privilege of working at a number of different tech companies in some capacity as a consultant, as an employee, whatever it may be. And I got to learn learn from the very best. And I'm here to share not just things that I've actually things that I've learned, but things that I've learned from experts and really great people. Um some of whom are sitting here today. Today I'm a developer relations engineer at DataStax. Uh we do GenAI stuff. In fact, we make a lot of open source GenAI stuff, which is why I joined the company. I believe in open source and I believe in GenAI and I think we can really combine these to make some great things.

01:52Today we're going to do a detailed look at AI engineering. AI engineer I want to start by looking at this term specifically AI engineer. Because before coming here, I talked to a bunch of folks and I was like, "Hey, do you kind of like we did just now? Do you feel like you could be an AI engineer?" And and like exactly like in this room, maybe 1% or half a percent of people said, "Yes." And the rest were like, "Ah, I don't know. I I I don't know Python. I I I don't know linear algebra. I don't TensorFlow. I don't I don't know about math. I didn't go to university." And so, there was a lot of like uncertainty there. And I want to eliminate that today, okay? So, I asked people and they the big mistake that they made was they conflated AI engineering with machine learning engineering, because that's all we've known so far. And even worse with machine learning research.

02:40Research and engineering are fundamentally different things as is science and the application of science to solve problems, which is called engineering. So, some semantics are not crossing. So, let's just start by establishing a working definition for our time together about what an AI engineer is. And and for this, I'm not really the best person to to come up with such a definition, but there are people more qualified to do so. Better suited to do so because of their companies that they have and the positions that they've had. There's a great blog and newsletter called Latent Space by my friend Shawn, Shawn Wang. Some of you may know him as Swyx. And he wrote this article called Rise of the AI engineer. And in it, he makes the definition of AI engineer. What is AI engineer? What What is this? He uses this really great diagram in his blog post and it's really a foundational post about what makes AI engineering. And this diagram I feel like is foundational for what makes up AI engineering. On the very left or on the origin, you've got machine learning research. That's very different. It's science. It's doing things like training models and doing evaluations on those models. One layer up is machine learning engineering. This is doing inference on the models reliably and coming up with new synthetic data that you can use to fine-tune those models. But, you see this this big line down the middle, the

03:52dotted line, is the API. And really everything after the API, Shawn puts it in his article, is AI engineering. So, it's the interface between software full-stack engineering, as you can see at the far end of that, and machine learning research. So, what is AI engineering? It's communicating with an API, behind which is a machine learning model. Fundamentally, right? Um and then at the end of the spectrum, after AI engineering, where also you find things like front-end engineering, back-end engineering, DevOps engineering, you've got full-stack engineering, which combines these across the stack, okay? Um this works well, this mental model, because fundamentally as an AI engineer you're applying the technology that is AI to solve problems.

04:33Engineering is really just problem-solving. And so, what if you want to make it really crude and just hand-wavy, you could say that when you make a fetch request over JavaScript in a React Native application to an HTTP API where some large language model is hidden behind, that's AI engineering. Doing a fetch to ChatGPT to get answers to solve a problem, you could call that AI engineering. This is Shawn's blog post. Underneath it though is is a really important sort of corroboration. There's this quote, it says this, it says, "In numbers, there's probably going to be significantly more AI engineers than there are machine learning or large language model engineers. One can be quite successful in this role, and pay attention to this last part, one can be quite successful in this role without ever training anything."

05:20Okay, and this this should really speak to a lot of the people who are like, "Oh, I don't know anything about training models." You don't need to to be an AI engineer. And I think this is very important because who says this is Andrej Karpathy? Anyone know Karpathy like about his track record? He was the I believe it was the VP of AI at Tesla, responsible for self-driving, and one of the co-founders of OpenAI, the company behind ChatGPT. The co-founder, the pretty much invented ChatGPT. Also is a YouTuber, he's got a ton of free content, hours and hours long, about how to build your own GPT-2. So, a great resource. And really, if anyone knows anything about LLMs, it's definitely Andre. And this is what he says. So, he sort of agrees with this. And so, what is an AI engineer? Call an API, use a large language model behind the API to solve a problem, boom, you're an AI engineer, okay?

06:05This article also goes on to say that this will likely be the highest demand engineering job of the decade. And it sounds very hypey, but there are people working as AI engineers earning like $600,000 a year. The literally this happens. And so, where there's hype, there's money. And this is not like crypto money. No offense to web 3 people, but like it value has been shown time and time again in the land of GenAI through protein research that is showing real promise in the medical industry to learning I I've learned a ton through using tools like Perplexity and Claude. So, there is value being demonstrated. Let's now go a little bit further. We we've talked about AI engineering. Let's talk about AI.

06:45Um what is AI? Why is it hyped? There's really multiple kinds of AI, right? There's There's rule-based AI, which is Yes, the system can be artificially intelligent with a bunch of if elses. You as a programmer write the rules ahead of time. And like Pac-Man is a great example of this. You eat the little cherries and the ghosts turn blue and start avoiding you if you've ever played Pac-Man. That's AI. It's just rule-based, meaning developers wrote those rules and the computer simulates intelligence. That's totally AI. There's also predictive AI, which finds its roots in a paper published in 1906 by a mathematician named Andre Markov. You may have heard of Markov chains, right? These are the basis of neural neurolinguistic programming and large language models, but that's where the roots are. And what they do is they help you predict the next state based on the current state of a system prob- probabilistically. So, on your phone, above your keyboard, if you have these little word suggestions for the next word based on what you've written. And you if you I don't know if you try this, but you just tap the middle word like a bunch of times, you end up with a nonsense sentence. That's because that's a big Markov chain. And the problem with Markov chains is that they're limited by memory. They can only give you the next state based on the current state, not all the states before it. Okay, this is

07:59called the Markov property. But that's an example of predictive AI, which isn't very intelligent, but it still has its place. The hyped AI is gen AI, generative AI, or sometimes called gen AI. And this finds its roots in a 2017 paper published by Google. If you ask me, it's just my opinion, but I think it was maybe a mistake for Google specifically to publish this paper because now, you know, a bunch of companies are eating their lunch. In any case, it it's it's a good thing that they published it. And this paper is called attention is all you need, right? And it it models a way of taking input tokens and paying multi-headed attention mechanistically with programming to not just the one piece of state, but the surrounding tokens in parallel. The surrounding by a token, you could say it's equivalent to like one English word. So, you take a word, you take the surrounding words, and you model literally attention to context. And based on that, you give three, four, five, you give a set of output tokens. And this paper laid the foundations for what we know as ChatGPT.

08:57Okay, quick lesson. So, gen AI is the hyped thing because of the Transformers architecture outlined in attention is all you need, this 2017 paper by Google Brain Scientists. Gen AI has a bunch of problems. Do we know this? The it's not perfect, nor is it even good, frankly. Um Google started rolling out this feature called AI Overviews in the US. It hasn't made it to Europe yet, thank goodness. But it's it's in continental Europe, unfortunately. So, it's in it's in the UK. Where if you ask a question to Google, they they their big marketing thing is let Google do the Googling for you. And they give you a generated AI answer. But this thing is often very wrong. So, somebody Googled, "Hey, how many cigarettes should a pregnant woman smoke per day?"

09:37And the answer is zero, in case you're curious. The answer is not just for pregnant women, for everybody is zero. If you're a smoker, I understand the addiction, but maybe stop. Anyway, um the the Google generated answer said confidently that a pregnant woman should smoke between one and three cigarettes a day. Um how much glass should I eat per day? And it gives you like answers that are wrong. So, a big problem with GenAI is they hallucinate. They {quote} "unquote" see things that are not there, and we as human beings attribute like authority to them for for whatever reason, and then we take this too seriously. But this is a huge problem. You can get misinformation quite easily. Problem number two with GenAI is the length of context it can hold in a given thread of messages is limited. The biggest context window we have today is Gemini from Google. It's a large language model and a bunch of other things. They share the same name.

10:28Um and that has up to 2 million tokens, where a token is equivalent to an English word, okay? What is 2 million tokens? That's like the last 10 years of text messages of yours, which is reasonable, but it's just 10 years. Like if you want to build Jarvis for yourself that knows everything about you. 10 years is nothing depending on how old you are. Some of you are children and it's your whole life. Um but for most of us, you know, we we need more context. And number three, there's a knowledge cut off. So, with GPT-3.5, you remember that? The the first one of the first models, it it it its training data ends in September of 2023. So, if you're like, "Hey, should I take an umbrella tomorrow?"

11:06your GPT-3.5 is not going to help you, right? Hey, I I received a paycheck for 100,000 Japanese yen. What is that in euros? Your large language model is not going to help. It doesn't know today's exchange rate. It do- So, real-time information doesn't exist. These are some real problems with GenAI. There's some ways to solve these problems. Can anyone tell me what they are? Well, this is the real AI engineering work. So, there's There's ways to do this. Some people think, "Well, I take a model and I fine-tune it. The problem with that is it's way too expensive. It's way too expensive to get a good amount of data to fix the model. Then you need to actually run the training process, which requires hardware that is quite expensive. And then what they don't tell you is once you have a model as output, you need to run evaluations on that model. You need to like test it.

11:50Okay, is it going to give me the right inference? And this all of this takes a ton of time and a lot of money. There's a better way that's actually free if not super cheap. It's called rag. Rag stands for retrieval augmented generation. And it's it sounds fancy because we like to give big words and acronyms to things, but honestly, it's not. It just means you retrieve, so you get data from somewhere that is true in the case of the exchange rate experiment. You use some API and get today's exchange rate. Step one, use JavaScript fetch. Fetch today's exchange rate. Cool, you retrieve. Then you use that retrieved data to augment or change the generated output from a large language model. Okay, but how do you change its generated output? You literally just give it context. So instead of what is 100,000 Japanese yen in euros being your prompt, instead of that, you you pull the exchange rate, and then you give it as a text string.

12:44You say, "Hey, today Here is today's exchange rates." Then you add your prompt. Now convert 100,000 Japanese yen to euros. So you just add English context based on some real-time data, and then write the rest of your prompt. This is all rag is, okay? And it's really great because you tell it the truth, and then you say, "Okay, now using this truth, give me more data. Give me the answer I want." That's rag. And there's many ways you can go about this, but this solves all of those problems. It solves the problem of hallucination cuz you're giving it the truth. It solves the problem of finite context length because it doesn't need a lot of context. You choose, right? In in how much rag data you pull.

13:20And it does it surpasses the knowledge cut-off because you're like getting real-time data yourself. Rag has components. Typically, you need data, of course, like authoritative good real data. You need an embeddings model, which is a way it's a specialized machine learning model that takes natural language, so English, and turns each word into a list of numbers, and then compares the proximity of these numbers with other numbers. What what that means is like it takes words like hot dog and food and cake and turns them into coordinates on a big map. And you can say it puts like like-minded words like hot dog and food in this area, and then words like car and guitar and like car would be here, guitar would be here.

14:00It would map them into space, so it can do similarity search. So, you need an embeddings model to do that. It turns text into numbers based on the meaning. And then you need a database to store those numbers. That's a vector store, okay? These components typically make up a RAG pipeline, but I've talked to you for way too long. Let me just show you RAG in action, and then we'll look at how we can use this to build web experiences that help users a lot. So, as I said, we work it on a bunch of open source stuff at DataStax. Um we have this tool called LangFlow that we make. It is fully open source. And the cool thing about that is you can actually just run it locally and use it in a self-hosted way. So, I'm going to fire up LangFlow here, and LangFlow is going to run.

14:38And what we're going to see is I've got a project here called vector store RAG, okay? And this is what a RAG pipeline looks like. Let's walk through this a little bit. So, you have a chat input, that's what the user types, give me the exchange rate, right? You have an embeddings model, we just talked about that, text to numbers. You have a vector search engine, that's going to take the user's input and search for it across your database. Once you get data from the database, you break it down into chunks so that you can sort them by how close is what. And from that you engineer a RAG prompt. And this Look at this prompt, it's exactly what I told you. It's like this context comes from the database, and given this context, now answer the question from the user, right? That's just RAG, okay?

15:20Once I take this prompt, I give it to OpenAI. So, I give it to GPT-3.5 Turbo with my API key. And then finally, I get a chat output. This is a rag pipe. This is the most basic rag pipeline in the world. So, let's run it and see what happens. So, I go to the playground and I'll ask it for something real-time, something that GPT-3.5 Turbo cannot know. Uh who was who won the Oscar for best picture in 2024, right? It's not going to know this. Um for sure not. And so, the answer we're going to see from 3.5 Turbo depending on the internet if you're on Wi-Fi, maybe get off. Um was I don't know. We'll see. The nice thing is it says thinking. You see? But we shall see. Also, oh, there we go. I'm sorry, but I'm unable to provide real-time information as I do not have access to This is pretty standard. It doesn't have access to the internet. So, let's do rag and fix this. How can we do that? Well, we need an embeddings model, we need data, and we need a vector store. We have all of those things. So, we can get data from like the BFI, the British Film Institute. This website this web page has all the list of the Oscars, right?

16:31So, we just take this and and crawl this website, break it up into a list of numbers that is embeddings, and store them in a vector store, okay? How might we do that? Well, this is this is Astra. It's a database. And I'm going to create a collection or a table. If you work with databases, you think of tables, right? I'll create one. I'll call it I don't know, stuff. And I can say I want embeddings from open AI. So, I'm going to choose an embeddings model. Um you don't really need to know much about this. So, we choose an embeddings model, and I've created a collection. And here, I'm going to store the contents of that web page as lists of numbers, okay? As vectors. So, now I need some way to get the content of that British Film Institute website and store it here so I can search it, okay? This is also modeled in LangFlow. So, if I come to LangFlow, just if I scroll down a bit, there's a this is a um write flow, not a read flow. So, let's look at this a little bit. So, I have a URL and this URL is the British Film Institute. It's that thing I showed you, the the blog post.

17:28We take that webpage and we break it down into chunks of text where it's no more than 1,000 characters. So, the we just break it we we get the English text, we break it down into chunks. We then take those chunks and insert them into Astra DB, into the collection that I just created called stuff, okay? We also attach an embeddings model to convert each chunk of text into a list of numbers. Is that clear so far? Yeah? So, we're just going to store this webpage in my database. So, I'm just going to click play and it's going to crawl that website. And as you can see, it's it's inserting it into my Astra collection that I just made. And in, I don't know, a few seconds, it should we should have the data.

18:08There we go. So, it took about 10 seconds um and I think now if I go to Astra, and maybe reload the page, I should have just that webpage but as a bunch of text in there. So, take a look quick look. Yeah, so this is the contents of that page and it's just chunked text split into chunks, right? There's also vectors. You see this? This is the big list of numbers. And it's it's a very long list. Like if I go to VS Code, like look at this, it's bananas. That's just like a bunch of text from the website. But the cool thing with vector search is we can search how similar is the user's query, represented as numbers, to this, right? And we find the closest match. Anyway, so now that we've ingested this data, let's go back to the playground and I will just ask the same question again.

18:50So, who won the Oscar for I know what it's going to do now is search through that data set, search for the words best picture. Remember, we talked about embeddings, they take natural language and turn them into numbers. So, when I say best picture here, it's going to take this and search the database for best picture. Where what what's close to best picture? Get the result and use that to generate text from GPT-3.5. So, I'll click play and we'll wait anxiously again um for like 2 minutes. I swear it's it's not it's not my fault. It's the internet. And there we go. Oppen- Look, it's it's so terse. Oppenheimer won It just knows this now. And the cool thing is I could even be like, "Give me the highlights highlights of the Oscars 2024, right?" And previously, it wouldn't know anything. But now, because it has the data from my database, the real-time data, it can generate new text based on the context I provide with my rag flow. So, it And and like how how do you write highlights?

19:53There you go. Some highlights include Oppenheimer winning seven awards, Emma Stone won Best Actress for Poor Things All of it just This is completely new stuff created from my abstract stuff from a website. This is rag, okay? Um I can see that some of you are thinking like, "This is cool. Um it's great, but I want to kind of now build a chatbot." Please do not build a chatbot. I think chatbots are cheap. I think you like it's it's unimaginative. It's like we have plenty of You can do more. And this is what I'm here to tell you. I work with gen AI and chatbots are so 2022, okay? I think we can do better. So, the Vercel team Um I used to work at Vercel. The Vercel team is doing some really great work with their AI SDK. And they're bringing what's called generative UI. Well, I don't know if you know this, but large language models like GPT-3, actually GPT-3.5 and onwards, can call functions.

20:47So, like human beings, like if you ask me, "Hey Tejas, what is 274 * 9 + 7 / 17.1?" I I'd be like, "I have no idea, but let me go use a tool." So, I will pick up a calculator and do that. And I'll get the answer. LLMs now have the capability to recognize when they just can't and when they need to call a tool. And with with open AI's and with Claude and with a bunch of APIs now you could include tools. Hey, call this function if you don't know the answer. And with the Vercel AI SDK they make that very easy with developer experience. So what you could do is call a function that returns some value. You know what a function that returns a value is? It's a React component. Oh. So now generative UI is your LLM will call a function that actually returns JSX that then you render in the render cycle. You see now we're thinking beyond chatbots. So let me show you I built something because I was annoyed.

21:47Like Kitson was saying we build the best things when we're angry. You can tweet that. So um I went to Netflix, right? And I'm like I want to watch something. And it was Saturday night, this is my family. We we share a Netflix account. And I want to watch Oh gosh, is this code of conduct violation? I want to I want to watch something. So I went on Netflix and I searched for uh movies with a strong female lead, right? Why not? And Netflix doesn't have really good search. Like it searches for keyword substring matches instead of like what I want. Instead of my semantic what I want. Um I could search for How about How about this one? How about um like roman- romantic comedies in Europe, right? It's like yeah, I don't I I don't the UI just broke. I I don't know. So um not very good. So using rag, right? And gen UI. So function calling with an LLM to return React components, I created a little project to see what happens. So if we come to here, this is movies plus plus. This is something I created just as a proof of concept. NPM run dev. Let me make sure it's actually running. And it's using those things that I showed you, rag, okay? So localhost 3000. And I could just choose my prompt here, movies with a strong female lead, and now it functions sort of like a chatbot. So, it's it's not the best because it's like sure, it's giving me this list, but the kind of nice thing here is it shows me who the strong

23:12female lead is, which is pretty cool, right? Um but using GenUI, we can do better. What might this look like as UI? Um and now we call the function that gets the posters and return Whoa, look at the that returns um All right, yeah, thanks. Yeah. And and it's and it's interactive. Like it can respond to my mouse cursor. In fact, I can click on this and I can like go to the website and read more if I want. And it knows the movies with strong female leads. I could even since it's since we're just rendering UI, I could um you know, show me the trailer for Wonder Woman. Um why not? And it's going to like find out and just straight up like render a little video that will start Yeah, just like right here. I just watch the trailer. Why not? And and then I could go back to text if I want. Did people like What was the critical reception of Wonder Woman, right? Um and you know, what did people say? Did they like it? Did they not? And then it goes back to text. So, sort of picking the right modality for the job. This user experience is far superior, right?

24:20To like I just feel like watching something with Keanu Reeves. And and like you just you don't know, right? So, um GenUI, no chatbots. I'm I'm sort of using chat here to illustrate the point, but you can generate actual user interfaces with React, even React Native and AI. And this is I think is the future. Let's I have like 5 minutes left. Let's look at the code that does this and then we'll wrap up. So, what we have is a big AI component. This is really where the magic happens. Um where we have this function stream UI and I'm saying talk to GPT-4o from OpenAI. You could put any model in here. You could put Claude, you could put whatever, right? Um messages is exactly what it sounds like. It's an array of a role, in this case it's user or assistant or system or tool, and content, which is your text as a message. Then, in the case that we return text, you just return you can return a React Markdown component.

25:12But, when you need to call a tool, this is the real secret sauce. So, you have this configuration object called tools, and each of these, show trailer, create genuine, these are tools. They're functions. So, what does a tool have? Has a description, some parameters, these are arguments, and a generate function. And this this is the secret sauce. So, the LLM will call this function based on this description. Literally, when the user asks for a trailer, use this tool. And that's how it knew to call the function, okay? Once it's it calls the function, we get we connect to our Astra database, that's the one I just used. And even it's since it's a generator function with a star, I can yield. While it's loading, I'm like, show a spinner.

25:52Yield a spinner, but then continue to fetch. And then yield again, another spinner. And then finally, return a video player React component. That's all. It's a function that returns JSX. And you can use this with React to really transform the UI of your applications, okay? One final thing I want to show you is this is backed by LangFlow that we looked at earlier, because this thing you click this API button, um, and you actually get an HTTP API. So, you could host this on AWS, wherever you want, and that rag pipeline becomes just accessible over HTTP. All right? And so, this is some ways that you can actually combine GenAI and GenUI to not just build another chatbot, but build something your users are actually going to want. Um, not that they would don't want chatbots, but, you know, um, something that pushes the boundaries a little bit. All right. It's time to start wrapping up.

26:45Um, let's just summarize a little bit, and we'll go from there. GenAI is hyped for a reason. There's a lot of room at the table if you can call an API that an LLM is hidden behind, you can be an AI engineer, and I would invite you to the space. Thanks so much for having me. I can't wait to see what you're going to build. It's been a pleasure, Dev Geeks Con.

More talks

All 23 transcripts

Elsewhere

There is every talk I have given, all 80 of them, ConTejas Code, the podcast, and Fluent React, the O'Reilly book on how React works inside.