Tejas Kumar

SmashingConf New York 2024

SmashingConf New York 2024

Watch this talk on YouTube

Transcript

43 paragraphs

This is an automatic transcript of the recording above. It is published in full and unedited, apart from correcting names the recogniser reliably mishears. It will contain mistakes. More than one person speaks in this recording, so every paragraph is marked with whoever is talking, exactly as the captions name them.

00:00Generating the Future Tejas Kumar

00:03VITALY:So, please, round of applause for Tejas. Come on.

00:08TEJAS:Thank you. Thank you so much for having me. And hey, what an honor to just goof around on stage with Vitaly who I consider a dear friend. Hi, everyone, I'm Tejas, that's pronounced like contagious. Since the lockdowns are over, I can say that again. And over the years, I've had the pleasure of working at a number different companies or with a number of different companies in various capacities and just absolutely love what I do. It's been an honor to build on the web for the past 20 years with learning from some of the greatest people I can call friends. I'm a developer for GenAI at component DataStax. And we're here to talk about generating the future. If you read the abstract on the website, it won't be hard until you recognize, okay, this is an AI talk. But set a context with the notes you're going to have in the discussion. The general vibe around AI lately is one of AI fatigue. I don't know about you, anybody just tired of this AI stuff? Look. It's every hand in the room. It's not pleasant anymore. It's over-hyped because the hype leads to valuation which is leads to large funding runs which leads to money which leads to hyper-capitalism which leads to why are we doing this? Yeah. Thank you. What I want to do is have a balanced, nuanced discussion with the details of AI that give a deeper understanding and maybe a deeper appreciation and highlight where it makes sense to use. And to give a little bit

01:44TEJAS:more context. This was posted on X, formerly Twitter, October 7th, a dear friend, Stephanie, dear brands, companies, I do not want or need AI injected into every single part of my life. I'm very likely to use your product if not shoving AI down my throat. This is a well-reflected sentiment across the room. The thesis as we step into the discussion: AI and its features are like salt. You sprinkle a little bit in a meaningful way, it can amplify flavor, give taste. Use it respectfully, you can get results. pour sat on your food, you can get sick. That's how we want to use AI today. In my introduction, Vitaly called me an AI engineer. What is that? What does that even mean? I so far this year have spoken at 16 different conferences. And at every conference, including this one, I take the opportunity to ask people around me: Do you think you could be an AI engineer? Do you think you could identify as that? And most of the time, nine out of ten times, people will say, no, I don't think so. Then I say, why? Because I don't know linear algebra. A lot ofs this gate kept behind academia. I don't know how to train models. I don't know what fine-tuning is. Does that resonate with any of you if I ask you if you could be an AI engineer?

03:07TEJAS:Show me your hands. That's almost everyone again. The truth is, they're entirely different beasts, you don't need to know Python or machine learning. AI, machine learning, engineering are very different. I point to Swyx, Shawn Wang, who owns Latent Space. He defines what AI engineering is in probably the most clear way. It's this -- you see this diagram here? You have got data that's research constrained and products on the other end of the spectrum. What you can see is machine learning research comes before machine learning engineering, which is training models and so on. And you've got this big dividing line down the middle which is the API. And after the API, you've got where the AI engineer falls, right? And then after the AI engineer you've got the full stack engineer. What this means is that if you're a full stack engineer, then AI is but one of the tools -- one of the elements -- of your stack. And oftentimes, probably in the most crude top contour way to be an AI engineer is to interact with an API that abstracts on top of a machine learning model. It's not training a machine learning model. It is not doing research on machine learning. It is far from academic. It's just talking to an API. In the most crude sense, if you can call a function like fetch in JavaScript and talk to OpenAI, congratulations.

04:34TEJAS:You're an AI engineer. You might think, whoa, this is strange. But I want to draw your attention to the bottom of this page where there's this quote which says: In numbers, there's probably going to be significantly more AI engineers than there are machine learning or large language model engineers. One can be quite successful in this role -- pay attention to this last piece -- without ever training anything. And it's also important to note who says this. Right? This is Andrej Karpathy. For those who don't know him, former VP of self-driving at Tesla, co-founder of OpenAI. Chief scientist at OpenAI, not chief scientist, was head of AI or something. But he had an instrumental role in creating ChatGPT and the AI models. In fact is now a YouTuber who has courses on how to make GPT2 for free. Many hours. If anyone can make a claim, it's Andrej Karpathy.

05:25TEJAS:It's not just my opinion, it's Andrej's, but I run a podcast, talk to experts in AI, Harper Carroll, studied at Stanford University, you Don't need to know machine learning to know AI. Do you think you could hypothetically possibly be an AI engineer? I'm hoping this becomes more accessible. I see a lot of you nodding. I'm glad that lands. Let's talk about AI. What is AI? When Google does the keynote, and no disparity to algorithm, summarize it, AI, AI, AI -- 122 times. What does that actually mean? There is many types of artificial intelligence. If we just take the semantics of it, artificial intelligence, then we arrive at things that look intelligent. Period. What looks intelligent? Logic gates. If you've played Pac-Man, right? For the Pac-Man rule. And you eat the cherries. Then the ghosts that previously approached you to eat you avoid you. You turn blue. I don't know if you remember your Pac-Man lore. But that -- that's an example of rule-based AI. It simulated intelligence. These ghosts avoid you, they're intelligent. The only difference is, the rules of avoid Pac-Man or eat Pac-Man are known by the developers ahead of time. And the if statements that indeed is an example of rule-based artificial intelligence. Rule-based, the rules are known ahead of time. There's predictive models like the autocomplete on your phone. Which is a big Markov chain. That's a statistical model that calculates the next most likely state based

07:11TEJAS:on the current state. If you use the autocomplete, you have the word suggestions. Have you played the game where you tap the middle of one constantly? That's a big Markov chain. The problem with Markov chains is what's called the Markov property. Which is the memory list. Meaning you take the current word and predict the next word, but you don't have memory of previous words. Therefore you end up with nonsense sentences when you tap the keyboard repeatedly. It's predictive AI, but it's a different model. Deep learning is the hype. It literally models the brain through neural networks. We can go into a lot of detail here, but we don't have the time for it. You have an input layer, a picture of something containing a dog, for example. And there's hidden layers which literally just model neurons and the connections between them in the brain. And you have your output layer. The machine, the computer, algorithmically figuring out the rules. Not that a developer writes the rules, the machine figures out the rules. And you don't need to know this to be an AI engineer. But setting the basis for what AI is. Specifically generative AI. You have the networks, the input is simple. Input, bunch of hidden layers that allow algorithms to arrive at probabilities and then the output layer. Inside these layers are architectures. And there are many different kinds of architectures that simulate attention. The one behind the GPT models comes out

08:46TEJAS:of a paper published by Google titled "Attention is all you need." It's not a long or difficult paper. It's available on the Internet. I highly recommend reading it. What they talk about in the paper, mostly Google brain scientists, and Google brain scientists, they're not brain scientists, the thing is called Google Brain. They have found a way through software, programming, to algorithmically model attention. Not just attention, but multi-headed attention. Instead of Markov chains predicting the chain, they can have a word on an input token and the surrounding context. Not just one word in the context, but all words in all contexts in parallel. This is part of the training process. And this happens in the model architecture. This is called a Transformer's architecture, which gave birth to what is known as generative AI. This is the over-hyped thing that none of us love anymore. Generative AI or GenAI. The big question a lot of us are asking, including people asked me at the party last right, was do I have a future? Someone asked me this morning. Nate asked me, give me job security. I'm a UX designer, am I gonna be used? The truth of it, and what I'm gonna convince you of through the remainder of this presentation is absolutely you'll probably have job security if you're working in some type of frontend or frontend-adjacent technology like UX. And the reason is models are like cheap imitations of humans. Right? Like this is a

10:22TEJAS:person looking into a mirror and seeing something that's not them.S in what I feel like when I talk to ChatGPT. It's not human. There are many problems with generative AI. Can we think of some of them? And these problems I will submit to you later are by design. Problem number one, they're static in their knowledge. Like by knowledge, we mean things they have learned from large corpuses of text. If you think about a model like GPT-3.5 Turbo, GPT-4.0 learned its last thing as of April 2023. We're in October 2024. So, you can't get up to date information. Problem number two, hallucinations. We have Google AI overview here in the US. And you've probably seen the memes, right? Where people ask Google, hey, how many cigarettes should a pregnant woman smoke every day? The answer is zero, in case you're curious. Also, not just for pregnant women, but for everyone. But in any case Google responded to this with a pregnant woman should smoke three to five cigarettes per day. And the reason for this is because the training data contained jokes and satire. But, of course, the large language model doesn't know that. Gemini doesn't know that. And so you have these weird scenarios where you actually have dangerous misinformation. Problem number three, the length of context is finite. Gemini, it's the largest context window of all time, 2 million tokens. A token is equivalent to one English word. Have you thought about 2 million English words is?

11:59TEJAS:This is really nothing like with a personal assistant like Jarvis. As you approach the maximum of the context link, you start to get useless or nonsensical in information. Finally, the biggest problem is it's not human. We are highly fundamental irrational beings. That's a feature, not a bug. Our ability to be creative, and hallucinate, but in a didn't way than the machines and fall in love and tell stories, this is something that GenAI will never, ever be able to replicate. Even though we can now model vocal emotion. Some of you may have seen ChatGPT advanced voice, we can't go much further than that. I want to talk to you about AI engineering and a solution to all of these problems and look at how we can do better through quality UX and design work. The main thing I want to show you is a technique called RAG. Anyone familiar with R AG? RAG solves -- five of you. This is great. RAG solves all of the problems. And RAG is the reason, those who work on the frontend even as UX or designers are needed and will be needed.

13:10TEJAS:Because here's the thing, GPT-3 when it came out as a large language model, nobody cared. OpenAI shipped it, and then nothing happened. And then they put a frontend in front of it that looks like a chat interface and then it took over the world. But if it wasn't for the frontend engineering, the UX of this thing, ChatGPT would be nothing. And so, RAG is building on top of this frontend. RAG stands for retrieval augmented generation. Which just means you get some context that the model will not have, like today's weather. Like an exchange rate. Like we're looking at some examples. So, you get some -- you retrieve some up to date information past its knowledge cutoff that cannot be hallucinated. You can retrieve it from a reliable source. And then use that retrieved information to augment, AKA exchange, the generated output. Is that clear? At its core, it manifests in prompt engineering. You retrieve some real context and use it to change the output of the LLM just by putting it in your prompt. RAG has these elements up to date author day Ty data.

14:20TEJAS:And embeddings models, this is a specialized model that can take natural language and embed it in dimensional space. Let's do it right now. We have three dimensions, right? Mention one, two, three -- depth. And so, if we embed the meaning of a word in dimensional space, we can say words like cats and dogs are over here because they're close. And words like iPhone are over here because it's different than cats and dogs. So, we can put things in space. That's what these models do. Take natural language and put them in space. Number three, these lists of numbers, these embeddings are vectors. The mathematical vectors, big lists of numbers. Vector store, you need somewhere to store them. Put them together, you can create reliable RAG applications. Let me show you RAG in practice in about 20 minutes. Let's look at RAG in practice. Example number one, ChatGPT. ChatGPT doesn't know today's weather. But it we ask what's the -- what's the weather in Berlin? Then it will perform RAG. It's literally -- it's getting -- it's retrieving and using that retrieved data to augment results. This is RAG. ChatGPT itself needs RAG because the underlying model doesn't know. It has not been trained on this. That's a frontend UX feature that makes ChatGPT useful. That the model alone cannot do, okay? So, that's value right there. RAG example number two, I have a podcast. It's -- people love it. But every episode is very long. It's 2 hours. Long

15:54TEJAS:form. And people go, I don't have the time for this. I need to know about resource hints. I built a RAG-style podcast search where it's like, tell me about resource hints. Which is a performance primitive in web engineering. It will search every conversation, we discussed this with Dan, I don't know if you know who Dan Shappir is. >> I don't know how familiar you were with resource hints... TEJAS: And he just continues, right? So, that's RAG. You see the intersection here of UX and AI. This whole thing, shoutout Josh Comeau helped me with the UX of this, thank you. This whole thing, making the text shiny and seeing the click on it. And the whole sparkly emoji comes together for the UX in meaningful ways for users.

16:42TEJAS:And Luke, yesterday, I love that he alluded to this as well, and gives follow-up questions. This whole thing on Luke's Website is just a big RAG app. Vitaly mentioned Perplexity yesterday, I haven't can Googled the whole thing in so long. He mentioned, after the talk. This is Perplexity, where is SmashingConf New York 2024. It will search, find and summarize and synthesize the information, take place at New World Stages in Manhattan. And cites the sources because it searched the Internet. This is just RAG and UX plus AI. You get the address and the date. And the thing about Perplexity, you can choose any large language model you want. The model doesn't matter, it does matter a little bit to generate text. But what matters more is the frontend and the UX. I hope I have convinced you of this fact. Some of you might be thinking, okay, this is cool. But how do I build this? The whole I are do what I do is to make all this accessible to everyone. How to build a RAG, you can fetch. But I want to make it. Although DataStax, I want to show you how RAG works in prod. It's called Langflow. Of course I need to log in. Let me show you this thing. I think it's the clearest way to explain RAG to someone and RAG's intersection of prompt engineering and more. So, if I sign in, and what I'm gonna do, just make a -- I'm gonna go to -- I'm gonna switch organizations. This should have been done ahead of time. I'm so sorry. This

18:28TEJAS:is so low quality. This is probably the worst talk at the conference. Anyway... please tell Vitaly. Anyway, let's make a new project. It's a blank flow and we're gonna model RAG here. So, if you look at any chat application, you've got a chat input. You've got a large language model, use one from OpenAI. And you can use any, you can use Ollama, that's a way to run a large language model locally. If we had more time, we would get into that. This is your basic quintessential chat within LLM application. And chat with GPT-3.5 Turbo. I've wired it up, this is not RAG. This is talking to an AI. If I go, which movie won the Oscar for best picture in 2024, right? In theory, ChatGPT is not going to know this, it's past the training date. It says, I'm sorry, but I don't know. It's nice that it does this instead of making something up. But we can fix this with RAG. Using DataStax and database, I have loaded every Oscar winner into a database. What I'm going to do, between the chat and the large language model, introduce RAG steps. Go to the database, my -- store, and say, I want you to talk to this database with the Oscars collection. And I'm going to grab an embeddings model. We talked about an embeddings model, right? Which is just a way to convert natural language into numbers. I'm going to get OpenAI embeddings and connect this over here. And I'm gonna connect this chat input to the search input on my database. Now the database is

20:15TEJAS:going to send me a bunch of records. I need some way to parse this. Just grab the parsed data. I'm gonna take the data from the search results -- can you see that? It's gonna take the search results from my database and turn it into text, okay? Now that this is all turned into a text, I'm going to construct and engineer a prompt. So, how do we do that? We have prompts. I'll just grab a prompt. Just like that. Just like that. And there's nowhere I can plug this into. What I'm gonna do is this is the most quintessential RAG prompt you can think of. Given the context -- and this is the context -- answer the -- answer the question. And I'm creating variables in my prompt here just like that. And so, now I'm gonna check and save this. And now I've got these things I can plug into. So, context comes from the data base. So, I'm just gonna draw that line, and the input, the chat input goes as the question. Finally, I get a cohesive prompt that I passed a input. Okay.

21:18TEJAS:I've just created like a little RAG flow. What that means, I've just added a search step here, you know? And so, now, if I go and run that query again, it will actually read the data from my database, put that in the prompt, send it to OpenAI, and we have... look at that. Just like solved all the problems. Yeah, thank you. We solved all the problems of hallucination, context link, knowledge cutoff. We did it. And we could do more. Tell me more about the 2024 Oscars. And I want you to see that we've given a more human touch by curating the information we've given to the large language model. Suddenly it knows all about the Oscars. Oppenheimer got 7, best director for Nolan. It had no idea about any of this before. We're to the training a new model here. We're not touching the model at all. We're just giving it more context. Is that clear?

22:18TEJAS:This is RAG. This is what we saw in ChatGPT, in my website. This is the technique. And it's all modeled beautifully for you with this flow. These are the steps. So, you have an input, an embeddings model, a vector database. You parse this output, construct a prompt, and then give it to the LLM, right? That's it. I think, though, we can do a little bit better. Let me see. And so, what we can do even better is talk about UX. Because what I just showed you was just a big chatbot. And I don't know about you -- I'm tired about chatbots. This dude. I hate chatbots. And I think -- this is a UX conference. I beg for better UX than a chatbot. I'm like, what can we do that's better than a chatbot? I want to introduce you in the little time we have left to generative UI. Not AI, have you seen this? This is what I want to talk about. Instead of talking about, let me just show you. I don't know about you, but when I search for a movie to watch, I don't think about the exact title. Especially not after a long week. Not, yes, I want to watch Journey to the Center of the Earth. I'm not this kind of person. You know what I feel like? Watch a romantic comedy, maybe one set in Europe. Unfortunately, the Internet is not made for people like me. If I go on Netflix, which I love Netflix, to be fair. I literally type that query: Romantic comedies in Europe. Look at this. Your search did not have any matches and the first

24:04TEJAS:suggestion is try different keywords. And that's because until the age of generative AI, this was the best we could do. Keyword-matching algorithms. But now that we can Capture not just the words, but the semantic meaning, AKA, the subjective vibe of the word, we can match on sentiment using GenAI. We can do better. In fact, we do do better. My job is at DataStax is 50% research, 50% teaching. This is the teaching mode, by the way, hi. As part of the research, I thought we could do better so I built an app that is what I considered to be superior movie search. Called Movies++, it looks similar to Netflix. But it works differently. Try that again, romantic comedies in Europe. It's going to ask Langflow. It's going give me this, which is kind of lame.

25:01TEJAS:And still kind of looks like a chatbot. But this is like step zero. We can do better. And this is what better looks like. What might this look like as UI? And I'm gonna send that over. And you see this? This is generative UI. Absolutely brilliant. And it's -- it's not just some server-rendered non-interactive stuff. It's beautiful. It shows me with detail what I'm mousing over. In fact, I can click on it. and I'm taken to more details on the movie database. But I could go to a different page in my application. This is what I wanted Netflix to get me. I type in a search query and I get that. I get like UI, right? We could even -- we could do something maybe even better. We could say show me the trailer for Notting Hill. I don't know if I'm gonna like this movie. It's do that -- show me this is the actually trailer for Notting Hill. That's the trailer right here. I could say where could I watch movies? This is endless, right? Today. And like a literal map of where -- walk distance from like New World Stages where we are right now. Absolutely bananas. This is what I want. Sure, it still looks like a chatbot. But I want you to extrapolate that this is not going to be a user of your application typing these, but it's gonna be the developer. Does that make sense? Like if you go on Amazon.com, you have product recommendations, a picture, a name and a price. That can be generated with AI. It's not like the developers here's your user

26:49TEJAS:account and generate AI. And serve your users. This is possible through React server components. These are generated by the large language model, weird. And sent over the network. If you want to geek out about the technical details about that, we can. But I want to land with one final thing. This is actually very practical for user support. Every time I go to support on a website, and I'm given a chatbot, I hate it. The first thing is where is the human? That's talking to a human. Because they're not capable, but if they were, let's imagine. Shift context a little bit, imagine this is now a support bot. I could say something like I forgot my password. Okay? We're not doing a movie thing anymore, just a proof of concept. Now, instead of how to reset your password, I get a form like this. This is what I want and I just type in my new password and I click this and it's done. This is -- this is -- whoa. No. But this is utility, right? This is more than just a useless chatbot. In fact, it's a very useful chatbot, because it just as a user, I just type what I want, and I get it instantly. As a developer, I can use it to generate portions of my application. Let's quickly talk about how this is built, excuse me, and we'll land the plane. I want to show you some code. What I'll preface the code with is this: As a human being with a brain, I know the limits of my knowledge. Like if you ask me to stand up here and multiply eight prime

28:29TEJAS:numbers together and give you the result, I'm gonna be like, no. I don't know. But what I do know is that there is a tool available that I can reach for and use. A calculator. Large language models have this exact capability. They're able -- like as you saw when I queried GPT and asked for the Oscar winners before RAG, it said I have no access to that. As per my knowledge cutoff, I don't know. You can specify tools. Here a bunch of tools, use them when you feel like it. Believe it or not. And they can use them. And the tools are just functions in programming code. With that, I want to show you the tools that give us the movie posters and the trailers and all of that. To do so, let's use cursor. Why not? Anyone using cursor? I'm glad. Just one person? Nice. So, it's this file. This is the code that gives us that. So, what we're looking at here is the Vercel AI SDK. You can do this with OpenAI or whatever you want. I used to work at Vercel so I'm a little bit biased. This is what I want to draw your attention to: tools. You pass in a configuration object, tools. And you've got key value pairs and a single tool, for example, show trailer, or get movies -- is actually, let's do show map. It's the simplest one. You have got three program terse. A description, and the large language model reads this, therefore knows when to use the tool. When a user asks for a location, where to watch movies? Use this tool. It then looks at

30:01TEJAS:parameters object and you can specify whatever you want here. Something like this. Which is a Zod validation schema. And finally, you've got the generate function which is a generator, literally. The star means it's a generators and it can do some really great things because it can yield to the user when it's actually making a query. And generate is you can just return a component. That's it. With forgot password I think is really cool because it's the forgot password tool, the user forgot their password, use this. To generate, return a React component. This isn't coupled to React. If you want to use web components, if you want to use Vue, if you want to use Svelte, doesn't matter. A function that returns UI is just a component. And an LLM could call it. This is all the magic that's happening here. And let's look inside forgot password. What's inside forgot password? It's literally HTML. With Tailwind -- I'm sorry, I'm sorry, I know we hate Tailwind.

30:59TEJAS:I'll die on that hill, though. And it's just UI. It could be literally anything. It could be like, oh, poor baby! I don't know. And let's just try that. Let's say I forgot my password, you know? I don't want to show you, this is not like some video. Look at that, that's so cool. The cool thing about this is -- I should make the large language model say "Cry me a river." The cool thing about this, though, is that large language models, their output is usually non-deterministic, right? Like you send a prompt to a large language model, and even if it's the same prompt ten times, you get back ten different answers. It's not deterministic. With function calling and returning your UI components, you add a layer of determinism. You will always get back the same forgot password component, always. And the only thing that changes is the input properties meaning parameters if your program has props. And you bring it to the otherwise chaotic world of large language models. Let's summarize. We entered into a discussion on GenAI which is over-hyped and we're all tired. We're tired because it's overused. My intention today was manifold, one, I wanted to make it access to believe you. Show you this is a space you can play in without being a Python expert with a master's in machine learning. In fact, you don't need to know anything about machine learning. Call an API. Start there. I hope I have made that accessible to you. Number

32:34TEJAS:two, explained some of the problems in the space, hallucinations and how to get over them with RAG. It's a technique to bring real world context and human-verified context into the generated output of machines removing some of the nonsense that they can hallucinate. Number three -- number three is chatbots suck, frankly. And we can do better through generating actual UI components. Not in response to user queries like we saw with my movies thing, but even ahead of time. The developers can generate UI using a large language model and just serve it over the wire to users. I hope all that have has become very clear. I've got two minutes left. And I want to spend the last two minutes actually talking about some hype that -- I get the impression that we're all cooked.

33:29TEJAS:And this is -- look. Just humor me a little bit because like generating the future is the title of the talk. Let's generate some stuff because I was very motivated by the first talk where he made Wordle. Have you all heard of v0? No? V0.dev. Make me Wordle with pure HTML and CSS only using the pattern input attribute -- attribute -- attribute. The correct word is -- what's a five letter word? Input. And then just do it. And so, I send this prompt off. And this is absolutely bananas. This one actually does scare me. And this is not -- look at that. That's absolutely -- and it will do it. And then when it's done generating, it will tell you how to use it. Pure CSS, no JavaScript. At all. And it's using the pattern property here for validating exactly my word. It just -- I mean, there we go. Wordle clone. Except a single -- yeah. And so, it tells you what's wrong, definitely. It's not okay, but it does at the end -- it's an iterative process. So, we can say like change this and fix that. But it's kind of bananas that this just exists. Without any JavaScript as well. The validation doesn't work. Anyway, so this is something I would play with for hours and in the end it will probably fix it. But the fact that you can go from zero to this as rapidly kind of blows me mind. And this is something I'm paying attention to very closely. I think we're not fully cooked, so to speak. Because you can see

35:28TEJAS:that there's human intervention needed. But I think as I mentioned earlier, when it comes to techniques like RAG, we always need augmentation by humans. Therefore we will never be obsolete. And with that, I want to extend really a heartfelt thank you for your time and attention today. Thanks so much. VITALY: Oh, here we go. Always a path to travel. Thank you so much, Tejas. Join me for your conversation. I think for me it's really kind of impress of because you also see some companies doing this sort of layering. Where they're bringing in a human as a kind of part of a process.

36:13TEJAS:Yep. VITALY: Like, for example, in situations where you have machine-generated output. Like imagine somebody is asking a similar question to a ChatGPT. The first response is generated by AI. If it's a mission critical task, finance or something, it will be human. If it's represented for the type of queries. And then if it's vetted, basically, by the human, then this is a style of the type of answers that will be kind of showed to the user in the end. And I'm wondering at this point -- we're kind of hitting the Bolt bottleneck, if you like. Right? Are we? Because if you're looking at dense, complex applications. You need to include a person, a person or human can make mistakes. How do you see that? I'm really plugging that we need to -- TEJAS: Verify?

37:13VITALY:We're gonna need to be in a position where we're vetting what needs to be introduced, especially if it has implications. But how do we make it all work without overcomplicated, over-engineering the code or investing too much time and effort into human checks?

37:29TEJAS:Yeah. That's a good question. Human checks are actually starting to disappear in some fields already. For example, a big part of training a model is performing what's called evals. Where you have validation data that once you get a finished model -- actually during the training process, you can give some validation data -- this is data that the machine learning model has never seen and test what its inference is on this data. And this used to be done with what's called a human in the loop. Meaning you just watched the rate of errors go down over time. This is changing to where now there's a methodology called LLM as judge. Where before a human verifies that the validation is actually appropriate, a large language model will do that. So, think of it this way. You're trying to train a model and you have inputs and outputs. And the model generates outputs for inputs it's never seen. You have a step before a human looks at it where you take the outputs that are novel and give it to GPT-4. And ask if it looks good, and GPT-4 had pass or fail that. If it fails, then a human it requested. If it's not, it does not. There's a company called Open Pipe, it's fully LLM is judged and watching the need for human evals to go down significantly. That's one way, just to use an LLM. That's workflows like this. Zac, I forget his last name. He's gonna kill me. Anyway, we works by TikTok. He has stockbroker AIs, buy this, sell that. But before he gets

39:11TEJAS:pinged, lower class models like GPT-3.5 watching and making recommendations. Before he's pinged, send to the larger one, does this make sense? Okay, now pinged me. He gets pinged when it's the right time to buy, but vetted by layers of LLMs. VITALY: Right. Is it then the specific language model that's focused on that domain? Or just the general one?

39:36TEJAS:That's a really good question. For the vast majority of people, unfortunately, it's a large general model. But this is not generally the best. The better is to have smaller, less generalized models. I believe this was a discussion with Luke. Yeah, that's the future, 100%. Large language models are kind of useless at special -- also why you made the -- [Audio is cutting out a bit] -- specialization. So, large language models aren't very useful for specialized tasks. This is where fine-tuning your own special model comes into play. I showcased my podcast, that's 100% my own model. It's important to generate the right HTML with the shiny text. GPT-4 can do that maybe 50% of the time, but I needed a model that could do that 100% of the time. I created mine. It's a matter of cost. A smaller model costs way less to host and to perform inference on. Yeah, this is 100% the future. VITALY: There is a question on in our company we want to automate many of the existing processes and train AI on our huge knowledge base for financial history and transactions. How much time does it take these days to train a model well? And how do you know it's trained well? And how expensive is it, actually?

41:00TEJAS:That's a good three questions. VITALY: Yeah, three questions.

41:03TEJAS:I would say before you embark on training a model, it's better to see how far you can get with RAG. Because the models that exist are very capable, right? And the only thing missing, really, is access to quality information. There's a great book titled: Developing Data-intensive Applications where they really teach you that garbage in, garbage out. If you can get not garbage in, if you can get quality in through your prompt, then probably get some good inferences. I think you have to really, really, really approve beyond a shadow of a doubt that a large language model is bad given a prompt. Then start trading something. Trading something is obviously orders of magnitude in time and money. Time because the algorithms need like a long time to run, depending on your dataset. VITALY: Like months? Or?

41:56TEJAS:Depends on the number of parameters. We talked about model architecture, an input and output layer and hidden layers. The hidden layers have a neuron so to speak, that's a diameter to arrive although the right output. GPT-4 has a billion parameters, the more parameters, the more the training time. It's a fine-tune of GPT-4, if you have a small model, it's not gonna take long. My podcast model has 7 billion parameters, trains in literally hours on my Apple M1. It depends. But it's complicated in time, but also money. It's very expensive. One, because you need personnel costs. Like you need somebody to do the evaluations. They tell you that training a model is the full end-to-end process. It's not. Once you have a trained model, you need to verify that it works. And that's expensive in terms of personnel costs. Someone has to prompt it and make sure it's not gonna fail. Then the added risk of something inaccurate, right? So, it's way too risky and expensive to actually train something. So, for this question, I'd say find an open source model, run it locally and then just construct the best prompts of your life and this will probably get you very far. VITALY: Would you say that the crux of it all is becoming more this way of defining, designing good prompts?

43:17TEJAS:Yes. VITALY: Is like prompt engineering really becoming a sort of major part of what we do? Because I can imagine, say you have a good UI, instead of typing it all in, you have a couple of buttons. Click on one of the buttons, it has one of the prompts being sent out, that's it, really. So, you really have to be very careful about parameters they include, the templating for prompts and things like that.

43:40TEJAS:Yes. VITALY: So, this is really everything, then? Like not everything, but this is the -- probably the most important thing that you need to do when it comes to this communication tool. TEJAS: Garbage in, garbage out. You're asking me high-quality questions, so I can give you high-quality answers. If you ask me broad, bad question, then I'm not gonna -- it's the same. It just models interaction.

44:03VITALY:Well, there were a couple of people who asked, where can I get the Movies++ application, please, please? TEJAS: It's not an application, but it's on GitHub.com/movies_plus -- and that's the signs. VITALY: How do we choose the right existing models? TEJAS: It depends. This is the answer. It depends on cost. If I'm a billionaire, find the most expensive model and use it. But I'm not. I wish I was. But I would do is -- and also, actually not even if I was a billionaire. Because there's environmental considerations. The stuff is killing the planet. We need to talk about that. And so, I would -- thank you. So, I would start with the smallest -- smallest model, see how I go. What does see how I go mean? It means send a bunch of prompts in an automated framework. Not just me typing stuff. A proper benchmarking methodology, a system, see how it performs and scale up from there. There's considerations, do you want to share your data with OpenAI? OpenAI will offer you the chance to opt out of them seeing your data and reusing that for training. But it's hidden in settings and you may not find it. Unless you look for it, right? And so, what I tend to do, also to manage costs, but also privacy, there's a tool called OlLaMA. Which I believe it was founded by people who were cofounders of Docker. And it's like Docker, but for LLMs. So, it runs, it abstracts away the infrastructure required to run an LLM. How it works, it's a commandline tool,

45:45VITALY:OlLaMA run model name, open source, and it runs it on any hardware, and in the cloud, Kubernetes cluster, whatever. It can do inference, with LLaMA, Mistral 7B, it's an underrated model. If you want to -- there's a model Mistral 8.72B I use that. Open source models plus OlLaMA is a good place to start for privacy-centric things and not feeding the sort of hyper-capitalist machine.

46:22VITALY:Excellent. Maybe just one final question by Nathan. Of course, we have a lot more questions, but I'm maybe you'll be available for questions outside after -- well, after the next session. Nathan is wondering: Generative UI sounds very cool. But how do we ensure it generates good code that does not create all the problems that junior developers run into? Form validation, correct API validation, accessible components. Just to finish the question. Thinking about the scary people, how do we -- scary people -- TEJAS: Do you mean like Satan?

46:58VITALY:I'm not sure. But to make sure generative AI does not get exploited to create malicious code. TEJAS: Thank you for the feedback on my talk, I could have done a better job showing you. I mentioned it brings determinism to the large language models. The components are components I wrote. I showed you the code how it was a form, I wrote that component. What does the LLM actually do, it creates the input parameters or the prop it is you speak React. It creates the props for the component. If you think about a blog post. A post component, title, date, content, or children. The component itself, structurally, I wrote. But the props, the title, the date, the content, those input parameters are generated by the LLM. In terms of re-creating code that will frustrate your team, the LLM is not writing any code. It's just calling my function with different parameters. VITALY: Well, I think that you have a lot of passion for AI and you're very excited about what's happening next. Is there anything that you think or you wish we had in this AI world now? TEJAS: So much.

48:13VITALY:With minus one minute. TEJAS: I'm sorry, yeah. I live in -- I live in Germany -- so, when I came to the US, I spoke to some -- you know, they said where are you based? I said, Germany. You guys, you regulate, you don't innovate. That's what I said. It hurt me hear that, but I think we need to find the right balance between innovation and regulation around data privacy, data stewardship, environmental constraints and energy requirements. I wish we had that. I think the US is too much towards the side of innovate at all costs. And I think Europe is too much towards the side of we need to be safe and not do anything. If we all as a global economy could find the middle, that's what I wish we had. VITALY: Well, thank you so much for being here, Tejas. Thank you so much.

More talks

All 19 transcripts

Elsewhere

There is every talk I have given, all 69 of them, ConTejas Code, the podcast, and Fluent React, the O'Reilly book on how React works inside.