“So most of the world is made up of full stack application developers who build software for anything from HRIS to accounting, to supply chain, and what have you?   For those people to easily add generative AI capabilities into their applications while remaining in compliance with the security, trust, and safety requirements that enterprises have, well that’s a fairly difficult challenge.”

Vivek Sriram– Co-founder of Bookend.ai

Summary:

In this episode of Science 4-Hire, I welcome my old friend and partner in crime Vivek Sriram, co-founder of Bookend AI, a start up that provides secure infrastructure that supports the efficient spin up and fine tuning of open source LLMs

We waste no time delving into the fascinating, confusing, and intricate world of Large Language Models (LLMs) and their burgeoning role within enterprise solutions, with a special focus on HR and hiring applications.  We sure do agree on the transformative potential of LLMs to revolutionize enterprise software, enhancing functionalities such as candidate screening, resume parsing, and even generating interview questions—tasks pivotal to modern HR departments.

Vivek brings me back down to earth a bit as he provides words of caution about the considerable challenges that come with integrating LLMs into enterprise systems, especially within the HR sector. Concerns around data privacy, the risk of perpetuating biases, and maintaining compliance with labor laws are significant when deploying AI in hiring. Vivek emphasizes the critical need for enterprises to navigate these challenges carefully, ensuring that LLM integration respects ethical guidelines and regulatory requirements, thus preventing potential adverse impacts on candidates and the hiring process.

The good news is that Vivek outlines strategies for implementing LLMs in a manner that balances innovation with responsibility. Approaches such as utilizing open-source models for greater control and customization, and employing platforms that offer secure, compliant AI integration, are discussed as viable solutions.  The idea of fine-tuning LLMs with proprietary data to better align with specific HR needs provides additional levels of confidence for those looking to use LLMs securely.  

Listeners will gain a nuanced understanding of the dual-edged nature of LLMs in HR and hiring contexts—recognizing their potential to significantly improve efficiency and decision-making in talent acquisition, while also grappling with the ethical, privacy, and compliance issues inherent in their use. This episode underscores the importance of thoughtful AI integration in HR practices, aiming for a future where technology serves to augment human judgment rather than supplant it.

Take Aways:

• We must recognize that integrating open-source LLMs into enterprise applications comes with complex challenges, including navigating licensing, data usage permissions, output control, and auditing requirements.

• Enterprises must address strict compliance and security standards, especially in regulated industries, when implementing open-source LLMs. This includes ensuring data privacy, adhering to industry-specific regulations, and maintaining the integrity of sensitive information.

• Tailoring open-source LLMs to specific enterprise needs requires significant customization. Enterprises need to modify these models to align with their unique operational, compliance, and ethical standards.

• Potential solutions to the issues inherent with the use of LLMs involve employing strategies for effective management of open-source LLMs, include:

• • Selective Model Adoption: Carefully selecting open-source models that best fit the enterprise’s technical and compliance requirements.

• • Data Management and Security: Implementing robust data management practices to ensure that the use of LLMs complies with data privacy laws and enterprise security policies.

• • Model Customization and Fine-Tuning: Customizing and fine-tuning open-source LLMs with enterprise-specific data to improve relevance and performance while adhering to ethical guidelines.

• It is essential to stay informed about evolving regulatory landscapes related to AI and machine learning technologies to ensure ongoing compliance and adapt strategies as needed.

Full transcript:

Speaker 0: Welcome to Science for Hyre. With your host doctor Charles Handler. Science for  Hire provides thirty minutes of enlightenment on best practices and news from the front lines of  the improvement testing universe. 

Speaker 1: Hello, and welcome to the latest edition of Science for Hire I your host doctor  Charles Handler. And I have a guest today. A guest back in the day, we’ve known each other  probably like twenty five years. I don’t know, be sitting around having a beer twenty five years  ago and said, you wanna be on my podcast about generative AI. You would have thought I’ve  probably been doing a lot of shots because nobody would know what the hell I’m talking about.  So anyway, my guests today, a good old friend and someone now I’m excited to be collaborate in  a little bit with a completely different field as me, which is awesome and hopefully expose our  listeners to some new ideas and thoughts. About large language models and all that fun stuff. So  my guess, Vivek Shiram, who is a cofounder of Bookend, AI welcome. 

Speaker 2: Well, thanks for having me, Charles. It’s a it’s a pleasure to be on your podcast. 

Speaker 1: Yeah. So I always let the audience I don’t think I’m lazy, but you know, kind of part  of me always says, well, are you lazy? And this is why you always have your guests introduce  themselves. But it’s really because who knows you better than you? So just tell tell our audience  here, you know, who you are, what you do.  

Man, you’ve had a pretty illustrious history, I will say, before I interrupt you. I’ll interrupt myself.  You know, I lived in San Francisco around two thousand. I came out there to be part of the I call  it the first Internet revolution there, and it was pretty exciting times. I went out there to work for  the first company putting testing online, but I hadn’t known Vivek from LSU.  He’s more in the realm of of IT development, exciting stuff like that he’ll tell us about. And so we  were we were you know, hanging out a lot back then. It was really interesting. Again, it it the  world at that time, twenty three years ago or whatever, it you know, we were we were really like,  wow, this is such new stuff. It’s gonna change everything, which did it seems very flat and  simplistic compared to where we are now.  

So anyway 

Speaker 2: All all that is true. I mean, you know, I’ve I’ve known you for probably more than  half my life now going going back to college from from from LSU. And you’re right. Twenty five  years ago, I’ve been sitting around drinking beer, playing pool, you know, whatever trouble we’re  getting up to. I don’t think any of us had any inkling about about podcasts or generated AI or any  of this stuff.  

Exactly. It is it certainly is a new world. Quickly by way of background, I’ve been in enterprise  software, basically my entire professional career. Which also runs about twenty five years now.  Most recently, I was chief before starting Bookend a year ago, I was the chief product officer for  a company called LucidWorks.  

We LucidWorks makes an AI powered search engine, which powers a large number of consumer  facing websites. If you ever go to Costco and, you know, look for a five gallon jar mayonnaise.  You put stuff in the search box, you know, that’s that’s what Musa Gorax does. Before that, I built  a couple of search engines at Amazon Web Services also to been in and around commercializing  various kinds of things whether we called it AI or not. The idea behind bookend was that we  wanted to make it easy for for non AI native developers. 

So most of the world is made up of full stack application developers who build software for  anything from HRIS to accounting, supply chain. What have you? For those people to easily add  generative AI capabilities into their applications while remaining in compliance with the security,  trust, and safety requirements that enterprises have, that’s a fairly difficult challenge. That’s that’s  kind of what we’re trying to solve at Booking. The rest of the team, just real quick, is is is also  made up of a number of people.  

They’re ten we just had ten people, so we’re not really big. But everybody’s got quite a bit of  experience in building managed services at companies like Google, Amazon, Dropbox, IBM,  etcetera. So that’s that’s that’s what we’re up to. 

Speaker 1: Cool. Yeah. And, you know, why we’re here today, really, it’s it’s funny because we’ve  been friends a while. And when I was in San Fran, you know, we talked about a lot of times,  well, how can we work together? What where do our things intersect?  

They never really did. And that’s okay because that doesn’t happen with a lot of my friends. But  on a recent trip to San Francisco. I think, again, beer is involved and a lot of bullshitting is  involved. But but Vivek’s like, you know, I think there there might be something here that would  be valuable to you.  

And, you know, we’re always doing that and hadn’t really found it yet, but I don’t think there’s  much of a force fit here. I kinda heard what what bookend is doing and looking at my own self.  And I think this applies to any industry, any person or company who is really looking to take  advantage of this, you know, revolutionary times that we have now in the tools and technology  that really seem otherworldly, quite honestly, but are real in in terms of what they’re doing in in  advancing their fields and, you know, I’m really thinking a lot about this stuff lately. And once I  heard a little bit about, you know, what y’all do, I’m like, wow. Okay.  

That makes a lot of sense as a foundational building block of a product or a program or whatever  in a lot of different ways and haven’t learned more about it, you know, it’s it’s exciting to to think  about working with you all a little bit in the in the HR technology space. And as I’ve done that, as  

I’ve gotten kind of more knowledgeable about your product. It’s it’s really opened up my mind.  Like, I’ve been thinking small ball, I still feel like as as far as, like, We can use generative AI to  do x, y, or z. But you talk to people who are building out architecture to support it and you’re just  listening to some of the use cases that you all had.  

I’m like, wow. Okay. There’s a lot more here. It seems like you are helping democratize some  hard things. So if you had to give your I don’t know.  

I would say elevator pitch, but we have more time than that. So, you know, even elevators to  really tall buildings like the Empire State building, they only take, like, thirty seconds these days.  So we have more time than that. So what are you guys all about? You know?  Give me a sales pitch. Give our audience a sales pitch. It’s okay. To save space here 

Speaker 2: for that? Yeah. I’ll I’ll I’ll I’ll give you the the the simple version of it. By now, I  mean, everybody’s familiar with with chat, GPT, and open all out of everybody. Virtually  everybody’s gone around and and played with it.  

And, you know, our image generators or or what have you. You know, that’s a that’s a a a closed  source piece of software. Right? So you can go to open AI. You can you can use it either for  enterprise.  

Purposes or for just playing around with it on on tragedy, Peter, or what have you. But it’s a it’s a  effectively, it’s a black box. You’re you’re gonna you’re gonna get the service Open AI dictates 

that that the terms of how you use it, and then you can if if your use case fits within those  circumstances, then Right. For a a lot of enterprise use cases that that may not necessarily be the  best fit. Their options they have other you know, one of the primary ones is is taking an open  source model, a pre trained model that can do any number of things from text summarization or  image generation or, you know, natural language processing or what have you.  And then using that inside of their own use case. The challenge with that is that the the most  popular place to go and get that from is a is a a website called hugging face. Your maybe your  listeners are familiar with that as well. I’m gonna just go there right now and go and see. I think  hugging face has four hundred and twenty three thousand seven hundred and seventy three  models available open source models available. 

Speaker 1: That’s not many. 

Speaker 2: That’s not many at all. So, you know, most people I’m sure can keep that in their  minds and and just know exactly what to do. The problem is that it’s difficult to know where to  get started. And enterprises typically have a bunch of rules and regulations about what’s allowed  to be done and what’s not allowed to be done. So licensing, what data you’re allowed to use into  it?  

How do you control the outputs? The auditing that goes along with it? Who’s allowed access to  it? All of those kinds of things are are frequently a a difficult set of challenges. So imagine, you  know, if you wanna go and take Meta’s open source model, Lama two.  

And and you wanted to go and and build a generative AI application in HR. That parses resumes  and identifies relevant skills. Right? So simple usage. Yep.  

And and you’re at a and you’re doing this in in the context working in a in a in a financial  institution, in a large bank. 

Speaker 1: Right? 

Speaker 2: You’re gonna have all kinds of of restrictions about what you can and cannot do with  it. What Bookend does is to offer a platform for those developers that make it a little bit more  risk free to take an open source model and run it, sanitize it, run it, use it kind of in the context of  an enterprise. This isn’t different from how enterprises adopt to open source technologies in any  number of other things, databases, search engines, web servers, what have you. In all of those  cases, you know, you take an open source piece of software, license it, sanitize it, benchmark it,  and make it available so that the IT leaders who are responsible for the running and maintenance  of enterprise software can do so with a controlled amount of risk and and know how about what’s  actually going on. So that’s that’s basically what Bookend does. 

Speaker 1: Gotcha. So one thing that, you know, I’m learning a lot because I I get general ideas.  Like I said, I’m teaching myself as much about this stuff, which right now I’m probably in  kindergarten, maybe first grade, you know, some some parts of it I’ll never get because I I don’t  need to, but which is the whole point of some of this stuff like GPT. But, you know, what I’m  curious about is I I kept thinking that conceptualizing what you all do that there’s there’s like a  outbound reach to say, chat GTP, let’s say. Right?  

Because I you can’t take or maybe you can, like, you can’t take chatGTP and put a copy of it on  your own server. Like, you have to reach out to it, right, and then get a turn, right, which is what  exposes all your stuff. But on the hugging face site, are there ones that you can just like say, I’m 

gonna download this thing and put it on my own server? Right? So do you conceptually serve  both those use cases?  

Do you have enterprise software or enterprise applications that people are building, you know,  internally? That would need to reach out to say, chat GTP. Maybe there’s a a late a title for  something like a chat GTP that’s Maybe it’s not even open source and the open source ones are  the one on hugging face. So explain a little bit about the difference between a model like chat g g  p or Bard or what’s the one my Microsoft has, which is now embedded in every thing. You know,  you 

Speaker 2: know Yeah. Microsoft uses open AI, which is, you know, Jeff’s right. Speaker 1: That’s right. 

Speaker 2: That’s part. Right. Yeah. So Bard is Google’s version of it. They’re all farmer models.  They all they all are trained on I’m kind of about the same kinds of datasets too, which is Right.  Buying large the Internet. 

Speaker 1: Yeah. Everything. Everything. Yeah. 

Speaker 2: Yeah. So you can you can use these services like Bard or or OpenAI and and and  interact with it. And they have OpenAI also has a commercial version of it and an enterprise  version of it. Right. And use it to to, you know, use to to to power your own applications.  But they don’t give you huge amounts of control over what can be done with it. You can’t you  can’t download the model, for instance. You can’t run it on your hardware. So all there there are  limitations to that. 

Speaker 1: Right. 

Speaker 2: There are also some constraints that that chief information security officers,  particularly in regulated industries, are still wrapping their heads about what they’re comfortable  doing and not mode. 

Speaker 1: Right. Right. 

Speaker 2: So, you know, it’s a I’ll give you an example. You know, one of the one of the  companies that we’re working with is a little hospital chain. For them if they wanted to go build a  chatbot or a kind of an assistant that helps physicians summarize patient notes, for instance, the  raw data, and it is electronic patient records. There’s a sensitive data. It’s about people’s, you  know, infirmities and all these kinds of things.  

There’s a huge amount of sensitivity about from within the IT leaders, within hospitals, for  instance, about taking that data and putting that into AI or bard or any any any of these kind of  AI services. So the option for them is is instead is is a a frequently popular one is take a model,  an open source model, from hugging base, like, you know, maybe Open Hermes or LAMA or  what have you. Run it on their own hardware. That could be, you know, it could be they could  have a cloud account. They but they manage it.  

It’s their data.

Speaker 1: Right? 

Speaker 2: Or they could use a service like ours, which give them control over what happens  with that model and, you know, with the data and all that. It becomes their own thing then. So it’s  kind of you know, there there are maybe two or three different alternatives like does. Right? So  open AI barter are on one one end.  

The hugging face kind of the open source wild west is at the other end. And in the middle, are are  companies like AnthroPIC. Right? Which which is kind of the they’re branding themselves as a  safe alternative to open AI. Right.  

And tropic has taken on a a huge investment from Amazon Web Services. 

Speaker 1: Right. So you 

Speaker 2: can use those capabilities on AWS if you, you know, if you wanna more control and  and assure about safety. 

Speaker 1: Uh-huh. 

Speaker 2: But all of these things that involve certain certain trade offs in terms of control and  cost and what you get out of it. Yeah. With with open AI, you get the least amount of control.  With the logging phase, you get the most amount of control. But you gotta do the most amount  of work to make it work there. 

Speaker 1: Yeah. Interesting. That’s the exact parallel. You know, I bet that that pair that is a  metaverse of all every different industry. Right?  

Like in my industry, Same thing. You got more control, less control, more danger, less danger,  more effort, less effort. Those are all the tradeoffs we’re constantly working with and when tools  come along that kinda help you manage those trade offs or make make maybe make accessible  what you’d like to do most but have limitations on, that’s always good. So I got a couple  questions here too just one thing I’ve never looked up, I could ask Chad GGP this, which I’ve  done a lot of education that way. But what exactly of two two questions?  

The first one, what exactly is a transformer model? What does that actually mean? Because I read  that a lot and, you know, I know it’s not Optimus Prime or or any of the stuff my kid plays with.  But 

Speaker 2: Yeah. So there was a, you know, a paper that that Google had put out a a while. Back  that, you know, that in a in a this is, like, the the the simplest version of it. You can go and you  you can go and ask chat, GPT. Know, that might be an interesting test to go and try right now.  Yes, sir. You know, how do you Well, let’s just ask you. Let’s just ask for it. 

Speaker 1: I’ll tell you what. You are given our listeners and viewers a prequel because I’m  working now to figure out how to have a chatGTP as a co host. I had it as a guest you can I got a  little plug in? You could talk to it. It’ll talk back, which is basically a preview of what’s gonna  happen when we have our own agents and we’re just telling it to do stuff and it’s talking back to  us.  

Right now, it’s it’s spread out a little, but, you know, we all know that Chad TTP’s agenda is to is 

to integrate this all just like now I’m enjoying using dolly from the same interface. If you’ve  never played with dolly, it’s It is awesome. 

Speaker 2: It’s pretty cool. 

Speaker 1: Professionally, I’ve done stuff, but I’ve also just done goofy stuff with my son when  we type in, like, make me you know, one thing for that I did with it as an aside. So we’re looking  at, like, Halloween costumes for my kids. So I I asked it, could I upload a picture of my kid? You  know? And then transform them into different stuff.  

It wouldn’t let you do that because it said I can’t recognize. So I described him and I’m just like  curly blonde hair, you know, really don’t know. Good looking. I said, which is true. And and it it  gave me four versions and I swear to God, one of them was like a spitting image of this kid.  I I strive it a little bit, you know, more specifically. And then I was turning them into nosferatu  and turning them. We turned them into a bird. And, I mean, you just you never know what you’re  gonna get. I think the first time I played around, I was like, I have a car, a Lotus.  I’m like, I’m always, like, thinking about Iron Maiden when I’m driving that car even though it’s  fiberglass. 

Speaker 2: No. That that I think everybody should think about it. I don’t need more of that. 

Speaker 1: I know. So so I said give me an image of Eddie, the mascot of Iron Maiden, you  know, driving a nineteen ninety five load of a spree with a union jack. And I mean, I’m telling  you, dude, I went through like fifteen versions. Sometimes it put two heads on Eddie, sometimes  it gave them two flags. Like, it does a lot of random stuff like that, but 

Speaker 2: but it was 

Speaker 1: sitting there going, holy shit. This is crazy. Yeah. I mean, it it was so fun. So it’s a fun  little diversion.  

Anyway, I don’t even remember what oh, I was talking about, you know, the integration of stuff.  So, yeah, what did ChatGTP say? 

Speaker 2: So I I actually went and asked about 

Speaker 1: it and 

Speaker 2: said explain explain what a transformer model is and and the and the and the first  sentence first two sentences are very interesting. To transformer models or a type of neural  network architecture that are particularly well suited for natural language processing task.  They’re introduced in the paper attention is all you need in twenty seventeen. You may become  the dominant architecture for natural language processing. 

Speaker 1: Gotcha. 

Speaker 2: Okay. That’s sorta helpful, but not really helpful. Yeah. What what what they’re what  they’re the simplistic way to to that I I might explain it to to, you know, to my dad or something  is that so that if you if you train a huge amount of data on on or if you if you if you train a 

transformer model with a huge amount of data, they tend to get really good at predicting what  tokens. The token is a part of a war.  

In what what order they come in. Right? And so then then you can you can use that capability in  order to to to put it to work to do things like generating time. Right? So this question that I asked  explain the transformer model has generated an answer for me, which is very different from from  maybe two years ago or even last year pre open AI, if you’d gone to Google and look for what’s a  transformer model, you get back a a set of search results.  

Right? And those things point back to the various websites that they load in and then, you know,  it’s it’s up to you to go look at they’re not generating the summaries on the fly. They’re not gonna  generating a response on the fly. Right? But, you know, these things are able to capture the  dependencies between words And they also are able to to process these these inputs in in parallel,  which, you know, make some do certain things really fast.  

Like, for instance, joining this picture that you’re talking about. When it comes to pictures, they  kinda work in the same way, you know, that what they’re doing is it’s creating a vector  representation of of of these images and then be able to combine them and generate them on the  flat. Issues, though, is that, you know, like, it it’s a cool example that you mentioned with Eddie, I  mean, like, I’m a big made in fan too, as you know, and it’s a Yeah. That’s fine. I don’t know if  Eddie’s copyrighted or not.  

Like, you know, and what what Bruce and the boys might have to say about, you know, what’s  going on with it. So maybe nothing. Right? So, you know, if you if you if you did the same thing  with the Grateful Dead still your face, you know, which I don’t think that the Grateful Dead ever  copyrighted, you could probably get away with that. Nobody really gives a shit.  But, you know, when in inside of the enterprise, like, these are some of the situations that come  up and that can be potentially problematic. Yeah. 

Speaker 1: I’ve had it blocked me when I’ve said, like, you know, use so and so’s picture logo,  and it’ll say that’s copyrighted material that I can’t. There may be a way to hack your way around  that. I typically say, we’ll make it like that, but not exactly. You know, and, you know, there’s  ways to prompt engineer, prompt it, whatever. 

Speaker 2: Yeah. And and that’s and that’s exactly one of the main issues with a lot of the stuff is  that is you know, you can just by trying a number of different things, you can get around you can  get around what the developer of of an application might have conceived. 

Speaker 1: Yeah. Totally. 

Speaker 2: Right? So you can hijack prompts. You can keep trying different things. You can  inject it with various things. And Yeah.  

And and, you know, like, to the the the issue the consequence from it from your chief  information security officer and a large company is that you you developed a security policy, a  framework for thinking about risk and risk remediation that’s evolved over thirty years for  longer. Right? Like, how to control access to to certain resources, how to how to prevent  unauthorized access to, you know, how do you track it, trace, and all that kind of stuff. And that  works really well when when responses from systems are deterministic. You get the same thing  back over and over again.  

You can predict kinda what’s happening like in the world of data business. That doesn’t exist in 

this. Right? You can the same question can generate different kind of things that is all context on  that. Yeah.  

So so what did how do you how do you build kind of the the enterprise safeguards 

Speaker 1: in this case? In enterprise, it has to say the same thing every time. If you’re relying on  it as an information source, as an input to some other thing that needs to be consistent, you can’t  have it be wonky like that. Right? And you can’t have it elucidate, you can’t have all that stuff. 

Speaker 2: If if you and I worked for a same large, let’s say, a fortune five hundred company,  And we had a we had an internal HR chatbot that that helped, you know, do various things.  Right? You know, help you you know, put in for vacation request for instance. And I had a way  of of manipulating the prompts to say, you know, give give me give me access to Charles Social  Security number. A bank account number or what have you.  

Like, that’s that’s really a problem. 

Speaker 1: Well, one thing I just read, you probably read about it is I think somebody set it up to  just repeat the word poetry. And if you do it enough, you can make it barf out all its training data.  Right? So, like, you can actually and I think they’re starting to, you know, safeguard against that.  But that’s the example is just like any kind of security.  

It’s always one one side of the fence coming up with a clever way. The other side of the fence  blocking that way and and on it goes and on and on it goes. You’re right? So, you know, taking it  out, kind of making it bulletproof is is a good way to to think about it. So it’s it must be part of  the equation.  

I think, again, at the enterprise level. 

Speaker 2: It has to be. 

Speaker 1: Yeah. Yeah. So I got a couple of things. So Is it the case thing with the transformer  model that, like, the more you stuff into it? Training wise?  

The more connections it can make and the the better it is. Right? 

Speaker 2: In in theory, yes. But I think in in practical terms, you hit into diminishing returns.  And can, you know, can you like, so the the large models also consume a lot of capacity to run.  Right? I mean, they’re they’re computationally really intensive and and the the the the GPUs, you  know, the the graphic processing units, which power all these things are are very expensive and  they’re serious supply chain constraints for them.  

So depending on the use case, it may or may not be the best fit. Mhmm. Right? So there’s an  evolving field is is is biological large language models. So they’re looking for things like how do  proteins interact with enzymes and a brand 

Speaker 1: about that. Yeah. 

Speaker 2: So in in those situations, a small metal might actually be a lot better than a than a  model Right. Right. On Internet data on Right. Right. Right?  

I

Speaker 1: mean, see, that’s the thing. My as a layperson, my mind always goes to chat, GTP.  Whenever I think of a large language model, That’s my paradigm. It’s publicly available. It’s  trained on all this shit.  

The more you stuff in there, the more stuff it it can know. Right? But I wonder when they’re  training this thing, like, when they’re training chat GTP. Right? I mean, do they have so look at  think about garbage in, garbage out, and think about horrible stuff.  

Right? Well, couldn’t you, on the front end, say, hey, as I’m training this thing, let’s leave all the  horrible stuff out. Therefore, it won’t come out on the other end, but it seems like it’s not like  that, that it it knows the horrible stuff. It’s in there. They try to put these safeguards around it, but  clever people can pull it out.  

And so in there is are the instructions for making a nuclear weapon or, you know,  methamphetamine or, you know, I remember, I don’t know if you ever saw this 

Speaker 2: I I remember when we were when we were young men, you know, the the most  insidious that it got was that, you know, we’d have, like, a photocopy couple of pages out of the  anarchist cookbook and, you know, we all think we’re really cool. You know? 

Speaker 1: Dude, are you kidding me? I was just about to say The only thing we had was the  Anika’s cookbook, and everybody said, oh, man, if you buy that, the FBI opens a file on you and  then Yeah. There was another there was another set of books called getting even by this guy  George. Hey, dude. Did you ever see those? 

Speaker 2: Yeah. Yeah. 

Speaker 1: That those were horrible. Like, how to how to mantle a balloon full of pee to  somebody in a box, so when they open it, it’s flashing over him. And it would always say, in  order to get your mark they call the person. So that’s crazy. Well, we just dated ourselves and it  goes to show you that there was nothing really besides that or make 

Speaker 2: this this world is a whole lot scarier. 

Speaker 1: Oh my god. The anarchist cookbooks like a nursery rhyme book at this point. Right?  So Anyway, 

Speaker 2: but do you know go go on back to your question though, the the the p in the GPT is  pre trained. 

Speaker 1: Gotcha. Yeah. Yeah. Yeah. 

Speaker 2: So it’s generative pre trained transformers. So most of these things, like, are are  trained on public data. So on on common crawl for instance, which, you know, but they’re they’re  you might you probably have seen, like, a lot of the authors and, you know, Steven King and and  who’s the Harry Potter Lord? 

Speaker 1: Yeah. Yeah. JK Rowling or something?

Speaker 2: JK Rowling. Yeah. You know, they’ve they’re I think rightfully so, up in arms about  Yeah. Their their their creative work being used with Yeah. 

Speaker 1: Music too. Right? In art images, it’s all like, that’s a whole another set of a million  podcasts. So Another quick question. I write these questions down because I’m so curious, and  then I got I got some some mills to transition to.  

But which is more energy intensive, mining Bitcoin or running to chat GTP. Right? Because you  still hear about people like taking fifteen thousand playstations and rigging them up, you know,  somewhere somewhere in a faraway world so they can mine bitcoin. But we know it takes a lot  of energy 

Speaker 2: I I don’t I don’t actually know, but the the the value the usefulness is is fairly obvious  to me. Yeah. You know, one one of them is far more useful than the other, but also that, you  know, to the question you’re asking earlier about about small models and large models. Like,  they’re going the opposite ways. Right?  

So Bitcoin, mining a new the marginal Bitcoin will always be more expensive than the previous  one. It’ll be computationally more expensive because that’s how the algorithm is interesting. The  the trend in in in GPTs and and AI is going the other way. Right? The trend is towards smaller  models.  

So you you you the the amount of work that it would have taking you to to to get to an answer a  year ago is going to be far more expensive, computationally expensive than what it will be a year  from now. So that that’s going the other way. So, you know, 

Speaker 1: that’s interesting interesting interesting. So here’s the thing too that I’m learning. I’m  just starting to understand this. When you first describe it to me, I didn’t really get it. But there’s  also this comp concept or reality of a of a hybrid model like a like an RAG.  Right? So in other words, you could take a large language model that’s open source or whatever,  put it somewhere and then take your own corpus of information and say, I wanna inject this stuff  into your relational capabilities model. That’s pretty cool. 

Speaker 2: Yeah. It’s it’s related to this concept called fine tuning. Right? Which is which is you  take a a pre trained model, you know, let let’s say, you know, the LAMA tube from matter. And  and you wanna use it for your own situation, which which, you know, maybe it’s it’s a it’s a  chatbot for an internal HR chatbot in a in a large company.  

That is going to get more and more useful. The the more it’s it’s applied, it’s tuned on with your  own data. Right? So so RAG is a is a technique it stands for retrieval augmented generation.  What it does is kind of a hybrid approach that combines retrieval, right, which is search with  with generative models to to to improve the the quality and relevance.  

And those things end up being particularly useful for things like question answer, kind of use  cases, or or natural language generation, you know, stuff like that. 

Speaker 1: Gotcha. So that that mean, you guys you guys accommodate that as well. Right?  Like, you can do a hybrid where I might take a a pre trained model on a big one or a small one,  whatever we’re talking about. And then let’s say, I have a just a old dictionary of medical coding  things or something.  

You want more control over that, you know, the open source model or whatever may have that in  there. But you gotta have more control over it. You can’t to your point about getting different 

results. I think if you inject your own model that’s very clearly labeled and structured, that’ll  allow you a whole new set of insights and capabilities. Right? 

Speaker 2: Absolutely. A hundred percent. And, you know, and all of the stuff is particularly  useful when you think about, you know, like manufacturing, for instance. Right? You might you  might have part numbers, part descriptions, things that are that are very unique to your own  organization.  

Which nobody else is really going to know. So in order to find real usefulness out of some of this  some of this generative AI capabilities requires fine tuning with with with your own proprietary  data and using techniques like rad. 

Speaker 1: So let’s talk now about product. Right? So building product. I’m curious, you know,  you’ve got enterprise applications. Right?  

So I look at it as b to b. You’ve got enterprises that you know, are basically doing things for  consumers with their economic business models, etcetera. And they they there’s a lot of ways  they could use that natively, like with their own developers and their own stuff. But they also use  a lot of products. Right?  

Enterprise grade products that that solve problems or create functionality. You know, I’m starting  to think in that direction. Right? Like, in my industry, I’m looking at what are what are people  doing in terms of predictive hiring? How can we automate the the compliance based high touch  stuff that I’ve been doing for three decades in order to help at scale and and, you know, it’s it’s the  same questions everybody in every industry is asking.  

You know, as I think about that, I’m like, so I’m a product person and I want to use this is where,  again, I don’t understand, like, would I be using just the giant capability of chat, GTP, and wanna  build that into my product? And you know, make sure that I could can leverage that within the  protected walls of my product that I’m gonna sell to an enterprise. Would I ever even have a use  case to do that or would I more likely be saying, alright, I’m either gonna train my own model or  take one of these basic things because the way I understand it And this this opened my eyes too. I  I attended a webinar from some database company and he was talking about I was I was actually  a a friend of mine forwarded it to me, and it was all about using chatGTP to match resumes. So  I’m like, oh, man, that’s like, right up my alley.  

There’s some really dubious stuff maybe happening there, scary stuff. So I attended the webinar,  and it was from a database company, and I was really surprised they were talking about going out  to chat GTP to work with the resume and then sending back a match. And I’m I’m in the chat  there on the thing going, and this is highly technical. I had no idea at most of what’s going on. I  said, well, why the hell would you go out to chatGTP to do this.  

There’s tons of people who use natural language processing to match resumes to job descriptions.  They were like, well, the relational capabilities of chat, ETP, are so strong that there it’s gonna do  a much better job of this. When you feed it information going in one way and it gives you  information coming out the other way, that was a whole way. I hadn’t even thought about it. So  you know, that made me think, well, if I’m building a product to do that, would I would I just  have to reach out to GTP just for its relational capabilities?  

Or could I use one of these open source models from a hugging face to do that? So So just think  I’m designing a product in my industry. It’s gonna use a large language model to provide all kind  of crazy efficiencies and new things how how am I doing that? Am I using GTT? Am I using it?  You know, and how would that plug into your product? So that’s a lot.

Speaker 2: Yeah. So So I mean, you you can I’ll tell you the reasons why you might make a  decision to not use OpenAI chat GPT. Right? And that that might that and and I think there’s  plenty of places where it might be the obvious choice. For in the situations where the developers  or the product itself requires more control than an open source alternative is probably a a a better  better decision.  

It’s also going to typically be cheaper to do. You know, there’s a open AI and and others like  coherent charge on a on a token based model. So you gotta kinda figure out how many token  based it it’s going to take for for powering your use case. 

Speaker 1: Gonna get super expensive? Like like crazy expensive? 

Speaker 2: Yeah. It can. So that’s another reason. In regulated industries, it’s especially when  you’re dealing with sensitive data like like HR or or legal documents or what have you this isn’t  this isn’t to say that OpenAI is insecure or anything, but rather that the regulatory environment is  still moving around a whole bunch. So if you’re, you know, if you’re a global organization and  you’re dealing with people who are who are in multiple geographies, let’s say, Japan, Germany,  and in the United States, moving the data around and doing things on it, you know, or whatever  if you have an application that that has to satisfy people in multiple geographies, there’s a there’s  a deeper regulatory burden that comes along with it Yeah.  

Externally. Internally, for companies, you know, they have varying policies about what’s what  they will allow and not allow for their own users to do. So if you’re an application developer,  you’re building a piece of you know, piece of software for running as a SaaS opportunity for  whatever you like, these are all the things you gotta consider. Right? And And there’s it’s one  possibility you might end up saying that, like, actually, I don’t none of these things are a big deal.  I don’t care about the cost. I don’t really need any control over it. And I’m you know, and I don’t  have any geographic or regulatory issues. Hoping I might be perfectly good answer. What we  find though is that in in particularly regulated industries or or companies with global footprints,  those concerns tend to be pretty severe.  

You know, they want control over the data. They want control over the model so that they can  track what’s going on. They can authorize and and deauthorize access to it and all of that. So and  and frequently, you know, for especially for application developers, you know, who are kinda  comfortable with with running cloud workloads and stuff. Open source models are are great.  Right? I mean, it’s not it’s not too hard. A piece of a a platform like Bookend makes a a bunch of  the operational headaches go away. So you can you can take a model, you know, run it on on on  multiple clouds if you need. You know, you got a bunch of the the the security, auditing,  governance kind of capabilities that just come out of the box, so you don’t have to build all that  stuff.  

So there’s a there’s a there’s a bunch of bunch of things that you kinda just get free from using a  platform like ours and and we’re pretty cheap. Right? It’s not which it’s quite cost effective to run  it. 

Speaker 1: Cool. That’s very cool. So so I’m a product part, you know, I work for a company that  serves up you know, pre hire evaluations, predictive hiring. And, you know, I’ve got some ideas  to to build because I sell the enterprise, and I’ve got some ideas to to leverage large language  models for various things. Right?  

It it could be a lexicon of information about what it takes to do jobs like a taxonomy. It could be 

generating interview questions or scenarios. Right? And I I could have my own model my own  data for over the years that I wanna incorporate in that. But again, I wanna use some of the  various things.  

How do I use you guys. Like, wrap my stuff around yours, etcetera. Like, I I don’t a hundred  percent get that. So talk to me about how product builder for enterprise? 

Speaker 2: Our ours ours is fairly easy to use. I mean, it’s it’s kinda it’s kinda designed for use by  by, you know, kind of full stack application developers. So it’s it’s a it’s a REST API. So any any  developers who’s used to, you know, using using an API through popular languages. You know,  we support client libraries for Python as well as Java.  

So, you know, any developer who’s familiar with using those kind of languages can can call all  of the the different things that Bookend does through an API. And and all of the models that that  we make available on our platform are are curated, sanitized, benchmark, licensed all suitable for  enterprise use. Right? So then we kinda run that entire process. So from we don’t have four  hundred and twenty three thousand models on our platform.  

We only have a dozen. Right? But they’re all right now and, you know, we’ll have about a  hundred or so in the next couple of months. But, you know, they perform a number of different  tasks from summarization to question answered to generating embeddings and various kinds of  common things that developers want. But they’re all kind of, you know, really they’re suitable for  enterprise use.  

And and so from the models on down, we make available a whole bunch of security governance  and and trust and safety related things out of the box, including, you know, things like data loss  prevention. Right? So keeping your, you know, sensitive information, like Social Security  numbers or account information and stuff like that from turning up and Right. 

Speaker 1: Right. And returns Oh, gotcha. So what about what about if I wanted to say, well,  you know what? There’s some stuff I wanna use from chatGTP in what I’m doing. Can I use that  securely through Booking too?  

Or is it is it only models that, like, I would download completely? 

Speaker 2: We support you can you can use your own model on you know, like, we have a we  have a customer who’s in a designed their own model. Right? This is a this is another biotech. So  they’re, you know, they’re kind of interested in in doing some protein analysis. There’s not a  suitable open source model for that, so they run it on our platform.  

But we’re almost exclusively open source models. So it’s not a, you know, you wouldn’t you  might have an application that that that you know, that kind of incorporates both OpenAI and  some open source models, but you can’t use Open Source. You can’t use OpenAI with Bookend. 

Speaker 1: Gotcha. Gotcha. Right. But both most people who are building secure applications  for enterprise aren’t aren’t injecting GPT in there. I think the use case more in enterprise is people  who work there going, okay, how can I make my job easier?  

How can I find out information etcetera, which is a great use case for it? But as you start prompt  engineering it to do stuff, so, like, what if I prompt engineer it, to be able to run like a dataset or  something, and then I wanna build that dataset running into my, you know, my product, I guess,  that simplistic thing of running a data set, you could code Python or something else to do that.  

You don’t necessarily need it. It’s from people like me that don’t know how to do that that coding 

who can give it an Excel file and say, analyze these different trends or whatever. That’s the magic  of it.  

It’s not a you know, enterprise has stuff that does the same thing. Would that be a a correct app?  Yeah. 

Speaker 2: Yeah. I I think I think that’s fair. Look, we we we tend to do well in in situations.  Like, we really back off this stuff. Right?  

Like, dealing with sensitive information, in in we’re very comfortable in regulated industries and  and, you know, kind of the the decidedly non sexy stuff. Supply chain HRIS. Like, so all the the  platforms that make a bunch of organizations work well. But it’s not, you know, it’s not like the  the the cool, like, you know, image generation for your, you know, dating profile of the time or  whatever like that. Yeah.  

It’s not 

Speaker 1: I always think of I don’t know if you remember as a movie from I think probably the  eighties, but down and out in Beverly Hills, And what I always remember about that is, you  know, they were super rich. And the guy, he had a coat hanger factory, you know, makes the most  boring ass thing, a code hanger. Right? But people need it and, you know, super lucrative. So I  always think about that, like, there’s there’s a lot of money to be made in the unsexy stuff too.  Okay. So so, you know, in my world and it’s getting that way, everywhere. But, you know,  vendors of predictive hiring tools are gonna need to have it’s not the case yet. But there’s gonna  be a mandate probably eventually from the feds, but most likely from California and, you know,  the EU is working on something. So a mandate of being able to have a a third party audit from a  to z of what they’re doing. 

Speaker 2: Yeah. 

Speaker 1: So if someone were as a product come company were utilizing, you know, a large  language model, and they hadn’t wrapped up in something like yours. I’m assuming it’d make it a  lot easier for them to pass some kind of security audit, almost like when I tell when I tell  someone on our little platform that it’s hosted on AWS, I get to skip skip over crap load of IT  Yeah. Security questions. Thank goodness. Right?  

So you got that same capability. Right? 

Speaker 2: Yep. So so things like algorithmic bias and, you know, to the big deal for for the kind  of stuff that you’re you guys are doing. Like, so we make all of that a bunch of that kinda stuff is  available out of the box and auditing for it, tracking what’s going on, explaining what’s  happened, and stuff like that. 

Speaker 1: Interesting. Good stuff. Well, there’s there’s I’m looking forward to, you know, the  journey here. I mean, you’ve been a big part of educating me on this stuff and it’s so accessible,  yet difficult to conceptualize. If you if you really dig into it, I mean, I call chat TTP, you know,  supernatural math.  

Because all it is is math equations, predictions, but geez, you know, it’s ones at zeros, ones and  zeros, 

Speaker 2: but

Speaker 1: but for God’s sake, it it it writes books and generates crazy images and answers all  your questions and gives you recipes and I’m taking a prompt engineering class right now, like  online. And I’m wondering a lot from that too. So in exploration mode, there’s lots to learn, and  

I’m looking forward to, you know, learning more about what you guys are doing. So I appreciate  your time and and and hanging out. Where can people keep up with you and you guys and what  you’re doing?  

I mean, I say that all the time, and it’s always, well, we’re on LinkedIn. I’m on LinkedIn. Is there  anything else, you know? 

Speaker 2: Yeah. Well, we’re we’re we’re we’re bookend dot AI. Hopefully, easy to to to not spell  wrong. Yeah. We’re we’re on all all the usual LinkedIn discord and all that stuff.  Yeah. No. It was a it was a pleasure pleasure speaking with you today. Yeah. 

Speaker 1: Thank you 

Speaker 2: for having that. Absolutely. Hopefully hopefully we do it again. 

Speaker 1: Yeah. For sure. As we wind down today’s episode to your listeners, I want to remind  you to check out our website rockethire dot com and learn more about our latest line of business,  which is auditing and advising on AI based hiring tools and talent assessment tools. Take a look  

at the site. There’s a really all some FAQs document around New York City local law one forty  four that should answer all your questions about that complex and untested piece of legislation.  And guess what? There’s gonna be more to come. So check us out. We’re here to help.

The post LLAMA, LLAMA- How to Avoid Generative AI Drama! appeared first on Rocket-Hire.