“Personnel selection is shaking on its foundations. Lots of the assessments that we have, that we do, suddenly we’re not quite sure if they’re still as effective.”
“I feel as if now it’s starting to become the responsibility of organizations who are hiring to make sure that people are using it (Chat GPT) in the same way so that that they can clearly state, “This is how you can use Chat GPT or cannot use Chat GPT”
“From what I’ve seen, not many talent acquisition teams that I’ve encountered have really thought carefully about what they what message they wanna send to candidates about the use of technology. And I I think one one concern I do have is if the reaction to it is, “Oh, we’ve gotta monitor everything.” I just worry about the candidate experience impact that will have.”
Summary:
In this episode I host Patrick Dunlop from Curtin University in Perth, Australia, and Djurre Holtrop from Tilburg University in the Netherlands, discussing the evolving role of AI in hiring and personnel selection and their research into the use of ChatGPT by job applicants.
We traverse across three continents, bringing together two interesting foods, Vegemite and Stroopwafles as well as global insights on the use of AI in the hiring process. Together myself, Dunlop, an organizational psychologist, and Hultra, an assistant professor with a background in assessment psychology, delve into the practical and ethical implications of using large language models like GPT in job applications and recruitment.
The conversation explores how AI is reshaping traditional hiring processes, with a focus on my guest’s research into the use of AI in enhancing cover letters and the potential biases and ethical considerations that emerge.
The guests share insights from their ongoing research, including the impact of AI on candidates whose primary language is not English and how organizations might need to adapt their selection processes in response to technologies such as Chat GPT.. The episode also touches upon the future of AI in hiring, contemplating the balance between technological advancements and maintaining fairness and inclusivity in recruitment.
Key Takeaways:
- AI is significantly changing organizational selection systems, particularly in enhancing application materials like cover letters.
- Research into the use of LLMs in the hiring process is critical, but is difficult because of how fast things are moving.
- We can all agree that LLMs and GPT are going to create seismic shifts in hiring, but the “how remains unknown”
- Insights from different continents show a diverse understanding of AI’s role in hiring and its broader implications.
So listen in and you will gain a comprehensive understanding of the impact of AI on global hiring practices, along with practical insights and future prospects in the field of AI and personnel selection.
Full Transcript
Transcription for: “S4H_P_and_D.mp3” (Uploaded File) (New Transcription)
Speaker 0: Welcome to Science for Hyre. With your host doctor Charles Handler. Science for Hire provides thirty minutes of enlightenment on best practices and news from the front lines of the improvement testing universe.
Speaker 1: Hello, and welcome to the latest edition of Science for Hire. I am your host, doctor Charles Handler. We don’t often have two guests on at the same time, but it’s always worth it when we do and we do today. And these folks I have been introduced to by boss from the Netherlands, he I have a a pretty, I think, a disproportionate number of people from the Netherlands, Dutch, people on my podcast because of him, but he introduces me to so many interesting people. And I don’t think I’ve had any had anyone from Australia yet, and we got three continents going, three time zones.I’ve done a lot of that usually somebody draws a short straw on that, but I think we worked it out really well. Seven AM here for me. That’s no big deal. I’m up at five every day. I’m I’m bright eyed and bushy tailed and heavily caffeinated.So we’re in really good shape and we’ll go ahead and just get into the intros. We’re gonna talk today about a subject that transcends assessment, I feel like, and really is a global issue for pretty much everyone because we’re talking about using GPT in job applications that basically is everywhere all the time in recruitment. Yeah. It’s important in I o psychology too. But if even for job applicants, even for the economy, I mean, there’s so many implications of this.And a lot of these podcasts I mean, I just should stop I start saying, well, we’re not gonna talk about generative AI because everybody’s talking about it. But, oh, no. No. It’s too powerful for that. And our whole conversation today is gonna be about that.So I am going to tee up my guess. I’m gonna let them introduce themselves. We’ll start with Patrick Dunlop, who is is tuned in from Australia. Patrick, let let our guests know a little bit about yourself and and what you do.
Speaker 2: Thanks, Charles. I’m based at the Future of Work Institute, which is at University, called Curtain University in Perth Australia. My background is organizational psychology. And in fact, I’m a registered organizational psychologist here in Australia as well. So I do a little bit of practice from time to time.But my primary role is an academic. And so, basically, I like I I get paid to nerd out about things. And the things I like to nerd out about are recruitment assessment and selection.
Speaker 1: Very nice. Okay. And now we have Jira Hultra. Did I get that right? Speaker 3: You’re right. You got it right.
Speaker 1: Okay. Good.
Speaker 3: Good to be here, Charles. And be here with Pat too. Yeah. I I work at Dobrik University in the Netherlands. I’m one of the Locust peoples spreading around the world, taking it over step by step.Like you said, you have duchies everywhere. But, yeah, work at Dobrik University is an assistant professor. One of my favorite topics is to study recruitment and selection worked as an assessment psychologist for, let’s say, seven years before moving into academia. Did a a PhD in developing a personality inventories and finding ways to make them more predictive for a personal selection, then slowly drift it into this AI. What can AI do in personal selection?And that really started for me when I met a friend in the train. And he said, I developed an a sentiment analysis, and we were chatting and he said, you know, maybe we could apply this to personality questionnaires. Can we maybe augment your sentiment analysis to assess people’s personality, and this is, oh, this is a long time ago, eight years or so, maybe even ten? Right now, and that’s how we slowly got started. And all these technological advancements of today, they they amazed me because when we started, it was really well, just using Excel spreadsheets and they’re he cannot compete with that anymore.
Speaker 1: No. No. Well, awesome. So and it’s not a random. That is a very serendipitous thing too, you know.Keep waiting for one of those that happened to me, but we don’t have trains really here in New Orleans that get us around. We have street cars, and they’re very, very slow. So you’re not you’re not taking that to commute anywhere. Not so much for me, but that’s very very awesome. And it’s not a coincidence that, you know, I’m not trying to to pack my giant list of guests into a more efficient thing by having two folks.I mean, you all have worked together obviously, on some research that we’re gonna talk about today, which is which is pretty cool. And again, I like having three continents this is truly a global thing. Obviously, the reach of chat g t p. Although, it’s really interesting, you know, I hang around with a lot of people who aren’t even in white collar jobs and not knowledge workers necessarily, not everybody but but a good slice and we don’t talk about this stuff. You know, when I when I get around other white collar folks, it’s all we’re talking about.And I think that like my wife, my kid, they just call chat GTP AI. I mean, it’s starting to become, like, the Kleenex GTP is, like, the Kleenex of AI because it’s so accessible to everybody. And they understand that there’s some greater, you know, force going on here. I I I I say it every episode. I I call it supernatural math just just because it’s like, all it is is math equations, but boy, oh, boy, it sure has a personality and its own ethos in some weird way.I mean, even, like, you know, reading about it slacking off over the holidays because that’s what people do. And I found it a couple times. Like, I had to tell you’re being lazy. Come on up up your game here a little bit. Whether that’s true or not or just my own perceptions, who knows.But we are gonna talk about in the pre call here, you guys were just laying on me some amazing research you’ve been doing and, man, I I I can’t wait to hear about it. There’s so much to pack in. So let’s start. It sounds like there’s a couple of different research projects you all are working on. But first, just I would like you to each just tell the audience what you find so magical and transformative about this technology we’re talking about.And this really large language models. GPT is the poster child because everybody can use it, but there’s a lot of other ones. A hugging face has probably thousands, you know, that’s the the open source LOM thing. So yeah. Come on, Patrick.What what what’s so exciting and transformative? What what blows your mind about this stuff?
Speaker 2: Yeah. Lots of things. People often ask me, well, what do you use it for? And I I’ve been having a hard time answering that because I use it for everything in the moment that I think it might be useful. To use it.So I use it, for example
Speaker 3: Yeah.
Speaker 2: For creative, you know, as a creative a, if I need some ideas for something, I use it as anonymous peer reviewer of my work. If I need if I’m looking for some criticism, I I find it helpful for trans transforming, like, you know, if you you sometimes just wanna send an email that’s really blunt and a bit rude, but you don’t wanna send a blunt and rude email so you could use it use it for that. I think in in the context of personal selection, I guess this is where I think about it, the voice speakers, I’ve I’ve been researching personal selection for so long. I’m often thinking about or, you know, how might a candidate use this? And and I suppose the in in Australia, I don’t know how it is elsewhere, but certainly in Australia, there are it it’s it’s quite common for selection processes to involve a lot of text based content.And and so, like, the use of common letters is pretty common here. In in in many government roles, you actually have to submit usually, it’s a five page or longer document where you writing narrative, which explains how you meet the selection criteria. And and I I I mean, I don’t know how valid these things were to begin with, but I I’m sort of imagining that the the availability of these large language models will change the way organizations well, in this country at least set up their selection systems because if they don’t, they’re probably going to find that they’re getting the same content from all the applicants who figured out how to use it. I think another another observation I’d I’d throw in there if it’s okay is is I I sometimes hear people say, you know, I tried to use chat JPT and it’s pretty I find it not that useful. It just produces really bland stuff.And my sense is what’s going on there is that they’ve only had exposure to the free
version. Right? They haven’t paid for a subscription, and maybe their prompting is very basic. And so I I see a lot of people sort of you know, trying it out, finding it’s a bit underwhelming compared to the hype that perhaps seeing on LinkedIn or Twitter or or wherever they get their socials. There’s actually daylight between that that GPT four model and and the three point five model.It it level of performance you get out of it is is very impressive. And, of course, prompting is an art form as well. And and, yeah, if you’re not doing that properly, it it will give you answers that are a bit overwhelming.
Speaker 1: Yeah. Yeah. I’ve been I’m taking a prompt engineering class right now. There’s a lot of use cases. First of all, it’s on it’s based on three five, but and it’s probably not that old, but a a lot of the stuff they’re teaching you to do is already irrelevant because I can do it without all the fancy stuff.I say that, like, certainly, don’t wanna be inappropriate or whatever. But twenty bucks a month, the delta you get from that twenty bucks a month you know, it may be hard for some people to afford that, and and I wanna recognize that. But twenty bucks a month, the productivity I’ve gotten, and it’s you know, this stuff’s gonna be multimodal and and GTP four is getting closer to that, having dolly in there being able to browse the web now. So Jura, I also we’re on a video too, so I saw you. Is that a was that a stroopwafel?Did I say it right that you’re munching on?
Speaker 3: Yeah. You saw that right.
Speaker 1: Yeah. Yeah. I’m the very good. Yeah. I love waffles and I tell my kids every time I go to the Netherlands there’s waffles everywhere.I’m the waffle master on our house. We I got a waffle iron. I make waffles all the but not the little ones like that. But, man, oh, man. Those things are great.So and it’s seven AM here and I didn’t eat yet. So that’s, like, I run into the coffee shop immediately. We may even finish early so I get the coffee shop. And we don’t have the I’m
Speaker 3: so sorry for doing this thing.
Speaker 1: No. It’s wonderful. But did I say it right? Is it’s Stroke waffle. How do you say that? Speaker 3: Well, the double o, we are pronounced o. Oh, okay. Now it says Stroke waffle.
Speaker 1: Thrath waffle. Okay. Well, I sure love those things. And oh, man, I just have real good memories of being in Amsterdam as the winter, it’s kind of snowy out and dark and the, you know, and you go to there’s a little square, there’s a lot of them I’m sure, and then there’s a little cart where they’re selling those things. And I can’t get enough.
Speaker 3: Well, when they’re fresh, they’re even better. I had one. Mine just came out of a package, but Uh-huh. They’re fresh. They’re pretty amazing.Yeah. Patrick was visiting me recently in the Netherlands here, and we went out and we went to one of those cards. Color, fresh through fourfold, sat down just outside, fresh air, fresh through fourfold. That was just one of the better moments. Wouldn’t it be?
Speaker 1: Good. And, Patrick, no offense, but I I I don’t have the same love for Vige in my I I don’t know. If that’s even again, am I being stereotypical that most Australians really dig the veg mite all the time?
Speaker 2: It’s it’s controversial. I mean, it’s very brave of you to admit to me that you don’t like it. You are taking a risk when you when you’ve raised that within Australia. I love it. I was born and raised with I don’t like, I imagine by on toast with tons of butter.It’s it’s great. If you like to drink, it’s also a good hangover queue up.
Speaker 1: Well, I’m gonna admit this. I’m afraid of it. I’ve never actually tried it. So I might like it. I’m fifty six.I got a ten year old. I make them try stuff. Next time I’m there, I promise to to try it. Say I’m a finicky eater, but there’s just some stuff that I’m afraid to try. So I’m gonna I’m I’m gonna promise you that next time I go, I’ll try it.I promise. So I wonder how it would taste
spread on a strip off strip strip waffle. Stoop waffle. Stoop waffle.
Speaker 3: No. Don’t do that. That’s blasphemy.
Speaker 1: Okay. Well, that you know, you guys collaborate. You know, maybe maybe that’s a good collaboration. So little detour there, but but, Jira, what what what’s your take on just that the general amazement that you have with this before we get into you guys research.
Speaker 3: Yeah. I’m happy that I had a moment to reflect on that while Beth was talking. So I think that basically, personnel selection is shaking on its foundations. Right? Lots of the assessments that we have, that we do, suddenly we’re not quite sure if they’re still as effective.And for lots of the assessments that we know are reading note that the fact if this is also a threat, but that may be good in a way. So what I mean is, let’s say we have work sample tests, which are always great way to to sort of assess somebody’s ability to do his job. Yeah. If if that is a writing test for for a job as an editor or something, And that that’s clearly now a little bit under the threat. But on the other hand, maybe people should use JBT anyway in that position.
Speaker 1: Yes.
Speaker 3: So it’s it’s all up in the air. Right? So organizations are adjusting or are incorporating these large language models into their workflow, are working out how can we use them? What can we put in them? Will the big brother be watching?Will they be taking our content? So there there are lots of things going on there. But in a way that that’s probably one of the more relevant questions, should we also be be trying to assess or allow people to use These things, when it comes to personal selection, I do see a lot of assessments that will change based on just it’s share like, the the fact that it exists, things like cover letters, CVs, and work sample tests are easy targets, but also the other stuff that probably be able to talk more about based on the research he is doing that even psychiatric tests are under threat. But what I also see, there’s also a good side. I think that having a large language model as an assistant might empower more people to to do proper personnel selection.So if you’re in a small to medium business, you’re an owner, it’s it’s really difficult to know how to do personal selection properly. Yes. And if you have an assistant that has all that knowledge and is able to guide you to make conduct a simple job analysis and then select instruments that will test the the case AS that you require. And and that can all be done sort of for you. Set up for you, you only need to execute it.I think that also could boost professionalism in in in personal selection in general.
Speaker 1: Yeah. That’s a really good insight, you know. I I just envision a, you know, somebody in a a contrarian locked in a room right in multiple choice items on a typewriter because that’s they’re gonna be the same technology, you know, typewriter, a good analogy there because it just will be superfluous based on technology. And I expect, especially with multimodal, that it’s just going to become there’s two directions for me One is it’s just gonna be completely transparent and digital exhaust or just your interactions in the hiring process even will be processed somehow to give some assessment and then it will be, you know, simulations that are really easy to create and and score and all that. I mean, that’s that’s just how it’s gonna happen.How long that takes? I don’t know. I mean, I I have several clients right now that we’re working on multiple choice test stuff. I mean, you know, it’s still it’s still there. It’s not gonna disappear that quickly.Maybe the maybe the parallel, I love drawing analogies, electric, you know, and soon to be I think hydrogen powered. Vehicles versus internal combustion engines. Right? We have both. We’re moving toward probably a cessation of one
but, you know, there’s plenty of oil in the ground and there’s plenty of people who can’t afford it.So I don’t know if the analogy is exactly perfect. So you all have some great perspectives. I love the grounding in academics and, you know, assessment stuff, but also the broader look because Look, for me, CVs are more of a cover letter. I mean, you know, I haven’t applied for job in a long time, but does anybody even look at those really, is that gonna sway anybody’s opinion? So that seems like an appendix almost now, resumes too.So let’s talk about that. Let’s talk because that’s the tip of the spear as well in the hiring process. Right? So if we think about it as a funnel and a workflow, there’s all kind of issues with AI determining or being used to help you move people from outside the funnel, inside the funnel. And we we don’t need to talk about those today.But let’s assume someone is applying. Right? They already know about the job and they’re applying. These it’s the tip of the spear. This is where generative AI is probably gonna come into play first.So let’s talk about it. You know? Let’s talk about that. And, Patrick, it seems like that’s something I don’t know if you all are both working on, I think. But Let’s just open up with that conversation.What are you looking at? What are you finding in this area that that I think is gonna be really interesting to know?
Speaker 2: Yeah. I mean, the the first point I’ll make is is there’s a a lot of claims and a lot of words and a lot of things being said, but but not a lot of hard evidence out there yet, at least not that I’ve been able to find. And and so one of the hypotheses that you might sort of start with is is this hypothesis that generative oh, sorry, large language model would be able to
assist people who with their writing. Right? So they could take some somebody’s writing, which is pretty average and make it sound more impressive, more professional, You can always give a large language model, a persona using a prompt, and so you can make it make your own writing sound like it’s been written by Crispy Hetchens or Albert Einstein or you know, Donald Trump, whoever your favorite celebrity is.And and and so, you know, one hypothesis might be that if if people are applying you you know, if it’s organizations are using text based information such as cover letters to act as a network, you know, the early part of the the filter, filtering process, then applicants who use a large language model to produce some of this content when their writing is not not terribly good normally. They might they might have a better chance of getting through that filter than otherwise. In in the study that Jira and I are working on at the moment, and I’ll I’ll get him to talk a bit more about it because he’s closer to the data. But our our Our goal was to test that hypothesis. So it’s a two phase study.And in the first phase, what we’re doing is we’re asking participants to share a cover letter that they have written previously, so a handcrafted one without the assistance of of a large language model. Then we’re asking them to prepare a new cover letter for a fairly generic management level job. So, like, you know, a a cover of what it was, Derek can give you the details because he’s he he’s very familiar with the Qualtrics survey we’re using. So we’re asking these participants to produce a cover letter for that job And we’ve encouraged them to use the previous cover letter they’ve shared with us already if if it would save them some time because we figure most people when they they don’t just write a brand cover later, like, pick up their previous one and and edit it. And then in the final phase, we encourage them to use GPU t three point five.And and we essentially say use this however you see fit So, you know, some people with no doubt just copy and paste it and put in the placeholders. Others probably read it and then think carefully about how how good it sounds, how relevant it sounds to the job and make edits. And and Duro will have a bit of data on on the extent to which people have done that. We’ve got two groups of participants. One group are people who who’s language, who primary language is English, and then we’ve deliberately targeted a second group who have indicated that English is not their primary language.So our hypothesis is that one hypothesis might be that the people whose language whose primary language is not English. Might produce cover letters of high quality, at least as as judged by a sample of recruiters. And that’s the second phase of the study. We we will get recruiters to evaluate these cover letters. Jared, did you wanna add anything to that?
Speaker 3: No. I think you’ve covered it. We’re also asking people to share their prompts. Because we’re really interested in how people how proficient people are. And, well, I’ve I’ve looked a little bit at the data and the things that people do to how people use chat, GPT at the moment.It’s only a small sample, but because we we we ran a pilot basically, before we are deploying the whole study. And I I see relatively or prompting strategies and some participants even saying, I didn’t quite know about how this worked and how well this works. And they’re very impressed with the quality that Chachypity is offering, but also they simply just ask for, can you write me a cover letter for this position? That’s it. Whereas if you look on LinkedIn, there are so many how to’s, like, adapt this persona, do this many like, it’s almost like a whole step by step book that you can go through.Right. But these prompts and strategies do not seem to exist much in practice.
Speaker 1: Well, you know, here’s the here’s the interesting thing. I I think for simple stuff like that, you don’t have to do a lot. I mean, maybe well, I did my my own little experiment, I had to rewrite my professional bio. And so I just said, hey, I need to write a professional bio. I’m an I O psychologist.You know, here’s my LinkedIn. Write this for me. That was it. And it was amazing. Like, I wouldn’t think of anything I needed to improve.So I think in a relatively straightforward simple task, if you give it just a little bit of background information, it can do a fantastic job. And I’ve just thought of something really quickly. If you haven’t tried this in audience too, if you can handle it, I don’t know if you’ve heard of it. There’s a there’s a thing called roast my career. And you go on and you feed it your LinkedIn profile, and you can adopt the persona of I think there’s four or five people.Donald Trump is the one that sticks out, but there’s other ones. Famous people. And it it comes back and just, like, make so much fun of you and your career that I’m laughing, but I’m also like, wow. Maybe I haven’t done enough. Wow.May maybe I am kind of a sham here.
Speaker 3: You know, Charles? We have we have that as part of our profession. It’s called the payer review. Yeah. And then people tell you
Speaker 1: I know. I know. Oh my gosh, peer review. Yeah. Or you’re just so, man.I I I love telling stories. I I showed up for my my oral finals on my dissertation was in the morning. I brought doughnuts and I brought orange juice. And I have one guy there. I won’t say any
names or anything.Pretty much a legendary jerk. He chewed me out. What are you trying to do? You’re trying to get our favoritism here? You’re trying to you’re trying to, like, make sure you pass by bribing everybody.Get that stuff out of here. You know, at the very beginning, Boy, I guess it prepares you for the real world where where people are people can be pretty harsh. But anyway, so that’s not exactly the same thing. But I I my my presence was roasted like that in a real situation. But you should try it out because it does show you what you can do with some clever engineering, you know, to do something like that, and it probably isn’t even that difficult.So I think I think the the lower the the complexity level. Now the English proficiency is a nice variable there. Right? Because that that changes how easy it might be or how much the advantage it gives you. I don’t know.That’s what you all are studying, but I feel like for for the basic, hey, make me look good, tell tell us this this read or something good about me based on here’s some info about me. I think I fed it my my Vida. I don’t even think it could do my LinkedIn profile at that point. I just gave it my Vida. And it it knew me.Pretty well to write a really amazing, you know, really amazing thing. So so have you found anything with that yet? I mean, do do you are you extending this to resumes too? Or is it is it just cover letters?
Speaker 3: Well, we decided to focus on cover letters because cover letters are a bit easier for us to control in an experimental condition. CVs are usually bit messy. Everybody formats them differently. They’re yeah. Everybody has different rules for how to format CVs.Corporate letters at least are more or less predictable. So that’s why we decided to go for corporate letters. But to be honest, I recently did a little bit of a prevalence study in the Netherlands, and we asked
basically which types of assessments do you use we do see that and this was with about a
thousand organizations that about eighty percent of the organizations in the Netherlands represent of samples still use cover letters as part of their personal selection procedures. And then I don’t have have any clarity on whether they really use it or not and even more use CVs as part of their selection procedure, ninety percent and then only only interviews are a more prevalent selection instrument. But what’s funny is that I also asked a research student to look into alright.So how do what what do people discuss on Reddit? When it comes to using GPT. There’s actually most topics are about precisely these three types of assessments. So people offer and ask for advice mostly about how to use chat for corporate letters, resumes and interviews. The the she found some some quite interesting information on how, why, and what people use it for and the outcomes.People experience from using it.
Speaker 1: Interesting. So there’s a democratization. I mean, I think that’s the first thing that I’m thinking about in your research of, hey, you’re you’re not in as good a command of, you know, what and people can see that and probably cognitively whether it’s good or bad, gonna downgrade you as far as you know, your acceptability, you don’t want that. So so but at some point, it’s like everybody looks the same. So will it be people who aren’t using chat GTP, you know, gonna have a disadvantage kinda thing.But, you know, I thought about and when this stuff first came out, at least came to my view, you know, I wrote a blog or something where I’m like, yeah. It’s leveling the playing field. But, boy, you can’t You can’t defeat assessments with it, especially like a nonverbal cognitive, you know, or ravens or, you know, something like that. And so, you know, we’re still safe. As far as that goes, they’ll still differentiate.I I don’t really believe that as much anymore. I think there’s still a mandate there to say, look, if you you know, it won’t work. But, you know, hey, you gotta you gotta answer honestly. Otherwise, you might end up with a job you’re not a good fit for. It also puts more pressure on the interview, but we know people are cheating, not cheating.That’s a bad word. People people are opening up a computer and, you know, and and having it help answer stuff and even I don’t know if it’s GPT, but there’s there’s ways you can fake the camera making it look like you’re there when you’re doing other stuff or, you know, so I feel like an in person interview where someone checks their device and has to answer. But that’s not always, you know, feasible. So so that’s really interesting stuff. And, you know, to me, that’s just gonna lead you all to continue to to move down the funnel a little.But is there you talked about some other research you’re doing too, which seems pretty fascinating. So let’s let’s hear about that unless there’s something else you wanna make sure everybody, you know, takes away on the on the cover letter research, you know.
Speaker 2: There was something else for that study that we haven’t mentioned yet, but Okay. We’re also asking people to describe how they were using the the chat JPT algorithm to to help write their cover letters. And we we haven’t we’ve got a free text box which we analyze once the data come in, but we’ve also developed a categorical question question format where Yeah. We’ve given them options. I can’t remember the exact wording, but the options are basically oh, I’ve used it to completely fabricate something.Yep. That sounds impressive.
Speaker 1: Right.
Speaker 2: Or I thought know, anyway, the best of intentions, and then it wrote something that was really nice. It’s not true, but it was so nice. I thought I would just leave it there. And and and I’ve I had this experience once when I was demoing chat equity for a presentation I was giving about this topic. It it just made up a line about my proficiency in in hyphen, which I thought was fabulous.So I’ll just leave that there. And then the other side, the more honest side of of impression management, you know, we we thought a lot of people might use it just to put a layer of polish on this. So I don’t wanna make anything up. I don’t want it to lie on my behalf. But what I wanted to do is improve my pros or improve the clarity or or or make it more concise.So it’s it’s kind of a more authentic use of it. So what we’re hoping to get is a better understanding of the strategies that people have in mind Having said that, when we started the
study, I think I don’t know about you, Jira, but I I expected I think this sort of harks back to what you said originally, Charles, about know who’s talking about this technology and and more to the point who isn’t talking about it. Maybe we’re overestimating the extent to which people are thinking about this out there. And And and and soey’re not even thinking about, you know, authenticity.Or authenticity. They’re just thinking, oh, cool this is your toy. I didn’t know I could do this. So, you know, may maybe we’ve just got so far, I was quite surprised to see, you know, how simple the prompting was. Like, they’re tha sort of yeah. There’s a lot of hype on LinkedIn and there’s a lot of sort of doom and gloom stuff out there.You know, this is about every podcast I listen to at the moment on recruiting. Talks about this as a big threat to our selection systems. And I I do think it’s it’s it is a potential threat to some of the assessments. But it’s it’s probably not as fast moving as as we might think. You know, the interview stuff.That’s something I do see as a real thread. I I can see this technology where, you know, you’re getting prompted by these the large language models, which have maybe have been primed with information about you or maybe they’ve been primed with information about some superstar that you know.
Speaker 3: Yeah.
Speaker 2: And and it’s really just, you know, gets used as an elaborate faking device. I actually think that’s a bigger threat because virtual interviews are quite common. You know, you used a lot, especially in Australia. I couldn’t believe it. Everyone’s America.Not and and not just the the asynchronous format. But the specific study that you mentioned so I I think you probably had somebody in on your podcast before who would have talked about a project that Arctic Shores had run where they assess Chachi PPT’s capability of completing a verbal test and a numeracy test. And So they were kind enough to share their method and results with myself and another academic called Lois Hickman. And so we’ve we’ve been sort of looking at their prompting methods and, you know, so rerunning the analysis and and we we’ve got a paper that’s close to being submitted. And as you mentioned, it it absolutely hits it out of park, especially the GPT four model, when it’s taking a verbal test.But the University test was quite interesting. Both both the free version of so this test involves sequences of five numbers and one of the numbers is missing. Yep. And that’s one, two, four, six, whatever. Yeah.And a lot of a lot of tests of in involving intelligence or mental ability have questions like this. Like, I I know the open source international cognitive ability resources got a few sequence. Let
Speaker 1: us understand. It’s all about how
Speaker 2: you can
Speaker 1: work with patterns and how your brain can can interpolate and and fill in the blanks. Right? And how quickly you can do that?
Speaker 2: Well, what surprised me was both GPT three point five and GPT four really struggled with this numeracy test and struggled to the point where you know, the the paper that ran this analysis initially thought, well, we’ve got to give it a chance. So and so it turned this open ended sequence, complete the sequence question into a multi choice test. To give a, you know, fighting chance. Maybe if there’s only five options, it can it can perform better. And even then, it didn’t perform better.And in the end, what they what they did was they reorganized the the sequence question such that the last item was the one that was missing. And when you did that, GPT four suddenly could figure out how to do numeracy numerical sequences. Not nowhere near as, you know, not like ninety nine percentile or like fifty, sixty percentile. But it it did make me think when when I was, you know, I was reflecting on this because we we’ve been working on this paper and it it made me so I realized a few things like, where test developers probably need to start thinking about not so much is this a verbal test or not. Like, obviously, if it is verbal, then then it seems it seems that these algorithms will be a
threat.But also, how do these algorithm think? How do these algorithms think? Right? How does a large language model? I know they don’t really think literally, but you know, the the the way they produce words, one after the other based on a model that just keeps getting fed more data based on what’s already out there.You know, that that sort of suggests that it needs to work in a sequence. And and so if you if you’re writing tests of mental ability, maybe the sequence thing matters. And I think we’re still trying to figure out if if it matters if it’s multi choice or or free text. I I I just think there’s a new dimension that test developers might might have to consider when when coming out with the items because little changes that we would normally not think of that important might prove to be. Yeah.
Speaker 1: It’s almost like you wanna test it out. So in this prompt engineering class, I’m taking I don’t know if this is exactly it. But but you can learn and I haven’t had a real world application for this yet. But you can show it pattern and say if this, then this is correct or incorrect. I would
imagine you could train it with your own incoming sequencing and prompts to be able to do better on a numerical pattern thing.But, you know, do you have time as a job applicant to do that? But, you know, who might somebody might make up free little GPT that’s available to people that they can just feed this. Right? So that was the thing in my prompt engineering class. Like, I know you could feed it stuff, but I hadn’t really seen these structured ways of eliciting things again, I I wish I had had an application.I will just a little you mentioned Python. So I’m not a coder at all. I suck it in. I can understand what’s going on, though, conceptually. And I found actually somewhere there is an open source text to three d environment stuff.Because I think that text to to video and and realistic is gonna be the future of how we build simulations. But it it’s like, oh, you gotta install it from GitHub and do all this stuff. I had no clue. First, I went to my friend that I’m working with who’s up, you know, over the top tech guy and he’s like, oh, see if one of my people can do it. Which to me is also like, I’m busy.I got other stuff going on. I’m trying to make money here. So I’m like, oh, because I have these light bulbs. Oh, why don’t I let GPT do this? I gave GPT the thing I said, how do I get this installed? And it it gave me a step by step, but they’re all kind of problems. And I hear I am debugging, figuring stuff out. Having to do these plugins, I’m almost there. Like, I went into the time tunnel, and I’m like, holy shit. I’m coding.I’m coding. This thing is piloting me through this process and I never and if I had had to figure out that and learn that on my own, it would have taken me forever. So just just the point is, like, maybe you do know Python better because you got GTT to help you with it. Anyway, an aside, a story that every day. I’m finding more applications where the light bulb goes off, and I’m like, wait a minute.Why am I not trying GPT for this? Cool. So that is super interesting, and I haven’t talked about that. I’ve talked to Robert Nury from Arctic Shores and looked at some of the stuff he’s doing, and he’s so vocal about this. I I love people really just standing up and going.This is this is it. Let’s let’s move in this direction. I try to associate with those people as as much as I can. But, Jerry, what do you think?
Speaker 3: I was reflecting on Pat’s. Pat’s last conclusion, and I thought should we recommend does developers to sort of commence in this arms race when the space is developing so quickly? So
Speaker 1: I know.
Speaker 3: I I am fascinated by those findings, but and every time the the like, every every few months that we have a a new breakthrough with those large language models. So it feels if you now develop a type of item that is currently resistant. To large language models. It might just be useless to resist these models in in half year from now, and we all know how how long it takes to validate a psychometric test.
Speaker 1: Yeah.
Speaker 3: So there’s also a risk there. When you when you you you commence in this arms race and I don’t have the solution. That’s because I’m I’m really good now at burning down this one constructive idea, but I also don’t have the solution for this.
Speaker 1: Yeah. Yeah.
Speaker 3: Possibly, we need to think outside of the box and develop tests that we we we don’t have currently. And and not not necessary fairly a new variant of the the existing psychometric tests. But Yeah. Would that be? I’m Yeah.
Speaker 1: I don’t know. That’s why the other thing I challenged myself, like, I’m sitting here trying to come up with product ideas. I am thinking but then I’m like, I am not thinking big picture enough. Because you have to zoom out, I think, a lot to say that. And you mentioned the arms race, and I think you’re you’re very right, but I don’t think it’s just in our field.I mean, it’s an arms race overall, and it may even be an arms race of the human race against this stuff. Who knows? Like, I don’t I don’t like I don’t like to get into that too much, but it it’s we’re we’ve uncorked the bottle, you know, the I said the bar on the rollercoasters slam down and we’re in because of all the things it’s doing, but AI is paired, obstacle. In that, there’s always trade offs, and and I think economics are driving the bus more so than regulation or anything like that. Right?So the arms race I don’t think we’re gonna have a a a NATO treaty that everybody’s gonna abide by or whatever on this stuff. So can we think about let’s see. We’re pretty three pretty smart guys who know about assessments. Can we think about a completely new paradigm. Let let’s and Patrick keeps getting to go first.It gives Jerry a lot a lot of time to reflect, and I don’t want it to be unequal here. How about I start with mine and then we can go from there, but you can both reflect while I’m doing this. To me, it’s it’s again there’s there’s two things I think of. And one is and they’re kind of converged. It’s all about simulation it’s all about just doing your thing, doing things related to the job and being evaluated.And I feel like there’s two ways that can go. One, first of all, is, you know, you’re immersed in a world and you have to interact and I think the more complex that is and the more you’re driving the bus, the harder it might be for GPT to do it, although you can make an agent that you just say, hey, agent, go go ace this test. But even beyond that, what I really like and there’s so many actual, like, paradigmatic, if that’s a word, things that may resist this. But it’s just okay, you’re applying for this job, you know, maybe install this app or whatever. Just use your phone.Use do do your thing for a week. You know, we’re watching that or maybe we give you, you know, a fake organization you work for. And you’re just doing stuff for, like, a week or so, you know, having meetings, have conference calls, dude. Reports, whatever. And you’re getting graded on all that stuff, you know.And I guess there’s a large language model play in there, many different places. But you’re just gonna decouple, I think, from here’s a bunch of questions to just do your thing. And and let’s see how well you do it. So and maybe there’s a broader perspective than that even, but that’s that’s as wide as I’ve got. Yeah.So so, Jerry, what do you think?
Speaker 3: I like your idea there. I’ve seen some examples of products that are maybe what you’re describing now, but then somewhat smaller scale where people are developing work sample tests in meta. I really like those ideas. It, to me, feels as if it might be harder, to use tools like GPT currently in that space. Maybe you can develop a sort of controlled environment but you can ward these things out or or give people access to them anyway because if they’re relevant for the job, then why not?And let them play around with it, but then in your own environment. So I feel maybe that is where we’re going. We we do know that, for example, tests are use for instruments. And as you said, you may even be able to use generative AI to quickly develop them because that’s that’s the big issue with them. Right?They’re kind of expensive to develop and you need to develop them each time over and over again for new positions. So there there are really good opportunities here too.
Speaker 1: Mhmm. Yeah. I mean, avatars, you know, look, avatars are we we gotta get past just the three d avatars, you know. There’s there’s not that compelling. I think there’s also gonna be a situation at some point where If you are serving up a job applicant, a traditional looking test, they’re gonna be like, what is this company?How the hell are they doing this? Now, again, we look at this from a white collar lens it may be very different when the the majority of the world workforce is not white collar. They may not have the the ability to to to they may not be just thinking in this direction, I don’t want to again try to put boundaries on what people in any demographic or social economic status are are capable of by any means. But but my experience has been, again, that we’re in this little bubble. It’s expanding.But anyway, so Patrick, what about you? What’s your big idea?
Speaker 2: Yeah. I mean, it’s it’s vague, which is which is great because it means it can be someone else’s problem if implementing it. Yeah. I I think, like, the the key I mean, look, a meta analysis just Paul Saket just released another meta analysis last week in the Journal of Applied Psychology, and and it basically just concluded that you know. Last year, I told you that the validity of cognitive ability was lower than than we originally thought from Schmidt and Hunter’s famous piece.Now, I’m telling you it’s even lower than that. Right? It’s it’s it’s actually close to a copy of what it was, but point two or something. So, you know, it’s it’s sort of getting close to personality. And and you know, you you don’t know if this is just because we’ve got better data or if that if it’s because we’ve got more recent data and and the nature of the work that we do.Correct. Changed so
Speaker 1: much. Another thing. I saw Paul give a talk at the lack about it. And, you know, I don’t correct any of my any of my stuff because I don’t believe in it. But, anyway, that’s another issue there.
Speaker 2: Yeah. Well, yeah. So the over correction explained the first drop, but then this paper we just released last week suggest to see a bit lower than than if you And I think that the key there is it involves more recent data, which sort of implies that these assessments that are trying to get science rather than samples of behavior just not, you know, not as not as useful. And so that sort of lets me makes me think, and I’m going well beyond the data here, but I sort of feel like the key really is to to is to get assessments that enable you to observe behaviors that matter on the job. So I think this sort of leans into the work samples idea.And as as Dura mentioned, and I’m I’m sure you’ve you’ve mentioned this as well before, Charles, that, you know, with with this the the creativity, aspect of of these GPT of of these large language models, it means that developing the content is is easier. And with Darla, you can even make you know, image content, etcetera, to supplement supplement that. I think that we’re not too far away too from seeing that these large language models embedded into assessments that you could do. Fantastic. Customer service simulation or, you know, group discussion simulation.Yeah. The the problem is the guardrails the guardrails problem is huge. Right? And I I think that’s that’s probably gonna take years to sort out. You don’t wanna have a candidate in the middle of assessment discover that this this customer this dissatisfied customer there.They’re trying to deal with starts talking about, you know, far right wing nuts. He probably can’t throw something because that’s that’s gonna be
Speaker 1: Oh, yeah. Yeah. Well, you can control that by I think having your own, like, open source model that you you tweak or whatever, you know. So I had an idea. The other thing is now the ideas of can you use GPT for this?Start flashing like I was saying? So I I don’t know what I was doing. I was reading about a minute analysis. I was like, wait a minute. And I’ve done one in grad school like participated in the coding and stuff.It’s a lot of work. So I’m like, well, could chatGTP doing meta analysis for you? Like, could you feed it a bunch of papers and have it code and then have it, you know, drive. I don’t know. It seems like a lot of work.I immediately emailed a friend of mine who’s an academic and this area and said, hey, hey, can can we ever get chat to give you the mental analysis? I I don’t know, but seems like there’s a
possibility, at least for the coding, just for you guys, you know. So that’s the thing. We just keep thinking of new ways, you know.
Speaker 2: Yeah. I had some experience get into code qualitative information for me. And it it just by coincidence, I I did two rounds of coding, one the day before for dev day and the one one round the day after dev day. And the the the batch that I did after dev day, the quality was was, like, so bad compared to the it was exactly the same code that I used. I used the API.I used the same model, but its performance was very bad the second time around. And that tells me that you can’t necessarily trust it for these kinds of high stakes things. It’s not a reliable machine.
Speaker 1: Yeah. Yeah. Yeah. I I agree. Yeah.I’m looking trying to figure out how to how to pull qualitative you know, themes out of qualitative stuff, and I thought it might be as easy as just feeding it all that stuff and asking it questions. That’s how I’m gonna start. But then I put a thing on LinkedIn and I get a lot of hey, we’re trying that. It’s hard. You know?For me, I’m I’m structuring my prompts to say, okay. Oh, that’s what my plan is. I’m looking for this. Look at all this stuff and tell me what the preponderance of this is, you know. But that’s an adventure.I’m getting more and more complex. So Well, awesome. So we’re we’re we’re rolling to the end here. I wanna give you guys a chance to any this is a great conversation and, you know, I feel like on all these, I could just keep going for a day or two day. It’s just so freaking interesting what we’re dealing with, and it’s resetting a lot of stuff and, you know, we’re all just trying to figure it out, which is great.And when we get research, you know, like you all are able to do, I think that’s great. I mean, we’ve always had a lag between research and practice. Now things are changing so fast to your point. It’s like, how do you how do you keep up? You don’t want your but you can’t stop doing the research at all.You know, maybe we should have ChatGTP have its own journal and be its own refree, and you could feed it stuff, and it could tell you I mean, not just keep coming up with wacky stuff, but, I mean, it it seems anything seems possible now. You could have told me that, you know, a while back, I’d be like, you’re crazy. Anyway, so let’s leave us with some parting thoughts and and I guess Yeah. Also, at the same time, to be economical. Let every let everybody know how they can keep up with you and your research and, you know, I’ll start with Jerry.What do you what do you got?
Speaker 3: Yeah. I really like that you just said that academics have already had a hard time keeping up. But I feel Finally, industry has joined us. They are also having a hard time keeping up. So the developers developers are are there.They’re they’re setting the pace. Industry is lagging behind. Academic are now, yeah, maybe holding on to this train as well. As well as we can, but it’s so difficult, especially doing proper research. But like I said, developing psychometric assessments, it takes time too.Industry is now also looking at this and thinking, oh, how do we deal with this? But but I remember it’s this this Aon report and they showed that the scores on their tests haven’t quite improved since the introduction of CHAT GPT. And they took away from that, that it doesn’t really affect how sort of psychometric assessments. What I’m thinking is maybe there are very few early adopters currently and especially you need to have quite some level of confidence to to start using chat GPT on a psychometric assessment for for your own job application. Right?But there there might be a few people who who managed to do this and they you will not see them in the average of the tens of thousands of people that are being assessed on a daily basis. So, and we live in our little double We see all things chat GPT coming by on LinkedIn and we think, all right, this is amazing. But in practice, what we see is when my research master student, Naomi Lang, was looking at Reddit. Most people just use it to help them practice for interviews. Or to write a cover letter a little bit better, not using it to get assistance as they’re doing the interview.Because that also requires a lot of confidence to to do that. Imagine having that second screen there and then having the guts to sit there and read up from a screen. Well, but we’re having a job on the line. Right?
Speaker 1: Yeah. Yeah.
Speaker 3: So I feel that on the one hand, it’s going very quickly. On the one on the other hand, people are just discovering it, and the majority of people is probably not even using it yet.
Speaker 1: Yeah. Yeah. Yeah.
Speaker 3: In my opinion and so you asked for a main takeaway or sort of closing thoughts I feel as if now it’s starting to become the responsibility of organizations who are hiring to make sure that people are using it in the same way. So that that they can clearly state or at this is how you can use Chegg GPT or cannot use Chegg GPT and these other things in in the selection procedure, so that every all the applicants are then at the same level again, hopefully, as long as people listen, obviously. But, yeah,
Speaker 1: Yeah. That makes sense. So I I agree. I mean, I think a lot of times, again, it comes with the territory here that people are just so, like, Oh my god. Oh my god.So recruiting, hiring assessment, it is not a smoking pile of, you know, anachronistic wreckage at this point. I mean, the the the stuff we’re doing is still happening in it. What you said reminds me of just the the the impression management thinking good on you know, test it that happens, but is it moving the needle against tests in a significant way in in the macro sense now, you know? And I would almost say, maybe maybe people are really clever or or we have, you know, with thinking good, we have little detection traps and stuff. But anyway so, Patrick, closing thoughts?
Speaker 2: Yeah. I I’ll I’ll just follow from that. Yeah. I everyone thought had occurred to me when I began my research journey with chat DBT was, are we just gonna end up with another? Is faking good?Is faking bad debate? That ultimately is impossible to settle without data that’s just too hard generally for researchers to get their hands on. You know, maybe maybe the ability to use chat JPT effectively to write a cover letter that stands out. Not with notwithstanding the legitimacy of the content, maybe that’s a sign that somebody is adaptable and clever and, you know, tech savvy and actually is exactly the sort of person that you want in your organization. I I do think at the moment organizations don’t have a good I mean, I I don’t wanna generalize, of course, but from what I’ve seen.It’s not not many talent acquisition teams that I’ve encountered have really thought carefully about what they what message they wanna send to candidates about the use of technology. And I I think one one concern I do have is if if the if the, you know, reaction to it is, oh, we’ve gotta we’ve gotta, like, you know, monitor everything. We’ve gotta make sure people aren’t, you know, loading up other browser windows or moving their mouse in a funny way. I I just worry about the candidate experience impact that’ll have. And also my my hunch is that this won’t be these these kinds of constraints and conditions won’t be given to all candidates.Right? They’ll be given candidates who don’t have very much power. And and I think that that’s just already a lot of people find applying for work, incredibly painful, stress for frustrating. And I I just would hate to see, you know, generally, if I I lead to more more lucky out of my big business and so forth.
Speaker 1: I think if people in these, you know, that are doing the hiring of those organizations try to fight against it, it’s a completely losing battle. You have to figure out how do we accept this and how do we still get variants in our in our, you know, applications because If you just try to do all that stuff, you’re gonna look bad, but you’re gonna lose probably as well. And I even, like, I have a friend that works for Cidelity, you know, and I remember asking him. I I may have been on the podcast. I can’t remember, but I think so.He’s like, yeah, we embrace it. We know. You know, there’s GitHub as a co pilot, so we just let people use it as part of their problem solving and stuff, you know. And so we’re seeing how proficient are you with it. I mean, that could be another thing because almost any job now.It has a value in efficiency and better outcome for the most part. We know it. You know, it gives false information, etcetera. So I think there’s the other thing is someone maybe we can detect someone who’s over relying on it
because it it itself fails. But but that would be intermittent.I don’t think that would be a good idea. So anyway, we could talk forever. We got an hour. I usually go forty five minutes. But, you know, these folks are doing great stuff.And maybe just remind each, you know, where can we go look at your research? Is there is there a place we can consume this, and then we’ll we’ll play on out of here.
Speaker 3: Basically, we we are quite Googleable. So if you hit Google and LinkedIn, you can find me there in path two. I’m always open to receiving emails. We’re always looking for industry partners to collaborate with. So if somebody is gained to work on a project around chat GPT and how applicants use it, please approach us and we can have a chat.So and also, for general other questions. Just send us an email. It’s pretty pretty easy to find for me.
Speaker 1: I’ll I’ll have GP GPT email you guys with with some ideas for research. But it’ll look like me that does that did that.
Speaker 2: Yeah. I’ll just say LinkedIn’s best way to keep up with my stuff. I I usually post once a week or so. I try to keep them short and and practitioner friendly. And as as Jiro mentioned, we’re always on the lookout for industry collaborations.But Charles, I did want to mention something specific I decided to ask Chachi a bit too what veggie might might taste like my street waffle. And Beautiful. It it’s given me a pretty good optimistic account So it says the combination would likely result in a unique flavor profile where the intense umami and salty notes of vegamide balance the sweetness and buttery texture of the strip waffle. This could either be surprisingly pleasant for those who enjoy contrasting flavors or potentially off putting for others who prefer more traditional pairings. So I don’t know which top you watch, Charles, but I I hope that
Speaker 1: sounds good. Actually, I’m I’m gonna do an idea. We can get vegemite here. We can get the the waffles here. And I’m gonna be brave and do it.I do strange things with waffles. I made a Thanksgiving leftover savory waffle one time where I took all the stuff Turkey and the and the corn bread and stuffing and cranberry sauce, pressed it in the waffle iron. That was pretty good. I make waffle sandwiches. So let’s see.I’ll I’ll let you all know. And then when we do this again next year, we’ll we’ll we’ll talk about it. So thank you all very much. Excellent excellent conversation today.
Speaker 3: Thank you, Charles.
Speaker 1: Yeah.
Speaker 2: Thank you, Charles.
Speaker 1: As we wind down today’s episode dear listeners, I want to remind you to check out our website rocket hire dot com. And learn more about our latest line of business, which is auditing and advising on AI based hiring tools and talent assessment tools. Take a look at the site. There’s a really awesome FAQs document around New York City local law one forty four that should answer all your questions about that complex and untested piece of legislation. And guess what?There’s gonna be more to come. So check us out. We’re here to help.
The post Vegemite, Stroopwafles, and AI based assessments: New sensation, or acquired taste? appeared first on Rocket-Hire.
