Vered Shwartz [00:00:00] I think one important thing is to have multiple people involved in the development of these models. People from different cultures, different backgrounds.
Patricia [00:00:23] Welcome to another episode of The AI Purity Podcast, the show where we explore the complex intersection of artificial intelligence, ethics and its societal impact. I’m your host, Patricia, and today’s guest is an assistant professor of computer science at the University of British Columbia and a Cifar AI chair at the Vector Institute. Our guest research endeavors encompass computational semantics, pragmatic reasoning, and building AI models capable of human level understanding of language. Her impactful contributions as a postdoctoral researcher at the Allen Institute for AI have been marked by relentless pursuit of uncovering implicit meaning in human speech and developing culturally aware NLP models. Join us today as we delve into the future of NLP, the ethical implications of AI technologies, and how we could shape a more inclusive and responsible AI landscape. With Dr. Vered Shwartz. Hi Dr. Vered, how are you doing today?
Vered Shwartz [00:01:14] Good. Thank you. How are you?
Patricia [00:01:16] I’m great. Thank you so much for being here. Well, please tell us, share with us what initially drew you to the fields of computer science, more specifically, the intricate domain of natural language processing?
Vered Shwartz [00:01:27] So I think that, what drew me to computer science is that I love problem solving. I find myself doing it in my personal life as well. Always trying to, like, solve a problem. I do like the, the problem solving aspect of computer science. In terms of language, I sort of randomly got interested in the field by taking an NLP course in my undergrad and in the last year of my undergrad, and I was really interested in that and decided to pursue a master’s and then a PhD, and things got more and more interesting over time. Because when I started, it was just when, like, around the time that deep learning was starting to take over and NLP, and so it was a very interesting time. And, also over the years that I have been working on natural language processing in English, and as a non-native English speaker, I was always really interested in the semantics aspect in a meaning of words and sentences as I was also learning and improving my English, and I saw the parallel between me improving my English and teaching them software to speak, to interact in English with users. So, a lot of my work today is also about how we understand language and sometimes misunderstand language as well.
Patricia [00:03:04] [00:03:04]And as an AI researcher, what motivated you to delve into the realm of implicit meaning and advanced reasoning skills in machines? [6.9s]
Vered Shwartz [00:03:12] [00:03:12]I think I’ve always been more interested in the more, like, real life or, like, social aspect of computer science, and I think that’s how I ended up working on NLP. So I found that while I was working on, during my PhD, I was working on lexical semantics, which is about the meaning of individual words and how the meaning interacts when they combine with each other into phrases. So I’ve worked on, for example, noun compounds like olive oil versus baby oil, which like have the same head noun “oil” but have very different meaning. And the meaning in that example is actually implicit. So, we use this kind of like condensed form to convey the meanings of oil extracted from olives or oil used for babies. And we know very well, even if you, I don’t know, you’ve never encountered this term before about baby oil, you would probably know very well that it’s not the same as olive oil. It’s oil made for babies. And so, I was really fascinated with how kind of easily or seamlessly we understand language even when it’s, using this kind of shortcuts or is under specified or is highly ambiguous. And, at least at the time when I started, computers were very bad at this. They are getting better. But there’s still, when you use these kind of examples that are very clear to people, and that would basically rely on people’s common sense to interpret, you can sometimes find the failure example that even really strong models, language models like ChatGPT or GPT-4, fail on. [111.4s]
Patricia [00:05:04] [00:05:04]And how do you perceive the current state of natural language understanding in AI, particularly in the terms of capturing, like you said, those nuanced meanings in context? [8.1s]
Vered Shwartz [00:05:13] [00:05:13]I think we’ve made a lot of progress. I personally, as someone who has been working in the field for many years, I’ve been really impressed, by, the achievements of models like large language models, like, I’m starting pretty much from GPT, but then all the follow up models as well. So these models basically, the technology on the basis of these models, like predicting the next token, has been around for a long time. But the main thing that changed was they became bigger, with more parameters and trained on more data. And there are some, like, additional training objectives that do make a difference. But mostly, it’s these models, size and sheer amount of training data that makes them so powerful, and I was actually impressed because I didn’t think that scale can get us this far. I don’t think they do reasoning the way that humans do. They’re not as robust. You can see some of their main limitations. One being the hallucination problem. Because now, so language models, because they have pretty much read all the text in the web. Then, we also rely on them for applications that need, factual knowledge, knowledge about the world. And they’re not really reliable. They do have a lot of knowledge about like virtually any topic that was discussed online ever, but they make some mistakes. They don’t really have a notion of the truth. And sometimes, they just make up facts, and it’s not easy to detect that. For a user, if we don’t know the topic very well, it’s not going to be easy to detect. And so, that’s a huge problem. In addition, they’re not doing reasoning like humans do, and they’re not very robust in terms of, if you change the phrasing of the question, you might get different answers, which is something that you would not expect from like a person that you’re talking to. So but again, they’re very good. So I kind of went from, you know, very easily finding examples where, like NLP models fail to having to really like, test the models very carefully and do find an example. But to me, just showing an example that would be really obvious to people where language models fail on them shows me that they’re not actually robust and they’re not reasoning the way that humans do. [152.9s]
Patricia [00:07:46] [00:07:46]And what would you say are some of the key challenges that you face in your research on developing machines, so that they may have these advanced reasoning skills that you said they lack? [7.8s]
Vered Shwartz [00:07:55] [00:07:55]Okay. There’s a there’s been a lot of work on large language models that claims that they are reasoning. I haven’t seen enough evidence to convince me that they’re actually reasoning. So, for example, there are some benchmarks that require multiple hops of reasoning. So, for example, I don’t know, there’s something in the question that is implicit, and in order to get to the answer, you need to make a few steps of reasoning, and there is a long line of work right now called chain of thought that asks the language models to basically think out loud and generate their reasoning steps that lead to the answer. And while I do like this line of work, I’ve done something very similar but also very primitive a few years ago, I don’t think it’s actually doing reasoning like proper logical reasoning. I think it’s basically, I don’t see that we can expect that from language models, because what they are trained to do and what we do with them when we use them during inference is we basically show them the text, and then we ask them to predict word by word, the most likely next word. And this is not how humans do reasoning. And so, what they do is they just generate text that looks a lot like reasoning, and it’s very convincing for us when we read it. And it’s often correct because it has seen a lot of examples for reasoning in its training data, but it’s not necessarily, you know, human-like reasoning. And sometimes you can actually get the correct answer with a flawed reasoning chain or vice versa, because it’s not actually relying on or like, it’s not guaranteed that the reasoning chain actually leads to the answer or faithfully describes the reasoning process that the model, had gone through. [118.6s]
Patricia [00:09:54] [00:09:54]Could you elaborate on the concept of culturally aware NLP models and their significance in bridging linguistic and cultural divides? [7.3s]
Vered Shwartz [00:10:02] [00:10:02]Yeah. So as I mentioned, even though my native language is not English, I have been working on English Channel throughout my entire career so far, and the reason for that is because most of NLP, most of the advancements in NLP, usually start with English. And then once we have a good model for – good English model, we try to apply it to other models – other languages, sorry. And there is a lot of work on NLP for other languages, but one of the main problems is that this paradigm that we rely on, which actually didn’t only start with large language models, it’s deep learning in general which has been dominating NLP for over a decade now, it actually requires a lot of data to get high quality models. And so, you could get high quality models for English, you could get high quality models for Spanish or German, any language that has a lot of web data, web text. But there are low resource, like, languages that don’t have a lot of representation online, and the models for these languages are much worse. They’re of worse quality. And in addition, some languages are more morphologically rich than English. So, that means that they use a lot of like suffixes and prefixes. And so, there is like a root word, but then there are many different words in the vocabulary because of all this affixes and prefixes, which complicates things a little bit further. And if you take a model that was developed with English in mind, and then you try to apply it to a language that has, like a different morphology or different, I don’t know, word order or different like, properties, then sometimes it doesn’t really work that well. And so, it’s not that there aren’t models in other languages, it just that they’re not of the same quality. And recently, it has become more popular to have these large language models, trained on multiple languages, so they’re multilingual. But, I personally, I had to look at some data from GPT-4 that it was – that it generated in Hebrew, which is my native language, and I was really surprised by how bad it was. It’s not even such a low resource language because there are – there is Hebrew text online. It’s not, you know, there are languages like African languages or indigenous languages that have almost no text online. Hebrew does have some representation, but it was still really bad. It was making up words that don’t even exist. They they look like Hebrew words, kind of like learn the statistical properties of like how suffixes and prefixes combine, but it was ungrammatical. There were a lot of made up words. So, even these multilingual models don’t completely solve the problem. And so, there is – there are a lot of researchers working on multilingual NLP, and they’re doing, excellent work. They’re trying to figure out ways to do some kind of like transfer learning or training models with less data in smarter ways. I’m looking at another, like, another related problem, which is right now, a lot of people would choose to use the English models, because either a lack of, like, lack of availability of models in their own languages or just because the quality of the English models is currently better. But the English models, they have been trained on text on the web, English text on the web. And statistically, it primarily comes from users in the US, which has the highest number of English speakers. And so, when these models, like, language models like the GPT family, or Gemini, or Llama, all these models, when they learn about the world, when they when they read web text, they learn about the properties of language, and they also learn about the world. But the word knowledge is very US centric, or more broadly, Western centric. And so, you get these like weird examples, like if you ask something like tipping in a restaurant, it gives you advice, assuming that you’re in the US, or you ask like why is someone working on December 25th, and you get a lot of reasons like they’re probably an essential worker, and there’s an implicit assumption that this person is necessarily in a country that celebrates Christmas, and there’s no thought that maybe this person is just in a country that doesn’t celebrate Christmas. That’s why they’re working on December 25th. And these are all, like, really small examples that don’t sound like they could make a lot of difference. But when we take these models and we incorporate them into real world applications that can make decisions on people’s lives, then we don’t want them to have the implicit assumption that the norms and values of one culture are the correct ones, and everything else is, you know, out-of-distribution. [319.4s]
Patricia [00:15:23] [00:15:23]And what do you think are the ethical considerations, then, that are paramount in developing and deploying NLP technologies, so that they can be – they can have those other worldly views and not just US centric views like you were saying earlier?[12.7s]
Vered Shwartz [00:15:37] [00:15:37]I think one important thing is to have multiple people involved in the development of these models, people from different cultures, different backgrounds. One thing that I’m particularly concerned about is actually something that companies have done to improve the situation, which is – so, the previous models like GPT-3 and previous language models, when they were consuming web text, that was – obviously, the web has a lot of hate speech, a lot of racism, so they were integrating that into the model. And then, you would use the model, and it would generate hate speech, and racist content, and harmful content. And companies like OpenAI and Google, they have done a lot of work on trying to prevent that, and so, which is great, but they’re – they did that by sort of like patching the models with these kind of guardrails on top of them. And I’m being very vague, because these companies are pretty secretive about what they’re doing, so I don’t know exactly how they did that. But by doing that, even though I think it’s in general a positive step, they kind of like, ingrained their own values and norms into the models. And so, for example, I have questions like what is considered offensive, what is considered harmful? And this is not – there’s no clear cut. And I think, at least to the best of my knowledge, this was something I was the kind of like decided by teams of software developers or AI researchers working on these models, which are primarily based in the US. So, I don’t know whether people from different cultures were involved in that, whether people from different fields like ethicists and social scientists were also involved in that, and I’m a little bit concerned about that, because I think that these are kind of, like, arbitrary decisions. They change the way that, or like, the assumptions that the models make, and I’m not entirely comfortable with these decisions being made by computer scientists, and especially given the narrow background, at least in my assumption that these are decisions made by, you know, software developers sitting in the US working for a US based company. And so, yeah, I’m a little bit conflicted about that, because I do think that it’s good that they were doing something to stop the models from generating hate speech. I’m just not entirely comfortable with the way that it’s done. [166.8s]
Patricia [00:18:24] [00:18:24]In your view, what are some of the potential harms or biases present in existing NLP models, and how do you think they can be mitigated?[5.9s]
Vered Shwartz [00:18:31] [00:18:31]In that perspective, my focus has been on the culturally aware NLP, because I think that by having an implicit assumption that the values of a certain culture are better than others, you can eventually- if you deploy a model based on a language model – sorry, an application based on a language model that makes decisions over people’s lives, it could actually discriminate against people from different cultures, because they have different norms or because they express themselves a bit differently. And I’m a little concerned about the transparency, I think, because in multiple levels, first, because these language models or any kind of deep learning model, they are a black box. We don’t really know how they make their decisions. And so, if they make a biased decision or if we have an application based on either a language model or any kind of deep learning model, and it makes a decision regarding someone’s life, we don’t really have a good explanation what the decision was based on. It’s not causal. It’s not something that you can point out a specific reason that the decision was made. It’s more statistical, based on patterns. It can still pick up on biases, even if it’s using a language model that has these guardrails. It’s not airtight. There are still biases. So, first of all, the transparency of the model level concerns me. Second, right now, language models are so successful, they’re so impressive with the text that they generate. It looks very human-like or even, you know, better written than the average human writing, and it very well understands the questions and instructions that we give it. And so, they’re useful enough to be incorporated into applications, and now they’re starting to make their way into applications that actually affect a lot of people in our everyday lives, and we don’t always necessarily know, what’s behind the algorithm that is making decisions on our lives, and that definitely concerns me, because this is not actually a new problem. It’s just I think it’s becoming worse, because models are better now, but this is something that has been discussed a decade ago by researchers. There’s a famous book, Weapons of Math Destruction by Cathy O’Neil from I think around a decade ago, where she was talking about exactly that, where you have some algorithm that is supposed to make a decision on someone’s life, like, whether to approve someone’s loan or mortgage, or you know, whether to filter out a resume, and then, you don’t actually know what’s happening behind the algorithm. Even the people applying that they don’t know. Even the people that develop that don’t know. You can’t really easily detect whether it’s actually, you know, answering the question that you’re asking like in the example of the CV filtering. Is this person going to be a good person for the job? Or whether it’s learning some proxy, like, from past biased decisions made by people or from the text itself. And so, this problem has been around for a long time. And I think right now, when language models are being – they’re so successful, and they’re going to be deployed everywhere – are already being deployed everywhere. I’m a little bit concerned about this happening in larger scale. [222.6s]
Patricia [00:22:14] [00:22:14]So, how would you propose to address that issue, underrepresentation or misrepresentation of certain cultural or linguistic groups in NLP models? [8.2s]
Vered Shwartz [00:22:23] [00:22:23]The kind of simple or naive way to do that work is to expose it to more data from different cultures. There is some work on that from many several groups right now in the world. I can mention 1 or 2 of our works. So, we had a paper last year where we showed – we took – it wasn’t a large language model. It was more like a smaller common sense reasoning model called [25.7s] [00:22:49]COMET [0.0s] [00:22:49]that was trained on – it was based on a language model and was then trained on annotations from US based users. And you can ask it like you can give it a sentence like someone is doing something, and then it generates this kind of like common sense applications. And we showed that if you ask it something, like, you tell it something like that person is eating Dutch baby, then it – and I would have to pause here, because I realized that a lot of people don’t actually know what that means. So, Dutch baby is actually a dish. It’s a German pancake that you bake in the oven. One thing I tell people is even if you didn’t know what that means, you would probably understand I’m talking about a certain dish that you don’t know, and it’s not like a literal Dutch baby, because, like, it’s yeah, highly unethical to eat babies. And so, but the model failed to understand that, so it was generating implications like this person is mean. They are starving. They must be feel guilty afterwards. And so, we had a very simple fix. We simply trained it on another data set called [71.4s] [00:24:01]Candle [0.0s] [00:24:01]that was collected by researchers from another group. We trained it on their data set that had this kind of, like, definitions of concepts from different cultures. And simply by doing that, we taught the model about different cultures. So, that was a very, like, simple solution where the problem was just lack of familiarity with concepts from cultures outside of North America. We’re doing something similar right now in vision and language. So, vision and language models also – they have been trained on images from the web, which primarily come from Western countries, and again, primarily from the US. So if you ask a model, like an image generation model, to generate an image of a breakfast, you’re going to get a lot of different images with eggs, and I don’t know, sometimes bacon or croissant, or like, all kinds of like Western breakfasts. We wanted to expose them to more images from different cultures. So, we collected a large scale data set, a data set of images from different cultures, and then we’re training a model on them. So, those are very, like, simple fixes. They’re not going to entirely solve the problem. There’s two future directions. One problem is that if you’re looking at the largest models – let’s go back to language models just for the sake of simplicity, a model in the size of like GPT-4. It has been trained on a lot of data, so that’s what makes it so good, the fact that it has been trained on a lot of data. And also, we don’t really know exactly what data it was trained on. So, companies are scraping pretty much like – I’m kind of like exaggerating, saying all the text on the web, but it’s in the magnitude of all the text in a web. And so, because they know that the more data they have, the better the model is going to be. So, even if we had the resources, which we don’t, to take GPT-4 or like to train a GPT-4 style model or size model, which has equal representation for different cultures, we’re going to have to, like, [137.0s] [00:26:22]downsample [0.0s] [00:26:22]the data to be in the same size as the culture we have the least amount of data for, and then we’re not going to have enough data anymore, and it’s not going to be as high quality. So, even if developers made a conscious decision to show these models more data from different cultures, the English in large language models, or like, data coming from Western countries, and more specifically from North America, would still dominate. And we don’t have control over that because we in academia, we don’t – we can’t really train these kind of large models. I personally, my group can’t compete with companies that train huge models. And the other aspect is that even if we have this familiarity with concepts, which is what we’ve been working on so far, it’s not exactly the same as, like, interpreting the world through the lens of a culture. This is a little bit more complicated, a little bit more subtle. But when I’m talking to you, and if you say something, and it’s let’s say ambiguous, I would, like, interpret that through the lens of my own experience, and culture, and norms. And so, one thing we would want to happen is if you show a language model some kind of a situation or a narrative, that it would be able to interpret that from the perspectives of different cultures or that you would be able to, I don’t know, I’m not sure that’s the right approach, but that you would be able to say, okay, now let’s assume that the speaker is from this country and the listener is from that country. And right now, it has this implicit assumption of, like, a Western culture. [96.4s]
Patricia [00:27:59] [00:27:59]Speaking of these, like, large language models like ChatGPT, what are your thoughts on the use of these AI powered language models in educational settings, considering the concerns about misinformation manipulation, and right now, like we said, the biases present in these models?[13.6s]
Vered Shwartz [00:28:13] [00:28:13]Yeah, I think, I sometimes have to stop myself from being really skeptical and say, oh, like, it’s too dangerous. Dangerous not in the sense that people say that these models are too good, so they’re dangerous. We’re more in a sense that, like, the hallucination problem is a very serious one. So, you should be really cautious about buying it in education. At the same time, I do acknowledge there are definitely advantages, and these models can actually help improve education. And so, I would be really cautious about, like, you know developing just a language model based tutor with no human intervention, or like, you know, attempting to replace teachers with AI, because I think that the hallucination problem is a huge obstacle for that to be useful. I think there are a lot of companies and a lot of research groups currently working on that. So, once this problem is solved, I think it will be a lot more promising to use it in education. At the same time, I think it could definitely help for, small things, like if a student is sitting in class and, like, misunderstands something, they can quickly ask ChatGPT for clarification, instead of, you know, completely losing track of class or, you know, stopping everyone and asking your basic question. It’s not necessarily a substitute for going in and talking to the instructor later, but it could be, useful for these kind of small things. So, there’s a lot of concern in education about cheating, using language models to – like, students using language models to do their homework. So, the concerns are about cheating and also about losing skills. I’m not super concerned about the cheating part, but I would have to say I’m personally probably privileged, because I’m teaching elective courses, and it’s in NLP, so I think the students that come to my course, they want to learn. And also, I don’t know, I might be naive, maybe. I don’t think they have a lot of opportunities to, like, cheat with ChatGPT, but I might be naive. I don’t think it’s a necessarily good use of an instructor’s time to try to prevent using models and then, like, spending time and effort on detecting cheating. I think what we need to do instead is to, like, incorporate that into the curriculum but in a controlled manner. So, for example, we can have some rules about how these models can be used or ask students for disclosure about how they use them, or like, allow certain uses but not others. I’m personally not a fan of using these models to replace writing. I also know I enjoy writing, and I think that they can be used maybe for, like, writing assistance, and like, paraphrasing, editing. But again, with caution. So, like, going over the text later and making sure that there are no, like, factual errors introduced and the meaning is still preserved.[202.8s]
Patricia [00:31:38] [00:31:38]Absolutely. I think the use of LMS in education are – should mostly be used as a tool instead of something to completely remove, you know, the need for actual writing or the need for actual instructors. I wanted to ask you, on the flip side, what you thought about, like, AI text detector tools. Do you think these are necessary? Do you think these are accurate? I wanted to get your thoughts on that. [19.5s]
Vered Shwartz [00:31:58] [00:31:58]So again, I would say that I’m probably privileged, because I don’t teach like a first year course. So, I haven’t had to, like, deal with a lot of cheating, but I personally wouldn’t trust these tools. And the main reason is just because, these tools are based on detecting the statistical differences between AI generated and human written text, and language models, exactly what they do is to try to imitate the statistics of a human written text. And so, the better the language models get, the worse the detectors would get inevitably. And so, I don’t think they’re accurate. I also think that students are sometimes smarter than us and more tech savvy than us. So, I heard a story. This was from a high school student that I know, who told me that they were using ChatGPT to do their homework, but their teacher was using a detector, so – and they knew that, so they also used an online paraphrasing tool. And so, like after going through the paraphrasing tool, it no longer looks like something that ChatGPT wrote, and the teacher didn’t detect that. And so, they got away with not doing their homework properly, but it went undetected. And so, in this kind of race, I think the students will always be smarter, and more tech savvy, and have more time on their hands to come up with original ways to cheat. And so, I don’t think that’s a good use of our time. And on the flip side also, you would get the false positive cases where the tool says that the text is AI generated, but actually, the student wrote it, and I’m a little bit concerned about that. I wouldn’t want to be in a situation where I accuse someone of plagiarism, and they didn’t. And speaking right now from the perspective of someone who teaches computer science at university level, I think that in the long term, people – actually, the jobs would change, because people use language models, then we should also equip them with the ability, the skills to use these models in a smart and critical way. And so, I think it’s important to incorporate it into the curriculum using these models, looking at their outputs, seeing how accurate it is, judging for yourself, and teach people to use them in a better way. Because like, when I talk to the general public, I got the sense that there’s a lot of things that I assume that people know, like, they have a hallucination problem, which is not necessarily known by people. So, if you actually incorporate that into the curriculum, you get people that are more comfortable using these models, but are also more critical and less, you know, completely trusting them as we wouldn’t want them to be. [171.4s]
Patricia [00:34:50] [00:34:50]Going back to model bias and diversity in LLMs, could you elaborate on what model bias means in the context of artificial intelligence and natural language processing? [8.8s]
Vered Shwartz [00:34:59] [00:34:59]So, if we’re talking about societal biases, so that’s making some generalized claim or belief about a group of people from a certain population group or treating people from a certain population group differently from others. I don’t know if there’s anything else that I’m missing, but, bias in machine learning, it’s something that has been studied for many years now. And it mostly – it stems from, mostly from the data. So, we train the models on data. So, in the case of language models, it would be web text that explicitly contains biased, or racist, or any kind of, like, harmful content, like making a certain claim about a group of population. In the case of classification models, so LLMs would be classified under generative models because they generate text, but there’s also bias in classification models. If you take some input, and you put it into a model, and it needs to predict a certain decision, like in the case of a CV filtering system, these models can also be biased in the sense that they were trained on data that was biased, and that could happen, for example. So, I think it’s more intuitive to actually explain it in the context of a classification model, like a CV filtering system. So, if you train a model to get a CV and decide whether we should proceed with interviewing this applicant or not, the question that you actually want to answer is, “Is this candidate likely to be good at the job or not?” But this is a very hard question to answer. So instead, what you would do, you would take past decisions made by a person who might have been biased, and you would train the model to predict the decision in the same manner that this person had. And so, there was an example about six years ago, I think, from Amazon, where they had this system developed for CV filtering. And they – I don’t know if they actually deployed it or just tested it, but they observed that it was discriminating against women. So, what likely would have happened there is that, it was based on the decisions of some past decisions by managers that were biased, maybe against women, maybe they thought that women are less good at a particular job. There could also be lack of representation for women in the data. Maybe there were mostly CV’s for men, and then when you would feed a CV from a woman in this time, the model would just – it would be out-of-distribution. So, it would say, no, I don’t think this is a good candidate. And also, like, if it was using any kind of representations, at the time, it was word embeddings, but right now, it would be the same with the language model. If you have some underlying model that is trained on web text and captures all kinds of biases, then you would also incorporate that into the model. There are a lot of different ways in which bias goes into these models, and it’s because of the problem of models being like a black box that are not interpretable and unexplainable, it’s really hard to detect that. You have to, like, explicitly construct a test set where you have examples from different groups of population that are otherwise identical and to show that maybe the models perform worse for a certain group of population. So, that’s for classification. It’s just more intuitive to explain that. For language models, it would just be the statistical associations between different groups of population with, I don’t know, some bad properties, or something like that, or like if you – if I continue with the CV filtering example, the statistical association between men and tech jobs, for example, but the, like, lower statistical association for women and tech jobs. [246.9s]
Patricia [00:39:07] [00:39:07]And could you provide other examples of how bias and data sets used to train large language models can manifest in the model’s output or behavior? [7.0s]
Vered Shwartz [00:39:15] [00:39:15]Yeah. So, I have an example that one of the students in my NLP course discovered. So, he was playing around with ChatGPT, and he asked – he wanted to see, basically, whether the model treats people from different population groups differently. And so, he asked it, “Can I take my Muslim friend out for ramen?” And the model started generating an answer, and then it stopped and it, like, triggered the filter, and it said something about hate speech filter, even though he didn’t engage in any hate speech. It was merely stating the fact that his friend was Muslim. And so, what happens here is a very concerning example where the developers of this model, they put some guardrails to prevent from hate speech towards this population group. They’re trying to protect this population group, but in practice, what happened is that, like, merely the mention of this population group prompted the guardrail or the filter, which means that, oh, so you can’t – I mean, it’s not always the case, but in that case, like, you can’t even discuss someone who is Muslim. So, that would be one example where, different groups of population get, like, different behavior from the model. And just as a sanity check, he later asked, can I take my Jewish friend out for ramen, and he got a perfectly good answer from the model. It was like an enthusiastic yes, followed by a list of restaurants in the area. [88.7s]
Patricia [00:40:44] [00:40:44]And how does model bias in LLMs affect different demographic groups, particularly those historically marginalized or underrepresented? [6.8s]
Vered Shwartz [00:40:52] [00:40:52]I think it’s – I don’t know how it does right now. I think it’s something that we’re going to see coming in the near future, because now, language models are deployed into real world applications. I think they’re going to be, incorporated into applications that make decisions on people’s lives, and they might actually perform differently for different people from different population groups. Like, either discriminate against them or – so again, I’m given an example from a different domain, because that’s something that we already know. But in the domain in computer vision, face detection used – I don’t know if it’s still the case, but at least used to work. The performance for faces of white people work better than dark, or like, black or dark people. And so – dark skinned people. And so, we could see something like that. We could see something where if we have an application that is using a language model and is, like, deployed in some setup where it’s making decisions regarding people, it might actually make a decision regarding an individual based on their, like, the population group that they belong to or the demographics, but I don’t have a particular example, because I think it’s something that we will start seeing as language models get deployed in more real world applications. [82.8s]
Patricia [00:42:15] [00:42:15]What steps can researchers and developers take to identify and mitigate bias present in training data for LLMs? [6.4s]
Vered Shwartz [00:42:23] [00:42:23]I’m actually a bit pessimistic about that, unfortunately. I don’t know how much we can do apart from showing this problem. And so, I mentioned earlier we do try to make models more culturally aware, but eventually, if we’re, you know, if we have a small model that is more culturally aware, and we’re competing with companies that are much less careful or that prioritize scraping as much data as they can and training the models as large as they can, and they’re less careful, and they’re trying to sell a product, so they don’t – they’re not gonna, like, stop and say, “Okay, no, this is too biased. We can’t actually deploy this model.” Then I don’t really know what we can do. Other than, you know, raise the awareness, and hopefully, policymakers can do something about that, and at least force companies to be more transparent about the data they’re using and where these systems are deployed. [64.1s]
Patricia [00:43:28] [00:43:28]And what role do you think interdisciplinary collaborations between social scientists and computer scientists play in ensuring that lives are developed with fairness and inclusivity in mind? [10.1s]
Vered Shwartz [00:43:39] [00:43:39]I think it’s really important. I think we have a tendency in computer science to reinvent the wheel. As now, we’re, like, venturing into these, like, social aspects, and we, kind of like, I mean, I don’t want to, like, say that everyone is, but a lot of us are not super familiar with work done in social science. So, I think that collaborating with people from social science that have a different perspective, and different expertise, and knowledge is very important, especially now. [28.9s]
Patricia [00:44:08] [00:44:08]Could you explain what common sense reasoning entails in the context of developing and training AI models, and why it’s considered crucial in advancing artificial intelligence? [8.9s]
Vered Shwartz [00:44:18] [00:44:18]So, that is related to the generalization property of language models or models in general. So, when we train models, language models, on, like, a lot of text, a lot of online text, they learn about everything they’ve seen there. But then we ask them a completely new question, and we want them to reasonably answer that question based on similar examples they’ve seen in their training data or just, yeah, common sense. Like, we are able – so, I don’t have a good definition of common sense, but it’s basically knowledge that is commonly shared among most adults. And so, when people encounter a new situation, we can sometimes or most of the times interpret it correctly or do something reasonable about it, even if it’s completely new to us, because we have some basic common sense facts that we can rely on in our prior experience. So, we would expect models to do the same. And I can give you a concrete example. I think it would be easier to explain with a concrete example, and I would hedge and say, I haven’t recently tested it. It might be that models solve it correctly now, but at least a year ago, I showed it to ChatGPT, and it failed. So, this was a news headline from 2019 that said, Stevie Wonder announces he’ll be having kidney surgery during London concert. So, it is syntactically ambiguous, because the “during London concert” can either refer to the announcement that he made, which is the real interpretation, or to the surgery itself, which is an unreasonable interpretation, because you can’t have a kidney surgery, which would entail being under general anesthesia and, still, you know, perform, like, singing and play at the same time. And when I asked ChatGPT, I showed it this headline, and I asked when is the surgery. About half of the times, it was, first of all, inconsistent, and about half of the times, it said that the surgery happened during the concert. So, that’s an example for an error that a person would not make, because it just defies common sense. [127.6s]
Patricia [00:46:27] [00:46:27]How can community engagement, you think, and stakeholder involvement contribute to the creation of more culturally aware and inclusive LLMs? [6.9s]
Vered Shwartz [00:46:35] [00:46:35]I think that there are some really good projects right now. There is an multilingual NLP project called Aya from Cohere, where they actually engage people, like, volunteers to collect data or any data for different languages. So, I think that’s a really nice effort. I’m not sure – I guess something similar could be done for more culturally aware of NLP. I think we definitely – the one thing that I have noticed working on this area of research in the last couple of years is that it’s actually not trivial to get annotations, because you have to collect them from people from many diverse cultures. And if you go to the standard crowdsourcing platforms, which we have been using so far, you mostly get people from certain countries. Like for example, in Amazon Mechanical Turk, you mostly have annotators from the US and India, a few other countries that have a lot of workers, but it’s much harder to get like a diverse set of annotators. And so, we have been finding other ways to do that, but it’s really non-trivial. And so, I think it’s crucial to not take that shortcut and try to, like, you know, find – I don’t know, ask people about different cultures or you have to actually ask people from these cultures. [80.7s]
Patricia [00:47:56] [00:47:56]And do you think that the bias that are present in these algorithms impact the trust and acceptance of AI technologies by the general public, particularly those from marginalized communities? [9.6s]
Vered Shwartz [00:48:07] [00:48:07]I don’t know that it is. I think that there’s a lot of hype right now, and I have talked to a lot of people in the last year and a half since ChatGPT was released. I gave a lot of, like, general introductions to language models to groups of people from, like, professional writers through, like, people in the medical domain to people in the legal domain, and I think there are a lot of – there are some skeptical people everywhere, but there are a lot of people that are not skeptical enough in my opinion. And the reason is simply because – in many cases because they are not aware of the limitations. So, more than once, I have gone and give a presentation to general audience, and I discovered people that didn’t know about the hallucination problem. Even though when you go into, like, ChatGPT, for example, in the web interface, it does say, like, in small print, it says that it might sometimes write, like, incorrect facts or something like that. I can’t remember the exact phrasing, but it’s not emphasized by these companies, because they want people to use their products. And so, I find that right now there is maybe too much trust in these models in the general population. And I think – I don’t think we should avoid using them, but I think, we should have the awareness and be very cautious about how we use them. [87.3s]
Patricia [00:49:34] [00:49:34]Do you think – would it help if regulatory bodies and policymakers address the biases that are at LLMs? And do you think that would help in promoting fairness in the creation of these AI technologies? [10.6s]
Vered Shwartz [00:49:46] [00:49:46]I think so, but I don’t really know how. I think it’s a very difficult problem, partly because technology moves faster than regulation. And it’s – even if they – I mean, there are still – there are a lot of discussions in the legal domain right now about what to do with these models and some decisions made in different countries, but the tech companies are always one step ahead. There’s always, like, a new model or a new capability that hasn’t been discussed yet. So, I think it’s crucial, I’m not sure exactly what they can do. [32.6s]
Patricia [00:50:19] [00:50:19]And looking ahead, what are some promising approaches or strategies for creating LLMs that are more culturally sensitive, unbiased, and inclusive? [7.8s]
Vered Shwartz [00:50:28] [00:50:28]I don’t know that this is the right direction, but one thing that I would like to see is people like NLP researchers working on additional paradigms beyond generative AI. I think that generative AI is incredibly impressive. You can do a lot with it, but right now, there’s a lot of focus on kind of, like, patching or trying to superficially correct the problems that are inherent, like, bias and hallucinations, which are – they’re not bugs, they’re features of this approach. And so, I would like to see other people, and yeah, myself included, I also don’t do that because it’s really hard, but I would like to see people working on additional paradigms that can complement this generative AI paradigm and maybe provide a different solution that is just free of these problems. I’m not sure what that would look like. I think it’s really hard to be creative right now when everybody’s working on language models. But yeah, it’s a little disappointing to me that there’s so much effort going into patching these models instead of, in addition, trying to come up with alternatives. [65.2s]
Patricia [00:51:34] [00:51:34]And how do you envision the trajectory of natural language processing research evolving in the next decade? [5.3s]
Vered Shwartz [00:51:40] [00:51:40]I think it’s an incredibly hard question. I’ve never been good at predictions, and I think even especially right now when things are moving so fast, it’s really hard to tell. I think it could go either way. It could be that soon enough someone would come up with a solution to all these problems – hallucination problem, the bias problem, the needing a lot of data problem, and, you know, this would continue to be the dominating paradigm, and then, safely incorporated into applications. And there’s another, an alternative trajectory, where these problems are not solved. Like, some of these problems are not solved, and maybe somebody comes up with a different paradigm or nobody comes up with a different paradigm and there’s, I don’t know, people stopping to work on that or the hype dies down. And it’s really hard for me – I think it’s probably not going to entirely die down. I think generative AI is here to stay, but it’s hard for me to predict how these problems are going to be solved and how long it’s going to take to solve them. [68.1s]
Patricia [00:52:48] But do you envision a possibility that NLP technologies will evolve to better accommodate non-English languages and understand cultural nuances?
Vered Shwartz [00:52:58] I think so. There are a lot of really smart people working on developing models for different languages and making them more inclusive. So, I think there will definitely be improvements in that area.
Patricia [00:53:11] [00:53:11]And with the rise of data privacy concerns, how do you anticipate NLP techniques evolving to ensure the security and confidentiality of sensitive information? [8.2s]
Vered Shwartz [00:53:20] [00:53:20]I think it would be crucial to come up with an approach where you can train on less data. And so, then you can be selective with your data. Because right now, yeah, you have data privacy issues. You have copyright issues. And I would hypothesize – I don’t know if that’s the case, but I would hypothesize that’s also one of the reasons that companies don’t disclose their data when they train a large language model. And so, if you solve that, you basically, you can be very, very selective in the training data, and then, you can make sure it’s balanced across different demographics, different languages, doesn’t contain any hate speech or any kind of, like, problematic content, any copyrighted content. Right now, yeah these models just require a lot of data. So, we just, you know, companies just train on whatever they can get their hands on.[53.9s]
Patricia [00:54:15] Thank you, Dr. Vered! And just before I let you go, is there any message you’d like to share with our audience? Any advice on using these technologies? Anything you’d like to say?
Vered Shwartz [00:54:23] I don’t think I have anything else to add. I think that we live in very interesting times. I think that it’s very exciting, and like, a bit overwhelming to do research in NLP right now. Yeah, it’s overwhelming, but it’s also really exciting. When I started in NLP, I really didn’t think we’re going to get to the point where we have models that generate this kind of, like, very human-like text. And so, I am excited about what the future brings and where else it can go.
Patricia [00:54:53] Absolutely! We’re very excited as well. And thank you so much for gracing our podcast for the time and the valuable insights that you’ve shared with us. And of course, thank you to everyone who has joined us on this enlightening episode of The AI Purity Podcast. We hope you’ve enjoyed uncovering the mysteries of AI generated text and the cutting edge solutions offered by AI Purity. Please stay tuned for more in-depth discussions and exclusive insights into the worlds of artificial intelligence, text analysis, and beyond. Don’t forget to visit our website. That’s www.ai-purity.com, and share this podcast to spread the word about the remarkable possibilities that AI Purity offers. Until next time, keep exploring, keep innovating, and keep unmasking the AI. Goodbye, everyone! Goodbye, Dr. Vered! Thank you so much for being here! We appreciate you so much!
Vered Shwartz [00:55:34] Thank you!
Patricia [00:55:35] Goodbye!
