Eric Hanselman

Welcome to Next in Tech, an S&P Global Market Intelligence podcast with the world of Emerging Tech Lives. I'm your host, Eric Hanselman, Chief Analyst for Technology, Media and Telecom at S&P Global Market Intelligence. And today, we're going to be kicking off Pride month with the discussion of language and how it gets used in corporate communications and some tools using artificial intelligence for taking a look at what that language really means and some things that we can intuit around it, and joining me to discuss this is returning guests, Emily Jasper, the Director of Data and Insights at SP Global Market Intelligence; and Ilan Attar, the Head Data Scientist at Pronto NLP. Welcome to you both.

Emily Jasper

So great to be back, Eric, especially for Pride month.

Ilan Attar

Yes, it's a pleasure to be here.

Eric Hanselman

It's great to have you both here because you've been doing some research around corporate language and expressions. And actually, why don't you dig into it and give our audience a little bit of background in terms of what you've been looking at.

Emily Jasper

Sure. We are continuing research that we've already conducted over the last 2 years to take a look at how data specifically for the LGBTQ+ community can be represented in various forms. Initially, we did survey collection on where companies actually collecting data about LGBTQ+ employees.

And last year, we looked at keyword mentions in ESG filings, so those documents that are specifically geared towards kind of the S or social components of ESG. This year, we wanted to take another look at it and in partnership with Pronto NLP, we looked at LGBTQ+ keyword mentions within earning transcripts or other kind of analysts and public calls, and we expanded the time period to start in 2010.

And our goal was to identify how these mentions could potentially be categorized, are they passive mentions, do they express intent? Or are they actually delivering results. And so we posed question, went through lot of rounds of review, and we're really excited to be bringing some great insights through this paper during Pride month.

Eric Hanselman

So coming from 2010, you've actually got some longitudinal breadth here in terms of what you're looking at. And I think we can all agree that the last year somewhat has been challenging for diversity, equity and inclusion efforts at the corporate level. So it's interesting to see the results that you put together around this. Some insights in terms of some directionality. But can you discuss a little bit about what you are actually putting to work in terms of doing the analysis.

Emily Jasper

Well, I am not the expert on the models, and I would love to have Ilan talk a little bit about that because I think this is an educational opportunity for me based on how we perceive these types of mentions and the types of keywords that make sense? And then having to train a model can kind of follow along in those perceptions.

Ilan Attar

Right. I think both of you really kind of hit it on 2 points, the 2 main points really, which is, given that whole history, do any patterns emerge and can we accurately using this cutting-edge technology, analyze these mentions for all those different vectors that Emily described. So we kind of expect companies to at least mention something about inclusivity and how diverse their companies are or how much they'd like them to be. But I think to Emily's point is how often does that materialize? I think it's a powerful question that needed to be asked, and I think this is a good first step to answering it.

Eric Hanselman

AI models, you heard a lot about natural language processing capabilities, NLP, of course, been using it for sentiment analysis in all certain dimensions. It sounds like the study though was looking not only at some of the sentiment, not only the mentions initially, but also some of that stronger sentiment, Emily you are identifying differentiating between passive use and more active use. It sounds like you're really digging into specifics beyond the -- just the mentions themselves.

Emily Jasper

And Ilan actually just said the perfect thing, which is, this is just a start. We were scratching the surface. The model was looking at mentions and we categorized the mention type. But then the model also added in some additional vectors, we didn't even get a chance to dig into yet, which is exciting because that means we already have our plans for next year.

But taking that step from the question based on all of the critics of diversity, equity and inclusion programs and one of the key phrases that happens around June when corporations change their logos is, are these firms pinkwashing or performing a mention of inclusion limited to a period of time in a sense of trying to appear socially inclusive while still maybe not following through the rest of the year and/or with other programs for their employees.

So that was the key focus. We know there's a lot of directions we can go. And there are a lot of other characterizations on the data itself that we can add utilizing the rest and breadth of the Capital IQ data sets related to terms, their sizes, who the speakers were, et cetera, but we really wanted to dial into this question right out of the gate. Is that mention something that has a little meat behind it? And that's where utilizing the model to help us identify that was very helpful.

Ilan Attar

Right. And I think it's a good point that you bring, Eric, is that doing keyword analysis and maybe counting how often these words appear, that was maybe NLP technology 10 years ago, 20 years ago. As you've mentioned, things have really come along and definitely at Pronto NLP, obviously, our tools are geared more towards the investment side of these calls.

But using these advanced models, we can go way beyond keyword count, right? We can really understand what's the sentiment here in terms of the nuance in the language, not just to throw away what our model calls puffery kind of statements, but have this company actually put their money where their mouth is. And I think it's a good way to use AI for good, right?

We have a lot of examples of, at least in the news of AI being used maybe for -- maybe not quite nefarious but questionable kind of outcomes. And I think it's important to show people that you can use AI to hold people and companies accountable, especially with this huge growth in hey, basically across the world, I think it's a good time to take a step back and to hold people accountable and to use tech to make the world better.

Eric Hanselman

Looking at the data, curious what was the corpus that you were looking at? And Emily, you mentioned Capital IQ Pro, some of our sources of data. What types of data and statements were you looking at? Were these primarily press or financial statements? Well, what's that corpus look like?

Emily Jasper

Our machine readable transcripts for data related to earnings call, company presentations, allows for these types of models to read across them. We have global English as a language for the model so that, that way, it can pull everything. And these transcripts include quite a bit of history. As we mentioned, we were able to start looking at 2010, but the transcripts go further back. And part of the reason we wanted to utilize these were this is a call often that has a CEO or an executive answering to the street or giving information about their progress and goals.

It also allowed us to see some analyst questions, which were very interesting in the back and forth in the early 2010 specifically related to Disney, for example, and their support of The Human Rights Campaign and having same sex couples participate in Disney events and a lot of criticism then, which obviously Disney, in some cases, is still experiencing today.

So we could see that history that far back. Whereas I think in some cases, things like social media that dramatically changed over the last, we knew that, in some cases, that was maybe not as meaningful of a commitment of where the firm was actually making these achievements. And so these transcripts are really where we wanted to dial in.

Eric Hanselman

Interesting. Consider the fact that you've now got both sides of those conversations aren't earning calls because you've got not only the corporate statements that are being made, but the analysts who are the ones who are doing the inquiry on the other side of that and being able to pull together some social context that's coming in from outside the organization from the analyst perspective.

So if we think about what some of these results are really indicating, Emily you used the term pinkwashing and I think we see a lot of categorize in these performance statements. Again, as we roll into June it is an area where I think organizations see they want to be able to make a statement and a positioning. But what were you really seeing in that data? And what are the results turn out?

Emily Jasper

I will say the surprising thing that is not surprising is that there is a lot of what I would consider umbrella use of diversity, equity and inclusion efforts in which you can imagine a firm or whoever is making this statement is assuming the LGBTQ+ population is included. And specifically, that comes up a lot with gender parity effort.

So I think, in particular, that is incredibly surprising because those efforts are still only based on a specific gender and are not necessarily pointing out these underserved communities and individuals. So there's definitely movement, and there's definitely indication of expanded health care benefits for same sex couples, et cetera. But it is pretty clear in some of these areas of measurement, we have room to grow.

Eric Hanselman

With these areas, because I think to your point, there are sort of first order statements and understandings that organizations have. And then as they're going through this journey, really understanding, especially issues around gender moving beyond just binary interpretations and getting into more mature understanding of the entire spectrum that's out there. Is that something you're seeing progress there? Are those an extent that you're looking at for some of this analysis?

Emily Jasper

I think one of the areas we have to maybe accept that this data can -- what they correlate with is that the critics of DEI programs might then force some of the firms to reduce their effort in measurable programs, which means we may have to assume that the best we can do is gender parity studies of executives or gender pay rates because it is a lot easier and in many countries, legal to track gender of employees versus sexual orientation.

And I think in my experience, knowing that everyone has the best of intentions, but so many hours in the day that might be the reality is we have to take into account that the tools to measure impact on LGBTQ+ programs might be severely limited just by resources and perception on kind of the legal stance collecting that data.

Eric Hanselman

That's something that you brought up on previous episodes and we've had a chance to discuss about just the extent to which organizations are able to define the metrics that they're using and to help them understand what their employee population, what their end customer populations look like, the fact that some of these really wind up being much more nuanced than they may actually have in terms of capabilities. And that may be a self-limiting factor in terms of how they actually put some of this together.

Emily Jasper

And think about the countries where some of these global firms either are based or working in, it might still be illegal. So a version of maybe coated language to demonstrate that inclusion might be related to more gender-neutral statements or programs as opposed to pointing one out or even intentionally indicating something related to the queer community as an effort to protect their employees.

And so we also have to kind of take that into account a little bit, which is why I think in some cases, we were hesitant on maybe that sentiment question, something is positive or negative because in some cases, what might have been considered negative might be actually a supporting statement that hate speech is not permitted, even if hate speech itself as a phrase might be considered a negative phrase, but a firm that has global support indicating that they do not tolerate hate speech, might be again a way that they can indicate for employees in those countries where it's either not permitted or illegal or still considered taboo that they can feel that work is a safe place for them to go.

Eric Hanselman

There is at least an ability to be able to signal that intent, whether or not explicit statements can be made in the framework in which they exist. Well, that's an important point. I think about sort of the directions in which you're looking at, you talked a little bit about the slate for the year ahead. What do you see as the principal accomplishments you're putting together so far and where you want to take?

Emily Jasper

Well, Ilan, I'd love for you to tell Eric, a little bit about just the fact that Pronto NLP has dialed in to ESG-related models specifically?

Ilan Attar

Right. As I mentioned before, let's say, main focus is sentiment from a financial perspective. We've had a lot of interest from our customers to kind of take a different look at these documents from the ESG perspective. Now there's a whole body of literature out there of how ESG relates to financial returns or even if that's the right question to be asking.

So we kind of turned some of our tech stack towards training an ESG LLM model, a large language model, which not only can understand context and be more sensitive to the nuances, but also to give us a little more background and analytics on these sentences.

So things like is this company giving any qualitative or quantitative evidence to their claims. So from a classical ESG perspective, you might think of a company says they're going to reduce their greenhouse emissions. That sounds great. But did they give a number? Did they tell you by when? Are they reducing them? Are they cutting them down totally? These kinds of questions are much more nuanced and much more interesting.

And so when Emily and the team approached us to kind of focus the analysis on diversity and the LGBTQ community specifically, we kind of took it as an opportunity to say, okay, let's see how these models can actually perform on something a little more nuanced and a little more directed. And I think it's important to mention, as we did before, is that these are not ESG-related documents.

So whereas you might expect a company to kind of blow air in their own sales in the ESG documents, these are documents speaking to the Street, we mentioned already. And so we kind of expect the company to maybe be a little more measured in the way they speak. There might be some blowback from analysts. There might be blowback from the market in general. And so we really want to measure, are these companies really doing what they say they're doing? And if they don't say anything, that's also kind of a red flag.

Eric Hanselman

A lack of statement is just as much of a statement as real statement itself.

Ilan Attar

Right.

Eric Hanselman

To geek out a little on the model side, building the training data for this seems like that would be a fairly complex undertaking because you're talking about nuance in language in ways that are really stepping in very complicated environments and, as you're pointing out, also measuring any lack of statement around this as well. That sounds like that can present some significant technical challenges.

Ilan Attar

Yes. It's a good point. The model itself is based on, let's say, the architecture is based on popular open source model released by meta. But of course, we had to kind of retrain it from the ground up, not only to focus on the things we're most interested in, but also for us to be able to use it kind of at will, right?

So I want to be able to use it on ESG documents and also all kinds of other documents that might be related. So it's a fairly intensive process. Obviously, we kind of went through a few iterations and built up a nice training set, and refined it over time. We had a few design partners as well to kind of point us in the right direction. But again, it's -- as with all of these models, it seems like it will be a work in progress, and we'll see what the paper looks like next year.

Emily Jasper

I will say one thing that I appreciated in this was questions we asked ourselves of how finite did we want to be in the definitions of these categories. And in some cases, we may have had instances where specifically because of the words like pride, there are so many statements that say our firm takes pride in and it can have any number of things that they take pride in and it has nothing to do with the LGBTQ+ community.

So we had to have mechanisms, we're kind of sorting that out a little bit, which was another interesting challenge. And then I think at the heart of it, and this was a little bit inspired by my experience with The Human Rights Campaign Equality Index and Stonewall's Equality Index is drawing a little bit of a line in the sand of what we consider action, is this something that was delivered, was this something that was planned?

And having that definition between something that was planned or had intention versus was delivered. And then as Ilan said, delivered with a quantitative measure was really important for us to gain, because while we understand there may be signals, there may be coated language, at the heart of it, if your firm is like standing strong on supporting LGBTQ+ employees, looking at the opportunities of, did you say it? Did you infer it or did you actually state it?

And I think that's a big difference when someone is working on their prepared remarks. And employees want to see themselves represented at this level. They want to know that the firm is supporting them and sometimes going ahead and just saying LGBTQ+ or same sex benefit is important, especially in these types of public forums.

Eric Hanselman

And the question though is, is that much more complicated piece of correlating the statements and determining action, which is no small feat, it seems, in terms of really understanding what was actually achieved. That seems like the correlation of that, the statement to action piece adds even more complexity to this project.

Emily Jasper

Well, I think that's something that we would have to dig in definitely in the future because that is one area with the history we have and the fact that the data set allows us to use a unique identifier for the firm, we could when we have time off the side of our desks look at were statements by the same firm 5 years ago that we classified as intention statements, did they show up as action with qualitative or quantitative evident later?

And that will be a big question down the line. And I think that will be an even bigger question when, as you might assume there was an increase in 2021, especially in light of George Floyd and some of the kind of very proactive inclusive efforts that companies made in their statements, there was a rise and an increase in these mentions. The question will be 5, 6 years from then did they deliver. And we have now at least a mechanism that we're in a position to keep an eye on that.

Eric Hanselman

It sounds like this has built a really solid base to work from. As you're pointing out, a lot of really interesting places to go with this. Well, this has been fascinating. I guess the report on this is going to be published probably sometime shortly after the podcast actually aired.

So I will point our audience members to that. We'll have links in the show notes pointing you to where you can actually get a hold of some of this data because it's fascinating to look at, at least in its preliminary state, all sorts of really useful insights.

Well, this has been great. We are unfortunately at time for this episode, but thank you both for coming on, and maybe we'll take a rain check to bring you back when the final study results are in and take a look at where this goes next. But thank you both. And that is it for this episode of Next In Tech. Thanks to our audience for staying with us.