Tim Baker: Good morning, everyone, and welcome to our panel discussion on how to deliver real business value through generative AI
Tim Baker: my name is Tim Baker. And I lead the financial services practice
Tim Baker: at expiry.
Tim Baker: let's go back one
Tim Baker: at Xpero. We build systems for expert users, and in the process we span a wide variety of disciplines ranging from product strategy through to full stack deployment. And you can see the industries that we cover there
Tim Baker: in financial services. The practice that I lead you can see that we have a number of large clients, and about half of our business is actually in financial services where we build systems either bespoke across these verticals,
Tim Baker: and various use cases, and we also use our connected framework product. Set that Chris, my colleague, who's on the call he leads, and this allows us to accelerate the development of delivery of a variety of these solutions.
Tim Baker: So that's the infomercial over with. Let's get into it. If you have any questions. By all means ask them on the zoom. There's a QA. Button at the bottom so please do that as we go, and we'll try and monitor those and feel those as we go, and then we'll do a final check at the end as well.
Tim Baker: So when I asked GPT. the other day to take a look at this image, which I uploaded it correctly, identified the chart as the gardener hype cycle.
Tim Baker: and also correctly observed that geni Gen. AI is indeed at the peak. So in theory, we need to prepare ourselves for the so called trough of disillusionment.
Tim Baker: When, according to Gartner, the interest wanes as experiments and implementations, implementations fail to deliver producers of the technology shake out or fail. Investments continue only if
Tim Baker: the surviving providers improve their products to the satisfaction of early adopters.
Tim Baker: I'm sorry. But I just don't think we're going to see this trough in this case. Given the continuous innovation and advancements and ever expanding adoption of the technology.
Tim Baker: Regardless. This panel is about the how to of dodging those bullets, how to choose the right project and get it off the ground, and ultimately to delight users.
Tim Baker: So we have a pretty amazing panel, and I'm really pleased that they could join us this morning.
Tim Baker: I wonder if we could start by just having you introduce yourselves
Tim Baker: and describe your role at your firm, and how your role relates to. Obviously this topic. Also tell us a little bit about how you got started looking at Ml. At your firm machine learning and more recently generative. AI. So
Tim Baker: let's start with that, Chris on your side, and then we'll work away across from left to right. Sure. Thanks, Tim. Chris Lacova. I'm the head of product here at Expiro
Chris LaCava: where I got into generative AI first and specifically large language models was actually through looking at user experience development. So a lot of stuff that we'll be talking to today around use cases about enhancing user experience. At Xpero, we use
Chris LaCava: we use this these types of technologies to really look at data analytics platforms. So that's the type of products that are in our wheelhouse and and we'll go into more detail. As the as the discussion follows.
Tim Baker: Right? Thanks, Chris Adam.
Adam Wheat: So, Adam Wheat, I am a CTO for the Enterprise Solutions group at Morningstar.
Adam Wheat: Morningstar, for for those of you that know it is. It has a a lot of different businesses and a lot of different services. But very much financial data and financial reporting centric type of business.
Adam Wheat: Morningstar's had a very long history with AI with Ml, we collect data on just about it. Every imaginable investment, vehicle and security that exists on the planet a lot of that comes from unstructured sources. So
Adam Wheat: machine learning and and AI approaches to data, extraction and normalization have been around for a very long time at Morningstar.
Another of morning stars.
Adam Wheat: large business units, is, is doing fundamental research on investments. And so we we've employed generative AI there for some time in in the sense of of auto writing and and helping analysts construct take data and turn it into to to written analytical reports. And and so it's been around for a long time. I I'd say that the leap in the last year is is is really remarkable, and and we launched. We'll talk about this later earlier this year a. A an AI assistant or chat bot around our research. And so it's been a
a long fuse, but a very, very strong uplift in in recent months, in terms of what we delivered to users on it.
Tim Baker: Great thanks, Adam Graham.
Graham Ganssle: Yo, I'm Graham. I support machine learning efforts at wayfare for both customer service and sales, as it relates to generative AI applications
like Adam's crew. We have a
Graham Ganssle: long history in in machine learning tech. I think we've been deploying Ml stuff for about years
Graham Ganssle: and just like everybody else in the world. We said, Hey, what can we do if we use generate? AI so we are using Lms. Jenny Itech to interface with
Graham Ganssle: customer interactions in in some way or another. And we'll we'll get into the details later on today.
Tim Baker: Brilliant thanks, Graeme. And finally, Michael.
Michael Young: Hi, everyone so obviously, I'm Michael young CTO Reuters. And our history really of getting involved and being involved in AI goes way back and it does start with machine learning and automations for those of you that I understand our business a little bit more deeply. You'll re you'll recognize that the London Stock Exchange
Michael Young: products, such as Workspace and Icon and others carry Reuters news exclusively. A. Or rather, we're exclusive to them for financial customers and products. So that's really where the genesis started in terms of automations. And delivering things like company data in re, relevant to stock exchange and postings and sec filings and and economic indicators. We get out of various economic reports globally.
Michael Young: Automations are really an important part of, you know, having millisecond delivery to that. So that's just evolve, naturally with AI over the last years. We've got more and more sophisticated. We've deployed developing tools and then, you know as we get into generative AI, we we've done some level of AI technology approaches in terms of
Michael Young: a A if you will cybernetically enhancing our journalist teams. But obviously those tool sets are evolving as we go along. So that that's our focus. Our interest now is how to take a journalist and make them times more productive in terms of their output.
Tim Baker: Brilliant thanks, Michael. So we're gonna talk a little bit about used cases first, as as folks alluded to. So I thought we'd kick off with a little bit of a grounding of what we're seeing as a software company here at Xpero in the field
Tim Baker: building complex systems for expert users. So so, Chris, I wonder if you can provide a brief overview of some of the high level use cases for generative AI and the tech, that kind of comes to play when Jenai is most applicable.
Chris LaCava: Yeah, absolutely. What we're looking at here is really just the continuum of how things can get complicated as you implement these things. But I think that what should probably be the most salient takeaway on this is that a lot of a lot of people tend to think when they're getting into this, that they've gotta train a large language model to get some of the stuff off the ground. And so if you look at the the list of these things.
Chris LaCava: the top, the top of the list is the most complicated. And I think, the point that I kinda wanna drive home with this is that you can actually do a lot just from programmatically engineering really smart prompts. And you can get a lot off the ground
Chris LaCava: doing that very quickly. And then there are ways that you can. Even enhance that with some vector based data,
Chris LaCava: services and fine tune stuff on an existing Llm before you get to to to really building something from the ground up at all. In fact, I would say, very few people move forward are, gonna are are not, are, gonna build
Chris LaCava: and train something from the ground up. That's it's it's probably cost prohibitive at this point, and anything out there is gonna outperform it. So I think the the takeaway here is just you actually can do a lot by being very pragmatic and quick and dirty.
Chris LaCava: by by just proving out of point, with with starting with prompt engineering and then moving up the ladder from there.
Graham Ganssle: million dollars is what it cost to train. JPT. .
Chris LaCava: Yeah, yeah.
Tim Baker: yeah, I think we still Bloomberg spent quite a bit of money as well in building that and building out their own proprietary model. Yeah. And I mean, I think there was less than
Chris LaCava: months between and and was like, you know, light years ahead. So even if whatever is out there can't do what you needed to do right now. You know, it's it's probably gonna be right around the corner
Chris LaCava: and then just going through a few use cases. This is not an exhaustive list, but it's stuff that we certainly see at expiry. And where th this is some of the technology applications that we've been using. So I think Adam mentioned this already. But digital assistant is usually the first thing that people think of kind of a
Chris LaCava: a chatbot on steroids, a co-pilot to help you navigate through. You know, different interfaces. If it's embedded in an application or do certain tasks, link things together, transport data from one place to another, or just explain things and have have someone you can ask questions to.
Next slide, Tim.
Chris LaCava: We do a lot of work with knowledge graphs and no knowledge. Graphs are very powerful for their analytical unstructured format. But
Chris LaCava: when you put an Llm. Next to them you actually can use those things and and in synchronicity where the Llm. Can gain a lot of semantic knowledge from the graph itself, but also the graph can, be used sometimes to fact. Check what the Lm is telling you. So we'll probably get into a little bit more details here. But there's a lot of really exciting things happening with knowledge graphs and large language models and generative AI in general. That I'm sure we'll discuss
Chris LaCava: next.
Chris LaCava: This is one at expiry that we focus on a lot, and I'll I'll be talking about a little bit more. But AI driven workflows. So being able to actually have a system into it, and use semantic modeling a lot of cases to understand what the user is gonna do next and build those workflows based on based on what's what's transpiring in real time and a lot of times within analytics app applications specifically. But I think a lot of things in general.
Tim Baker: Yeah, I love this world because it's I think a lot of the complex systems we build
Tim Baker: take quite a lot of learning and navigation. But if you can pose a question and have those components
Tim Baker: put in front of you depending on the use case that you're looking at. That's incredibly powerful. I think.
Chris LaCava: yeah. And investigatory workflows and data analytics is not linear. So I think that especially in this case, when you're doing scenario building in in conjunction with dynamic workflows a lot of times to have that. That analysis be responsive is something that helps a lot in in these analytical workflows.
Chris LaCava: And you know, of course, like you, wanna whenever you get information
Chris LaCava: in real time, you may want to build scenarios, or you may want to try. Or what if that had just occurred to you based on that information? So it's kind of taking what you would do in a Jupiter notebook, for example, and putting, you know, a conversational piece around it. And I think that's that's that's super powerful.
Chris LaCava: And then I think we've got one more scenario, maybe a couple of more that I that I pulled out, but a large amounts of data, especially unstructured data or complicated data like time series. Even if you can visualize it, or you can bring that to the foreground. It's really hard for someone to digest just a lot of cognitive load. So they miss some of the non obvious pieces of of complex data data like that. And large language models
Chris LaCava: can look at that stuff and pull out a lot of patterns that a yeah, a human may be able to see. But at scale it's really hard to see. So we use this a lot when there are non obvious connections, or there are there are things, you know, bundled up in time series. Or there are a lot of filters to be able to to get through data. A lot of times. A large language model can sift through that and give you some hints.
Chris LaCava: And then the last scenario that I just want to touch on here before we jump into discussion on the next slide. Is it being able to? It's kind of an offshoot from what I said, but being able to take a lot of information, both structured and unstructured, and in be able to summarize and give the user.
Chris LaCava: You know, a digestible interpretation of that and hopefully, some of the folks in journalism will kind of talk about this. But it's a way to digest a lot of information and suit in in in Turbo charge a lot of information.
Chris LaCava: summarization, so that person can then convey what? What a lot of this stuff that had been crunched down says. So it, you know, in the in the case of journalism, you can take a lot of disparate information, pull it together, structure it, and give that person basically the bones to be able to go through and then enhance it and and tell that story.
Tim Baker: Yeah, I've been hearing a lot of use cases on the by side.
Tim Baker: You know, Web, you know when I was an analyst you'd have to pull together lots of pieces of information, build a model read lots of filings, assimilate lots of things, and you know, I think someone said to me the other day.
Tim Baker: you know, you can now do the work of with just people.
Tim Baker: So it's gonna be interesting to see how these types of tools are adopted. And I'm sure Adam will talk a little bit more about this. And and Michael as well.
Tim Baker: So let's let's allow our speakers now to kind of double click on some of that. So I'd love to dive into the specific use cases and examples that you guys have been experimenting with and building products with.
Tim Baker: So let's go around the group. Because I know you've all got fabulous stories. and if you can kind of talk about the genesis of each project or the project that you've been focused on.
Tim Baker: you know, what were you solving for how do you get the project underway? And ultimately you know what was delivered? Let's start with. Adam
Adam Wheat: W. Wonderful I'll I'll tell what what is
Adam Wheat: Hopefully a a story of inspiration with some some cautionary tales sprinkled in along the way.
Adam Wheat: So Morningstar launched a generative AI Chatbot called mo Morningstar mo earlier this year. And and I'll go through the the process that that led to that, and and some of what we learned.
Adam Wheat: It was. It was a very rapidly developed application, a very short timeline. So we were working with
Adam Wheat: you know it. It's funny to to sit here today and realize that it hasn't even been a year since Chachi Bt. Was released.
Adam Wheat: Let let that sink in for a minute. And so, and the amount of of of generative model progress that's been made in that period of time has been incredible. So if we take ourselves all the way back to the first quarter of this year when kind of chat Gp was very new, and it was, you know, churning up most of the headlines around AI and the access to the Apis for GPT.
Adam Wheat: W. Was a relatively new and novel capability. At at least in terms of easy access. We we'd started working with chat with your documents. Type concepts. Right? So what's what? At the time, I don't think had a name, and and has become.
Adam Wheat: placed under this moniker of rag retrieval, augmented generate of AI rag and so we were working on that. You know, what could we do with this? We have a lot of products. Maybe we could do a smarter assistant to read product manuals and and help people work with that. So we had some Pocs going, and it was. It was very impressive, very exciting, and successful at that level, and and in in the vein of no good deed goes unpunished that kind of went up the chain, and
Adam Wheat: our our the CEO of Morningstar Kana. Kapor was very interested in. What are we doing with AI? And and this came across his radar, and he's like, Oh, that's really cool. You know what? Where we have a lot of documents is our entire research library, hundreds of thousands, millions of documents that we've written over years. Let's use it for that.
Adam Wheat: And so II took a deep breath. Sure, sure we could do that and and and I wanna use it for our our Morning Star Investment Conference in April? And and this was I think we were or weeks ahead of the conference. Wh, when this came through. So we were able to to bring together a small team, build a pipeline process. All these documents put them in a a vectorized embedded database for retrieval.
Adam Wheat: set up a an architecture and prompts and an Api back end. And found a partner that could do a talking
Adam Wheat: speech to text and text to speech overlay. So we had within a handful of weeks a
Adam Wheat: a I powered
Adam Wheat: human looking avatar talking to the CEO on stage at the Morningstar Investment Conference, and and it was amazing I so I remember seeing it. I thought, how on earth was that built so quickly.
Tim Baker: It was a very brave thing to do.
Adam Wheat: I it was brave, crazy something like that. But but we pulled it off and and didn't didn't run into the the Murphy's law of of high high stress, Demos and and everything went well. But that turned into everyone loves this.
Adam Wheat: We had something like
Adam Wheat: over , questions at our booth, at the conference of people walking up and talking to to this AI and and so then the mandate was, well. let's put this in our products within a month and and so so we we did. We. We continued to harden the system, did a lot of testing
expanded the documents that it had access to, improved a lot of our data pipelines.
Adam Wheat: and so what we ended up with was a very capable research assistant that someone could ask natural language questions to And and really dive into any number of topics, whether it's morning stars research on a specific fund, morning stars, research on a specific equity
general investment?
Adam Wheat: educational materials, which is, how can I plan for my children's college education? What's the right strategy for my investment savings accounts? All of those sorts of things became a conversational interaction that could be ha! Happening within a manner of seconds, of someone arriving on our our research portal site.
Adam Wheat: In the past, they would have gone through a traditional search experience might have found a list of Pdfs that they might have spent hours reading through to get that same answer. So II think in terms of
Adam Wheat: extracting unlock unlocking value out of a really big library of research. We, you know, prolific producers of research that no human being could read all of it. I think, having a an assistant waiting there that's already. Read it all, and can tell you the right answer in a matter of seconds and provide you with links to the full document if you need it. Was a really powerful thing for our users, and really exciting thing for our business?
Tim Baker: Does it interact with its users in voice? Or was it just for the one user?
Adam Wheat: So so we have what I would call the stage show version which is, is hooked up to to the the human avatar. And and we roll that around at a lot of conferences and and do stage shows our in product implementation is the same exact Api
Adam Wheat: with some very tiny modifications on on kind of the personality of it. In a chatbot traditional Chatbot version, we we did not put the human avatar in in in the web based. Product.
Chris LaCava: Yeah.
Chris LaCava: yeah, it's really good. So there's been. There's been some work in like in bilingual education, actually, where they're doing like, you know, proficiency assessments and things like that for student intake. And there is a lot of
Chris LaCava: voice, you know, to to figure out inflections and things like that. And there, I, Google, actually, I'm aware of Google. At least, I'm sure a lot of other vendors have this, but they have a whole set of models that are based on inflection and speech. For that analysis.
Chris LaCava: So yeah.
Adam Wheat: I will add one thing that may be interesting to the audience here. That was
Adam Wheat: something we discovered along the way that was somewhat unexpected is
Adam Wheat: Of all of those hundreds of thousands of documents that we used with Moe.
Adam Wheat: They're all in English,
Adam Wheat: but
Adam Wheat: foreign language support as just an emergent capability of the large language model was there, right? So we had. We didn't expect this, and and it started coming through. Once we released it is, people would ask their questions in Mandarin, Korean, Japanese.
Adam Wheat: Spanish, French, whatever, and and get high quality answers out of English based documents. So we have. We have a lot of in our business effort around translating our sources for foreign markets
Adam Wheat: and even then it's it's a relatively targeted subset. But the ability to to create that capability and and give the person the valuable content, or inside out of the document without pre translating, was unexpected and and really, really powerful. I think that's a great example. I think we keep seeing unexpected things that this technology can do
Tim Baker: and a lot of people struggle to explain how well it does it.
Tim Baker: Mike, Michael, do you wanna kind of run us through some of the use cases you've been playing with.
Michael Young: sure. And and you know, in all due respect, II will warn that to be careful about what I discussed, because we've got a lot of things under play here that are are not released to market yet. But one area I can really talk about. I think that. Well, first of all, I just wanted to. You know, pile on for a second on Adam's use cases because all of those are are valid for
Michael Young: Reuters as well. We're certainly you know, directionally working on all the same type of issues. And I think the idea of what I would call you know, a, a, a interactive search. In other words, search really becomes that that to a voice. AI, is something that this technology. This is this latest kind of plateau of AI technology is definitely enabling for all of us. We're all working toward that on various levels.
Michael Young: We're also we have a huge business. Again. This is something people aren't necessarily aware of unless you're in the media and broadcast industry. But a large portion of our business is selling. B twob, so you, that's why you'll see our stories in our videos sometimes tagged on various major broadcasters and distributed around the world.
Michael Young: But it means we're very video centric and picture centric. And that's actually where these technologies from a used case, different level and become really interesting. We've already got a lot of these baseline technologies deployed in the last months in A, in a video on demand product.
Michael Young: In our B Twob platform we call connected major broadcasters and and news agencies
Michael Young: leverage the content that we create distribute on that platform, and what that means in in a a real use case is the ability to look at a a video. Go, wow, that's interesting. There's
Michael Young: there's some, you know, video of of the coordination. The Queen's core. Sorry the King's coronation
Michael Young: and there's some people speaking, and so forth. Well, you know, at the click of a button in real time you can generate a shot list or transcript down the side specific to industry.
Michael Young: and then you can turn around and click another button and translate that live into different languages
Michael Young: quite accurately, whereas, you know, we could have tried to do that or years ago, and you would have had maybe % accuracy at best, particularly around the language translation. And now it's pushing anywhere from to % accurate to the point where people can use it almost completely out of the gate. So that's really game changing in a lot of ways.
Michael Young: And we think about how we're leaning into video and photos as part of our and and you know less weight on text as we go forward in in society in general. You know, I think that's a fantastic, interesting way that this will continue to expand. And the difficulty in the in in AI
Michael Young: generating curation and meaning out of video is exponentially more work, compute, more compute ultimately than it is to generate curation and and understanding out of text which, you know. Remind everybody that's with the large language models. Predominantly do. They can be used in other ways.
Michael Young: which can help such as ultimately generate code. You know, automatically. But it's it's really interesting to see where it will go in terms of that space, and how it will transform everyday life.
Tim Baker: I love those examples from from you and Adam where it was. It's really about taking an existing capability, a set of content, a corpus of information, be it real time.
Tim Baker: or just going back for well, voices, goes back over a hundred years and actually making it navigable and and creating value added products and services on on top of it.
Tim Baker: Graham. You're in a quite different, you know. Industry. Give us some examples of of how you've been using geneai.
Graham Ganssle: Yes. So for context, a lot of the work that we do here at Wayfair serves not business customers, but but consumers. And so a lot of the focus. Maybe the tone of some of these outputs could be different. If you deploy technologies like this directly into the hands of consumer. So not to say we don't have a B twob business because we do but maybe some of these applications are
Graham Ganssle: tone matched a little bit differently to the end user, which is interesting, because, as we all know, from playing with Chat Gp, it's easy to change the tone. You just ask it to use a different tone.
Graham Ganssle: In our business as as you could read from our CTO's recent Forbes article or Q. earnings. When our CEO talked about this
Graham Ganssle: we are working to deploy Genai Tech into a bunch of different areas of the business. And one of those areas is the area where I work.
Graham Ganssle: And in in the world of of both customer service and in sales. We have Dor, a directionally similar product to what Adam was talking about.
Graham Ganssle: Fundamentally, our position right now is to try to help out our our associates, our field staff, as they either help our customers find the right product that makes their home the right one.
Graham Ganssle: or once they have that product to remediate any issues, they've had to set up any further contacts or purchases with wayfair. So in to that end.
Graham Ganssle: And with respect to what Adam said, the products that we're working on are a set of augmentation tools that help the agents do their jobs more efficiently in terms of giving them access to underlying information.
Graham Ganssle: In our case it's much less about the retrieval of let's say, like, expert user based financial documentation
Graham Ganssle: and much more about generalized policies that wayfair has for our interactions with our customers. It's also about retrieving information for our customers. Again, this is through our associates that gives the customers and gives the associates themselves a
Graham Ganssle: richer understanding of what our product catalog
Graham Ganssle: is, is like in reality, when it gets to gets into the hands of our customers, as as
Graham Ganssle: some may know, we have a rich set of customer reviews on our products. Millions and millions of skews live on the site every day, and the the best way to sort of represent what those products are like to our customers is to have our customers talk about them.
Graham Ganssle: And if you're a customer service associate who is responsible for communicating some of that stuff to folks who rather communicate on the phone with you, and then sort of browse through the website themselves. It's impossible to do that as as a human being to know about millions queues? So, being able to summarize some of that information and give the access to that information into our associates hands in real time is
Graham Ganssle: super powerful. And and it helps out with
Graham Ganssle: with getting that information out in the right channel, whether it be phone chat whatever that our customers
Graham Ganssle: prefer rather than forcing them into a channel like a website, if that's not their purpose.
Tim Baker: So is your solution. Listening in to conversations on the phone as well. Is it? Is it learning from those? Is it
Tim Baker: prompting your customer support people
Tim Baker: based on what it's hearing?
Graham Ganssle: It's less about prompting the the associates, and it's more about enabling them so it's they can choose to use the the features as they wish. Right? So it's not like a whisper into their ears as they're on the phone. It's more like a platform that allows them to ask questions about a product or a policy. So the the thing that we don't want to do is overburden
Graham Ganssle: our associates with erroneous or just. You know, the luminous pile of information. And so it's.
Graham Ganssle: I don't want to use the word opt in, because it's there for them whenever they wish to use it, but it's it's at their disposal, and they can ignore it if they choose to.
Tim Baker: Brilliant. So you've you've all. You've all kind of spelt out some really big and interesting and value-added use cases. But I can't believe that you didn't have some problems along the way.
Tim Baker: because this space is moving so fast. There are guide rails. There are technology challenges can can. Maybe we. We stay with you, Graham, at first, and just talk about some of the
Tim Baker: you know, as a veteran implementer. What were the specific challenges that that you had to address with with Gen. Ai in particular?
Graham Ganssle: Sure, II think that our most surprising we we had a lot of learnings. But our most surprising learning was that technology is easiest piece. The. It's. It is a straightforward path to build a tool. Using gen, AI based tech
Graham Ganssle: but it's a much more difficult path to educate associates, or or, you know, in the future. And customers. On
Graham Ganssle: how and when to use the tech.
Graham Ganssle: So, for instance, if you think about this from a consumer perspective you might, if you had a a live chat bot out on a website. And you're you are on the Wayfare website. But this is hypothetical because we didn't haven't deployed in this way yet.
Graham Ganssle: You could be looking at a product page, and you could have that chat open, and it may not be obvious to scroll whether you should scroll down to look for product dimensions on the web page, or ask the virtual assistant for product dimensions. And the II mentioned that as a an analogy, because the same thing is true in
Graham Ganssle: as as a as an employee of wafer. So in our augmentative technology, it's a little bit difficult for some of the associates to sort of recognize that it's the right moment to ask the question. And so I mentioned at the beginning of the answer that technology is the easiest part because
Graham Ganssle: changing human behaviors. That change management work is, I think, and we have found to be
Graham Ganssle: an even more important and fascinating challenge.
Tim Baker: Interesting!
Tim Baker: Michael, what are some of the challenges and blockers you had to overcome?
Michael Young: II think the for those of you again that they've ever looked at the history of the long history of Reuters. You'll know that we have something called the Trust Principles, and you might want to Google that because it's an interesting story about them, they they trace all the way back to World War , but you know we're we. We follow those trust principles which basically ensure that our corpus is very fact-driven and verified. So obviously, we have a standard and a bias
Michael Young: control element to how we do things. So how do you translate that into these new tools? You know. How do you ensure that hallucination is under level control? A in a in any kind of base Llm that you're leveraging. You know there are ways to do that, but you almost have to build monitoring systems on the side to ensure you're staying within those guardrails that you mentioned?
Michael Young: So that turns out to a be complicated doable, but complicated so that slows you down to market to be honest and be expensive.
Michael Young: so that, you know, I think the bigger challenge really for us has been.
Michael Young: you know. Sure no one's going to pay anything more on a margin for a product that's being offered to them. They would love to have all these extra capabilities and feature sets around how they discover content, use it, leverage it insights curation so forth. But are they willing to actually pay more
Michael Young: for than they do today? Probably not out of the gate. And this is still a relatively expensive technology. Right? So I think honestly, the business model, the commercial model is one of the biggest challenges around this and over the next few years. There's a few reasons.
Michael Young: to think that will become a lot more cost effective. To apply the same technologies just even in the way that chips are produced and offered will make a make a huge difference. But it's gonna take a little bit of time. So you know where we have high profit. High margin products is where or high volumes of operations
Michael Young: where cost controls really start to make a huge difference. In terms of automations. That that's where you'll see the technologies. First, because the business case justifies it. But there's a lot of cool things I would do with Reuters news corpus going back years. I'm not going to do out of the gate because it's too expensive. Eventually we'll get there. So I think that's one of the the biggest areas of challenges.
Tim Baker: You know, a human in the loop in the process as well, is that part of the the way you solve for this near term?
Michael Young: Well, we've always. We've always done that. And, as I mentioned earlier, we've used evolving levels or capabilities in the AI space and in the automations, machine learning and automation space.
Michael Young: And so if something falls below a different. You know, we have these controls on these tests that happen on everything that we do. If something falls below a marked confidence level around a piece of content that we're handling, or we have some automation around it. Does, you know, go across a workflow tool in front of a human being to confirm it.
Michael Young: And certainly it would be the same thing in this case. Anything that falls below what we think the the confidence level is, you know, something's must be highly accurate, because it's a piece of market data, for example, that could be markets if we miss you know, if if we misjudge it and then we publish that, and then that could move markets erroneously the wrong way, and that has to be an extremely high confidence. But if it's something that say you know,
Michael Young: an insight piece, or something like that, and then the confidence can be lower, although for our standards still must be relatively high. So absolutely, that's the other. Complexity is, how do you use human beings productively in this loop as well? And when do you put that intervention in place?
Tim Baker: Yeah, I remember when I was at Tr, there was a product called Sigdevs.
Tim Baker: which was a team of humans who were
Tim Baker: reading there would be a significant development, and they would have to put it into bullet points. And that seems like a a really great use case, for you know, for for you guys
Tim Baker: thought you're still on mute, Michael
Michael Young: apologies. I lost my controls there for a second.
Michael Young: you know we have tools like, I said, pretty extreme workflow tools that you know. And we have large operations, content operations, teams. That we're, you know we're very proud of having built that capability. But you know, if you take one of them as an example, one of our productivity verification tools exactly in the vein of you're talking about is called Fastwire, and someone operating with fast wire. You know.
Michael Young: Terminal, if you will, for lack of a better term or station and and most of those teams are in India these days.
Michael Young: They'll process or verify. You know, a a piece that will be published. every one to s. So that's the speed or the capability or the level of automation in combination with human that you have to arrive at to make this this useful.
Tim Baker: amazing Adam. I'm sure you faced a lot of barriers with yours. You're very short time frames. You must have had to kind of get in front of those and put guide rails in in place.
Tim Baker: Give us some of the things that you have to deal with getting that product out so quickly.
Adam Wheat: Sure. And maybe I'll I'll I'll start with one that that I didn't have to deal with. But II it was easy to recognize how fortunate we were, but might be a challenge for others that are entering into this space. So maybe useful for the audiences is the
Adam Wheat: most of these applications need to have be fed with live data, right? Like. And right.
Adam Wheat: everyone knows that there are these training windows of, you know. So Chat Gp doesn't know things. After some point in , a and a lot of a lot of our business cases need to be Updated with fresh data that can be retrieved. And so having the right documents and data sources available
Adam Wheat: to start building on is a really fundamental problem. I think Michael and I are like, had the best case scenario. As we think about it, right? We, we sell our data and research as products. So there's there's lots of of production, pipelines and Apis and and easy ways to get that into an AI system.
Adam Wheat: But a lot of organizations haven't invested in their information architecture and infrastructure in that way. So if you're thinking about how AI is going to play a role in in your business story, start with, where's the the information gonna come from? If that, if that's a use case, you need start solving that problem. That's not an AI problem. It's it's it's a general
Adam Wheat: data and technology problem that that that many of us share. But that that's a really foundational thing. we were lucky to start with, I can just point out an Api that we already have. Take all the documents I need, and and we're off to the races.
Adam Wheat: so that that's something to be aware of. I'd also say and and this is similar to something Michael said around kind of the
the trust principles are right that the the
Adam Wheat: independent integrity and and research that we that morningstar really is proud of and has developed a reputation around for a number of years
Adam Wheat: is a concern. Right? So what would the worst case scenario, putting an AI Chatbot out be? Is it? You know, says something horrible, and there's screen captures on Twitter and and whatever else like they do with Chat Gp, when they make it say terrible stuff. Right, that that was a big concern. So you know.
Adam Wheat: putting the right guardrails in place and making sure that we had
Adam Wheat: the right type to of of prompt engineering and guard rails around it, so that you know things that would be damaging to our reputation.
Adam Wheat: We would have a high confidence that those would be eliminated from the system. And also you know, maybe
Adam Wheat: a more nuanced version of that is incorrect or inappropriate responses. Even in the context of rate.
Adam Wheat: our data and our tools get used by a lot of participants in regulated financial markets. Right? Give giving financial advice
Adam Wheat: outside of the right type of contractual relationship
Adam Wheat: is a big problem so making sure that our tools didn't do things that that they shouldn't be doing and and avoided that type of behavior. So that was a big, a big part of it. I'd also say that accuracy is is an interesting concept. When you get into AI, particularly with document retrieval.
Adam Wheat: Right? We don't. We don't really think about it. Our brains have been wired to handle this like when you Google for something. The fact that there, you know, you're Googling for something recent. But you find an article from years ago that would be considered wrong today. Right? That's old and dated and wrong.
Adam Wheat: we just filter through it, and you know. give me the new stuff and I'll I'll focus on that.
Adam Wheat: But when you, when you start feeding documents into an AI that might have longer lives.
Adam Wheat: That this question of what's a hallucination
Adam Wheat: and wrong? Because the AI made it up versus. This is an absolutely correct factual statement that somebody wrote in a research paper years ago. It just doesn't happen to reflect the current
Adam Wheat: opinion market environment of of a morning star. Right? So, that that was, that's a tricky problem to think about in terms of of of both. How do you identify and guard against those and and as more data comes in on a consistent basis, you know what? What's the data management strategy to make sure.
Adam Wheat: The right and most fresh information is is is relied upon first. And I think there's a lot of technical things. But II to Graham's point of of the technology is often the easy part.
Adam Wheat: an engineering team is not going to understand the nuance of why that's wrong. Right? So
Adam Wheat: yeah, I did what it was supposed to do, and it read it out of this article. Perfect mission accomplished what we found very early on was helpful is you need subject matter experts for a lot of this, you know. How do you define what's right? What is correct? What what would the human expect to see. And and a lot of times that expertise is outside of the technology team in your organization. So it really does take good pathways and partnership in the business
Adam Wheat: to get you know, to have that feedback loop of. Here's what we built, and then go use it and and have that human in the loop feedback. As you're building to understand. You know
Adam Wheat: where the gaps are, where the the challenges are, and and make sure that that gets back into the engineering process to improve and improve the quality and and reliability results.
Tim Baker: When did you get the compliance team involved? Because they, you know, they must have been sensitive to something that's pushing out recommendations to an audience.
Adam Wheat: Yeah, we we had, it happened very quickly. Given the schedule that we were trying to deliver in product
Adam Wheat: capabilities so legal and compliance including we relied on on some very good external
Adam Wheat: legal advice from folks that have have spent a lot of time learning about AI and and and
Adam Wheat: came to the table with a lot of do's and don'ts and knowledge. That was that was really helpful. But yeah, the the making sure, particularly in regulated financial space, that we understand
Adam Wheat: what the requirements are.
Adam Wheat: where things are going with respect to not only legal precedent. There's there's certainly a lot of of things going on around.
Adam Wheat: Where did the AI training data come from, and who owned it, and and who still what? From Reddit? All right all of those things going on, so understanding the backdrop and and things to that. We need to protect ourselves from both from
Adam Wheat: What are we building? But also we have a lot of content out there that we and others might use as as fuel in an AI engine. And and what do we need to be thinking about there? So there, there are a lot of considerations, and we had
Adam Wheat: a lot of great resources. II if it was up to me, we would have failed miserably. I'm I'm glad to have had such great business partners and and legal and compliance to to do the heavy lifting. But it's it's a
Adam Wheat: very consequential work stream. If you're depending on how you're launching and who the the audience is internal is clearly easier than external.
Tim Baker: Yup. We've had a question about costs. So you've all been building these systems. Would you know, as you went into them?
Tim Baker: Did you kind of need to do an assessment of the costs? And and what are some of the benchmarks. The question is, do you have any rough estimates for planning purposes?
Adam Wheat: I could kind of go through the
Adam Wheat: so where does the cost come from?
Adam Wheat: Right? So if you're using a hosted foundation model, let's say you want to build on Gpt. which you could could have an Api relationship directly with Openai. You could access it through azure Openai Api endpoints. So there's per token cost that will arise from there. So that that's really a volume based. Question, right? So how much?
Adam Wheat: How much context are you going to feed in? Per query? How big are the answers, you know, for a lot of chat based with relatively modest sized content.
Adam Wheat: It's it's unbelievably cheap to do to that type of workflow and if you need higher throughput and guaranteed compute for it. It gets it does get more expensive. You can. Microsoft now offers dedicated compute instances. If you need to guarantee high throughput.
Adam Wheat: Instead of having this multi-tenant, maybe you get rate limited at some point setup so so
Adam Wheat: fairly cheap on that side. The embedding of documents and and storing and retrieving things from databases, even at pretty large volumes, I'd say in the scale of of things is is relatively inexpensive.
Adam Wheat: Where you do get into much more meaningful cost is if you did want to do model training, right? So even fine tuning over with large models and large training sets can can
Adam Wheat: can be pretty expensive in training a model from scratch. Then you're right. You know, even small models are in the thousands of dollars to to train from scratch
Adam Wheat: I I'd say that the compute costs are real there. But to build a corpus and and manage it a a full Ml. Ops pipeline for training your own models. I think there's a big compute cost, and then there's a big people cost to to work with that data and and make sure that that, you have the right hands, both from a data science perspective as well as a content management perspective involved to do it right.
Tim Baker: Perfect thanks, Adam. So we've only got a few a few minutes left. But I did want to.
Tim Baker: have you guys just talk a little bit about what strategies for success might be out there for members of the audience thinking about, okay, we're gonna jump in, or maybe we're going to wait. And then who wants to go first with this one? But you know what or kind of bullets would you give in terms of as you're embarking? What strategy should you think about?
Tim Baker: And Tim not sorry to interrupt? We do have one more question. And so if we can leave some time for that as well, thank you.
Tim Baker: Thanks, Lisa, so so quickly. Maybe I'll give an answer. Yeah.
Graham Ganssle: do something don't do nothing
Graham Ganssle: that's important, because testing these things especially live is the only way that you'll understand whether or not they're affected for your business.
Graham Ganssle: The other is beef. II actually would.
Graham Ganssle: I have a I have a different opinion opinion. To Adam's answer about cost, I would say, be flexible, or at least plan for more cost than you think you might need in terms of just talking to
Graham Ganssle: external lms. Because
Graham Ganssle: through testing, especially Ab, testing, you can easily run up your bill there. And finally, I would say II would I would actually go back to something, Adam said earlier, which is in terms of risk, be prepared to
Graham Ganssle: spend a great deal of time and and effort, making sure that the outputs are for for whatever your audiences are, you know, hardened to certain types of hallucinations or incorrect information.
Michael Young: It's a good one, Michael. Yeah. So it's a
Michael Young: sorry sherry. Go ahead. Go ahead, Micah.
Michael Young: I was just gonna say and following the theme, II won't throw out, but I'll focus on one, and that is you know. Stay agile and don't do strict and inflexible integrations, because the players that are out there now, and the first offerings of this technology will change massively over the next few years.
Michael Young: So whatever you build, you wanna make sure that you can. If if it's desirable and beneficial to you, you're able to switch to other providers and other solutions. So for example, when you think about vector
Michael Young: mapping, you wanna look for embedding standards that there are a couple of open source standards that are evolving. And you know there's or big players that are saying their own version for those standards. But you wanna make sure that you're abstracting whatever you build for your organization away from a technical point of view, away from being dependent or highly invested in one or players.
Chris LaCava: That's a great one. Thanks, Michael Chris. That was my point exactly. Is that the landscapes changing so quickly. There's so many new players every day coming into into view, and they're aggressively attacking a lot of the objections out there. So
Chris LaCava: you know a lot of the contortions you have to do to address those objections will disappear, and you don't want to build anything that's too rigid.
Tim Baker: Yeah, I know, Adam. We've talked about this before taking a kind of a platform approach. I assume that implies that you've got you can. You can pull in a different model or different set of capabilities onto that platform.
Adam Wheat: Yeah, that's and that's that's what
Adam Wheat: that's the approach morning starts taking. It's very much aligned with with what Michael just said about think about it in terms of abstractions where models are hot, swappable retrieval pathways are hot, swappable. Ii also like to talk to my teams a lot about, you know, we call ourselves computer scientists and data scientists like, let's not forget what science is. It's it's defining experiments and trying things with the knowledge that they can fail and and so run, you know, to Graham's Point try stuff, run experiments.
Adam Wheat: and if they don't work now
Adam Wheat: come back to them in a year, and they might. I think, the way things are changing, taking that that experimental mindset and making sure that you have a good process that allows for that over time is gonna really add value.
Tim Baker: So we have another question, is there an optimal amount of data to train a model with? IE. Is there a point where too much data can cause issues? Or is more data always better than less?
Tim Baker: Anyone have a view on that.
Adam Wheat: I think that's very domain specific. Ii think what we're seeing is for some use cases, very small models or very small data sets for fine tuning can be very successful. Just one example is,
Adam Wheat: yeah. And this is at not really a new generative AI problem. It's like an old machine. Learning problem is, is entity recognition
Adam Wheat: is hard, right? And and in our context, it's, you know, someone says.
Adam Wheat: Tell me about Apple's latest earnings, and you know you need to know that apple is a company in the and not produce at at the store, and you need to map that to an identifier and go retrieve a bunch of data
Adam Wheat: to tell them the right answer right? And and so
Adam Wheat: the language model extraction of things like that. can be really helped with fine tuning and training.
Adam Wheat: But can we? What we found, at least in in some of the Pocs, we've run so far is you don't need a huge volume to make make a difference on them, I think, for new models.
Adam Wheat: They can be infinitely big but yeah, way, the use case and and the data set you have access to to think about where the optimal point is of
Adam Wheat: return for the cost you have to put into that training exercise.
Tim Baker: Well, guys, I think we could probably carry on for another couple of hours on this topic. But I'd just like to thank all of you for for your insights. Filling the questions.
Tim Baker: If anyone on the call has any other questions. Then, please let me know, and I'll get those out to the to the panelists. So again a huge thank you folks for coming on it felt like we covered a lot of
Tim Baker: a lot of interesting topics. And hope to speak to you guys soon.
Chris LaCava: Thanks everyone. Thank you.
Adam Wheat: Thanks.
Michael Young: Thank you.
