This week, we sit down with Sam Flynn, COO and co-founder of Josef, to separate substance from hype in the rapidly evolving world of legal tech. Sam shares his passionate stance that “RAG is not dead,” defending Retrieval-Augmented Generation (RAG) as a foundational and still deeply relevant method for deploying AI in the legal industry—despite the flashy allure of agentic AI. His nuanced take reminds listeners that success in this space depends not only on the sophistication of the technology, but on doing the “boring” foundational work: ensuring data integrity, context-aware chunking, and responsible workflows.
Throughout the discussion, Sam champions the idea that great legal technology should not just enhance expert workflows but make legal information accessible to non-experts. With examples from Josef’s clients like L’Oréal, Bumble, and Bupa, Sam illustrates how Josef’s tools allow legal departments to offload routine work through reliable self-service systems—freeing up time for more strategic thinking while improving speed, compliance, and consistency across organizations. He makes the case that empowering end users with trustworthy tools isn’t just good tech—it’s a new model for scaling legal and compliance services.
A key highlight is Josef’s Roxanne project, developed in collaboration with Housing Court Answers and NYU. Roxanne is an AI-powered tool designed to help tenants in New York navigate the complexities of housing law. Sam outlines the safeguards that ensure Roxanne’s answers are accurate and compliant, such as closed-domain data sources, human-in-the-loop validation, and smart escalation workflows. The conversation touches on the broader access to justice (A2J) implications of this technology—arguing that when designed carefully, AI can amplify the reach and impact of legal aid organizations by orders of magnitude.
The episode doesn’t shy away from the tensions legal professionals feel when automation enters their domain. Sam offers a powerful reframing: instead of seeing these tools as a threat, lawyers should view them as opportunities to offload low-value tasks and expand their influence. The goal, he says, is not to cut jobs—but to redefine the kind of work legal professionals do, making space for more proactive, strategic, and meaningful engagements within organizations and communities.
As the conversation wraps, Sam shares his optimism about the future—tempered by a clear-eyed understanding of the human factors that will determine success. While the technology is ready, the question is whether legal professionals will step up and take the lead. His call to action is clear: focus less on the hype, and more on the systems, safety, and trust that make tech transformative. Whether you’re a legal technologist, innovator, or cautious observer, this episode offers a grounded and inspiring look at what it takes to build legal tech that actually works.
Listen on mobile platforms: Apple Podcasts | Spotify | YouTube
[Special Thanks to Legal Technology Hub for their sponsoring this episode.]
Blue Sky: @geeklawblog.com @marlgeb
Email: geekinreviewpodcast@gmail.com
Music: Jerry David DeCicca
Transcript
Greg Lambert (00:00)
Hey everyone, I’m Greg Lambert with the Geek in Review and I’m here today with Nikki Shaver and Stephanie Wilkins and we’re going to talk about the generative AI map. So Nikki, what’s going on with that?
Nikki Shaver (00:10)
So Legal Tech Hub launched a generative AI map for Legal Tech on February 19th. Steph, do you want to talk a little bit about it?
Stephanie Wilkins (00:19)
Sure, it was a big undertaking. It was exciting for us. We compiled all of our research through February 19th of all the solutions in our directory that incorporated generative AI and we actually published it on March 3rd. And at the time it showed 400 different legal tech products using generative AI across 17 broad categories. We published it and the response was overwhelming to say the least. There was so much interest after it was published from companies who wanted their products included.
that we decided to create an updated version for the first day of legal week just two weeks after our initial publication. And in that two weeks, had then over 500 solutions across 18 broad categories.
with the top areas for Gen.AI development in legal tech being broad legal AI assistant tools, contracts, and litigation solutions. And Nikki, maybe if you want to talk we take away from this.
Nikki Shaver (01:13)
Yeah, I mean, it’s interesting. I think there are still people who wonder whether maybe generative AI is kind of fringe and like blockchain, it will disappear. I think what this really shows when you have over 500 solutions with generative AI already, this is not a fringe area. There’s not an if as to whether law firms and corporate legal departments will adopt generative AI. It’s really when being built into so many products that if your practice uses any kind of technology, inevitably you will also
be using generative AI. So that means all of you need AI strategies and AI policies in place if that hasn’t already happened. So we’ll be launching a map update every quarter. I’ve gone out on a limb and said that I believe a year from now we’ll have a thousand solutions with generative AI. And we have various other maps and graphics plans. So stay tuned to legaltechnologyhub.com to ensure you stay up to date with the market and make sure you also
sign up to our newsletters.
Greg Lambert (02:10)
Great. All right. Thanks, Nikki. Thanks, Stephanie.
Marlene Gebauer (02:19)
Welcome to The Geek and Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gabauer.
Greg Lambert (02:26)
and I’m Greg Lambert.
Marlene Gebauer (02:28)
So today we welcome Sam Flynn, COO and co-founder of Josef. Sam, welcome to The Geek in Review.
Sam (02:33)
Thanks Marlene and Greg, it’s very exciting to be here.
Greg Lambert (02:37)
We’re excited to have you here. Sam, before we jump in and talk about Josef, I know that you released some new functionality recently with Josef, and there was something that you said in the video announcement that I wanna kinda have a little back and forth on if you don’t mind. And that was, you are…
Marlene Gebauer (02:38)
Yeah.
Sam (02:54)
Okay.
Greg Lambert (03:00)
leveraging the retrieval augmented generation in your AI product, which essentially allows you to chunk information and use that information to ground the results that you have to that information. So what we refer to as RAG. And a lot of people think that RAG is dead because agentic AI is something that many folks feel is
Marlene Gebauer (03:28)
hot new thing.
Greg Lambert (03:29)
Yeah, it’s a hot new thing. But you specifically said RAG is not dead. So I want to give you a chance to defend your thesis here and tell us why RAG is not dead.
Sam (03:29)
Enjoy your
Yeah
Marlene Gebauer (03:43)
Prove
the naysayers wrong.
Sam (03:44)
Yeah, that’s it. This is my hot take everyone. RAG is I can say without a shadow of a doubt, not dead. Agenetic AI is very exciting technology. and I’m thrilled to see where we take it. That said, I’ll make a little note here. and we use workflows at Josef. So yes, you use.
Marlene Gebauer (03:48)
Hahaha.
Sam (04:14)
form of RAG to chunk the documents but you can build workflows on top of it.
I have heard from, this is a couple of weeks ago, from someone who built one of the very first agentic AI workflows in legal for one of the big five tech companies in the US, that he can count on one hand, the number of agentic AI workflows that are out there in the wild in legal and compliance that are delivering value. So it’s very exciting technology, but I want to…
For all the geeks out there in LegalTech I do want everyone to take a breath and to look at what we have, which is profound technology, and to start to deliver value rather than jumping from hype cycle to hype cycle. That’s my little rant. I’m bringing it back to answering your-
Greg Lambert (05:09)
But it’s so
much fun to jump on the brand new shiny thing.
Sam (05:14)
It is!
I mean, I’m a geek. I love it too, right? So does everyone at Josef.
Marlene Gebauer (05:18)
Yep. We want to believe, you know, we just want to
believe that, this is the next, this is the, this is the thing that’s going to save us every time.
Greg Lambert (05:24)
This is it.
Sam (05:25)
This is our Hail Mary. But to
get boring for a moment, what actually sits under my hot take is it doesn’t matter how sophisticated the technology that sits on top of it or how sophisticated the workflow that we build. If you don’t get the knowledge source right, all of that is going to be for naught.
And we’ve all used a Microsoft Copilot or a generic RAG tool that is splitting our legal documents in strange places, halfway through a clause or halfway through a paragraph. It doesn’t understand the hierarchy of the document. That’s the thing that we have spent two years getting right across thousands of legal and compliance documents with research partners at organizations like NYU and Cornell and with our customers ranging from L’Oreal through
to Orrick the law firm.
Greg Lambert (06:22)
Yeah,
I’m curious as the AI models advance and it seems like every week or two, there’s some big launch, OpenAI just came out with 4.5. I think Claude came out with 3.7. Each one doing, and I think at this point it’s, you know.
Sam (06:38)
Hello.
Greg Lambert (06:48)
they’re significant, but not huge leaps at this point. But when these changes come out, does it affect the workflow model on your side of things? And how difficult is it to adjust to these changes that you’re doing? Especially if you’re basing, you’ve worked two years on getting your RAG structure just right, and then…
know, foundation models may change and, you know, kind of throw that off.
Sam (07:22)
Yeah, it’s a very good question. To your first point, I agree. I think we’ve seen the end of the great leaps in the foundational layers. Now, all of the advances that we’re seeing from deep-seek through to reasoning models are differences in the way that we’re interacting with the underlying technology. I think that’s really critical for people to understand at this point.
The relevance of that to your actual question is the changes in the underlying models are less and less impacting what we’re building on top because of that fact. But every tweak in the underlying model can fundamentally change the way that the output is created or generated. And if you are going to be deploying this technology,
in the enterprise or in a high-risk environment like ours, then you need to have very solid safety and maintenance checks in place to make sure that the answers and the outputs are still coming out in the way that you want it to. So we have lots of those built in at Josef. We’ve seen lots of sort gold standards out there in the environment. It takes a lot of work. So find a partner you trust if you want to do that work.
Marlene Gebauer (08:48)
I was just really interested in the, the information you got from your, your source that there’s really what was it five that are delivering value and, know, cause you know, I think you’re hearing more about that. What? No, no, no, no, no. It’s like, we wouldn’t ask you to do that, but you know, you know, in terms of what we’re hearing outside of that, mean, is it more like innovation theater?
Sam (08:57)
huh.
Greg Lambert (09:00)
Spill the tea, Sam.
Sam (09:04)
I’ll never give up my sources.
Greg Lambert (09:06)
Ha ha ha ha.
I would, I would.
Marlene Gebauer (09:17)
that truly there are only a few things that are working right now, because it seems like when you look at the news and the industry reports that it sounds like a lot of organizations are trying this out.
Sam (09:33)
Yeah, I think, I think partly it’s absolutely innovation theater. Our peers, like it or not, get a lot of buzz and interest and excitement, right? We trade in engagement.
So many professionals these days trade in engagement and what’s one of the best ways of getting engagement right now, particularly in the era of AI, new stuff, right? I don’t want to be too cynical, but I’m sure that’s a big part of it. That said, I don’t, I really don’t want to be read as undermining the technology either. There are extraordinary opportunities here, but I do want everyone to pause and get to work, kind of do the boring work.
is my call to action and show people that with the foundational, with the transformational rather, technology that we have today, we can deliver real value to the people who don’t care about the tech or the shiny new object. I think that’s what builds us the trust to then move forward into this sci-fi future that awaits.
Marlene Gebauer (10:43)
There’s a phrase I like that of reminded me, like no mud, no lotus. That basically you have to do the hard work in order to get the big reward. So do the basic stuff.
Sam (10:49)
Yes. Yes.
Mm-hmm. And I’m interested.
I think that’s perfect phrasing, Marlene. And I’m fascinated that the two of you picked up on this product release of ours because it was kind of a boring product release, right? There’s no fancy new buttons or like shiny new features. It was under the hood. So I’m fascinated at the response it’s got in the market.
Marlene Gebauer (11:03)
Mm-hmm.
Greg Lambert (11:18)
Well, speaking of the response in the market, let’s jump over and talk more about Josef. So what’s the core problem that you’re solving in legal, and what’s kind of the story behind the creation of Josef and products like Josef Q?
Sam (11:40)
So the way I think about Josef and why I started the company in 2019 is to create self-service tools.
That might not mean something to people in your audience, but for me, that is at the core of what we need to do in the legal tech space. I’ll tell you an anecdote to make that real. general counsel at a customer of ours, who’s a Bay Area health tech company, said to me two weeks ago, I haven’t seen a sales contract in six months. And he used to see several every day if they got too complex. That to me,
says two things. One, this very clever man who has a lot of education and a lot of strategic value to give to his organization can now, now has the space and time to do that. And his sales team who are keeping the money flowing for the company now get to generate their complex agreements in a matter of seconds, if it’s simple or minutes, if it’s complex, and not have to wait for a day, two days a week for their contract to go through legal.
Those two pieces are symbiotic. And I think in the middle or at the center of that sits this idea of self-service tools. We work some other examples of tools are L’Oreal has a number of document automation tools that do exactly that kind of work. Bupa, a global insurance company with 80,000 employees has built Q &A tools, so AI powered Q &A tools that need to be really reliable.
that answer questions for the business without any lawyer interaction on topics as complex as private health insurance regulations through to HR and people policies. Bumble, the social media company or the dating app, has built a suite of privacy tools, some of which are internal to the privacy team and some of which are potentially going to go to consumers. So that’s kind of the breadth of the work that we do.
If you’ll let me wax lyrical for one moment longer, don’t give it, don’t give a founder a stage. They’ll never shut up. the genesis of all of this and a big part of what, what I wanted to talk to the two of you about today is this idea of accessibility or access to justice. And me and my co-founders, Tom [Dreyfus] and Kirill [Kliavin] actually started Josef building tools for legal aid organizations to help everyday people get access to legal services.
Marlene Gebauer (13:54)
Please.
Greg Lambert (13:54)
Go for it. Go for it.
Sam (14:23)
So many people see that, like justice tech are so distinct from the tech that we talk about in the kinds of companies that I just mentioned, but I think there is a straight line. It is all about getting that low level work off your desk as an expert and making legal services, the work we do more accessible. That’s my vision.
Marlene Gebauer (14:46)
Well, that is a perfect segue into the next question, Sam. So Josef has a collaboration with Housing Court Orders, HCA, and New York University on Roxanne. apparently, what Roxanne does, and I know you’ll tell us more about it, provides actual information. It assists by assisting tenants.
with documenting issues and preparing letters. tell us a little bit about that. Tell us about how this advice remains accurate, up to date, and compliant in this landscape of housing law. Because I know that is a very fiery area often.
Greg Lambert (15:33)
Yeah. What, what
I want to know Sam is when you, when you say Roxanne, do you actually sing the lyric from the police? Do that.
Marlene Gebauer (15:38)
Roxy!
Sam (15:40)
Yeah. Everybody
Marlene Gebauer (15:43)
Everybody now.
Sam (15:45)
we find there’s a generational divide. It’s a very good, a very good tool for, for putting you on, on the map. no, it’s a very, it’s a very sweet name. like it. the, question Marlene is, is everything in this space. How do you make sure that your LLM powered tool is actually reliable and trustworthy?
Marlene Gebauer (15:49)
Hahaha!
gotta be right. These are people who
won’t know the difference, right?
Sam (16:10)
Literally,
right? It’s got to be right. And that’s something that we have thought about a lot. And that’s why we were very excited by the like under the hood changes when people are sort of running off and building Agentic AI workflows that introduce even more variables that mean validating the output is going to be harder. There’s a bunch of different ways of thinking about it. There are the technical controls.
And so these are, I think the key ones for people to know, even if you’re not super familiar with the space, are having a trusted data source. So Josef is what is called a closed domain system. It’s not like a chat GPT, which can draw from any source anywhere. This is only taking from the existing knowledge resources that you have in your organization. So for the insurer example I gave you before, they’ve got their internal wikis.
on private health insurance regulations. It’s just that the marketing team or the sales team or the research team wasn’t interacting with that information, right? Same thing here with housing court answers and NYU. They have their tenancy law explainers and websites and whatever else. People just weren’t engaging with them for many reasons. That information is already being kept up to date and complete. And then you sit Josef Q on top and then people have the ability to ask.
questions and get answers and do whatever work they need to do. So I think that’s a critical component. I think another one is, is this, have either of you heard of Megan Ma from Codex? No way. Okay. Like she’s, she’s one of my favorite thinkers in this space, right? And she draws this beautiful analogy with, self-driving cars. We were five years ago, 10 years ago, we were kind of in this, in this environment where everyone’s trying to make.
Marlene Gebauer (17:48)
yeah, we’ve, she has been on the podcast. Yes. Yes.
Sam (18:07)
the car’s perfect. They were like, there’s gonna be self-driving cars everywhere, right? Then they’ve got to be perfect. They can’t have any accidents. And it’s just like, this doesn’t make sense, right? Humans have accidents, but we couldn’t quite figure out how to bridge that gap. For her, now, not for everyone, but for a lot of people, the great technical challenge shifted from how do we make the cars perfect to how do we build the best alarm system to tell the driver when to take over? Similar kind of features are built into Josef, right?
pre-launch and post-launch. So pre-launch, you have what’s called human in the loop training, where the experts at Housing Court Answers are getting real world questions coming through from tenants in the city. They feed them into the tool, they get the answer, and they validate it. Yes, no, change it, right? So in that way, you’re improving the performance of the tool. And then post-launch, you have workflows, triggers, access to data that helps you track performance of the tool as well. So it’s not about getting humans out of the workflow. It’s about
building them into the workflow to ensure that the tool is going to operate effectively and safely.
Greg Lambert (19:13)
Yeah, I know one of the folks that helped develop Roxanne Sateesh Nor iLegal aid lawyer, adjunct at NYU Law, has a very impressive resume. But he talked about the potential for Josef Q in these types of having AI tools bridge the access to justice gap that it could.
increase the capacity by 10 times, 100 times, 1,000 times. So can you kind of elaborate on what Sateesh is talking about with this jump in capacity for A2J?
Sam (19:47)
Thank
Yeah, it’s a lot in there. I mean, look, at base what he’s talking about, right, is what we all know to be true. If you automate a task subject to CPU and distribution, that task can then be done an infinite number of times. So if you’ve got one person on the phone or at housing court from housing court answers, answering one individual’s question.
If you then have Josef Q take the knowledge base and that expert teach Josef Q how to answer that question, it can be answering that question, you know, a thousand times a day without any extra effort from the individuals involved. So just that scale piece. It’s about scaling the work that you do, scaling your services. That’s, that’s the idea that he’s getting at.
What sits underneath that though, I think is very interesting and something we need to grapple with quite seriously as a legal industry and as a compliance industry as well. And that is what is the value of the work that we’re doing and what is the value that we can give to society or the organizations that we work with. So a lot of lawyers and compliance officers, when they hear those kinds of statements, right? Well, you don’t have to answer the question anymore because it’s going to do it for you.
can lead to a lot of fear, right? Because you’re saying, well, that’s currently 30 % of my day. And that’s from actual research. 30 % of an expert’s day has been answering questions. That’s a threat. But I think we have to pause and say, well, if the tool can answer the question as well, or sometimes better than you can, and that’s not a very valuable question for the organization, what other value can you bring here? What else should we be looking at? And in my experience,
launching these tools actually increases the TAM for lawyers and compliance officers. And when I say TAM, mean, it’s a total addressable market for anyone not in the tech world. It’s the question you always get asked when you go in, when you speak to investors, they say, well, what’s your total addressable market? How big could this be? Right? And when you think about in, let’s go to an organization for a second, right? You’ve spent all this time building a beautiful code of conduct.
or an insider trading policy and You think about how many people are actually engaging with that policy and Often of the people who should be engaging with it. It’s sitting at like five ten percent, right? What are all those other people doing when they come across these issues? Inevitably are they just taking the bottle of wine? Are they just making the trade? Are they just taking the customer out for dinner and not checking a lot of the time? Yes by
encouraging people to engage with these materials and increasing their engagement, you’re increasing the TAM. And suddenly as a lawyer and a compliance professional, yes, you’ve lost a few basic questions from your day, but at the end of that day, you can now see exactly what’s happening across the organization. We had three questions about bullying from the engineering team today. We had two about a data breach from X team. Let’s go and do that work. Let’s be proactive. Let’s report this to the executive so that they can see what’s happening.
And that changing the way that we think about the value we bring to the organization, I think is critical. And the same thing in the access to justice space, right? This now gives housing court answers and other organizations access to data to help them improve their resources, but also to go and do strategic advocacy initiatives, right? These are the, we’re seeing a spate of issues around X, Y mold, for example, or heating, right?
they can then go and advocate for better circumstances for tenants.
Marlene Gebauer (23:49)
So you mentioned that Josef is a closed system, which there’s an inherent degree of safety just by being a closed as opposed to an open system. Are there additional sort of safeguards and validation processes that Josef employs to prevent bias, ensure data privacy, and maintain user trust in Roxanne?
Sam (24:15)
Yes, there are lots. Yeah. Yeah. Yeah. There are lots that this is the thing we care about the most deeply, right? you can build a chat to PDF document in seconds for free on the internet. What we’re about is about building a powerful Q&A tool on Josef or a doc automation tool or whatever you’re building, that you can trust. That’s the key. And what goes into that is lots of the boring work that we spoke about earlier and lots of the stuff you don’t see.
Marlene Gebauer (24:16)
or Josef in general, guess.
Greg Lambert (24:18)
Thank
Sam (24:46)
The, I think if we go to like those, those concerns, right? And these are common concerns, bias, privacy, actually see them dropping out of the news a little bit lately, which is really interesting. on the, the bias side, I don’t think it’s a very, very important question. the way that we control for that is kind of in the same way as you would control for that in a human, right? I often draw that analogy. I think it’s a bad analogy sometimes, but it’s a good analogy when we’re thinking about
how we govern AI. How do we govern bias in humans or how do we try and prevent it? We educate them. So we feed the right knowledge into the tool, the corpus. We train them over time. we don’t say that. that might offend someone. You set rules above that, right? So knowledge, education, rules. You set rules for what the tool can and can’t do.
And part of that is built in at the LLM level, but you can also do that in the tuning lab in Josef. And then finally, you have a layer of accountability. So that is where if it’s saying things that you, should not be saying, you can moderate those out and make sure it’s not doing it. That for me applies to so many of these questions about AI. Treating it like a human or training it in the same way as you would a human, I think is a really good way of analogizing.
how to deal with the technology. On the privacy front, there’s only like privacy and data control, right? There’s only one additional consideration for AI powered tools that you would already be considering for your CLM or your intake tool. And that is, are you storing the data in the model? The reason why that makes people nervous is because there’s still stuff like…
There’s a few gray areas with the models, right? And you just don’t want to be storing it there. And people are particularly concerned if they’re using it to train the model or fine tune it, to use the technical term. And then there are concerns about what happens with that data. I won’t go into the details today. But if you can confirm with whoever you’re working with that the data isn’t stored in the model, then in my humble opinion, you’re good to go. And I think most InfoSec and privacy experts would agree.
Marlene Gebauer (27:09)
curious, mean, it make sense, because you’re talking about a lot of the work that’s sort of behind the scenes, under the hood, does it make sense to make that more public to customers, or is it just overload?
Sam (27:27)
I mean, you know, it’s interesting. Like we did this product release recently and I was of the view that it wasn’t, I mean, I know it’s, it’s extraordinary from a technical perspective, but two years ago, if I told the world or two and a bit years ago, pre GPT, if I told the world, and are we changed the way that our, you conditional logic works? Like, like that would be
Marlene Gebauer (27:33)
Nobody who’s gonna pay attention.
Like, yeah, whatever.
Sam (27:51)
Literally would not
Greg Lambert (27:51)
you
Sam (27:52)
have gotten a second look whereas we just sort of quietly and outsiders did a quick video and a quick loom and And it got picked up by news publications around the world and I’m like, the world’s changed. So short answer is yes, absolutely. The longer answer is I continue to be amazed by how transformative this technology is and I think
the legal world for the first time. I’ve been doing this for a long time now and I was an attorney before this. I think the legal world for the first time is sitting up and taking notice of the tax.
Greg Lambert (28:31)
I want to ask you about some of the lessons you’re learning with the implementation, but I want to morph this a little bit. You talked about your desire for self-service, that people can do things for themselves. Now with AI tools,
I’ve been talking more lately with people that where AI tools tend to give the most benefit to lawyers is in the areas that they’re experts in, which may seem a little off, but it really kind of helps them with the things that they do day in, day out, because…
They know that subject matter and they would know when the AI tools are moving in the right direction so they could use it. And then they also know when it’s kind of varied off course. Now when you talk about self-help, these are not necessarily experts in the area. And one situation recently, and I don’t know if you’ve heard the Canada Air,
Sam (29:49)
Yeah.
Greg Lambert (29:49)
issue
Sam (29:49)
Yeah.
Marlene Gebauer (29:49)
Yeah.
Greg Lambert (29:50)
where that.
So I mean here, you know, there was self help for the for the customers. And then the AI tool gave incorrect information when it came to bereavement travel. And Air Canada had to pay a hefty amount. So I mean, how how do you kind of monitor for these challenges?
Sam (30:06)
Yeah.
Greg Lambert (30:14)
when you are putting the product in front of somebody that may not be the expert in that area.
Sam (30:19)
Yeah.
Yeah. The million dollar question, Greg. That’s right. I mean, it is the question and I’m glad you asked it. The…
Greg Lambert (30:24)
That’s why they pay me the big bucks on this podcast right Marlene
Sam (30:38)
Our great challenge that we set ourselves and kind of no one else in the space set themselves is we need this tool to be accurate enough. I’m going to scrap accurate. We need this tool to be reliable enough to give responses to non-experts, which many people thought was a crazy thing to do. Right? Everyone think about co-counseling, think about Harvey, right? It’s your co-pilot or co-pilot. It’s a co-pilot. It’s sitting there with the expert, helping you work a little bit faster, a little bit better.
Great, that is nothing to sneeze at. But our vision has always been we want to empower clients, people on the street, tenants in New York City to do the legal work that they need to do in order to achieve an outcome. It needs to be done safely. And that is what we’ve spent two years working on, making sure that it can be done safely.
There are few different ways to think about it. So one is all the technical controls that we’ve talked about, right? And there are two phases. One is in building up the tool. And we have a reliability assessment framework that we presented at Stanford at Maastricht University. It is cutting edge. And what it does is it helps you know that the tool is performing at a level that you’re comfortable at.
And that’s much harder than it feels, than it seems when it comes to AI tools. So knowing that your tool is performing at a level, and that requires a bunch of different things, but primarily it’s that expert in the loop to validate it, right? Then there’s post-launch. And so post-launch brings me back to that Megan Ma comment, right? You want to build in controls, triggers, escalation pathways, feedback loops.
so that the experts can jump in when they need to jump in. So it’s definitely not about removing the experts completely from the loop. It’s about removing the experts from the loop where you can safely.
One of the very interesting things that we’ve learned through this project is you can have an internal copilot that is trained on much more complex, high risk data for the experts. And they can use that to help them answer questions. And it’s very interesting that you said that, because I’ve also found that experts in certain areas who know everything, right?
start to use these tools. And I’ve heard them. went to a, for Roxanne, we went to a conference at Fordham and it was every big housing attorney lawyer in the city, housing attorney in the city. And they were using the tool. And I heard multiple times a day, I didn’t know we had that. is there such thing as a HP lawyer? What are they? And then it would spark these conversations, right? I think as experts, we get very comfortable and we stop questioning what we know. And these tools are really good at helping us to question what we know.
But what’s profound and what most people miss is the Harvey’s, the copilots, whatever. What you’re doing there is you’re actually creating a data set that can safely be surfaced to non-experts because you’re creating validated Q and A pairs, which is perfect fodder for an LLM. So there is often a symbiotic relationship between those two applications of the technology that I think people miss.
unless they’re thinking about, how do we empower people outside of this space? I could talk about that for an hour, but I’ll stop there.
Greg Lambert (34:19)
Hahaha
Marlene Gebauer (34:20)
you
you know, Sam, well, before we started recording, you mentioned Roxanne is is a passion project. And I’m wondering what future plans you have for Roxanne or for other A2J initiatives.
Sam (34:38)
Yeah, lots. So this runs through the pro bono arm of Josef. So we started the organization for legal aid and we do that every day. The future plans are to roll this out elsewhere. So we have similar collaborations happening across the country. So Cornell and the Ithaca Tenants Union is ready to launch their tool. We have similar projects about to kick off in Michigan.
Pennsylvania and potentially Tennessee. So it’s really starting to gain traction. Outside of that though, we just work with lot of legal aid organizations. And so I want more and more organizations to see how easy it is, if you have the right partner and the right tool, how easy it is to stand these tools up. We spent a year doing this work, learning, failing, so that we could…
give this off to other legal organizations so that they could run. We’ve now, we can now go from conception of an idea to launch ready tool in a couple of weeks. that is what I want people to take away from this conversation. If they’re listening from a legal aid organization, and that you can do it safely to, to Greg’s question from before. yeah, there’s lots of other obstacles like regulatory change in the U S is slowly starting to happen as well, which I think is really critical. So like,
Bar associations not seeing this as a threat because it’s not. None of the, none of these questions that are being answered by the tool were coming to attorneys before this, right? So again, it’s that abundance mindset. We’re not cutting each other’s lunch. We’re increasing the town. that is if we’re going to participate in this transformation, that is the mindset shift that we need to undertake as attorneys.
Greg Lambert (36:33)
Sam, are at the point now where we ask you to pull out a crystal ball and peer into the future for us. So over the next couple of years or so, what are you seeing as some of the opportunities or challenges that you’re going to be facing there with Josef and all the products, the commercial side and the
access to justice side. What’s going to be good and what’s going to be bad that you think is on the horizon?
Sam (37:09)
The opportunity is the technology.
This is, I was gonna say a once in a generation, but I think it’s even rarer than that. This moment is extraordinary. And I do not think that we will recognize the workplace in three years time.
scary, right? Like it scares me and I’m doing this work. Like that’s, that’s profound. And that’s, that’s the opportunity. So all of these things that we’ve been talking about us geeks, right? That we’ve been talking about for decades now, how we can transform the industry that is now within our grasp and we don’t need to wait another year or two or three for this to get perfect. It’s within our grasp today.
Marlene Gebauer (37:33)
with you on that.
Sam (38:04)
Cool. The challenge is…
are people and are the right people going to pick that up and run with it? And so here what I mean is
Marlene, asked earlier about trust and I want to finish with that because I think that’s critical here. My belief and it has always been this at Josef is we are only going to build the best tools and trusted tools in the hands of experts. When I say the right people, that’s who I mean. I mean, people who understand the underlying content, they understand the clients, they understand the ecosystem.
Getting their buy-in to use this tech to drive that change is going to be critical. I’ll give you an example. ChatGPT today, in the space of the last 30 minutes that we’ve been speaking, how many times has it given legal advice?
Like I shudder to think, right? How much of it was real?
Marlene Gebauer (39:12)
According
to the survey that we were just talking about yesterday, it’s like apparently all of the attorneys are using it for legal questions.
Sam (39:17)
Yeah, literally.
Greg Lambert (39:18)
least at least two thirds.
Sam (39:21)
There you go, right?
so, and I mean, that’s, that’s in and of itself speaks to the opportunity of the tech, but also that’s a worry, right? Because a lot of that’s going to be wrong. and so Greg, to your point, right? We’re surfacing this tech to non-experts. They’re already using it. They’re already going to Google and getting that little response. And actually the answer came from San Bernardino, but they’re sitting in Brooklyn and they’re about to go off and do something that’s, that’s incorrect. So the, what I want is for our community.
to be a part of that push. And if we are a part of that push, then we’re gonna drive profound and positive generational change. And if we’re not, then I think there’s a lot of risk.
Greg Lambert (40:03)
All right, well Sam Flynn, thank you very much for taking the time to talk with us today. This has been a lot of fun.
Sam (40:09)
My pleasure. My pleasure, Greg and Marlene. Thanks for having me.
Marlene Gebauer (40:09)
Yeah, thank you, Sam.
And thanks to all of you, our listeners, for taking the time to listen to the Geek in Review podcast. If you enjoy the show, share it with a colleague. We’d love to hear from you on LinkedIn and Blue Sky.
Greg Lambert (40:24)
And Sam, we’ll make sure that we put things on the show notes, but if listeners want to reach out or learn more about Josef and what you’re doing there, what’s the best place for them to look?
Sam (40:36)
Great, so you can check out our website, so joseflegal.com and if you want to follow me, I talk about all this kind of stuff every week on LinkedIn. It’s Samuel-Flynn, F-L-Y-N-N, and you can find me on LinkedIn.
Marlene Gebauer (40:54)
Terrific. And as always, the music you hear is from Jerry David DeCicca Thank you, Jerry.
Greg Lambert (40:59)
Thanks Jerry. Alright, talk to you right later.
Marlene Gebauer (41:02)
Okay, bye.
Sam (41:03)
Bye.