Predictions about artificial intelligence often focus on job losses and shrinking demand for lawyers. Filevine CEO and co-founder Ryan Anderson and product manager John Rizner offer a sharply different forecast. Drawing on the Jevons paradox, they argue greater efficiency will make legal services accessible to more people, encourage deeper legal research, and create work once excluded by cost. AI might reduce the effort required for individual tasks while expanding the overall volume and ambition of legal representation.
The shift holds major implications for the access-to-justice gap. Faster drafting, research, and document review would allow lawyers to serve more clients without sacrificing professional judgment. Anderson expects family law, immigration, bankruptcy, criminal defense, and employment litigation to experience some of the earliest growth. Motions, witnesses, and legal theories once abandoned over expense become economically viable, although courts face their own capacity crisis as more disputes and arguments enter the system.
Rizner explains how Filevine’s legal AI platform, Lois, applies machine learning to one of legal research’s oldest problems: traditional citators often return different results. Lois combines citation graphs with semantic analysis to locate opinions discussing related legal doctrines even when no direct citation connects the cases. A panel of models then evaluates potential conflicts and produces a structured memo. The goal is richer legal analysis focused on the precise holding or proposition a lawyer needs, rather than a simple flag attached to an entire opinion.
Accuracy still demands disciplined human review. Filevine organizes citation verification into three levels: confirming the cited case exists, determining whether the case supports the claimed proposition, and checking whether the authority is still good law. The conversation also examines Rizner’s research into how different large language models approach efficient breach of contract. OpenAI, Google, and Anthropic models produced dramatically different recommendations, revealing embedded legal and economic preferences beneath seemingly neutral answers.
The guests also explore how AI changes legal drafting, law firm economics, and the billable hour. Filevine’s acquisition of Pincites, now Lois for Word, reflects Microsoft Word’s continuing role as the shared language of legal documents, redlines, formatting, and negotiations. Efficiency does not automatically eliminate hourly billing. Lawyers might instead use saved time to produce more thoroughly researched arguments, stronger contracts, and work product approaching senior-level depth. Firms still need incentives rewarding efficiency rather than treating faster work as lost revenue.
Looking ahead, Anderson and Rizner predict a proliferation of frontier and open-source models tailored to firms, individual lawyers, and specific client relationships. Legal teams will increasingly pair proprietary knowledge with selected models to produce highly specialized analysis. Yet model choice introduces jurisprudential bias, accuracy risks, and serious training concerns for junior lawyers. AI expands the range of available options, while experienced legal judgment decides which arguments deserve trust, which sources require verification, and which advice should reach the client.
John Rizner Slides Filevine Primary Presentation – 2026
Listen on mobile platforms: Apple Podcasts | Spotify | YouTube | Substack
[Special Thanks to Legal Technology Hub for their sponsoring this episode.]
Email: geekinreviewpodcast@gmail.com
Music: Jerry David DeCicca
Transcript:
Nikki Shaver (00:00) Hi Greg and Marlene. Ever since Anthropic launched Claude for Legal, a lot of focus has been on whether firms and legal departments should be using that in place of, or as well as, legal-specific applications like Harvey and Legora. But Anthropic is not the only big player to brush up against the legal market. Microsoft has launched its legal agent, embedded in Word and other 365 applications. OpenAI has reportedly hired someone to help build in legal. Perplexity has launched Computer for Counsel with a host of legal MCP connectors. And now it seems Amazon may also have its sights on our vertical with Amazon Quick for Legal. It was already a tough market to navigate when all you had to worry about were the thousands of legal applications; now you also need to track what’s happening in the broader tech ecosystem. We’ll soon be publishing a helpful comparison of the big tech offerings for legal, with an examination of where in the legal tech stack they might be useful. Stay tuned for that on our site at legaltechnologyhub.com, or follow along on LinkedIn at Legaltech Hub. It’s a pleasure to see all of you. Until next week.
Marlene Gebauer (01:13) Welcome to The Geek in Review, the podcast focused on innovative and creative ideas in the legal industry. I’m Marlene Gebauer,
Greg Lambert (01:20) and I’m Greg Lambert. And today we are going to be digging into a fascinating and probably somewhat counterintuitive theory that’s turning the traditional panic over AI on its head. Instead of asking how many legal jobs AI will destroy, we’re actually going to explore why AI may actually trigger an unprecedented explosion of legal work, opening up massive markets that were historically priced out.
Marlene Gebauer (01:48) In hearing what our guests have to say about this, and to lead our conversation, we are absolutely thrilled to welcome John Rizner, product manager at Filevine. And joining John is Ryan Anderson, the CEO and co-founder of Filevine. John and Ryan, welcome to The Geek in Review.
Greg Lambert (02:03) Welcome, guys.
Ryan Anderson (02:04) Good to be here, Greg and Marlene.
Greg Lambert (02:05) All right, John, let’s start off with you, because we wanted to talk about the Jevons paradox, which I know a lot of our listeners have heard before. But let’s look at the big macroeconomic picture that you guys have brought to our attention as well. Many pundits out there are predicting that AI will contract the legal industry, but you guys are arguing the exact opposite of that using the Jevons paradox — the economic theory that technology makes resources cheaper and more efficient to produce, and then consumption actually skyrockets. So do you mind just talking a little bit about how the paradox applies to legal work, and how we should be prepared for that?
John Rizner (02:51) Yeah, totally. So I’m going to talk about it in connection with a product that I’m working on: a citator, à la Shepard’s-style citations, within LOIS. When we started off investigating our tooling, I had a number of conversations with some old law school classmates of mine that did far better in law school than I did — appellate litigators who went through the elite clerkship rounds. And I was talking with them: “If you had the ability, with large language models and new AI tooling, would you be excited to chew through fewer research materials as part of your work?” What was interesting is all of them said, actually, if I had unlimited time and unlimited resources, I would want to find more and read more, as opposed to find less and read less. If they could, they would love to boil the ocean — exhaust every possible resource in reading through opinions and relevant literature on their case before drafting or preparing anything. So I think as AI tools, especially in the legal research space, the citator space, make finding the relevant literature easier — even making more of the relevant literature an immediate delivery to the user — rather than lawyers spending less time doing research, we’re going to see lawyers spend more time doing substantive research. The difference being, instead of lawyers saying, “All right, I’ve got X amount of time, I found this handful of cases, let’s quickly draft this and get it out the door,” I think you’re going to see far more in-depth and nuanced arguments in legal work product, due to the fact that lawyers will have easier access to greater amounts of research material. And as a consequence, you’ll probably have better opinions come out of courts, because the arguments lawyers are making are going to be more nuanced, reflecting the deeper research. So I hope lawyers like doing research — I hope that’s why they went to law school — because in my view of it, you’re going to see greater and greater research, and far higher quality research, as opposed to lesser or more shallow research.
Greg Lambert (05:23) Yeah, that was going to be my follow-up question, on a conversation we had last week — one of the things she mentioned was: we used to get these one-page surveys or reports, and now we’re getting these 40-page surveys. And it’s not necessarily better; it’s just more. But I think your argument is that the outcome, the results, the research is actually better rather than just more research. Am I interpreting that correctly?
John Rizner (05:55) Yeah. So I think, at their core, and obviously we all know as part of our ethical duties, lawyers should be diving through the cases that they’re possibly looking at. And I think as these tools evolve and become more mature and can find more relevant and more on-point items for the lawyers to read, they will read the same amount, maybe even read more. But the conclusions they come to will be stronger, because they have that better picture of how the law is at that particular moment. So volume may grow, and in addition to volume growing, I think quality becomes far better in the long run.
Marlene Gebauer (06:38) Yeah, I think that makes sense, because AI kind of allows you to separate the wheat from the chaff, and then you can focus more deeply on the things that you need to focus on and sort of leave the other ones.
Ryan Anderson (06:52) Yeah, I’ll speak to that for just a moment too, from a market perspective. Two, three years ago there were a lot of predictions around what would happen to engineers, coders. Filevine employs a lot of coders — I think we will end the year with something like 400 to 450 engineers, about half of which are ML engineers, the other half your traditional coders. And there were quite a few predictions that AI would reduce the need for corporations, for tech companies specifically, to have as many engineers. The theory was that there will be so much code written, and it’ll be so much easier to write, that you won’t need as many engineers — so you’ll see some layoffs and fewer engineers hired. Well, it’s been the case at our company, and I think broadly across the industry, that the exact opposite has happened. The engineering market sort of hit a low maybe two or three years ago and has actually climbed out of that. More are being hired — not more than ever, but certainly more than the post-COVID lows. More companies are bringing on more engineers to do more work, and there’s a very logical reason for this. We definitely write 10 times more code today than we did before — it might even be more than 10 times more. But for every piece of code that we write, it has to be reviewed, it has to be QA’d, it has to be tested. And as the aperture of the product increases, the surface area for problems increases, but the ambition of the product also grows. What that has allowed us to do as a company is build a much richer, broader, more end-to-end product offering that we think is really unique, but I think it’s the same for a lot of companies. The analogy really holds true for legal. As John brought up, you’re going to have lawyers able to do research more quickly and find truth for their clients in a much broader way, and be able to zero in and find creative ways to apply the facts of their case to the law. But even consider the lawyer who says, “Instead of bringing that motion the client just didn’t want us to do because it was going to be expensive, I didn’t bring the motion to compel the deposition of that witness. We just let it go — we just didn’t do it in this case.” That happens all the time. If you talk to litigators, they’ll say their clients didn’t want to pay for witness number six, who might have had some interesting information in the case. But boy, now that you can bring that motion to compel, and it’s much easier to draft and much quicker, the opportunity costs change, and the search for truth becomes sort of superpowered. And of course, litigation in particular is a counterparty affair — what one side does, the other side is having to respond to, and then maybe bringing a counterattack themselves. So for better or for worse, I think truth will get sought out more quickly. But also, I think lawyers are going to be very busy. And honestly, I wouldn’t want to be a judge in this atmosphere — I think it is really tricky for the judiciary at this time.
Greg Lambert (09:47) Yeah.
John Rizner (09:50) And I think one other item to bring in on this truth question: it’s not like we are looking at a field where all the buried treasure has been dug up. One of the things we were looking at early on was research by Paul Hellyer and Susan Mart, two legal scholars, who had done quantitative analysis of how often, say, Lexis and Westlaw agree or disagree on a particular citation, or how often they return the same relevant opinions for a particular citation analysis. What’s really interesting is that the differences between the two could be substantial, depending on the opinions looked at. So if you’re a subscriber only to Westlaw or only to Lexis, you may be missing items that Lexis is returning but Westlaw isn’t, or vice versa. And so where AI tooling makes it cheaper and easier to apply different methodologies — so you can have your multiple citators taking different approaches at a larger scale — I think you start to discover more of those opinions that were otherwise hidden, because you were stuck with one citator and only uncovering those hidden truths via one type of methodology.
Greg Lambert (11:10) Ryan, I want to go back to something that you mentioned about using the engineers as a parallel path to some of the things that we’re seeing in legal. I was listening — I believe it was Adam Mosseri from Instagram — talking about how his engineering teams are made up now versus how they were pre-AI, where he was saying every project had like a baker’s dozen of people on it, ranging from engineers to project managers to researchers and data scientists. And he says now it’s essentially a team of three. Are you seeing similar things on your engineering team? And do you think there’s a parallel for how lawyers will be doing it?
Ryan Anderson (12:00) Yeah — and sorry, sorry to interrupt you, but yes, we are definitely seeing a reduction in what a single person can do. Let me give an example. This just happened today, about two hours ago. I was going through a new feature with our team. We were thinking through how to design an experience on our deep retrieval engine. We think we have really world-class retrieval — we think we can search through more documents than almost any product out there; we literally think we’re the best in the world at this. But you might imagine there are some UI constraints: telling the user where you are when you’re searching through millions of pages of documents can be tricky. There was some back and forth about how to do this, the team’s debating, there are about 10 people on the call, and we’re saying, “Well, how about this method to tell the user what’s going on? Here are some other methods.” And about 15 minutes into the conversation, one of the engineers goes, “Well, I actually just coded it. Does anyone want to see what I just did?” And we literally shared a screen, and he showed something — not in a production environment, but on his local machine using real live code — and said, “Here’s how I think we should do it.” It was working, and we could actually play with it right there during the meeting. I can tell you that kind of interaction never happened before. Never. We have lawyers who say, “I was exploring an argument that I didn’t think would be a good idea, but I used LOIS, and all of a sudden it found a couple cases, and it was a line of case law theory that I didn’t think applied to my case, but it does. And so we’re going to actually attack the other side in a way that we hadn’t realized at all.” When you lower the cost for exploration, for creative exploration, people become much more creative. I’ve always thought the best lawyers are actually exceptionally creative. It’s funny — lawyers don’t think of themselves as creatives. They think of artists and movie stars and musicians as creative people. But I think great lawyers are exceptionally creative, and by lowering the cost to go on a creative thread with a legal theory, you enable much more creativity. It’s very exciting.
Marlene Gebauer (14:09) So Ryan, when we talk about lowering costs to capture a dormant market of middle-class clients and small businesses, what specific practice areas or legal needs do you think will see the quickest influx of new work? How does Filevine view this unfolding from a macro strategy perspective?
Ryan Anderson (14:29) There are many great things about AI, and there are some scary things about AI. But in the legal industry, perhaps the most exciting is — we all know that getting a lawyer is actually quite a challenge. My brother, who’s our chief product officer, talks about how he needed a lawyer to help with a relatively small real estate dispute. Here’s a guy who literally leads product for a pretty significant legal tech company, has a brother who was a lawyer, and knows many, many lawyers in his day-to-day job, and he could barely get somebody to answer the phone. It is hard sometimes to find a lawyer — and that’s a well-connected guy. If you’re somebody who’s indigent, working class, maybe you don’t grow up with the same kind of privileges or connections, it can be extremely challenging to find good legal representation. And it’s pretty awesome that lawyers are going to have much more capacity now. You asked what industries: I think you’ll see a lot more in family law, a lot more immigration law, a lot more bankruptcy, and I think underrepresented criminal defendants are going to be in much better shape than they are today. In fact, a lot of those state public defenders’ offices are clients of Filevine. The Innocence Project is a client of Filevine. You’re going to be able to have more people who have meritorious claims for wrongful prosecution — or for having a lawyer who didn’t do the job they should have been doing — win their case, or at least get them a lower sentence. Many more of these cases can be taken on. It’s incredibly exciting. So I think you’ll see it across the board in those areas. The example I give is: these are lawyers, so we’ll say they have 10 hours in their workday, and it now takes two hours to draft a will instead of eight. Well, how many more people can they serve in a given day? If they could draft one will in eight hours, or maybe it was four, and now can do it in two, you’re now serving something like double the number of people during a day. You don’t even have to reduce your rates that much — maybe your rates stay the same, they might even go higher — but you’re able to serve more people, and those people wind up paying less because they’re taking less of the lawyer’s time. So it’s a very exciting world we’re going to live in. We don’t know exactly how it will all play out, but more legal customers will be served than ever before. It’s very exciting.
Marlene Gebauer (16:55) What do you think about labor and employment? Would that be an area as well?
Ryan Anderson (16:59) Oh, for sure — undoubtedly. I would love to say that this is maybe good news for corporate defendants. It’s probably not good news for corporate defendants. They’ll have more tools at their disposal to investigate these claims, and they’re often not meritorious, and Filevine wants to help those customers. But also plaintiffs’ lawyers who have meritorious claims against corporations use our products. We really want to help lawyers find the truth; we want to help the justice system find the truth. We are huge believers that the American system of justice not only is the best in the world, but forms the infrastructure for a fair and just capitalist society. We think the lawyer’s role in a system of capitalism is really critical, because otherwise it’s hard to keep business in check. So we’re really proud to serve our customers, whatever side of the “v” they may be on. But yes, I think you can expect an increase in litigation across the board.
Greg Lambert (17:52) I’ll go back to when you were saying it’s going to be a difficult time to be a judge right now, because I think that’s one area where, if we don’t figure out how they are going to handle this massive influx of new cases, it doesn’t matter really how much of the efficiency gets on the plaintiff/defendant side if the court system is inaccessible.
Marlene Gebauer (18:20) I just wonder if they’re going to get like an AI version of Judge Judy — people will agree to the AI judge and let them decide in smaller matters.
Greg Lambert (18:24) Are you guys working on that, Ryan?
Ryan Anderson (18:24) Yeah, you could see kind of smaller matters, private matters. Obviously, I think there’s a different need to be there, but —
Marlene Gebauer (18:38) I could see that for mediation, sure, stuff like that.
Ryan Anderson (18:55) Yeah, I think so. I think you can envision small arbitrations that are private, where at least they agree to maybe some AI tooling being used. I think that’s almost certainly going to happen — it probably should, to a degree. Let me be clear, though: Filevine and LOIS, which is our AI product, is the core economic engine of this business. We basically only sell LOIS and AI products today, which is quite a difference from where we were three to four years ago. So AI is incredibly important to me; it’s incredibly important to this company. We are fully AI-pilled in how bullish we are about this industry. Having said all of that, I think the judgment of the lawyer, the judge, the legal professional isn’t going away for a really long time. It is one thing to see an AI output as an unsophisticated consumer of that information and say, “Geez, this kind of looks pretty good and pretty persuasive.” But I think all of us have seen enough AI results and prompts and, frankly, slop to go: hold on, hold on — this looks like it’s right, but it is in fact not right, and sometimes in really critical but perhaps nuanced ways. That kind of legal judgment is going to exist for quite some time. So I think we’re a very far ways off from lawyers being replaced, from judges being replaced. But as I’m sure everyone in your audience already realizes, the age of AI in legal is here — it has been here now for a couple of years and is very squarely in our era. We will talk to our grandkids about this transition; it is a very big deal. But we’re huge believers in the primacy of human judgment when it comes to the law. We believe that lawyers play a really critical role.
John Rizner (18:55) And I want to tie back to some research that we did this last year on how LLMs respond — we may get into the paper in a little bit in this podcast — but we were looking at how different LLMs, or different families of LLMs, react to the same law-and-economics breach of contract issue, and how they compare to humans. What was maybe surprising, maybe unsurprising, was that depending on what model or what family of models you were using, you could see dramatically different results on whether the LLM was pushing the judge or the practitioner to push for a breach of contract or to keep a promise. So I think there could even be a future in which — if we’re talking about our LLM-based arbitrators — folks are fighting over which LLMs or which tech tools to use, because there is that risk that the tool you use could affect what answers you’re coming to. And I think that’s even another argument why the human attorney needs to remain a core part of the practice. I don’t think we’re yet comfortable giving over our judgment to tools that might already have a judgment built in that we don’t even agree with.
Ryan Anderson (21:45) Yeah, and of course — yeah, go ahead.
Marlene Gebauer (21:47) No, no, go ahead. Sorry.
Ryan Anderson (21:50) No, no. I mean, look, we all see the hallucination news daily, and it’s everywhere. I think lawyers are right to be very concerned about it. I think judges are right to take a really strict view of hallucination. It’s out there all the time; it is an extremely challenging problem. Hopefully we’ll get a chance to discuss what we’re doing on anti-hallucination — that’s an entire team at Filevine, the anti-hallucination team, certainly with respect to case law. But I’ll just give you one example. We had a customer put some data through Claude versus LOIS — we think LOIS is much more precise than Claude — and Claude came back, and it had taken some testimony and put in quotations something that a witness had said. Literal quotation marks around what a witness had said. And it turned out that witness hadn’t said that; it was sort of an amalgamation of three or four things the witness had said. If you looked at each statement the witness had made, you can understand how Claude would have arrived at the conclusion that the witness had said this quote — but the witness had not said that. And any lawyer looking at the quote versus what the witness had actually spoken and verbalized in the deposition: totally different. Totally different. So the legal judgment to know the difference between what is close and what is precise is going to be needed for a long time. We’re not there yet. AI tools are very important, but they have to be watched over, for sure.
Marlene Gebauer (23:15) Well, I want an anti-hallucination T-shirt — an anti-hallucination team T-shirt, that’s what I want. So John, Greg and I saw your presentation at Texas Trailblazers a few months back, and you noted that AI adoption follows incentives, and that trust in AI is ultimately a workflow problem. Since of course we’re talking about Gen AI, it wouldn’t be fair to not have a billable hour question. If efficiency gains allow lawyers to draft a complex contract or review a file in a fraction of the time, how do firms need to restructure their incentives so they aren’t punishing efficiency under this traditional hourly model? And honestly, if you have this, that’s the secret sauce, because we’re all grappling with that.
Greg Lambert (24:05) This is the billion dollar — maybe trillion dollar — question.
Marlene Gebauer (24:07) Yeah, so here you go.
John Rizner (24:09) I think, on the one hand, the billable hour has survived a lot of technological change so far and has not disappeared yet, so I think the billable hour has some weight to it that will be hard to remove. Coming back to Ryan’s point on the will piece, as well as our opening point about quality: say producing a particular piece of work product today takes eight hours, and we push it down to two to get that same exact work product. Well, there’s still runway to improve on that particular document — a better-researched argument, a more ironclad contract, a will that really thinks through all the elements of this particular family situation and gets it exactly how the — this is throwing back to my law school days — the testator, is that the right term? Got to go back to trusts and estates. But I think with that extra time, the associate or the partner now has the ability to provide a better work product, something that really reflects what their particular client needs, at the same price or less, or maybe slightly more, but with a depth that wasn’t even close to what we could achieve today. The deep nuance that an associate may not have been required or expected to have today will be a requirement in the very near future and in the long run. So those extra six hours will be spent producing that higher-level, almost partner-level deep quality that just wasn’t accessible or doable today. So I think the billable hour survives for a while — unfortunately or fortunately, depending on what side of the debate you’re on. I know I never enjoyed the billable hour, but I think the economics of it still work, especially if you think that volume and quality become a greater and greater focus.
Greg Lambert (26:09) Yeah, I’ve yet to meet someone that loves the billable hour, but yet here we are.
Marlene Gebauer (26:15) When you find one, let us know. We want them on the podcast. It’s like: explain yourself.
Greg Lambert (26:21) Yeah. John, I want to also jump in on your presentation that you did a couple months ago at Texas Trailblazers, because I wasn’t sure how well that was going to go over — it was a pretty in-depth, very, very deep dive on citation systems.
Marlene Gebauer (26:38) We loved it.
Greg Lambert (26:44) But I asked a couple of partners at my firm who were in the room there, and they were like, “This was the best part of the whole conference. I love this.” So let’s jump into that, and more on the LOIS legal research product that you guys have as well. You’ve noted that traditional citators tell you about the case — we kind of talked about that earlier — but lawyers really care about the specific holding. Do you mind giving us almost a Reader’s Digest version of what you presented there, and talking about these dual-pathway retrieval systems under the hood? I’ll turn it over to you.
John Rizner (27:26) Yeah, absolutely. So our citator approach all came from the idea of: are there workflows in the current legal system that we think we could mimic using ML tooling and LLMs, to come to the same answer that the current legal process works through? What we ended up falling into is: on the one hand, citation graphs — your traditional citation approach — do a really good job when we’re looking at those cases that directly engage one another. Chevron and Loper Bright, for example, where you have Loper Bright talking expressly about Chevron. But there are all those cases that might talk to issues around the issue that you really care about, but might not expressly be cited by the potential treating cases — because it may be only a part of the larger case, or, for whatever reason, a clerk didn’t include it or a judge wanted to make a strategic decision, and those links just might not exist. So we thought: all right, can we find a bunch of opinions that might be relevant, either on a citation graph or, as we came to it, via semantic similarity — the inherent meaning of the text as turned into a kind of mathematical representation? Could we find all of the opinions that could exist and be relevant to the lawyer’s issue? And then from that, let’s pass that into almost a mock en banc process, where we have a bunch of LLMs operating as judges trying to decide: do we care about these opinions? Is this a relevant opinion to the issue we’re looking at? Do we think it conflicts with the issue we’re looking at? And from those mass LLM runs, produce a structured memo for the user saying: in this universe, we found these opinions that might cut across your specific chosen issue, for these reasons. What we’re really excited about is finding those opinions that do negatively engage one another but are really hard to find, because they’re not cleanly on a citation graph. The one I always go back to — I’ll look here in Utah — there’s this case called Brinkerhoff v. Salt Lake City, and the last few paragraphs of Brinkerhoff talk about this old doctrine of governmental immunity. There’s this test where the question of governmental immunity turns on whether the government activity is proprietary or governmental. If you pull up Brinkerhoff on some traditional citators, you’re not going to see any negative flags on it. And the reason is that Brinkerhoff has been talked about in a negative way in later-down-the-road Utah opinions, but the doctrine that Brinkerhoff used in those final few paragraphs was ultimately cut down by the Utah Supreme Court and the Utah legislature. Because we are, one, surfacing the fact that Brinkerhoff has talked about this doctrine, regardless of whether those later cases mention Brinkerhoff, and two, because we’re giving it to this LLM panel approach — this voting and memo-writing approach — the LLMs can recognize: wow, this doctrine is being used in Brinkerhoff; wait a minute, here’s Standiford, this later-in-time case that attacked the doctrine. We think there’s a connection there — a conflict there. We are able to find these items that are, again, otherwise unfindable on a traditional citation graph. That’s the thing I’m really excited about: we’re able to find for lawyers these ideas and conflicts that may be relevant to the other side’s point or opinion. What we want to surface to the lawyers is: hey, there might be other attacks on that particular proposition of law that we can now help you find, that were just undiscoverable before.
Greg Lambert (31:36) Yeah, man. It just brought that presentation all back to me. I remember how I geeked out on that train.
Ryan Anderson (31:44) It’s very cool. You know, I know this will be posted in transcript form — if there’s a way to show some of John’s slides, I mean, the slide that shows that…
Marlene Gebauer (31:53) Yeah, if you can give us a link to the slides, we will post that up in the show notes, absolutely. It’s so cool.
Ryan Anderson (31:58) Yeah — I mean, I think as you all probably were when you saw the presentation, just the notion that Westlaw and Lexis find different sets of cases. Good night, that’s terrifying.
Marlene Gebauer (32:09) We’re just like, yeah — someone’s finally saying it out loud.
Ryan Anderson (32:13) Right, right. And we’ve just shown that this method actually picks up a lot of cases that, in some cases, neither of them found. So it’s really interesting.
John Rizner (32:13) When we were starting to first benchmark our tooling, we had been pulling down those studies by Susan Mart in Colorado about the different cases they were finding. And we were like: some of this research was done eight years ago or five years ago — I wonder if, because of technological change, these differences have started to go away. In our current benchmarking: not so much. Still the dramatic divergence — which citator you happen to be using is going to give you a particular answer, and if you’re using a different one, you’re going to get a different answer. So it’s amazing just how much opportunity there is to help fill those gaps and give folks an ability to find things that they otherwise aren’t finding right now with their current tool.
Ryan Anderson (33:20) I’ll just briefly note: we have such a wealth of case law, a rich history of case law in this country. It’s so cool that most things have been decided — the vast majority of things have been discussed at some point by some court somewhere — which gives such stability to businesses, to human beings, to politicians, to people operating in our country. It gives everyone a tremendous amount of confidence when they make decisions, especially business decisions, to understand what the nature of the law is. And yet the citation system that we use was written — and it was the best we had — in a really deterministic, code kind of way. Here’s a citation; it links to this other citation, which links to this other citation. And that’s great — that’s what we had available to us, and it took humans to chain those cases together in logical trees, progenies of different cases. But LLMs are particularly good at saying: these words, these phrases, are semantically similar — they have similar meaning to this other set of cases over here — and even though there’s no hard-coded citation, we find similarity here, and you should take a look at it. It’s really fascinating. It’s a great use of LLMs. Go ahead, Marlene.
Marlene Gebauer (34:22) I just think about the discrepancies that you’re talking about and how that impacts — and we’re going to talk about the hallucinations — but when you’re trying to check for hallucinations and for legitimate cites, how does this play into that? And with how the courts are coming down on this, how do people rely on the tools that they have? Or should they?
Ryan Anderson (34:48) Yeah, I mean, we think we have a solution. I’ll let John talk about that.
John Rizner (34:52) So, in my view, this isn’t even specific to legal research in terms of case law: there’s such an importance to having corpuses of the information you care about available and engineered in a way that is best presented to the LLM, in the right way, at the right time, and in the right process. The real question I think lawyers need to be asking their tech providers is: all right, what corpuses are you using? How are you presenting them to your AI technology? And what benchmarks, what kind of evidence do you have that your approach is working well? It doesn’t matter if you’re using AI in your case file, trying to find the right document that outlines a scientific expert’s findings on something, or a point from a deposition that you really care about, or, on the legal research side, a particular opinion or particular citation that you care about — either way, it’s really important that your legal tech has engineering designed to get to the right point in the corpus for the answer you care about. For us, on the document side — Ryan was talking about that earlier with our enormous data science groups — they’re really focused on our corpus of case file material, and on stopping hallucinations by using what we call the knowledge engine to get the right answers surfaced to the LLM. On the citation side, what we’re focused on doing right now is taking open-source corpuses of grounded decisions — in this case, CourtListener being our big partner in that — and any time text is produced within LOIS in the chat, whenever we see a citation pop up, having tooling then check: okay, this has been brought up — does this case exist? And more importantly, does the user have an opportunity to check if the discussed opinion is discussing the issue we care about? Because one of the things we’re now seeing judges talk about is: okay, the case you cited exists, but it doesn’t support all of the propositions you’re trying to support. And in my mind — and this is what we’re building towards — there are almost three levels of evaluation you have to do for any particular piece of case law.
Marlene Gebauer (37:10) It doesn’t say that.
John Rizner (37:21) On the first level: is this case actually real? Does this citation actually exist somewhere in the corpus? That’s your very base level, right — is “John v. Ryan in the Territory of Guam” entirely made up? Was that citation made up? On the second level, you have that second-order hallucination: does that citation actually refer to and support the proposition you care about? Sure, the case exists, but are we talking about, I don’t know, Roe v. Wade in a contracts dispute? What’s being cited doesn’t support what you care about. And on the third level is that citational analysis: okay, the opinion exists, it supports what you care about — now, is it good law in the larger common law analysis? So that’s how we’re thinking about it: are we building towards hitting all three of those evaluation levels?
Marlene Gebauer (38:16) All right, I want to switch gears for a little bit and move away from litigation and more into the transactional area. Filevine started with a strong focus in plaintiff litigation, but this January you acquired Pincites, which is a legal drafting and redlining tool, and brought in Sona and Mariam Sulakian into your executive team, rebranding the tool as LOIS for Word. As a little bit of background for listeners who might not know: Sona was formerly a legal strategy expert at Ropes & Gray, and Mariam is a former GitHub and Meta product engineer. So congratulations — you brought in the dynamic duo. And I have a couple questions here. Why is it critical to corporate and transactional expansion to have something inside of Word? Particularly because I think we’re seeing now, with some of the larger LLMs, that they’re actually doing a lot of the work inside that environment — and I realize they’re doing stuff in Word too — but you do see a lot, particularly when you’re working with larger sets, that they’re actually doing it within the LLM environment.
Ryan Anderson (39:31) Look, it’s a great question. I don’t think Word is the final surface for all legal drafting, but it is definitely the dominant surface still today — and we’re not even talking about AI legal drafting, just legal drafting. The lingua franca of law is still Word, and probably will be for some time. If I said to you, “Hey, here are the redlines to this agreement. Go ahead and open this link with this other document type you’ve never heard of. Don’t worry about it — you’re going to have to sign in and make an account to see it,” Marlene, you would say, “No, thank you. Yes, hard pass. I need the Word document, please.” And I think pretty much every lawyer feels the same way. They have all learned to use Word. They understand it. They understand how comments work in Word. They understand how redlines work in Word. So getting an entire industry to re-platform may be challenging — certainly in the short term, probably in the medium term, maybe not in the long term. But re-platforming on a different drafting modality is just going to be really hard. So I think we’re stuck with Word — maybe that’s the right way to say it, maybe not. Everyone uses Word because it is the most fully featured drafting product the world has ever seen. So there are some good things about Word.
Marlene Gebauer (40:52) Greg likes WordPerfect. Sorry. He’s a big fan.
Greg Lambert (40:53) I was going to say — WordPerfect 4.2 on DOS. I mean, I’m up for bringing that back.
Ryan Anderson (40:56) There we go — you’d be surprised. Recently we still got a lot of customers asking us to integrate with —
Greg Lambert (41:01) Yeah, those reveal codes, man. Still need those.
Ryan Anderson (41:06) So I think that’s really critical. First of all, Sona and Mariam would be the first to tell you — and by the way, Marlene, you are right, they are the dynamic duo. These are two very sharp women — sisters, of course. First of all, lovely human beings, just a delight to work with, but as smart of folks as I’ve ever worked with in my career, and really fun to have on the team. They’ll anchor our San Francisco office. That was some time ago now — six months ago — so we’ve actually built quite a large team around them at this point, and that team is solely focused on AI legal drafting. Of course, they use a lot of the work that John and others have done, but they would be the first to tell you that Word is not the only surface they’re going to work on, and they already work on other ways to draft. But given the gravity that Word has in the industry, I think it’s going to be around for a while. There is a benefit to having everything be in Word — getting the formatting right is almost impossible without Word, depending on the court.
Marlene Gebauer (42:04) Formatting is a tough thing.
Ryan Anderson (42:06) It is very challenging. And look, I fully understand judges’ particularity and frustration around hallucinations — but they get equally as frustrated around some of the tiniest formatting issues. I’m a little less like — okay, does it really matter if the margins are 1.25 instead of 1.1 or something? But I can tell you that it does to them, and if it does to them, it does to the lawyer practicing in their courtroom. So it’s really not optional, and Word can reliably produce the best-formatted documents for legal in the world. So we need to be on that surface, and we want to be world class there. And we do think LOIS for Word is world class — we’ll put it up against Claude for Legal or Harvey or Legora or any of our other competitors. We think it’s the best redlining and drafting tool out there. And to that end, we should note that not only is LOIS for Word very good at redlining — we think it’s the best at redlining — but the ambition is much stronger than that, much broader. It is drafting long-form, sophisticated, complex legal documents grounded in evidentiary citations and case law citations. That is an ambition that I don’t think many of our other competitors have gotten to quite yet — maybe some at the top of the market are with us there. That’s the product we’re building, and we feel really proud to build it with those two at the helm. They’re doing great, and will do great.
Greg Lambert (43:37) Yeah, sounds like they’re a great team to work with.
Ryan Anderson (43:41) They’re awesome. I don’t know if you’ve met them individually, but sharp, lovely, fun, and incredible — I would say impeccable product taste. We think their thought processes and intuition around how to build products that lawyers really love and want to work with day in, day out is second to none.
Greg Lambert (44:00) Well, now I’m regretting bringing you two on. We should have brought those two on.
Ryan Anderson (44:01) Sure — well, I can tell you, they would be better.
Marlene Gebauer (44:02) I was going to say, now we have to bring them on. Were you teeing them up? You are teeing them up. Good, because I think that would be a fascinating discussion — why lawyers like tools. I don’t even know that they know why they like tools. So I think that would be cool.
Ryan Anderson (44:21) I think you’re right. Well, we’d love to have Sona or Mariam come on the podcast sometime.
Greg Lambert (44:27) Well, John, I’m going to jump ahead because I want to do one more geek-out with you while we’ve got you here.
John Rizner (44:32) Sounds great.
Greg Lambert (44:33) I want to bring up a working paper that you co-authored called “The AI’s Philosophy of Contract,” where you empirically studied how frontier large language models are handling classic concepts like efficient breach and remedies. So here’s our chance to geek out: what did this reveal, and what interested you in writing this paper?
John Rizner (45:04) Yeah, so we came across a paper that was basically an empirical study of how humans respond to your classic efficient breach scenario — where it is more economical to breach the contract as opposed to adhere to and follow the contract. Humans, interestingly enough, had certain breach rates, but if you included a specific remedy in the contract, you could change how humans, in the behavioral economics sense, respond to those efficient breach scenarios. So we were curious: as LLMs become a place where you go to ask for legal advice, how might they respond? Will they follow humans in how they respond to efficient breach, or will they take a colder, law-and-economics view, or a softer view? That was our big question. So what we did is we took what were, at the time, the main frontier LLMs — Anthropic, Google, OpenAI — and created large sets of efficient breach scenarios, and tracked how these LLMs would respond to those scenarios. And the divergence was wild. You had Google and OpenAI, who were more like, “All right, it’s economical — go ahead and breach.” Well, Anthropic was like, “We are never breaching. We cannot breach.” You might have a swing from high-90-percent breach rates to under-10-percent breach rates depending on what model you were choosing. One of our big takeaways was that, for practitioners, this should be a thing to think about, because what model you’re using might determine whether you’re getting advice of “let’s tell the client to do X” versus “let’s tell the client to do Y.” The other big note we found is that the LLMs likewise respond to whether you had specific breach remedies within the contract, in a way that a human would also respond to those types of hints within the contract. There’s a lot of discussion in the frontier labs in San Francisco about alignment of humanity generally with these tools. I think lawyers need to start thinking about alignment as in: does this particular LLM align with my particular jurisprudential philosophy, or how I would approach this particular question at a more theoretical, more philosophical element of judgment? Because if you just hand it over to the LLM, what answer you’ll get and what legal advice you might get will be determined by whether you happen to be talking to Anthropic or talking to OpenAI on a particular day. A really interesting question as well: when we ran that empirical data, the open-source models hadn’t yet had their big day in the limelight. It’ll be interesting to see to what extent open-source models become a way to choose a particular piece of jurisprudence baked into the model — that either you prefer, or a particular court prefers, or a particular judge prefers. I think there are real questions on that, as models proliferate, of folks choosing models that might fit their particular jurisprudential philosophy. Say that five times fast.
Greg Lambert (48:33) It just made me think whether it follows the University of Chicago economics theory or Berkeley economic theory. So I guess it’s how it’s trained, right?
Marlene Gebauer (48:43) It’s super interesting, because I just did a client presentation this morning, and they were talking about: okay, even if it’s an approved enterprise foundational model, if people put things in and the clauses are not what they would put in, but that’s what’s recommended — it just highlights how important it is to have a playbook, or some sort of template or guidance to use in addition to the model, because you may be doing something wrong. That’s one point. And the other point is, I think about more junior people using these tools. Someone who’s got more extensive experience would look at this and automatically say, “Absolutely not.” But somebody who’s more junior, and who’s actually using this as a learning tool in addition to a drafting tool, is really going down the wrong path.
Ryan Anderson (49:51) I couldn’t agree [more]. I think it’s very scary. Junior attorneys — boy, you almost wonder how much they should be using it. It’s very challenging. And we see it at the company: some of our employees who are junior at the company or new to the company often rely far too heavily on AI analyses and responses from frontier models, and gosh, you wonder — did they even think about these things? So it’s a big challenge, and for that very reason, we’re huge fans of lawyers and legal judgment. We think it’s going to be around for a while.
Greg Lambert (50:28) All right, guys, we’re going to jump to the crystal ball question. Looking out into the near future, what are some challenges or changes that you think we’re going to have to be prepared for as we move along in this age of AI?
Ryan Anderson (50:46) You will see model proliferation. We’re speaking to you on a day that Meta just came out with maybe their first really good model, and I don’t think anybody had that on their bingo card. But here they are with a model that looks — at least initially, based on the commentary online and even some testing done — like a pretty good model, and pretty good at legal. So now all of a sudden there’s at least a fourth player, probably a fifth: you’ve got Grok, Gemini, Claude, OpenAI, and now Meta. You’re going to see a lot more of these. We internally are using more open-source models than ever before — it is now part and parcel of the work we do to fine-tune and train open-source models. A lot of folks believed there were going to be maybe two, maybe three dominant labs, and people were going to mostly build on top of them. That does not look like it’s going to be the case. In fact, we’re seeing the open-source models be as good or better, with relatively limited fine-tuning, than even some of what the frontier models can do. So that really changes the economics for everybody. First of all, it means we can use a lot more inference — it’s not as costly as before. It also means there will probably be a lot more folks — end customers, law firms — choosing to have a stake in helping to build their own models, and we want to be there when customers do that. That’s part of the beauty of LOIS: law firms can really customize it to them and keep what makes their firm great within LOIS. So I think you’re going to see a huge proliferation of models, which is going to be different, it’s going to be confusing, and it’s not, I think, the environment a lot of people expected to see even just a year ago. So that’s my big crystal ball prediction.
John Rizner (50:46) And the one piece I would add to that: with the proliferation of models, you’ll see, increasingly, what I like to call synthetic secondary sources. With the proliferation of models, and folks having their preferred work product — really, what some of these firms have spent decades upon decades tuning and building as their particular approach to the law — I think you’ll have tooling and models emerge where the analysis provided to a particular lawyer is so customized, where it is the specific model and specific data source coming together. If the firm name is, say, Ryan and John LLP, it is a technological marriage of our exact corpus with our exact preferred models to get our exact answer, at a scale that’s just not been achievable, because the knowledge and the style and the preferences were locked in the brains of the partner or maybe the senior associates. A crazy ability to get ultra-custom legal outputs for users.
Ryan Anderson (53:40) I think you’re maybe not even going far enough — it may not even be at the firm level. It might be the lawyer level, and it might even be the lawyer paired with the client. You could envision a world in which you say, “For this client, we prefer this model.” I can see that very easily being the case. We already orchestrate through multiple models just to give you a certain response to a query. So it’s going to be a world where you see a lot of models, they’re going to be used in a lot of different ways, and they’re going to become more and more specialized. It’s very exciting. Thank you both so much for the time today.
Greg Lambert (54:22) Yeah, you got it. John and Ryan, thank you very much. Appreciate it.
Marlene Gebauer (54:26) Yeah, and thanks to all of you for listening to The Geek in Review. If you’ve enjoyed the show, please share it with a colleague. We’d love to hear from you on LinkedIn and Substack.
Greg Lambert (54:35) And real quick — if they want to learn more, where’s the best place to reach out?
Ryan Anderson (54:40) Probably just hit me up on Twitter — DM me on Twitter. Ryan Filevine is my username on Twitter, and you can find me there. I guess we call it X now, right? So it’s X.
Marlene Gebauer (54:48) And as always, the music you hear is from Jerry David DeCicca. Thank you, Jerry, and goodbye, everybody.