Reading Time: 9 minutes
I was able to participate recently in a professional event involving lots of smart librarians discussing AI and law libraries. At the end, one outcome I committed to was to list what my law library was doing and how AI might impact those tasks. Here’s how that progressed.
Sometimes saying something out loud to a group or telling a friend is a way to get yourself to act. I know that no one is really going to follow up with me on this but I tried to think how I would do it as I traveled back to San Diego. Also, volunteering something at the event made me extremely uncomfortable so doing something felt like I was relieving that discomfort a bit.
I really don’t know how AI is going to affect our operations. In reality, like automation, which I think is the closest comparison for libraries when it comes to operational change, it will be creeping forward, landing in some parts of our operations but ignored in others.
One thing that was clear from the discussions: we are not planning for the future any longer. It just may not have trickled down into the crevices of libraries equally. We will still be able to opt-in (or not) in some cases but in others, AI will either be available or not without our input or agency.
Making A List
As a director, I have both the obligation to have a strategy for my organization but also the time and bird’s eye view of our operations to initiate this process. At some point, though, I will need my staff to contribute time and resources to ensure we have really captured a complete picture. But I like to rough out ideas a bit, if only to know what I’m trying to talk about.
So, while the list idea was really my “thing I’ll do as soon as I return to the office” take away action item, it was in fact my second action on my return. The first was to give my folks an update on the meeting, an overview of what I’d been doing while I was away, and to plant some seeds for possible future activity on our Teams channel. Then I started on my list.
There are a lot of ways to approach this, I think. I merged two approaches. I thought about what we do in a typical week and started with the very first action on a Monday morning: our opening process. I supplemented this chronological approach with staff job descriptions.
Job descriptions are a great resource for this sort of project too. In theory, they are reviewed at least annually during the performance review. They are agreed to by employer and employee. They should reflect what everyone agrees are the priorities of that employee’s role. They should provide a detailed, but not granular, list of what busy people do all day.
We had just gone through everyone’s job descriptions last year, rewriting most of them to reflect clearer priorities and normalize some of the language over the organization. The job descriptions set out what it is that our people are supposed to be doing and that we have documented and that we are measuring their performance against. All of those items were dropped into the list too.
With these two inputs—a chronology of our work, a documented list of each job’s priorities—it was pretty easy, actually. We have a lot of moving parts but we’re not a huge organization, with just under 20 people. I know a lot about our processes because, in addition to working in libraries for decades, I’m responsible for our administrative processes: finance, payroll, facilities, HR. I look at a lot of internal data too, so I know how a lot of our technical and public services processes work.
I was able to build a list of 75 items relatively quickly. As I selected each one, I tried to make it as generic as possible. For example, some of our public services staff create videos for online tutorials that go on our YouTube channel. But some of our IT and technical services staff create videos for internal tutorials. If we are looking at the impact of artiificial intelligence, it probably isn’t going to make a diistinction between those sorts of content since the underlying process is the same even if the expert input is different. With staff input, I’ll get a better sense of distinctions that might be important.
For those of you just interested in the list, feel free to grab a copy from my OneDrive and re-purpose it or download from the embed below. Your law library’s choices won’t necessarily be the same as mine (or as our law library might eventually land, since this is draft 0 and just my own reflections) and I’m sure you can identify more activities tailored to your environment.
Checking It Twice
We talk about artificial intelligence as if it will change things but AI is really just the latest potential impact. When I think of AI and library operations, it strikes me as just the furthest end of automation. In theory, it adds a learning component to systems that can perform repetitive tasks in lieu of a person. As I was building my list, I added two columns: automation and AI. If I am going to do an assessment for AI, I might as well consider the entire spectrum between 100% human mediation and 0% human mediation. Also, if we have already automated something, then it may be further along or more amenable to whatever artificial intelligence may offer.
I am thinking of AI as a 100% human replacement. If it isn’t able to do that, and automation is possible, than AI doesn’t offer any new value for me. If there still needs to be a human validation step—as lawyers are seeing with legal research—then the value of going from a rote tool like automation to an intelligent tool like AI is rapidly diminished.
This was interesting because, as I went along, there were some tasks that could clearly be automated. What I mean is that, technically, there is a way to automate that step of a process. It was possible. There were also tasks that I thought could partially be automated but would absolutely still need a human involved. Think of book self-check-in. Sure, a patron can scan the book to record the book as being returned. In most cases, though, a human is involved in replacing the book on the shelf or confirming it was received or reviewing that its contents are intact with pages in order and CDs in envelopes.
AI and automation are different though. Automation is just shifting something a human can do manually and discretely and allow you to do it more quickly: merge a list of addresses with a letter, dial phone numbers and only connect the human when someone picks up. Additional functionality can cause that activity to happen on its own cycle. But it will merely follow the rules it has been given.
AI has “intelligence” or at least “learning” baked in. It goes beyond the rote and should provide enhanced, growing ability. It will see the rules and, one would hope, adapt for glitches in the matrix. If it doesn’t, it’s not any different than automation. It would still be reactive (seeing a need for change within the context and rules given) rather than proactive (deciding to extend the context and add rules on its own).
For example, I can automate a mailing list. I drop in the content, and it then mails it on the schedule I set and to the people who are subscribers. I don’t send or address each email. But I don’t want AI broadening that list of subscribers or sending new, unedited or unapproved content based on, say, email responses or other inputs it gathers.
The difficulty with AI is that it doesn’t really exist for a library when we leave the world of generating content (text, images, sound). It’s hard to know if AI can help me run my building more efficiently. Automation allows me to automatically turn on and off the lights based on a 9 to 5, Monday to Friday schedule. AI would, in theory, learn that, in fact, staff come in at 8:30 most Mondays but 8:45 most Fridays, that Prima stays until 6 on every third Thursday to pick up ice cream after work and Secundo sometimes works on Saturdays because there’s no one to bother them and they can get more work done. Is that possible? I don’t know. But I’m guessing the technology should be able to do that. And then adapt again when the weather gets cold and Prima doesn’t want ice cream any longer.
But I got stuck at this point and I need to think this through more. My automation and AI columns had five responses: yes, maybe, no, possible, partial. In this case, possible meant that it could be entirely automated or given over to AI because, technically, there was no reason for a human to be involved. Self-checkout is a great example of this. We don’t do self-checkout but it is possible. So that is a possible for us, not a no. We could buy self-checkout kiosks and so on. But I think that’s not granular enough to plan on. I think there is a more finely crafted spectrum between yes and no to craft.
One of the things that occurred to me as I was building the list was not to get too granular. At some point, you won’t automate or use AI in a process for just one step unless the automation or AI is built into the process tools. Otherwise, going to and fro between what’s automated and what’s not eliminates the productivity bump that AI or automation gives.
In most cases, tasks were the same across both columns. But not always. When we close a patron’s account, we can partially automate that closure. If an account is unused after 5 years then delete, that sort of thing. We use rules all the time in our email and other systems. We have compliance rules (how long to keep tax documents, how long to retain public records, etc.). We often keep a manual check though. We don’t want something like Google deleting both the primary and backup copies of all of our 650,000 accounts.
During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.
Sharing details on a recent incident impacting one of our customers (where we deleted their account and all their backups), Google, May 24, 2024
Don’t automate your destruction.
But perhaps AI offers the ability to learn patterns so that, in fact, we could complete all account deletions without a human intervening. We could shift from ILS-based rules where, after 3 years in a 3-year law school, a law student is deleted or converted to an alumni account, and get additional inputs that some students are staying for 4 years, for example, and have the system adjust to that.
What about serials claims? We have rules to let us know when things should arrive so we can claim them when they don’t. What if legal publisher X is always late in January because they close for 3 weeks in December or our mail room does, throwing off the aspirational publishing schedule? We may adapt our rule to meet that unwritten variance. Some of our systems already support automated claims (not every law library has an ILS).
How much of that automation is still intermediated by our systems librarians? What percentage of that remainder can be delegated to automation or AI? I could envision a point at which rule variations are anticipated based on prior experience without having to change a rule, though.
Who Is Naughty or Nice?
Let’s emphasize this point though. Automation and artificial intelligence create the technical potential to do some things. As we are seeing with lawyers, AI, and legal research though, just because you can, doesn’t mean you should. Librarianship is considered a profession because of the human intelligence involved. We may not suffer Rule 11 sanctions or bar discipline for relying on AI that is itself not 100% reliable. But we can still negatively impact those who rely on our systems and services. I don’t want any AI-based systems automatically processing invoices in case the invoice is fraudulent, and how would the AI know?
No law library is an island. We have free space from our regional government. We may want AI-infused lighting or heating but it’s not really our choice. No matter what we put on our list, and our assessment of its susceptibility for automation or AI, we will have infrastructure and political limits on our operational perimeter.
When we talk about AI, there is also this default assumption that it has 100% accuracy. We know that’s not true but it is often not part of the assessment. If we use an AI tool, what is its error rate? Are we better off with automation that we know will be 100% accurate—the lights will ALWAYS turn on at 9am and off at 5pm—because it is using rules we have designed and tested and enveloped in a system that allows us to catch anomalies?
How does 65% sound? 42%?
We also document substantial variation in system performance. LexisNexis’s Lexis+ AI is the highest-performing system we test, answering 65% of our queries accurately. Westlaw’s AI-Assisted Research is accurate 42% of the time, but hallucinates nearly twice as often as the other legal tools we test. And Thomson Reuters’s Ask Practical Law AI provides incomplete answers (refusals or ungrounded responses; see Section 4.3) on more than 60% of our queries, the highest rate among the systems we tested.
Hallucination-Free? Assessing the Reliability of
Leading AI Legal Research Tools, Varun Magesh, Faiz Surani, Matthew Dahl, Mirac Suzgun, Christopher D. Manning, Daniel Ho
I was working on an Excel spreadsheet recently. It was designed maybe 5 years ago and has been working fine. The calculations were simple =sum() formulas, nothing fancy. But it had been passed from one person to another and had accreted new information. Over time, the automation of the formulas broke down because the input data was changing. Over time, the spreadsheet outputs were no longer reliable.
We found the problems because we saw that the document was no longer able to reconcile with an external check. Two cells that should have been equal, validating each other, started to fail. The automation worked but the context had changed. If artificial intelligence was brought in, it would ideally flag those anomalies.
As I was rebuilding the spreadsheet, I saw that Excel had started to flag boxes. These are common if you create a formula that skips boxes (“Formula omits adjacent cells”). It’s letting you know that you’ve missed something, which may have been intentional or not.
In my case, I was putting in values that were inconsistent in size, and Excel warned me that something didn’t look right. It was pattern recognition. I suppressed the warnings—because the data was correct—but I was interested to see that it would flag them. It was not smart enough, though, to see that I was repeating the “error” and that, perhaps, it wasn’t actually an error. It just looked and saw a group of As and a group of Bs and flagged all of the Bs.
Anything on my list that includes a human step like this will remain only partially automated or a possible benefactor of AI. In fact, the closer it gets to high value activities—paying staff, giving legal advice, putting a book in the right place on a shelf—the less likely I think we will move from the technically possible to implementation. Even humans aren’t 100% but we can take responsibility for our failings. Technology just fails without accountability.