Following my previous
post, I am again looking at how to employ R and Python seamlessly to
use large language models (LLMs). Last time, I scraped information off
of Wikipedia using the rvest package, fed that information
to OpenAI’s Python API, and asked it to extract information for me.
But what if we could skip that scraping step? What if we had a more
complex question where writing an rvest or
RSelenium script was not feasible?
Enter the LangChain
Python library. I recently read Generative
AI with LangChain and Developing
Apps with GPT-4 and ChatGPT, both of which do a fabulous job of
introducing LangChain’s capabilities.
I’ve been thinking of LangChain as an LLM version of scikit-learn: It
is a model-agnostic framework where you can build LLM pipelines. Most
relevant to our needs here, though, is that you can employ tools
in these pipelines. Tools allow the LLM to rely on integrations to
answer prompts. One tool is Wikipedia, which allows the LLM to search
and read Wikipedia in trying to answer the question it’s been given.
This is especially useful if you want to ask it information about
something that happened after it was trained.
I’m continuing to use my Best Picture model as
the project, using LLMs to get more features for me to add to it. This
means I’m mostly using LLMs as information extractors instead
of information generators. Does this really map one-to-one with
what something like GPT 3.5 Turbo was meant to do? I truly don’t know. I
don’t think many people know what the hell these things can and should
be used for. Which is why I’m testing it and reporting out the accuracy
here!
Question
Past Lives (my favorite
movie of last year) was nominated for Best Picture, even though it
was director Celine Song’s debut feature-length film. This is rare, and
I think a helpful feature for my Best Picture model would be how many
films the director had directed before the nominated film.
Getting this information is a bit more complicated. It would involve
an RSelenium script of going to the movie’s page, finding
the director, clicking on their profile, and then either pulling down
filmography information from there or by clicking into their filmography
page. Pages aren’t formatted the same, either. Sometimes the section is
“Filmography,” sometimes it is “Works,” sometimes the information is in
a table, while other times it is in a bulleted list.
The idea here is to use LangChain to give an LLM the
Wikipedia tool to find this information for me. As opposed to my last
post, I am not giving it the relevant slice of info from Wikipedia
anymore. Instead, I asking it a question and giving it Wikipedia as a
tool to use itself.
Methodology
I took the following steps:
Wrote a prompt that asks an LLM to figure out how many
feature-length films the director of a movie had directed
before making a film. The prompt is a template that takes a
film’s name and release year.
Give the LLM the Wikipedia tool and a calculator (my thinking was
it might need this to sum up the number of movies, since these models
are optimized on language, not math).
Collect and clean the responses for every movie that’s been
nominated for Best Picture.
Test the accuracy by hand-checking 100 of these cases.
The Prompt and Function
I started by making a file named funs.py:
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents import load_tools, AgentExecutor, create_openai_tools_agent
from re import sub
model = ChatOpenAI(
model="gpt-3.5-turbo",
temperature=0,
openai_api_key="API_KEY_GOES_HERE"
)
tools = load_tools(["wikipedia", "llm-math"], llm=model)
prompt = ChatPromptTemplate.from_messages(
[
("system", """You are a helpful research assistant, and your job is to
retrieve information about movies and movie directors. Think
through the questions you are asked step-by-step."""),
("human", """How many feature films did the director of the {year} movie
{name} direct before they made {name}? First, find who
directed {name}. Then, look at their filmography. Find all the
feature-length movies they directed before {name}. Only
consider movies that were released. Lastly, count up how many
movies this is. Think step-by-step and write them down. When
you have an answer, say, 'The number is: ' and give the
answer."""),
MessagesPlaceholder("agent_scratchpad")
]
)
agent = create_openai_tools_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, max_iterations=6)
def prior_films(year, film):
resp = agent_executor.invoke({"year": year, "name": film})
return(resp['output'])
It turns out that many of the functions in the two books above,
despite being Published in October and December of 2023, have been
deprecated (but the author of Developing Apps has published updated code and
a second version of the book is being released this year). This is a
good reminder of how quickly this field is moving, and it’s OK to not be
entirely sure how to use these models—so long as you have appropriate
respect for that lack of knowledge. This is to say: I’m no expert here.
What I have above is cobbled together from LangChain docs, StackOverflow
posts, and GitHub threads. The prompt here lays out the basic steps it
should be following to get the information.
Bringing It To R
I wrote an R script to use this function in an R session. Why?
Because I think the tidyverse makes it easier to inspect,
clean, and wrangle data than anything in Python currently.
We start off by loading the R packages, sourcing the R script,
activating the Python virtual environment (the path is relative to my
file structure in my drive), and sourcing the Python script. I read in
the data from a Google Sheet of mine and do one step of cleaning, as the
read_sheet() function was bringing the title variable in as
a list of lists instead of a character vector. I then initialize a new
column, resp, where I will collect the responses from the
LLM.
library(googlesheets4)
library(tidyverse)
library(reticulate)
use_virtualenv("../../")
source_python("funs.py")
dat <- read_sheet("SHEET_ID_GOES_HERE") %>%
mutate(film = as.character(film)) %>%
select(year, film) %>%
mutate(resp = NA)
I iterate through each film using a for loop. This is
not the R way, I know, but if something snags, I want to catch the
response that I’m paying for. You’ll see that I take extra precautions
to catch everything by writing out the results in .csv row-by-row. (My
solution because I had been running this script and got an aborted R
session in the middle and lost everything.)
for (r in 1:nrow(dat)) {
cat("prompting", r, "\n")
dat$resp[r] <- str_replace_all(
prior_films(dat$year[r], dat$film[r]),
fixed("\n"),
" "
)
cat("writing", r, "\n")
if (r == 1) {
write_csv(dat[r, ], "prior_films.csv")
} else {
write_csv(dat[r, ], "prior_films.csv", append = TRUE)
}
Sys.sleep(runif(1, 2, 10))
}
As I said in my previous post, this is an example of how we can use R
and Python together in harmony. prior_films is a Python
function, but we use it inside of an R script.
An example call that returns the correct information:
> prior_films(2017, "phantom thread")
[1] "The director of the movie \"Phantom Thread\" is Paul Thomas Anderson. Before directing \"Phantom Thread\" in 2017, Paul Thomas Anderson directed the following feature-length movies that were released:\n\n1. Hard Eight (1996)\n2. Boogie Nights (1997)\n3. Magnolia (1999)\n4. Punch-Drunk Love (2002)\n5. There Will Be Blood (2007)\n6. The Master (2012)\n7. Inherent Vice (2014)\n\nThe number of feature films that Paul Thomas Anderson directed before making \"Phantom Thread\" in 2017 is 7."
I then pulled in this .csv to a separate cleaning script. Since I
asked it for standardized feedback, I could use a regular expression to
clean most of the responses. I read the rest myself. This went into a
new column called resp_clean, with only the integer
representing the number of movies the director had directed before the
movie in question.
library(tidyverse)
dat <- read_csv("prior_films.csv")
dat <- dat %>%
mutate(
resp_clean = case_when(
str_detect(tolower(resp), "the number(.*)is") ~
str_split_i(tolower(resp), "the number(.*)is", 2)
),
resp_clean = str_remove_all(resp_clean, "[^0-9]")
)
# manual code the rest
lgl <- !is.na(dat$resp) &
dat$resp != "Agent stopped due to max iterations." &
is.na(dat$resp_clean)
dat$resp[lgl] # read with mine own human eyes
dat$resp_clean[lgl] <- c(NA, NA, 44, NA, NA,
NA, NA, NA, NA, NA,
4, NA, NA, 86, NA,
9, NA, 4, 33, 26)
Oof, some huge misses. Not great performance from the LLM here. Most
of these huge misses are due to early filmmakers and the studio system,
which would churn out massive amounts of films, especially during the
silent era. So, let’s look at absolute error by year.
Error by Year
dat <- dat %>%
mutate(abs_err = abs(resp_clean - check))
dat %>%
summarise(mae = mean(abs_err))
## # A tibble: 1 × 1
## mae
## <dbl>
## 1 9.7
dat %>%
mutate(err_disc = case_when(
(check - resp_clean) > 0 ~ "Undercount",
(check - resp_clean) < 0 ~ "Overcount",
(check - resp_clean) == 0 ~ "Correct"
)) %>%
count(err_disc)
## # A tibble: 3 × 2
## err_disc n
## <chr> <int>
## 1 Correct 28
## 2 Overcount 15
## 3 Undercount 57
We get a mean absolute error of nearly ten films. We also
see that the model gave us the correct answer 28% of the time, an
undercount 57%, and an overcount 15%.
When we plot absolute error against release year, we can see the poor
performance is driven by earlier films:
ggplot(dat, aes(x = year, y = abs_err)) +
geom_abline(aes(intercept = 0, slope = 1), linetype = "dotted") +
geom_point(alpha = .5) +
geom_smooth(method = "loess", se = FALSE, span = .95, color = "purple") +
geom_smooth(method = "lm", se = FALSE, color = "forestgreen") +
theme_light() +
labs(x = "Year", y = "Absolute Error")
So how about we remove the studio-era films, since that wouldn’t be a
good input into a model trying to predict Best Picture next year
anyways? The cleanest cutoff I could think of is 1970 and later, since
RKO
closed in 1969:
dat %>%
filter(year > 1969) %>% # year RKO closed
summarise(mae = mean(abs_err))
## # A tibble: 1 × 1
## mae
## <dbl>
## 1 1.52
We’re still off by 1.5 films, which is still more error than I’d like
to include in my model. (Spoiler: I won’t be using the data generated
here for my Best Picture model.)
Problematic Films and Directors
Let’s look at which films were the biggest misses, with an error of
more than thirty films.
ggplot(dat, aes(x = resp_clean, y = check)) +
geom_abline(aes(intercept = 0, slope = 1), linetype = "dotted") +
geom_point(alpha = .5) +
geom_smooth(method = "loess", se = FALSE, span = .95, color = "purple") +
geom_smooth(method = "lm", se = FALSE, color = "forestgreen") +
theme_light() +
labs(x = "Hand-Checked", y = "GPT 3.5 Turbo") +
ggrepel::geom_text_repel(aes(label = ifelse(abs_err > 30, film, "")))
The directors in question here are W.S. Van Dyke, Michael Curtiz,
John Ford, William Wyler, and Edmund Goulding. I would invite you to
visit their Wikipedia pages and try to make sense of their filmography
sections; it’s a lot. John Ford’s page, for example, lists all of the
informational “short films” he made with the military, including Sex
Hygiene and How
to Operate Behind Enemy Lines. These pages were hard for me to
hand-code according to the prompt.
Overcounts
Lastly, let’s examine overcounts. It makes more intuitive sense to
get an undercount: The model didn’t pick up on films the person already
directed—it missed them. But an overcount is stranger: How does that
happen? A few examples:
> prior_films(2009, "district 9")
[1] "The feature-length movies directed by Neill Blomkamp before he made \"District 9\" in 2009 are:\n1. \"Chappie\" (2015)\n\nThe number of feature films Neill Blomkamp directed before making \"District 9\" is 1."
> prior_films(1999, "american beauty")
[1] "Before directing the movie \"American Beauty\" in 1999, Sam Mendes directed the following feature-length movies:\n1. American Beauty (1999)\n2. Road to Perdition (2002)\n3. Jarhead (2005)\n4. Revolutionary Road (2008)\n\nThe number is: 4"
We can see it didn’t make up new movies. Instead, listed movies that
came out after the movie in question. This is a great example
of how LLMs are trained on language and not mathematical reasoning. It
doesn’t understand the temporal sequence here of 2008 being after 1999
and thus couldn’t be before American Beauty.
Conclusion
A few takeaways:
Again, we can see R and Python work together seamlessly.
LangChain provides an LLM with tools, but this comes at greater
cost; GPT 4 probably would have done much better here, but it would have
cost much more money in context tokens.
Prompt engineering is important: I could have been more explicit
in the language Wikipedia tends to use, I could have asked it to use a
calculator to check that the years didn’t difference out to below zero
(e.g., 1999 - 2008 < 0), and I could have asked it to ignore silent
films (even though the first Best Picture winner had no
dialogue).
Domain expertise remains huge in data science: I think I’m pretty
knowledgeable about film, but I don’t know the silent era. I wasn’t
aware how many of the early directors had many dozens of silent films. I
didn’t know about quota
quickies. Understanding domain knowledge is vital for a data
scientist.
KEEP HUMANS IN THE LOOP. What I did here is kept
myself in the loop by checking performance against 100 hand-coded
examples. This is a very nascent field using technology that has only
been available to the public for a few years. Keep humans in the loop to
make sure things don’t go off-track. For example, I won’t be adding
these data to my model due to the error being too great.
