
> Note: This is a crosspost from [the ZenSearch blog](https://zensearch.jobs/blog/2024-08-19-lessons-learned-from-building-zensearch-part-ii-running-async-tasks?utm_source=matt_personal_site&utm_medium=referral)

## Introduction

In [the last post in this series](/blog/building-zensearch-part-i), I wrote in a little more detail about why I began building ZenSearch. In this post, I'll talk about how we handle async work, and how the async task processing stack has evolved over time as it approached its first "stable" state. Future posts will talk about how the backend and frontend have evolved over time, and decisions I wish I'd made from the get-go.

## The Initial Stack

The initial stack backing ZenSearch was basically:

1. A Flask server, running on a [DigitalOcean App Platform box](https://www.digitalocean.com/products/app-platform) for about $5 / month.
2. A [DigitalOcean-hosted Postgres database](https://www.digitalocean.com/products/managed-databases-postgresql), for about $12 / month.

And that was pretty much it. Flask would serve some rendered HTML templates and process form requests (think someone updating which companies they want to see jobs at in a basically unstyled HTML dropdown), and then we'd use [flask-apscheduler](https://pypi.org/project/Flask-APScheduler/) to run the jobs that collected and processed postings, and that sent the notification emails. The whole app costed less than $20 / month.

## Scaling

At the time, we only had about 20 companies in the database, and they were ones that I was interested in (the app was initially for me, after all). This meant that running posting collection and parsing in a background process on the same App Platform app that was running the web server worked fine. If we, for example, ran out of memory, the box would crash, the server would restart, and all would be well after a few minutes of downtime. This was fine for me - I didn't care if the app was down periodically as long as I got my notification emails.

This changed as we started getting more users, and, most importantly, adding new companies to the database. Over the past year or so, we've gone from 20 companies in the database to about 10,000, and we now have about 1.5 million postings (~300k open) as well.

For 20 companies, running posting ingestion tasks in a background process was fine. If each one took a minute or so, we'd need 20 minutes of background process time, and all was well. Once we hit a few hundred companies, this began to change quickly. It became apparent that we needed dedicated tooling for processing background tasks.

For that, we reached for [Celery](https://docs.celeryq.dev/en/stable/getting-started/introduction.html).

## Celery

We introduced Celery (and Redis, to use as a broker and a backend) last December and farmed our async task processing out to it instead of running it on the main app webserver. This meant we needed to spin up three new services in DigitalOcean: One for Celery itself, one for [Beat](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html), which is essentially a cron scheduler for Celery, and a Redis instance to serve as the broker and backend for Celery.

This new async task processing rig more than tripled our costs overnight - we were now paying something like $10 / month for Celery, $5 for Beat, and $30 for Redis.

The benefit, though, was that introducing Celery let us scale up significantly. All of a sudden, we were capable of processing jobs for hundreds (soon to be thousands) of companies without needing to worry about taking down our main webserver. We could also scale Celery independently of the main server, which has been beneficial to us over time, since today we're processing tens of thousands of async tasks per day.

## Growing Pains

Over time, we ran into more headaches. As we went from a few hundred companies to a few thousand, we started hitting a few issues we could've foreseen, but decided to solve when we eventually ran into them. First, having thousands of companies in the database means thousands of tasks to ingest job postings. For reasons beyond the scope of this post, these ingestion tasks can take anywhere from fifteen seconds to a few minutes, depending on the company. The distribution is heavily right-skewed (it has a long tail), but the p50 runtime for a task to ingest new postings into the database might've taken 30 seconds. If you do the back-of-the-napkin math, you quickly realize it'll only take about 3,000 companies in the database (one task per company per day) before we'd run out of time in the day trying to process each company sequentially.

At the same time, allowing for concurrency gets expensive too. If we try to scale up Celery to run more concurrent processes, we start hitting memory and CPU caps imposed by DigitalOcean.

The initial solution to this was to scale up the size of the box we were running Celery on, but it became clear pretty quickly that this wasn't a solution that would work over the longer term. We'd likely be paying over $100 / month, which is pennies to a corporation, but wasn't something I wanted to fund out of pocket.

## Migrating Celery to ECS

At this point, it started to become clear that the better strategy for running Celery would be to add a couple of cost-saving (and sometimes performance-improving) measures.

First, we migrated off of DigitalOcean to [Amazon ECS](https://aws.amazon.com/ecs/). There were a few reasons for this, but the main one was that ECS allows for autoscaling based on the depth of the broker queue, meaning we could run our posting ingestion tasks, scale up arbitrarily, and then shut off our instances once we'd processed all the tasks. This allows us to run our tasks on bigger boxes (with more concurrency) without needing to keep the boxes on all the time.

The other major benefit of moving to ECS is the ability to use [Fargate Spot](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/fargate-capacity-providers.html) instances, which often cost less than half as much as on-demand ones. Running spot instances has saved us a significant amount of money.

At the same time, we migrated from Redis to RabbitMQ as the broker for Celery, since we really only needed a message broker and Rabbit was far cheaper, and also gave us a nice management UI with which we could monitor tasks being picked up and completed.

All told, running Celery today costs us about $15 / month today.

## Celery Headaches and Lessons Learned

There have been a few issues I've run into and spent a fair amount of time debugging with Celery that made me learn some lessons the hard way, so to speak.

The first is that in general, it's beneficial to make tasks small, idempotent pieces of work whenever possible. For example, instead of one task that loops through 10,000 companies and collects postings at each of them, write a task that does it for each _individual_ company, and then trigger each of them. Long-running tasks can cause problems with retries, workers dying, restarts from failure, and so on.

For instance, Celery by default will acknowledge (ack) a task when it's picked up from the broker, so if you have a long-running task that dies halfway through, Celery will generally think it's completed.

Another footgun was that with SQS, there's a broker visibility timeout that you (the Celery user) need to set. For instance, you can set this timeout to 30 seconds, which means that if you have a Celery worker pick up a task and you (e.g.) have `task_acks_late=True` set, then if that task does not complete after 30 seconds, it will be re-enqueued and re-processed. This means that a long-running, non-idempotent task could be retried over and over, which might (hypothetically) result in a result like ten emails going out to a single user one morning instead of just one.

In general, writing tasks that can be rerun and finish quickly is a good way to combat some of these issues. We've made heavy use of techniques like fanout (where a single task enqueues a number of other tasks concurrently) and pagination (also sometimes known as batching, where a single task re-enqueues itself but with a "next page" pointer, such as incrementing an offset), in order to process large numbers of what would otherwise be long-running tasks individually with no loss of performance. This is extremely useful for things like indexing a large number of records into OpenSearch (more on this in a future post), where paginating through the records in each task is a great way to keep the individual task run times down, or using fanout to run a large number of posting ingestion tasks concurrently while keeping one process responsible for one task (company, in this case) at a time.

The last issue we've repeatedly had issues with is a [known issue](https://docs.hatchet.run/blog/problems-with-celery) [among Celery users](https://steve.dignam.xyz/2023/05/20/many-problems-with-celery/), which is Celery's lack of `async` support. For our tasks, we would (if we could) make heavy use of `async / await`, since so much of what we're doing are I/O bound operations (collecting job postings). However, since Celery does not support async, we instead need to use Celery's [concurrency](https://docs.celeryq.dev/en/stable/userguide/workers.html#concurrency) mechanisms to improve the performance of the workers. Largely, this has worked fine, but we did run into some issues periodically when trying to use the `gevent` pool where we'd get errors like this:

`Couldn't ack 8360, reason:"SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2423)')"`

This would cause our Celery workers to hang without actually crashing, which would mean no tasks would be processed until we restarted the worker. As a workaround, we switched to using the `threads` pool instead of `gevent`.

## Doing It Again

If I could do it again, I'd definitely make some choices differently early on. With the experience I have now, I would've reached straight for Celery instead of using the APScheduler. The APScheduler worked well and was easy to set up with the knowledge I had at the time, but knowing what I know now, Celery is very straightforward to orchestrate (at least when you're only using a few workers, have a single broker, are running on a managed service like ECS, etc.), and it would've saved a lot of refactoring to be able to go in that direction right away.

There have been also a few side quests along the way that I probably wouldn't repeat, such as using Redis and then SQS as the brokers for Celery before ultimately deciding on Rabbit.

One other viable option that I never considered would have been doing something even simpler, and using a tool like [PGQueuer](https://pgqueuer.readthedocs.io/en/latest/dashboard.html) early on to keep the stack as simple as possible. Since we already rely heavily on Postgres, this would've required no new infrastructure for setting up a broker, and since our needs are pretty straightforward (basically just running cron jobs periodically), it likely would've worked just fine.

## Up Next

In the next post, I'll discuss some things I've learned over time about making frontend code more friendly to work with, and choices I wish I'd made earlier on. Stay tuned!


Lessons Learned from Building ZenSearch, Part II: Async Tasks


> Note: This is a crosspost from [the ZenSearch blog](https://zensearch.jobs/blog/2024-08-14-lessons-learned-building-zensearch-part-1?utm_source=matt_personal_site&utm_medium=referral)

# Introduction

I started building ZenSearch about a year and a half ago, which is hard to believe. In hindsight, I wish I'd taken more screenshots - the app was nothing like it is now at the time.

This post is going to be the first of a few talking about things I've learned along the way. I've grown more than I can imagine as an engineer (and as a product manager, designer, and _founder_) over the past 17 months, and I've learned a lot of valuable lessons in that time.

## Why Zen?

I started building Zen in April 2023 when I was casually looking for a job. At the time, I was working for a small EdTech startup called [CollegeVine](https://www.collegevine.com/), and had been there about three years. I wasn't in a rush to leave, but it was starting to feel like a good time for me to start thinking about looking for opportunities to work on new types of problems and experience a new place.

So I figured I might try passively job hunting, which basically translates to "wait and see if something I'd be very interested in comes up." I had a shortlist of companies - many in Boston, some fully remote - that I thought I'd be interested in working at, based on my interests and their products or services. Some of them were the likes of Klaviyo (where I am now), Strava, AllTrails, and a few others.

The problem, though, was that I didn't have a good way to find out when those companies posted a new job that was relevant to me without just checking their job boards every day, which I didn't want to do. Even at the simplest level: I was looking for data scientist jobs at those companies. But I wasn't aware of any sites or tools (think LinkedIn, etc.) that would notify me when _those specific companies_ posted new jobs that I'd be interested in _without spamming me with promoted content, etc._

Out of this, ZenSearch was born.

## Building a Prototype

My initial pass at what's now ZenSearch was simple - basically as simple as possible. I reached into my then data scientist's toolbox and pulled out Selenium and cron, and deployed an app on DigitalOcean for $5 / month that ran a [Flask](https://flask.palletsprojects.com/en/3.0.x/) backend with [APScheduler](https://apscheduler.readthedocs.io/en/3.x/#) for scheduling tasks, and a Postgres database for another $12 for storing companies and postings. I created a couple tables for said companies and postings, and went off writing some scraping logic to collect and parse job postings.

And it worked great! It did basically exactly what I wanted - I had an extremely rudimentary way to add new companies to the database (manually), and the scraping job would run once a day and collect each company's jobs.

Once that was shipped, I started receiving emails (on no particular cadence). I'd run my background tasks to collect postings once a day, and, eventually, I got emailed about [this one at Klaviyo](https://zensearch.jobs/postings/cab3e9fb-dc39-4be8-a132-970c1ea08281), which I ultimately accepted.

Largely because of Zen, I only applied to a few companies, interviewed with three (a small health tech startup, Whoop, and Klaviyo), and ended up accepting an offer from Klaviyo.

In short, my job search was not what I had been used to in the past. Instead of unleashing a flood of applications, I let the jobs flow to me, mindfully applied to a few I was particularly interested in, and landed one.

## The Initial Stack

As I mentioned, the app was initially built in Flask. It was (and still is!) backed by a Postgres database, served some Jinja-templated HTML from the Flask backend, and that was about it. There was a tiny bit of interactivity to allow a user to mark a company as one they were interested in and to log in and log out, but it was extremely barebones. At that point, the site wasn't even responsive, meaning that if I wanted to view it on my phone, I'd need to click `Request Desktop Site` in my browser's settings.

The database and the web server both lived on DigitalOcean, which made up the extent of the infrastructure. All in all, it was $17 / month. At this point, we had about 20 companies in the database, a few hundred postings, and one user.

Then, I got addicted.

## What's Next?

The next post in this series will talk about some of the early ways ZenSearch evolved, including adding React, migrating to FastAPI, and adding too many new features to support, all far too quickly. I'll also address some early decisions that I would make differently if I could do it again.


Lessons Learned from Building ZenSearch, Part I: Backstory


## Introduction

I've been working on [ZenSearch](https://zensearch.jobs) for the past six months or so, and I'm proud of where it's at and thought it might be a good time to share it with all of you (cough, potential beta users). This post will be a brief intro to what the app does, why I built it, and some roadmap items for the future. As a disclaimer: I'm still _not_ an engineer (especially on the front end), so please don't read through my largely GPT-generated React code with too critical of an eye.

## What's Zen?

Zen is an app that I started building when I was passively job hunting from about April until June. I had a problem that I didn't know of a good solution to: I had a shortlist of companies I thought I might be interested in working for, and I found myself needing to repeatedly check their job boards very often to see if they posted a new job that matched my interests (mostly data science and ML engineering).

This was a painful, time-consuming, and, largely, easily automatable process. So I built Zen to automate it, by fetching new jobs from company boards each day and sending me an email containing ones that were relevant to me.

The app has evolved pretty significantly since then. Now, the app has a handful of core features:

1. You can [view the companies in its database](https://zensearch.jobs/companies) and request new ones to be added. We have about 2,000 companies in the database.
2. You can [set up email and search preferences](https://zensearch.jobs/jobs), which will let you filter jobs to relevant ones for you, and will trigger emails to be sent to you when a company you're interested in posts a job that matches your search query.
3. You can [peruse the job boards of all companies in the database](https://zensearch.jobs/jobs) or of [a single company](https://zensearch.jobs/companies/Airbnb) and filter for postings matching your interests. You can also filter by only postings posted in the past day, three days, week, etc.
4. You can get [job recommendations](https://zensearch.jobs/jobs) for new jobs you might not have seen yet that match your interests.

## Why Should I Use It?

Basically for the reason it says on the tin: It's a no-bullshit job searching app. I hate LinkedIn, and I largely built this so I could stay off of LinkedIn and sites like it. On Zen, there aren't any ads, there's no recruiter spam, there aren't any stale postings that aren't actually open anymore. It just works directly with company job boards and gets rid of the intermediary.

It's also free, and I'm not going to spam you about paying for an account (at least until my hosting costs are higher than the $30 / month that they are now), I'm not going to sell your data, I'm not going to promote jobs from companies that are paying me so it's impossible to find anything else, and so on.

And most importantly: I think it'll make your job search easier and more delightful! It did for me. I've found that the tooling that's available for job searching needs quite the refresh, so this app (which is very much a work in progress!) has been my attempt at filling in some of the holes there. I've basically built something that I wanted when I was job hunting because I found it useful, and I hope others will too!

## Roadmap

I have a bunch of ideas for some future things to build, to improve, etc. In no particular order, some things I'll be working on in the future include:

1. Expanding the database of companies and postings. Right now, there are about 550 companies and >12,000 postings.
2. Adding more analytics.
3. Adding some cover letter and resume assistant functionality to help you write and edit your cover letters and resume, maybe powered by GPT.
4. A whole bunch of miscellaneous improvements to the codebase to improve the UX of the app.
5. Design improvements!

## Wrapping Up

If there are any VCs reading this who think "Wow, this is an amazing start up idea!" or any designers or front-end engineers who like the idea and want to help out, please reach out to me 😉.

But in all seriousness: I hope someone comes across this post and finds something here useful, and I'd love to hear any feedback from anyone who's tried the app!


Introducing ZenSearch

Unit Testing dbt Models

Lessons Learned From Running R in Production


## Introduction

In [my last post](/blog/calibration) I wrote about common classification metrics and, especially, calibration.

With calibration in mind, this post will show why balancing your classes -- which is an all-too-common practice when working on classification problems -- is generally a bad idea and leads to poorly calibrated models.

## Some Example Data

For the purposes of this example, I'll use the Wisconsin breast cancer data. The data is built into the `mlbench` package in R and `scikit-learn` in python. You can also get it from the [UCI Machine Learning Repository](<https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic)>).

I'll only be using `cl_thickness`, which is the indicator for clump thickness.

```r
library(readr)
library(dplyr)
library(janitor)
library(purrr)
library(mlbench)

data(BreastCancer)

data <- BreastCancer %>%
  clean_names() %>%
  transmute(
    cl_thickness = as.numeric(cl_thickness),
    class
  ) %>%
  as_tibble()

data %>%
  slice_sample(n = 5) %>%
  pretty_print()
```

| cl_thickness |   class   |
| :----------: | :-------: |
|      5       |  benign   |
|      5       |  benign   |
|      4       |  benign   |
|      6       | malignant |
|      1       |  benign   |

The data is imbalanced: There are far more (about 2x) benign tumors than malignant ones in the sample.

```r
data %>%
  count(class) %>%
  mutate(prop = n / sum(n)) %>%
  pretty_print()
```

|   class   |  n  |   prop    |
| :-------: | :-: | :-------: |
|  benign   | 458 | 0.6552217 |
| malignant | 241 | 0.3447783 |

## Model Fitting

With that class imbalance in mind, let's get to model fitting. The first thing I'll do is fit a simple logistic regression model to predict the class (either malignant or benign) from the clump thickness.

First, I've written a bit of `tidymodels` helper code below for reuse later.

```r
library(tidymodels)

fit_model <- function(data, spec) {
  spec <- set_mode(spec, "classification")

  rec <- recipe(
    class ~ cl_thickness,
    data = data
  )

  wf <- workflow() %>%
    add_model(spec) %>%
    add_recipe(rec)

  fit(wf, data)
}

predict_prob <- function(model, data) {
  predict(model, data, type = "prob")$.pred_malignant
}
```

Now, I'll fit a simple logistic regression model by specifying `logistic_reg()` as the model specification in `fit_model()`.

```r
library(probably)

unbalanced_model <- fit_model(
  data,
  logistic_reg()
)

preds <- tibble(
  truth = data$class,
  truth_int = as.integer(data$class) - 1,
  estimate = predict_prob(unbalanced_model, data)
)
```

And now we can make a calibration plot of our predictions. Remember, the goal is to have the points on the plot lie roughly along the line `y = x`. Lying below the line means that our predictions are too high, and above the line means our predictions are too low.

```r
cal_plot_breaks(preds, truth = truth, estimate = estimate, event_level = "second")
```

![](/posts/balancing-classes/unnamed-chunk-6-1.png)

Awesome! Even with the class imbalance, our model's probability predictions are well-calibrated. In other words, when we predict that there's a 25% chance that a tumor is malignant, it's actually malignant about 25% of the time.

## Balancing the Training Data

So then, what happens if we balance the training data as we're so often told to do? First, let's balance by undersampling from the majority class.

```r
minority_class <- data %>%
  count(class) %>%
  filter(n == min(n))

balanced <- data %>%
  group_split(class, .keep = TRUE) %>%
  map_dfr(
    ~ {
      if (.x$class[1] == minority_class$class) {
        .x
      } else {
        slice_sample(
          .x,
          n = minority_class$n,
          replace = FALSE
        )
      }
    }
  )

balanced %>%
  count(class) %>%
  pretty_print()
```

|   class   |  n  |
| :-------: | :-: |
|  benign   | 241 |
| malignant | 241 |

Now we have the same number of observations for each class. Let's go ahead and fit another logistic regression model, but this time on the balanced data.

```r
balanced_model <- fit_model(balanced, logistic_reg())

preds$balanced_preds <- predict_prob(
  balanced_model,
  data
)

cal_plot_breaks(preds, truth = truth, estimate = balanced_preds, event_level = "second")
```

![](/posts/balancing-classes/unnamed-chunk-8-1.png)

All of a sudden, our model is very poorly calibrated. We're consistently overpredicting the probability of a tumor being malignant. Why is that? Think back to what we just did: We removed a bunch of examples of benign tumors from our training data.

Let's think about that from first principles for a minute. If you had no information at all, a reasonable guess for whether or not a tumor is malignant would be the overall proportion of tumors that are malignant. In the unbalanced data, that number was about 34%. But after balancing, it's now 50%. That means that we've just biased our "no-information" prediction upwards by about 16 percentage points (or 50%). And so it shouldn't be surprising that in our calibration plot above, we see that we're consistently over-predicting. Our probabilities are too high because the baseline rate at which the true class appears in our training data has just increased significantly.

> An important note which I'll circle back to later is that this intuition about a baseline guess is directly rated to the intercept term of the logistic regression model you fit.

## SMOTE

"But no!" you might be thinking. "Why would you just undersample directly? You're supposed to use an algorithm like SMOTE to overcome your class imbalance problem."

Great! Let's see if using SMOTE fixes our calibration issues. I'll first use SMOTE to intelligently oversample the minority class.

```r
library(themis)

smote <- recipe(class ~ cl_thickness, data = data) %>%
  step_smote(class) %>%
  prep() %>%
  bake(new_data = NULL)

smote %>%
  count(class) %>%
  pretty_print()
```

|   class   |  n  |
| :-------: | :-: |
|  benign   | 458 |
| malignant | 458 |

Now that we have balanced classes thanks to SMOTE, let's fit another logistic regresion model and see if it's any better-calibrated.

```r
smote_model <- fit_model(smote, logistic_reg())

preds$smote_preds <- predict_prob(
  smote_model,
  data
)

cal_plot_breaks(preds, truth = truth, estimate = smote_preds, event_level = "second")
```

![](/posts/balancing-classes/unnamed-chunk-10-1.png)

Interesting -- same problem. With SMOTE, we still make very similar errors to the ones we made in the case where we naively undersampled from our majority class. But let's think back to first principles again, because the exact same rationale applies. When we undersampled, we ended up artificially increasing the baseline rate of malignant tumors in our training data, which resulted in predictions that were too high. With SMOTE, we're doing the exact same thing: We've stil rebalanced our data to 50/50, we've just done it a fancier way. So of course we'll have the same problem with overprediction.

## A Random Forest

"But no!" you might be thinking. "You need to use a more complicated model like a random forest, because logistic regression won't pick up on complexities in your data well enough to be well-calibrated."

Great! Let's try a random forest:

```r
library(ranger)

rf <- fit_model(smote, rand_forest())

preds$rf_preds <- predict_prob(
  rf,
  data
)

cal_plot_breaks(preds, truth = truth, estimate = rf_preds, event_level = "second")
```

![](/posts/balancing-classes/unnamed-chunk-11-1.png)

Same issue again, albeit not quite as severe. And the same logic holds. In fact, it's even more straightforward with tree-based models. In a decision tree, you would determine a predicted probability by seeing what proportion of the labels in the leaf node that you end up in based on the features belong to the positive class. But then there's the same logic as before: We've just artifically increased the number of instances of the positive class dramatically, so of course the proportion of labels belonging to the positive class in our leaf nodes will increase.

## Coefficients

Looking at the coefficients of our three models can help understand what's going on here.

```r
list(
  "Original" = unbalanced_model,
  "Undersampled" = balanced_model,
  "Smote" = smote_model
) %>%
  map(tidy) %>%
  imap(~ select(.x, "Term" = term, !!.y := "estimate")) %>%
  reduce(inner_join, by = "Term") %>%
  pretty_print()
```

|     Term     |  Original  | Undersampled |   Smote    |
| :----------: | :--------: | :----------: | :--------: |
| (Intercept)  | -5.1601677 |  -4.4911164  | -4.4639646 |
| cl_thickness | 0.9354593  |  0.9400247   | 0.9173742  |

In all three models, the coefficient associated with the clump thickness is very similar. This should make sense intuitively: Our sampling was at random, so the relationship between the clump thickness and whether or not the tumor was malignant shouldn't change at all.

The thing that does change, though, is the intercept term. In both the model where we undersampled from the majority class and in the SMOTE model, the intercept term is significantly higher than it is in the original model on the unbalanced data. This should feel similar intuitively to the idea of the baseline guess from before. As its core, the intercept term in your logistic regression model is the guess you'd make with "no" information (in this particular case, no information means a clump thickness of 0).

We can illustrate this more clearly with three intercept-only models:

```r
unbalanced_intercept_only <-  glm(class ~ 1, data = data, family = binomial)
undersampled_intecept_only <- glm(class ~ 1, data = balanced, family = binomial)
smote_intercept_only <-       glm(class ~ 1, data = smote, family = binomial)
```

Now, let's compare the intercept coefficients of these three models on a probability (read: not a log-odds) scale.

```r
convert_log_odds_to_probability <- function(x) {
  odds <- exp(x)
  odds / (1 + odds)
}

unbalanced_intercept <-   coef(unbalanced_intercept_only)
undersampled_intercept <- coef(undersampled_intecept_only)
smote_intercept <-        coef(smote_intercept_only)

intercepts <- tibble(
  original =     convert_log_odds_to_probability(unbalanced_intercept),
  undersampled = convert_log_odds_to_probability(undersampled_intercept),
  smote =        convert_log_odds_to_probability(smote_intercept)
)

pretty_print(intercepts)
```

| original  | undersampled | smote |
| :-------: | :----------: | :---: |
| 0.3447783 |     0.5      |  0.5  |

And it's just as we expected: The intercept coefficient in the SMOTE model and the undersampling model are exactly 1/2, which corresponds to the fact that we balanced the classes to be exactly 50/50. And the intercept in the original model with the unbalanced classes is exactly the percentage of the data made up by the true class (malignant).

## When To Rebalance

There are some times where re-balancing the classes in your training data might make sense. One application that comes to mind is if you have strong prior information that your training data is actually biased, and is over-representing one of the two classes.

For instance, let's imagine we have data from 1000 breast cancer patients and we know a priori that about 20% of tumors are malignant, but in the training data, maybe 40% of the tumors we have are malignant. Depending on the long-term goal of the project, it might make sense to undersample from the malignant cases to get the overall rate of tumors being malignant down to around the 20% prior.

The rationale behind doing this would be that if you wanted your model to generalize well to future cases (outside of your training set) and you knew that in the broader population about 20% of cases are malignant, your biased training data could very well result in biased predictions out-of-sample even if your predictions look good in-sample.

Another case where rebalancing can make sense is if you plan to use a technique like [Platt Scaling](https://en.wikipedia.org/wiki/Platt_scaling) or [Isotonic Regression](https://en.wikipedia.org/wiki/Isotonic_regression#:~:text=In%20statistics%20and%20numerical%20analysis,to%20the%20observations%20as%20possible.) to re-calibrate your predictions ex-post. These methods are a bit beyond the scope of this post, but they're both fantastic ways to make sure a model is giving well-calibrated predictions while using classifiers that don't guarantee calibration, such as tree-based models. For instance, a combination of upsampling with SMOTE, using an often poorly-calibrated classifier like a tree booster such as XGBoost, and then re-calibrating ex-post with a Platt Scaler can result in a rare win-win scenario: Well-calibrated predictions, but also improved performance on normal classification metrics like the F1 score or AUC.

## Conclusion

In the [previous post](/blog/calibration), I wrote about calibration. In short, I think calibration is the single most important metric in evaluating the performance of classification models.

And so with that in mind, the main takeaway of this post is that you should be _very_ careful about trying to "fix" the "problem" of class imbalances when you're working on classification problems. If I could summarize the principle that I would follow in just a sentence, it would be that class imbalances often reflect important information about the prevalence of your classes in the real world, and it's often risky to dismiss that information in the name of having data that's split equally. In other words, class imbalances are usually not a problem at all. They're a feature, not a bug.


Balancing Classes in Classification Problems


## Introduction

There's something of a complexity trajectory in evaluating classification models that I've observed over the past few years. It starts with accuracy. But soon after learning about accuracy, data scientists are taught that accuracy is problematic for two reasons:

1. It doesn't work well with unbalanced classes. This is the "if 95% of people don't have cancer and you always predict 'no cancer', your model isn't actually good" argument.
2. It doesn't make any distinction between types of errors. In particular, it weighs false positives and false negatives equally, which may not be appropriate for the problem being solved.

These are both perfectly valid drawbacks of using accuracy as a metric. So then we move on. Next stop: precision and recall.

## Precision and Recall

Precision and recall are the two _next_ most common classification metrics I've seen. _Precision_ is the percentage of the time that your model is correct when it labels something as _true_. Recall is the percentage of the actual _true_ examples that your model labels as true. These metrics are important for different reasons.

A very precise model doesn't make very many Type I errors. For instance, if you're predicting whether or not someone has cancer, a very precise model is "trustworthy" in the sense that if it tells you that they do have cancer, they most likely do. You might think about precision as a metric in hiring: You probably want your hiring process to be very good at evaluating good candidates. A high-precision hiring process would mean that when you think you've found a person who would be a great fit on your team, you're very likely correct.

Recall is a bit different: It's the percentage of the true labels that your model finds. A high-recall model suffers few false _negatives_: When something actually belongs to the true class, your model very often predicts it as such. You might think about this in the context of our cancer example from before. A higher recall model would mean that your model catches more of the cancer cases.

Depending on your use case, you might optimize for one of these or the other. Or you could use a blend of the two, the most common of which is the _F1 score_, which is the harmonic mean of precision and recall. The idea of the F1 score is to optimize for a balance of both precision and recall, as opposed to optimizing for one at the cost of the other.

## Predicting Probabilities

You might be reading this thinking about how this is all about predicting classes, but very often we care about predicting probabilities. For instance, at CollegeVine we make predictions about each student's chances of getting into their favorite colleges and universities. It's not useful for students if we predict an acceptance or a rejection. After all, they want to know their _chance_, and to make a class prediction would mean that we determine a cutoff point at which we decide that if someone's chance is above that threshold, they'll get in. And if not, they won't.

The problem is that there is no such threshold. More likely, college admissions is a bit of a game of chance: If five students each have a 20% chance of getting in to Carleton, for instance, I'd expect that about one of the five would get in on average. But it'd be disingenuous to make a class prediction, and I'm not sure how we'd even do that. For the five previously mentioned students, we'd expect one to get in. But if we were predicting classes we'd either predict all five to be accepted or all five to be rejected depending on where we set our threshold, and neither of those is the most likely scenario.

With all of that in mind, what metrics do we use instead? There are three metrics that we look at when we're evaluating our models: The [Brier Skill Score](https://en.wikipedia.org/wiki/Brier_score), [AUC](/blog/on-auc-roc), and calibration. I've already written about AUC, and for the purposes of this post, I'm going to focus on the one I view as the most important: Calibration.

## Calibration

The most important metric we track is calibration, which we'll often evaluate by looking at a calibration plot. Below is an example of such a plot from [FiveThirtyEight](https://projects.fivethirtyeight.com/checking-our-work/us-senate-elections/)

![](/posts/calibration/fivethirtyeight-calibration.png)

The idea behind calibration is to answer the basic question "When my model said that someone had a 31% chance of winning an election, did they actually win about 31% of the time?" As you get more data, you can group your predictions into buckets to answer this question. For instance, if we're predicting college chances we might take all the times that we said someone had between a 30 and 35% chance of getting accepted, and we'd calculate the actual proportion of the time that they were accepted. If it's about 32.5% (close to the mean prediction in our bucket), we'd say that our model is making well-calibrated predictions in that bucket. If our model makes well-calibrated predictions of admissions chances across all buckets, it's fair to say that it's well-calibrated in general.

Calibration is important because it directly affects how we interpret our model's predictions. If we're making probability predictions and they're _not_ well-calibrated, then when we say something has a 25% chance of happening, or a 50% chance of happening, or a 90% chance of happening, those numbers aren't actually meaningful. It might be the case that the 50% probablity event happens more than the 25% one and less that the 90% one, but that isn't even a guarantee. It also is probably not the case with a badly-calibrated model that the 50% predicted probability event happens twice as often as the 25% one.

For instance, let's imagine we're working on a classic machine learning example (and real-world) problem: Email spam detection. Ultimately, we need to predict a class: Given an email, we need our model to tell us if it's spam or not. But for a large proportion of classifiers, this requires setting a probability threshold. For instance, if our model says there's a greater than 50% chance that some email is spam, we mark it as spam. If false positives (a real email being marked as spam) are more problematic than false negatives (a spam email being marked as not spam), then we might increase our probability threshold to 80%, for example, which would make it "harder" for an email to be marked as spam. But with the higher threshold, the emails that we do mark as spam we're more confident about. Often times, we'll use domain knowledge to determine this threshold. I've had many conversations where we set arbitrary thresholds based on our experience or our gut instincts about how the system we're working in should work. Often, those conversations end with something like "80% feels about right" and we go with that.

Hopefully you're starting to see the issue here: If our model's predictions are poorly calibrated, then it's not possible to make a decision like that. We can't lean into our domain knowledge about any particular number being a threshold that makes sense, because the probabilities we're predicting don't actually _mean_ anything in practice. In other words, the fact that we don't know if when we say 80%, it's actually 95% or 35% or some other number makes it impossible to make decisions based on our predictions. In short, if our predictions aren't well-calibrated, it's not possible to reason about them in any meaningful way. They can also very easily be misleading.

As I've mentioned before, this is especially important is the probabilities themselves _are_ the prediction. If you're telling a student that their chances of getting into Carleton are 27%, it goes without saying that when you say they have a 27% chance, that if they were to apply four times they'd get in about once on average. If they get in about one in eight times instead, the miscalibrated prediction could have a meaningful, negative effect on their college outcomes. For instance, if you severely overpredict a student's chances and they end up applying to fewer schools as a result, there's an increased likelihood of them getting in _nowhere_, which would be a particularly bad outcome. In this case, better-calibrated predictions lead directly to better decision-making.

In statistics world, it might be helpful to think of poor calibration as a bias issue: You might think of better-calibrated predictions as being less biased, in the sense that the expected value of the outcome (the actual long-run frequency of a student being accepted) is closer to what your prediction was.

## Wrapping Up

Hopefully I've convinced you that the calibration of your classifier's predictions is important and practically meaningful. If a model's predictions are poorly calibrated, it's difficult (or impossible) to reason about them in a practical sense. Miscalibrated predictions can also be misleading, because we often naturally interpret probabilities as long-run frequencies. For instance "If I flip this coin 100 times, it'll come up heads about half of the time." You might think about working with poorly calibrated predictions being similar to flipping a biased coin when you're unaware of its biases.

In the next post, I'll talk about class imbalances and why balancing your classes can be a particularly bad idea.


Calibration and Evaluating Classification Models


**AUC** goes by many names: AUC, AUC-ROC, ROC-AUC, the area under the curve, and so on. It's an extremely important metric for evaluating machine learning models and it's an uber-popular data science interview question. It's also, at least in my experience, the single most commonly misunderstood metric in data science.

I've heard several common misunderstandings or flat-out falsehoods from people in all kinds of roles discussing AUC. The biggest offenses tend to come from overcomplicating the topic. It's easy to see the [Wikipedia page for the ROC curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic) and be confused, intimidated, or some combination of the two. ROC builds off of other fundamental data science concepts -- the true and false positives rates of a classifier -- so it's certainly not a good place to _start_ learning about metrics for evaluating the performance of models.

The most common cause for confusion about AUC seems to come from the plot of the _ROC_ curve, and nothing particularly special about AUC itself. Generally, I'll hear AUC explained as being the area under the ROC curve, and that it's all about testing how well your model balances false positives and false negatives. That's all well and good, but it doesn't give someone new to AUC any intuition about what AUC _actually_ means in practice. For instance, let's imagine we're trying to predict the chance that a student is accepted at Carleton College -- a quite common problem at CollegeVine! How does saying "AUC tells me about how my model is balancing false negatives and false positives" tell me anything about how well my model is doing at predicting that student's chances?

The main issue I have with this factual-yet-unhelpful explanation of AUC is just that: While it may be true, it doesn't get to the point. And even worse, it's sometimes used as a crutch: A fallback answer when someone feels stuck when asked how to interpret AUC in real, practical terms.

So in this post, I'll focus on just one thing, then: Answering the question above about how to interpret AUC.

## What is AUC?

As I mentioned, it's usually not helpful to try to explain AUC to someone by telling them that it's just the area under the ROC curve, or that it's a metric you can use for predicting _probabilities_ as opposed to predicting _classes_, or that it's a metric trying to balance false positives and false negatives. None of those things get to the crux of the problem.

So what _is_ AUC, then? It's pretty simple: Let's imagine a model `M` being evaluated on data `X` where `X` contains some instances of the true class and some instances of the false class. The AUC of `M` on `X` is the probability that given a random item from `X` belonging to the _true_ class (`T`) and another random item from `X` belonging to the _false_ class (`F`), that the model predicts that the probability of `T` being true (belonging to the true class) is higher than the probability of `F` being true (belonging to the true class).

Let's go back to the example about Carleton admissions, and let's imagine that we have a model that gives a probability of admission to Carleton given some information about a student. If I give the model one random _accepted_ student and one random _rejected_ student, the AUC of the model is the probability that the accepted student had a higher chance of acceptance (as estimated by the model) than the rejected student did.

For more on this, I'd refer everyone to [this fantastic blog post](https://developers.google.com/machine-learning/crash-course/classification/roc-and-auc) by the team at Google, which does a great job at explaining further and/or better.

## A Simple Implementation

The easiest way to convey this idea might be to show a simple implementation of AUC. Below is some R code.

First, let's start by writing a function to do exactly what's described above. Again, here's the algorithm given some evaluation data:

1. Choose a random item from the `true` class.
2. Choose a random item from the `false` class.
3. Make a prediction on each of the two items.
4. If the predicted probability for the actually true item is greater than the predicted probability for the actually false item, return true. Otherwise, return false. If they're equal, flip a coin.
5. Repeat 1-4 many times, and calculate the proportion of the time your model guessed correctly. This is your AUC.

Now, let's write this in R with a little help from some vectorization.

```r
library(rlang)
library(dplyr)
library(tibble)
## Our AUC implementation
## In this implementation, we take a data frame containing a "truth" (i.e. whether
## the example is _actually_ in either the true class or the false class)
## and an "estimate" (our predicted probability).
## This implementation is in line with how {{yardstick}} implements all of its metrics
interpretable_auc <- function(data, N, truth_col = "truth", estimate_col = "estimate") {

  ## First, subset the data down to just trues and just falses, separately
  trues <- filter(data, .data[[truth_col]] == 1)
  falses <- filter(data, .data[[truth_col]] == 0)

  ## Sample the predicted probabilities for N `true` examples, with replacement
  random_trues <- sample(trues[[estimate_col]], size = N, replace = TRUE)

  ## Do the same for N `false` examples
  random_falses <- sample(falses[[estimate_col]], size = N, replace = TRUE)
  ## If the predicted probability for the actually true
  ##  item is greater than that of the actually false item,
  ##  return `true`.
  ## If the two are equal, flip a coin.
  ## Otherwise, return false.
  true_wins <- ifelse(
    random_trues == random_falses,
    runif(N) > 0.50,
    random_trues > random_falses
  )

  ## Compute the percentage of the time our model was "right"
  mean(true_wins)
}
```

Next, we can test our simple implementation against `yardstick` on some real data. For the sake of demonstration, I just used the built-in `mtcars` data. Here's how the data looks:

```r
library(knitr)
library(kableExtra)
## Doing a little data wrangling
data <- mtcars %>%
  transmute(
    vs = as.factor(vs),
    mpg,
    cyl
  ) %>%
  as_tibble()
data %>%
  slice_sample(n = 6) %>%
  kable("html", caption = 'Six rows of our training data') %>%
  kable_styling(position = "center", full_width = TRUE)
```

| vs  | mpg  | cyl |
| :-: | :--: | :-: |
|  1  | 27.3 |  4  |
|  0  | 19.2 |  8  |
|  0  | 21.0 |  6  |
|  0  | 15.0 |  8  |
|  0  | 13.3 |  8  |
|  0  | 10.4 |  8  |

Now, let's fit a few logistic regression models to the data to see how our AUC implementation compares to the `yardstick` one.

```r
library(purrr)
library(yardstick)
## Simplest model -- Just an intercept. AUC should be 50%
model1 <- glm(vs ~ 1, data = data, family = binomial)
## Adding another predictor
model2 <- glm(vs ~ mpg, data = data, family = binomial)
## And another
model3 <- glm(vs ~ mpg + cyl, data = data, family = binomial)
## Make predictions for all three models
preds <- tibble(
  truth = data$vs,
  m1 = predict(model1, type = "response"),
  m2 = predict(model2, type = "response"),
  m3 = predict(model3, type = "response")
)
## For each model, compute AUC with both methods: Yardstick (library) and "homemade"
map_dfr(
  c("m1", "m2", "m3"),
  ~ {
    yardstick <- roc_auc(preds, truth = truth, estimate = !!.x, event_level = "second")$.estimate
    homemade <- interpretable_auc(preds, N = 100000, truth_col = "truth", estimate_col = .x)
    tibble(
      model = .x,
      yardstick = round(yardstick, digits = 2),
      homemade = round(homemade, digits = 2)
    )
  }
) %>%
  kable("html", caption = 'Yardstick vs. Our Implementation') %>%
  kable_styling(position = "center", full_width = TRUE)
```

| model | yardstick | homemade |
| :---: | :-------: | :------: |
|  m1   |   0.50    |   0.50   |
|  m2   |   0.91    |   0.91   |
|  m3   |   0.95    |   0.95   |

As we've seen here, AUC actually shouldn't be all that much of a cause for confusion! The way I like to frame it is this: The AUC of your model is how good your model is at making even-odds bets. If I give your model two options and ask it to pick which one it thinks is more likely, a "better" model (by AUC standards) will be better at identifying the true class more often.

In real terms, that's a meaningful, good thing. If we're trying to predict the probability of a cancer patient having cancer, it's important that our model can distinguish between people with cancer and people without it when given one person from each class. If it couldn't - meaning the model was either randomly guessing or doing worse than random - the AUC would be 50% (or below 50%, in the worse-than-random disaster scenario).

## Additional Thoughts

I also often hear the misconception that AUC is sensitive to things like class imbalance. This means that if the true class makes up a disproportionately large (or small) proportion of the evaluation data, that can skew the AUC. But based on the intuition we just built before, that's of course not true. The key thing to remember is that the model is given one true and one false example. In choosing those, it doesn't matter if the true class only makes up 0.005% of all of the examples in the evaluation data: AUC is only evaluating the model on its ability to determine _which_ of the two is the true class.

However, there is one thing related to class imbalance, and just sample size in general, that _would_ affect AUC, which is the raw number of examples of each class in the evaluation data. If, for instance, you had only a single instance of the `true` class in the evaluation set, then the AUC of the model is entirely determined by how good the predictions of the model are on that single example. For instance, if we have a single `true` class and the model predicts a 100% probability of it being true, then, assuming the predictions for all of the other examples in the evaluation set are _not_ 100%, the AUC of the model as evaluated on that data is 100%. This isn't necessarily because the model is "good" in any sense, but just because the model is over-indexing to a single good prediction in the evaluation set. In practice though, this AUC estimate wouldn't generalize. As we got more data, the predictions for all the true classes would certainly not all be 100%, so the AUC of the model would go down over time.

Fortunately, there's an easy fix for this problem. AUCs are a point estimate, but we could also estimate a margin of error or a confidence interval for our AUC. For a situation where we only have a single instance of the true class in the evaluation set, the margin of error for our AUC would be very wide.

## Wrapping Up

Hopefully this post helped give a better intuition for what AUC actually is! A couple of major takeaways:

1. AUC doesn't need to be this super complicated thing about trading off between false positives and negatives and trying many different classification thresholds and such. In my opinion, it's much simpler to just think about it as the likelihood of a guess that your model makes between two choices being correct.
2. AUC isn't affected by class imbalances.


2024-08-24

Lessons Learned from Building ZenSearch, Part II: Async Tasks

2024-08-14

Lessons Learned from Building ZenSearch, Part I: Backstory

2023-11-10

Introducing ZenSearch

2023-07-09

Unit Testing dbt Models

2023-06-29

Lessons Learned From Running R in Production

2023-04-01

Balancing Classes in Classification Problems

2023-03-20

Calibration and Evaluating Classification Models

2023-03-09

Interpreting AUC-ROC