• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
The Analysis Factor

The Analysis Factor

Statistical Consulting, Resources, and Statistics Workshops for Researchers

  • Home
  • Our Programs
    • Membership
    • Online Workshops
    • Free Webinars
    • Consulting Services
  • About
    • Our Team
    • Our Core Values
    • Our Privacy Policy
    • Employment
    • Collaborate with Us
  • Statistical Resources
  • Contact
  • Blog
  • Login

Measures of Predictive Models: Sensitivity and Specificity

by Karen Grace-Martin 5 Comments

Not too long ago, I was  in Syracuse for a family trip to the zoo. Syracuse is about 60 miles from where I live and it has a very nice little zoo.

This year was particularly exciting because a Trader Joe’s just opened in Syracuse.  We don’t have one where we live (sadly!)  so we always stock up on our favorite specialty groceries when we’re near a Trader Joe’s.

On this particular trip, though, we had an unwelcome surprise.  My credit card card company believed my Trader Joe’s spree was fraudulent and declined the transaction.  I got a notice on my phone and was able to fix it right away, so it wasn’t the big inconvenience it could have been.

But this led us to wonder what it was about the transaction that led the bank to believe it was fraudulent.  Do credit card thieves often skip town and go grocery shopping?

The bank was clearly betting so.  It must have a model for aspects of a transaction that are likely enough to be fraudulent that it shuts it down. 

With any model, though, you’re never going to to hit 100% accuracy.  And if you’re wrong, there’s a tradeoff between tightening standards to catch the thieves and annoying your customers.

Measuring Accuracy of Model Predictions

There are many ways to measure how well a statistical model predicts a binary outcome.  Three very common measures are  accuracy, sensitivity, and specificity.

Accuracy is one of those rare terms in statistics that means just what we think it does, but sensitivity and specificity are a little more complicated.  To understand all three, first we have to consider the situation of predicting a binary outcome.

The basic situation is this:  for each trial (in our example, each transaction), there is only one true outcome: a Positive or a Negative.  In my example, we’ll assume a Positive is a stolen credit card.  That’s what the bank is on the lookout for.

And of course we need the model to predict the outcome better than randomly guessing.  Imagine if your credit card transactions were randomly declined for fraud.  You would stop using the card.

So here is a little table of all the possible situations:
confusion-matrix

This table has the fancy name of a confusion matrix, but you can see that it’s pretty simple.

The Test Indicator is whatever process we’re using to predict whether each individual is a Yes or a No on the outcome we’re interested in.  Does the bank’s model predict a thief?

The Outcome is what actually eventually happens.  Was the card really stolen?

For some decisions, those in box a, the model correctly predicted a no.  All good.  This is a true negative.  The customer is using their own card and the bank believes it.  The transaction goes through and the family happily snacks on dried mango on the drive home.

Box b is the count of those who were predicted to be Yeses but were actual Nos.  Not so good.  False positive.  The customer isn’t happy that their real grocery shopping gets declined.  Only some quick followup on the phone saves them.

Box c is the opposite of b.  Those who were predicted to be Nos but were actual Yeses.  This one is not so good either.  False negative.  The thief gets away with a load of risotto and dark chocolate covered almonds.

And finally some transactions were predicted to be Yeses and truly were Yeses.  These individuals are all in box d.  The thief is shut down.  Justice is served.

Accuracy of Models

A perfectly accurate model would put every transaction into boxes a and d.  Thieves are stopped but customers are not.

A model that is so bad it’s worthless would have a lot of b’s (angry customers without groceries) and c’s (happy thieves with groceries) and possibly both.

One simple way of measuring Accuracy is simply the proportion of individuals who were correctly classified–the proportions of True Positives and True Negatives.

This is helpful for sure, but sometimes it matters whether we’re correctly getting a Positive or a Negative correct.  It may be worth annoying a few customers to make sure no thieves get away.

Another issue is we can generally increase one simply by decreasing the other.  This may have important implications but the overall Accuracy rate won’t change.

Or worse, we could improve overall Accuracy just by making the test more able to find the more common category.

So a better approach is to look at the accuracy for Positives and Negatives separately.

These two values are called Sensitivity and Specificity.

Sensitivity = d/(c+d)
: The proportion of observed positives that were predicted to be positive.  In other words, of all the transactions that were truly fraudulent, what percentage did we find?

Specificity = a/(a+b): The proportion of observed negatives that were predicted to be negatives.  In other words, of all the transactions that were legitimate, what percentage did we predict to be so?

Ideally, the test will result in both being high, but usually there is a tradeoff.  Every test needs to pick a threshold for how high a probability of fraud has to be before we call it a fraud.

Lowering that threshold to increase Sensitivity will decrease Specificity and vice versa.  It’s important to understand this as you’re choosing that threshold and evaluating a model.

Bookmark and Share

 

Binary, Ordinal, and Multinomial Logistic Regression for Categorical Outcomes
Get beyond the frustration of learning odds ratios, logit link functions, and proportional odds assumptions on your own. See the incredible usefulness of logistic regression and categorical data analysis in this one-hour training.

Tagged With: logistic regression, model accuracy, ROC Curve, sensitivity, specificity

Related Posts

  • What Is an ROC Curve?
  • Member Training: Explaining Logistic Regression Results to Non-Researchers
  • How to Decide Between Multinomial and Ordinal Logistic Regression Models
  • When to Use Logistic Regression for Percentages and Counts

Reader Interactions

Comments

  1. Diane Yelle says

    December 7, 2020 at 12:15 pm

    Great way of explaining the concepts – clear and easy to understand!!

    Reply
  2. Mehul says

    April 16, 2020 at 7:04 am

    Ty very much for this very clear explanation. I have seen explanations of sensitivity and specificity, be all over the place in many articles.

    Reply
  3. Manish Kumar says

    February 9, 2020 at 1:16 am

    very clear explanation . Trader Joe’s example is incredible. Kind of revises my concepts every time I go there 🙂 > I hope they open a new store in your neighborhood

    Reply
  4. Becky says

    July 18, 2018 at 5:43 am

    Incredibly useful and logical explanation!

    Reply
  5. rperkins says

    December 3, 2017 at 2:13 pm

    At last! A logical mind and a clear writer. This explanation (like so many others on the site) is incredibly useful.

    Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Please note that, due to the large number of comments submitted, any questions on problems related to a personal study/project will not be answered. We suggest joining Statistically Speaking, where you have access to a private forum and more resources 24/7.

Primary Sidebar

This Month’s Statistically Speaking Live Training

  • Member Training: Analyzing Pre-Post Data

Upcoming Free Webinars

Poisson and Negative Binomial Regression Models for Count Data

Upcoming Workshops

  • Analyzing Count Data: Poisson, Negative Binomial, and Other Essential Models (Jul 2022)
  • Introduction to Generalized Linear Mixed Models (Jul 2022)

Copyright © 2008–2022 The Analysis Factor, LLC. All rights reserved.
877-272-8096   Contact Us

The Analysis Factor uses cookies to ensure that we give you the best experience of our website. If you continue we assume that you consent to receive cookies on all websites from The Analysis Factor.
Continue Privacy Policy
Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Non-necessary
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
SAVE & ACCEPT