Sentiment Analysis with NLTK

Sentiment analysis tells you if a text is positive, negative, or neutral. It is one of the most popular tasks in natural language processing (NLP).

In this guide, you will learn how to do sentiment analysis with NLTK, a classic Python NLP library. We will cover the basics, build a small classifier, and look at real outputs.

What Is Sentiment Analysis?

Sentiment analysis is the process of detecting emotion in text. Companies use it to read customer reviews at scale.

You can analyze tweets, product feedback, support tickets, or survey answers. The goal is simple: turn words into a label like positive or negative.

There are two common approaches. The first uses a prebuilt lexicon of scored words. The second trains a machine learning model on labeled data.

Why Use NLTK for Sentiment Analysis?

NLTK is free, well documented, and beginner friendly. It ships with a built-in sentiment lexicon called VADER.

VADER is tuned for social media text. It handles slang, emojis, and punctuation like exclamation marks. That makes it a great starting point.

NLTK also gives you the tools to build your own classifier. You get tokenizers, feature extractors, and evaluation helpers in one package.

Installing NLTK

First, install the library with pip. Then download the data packages you need.

pip install nltk

Next, open a Python shell and download the resources. This only needs to happen once.

import nltk

# Download the VADER lexicon used for sentiment scoring
nltk.download('vader_lexicon')

[nltk_data] Downloading package vader_lexicon to /root/nltk_data...
[nltk_data]   Package vader_lexicon is already up-to-date!
True

Getting Started with VADER

VADER stands for Valence Aware Dictionary and sEntiment Reasoner. It returns four scores: negative, neutral, positive, and compound.

The compound score is the most useful. It ranges from -1 (very negative) to +1 (very positive).

Let's score a few short sentences.

from nltk.sentiment import SentimentIntensityAnalyzer

# Create the analyzer object
sia = SentimentIntensityAnalyzer()

# Score a positive sentence
print(sia.polarity_scores("I love this product, it is amazing!"))

{'neg': 0.0, 'neu': 0.402, 'pos': 0.598, 'compound': 0.8439}

The compound score of 0.84 is clearly positive. Now try a negative sentence.

print(sia.polarity_scores("This is the worst experience I have ever had."))

{'neg': 0.512, 'neu': 0.488, 'pos': 0.0, 'compound': -0.6249}

A compound score of -0.62 signals strong negative sentiment. The model works well on short, informal text.

Turning Scores into Labels

Raw scores are useful, but labels are easier to work with. A common rule uses thresholds on the compound score.

Scores above 0.05 are positive. Scores below -0.05 are negative. Everything in between is neutral.

def get_label(text):
    score = sia.polarity_scores(text)["compound"]
    if score >= 0.05:
        return "positive"
    elif score <= -0.05:
        return "negative"
    return "neutral"

# Test the helper on three inputs
for sentence in ["Great service!", "It was okay.", "Terrible support."]:
    print(sentence, "->", get_label(sentence))

Great service! -> positive
It was okay. -> neutral
Terrible support. -> negative

This small function is enough for many real projects. You can wrap it around a loop and process thousands of reviews in seconds.

Preparing Text Before Scoring

Clean text usually gives better results. Splitting text into words is the first step. You can learn more in this guide on tokenization with NLTK in Python.

Lowercasing helps too. So does removing URLs and extra whitespace. Keep negations like "not" though, because they flip meaning.

import re

def clean_text(text):
    text = text.lower()                 # normalize case
    text = re.sub(r"http\S+", "", text) # remove links
    text = re.sub(r"\s+", " ", text)    # collapse spaces
    return text.strip()

print(clean_text("I LOVE this!!  Visit http://example.com now"))

i love this!! visit now

Notice that punctuation like "!!" stays. VADER uses it as a signal of intensity, so do not strip it.

Building a Custom Classifier

VADER is great, but sometimes you need a model trained on your own data. NLTK makes this easy with the Naive Bayes classifier.

You need labeled examples. Each example is a tuple of words and a label.

from nltk.tokenize import word_tokenize
from nltk.classify import NaiveBayesClassifier

# Small labeled dataset for demonstration
train_data = [
    (word_tokenize("I love this movie"), "pos"),
    (word_tokenize("This film is wonderful"), "pos"),
    (word_tokenize("Amazing acting and story"), "pos"),
    (word_tokenize("I hate this movie"), "neg"),
    (word_tokenize("This film is boring"), "neg"),
    (word_tokenize("Terrible acting and plot"), "neg"),
]

# Convert words into a feature dictionary
def features(words):
    return {word: True for word in words}

# Train the classifier
training_set = [(features(words), label) for words, label in train_data]
classifier = NaiveBayesClassifier.train(training_set)

# Predict on new text
print(classifier.classify(features(word_tokenize("wonderful story"))))

pos

The model correctly labels the new text as positive. With a real dataset of thousands of rows, accuracy improves a lot.

You can inspect which words matter most. The show_most_informative_features method prints the strongest signals.

# Show the top 5 words that influence the prediction
classifier.show_most_informative_features(5)

Most Informative Features
               wonderful = True              pos : neg    =      1.7 : 1.0
                 amazing = True              pos : neg    =      1.7 : 1.0
                terrible = True              neg : pos    =      1.7 : 1.0
                 boring = True               neg : pos    =      1.7 : 1.0
                   hate = True               neg : pos    =      1.7 : 1.0

Evaluating Your Model

Never trust a model without testing it. Split your data into a training set and a test set.

Use accuracy to measure how many predictions are correct. For balanced datasets, this is a good first metric.

from nltk.classify.util import accuracy

# Evaluate on the same small set (for demo only)
print(accuracy(classifier, training_set))

1.0

A perfect score here is expected because the test set equals the training set. Always evaluate on unseen data in real projects.

Working with Large Text Pipelines

Sentiment analysis often runs inside a bigger system. You may stream reviews from a message queue and store results in a database.

For event pipelines, tools like Kafka are common. If you build one, these guides on how to produce and consume Kafka messages in Python will help you move text through your system.

You can also score rows pulled from a database. This walkthrough on how to query an Oracle database from Python shows how to fetch data and feed it into your analyzer.

Common Pitfalls

Sarcasm is hard. "Great, another delay" reads positive to VADER but is clearly negative to a human.

Domain words cause trouble too. In finance, "volatile" is neutral. In reviews, it sounds negative.

Keep sentences short when possible. Long, mixed sentences often get a muddy compound score.

Finally, always check a sample by hand. Automated scores are useful, but human review catches blind spots.

Conclusion

Sentiment analysis with NLTK is a great first NLP project. VADER gives you instant results with almost no setup.

When you need more control, NLTK's Naive Bayes classifier lets you train on your own labeled data. Clean your text, evaluate on unseen data, and watch for sarcasm.

Start small, test often, and your sentiment pipeline will be reliable in no time.