Stop words are extremely common, low-information words — such as "the", "is", "and", "to", and "of" — that natural language processing pipelines often filter out so that analysis focuses on the rarer words that actually distinguish one document from another. Removing them shrinks the vocabulary, cuts noise, and speeds up classic techniques like bag-of-words, TF-IDF, and keyword search. There is no single official list: NLTK ships 179 English stop words, scikit-learn about 318, and spaCy about 326. The critical rule is that removal is optional and task-dependent — you strip stop words for statistical keyword models, but you keep them for sentiment analysis, translation, and any modern transformer model, because words like "not" and "never" carry the meaning.
That paragraph is the summary an AI overview will give you. What it can't give you is the judgment call underneath it: stop word removal is a 1990s optimization that quietly became harmful for the models most people now use. Below is the pipeline it fits, the exact library differences, working Python for NLTK and spaCy, and a clear decision table for when to remove versus keep.
The pipeline: where stop word removal fits
Stop word removal is one optional stage in a preprocessing pipeline. Text is first broken into tokens, then the tokens that match a stop word list are dropped, leaving a smaller set of content-bearing tokens for the model or index.
Understanding Stop Words
Stop words are high-frequency, low-semantic-value words that can be filtered out to improve NLP processing efficiency. Common examples include articles, prepositions, and conjunctions that appear across most documents but don't contribute to distinguishing content or meaning. The NLTK library provides a standard list including words like "i", "me", "my", "we", "our", "just", "don", and "should".
For example, the sentence "Come over to my house" becomes "Come house" when stop words are removed. While not grammatically correct, the core intent remains understandable, demonstrating the trade-off between processing efficiency and linguistic completeness.
Stop words at a glance
The single most common mistake is treating "remove stop words" as a fixed rule. It is a tunable choice, and the right answer depends on the model and the task. This table is the reference to keep.
| Aspect | Detail |
|---|---|
| What they are | Extremely frequent words ("the", "is", "and", "to", "of") that appear in nearly every document and carry little standalone meaning. |
| NLTK list | 179 English stop words. Load with stopwords.words('english') after nltk.download('stopwords'). Small and conservative. |
| spaCy list | ~326 English stop words in spacy.lang.en.stop_words.STOP_WORDS. Each token also exposes token.is_stop. |
| scikit-learn list | ~318 words in the built-in 'english' list. Maintainers warn it has known issues — supply your own list for serious work. |
| Remove for | Bag-of-words, TF-IDF, topic modeling (LDA), keyword search / information retrieval — order-independent statistical methods. |
| Keep for | Sentiment analysis, negation-sensitive text, translation, question answering, and all transformer models (BERT, GPT). |
| Main risk | Negation and function words ("not", "never", "no", "but") flip meaning; removing them corrupts sentiment and legal/medical text. |
| Customize by | Adding domain filler that is uninformative in your corpus and removing defaults (like "not") that carry signal for your task. |
Why modern transformers do NOT remove stop words
This is the part most older tutorials get wrong. Stop word removal was designed for statistical models that treat a document as an unordered bag of words, where "the" genuinely adds nothing. Modern transformer models — BERT, RoBERTa, GPT, and their relatives — work in the opposite way: they read the entire sequence and use self-attention to derive each word's meaning from its neighbors. In that setting, function words are not noise; they are grammatical signal the model was trained on.
Two concrete reasons not to strip stop words before a transformer:
- They use subword tokenizers, not word lists. BERT uses WordPiece and GPT uses byte-pair encoding. They never operate on a clean list of English words, so a stop word filter doesn't even map cleanly onto their inputs — it just deletes text the model expects to see.
- Context depends on the "filler." "The bank of the river" versus "a deposit at the bank" — prepositions and articles are part of how attention disambiguates meaning. Removing them degrades accuracy.
The rule of thumb: stop word removal belongs to the bag-of-words / TF-IDF era. If you are fine-tuning or prompting a contextual model, feed it the raw text.
When Stop Words Can Be Problematic
Aggressive stop word removal can cause significant issues when context and sentiment matter. Consider sentiment analysis scenarios where phrases like "not happy" or "never good" carry completely different meanings than "happy" or "good" alone. Removing "not" or "never" because they appear in stop word lists completely reverses the intended emotion.
Critical warning: Context matters. Blindly applying generic stop word lists can distort meaning, especially in sentiment analysis, legal text interpretation, or applications requiring precise semantic understanding.
Benefits of Using Stop Words
Stop words optimize NLP tasks by reducing noise and computational overhead. High-frequency words like "the", "is", "on", and "and" appear disproportionately often but carry minimal semantic weight. Removing them leads to more efficient text processing, reduced storage requirements, and improved model focus on meaningful content.
-
Performance improvement: Faster tokenization and processing
-
Storage efficiency: Smaller indexes and reduced memory usage
-
Model accuracy: Focus on distinguishing keywords rather than filler words
-
Search relevance: Better document matching in information retrieval
Best practice: Tailor your stop word strategy to your specific use case. Search engines benefit from aggressive filtering, while chatbots and sentiment analysis systems require more conservative approaches.
Removing stop words in Python
The three most common libraries each take a slightly different approach. Use a set for the stop word lookup — membership tests are O(1), which keeps removal fast across large corpora.
NLTK
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
nltk.download("stopwords") # run once
nltk.download("punkt") # tokenizer models
stop_words = set(stopwords.words("english")) # 179 words
text = "Come over to my house and we can build the model together"
tokens = word_tokenize(text)
filtered = [w for w in tokens if w.lower() not in stop_words]
print(filtered)
# ['Come', 'house', 'build', 'model', 'together']
spaCy
spaCy needs no separate corpus download — every token carries an is_stop flag:
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Come over to my house and we can build the model together")
filtered = [token.text for token in doc if not token.is_stop]
print(filtered)
# ['Come', 'house', 'build', 'model', 'together']
scikit-learn (TF-IDF)
For a bag-of-words or TF-IDF pipeline, the vectorizer removes stop words for you. Prefer passing your own list over the built-in 'english' set, which its maintainers flag as imperfect:
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
"Come over to my house",
"The model is training on the data",
]
# Built-in list (quick, but has known issues):
vectorizer = TfidfVectorizer(stop_words="english")
# Recommended: supply a curated list you control
my_stop_words = ["the", "is", "on", "to", "my", "over"]
vectorizer = TfidfVectorizer(stop_words=my_stop_words)
X = vectorizer.fit_transform(corpus)
print(vectorizer.get_feature_names_out())
To keep negation words for a sentiment task, subtract them from the list before you build the vectorizer:
keep = {"not", "no", "never", "against"}
custom = [w for w in stop_words if w not in keep]
See the effect on real text
A word cloud is the fastest way to see what stop word removal does: with stop words left in, "the" and "and" dominate; strip them and the meaningful terms surface. Paste any text below and toggle stop word filtering to watch the vocabulary change.
Key takeaways
- Stop words are common, low-information words; there is no canonical list, and NLTK (179), scikit-learn (~318), and spaCy (~326) all differ.
- Remove them for bag-of-words, TF-IDF, topic modeling, and keyword search — order-independent statistical methods that benefit from a smaller, cleaner vocabulary.
- Keep them for sentiment analysis, translation, question answering, and every transformer model; negation words like "not" and "never" carry the meaning.
- Always customize: start from a standard list, add domain filler, and remove defaults that matter for your task. Test accuracy with and without removal on your own data.