Python nltk — A list of stems of sentences/phrases

Python nltk — A list of stems of sentences/phrases … here is a solution to the problem.

Python nltk — A list of stems of sentences/phrases

I

have a bunch of sentences in the list and I want to use the nltk library to block it. I can extract one sentence at a time, but I’m having trouble extracting sentences from a list and putting them back together. Am I missing a step? The nltk library is new. Thanks!

import nltk 
from nltk.stem import PorterStemmer 
ps = PorterStemmer()

# Success: one sentences at a time 
data = 'the gamers playing games'
words = word_tokenize(data)
for w in words:
    print(ps.stem(w))

# Fails: 

data_list = ['the gamers playing games',
            'higher scores',
            'sports']
words = word_tokenize(data_list)
for w in words:
    print(ps.stem(w))

# Error: TypeError: expected string or bytes-like object
# result should be: 
['the gamer play game',
 'higher score',
 'sport']

Solution

You are passing the list to the word_tokenize, but you cannot do so.

The solution is to wrap your logic in another for-loop

data_list = ['the gamers playing games','higher scores','sports']
for words in data_list:
    words = tokenize.word_tokenize(words)
    for w in words:
        print(ps.stem(w))

>>>>the
gamer
play
game
higher
score
sport

Related Problems and Solutions