How do I tokenize a string sentence in NLTK?

I am using nltk, so I want to create my own custom texts just like the default ones on nltk.books. However, I’ve just got up to the method like

my_text = ['This', 'is', 'my', 'text']

I’d like to discover any way to input my “text” as:

my_text = "This is my text, this is a nice way to input text."

Which method, python’s or from nltk allows me to do this. And more important, how can I dismiss punctuation symbols?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

This is actually on the main page of nltk.org:

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

Method 2

As @PavelAnossov answered, the canonical answer, use the word_tokenize function in nltk:

from nltk import word_tokenize
sent = "This is my text, this is a nice way to input text."
word_tokenize(sent)

If your sentence is truly simple enough:

Using the string.punctuation set, remove punctuation then split using the whitespace delimiter:

import string
x = "This is my text, this is a nice way to input text."
y = "".join([i for i in x if not in string.punctuation]).split(" ")
print y

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0