getting token with a simple function

  function, nlp, python, token

could you please tell me what is the difference of these functions?

I saw this function somewhere;

def separate_punc(doc_text):
    return [token.text.lower() for token in nlp(doc_text) if token.text not in 'nn nnn!"-#$%&()--.*+,-/:;<=>[email protected][]^_`{|}~tn ']

I tried to rewrite it with my own knowledge like this:

def clean_punch(Doc):
    D=[]
    for i in nlp(Doc):
        if i.text not in 'nn nnn!"-#$%&()--.*+,-/:;<=>[email protected][]^_`{|}~tn ':
             D.append(i.text.lower())
        return D

But they are not the same. I get just the first word of document. but the first one get the whole tokens.

Source: Python Questions

LEAVE A COMMENT