/ NLP  

NLP系列

基于fasttext的文本情感分析

1
import fastText
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import re  #正则表达式
from bs4 import BeautifulSoup #html标签处理
import pandas as pd

def review_to_wordlist(review):
'''
把IMDB的评论转成词序列
'''
# 去掉HTML标签,拿到内容
review_text = BeautifulSoup(review).get_text()
# 用正则表达式取出符合规范的部分
review_text = re.sub("[^a-zA-Z]"," ", review_text)
# 小写化所有的词,并转成词list
words = review_text.lower().split()
# 返回words
return words

# 使用pandas读入训练和测试csv文件
train = pd.read_csv('data/labeledTrainData.tsv', header=0, delimiter="\t", quoting=3)
1
num = len(train['review'])
1
train_num = int(num/5*4)
1
2
3
train_data = []
for i in range(0,train_num):
train_data.append(" ".join(review_to_wordlist(train['review'][i])))
1
2
3
test_data = []
for i in range(train_num,num):
test_data.append(" ".join(review_to_wordlist(train['review'][i])))
1
2
y_train = train['sentiment'][:train_num]
y_test = list(train['sentiment'][train_num:])
1
train_data[:10]
['with all this stuff going down at the moment with mj i ve started listening to his music watching the odd documentary here and there watched the wiz and watched moonwalker again maybe i just want to get a certain insight into this guy who i thought was really cool in the eighties just to maybe make up my mind whether he is guilty or innocent moonwalker is part biography part feature film which i remember going to see at the cinema when it was originally released some of it has subtle messages about mj s feeling towards the press and also the obvious message of drugs are bad m kay visually impressive but of course this is all about michael jackson so unless you remotely like mj in anyway then you are going to hate this and find it boring some may call mj an egotist for consenting to the making of this movie but mj and most of his fans would say that he made it for the fans which if true is really nice of him the actual feature film bit when it finally starts is only on for minutes or so excluding the smooth criminal sequence and joe pesci is convincing as a psychopathic all powerful drug lord why he wants mj dead so bad is beyond me because mj overheard his plans nah joe pesci s character ranted that he wanted people to know it is he who is supplying drugs etc so i dunno maybe he just hates mj s music lots of cool things in this like mj turning into a car and a robot and the whole speed demon sequence also the director must have had the patience of a saint when it came to filming the kiddy bad sequence as usually directors hate working with one kid let alone a whole bunch of them performing a complex dance scene bottom line this movie is for people who like mj on one level or another which i think is most people if not then stay away it does try and give off a wholesome message and ironically mj s bestest buddy in this movie is a girl michael jackson is truly one of the most talented people ever to grace this planet but is he guilty well with all the attention i ve gave this subject hmmm well i don t know because people can be different behind closed doors i know this for a fact he is either an extremely nice but stupid guy or one of the most sickest liars i hope he is not the latter',
 'the classic war of the worlds by timothy hines is a very entertaining film that obviously goes to great effort and lengths to faithfully recreate h g wells classic book mr hines succeeds in doing so i and those who watched his film with me appreciated the fact that it was not the standard predictable hollywood fare that comes out every year e g the spielberg version with tom cruise that had only the slightest resemblance to the book obviously everyone looks for different things in a movie those who envision themselves as amateur critics look only to criticize everything they can others rate a movie on more important bases like being entertained which is why most people never agree with the critics we enjoyed the effort mr hines put into being faithful to h g wells classic novel and we found it to be very entertaining this made it easy to overlook what the critics perceive to be its shortcomings',
 'the film starts with a manager nicholas bell giving welcome investors robert carradine to primal park a secret project mutating a primal animal using fossilized dna like jurassik park and some scientists resurrect one of nature s most fearsome predators the sabretooth tiger or smilodon scientific ambition turns deadly however and when the high voltage fence is opened the creature escape and begins savagely stalking its prey the human visitors tourists and scientific meanwhile some youngsters enter in the restricted area of the security center and are attacked by a pack of large pre historical animals which are deadlier and bigger in addition a security agent stacy haiduk and her mate brian wimmer fight hardly against the carnivorous smilodons the sabretooths themselves of course are the real star stars and they are astounding terrifyingly though not convincing the giant animals savagely are stalking its prey and the group run afoul and fight against one nature s most fearsome predators furthermore a third sabretooth more dangerous and slow stalks its victims the movie delivers the goods with lots of blood and gore as beheading hair raising chills full of scares when the sabretooths appear with mediocre special effects the story provides exciting and stirring entertainment but it results to be quite boring the giant animals are majority made by computer generator and seem totally lousy middling performances though the players reacting appropriately to becoming food actors give vigorously physical performances dodging the beasts running bound and leaps or dangling over walls and it packs a ridiculous final deadly scene no for small kids by realistic gory and violent attack scenes other films about sabretooths or smilodon are the following sabretooth by james r hickox with vanessa angel david keith and john rhys davies and the much better bc by roland emmerich with with steven strait cliff curtis and camilla belle this motion picture filled with bloody moments is badly directed by george miller and with no originality because takes too many elements from previous films miller is an australian director usually working for television tidal wave journey to the center of the earth and many others and occasionally for cinema the man from snowy river zeus and roxanne robinson crusoe rating below average bottom of barrel',
 'it must be assumed that those who praised this film the greatest filmed opera ever didn t i read somewhere either don t care for opera don t care for wagner or don t care about anything except their desire to appear cultured either as a representation of wagner s swan song or as a movie this strikes me as an unmitigated disaster with a leaden reading of the score matched to a tricksy lugubrious realisation of the text it s questionable that people with ideas as to what an opera or for that matter a play especially one by shakespeare is about should be allowed anywhere near a theatre or film studio syberberg very fashionably but without the smallest justification from wagner s text decided that parsifal is about bisexual integration so that the title character in the latter stages transmutes into a kind of beatnik babe though one who continues to sing high tenor few if any of the actors in the film are the singers and we get a double dose of armin jordan the conductor who is seen as the face but not heard as the voice of amfortas and also appears monstrously in double exposure as a kind of batonzilla or conductor who ate monsalvat during the playing of the good friday music in which by the way the transcendant loveliness of nature is represented by a scattering of shopworn and flaccid crocuses stuck in ill laid turf an expedient which baffles me in the theatre we sometimes have to piece out such imperfections with our thoughts but i can t think why syberberg couldn t splice in for parsifal and gurnemanz mountain pasture as lush as was provided for julie andrews in sound of music the sound is hard to endure the high voices and the trumpets in particular possessing an aural glare that adds another sort of fatigue to our impatience with the uninspired conducting and paralytic unfolding of the ritual someone in another review mentioned the bayreuth recording and knappertsbusch though his tempi are often very slow had what jordan altogether lacks a sense of pulse a feeling for the ebb and flow of the music and after half a century the orchestral sound in that set in modern pressings is still superior to this film',
 'superbly trashy and wondrously unpretentious s exploitation hooray the pre credits opening sequences somewhat give the false impression that we re dealing with a serious and harrowing drama but you need not fear because barely ten minutes later we re up until our necks in nonsensical chainsaw battles rough fist fights lurid dialogs and gratuitous nudity bo and ingrid are two orphaned siblings with an unusually close and even slightly perverted relationship can you imagine playfully ripping off the towel that covers your sister s naked body and then stare at her unshaven genitals for several whole minutes well bo does that to his sister and judging by her dubbed laughter she doesn t mind at all sick dude anyway as kids they fled from russia with their parents but nasty soldiers brutally slaughtered mommy and daddy a friendly smuggler took custody over them however and even raised and trained bo and ingrid into expert smugglers when the actual plot lifts off years later they re facing their ultimate quest as the mythical and incredibly valuable white fire diamond is coincidentally found in a mine very few things in life ever made as little sense as the plot and narrative structure of white fire but it sure is a lot of fun to watch most of the time you have no clue who s beating up who or for what cause and i bet the actors understood even less but whatever the violence is magnificently grotesque and every single plot twist is pleasingly retarded the script goes totally bonkers beyond repair when suddenly and i won t reveal for what reason bo needs a replacement for ingrid and fred williamson enters the scene with a big cigar in his mouth and his sleazy black fingers all over the local prostitutes bo s principal opponent is an italian chick with big breasts but a hideous accent the preposterous but catchy theme song plays at least a dozen times throughout the film there s the obligatory we re falling in love montage and loads of other attractions my god what a brilliant experience the original french title translates itself as life to survive which is uniquely appropriate because it makes just as much sense as the rest of the movie none',
 'i dont know why people think this is such a bad movie its got a pretty good plot some good action and the change of location for harry does not hurt either sure some of its offensive and gratuitous but this is not the only movie like that eastwood is in good form as dirty harry and i liked pat hingle in this movie as the small town cop if you liked dirty harry then you should see this one its a lot better than the dead pool',
 'this movie could have been very good but comes up way short cheesy special effects and so so acting i could have looked past that if the story wasn t so lousy if there was more of a background story it would have been better the plot centers around an evil druid witch who is linked to this woman who gets migraines the movie drags on and on and never clearly explains anything it just keeps plodding on christopher walken has a part but it is completely senseless as is most of the movie this movie had potential but it looks like some really bad made for tv movie i would avoid this movie',
 'i watched this video at a friend s house i m glad i did not waste money buying this one the video cover has a scene from the movie capricorn one the movie starts out with several clips of rocket blow ups most not related to manned flight sibrel s smoking gun is a short video clip of the astronauts preparing a video broadcast he edits in his own voice over instead of letting us listen to what the crew had to say the video curiously ends with a showing of the zapruder film his claims about radiation shielding star photography and others lead me to believe is he extremely ignorant or has some sort of ax to grind against nasa the astronauts or american in general his science is bad and so is this video',
 'a friend of mine bought this film for and even then it was grossly overpriced despite featuring big names such as adam sandler billy bob thornton and the incredibly talented burt young this film was about as funny as taking a chisel and hammering it straight through your earhole it uses tired bottom of the barrel comedic techniques consistently breaking the fourth wall as sandler talks to the audience and seemingly pointless montages of hot girls adam sandler plays a waiter on a cruise ship who wants to make it as a successful comedian in order to become successful with women when the ship s resident comedian the shamelessly named dickie due to his unfathomable success with the opposite gender is presumed lost at sea sandler s character shecker gets his big break dickie is not dead he s rather locked in the bathroom presumably sea sick perhaps from his mouth he just vomited the worst film of all time',
 'this movie is full of references like mad max ii the wild one and many others the ladybug s face it s a clear reference or tribute to peter lorre this movie is a masterpiece we ll talk much more about in the future']
1
2
ftrain = open("data/sentiment_train.txt","w")
ftest = open("data/sentiment_test.txt","w")
1
2
3
4
5
def to_file(file, x, y):
for i, line in enumerate(x):
outline = line + "\t__label__" + str(y[i]) + "\n"
file.write(outline)
file.close()
1
to_file(ftrain, train_data, y_train)
1
to_file(ftest, test_data, y_test)
1
2
3
4
def print_results(N, p, r):
print("N\t" + str(N))
print("P@{}\t{:.3f}".format(1, p))
print("R@{}\t{:.3f}".format(1, r))
1
2
3
4
from fastText import train_supervised

model = train_supervised(
input="data/sentiment_train.txt", epoch=25, lr=1.0, wordNgrams=2, verbose=2, minCount=1)
1
print_results(*model.test("data/sentiment_test.txt"))
N    5000
P@1    0.891
R@1    0.891
1
model.predict('the creators of south park in their own film here this is a brilliant film with a huge entertainment factor if you like naked gun films and are not young and not too mature or serious on your humor you ll love this')
(('__label__1',), array([1.00001001]))