下面的Python程序实现了通过从网页抓取一篇文章，然后根据这篇文章来生成新的文章，这其中的原理就是基于概率统计的文本分析。
过程大概就是网页抓取数据->统计分析->生成新文章。网页抓取数据是通过BeautifulSoup库来抓取网页上的文本内容。统计分析这个首先需要使用ngram模型来把文章进行分词并统计频率。因为文章生成主要依据马尔可夫模型，所以使用了2-gram，这样可以统计出一个单词出现在另一个单词后的概率。生成新文章是基于分析大量随机事件的马尔可夫模型。随机事件的特点是在一个离散事件发生之后，另一个离散事件将在前一个事件的条件下以一定的概率发生。
fromurllib.requestimporturlopenfromrandomimportrandintfrombs4importBeautifulSoupimportredefwordListSum(wordList):sum =0forword, valueinwordList.items(): sum = sum + valuereturnsumdefretrieveRandomWord(wordList):randomIndex = randint(1, wordListSum(wordList))forword, valueinwordList.items(): randomIndex -= valueifrandomIndex <=0:returnworddefbuildWordDict(text):text = re.sub('(\n|\r|\t)+'," ", text) text = re.sub('\"',"", text) punctuation = [',','.',';',':']forsymbolinpunctuation: text = text.replace(symbol," "+ symbol +" ") words = text.split(' ') words = [wordforwordinwordsifword !=""] wordDict = {}foriinrange(1, len(words)):ifwords[i-1]notinwordDict: wordDict[words[i-1]] = {}ifwords[i]notinwordDict[words[i-1]]: wordDict[words[i-1]][words[i]] =0wordDict[words[i-1]][words[i]] = wordDict[words[i-1]][words[i]] +1returnwordDictdefrandomFirstWord(wordDict):randomIndex = randint(0, len(wordDict))returnlist(wordDict.keys())[randomIndex]html = urlopen("http://www.guancha.cn/america/2017_01_21_390488_s.shtml")bsObj = BeautifulSoup(html,"lxml")ps = bsObj.find("div", {"id":"cmtdiv3523349"}).find_next_siblings("p");content =""forpinps: content = content + p.get_text()text = bytes(content,"UTF-8")text = text.decode("ascii","ignore")wordDict = buildWordDict(text)length =100chain =""currentWord = randomFirstWord(wordDict)foriinrange(0, length): chain += currentWord +" "currentWord = retrieveRandomWord(wordDict[currentWord])print(chain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
buildWordDict(text)函数接收文本内容，生成的内容如下
{‘itself’: {‘,’: 1}, ‘night’: {‘sky’: 1}, ‘You’: {‘came’: 1, ‘will’: 1}, ‘railways’: {‘all’: 1}, ‘government’: {‘while’: 1, ‘,’: 1, ‘is’: 1}, ‘you’: {‘now’: 1, ‘open’: 1, ‘down’: 1, ‘with’: 1, ‘.’: 6, ‘,’: 1, ‘that’: 1},
主要就是生成一个字典，键是文章中所有出现的词语，值其实也是一个字典，这个字典是所有直接出现在键后边的词语及其出现的频率。这个函数就是ngram模型思想的运用。
retrieveRandomWord(wordList)函数的wordList代表的是出现在上一个词语后的词语列表及其频率组成的字典，然后根据统计的概率随机生成一个词。这个函数是马尔可夫模型的思想运用。
然后运行这个程序会生成一个长度为100的文章，如下面所示
fail . We will stir ourselves , but we will never before . Do not share one heart and pleasant it back our jobs . We are infused with the orderly and railways all of the gangs and robbed our jobs for their success will determine the civilized world . We will their success will be a great men and highways and millions to all bleed the world . It belongs to great national effort to defend our products , constantly complaining , D . We will be ignored again . It belongs to harness the expense of America .
生成的文章看起来语法混乱，这也难怪，因为只是抓取分析统计了一篇的文章。我想如果可以抓取足够多的英文文章，数据集足够大那么语法准确度会大大提高。
