1.数据挖掘简介(略)

2.使用Python和IPython Notebook

2.1.安装Python

2.2.安装IPython

2.3.安装scikit-learn

scikit-learn是用Python开发的机器学习库，它包含大量机器学习算法、数据集、工具和框架。它以Python科学计算的相关工具集为基础，其中numpy和scipy等都针对数据处理任务进行过优化，因此scikit-learn速度快、扩展性强，为此做数据挖掘很实用。
scikit-learn可以用Python3提供的pip工具进行安装，之前没有安装Numpy和Scipy的话也会顺便安装。安装命令如下：

pip install scikit-learn

3.亲和性分析示例

3.1什么是亲和性分析

亲和性分析根据样本个体（物体）之间的相似度，确定它们关系的亲疏。亲和性分析的应用场景如下。
(1)向网站用户提供多样化的服务或投放定向广告
(2)为了向用户推荐电影或商品，而卖给他们一些与之相关的小玩意。
(3)根据基因寻找有亲缘关系的人。
......
亲和性有哪些测量方法？
(1)统计两件商品一起出售的频率，或者统计顾客购买商品1后再买商品2的比率。
(2)计算两个体之间的相似度
......

3.2商品推荐

商品销售从线下搬到线上后，很多之前靠人工完成的工作只有实现自动化，才有望将生意做大，向上销售出自英文up-selling，指的是向已经购买商品的顾客推销另一种商品。原来线下由人工完成的商品推荐工作，现在依靠数据挖掘技术就能完成，而且利润大，助推电子商务革命的发展！
我们一起看一个简单的推荐服务：人们之前经常购买的两件商品，以后也很可能同时购买。 $\color{red}{**很简单但很实用的思路**}$
作为数据挖掘入门性质的例子，我们希望得到下面的规则：
如果一个人买了商品X，那么他很有可能购买商品Y

3.3在Numpy中加载数据集

import numpy as np
dataset_filename = 'affinity_dataset.txt'
X = np.loadtxt(dataset_filename)

3.4实现简单的排序规则

规则的优劣有多种衡量方法，常用的是支持度（support）和置信度（confidence）。
支持度指数据集中应验的次数，有时候需要对支持度进行规范化。
支持度衡量给定规则应验的比例，置信度衡量规则准确率如何，即符合给定条件的所有规则里，跟当前规则结论一致的比例有多大，计算方法为首先统计当前规则的出现次数，再用它除以条件相同的规则数量
如果顾客买了苹果，他们也会购买香蕉的支持度和置信度

num_apple_purchases = 0
for sample in X:
    if sample[3] ==1: #This person bought apples
        num_apple_purchases += 1
#    print('{0} people bought apples'.format(num_apple_purchases)) #ou can try the print way to find difference
print('{0} people bought apples'.format(num_apple_purchases))

image.png

同理，检测sample[4]的值是否为1，就可以确定顾客是否也买了香蕉，进而可以计算支持度和置信度。
我们需要统计数据集中所有规则的相关数据，首先分别为规则应验和规则无效两种情况构建字典。字典的键是由条件和结论组成的元组，元组元素为特征在特征列表中的索引值，不要用实际特证名，比如“顾客如果购买了苹果，也买了香蕉”就用（3，4）表示。如果某个个体的条件和结论均与给定规则相符，则表示给定规则对该个体适用，反之无效。
为了计算所有规则的置信度和支持度，首先创建几个字典，用来存储计算结果。这里使用defaultdict，好处是如果查找的键不存在，则返回默认值。需要统计的量有规则应验、规则无效、条件相同的规则数量。

from collections import defaultdict
vaild_rules = defaultdict(int)
invaild_rules = defaultdict(int)
num_occurances = defaultdict(int)
#依次对样本的每个个体及个体的每个特征值进行处理。第一个特征为规则的前提条件-----顾客购买了某种商品
for sample in X:
    for premise in range(5):
#检测个体是否满足条件，如果不满足则检测下一个条件
        if sample[premise] ==0:continue
#如果条件满足（即值为1），该条件的出现次数加1，在遍历过程中跳过条件和结论相同的情况，比如“如果顾客购买了苹果，他们也买苹果”，这样的规则无用
        num_occurances[premise] += 1
        n_sample,n_features = X.shape
        for conclusion in range(n_features):
            if premise ==conclusion:continue
#如果规则适用于个体，规则应验这种情况（vaild_rules字典中，键为由条件和结论组成的元组）增加一次，反之，违反规则情况（invaild_rules字典中）就增加一次

https://github.com/datawhalechina/joyful-pandas datawhale pandas教程
https://space.bilibili.com/631186842?from=search&seid=16882960572917617056 Rachel's english
https://www.liulishuo.com/liulishuo.html 流利说英语 有app直接下
https://github.com/fengdu78/lihang-code 李航Python实现

[ch1_affinity_create]

X = np.zeros((100, 5), dtype='bool')
#dtype can change,such as int,float

X.shape[0]
#0 is row,1 is col

#数组的索引方式是和列表一样的

np.savetxt("affinity_dataset.txt", X, fmt='%d')
#parameters
fmt : str or sequence of strs, optional
        A single format (%10.5f), a sequence of formats, or a
        multi-format string, e.g. 'Iteration %d -- %10.5f', in which
        case `delimiter` is ignored. For complex `X`, the legal options
        for `fmt` are:

#create a random float from 0 to 1
a = np.random.random()

print(X[:5].astype(np.int))

[ch1_affinity]

n_samples, n_features = X.shape
print("This dataset has {0} samples and {1} features".format(n_samples, n_features))
#这种print格式用print('{0},{1}'.format(a,b))

#count the people who bought apples
num_apple_purchases = 0
for sample in X:
    if sample[3] = 1:
        num_apple_purchases += 1
print('{0} people bought apples'.format(num_apples_purchases))

####################################################################################################
##bought 3 but not bought 4
rule_valid = 0
rule_invalid = 0
for sample in X:
    if sample[3] == 1:  # This person bought Apples
        if sample[4] == 1:
            # This person bought both Apples and Bananas
            rule_valid += 1
        else:
            # This person bought Apples, but not Bananas
            rule_invalid += 1
print("{0} cases of the rule being valid were discovered".format(rule_valid))
print("{0} cases of the rule being invalid were discovered".format(rule_invalid))
####################################################################################################
## not bought 3 
rule_valid = 0
rule_invalid = 0
for sample in X:
    if sample[3] == 1:
        if sample[4] == 1:
            rule_valid += 1
    else:
        rule_invalid += 1
print('{0} rule_valid'.format(rule_valid))
print('{0} rule_invalid'.format(rule_invalid))
####################################################################################################
规则是  如果买了苹果，可能也买了香蕉。
规则无效是 如果买了苹果，但没买香蕉

print("The support is {0} and the confidence is {1:.3f}.".format(support, confidence))
# Confidence can be thought of as a percentage using the following:
print("As a percentage, that is {0:.1f}%.".format(100 * confidence))

####################################################################################################
from collections import defaultdict
# Now compute for all possible rules
valid_rules = defaultdict(int)
invalid_rules = defaultdict(int)
num_occurences = defaultdict(int)

for sample in X:
    for premise in range(n_features):
        if sample[premise] == 0: continue
        # Record that the premise was bought in another transaction
        num_occurences[premise] += 1
        for conclusion in range(n_features):
            if premise == conclusion:  # It makes little sense to measure if X -> X.
                continue
            if sample[conclusion] == 1:
                # This person also bought the conclusion item
                valid_rules[(premise, conclusion)] += 1
            else:
                # This person bought the premise, but not the conclusion
                invalid_rules[(premise, conclusion)] += 1
support = valid_rules
confidence = defaultdict(float)
for premise, conclusion in valid_rules.keys():
    confidence[(premise, conclusion)] = valid_rules[(premise, conclusion)] / num_occurences[premise]
####################################################################################################
from collections import defaultdict
rule_valid = defaultdict(int)
rule_invalid = defaultdict(int)
num_premise = defaultdict(int)
n_features = X.shape[1]
for sample in X:
    for premise in range(n_features):
        if sample[premise] == 0:continue
        if sample[premise] == 1:
            num_premise[premise] += 1
        for conclusion in range(n_features):
            if premise == conclusion:continue
            if sample[conclusion] == 1:
                rule_valid[(premise,conclusion)] += 1
            else:
                rule_invalid[(premise,conclusion)] += 1
support = rule_valid
confidence = defaultdict(float)
for premise,conclusion in rule_valid.keys():
    confidence[(premise,conclusion)] = rule_valid[(premise,conclusion)] / num_premise[premise]

####################################################################################################
for premise, conclusion in confidence:
    premise_name = features[premise]
    conclusion_name = features[conclusion]
    print("Rule: If a person buys {0} they will also buy {1}".format(premise_name, conclusion_name))
    print(" - Confidence: {0:.3f}".format(confidence[(premise, conclusion)]))
    print(" - Support: {0}".format(support[(premise, conclusion)]))
    print("")
####################################################################################################
for premise,conclusion in confidence:
    features = ["bread", "milk", "cheese", "apples", "bananas"]
    premise_name = features[premise]
    conclusion_name = features[conclusion]
    print('If someone buy {0} then they may buy {1}'.format(premise_name,conclusion_name))
    print('confidence is {0:.3f}'.format(confidence[(premise,conclusion)]))
    print('support is {0}'.format(support[(premise,conclusion)]))

#用于打印特定的数据结构，整齐好看
from pprint import pprint
pprint(list(support.items()))
#example
import pprint

data = ("test", [1, 2, 3,'test', 4, 5], "This is a string!",
        {'age':23, 'gender':'F'})
print(data)
pprint.pprint(data)

image.png

注意

# [Python: 字典列表: itemgetter 函数: 根据某个或某几个字典字段来排序列表](https://www.cnblogs.com/baxianhua/p/8182627.html)
from operator import itemgetter
sorted_support = sorted(support.items(), key=itemgetter(1), reverse=True)
# [python中sorted和sort 、reversed和reverse的使用](https://www.cnblogs.com/shengguorui/p/10863988.html)

[OneR]

[ch1_oner_application]

#iris.describe
print(dataset.DESCR)

在进行OneR算法分类前需要将数据进行离散化

# Compute the mean for each attribute
attribute_means = X.mean(axis=0)
assert attribute_means.shape == (n_features,)#assert:断言
X_d = np.array(X >= attribute_means, dtype='int')
#X.means(axis):axis = 0 is symbol take col
#assert 1==1    # 条件为 true 正常执行
#assert 1==2    # 条件为 false 触发异常

#sklearn中已经废弃cross_validation,将其中的内容整合到#model_selection中
#将sklearn.cross_validation 替换为 sklearn.model_selection
##origin
# Now, we split into a training and test set
from sklearn.cross_validation import train_test_split

# Set the random state to the same number to get the same results as in the book
random_state = 14

X_train, X_test, y_train, y_test = train_test_split(X_d, y, random_state=random_state)
print("There are {} training samples".format(y_train.shape))
print("There are {} testing samples".format(y_test.shape))
##new
# Now, we split into a training and test set
from sklearn.model_selection import train_test_split
# Set the random state to the same number to get the same results as in the book
random_state = 14
X_train, X_test, y_train, y_test = train_test_split(X_d, y, random_state=random_state)
print("There are {} training samples".format(y_train.shape))
print("There are {} testing samples".format(y_test.shape))
#train_X,test_X,train_y,test_y = train_test_split(train_data,train_target,test_size=0.3,random_state=5)
#train_test_split()函数是用来随机划分样本数据为训练集和测试集的，当然也可以人为的切片划分
#优点：随机客观的划分数据，减少人为因素
#test_size：测试数据占样本数据的比例，若整数则样本数量

#zip() 函数用于将可迭代的对象作为参数，将对象中对应的元素打包成一个个元组，然后返回由这些元组组成的列表。
#如果各个迭代器的元素个数不一致，则返回列表长度与最短的对象相同，利用 * 号操作符，可以将元组解压为列表
>>>a = [1,2,3]
>>>b = [4,5,6]
>>>c = [4,5,6,7,8]
>>>zipped = zip(a,b)     # 打包为元组的列表
[(1, 4), (2, 5), (3, 6)]
>>>zip(a,c)              # 元素个数与最短的列表一致
[(1, 4), (2, 5), (3, 6)]
>>>zip(*zipped)          # 与 zip 相反，*zipped 可理解为解压，返回二维矩阵式
[(1, 2, 3), (4, 5, 6)]
class_counts = defaultdict(int)
#Iterate through each sample and count the frequency of each class/value pair
for sample, y in zip(X, y_true):
    if sample[feature] == value:
        class_counts[y] += 1

a = zip(X, y)
for b in a:
    print(b)

image.png

a = zip(X, y)
for b,c in a:
    print(b)

image.png

a = zip(X, y)
for b,c in a:
    print(c)

image.png

error = sum([class_count for class_value, class_count in class_counts.items()
                 if class_value != most_frequent_class])

[Python数据挖掘入门与实践]-第一章开启数据挖掘之旅

[Python数据挖掘入门与实践]-第一章开启数据挖掘之旅

1.数据挖掘简介(略)

2.使用Python和IPython Notebook

2.1.安装Python

2.2.安装IPython

2.3.安装scikit-learn

3.亲和性分析示例

3.1什么是亲和性分析

3.2商品推荐

3.3在Numpy中加载数据集

3.4实现简单的排序规则

[ch1_affinity_create]

[ch1_affinity]

注意

[OneR]

[ch1_oner_application]