使用numpy进行数据分析

  1. 生成矩阵X1和X2,两者随机从[-10,10]内整数中抽取10个样本,且形状分别为2 $\times$ 5和5 $\times$ 2,随机种子分别为0和1,输出结果;

  2. 将X1和X2的转置用两种方法实现按行合并,分别赋值给Xnew1和Xnew2,输出结果,并比较结果的差异;

  3. 对矩阵X1实现按行求和和按列求和;

  4. 计算X1 $\times$ X2;

  5. 生成矩阵X,两者随机从[-10,10]内整数中抽取25个样本,形状为5 $\times$ 5,随机种子为0,同时生成向量y=[3,2,4,6,1],求解$X\beta=y$的方程解;

In [1]:
import numpy as np

#1
np.random.seed(0)
X1 = np.random.randint(-10,11,(2,5))
np.random.seed(1)
X2 = np.random.randint(-10,11,(5,2))
print(X1,"X1")
print(X2,'X2')

#2
Xnew1 = np.concatenate((X1,X2.T),axis = 1)  #axis:1为按行,0为按列
Xnew2 = np.stack((X1,X2.T),axis = 1)
print(f"Xnew1 has shape {Xnew1.shape}: {Xnew1}")
print(f"Xnew2 has shape {Xnew2.shape}: {Xnew2}")

#3
X1.sum(axis = 1)  #行
X1.sum(axis = 0)  #列

#4
print(np.matmul(X1,X2))

#5
np.random.seed(0)
X = np.random.randint(-10,11,(5,5))
y = np.array([3,2,4,6,1])
np.linalg.solve(X,y)
[[  2   5 -10  -7  -7]
 [ -3  -1   9   8  -6]] X1
[[ -5   1]
 [  2  -2]
 [ -1   1]
 [ -5   5]
 [-10   6]] X2
Xnew1 has shape (4, 5): [[  2   5 -10  -7  -7]
 [ -3  -1   9   8  -6]
 [ -5   2  -1  -5 -10]
 [  1  -2   1   5   6]]
Xnew2 has shape (2, 2, 5): [[[  2   5 -10  -7  -7]
  [ -3  -1   9   8  -6]]

 [[ -5   2  -1  -5 -10]
  [  1  -2   1   5   6]]]
[[115 -95]
 [ 24  12]]
Out[1]:
array([-0.21997202,  0.31466924, -0.4899185 ,  0.61881287, -0.18558613])

使用pandas进行数据分析

shops_nm.xlsx文档里面记录了有关火锅店铺的基本信息,包括点名,评分,菜品等。

  1. 把shops_nm.xlsx读入到数据框raw_data中, 输出其列名, 行数和列数大小, 前五行的内容;

  2. 挑选出数据框raw_data中评价数不缺失的行,挑选出的数据赋值给数据框raw_data2,输出行数, 列数以及前五行的内容;

  3. 对于数据框raw_data2,利用apply函数以及自定义函数clean_price(x)(函数作用为去除字符串x中的中文内容)去除"人均"列中的所有中文字符,并该列数据格式转化为浮点型数据,输出raw_data2前五行的内容;

  4. 对于数据框raw_data2,新建一列数据,列名为"商家等级", 若'评分'<=2,商家等级为0,若2<'评分'<=3.5,商家等级为1,其余'评分'的商家等级为2,输出raw_data2前五行的内容;

  5. 对于数据框raw_data2,根据'关键词'和'商家等级'分组计算平均评分,并输出结果;

In [5]:
import pandas as pd

#1
raw_data = pd.read_excel("shops_nm.xlsx")
list(raw_data.columns)
raw_data.shape
raw_data.head()

#2
raw_data2=raw_data.loc[raw_data["评价数"].isnull()==False]
raw_data2.shape
raw_data2.head()

#3
def clean_price(x):
    filter_words = ["人均:", "人均", "大概", "左右", "差不多"]
    if type(x) is int or type(x) is float:
        return x
    for word in filter_words:
        if word in x:
            x = x.replace(word, "")
    return x

raw_data2["人均"] = raw_data2["人均"].apply(clean_price)
raw_data2["人均"] = raw_data2["人均"].astype(float)
raw_data2.head()

#4
def rating(x):
    if x<=2:
        return 0
    elif x<=3.5:
        return 1
    else:
        return 2
raw_data2['商家等级']=raw_data2['评分'].apply(rating)
raw_data2.head()

#5
raw_data2.groupby(by = ["关键词", "商家等级"]).agg({"人均": "mean"})
<ipython-input-5-4fd5d61b37f2>:24: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  raw_data2["人均"] = raw_data2["人均"].apply(clean_price)
<ipython-input-5-4fd5d61b37f2>:25: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  raw_data2["人均"] = raw_data2["人均"].astype(float)
<ipython-input-5-4fd5d61b37f2>:36: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  raw_data2['商家等级']=raw_data2['评分'].apply(my_star)
Out[5]:
人均
关键词 商家等级
火锅 0 64.645313
1 48.153846
2 49.799388
烧烤 0 56.884615
1 73.000000
2 63.058182
In [ ]:
 

使用Matplotlib绘图

  1. 将merge_shop_coupon_nm.xls读入到数据框merge_data中, 输出其列名, 行数和列数大小, 前五行的内容;

  2. 设置全局中文字体为'SimHei';

  3. 设置画布大小为(15,10),绘制如下四幅子图,第一幅子图展示人均评价关系图,第二幅子图展示人均评价关系局部图,评价数截取500-6500之间,人均截取20-60,第三幅子图按关键词分组绘制人均评价关系局部图,截取范围与第二幅子图类似,第四幅绘制购买人数评价数目关系图,购买人数和评价数绘图范围都截取0-3000;

  4. 设置画布大小为(10,5),绘制如下纵向排列和横向排列的直方图;

  5. 设置画布大小为(10,8),绘制如下反映购买人数评价数关系的箱型图

In [20]:
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt

#1
merge_data = pd.read_excel("merge_shop_coupon_nm.xlsx")
list(merge_data.columns)
merge_data.shape
merge_data.head()

#2
matplotlib.rcParams["font.sans-serif"] = ["SimHei"]
In [24]:
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt

matplotlib.rcParams["font.sans-serif"] = ["SimHei"]   #设置字体样式,正常显示中文标签
matplotlib.rcParams["axes.unicode_minus"] = False   #设置正常显示字符
#3
fig = plt.figure(figsize=(15, 10))

# ax1
ax1 = fig.add_subplot(221)  #创建子图
ax1.set_title("人均评价关系图")
ax1.set_xlabel("人均")
ax1.set_ylabel("评价数")
ax1.scatter(x=merge_data["人均"], y=merge_data["评价数"], color="r", marker="v")

# ax2
ax2 = fig.add_subplot(222)
ax2.set_title("人均评价关系图")
ax2.set_xlabel("人均")
ax2.set_ylabel("评价数")
ax2.scatter(x=merge_data["人均"], y=merge_data["评价数"], color="r", marker='v')
ax2.set_ylim((500,6500))
ax2.set_xlim((20,60))
ax2.hlines(2000, 20, 60, colors="r", linestyle="dashed")

# ax3
ax3 = fig.add_subplot(223)
ax3.set_title("人均评价关系图")
ax3.set_xlabel("人均")
ax3.set_ylabel("评价数")
ax3.scatter(x=merge_data["人均"][merge_data["关键词"]=="火锅"],
           y=merge_data["评价数"][merge_data["关键词"]=="火锅"],
           color="r", marker='v',label = "火锅")
ax3.scatter(x=merge_data["人均"][merge_data["关键词"]!="火锅"],
           y=merge_data["评价数"][merge_data["关键词"]!="火锅"],
           color="b", marker='o',label = "非火锅")
ax3.set_ylim((500,6500))
ax3.set_xlim((20,60))
ax3.legend(loc = "upper left", bbox_to_anchor = (0,1))  #loc:指定图例的位置作为锚点,bbox_to_anchor:锚点相对坐标
ax3.hlines(2000, 20, 60, colors="r", linestyle="dashed")

# ax4
ax4 = fig.add_subplot(224)
ax4.set_title("购买人数评价数目关系图")
ax4.set_xlabel("购买人数")
ax4.set_ylabel("评价数")
ax4.set_xlim((0,3000))
ax4.set_ylim((0,3000))
ax4.scatter(x=merge_data["购买人数"], y=merge_data["评价数"], color="r")
ax4.plot(np.arange(3000), np.arange(3000), color="blue")

plt.show()
In [11]:
#4
fig = plt.figure(figsize=(10,5))
ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122)
ax1.hist(merge_data["评分"], bins=30, color="r", rwidth=0.9)  #bins:bin的条数;rwidth:控制条形的相对宽度
ax2.hist(merge_data["评分"], bins=30, color="b", rwidth=0.9,
         cumulative="True", orientation="horizontal")     
ax1.set_xlabel("评分")
plt.show()
D:\anaconda\lib\site-packages\matplotlib\backends\backend_agg.py:214: RuntimeWarning: Glyph 20998 missing from current font.
  font.set_text(s, 0.0, flags=flags)
D:\anaconda\lib\site-packages\matplotlib\backends\backend_agg.py:183: RuntimeWarning: Glyph 20998 missing from current font.
  font.set_text(s, 0, flags=flags)
In [19]:
#5
d1 = merge_data["评价数"][merge_data["购买人数"] < 500]
d2 = merge_data["评价数"][merge_data["购买人数"] >= 500][merge_data["购买人数"] < 1000 ]
d3 = merge_data["评价数"][merge_data["购买人数"] >= 1000][merge_data["购买人数"] < 1500 ]
d4 = merge_data["评价数"][merge_data["购买人数"] >= 1500][merge_data["购买人数"] < 2000 ]
d = [d1, d2, d3, d4]

fig = plt.figure(figsize=(10, 8))

ax = fig.add_subplot(111)
ax.boxplot(d,
           patch_artist='Patch',
           labels = ['<500', '[500,1000)', '[1000,1500)', '[1500,2000)'],
           boxprops = {'color':'#2196F3','facecolor':'#BBDEFB'}  #facecolor:填充颜色
          )
ax.set_title('购买人数评价数目关系图')
ax.set_xlabel('购买人数')
ax.set_ylabel('评价数')
plt.show()
C:\Users\cy\AppData\Roaming\Python\Python38\site-packages\numpy\core\_asarray.py:83: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
  return array(a, dtype, copy=False, order=order)
In [ ]: