任务2.1 学习BeautifulSoup
英语生词本
parser n. 剖析器;
prettify v. 修饰;
sibling n. 兄弟，姐妹; [生] 同科，同属; [人] 氏族成员;
在cmd命令行窗口安装BeautifulSoup库：
pip install beautifulsoup4
如何使用BeautifulSoup
from bs4 import BeautifulSoup
soup = BeautifulSoup('<p>data</p>' , 'html.parser')
BeautifulSoup库的安装小测
演示HTML页面地址：http://python123.io/ws/demo.html
>>> r = requests.get("http://python123.io/ws/demo.html")
>>> r.text
'<html><head><title>This is a python demo page</title></head>\r\n<body>\r\n<p class="title"><b>The demo python introduces several python courses.</b></p>\r\n<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\r\n<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p>\r\n</body></html>'
>>> demo = r.text
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(demo, "html.parser")
>>> print(soup.prettify())
<html>
<head>
<title>
This is a python demo page
</title>
</head>
<body>
<p class="title">
<b>
The demo python introduces several python courses.
</b>
</p>
<p class="course">
Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
Basic Python
</a>
and
<a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
Advanced Python
</a>
.
</p>
</body>
</html>
>>>
2.1.1 BeautifulSoup库的基本元素
HTML文档 <==> 标签树 <==> BeautifulSoup类
即，BeautifulSoup类 对应一个HTML/XML文档的全部内容。
BeautifulSoup类有5种基本元素：
-
Tag
标签，最基本的信息组织单元，分别用<></>标明开头和结尾 -
Name
标签的名字，<p>...</p>的名字是'p'，格式：<tag>.name -
Attributes
标签的属性，字典形式组织，格式：<tag>.attrs -
NavigableString
标签内非属性字符串，<>...</>中字符串，格式：<tag>.string -
Comment
标签内字符串的注释部分，一种特殊的Comment类型
标签树的平行遍历: （上行遍历，下行遍历略）
for sibling in soup.a.next_siblings:
print(sibling) #遍历后续节点
for sibling in soup.a.previous_siblings:
print(sibling) #遍历前续节点
问题：如何让<html>内容更加“友好”的显示？
---bs4库的prettify()方法
>>> print(soup.a.prettify())
<a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
Basic Python
</a>
2.1.2 BeautifulSoup库实践案例
使用beautifulsoup提取丁香园论坛的回复内容
- 用户浏览器访问目标网站并检查目标内容所在标签
目标网址如下：http://www.dxy.cn/bbs/thread/626626#626626
用Chrome访问的，按F12可看见网站结构及回复内容所在标签如下图：
- 获取回复内容
我们所需的评论内容就在td class="postbody"标签下，利用BeautifulSoup获取内容
content = data.find("td", class_="postbody").text
参考代码：
import urllib.request
from bs4 import BeautifulSoup as bs
def main():
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/58.0.3029.110 Safari/537.36 SE 2.X MetaSr 1.0"
}
url = 'http://www.dxy.cn/bbs/thread/626626'
request = urllib.request.Request(url, headers=headers)
response = urllib.request.urlopen(request).read().decode("utf-8")
html = bs(response, 'lxml')
getItem(html)
def getItem(html):
datas = [] # 用来存放获取的用户名和评论
for data in html.find_all("tbody"):
try:
userid = data.find("div", class_="auth").get_text(strip=True)
print(userid)
content = data.find("td", class_="postbody").get_text(strip=True)
print(content)
datas.append((userid,content))
except:
pass
print(datas)
if __name__ == '__main__':
main()
参考链接：https://blog.csdn.net/wwq114/article/details/88085875
说实话，没看懂。。。还要认真学习呀
任务2.1 学习xpath
要交作业了，来不及学了，待补充。。。
