爬虫学习笔记(二)

Posted on | In , , |
Beautiful Soup 库

Beautiful Soup 库

demo html: http://python123.io/ws/demo.html

这里是基本用法示例。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
>>> import requests
>>> r = requests.get("http://python123.io/ws/demo.html")
>>> r.text
'<html><head><title>This is a python demo page</title></head>\r\n<body>\r\n<p class="title"><b>The demo python introduces several python courses.</b></p>\r\n<p class="course">Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:\r\n<a href="http://www.icourse163.org/course/BIT-268001" class="py1" id="link1">Basic Python</a> and <a href="http://www.icourse163.org/course/BIT-1001870001" class="py2" id="link2">Advanced Python</a>.</p>\r\n</body></html>'
>>> demo = r.text
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup(demo,'html.parser')
>>> print(soup.prettify())
<html>
 <head>
  <title>
   This is a python demo page
  </title>
 </head>
 <body>
  <p class="title">
   <b>
    The demo python introduces several python courses.
   </b>
  </p>
  <p class="course">
   Python is a wonderful general-purpose programming language. You can learn Python from novice to professional by tracking the following courses:
   <a class="py1" href="http://www.icourse163.org/course/BIT-268001" id="link1">
    Basic Python
   </a>
   and
   <a class="py2" href="http://www.icourse163.org/course/BIT-1001870001" id="link2">
    Advanced Python
   </a>
   .
  </p>
 </body>
</html>
>>> soup = BeautifulSoup('<p>data</p>','html.parser')
>>> print(soup.prettify())
<p>
 data
</p>

Beautiful Soup库的理解

Beautiful Soup库的引用

Beautiful Soup也叫beautifulsoup4或bs4

1
2
from bs4 import BeautifulSoup
import bs4

Beautiful Soup类

HTML/XML <-> 标签树 <-> Beautiful Soup类

1
2
3
>>> from bs4 import BeautifulSoup
>>> soup = BeautifulSoup("<html>data</html>", "html.parser")
>>> soup2 = BeautifulSoup(open("D://demo.html"), "html.parser")
  • BeautifulSoup对于一个HTML/XML文档的全部内容。

Beautiful Soup库解析器

解析器 使用方法 条件
bs4的HTML解析器 BeautifulSoup(mk,’html.parser) 安装bs4库
lxml的HTML解析器 BeautifulSoup(mk,’lxml’) pip install lxml
lxml的XML解析器 BeautifulSoup(mk,’xml’) pip install lxml
html5lib的解析器 BeautifulSoup(mk,’html5lib’) pip install html5lib

Beautiful Soup类的基本元素

基本元素 说明
Tag 标签,最基本的信息组织单元,分别用<></>标明开头和结尾
Name 标签的名字,<p>…</p>的名字是’p’,格式:<tag>.name
Attributes 标签的属性,字典形式组织,格式:<tag>.attrs
NavigableString 标签内非属性字符串,<>…</>中字符串,格式<tag>.string
Comment 标签内字符串的注释部分,一种特殊的Comment类型 
1
2
3
4
5
<p>..</p>:标签tag

<p class='title'>...</p>
 ↑    ↑
名称  属性
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from bs4 import BeautifulSoup
import requests
r = requests.get("http://python123.io/ws/demo.html")
demo = r.text
soup = BeautifulSoup(demo, 'html.parser')
print(soup.a.name)
print(soup.a.parent.name)
print(soup.a.parent.parent.name)
tag = soup.a
print(tag.attrs)
print(tag.attrs['class'])
print(tag.attrs['href'])
print(type(tag.attrs))
print(type(tag))
print(tag.string)
print(soup.p)
print(soup.p.string)
print(type(soup.p.string))

标签树的下行遍历

属性 说明
.contents 子节点的列表，将所有儿子结点存入列表
.children 子节点的迭代类型，与.contents类似，用于循环遍历
.descendants 子孙节点的迭代类型，包含所有子孙节点，用于循环遍历 
1
2
3
4
5
for child in soup.body.children:   # 遍历儿子节点
	print(child)
	
for child in soup.body.descendants:	# 遍历子孙节点
	print(child)

标签树的上行遍历

属性 说明
.parent 节点的父亲标签
.parents 节点的先辈标签的迭代类型，用于循环遍历先辈节点 
1
2
3
4
5
6
7
>>> soup = BeautifulSoup(demo, "html.parser")
>>> for parent in soup.a.parents:
 		 if parent is None:
 				print(parent)
 		 else:
	 			print(parent.name)
# 遍历所有先辈节点，包括soup本身，所以要区别判断

标签树的平行遍历

属性 说明
.next_sibling 返回按照HTML文本顺序的下一个平行节点标签
.previous_sibling 返回按照HTML文本顺序的上一个平行节点标签
.next_siblings 迭代类型，返回按照HTML文本顺序的后续所有平行节点标签
.previous_siblings 迭代类型，返回按照HTML文本顺序的前续所有平行节点标签 
1
2
3
4
5
for sibling in soup.a.next_siblings: # 遍历后续节点
	print(sibling)
	
for sibling in soup.a.previous_siblings:  # 遍历前续节点
	print(sibling)
Buy me a coffe. XD