实习僧

一：前言

继续练习Scrapy框架，这次抓取的是实习僧网最新的招聘信息，包括招聘岗位，时间，工资，学历要求，职位诱惑和职位描述等等。之后保存到mongodb和json文件中以备后续使用。爬虫中遇到了很多问题，比如不同解析函数传值、xpath配合正则、职位描述的网页结构多变，这些都一一解决了。代码地址：https://github.com/rieuse/ScrapyStudy

二：运行环境

• IDE：Pycharm 2017
• Python3.6
• pymongo 3.4.0
• scrapy 1.3.3

三：实例分析

1.首先进入官网：shixiseng.com 然后点击进入更多职位后就可以看见下面这样网页。

招聘主页

start_urls = ['http://www.shixiseng.com/interns?t=zj&p={}'.format(n) for n in range(1, 501)]

2.知道主网址构成后，就进入开发者模式看看每个招聘信息的具体网址。这里只是一个不含有域名的链接，那么就需要自己构成对应详细的招聘信息网址

links = response.xpath('//div[@class="po-name"]/div[1]/a/@href').extract()
for link in links:
    dlink = 'http://www.shixiseng.com' + link

网址

        item['name'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[1]/span[1]/span/text()').extract_first()
        item['link'] = response.meta['link']
        item['company'] = response.xpath('//*[@id="container"]/div[1]/div[2]/div[1]/p[1]/a/text()').extract_first()
        item['place'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[2]/@title').extract_first()
        item['education'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[3]/text()').re_first(r'[\u4e00-\u9fa5]+')
        item['people'] = response.xpath('//*[@id="container"]/div[1]/div[2]/div[1]/p[2]/span/text()').extract()[1]
        item['money'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[1]/text()').re_first(r'[^\s]+')
        item['week'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[4]/text()').extract_first()
        item['month'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[5]/text()').extract_first()
        item['lure'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/p/text()').extract_first()
        item['description'] = response.xpath('//div[@class="dec_content"]/*/text()|//div[@class="dec_content"]/text()').extract()
        item['data'] = response.xpath('//*[@id="container"]/div[1]/div[1]/p[3]/text()').extract()

数据分布

四：实战代码

前面已经完成了网页结构分析和所需数据的抓取方法，下面就开始使用Scrapy框架来完成这次任务。完整代码位置：github.com/rieuse/ScrapyStudy
1.首先使用命令行工具输入代码创建一个新的Scrapy项目，之后创建一个爬虫。

scrapy startproject ShiXiSeng
cd ShiXiSeng\ShiXiSeng\spiders
scrapy genspider shixi shixiseng.com

2.打开ShiXiSeng文件夹中的items.py，改为以下代码，定义我们爬取的项目。

import scrapy


class ShixisengItem(scrapy.Item):
    name = scrapy.Field()
    link = scrapy.Field()
    company = scrapy.Field()
    people = scrapy.Field()
    place = scrapy.Field()
    education = scrapy.Field()
    money = scrapy.Field()
    week = scrapy.Field()
    month = scrapy.Field()
    lure = scrapy.Field()
    description = scrapy.Field()
    data = scrapy.Field()

3.配置middleware.py配合settings中的User_Agent设置可以在下载中随机选择UA有一定的反ban效果，在原有代码基础上加入下面代码。这里的user_agent_list可以加入更多。

import random
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware


class RotateUserAgentMiddleware(UserAgentMiddleware):
    def __init__(self, user_agent=''):
        self.user_agent = user_agent

    def process_request(self, request, spider):
        ua = random.choice(self.user_agent_list)
        if ua:
            print(ua)
            request.headers.setdefault('User-Agent', ua)

    user_agent_list = [
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1"
        "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

    ]

4.明确一下目标，这是抓取的数据保存到mongodb数据库中和本地json文件。所以需要设置一下Pipelines.py

import json
import pymongo
from scrapy.conf import settings


class ShixisengPipeline(object):
    def __init__(self):
        self.client = pymongo.MongoClient(host=settings['MONGO_HOST'], port=settings['MONGO_PORT'])
        self.db = self.client[settings['MONGO_DB']]
        self.post = self.db[settings['MONGO_COLL']]

    def process_item(self, item, spider):
        postItem = dict(item)
        self.post.insert(postItem)
        return item


class JsonWriterPipeline(object):
    def __init__(self):
        self.file = open('shixi.json', 'w', encoding='utf-8')

    def process_item(self, item, spider):
        line = json.dumps(dict(item), ensure_ascii=False) + "\n"
        self.file.write(line)
        return item

    def spider_closed(self, spider):
        self.file.close()

5.然后setting.py里面也要修改一下，这样才能启动Pipeline相关配置，最后可以保存相关数据。

BOT_NAME = 'ShiXiSeng'
SPIDER_MODULES = ['ShiXiSeng.spiders']
NEWSPIDER_MODULE = 'ShiXiSeng.spiders'

# 配置mongodb
MONGO_HOST = "127.0.0.1"  # 主机IP
MONGO_PORT = 27017  # 端口号
MONGO_DB = "Shixiseng"  # 库名
MONGO_COLL = "info"  # collection名

# pipeline文件的入口,这里进
ITEM_PIPELINES = {
    'ShiXiSeng.pipelines.JsonWriterPipeline': 300,
    'ShiXiSeng.pipelines.ShixisengPipeline': 300,
}

# 设置随机User_Agent
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
    'ShiXiSeng.middlewares.RotateUserAgentMiddleware': 400,
}

ROBOTSTXT_OBEY = False  # 不遵循网站的robots.txt策略
DOWNLOAD_DELAY = 1  # 下载同一个网站页面前等待的时间，可以用来限制爬取速度减轻服务器压力。
COOKIES_ENABLED = False  # 关闭cookies

6.这次最重要的部分就是spider中的shixi.py

# -*- coding: utf-8 -*-
import scrapy
from ShiXiSeng.items import ShixisengItem
from scrapy import Request


class ShixiSpider(scrapy.Spider):
    name = "shixi"
    allowed_domains = ["shixiseng.com"]
    start_urls = ['http://www.shixiseng.com/interns?t=zj&p={}'.format(n) for n in range(1, 501)]
    headers = {
        'Host': 'www.shixiseng.com',
        'Connection': 'keep-alive',
        'Cache-Control': 'max-age=0',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        'Upgrade-Insecure-Requests': '1',
        'Referer': 'http://www.shixiseng.com',
        'Accept-Encoding': 'gzip,deflate,sdch',
        'Accept - Language': 'zh-CN,zh;q=0.8,en;q=0.6'
    }

    def parse(self, response):
        links = response.xpath('//div[@class="po-name"]/div[1]/a/@href').extract()
        for link in links:
            dlink = 'http://www.shixiseng.com' + link
            yield Request(dlink, meta={'link': dlink}, headers=self.headers, callback=self.parser_detail)

    def parser_detail(self, response):
        item = ShixisengItem()
        item['name'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[1]/span[1]/span/text()').extract_first()
        item['link'] = response.meta['link']
        item['company'] = response.xpath('//*[@id="container"]/div[1]/div[2]/div[1]/p[1]/a/text()').extract_first()
        item['place'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[2]/@title').extract_first()
        item['education'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[3]/text()').re_first(r'[\u4e00-\u9fa5]+')
        item['people'] = response.xpath('//*[@id="container"]/div[1]/div[2]/div[1]/p[2]/span/text()').extract()[1]
        item['money'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[1]/text()').re_first(r'[^\s]+')
        item['week'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[4]/text()').extract_first()
        item['month'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/span[5]/text()').extract_first()
        item['lure'] = response.xpath('//*[@id="container"]/div[1]/div[1]/div[2]/p/text()').extract_first()
        item['description'] = response.xpath('//div[@class="dec_content"]/*/text()|//div[@class="dec_content"]/text()').extract()
        item['data'] = response.xpath('//*[@id="container"]/div[1]/div[1]/p[3]/text()').extract()[0]
        yield item

①这里面涉及到了不同解析函数的传值问题，首先是从start_urls开始请求每一个链接然后得到的respose传给parser()函数进行第一次解析，这个函数的作用就是获取该页面上的每一个招聘页面的链接，之后yield一个Request()函数，使用meta这个参数即可传值，类型是dict，这里就把链接传给了parser_detail() 函数然后做进一步的解析获取最后想要的数据。
②使用了正则+xpath来获取准确的数据

五：总结

运行代码后就会获取实习僧最新的全部的招聘数据，一共大概是5000条数据很快就可以抓取下来保存到mongodb数据库和本地的json文件中。
代码都放在github中了，有喜欢的朋友可以点击 start follw，欢迎一起学习交流。**https://github.com/rieuse **

加油