浏览器抓取网页( 这个是想要跳过这个，直接把你最终打开的页面里的cookie拷贝出来回答)

　　浏览器抓取网页(

这个是想要跳过这个，直接把你最终打开的页面里的cookie拷贝出来回答)

　　

　　这就是我要抢的网站

　　答案：

　　在你最终直接打开的页面中复制cookie

　　import requests

headers = {

'Cookie': '__cfduid=d1e0cc7faadf90af5d64ff45aa2aba92c1503910637; cf_clearance=37fa4932d1e2dde34e9074b0f03aabb0d5f7e8ab-1503910642-31536000',

'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36'

}

url = 'http://www.agronet.com.cn/'

r = requests.get('http://www.agronet.com.cn/', headers=headers)

r.encoding = 'utf'

print r.text

　　答案：

　　访问域名时，做了302重定向，

　　Request URL:http://www.agronet.com.cn/cdn-cgi/l/chk_jschl?jschl_vc=d8a99f573078471437148bcdc30097b1&pass=1503973845.042-jbbgNU0opS&jschl_answer=3984273

Request Method:GET

Status Code:302 Moved Temporarily

Remote Address:119.147.134.219:80

Referrer Policy:no-referrer-when-downgrade

Response Headers

view source

CF-RAY:395c1714639389e9-SZX

Connection:keep-alive

Content-Length:165

Content-Type:text/html

Date:Tue, 29 Aug 2017 02:30:45 GMT

Location:http://www.agronet.com.cn/

Server:yunjiasu-nginx

Set-Cookie:cf_clearance=df2e88c103783c5b5a9524bbd4105fbd74b4b4cd-1503973845-31536000; path=/; expires=Wed, 29-Aug-18 03:30:45 GMT; domain=.agronet.com.cn; HttpOnly

X-Frame-Options:SAMEORIGIN

　　因为处理302的参数未知，所以使用request.Session来处理请求

　　答案：

　　对于python，有一个很好的库：requests

　　可以解决301、302跳转、UA、cookie等问题，

　　文档地址：...

　　答案：

　　如果你要爬取的数据不是js执行生成的，可以使用以下方法

　　import requests

session = requests.Session()

headers = {"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Firefox/52.0","Connection":"close","Accept-Language":"zh-CN,zh;q=0.8,en-US;q=0.5,en;q=0.3","Accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","Upgrade-Insecure-Requests":"1"}

cookies = {"safedog-flow-item":"A7C62592B3B3C821E8940811D417910B","__utmz":"63149887.1503977571.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)","ctrl_time":"1","Hm_lvt_8f396c620ee16c3d331cf84fd54feafe":"1503977516","__utmt":"1","__cfduid":"d61c*敏*感*词*43d7df1327ad4ee0c17da83adcf1503977565","Hm_lpvt_8f396c620ee16c3d331cf84fd54feafe":"1503977628","cf_clearance":"6c62a4b3214679cbbf0e76f7feb17fcbe1e9955a-1503977569-31536000","agronetCookie":"D98233C159EF3AEA333D883B971C1FCA9B6F08A7476FD809","__utmb":"63149887.2.10.1503977571","bdshare_firstime":"1503977570334","__utmc":"63149887","yjs_id":"Mozilla%2F5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010.12%3B%20rv%3A52.0)%20Gecko%2F20100101%20Firefox%2F52.0%7Cwww.agronet.com.cn%7C1503977570020%7Chttp%3A%2F%2Fwww.agronet.com.cn%2F","__utma":"63149887.720670875.1503977571.1503977571.1503977571.1"}

response = session.get("http://www.agronet.com.cn/", headers=headers, cookies=cookies)

print("Status code: %i" % response.status_code)

print("Response body: %s" % response.content)

　　如果数据是js生成的，建议使用webdriver

　　答案：

　　如果 cookie 是动态变化的呢？

　　本文地址：H5W3 » 爬虫爬取网页，返回的页面是，请打开浏览器的javascript，然后刷新浏览器。如何解决这个问题。要跳过这个，请抓住 网站