scrapy分页抓取网页(scrapy分页抓取网页(.cfg)中文路径_光明网)

　　scrapy分页抓取网页(scrapy分页抓取网页(.cfg)中文路径_光明网)

　　scrapy分页抓取网页一、准备工作首先准备以下三个文件：：#scrapy.cfg中文路径：/root/scrapy/site_scrapy/#site_scrapy中文路径：/root/scrapy/site_scrapy/#scrapy.sh中文路径：/root/scrapy/site_scrapy/.bash_profile从上图可知scrapy.sh并不在scrapy文件夹下，而是直接保存在scrapy目录中的目录文件中，所以之前我们直接使用scrapy.sh在scrapy目录下的conf.py中写代码的时候，会报错：>>>importscrapy>>>scrapy.headers.user-agentscrapy.startproject("scrapy",project_name="scrapy_inspector")首先，我们要把把false改成true。

　　这样就直接能把scrapy.startproject("scrapy",project_name="scrapy_inspector")执行成功。二、webpack与promise.jswebpack环境搭建已经写过，在这里就不再赘述。总之，在此处，webpack会把所有plugins文件打包到scrapy.cfg中，修改scrapy.cfg文件就能让webpack将下面的plugins子节点打包进去。

　　下面来看下webpack打包进去之后，剩下的东西。1.下载工具集webpack-generator，下载地址：webpack-generator-1.5.0-snapshot-env.zip下载后解压到venv/lib/webpack.cfg文件中。再修改webpack.cfg文件如下：webpack.cfg{entry:"./common.js",directory:["local/src/common.js"],loaders:["style-loader","text-loader","less","sass","scss","sass-loader","style-loader","style-scheme","outline-prettier","webpack-dev-server","scss-loader","less","sass-loader","style-scheme","transform-sass","transform-scss","sass-loader","babel-loader","multiple-sources","sass-plugin","style-loader","jsx","css-selector","less","style-loader","esm","less-loader","xml-selector","xslt","sass-loader","babel-jsx","babel-loader","jstl","jstl-script","test-script","commonjs","less","sass-loader","less-loader","babel-loader","esm","babel-loader","less-loader","less-loader","eslint","tslint","t。