scrapy

all

Web-crawling framework.

Examples (7)

Create a project

scrapy startproject project_name

Create a spider (in project directory)

scrapy genspider spider_name website_domain

Edit spider (in project directory)

scrapy edit spider_name

Run spider (in project directory)

scrapy crawl spider_name

Fetch a webpage as Scrapy sees it and print the source to `stdout`

scrapy fetch url

Open a webpage in the default browser as Scrapy sees it (disable JavaScript for extra fidelity)

scrapy view url

Open Scrapy shell for URL, which allows interaction with the page source in a Python shell (or IPython if available)

scrapy shell url