scrapy-ctl(1) Python Scrapy control script


scrapy-ctl [command] [OPTIONS] ...


Scrapy is controlled through the scrapy-ctl control script. The script provides several commands, for different purposes. Each command supports its own particular syntax. In other words, each command supports a different set of arguments and options.


fetch [OPTION] URL

Fetch a URL using the Scrapy downloader
Print response HTTP headers instead of body

runspider [OPTION] spiderfile

Run a spider
Store scraped items to FILE in XML format

settings [OPTION]

Query Scrapy settings
Print raw setting value
Print setting value, intepreted as a boolean
Print setting value, intepreted as an integer
Print setting value, intepreted as an float
Print setting value, intepreted as an float
Print initial setting value (before loading extensions and spiders)

shell URL | file

Launch the interactive scraping console

startproject projectname

Create new project with an initial project template

--help, -h

Print command help and options


Print Scrapy version and exit


Log file. if omitted stderr will be used

--loglevel=LEVEL, -L LEVEL

Log level (default: None)


Disable logging completely


Always use this spider when arguments are urls


Write python cProfile stats to FILE


Write lsprof profiling stats to FILE


Write process ID to FILE


Set/override setting (may be repeated)


Python path to the Scrapy project settings


Scrapy-ctl was written by the Scrapy Developers <[email protected]>.

This manual page was written by Ignace Mouzannar <[email protected]>, for the Debian project (but may be used by others).