scrapy-ctl(1) Python Scrapy control script

SYNOPSIS

scrapy-ctl [command] [OPTIONS] ...

DESCRIPTION

Scrapy is controlled through the scrapy-ctl control script. The script provides several commands, for different purposes. Each command supports its own particular syntax. In other words, each command supports a different set of arguments and options.

OPTIONS

fetch [OPTION] URL

Fetch a URL using the Scrapy downloader
--headers
Print response HTTP headers instead of body

runspider [OPTION] spiderfile

Run a spider
--output=FILE
Store scraped items to FILE in XML format

settings [OPTION]

Query Scrapy settings
--get=SETTING
Print raw setting value
--getbool=SETTING
Print setting value, intepreted as a boolean
--getint=SETTING
Print setting value, intepreted as an integer
--getfloat=SETTING
Print setting value, intepreted as an float
--getlist=SETTING
Print setting value, intepreted as an float
--init
Print initial setting value (before loading extensions and spiders)

shell URL | file

Launch the interactive scraping console

startproject projectname

Create new project with an initial project template

--help, -h

Print command help and options

--version

Print Scrapy version and exit

--logfile=FILE

Log file. if omitted stderr will be used

--loglevel=LEVEL, -L LEVEL

Log level (default: None)

--nolog

Disable logging completely

--spider=SPIDER

Always use this spider when arguments are urls

--profile=FILE

Write python cProfile stats to FILE

--lsprof=FILE

Write lsprof profiling stats to FILE

--pidfile=FILE

Write process ID to FILE

--set=SET

Set/override setting (may be repeated)

--settings=MODULE

Python path to the Scrapy project settings

AUTHOR

Scrapy-ctl was written by the Scrapy Developers <[email protected]>.

This manual page was written by Ignace Mouzannar <[email protected]>, for the Debian project (but may be used by others).