SYNOPSIS
clive [options]... [URL]...DESCRIPTION
clive is a command line video extraction utility for Youtube and other similar video-sharing websites. It was written to work around the Adobe Flash plugin requirement as the technology is poorly supported on Unix-like systems.clive is not an universal video extraction utility. In fact, it supports only a number of video websites. Each website typically exposes access to the video content in a very different way, meaning that clive has to be customized for each website in order to download any videos from them.
OPTIONS
-h, --help print help and exit -v, --version print version and exit --hosts print supported hosts and exit --upgrade-config upgrade 2.0/2.1 config to 2.2+ format -l, --last recall last input --last-file=FILE read/write FILE instead of default path Output Options: --emit-csv emit video details in CSV to stdout --debug print cURL debug messages -q, --quiet turn off all output --stderr redirect all output to stderr --print-fname print filename before download starts HTTP Options: --agent=STRING identify as STRING to http server --connect-timeout=SECS max time allowed connection to take --connect-timeout-socks=S same as above, tries to workaround SOCKS --proxy=ADDR use ADDR for http proxy --no-proxy disable all use of http proxy --cookie-jar=FILE enable cookies, write them to FILE Cache Options: --cache-file=FILE read/write FILE instead of default path -r, --cache-read enable reading from cache -d, --cache-dump dump cache records to stdout --cache-dump-format=STRING print format for dumping cache records --cache-grep=PATTERN grep cache records for PATTERN -i, --cache-ignore-case ignore case while matching records -D, --cache-remove-record remove matched records from cache --cache-clear truncate cache records --no-cache disable cache all (read and write) use Download Options: -f, --format=FORMAT extract FORMAT of video -O, --output-file=FILE write video to file -c, --continue continue partially downloaded file -n, --no-extract do not extract any videos --save-dir=DIR save video files to DIR --cclass=CLASS use character CLASS to filter titles -C, --no-cclass do not apply character class --filename-format=STRING use STRING to format output filename --exec=CMD command to run when transfer finishes -e, --exec-run invoke command defined with --exec --stream-exec=CMD stream command to be invoked --stream=PERCENT invoke --stream-exec when transfer reaches % -s, --stream-pass pass video link to --stream-exec command --limit-rate=AMOUNT limit transfer rate to AMOUNT (KB/s) --stop-after=SIZE|PERCENT stop file transfer after SIZE or PERCENT
OPTION SYNTAX
You may freely mix different option styles and specify options after the command line arguments, e.g.:% clive -c URL --format=best
You may also put several options together that do not require arguments:
% clive -cnrf best URL
Note that the ``dashed'' options have aliases. For example:
% clive --no-extract --no_extract --noextract
% clive --cache-read --cache_read --cacheread
OPTION DESCRIPTIONS
- -h, --help
- Print help and exit.
- -v, --version
- Print version and exit.
- --hosts
- Print supported hosts with available formats and exit.
- --upgrade-config
- Upgrade clive 2.0/2.1 config to current 2.2+ format and exit.
- -l, --last
- Re-feed the previously fed video page links from the last run time.
- --last-file=path
- Use path instead of the default path. See also ``FILES''.
Output options
- --emit-csv
- Print (or emit) video details in CSV format to standard output. Implies --no-extract.
- --print-fname
- Print filename on a separate line before download starts.
- --debug
- Print cURL debug (or verbose) messages to standard error.
- -q, --quiet
- Turn off all output to standard output and error.
- --stderr
- Direct all output to standard error.
HTTP Options
- --agent=string
- Identify clive as string to HTTP servers. Defaults to ``Mozilla/5.0''.
- --connect-timeout=seconds
- Maximum time in seconds allowed for connection to take. Defaults to 30.
- --connect-timeout-socks=seconds
- Same as above but tries to workaround the SOCKS proxy bug in cURL. Defaults to 30.
- --proxy=address
- Use address for HTTP proxy. Example: ``http://foo:1234''.
- --no-proxy
- Disable all use of HTTP proxy, even if http_proxy environment variable is set.
- --cookie-jar=file
- Enable cookies, which are otherwise rejected by default, and have libcurl to write them to file. Specify ``-'' to instead to have the cookies written to stdout.
Cache Options
- --cache-file=path
- Use path instead of the default path. See ``FILES''.
- -r, --cache-read
- Read video details from cache record if it exists. Allows clive to skip video page fetching and parsing again. See ``CACHE'' section for more on this.
- -d, --cache-dump
- Dump cache records to standard output.
- --cache-dump-format=format-string
-
Used to format the output of the above. Defaults to ``%n: %t [%f, %mMB]''.
Example:
% clive --cache-dump --cache-dump-format=``%d: %t''Supported format specifiers:
%t .. video page title
%i .. video id
%h .. video host
%l .. video file length (bytes)
%m .. video file length (MB)
%d .. date (last update)
%T .. time (last update)
%s .. time stamp (same as ``%d %T'')
%f .. video file format
%n .. index - --cache-grep=pattern
- Grep stored cache records for pattern. See also ``EXAMPLES - ADVANCED USE''.
- -i, --cache-ignore-case
- Ignore case-differences while matching records.
- -D, --cache-remove-record
- Remove matched records from cache.
- --cache-clear
- Truncate cache records.
- --no-cache
- Disable all (read and write) cache use.
Download Options
- -f, --format=format
-
Download the format of the video. If format is set to best, clive
will attempt to download the best quality of the video.
Note that the format is strictly host specific. See the ``FORMATS'' section for more on this.
- -n, --no-extract
- Do not extract the video. In other words: simulate only to the point that clive verifies the video link after fetching and parsing the video page.
- -O, --output-file=file
-
Write video to file. Overwrites an already existing file.
Do not use this option when you are downloading more than one video
on one go.See also the note below.
- -c, --continue
-
Continue partially downloaded video file.
Note that, by default, clive appends a numeric suffix to the filename if the file exists already. That is unless:
* file is already completely retrieved, or: * -c or -O is used
- --save-dir=dir
- Save extracted videos to dir. clive defaults to the current working directory.
- --cclass=class
- Use character-class to filter video page titles. Defaults to ``\w''. This is a Perl regular expression character class. For example: ``[A-Za-z0-9]''.
- -C, --no-cclass
- Disables the use of --cclass. Causes clive to use the video page title as it is for output filename.
- --filename-format=format-string
-
Use format-string to format output video filenames. Default is ``%t.%s''.
Supported format specifiers:
%t .. video page title (after applying character-class filter)
%s .. video file suffix (e.g. ``flv'')
%i .. video id
%h .. video host - --exec=command;
-
Defines the command to run when video file transfer completes.
Note that --exec-run must be used to actually cause clive
to invoke the defined command.
Optional arguments may be passed to the command. The expression must be terminated by a semicolon (``;''). If the specifier ``%i'' appears anywhere in the command, it is replaced by the pathname of the extracted video file.
- --exec=command+
- Same as above but ``%i'' is replaced with as many path names as possible for the invocation of command.
- -e, --exec-run
- Causes clive to invoke the command defined with --exec when transfer finishes.
- --stream-exec=command
- Define the command to be invoked with --stream and --stream-pass. If a ``%i'' specifier is used in the command, it will be replaced with either video file path name (--stream) or parsed video link (--stream-pass).
- --stream=percent
-
Causes clive to fetch, parse, start download and eventually invoke
the command defined with --stream-exec when the transfer reaches
the percentage defined with this option. See also ``EXAMPLES''.
Note that clive does nothing to check if there is enough data buffered before invoking the --stream-exec defined command. If the transfer rate drops significantly after starting the process and it runs out of data, clive does nothing to fix this.
Also note that clive will not continue to download another file before the child process exits.
This mode is supported for historical reasons. Consider using --stream-pass instead.
- -s, --stream-pass
-
Otherwise the same as above but instead of starting the download,
clive passes the parsed video link to the command defined with
--stream-exec. See also ``EXAMPLES''.
This option was inspired by a clive(1) wrapper script contributed by Bill Squire.
- --limit-rate=amount
- Limit transfer rate to amount KB/s.
- --stop-after=size|percent
- Stop file transfer after size or percent. The value must be terminated by either '%' or 'M'.
EXAMPLES - BASIC USE
- clive "http://youtube.com/watch?v=3HD220e0bx4"
- Extracts video (flv) from the above video page link. You can then play the flv video file in a media player.
- cat > url.lst
-
http://en.sevenload.com/videos/IUL3gda-Funny-Football-Clips http://youtube.com/watch?v=3HD220e0bx4 http://break.com/index/beach-tackle-whip-lash.html http://www.liveleak.com/view?i=704_1228511265
- cat url.lst | clive
- You can feed clive multiple video page links like this or as command line arguments.
- clive URL1 URL2 URL3 URL4
- When you are using the pipes, be sure to separate each link with a newline.
- xclip -o | clive
- There are many X clipboard utilities. The above example uses xclip(1) and a pipe to paste (or feed) the contents to clive.
- clive -l
- Recall last video page link input. Regardless the way they were fed to clive.
EXAMPLES - ADVANCED USE
- clive -f best "http://youtube.com/watch?v=3HD220e0bx4"
- Extract the best format of the video.
- clive -r -f best "http://youtube.com/watch?v=3HD220e0bx4"
- Same as above but read the cache record without fetching and parsing the video page again.
- clive --cache-dump
- Dump all cache records to stdout. You can use --cache-dump-format to format the output.
- clive -ig 3hd2
- Grep for ``3hd2'' pattern in cache records. If pattern matches, clive continues to extract the matched videos. Note the use of ``-i'' (--cache-ignore-case).
- clive -ig 3hd2 -D
- Same as above but removes the record from cache instead of extracting the video.
- clive --exec="ffmpeg -i %i -acodec libvorbis %i.ogg;" -e URL
- Copy audio from downloaded video to ogg with ffmpeg(1).
- clive --stream-exec="mplayer -really-quiet %i" --stream=25 URL
- Start playing the video being extracted when the transfer reaches 25% complete.
- echo '--stream-exec="mplayer -really-quiet %i"' >> ~/.cliverc
- clive -s URL
- Alternative to Adobe Flash. vlc(1) and totem(1) have been reported to work also.
FORMATS
clive downloads ``flv'' by default from all of the supported websites.- youtube.com
- last.fm
-
Format: (flv|fmt17|fmt18|fmt22|fmt35)
If --format option is not unused, clive defaults to whatever Youtube defaults to. Technically speaking, we leaving the ``&fmt='' from the video link.
Youtube likes to rehash these from time to time so don't be surprised if, for example, the quality is not what you expected. The same applies to the suffices listed below.
YoutubeID Alias Suffix Resolution fmt22 hd mp4 1280x720 fmt35 hq flv 640x380 fmt18 mp4 mp4 480x360 fmt34 - flv 320x180 (quality reportedly varies) fmt17 3gp 3gp 176x144
You can use either, the ``alias'' (e.g. ``hd'') or the ``YoutubeID'' (e.g. ``fmt22'') with --format. The aliases exist for historical reasons. The suffix is parsed from the content-type field of the returned HTTP header.
clive can also download videos that last.fm lists as Youtube hosted videos.
- video.google.com
-
Format: (flv|mp4)
mp4 format is available for a limited number of videos.
- dailymotion.com
-
Format: (flv|hq|hd)
The HD and HQ videos may not always be available.
hd (1280x720) hq (848x480) flv (320x240) aka "sd"
- spiegel.de
-
Format: (flv|vp6_(64|576|928)|h264_1400)
h264_1400 .. mp4 (996x560) vp6_928 .. flv (996x560) vp6_576 .. flv (560x315) flv .. flv (180x100) vp6_64 .. flv (180x100)
Format: (3gp|small|iphone|podcast)
The data that clive parses indicates that these formats should be available although we are yet to find a video with these formats available. If you find one, let us know, too.
3gp .. 3gp (?) small .. 3gp (?) iphone .. mp4 (?) podcast .. mp4 (?)
- golem.de
- Format: (flv|high|ipod)
- vimeo.com
-
Format: (flv|hd)
HD should be available for the vimeo.com/hd channel videos at least. Note that ``flv'' only means the ``default flv''. Some of the hosted ``default'' videos are actually ``mp4'', not ``flv''.
For further reading:
http://vimeo.com/help/hd - Other
- All other supported websites (see --host output) support the flv format only.
FILES
Should HOME environment variable be undefined for some reason, clive will use the current working directory instead.- $HOME/.cliverc, $HOME/.clive/config, $HOME/.config/clive/config
-
User configuration file. For example:
% cat >> ~/.cliverc
-f best
--proxy=http://foo:1234 - $HOME/.cache/clive/last
-
File containing the last user input (video page links).
You can use --last-file to override the path, e.g.:
--last-file=/path/to/last/file.You can also define this option in the config file.
See also CLIVE_CACHE notes below.
- $HOME/.cache/clive/cache
-
BerkeleyDB based cache file containing the records of fetched
and parsed video pages.
You can use --cache-file to override the path., e.g.:
--cache-file=/path/to/cache/file.You can also define this option in the config file.
See also CLIVE_CACHE notes below.
- Notes: CLIVE_CACHE
-
clive defaults to use $HOME/.cache/clive/ for ``last'' and
``cache'' files described above.
The use of the default path can be overridden by CLIVE_CACHE environment variable. Note that clive will attempt to create the specified path recursively.
Examples:
setenv CLIVE_CACHE /home/user/cachedata (in csh terms)
clive # will read/write /home/user/cachedata/(last|cache)unsetenv CLIVE_CACHE clive # read/write $HOME/clive/(last|cache) clive --last-file=mylast --cache-file=cachedata/mycache # read/write "mylast" file, read/write cachedata/mycache file
CACHE
The purpose of the cache is to allow clive to skip fetching and parsing the video page again. It does not contain any actual video data so one should not expect to recover a deleted video file from the cache. Only some of the parsed details are stored as records to the cache.By now, it is should be a well known fact that the cache fails with some of the supported hosts. For example Youtube video links expire after some time, this causes the re-extraction to fail if the cached video link is used later again.
This was the main reason why in 2.2.0 reading from cache was disabled by default. Many users reported the reuse of expired video links as a bug previously even though it was well documented in the manual page explaining that most of the HTTP 403/404 errors were actually caused by expired video links.
It is, of course, still possible to read from cache. You can enable this by invoking the --cache-read option. This causes clive to look up a saved cache record and reuse the stored video details if they are found instead of fetching the video page.
The use of the cache can be disabled with the --no-cache option. This disables both read and write. Note that if the BerkeleyDB Perl module is not installed, clive will not use the cache.
See also the --cache-grep option.
UNICODE
Q: Why am I seeing mangled video filenames?
A: Make sure you have set appropriate locale. For example (in csh/urxvt terms):
% setenv LANG en_US.UTF-8
% urxvt &
You can get a list of supported locales on your typical Unix-like system with:
% locale -a
DEBUGGING
Some tips that we have found useful:
% clive --debug URL
Causes libcurl to be verbose.
% clive -n URL
Simulates video extraction only.
BUGS
Sure to be some.
Please report them:
<http://code.google.com/p/clive/issues/>
EXIT STATUS
clive exits 0 on success, and >0 if an error occurs.
CLIVE_OK = 0 CLIVE_NOTHINGTODO = 1 # file already retrieved CLIVE_NOSUPPORT = 2 # host not supported CLIVE_READ = 3 # file open/read error CLIVE_GREP = 4 # grep: nothing matched in cache CLIVE_OPTARG = 5 # invalid option argument CLIVE_SYSTEM = 6 # system call failed (e.g. fork) CLIVE_REGEXP = 7 # regexp pattern matching failed CLIVE_FORMAT = 8 # requested format unavailable CLIVE_NET = 9 # network error CLIVE_STOP = 10 # --stop-after CLIVE_MARKEDBROKEN = 11 # support marked broken
OTHER
Project page:<http://clive.googlecode.com/>
Front-end (GUI):
<http://abby.googlecode.com/>
Development code:
% git clone git://repo.or.cz/clive.git
HISTORY
* Originally written in Python * Rewritten in Perl for 2.0.0
AUTHOR
Toni Gundogdu <[email protected]>Thanks to all those who have contributed to the project by sending patches, reporting bugs and writing feedback. You know who you are.