SYNOPSIS

# A complete screen-scraper and RSS generator here:

use strict;
use XML::RSS::SimpleGen;
my $url = q<http://www.exile.ru/>;

rss_new( $url, "eXile", "Moscow-based Alternative Newspaper" );
rss_language( 'en' );
rss_webmaster( '[email protected]' );
rss_twice_daily();

get_url( $url );

while(
m{<h4>\s*<a href='/(.*?)'.*?>(.*?)</a>\s*</h4>\s*<p.*?>(.*?)<a href='/}sg
) {
rss_item("$url$1", $2, $3);
}

die "No items in this content?! {{\n$_\n}}\nAborting"
unless rss_item_count();

rss_save( 'exile.rss', 45 );
exit;

DESCRIPTION

This module is for writing RSS files, simply. It transparently handles all the unpleasant details of RSS, like proper XML escaping, and also has a good number of Do-What-I-Mean features, like not changing the modtime on a written-out RSS file if the file content hasn't changed, and like automatically removing any HTML tags from content you might pass in.

This module isn't meant to have the full expressive power of RSS; instead, it provides functions that are most commonly needed by RSS-writing programs.

INTERFACE

This module provides a bunch of functions for starting an RSS feed in memory, putting items into it, and saving it to disk (or printing it as a string, as in a CGI). If you prefer an object-oriented interface (obviously more useful if you're composing several feeds at once), then you can use this module as a class whose methods are the same as the function names minus ``rss_''. Except for this detail of the naming, the functions and methods are the same, behave the same, and take the same arguments.

That is, this functional code:

  use XML::RSS::SimpleGen;
  my $url = q<http://www.exile.ru/>;
  
  rss_new( $url, "eXile" );
  rss_language( 'en' );
  get_url( $url );
  ...

does the same work as this OO code:

  use XML::RSS::SimpleGen ();
  my $url = q<http://www.exile.ru/>;
  my $rss = XML::RSS::SimpleGen->new( $url, "eXile");
  $rss->language( 'en' );
  $rss->get_url( $url );
  ...

(Note that the function "get_url" doesn't have a leading ``rss_'', so its method name is the same as its function name. It's the one exception.)

If this talk of objects puzzles you, see HTML::Tree::AboutObjects in the "HTML-Tree" dist, and/or see the chapter ``User's View of Object-Oriented Modules'' in my book Perl & LWP (<http://www.amazon.com/exec/obidos/ASIN/0596001789>). (The book is also useful as an extended discussion of screen-scraping.)

Note: in the code below, I use the word ``accessor'' a lot, to refer to a function or method that you can call two possible ways: 1) like "foo(val)" to set the ``foo'' attribute to the value val, or 2) like "foo()" to return the value of the ``foo'' attribute.

FUNCTIONS

"rss_new( url );"

"rss_new(

url, title );"

"rss_new(

url, title, description );"

or: "$rss = XML::RSS::SimpleGen->new(...);"

This function creates a new RSS feed in memory. This should be the first "rss_whatever" function you call in your program. If you call it again, it erases the current object (if any) and sets up a new one according to whatever parameters you pass.

The parameters are the full URL, the title, and the description of the site (or page) that you're providing an RSS feed of. The description is optional, but you should provide at least a URL and title.

Examples:

  rss_new( $url, "eXile", "Moscow-based Alternative Newspaper" );
  rss_new( 'http://www.mybazouki.com/news/', "Bazouki News!" );

(As a method, XML::RSS::SimpleGen->new simply returns a new RSS object.)

the accessor "rss_language(language_tag)"

This declares what language this

RSS feed is in. It must be an RFC3066-style language tags like ``en'', or ``en-US'', or ``zh-TW''. (See I18N::LangTags::List for a list.) If you don't set the feed's language, it defaults to ``en'', for generic English.

If you call this function without a parameter, it returns the current value of the RSS feed's language. For example:

  print "I'm making an RSS feed for ", rss_language(), "!\n";

The same is true for all the functions that I label as ``accessors''.

the accessor "rss_item_limit(number)"

This sets the maximum number of items that this feed will show.

The default value is 0, meaning that there is no maximum.

If you set it to a positive number

N, then the feed will show only the first N items that you declare with "rss_item". (Or, if you set "rss_history_file", then the newest N items that you declare with "rss_item".)

If you set it to a negative number -N, then the feed will show only the last N items that you declare with "rss_item". (Or, if you set "rss_history_file", then the oldest N items you declare with "rss_item", which is unlikely to be useful!)

the accessor "rss_webMaster(email-address)"

This declares what email address you, the

RSS generator manager, can be reached at. Example:

  rss_webMaster( '[email protected]' );

"rss_history_file( filename )"

This declares that you want this

RSS feed to keep track of what items are new, and to list them first when the RSS is emitted. To do this, the RSS generator has to store information in a file, where it tracks its ``history'', i.e., when was the first time it saw given URLs, and the most recent time it saw given URLs.

Typical usage is:

  rss_history_file( 'thisrssfeed.dat' );

You should call "rss_history_file" before you make any calls to "rss_item".

The history-file feature is meant for cases where your RSS-generator program calls "rss_item" on every link it sees, but only wants the new links to appear in the RSS output. (This can be a good approach if you're making an RSS feed of a page like "http://www.guardian.co.uk/" where there's some new links (to the recently added stories), but also links to some days-old stories, and also links to some always-there things like ``Archive Search'' and ``Contact Us'' pages.

Once you call rss_history_file, the specified file is read in. The in-memory history (stored in the RSS object) is updated as you call "rss_item". But the file isn't updated until you call rss_save.

(A do-what-I-mean side effect of calling "rss_history_file" is that it sets rss_item_limit to 25 if it is currently 0.)

(Incidentally, if you're using rss_history_file as part of a CGI that emits RSS data, instead of a program that just saves to an RSS file, then things will get complicated. You'll need to call an internal method to explicitly commit the history file to disk, and you'll need a semaphore file to avoid race conditions. Email me for full info.)

"rss_item( url );"

"rss_item(

url, title );"

"rss_item(

url, title, description );"

This adds a new item to the current feed. You will need to specify the

URL to add (and it should be a valid-looking URL, starting with "something:", and not containing any spaces). You may also specify the title, but it's optional. And finally, you can optionally specify a description. (You can remember this because it starts with the essential item first, and progresses toward the most optional.)

Leading and tailing whitespace is removed from whichever of url, title, and description are defined values, and HTML is parsed out.

A simple usage:

  rss_item(
    "http://www.harpers.org/MostRecentWR.html",
    "Harper's Magazine's Weekly Review"
  );

Although in practice, a typical call won't have string constants, but will instead be like the example in the Synopsis sectios, namely:

  rss_item("$url$1", $2, $3);

Incidentally, as a do-what-I-mean feature, if the first parameter doesn't look like a URL but one of the others does, then this error is silently forgiven. This is so you can occasionally slip up and forget the order of the parameters.

(In the unlikely event where you need to avoid the HTML-removal features, you can do this by passing scalar-references instead of normal strings, like so: "rss_item($url, $title, \$not_to_be_escaped)".)

"rss_item_count()"

This returns the number of items you've declared. I anticipate that its main usage will be something like:

  die "What, no objects found at $url ?!"
   unless rss_item_count();

or, maybe...

  exit unless rss_item_count();

...depending on how/whether you'd want to react to cases where you don't see anything to put into an RSS feed.

Note that the parens are optional, since this command takes no options (just like Perl's "time()" function).

"rss_image( url, h, w );"

This declares that you want to declare a particular image as the logo for this feed. Most feeds don't have such a thing, and most readers just ignore it anyway, but if you want to declare it, this function is how. The three parameters, which are all required, are: the image's

URL, its height in pixels, and its width in pixels. According to various specs, the width should/must be between 1 and 144, an the height should/must be between 1 and 400.

A typical usage:

  rss_image("http://interglacial.com/rss/weebl.gif", 106, 140);

Be careful not to mix up the height and width.

"rss_save( filename );"

"rss_save(

filename, max_age_days );"

This saves the

RSS date to the file you specify. If the RSS data hasn't changed, the file (and its modtime) aren't altered. The optional max_age_days parameter means that if ever the file exists, and its content hasn't changed for that many days or longer, then the program should die with a warning message. For example, in the case of a screen-scraper for a site that we know should (in theory) change its content at least weekly, we might save the RSS file with:

  rss_save("whatever.rss", 17);
   # Scream if the feed is unchanged for 17 days.

The seventeen there is gotten by assuming that just maybe the site might skip two weeks for a vacation now and then, and might even put out the pre-vacation issue a few days early --- but that if ever the program notices that the data hasn't changed for 17 days, then it should emit error messages. If you want to disable this feature on a one-time basis, just change the modtime (like via "touch") on the whatever.rss file.

If you don't specify a "max_age_days" value, then this whole complain-if-it's-old feature is disabled.

"rss_as_string();"

This returns the RSS-XML data as a string. This function is called internally by the rss_save function; but you might want to call it explicitly, as in a CGI, where your CGI would probably end like this:

  print "Content-type: application/xml\n\n", rss_as_string();
  exit;

"get_url( url );"

"$content = get_url(

url );"

or: "$content = $rss->get_url(...);"

or: "$content->get_url(...);"

This tries to get the content of the given url, and returns it.

This is quite like LWP::Simple's "get" function, but with some additional features:

• If it can't get the URL's content at first, it will sleep for a few seconds and try again, up to about five times. (This is to avoid the case of the URL being temporarily inaccessible simply because the DNS is a bit slow, or because the server is too busy.)
• If it can't get the content, even after several retries, it will abort the program (like a "die"). If you want to override this behavior, then call it as "eval { get_url($url) };"
• If you call the function in void context (i.e., not using its return value), then the function assigns the URL's content to $_. That's so you can write nice concise code like this:

             get_url $thatsite;
             m/Top Stories Tonight/ or die "What, no top stories?";
             while( m{<a class="top" href="(.*?)">(.*?)</a>}g ) {
               rss_item("$thatsite/$1", $2);
             }
  

• This returns the content of the URL not exactly as-is, but after changing its newlines to native format. That is, if the contents of the URL use CR-LF pairs to express newlines, then "get_url" changes these to "\n"'s before returning the content. (Similarly for old MacOS newline format.) Clearly this is wrong in you're dealing with binary data; in that case, use LWP::Simple's "get" directly.
• Finally, as a resource-conversation measure, this function will also try to call "sleep" a few times if it sees several quick calls to itself coming from a program that seems to be running under crontab. As most of my RSS-generators are crontabbed, I find it very useful that I can have however many "get_url"'s in my crontabbed programs without worrying that they'll take even a noticeable part of the server's bandwidth.

"rss_hourly" or "rss_daily" or "rss_twice_daily" or "rss_thrice_daily" or "rss_weekly" or "rss_every_other_hour"

Calling one of these functions declares that this feed is usually generated at the same time(s) every day (or every week, in the case of "rss_weekly"). And, where it's not just once a day/week, these multiple times a day are evenly spaced. These functions then set the feed's "updatePeriod", "updateBase", "updateFrequency", "skipHours", "skipDays", and "ttl" elements appropriately, so that RSS readers can know at at what times there could (or couldn't) be new content in this feed.

In other words: use "rss_twice_daily" if this feed is updated at about the same time every day and then again 12 hours later. Use "rss_thrice_daily" if this feed is updated at the same time daily, and then 8 hours later, and then 8 hours later. And use "rss_every_other_hour" if the feed updates at about n minutes past every even numbered hour, or every odd-numbered hour.

Clearly I mean these functions to be used in programs that are crontabbed to run at particular intervals, as with a crontab line like one of these:

       52 * * * *         ~/thingy   # => rss_hourly
       52 23 * * *        ~/thingy   # => rss_daily
       52 4,16 * * *      ~/thingy   # => rss_twice_daily
       52 5,13,21 * * *   ~/thingy   # => rss_thrice_daily
       52 23 * * 3        ~/thingy   # => rss_weekly
       52 */2 * * *       ~/thingy   # => rss_every_other_hour

Clearly there aren't "rss_interval" functions for all the scheduling possibilities programs --- if you have a program that has to run at 6am, 8am, 1pm, and 4pm, there's no function for that. However, the above crontab lines (or with minor changes, like "1,9,17" instead of "5,13,21") are just fine for almost every RSS feed I've run.

An aside: I recommend running the programs at about 52 minutes past the hour, generally in series, like so:

       52 5,13,21 * * *   ~/thingy ; ~/dodad ; ~/makething ; ~/gizmo

However, your mileage may vary.

Incidentally, these functions take no arguments, so the parentheses are optional. That is, these two lines do the same thing:

       rss_hourly;
       rss_hourly();

MINOR FUNCTIONS

These are functions that you probably won't need often, or at all. I include these for the sake of completeness, and so that advanced users might find them useful in some cases.

"rss_skipHours( gmt_hour_num, gmt_hour_num, ... );"

This function directly sets the "skipHours"

element's values to the specified GMT hour numbers.

"rss_updateHours();"

"rss_updateHours( gmt_hour_num, gmt_hour_num, ... );"

This function is a wrapper around "rss_skipHours"

--- you call "rss_updateHours" with a list of GMT hour numbers, and "rss_updateHours" will call "rss_skipHours(0 .. 23)" except without whatever hour numbers you specified.

If you call with an empty list (i.e., "rss_updateHours();"), then we uses "gmtime" to find out the current hour (and rounds it up if it's after 50 minutes past), basically just as if you'd called:

      rss_updateHours( (gmtime(600+time()))[2] );

"rss_skipDays();"

"rss_skipDays( gmt_day_num, gmt_day_num, ... );"

"rss_skipDays(

gmt_day_name, gmt_day_name, ... );"

This function directly sets the "skipDays"

element's values to the specified weekdays. Note that this accepts either integers (like 6 for Saturday, Sunday being either 0 or 7), or their exact English names.

If you use the "skipDays" field, consider that it refers to days figured by GMT, not local time. For example, if I say to skip Saturdays, that means Saturdays GMT, which in my timezone (Alaska) starts in the middle of Friday afternoon.

"rss_updateDays();"

"rss_updateDays( gmt_day_num, gmt_day_num, ... );"

"rss_updateDays(

gmt_day_name, gmt_day_name, ... );"

This function is a wrapper around "rss_skipDays"

--- you call "rss_updateDays" with a list of GMT day names/numbers, and "rss_updateDays" will call "rss_skipDays(0 .. 6)" except without whatever days you specified.

If you call with an empty list (i.e., "rss_updateDays();"), then we uses "gmtime" to find out the current day (GMT!), basically just as if you'd called:

      rss_updateDays( (gmtime(600+time()))[6] );

"rss_updatePeriod( periodstring );"

This function directly sets the "sy:updatePeriod"

element's value to the period specified. You must specify one of the strings: ``yearly'', ``monthly'', ``weekly'', ``daily'', ``hourly''. I advise using ``weekly'' only if you know what you're doing, and ``yearly'', ``monthly'' only if you really know what you're doing.

"rss_updatePeriod( periodstring, int, base );"

This is a shortcut for "rss_updatePeriod(

periodstring); rss_updateFrequency(int)"

"rss_updatePeriod( periodstring, int, base );"

This is a shortcut for "rss_updatePeriod(

periodstring); rss_updateFrequency(int); rss_updateBase(base)"

"rss_updateBase( iso_date_string );"

"rss_updateBase(

epoch_time );"

This function directly sets the "sy:updateBase"

element's value to the moment specified. If you pass in an epoch time, it is converted to an ISO date string.

the accessor "rss_updateFrequency( integer );"

This function directly sets the "sy:updateFrequency"

element's value to the value specified. The value has to be a nonzero positive integer.

For example, this means that this feed updates at/by the start of every hour and 30 minutes past:

  rss_updateBase('2000-01-01T00:00-00:00');
  rss_updateFrequency(2);
  rss_updatePeriod('hourly');  # 2*hourly means "twice an hour"

Recall that this can also be done with the the "rss_updatePeriod( per, freq, base )" shortcut, like so:

  rss_updateBase('hourly', 2, '2000-01-01T00:00-00:00');

the accessor "rss_retention(number)"

If you are using an "rss_history_file(

file)", the history file will accrete a list of all URLs it has seen. But to keep this file from potentially getting immense, items that haven't been seen for a while are thrown out. The period of time a feed's items go unseen before each is forgotten is called that feed's retention, and is expressed in seconds.

The default retention value is 32 days (i.e., 32*24*60*60, the number of seconds in 32 days). If you wanted to change it to just a week, you would do this with "rss_retention(7*24*60*60)".

As a special case, a zero or negative value for the retention means to never clear anything from the history file, no matter how long it has gone unseen.

"rss_add_comment( strings );"

Call this function if you want to add extra

XML comments to this RSS file. For example, if you call this:

        rss_add_comment(
          "Our terms of use: http://wherever.int/rsstou.html",
          "Any questions? Ask [email protected]",
        );

...then this RSS feed will contain this XML fairly early on in the file:

        <!-- Our terms of use: http://wherever.int/rsstou.html -->
        <!-- Any questions? Ask [email protected] -->

the accessor "rss_css( url )"

This defines the given

URL as being the XML-CSS stylesheet for this RSS feed. The default value is ``./rss.css'' if "-e "rss.css"" is true, otherwise is the value http://www.interglacial.com/rss/rss.css

the accessor "rss_xsl( url )"

This defines the given

URL as being the XML-XSL stylesheet for this RSS feed. The default value is none.

The accessors "rss_url(

string ), rss_title( string ), rss_description( string )"

These define this feed's

URL, title, and description. These functions are just for completeness, since it's simpler to just specify any/all of these parameters in the call to "rss_new".

the accessor "rss_ttl( number )"

This sets the parameter of this

RSS feed's "ttl" element, which suggests how long (in minutes, not seconds!) an RSS reader should wait after it polls a feed until it polls it again. For example, "rss_ttl(90)" would suggest that a reader should not poll this feed more often than every 90 minutes.

(This element is somewhat obsolescent next to the newer and more informative "sy:update*" elements, but is included for backward compatibility.)

the accessor "rss_allow_duplicates( boolean )"

This controls whether or not duplicate items are filtered out out the feed. By default this is

on. Note that duplicates are detected only by their URL, so if you call this:

        rss_item('http://foo.int/donate', "Give!");
        rss_item('http://foo.int/donate', "We need money!");
        rss_save('begging.rss');

...then only the first will appear in the feed, since the second item has a URL that is already being saved in this feed. (However, "rss_item_count" is still 2, because filtering out duplicates is something that only happens as the feed is saved.)

the accessor "rss_docs( url )"

This sets the value of the not-generally-useful "doc"

RSS element. The default value is ``./about_rss.html'' if "-e "about_rss.html"" is true, otherwise ``http://www.interglacial.com/rss/about.html''.

the accessors "rss_image_url(url), rss_image_width(number), rss_image_height(number), rss_image_title(text), rss_image_link(url), rss_image_description(text)"

These are for manually setting the values of this feed's image element's subelements:

  <image>
              <url> (rss_image_url)         </url>
            <width> (rss_image_width)       </width>
           <height> (rss_image_height)      </height>
            <title> (rss_image_title)       </title>
             <link> (rss_image_link)        </link>
      <description> (rss_image_description) </description>
  </image>

You rarely need to call any of these "rss_image_whatever" functions --- usually just calling "rss_image( url, h, w );" is enough.

RSS VERSION

RSS feeds emitted by this module are basically according to v0.92 RSS, with a very few extensions from v2.0 RSS. They are not RDF files.

COPYRIGHT AND DISCLAIMERS

Copyright (c) 2003,4 Sean M. Burke. All rights reserved.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose.

Portions of the data tables in this module are derived from the entity declarations in the W3C XHTML specification.

Currently (January 2004), that's these three:

       http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent
       http://www.w3.org/TR/xhtml1/DTD/xhtml-special.ent
       http://www.w3.org/TR/xhtml1/DTD/xhtml-symbol.ent

Portions of the code in this module were adapted from parts of Gisle Aas's LWP::Simple and the old (v2.x) version of his HTML::Parser.

AUTHOR

Sean M. Burke "[email protected]"