General Questions

And in XML format (can also be used in HTML documents when using libxml2):

    <meta2>
        Some Content
    </meta2>

Then, to inform Swish-e about the existence of the meta name in your documents, edit the line in your configuration file:

    MetaNames DC.subject meta1 meta2

When searching you can now limit some or all search terms to that MetaName. For example, to look for documents that contain the word apple and also have either fruit or cooking in the DC.subject meta tag.

and "creator" is defined as a property (see "PropertyNames" in SWISH-CONFIG) Swish-e can return "accounting department" with the result for that document.

    swish-e -w foo -p creator

Or for sorting:

    swish-e -w foo -s creator

placed in your documents and then using configuration settings of:

    PropertyNames creator
    MetaNames creator

will allow you to limit your searches to documents created by accounting:

    swish-e -w 'foo and creator=(accounting)'

That will find all documents with the word "foo" that also have a creator meta tag that contains the word "accounting". This is using MetaNames.

And you can also say:

    swish-e -w foo -p creator

which will return all documents with the word "foo", but the results will also include the contents of the "creator" meta tag along with results. This is using properties.

You can use properties and meta names at the same time, too:

    swish-e -w creator=(accounting or marketing) -p creator -s creator

That searches only in the "creator" meta name for either of the words "accounting" or "marketing", prints out the contents of the contents of the "creator" property, and sorts the results by the "creator" property name.

(See also the "-x" output format switch in SWISH-RUN.)

And in "/path/to/other/config" you can say:

    IndexDir file1 file2 file3 file4 file5 ...
    IndexDir file20 file21 file22

You may also specify more than one configuration file on the command line:

    ./swish-e -c config_one config_two config_three

Another option is to create a directory with symbolic links of the files to index, and index just that directory.

You may also wish to index a single file that contains words that are or are not indexing as you expect and use -T to output debugging information about the index. A useful command might be:

    ./swish-e -f index.swish-e -T INDEX_FULL

Once you see how Swish-e is parsing and indexing your words, you can adjust the configuration settings mentioned above to control what words are indexed.

Another useful command might be:

     ./swish-e -c my.conf -i problem.file -T PARSED_WORDS INDEXED_WORDS

This will show white-spaced words parsed from the document (PARSED_WORDS), and how those words are split up into separate words for indexing (INDEXED_WORDS).

and/or these may be used also:

       <!-- noindex -->
       <!-- index -->

which says to run the program ``pdftotext'' passing it the pathname of the file (%p) and a dash (which tells pdftotext to output to stdout). Then for each .pdf file Swish-e runs this program and reads in the filtered document from the output from the filter program.

This has the advantage that it is easy to setup --- a single line in the config file is all that is needed to add the filter into Swish-e. But it also has a number of problems. For example, if you use a Perl script to do your filtering it can be very slow since the filter script must be run (and thus compiled) for each processed document. This is exacerbated when using the -S http method since the -S http method also uses a Perl script that is run for every URL fetched. Also, when using -S prog method of input (reading input from a program) using FileFilter means that Swish-e must first read the file in from the external program and then write the file out to a temporary file before running the filter.

With -S prog it makes much more sense to filter the document in the program that is fetching the documents than to have swish-e read the file into memory, write it to a temporary file and then run an external program.

The Swish-e distribution contains a couple of example -S prog programs. spider.pl is a reasonably full-featured web spider that offers many more options than the -S http method. And it is much faster than running -S http, too.

The spider has a perl configuration file, which means you can add programming logic right into the configuration file without editing the spider program. One bit of logic that is provided in the spider's configuration file is a ``call-back'' function that allows you to filter the content. In other words, before the spider passes a fetched web document to swish for indexing the spider can call a simple subroutine in the spider's configuration file passing the document and its content type. The subroutine can then look at the content type and decide if the document needs to be filtered.

For example, when processing a document of type ``application/msword'' the call-back subroutine might call the doc2txt.pm perl module, and a document of type ``appliation/pdf'' could use the pdf2html.pm module. The prog-bin/SwishSpiderConfig.pl file shows this usage.

This system works reasonably well, but also means that more work is required to setup the filters. First, you must explicitly check for specific content types and then call the appropriate Perl module, and second, you have to know how each module must be called and how each returns the possibly modified content.

In comes SWISH::Filter.

To make things easier the SWISH::Filter Perl module was created. The idea of this module is that there is one interface used to filter all types of documents. So instead of checking for specific types of content you just pass the content type and the document to the SWISH::Filter module and it returns a new content type and document if it was filtered. The filters that do the actual work are designed with a standard interface and work like filter ``plug-ins''. Adding new filters means just downloading the filter to a directory and no changes are needed to the spider's configuation file. Download a filter for Postscript and next time you run indexing your Postscript files will be indexed.

Since the filters are standardized, hopefully when you have the need to filter documents of a specific type there will already be a filter ready for your use.

Now, note that the perl modules may or may not do the actual conversion of a document. For example, the PDF conversion module calls the pdfinfo and pdftotext programs. Those programs (part of the Xpfd package) must be installed separately from the filters.

The SwishSpiderConfig.pl examle spider configuration file shows how to use the SWISH::Filter module for filtering. This file is installed at $prefix/share/doc/swish-e/examples/prog-bin, where $prefix is normally /usr/local on unix-type machines.

The SWISH::Filter method of filtering can also be used with the -S http method of indexing. By default the swishspider program (the Perl helper script that fetches documents from the web) will attempt to use the SWISH::Filter module if it can be found in Perls library path. This path is set automatically for spider.pl but not for swishspider (because it would slow down a method that's already slow and spider.pl is recommended over the -S http method).

Therefore, all that's required to use this system with -S http is setting the @INC array to point to the filter directory.

For example, if the swish-e distribution was unpacked into ~/swish-e:

   PERL5LIB=~/swish-e/filters swish-e -c conf -S http

will allow the -S http method to make use of the SWISH::Filter module.

Note that if you are not using the SWISH::Filter module you may wish to edit the swishspider program and disable the use of the SWISH::Filter module using this setting:

    use constant USE_FILTERS  => 0;  # disable SWISH::Filter

This prevents the program from attempting to use the SWISH::Filter module for every non-text URL that is fetched. Of course, if you are concerned with indexing speed you should be using the -S prog method with spider.pl instead of -S http.

If you are not spidering, but you still want to make use of the SWISH::Filter module for filtering you can use the DirTree.pl program (in $prefix/lib/swish-e). This is a simple program that traverses the file system and uses SWISH::Filter for filtering.

Here's two examples of how to run a filter program, one using Swish-e's "FileFilter" directive, another using a "prog" input method program. See the SwishSpiderConfig.pl file for an example of using the SWISH::Filter module.

These filters simply use the program "/bin/cat" as a filter and only indexes .html files.

First, using the "FileFilter" method, here's the entire configuration file (swish.conf):

    IndexDir .
    IndexOnly .html
    FileFilter .html "/bin/cat"   "'%p'"

and index with the command

    swish-e -c swish.conf -v 1

Now, the same thing with using the "-S prog" document source input method and a Perl program called catfilter.pl. You can see that's it's much more work than using the "FileFilter" method above, but provides a place to do additional processing. In this example, the "prog" method is only slightly faster. But if you needed a perl script to run as a FileFilter then "prog" will be significantly faster.

    #!/usr/local/bin/perl -w
    use strict;
    use File::Find;  # for recursing a directory tree

    $/ = undef;
    find(
        { wanted => \&wanted, no_chdir => 1, },
        '.',
    );

    sub wanted {
        return if -d;
        return unless /\.html$/;

        my $mtime  = (stat)[9];

        my $child = open( FH, '-|' );
        die "Failed to fork $!" unless defined $child;
        exec '/bin/cat', $_ unless $child;

        my $content = <FH>;
        my $size = length $content;

        print <<EOF;
    Content-Length: $size
    Last-Mtime: $mtime
    Path-Name: $_

    EOF

        print <FH>;
    }

And index with the command:

    swish-e -S prog -i ./catfilter.pl -v 1

This example will probably not work under Windows due to the '-|' open. A simple piped open may work just as well:

That is, replace:

    my $child = open( FH, '-|' );
    die "Failed to fork $!" unless defined $child;
    exec '/bin/cat', $_ unless $child;

with this:

    open( FH, "/bin/cat $_ |" ) or die $!;

Perl will try to avoid running the command through the shell if meta characters are not passed to the open. See "perldoc -f open" for more information.

Or run a filter like:

    FileFilter .foo e:/perl/bin/perl.exe 'myscript.pl "%p"'

It's often easier to just install Linux.

This is with "-e", and used about 26M or memory:

    270279 unique words indexed.
    23841 files indexed.  177640166 total bytes.
    Elapsed time: 00:06:43 CPU time: 00:04:12

You can also build a number of smaller indexes and then merge together with "-M". Using "-e" while merging will save memory.

Finally, if you do build a number of smaller indexes, you can specify more than one index when searching by using the "-f" switch. Sorting large results sets by a property will be slower when specifying multiple index files while searching.

Or under an old Sparc

  % unlimit openfiles

Now you can search for words or phrases in the path name:

    swish-e -w 'foo AND swishdocpath=(sales)'

So that will only find documents with the word ``foo'' and where the file's path contains ``sales''. That might not works as well as you like, though, as both of these paths will match:

    /web/sales/products/index.html
    /web/accounting/private/sales_we_messed_up.html

This can be solved by searching with a phrase (assuming ``/'' is not a WordCharacter):

    swish-e -w 'foo AND swishdocpath=("/web/sales/")'
    swish-e -w 'foo AND swishdocpath=("web sales")'  (same thing)

The second option is a bit more powerful. With the "ExtractPath" directive you can use a regular expression to extract out a sub-set of the path and save it as a separate meta name:

    MetaNames department
    ExtractPath department regex !^/web/([^/]+).+$!$1/

Which says match a path that starts with ``/web/'' and extract out everything after that up to, but not including the next ``/'' and save it in variable $1, and then match everything from the ``/'' onward. Then replace the entire matches string with $1. And that gets indexed as meta name ``department''.

Now you can search like:

    swish-e -w 'foo AND department=sales'

and be sure that you will only match the documents in the /www/sales/* path. Note that you can map completely different areas of your file system to the same metaname:

    # flag the marketing specific pages
    ExtractPath department regex !^/web/(marketing|sales)/.+$!marketing/
    ExtractPath department regex !^/internal/marketing/.+$!marketing/

    # flag the technical departments pages
    ExtractPath department regex !^/web/(tech|bugs)/.+$!tech/

Finally, if you have something more complicated, use "-S prog" and write a perl program or use a filter to set a meta tag when processing each file.

All Schemes share the following characteristics:

Here is a brief overview of how the different Schemes work. The number in parentheses after the name is the value to invoke that scheme with "swish-e -R" or "RankScheme()".

Ranking is a complicated subject. SWISH-E allows for more Ranking Schemes to be developed and experimented with, using the -R option (from the swish-e command) and the RankScheme (see the API documentation). Experiment and share your findings via the discussion list.

Document Info

.