IncludeConfigFile /usr/local/swish/conf/site_config.config

This may be overridden from the command line via the "-v" switch (see SWISH-RUN).

    0 = no report
    1 = fatal errors
    2 = errors
    3 = warnings

Currently (as of 2.4.4 - early 2005) libxml2 only reports errors at level 2. The default as of 2.4.4 is ``2'' which should report any errors that might indicate a problem parsing a document.

The exception to this is UTF-8 to Latin-1 conversion errors are reported at level 3 (changed from 1 in 2.4.4). Although these errors indicate a problem indexing text, they are only reported at level 3 because they can be very common.

It is recommended that you index at ParserWarnLevel 3 when first starting out to see what errors and warnings are reported. Then reduce the level when you understand what documents are causing parsing problems and why.

    IndexFile /usr/local/swish/site.index

    <meta name="robots" content="noindex">

Enable alternate search syntax. Allows the usage of a basic ``Altavista(c)'', ``Lycos(c)'', etc. like search syntax. This means a search query can contain ``+'' and ``-'' as syntax parameter.

Example:

    swish-e -w "+word1 +word2 -word3  word4 word5"
    "+"  = following word has to be in all found documents
    "-"  = following word may not be in any document found
    " "  = following word will be searched in documents

Using this config directive you can change the boolean search operators of Swish-e, e.g. to adapt these to your language. The default is: AND OR NOT

Example (german):

    SwishSearchOperators   UND  ODER  NICHT

"SwishSearchDefaultRule" defines the default Boolean operator to use if none is specified between words or phrases. The default is "AND".

The word you specify must match one of the available "SwishSearchOperators".

Example:

    SwishSearchOperators   UND  ODER  NICHT
    # Make it act like a web search engine
    SwishSearchDefaultRule ODER

The output of Swish-e can be defined by specifying a format string with the "-x" command line argument. Using "ResultExtFormatName" you can assign a predefined format string to a name.

Examples:

    ResultExtFormatName  moreinfo   "%c|%r|%t|%p|<author>|<publishyear>\n"

Then when searching you can specify the format string's name

    swish-e   ...  -x moreinfo  ...

See the "-x" switch in SWISH-RUN for more information about output formats.

Examples:

The "-S" command line argument is used to select the file access method.

    swish-e -c swish.config -S fs    - file system
    swish-e -c swish.config -S http  - internal http spider
    swish-e -c swish.config -S prog  - external program of any type

For the fs method of access IndexDir is a space-separated list of files and directories to index. Use a forward slash as the path separator in MS Windows.

For the http method the IndexDir setting is a list of space-separated URLs.

For the prog method the IndexDir setting is a list of space-separated programs to run (which generate documents for swish to index).

You may specify more than one IndexDir directive.

Any sub-directories of any listed directory will also be indexed.

Note: While processing directories, Swish-e will ignore any files or directories that begin with a dot (``.''). You may index files or directories that begin with a dot by specifying their name with "IndexDir" or "-i".

Examples:

    # Index this directory an any subdirectories
    IndexDir /usr/local/home/http

    # Index the docs directory in current directory
    IndexDir ./docs

    # Index these files in the current directory
    IndexDir ./index.html ./page1.html ./page2.html
    # and index this directory, too
    IndexDir ../public_html

For the HTTP method of access specify the URL's from which you want the spidering to begin.

Example:

If the file's type is HTML or HTML2 (as set by "IndexContents" or "DefaultContents") then the file will be parsed for a HTML title and that title will be indexed. Note that you must set the file's type with "IndexContents" or "DefaultContents": ".html" and ".htm" are NOT type HTML by default. For example:

   IndexContents HTML* .htm .html

If a title is found, it will still be checked for "FileRules title", and the file will be skipped if a match is found. See "FileRules".

If the file's type is not HTML, or it is HTML and no title is found, then the file's path will be indexed.

For example, this will allow searching by image file name.

    NoContents .gif .xbm .au .mov .mpg .pdf .ps

Note: Using this directive will not cause files with those suffixes to be indexed. That is, if you use "IndexOnly" to limit the types of files that are indexed, then you must specify in "IndexOnly" the same suffixes listed in "NoContents".

This does not work:

    # Wrong!
    IndexOnly .htm .html
    NoContents .gif .xbm .au .mov .mpg .pdf .ps

A "-S prog" program may set the "No-Contents:" header to enable this feature for a specific document (although it would be smarter for the "-S prog" program to simply only send the pathname or title to be indexed.

For example, you may index your files locally (with the File system indexing method), yet return a URL in search results. This directive can be used to map the file names to their respective URLs on your web server.

There are five operations you can specify: replace, append, remove, prepend, and regex They will parse the pathname in the order you've typed these commands.

This directive uses C library regex.h regular expressions.

   replace "the string you want replaced" "what to change it to"
   remove "a string to remove"   
   prepend "a string to add before the result"
   append "a string to add after the result"
   regex  "/search string/replace string/options"

Remember, quotes are needed if an expression contains white space, and backslashes have special meaning.

Regex is an Extended Regular Expression. The first character found is the delimiter (but it's not smart enough to use matched chars such as [], (), and {}).

The replace string may use substitution variables:

    $0      the entire matched (sub)string
    $1-$9   returns patterns captured in "(" ")" pairs
    $`      the string before the matched pattern
    $'      the string after the matched pattern

The options change the behavior of expression:

    i       ignore the case when matching
    g       repeat the substitution for the entire pattern

Examples:

    ReplaceRules replace testdir/ anotherdir/
    ReplaceRules replace [a-z_0-9]*_m.*\.html index.html

    ReplaceRules remove testdir/

The XML2, HTML2, and TXT2 parsers are currently only available when Swish-e is configured to use libxml2.

You may use XML*, HTML*, and TXT* to select the parser automatically. If libxml2 is installed then it will be used to parse the content. Otherwise, Swish-e's internal parsers will be used.

Documents that are not assigned a parser with "IndexContents" will, by default, use the HTML2 parser if libxml2 is installed, otherwise will use Swish-e's internal HTML parser. The "DefaultContents" directive may be used to assign a parser to documents that do not match a file extension defined with the "IndexContents" directive.

Example:

    IndexContents HTML* .htm .html .shtml
    IndexContents TXT*  .txt .log .text
    IndexContents XML*  .xml

HTML* is the default type for all files, unless otherwise specified (and this default can be changed by the DefaultContents directive. Swish-e parses titles from HTML files, if available, and keeps track of the context of the text for context searching (see "-t" in SWISH-RUN).

If using filters (with the "FileFilter" directive) to convert documents you should include those extensions, too. For example, if using a filter to convert .pdf to .html, you need to tell Swish-e that .pdf should be indexed by the internal HTML parser:

    FileFilter  .pdf   pdf2html
    IndexContent  HTML  .pdf

See also Document Filter Directives.

Note: Some of this may be changed in the future to use content-types instead of file extensions. See SWISH-3.0

The XML2, HTML2, and TXT2 parsers are currently only available when Swish-e is configured to use libxml2.

You may use XML*, HTML*, and TXT* to select the parser automatically. If libxml2 is installed then it will be used to parse the content. Otherwise, Swish-e's internal parsers will be used.

Example:

    DefaultContents HTML

The "DefaultContents" directive should be used when spidering, as HTML files may be returned without a file extension (such as when requesting a directory and the default index.html is returned).

Setting FileInfoCompression to "yes" will compress the index file to save disk space. This may result in longer indexing times. The default is "no".

Also see the "-e" switch in SWISH-RUN for saving RAM during indexing.

If "ConvertHTMLEntities" is set "no" the entities will be indexed without conversion.

NOTE: Entities within XML files and files parsed with libxml2 (HTML2) are converted regardless of this setting.

    swish-e -w subjects=foo

(See also the "-t" switch in SWISH-RUN for information about context searching in HTML documents.)

The MetaNames directive is a space separated list. For example:

    MetaNames meta1 meta2 keywords subjects

You may also use "UndefinedMetaTags" to specify automatic extraction of meta names from your HTML and XML documents, and also to ignore indexing content of meta tags.

META tags can have two formats in your HTML source documents:

    <META NAME="meta1" CONTENT="some content">

and (if using the HTML2/libxml2 parser)

    <meta1>
        some content
    </meta1>

But this second version is invalid HTML, and will generate a warning if ParserWarningLevel is set (libxml2 only).

And in XML documents, use the format:

    <meta1>
        Some Content
    </meta1>

Then you can limit your search to just META meta1 like this:

    swish-e -w 'meta1=(apples or oranges)'

You may nest the XML and the start/end tag versions:

    <keywords>
        <tag1>
            some content
        </tag1>
        <tag2>
            some other content
        </tag2>
    <keywords>

Then you can search in both tag2 and tag2 with:

    swish-e -w 'keywords=(query words)'

Swish-e indexes all text as some metaname. The default is "swishdefault", so these two queries are the same:

    swish-e -w foo
    swish-e -w swishdefault=foo

When indexing HTML Swish-e indexes the HTML title as default text, so when searching Swish-e will find matches in both the HTML body and the HTML title. Swish also, by default, indexes content of meta tags. So:

    swish-e -w foo

will find ``foo'' in the body, the title, or any meta tags.

Currently, there's no way to prevent Swish-e from indexing the title contents along with the body contents, but see "UndefinedMetaTags" for how to control the indexing of meta tags.

If you would like to search just the title text, you may use:

    MetaNames swishtitle

This will index the title text separately under the built-in swish internal meta name ``swishtitle''. You may then search like

    swish-e -w foo  -- search for "foo" in title, body (and undefined meta tags)
    swish-e -w swishtitle=foo -- search for "foo" in title only

In addition to swishtitle, you can limit searches to documents' path with:

   MetaNames swishdocpath

Then to search for ``foo'' but also limit searches to documents that include ``manual'' or ``tutorial'' in their path:

   swish-e -w foo swishdocpath=(manual or tutorial)

See also "ExtractPath".

    MetaNames summary
    MetaNameAlias summary description overview

Then all three tags will get indexed as meta tag ``summary''. You can then search all the fields as:

    -w summary=foo

The Alias work at search time, too. So these will also limit the search to the ``summary'' meta name.

    -w description=foo
    -w overview=foo

    MetaNamesRank 4 subject
    MetaNamesRank 3 swishdefault
    MetaNamesRank 2 author publisher
    MetaNamesRank -5 wrongwords

This feature is still considered experimental. If you use it, please send feedback to the discussion list.

Example:

    HTMLLinksMetaName links

Now, to limit searches to files with a link to ``home.html'' do this:

    -w links='"home.html"'

The double quotes force a phrase search.

To make Swish-e index links as normal text, you may use:

    HTMLLinksMetaName swishdefault

This feature is only available with the libxml2 HTML parser.

Example:

    ImagesLinksMetaName images

Now, if you would like to find pages that include a nice image of a beach:

    -w images='beach'

To make Swish-e index links as normal text, you may use:

    ImageLinksMetaName swishdefault

This feature is only available with the libxml2 HTML parser.

For example, by specifying a tag name:

   IndexAltTagMetaName bar

would make this markup:

    <foo>
        <img src="/someimage.png" alt="Alt text here">
    </foo>

appear like

    <foo>
        <bar>Alt text here</bar>
    </foo>

Then the normal rules ("MetaNames" and "PropertyNames") apply to how that text is indexed.

If you use the special tag ``as-text'' then

    <foo>
        <img src="/someimage.png" alt="Alt text here">
    </foo>

simply becomes

    <foo>
        Alt text here
    </foo>

This feature is only available when using the libxml2 parser (HTML2 and XML2).

For example, say you wish to index image links under the metaname ``images''.

    ImageLinksMetaName images

    <person age="23">
          John Doe
    </person>

look like the following to swish:

    <person>
        <person.age>
            23
        </person.age>
        John Doe
    </person>

What happens to the text ``23'' will depend on the setting of "UndefinedXMLAttributes":

    XMLClassAttributes class

    <person class="first">
        John
    </person>
    <person class="last">
        Doe
    </person>

Will appear to Swish-e as:

    <person>
        <person.first>
        John
        </person.first>
    </person>
    <person>
        <person.last>
        Doe
        </person.last>
    </person>

How the data is indexed depends on "MetaNames" and "UndefinedMetaTags".

Here's an example using the following configuration which combines the two directives "XMLClassAttributes" and "UndefinedXMLAttributes".

    XMLClassAttributes class
    UndefinedMetaTags auto
    UndefinedXMLAttributes auto
    IndexContents XML2 .xml

The source XML file looks like:

    <xml> <person class="student" phone="555-1212" age="102"> John </person>
    <person greeting="howdy">Bill</person> </xml>

Swish-e parses as:

    ./swish-e -c 2 -i 1.xml -T parsed_tags  parsed_text  -v 0
    Indexing Data Source: "File-System"

    <xml> (MetaName)

        <person> (MetaName)
            <person.student> (MetaName)
                <person.student.phone> (MetaName)
                    555-1212
                </person.student.phone> 
                <person.student.age> (MetaName)
                    102
                </person.student.age> 
                John
        </person>

        <person> (MetaName)
            <person.greeting> (MetaName)
                howdy
            </person.greeting> 
            Bill
        </person>

    </xml> 
    Indexing done!

One thing to note is that the first <person> block finds a class name ``student'' so all metanames that are created from attributes use the combined name ``person.student''. The second <person> block doesn't contain a ``class'' so, the attribute name is combined directly with the element name (e.g. ``person.greeting'').

The extracted string will be indexed under the specified meta name.

See "ReplaceRules" for a description of the various pattern replacement methods, but you will use the regex method.

For example, say your file system (or web tree) was organized into departments:

    /web/sales/foo...
    /web/parts/foo...
    /web/accounting/foo...

And you wanted a way to limit searches to just documents under ``sales''.

    ExtractPath department regex !^/web/([^/]+)/.*$!$1!

Which says, extract out the department name (as substring $1) and index it as meta name "department". Then to limit a search to the sales department:

    swish-e -w foo AND department=sales

Note that the "regex" method uses a substitution pattern, so to index only a sub-string match the entire document path in the regular expression, as shown above. Otherwise any part that is not matched will end up in the substitution pattern.

See the "ExtractPathDefault" option for a way to set a value if not patterns match.

Although unlikely, you may use more than one "ExtractPath" directive. More than one directive of the same meta name will operate successively (in order listed in the configuration file) on the path. This allows you to use regular expressions on the results of the previous pattern substitution (as if piping the output from one expression to the patter of the next).

    ExtractPath foo regex !^(...).+$!$1!
    ExtractPath foo regex !^.+(.)$!$1!

So, the third letter is indexed as meta name ``foo'' if both patterns match.

    ExtractPath foo regex !^X(...).+$!$1!
    ExtractPath foo regex !^.+(.)$!$1!

Now (not the ``X''), if the first pattern doesn't match, the last character of the path name is indexed. You must be clear on this behavior if you are using more than one "ExtractPath" directive with the same metaname.

The document path operated on is the real path swish used to access the document. That is, the "ReplaceRules" directive has no effect on the path used with "ExtractPath".

The full path is used for each meta name if more than one "ExtractPath" directive is used. That is, changes to the path used in "ExtractPath foo" do not affect the path used by "ExtractPath bar".

For example, say your want to index each document with a metaname ``department'' based on the following path examples:

    /web/sales/foo...
    /web/parts/foo...
    /web/accounting/foo...

But you are also indexing documents that do not follow that pattern and you want to search those separately, too.

    ExtractPath department regex !^/web/([^/]+)/.*$!$1!
    ExtractPathDefault department other

Now, you may search like this:

    -w foo department=(sales)      - limit searches to the sales documents
    -w foo department=(parts)      - limit searches to the parts documents
    -w foo department=(accounting) - limit searches to the accounting documents
    -w foo department=(other)      - everything but sales, parts, and accounting.

This basically is a shortcut for:

    -w foo not department=(sales or parts or accounting)

but you don't need to keep track of what was extracted.

Properties are useful for returning additional data from documents in search results --- this saves the effort of reading and parsing the source files while reading Swish-e search results, and is especially useful when the source documents are no longer available or slow to access (e.g. over http).

Another feature of properties is that Swish-e can use the PropertyNames for sorting the search results (see the "-s" switch).

    PropertyNames author subjects

Two variations are available. "PropertyNamesCompareCase" and "PropertyNamesIgnoreCase". These tell Swish-e to either ignore or compare case when sorting results. The default for "PropertyNames" is to ignore the case.

    PropertyNamesIgnoreCase subject
    PropertyNamesCompareCase keyword

The defaults for ``internal'' properties are:

    swishtitle          --  ignore the case
    swishdocpath        --  compare case
    swishdescription    --  compare case

These can be overridden with "PropertyNamesCompareCase" and "PropertyNamesIgnoreCase".

    PropertyNamesCompareCase swishtitle

Use of PropertyNames will increase the size of your index files, sometimes significantly. Properties will be compressed if Swish-e is compiled with zlib as described in the INSTALL manual page.

If Swish-e finds more than one property of the same name in a document the property's contents will be concatinated for strings, and a warning issues for numeric (or date) properties.

When printing properties with the swish-e binary newlines are replaced with a space character. Use the swish-e library (or SWISH::API perl module) to fetch properties without newlines replaced.

Swish-e will not parse dates when indexing; you must use a timestamp.

    PropertyNameAlias swishtitle title titel título titulo

Note that ``swishtitle'' is the built-in property used to store the title of a document, and therefore you do not need to specify it as a PropertyName before use.

If any of the property names do not exist they will be created (e.g. you do not need to define the property with PropertyNames first).

In general, this feature will only be useful when parsing HTML or XML with the libxml2 parser.

For example:

    PropertyNamesMaxLength 1000 swishdescription
    PropertyNameAlias swishdescription body

Is somewhat like

    StoreDescription HTML <body> 1000
    StoreDescription XML <body> 1000
    StoreDescription HTML2 <body> 1000
    StoreDescription XML2 <body> 1000

but StoreDescription allows setting the tag for each parser type.

    PropertyNamesMaxLength 1000 headings
    PropertyNameAlias headings h1 h2 h3 h4

collects all the heading text into a single property called ``headings'', not to exceed 1000 characters.

This setting is used when sorting during indexing, and perhaps when sorting while searching. It also effects the order when limiting to a range of values with the -L option.

If "PreSortedIndex" it is not present in the config file (default action), all the properties will be presorted at indexing time. If it is present without any parameter, no properties will be presorted. Otherwise, only the property names specified will be presorted.

For example, if you only wish to sort results by a property called "title":

    PropertyNames title age time
    PreSortedIndex  title

The document type (XML, HTML and TXT) must match the document type currently being indexed as set by "IndexContents" or "DefaultContents". See those directives for possible values. A common problem is using "StoreDescription" yet not setting the document's type with "IndexContents" or "DefaultContents". Another problem is different types:

    IndexContents HTML2 .html
    StoreDescription HTML <body>

Then .html documents are assigned a type of HTML2 (and parsed by the libxml2 parser), but the description will not be stored since it is type HTML instead of HTML2.

For text documents you specify the type TXT (or TXT2 or TXT*) and the number of characters to capture.

    StoreDescription TXT 20

The above stores only the first twenty characters from the text file in the Swish-e index file.

For HTML, and XML file types, specify the tag to use for the description, and optionally the number of characters to capture. If not specified will capture the entire contents of the tag.

    StoreDescription HTML <body> 20000
    StoreDescription XML  <desc> 40

Again, note that documents must be assigned a document type with "IndexContents" or "DefaultContents" to use this feature.

Swish-e will compress the descriptions (or any other large property) if compiled to use zlib (see INSTALL). This is recommended when using StoreDescription and a large number of documents. Compression of 30% to 50% is not uncommon with HTML files.

The default depends on the default setting compiled with zlib, but is typically six.

This option is useful when using "StoreDescription" to store a large amount text in properties (or if using "PropertyNames" with large property sizes).

Properties must be over a value defined in config.h (100 is the default) before compression will be attempted. Swish-e will never store the results of the compression if the compressed data is larger than the original data.

This option is only available when Swish-e is compiled with zlib support.

Example:

    TruncateDocSize    10000000

The default is zero, which means read all data.

Warning: If you use TruncateDocSize, use it with care! TruncateDocSize is a safety belt only, to limit e.g. filteroutput, when accessing databases, or to limit ``runnaway'' filters. Truncating doc input may destroy document structures for Swish-e (e.g. swish may miss closing tags for XML or HTML documents).

TruncateDocSize does not currently work with the "prog" input source method.

It's a good idea to create both a normal index and a fuzzy index and allow your search interface select which index to use. Many people find the fuzzy searches to be too fuzzy.

The available fuzzy indexing options can be displayed by running

   swish-e -T LIST_FUZZY_MODES

Available options include:

    UseStemming no
    UseStemming yes

When UseStemming is set to "yes" every word is stemmed before placing it in to the index.

This option is deprecated. It has been superceded by "FuzzyIndexingMode".

This option is deprecated. It has been superceded by "FuzzyIndexingMode".

    IgnoreTotalWordCountWhenRanking no

The default was changed from no to yes in version 2.2.

NOTE: must be set to no if you intend to use the -R 1 option when searching.

    MinWordLimit 5

When indexing Swish-e uses WordCharacters to split up the document into words. Words are defined by any string of non-blank characters that contain only the characters listed in WordCharacters. If a string of characters includes a character that is not in WordCharacters then the word will be spit into two or more separate words.

For example:

    WordCharacters abde

Would turn ``abcde'' into two words ``ab'' and ``de''.

Buzzwords are case insensitive.

Buzzwords should be separated by spaces and may span multiple directives. If the special format "File:filename" is used then the Buzzwords will be read from an external file during indexing.

Examples:

    Buzzwords C++ TCP/IP

    Buzzwords File: ./buzzwords.lst

If a Buzzword contains search operator characters they must be backslashed when searching. For example:

    Buzzwords C++ TCP/IP web=http

    ./swish-e -w 'web\=http'

Buzzwords are found by splitting the text on whitespace, removing "IgnoreFirstChar" and "IgnoreLastChar" characters from the word, and then comparing with the list of "Buzzwords". Therefore, if adding "Buzzwords" to an index you will probably want to define "IgnoreFirstChar" and "IgnoreLastChar" settings.

Note: Buzzwords specific settings for "IgnoreFirstChar" and "IgnoreLastChar" may be used in the future.

Enabling this option can reduced the size of the index file, but at the expense of slower wildcard search times.

The default changed from YES to NO starting with version 2.4.3.

Words should be separated by spaces and may span multiple directives. If the special format "File:filename" is used then the stop words will be read from an external file during indexing.

In previous versions of Swish-e you could use the directive

    IgnoreWords swishdefault - obsolete!

to include a default list of compiled in stopwords. This keyword is no longer supported.

Examples:

    IgnoreWords www http a an the of and or

    IgnoreWords File: ./stopwords.de

You can specify a list of words following the directive (you may specify more than one "UseWords" directive in a config file), and/or use the "File:" form to specify a path to a file containing the words:

    UseWords perl python pascal fortran basic cobal php
    UseWords File: /path/to/my/wordlist

Please drop the Swish-e list a note if you actually use this feature. It may be removed from future versions.

    IgnoreLimit 50 1000

Swish-e must do extra processing to adjust the entire index when this feature is used. It is recommended that instead of using this feature that you decided what words are stopwords and add them to IngoreWords in your configuration file. To do this, use IgnoreLimit one time and note the stop words that are found while indexing. Add this list to IgnoreWords, and then remove IgnoreLimit from the configuration file.

This option is useful to avoid indexing specific data from a file. For example:

    <person>
        <first_name>
            William
        </first_name> <last_name>
            Shakespeare
        </last_name> <updated_date>
            April 25, 1999
        </updated_date>
    </person>

In the above example you might not want to index the updated date, and therefore prevent finding this record by searching

    -w 'person=(April)'

This is solved by:

    IgnoreMetaTags updated_date

See also "UndefinedMetaTags".

This experimental feature can be used to define a set of characters that describe a number. If a word is found to contain only those characters it will not be indexed. The characters listed must be part of "WordCharacters" settings. In other words, the ``word'' checked is a word that Swish-e would otherwise index.

For example,

    IgnoreNumberChars 0123456789$.,

Then Swish-e would not index the following:

    123
    123,456.78
    $123.45

You might be tempted to avoid indexing hex numbers with:

    IgnoreNumberChars 0123456789abcdef

which will not index 0D31, but will also not index the word ``bad''.

This is an experimental feature that may change in future versions. One possible change is to use regular expressions instead.

    IndexComments yes

Note: This is a change in the default behavior prior to version 2.2.

For example:

    # This will index a_b as a-b and ámo as amo
    TranslateCharacters _á -a

"TranslateCharacters :ascii7:" is a predefined set of characters that will translate eight bit characters to ascii7 characters. Using the :ascii7: rule will translate ``Ääç'' to ``aac''. This means: searching ``Çelik'', ``çelik'' or ``celik'' will all match the same word.

TranslateCharacters is done early in the indexing process, after converting HTML entities but before splitting the input text into words based on WordCharacters. So characters you are translating from do not need to be listed in word characters.

The same character translations take place when searching.

For example, if you have a tag:

    <subjects>
        computer programming | apple computers
    </subjects>

You might want to prevent matching ``programming apple'' in that meta name.

    BumpPositionCounterCharacters |

There is no default, and you may list a string of characters.

For example,

    <person>
        <first_name>
            William
        </first_name>
        <last_name>
            Shakespeare
        </last_name>
        <updated_date>
            April 25, 1999
        </updated_date>
    </person>

In the configuration file:

    DontBumpPositionOnEndTags first_name
    DontBumpPositionOnStartTags last_name

This configuration allows this phrase search

    -w 'person=("william shakespeare")'

but this phrase search will fail

    -w 'person=("shakespeare april")'

    # Only index .html .htm and .q files
    IndexOnly .html .htm .q

"IndexOnly" checks that the file end in the characters listed. It does not check ``extensions''. "IndexOnly" is tested right before "FileRules" is processed.

    FollowSymLinks no
    FollowSymLinks yes

Except for "FileRules title ...", this feature is only available for file access method (-S fs), which is the default indexing mode. Also, any pathname modification with "ReplaceRules" happens after the check for "FileRules". (It's unlikely that you would exclude files with "FileRules" based on text you added with "ReplaceRules"!)

The regular expression is a C regex.h extended regular expression. You may supply more than one regular expression per line, or use separate directives. Preceding the regular expression with the word ``not'' negates the match.

The regular expression is compared against [type] as described below.

For historical reasons, you can specify "contains" or "is". "is" simply forces the regular expression to match at the start and end of the string (by internally prepending ``^'' and appending ``$'' to the regular expression).

The "regex" option requires delimiter characters:

    FileRules title regex /^private/i

The only advantage of "regex" is if you want to do case insensitive matches, or simply like your regular expressions to look like perl regular expressions. You must use matching delimiters; (), {}, and [], are not currently supported for no good reason other than laziness.

Use quotes (" or ') around a pattern if it contains any white space. Note that the backslash character becomes the escape character within quotes.

For example, these sets generate the same regular expressions.

    FileRules title is hello
    FileRules title contains ^hello$
    FileRules title regex /^hello$/

These all need quotes due to the included space character

    FileRules title is "hello there"
    FileRules title contains "^hello there$"
    FileRules title regex "!^hello there$!"

These show how the backslash must be doubled inside of quotes. Swish-e converts a double-backslash into a single backslash, and then passes that single onto the regular expression compiler.

    FileRules filename regex /\.pdf/
    FileRules filename regex "/\\.pdf/"

    FileRules filename regex !hello\\there!     # need double for real backslash 
    FileRules filename regex "!hello\\\\there!" # need double-double inside of quotes

Matching Types

The following types of match strings my be supplied:

    FileRules pathname
    FileRules dirname
    FileRules filename
    FileRules directory
    FileRules title

    FileMatch pathname
    FileMatch filename
    FileMatch dirname
    FileMatch directory

pathname matches the regular expression against the current pathname. The pathname may or may not be absolute depending on what you supplied to "IndexDir".

Example:

    # Don't index paths that contain private or hidden
    FileRules pathname contains (private|hidden)

    # Same thing
    FileRules pathname regex /(private|hidden)/

    # Don't index exe files
    FileRules pathname contains \.exe$

dirname and filename split the path name by the last delimiter character into a directory name, and a file name. Then these are compared against the patterns supplied. Directory names do not have a trailing slash. All path names use the forward slash as a delimiter within Swish-e.

Example:

    # Same as last example - don't index *.exe files.
    FileRules filename contains \.exe$

    # Don't index any file called test.html files
    FileRules filename contains ^test\.html$

    # Same thing
    FileRules filename is test\.html

    # Don't index any directories that contain "old"  (/usr/local/myold/docs)
    FileRules dirname contains old

    # Don't index any directories that contain the path segment "old" (/usr/local/old/foo)
    FileRules dirname contains /old/

    # Index only .htm, .html, plus any all-digit file names
    IndexOnly .htm .html
    FileMatch filename contains ^\d+$

    # Same as previous, but maybe a little slower
    FileRules filename regex not !\.(htm|html)$!
    FileMatch filename contains ^\d+$

Swish-e checks these settings in the order of "pathname", "dirname", and "filename", and "FileMatch" patterns are checked before "FileRules", in general. This allows you to exclude most files with "FileRules", yet allow in a few special cases with "FileMatch". For example:

    # Exclude all files of .exe, .bin, and .bat
    FileRules filename contains \.(exe|bin|bat)$
    # But, let these two in
    FileMatch filename is baseball\.bat incoming_mail\.bin

    # Same, but as a single pattern
    FileMatch filename is (baseball\.bat|incoming_mail\.bin)

The "directory" type is somewhat unique. When Swish-e recurses into a directory it will compare all the files in the directory with the pattern and then decide if that entire directory should or should not be indexed (or recursed). Note that you are matching against file names in a directory --- and some of those names may be directory names.

A "FileRules directory" match will cause Swish-e to ignore all files and sub-directories in the current directory.

Warning: A match with "FileMatch directory" says to index everything in the *current* directory and ignore any FileRules for this directory.

Example:

    # Don't index any directories (and sub directories) that contain
    # a file (or sub-directory) called "index.skip"
    FileRules directory contains ^index\.skip$

    # Don't index directories that contain a .htaccess file.
    FileRules directory contains ^\.htaccess

Note: While processing directories, Swish-e will ignore any files or directories that begin with a dot (``.''). You may index files or directories that begin with a dot by specifying their name with "IndexDir" or "-i".

"title" checks for a pattern match in an HTML title.

Example:

    FileRules title contains construction example pointers

    # This example says to ignore case
    FileRules title regex "/^Internal document/i"

Note: "FileRules title" works for any input method (fs, prog, or http) that is parsed as HTML, and where a title was found in the document.

In case all this seems a bit confusing, processing a directory happens in the following order.

First the directory name is checked:

    FileRules dirname - reject entire directory if matches

Next the directory is scanned and each file name (which might be the name of a sub-directory) is checked:

    FileRules directory - reject entire dir if *any* files match
    FileMatch directory - accept entire dir if *any* files match

Then, unless "FileMatch directory" matched, each file is tested with FileMatch. A match says to index the file without further testing (i.e. overrides FileRules and IndexOnly):

    FileMatch pathname  \
    FileMatch dirname   - file is accepted if any match
    FileMatch filename  /

otherwise

    IndexOnly - file is checked for the correct file extension

    FileRules pathname  \
    FileRules dirname   - file is rejected if any match
    FileRules filename  /

finally, the file is indexed.

Files (not directories) listed with "IndexDir" or "-i" are processed in a similar way:

    FileMatch pathname  \
    FileMatch dirname   - file is accepted if any match
    FileMatch filename  /

otherwise, the file is rejected if it doesn't have the correct extension or a FileRules matches.

    IndexOnly - file is checked for the correct file extension

    FileRules pathname  \
    FileRules dirname   - file is rejected if any match
    FileRules filename  /

Note: If things are not indexing as you expect, create a directory with some test files and use the "-T regex" trace option to see how file names are checked. Start with very simple tests!

    MaxDepth 5

Note: The default was changed from 5 to 0 in release 2.4.0

    Delay 1

Note: The default was changed from 60 to 5 seconds in release 2.4.0

    TmpDir /tmp/swish

If this directory does not exist or is not writable Swish-e will fail with an error during indexing.

Note, the environment variables of "TMPDIR", "TMP", and "TEMP" (in that order) will override this setting.

    SpiderDirectory /usr/local/swish

    SwishProgParameters /path/to/config hello there
    IndexDir /path/to/program.pl

Then running:

    swish-e -c config -S prog

Swish-e will execute "/path/to/program.pl" and pass "/path/to/config hello there" as three command line arguments to the program. This directive makes it easy to pass settings from the Swish-e configuration file to the external program.

For example, the "spider.pl" program (included in the "prog-bin" directory) uses the "SwishProgParameters" to specify what file to read for configuration information.

    SwishProgParameters spider.config
    IndexDir ./spider.pl

The "spider.pl" program also has a default action so you can avoid using a configuration file:

This is the path to a directory where the filter programs are stored. Swish-e looks in this directory to find the filter specified in the FileFilter directive.

This directive is not needed if the filter program can be found in your system's path. Even if your filter is not in your system's path you can specify the full path to the filter in the FileFilter or FileFilterMatch directives.

Example:

    FilterDir /usr/local/swish/filters

Examples:

    FileFilterMatch ./pdftotext "%p -" /\.pdf$/

Note that will also match a file called ``.pdf'', so you may want to use something that requires a filename that has more than just an extension. For example:

    FileFilterMatch ./pdftotext "%p -" /.\.pdf$/

To specify more than one extension:

    FileFilterMatch ./check_title.pl "%p" /\.html$/  /\.htm$/

Or a few ways to do the same thing:

    FileFilterMatch ./check_title.pl %p /\.(html|html)$/
    FileFilterMatch ./check_title.pl %p /\.html?$/

And to ignore case:

    FileFilterMatch ./check_title.pl %p /\.html?$/i

You may also precede an expression with ``not'' to negate regular expression that follow. For example, to match files that do not have an extension:

    FileFilterMatch ./convert "%p %P" not /\..+$/