man Marpa::R2::Scanless::R (3): Scanless interface recognizers

Synopsis

    my $recce = Marpa::R2::Scanless::R->new( { grammar => $grammar } );
    my $self = bless { grammar => $grammar }, 'My_Actions';
    $self->{recce} = $recce;
    if ( not defined eval { $recce->read($p_input_string); 1 }
        )
    {
        ## Add last expression found, and rethrow
        my $eval_error = $EVAL_ERROR;
        chomp $eval_error;
        die $self->show_last_expression(), "\n", $eval_error, "\n";
    } ## end if ( not defined eval { $event_count = $recce->read...})
    my $value_ref = $recce->value( $self );
    if ( not defined $value_ref ) {
        die $self->show_last_expression(), "\n",
            "No parse was found, after reading the entire input\n";
    }

    package My_Actions;
    sub do_parens    { shift; return $_[1] }
    sub do_add       { shift; return $_[0] + $_[2] }
    sub do_subtract  { shift; return $_[0] - $_[2] }
    sub do_multiply  { shift; return $_[0] * $_[2] }
    sub do_divide    { shift; return $_[0] / $_[2] }
    sub do_pow       { shift; return $_[0]**$_[2] }
    sub do_first_arg { shift; return shift; }
    sub do_script    { shift; return join q{ }, @_ }

About this document

This page is the reference document for the recognizer objects of Marpa's SLIF (Scanless interface).

Internal and external scanning

The Scanless interface is so-called because it does not require the application to supply a scanner (lexer). The SLIF contains its own lexer, one whose use is integrated into its syntax. In this document, use of the SLIF's internal scanner is called internal scanning.

The SLIF allows applications that find it useful to do their own scanning. When an application bypasses the SLIF's internal scanner and does its own scanning, this document calls it external scanning. An application can use external scanning to supplement internal scanning, or to replace the SLIF's internal scanner entirely.

Locations

Input stream locations

An input stream location is the offset of a codepoint in the input stream. When the input stream is being treated as a string, input stream location corresponds to Perl's "pos()" location. In this document, the word ``location'' refers to location in the input stream unless otherwise specified.

Negative locations

Several methods allow locations and lengths to be specified as negative numbers. A negative location is a location counted from the end, so that -1 means location before the last character of the string, -2 the location before the second to last character of a string, etc. A negative length indicates a distance to a location counted from the end. A length of -1 indicates the distance to the end of the string, -2 indicates the distance to the location just before the last character of the string, etc.

For example, suppose that we are dealing with input stream locations. The span ("0, -1") is the entire input stream. The span ("-1, -1") is the last character of input stream. The span ("-2, -1") is the last two characters of the input stream. The span ("-2, 1") is the second to last character of the input stream.

G1 locations

In addition to input stream location, the SLIF also tracks G1 location. G1 location starts at zero, and increases by exactly one as each lexeme is read. G1 location is usually not the same as input stream location. There is also a concept of G1 length, which is simply length calculated in terms of G1 locations.

G1 location can be ignored most of the time, but it does become relevant to a small degree when dealing with ambiguous terminals, and to a greater degree when tracing the G1 grammar. (For those more familiar with Marpa's internals, the G1 location is the G1 Earley set index.)

Current location

The SLIF tracks the current location in the input stream, more usually simply called the current location. Locations are zero-based, so that location 0 is the start of the input stream. A location is said to point to the character after it, if there is such a character. For example, location 0 points to the first character of the input stream, unless the stream is of zero length, in which case there is no first character.

A current location equal to the length of the input stream indicates EOS (end of stream). In a zero length stream, location 0 is EOS. The EOS location never points to a character.

In the SLIF, when the current input stream location moves, it does not necessarily advance --- it can skip forward, or can be positioned to an earlier location. The application can skip sections of the input stream. The application is also free to revisit spans of the input stream as often as it wants.

Here are the guarantees:

Initially, the current location is 0.
The current location will never be negative.
The current location will never be greater than EOS.

Literals and G1 spans

Often it is useful to find the literal substring of the input which corresponds to a span of G1 locations. If an application reads the input monotonically within the G1 span this presents no complications.

``Monotonically'' here means that, for the G1 span "$g1_start, $g1_length", the application reads the G1 locations in sequence and one-by-one, starting at $g1_start and ending at "$g1_start+$g1_length". This is the usual case.

Reading the input monotonically is the default, and by far the most common case. But Marpa applications are free to skip forward in the stream, to jump backward, to reread the same input multiple times, etc., etc. It is entirely possible the final input stream location of a G1 span will be before the start of the G1 span.

In precise terms, the substring returned for a G1 span "$g1_start, $g1_length" is determined as follows: The string will start at the first input stream location in the span for G1 location "$g1_start+1". The end of the string will be at the last input stream location in the span for G1 location "$g1_start+$g1_length". When an application moves backward in the input, the end of the string, as calculated above, may be before the start of the string. When the end of a string is before its start, the substring returned will be the zero-length string.

Applications which do not read monotonically, but which also want to associate spans of G1 locations with the input stream, may need to reassemble the input based on their own ideas. The ``literal()'' method can assist in this process.

How internal scanning works

The SLIF always starts scanning using the "read()" method. Pedantically, this means scanning always begins with a phase of internal scanning. But that first phase may be of zero length, and after that, internal scanning does not have to be resumed.

Internal scanning can be resumed with the "resume()" method. Both the "read()" and "resume()" methods require the application to specify a span in the input stream. The "read()" method sets the input stream, and that input stream is the one used by all "resume()" method calls for that recognizer.

In what follows, the term ``internal scanning method'' refers to either the "read()" or the "resume()" method. After an internal scanning method, the current location will indicate how far in the input stream the internal scanning method actually read. If the internal scanning method paused before EOS, the current location will be the one at which it paused. If the internal scanning method pauses at EOS, the current location will be EOS. The return value of the "read()" and the "resume()" method is the current location.

EOS

The location of EOS depends on the $start and $length arguments to the last internal scanning method, and on the length of the input string.

If the $length argument of the last internal scanning method was non-negative, EOS will be at "$start+$length".
If the $length argument was negative, EOS will be at "$length + 1 + length $input_string".
The default length for the internal scanning methods is always -1, so that the default EOS is always at "length $input_string", the end of the input string.

Pauses in internal scanning

When a "read()" and the "resume()" method pauses, one of more of the following occurred.

A named event
One or more named events may have triggered. Named events are created by named event statements. They can also be created by lexeme pseudo-rules. Named events may be queried using the events() method().
A unnamed lexeme pause event
A lexeme pause that is not a named event may have triggered. Lexeme pauses are created by lexeme pseudo-rules. Applications can always name lexeme pause events, using the event adverb, and are strongly encouraged to do so. If all lexeme pauses are named, the check for unnamed events can be omitted. The presence or absence of an unnamed lexeme pause event may be checked for using the lexeme_pause() method.
EOS
EOS may have been reached. This may be checked for by comparing the current location with the expected EOS.

The input stream

For error message and other purposes, even external lexemes are required to correspond to a span of the input stream. An external scanner must set up a relationship to the input stream, even if that relationship is completely artificial.

One way to do this is to put an artificial preamble in front of the input stream. For example, the first 7 characters of the input stream could be a preamble containing the characters ""NO TEXT"". This preamble could be immediately followed by what is seen as the text from a more natural point of view. In this case, the initial call to the "read()" method could take the form "$slr->read($input_string, 7)". Lexemes corresponding to the artificial preamble would be read using a method call similar to "$slr->lexeme_read($symbol_name, 0, 7, $value)".

Constructor

    my $recce = Marpa::R2::Scanless::R->new( { grammar => $grammar } );

The "new()" method is the constructor for SLIF recognizers. The "new()" constructor accepts a hash of named arguments. The "grammar" named argument is required. All other named arguments are optional.

The following named arguments are allowed:

end

Most users will want to ignore this argument. It is an advanced argument, mainly for use in testing. The "end" named argument specifies the parse end, as a G1 location. The default is for the parse to end where the input did, so that the parse returned is of the entire input. The "end" named argument is not allowed once a parse series has begun.

grammar

The "new" method is required to have a "grammar" named argument. Its value must be an SLIF grammar object.

max_parses

If non-zero, causes a fatal error when that number of parse results is exceeded. "max_parses" is useful to limit CPU usage and output length when testing and debugging. Stable and production applications may prefer to count the number of parses, and take a less Draconian response when the count is exceeded.

The value must be an integer. If it is zero, there will be no limit on the number of parse results returned. The default is for there to be no limit.

ranking_method

The value must be a string: one of ""none"``, ''"rule"``, or ''"high_rule_only"``. When the value is ''"none"", Marpa returns the parse results in arbitrary order. This is the default. The "ranking_method" named argument is not allowed once evaluation has begun.

The ""rule"`` and ''"high_rule_only"" ranking methods allows the user to control the order in which parse results are returned by the "value" method, and to exclude some parse results from the parse series. For details, see the document on parse order.

semantics_package

Sets the semantic package for the recognizer. The setting of this argument takes precedence over any package implied by the blessing of the per-parse arguments to the SLIF recognizer's "value()" method. The semantics package is used when resolving action names to fully qualified Perl names. For more details on the SLIF semantics, see the document on SLIF semantics.

too_many_earley_items

The "too_many_earley_items" argument is optional, and very few applications will need it. If specified, it sets the Earley item warning threshold to a value other than its default. If an Earley set becomes larger than the Earley item warning threshold, a recognizer event is generated, and a warning is printed to the trace file handle.

Marpa parses from any BNF, and can handle grammars and inputs which produce very large Earley sets. But parsing that involves very large Earley sets can be slow. Large Earley sets are something most applications can, and will wish to, avoid.

By default, Marpa calculates an Earley item warning threshold for the G1 recognizer based on the size of the G1 grammar, and for each L0 recognizer based on the size of the L0 grammar. The default thresholds will never be less than 100. The default is the result of considerable experience and almost all users will be happy with it.

If the Earley item warning threshold is changed from its default, the change applies to both L0 and G1 --- currently there is no way to set them separately. If the Earley item warning threshold is set to 0, no recognizer event is generated, and warnings about large Earley sets are turned off. An Earley item threshold warning almost always indicates a serious issue, and turning these warnings off will rarely be what an application wants.

trace_terminals

If non-zero, traces the lexemes --- those tokens passed from the L0 parser to the G1 parser. This named argument is the best way to follow what the L0 parser is doing, and it is also very helpful for tracing the G1 parser.

trace_values

The trace_values named argument is a numeric trace level. If the numeric trace level is 1, Marpa prints tracing information as values are computed in the evaluation stack. A trace level of 0 turns value tracing off, which is the default. Traces are written to the trace file handle.

trace_file_handle

The value is a file handle. Trace output and warning messages go to the trace file handle. By default, the trace file handle is inherited from the grammar.

Basic mutators

read()

    $recce->read($p_input_string);

    $recce->read( \$string, 0, 0 );

Given a pointer to an input stream, "read()" parses it according to the grammar. Only a single call to "read()" is allowed for a scanless recognizer.

"read()" recognizes optional second and third arguments. The second argument is a location in the input stream at which internal scanning will start. The third argument is the length of the section of the input stream to be scanned before pausing. The default start location is zero. The default length is -1. Negative locations and lengths have the standard interpretation, as described above.

Start location and length can both be zero. This pauses internal scanning immediately and can be used to hand complete control of scanning over to an external scanner.

Completion named events can occur during the "read()" method. When a named event occurs, the "read()" method pauses. Named events can be queried using the Scanless recognizer's events() method. The "read()" method also pauses as specified with the Scanless DSL's pause adverb.

On failure, throws an exception. The call is considered successful if it ended because a parse was found, or because internal scanning was paused. On success, "read()" returns the location in the input stream at which internal scanning ended. This value may be zero.

series_restart()

    $slr->series_restart( { end => $i } );

The "series_restart()" method ends the current parse series, and starts another. It allows, as optional arguments, hashes of named arguments for the SLIF recognizer. These named arguments can be any of those allowed by the "set()" method.

"series_restart()" resets all the named arguments to their defaults. An application that wants a non-default named argument to have effect in each of its parse series must respecify it at the beginning of each parse series. "series_restart()" is particularly useful for the "end" and "semantics_package" named arguments, which cannot be changed once a parse series is underway. To change their values, an application must start a new parse series.

set()

    $slr->set( { max_parses => 42 } );

This method allows the named arguments to be changed after an SLIF grammar is created. Currently, the arguments that may be changed are "end", "max_parses", "semantics_package" and "trace_file_handle".

value()

    my $value_ref = $recce->value( $self );

The "value" method call evaluates the next parse tree in the parse series, and returns a reference to the parse result for that parse tree. If there are no more parse trees, the "value" method returns "undef".

Because Marpa parses ambiguous grammars, every parse is a series of zero or more parse trees. This series of zero or more parse trees is called a parse series. There are zero parse trees if there was no valid parse of the input according to the grammar.

The "value()" method allows one, optional argument. This argument can be a Perl scalar of any kind, but the most useful possibilities are references (blessed or unblessed) to hashes or array. If provided, the argument of the "value()" method explicitly specifies the per-parse argument for the parse tree. The per-parse argument will be the first argument of all Perl semantics closures, and can be used to share data within the tree, when that data does not conveniently fit into the bottom-up flow of parse tree evaluation. Symbol tables are one example of the kind of data which parses often require, but which it is not convenient to accumulate bottom-up.

If the "semantics_package" named argument of the SLIF recognizer was not specified, Marpa will use the package into which the per-parse argument was blessed as the semantics package --- the package in which to look for the parse's Perl semantic closures. In this case, Marpa will regard the per-parse arguments of all calls in the same parse series as the source of the semantics package, and it will require that the calls be consistent --- each call must have a per-parse argument, and that per-parse argument must be blessed into the semantics package.

Mutators for external scanning

activate()

        $slr->activate($_, 0) for @events;

The "activate()" method allows the recognizer to deactivate and reactivate named events. Named events allow the recognizer to stop for external scanning at conveniently defined locations. Named events can be defined for the prediction and completion of non-zero-length symbols, and nulled events can be defined to trigger when zero-length symbols are recognized.

The "activate()" method takes two arguments. The first is the name of an event, and the second (optional) argument is 0 or 1. If the argument is 0, the event is deactivated. If the argument is 1, the event is reactivated. An argument of 1 is the default. but, since an SLIF recognizer always starts with all defined events activated, 0 will probably be more common as the second argument to "activate()"

Location 0 events are triggered in the SLIF recognizer's constructor, before the "activate()" method can be called. This means that currently there is no way to deactivate location zero events.

The overhead imposed by events can be reduced by using the "activate()" method. But making many calls to the the "activate()" method purely for efficiency purposes will be counter-productive. Also, deactivated events still impose some overhead, so if an event is never used it should be commented out in the SLIF DSL.

lexeme_alternative()

            if ( not defined $recce->lexeme_alternative($token_name) ) {
                die
                    qq{Parser rejected token "$long_name" at position $start_of_lexeme, before "},
                    substr( $string, $start_of_lexeme, 40 ), q{"};
            }

The "lexeme_alternative()" method allows an external scanner to read ambiguous tokens. Most applications will prefer the simpler "lexeme_read()".

"lexeme_alternative()" takes one or two arguments. The first argument, which is required, is the name of a symbol to be read at the current location. The second argument, which is optional, is the value of the symbol. The value argument is interpreted as described for "lexeme_read()".

Any number of tokens may be read using "lexeme_alternative()" without advancing the current location. This allows an application to use ambiguous tokens. To complete reading at a G1 location, and advance the current G1 location to the next G1 location, use the "lexeme_complete()" method.

On success, returns a non-negative number. Returns "undef" if the token was rejected. Failures are thrown as exceptions.

lexeme_complete()

            next TOKEN
                if $recce->lexeme_complete( $start_of_lexeme,
                        ( length $lexeme ) );

The "lexeme_complete()" method allows an external scanner to read ambiguous tokens. Most applications will prefer the simpler "lexeme_read()".

The "lexeme_complete()" method requires two arguments, a input stream start location and a length. These are interpreted as described for the corresponding second and third arguments to "lexeme_read()". The "lexeme_complete()" method completes the reading of alternative tokens at the current G1 location, and advances the current G1 location by one. Current location in the input stream is moved to the location after the new lexeme, as indicated by the arguments.

Completion named events can occur during the "lexeme_complete()" method. Named events can be queried using the Scanless recognizer's events() method.

Return value: On success, "lexeme_complete()" returns the new current location. This will never be location zero, because a succesful call of "lexeme_complete()" always advances the location. On unthrown failure, "lexeme_complete()" returns 0.

lexeme_read()

    $re->lexeme_read( 'lstring', $start, $length, $value ) // die;

The "lexeme_read()" method reads a single, unambiguous, lexeme. It takes four arguments, only the first of which is required. The first argument is the lexeme's symbol name. The second and third arguments specify the span in the input stream to be associated with the lexeme. The last argument indicates its value.

The second and third arguments are, respectively, the start and length of a span in the input stream. The start defaults to the current location. If the pause span is defined, and the start of the pause lexeme is the same as the current location, length defaults to the length of the pause span. Otherwise length defaults to -1.

Negative values are allowed and are interpreted as described above. This span will be treated as the section of the input stream that corresponds to the tokens read at the current location. This correspondence may be artificial, but a span must always be specified.

The fourth argument specifies the value of the lexeme. If the value argument is omitted, the token's value will be a string containing the corresponding substring of the input stream. Omitting the value argument does not have the same effect as passing an explicit Perl "undef". If the value argument is an explicit Perl "undef", the value of the lexeme will be a Perl "undef".

    $slr->lexeme_read($symbol, $start, $length, $value)

is the equivalent of

    $slr->lexeme_alternative($symbol, $value)
    $slr->lexeme_complete($start, $length)

Current location in the input stream is moved to the place where "read()" paused or, if it never pauses, to "$start+$length". Current G1 location is advanced by one.

Completion named events can occur during the "lexeme_read()" method. Named events can be queried using the Scanless recognizer's events() method.

Return value: On success, "lexeme_read()" returns the new current location. This will never be location zero, because lexemes cannot be zero length. If the token was rejected, returns a Perl "undef". On other unthrown failure, returns 0.

resume()

    my $re = Marpa::R2::Scanless::R->new(
        {   grammar           => $parser->{grammar},
            semantics_package => 'MarpaX::JSON::Actions'
        }
    );
    my $length = length $string;
    for (
        my $pos = $re->read( \$string );
        $pos < $length;
        $pos = $re->resume()
        )
    {
        my ( $start, $length ) = $re->pause_span();
        my $value = substr $string, $start + 1, $length - 2;
        $value = decode_string($value) if -1 != index $value, '\\';
        $re->lexeme_read( 'lstring', $start, $length, $value ) // die;
    } ## end for ( my $pos = $re->read( \$string ); $pos < $length...)
    my $per_parse_arg = bless {}, 'MarpaX::JSON::Actions';
    my $value_ref = $re->value($per_parse_arg);
    return ${$value_ref};

The "resume()" method takes two arguments, a start location and a length. The default start location is the current location. The default length is -1. Negative arguments are interpreted as described above.

The "resume()" method resumes the SLIF's internal scanning, as described above.

Completion named events can occur during the "resume()" method. When a named event occurs, the "resume()" method pauses. Named events can be queried using the Scanless recognizer's events() method. The "resume()" method also pauses as specified with the Scanless DSL's pause adverb.

On success, "resume()" moves the current location to where it paused, or to the EOS. The return value is the new current location. On unthrown failure, "resume()" return a Perl "undef".

Accessors

ambiguity_metric()

    my $ambiguity_metric = $slr->ambiguity_metric();

Returns 1 if there is an unambiguous parse, and 2 or greater if there is a ambiguous parse. Returns 0 if called before parsing. Returns 0 or less than zero on other unthrown failure.

current_g1_location()

    my $current_g1_location = $slr->current_g1_location();

Returns the current G1 location.

events()

        EVENT:
        for my $event ( @{ $slr->events() } ) {
            my ($name) = @{$event};
            push @actual_events, $name;
        }

The "events()" method takes no arguments, and returns an array of event descriptors. It returns the empty array if there were no event.

Each named event descriptor is a reference to an array of one, and potentially more, elements. The first element of every named event descriptor is a string containing the name of the event, and this is typically the only element. In certain cases, there could be other elements of a named event descriptor, which will be as described for the type of named event. Named events are described in the SLIF DSL.

Events occur during the the Scanless recognizer's read(), resume(), lexeme_complete(), and lexeme_read() methods. Any subsequent call to an SLIF recognizer mutator may clear the list of triggered events, The assumption is that an application interested in events will call the "events()" method almost as soon as control is returned to it.

Named events are returned in order by type. Completion events are first. They are followed by the nulled events. These are in turn followed by prediction events. Within each type, the order of events is arbitrary.

Applications may find it convenient to turn specific events off, temporarily or permanently. Events may be activated or deactivated with the SLIF recognizer's activate() method.

exhausted()

    my $exhausted_status = $slr->exhausted();

The exhausted method returns a Perl true if parsing in a SLIF recognizer is exhausted, and a Perl false otherwise. Parsing is exhausted when the recognizer will not accept any further input.

An attempt to read input into an exhausted parser causes an exception to be thrown. The exception is all that most applications require, but this method allows the recognizer's exhaustion status to be discovered directly.

g1_location_to_span()

        my ( $span_start, $span_length ) =
            $slr->g1_location_to_span($g1_location);

G1 locations do not correspond to a single input stream location, but to a span of them. The "g1_location_to_span()" method returns an array of two elements, representing a span in the input stream. The first element of the array is the input stream location where the span starts. The second element of the array is the length of the span. As a special case, the input stream span for G1 location 0 is always (0,0).

Sometimes it is convenient to think of G1 location as corresponding to a single input stream location. When this is the case, what is usually intended is the last input stream location of the span. The last input stream location of the span will always be "$span_start+$span_length".

input_length()

    my $input_length = $slr->input_length();

The "input_length()" method accepts no arguments, and returns the length of the input stream.

last_completed()

    sub show_last_expression {
        my ($self) = @_;
        my $recce = $self->{recce};
        my ( $g1_start, $g1_length ) = $recce->last_completed('Expression');
        return 'No expression was successfully parsed' if not defined $g1_start;
        my $last_expression = $recce->substring( $g1_start, $g1_length );
        return "Last expression successfully parsed was: $last_expression";
    } ## end sub show_last_expression

    my ( $g1_start, $g1_length ) = $recce->last_completed('Expression');

Given the name of a symbol, returns the start G1 location and the length in G1 locations of the most recent match. If there was more than one most recent match, it returns the longest. If there was no match, returns the empty array in array context and a Perl false in scalar context.

line_column()

    my ( $start, $span_length ) = $re->pause_span();
    my ( $line,  $column )      = $re->line_column($start);

The "line_column()" method accepts one, optional, argument: a location in the input stream. The location defaults to the current location. "line_column()" returns the corresponding line and column position, as a 2-element array. The first element of the array is the line position, and the second element is the column position.

Numbering of lines and columns is 1-based, following UNIX editor tradition. Except at EOF, the line and column will be that of an actual character. At EOF the line number will be that of the last line, and the column number will be that of the last column plus one. Applications which want to treat EOF as a special case can test it for using the "pos()" method and the "input_length()" method.

A line is considered to end with any newline sequence as defined in the Unicode Specification 4.0.0, Section 5.8. Specifically, a line ends with one of the following:

a LF (line feed U+000A);
a CR (carriage return, U+000D), when it is not followed by a LF;
a CRLF sequence (U+000D,U+000A);
a NEL (next line, U+0085);
a VT (vertical tab, U+000B);
a FF (form feed, U+000C);
a LS (line separator, U+2028) or
a PS (paragraph separator, U+2029).

literal()

    my $literal_string = $re->literal( $start, $span_length );

The "literal()" method accepts two arguments, the start location and length of a span in the input stream. It returns the substring of the input stream corresponding to that span.

pause_lexeme()

   my $lexeme = $re->pause_lexeme();

The "pause_lexeme()" method accepts no arguments, and returns the name of the lexeme which caused the most recent pause. The pause lexeme is initially undefined and it is reset to undefined at the beginning of each call to the "read()" or "resume()" methods.

More than one lexeme may cause a pause. When this is the case, all the causal lexemes will be acceptable to the G1 grammar, and all the causal lexemes will have the same lexeme priority. When more than one lexeme causes a pause, the choice of pause lexeme is arbitrary. Applications may not rely on a particular choice, or on that choice being repeated, even when the choice is made in similar or identical circumstances.

Not every pause is caused by a lexeme. A pause often occurs because of the length argument of an internal scanning method. When the most recent pause was not caused by a lexeme, the pause lexeme is undefined. "pause_lexeme()" returns a Perl "undef" when the pause lexeme is undefined.

pause_span()

    my ( $start, $length ) = $re->pause_span();

The "pause_span()" method accepts no arguments, and returns the ``pause span'' as a 2-element array. The ``pause span'' is the start location and length of the lexeme which caused the most recent pause. The pause span is initially undefined and it is reset to undefined at the beginning of each call to the "read()" or "resume()" methods.

A pause is not always caused by a lexeme --- internal scanning may be paused because of the length argument of an internal scanning method. When the most recent pause was not caused by a lexeme, no span can be associated with it, and the pause span is undefined. "pause_span()" returns a Perl "undef" if the pause span is undefined.

pos()

    my $pos = $slr->pos();

The "pos()" method accepts no arguments, and returns the current input stream location.

progress()

    my $progress_output = $slr->progress();

Returns an array that describes the progress of a parse at a location. With no argument, "progress()" reports progress at the current location. If a G1 location is given as its argument, "progress()" reports progress at that G1 location. The G1 location may be negative. An argument of -X will be interpreted as location N+X+1, where N is the current G1 location. In other words, an argument of -1 indicates the current G1 location, an argument of -2 indicates the G1 location just before the current one, etc.

The progress reports returned by the "progress()" method identify rules by their G1 rule ID. G1 rules IDs can be converted to a list of the rule's symbols using the "rule()" method of the SLIF grammar. Details on progress reports can be found in their own document.

show_progress()

    my $show_progress_output = $slr->show_progress();

Shows the progress of the G1 parse. For a description of its output, see Marpa::R2::Progress.

With no arguments, the string contains reports for the current location. If locations are specified as arguments to "show_progress()", they need to be G1 locations.

With a single integer argument N, the string contains reports for G1 location N. With two numeric arguments, N and M, the arguments are interpreted as the start and end points of a range of G1 locations and the returned string contains reports for all locations in the range.

If an argument is negative, -N, it indicates the Nth location counting backward from the furthest location of the parse. For example, if 42 was the furthest G1 location, -1 would be G1 location 42 and -2 would be location 41. For example, the method call "$recce->show_progress(-3, -1)" returns reports for the last three G1 locations of the parse. The method call "$recce->show_progress(0, -1)" will print progress reports for the entire parse.

Locations are G1 locations instead of string offsets, for two reasons. First, G1 parse state is only defined at the start of parsing, and at the end of a non-discarded lexeme. Therefore many strings offsets will not have a G1 parse state. Second, SLIF recognizers using external scanning are allowed to rescan the same string repeatedly. Therefore, a single string offset may have many G1 parse states.

substring()

    my $last_expression = $recce->substring( $g1_start, $g1_length );

Given a G1 span --- that is, a G1 start location and a length in G1 locations --- the "substring()" method returns a substring of the input stream. A G1 length of zero will produce the zero-length string.

The substring of the input stream is determined on the assumption that the application reads the input monotonically. When this is not the case, the substring is determined as described above.

terminals_expected()

    my @terminals_expected = @{$slr->terminals_expected()};

Returns a reference to a list of strings, where the strings are the names of the lexemes acceptable at the current location. The presence of a lexeme in this list means that lexeme will be acceptable in the next call of the "resume()" method.

This is highly useful for Ruby Slippers parsing. A more fine-tuned approach is to identify the lexemes of interest and create ``predicted symbol'' events for them.

Discouraged methods

Methods in this section continue to be supported, but their use is discouraged in favor of other, better solutions. New applications should avoid using discouraged methods.

event()

            my $event    = $slr->event($event_ix);

Use of this method is discouraged in favor of the more efficient events() method. The "event()" method requires one argument, an event index. It returns a descriptor of the named event with that index, or a Perl "undef" if there is no such event. For more details on events, see the description of the events() method.

last_completed_range()

Use of this method is discouraged in favor of ``last_completed()''. Given the name of a symbol, "last_completed_range()" returns the G1 start and G1 end locations of the most recent match. If there was more than one most recent match, "last_completed_range()" returns the longest. If there was no match, "last_completed_range()" returns the empty array in array context and a Perl false in scalar context.

range_to_string()

Use of this method is discouraged in favor of ``substring()''. Given a G1 start and a G1 end location, "range_to_string()" returns the substring of the input stream that is between the two. The "range_to_string()" method assumes that the application read forward smoothly in the input stream, while reading the sequence of G1 locations. When that is not the case, "range_to_string()" behaves in much the same way as described above for ``substring()''.

Copyright and License

  Copyright 2014 Jeffrey Kegler
  This file is part of Marpa::R2.  Marpa::R2 is free software: you can
  redistribute it and/or modify it under the terms of the GNU Lesser
  General Public License as published by the Free Software Foundation,
  either version 3 of the License, or (at your option) any later version.
  Marpa::R2 is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  Lesser General Public License for more details.
  You should have received a copy of the GNU Lesser
  General Public License along with Marpa::R2.  If not, see
  http://www.gnu.org/licenses/.