This class does very little of its own tokenization. Instead the token classes contain external
tokenization routines, whose name is '__PPIX_TOKENIZER__' concatenated with the current mode of the
tokenizer ('regexp' for regular expressions, 'repl' for the replacement string).
These external tokenizers are called as static methods, and passed the "PPIx::Regexp::Tokenizer" object
and the current character in the character stream.
If the external tokenizer wants to make one or more tokens, it returns an array containing either length
in characters for tokens of the tokenizer's own class, or the results of one or more "make_token" calls
for tokens of an arbitrary class.
If the external tokenizer is not interested in the characters starting at the current position it simply
returns.
The following methods are for the use of external tokenizers, and arenotpartofthepublicinterfacetothisclass.capture
if ( $tokenizer->find_regexp( qr{ \A ( foo ) }smx ) ) {
foreach ( $tokenizer->capture() ) {
print "$_\n";
}
}
This method returns all the contents of any capture buffers from the previous call to "find_regexp". The
first element of the array (i.e. element 0) corresponds to $1, and so on.
The captures are cleared by "make_token", as well as by another call to "find_regexp".
cookie
$tokenizer->cookie( foo => sub { 1 } );
my $cookie = $tokenizer->cookie( 'foo' );
my $old_hint = $tokenizer->cookie( foo => undef );
This method either creates, deletes, or accesses a cookie.
A cookie is a code reference which is called whenever the tokenizer makes a token. If it returns a false
value, it is deleted. Explicitly setting the cookie to "undef" also deletes it.
When you call "$tokenizer->cookie( 'foo' )", the current cookie is returned. If you pass a new value of
"undef" to delete the token, the deleted cookie (if any) is returned.
When the "make_token" method calls a cookie, it passes it the tokenizer and the token just made. If a
token calls a cookie, it is recommended that it merely pass the tokenizer, though of course the token can
do whatever it wants.
The cookie mechanism seems to be a bit of a crock, but it appeared to be more work to fix things up in
the lexer after the tokenizer got something wrong.
The recommended way to write a cookie is to use a closure to store any necessary data, and have a call to
the cookie return the data; otherwise the ultimate consumer of the cookie has no way to access the data.
Of course, it may be that the presence of the cookie at a certain point in the parse is all that is
required.
expect
$tokenizer->expect( 'PPIx::Regexp::Token::Code' );
This method inserts a given class at the head of the token scan, for the next iteration only. More than
one class can be specified. Class names can be abbreviated by removing the leading 'PPIx::Regexp::'.
If no class is specified, this method does nothing.
The expectation lasts from the next time "get_token" is called until the next time "make_token" makes a
significant token, or until the next "expect" call if that is done sooner.
find_regexp
my $end = $tokenizer->find_regexp( qr{ \A \w+ }smx );
my ( $begin, $end ) = $tokenizer->find_regexp(
qr{ \A \w+ }smx );
This method finds the given regular expression in the content, starting at the current position. If
called in scalar context, the offset from the current position to the end of the matched string is
returned. If called in list context, the offsets to both the beginning and the end of the matched string
are returned.
find_matching_delimiter
my $offset = $tokenizer->find_matching_delimiter();
This method is used by tokenizers to find the delimiter matching the character at the current position in
the content string. If the delimiter is an opening bracket of some sort, bracket nesting will be taken
into account.
When searching for the matching delimiter, the back slash character is considered to escape the following
character, so back-slashed delimiters will be ignored. No other quoting mechanisms are recognized,
though, so delimiters inside quotes still count. This is actually the way Perl works, as
$ perl -e 'qr<(?{ print "}" })>'
demonstrates.
This method returns the offset from the current position in the content string to the matching delimiter
(which will always be positive), or undef if no match can be found.
get_mode
This method returns the name of the current mode of the tokenizer.
get_start_delimiter
my $start_delimiter = $tokenizer->get_start_delimiter();
This method is used by tokenizers to access the start delimiter for the regular expression.
get_token
my $token = $tokenizer->make_token( 3 );
my @tokens = $tokenizer->get_token();
This method returns the next token that can be made from the input stream. It is not part of the external
interface, but is intended for the use of an external tokenizer which calls it after making and retaining
its own token to look at the next token ( if any ) in the input stream.
If any external tokenizer calls get_token without first calling make_token, a fatal error occurs; this is
better than the infinite recursion which would occur if the condition were not trapped.
An external tokenizer must return anything returned by get_token; otherwise tokens get lost.
interpolates
This method returns true if the top-level structure being tokenized interpolates; that is, if the
delimiter is not a single quote.
make_token
return $tokenizer->make_token( 3, 'PPIx::Regexp::Token::Unknown' );
This method is used by this class (and possibly by individual tokenizers) to manufacture a token. Its
arguments are the number of characters to include in the token, and optionally the class of the token. If
no class name is given, the caller's class is used. Class names may be shortened by removing the initial
'PPIx::Regexp::', which will be restored by this method.
The token will be manufactured from the given number of characters starting at the current cursor
position, which will be adjusted.
If the given length would include characters past the end of the string being tokenized, the length is
reduced appropriately. If this means a token with no characters, nothing is returned.
match
if ( $tokenizer->find_regexp( qr{ \A \w+ }smx ) ) {
print $tokenizer->match(), "\n";
}
This method returns the string matched by the previous call to "find_regexp".
The match is set to "undef" by "make_token", as well as by another call to "find_regexp".
modifier_duplicate
$tokenizer->modifier_duplicate();
This method duplicates the modifiers on the top of the modifier stack, with the intent of creating a
locally-scoped copy of the modifiers. This should only be called by an external tokenizer that is
actually creating a modifier scope. In other words, only when creating a PPIx::Regexp::Token::Structure
token whose content is '('.
modifier_modify
$tokenizer->modifier_modify( name => $value ... );
This method sets new values for the modifiers in the local scope. Only the modifiers whose names are
actually passed have their values changed.
This method is intended to be called after manufacturing a PPIx::Regexp::Token::Modifier token, and
passed the results of its "modifiers" method.
modifier_pop
$tokenizer->modifier_pop();
This method removes the modifiers on the top of the modifier stack. This should only be called by an
external tokenizer that is ending a modifier scope. In other words, only when creating a
PPIx::Regexp::Token::Structure token whose content is ')'.
Note that this method will never pop the last modifier item off the stack, to guard against unmatched
right parentheses.
modifier_seen
$tokenizer->modifier_seen( 'i' )
and print "/i was seen at some point.\n";
Unlike modifier(), this method returns a true value if the given modifier has been seen in any scope
visible from the current location in the parse. There is no magic for group match semantics ( /a, /aa,
/d, /l, /u) or modifiers that can be repeated, like /x and /xx, or /e and /ee.
peek
my $character = $tokenizer->peek();
my $next_char = $tokenizer->peek( 1 );
This method returns the character at the given non-negative offset from the current position. If no
offset is given, an offset of 0 is used.
If you ask for a negative offset or an offset off the end of the sting, "undef" is returned.
ppi_document
This method makes a PPI document out of the remainder of the string, and returns it.
prior_significant_token
$tokenizer->prior_significant_token( 'can_be_quantified' )
and print "The prior token can be quantified.\n";
This method calls the named method on the most-recently-instantiated significant token, and returns the
result. Any arguments subsequent to the method name will be passed to the method.
Because this method is designed to be used within the tokenizing system, it will die horribly if the
named method does not exist.
If called with no arguments at all the most-recently-instantiated significant token is returned.
strict
say 'Parse is ', $tokenizer->strict() ? 'strict' : 'lenient';
This method simply returns true or false, depending on whether the 'strict' option to "new()" was true or
false.
Install Now
Emojis
PNG Icons