MKDoc::XML::Tokenizer - Tokenize XML the REX way

my$tokens=MKDoc::XML::Tokenizer->process_data($some_xml);
       Splits $some_xml into a list of MKDoc::XML::Token objects and returns an array reference to the list of
       tokens.

   my$tokens=MKDoc::XML::Tokenizer->process_file('/some/file.xml');
       Same as MKDoc::XML::Tokenizer->process_data ($some_xml), except that it reads $some_xml from
       '/some/file.xml'.

       Copyright 2003 - MKDoc Holdings Ltd.

       Author: Jean-Michel Hiver

       This module is free software and is distributed under the same license as Perl itself. Use it at your own
       risk.

ThismoduledoeslowlevelXMLmanipulation.ItwillsomehowparseevenbrokenXMLandtrytodosomethingwithit.Donotuseitunlessyouknowwhatyou'redoing.

       MKDoc::XML::Tokenizer - Tokenize XML the REX way

       MKDoc::XML::Tokenizer works with MKDoc::XML::Token, which can be used when building a full tree is not
       necessary. If you need to build a tree, look at MKDoc::XML::TreeBuilder.

       MKDoc::XML::Token MKDoc::XML::TreeBuilder

perl v5.36.0                                       2022-10-13                         MKDoc::XML::Tokenizer(3pm)

       MKDoc::XML::Tokenizer is a module which uses Robert D. Cameron REX technique to parse XML (ignore the
       carriage returns):

         [^<]+|<(?:!(?:--(?:[^-]*-(?:[^-][^-]*-)*->?)?|\[CDATA\[(?:[^\]]*](?:[^\]]+])
         *]+(?:[^\]>][^\]]*](?:[^\]]+])*]+)*>)?|DOCTYPE(?:[ \n\t\r]+(?:[A-Za-z_:]|[^\
         x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x00-\x7F])*(?:[ \n\t\r]+(?:(?:[A-Za-z_:]|[^\
         x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x00-\x7F])*|"[^"]*"|'[^']*'))*(?:[ \n\t\r]+)
         ?(?:\[(?:<(?:!(?:--[^-]*-(?:[^-][^-]*-)*->|[^-](?:[^\]"'><]+|"[^"]*"|'[^']*'
         )*>)|\?(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x00-\x7F])*(?:\?>|[\
         n\r\t ][^?]*\?+(?:[^>?][^?]*\?+)*>))|%(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0
         -9_:.-]|[^\x00-\x7F])*;|[ \n\t\r]+)*](?:[ \n\t\r]+)?)?>?)?)?|\?(?:(?:[A-Za-z
         _:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x00-\x7F])*(?:\?>|[\n\r\t ][^?]*\?+(?
         :[^>?][^?]*\?+)*>)?)?|/(?:(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.-]|[^\x
         00-\x7F])*(?:[ \n\t\r]+)?>?)?|(?:(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.
         -]|[^\x00-\x7F])*(?:[ \n\t\r]+(?:[A-Za-z_:]|[^\x00-\x7F])(?:[A-Za-z0-9_:.-]|
         [^\x00-\x7F])*(?:[ \n\t\r]+)?=(?:[ \n\t\r]+)?(?:"[^<"]*"|'[^<']*'))*(?:[ \n\
         t\r]+)?/?>?)?)

       That's right. One big regex, and it works rather well.

         my $tokens = MKDoc::XML::Tokenizer->process_data ($some_xml);
         foreach my $token (@{$tokens})
         {
             print "'" . $token->as_string() . "' is text\n" if (defined $token->text());
             print "'" . $token->as_string() . "' is a self closing tag\n" if (defined $token->tag_self_close());
             print "'" . $token->as_string() . "' is an opening tag\n" if (defined $token->tag_open());
             print "'" . $token->as_string() . "' is a closing tag\n" if (defined $token->tag_close());
             print "'" . $token->as_string() . "' is a processing instruction\n" if (defined $token->pi());
             print "'" . $token->as_string() . "' is a declaration\n" if (defined $token->declaration());
             print "'" . $token->as_string() . "' is a comment\n" if (defined $token->comment());
             print "'" . $token->as_string() . "' is a tag\n" if (defined $token->tag());
             print "'" . $token->as_string() . "' is a pseudo-tag (NOT text and NOT tag)\n" if (defined $token->pseudotag());
             print "'" . $token->as_string() . "' is a leaf token (NOT opening tag)\n" if (defined $token->leaf());
         }