xtokid - Print all tokens found in a source file

       Print all tokens found in a source file.

       -i, --include=LANGS
              include languages in LANGS (default: "C C++ asm")

       -x, --exclude=LANGS
              exclude languages in LANGS

       -l, --lang-option=L:OPT pass OPT as a default for language L (see below)

       -m, --lang-map=MAPFILE
              use MAPFILE to map file names onto source language

       -d, --default-lang=LANG
              make LANG the default source language

       -p, --prune=NAMES
              exclude the named files and/or directories

       --files0-from=F
              tokenize only the files specified by NUL-terminated names in file F

       --help display this help and exit --version           output version information and exit

       The following arguments apply to the language-specific scanners:

   Clanguage:-k,--keep=CHARS
              Allow CHARS in single-token strings, keep the result

       -i,--ignore=CHARS
              Allow CHARS in single-token strings, toss the result

       -u,--strip-underscore
              Strip a leading underscore from single-token strings

   C++language:-k,--keep=CHARS
              Allow CHARS in single-token strings, keep the result

       -i,--ignore=CHARS
              Allow CHARS in single-token strings, toss the result

       -u,--strip-underscore
              Strip a leading underscore from single-token strings

   Javalanguage:-k,--keep=CHARS
              Allow CHARS in single-token strings, keep the result

       -i,--ignore=CHARS
              Allow CHARS in single-token strings, toss the result

       -u,--strip-underscore
              Strip a leading underscore from single-token strings

   Assemblylanguage:-c,--comment=CHARS
              Any of CHARS starts a comment until end-of-line

       -k,--keep=CHARS
              Allow CHARS in tokens, and keep the result

       -i,--ignore=CHARS
              Allow CHARS in tokens, and toss the result

       -u,--strip-underscore
              Strip a leading underscore from tokens

       -n,--no-cpp
              Don't handle C pre-processor directives

   Textlanguage:-i,--include=CHAR-CLASS
              Treat characters of CHAR-CLASS as token constituents

       -x,--exclude=CHAR-CLASS
              Treat characters of CHAR-CLASS as token delimiters

   Perllanguage:-i,--include=CHAR-CLASS
              Treat characters of CHAR-CLASS as token constituents

       -x,--exclude=CHAR-CLASS
              Treat characters of CHAR-CLASS as token delimiters

       -d,--dtags
              Include documentation tags

       Lisp language:

       xtokid - Print all tokens found in a source file

       Report bugs to bug-idutils@gnu.org

       The  full  documentation for the id-utils package is maintained as a Texinfo manual.  If the info and id-utils packages are properly installed, the command

              info'IDdatabase'

       should give you access to the complete manual.

xtokid - 4.6.28-15da                                May 2020                                           XTOKID(1)

xtokid [OPTION]... [FILE]...