ids2ngram - generate n-gram data file from ids file

       Originally   written   by   Phill.Zhang   <phill.zhang@sun.com>.    Currently   maintained   by  Kov.Chai
       <tchaikov@gmail.com>.

ids2ngram generates idngram file, which is a sorted [id1,..,idN,freq] array, from binary id stream files.
       Here, the id stream files are always generated by mmseg or slmseg. Basically, it finds all occurrence of
       n-words tuples (i.e. the tuple of (id1,..,idN)), and sorts these tuples by the lexicographic order of the
       ids make up the tuples, then write them to specified output file.

       Following  example  will  use  three  input  idstream  file  idsfile[1,2,3]  to generate the idngram file
       all.id3gram. Each para (internal map size or hash size) would  be  1024000,  using  swap  file  for  temp
       result. All temp para result would eventually be merged to got the final result.

       ids2ngram-n3-s/tmp/swap-oall.id3gram-p1024000idsfile1idsfile2idsfile3

       The input file is presented as a binary id stream, which looks like:
           [id0,...,idX]

       ids2ngram - generate n-gram data file from ids file

       All the following options are mandatory.

       -n,--NMaxN
           Generates  N-gram  result.  ids2ngram does only support uni-gram, bi-gram, and trigram, so any number
           not in the range of 1..3 is not valid.

       -s,--swapswap-file
           Specify the temporary intermediate file.

       -o, --outoutput-file
           Specify the result idngram file, e.g. the array of [id1, ..., idN, freq]

       -p, --paraN
           Specify the maximum n-gram items per paragraph. ids2ngram writes to the  temporary  file  on  a  per-
           paragraph  basis.  Every  time it writes a paragraph out, it frees the corresponding memory allocated
           for it. When your computer system permits, a higher N is suggested. This can speed up the  processing
           speed because of less I/O.

mmseg(1), slmseg(1), slmbuild (1).

perl v5.36.0                                       2023-08-08                                       IDS2NGRAM(1)

       ids2ngram [option]... ids_file...