Understanding the Uniq Command

The uniq command in Unix-like operating systems is used to filter adjacent matching lines of a file. It's often used in conjunction with the sort command to remove duplicate lines from a file, as uniq only compares adjacent lines. This tool is essential for text processing and data cleaning tasks for developers.

Common Uniq Command Usages

Here are some of the most common ways to use the uniq command:

Show All Lines Without Duplication

To display all lines from a file, removing any adjacent duplicate lines, you can pipe the output of sort to uniq. This is the most basic usage and effectively gives you a list of unique entries.

# To show all lines without duplication:
# (`sort -u` and `uniq` have the same effect.)
sort <file> | uniq

Show Only Unique Lines

If you want to see only the lines that appear exactly once in the sorted file, use the -u option with uniq.

# To show not duplicated lines:
sort <file> | uniq -u

Show Only Duplicated Lines

To identify and display only the lines that are duplicated (i.e., appear more than once), use the -d option.

# To show duplicated lines only:
sort <file> | uniq -d

Count All Lines

To count the occurrences of each unique line in a file, use the -c option. This is useful for frequency analysis.

# To count all lines:
sort <file> | uniq -c

Count Not Duplicated Lines

Combine -u and -c to count only those lines that appear exactly once.

# To count not duplicated lines:
sort <file> | uniq -uc

Count Only Duplicated Lines

To count only the lines that are duplicated, use the -d and -c options together.

# To count only duplicated lines:
sort <file> | uniq -dc

Further Resources

• Man page for uniq
• GNU Coreutils uniq documentation
• Stack Overflow: How to get unique lines from a file in Linux