Linux uniq Command: Removing Duplicate Lines from Sorted Files

The uniq command in Linux removes duplicate lines from a sorted file. That's why it is often used together with the sort command.

The most common options are:

• -c: Indicates at the beginning of the lines the number of occurrences;
• -d: Prints only duplicate lines;
• -i: Ignores the difference between uppercase and lowercase;
• -u: Prints only unique lines, which don't have duplicates.

Without sorting, uniq cannot eliminate duplicates. See the example below:

$ uniq supermercado arrozfeijãocarnebatataalfacetomatearrozcarne

$ sort supermercado | uniqalfacearrozbatatacarnefeijãotomate

The uniq command is very useful for removing duplicates from files with data sources, especially text files, CSV, etc.

The "-i" option makes uniq ignore whether items are in uppercase or lowercase.

The "-D" option shows only duplicate lines:

$ sort supermercado | uniq -D arrozarrozcarnecarne

Learn much more about Linux in our online course. You can enroll here. If you already have an account, or want to create one, just log in or create your user here.

Did you like it? Share