Text Filters in Linux: sort, cut, grep, awk, and sed with Examples

Linux has several tools for working with and transforming text files. These tools are called text filters and are useful for working with scripts, verifying log files, etc.

cat command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=7

Usage:

$ cat [options] file

When the options appear between brackets, they are indeed optional.

The cat command concatenates files, prints their contents to the screen, and can also receive text typed from the keyboard to a file.

Let's see how to create a file with just a few lines of text:

$ cat > teste.txt

Now you can type any text. When finished, press Ctrl d on an empty line to end data entry and save the file teste.txt.

To see the contents of the newly created file:

$ cat teste.txt

cat can also be used to concatenate files.

$ cat texto1.txt > texto.txt

Note that in this example the contents of the file texto.txt is replaced by texto1.txt.

To append the contents of texto1.txt to the end of the file texto.txt the correct way would be:

$ cat texto1.txt >> texto.txt

cut command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=15

Usage:

$ cut options file

The cut command literally translated means to cut. It reads the contents of one or more files and outputs a vertical column.

Its most common options are:

• -b number: Prints a vertical list with byte number (from left to right);
• -c number: Prints a vertical list with character number (from left to right);
• -d delimiter: Sets the delimiter that separates one column from another. The default is Tab;
• -f number: Prints column number.

Examples:

$ cut –d : -f 1 /etc/passwd

$ cut –b 1 /etc/passwd

$ cat /etc/hosts | cut –f 2

expand command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=24

Usage:

$ expand [options] file

The expand command converts Tab (tabulation) within texts to the corresponding number of spaces. It is useful to make a text using tabulation more suitable for certain devices, such as video, printer, files, etc.

Frequently used options are:

• -t number: Specifies the number of spaces that the tab contains. The default is 8;
• -i: Converts only at the beginning of lines.

Example:

$ expand LEIAME.TXT

unexpand command

Usage:

$ unexpand [options] file

The unexpand command converts simple space to TAB (tabulation) within texts, at the beginning of lines. It is the inverse of the expand command.

Frequently used options are:

• -t number: Specifies the number of spaces that the tab contains. The default is 8;
• -a: Converts all spaces instead of only spaces at the beginning of lines.

Example:

$ unexpand LEIAME.TXT

fmt command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=28

Usage:

$ fmt [options]

The fmt command (format) formats a text with a specific width. It can remove spaces or add spaces as needed for the desired width. The default is 75 characters.

The frequently used option is:

• -w number: Sets the desired width for the text.

Example:

$ fmt –w 50 LEIAME.TXT

head command

&index=38&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb

Usage:

$ head [options]

The head command (header) displays the first 10 lines from the beginning of the text.

The frequently used option is:

• -n number: Sets the number of lines that head will display.

Example:

$ head –n 50 LEIAME.TXT

join command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=45

Usage:

$ join [options] file1 file2

The join command joins the lines from both files that have a common index. The join command can be used as a simple database.

The frequently used options are:

• -j1 number: Chooses the number field as the index for file1.
• -j2 number: Chooses the number field as the index for file2.
• -j number: Chooses the number field as the index for both files.

Example:

Assume that file1 contains the following:

1      GZH-1234

2      HYD-2389

3      GIS-2348

And file2 has the following:

1      Fiat Uno Mille Smart

2      Audi A3

3      Monza

After running the command:

$ join –j 1 file1 file2

The output will be:

1      GZH-134 Fiat Uni Mille Smart

2      HYD-2389 Audi A3

3      GIS-2348 Monza

TABLE 5 - Symbols to differentiate header and footer

Symbol	Description
\:\:\:	Symbol used to start the text header
\:\:	Symbol used to start the body  of the text
\:	Symbol used to start the text footer

Note that the join command requires both files to contain an index, as shown in the example.

nl command

Usage:

&index=65&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb

$ nl [options] [file]

The nl command (number lines) is used to number the lines of a file. The command considers special conditions for the file header and footer.

The frequently used options are:

• -h suboption: Used to format the text header. The default is not to number the header;
• -b suboption: Used to format the text body. The default is to number only non-empty lines;
• -f suboption: Used to format the text footer. The default is not to number the footer.

The suboptions are:

• A: Number all lines;
• t: Number only filled lines;
• n: Do not number lines.

Example:

Assume that file.txt has the following:

\:\:\:

Software Engineering Students Grades and Attendance Report

—————————————————————————————————

Name   Grade          Freq.         Result

—————————————————————————————————

\:\:

Carlos Torres         8,5             80%           Approved

José Antônio          10,0           100%         Approved

Maria de Lourdes  10,0           100%         Approved

Mário Cabral          9,5             100%         Approved

\:

—————————————————————————————————

And we use the nl command:

$ nl file.txt

The result will be:

Software Engineering Students Grades and Attendance Report

—————————————————————————————————

Name   Grade          Freq.         Result

—————————————————————————————————

1 Carlos Torres      8,5             80%           Approved

2 José Antônio       10,0           100%         Approved

3 Maria de Lourdes                 10,0           100%         Approved

4 Mário Cabral       9,5             100%         Approved

—————————————————————————————————

od command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=69

Usage:

$ od [options] [file]

The od command (Octal Dump) is used to display the contents of a file in hexadecimal, octal, ASCII formats and character names.

The frequently used option is:

• -t type: Specifies the output type that the od command should generate.

The possible types are:

• a: Character name;
• c: ASCII;
• o: Octal;
• x: Hexadecimal.

Example:

$ cat > file.txt

Linux Certification

Ctrl d

$ od –t x file.txt

0000000 74726543 63696669 6fe3e761 6e694c20

0000020 00007875

0000022

Comando paste

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=72

Usage:

$ paste [options] file1 file2

The paste command is used to concatenate lines from multiple files into vertical columns.

The frequently used options are:

• -d's': Separates columns with the symbol s inside single quotes;
• -s: Concatenates all content of a file with one line per file.

Example:

Suppose file1 has the following content:

lmmoraes

rrodrigues

aduarte

And file2 has the following content:

provider.com.br

provider2.com.br

provider3.com.br

When we use the paste command, the result will be:

$ paste –d'@' file1 file2

[email protected]

[email protected]

[email protected]

Comando pr

&index=76&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb

Usage:

$ pr [options] file

The pr command (printing) formats a text file for paginated output with defined headers, margins, and width. It is useful for formatting raw text for printing. The header consists of date, time, filename, and page number.

The frequently used options are:

• -d: Specifies double spacing;
• -l number: Specifies the number of characters of page width. The default is 66 characters;
• -o number: Specifies the number of spaces for the left margin.

Example:

$ pr –l 75 –o 5 file.txt

Comando split

Usage:

$ split [options] input_file output_file

The split command is used to split large files into n-smaller files. The output files are generated according to the number of lines of the input file.

The default is to split the file every 1000 lines. The output filenames follow the pattern outputfileaa outputfileab outputfileac, and so on.

The most frequent option is:

• -n: Where n is the number of lines that will split the input file.

Example:

$ split -20 file1.txt outputfile.txt

Comando tail

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=102

Usage:

$ tail [options] file

The tail command displays the last 10 lines of a file. It works as the opposite of the head command.

The most frequent options are:

• -n number: Specifies the number of last lines that tail will show from a file;
• -f: Shows the last lines of a file continuously while another process writes more lines. Very useful for viewing LOG files.

Example:

$ tail –n 50 /var/log/messages

$ tail –f /var/log/messages

Comando tr

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=108

Usage:

$ tr [options] search_variable replace_variable

The tr command swaps one variable for another specified one. This command does not work directly with files, therefore it must be used with the standard output of another command.

The most common options are:

• -d: Deletes occurrences of the search variable;
• -s: Suppresses repeated occurrences of the search variable.

Example:

$ cat arquivo1 | tr a-z A-Z

In this example, the tr command swaps all letters from a to z to uppercase.

$ cat arquivo1 | tr –d a

In this example, the tr command deletes the letter a.

$ cat arquivo1 | tr –s 1

In this example, the tr command suppresses the repeated occurrences of the number 1.

The tr command can swap characters from the search variable to the swap variable, but the number of characters must be the same in both.

wc command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=125

Usage:

$ wc [options] [files]

The wc command counts the lines, words, and characters in one or more files. If more than one file is passed as an argument, it will display statistics for each file and also the total.

The most common options are:

• -c: Counts the number of characters in one or more files;
• -l: Counts the number of lines in one or more files;
• -L: Counts the number of characters in the longest line of the file;
• -w: Counts the words in one or more files.

Example:

$ wc LEIAME.TXT

sort command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=96

Usage:

$ sort [-b] [-d] [-f] [-i] [-m] [-M] [-n] [-r] [-u] file [-o output_file]

The sort command sorts the lines of a text file.

Its options are:

• -b: Ignores leading spaces on the line;
• -d: Sorts lines alphabetically and ignores punctuation;
• -f: Ignores the difference between uppercase and lowercase;
• -I: Ignores control characters;
• -m: Merges two or more files into one sorted output file;
• -M: Treats the first three letters of lines as a month (e.g. JAN);
• -n: Sorts by numbers at the start of lines;
• -r: Sorts in reverse order;
• -u: If a line is duplicated, shows only the first one;
• -o: Sends the command output to the file.

Example:

$ sort arquivo

uniq command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=116

Usage:

$ uniq [Options]... [Input_File [Output_File]]

The uniq command removes duplicate lines from a sorted file.

The most common options are:

• -c: Indicates at the beginning of lines the number of occurrences;
• -d: Prints only duplicate lines;
• -i: Ignores the difference between uppercase and lowercase;
• -u: Prints only unique lines, which have no duplicates.

sed command

&list=PL6IznNGWCdD89jP8kiacplFQcJlmMmjhb&index=90

Usage:

$ sed [Options] {script} file

The sed command is a powerful string editor for filtering or editing text streams.

The most common options are:

• -e: Adds a script to the commands to be executed;
• -f file: Adds the content of a file as a script to be executed;
• -r: Uses regular expressions in the script.

Examples:

To substitute expressions with others, we use "s", with "/" as delimiters, so that the first occurrence is the text to search for, and the second the text that will be replaced. Note that sed will only replace the first occurrence of each line, and it is case-sensitive. The standard output of sed will be the terminal.

$ cat arquivo

Hoje fará calor de noite. A Noite é bela.

$ sed s/noite/dia/ arquivo

Hoje fará calor de dia. A Noite é bela.

Learn much more about Linux in our online course. You can enroll here. If you already have an account, or want to create one, just log in or create your user here.

Liked it? Share