Understanding Linux Streams, Pipes, and Redirects

In the Linux command-line environment, understanding how data flows is crucial for efficient task management. This involves mastering the concepts of streams, pipes, and redirects. Streams represent sequences of characters for input and output, while pipes and redirects allow you to control and manipulate this data flow between commands and files.

3.1. Standard I/O Streams

A Linux shell's inputs and outputs are sequences of characters called streams. There are three standard I/O streams:

3.2. Input and Output Redirections

Input and output redirections are fundamental operations performed using angular brackets (<>) to control where command output goes and where command input comes from.

3.2.1. Types of Redirection

cat << EOF > output.txt
line 1
line 2
line 3
EOF
echo done

wc <<< $word

3.2.2. Additional Redirection Examples

3.2.2.1. Separate Standard Output and Standard Error

Send standard output to sout.txt and standard error to serr.txt:

command1 > sout.txt 2> serr.txt

3.2.2.2. Combine Standard Output and Standard Error

Send both standard output and standard error streams to the same file sone.txt:

command1 > sone.txt 2>&1

Alternatively, using Bash 4+:

command1 &> sone.txt

3.2.2.3. Check for String Presence in a Variable

Use grep with a herestring to check if a string is present in a shell variable:

grep "ABC" <<< $var

3.2.2.4. Discarding All Output

Ignore both standard output and standard error by redirecting them to /dev/null:

command1 &> /dev/null

/dev/null is a special null device file. Anything written to it is discarded, and reading from it returns an end-of-file (EOF) immediately.

3.3. Command Piping

Piping allows you to redirect the standard output of one command to the standard input of another command, creating a chain of operations.

command1 | command2 parameter1 | command3 parameter1 parameter2 | command4

Examples:

• Sort a file and remove duplicate records:

  sort file1 | uniq

• Print the 5 most frequently used commands:

  history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head -5

• Print file types and their frequencies:

  ls | rev | cut -f1 -d'.' | rev | sort | uniq -c | sort -n

3.4. The xargs Command

• xargs is short for "extended arguments."
• It is used to build and execute command lines from standard input. Some commands can accept arguments from both standard input and as command-line arguments, while others only accept arguments. For the latter, xargs is essential.
• xargs converts input from standard input into arguments for a specified command. It can divide the input into manageable chunks and run the command repeatedly for each group of arguments.
• The -n option allows you to specify the number of arguments per command execution.
  Example: find . | xargs -n1 basename
• To assign standard input to a placeholder within a command, use the -I option. This is useful when you need to place the input in the middle of a command.
  Example: ls | xargs -I{} echo "File {} was found"
• By default, xargs uses whitespace to delimit arguments. When the -0 option is used, xargs uses null termination to identify arguments, which is safer for filenames containing spaces or special characters.

Examples:

• Print the number of lines, words, and characters in files within a directory (note: ls | wc is not the typical use case for xargs, but demonstrates piping):
• Print file types and their frequencies using find and xargs:
  find . -type f | xargs basename -a | grep "\." | rev | cut -f1 -d'.' | rev | sort | uniq -c | sort -n
• Rename all files in a directory by appending .bkp:
  ls | xargs -I{} mv {} {}.bkp
• Copy all .c files to a specific directory using null termination for safety:
  find . -name "*.c" -print0 | xargs -0 -n1 -I{} cp {} some/folder/
  The -print0 option of the find command prints filenames terminated by a null character. Using the -0 option with xargs tells it to expect null-terminated input, ensuring correct handling of filenames with spaces or special characters.

3.5. The tee Command

• The tee command reads from standard input and writes it to both standard output and one or more files simultaneously.
• This is extremely useful when you want to view the output of a command on the screen in real-time while also saving that output to a file for later analysis or logging.

Examples:

• Display the output of ls on the screen and save it to fileList.txt:
  ls | tee fileList.txt

Further Reading and Resources

• MDN Web Docs: MIME types (Related to data handling)
• Bash Manual: Redirection Builtins
• Bash Manual: Pipelines
• xargs man page
• tee man page