Pipes and Filters
Challenge Gallery
Quick Reference
The pipe operator:
command1 | command2 | command3
stdout of command1 flows into stdin of command2, and so on.
Filter commands:
| Command | What It Does |
|---|---|
sort |
Sort lines alphabetically |
sort -n |
Sort lines numerically |
sort -r |
Sort in reverse order |
sort -u |
Sort and deduplicate in one step |
uniq |
Remove adjacent duplicate lines |
uniq -c |
Count occurrences of each line |
wc -l |
Count lines |
wc -w |
Count words |
head -n N |
Show first N lines |
tail -n N |
Show last N lines |
grep pattern |
Keep only lines matching pattern |
grep -v pattern |
Keep only lines NOT matching pattern |
cut -d',' -f1,3 |
Extract fields 1 and 3 (comma-delimited) |
tr 'a-z' 'A-Z' |
Translate lowercase to uppercase |
tee file.txt |
Copy stream to file AND stdout |
How It Works
Basic Pipes
# Count files in a directory
student@ubuntu:~$ ls | wc -l
12
# Sort a file and remove duplicates
student@ubuntu:~$ sort names.txt | uniq
Alice
Bob
Charlie
# Find your gcc commands in history
student@ubuntu:~$ history | grep gcc
42 gcc hello.c -o hello
67 gcc -Wall main.c -o main
student@ubuntu:~$ ls | wc -l
12
# Sort a file and remove duplicates
student@ubuntu:~$ sort names.txt | uniq
Alice
Bob
Charlie
# Find your gcc commands in history
student@ubuntu:~$ history | grep gcc
42 gcc hello.c -o hello
67 gcc -Wall main.c -o main
Building Pipelines Step by Step
Start with one command, verify the output, add the next stage.
# Goal: find the 3 most common words in a file
# Step 1: see the raw data
student@ubuntu:~$ cat words.txt
apple
banana
apple
cherry
banana
apple
# Step 2: sort groups duplicates together
student@ubuntu:~$ sort words.txt
apple
apple
apple
banana
banana
cherry
# Step 3: count unique lines
student@ubuntu:~$ sort words.txt | uniq -c
3 apple
2 banana
1 cherry
# Step 4: sort by count descending, take top 3
student@ubuntu:~$ sort words.txt | uniq -c | sort -rn | head -3
3 apple
2 banana
1 cherry
# Step 1: see the raw data
student@ubuntu:~$ cat words.txt
apple
banana
apple
cherry
banana
apple
# Step 2: sort groups duplicates together
student@ubuntu:~$ sort words.txt
apple
apple
apple
banana
banana
cherry
# Step 3: count unique lines
student@ubuntu:~$ sort words.txt | uniq -c
3 apple
2 banana
1 cherry
# Step 4: sort by count descending, take top 3
student@ubuntu:~$ sort words.txt | uniq -c | sort -rn | head -3
3 apple
2 banana
1 cherry
Working With Structured Data
# Extract the second column from a CSV and sort numerically
student@ubuntu:~$ cut -d',' -f2 grades.csv | sort -n
82
91
95
# Convert to uppercase
student@ubuntu:~$ echo "hello world" | tr 'a-z' 'A-Z'
HELLO WORLD
# Save intermediate output while continuing the pipeline
student@ubuntu:~$ sort words.txt | tee sorted.txt | uniq -c
3 apple
2 banana
1 cherry
student@ubuntu:~$ cut -d',' -f2 grades.csv | sort -n
82
91
95
# Convert to uppercase
student@ubuntu:~$ echo "hello world" | tr 'a-z' 'A-Z'
HELLO WORLD
# Save intermediate output while continuing the pipeline
student@ubuntu:~$ sort words.txt | tee sorted.txt | uniq -c
3 apple
2 banana
1 cherry
Common Pitfalls
uniqwithoutsort–uniqonly removes adjacent duplicates. Alwayssort | uniq, or usesort -u.- Lexicographic vs. numeric sort –
sortputs10before2. Usesort -nfor numbers. - Debugging multi-stage pipelines all at once – Build left to right, verifying each stage. Never debug a five-stage pipeline blind.
cat file | sort– Works, butsort fileis cleaner.catis only needed when combining multiple files or when there’s no file argument.