Linux Search & Pipes

Everything below runs on one sample file: a fake web server log called access.log. A log is ideal because it has repeated IPs and status codes, so counting and ranking commands produce real, interesting output.

Each line looks like this:

203.0.113.42 - - [10/Oct/2025:09:14:28 +0000] "GET /admin HTTP/1.1" 200 47045 "-" "PostmanRuntime/7.29"

The whitespace-separated fields you'll use most:

• field 1 — the IP address
• field 7 — the URL path
• field 9 — the HTTP status code (200, 404, 500…)
• field 10 — the response size in bytes

Grab the ready-made file straight onto your Linux box and follow along — no setup, no scripts:

wget https://golu.net/access.log
# or, if you prefer curl:
curl -O https://golu.net/access.log

It's 500 lines of realistic traffic — repeated IPs, mixed status codes and response sizes — so every command below produces real, interesting output. ⬇ Download access.log directly

The pipe is the whole trick. It takes the output of one command and feeds it as the input of the next. Instead of one giant command, you chain small ones, each doing one job:

command1 | command2 | command3

Read it left to right: do this, then pass it to that, then to that. This one idea powers every recipe on this page.

head shows the start of a file, tail shows the end. Both default to 10 lines.

head access.log          # first 10 lines (default)
head -n 5 access.log     # first 5
tail access.log          # last 10 lines
tail -n 20 access.log    # last 20
tail -f app.log          # "follow" — stream new lines as they arrive

Your "top 10" is literally head — the default count. tail -f is the one to remember for live logs: it keeps printing new lines until you hit Ctrl-C.

This is the one you'll use most. It keeps only the lines that match a pattern.

grep "404" access.log        # lines containing 404
grep -i "post" access.log    # -i: case-insensitive (matches POST)
grep -n "500" access.log     # -n: show line numbers
grep -v "200" access.log     # -v: invert — lines that DON'T match
grep -c "404" access.log     # -c: count matching lines
grep -r "TODO" ./src/        # -r: search a folder recursively

Different job from grep. find looks for files by name, size, or type — it doesn't look at what's inside them.

find . -name "*.log"          # all .log files under here
find . -type f -size +10M     # files bigger than 10 MB
find /etc -name "ssh*"        # files starting with "ssh"

The single most important habit. Whatever comes first in the pipe runs first — so where you put head relative to grep decides what you actually get.

head -n 10 access.log | grep 404   # 404s WITHIN the first 10 lines only
grep 404 access.log | head -n 10   # search the whole file, then first 10 matches

Same tools, opposite meaning. Rule of thumb: head before grep = "search only the top of the file." grep before head = "find matches everywhere, then trim to 10." The second is almost always what you want.

Reorders lines. Alphabetical by default; the flags are where the power is.

sort access.log          # alphabetical (text)
sort -r access.log       # -r: reverse
sort -n numbers.txt      # -n: NUMERIC sort
sort -k9 -n access.log   # -k9: sort by the 9th field
sort -u names.txt        # -u: sort + drop duplicates

Find the five heaviest responses by sorting on the size field:

awk '{print $10, $7}' access.log | sort -rn | head -n 5

uniq -c collapses duplicate lines and prepends a count. But there's a catch that trips everyone up:

Put it together and you get the classic "top N most frequent" idiom — here, the most active IPs:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -n 10

Reading it stage by stage:

• awk '{print $1}' — pull out just the IP
• sort — group identical IPs together (required before uniq)
• uniq -c — collapse duplicates, prepend a count
• sort -rn — rank by count, biggest first
• head -n 10 — keep the top 10

    131 203.0.113.42
     93 192.168.1.10
     71 10.0.0.5
     69 198.51.100.7
     46 8.8.8.8
     31 172.16.0.1
     30 192.168.1.55
     29 10.0.0.99

Notice this uses two different sorts doing two different jobs: the first groups identical lines; the last ranks the counts. Swap $1 for $9 to rank status codes instead.

The one idea

awk reads a file one line at a time and runs your instructions on each. The whole model is:

awk 'PATTERN { ACTION }' file

"For every line where PATTERN is true, do ACTION." Leave out the pattern and the action runs on every line. Leave out the action and it just prints the line.

Fields — why awk beats cut

awk splits each line on whitespace and hands you numbered pieces:

• $1, $2, … — the fields
• $0 — the entire line
• NF — number of fields; $NF — the last field
• NR — the current line number

awk '{print $1}' access.log        # the IP
awk '{print $1, $9}' access.log    # IP and status (comma = space between)

Patterns — filtering by value

This is where awk replaces grep: it can compare a column to a number.

awk '$9 == 404 { print $0 }' access.log        # status equals 404
awk '$10 > 40000 { print $7, $10 }' access.log # responses over 40 KB
awk '$9 == 404 && $10 > 20000' access.log      # two conditions
awk '/admin/ { print $1 }' access.log          # regex match, like grep

Math — BEGIN and END

BEGIN runs once before any lines; END runs once after all of them. Perfect for totals.

awk '{ total += $10 } END { print total }' access.log          # sum all bytes
awk '{ t += $10; n++ } END { print t/n }' access.log          # average size
awk '$9 == 404 { c++ } END { print c }' access.log            # count 404s

Arrays — replacing sort | uniq -c

An associative array is a tally sheet keyed by anything you choose:

# bytes sent to each IP — grep/cut simply can't do this
awk '{ bytes[$1] += $10 } END { for (ip in bytes) print bytes[ip], ip }' access.log | sort -rn

Ragged output is hard to read. column -t pads every column to line up like a table.

awk '{print $1, $9}' access.log | column -t | head

Add a header row via awk's BEGIN and let column align it with the data:

awk 'BEGIN {print "IP","STATUS"} {print $1,$9}' access.log | column -t | head

IP            STATUS
198.51.100.7  404
203.0.113.42  200
10.0.0.5      404

For fixed widths you control yourself, use printf inside awk — %-16s means "string, 16 wide, left-aligned":

awk '{printf "%-16s %-8s\n", $1, $9}' access.log | head

There is no count command in Linux. The line counter is wc -l (word count, lines).

grep 404 access.log | wc -l    # how many lines matched

Since counting matches is so common, grep does it in one step with -c:

grep -c 404 access.log         # same result, no extra pipe

Workflow: look first, then count. Run without wc -l to confirm you're matching the right stuff, then add | wc -l once you trust it.

Pipes send output to another command. Redirection sends it to a file.

grep 404 access.log > errors.txt    # write to file (OVERWRITES)
grep 500 access.log >> errors.txt   # APPEND to the end
head 10 2> whoops.txt               # send ERRORS to a file (stream 2)

Normal output is "stream 1," errors are "stream 2" — hence 2> for capturing error messages separately. You can redirect the end of any pipe:

awk '{print $1}' access.log | sort | uniq -c | sort -rn | head > top10.txt

grep context & extraction

grep -A 2 500 access.log   # match + 2 lines After
grep -B 2 500 access.log   # 2 lines Before + match
grep -C 2 500 access.log   # 2 lines of Context both sides
grep -oE 'HTTP/1.1" [0-9]+' access.log  # -o: print only the matching part

cut — a lighter awk for simple columns

cut -d' ' -f1 access.log | head   # -d' ': split on space, -f1: field 1

Same as awk '{print $1}' here. Use cut to slice a column; use awk when you need logic or math.

less — read long output like a document

grep GET access.log | less   # space=scroll, /word=search, q=quit

These mistakes teach more than the commands. Every one traces back to the same few causes.

One small challenge a day, all on access.log. Try each yourself first, then reach for a hint only if stuck.

• Mon — Show the last 20 lines. Then the first 3.
• Tue — Count how many requests returned a 404. Do it two ways (with wc -l and with grep -c).
• Wed — Print just the IP and status columns, aligned as a table with a header.
• Thu — Rank the status codes from most to least common (top-10 pattern, field 9).
• Fri — Find every response bigger than 40000 bytes and show its URL and size.
• Sat — Total the bytes sent to each IP, ranked biggest first (awk array).
• Sun — Save the top-10 busiest IPs to a file called top10.txt, then cat it back.