DEV Community: devTalk

Laravel Context: Share Data Across Logs and Queued Jobs

devTalk — Sun, 01 Nov 2026 13:44:32 +0000

You know which tenant a request belongs to in the middleware. Then the request writes a log line, dispatches a job, and the job writes more log lines on a worker with no idea who the tenant was. People usually fix this by adding $tenantId to every job constructor and every Log::info() call. Laravel's Context facade does it for you: set the value once, and it is attached to every log entry and carried into every job you dispatch.

The quick answer

use Illuminate\Support\Facades\Context;

Context::add('tenant_id', 42);   // in middleware, once per request

// anywhere later, including inside a queued job:
$tenantId = Context::get('tenant_id');   // 42

Every log entry written after that carries tenant_id as metadata, and every job dispatched after that carries it to the worker.

Step 1: set it in middleware

<?php

namespace App\Http\Middleware;

use Closure;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Context;
use Symfony\Component\HttpFoundation\Response;

class AddTenantContext
{
    public function handle(Request $request, Closure $next): Response
    {
        if ($tenantId = $request->user()?->tenant_id) {
            Context::add('tenant_id', $tenantId);
        }

        return $next($request);
    }
}

This assumes your users table has a tenant_id column. Use whatever identifies the tenant in your app, such as the subdomain or a route parameter. Register the middleware where the user is already available. In a Laravel 11+ app, bootstrap/app.php is the place:

->withMiddleware(function (Middleware $middleware) {
    $middleware->appendToGroup('web', \App\Http\Middleware\AddTenantContext::class);
})

Adding it to the web group rather than the global stack matters here: the session, and with it $request->user(), is only available after the web group's session middleware has run. If you take the tenant from the subdomain instead, ordering does not matter.

Step 2: read it inside the job

<?php

namespace App\Jobs;

use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Illuminate\Support\Facades\Context;
use Illuminate\Support\Facades\Log;

class ProcessOrder implements ShouldQueue
{
    use Queueable;

    public function __construct(public int $orderId) {}

    public function handle(): void
    {
        $tenantId = Context::get('tenant_id');   // 42, set in the request

        Log::info('Processing order', ['order_id' => $this->orderId]);
    }
}

Dispatch it from a controller as usual with ProcessOrder::dispatch($order->id). The job constructor does not mention the tenant at all.

What the log looks like

I ran this against the Laravel 13 context classes and a real database queue. The request logged a line, dispatched a job, and a separate PHP process (the "worker") started with an empty context and ran the job:

Order created {"order_id":7} {"tenant_id":42,"request_id":"req-abc"}      ← written in the request
context before job: []                                                    ← worker starts empty
Processing order {"order_id":7} {"tenant_id":42,"request_id":"req-abc"}   ← written in the worker
tenant_id read inside the job: 42

The first {...} is the log context you passed to the call. The second is the shared Context data, which Laravel adds as log metadata. In a default file log both appear after the message, as in this example and in the Laravel documentation. Your log formatter may place the second part differently.

How it travels to the worker

There is no magic beyond two hooks that Laravel registers itself, both visible in Illuminate\Log\Context\ContextServiceProvider:

On dispatch, Laravel adds the current context to the job's payload under the key illuminate:log:context. Each value is PHP-serialized.
When a worker starts a job, it reads that key and loads the data back into the worker's context before your handle() runs. The existing context is cleared first, so one job's data does not leak into the next job on a long-running worker.

Here is the payload that was stored in the jobs table in my test:

{"data":{"tenant_id":"i:42;","request_id":"s:7:\"req-abc\";"},"hidden":{"api_token":"s:10:\"secret-xyz\";"}}

A job dispatched with an empty context has no illuminate:log:context key at all.

Hidden context

Some data should travel with the request but stay out of the logs. Use addHidden():

Context::addHidden('api_token', $token);

Context::getHidden('api_token');   // the token
Context::get('api_token');         // null

Hidden values are not added to log entries and are not returned by get() or all(). They are still carried to queued jobs. Note the payload above: secret-xyz is readable in the stored job. In my test with the database queue it was plain serialized text, not encrypted. "Hidden" means hidden from logs, not protected from anyone who can read your queue storage, so do not put real secrets in context.

Things to know

The context is captured when you dispatch. A value added after dispatch() does not reach that job.
Values must be serializable. The context is serialized into the payload. Store IDs, strings and numbers, not closures or large objects. Eloquent models are stored by identifier, and if the record is deleted before the job runs, the value comes back as null, so an ID is simpler and clearer.
Keep it small. It is added to every job payload you dispatch, so a large value is stored and transferred once per job.
It does not replace explicit job data. If a job cannot do its work without a value, such as the order it must process, pass it through the constructor. Use context for ambient data: tenant, trace ID, locale, request ID.
It is not a security boundary. Scope your queries by tenant as you do today. A shared value in a payload is data your app wrote, not proof of who may see what.
Console commands and schedulers have no request. Nothing sets the tenant there. Set it explicitly where the command starts. Context::scope() sets values for the duration of a callback and restores the previous state afterwards.

Common questions

Does Context work with every queue driver?

The hook is registered with Queue::createPayloadUsing() on the base Queue class, and the built-in database, Redis, SQS, Beanstalkd and sync queues all extend that class, so they build their payloads the same way. I only ran the database driver.

How do I add a request ID or trace ID to every log line?

Add it in the same middleware with Context::add('trace_id', (string) Str::uuid()). The Laravel documentation uses exactly this pattern for a URL and trace ID.

What is the difference between `Log::withContext()` and `Context::add()`?

Log::withContext() adds data to the log entries of the current process. Context::add() also reaches the log entries, as metadata, and in addition travels with the jobs you dispatch.

Can I change what is carried to jobs?

Yes. Laravel provides Context::dehydrating() and Context::hydrated() callbacks. Inside them, change only the repository passed to your callback, not the Context facade, as the documentation warns.

Sources

Originally published on DEV Talk.

AWK gsub() Examples: Find and Replace Text in Linux

devTalk — Sun, 01 Nov 2026 13:44:00 +0000

gsub() is AWK's find-and-replace. It replaces every match of a pattern in a line or in a single column, and it tells you how many replacements it made. It looks like sed 's/old/new/g', with one big advantage: it can work on one column and leave the rest alone. This guide covers the syntax, the special characters that trip people up, and how to use it with columns and shell variables, with the exact output of each example.

The quick answer

awk '{gsub(/foo/, "X"); print}' file.txt        # replace every foo with X on every line
awk '{sub(/foo/, "X"); print}' file.txt         # replace only the first foo on each line
awk '{gsub(/foo/, "X", $2); print}' file.txt    # replace only inside column 2

If AWK is new to you, read the beginner's guide first.

Sample file

printf 'foo bar foo\nbaz foo\nno match\n' > g.txt

Syntax

gsub(regex, replacement)            # works on the whole line, $0
gsub(regex, replacement, target)    # works on $2, a variable, or any other target
sub(regex, replacement, target)     # same, but replaces only the first match

Both functions change the target in place and return the number of replacements (sub returns 0 or 1). The g in gsub stands for "global", meaning all matches.

`sub()` versus `gsub()`

awk '{sub(/foo/, "X"); print}' g.txt
# X bar foo
# baz X
# no match

awk '{gsub(/foo/, "X"); print}' g.txt
# X bar X
# baz X
# no match

The first line has two foo. sub() changed one, gsub() changed both.

Count the replacements

awk '{n = gsub(/foo/, "X"); print n}' g.txt
# 2
# 1
# 0

awk '{n += gsub(/foo/, "X")} END {print n}' g.txt
# 3

Because gsub() returns the count, it doubles as a way to count occurrences. There is more on that in the counting occurrences guide.

Change only some lines

Put a pattern before the braces, or use the return value as the condition. A rule with no braces prints the line when the condition is true, so this prints only the lines that were changed:

awk '/foo/ {gsub(/foo/, "X"); print}' g.txt     # change and print only matching lines
awk 'gsub(/foo/, "X")' g.txt                    # same result, shorter
# X bar X
# baz X

To change matching lines but still print every line, put a lone 1 after the replacement, which is an always-true condition that prints the line:

awk '{gsub(/foo/, "X")} 1' g.txt

Replace inside one column

Pass the field as the third argument:

echo 'a,foo,c' | awk -F, '{gsub(/foo/, "X", $2); print}'
# a X c

Look at the output. The commas turned into spaces. When you change a field, AWK rebuilds the whole line from its fields and joins them with the output field separator, OFS, which is a space by default. Set OFS to match the input:

echo 'a,foo,c' | awk -F, -v OFS=, '{gsub(/foo/, "X", $2); print}'
# a,X,c

The same applies to space-separated input with uneven spacing. a foo c comes out as a X c, with single spaces, after a field is modified. This is the most common surprise with gsub() on columns. For more on separators, see AWK field separators.

A practical example, converting the dashes in the first column of a CSV line to slashes:

echo '2026-10-30,x' | awk -F, -v OFS=, '{gsub(/-/, "/", $1); print}'
# 2026/10/30,x

Use the matched text with `&`

In the replacement, & stands for whatever the regex matched:

echo 'cat dog' | awk '{gsub(/[a-z]+/, "<&>"); print}'
# <cat> <dog>

To get a literal ampersand, escape it with two backslashes inside the AWK string:

echo 'cat' | awk '{gsub(/cat/, "\\&"); print}'
# &

A replacement of a&b on the text cat gives acatb, not a&b. This matters when the replacement comes from user data, such as a URL with & in it.

Special characters: the dot, slashes and plus signs

The pattern is a regular expression, so some characters mean something other than themselves.

echo 'a.b.c' | awk '{gsub(".", "-"); print}'
# -----

echo 'a.b.c' | awk '{gsub(/\./, "-"); print}'
# a-b-c

echo 'a.b.c' | awk '{gsub("\\.", "-"); print}'
# a-b-c

An unescaped . matches any character, so the first command replaced all five characters. Escape it with a backslash in a regex literal (/\./) or with two backslashes in a string ("\\."). A character class also works: [.].

A slash in a regex literal must be escaped, or you can use a string instead:

echo '/usr/local/bin' | awk '{gsub(/\//, "_"); print}'     # _usr_local_bin
echo '/usr/local/bin' | awk '{gsub("/", "_"); print}'      # _usr_local_bin

Other characters that need escaping or brackets to be treated literally include + * ? ( ) [ ] ^ $ | and the backslash itself.

Common recipes

# delete characters
echo 'a1b22c333' | awk '{gsub(/[0-9]/, ""); print}'                 # abc

# trim spaces from both ends
echo '   hi there   ' | awk '{gsub(/^ +| +$/, ""); print "[" $0 "]"}'   # [hi there]

# squeeze repeated spaces into one
echo 'a    b     c' | awk '{gsub(/ +/, " "); print}'                 # a b c

# tabs to commas
printf 'a\tb\tc\n' | awk '{gsub(/\t/, ","); print}'                  # a,b,c

# replace every run of digits
echo 'order 123 and 45' | awk '{gsub(/[0-9]+/, "#"); print}'         # order # and #

# strip punctuation
echo 'Hi, there! 99' | awk '{gsub(/[[:punct:]]/, ""); print}'        # Hi there 99

# remove Windows line endings
printf 'ab\r\n' | awk '{sub(/\r$/, ""); print}'                      # ab

Anchors work as in other regex tools. On foo foo, sub(/^foo/, "X") gives X foo and sub(/foo$/, "X") gives foo X.

Use shell variables with gsub

Pass the value with -v and use it as the replacement or as the pattern:

awk -v r=NEW '{gsub(/foo/, r); print}' g.txt
# NEW bar NEW
# baz NEW
# no match

awk -v p=foo '{gsub(p, "X"); print}' g.txt
# X bar X
# baz X
# no match

Three things to watch:

A pattern in a variable is still a regex. With -v p='a+b', the text a+b was not replaced, because the pattern means "one or more a, then b". With -v p='a[+]b' it was replaced. Escape special characters in the variable.
An&in the variable is special again. With -v r='A&B', foo became AfooB. To get a literal &, pass -v r='A\\&B'; the result was A&B.
-vprocesses backslash escapes. -v r='a\tb' puts a real tab in the variable.

Case-insensitive replacement

AWK regexes are case sensitive. The portable approach is to work on a lowercased copy, which changes the case of the whole line:

echo 'Foo FOO foo' | awk '{t = tolower($0); n = gsub(/foo/, "X", t); print n, t}'
# 3 X X X

This counts all three variants. If you need to keep the rest of the line's original case, use gawk and its IGNORECASE variable. mawk silently ignores that variable, so the output would be wrong without any error.

Capture groups: `gensub()` in gawk only

sub() and gsub() cannot reuse parts of the match, as \1 does in sed. gawk adds gensub() for that:

echo 'john smith' | gawk '{print gensub(/(\w+) (\w+)/, "\\2, \\1", "g")}'
# smith, john

On mawk, the same function fails with function gensub never defined. For portable code, split the line into fields instead:

echo 'john smith' | awk '{print $2 ", " $1}'
# smith, john

To pull a piece out of a line rather than replace it, match() with substr() works everywhere:

echo 'id=42;x' | awk '{if (match($0, /[0-9]+/)) print substr($0, RSTART, RLENGTH)}'
# 42

Change the file itself

gsub() only changes the text AWK holds in memory. The file on disk stays the same until you save the output. Write to a new file and move it into place:

awk '{gsub(/foo/, "X")} 1' g.txt > g.tmp && mv g.tmp g.txt

On gawk you can edit in place. I ran this on gawk 5.2.1:

gawk -i inplace '{gsub(/foo/, "X")} 1' g.txt

Never redirect to the file you are reading (awk ... g.txt > g.txt), because the shell empties it first.

gsub or sed?

For a plain global replacement on whole lines, sed 's/foo/X/g' file is shorter and gives the same result. Choose AWK when you need to change only one column, replace only on lines that meet a condition on another column, count replacements, or combine the replacement with other calculations. Capture groups are easier in sed (\1), or in gawk with gensub().

Common problems

Every character was replaced. You used "." or /./. Escape the dot.
Commas or spacing changed after editing a column. Set OFS to the same value as -F.
The file did not change. gsub() only affects the output. Save it with a new file or -i inplace on gawk.
Nothing matched with a variable pattern. The variable holds a regex. Escape + ? . ( ) [ ] ^ $ | or use a bracket class.
Unexpected&in the output. It is replaced by the matched text. Escape it as \\&.
Whole line changed after modifying$0. If you change $0 with gsub(), AWK splits it into fields again. After gsub(/ /, ",") on a b c, NF was 1 and $1 was a,b,c.

Frequently asked questions

What is the difference between sub and gsub in AWK?

sub() replaces the first match in the target. gsub() replaces every match.

How do I replace text in one column only?

Give the column as the third argument: gsub(/old/, "new", $3). Set OFS to your delimiter so the line is rebuilt correctly.

How do I use a variable in gsub?

Pass it with -v, for example awk -v r=NEW '{gsub(/foo/, r)} 1' file. The pattern and the replacement can both be variables.

Does AWK gsub support regex groups like \1?

No. Standard AWK has no backreferences in the replacement. gawk provides gensub(), which does, but it is not available in mawk.

How do I replace a literal dot or slash?

Escape it: /\./ and /\// in a regex literal, or "\\." in a string. A bracket such as [.] also works.

Next in this series

Originally published on DEV Talk.

AWK Remove Duplicates: Lines, Columns and CSV Rows

devTalk — Sun, 01 Nov 2026 02:27:21 +0000

The shortest way to remove duplicate lines from a file in Linux is an AWK one-liner that most people copy without ever reading. It is worth understanding, because the same trick handles duplicates by column, duplicate CSV rows, keeping the last occurrence instead of the first, and listing only the lines that repeat. This guide covers each case with sample input and the exact output.

The quick answer

awk '!seen[$0]++' file.txt

This prints each distinct line the first time it appears and drops every later copy. Unlike sort -u, it keeps the original order of the lines and does not need the file to be sorted.

Sample files

printf 'apple\nbanana\napple\ncherry\nbanana\napple\n' > dup.txt

cat > emails.csv <<'EOF'
id,email,city
1,ann@x.com,Colombo
2,bob@x.com,Kandy
3,ann@x.com,Galle
4,cy@x.com,Kandy
5,bob@x.com,Colombo
EOF

# used further down
printf 'a\na\nb\na\nc\nc\n' > con.txt
printf 'Apple\napple\nAPPLE\nbanana\n' > case.txt
printf 'apple\napple \nbanana\n' > sp.txt
printf 'x\r\nx\nx\r\n' > crlf.txt

Remove duplicate lines and keep the order

awk '!seen[$0]++' dup.txt
# apple
# banana
# cherry

sort -u dup.txt gives the same three lines here, but only because they happen to be in alphabetical order already. sort -u always returns sorted output; the AWK version returns the lines in the order they first appeared.

How `!seen[$0]++` works

Read it from the inside out:

$0 is the whole current line, used as a key in the array seen.
seen[$0]++ adds one to that entry, but the value of the expression is the number before the increment. A key that has never been used counts as 0.
! turns that into true when it was 0 (first time) and false for 1, 2, 3... (a repeat).
An AWK rule with a condition and no braces prints the line when the condition is true.

So the first apple sees 0, passes, and is printed. The second sees 1 and is skipped. The condition is true only for the first copy of each line.

Remove duplicates by one column

Use the column as the key instead of the whole line. This keeps the first row for each email address:

awk -F, '!seen[$2]++' emails.csv
# id,email,city
# 1,ann@x.com,Colombo
# 2,bob@x.com,Kandy
# 4,cy@x.com,Kandy

Rows 3 and 5 are gone because their emails had already appeared. -F, sets the comma as the field separator (see AWK field separators). The header row was kept only because no earlier line had the key email. If a data row could ever match the header text, protect the header explicitly:

awk -F, 'NR==1 || !seen[$2]++' emails.csv

Remove duplicates by several columns

Join the columns with a comma inside the brackets. AWK builds one key from them:

printf 'a,1\na,1\na,2\nb,1\n' | awk -F, '!seen[$1,$2]++'
# a,1
# a,2
# b,1

Two rows count as duplicates only when both columns match. a,1 and a,2 share the first column but are different rows.

Keep the last occurrence instead of the first

Reverse the file, keep the first of each, and reverse it back. tac prints a file from the last line to the first:

tac emails.csv | awk -F, '!seen[$2]++' | tac
# id,email,city
# 3,ann@x.com,Galle
# 4,cy@x.com,Kandy
# 5,bob@x.com,Colombo

Each email now appears once, with its last row. The result is ordered by where each row last appeared, not by where it first appeared. tac is part of GNU coreutils; on macOS use tail -r instead.

Remove only consecutive duplicates

This is what uniq does. Compare each line with the previous one:

awk '$0 != prev {print} {prev = $0}' con.txt

On a file with the lines a a b a c c, this prints a b a c. The second a after b stays, because it is not next to another a. Plain uniq gave the same result. Unlike !seen[$0]++, this uses almost no memory because it only remembers one line.

Show only the duplicates

awk 'seen[$0]++' dup.txt             # every repeated copy (2nd, 3rd, ...)
# apple
# banana
# apple

awk 'seen[$0]++ == 1' dup.txt        # each duplicated value, once
# apple
# banana

Without the !, the condition is true for the second and later copies. Adding == 1 selects only the second copy, so each duplicated value is printed once.

Show lines that appear only once

This needs two passes, so name the file twice. The first pass counts, the second prints the lines whose count is 1:

awk 'NR==FNR {c[$0]++; next} c[$0] == 1' dup.txt dup.txt
# cherry

NR==FNR is true only while AWK reads the first copy of the file. next skips to the following line without running the second rule.

Count how many times each line appears

awk '{c[$0]++} END {for (k in c) print c[k], k}' dup.txt | sort -k1,1nr -k2
# 3 apple
# 2 banana
# 1 cherry

This is the same technique as in counting occurrences with AWK.

Ignore case or stray whitespace

Two lines that look the same to you are different keys to AWK if their case, trailing spaces or line endings differ. Normalise the key, and keep the original line for printing:

awk '!seen[tolower($0)]++' case.txt          # Apple, apple, APPLE, banana  ->  Apple, banana
awk '{sub(/[ \t]+$/, "")} !seen[$0]++' sp.txt   # "apple" and "apple " become one line

In the first command the key is lowercase, but the line printed is the original one, so the first spelling (Apple) is kept. In the second, sub() strips trailing spaces from the line before the duplicate check, and the cleaned line is what gets printed.

A file from Windows has a hidden carriage return at the end of each line. A test file with x\r\n, x\n and x\r\n produced 2 lines with the plain command and 1 after removing the carriage return:

awk '{sub(/\r$/, "")} !seen[$0]++' crlf.txt

Remove duplicates across several files

awk '!seen[$0]++' file1.txt file2.txt > merged.txt

The seen array is shared across all the files in one run, so a line from the second file that already appeared in the first is dropped.

Save the result

Never redirect output into the file you are reading, as in awk '...' file > file. The shell empties the file before AWK starts, and you lose the data. Write to a new file and move it into place:

awk '!seen[$0]++' dup.txt > dup.clean && mv dup.clean dup.txt

With gawk you can edit in place instead. I ran this on gawk 5.2.1:

gawk -i inplace '!seen[$0]++' dup.txt

-i inplace is a gawk feature. mawk does not have it, so use the temporary file method there.

Count how many lines were removed

awk '!seen[$0]++ {kept++} END {print NR - kept, "removed"}' dup.txt
# 3 removed

AWK, sort -u or uniq?

Tool	Keeps original order	Needs sorted input	Memory	Best for
`awk '!seen[$0]++'`	Yes	No	Stores every distinct line	Order matters, or dedupe by column
`sort -u`	No (sorted output)	No	Spills to disk when large	Very large files, sorted output is fine
`uniq`	Yes	Yes (only removes adjacent copies)	One line	Already sorted input

The AWK array holds every distinct line in memory. For a file with millions of mostly unique lines that can be large, and sort -u is the safer choice. For the usual log, list or CSV, AWK is fast and simple.

Common problems

"Duplicates" that are not removed. Trailing spaces, different case or Windows line endings make lines differ. Normalise the key as shown above. cat -A file shows hidden characters.
The header disappeared or was duplicated. Use NR==1 || !seen[$2]++ to always keep it.
Quoted CSV fields. -F, splits inside quotes, so a value like "Smith, John" becomes two fields and the wrong key is used. For quoted CSV, deduplicate on a column that has no commas, or use a CSV-aware tool.
The file came out empty. You redirected into the input file. Write to a new file first.

Frequently asked questions

How do I remove duplicate lines in Linux without sorting?

Use awk '!seen[$0]++' file. It keeps the first copy of each line and preserves the original order.

How do I remove duplicates based on one column in AWK?

Use that column as the array key, for example awk -F, '!seen[$2]++' file.csv to keep the first row for each value in column 2.

How do I keep the last duplicate instead of the first?

Reverse the file, deduplicate, and reverse again: tac file | awk '!seen[$0]++' | tac.

What does `!seen[$0]++` mean?

It is true only the first time a line is seen. seen[$0]++ returns the previous count, and the ! turns a count of 0 into true, so the default action (print) runs for first occurrences only.

Is the AWK method case sensitive?

Yes. Use !seen[tolower($0)]++ to treat Apple and apple as the same line.

Next in this series

Originally published on DEV Talk.

AWK Count Occurrences: Lines, Words, Characters & Values

devTalk — Sat, 31 Oct 2026 05:19:30 +0000

"How many times does this appear?" can mean four different things: how many lines contain it, how many times it appears in total, how many times each value shows up in a column, or how many distinct values there are. AWK handles all four with a few short commands. This guide shows each one with sample input and the exact output.

The quick answers

awk '/error/ {c++} END {print c+0}' file.txt                 # lines containing "error"
awk '{n += gsub(/error/, "&")} END {print n+0}' file.txt     # every occurrence of "error"
awk '{c[$1]++} END {for (k in c) print c[k], k}' file.txt    # how often each value of column 1 appears

The first counts lines, the second counts every occurrence even when one line has several, and the third counts per value. If AWK itself is new to you, start with the beginner's guide.

Sample files

cat > words.txt <<'EOF'
error: disk full
ok
Error: timeout, error again
warning: low memory
error error error
ERROR: failed
EOF

cat > sales.csv <<'EOF'
date,region,product,qty,price
2026-10-01,north,pen,10,1.50
2026-10-01,south,book,3,12.00
2026-10-02,north,book,2,12.00
2026-10-02,east,pen,25,1.50
2026-10-03,south,pen,5,1.50
EOF

cat > access.log <<'EOF'
10.0.0.1 - - [24/Oct/2026:10:00:01 +0000] "GET /index.php HTTP/1.1" 200 512
10.0.0.2 - - [24/Oct/2026:10:00:02 +0000] "GET /about HTTP/1.1" 404 128
10.0.0.1 - - [24/Oct/2026:10:00:03 +0000] "POST /login HTTP/1.1" 302 0
10.0.0.3 - - [24/Oct/2026:10:00:04 +0000] "GET /index.php HTTP/1.1" 200 512
10.0.0.2 - - [24/Oct/2026:10:00:05 +0000] "GET /missing HTTP/1.1" 404 128
10.0.0.1 - - [24/Oct/2026:10:00:06 +0000] "GET /index.php HTTP/1.1" 500 64
EOF

Count lines that match

awk '/error/ {c++} END {print c+0}' words.txt
# 3

Each matching line adds one to c, and the END block prints the total. The +0 matters. If nothing matches, c was never set and plain print c prints an empty line instead of a number. With c+0 you get 0.

Only lowercase error matched here, so the lines with Error and ERROR were not counted. The case-insensitive version is below.

grep -c error words.txt gives the same 3. AWK earns its place when the count needs a condition on a column. In the sample web server log, the status code is the second to last field, so the number of error responses (status 400 or higher) is:

awk '$(NF-1) >= 400 {c++} END {print c+0}' access.log
# 3

NF is the number of fields, so $(NF-1) is the one before the last. The parentheses are required.

Count every occurrence, not just lines

grep -c and the first command above count lines. The line error error error counts once. To count every occurrence, use the return value of gsub(), which is the number of replacements it made:

awk '{t = $0; n += gsub(/error/, "&", t)} END {print n}' words.txt
# 5

Replacing a match with & means "put back the text that matched", so nothing changes, but you still get the count. Working on a copy (t) leaves the original line untouched. The same count from the shell is grep -o error words.txt | wc -l, which also gives 5.

To see the count per line:

awk '{n = gsub(/error/, "&"); print NR": "n}' words.txt
# 1: 1
# 2: 0
# 3: 1
# 4: 0
# 5: 3
# 6: 0

Ignore case

Lowercase a copy of the line first. This works in every awk:

awk '{t = tolower($0); n += gsub(/error/, "&", t)} END {print n}' words.txt
# 7

gawk also has an IGNORECASE variable, but mawk silently ignores it, so tolower() is the safer habit.

Whole words only

A regex match also finds error inside errors or error:. To count only fields that are exactly the word, compare the fields:

awk '{for (i = 1; i <= NF; i++) if ($i == "error") n++} END {print n+0}' words.txt
# 4

This gives 4, not 5, because error: on the first line has a colon attached and is not equal to error.

Count one character

echo 'banana' | awk '{print gsub(/a/, "&")}'
# 3

Two traps. A quoted string is a regular expression, so gsub(".", ...) matches every character; on a.b.c it returned 5, while gsub(/\./, ...) returned 2. And matches do not overlap: gsub(/aa/, ...) on aaaa returned 2, not 3.

Count how often each value appears

This is the most useful pattern. Use the value as an array key and add one every time you see it:

awk -F, 'NR>1 {c[$2]++} END {for (k in c) print k, c[k]}' sales.csv | sort
# east 1
# north 2
# south 2

c[$2]++ reads as "increase the counter for this region". A key that has not been seen yet starts at zero, so there is no setup. NR>1 skips the header, and -F, sets the comma as the separator (see AWK field separators).

The order of for (k in c) is not defined and differs between awks, so pipe the result to sort as above.

Sort by count, highest first

Print the count first, then sort numerically in reverse:

awk -F, 'NR>1 {c[$3]++} END {for (k in c) print c[k], k}' sales.csv | sort -k1,1nr -k2
# 3 pen
# 2 book

The second sort key, -k2, puts names with equal counts in a stable order.

Top visitors in a web server log

awk '{c[$1]++} END {for (k in c) print c[k], k}' access.log | sort -k1,1nr -k2
# 3 10.0.0.1
# 2 10.0.0.2
# 1 10.0.0.3

Add | head -10 for a top 10. Change $1 to $7 to count requested paths instead of IP addresses, or to $(NF-1) to count status codes:

awk '{c[$(NF-1)]++} END {for (k in c) print k, c[k]}' access.log | sort
# 200 2
# 302 1
# 404 2
# 500 1

The column numbers assume the common access log layout shown in the sample, where the quoted request is split by spaces into method, path and protocol. If your log format differs, check with awk '{print $7}' access.log | head first.

Add percentages

awk -F, 'NR>1 {c[$2]++; t++} END {for (k in c) printf "%-6s %d %.0f%%\n", k, c[k], c[k]/t*100}' sales.csv | sort
# east   1 20%
# north  2 40%
# south  2 40%

Count distinct values

To know how many different regions exist, not how often each appears:

awk -F, 'NR>1 && !seen[$2]++ {n++} END {print n}' sales.csv
# 3

awk -F, 'NR>1 {s[$2]} END {print length(s)}' sales.csv
# 3

The first counts a value the first time it is seen. The second only touches each key, then asks for the size of the array. length(array) works in gawk and mawk.

Count lines and words

awk 'END {print NR}' words.txt          # 6   (lines)
awk '{w += NF} END {print w}' words.txt # 16  (words)
awk '!NF {c++} END {print c+0}' file    # blank lines

These match wc -l and wc -w. NR is the line number of the last line read, and NF is the field count of each line.

Count words across a file

awk '{for (i = 1; i <= NF; i++) {w = tolower($i); gsub(/[^a-z]/, "", w); if (w != "") f[w]++}}
     END {for (k in f) print f[k], k}' words.txt | sort -k1,1nr -k2 | head -4
# 7 error
# 1 again
# 1 disk
# 1 failed

For each word, lowercase it, strip everything that is not a letter so error: and Error both become error, and count it. Because of the [^a-z] filter this removes digits and non-English letters too, so adjust the pattern for your text.

Use a shell variable as the search term

awk -v w=error '$0 ~ w {c++} END {print c+0}' words.txt
# 3

With ~ the variable is used as a regular expression, so special characters such as . keep their regex meaning. Do not put the shell variable inside the program text; pass it with -v.

Common problems

A blank line instead of 0. You printed a counter that was never set. Print c+0.
The count is lower than expected. Counting lines (c++ on a match) and counting occurrences (gsub) give different totals when a line has the term more than once.
Header row counted. Add NR>1 to the rule.
Case variants counted separately. Error, error and ERROR are different keys. Lowercase the key with c[tolower($1)]++.
Trailing spaces or Windows line endings. A value with a hidden \r or a trailing space is a different key from the same value without it. Clean it first, for example with sub(/\r$/, "").
Output order changes between machines. It comes from the for-in loop. Sort the output.

Frequently asked questions

How do I count occurrences of a word in a file with AWK?

Use awk '{t=$0; n += gsub(/word/, "&", t)} END {print n+0}' file. It counts every occurrence, including several on one line. To count only lines, use awk '/word/ {c++} END {print c+0}' file.

How do I count how many times each value appears in a column?

Use an array: awk '{c[$1]++} END {for (k in c) print c[k], k}' file | sort -nr. Replace $1 with the column you need.

What is the difference between `grep -c` and counting with AWK?

grep -c counts matching lines only. AWK can also count every occurrence (gsub), count per value (arrays), and add conditions on specific columns.

How do I count unique values in a column?

Use awk '!seen[$1]++ {n++} END {print n}' file, or collect the keys in an array and print its length().

Originally published on DEV Talk.

AWK Cheat Sheet: Commands, Variables, Functions & One-Liners

devTalk — Sat, 31 Oct 2026 05:12:54 +0000

This is a reference to keep open while you work: the AWK syntax you use most, grouped by task, with the result of each example. Every entry was run on gawk 5.2.1 and mawk 1.3.4. Where the two disagree, the entry says so, and a portability table near the end collects the differences in one place. Print this page from your browser if you want a paper copy.

If you are new to AWK, read the beginner's guide first. This sheet assumes you know what a field and a record are.

Sample files used below

printf 'start\nx\ny\nend\nz\n' > rng.txt
printf 'one two\nthree four five\n\nsix\n' > t4.txt
printf 'p1 l1\np1 l2\n\np2 l1\n' > para.txt

Run AWK

What	Command
Program on the command line	`awk 'pattern {action}' file`
Read from a pipe	`command \
Field separator	{% raw %}`awk -F, '...' file` (details in AWK field separators)
Set a variable	`awk -v name=value '...' file`
Program from a file	`awk -f script.awk file`
Show the version	`awk --version` or `awk -W version`

Wrap the program in single quotes. Double quotes let the shell replace $1 before AWK sees it.

Program structure

BEGIN { ... }          # once, before any input
pattern { action }     # for every line that matches the pattern
{ action }             # no pattern: every line
pattern                # no action: print the matching lines
END { ... }            # once, after the last line

Fields and built-in variables

Variable	Meaning
`$0`	The whole current line
`$1`, `$2`, ...	Field 1, field 2, ...
`NF`	Number of fields on this line (`$NF` is the last, `$(NF-1)` the one before)
`NR`	Line number across all input
`FNR`	Line number within the current file
`FILENAME`	Name of the current file
`FS`	Input field separator (default: runs of spaces and tabs)
`OFS`	Output field separator, used between items in `print a, b` (default: a space)
`RS`	Input record separator (default: newline). `RS=""` reads paragraphs separated by blank lines
`ORS`	Output record separator (default: newline)
`ENVIRON["NAME"]`	An environment variable
`ARGC`, `ARGV[i]`	The command-line arguments
`RSTART`, `RLENGTH`	Set by `match()`
`SUBSEP`	Joins the parts of a multi-dimensional array key

awk '{print FILENAME, NR, FNR}' t4.txt rng.txt | sed -n '3,6p'
# t4.txt 3 3
# t4.txt 4 4
# rng.txt 5 1
# rng.txt 6 2

awk 'BEGIN{RS=""} {print NR ": " $1, NF}' para.txt
# 1: p1 4
# 2: p2 2

awk 'BEGIN{ORS="|"} {print $1} END{printf "\n"}' t4.txt
# one|three||six|

echo 'a b c' | awk '{NF=2; print}'        # a b    (assigning NF cuts the line)
echo 'a b'   | awk '{$5="e"; print NF}'    # 5      (assigning past the end adds empty fields)

Patterns

Pattern	Selects	Result on the sample files
`NR==2`	Line number 2	`awk 'NR==2' t4.txt` gives `three four five`
`NR%2`	Odd lines	`rng.txt`: `start`, `y`, `z`
`NR%2==0`	Even lines	`rng.txt`: `x`, `end`
`NR>=2 && NR<=3`	A range of line numbers	`rng.txt`: `x`, `y`
`/start/,/end/`	From a matching line to the next matching line	`rng.txt`: `start`, `x`, `y`, `end`
`/regex/`	Lines matching a regular expression	`awk '/x/{c++} END{print c+0}' rng.txt` gives `1`
`!/regex/`	Lines that do not match	`awk '!/x/' rng.txt` skips `x`
`$1 ~ /^t/`	Field 1 matches	`t4.txt`: `three four five`
`$1 !~ /^t/`	Field 1 does not match	`t4.txt`: `one two`, the blank line, `six`
`NF`	Non-empty lines	`t4.txt`: 3 lines (the blank one is dropped)
`!NF`	Blank lines	`awk '!NF' t4.txt \
{% raw %}`length > 8`	Lines longer than 8 characters	`t4.txt`: `three four five`

Operators

Kind	Operators	Example and result
Arithmetic	`+ - * / % ^`	`2^10` is `1024`, `7%3` is `1`, `7/2` is `3.5`, `int(7/2)` is `3`, `-7%3` is `-1`
Assignment	`= += -= *= /= ++ --`	`x=5; x++; x+=10; x*=2` leaves `x` at `32`
Comparison	`< <= == != >= >`	`("10" < "9")` is `1` (text compare), `(10 < 9)` is `0` (number compare)
Concatenation	(nothing between values)	`a="foo"; b="bar"`: `a b` is `foobar`, `a "-" b` is `foo-bar`
Conditional	`cond ? yes : no`	`n=5; (n%2==0 ? "even" : "odd")` gives `odd`
Membership	`key in array`	`("k" in a)` is `1`, `("z" in a)` is `0`
Logic	`&& \	\

The {% raw %}** power operator works in gawk only; mawk gives a syntax error. Use ^, which works in both.

String functions

Function	Does	Example and result
`length(s)`	Length of a string	`length("Hello, World")` is `12`
`substr(s, start, len)`	Part of a string, counting from 1	`substr(s,8)` is `World`, `substr(s,1,5)` is `Hello`
`index(s, t)`	Position of `t` in `s`, or 0	`index(s,"World")` is `8`
`toupper(s)`, `tolower(s)`	Change case	`HELLO, WORLD` and `hello, world`
`sub(re, repl, s)`	Replace the first match in `s`; returns 1 or 0	`s="foo bar foo"` becomes `X bar foo`
`gsub(re, repl, s)`	Replace every match; returns the count	`s="foo bar foo"` becomes `X bar X` (returns `2`)
`match(s, re)`	Find a match; sets `RSTART` and `RLENGTH`	On `price: 42 USD` with `/[0-9]+/`: `8 2`, and `substr` gives `42`
`split(s, arr, sep)`	Split into an array; returns the count	`split("a,b,c",p,",")` returns `3`, with `p[1]="a"`, `p[3]="c"`
`sprintf(fmt, ...)`	Format like `printf`, return a string	`sprintf("%05.1f\

{% raw %}

awk 'BEGIN{s="a-b-c"; gsub(/-/,"&&",s); print s}'                 # a--b--c   (& means "the matched text")
awk 'BEGIN{n=split("a1b22c",p,/[0-9]+/); print n, p[1], p[2], p[3]}'  # 3 a b c   (separator can be a regex)
awk 'BEGIN{s="  trim me  "; gsub(/^ +| +$/,"",s); print "[" s "]"}'   # [trim me]

Two things to remember. First, a string used as a pattern is a regular expression, so gsub(".", "-", s) replaces every character, not every dot. On a.b.c it gave -----, while gsub(/\./, "-", t) gave a-b-c. Second, substr() with a start below 1 is not portable: substr("hello", -1, 3) gave hel in gawk and hell in mawk. Keep the start at 1 or more.

Numbers and printf

Function	Result
`int(x)`	`int(3.9)` is `3`, `int(-3.9)` is `-3`
`sqrt(x)`, `exp(x)`, `log(x)`	`sqrt(16)` is `4`, `exp(0)` is `1`, `log(1)` is `0`
`sin(x)`, `cos(x)`, `atan2(y,x)`	`sin(0)` is `0`, `cos(0)` is `1`, `atan2(0,-1)` is `3.14159`
`rand()`, `srand(seed)`	`rand()` returns a number from 0 up to (not including) 1. The actual numbers differ between awks, so do not compare them

Plain print shows at most six significant digits, so use printf when exact output matters. A single line that uses most formats:

awk 'BEGIN{printf "%d %5.2f %-6s| %05d %x %o %e %c %%\n", 42.9, 3.14159, "ab", 42, 255, 8, 12345.678, 65}'
# 42  3.14 ab    | 00042 ff 10 1.234568e+04 A %

Format	Meaning
`%d`	Integer (42.9 prints as 42)
`%5.2f`	Decimal number, width 5, 2 digits after the point
`%-6s`	String, width 6, left-aligned
`%05d`	Integer padded with zeros to width 5
`%x`, `%o`	Hexadecimal, octal
`%e`	Scientific notation
`%c`	Character from a number (65 gives A)
`%%`	A literal percent sign

printf does not add a newline. End the format with \n.

Control flow

awk 'BEGIN{for(i=1;i<=3;i++) printf "%d ", i; print ""}'                       # 1 2 3
awk 'BEGIN{i=1; while(i<=3){printf "%d ", i; i++} print ""}'                  # 1 2 3
awk 'BEGIN{i=5; do {printf "%d ", i; i++} while(i<3); print ""}'              # 5   (do-while runs once first)
awk 'BEGIN{for(i=1;i<=10;i++){if(i%2) continue; if(i>6) break; printf "%d ", i} print ""}'   # 2 4 6
awk 'BEGIN{x=7; if (x<5) print "low"; else if (x<10) print "mid"; else print "high"}'          # mid

Statement	Effect
`next`	Stop with this line and go to the next one. `NR==2{next} {print}` skips line 2
`nextfile`	Skip the rest of the current file. `FNR==2{nextfile} {print FILENAME": "$0}` printed one line from each of two files
`exit n`	Stop and set the exit status. `NR==2{exit 3}` exited with 3. An `END` block still runs after `exit`: `BEGIN{exit 5} END{print "end still runs"}` printed that text and exited with 5

Arrays

Task	Code
Set and read	`a["key"]=1; print a["key"]`
Test for a key	`if ("key" in a) ...` (does not create the key)
Count the entries	`length(a)` (works in gawk and mawk)
Remove one key / all keys	`delete a["key"]` / `delete a`
Loop over keys	`for (k in a) print k, a[k]` (order is not guaranteed, pipe to `sort`)
Split into an array	`n=split("c a b", w)` gives `w[1]="c"`, `w[2]="a"`, `w[3]="b"`
Two-part key	`a[1,2]="v"`; split the key back with `split(k, p, SUBSEP)`
Count by value	`{c[$1]++} END {for (k in c) print k, c[k]}`
Remove duplicate lines	`!seen[$0]++`

Input and output

Task	Code
Write to a file	`print $1 > "out.txt"`
Pipe to a command	`print $1 \
Write to standard error	{% raw %}`print "message" > "/dev/stderr"`
Run a shell command	`system("echo from shell")`
Read a command's output	`"echo hello" \
Read a file line by line	{% raw %}`while ((getline line < "rng.txt") > 0) n++` counted 5 lines

Close a file with close("out.txt") before you read it back in the same program. I wrote $1 of every line of t4.txt to a file, closed it, and read it back with getline, and got one, three, an empty line and six.

User-defined functions

function max(a, b) { return a > b ? a : b }
function fact(n)   { return n <= 1 ? 1 : n * fact(n - 1) }
function f(x,  tmp) { tmp = x * 2; return tmp }   # extra parameters are local variables

BEGIN { print max(3, 9), max("b", "a") }          # 9 b
BEGIN { print fact(5) }                           # 120
BEGIN { tmp = "keep"; print f(4), tmp }           # 8 keep

AWK has no local keyword. Extra parameters you never pass behave as local variables, which is why tmp kept its value in the last line. By habit, put two spaces before them to show they are locals.

One-liners

Goal	Command
Number the lines	`awk '{print NR": "$0}' rng.txt` (first line: `1: start`)
Print line 3	`awk 'NR==3' rng.txt` (`y`)
Print lines 2 to 3	`awk 'NR>=2 && NR<=3' rng.txt`
Print everything after line n	`awk -v n=2 'NR>n' rng.txt` (`y`, `end`, `z`)
Print the last line	`awk 'END{print}' rng.txt` (`z`)
Print every other line	`awk 'NR%2' rng.txt` (`start`, `y`, `z`)
Delete blank lines	`awk 'NF' t4.txt`
Remove duplicate lines	`awk '!seen[$0]++' file`
Reverse the lines	`awk '{a[NR]=$0} END{for(i=NR;i>0;i--) print a[i]}' rng.txt` (starts `z`, `end`, `y`)
Join lines with commas	`awk '{printf "%s%s", sep, $0; sep=","} END{print ""}' rng.txt` (`start,x,y,end,z`)
Trim leading and trailing space	`awk '{gsub(/^[ \t]+\
Find the longest line	{% raw %}`awk '{if (length > max) {max=length; line=$0}} END{print max, line}' t4.txt` (`15 three four five`)
Count lines and words	`awk '{w+=NF} END{print NR, w}' t4.txt` (`4 6`)
Swap the first two fields	`echo 'a b c' \
Capitalise the first word	{% raw %}`echo 'hello world' \
Print all but the first field	{% raw %}`echo 'a b c' \
Count matching lines	{% raw %}`awk '/x/{c++} END{print c+0}' rng.txt` (`1`)
Sum a column	`awk '{s+=$1} END{print s}' file` (see summing a column)
Print a column	`awk '{print $2}' file` (see printing a column)

What works in gawk but not in mawk

Most of this sheet runs on every awk. These features do not. The worst ones fail with no message at all, so your script runs and gives wrong output.

Feature	gawk	mawk 1.3.4
`gensub()`	Works	Error: function never defined
`asort()`, `asorti()`	Work	Error: function never defined
`**` power operator	Works	Syntax error
`IGNORECASE=1`	Case-insensitive matching	Silently ignored
`PROCINFO["sorted_in"]`	Sorts `for (k in a)`	Silently ignored (loop order differs)
`BEGINFILE`, `ENDFILE`	Run per file	Silently do nothing
`FIELDWIDTHS`, `FPAT`	Work	Silently ignored
UTF-8 characters	With a UTF-8 locale, `length("héllo")` is `5`	`6` (counts bytes), even in a UTF-8 locale
Division by zero	Fatal error	Prints `-nan` and carries on
`0x1A + 0`	`0`	`26`
`-i inplace`, `--csv` (5.3+)	gawk options (`-i inplace` was run on gawk 5.2.1; `--csv` needs 5.3+ and was not run)	Not tested

These do work in both: length(array), nextfile, delete array, systime(), strftime(), /dev/stderr, -v and -f.

Common mistakes

Double quotes around the program. awk "{print $1}" prints whole lines, because the shell empties $1.
Header row in a numeric test. $4 > 5 also matches a header like qty. Add NR>1 &&.
Array loop order. for (k in a) has no guaranteed order. Pipe to sort.
SettingFSinside a rule. It only affects the next line. Set it in BEGIN or use -F.
$(NF-1)on a blank line. It stops AWK with an error. Guard with NF>1.
A string where you meant a regex. gsub(".", ...) matches any character.

Originally published on DEV Talk.

AWK Sum Column: Total, Average, Min, Max and Group By

devTalk — Fri, 30 Oct 2026 15:11:33 +0000

Adding up a column of numbers is one of the best reasons to learn AWK. There is no spreadsheet to open and no script to write: one line reads the file, adds each value to a running total, and prints the result at the end. This guide starts with that one line and then covers the things people actually need next: average, minimum and maximum, totals per group, and what to do when the data is messy. Every example comes with the input and the exact output.

The quick answer

awk '{sum += $1} END {print sum}' numbers.txt                # sum of column 1
awk -F, 'NR>1 {sum += $4} END {print sum}' sales.csv         # CSV: skip header, sum column 4

New to AWK? The beginner's guide explains the pattern {action} idea used here, and how to print a column explains $1, $4 and NR.

Sample files

printf '10\n20\n30.5\n' > nums.txt
printf '100\n9\n25\n' > n3.txt
printf '5\nN/A\n7\n' > bad.txt
printf '1 2 3\n4 5 6\n' > grid.txt
: > empty.txt        # an empty file

cat > sales.csv <<'EOF'
date,region,product,qty,price
2026-10-01,north,pen,10,1.50
2026-10-01,south,book,3,12.00
2026-10-02,north,book,2,12.00
2026-10-02,east,pen,25,1.50
2026-10-03,south,pen,5,1.50
EOF

Sum a column

awk '{sum += $1} END {print sum}' nums.txt
# 60.5

Here is what happens. For every line, sum += $1 adds the first field to sum. You never declared sum: AWK creates a variable the first time you use it and starts it at zero. The END block runs once, after the last line, and prints the total.

For a CSV with a header, add -F, for the separator and NR>1 to skip the header line:

awk -F, 'NR>1 {s += $4} END {print s}' sales.csv
# 45

Leaving out NR>1 does not change this total, because the header text qty counts as 0. But it quietly breaks the next calculation.

Count rows and calculate the average

Keep a counter next to the sum, then divide at the end:

awk -F, 'NR>1 {s += $4; n++} END {print s, n, s/n}' sales.csv
# 45 5 9

Now see what happens without the header skip:

awk -F, '{s += $4; n++} END {print s, n, s/n}' sales.csv
# 45 6 7.5

The sum is the same, but the header was counted as a sixth row, so the average is wrong and nothing warns you.

Guard against an empty file

If there is no input, n is never set, and the division goes wrong. The two awks I tested disagree on how:

awk '{s += $1; n++} END {print s/n}' empty.txt
# gawk: fatal: division by zero attempted
# mawk: -nan

gawk stops with an error. mawk prints -nan and carries on, which is easy to miss in a script. Check before you divide:

awk '{s += $1; n++} END {if (n > 0) print s/n; else print "no data"}' empty.txt
# no data

Minimum and maximum

Start both from the first data row, then compare every row against them:

awk -F, 'NR==2 {min=max=$4}
         NR>2  {if ($4<min) min=$4; if ($4>max) max=$4}
         END   {print "min", min, "max", max}' sales.csv
# min 2 max 25

The comparison is numeric, not alphabetical. On the file n3.txt (100, 9, 25), the minimum comes out as 9 and the maximum as 100, which is correct. If the comparison were on text, "9" would beat "100".

Do not start the minimum at 0 in a BEGIN block. Then nothing in a list of positive numbers is ever smaller, and you get 0:

awk 'BEGIN{min=0} {if ($1<min) min=$1} END {print min}' n3.txt
# 0   (wrong, the real minimum is 9)

If the data may not have a header, or you do not know the first row's line number, use a version that treats "not set yet" as the starting point:

awk '{if (min=="" || $1<min) min=$1; if (max=="" || $1>max) max=$1} END {print min, max}' n3.txt
# 9 100

Sum only some rows

Put a condition in front of the braces. Total quantity sold in the north region:

awk -F, '$2=="north" {s += $4} END {print s}' sales.csv
# 12

To count matching rows, use a counter. Add +0 when you print it, so that zero matches prints 0 instead of an empty line:

awk -F, 'NR>1 && $2=="north" {n++} END {print n+0}' sales.csv
# 2

Totals per group

An array indexed by the group name gives you one total per group. This is the AWK version of a spreadsheet pivot table:

awk -F, 'NR>1 {s[$2] += $4} END {for (k in s) print k, s[k]}' sales.csv | sort
# east 25
# north 12
# south 8

The | sort is there because AWK does not promise any order when looping over an array; gawk and mawk printed the same groups in a different order when I left it out.

The same idea works for money. Revenue is quantity times price, so add that product per region, plus a grand total:

awk -F, 'NR>1 {r[$2] += $4 * $5; t += $4 * $5}
         END {for (k in r) print k, r[k]; print "total", t}' sales.csv | sort
# east 37.5
# north 39
# south 43.5
# total 120

Sum across each row, or down each column

The file grid.txt has two lines of three numbers. A loop over the fields gives the total of each row:

awk '{t=0; for (i=1; i<=NF; i++) t += $i; print t}' grid.txt
# 6
# 15

To total every column at once, keep an array indexed by column number:

awk '{for (i=1; i<=NF; i++) c[i] += $i}
     END {for (i=1; i<=NF; i++) printf "%s%s", c[i], (i<NF ? OFS : ORS)}' grid.txt
# 5 7 9

Inside END, NF still holds the field count of the last line, which is fine when every line has the same number of columns.

Running total

Print inside the main rule, not in END, to show the total so far on every line:

awk -F, 'NR>1 {s += $4; print $4, s}' sales.csv
# 10 10
# 3 13
# 2 15
# 25 40
# 5 45

Percent of total (reading the file twice)

To show each row as a share of the total, AWK needs the total first. Pass the same file twice. NR==FNR is true only while the first copy is being read, so you can add things up on the first pass and print on the second:

awk -F, 'NR==FNR { if (FNR>1) t += $4; next }
         FNR>1  { printf "%-5s %5.1f%%\n", $3, $4/t*100 }' sales.csv sales.csv
# pen    22.2%
# book    6.7%
# book    4.4%
# pen    55.6%
# pen    11.1%

Print the result properly

AWK's plain print shows a number with at most six significant digits, so a large decimal total turns into scientific notation. I tested this on both awks:

awk 'BEGIN {print 1234567.89; printf "%.2f\n", 1234567.89}'
# 1.23457e+06
# 1234567.89

For money and anything you plan to copy into a report, use printf "%.2f\n". Whole numbers are fine either way (100000 * 3 prints 300000), and 0.1 + 0.2 prints 0.3 because of the same six-digit rounding.

When the data is messy

Text where a number should be

AWK converts text to the number it starts with, and to 0 if it starts with no number. A row of N/A is added as 0, with no error:

awk '{s += $1; n++} END {print s, n, s/n}' bad.txt
# 12 3 4

The sum is right, but the average divides by three rows when only two have numbers. Filter with a pattern, and the count and the sum both agree:

awk '$1 ~ /^[0-9.]+$/ {s += $1; n++} END {print s, n, s/n}' bad.txt
# 12 2 6

Currency symbols and thousand separators

A value like $1,200.50 starts with a dollar sign, so AWK reads it as 0. Strip the symbol and the comma first with gsub:

printf 'price\n$1,200.50\n$99.25\n' > money.txt
awk '{s += $1} END {print s}' money.txt
# 0

awk 'NR>1 {gsub(/[$,]/, "", $1); s += $1} END {printf "%.2f\n", s}' money.txt
# 1299.75

(I skipped the price header with NR>1; it would add 0 either way.) A comma used as a thousands separator also clashes with -F,. A quoted "1,200" in a CSV file is split into two fields, which is a job for the CSV article in this series.

Numbers with a unit

AWK reads the leading number and ignores the rest, so 20G plus 120G adds to 140:

printf 'Used\n20G\n0\n120G\n' | awk 'NR>1 {s += $1+0} END {print s}'
# 140

That works only when every value has the same unit. A column that mixes M and G will add the digits as if they were the same thing.

Strings that look like hex

One difference between the two awks: a value such as 0x1A is 0 in gawk and 26 in mawk when used as a number. If your data contains hex-looking codes, do not rely on arithmetic on them.

Quick reference

Goal	Command
Sum	`awk '{s+=$1} END {print s}'`
Sum, skipping a header	`awk 'NR>1 {s+=$1} END {print s}'`
Average	`awk '{s+=$1; n++} END {if (n) print s/n}'`
Count	`awk 'END {print NR}'`
Max	`awk '{if (max=="" \
Min	{% raw %}`awk '{if (min=="" \
Sum per group	{% raw %}`awk '{s[$1]+=$2} END {for (k in s) print k, s[k]}' \
Two decimals	{% raw %}`printf "%.2f\n", s`

Frequently asked questions

How do I sum a column in AWK?

Add each value to a variable in the main rule and print it in END: awk '{s += $3} END {print s}' file.

How do I sum a column in a CSV file?

Add the separator and skip the header: awk -F, 'NR>1 {s += $4} END {print s}' file.csv.

Why does my total print as 1.23457e+06?

Plain print limits numbers to six significant digits. Use printf "%.2f\n", total.

Can AWK sum a column that has a header?

Yes. Skip the first line with NR>1, so the header is neither added nor counted.

Originally published on DEV Talk.

AWK Field Separator: -F, FS and Multiple Delimiters

devTalk — Fri, 30 Oct 2026 10:55:36 +0000

The most common reason an AWK command prints the wrong thing is that AWK split the line somewhere you did not expect. The rule that decides where to split is the field separator. Get it right and $1, $2 and $NF point at the columns you meant. This guide covers the default behaviour, single and multiple delimiters, the four ways to set the separator, and a few traps, with sample input and the exact output of each example.

The quick answer

awk -F, '{print $2}' file.csv                 # comma
awk -F: '{print $1}' /etc/passwd              # colon
awk -F'\t' '{print $2}' file.tsv              # tab
awk -F'[,;]' '{print $2}' file.txt            # comma OR semicolon
awk -F', ' '{print $2}' file.txt              # the two-character separator ", "

If you are new to AWK, start with the beginner's guide. To see how column numbers work once the line is split, read how to print a column with AWK.

Sample files

Create these once. Every example below uses them.

printf 'root:x:0:0:root:/root:/bin/bash\nada:x:1000:1000:Ada L:/home/ada:/bin/zsh\n' > passwd.txt
printf 'id\tname\tcity\n1\tAda Lovelace\tLondon\n2\tAlan Turing\tWilmslow\n' > t.tsv
printf 'a, b;c ,d\n' > mix.txt
printf 'a,,c,\n' > empty.csv
printf '  one   two\tthree  \n' > ws.txt
printf '2026-10-01 ERROR disk full\n2026-10-02 INFO ok\n' > app.log

cat > sales.csv <<'EOF'
date,region,product,qty,price
2026-10-01,north,pen,10,1.50
2026-10-01,south,book,3,12.00
2026-10-02,north,book,2,12.00
2026-10-02,east,pen,25,1.50
2026-10-03,south,pen,5,1.50
EOF

cat > access.log <<'EOF'
10.0.0.1 - - [24/Oct/2026:10:00:01 +0000] "GET /index.php HTTP/1.1" 200 512
10.0.0.2 - - [24/Oct/2026:10:00:02 +0000] "GET /about HTTP/1.1" 404 128
10.0.0.1 - - [24/Oct/2026:10:00:03 +0000] "POST /login HTTP/1.1" 302 0
EOF

What AWK does when you set nothing

By default, fields are separated by runs of spaces and tabs, and leading and trailing whitespace is ignored. The file ws.txt has two leading spaces, three spaces between words, a tab, and trailing spaces:

awk '{print NF, "[" $1 "]"}' ws.txt
# 3 [one]

Three fields, and the first one is clean. That is why plain awk '{print $2}' works well on column-aligned output like df or ps, and why it fails on anything else.

Set a single-character separator

Use -F followed by the character. A single character other than a space is taken literally, so you do not need to escape it:

awk -F: '{print $1 " -> " $NF}' passwd.txt
# root -> /bin/bash
# ada -> /bin/zsh

printf 'id|name|city\n1|Ada|London\n' | awk -F'|' 'NR>1 {print $2}'
# Ada

echo 'a.b.c' | awk -F. '{print $2}'
# b

The pipe and the dot are special characters in regular expressions, yet both worked as plain characters here. I tested that on gawk and mawk. Quote the pipe anyway, because the shell treats an unquoted | as a pipe.

Tabs

A tab-separated file needs -F'\t'. Without it, a name like "Ada Lovelace" is split at the space:

awk -F'\t' 'NR>1 {print $2}' t.tsv
# Ada Lovelace
# Alan Turing

awk 'NR>1 {print $2}' t.tsv        # default splitting
# Ada
# Alan

In bash you can also pass a real tab character with -F$'\t'; it gave the same result.

Multi-character and multiple delimiters

When the separator is more than one character, AWK treats it as a regular expression. That is how you get several delimiters at once.

Goal	Option	Input → result
Exactly two characters	`-F', '`	`x, y, z` → `$2` is `y`
Comma or semicolon	`-F'[,;]'`	`a,b;c` → `a`, `b`, `c`
Any run of spaces, commas, semicolons	`-F'[ ,;]+'`	`a, b;c ,d` → 4 fields
A word as the separator	`-F'one\	two'`
Delimiter with optional spaces around it	`-F' , '`	`a, b ,c` → `a`, `b`, `c`

For example, with mix.txt (a, b;c ,d):

awk -F'[ ,;]+' '{print NF; print $1 "|" $2 "|" $3 "|" $4}' mix.txt
# 4
# a|b|c|d

Trimming is the other common need. A plain -F, on a, b ,c leaves the spaces inside the field ($2 is " b "). Put the spaces in the pattern and they disappear:

echo 'a, b ,c' | awk -F' *, *' '{print "[" $1 "][" $2 "][" $3 "]"}'
# [a][b][c]

Empty fields and leading spaces

With a single-character separator, every delimiter counts, even next to another one. The line a,,c, has four fields, two of them empty:

awk -F, '{print NF; print "[" $2 "]"}' empty.csv
# 4
# []

Default splitting behaves differently: it collapses runs of whitespace. You can see the difference by asking for a single space as the separator:

echo 'a  b' | awk '{print NF}'          # 2   (default: runs collapse)
echo 'a  b' | awk -F'[ ]' '{print NF}'  # 3   (each space separates, so there is an empty field)

This leads to a trap. If you use a regex like [ ]+ on a line that starts with spaces, you get an empty first field, which the default splitting would have skipped:

echo '  a b' | awk '{print NF, "[" $1 "]"}'          # 2 [a]
echo '  a b' | awk -F'[ ]+' '{print NF, "[" $1 "]"}'  # 3 []

When you only want whitespace splitting, leave the separator alone.

Four ways to set the separator

Way	Example	Good for
`-F` option	`awk -F, '...' file`	One-liners
`-v FS=`	`awk -v FS=, '...' file`	Same effect, handy in scripts
`BEGIN` block	`awk 'BEGIN{FS=","} ...' file`	Program files
After the program	`awk '...' FS=, file`	Different separator per file

All four split sales.csv on commas when I tested them. The last form is the odd one out, because the assignment takes effect when AWK reaches it in the argument list. That lets you read files with different formats in one run:

awk '{print $1}' FS=, sales.csv FS=: passwd.txt

The first file's lines print their first comma-separated field (date, 2026-10-01, ...) and then, once AWK reaches the second assignment, passwd.txt lines print their first colon-separated field (root, ada).

Gotcha: changing FS inside a rule is one line too late

AWK splits a line into fields as soon as it reads it. If you set FS in the main rule, the current line has already been split, so the change applies from the next line:

awk '{FS=":"; print $1}' passwd.txt
# root:x:0:0:root:/root:/bin/bash     (first line: still split on whitespace)
# ada

Set it in BEGIN instead, or force a re-split with $0=$0:

awk '{FS=":"; $0=$0; print $1}' passwd.txt
# root
# ada

FS reads, OFS writes

FS controls how input is split. OFS controls what goes between fields when you print with commas. Combine them to convert formats. Assigning to a field (even to itself, with $1=$1) makes AWK rebuild the line using OFS:

awk 'BEGIN{FS=","; OFS=";"} {$1=$1; print}' sales.csv | head -2
# date;region;product;qty;price
# 2026-10-01;north;pen;10;1.50

awk -F, -v OFS='\t' '{$1=$1} 1' sales.csv | head -2 | cat -A
# date^Iregion^Iproduct^Iqty^Iprice$        (^I is a tab, $ marks the line end)
# 2026-10-01^Inorth^Ipen^I10^I1.50$

The trailing 1 is a pattern that is always true, so AWK prints the line.

Real examples

A timestamp inside square brackets

In a web server log, the date sits between [ and ]. Use both brackets as separators. In a bracket expression, put ] first so it counts as a character:

awk -F'[][]' '{print $2}' access.log
# 24/Oct/2026:10:00:01 +0000
# 24/Oct/2026:10:00:02 +0000
# 24/Oct/2026:10:00:03 +0000

The quoted request line

Splitting on the double quote isolates the request. Then split() breaks it into method and path:

awk -F'"' '{split($2, r, " "); print r[1], r[2]}' access.log
# GET /index.php
# GET /about
# POST /login

A word as the separator

awk -F' ERROR ' 'NF>1 {print $2}' app.log
# disk full

Only the first line contains ERROR, so it is the only one with more than one field.

Split into characters, or into fixed widths

An empty separator splits every character into its own field. This worked on both gawk and mawk:

echo abc | awk -v FS= '{print NF, $2}'
# 3 b

For fixed-width columns, gawk has FIELDWIDTHS. This is a gawk-only feature, and the failure on other awks is quiet. Create a one-line file whose first 10 characters are a date and next 5 are a level:

printf '2026-10-01ERROR\n' > fixed.txt
awk 'BEGIN{FIELDWIDTHS="10 5"} {print "[" $1 "][" $2 "]"}' fixed.txt
# gawk: [2026-10-01][ERROR]
# mawk: [2026-10-01ERROR][]        (setting ignored, no error message)

If a script must run on mawk too, use substr($0, 1, 10) and substr($0, 11, 5) instead.

When a separator is not enough: quoted CSV

A separator cannot understand quoting. On the line Ada,"hi, there", -F, gives you three fields, because the comma inside the quotes also splits. For real CSV with quoted fields you need FPAT in gawk, or the built-in CSV mode that gawk added in version 5.3.0. I could only test gawk 5.2.1 here, so I have not run the CSV mode. The CSV article in this series covers it.

Quick reference

Input format	Option
Spaces and tabs (default)	nothing
Comma	`-F,`
Colon	`-F:`
Pipe	`-F'\
Tab	{% raw %}`-F'\t'`
Either of several characters	`-F'[,;]'`
Delimiter with spaces around it	`-F' , '`
Square brackets	`-F'[][]'`
Each character	`-v FS=`

Frequently asked questions

How do I use multiple delimiters in AWK?

Give -F a regular expression: -F'[,;]' for a comma or a semicolon, or -F'one|two' for two words. Add a + after the brackets to treat a run of them as one separator.

What is the difference between `-F` and `FS`?

Nothing for your program. -F, on the command line sets the same FS variable that BEGIN{FS=","} sets.

Why does `$1` print the whole line?

Because the separator never matched. If you set -F, but the file uses semicolons, there is only one field, and $1 is the whole line. Check with awk -F, '{print NF}' file.

How do I set the separator to a tab?

Use -F'\t', or in bash -F$'\t'

Originally published on DEV Talk.

AWK Print Column: $1, $NF, Last Column and More

devTalk — Fri, 30 Oct 2026 09:37:56 +0000

Pulling one column out of a text file is the most common thing people use AWK for. It takes one short command, but a few small details decide whether you get the right column, a broken one, or a crash on a blank line. This guide covers the command, the options around it, and the mistakes worth knowing about, with sample input and the exact output of each example.

The quick answer

awk '{print $2}' file.txt          # second column, split on spaces and tabs
awk -F, '{print $2}' file.csv      # second column of a comma-separated file
awk '{print $NF}' file.txt         # last column

$1 is the first column, $2 the second, and $NF the last one. If you are new to AWK, read the AWK beginner's guide first for how lines and fields work.

Sample files

Create these once and every example below will work as shown. sales.csv is the same file used in the rest of this series.

echo 'a b c d' > w.txt

cat > sales.csv <<'EOF'
date,region,product,qty,price
2026-10-01,north,pen,10,1.50
2026-10-01,south,book,3,12.00
2026-10-02,north,book,2,12.00
2026-10-02,east,pen,25,1.50
2026-10-03,south,pen,5,1.50
EOF

cat > df.txt <<'EOF'
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   20G   28G  42% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
/dev/sdb1       200G  120G   70G  63% /data
EOF

Print one or more columns

awk '{print $2}' w.txt          # b
awk '{print $1, $3}' w.txt      # a c
awk '{print $1 $3}' w.txt       # ac
awk '{print $1 "-" $3}' w.txt   # a-c

A comma inside print inserts the output separator, which is a space by default. Without the comma the values are glued together. If you want to join them with something else, either put your own text between them (the last line) or change the output separator once in a BEGIN block:

awk 'BEGIN {OFS=","} {print $1, $3}' w.txt    # a,c

Choose the delimiter

AWK splits on runs of spaces and tabs unless you tell it otherwise. Use -F for anything else:

awk -F, '{print $2}' sales.csv                 # CSV: region, north, south ...
awk -F: '{print $1, $NF}' <<< 'root:x:0:0:root:/root:/bin/bash'
# root /bin/bash
awk -F'\t' '{print $2}' file.tsv               # tab-separated

The <<< in the second example is a bash and zsh here-string that feeds a short text to the command; in plain sh, use echo '...' | awk ... instead. The details of separators, including several at once, are in the field separator guide.

The last column and the one before it

NF is the number of fields on the current line, so $NF is the last field and $(NF-1) is the one before it. This works even when lines have different numbers of columns:

awk '{print $NF}' w.txt         # d
awk '{print $(NF-1)}' w.txt     # c

The parentheses matter. Without them, AWK reads $NF-1 as "the last field, minus one":

echo '5 10' | awk '{print $NF-1}'     # 9, not the 5

Print everything except the first column

There is no "all but the first" shortcut, but there are three ways to do it. Setting $1 to an empty string leaves a stray leading space, because AWK rebuilds the line with its output separator:

awk '{ $1=""; print }' w.txt                  # " b c d"  (note the leading space)
awk '{ $1=""; print substr($0,2) }' w.txt     # b c d

The loop version has no such problem, and it also lets you print any range of columns:

awk '{ for (i=2; i<=NF; i++) printf "%s%s", $i, (i<NF ? OFS : ORS) }' w.txt
# b c d

printf 'a b c d e f\n' | awk -v from=2 -v to=4 \
  '{ for (i=from; i<=to; i++) printf "%s%s", $i, (i<to ? OFS : ORS) }'
# b c d

Skip the header and filter rows

Put a condition in front of the braces. NR>1 skips the first line; a field test keeps only some rows:

awk 'NR>1 {print $1, $5}' df.txt
# /dev/sda1 42%
# tmpfs 0%
# /dev/sdb1 63%

awk -F, '$2=="north" {print $3}' sales.csv
# pen
# book

Pick a column by its header name

Counting columns gets fragile when someone adds one to the file. If the first line is a header, you can look columns up by name instead. Read the header into an array on line 1, then use the array as the column number:

awk -F, 'NR==1 {for (i=1; i<=NF; i++) col[$i]=i; next}
         {print $col["product"], $col["price"]}' sales.csv

pen 1.50
book 12.00
book 12.00
pen 1.50
pen 1.50

next tells AWK to stop processing the header line and move on to the next one. If the file gains a new column, this command keeps working.

Take the column number from a shell variable

awk -v col=3 '{print $col}' w.txt     # c

Because the number comes from outside, one script can print a different column each time, for example awk -v col="$1" '{print $col}' file.txt inside a shell script.

Unique values from a column

awk -F, 'NR>1 {print $2}' sales.csv | sort -u
# east
# north
# south

awk -F, 'NR>1 && !seen[$2]++ {print $2}' sales.csv     # keeps first-seen order
# north
# south
# east

Problems and fixes

The column has spaces inside it

With the default splitting, a tab-separated file where a name contains a space gives you half the name:

printf 'id\tname\tcity\n1\tAda Lovelace\tLondon\n' | awk 'NR>1 {print $2}'
# Ada

printf 'id\tname\tcity\n1\tAda Lovelace\tLondon\n' | awk -F'\t' 'NR>1 {print $2}'
# Ada Lovelace

Set the separator to a tab with -F'\t'.

Parsing `ls -l` output

A file name with a space is split into several fields, so $9 returns only the first word and $NF returns only the last one. In my test, a file called my report.txt gave my for $9 and report.txt for $NF. Do not parse ls output when names may contain spaces; use find or a shell glob instead.

A blank line crashes `$(NF-1)`

On an empty line NF is 0, so $(NF-1) asks for field -1. Both gawk and mawk stop with an error and exit code 2 (gawk says "attempt to access field -1", mawk says "negative field index $-1"). On a line with only one field, $(NF-1) is $0, the whole line. Guard it with a condition:

awk 'NF>1 {print $(NF-1)}' file.txt

Files from Windows (CRLF line endings)

Each line ends in a hidden carriage return, which becomes part of the last field. od -c shows it:

awk -F, '{print $NF "|"}' crlf.csv | od -c
# b  \r   |  \n

Remove it first:

awk -F, '{sub(/\r$/,""); print $NF "|"}' crlf.csv
# b|
# d|

CSV fields with commas inside quotes

Splitting on -F, cannot know that "hi, there" is one value. On the line Ada,"hi, there", field 2 is "hi and field 3 is there". For real CSV files you need a parser; this is covered in the CSV article of this series.

Frequently asked questions

How do I print the last column in AWK?

Use $NF. For the second to last, use $(NF-1).

How do I print columns 2 to 4?

Use the range loop shown above, with from=2 and to=4.

How do I print a column when the delimiter is a comma?

Add -F,, for example awk -F, '{print $3}' file.csv.

What is the difference between `print $1, $2` and `print $1 $2`?

The comma puts the output separator (a space by default) between the values. Without it they are joined with nothing in between.

Originally published on DEV Talk.

PHP json_decode() Returns null: Catch Invalid JSON Errors

devTalk — Wed, 28 Oct 2026 15:00:04 +0000

Here is a bug that looks like a data problem and is really an error-handling problem. A payment webhook arrives, your code decodes it, and the order never gets marked as paid. No exception, no log line, nothing. The cause is one function that fails quietly: json_decode() returns null when the JSON is broken, which is exactly what it also returns for a perfectly valid null.

The silent failure

$payload = '{"event":"paid","amount":1999,}';   // trailing comma: invalid JSON

$data = json_decode($payload, true);

echo $data['event'] ?? 'none';   // prints: none

The variable $data is NULL, the ?? operator hides the problem, and the script carries on as if the webhook was empty. Now compare with a valid payload:

Input	`json_decode()` returns	Is it an error?
`{"event":"paid",}`	`NULL`	Yes (syntax error)
`null`	`NULL`	No, it is valid JSON
`''` (empty string)	`NULL`	Yes (syntax error)

You cannot tell these apart from the return value alone. The old workaround is to call json_last_error() straight afterwards, which does work (it returned 4 for the broken payload and 0 for null when I ran it). But it relies on a global error state that you have to remember to check, in the right place, every single time.

The fix: JSON_THROW_ON_ERROR

Pass the JSON_THROW_ON_ERROR flag and PHP throws a JsonException instead of returning null:

try {
    $data = json_decode($payload, true, 512, JSON_THROW_ON_ERROR);
} catch (JsonException $e) {
    // $e->getMessage() === 'Syntax error'
    // $e->getCode()    === 4   (the value of JSON_ERROR_SYNTAX)
    http_response_code(400);
    exit('Invalid JSON: ' . $e->getMessage());
}

Now a broken payload stops at the catch block with a reason, instead of travelling further into your code as null. The exception message and code are the same ones json_last_error_msg() and json_last_error() would give you.

The named-argument shortcut

The signature of the function is json_decode(string $json, ?bool $associative = null, int $depth = 512, int $flags = 0). The flag is the fourth parameter, so the positional version forces you to type the default depth just to reach it: 512. With a named argument you can skip it:

$data = json_decode($json, true, flags: JSON_THROW_ON_ERROR);

This needs PHP 8.0 or newer, because that is when named arguments arrived. I checked that it behaves the same as the positional call, and that a broken payload throws with the same message.

One more detail: the second parameter is optional too. If you leave out true, you get objects instead of arrays, and you can still ask for arrays through a flag:

json_decode($json, flags: JSON_THROW_ON_ERROR);                          // stdClass object
json_decode($json, flags: JSON_OBJECT_AS_ARRAY | JSON_THROW_ON_ERROR);   // array

Things that still catch people out

A validnullis stillnull. The flag only throws on errors, and null is not an error. If you expect an object or an array, check the type after decoding: if (!is_array($data)) { ... }.
An empty string throws. json_decode('', true, flags: JSON_THROW_ON_ERROR) gives a syntax error, so an empty request body ends up in your catch block too. Usually that is what you want.
The flag leavesjson_last_error()alone. In my test, a call with the flag neither set nor reset the global error state. Do not mix the two styles in the same code path.
Depth is an error as well. json_decode('[[1]]', true, 1, JSON_THROW_ON_ERROR) throws "Maximum stack depth exceeded".
json_encode()has the same problem and the same fix. Without the flag, json_encode("\xB1\x31") returns false. With it, you get a JsonException saying "Malformed UTF-8 characters, possibly incorrectly encoded".

Only need to know if it is valid?

From PHP 8.3 there is json_validate(), which returns a boolean without building the decoded data:

json_validate('{"a":1}');     // true
json_validate('{"a":1,}');    // false
json_validate('null');        // true

Use it when you only want to check a string, for example before storing it. If you need the data, decode once with JSON_THROW_ON_ERROR and skip the extra pass.

PHP 8.6 makes the message more useful

On PHP 8.6.0 RC4 (a release candidate, so details may still change), the same broken payload produced Syntax error near location 1:16 instead of plain Syntax error, and the depth error added a location too. On PHP 8.3.6 and 8.4.27 the messages were the short ones. If you log or test against exact message text, expect it to change when you upgrade.

A handy rule

If the JSON comes from outside your code (a request body, a webhook, a queue message, an API response), decode it with JSON_THROW_ON_ERROR. If you wrote the string yourself a line earlier, plain json_decode() is fine.

References

Originally published on DEV Talk.

Node.js October 2026 Security Release: What to Patch Now

devTalk — Wed, 28 Oct 2026 14:02:07 +0000

On Wednesday, October 21, 2026, the Node.js project shipped security releases for three lines: 26.x, 24.x and 22.x. The advisory fixes 10 vulnerabilities, 4 rated High, 3 Medium and 3 Low. One of the High ones, a memory-corruption bug in the bundled Brotli decoder, is described as allowing remote code execution, and that is the reason to patch this week rather than at the next maintenance window.

The short version

Release line	Upgrade to at least
Node.js 26.x	`v26.12.1`
Node.js 24.x	`v24.22.1`
Node.js 22.x	`v22.23.4`

The advisory lists no workarounds for any of the 10 issues, so upgrading is the fix. It also says end-of-life versions are always affected when a security release ships. Node.js 20 and earlier are end of life and get no patch, so on those the only route is moving to a supported line.

Check what you are running

# on a server or CI runner
node --version

# inside a container image
docker run --rm your-image:tag node --version

# are you on Express 5 / body-parser 2?
npm ls express body-parser

# do you call the Brotli API directly?
grep -rn "brotliDecompress\|createBrotliDecompress" src/

An image pinned to an old tag stays vulnerable until you rebuild it. After upgrading, restart every Node.js process, containers included, so the patched runtime is actually loaded. If you use your distribution's Node.js package, also check your distro's advisory: the Aikido write-up notes that packaged builds may use the system Brotli library.

# nvm example
nvm install 24
nvm alias default 24
node --version     # expect v24.22.1 or newer

# Docker: rebuild from a fresh base image
docker build --pull -t my-app .

All 10 vulnerabilities

CVE	Severity	Lines	What it is
CVE-2026-111497	High	26, 24, 22	Brotli decoder out-of-bounds write in `node:zlib`
CVE-2026-105482	High	26, 24, 22	`dns.resolve*()` truncates a hostname at a NUL character
CVE-2026-105483	High	26, 24	`tls.checkServerIdentity()` fails to treat an IP written with fullwidth digits as an IP
CVE-2026-106069	High	26, 24, 22	TLS hostname check bypass when a host "loosely" resolves to an IP address
CVE-2026-105481	Medium	26, 24, 22	Keep-alive socket treats data that arrived early as the response to the next request
CVE-2026-106072	Medium	26, 24, 22	DNS names in a certificate can authenticate non-canonical IPv4 spellings
CVE-2026-106074	Medium	26, 24, 22	`crypto.randomUUID()` and `randomInt()` caches saved into snapshots and single executable apps
CVE-2026-106070	Low	26 only	`zlib.ZipFile` returns attacker-controlled bytes after a failed header validation
CVE-2026-106071	Low	26 only	Permission Model wildcard grant allows more than intended
CVE-2026-106073	Low	26, 24, 22	Permission Model bypass through `fs.realpathSync()`

The release also updates c-ares, undici, the CA certificate bundle (the advisory lists "eMudhra Intermediate CA Compromise"), OpenSSL on 24.x and 22.x, and SQLite on 22.x.

The Brotli bug is the urgent one

CVE-2026-111497 is an out-of-bounds write in the Brotli decoder that Node.js bundles. According to the advisory, an invalid block switch was treated as "need more input" instead of a hard format error, so the streaming decoder kept appending data to a small internal buffer. Malicious Brotli data passed to node:zlib decompression can cause a crash, disclose process memory, or lead to remote code execution.

The question that decides your exposure is whether untrusted Brotli data ever reaches your process. I checked the common paths on Node.js 22.22.0, using harmless data:

fetch()decodes Brotli responses on its own. A test server answering with Content-Encoding: br came back as plain text. So any server your code fetches from, including a user-supplied webhook or URL, can send your app Brotli data.
A barehttpserver does not decode request bodies. My test server saw the raw compressed bytes.
Express 5 and body-parser 2 do. An Express 5.3.0 app using express.json() (with body-parser 2.4.0, which has a br case in its body reader) parsed a Brotli-compressed JSON body without any extra setup. So any client can feed Brotli data to such an app. That matches what Aikido reports.

In short: if you run Express 5 or call fetch() against servers you do not control, assume you are exposed.

How bad is it in practice?

Public reports disagree on how easily it can be exploited, so I will not over-claim either way. The Hacker News says a public proof of concept appeared on October 22 and reached code execution only against a deliberately weakened test build, with a crash and a limited memory read on a standard build. Aikido reports later code execution against an unmodified Node.js 24.22.0 image on Linux arm64, not on every attempt, and says 24.22.1 rejects the same input. Both say no exploitation in the wild had been reported as of October 22, and Qualys notes the CVE records did not yet have CVSS scores. A memory-safety flaw in a runtime that handles untrusted network input, with a public proof of concept and a routine version bump as the fix, is a good reason to patch now.

A stopgap if you cannot upgrade today

It does not replace the upgrade, and it does not cover Brotli responses your app fetches from other servers. But for an Express 5 API you can refuse Brotli request bodies before any body parser runs:

app.use((req, res, next) => {
  if (/\bbr\b/i.test(req.headers['content-encoding'] ?? '')) {
    return res.status(415).send('Unsupported Content-Encoding');
  }
  next();
});

app.use(express.json()); // must come after the check

I ran this with Express 5.3.0: a plain JSON body and a gzip body returned 200, and a Brotli body returned 415. The same rule can also live at your proxy or load balancer.

The TLS hostname bugs: real, but narrower

Three of the High and Medium issues (CVE-2026-105483, CVE-2026-106069, CVE-2026-106072) are in how tls.checkServerIdentity() decides that a certificate matches the host you connected to. The advisory describes them as IP-address edge cases: an IP written in an unusual way (fullwidth digits, leading zeros, hex or a bare integer) gets matched against DNS names in the certificate instead of its IP entries. For CVE-2026-106072 the advisory says exploitation needs a numeric DNS name in a certificate from a CA the client trusts, plus a resolver that accepts those spellings, such as glibc.

The leading-zero case rests on a real parsing difference. WHATWG URL parsing reads 0127 as octal, and some operating system resolvers read it as decimal:

new URL('http://0127.0.0.1/').hostname   // '87.0.0.1'  (checked on Node 22.22.0)

So the certificate can be validated against a different address than the one the socket connects to. The practical takeaway: if your services connect to other services by IP address, or take hostnames or URLs from users, these matter. If you only connect to normal DNS names, your exposure is much smaller, but the patch is still worth taking.

The DNS bug: validate, then resolve

CVE-2026-105482 means dns.resolve*() and dns.reverse() cut a name off at an embedded NUL byte, because the name is handed to c-ares as a C string. The advisory's example: attacker.example\0.trusted.example resolves attacker.example. The risky pattern is checking a hostname against an allowlist and then resolving it, which is common in SSRF protection code. The advisory calls it an incomplete fix of CVE-2026-48930, which covered only dns.lookup() and the net/tls host options. The fix is the upgrade. As defence in depth (my suggestion, not from the advisory), reject names containing a NUL byte before you validate them:

if (hostname.includes('\0')) throw new Error('invalid hostname');

The rest, and who can skip them

Keep-alive socket (CVE-2026-105481): unsolicited data that arrived on an idle reused socket can be parsed as the response to your next request. It is described as an incomplete fix of CVE-2026-48931.
randomUUID()/randomInt()(CVE-2026-106074): only matters if you build user-land startup snapshots (--build-snapshot) or single executable apps with useSnapshot. Every process started from the snapshot returns the same values until the cache refills, so IDs can collide or be predicted.
Permission Model (CVE-2026-106071, CVE-2026-106073): only matters if you run with --permission and --allow-fs-read or --allow-fs-write. 106071 is on 26.x only.
zlib.ZipFile(CVE-2026-106070): 26.x only.

Frequently asked questions

Is this being exploited?

As of October 22, the sources I read reported no exploitation in the wild. A public proof of concept exists, so do not treat that as permission to wait.

I don't use Brotli. Am I safe?

Not necessarily. Brotli decoding can happen without you calling it: through fetch() responses, or through Express 5 request body parsing, as shown above. And the other nine fixes apply regardless.

Can I stay on Node.js 20?

It is end of life and will not receive these fixes. Plan the move to a supported line.

References

Originally published on DEV Talk.

PHP 8.2 End of Life: Dates, Risks and How to Upgrade

devTalk — Wed, 28 Oct 2026 12:17:26 +0000

PHP 8.2 stops receiving security fixes on 31 December 2026. Your site will not break on 1 January. What changes is that the next serious PHP vulnerability will be fixed in newer versions only, and 8.2 will stay exposed. If you still run 8.2 in production, you have about two months. That is enough time to upgrade properly, as long as you start by finding out what is blocking you.

The dates

PHP gives each release branch two years of active support (bug fixes and security fixes), then two more years of security fixes only. After that the branch reaches end of life. This is the table from php.net:

Branch	Released	Active support ended / ends	Security support ends
8.2	8 Dec 2022	31 Dec 2024	31 Dec 2026
8.3	23 Nov 2023	31 Dec 2025	31 Dec 2027
8.4	21 Nov 2024	31 Dec 2026	31 Dec 2028
8.5	20 Nov 2025	31 Dec 2027	31 Dec 2029

Note the second column for 8.2: it already lost active support in December 2024. For the past two years it has received security fixes only, and in two months it will receive nothing.

What end of life means in practice

No more patches for new vulnerabilities. The latest 8.2 release at the time of writing is 8.2.35. After the end-of-life date, the PHP project stops issuing security fixes for the branch.
Your host may move before the deadline. Acquia, for example, set its own cutoff at 1 October 2026 and said it would upgrade remaining PHP 8.2 applications to 8.4 itself. Check your provider's notices, because a forced upgrade you did not test is worse than one you planned.
Audits and scanners will flag it. Security scanners and compliance reviews usually treat an end-of-life runtime as a finding by itself.

Which version to move to

The smallest hop is 8.3, but it only buys you one more year (security fixes until 31 December 2027). If you are going to do the work anyway, aim higher:

PHP 8.4 is a good default. Security support runs to the end of 2028.
PHP 8.5 gives the longest runway (end of 2029). Choose it if your framework and dependencies already support it.
PHP 8.3 makes sense only as a stepping stone when something blocks you from going further.

If you use Laravel, look at the php constraint in the framework's own composer.json. On the branch heads I checked, Laravel 13.x requires ^8.3, while 12.x and 11.x require ^8.2. So on Laravel 12 you can upgrade PHP first and Laravel later, but a move to Laravel 13 already means leaving 8.2.

Step 1: find every place that runs 8.2

Production is the obvious one. The forgotten ones are CI images, queue workers, cron containers and staging.

php -v                                   # on every server and container

# what your repo pins
grep -n '"php"' composer.json
grep -rn "php:8.2\|php8.2\|8\.2" Dockerfile* docker-compose* .github 2>/dev/null

Step 2: let Composer tell you what is blocking you

Before changing anything, ask Composer which package would stop you from running a newer PHP:

composer why-not php 8.4.0

I tested this with a package that declares "php": "~8.2.0", and the output points straight at it:

acme/old-lib 1.0.0 requires php (~8.2.0)

Each line is a package you must upgrade, replace, or ask the maintainer about. Two details matter. If your composer.json sets config.platform.php to fake an older PHP, remember to raise or remove it when you upgrade. And composer check-platform-reqs checks the PHP that is actually running, so run it on the new version, inside the new environment.

Step 3: catch deprecations without running the app

The cheapest test is to lint your whole codebase with the new PHP. Linting compiles each file, and some deprecations are raised at compile time. The most common one when moving to 8.4 is the implicit nullable parameter:

function send(string $to, array $options = null): void { /* ... */ }

On PHP 8.4.27 a lint run reports:

Deprecated: send(): Implicitly marking parameter $options as nullable is deprecated, the explicit nullable type must be used instead in nullable.php on line 2

PHP 8.3.6 printed nothing for the same file, which is why this appears only after you move to 8.4. The fix is to add the question mark:

function send(string $to, ?array $options = null): void { /* ... */ }

To check a whole project with the new PHP binary:

find app src -name '*.php' -print0 \
  | xargs -0 -n1 php -d error_reporting=E_ALL -d display_errors=1 -l 2>&1 \
  | grep -v "No syntax errors"

A second one is calling get_class() with no arguments inside a class. Both PHP 8.3.6 and 8.4.27 reported Calling get_class() without arguments is deprecated. Use self::class, static::class or get_class($this) instead.

Linting only finds what the compiler sees. Deprecations that happen when code runs will appear only if that code path executes, so also run your full test suite on the new PHP version with error_reporting set to E_ALL, and read the output.

Step 4: switch the runtime and roll out

Update the Docker base image, for example FROM php:8.2-cli to FROM php:8.4-cli, and rebuild with docker build --pull.
Add the new version to your CI matrix first, and make it pass before touching production.
Check that every PHP extension you need is available for the new version, including ones you install separately. If you build extensions yourself, see the PIE installation guide. Xdebug, for instance, has to be a release that supports your PHP version.
Deploy to staging, then to one production node, then everywhere. Keep the old image tag so you can roll back.

Frequently asked questions

Will my site stop working on 1 January 2027?

No. PHP 8.2 keeps running. You just stop getting security fixes from the PHP project.

Is PHP 8.3 a safe place to land?

It is supported until 31 December 2027, so it is safe for now, but you would be repeating this exercise in a year. If 8.3 is the most your dependencies allow today, use it, and plan the next hop.

My Linux distribution still ships PHP 8.2. Am I covered?

That depends on your distribution's own support policy, not on php.net. Check your vendor's security notices for the exact package version before assuming you are covered.

References

Originally published on DEV Talk.

PHP 8.6 Release Date, New Features and Deprecations

devTalk — Mon, 26 Oct 2026 02:54:22 +0000

PHP 8.6 is scheduled for general availability on 19 November 2026. The feature freeze has passed, release candidate 4 is out, and the headline features are known: partial function application, a clamp() function, a proper Duration class, and a batch of deprecations that may light up your logs. I built PHP 8.6.0 RC4 from the official source and ran every example below, so you can see what the code does today instead of what a blog post promises. One warning first: it is a release candidate, and it is not for production.

Dates

From the schedule on wiki.php.net: RC4 shipped on 21 October, RC5 is planned for 5 November, and GA is planned for 19 November 2026. The hard feature freeze was 22 September, so new features should not appear now, but dates and small details can still move. PHP 8.2 reaches end of life on 31 December 2026, which makes this a good moment to plan your upgrade path anyway.

1. Partial function application: the ? placeholder

You can now create a function from another function by leaving some arguments open with ?:

$slug = str_replace(' ', '-', ?);

$slug('Hello World');            // "Hello-World"  ($slug is a Closure)

array_map(str_repeat(?, 2), ['ab', 'c']);   // ["abab", "cc"]

It works on methods too ($greeter->greet('Hello', ?) gave "Hello, Ada!"), and with several placeholders (array_slice(?, ?, 2) called with [1,2,3,4,5] and 1 returned [2,3]). Use ... to leave all remaining arguments open:

$pad = str_pad('7', ...);
$pad(3, '0', STR_PAD_LEFT);      // "007"

Two things I hit while testing:

...must come last. Writing str_pad(..., pad_type: STR_PAD_LEFT) is a fatal error: "Variadic placeholder must be last".
Every?becomes a required parameter, even if the original parameter was optional. Calling a closure made with str_replace(' ', '-', ?, ?) with one argument threw ArgumentCountError.

Named arguments work alongside placeholders: str_pad(?, 3, pad_type: STR_PAD_LEFT, pad_string: '0') then called with '7' gave 007. According to the upgrade notes, array_map() with a partial application or first-class callable is compiled into a plain loop, which avoids creating extra closures. I did not benchmark that claim.

2. clamp()

clamp(101, min: 0, max: 100);   // 100
clamp(-5, 0, 100);              // 0
clamp(42, 0, 100);              // 42
clamp(5, 10, 1);                // ValueError

The error message was: "clamp(): Argument #2 ($min) must be smaller than or equal to argument #3 ($max)". No more hand-written max($min, min($max, $value)).

3. Time\Duration

An immutable class for lengths of time. It has factory methods (fromSeconds, fromMilliseconds, fromMinutes, fromHours, and fromIso8601DurationString), arithmetic (add, sub, multiplyBy, divideBy, negate, absolute) and a static compare(). The readable properties are seconds, nanoseconds and negative:

use Time\Duration;

// exponential backoff: 100ms, 200ms, 400ms, 800ms
foreach ([0, 1, 2, 3] as $attempt) {
    $delay = Duration::fromMilliseconds(100)->multiplyBy(2 ** $attempt);
}

Duration::fromMinutes(90)->seconds;                        // 5400
Duration::compare(
    Duration::fromMinutes(90),
    Duration::fromIso8601DurationString('PT1H30M')
);                                                          // 0 (equal)

The backoff loop produced 100,000,000, 200,000,000, 400,000,000 and 800,000,000 nanoseconds. Trying to assign $duration->seconds = 9 threw "Cannot modify readonly property". Note that the class has no toMilliseconds() method in RC4: you read seconds and nanoseconds yourself.

4. Smaller additions that worked in my tests

Readonly properties can have defaults: public readonly array $tags = ['a']; is now valid, and changing it still throws.
Enums can define__debugInfo(): var_dump(Level::Low) printed my custom label and value.
#[\Override]works on class constants. On a constant that does not exist in a parent or interface, PHP stops with: "Lone::X has #[\Override] attribute, but no matching parent constant exists".
Objects stored in constants can be mutated: after const BOX = new Box;, BOX->n = 5; worked.
JSON errors say where: json_last_error_msg() returned "Syntax error near location 1:15", and JsonException carries the same detail.
pack()andunpack()accept<and> on signed integers: pack('l>', -2) gave fffffffe and pack('l<', -2) gave feffffff.
New in core: a SortDirection enum (cases Ascending and Descending) and a polling API (Io\Poll). The upgrade notes list both; I confirmed the enum exists and did not exercise the polling API.

Deprecations: run your test suite with E_ALL

I triggered each of these on RC4. They are deprecations, not removals, but they will add noise to logs:

Code	What PHP 8.6 says
`return` inside `finally`	"Returning from a finally block is deprecated"
`return 5;` in `__construct()`	"Returning a value from a constructor is deprecated"
`define('X', 1, true)`	Third argument deprecated (it also emitted a warning that this will be an error in PHP 9.0)
A class named `let` or `is`	"Using "let" as a class name is deprecated since 8.6"
`const _ = 1;`	"Calling a constant "_" is deprecated since 8.6"
Class constant named `namespace`	"Declaring class constant called 'namespace' is deprecated"
A function named `readonly`	"Calling a function "readonly" is deprecated since 8.6"
`is_a($str, $cls, false)` with a string	Deprecated when `$allow_string` is false
`__debugInfo(): ?array`	Deprecated: declare a non-nullable `array` return type

The upgrade notes list more, including Mbregex (the Oniguruma library behind it is no longer maintained) and several mysqli and Reflection calls. Two that older codebases may hit are return inside finally and a third argument to define().

Changes that affect behavior

Safer session defaults. session.use_strict_mode now defaults to 1, session.cookie_httponly to 1, and session.cookie_samesite to "Lax". I confirmed the new defaults in the source. If your app reads the session cookie from JavaScript, or depends on cross-site POSTs carrying it, set the old values explicitly. Laravel manages its own session cookie settings, so most Laravel apps are unaffected; apps on native PHP sessions should check.
By-referenceforeach can now visit elements it previously skipped after array compaction, and ?? or empty() on a magic property no longer calls __get() in one specific case. Both are in the upgrade notes under backward-incompatible changes.
preg_grep()now returnsfalse instead of a partial array when a PCRE error occurs, like the other preg_* functions.

Is Laravel ready?

At the 13.38.0 tag, Laravel's composer.json requires ^8.3, which allows PHP 8.6, but its test workflow covers PHP 8.3, 8.4 and 8.5, with no 8.6 in the list. So the framework is not yet tested against 8.6 in that release, and your packages probably are not either. A sensible plan: add an optional CI job on the 8.6 release candidate that is allowed to fail, run it with error_reporting set to E_ALL, and fix the deprecations that appear. Wait for your dependencies before moving production.

References

Originally published on DEV Talk.

DEV Community: devTalk

Laravel Context: Share Data Across Logs and Queued Jobs

The quick answer

Step 1: set it in middleware

Step 2: read it inside the job

What the log looks like

How it travels to the worker

Hidden context

Things to know

Common questions

Does Context work with every queue driver?

How do I add a request ID or trace ID to every log line?

What is the difference between Log::withContext() and Context::add()?

Can I change what is carried to jobs?

Sources

AWK gsub() Examples: Find and Replace Text in Linux

The quick answer

Sample file

Syntax

sub() versus gsub()

Count the replacements

Change only some lines

Replace inside one column

Use the matched text with &

Special characters: the dot, slashes and plus signs

Common recipes

Use shell variables with gsub

Case-insensitive replacement

Capture groups: gensub() in gawk only

Change the file itself

gsub or sed?

Common problems

Frequently asked questions

What is the difference between sub and gsub in AWK?

How do I replace text in one column only?

How do I use a variable in gsub?

Does AWK gsub support regex groups like \1?

How do I replace a literal dot or slash?

Next in this series

AWK Remove Duplicates: Lines, Columns and CSV Rows

The quick answer

Sample files

Remove duplicate lines and keep the order

How !seen[$0]++ works

Remove duplicates by one column

Remove duplicates by several columns

Keep the last occurrence instead of the first

Remove only consecutive duplicates

Show only the duplicates

Show lines that appear only once

Count how many times each line appears

Ignore case or stray whitespace

Remove duplicates across several files

Save the result

Count how many lines were removed

AWK, sort -u or uniq?

Common problems

Frequently asked questions

How do I remove duplicate lines in Linux without sorting?

How do I remove duplicates based on one column in AWK?

How do I keep the last duplicate instead of the first?

What does !seen[$0]++ mean?

Is the AWK method case sensitive?

Next in this series

AWK Count Occurrences: Lines, Words, Characters & Values

The quick answers

Sample files

Count lines that match

Count every occurrence, not just lines

Ignore case

Whole words only

Count one character

Count how often each value appears

Sort by count, highest first

Top visitors in a web server log

Add percentages

Count distinct values

Count lines and words

Count words across a file

Use a shell variable as the search term

What is the difference between `Log::withContext()` and `Context::add()`?

`sub()` versus `gsub()`

Use the matched text with `&`

Capture groups: `gensub()` in gawk only

How `!seen[$0]++` works

What does `!seen[$0]++` mean?

What is the difference between `grep -c` and counting with AWK?

What is the difference between `-F` and `FS`?

Why does `$1` print the whole line?