DEV Community: Chris

NGINX Log Taming - Structured Data & Statistical Analysis

Chris — Sun, 25 Oct 2026 00:42:54 +0000

Logs are great! They provide a lot of information that assists in understanding the current environment and provide insight when trying to solve problems. It seems every application has its own logging style. Some log to a file, some log to the system journal, some log to standard out, etc. In addition to the variance in where they log, they generally have their own logging format. Some attempt to log in plain English while some log using application specific codes that must be cross referenced in order to understand what is happening.

At some point logging collided with the concept of structured data and their was much rejoicing! Suddenly we could shelve our custom parsers, as proud of them as we were, and enjoy logs that have predictable key/value pairs. The most common way applications offer logs in a structured data format is via JSON output.

Not only are JSONified logs wonderful for scripts that ingest them, when we are hacking for our lives during an outage we can pipe the log file through a utility such as jq and find inspiration to keep living during dark times.

Now, on to the namesake of this article, NGINX! A lot of readers here may admin NGINX servers that produce giga bytes of logs every day. processing large logs for meaningful information can be challenging. NGINX has an option to change the logging format to JSON versus the difficult to read/parse default format. This maneuver involves creating a custom log_format and then associating it with the access_log.

The custom log_format definition must be in the /etc/nginx/nginx.conf file, while the access_log directive that invokes it may be in the same file for global effect, or in an a specific site file for more granular control of logs.

The log_format definition belongs in the http context of the /etc/nginx/nginx.conf file. Here is an example custom log_format that is named custom_json:

http {
    <-- snip -->

    log_format custom_json escape=json '{'
        '"time_local":"$time_local",'
        '"remote_addr":"$remote_addr",'
        '"host":"$host",'
        '"request":"$request",'
        '"status": "$status",'
        '"body_bytes_sent":"$body_bytes_sent",'
        '"request_time":"$request_time",'
        '"http_referrer":"$http_referer",'
        '"http_user_agent":"$http_user_agent"'
    '}';

    <-- snip -->
}

Next we invoke the custom parser, this can be defined in the http, server, or location contexts. Here is an invocation example:

access_log /var/log/nginx/access.log custom_json;
Once these changes have been made, restart the service, and have a look at the new log format. Now a line in the log file looks like this:

root@pagoda:/var/log/nginx# tail -1 access.log
{"time_local":"25/Oct/2026:08:30:50 +0900","remote_addr":"192.168.82.1","host":"activebit.net","request":"GET / HTTP/1.0","status": "200","body_bytes_sent":"65","request_time":"0.000","http_referrer":"","http_user_agent":""}

This is arguably harder on the eyes than the default format, but if we pipe it to a utility such as jq in renders in this way:

root@pagoda:/var/log/nginx# tail -1 access.log | jq .
{
  "time_local": "25/Oct/2026:08:30:50 +0900",
  "remote_addr": "192.168.82.1",
  "host": "activebit.net",
  "request": "GET / HTTP/1.0",
  "status": "200",
  "body_bytes_sent": "65",
  "request_time": "0.000",
  "http_referrer": "",
  "http_user_agent": ""
}

This is much easier to read and quickly understand what all the parts of the log message mean. Since it is structured data we do not have to write a custom parser that's prone to errors in order to programmatically analyze the data. Here is an example using python3 of how to quickly get statistics for a given key seen in the log file:

#!/usr/bin/env python3

import json
import sys 
from pprint import pprint

subjFile = sys.argv[1]
keyName = sys.argv[2]

def getFlatFile(ourFile):
    hook = open(ourFile, errors='ignore')
    raw = hook.read()
    hook.close()
    return raw.splitlines()

logList = getFlatFile(subjFile)

statHash = {}
linesProcessed = 0 
for l in logList:
    l = json.loads(l)
    linesProcessed += 1
    if l[keyName] not in statHash:
        statHash[l[keyName]] = 0 
    statHash[l[keyName]] += 1

print('stats for key: ' + keyName)
for seenKey in sorted(statHash):
    percent = round((statHash[seenKey] / linesProcessed) * 100)
    print(seenKey + ' occurances: ' + str(statHash[seenKey]) + ' percent: ' + str(percent) + '%')

This is an example of evaluating the 'status' key value:

root@pagoda:/home/chris/dev.to# ./json_log_eval.py /var/log/nginx/access.log status
stats for key: status
200 occurances: 61 percent: 29%
400 occurances: 11 percent: 5%
403 occurances: 9 percent: 4%
404 occurances: 126 percent: 60%
405 occurances: 1 percent: 0%
444 occurances: 1 percent: 0%

Fear No Log File!

Network Traffic Awareness

Chris — Thu, 22 Oct 2026 10:41:40 +0000

We have many connected devices such as phones, watches, tablets, laptops, cameras, refrigerators, air conditioners, etc; yet do we know who these devices are communicating with or how? Shouldn't we know? This train of thought inspired the 'Traffic Evaluator' project.

We can know what internet hosts our devices are communicating with and how.

Recording the network traffic and analyzing it reveals who our devices are communicating with. This also gives us insight into how much our ISP can know about us. Any organization in the 'network path' can easily harvest all this information.

In order to demonstrate the depth of information that is easily available through packet analysis we need a test point between the user and the internet. Providing a VPN tunnel with a respected protocol such as WireGuard allows this tool to have such a test point.

If you are interested the 'Traffic Evaluator' project is at https://activebit.net/te . This site allows a user to download and install a WireGuard tunnel configuration where all the traffic is under surveillance and a usage report can be generated at the users request. The report contains details such as:

Organizations the connected device is communicating with
Countries where the remote servers are located
Second Level Domain Summary
FQDNs seen in dns requests, TLS & QUIC Headers
Understanding of what protocols are in use

Peer Review - Another Dimension to Consider

Chris — Wed, 21 Oct 2026 08:38:31 +0000

When people think of peer review it is often associated with reducing risk for upcoming changes. While this is true there is another dimension that deserves consideration.

When I supported financial institutions and ISPs change control was very serious and peer review was part of that process. I found it very insightful as I learned much about my coworkers through the review. The questions someone poses reveals much more than their answers. The peer review process created an environment that allowed us to better understand our coworkers and suddenly we could realize and respect the depth of their experience, insight, and ingenuity in a real way. This process creates TEAM!

My path is somewhat opposite of others in that I moved from 'stable' organizations to startups. I found that change control and consequently peer review was not much wanted in startups. In some cases I found this rational as a fellow in a startup may be a one person department and conducting peer review with oneself is tiresome.

When there is more than one person working in the same space or concern startups often avoid peer review in the name of speed and agility; DO NOT SLOW ME DOWN!

I argue waiting a day will never kill a project and the respect that can be developed during the peer review process will in the long term be much more valuable than any 'shoot from the hip' change. Peer review is an opportunity to create a team in a natural way. Take advantage of this and become a stronger organization.