# Network logs

> See every request your units answer, every name they look up and every connection they make, and search all of it. Nothing to install or configure; it works for every unit.

Web page: https://kuiper.sh/docs/network/

## What Kuiper records

Open a unit and choose **Network**. Three views show what is going in and out of it.

| View | A row is | It comes from |
| --- | --- | --- |
| **HTTP** | A request to your unit's address: method, path, status, how long it took, who sent it | The edge proxy, which every request passes through |
| **DNS** | A name your unit's containers looked up, and what they were told | A small resolver Kuiper runs on the node |
| **Connections** | A connection to or from your unit's containers, with bytes each way and how long it lasted | The node's connection table |

There is nothing to switch on and nothing to add to your app. It works for every unit, on its Kuiper address and on your own domains. Only traffic to and from your units is recorded. The dashboard, the API and other people's units never appear.

At the top of the page a short summary says whether the unit is healthy: requests, the share that failed, how long they took, and a graph of the last hour.

## Requests

Each row shows the time, method, path, status, duration and the client's address. Click a row to see the rest: the query string, host, user agent, referer, response size, the deployment that answered, and the address the edge sent the request to.

When the edge cannot get an answer from your app, the row says why, for example `dial tcp 10.0.0.2:30000: connect: connection refused`. A 502 is easier to understand with that sentence next to it.

### Request IDs

Kuiper adds an `X-Request-Id` header to each request that does not already carry one, passes it to your app, and sends it back to the client in the response. If your app logs that header, you can go from a request to the app's own output:

1. Open the request in the dashboard.
2. Press **App logs around this time**. The link carries the request ID.

A request ID that the client sent is kept as it was. Send your own and you can find that exact request later with `@rid:your-id`.

### Copy as curl

The details panel has **Copy as curl**. It repeats the method and URL. Headers and bodies are not recorded, so add your own `-H` and `-d`. Query parameters that were redacted (see [What is kept](#what-is-kept-and-what-is-not)) are copied as `REDACTED`.

## Search

Every filter box on this page uses the same language as the rest of the dashboard.

| Write | It matches |
| --- | --- |
| `@status:503` | Rows whose status is 503. Text is compared ignoring case. |
| `@status:5xx` | A class: 500 to 599. |
| `@status:400..499` | A range, ends included. |
| `@duration:>1s` | A comparison: `>`, `>=`, `<`, `<=`. Times take `ms` and `s`. |
| `@status:502,503,504` | Any of several values. |
| `@path:/api/*` | `*` matches anything. `*.openai.com` works too. |
| `-@method:OPTIONS` | A leading `-` leaves rows out. It works on any term. |
| `timeout` or `"connection reset"` | Words in the row: the path, host, client and request ID for HTTP; the name and answers for DNS; the peer and ports for connections. |

Terms are combined: `@status:5xx @path:/api/* -@method:POST` means all three. A key can be written in any case. If you write a key Kuiper does not know, or a number it cannot read, the box tells you what to write instead of showing nothing.

Start typing `@` in the dashboard for the keys, and `@status:` for values to pick from. The chips under the box are common searches. Press `/` to jump to the box.

### Keys

| View | Keys |
| --- | --- |
| HTTP | `status` (also `code`), `method`, `path`, `host`, `ip` (the client), `duration` (also `ms`, `latency`), `bytes` (response size), `ua`, `referer`, `rid` (request ID), `upstream`, `deployment`, `query`, `error`, `proto` |
| DNS | `name`, `type`, `rcode`, `answer`, `duration`, `client`, `upstream`, `transport`, `deployment` |
| Connections | `ip` (the other end), `dst`, `src`, `port` (destination), `proto`, `direction` (`egress` or `ingress`), `peer`, `bytes` (both ways), `duration`, `state` (`open` or `closed`), `kind`, `deployment` |

Some searches to start from:

- `@status:5xx -@path:/healthz` shows failures that are not the health check.
- `@ip:203.0.113.9` shows everything one client did.
- `@rcode:NXDOMAIN` shows names that did not exist.
- `@direction:egress @bytes:>1000000` shows where your unit sends the most data.
- `@peer:*.openai.com` shows every connection to a provider.

### Live and history

New rows appear as they arrive while **Live** is on. If you have scrolled down, they wait behind a **new rows** button instead of moving the page. Scroll to the end of the table and older rows load. **Paused** stops the table moving so you can read it.

## The summary

The strip above the table answers whether the unit is healthy.

- **Requests** and **Errors (5xx)** for the period, and the **latency** at the 50th, 95th and 99th percentile.
- A graph of requests per minute, with the failed part in red, and a bar of responses by class.
- **Busiest paths, clients and destinations.** Numeric and UUID parts of a path are grouped, so `/users/17` and `/users/18` are one row, `/users/:id`. Click any entry to search for it.

The health label follows simple rules. **Failing** is 10% or more of requests ending in a 5xx, from at least five requests. **Degraded** is 1% or more of at least twenty, or a 95th percentile of five seconds or more. **No traffic** means there were no requests in the period. Change the period at the top right, from 15 minutes to 7 days.

## DNS lookups

Containers of your unit are given Kuiper's resolver on their network as their first DNS server and the host's own resolvers as the second. A lookup that the container's built-in resolver cannot answer goes to Kuiper's resolver, which passes it on unchanged and writes down the name, the type, the response code (`NOERROR`, `NXDOMAIN`, `SERVFAIL`), the answers and how long the real resolver took.

The resolver can only make DNS better. If it is stopped, or the node's firewall will not let containers reach it, the container uses the host's resolvers as before and logging stays off for that network.

Containers pick this up when they start. Units already running keep the DNS they started with until their next deploy or restart.

Sandboxes and browsers that a unit starts use their own resolver settings and are not logged.

## Connections

A row is written when a connection ends, and again every minute for a connection that is still open, so a connection that has stopped moving bytes shows up with a growing age and a flat byte count. A later row for the same connection replaces the earlier one: the list shows each connection once, in its latest state.

- **Direction** is `egress` when your unit opened the connection and `ingress` when it accepted one.
- **Peer** is the other end. It is the unit's name when that is another unit of the project, **edge** for Kuiper's proxy passing a request on, and otherwise the name your unit last looked up for that address (`api.openai.com`), or the address.
- **Bytes** are from your unit's side: `↑` is what it sent and `↓` what it received.

Connections come from the node's connection table. A connection that was already open when the node's daemon started is timed from then. Volumes need the kernel to count bytes; the node setup does that, and the page says so when it is off.

## From the CLI

```sh
kuiper network http -u web
kuiper network http -u web -q '@status:5xx -@path:/healthz' --since 6h
kuiper network http -u web -q '@rid:client-req-0001'
kuiper network dns -u web -q '@rcode:NXDOMAIN'
kuiper network flows -u web -q '@direction:egress @bytes:>1000000'
kuiper network summary -u web --window 24h
```

Rows print oldest first, like `kuiper logs`. `-u` limits to a unit; without it the whole project is searched. `--since` takes `15m`, `6h`, `7d` or a time, and `--until` ends the range. `--limit N` sets how many rows to start with (default 100). `-f` follows new rows, and with `--json` prints one JSON object per line. See the [CLI reference](https://kuiper.sh/docs/cli/#network-logs).

## What is kept, and what is not

Recorded: the fields above. Not recorded: request and response bodies, cookies, and headers other than the user agent, the referer and the request ID.

Secrets are removed before a row leaves the machine it was written on:

- `Authorization`, `Cookie` and similar headers are replaced by the edge and never recorded.
- The value of a query parameter named `token`, `key`, `secret`, `password`, `code`, `sig`, `signature`, `auth`, `access_token` or `api_key` (in any capitalisation, along with a few close relatives such as `refresh_token` and names ending in `_token`, `_secret` or `_password`) is replaced by `REDACTED`, in the query string and in the referer. The name stays, so you can see that it was sent. The control plane does this again when a row arrives.
- A path is recorded as it was sent. Do not put secrets in paths.

Rows are visible to the people in your organisation and kept for a limited time: seven days for requests, three days for lookups and connections. A project that is very busy keeps its newest rows only: 500,000 requests, 300,000 lookups and 300,000 connections.

## When nothing shows up

An empty view says why. These are the usual reasons.

| The page says | What it means |
| --- | --- |
| No node has reported network logs yet | The node daemon is older than this feature. Update Kuiper and re-run the installer. |
| The edge has not logged a request yet | The edge is running and has not served a request since it started logging. If requests have been made and this stays, re-run the installer so the edge writes an access log. |
| The edge's log directory does not exist | The edge was started before the log directory was mounted. Re-run the installer. |
| Byte counts are off | The kernel is not counting bytes per connection. Run the node setup again, or `sysctl -w net.netfilter.nf_conntrack_acct=1`. |
| DNS is unavailable | The node could not set up the resolver for this project's network. The message says why. |

Requests that reach no running instance of a unit, for example while it is stopped, get Kuiper's 404 page and are not logged.

## Turning it on in a self-hosted Kuiper

On a hosted Kuiper there is nothing to do. On your own Droplet or server, update the checkout and run the installer again, as for any update:

```sh
git pull
sudo deploy/droplet/install.sh
```

The installer rebuilds the control plane and `kuiperd`, and then:

- creates `/var/lib/kuiper/edge-logs` and recreates the `kuiper-caddy` container once with that directory mounted at `/var/log/kuiper-edge`. The edge is unavailable for a few seconds. The routes it was given are kept in the `kuiper-caddy-config` volume (the container runs `caddy run --resume`) and the certificates in `kuiper-caddy-data`, and the installer pushes the routes again straight after.
- pulls the current `caddy:2` when it recreates the container. The access log needs Caddy 2.8 or newer. If an older Caddy refuses it, the control plane pushes the routes without logging so they keep updating, and says so in its log.
- on every node it sets up, sets `net.netfilter.nf_conntrack_acct=1` (in `/etc/sysctl.d/99-kuiper-network.conf`) and loads `nf_conntrack` at boot. `kuiperd` also sets it when it starts.

Then redeploy or restart units so their containers use the resolver. Other nodes: update `kuiperd` and run `deploy/node/setup-node.sh` again.

An edge you run yourself, for example a regional one, needs the same mount on its Caddy container, `-v /var/lib/kuiper/edge-logs:/var/log/kuiper-edge`, on the node that runs `kuiperd` next to it. The control plane pushes the logging config to every edge it manages.

### The resolver and the firewall

The resolver listens on each project network's gateway address, port 53, so it never collides with `systemd-resolved` on `127.0.0.53`. Containers reach it through the host's INPUT chain. Before it points a container at the resolver, `kuiperd` adds two rules, UDP and TCP port 53 from that network's subnet to its gateway, ahead of `ufw`'s rules, and re-checks them every minute. If it cannot (no `iptables`, for example) it leaves the container's DNS alone. If you manage that firewall yourself, allow the same traffic and set `KUIPER_DNS_FIREWALL=off`.

### Settings

On the control plane:

| Variable | Default | Meaning |
| --- | --- | --- |
| `KUIPER_EDGE_ACCESS_LOG` | `on` | `off` leaves the edge without an access log. |
| `KUIPER_EDGE_ACCESS_LOG_FILE` | `/var/log/kuiper-edge/access.log` | Where the edge's Caddy writes it (inside its container). |
| `KUIPER_NETWORK_HTTP_RETENTION_DAYS` | `7` | Days to keep requests. |
| `KUIPER_NETWORK_DNS_RETENTION_DAYS` | `3` | Days to keep lookups. |
| `KUIPER_NETWORK_FLOW_RETENTION_DAYS` | `3` | Days to keep connections. |
| `KUIPER_NETWORK_HTTP_MAX_ROWS` | `500000` | Most request rows one project keeps. |
| `KUIPER_NETWORK_DNS_MAX_ROWS` | `300000` | Most lookup rows one project keeps. |
| `KUIPER_NETWORK_FLOW_MAX_ROWS` | `300000` | Most connection rows one project keeps. |

On `kuiperd`:

| Variable | Default | Meaning |
| --- | --- | --- |
| `KUIPER_NETWORK_LOGS` | `on` | `off` switches all three off. |
| `KUIPER_EDGE_LOG_FILE` | `/var/lib/kuiper/edge-logs/access.log` | The edge's log on this host. Empty switches HTTP logs off. |
| `KUIPER_DNS_LOGGING` | `on` | `off` leaves Docker's default DNS untouched. |
| `KUIPER_DNS_FIREWALL` | `on` | Let `kuiperd` open port 53 from a project network to its gateway. |
| `KUIPER_FLOW_LOGGING` | `on` | Log connections. |
| `KUIPER_FLOW_INTERVAL` | `10` | Seconds between readings of the connection table. |

A node that cannot reach the control plane keeps up to 20,000 rows of each kind and drops the oldest beyond that, then reports how many it dropped. Nothing else about the node is affected.