Network logs

What Kuiper records#

Open a unit and choose Network. Three views show what is going in and out of it.

There is nothing to switch on and nothing to add to your app. It works for every unit, on its Kuiper address and on your own domains. Only traffic to and from your units is recorded. The dashboard, the API and other people's units never appear.

At the top of the page a short summary says whether the unit is healthy: requests, the share that failed, how long they took, and a graph of the last hour.

Requests#

Each row shows the time, method, path, status, duration and the client's address. Click a row to see the rest: the query string, host, user agent, referer, response size, the deployment that answered, and the address the edge sent the request to.

When the edge cannot get an answer from your app, the row says why, for example dial tcp 10.0.0.2:30000: connect: connection refused. A 502 is easier to understand with that sentence next to it.

Request IDs#

Kuiper adds an X-Request-Id header to each request that does not already carry one, passes it to your app, and sends it back to the client in the response. If your app logs that header, you can go from a request to the app's own output:

1. Open the request in the dashboard.
2. Press App logs around this time. The link carries the request ID.

A request ID that the client sent is kept as it was. Send your own and you can find that exact request later with @rid:your-id.

Copy as curl#

The details panel has Copy as curl. It repeats the method and URL. Headers and bodies are not recorded, so add your own -H and -d. Query parameters that were redacted (see What is kept) are copied as REDACTED.

Search#

Every filter box on this page uses the same language as the rest of the dashboard.

Terms are combined: @status:5xx @path:/api/* -@method:POST means all three. A key can be written in any case. If you write a key Kuiper does not know, or a number it cannot read, the box tells you what to write instead of showing nothing.

Start typing @ in the dashboard for the keys, and @status: for values to pick from. The chips under the box are common searches. Press / to jump to the box.

Keys#

Some searches to start from:

• @status:5xx -@path:/healthz shows failures that are not the health check.
• @ip:203.0.113.9 shows everything one client did.
• @rcode:NXDOMAIN shows names that did not exist.
• @direction:egress @bytes:>1000000 shows where your unit sends the most data.
• @peer:*.openai.com shows every connection to a provider.

Live and history#

New rows appear as they arrive while Live is on. If you have scrolled down, they wait behind a new rows button instead of moving the page. Scroll to the end of the table and older rows load. Paused stops the table moving so you can read it.

The summary#

The strip above the table answers whether the unit is healthy.

• Requests and Errors (5xx) for the period, and the latency at the 50th, 95th and 99th percentile.
• A graph of requests per minute, with the failed part in red, and a bar of responses by class.
• Busiest paths, clients and destinations. Numeric and UUID parts of a path are grouped, so /users/17 and /users/18 are one row, /users/:id. Click any entry to search for it.

The health label follows simple rules. Failing is 10% or more of requests ending in a 5xx, from at least five requests. Degraded is 1% or more of at least twenty, or a 95th percentile of five seconds or more. No traffic means there were no requests in the period. Change the period at the top right, from 15 minutes to 7 days.

DNS lookups#

Containers of your unit are given Kuiper's resolver on their network as their first DNS server and the host's own resolvers as the second. A lookup that the container's built-in resolver cannot answer goes to Kuiper's resolver, which passes it on unchanged and writes down the name, the type, the response code (NOERROR, NXDOMAIN, SERVFAIL), the answers and how long the real resolver took.

The resolver can only make DNS better. If it is stopped, or the node's firewall will not let containers reach it, the container uses the host's resolvers as before and logging stays off for that network.

Containers pick this up when they start. Units already running keep the DNS they started with until their next deploy or restart.

Sandboxes and browsers that a unit starts use their own resolver settings and are not logged.

Connections#

A row is written when a connection ends, and again every minute for a connection that is still open, so a connection that has stopped moving bytes shows up with a growing age and a flat byte count. A later row for the same connection replaces the earlier one: the list shows each connection once, in its latest state.

• Direction is egress when your unit opened the connection and ingress when it accepted one.
• Peer is the other end. It is the unit's name when that is another unit of the project, edge for Kuiper's proxy passing a request on, and otherwise the name your unit last looked up for that address (api.openai.com), or the address.
• Bytes are from your unit's side: ↑ is what it sent and ↓ what it received.

Connections come from the node's connection table. A connection that was already open when the node's daemon started is timed from then. Volumes need the kernel to count bytes; the node setup does that, and the page says so when it is off.

From the CLI#

kuiper network http -u web
kuiper network http -u web -q '@status:5xx -@path:/healthz' --since 6h
kuiper network http -u web -q '@rid:client-req-0001'
kuiper network dns -u web -q '@rcode:NXDOMAIN'
kuiper network flows -u web -q '@direction:egress @bytes:>1000000'
kuiper network summary -u web --window 24h

Rows print oldest first, like kuiper logs. -u limits to a unit; without it the whole project is searched. --since takes 15m, 6h, 7d or a time, and --until ends the range. --limit N sets how many rows to start with (default 100). -f follows new rows, and with --json prints one JSON object per line. See the CLI reference.

What is kept, and what is not#

Recorded: the fields above. Not recorded: request and response bodies, cookies, and headers other than the user agent, the referer and the request ID.

Secrets are removed before a row leaves the machine it was written on:

• Authorization, Cookie and similar headers are replaced by the edge and never recorded.
• The value of a query parameter named token, key, secret, password, code, sig, signature, auth, access_token or api_key (in any capitalisation, along with a few close relatives such as refresh_token and names ending in _token, _secret or _password) is replaced by REDACTED, in the query string and in the referer. The name stays, so you can see that it was sent. The control plane does this again when a row arrives.
• A path is recorded as it was sent. Do not put secrets in paths.

Rows are visible to the people in your organisation and kept for a limited time: seven days for requests, three days for lookups and connections. A project that is very busy keeps its newest rows only: 500,000 requests, 300,000 lookups and 300,000 connections.

When nothing shows up#

An empty view says why. These are the usual reasons.

Requests that reach no running instance of a unit, for example while it is stopped, get Kuiper's 404 page and are not logged.

Turning it on in a self-hosted Kuiper#

On a hosted Kuiper there is nothing to do. On your own Droplet or server, update the checkout and run the installer again, as for any update:

git pull
sudo deploy/droplet/install.sh

The installer rebuilds the control plane and kuiperd, and then:

• creates /var/lib/kuiper/edge-logs and recreates the kuiper-caddy container once with that directory mounted at /var/log/kuiper-edge. The edge is unavailable for a few seconds. The routes it was given are kept in the kuiper-caddy-config volume (the container runs caddy run --resume) and the certificates in kuiper-caddy-data, and the installer pushes the routes again straight after.
• pulls the current caddy:2 when it recreates the container. The access log needs Caddy 2.8 or newer. If an older Caddy refuses it, the control plane pushes the routes without logging so they keep updating, and says so in its log.
• on every node it sets up, sets net.netfilter.nf_conntrack_acct=1 (in /etc/sysctl.d/99-kuiper-network.conf) and loads nf_conntrack at boot. kuiperd also sets it when it starts.

Then redeploy or restart units so their containers use the resolver. Other nodes: update kuiperd and run deploy/node/setup-node.sh again.

An edge you run yourself, for example a regional one, needs the same mount on its Caddy container, -v /var/lib/kuiper/edge-logs:/var/log/kuiper-edge, on the node that runs kuiperd next to it. The control plane pushes the logging config to every edge it manages.

The resolver and the firewall#

The resolver listens on each project network's gateway address, port 53, so it never collides with systemd-resolved on 127.0.0.53. Containers reach it through the host's INPUT chain. Before it points a container at the resolver, kuiperd adds two rules, UDP and TCP port 53 from that network's subnet to its gateway, ahead of ufw's rules, and re-checks them every minute. If it cannot (no iptables, for example) it leaves the container's DNS alone. If you manage that firewall yourself, allow the same traffic and set KUIPER_DNS_FIREWALL=off.

Settings#

On the control plane:

On kuiperd:

A node that cannot reach the control plane keeps up to 20,000 rows of each kind and drops the oldest beyond that, then reports how many it dropped. Nothing else about the node is affected.