Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@

### Breaking

- **Listeners: no TCP port by default, auth on by default.** xtcp2 now serves health, metrics and the link monitor's detail paths on one Unix socket, `/run/xtcp2/metrics.sock` (mode `0600`), with a raw bearer token from `LISTENER_RAW_TOKEN_FILE`, and exits 64 without one. It no longer listens on `:9088` or `:8889`: gRPC, pprof and `/ip` are served only by a listener that names them. `-listener` (repeatable) / `LISTENERS` describes each listener, its credentials (files only) and the classes it serves: `health`, `metrics`, `metrics-detail`, `ip`, `debug`, `grpc-read`, `grpc-admin` ([listeners](docs/security.md#listeners)). TCP must be typed (`network=tcp`) and needs a credential unless it is loopback or serves health alone; `debug` and `grpc-admin` need an admin credential; pprof never serves `cmdline`. A `grpc-read` listener answers admin RPCs `Unimplemented`. Bad listener config, a missing token or token file exits 64; `ConfigService.Set` refuses the same.
- **The per-listener flags are deprecated.** `-promListen*`, `-grpcPort`, `-grpcListen*`, `-grpc*UnixSocket*`, `-linkMonitorIPListen` and their variables still work, expanded to listeners with a `DEPRECATED` line naming the `-listener` equivalent; the Prometheus listener they describe no longer serves pprof, and as TCP it now needs a token. Giving them with `-listener` exits 64. An empty `-listenerAuthMode` / `LISTENER_AUTH_MODE=` now means unset (raw), not disabled.
- **Clients dial the gRPC socket.** `xtcp2ctl` and `xtcp2client` default to `unix:/run/xtcp2/grpc.sock`; pass `-network tcp` for a TCP listener. The OCI images expose no ports, and `-healthcheck` probes the listener serving `health` from the same settings as the daemon. The NixOS module gains `listeners` and `listenerTokenFile` (null generates a token once, at `/var/lib/xtcp2/listener-token`); `linkMonitor.ipListen` becomes a loopback `ip` listener. New gauge `xtcp2_listener_active{name,network,class}`.
- **S3 uses TLS unless told otherwise.** A bare `host:port` S3 endpoint now uses TLS (it was plaintext), and `http://` is refused at startup (exit 64) unless `-s3AllowInsecure` / `S3_ALLOW_INSECURE=true`, which logs a warning and sets `xtcp2_s3_insecure 1`. TLS 1.2 is the minimum; `-s3CAFile` / `S3_CA_FILE` adds a private CA. `ConfigService.Set` refuses a plaintext endpoint without the opt-in.
- **Credentials on the command line.** The production (slim) flavors refuse `-s3AccessKey`, `-s3SecretKey` and the four `-listener*` secret flags with exit 64; development builds warn, now for the S3 flags too and even when the env var is also set. Use the `_FILE` variables.
- **`_FILE` secrets are checked.** A missing or unreadable `*_FILE` path, a non-regular file, or one readable by group or others stops startup with exit 64. It used to be ignored silently (listener tokens) or logged (S3 keys), leaving the credential empty. Mount secrets mode `0400`.
Expand Down
31 changes: 17 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,29 +28,33 @@ cd xtcp2
# 2. Build the main daemon (or `nix develop` for a full dev shell)
nix build .#xtcp2

# 3. Run it against the local host, discarding output, with verbose logging
sudo ./result/bin/xtcp2 -dest null -d 333
# 3. A bearer token for its listeners, readable only by root
sudo install -d -m 0700 /etc/xtcp2
sudo sh -c 'umask 077; head -c 32 /dev/urandom | base64 > /etc/xtcp2/token'

# 4. Run it against the local host, discarding output, with verbose logging.
# Metrics on /run/xtcp2/metrics.sock (the default), gRPC on /run/xtcp2/grpc.sock.
sudo LISTENER_RAW_TOKEN_FILE=/etc/xtcp2/token ./result/bin/xtcp2 -dest null -d 333 \
-listener name=metrics,serves=health+metrics+metrics-detail,address=/run/xtcp2/metrics.sock \
-listener name=grpc,serves=grpc-admin,address=/run/xtcp2/grpc.sock
```

Stream to Kafka/Redpanda instead:

```sh
sudo ./result/bin/xtcp2 -dest kafka:127.0.0.1:9092 -topic xtcp
```
xtcp2 opens no TCP port unless a listener asks for one, and refuses to start without a token; see [listeners](docs/security.md#listeners). Stream to Kafka/Redpanda instead with `-dest kafka:127.0.0.1:9092 -topic xtcp`.

Inspect TCP records live over gRPC, in a second terminal:

```sh
nix build .#xtcp2client -o result-client
./result-client/bin/xtcp2client # streams XtcpFlatRecord from the running daemon
sudo XTCP2_GRPC_AUTH_TOKEN_FILE=/etc/xtcp2/token ./result-client/bin/xtcp2client # streams XtcpFlatRecord
```

The daemon also enables its embedded link monitor by default. Inspect it from another terminal:

```sh
curl -fsS http://localhost:9088/metrics # daemon metrics and link-health summary
curl -fsS http://localhost:9088/metrics/link # link detail
curl -fsS http://localhost:9088/metrics/driver # detailed driver/PHY counters
xc() { sudo curl -fsS --unix-socket /run/xtcp2/metrics.sock -H "Authorization: Bearer $(sudo cat /etc/xtcp2/token)" "http://xtcp2$1"; }
xc /metrics # daemon metrics and link-health summary
xc /metrics/link # link detail
xc /metrics/driver # detailed driver/PHY counters
```

For link monitoring alone, run the standalone exporter:
Expand Down Expand Up @@ -79,8 +83,7 @@ Run `./result/bin/xtcp2 -help` for the full flag list. Common flags:
| `-marshal` | `protobufList` | Wire format (`protobufList`, `protoJson`, `protoText`, `msgpack`, `jsonl`, `csv`, `tsv`) |
| `-netlinkers` | `2` | Parallel decoders per socket (one receiver reads each socket) |
| `-deserializers` | `all` | Which inet_diag attributes to decode |
| `-promListen` | `:9088` | Prometheus metrics listener |
| `-grpcPort` | `8889` | gRPC API port |
| `-listener` | `/run/xtcp2/metrics.sock` | A listener, repeatable: socket or TCP address, credentials, and what it serves ([listeners](docs/security.md#listeners)) |
| `-linkMonitor` | `true` | Embedded Ethernet/RDMA link monitor |
| `-d` | `111` | Debug verbosity (higher = more) |

Expand All @@ -102,7 +105,7 @@ The `goip` tool and its supporting Go code implement an expanding subset of ipro

Optional `/ip/...` HTTP paths expose supported read-only goip commands for remote investigation, including forms such as `/ip/addr/show/dev/eth0` and `/ip/link`. Enable them with `-linkMonitorIPPaths` in xtcp2 or `-ip.paths` in the standalone monitor. `goip-parity` checks requests and output against iproute2; coverage is evolving, so goip is not yet a complete replacement. Its implemented write verbs currently produce **dry-run request files only**, not kernel changes.

xtcp2 can instead move `/ip` and `/i` to a dedicated loopback listener with `-linkMonitorIPListen 127.0.0.1:9089`. That listener deliberately has no bearer authentication and rejects non-loopback peers; see the [security tradeoffs](docs/security.md#optional-loopback-inspection-listener) before enabling it.
xtcp2 can serve `/ip` and `/i` on a dedicated loopback listener, `-listener name=ip,serves=ip,address=127.0.0.1:9089,network=tcp,auth=disabled`. That listener deliberately has no bearer authentication and rejects non-loopback peers; see the [security tradeoffs](docs/security.md#optional-loopback-inspection-listener) before enabling it.

Details: [netlink and goip overview](docs/netlink/README.md), [coverage and parity status](docs/netlink/coverage-status.md), [HTTP command paths](cmd/go-link-monitor/IP-ARGV-PATHS-DESIGN.md).

Expand Down
6 changes: 5 additions & 1 deletion cmd/xtcp2/exit.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ package main
import (
"errors"

"github.com/randomizedcoder/xtcp2/pkg/listeners"
"github.com/randomizedcoder/xtcp2/pkg/secretfile"
"github.com/randomizedcoder/xtcp2/pkg/xtcp"
)

Expand All @@ -22,7 +24,9 @@ func exitCodeFor(err error) int {
return exitOK
case errors.Is(err, errUsage),
errors.Is(err, xtcp.ErrNotCompiledIn),
errors.Is(err, xtcp.ErrInsecureTransport):
errors.Is(err, xtcp.ErrInsecureTransport),
errors.Is(err, listeners.ErrConfig),
errors.Is(err, secretfile.ErrSecretFile):
return exitUsage
default:
return exitFailure
Expand Down
9 changes: 4 additions & 5 deletions cmd/xtcp2/exit_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import (
"os"
"path/filepath"
"strings"
"sync"
"testing"

"github.com/randomizedcoder/xtcp2/gen/go/xtcp_config"
Expand Down Expand Up @@ -73,13 +72,13 @@ func TestRunMain_debugWritersNotCompiledIn(t *testing.T) {
os.Args = append([]string{"xtcp2"}, tc.args...)
t.Cleanup(func() { os.Args = prevArgs })

prevProm := promHandlerStarter
promHandlerStarter = func(context.Context, *sync.WaitGroup, promParams) error { return nil }
t.Cleanup(func() { promHandlerStarter = prevProm })
stubListeners(t)

prevDaemon := daemonRunner
started := false
daemonRunner = func(context.Context, context.CancelFunc, *xtcp_config.XtcpConfig) { started = true }
daemonRunner = func(context.Context, context.CancelFunc, *xtcp_config.XtcpConfig, []xtcp.GRPCListener) {
started = true
}
t.Cleanup(func() { daemonRunner = prevDaemon })

wantRC, wantStarted := exitUsage, false
Expand Down
59 changes: 59 additions & 0 deletions cmd/xtcp2/http_server.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
package main

import (
"context"
"errors"
"log"
"net"
"net/http"
"sync"
"time"
)

// httpShutdownGrace bounds the drain of in-flight requests at shutdown. It is
// under cancelSleepTimeCst, so the drain finishes before the signal handler
// gives up and exits.
const httpShutdownGrace = 3 * time.Second

// fatalf is the package-level abort handler. Defaults to log.Fatalf;
// tests swap this in for a capture so the serve error branch is
// exercisable without exiting.
var fatalf = log.Fatalf

func newHTTPServer(handler http.Handler) *http.Server {
return &http.Server{
Handler: handler,
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 10 * time.Second,
// The link monitor's detail paths answer within their own 10s
// timeout; the write deadline leaves room to send the body.
WriteTimeout: 15 * time.Second,
IdleTimeout: 30 * time.Second,
}
}

// serveHTTP serves ln until ctx ends, then drains in-flight requests for at
// most grace before closing what remains. A serve failure while ctx is live
// calls failed. name prefixes the shutdown logs.
func serveHTTP(ctx context.Context, wg *sync.WaitGroup, name string, srv *http.Server, ln net.Listener, grace time.Duration, failed func(error)) {
defer wg.Done()
served := make(chan error, 1)
go func() { served <- srv.Serve(ln) }()
select {
case err := <-served:
failed(err)
return
case <-ctx.Done():
}
shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), grace)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
log.Printf("%s shutdown: %v; closing", name, err)
if err := srv.Close(); err != nil {
log.Printf("%s close: %v", name, err)
}
}
if err := <-served; !errors.Is(err, http.ErrServerClosed) {
log.Printf("%s serve: %v", name, err)
}
}
Loading