diff --git a/CHANGELOG.md b/CHANGELOG.md index 22015ca2a..035922a9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,8 @@ ### New +- **The S3 upload queue never blocks polling.** It used to block the poller once 16 envelopes were waiting for a slow or unreachable S3. Now `-s3QueueDepth` (default 16, max 4096) and `-s3MaxBufferedBytes` (default 64 MiB, `0` = no byte bound) bound it, and past either it drops the oldest envelope. Drops are counted in `xtcp_s3_dropped_total{reason}` and `xtcp_s3_dropped_bytes_total{reason}`; `xtcp_s3_buffered_bytes{stage}` shows what is held, and `queueFull` now counts drops. Env `S3_QUEUE_DEPTH`, `S3_MAX_BUFFERED_BYTES`; startup-only (`ConfigService.Set` keeps the running values). Budget ≈ `-s3MaxBufferedBytes` + the flush threshold + one upload ([design](docs/design-jitter-and-backoff.md#back-pressure--memory)). +- **Strings from tenants and the network are cleaned.** Container names and images, a tenant's interface names, LLDP and NIC strings, the ASN feed's network owner, netns names and congestion names are cut to 256 bytes (16 for interface and congestion names) at a UTF-8 boundary. Invalid UTF-8 and control characters (C0, DEL, C1) become U+FFFD. Each is cleaned once, where it enters, not per row. - **Per-namespace caps.** `-maxDumpBytes` (default 256 MiB) and `-maxRecordsPerNetns` (default 262144) bound one namespace's dump per poll: packets past the byte cap are drained undecoded, and records past the record cap are dropped before decode. `-maxNetns` (default 4096) bounds the namespaces tracked. At the cap, tracked namespaces keep their slots, and new ones wait for a freed slot, first come, first served. Alert on `xtcp_netns_dropped_total`. Drops are counted in `xtcp_dump_dropped_total{reason}` and `xtcp_netns_dropped_total`, with one log line per truncated dump or short reconcile. `0` turns a cap off; the matching env vars are `MAX_DUMP_BYTES`, `MAX_RECORDS_PER_NETNS` and `MAX_NETNS`. The caps are startup-only: `ConfigService.Set` keeps the running values. No healthy host should reach the defaults; a host that does now loses that namespace's excess rows, where it used to collect them all ([polling and batching](docs/polling-and-batching.md#per-namespace-caps)). - **Socket-count gate, reconcile limits, per-owner quota.** Before each poll xtcp2 reads each namespace's sockstat (an fd opened in the namespace; `pread`, or io_uring batches with `-ioUring`) and, when the total is over `-maxRecordsPerPoll` (default 1,048,576), degrades only the namespaces over their fair share: first TIME_WAIT/SYN_RECV are not asked for, then one of K port shards is dumped per poll. `-maxNetnsPerOwner` (default 64) caps namespaces tracked per owner (the container runtime's cgroup), `-maxNetnsAddsPerReconcile` (256) the adds per reconcile, and `-reconcileTimeout` (1s) the namespace scan; a timed-out scan deletes nothing. New metrics `xtcp_poll_sockets`, `xtcp_poll_degraded_total{level}` and `xtcp_reconcile_incomplete_total{reason}`; `xtcp_netns_dropped_total` gains `reason`. `0` turns each off; all are startup-only. Each tracked namespace now holds four fds (was two: sockstat and `ip_local_port_range` are new), and xtcp2 warns at startup when `RLIMIT_NOFILE` can't cover them ([polling and batching](docs/polling-and-batching.md#socket-count-gate), [network namespaces](docs/network-namespaces.md#configuration)). The analysis behind an upstream all-namespace dump is in [docs/kernel](docs/kernel/inet-diag-all-netns.md). - **Vulnerability check.** `checks.govulncheck` runs govulncheck against a pinned snapshot of the Go vulnerability database, with no tags and with every flavor tag, and fails on a reachable finding not in its accepted list. One is accepted: GO-2026-6859 in `net/http`, fixed in go1.27.4, until the toolchain moves. `nix run .#govulncheck-fresh` checks against the live database and runs `go mod verify`; `nix build .#sbom-` writes a CycloneDX SBOM for any binary package. diff --git a/cmd/xtcp2/testdata/defineflags-golden.txt b/cmd/xtcp2/testdata/defineflags-golden.txt index 46b722284..3cf211374 100644 --- a/cmd/xtcp2/testdata/defineflags-golden.txt +++ b/cmd/xtcp2/testdata/defineflags-golden.txt @@ -120,8 +120,10 @@ s3Endpoint s3parquet: S3-compatible endpoint URL (e.g. http://127.0.0.1:9000 fo s3FlushInterval 0s s3parquet: staleness ceiling — force-flush the in-memory Parquet object after this long even if under the byte cap. 0 = derive as max(-frequency, 30m). s3FlushJitterPct 20 s3parquet: jitter as a percent (0-100) of -s3FlushInterval, applied to the timed flush so the fleet doesn't ceiling-flush in lockstep. 0 disables. s3FlushThresholdJitterPct 20 s3parquet: per-object downward jitter as a percent (0-100) of the byte cap; each object finalizes at threshold*(1-rand[0,pct/100]) to de-sync the size-cap upload path. 0 disables. +s3MaxBufferedBytes 67108864 s3parquet: marshaled envelope bytes the upload queue may hold. Past it the oldest envelope is dropped (reason="buffered_bytes"); the newest is always kept. 0 = no byte bound. Falls back to S3_MAX_BUFFERED_BYTES env. s3ParquetFlushBytes 0 s3parquet: soft cap on the in-memory Parquet builder's uncompressed row bytes before finalize+upload. 0 = daemon default (63 MiB). s3Prefix s3parquet: optional key prefix within the bucket. Combined with Hive-style partitioning host=…/date=…/hour=…/.parquet. +s3QueueDepth 16 s3parquet: envelopes queued for the upload worker (max 4096). Past it the oldest is dropped and counted in xtcp_s3_dropped_total{reason="queue_depth"}, so an S3 outage never blocks polling. 0 = 16. Falls back to S3_QUEUE_DEPTH env. s3Region s3parquet: S3 region. Defaults to 'us-east-1' when empty; required by AWS, ignored by most MinIO setups. s3SecretKey s3parquet: S3 secret key. Prefer S3_SECRET_KEY_FILE (path to a file, mode 0400, holding the key): a flag value shows in ps and docker inspect, warns, and is refused by production builds. Never logged. s3SkipBucketProbe false s3parquet: skip the startup BucketExists probe (a HeadBucket needing s3:ListBucket). Set true for a write-only, s3:PutObject-only credential. Falls back to S3_SKIP_BUCKET_PROBE env. diff --git a/cmd/xtcp2/testdata/printflags-golden.txt b/cmd/xtcp2/testdata/printflags-golden.txt index 360ac781d..4c5cc6bb9 100644 --- a/cmd/xtcp2/testdata/printflags-golden.txt +++ b/cmd/xtcp2/testdata/printflags-golden.txt @@ -28,6 +28,8 @@ *s3FlushThresholdJitterPct: 117 *s3UploadMaxAttempts: 118 *s3UploadBackoffCap: 119ms +*s3QueueDepth: 1191 +*s3MaxBufferedBytes: 1192 *reconcileFrequency: 120ms *reconcileBeforePoll: false *maxDumpBytes: 1201 diff --git a/cmd/xtcp2/xtcp2.go b/cmd/xtcp2/xtcp2.go index 98bf1bf63..53b0a0a92 100644 --- a/cmd/xtcp2/xtcp2.go +++ b/cmd/xtcp2/xtcp2.go @@ -116,12 +116,14 @@ const ( // docs/design-jitter-and-backoff.md. A 0 pct disables the corresponding // jitter (restoring deterministic behavior); the two 0-valued durations // mean "derive from poll_frequency" (see the s3parquet destination). - pollJitterPctCst uint = 20 - s3FlushIntervalCst = 0 * time.Second - s3FlushJitterPctCst uint = 20 - s3FlushThresholdJitterPctCst uint = 20 - s3UploadMaxAttemptsCst uint = 10 - s3UploadBackoffCapCst = 0 * time.Second + pollJitterPctCst uint = 20 + s3FlushIntervalCst = 0 * time.Second + s3FlushJitterPctCst uint = 20 + s3FlushThresholdJitterPctCst uint = 20 + s3UploadMaxAttemptsCst uint = 10 + s3UploadBackoffCapCst = 0 * time.Second + s3QueueDepthCst uint = 16 + s3MaxBufferedBytesCst uint64 = 64 * 1024 * 1024 // Namespace-reconcile cadence defaults (Method B /proc-scan discovery). // reconcileBeforePoll ties discovery to poll cadence (reconcile before each @@ -309,6 +311,8 @@ type mainFlags struct { s3FlushThresholdJitterPct *uint s3UploadMaxAttempts *uint s3UploadBackoffCap *time.Duration + s3QueueDepth *uint + s3MaxBufferedBytes *uint64 reconcileFrequency *time.Duration reconcileBeforePoll *bool @@ -550,6 +554,8 @@ func defineS3Flags(f *mainFlags) { f.s3FlushThresholdJitterPct = flag.Uint("s3FlushThresholdJitterPct", s3FlushThresholdJitterPctCst, "s3parquet: per-object downward jitter as a percent (0-100) of the byte cap; each object finalizes at threshold*(1-rand[0,pct/100]) to de-sync the size-cap upload path. 0 disables.") f.s3UploadMaxAttempts = flag.Uint("s3UploadMaxAttempts", s3UploadMaxAttemptsCst, "s3parquet: max upload attempts (original + retries) before dropping the object. Retries use full-jitter exponential backoff.") f.s3UploadBackoffCap = flag.Duration("s3UploadBackoffCap", s3UploadBackoffCapCst, "s3parquet: cap on a single upload retry's backoff window (full jitter draws in [0,window]). 0 = derive as clamp(-frequency/10, 1s, 1h).") + f.s3QueueDepth = flag.Uint("s3QueueDepth", s3QueueDepthCst, "s3parquet: envelopes queued for the upload worker (max 4096). Past it the oldest is dropped and counted in xtcp_s3_dropped_total{reason=\"queue_depth\"}, so an S3 outage never blocks polling. 0 = 16. Falls back to S3_QUEUE_DEPTH env.") + f.s3MaxBufferedBytes = flag.Uint64("s3MaxBufferedBytes", s3MaxBufferedBytesCst, "s3parquet: marshaled envelope bytes the upload queue may hold. Past it the oldest envelope is dropped (reason=\"buffered_bytes\"); the newest is always kept. 0 = no byte bound. Falls back to S3_MAX_BUFFERED_BYTES env.") } // defineCapFlags registers the three caps on what a tenant can cost the @@ -742,6 +748,8 @@ func printS3Flags(f *mainFlags) { fmt.Println("*s3FlushThresholdJitterPct:", *f.s3FlushThresholdJitterPct) fmt.Println("*s3UploadMaxAttempts:", *f.s3UploadMaxAttempts) fmt.Println("*s3UploadBackoffCap:", *f.s3UploadBackoffCap) + fmt.Println("*s3QueueDepth:", *f.s3QueueDepth) + fmt.Println("*s3MaxBufferedBytes:", *f.s3MaxBufferedBytes) fmt.Println("*reconcileFrequency:", *f.reconcileFrequency) fmt.Println("*reconcileBeforePoll:", *f.reconcileBeforePoll) } @@ -871,6 +879,8 @@ func buildConfig(f *mainFlags, des *xtcp_config.EnabledDeserializers) *xtcp_conf S3FlushThresholdJitterPct: uint32(*f.s3FlushThresholdJitterPct), S3UploadMaxAttempts: uint32(*f.s3UploadMaxAttempts), S3UploadBackoffCap: durationpb.New(*f.s3UploadBackoffCap), + S3QueueDepth: uint32(*f.s3QueueDepth), + S3MaxBufferedBytes: *f.s3MaxBufferedBytes, ReconcileFrequency: durationpb.New(*f.reconcileFrequency), ReconcileBeforePoll: *f.reconcileBeforePoll, MaxDumpBytes: *f.maxDumpBytes, @@ -1651,6 +1661,7 @@ func environmentOverrideConfig(c *xtcp_config.XtcpConfig, debugLevel uint) { envOverridePacket(c, debugLevel) envOverrideMarshalAndDest(c, debugLevel) envOverrideCaps(c, debugLevel) + envOverrideS3Queue(c, debugLevel) envOverrideKafka(c, debugLevel) envOverrideLabeling(c, debugLevel) envOverrideEnrichment(c, debugLevel) @@ -2380,6 +2391,19 @@ func checkNoPositional(args []string) error { // envOverrideCaps applies MAX_DUMP_BYTES, MAX_RECORDS_PER_NETNS and // MAX_NETNS; see defineCapFlags. +// envOverrideS3Queue applies S3_QUEUE_DEPTH and S3_MAX_BUFFERED_BYTES, the +// s3parquet upload queue's bounds. +func envOverrideS3Queue(c *xtcp_config.XtcpConfig, debugLevel uint) { + if v, ok := envUint32("S3_QUEUE_DEPTH"); ok { + c.S3QueueDepth = v + logEnv("S3_QUEUE_DEPTH", fmt.Sprintf("c.S3QueueDepth:%d", v), debugLevel) + } + if v, ok := envUint64("S3_MAX_BUFFERED_BYTES"); ok { + c.S3MaxBufferedBytes = v + logEnv("S3_MAX_BUFFERED_BYTES", fmt.Sprintf("c.S3MaxBufferedBytes:%d", v), debugLevel) + } +} + func envOverrideCaps(c *xtcp_config.XtcpConfig, debugLevel uint) { if v, ok := envUint64("MAX_DUMP_BYTES"); ok { c.MaxDumpBytes = v @@ -2436,4 +2460,6 @@ func printConfigCaps(c *xtcp_config.XtcpConfig) { fmt.Println("c.ReconcileTimeout:", c.ReconcileTimeout.AsDuration()) fmt.Println("c.MaxNetnsPerOwner:", c.MaxNetnsPerOwner) fmt.Println("c.MaxNetnsAddsPerReconcile:", c.MaxNetnsAddsPerReconcile) + fmt.Println("c.S3QueueDepth:", c.S3QueueDepth) + fmt.Println("c.S3MaxBufferedBytes:", c.S3MaxBufferedBytes) } diff --git a/cmd/xtcp2/xtcp2_flags_helpers_test.go b/cmd/xtcp2/xtcp2_flags_helpers_test.go index 183c84dc0..6c43d8bbb 100644 --- a/cmd/xtcp2/xtcp2_flags_helpers_test.go +++ b/cmd/xtcp2/xtcp2_flags_helpers_test.go @@ -89,7 +89,7 @@ const ( // mainFlags field must be a registered flag and every registered flag must // have a field. A flag added without a field, or a field that stops being // registered, is the drift this file exists to notice. - mainFlagsFieldCountCst = 139 // + treatmentConfig (embedded agent) and treatmentAttribution (journal reader) + mainFlagsFieldCountCst = 141 // + treatmentConfig (embedded agent) and treatmentAttribution (journal reader) printFlagsGoldenCst = "testdata/printflags-golden.txt" defineFlagsGoldenCst = "testdata/defineflags-golden.txt" @@ -190,6 +190,8 @@ func populatedMainFlags() *mainFlags { s3FlushThresholdJitterPct: u(117), s3UploadMaxAttempts: u(118), s3UploadBackoffCap: ms(119), + s3QueueDepth: u(1191), + s3MaxBufferedBytes: u64(1192), reconcileFrequency: ms(120), reconcileBeforePoll: b(false), diff --git a/cmd/xtcp2/xtcp2_test.go b/cmd/xtcp2/xtcp2_test.go index 43dbbcd34..9c6094ec0 100644 --- a/cmd/xtcp2/xtcp2_test.go +++ b/cmd/xtcp2/xtcp2_test.go @@ -316,7 +316,10 @@ func TestEnvOverrideMarshalAndDest(t *testing.T) { t.Setenv("S3_FLUSH_THRESHOLD_JITTER_PCT", "15") t.Setenv("S3_UPLOAD_MAX_ATTEMPTS", "7") t.Setenv("S3_UPLOAD_BACKOFF_CAP", "90s") + t.Setenv("S3_QUEUE_DEPTH", "64") + t.Setenv("S3_MAX_BUFFERED_BYTES", "1048576") envOverrideMarshalAndDest(c, 0) + envOverrideS3Queue(c, 0) if c.MarshalTo != "protoJson" || c.Dest != "null" || c.DestWriteFiles != 3 { t.Errorf("envOverrideMarshalAndDest mismatch: %+v", c) @@ -333,6 +336,9 @@ func TestEnvOverrideMarshalAndDest(t *testing.T) { if c.S3UploadBackoffCap.AsDuration() != 90*time.Second { t.Errorf("S3UploadBackoffCap = %v, want 90s", c.S3UploadBackoffCap.AsDuration()) } + if c.S3QueueDepth != 64 || c.S3MaxBufferedBytes != 1048576 { + t.Errorf("S3QueueDepth/S3MaxBufferedBytes = %d/%d, want 64/1048576", c.S3QueueDepth, c.S3MaxBufferedBytes) + } } // TestEnvOverrideS3SecretFile covers the Docker `_FILE` convention: the S3 @@ -1048,6 +1054,8 @@ func TestPrintFlags(t *testing.T) { f.s3FlushThresholdJitterPct = &n f.s3UploadMaxAttempts = &n f.s3UploadBackoffCap = &d + f.s3QueueDepth = &n + f.s3MaxBufferedBytes = &n64 f.reconcileFrequency = &d f.reconcileBeforePoll = &b f.maxDumpBytes = &n64 @@ -1278,6 +1286,8 @@ func TestBuildConfig(t *testing.T) { s3FlushThresholdJitterPct: &sftj, s3UploadMaxAttempts: &suma, s3UploadBackoffCap: &subc, + s3QueueDepth: &wf, + s3MaxBufferedBytes: &nl, reconcileFrequency: &rf, reconcileBeforePoll: &rbp, maxDumpBytes: &nl, @@ -1367,6 +1377,8 @@ func TestBuildConfig(t *testing.T) { {"S3FlushThresholdJitterPct", c.S3FlushThresholdJitterPct, uint32(15)}, {"S3UploadMaxAttempts", c.S3UploadMaxAttempts, uint32(7)}, {"S3UploadBackoffCap", c.S3UploadBackoffCap.AsDuration(), 90 * time.Second}, + {"S3QueueDepth", c.S3QueueDepth, uint32(11)}, + {"S3MaxBufferedBytes", c.S3MaxBufferedBytes, uint64(150)}, {"ReconcileFrequency", c.ReconcileFrequency.AsDuration(), 3 * time.Minute}, {"ReconcileBeforePoll", c.ReconcileBeforePoll, false}, {"IpmetaBootstrapPath", c.IpmetaBootstrapPath, ipmetaBootstrap}, @@ -1496,6 +1508,12 @@ func TestConfigValidation_bounds(t *testing.T) { {"boundary s3_upload_max_attempts=100", func(c *xtcp_config.XtcpConfig) { c.S3UploadMaxAttempts = 100 }, false}, {"negative s3_upload_max_attempts=0", func(c *xtcp_config.XtcpConfig) { c.S3UploadMaxAttempts = 0 }, true}, {"negative s3_upload_max_attempts=101", func(c *xtcp_config.XtcpConfig) { c.S3UploadMaxAttempts = 101 }, true}, + // s3_queue_depth: lte 4096, 0 = default + {"boundary s3_queue_depth=0", func(c *xtcp_config.XtcpConfig) { c.S3QueueDepth = 0 }, false}, + {"boundary s3_queue_depth=4096", func(c *xtcp_config.XtcpConfig) { c.S3QueueDepth = 4096 }, false}, + {"negative s3_queue_depth=4097", func(c *xtcp_config.XtcpConfig) { c.S3QueueDepth = 4097 }, true}, + // s3_max_buffered_bytes: any value, 0 = no byte bound + {"positive s3_max_buffered_bytes=64MiB", func(c *xtcp_config.XtcpConfig) { c.S3MaxBufferedBytes = 64 << 20 }, false}, // durations: gte 0, with 0 = derive (valid) {"positive s3_flush_interval=45m", func(c *xtcp_config.XtcpConfig) { c.S3FlushInterval = durationpb.New(45 * time.Minute) }, false}, {"corner s3_flush_interval=0 (derive)", func(c *xtcp_config.XtcpConfig) { c.S3FlushInterval = durationpb.New(0) }, false}, diff --git a/docs/design-jitter-and-backoff.md b/docs/design-jitter-and-backoff.md index 050b3199a..c05095dc5 100644 --- a/docs/design-jitter-and-backoff.md +++ b/docs/design-jitter-and-backoff.md @@ -138,8 +138,8 @@ defer timer.Stop() for { select { case <-d.closedCh: - // drain queueCh, finalize, return (unchanged) - case item := <-d.queueCh: + // drain the queue, finalize, return (unchanged) + case <-d.queue.notify: processItem(item) // finalizes when accumBytes >= effectiveThreshold (jittered cap) case <-timer.C: finalize() // staleness ceiling @@ -185,7 +185,9 @@ For a `24h` poll: `cap = 24h/10 → clamp → 1h`; 10 attempts of full-jittered ### Back-pressure & memory -While the worker is in `uploadWithRetry`, it isn't draining `queueCh` (capacity 16). During an outage the queue fills, `Send` blocks, and `flushEnvelope` in the poller blocks — back-pressuring collection. This is *intended* bounded behavior: memory is capped at the queue + one in-flight Parquet object. To keep an outage from stalling collection past one cycle, bound total retry time to roughly one `PollFrequency` — with `base=1s`, `cap=PollFrequency/10`, `attempts=10`, the worst-case sum of windows is ≈ `PollFrequency` (full jitter averages half that). Outages longer than that fall back to today's documented drop-and-continue. Persistent on-disk spooling is explicitly [future work](#open-questions--future-work). +While the worker is in `uploadWithRetry`, it isn't draining its queue. `Send` never blocks: the queue holds at most `-s3QueueDepth` envelopes (default 16, max 4096) and `-s3MaxBufferedBytes` of them (default 64 MiB, `0` = no byte bound), and past either bound it drops the oldest, counted in `xtcp_s3_dropped_total{reason}` and `xtcp_s3_dropped_bytes_total{reason}`. The newest envelope is always kept. Polling keeps its cadence through an outage; the cost is the oldest data. (Before this bound, a full queue blocked `Send`, and with it `flushEnvelope` mid-poll, stalling collection.) + +Memory budget: ≈ `-s3MaxBufferedBytes` (queued envelopes) + the flush threshold (the object being built, ≈ 63 MiB by default) + one object during upload. With the defaults that is about 2–3× the threshold, ≈ 190 MiB; size `GOMEMLIMIT` and the container limit above it. `xtcp_s3_buffered_bytes{stage}` shows the first two. To keep an outage from stalling collection past one cycle, bound total retry time to roughly one `PollFrequency` — with `base=1s`, `cap=PollFrequency/10`, `attempts=10`, the worst-case sum of windows is ≈ `PollFrequency` (full jitter averages half that). Outages longer than that fall back to today's documented drop-and-continue. Persistent on-disk spooling is explicitly [future work](#open-questions--future-work). ## New configuration surface diff --git a/docs/metrics.md b/docs/metrics.md index fd04c3961..4bfb486d9 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -200,7 +200,7 @@ All are in `xtcp_counts`. | `destS3Parquet` | `uploadRetry` | error | an upload attempt failed and will be retried | | `destS3Parquet` | `uploadErrorClass` | `tls`, `timeout`, `canceled`, `dns`, `refused`, `other` | why an attempt failed | | `destS3Parquet` | `uploadRowsDropped`, `uploadBytesDropped` | error | rows and bytes lost with a failed upload | -| `destS3Parquet` | `queueFull` | error | the upload queue was full | +| `destS3Parquet` | `queueFull` | error | an envelope was dropped from the upload queue, past `-s3QueueDepth` or `-s3MaxBufferedBytes` (`Send` never blocks; see below) | | `destS3Parquet` | `unmarshal`, `write`, `writerClose` | error | building the Parquet file failed | | `destS3Parquet` | `setS3Flush` | count | the flush settings were changed over gRPC | | `destKafka` | `Produce` | count / error | produces and produce failures; also in histograms, as how long a produce took | @@ -217,6 +217,14 @@ All are in `xtcp_counts`. | `destNull`, `destStdout` | `start` | count | sends | | marshaller name, such as `protobufListMarshal` | `marshal` | error | the envelope couldn't be marshalled | +The s3parquet upload queue has its own series. Past either bound it drops the oldest envelope, so an S3 outage costs data rather than stalling the poller: + +| metric | labels | meaning | +|---|---|---| +| `xtcp_s3_dropped_total` | `reason="queue_depth"\|"buffered_bytes"` | envelopes dropped past `-s3QueueDepth` or `-s3MaxBufferedBytes` | +| `xtcp_s3_dropped_bytes_total` | `reason` | their marshaled bytes | +| `xtcp_s3_buffered_bytes` | `stage="queue"\|"builder"` | bytes held: envelopes awaiting the worker, and approximate row bytes in the Parquet object being built | + ### Enrichment `enrich*.go`. Updates per refresh or per event, not per record. diff --git a/docs/security-hardening/focused-container-hardening.md b/docs/security-hardening/focused-container-hardening.md index af48ea8ab..66643a4dd 100644 --- a/docs/security-hardening/focused-container-hardening.md +++ b/docs/security-hardening/focused-container-hardening.md @@ -83,7 +83,7 @@ What to tighten (each is in the focused set below): - **Secrets:** `S3_SECRET_KEY_FILE` / `S3_ACCESS_KEY_FILE` from a mounted secret, never the flag. Make the flag a startup error in the production flavor. - **Transport:** refuse `http://` endpoints unless `-s3AllowInsecure`; TLS ≥ 1.2; optional CA bundle path for private endpoints. - **Bounds:** cap the pending-upload queue depth and total buffered bytes (`queueCh` exists; make the depth explicit and metered), cap rows per object, and cap every tenant-controlled string column (length, valid UTF-8, no C0/C1 control characters) in `rowFromProto`. -- **Memory:** `parquet-go` holds column buffers plus the serialized page; budget ≈ 2–3× the flush threshold and size `GOMEMLIMIT` accordingly, or lower the threshold for small hosts. +- **Memory:** `parquet-go` holds column buffers plus the serialized page; budget ≈ `-s3MaxBufferedBytes` (64 MiB) + the flush threshold + one object in upload, about 2–3× the threshold (≈ 190 MiB with the defaults), and size `GOMEMLIMIT` accordingly, or lower the threshold and `-s3MaxBufferedBytes` for small hosts. `xtcp_s3_buffered_bytes{stage}` shows what is held. ## Focused hardening set diff --git a/docs/security-hardening/focused-implementation-plan.md b/docs/security-hardening/focused-implementation-plan.md index 8c614809c..7bcad518e 100644 --- a/docs/security-hardening/focused-implementation-plan.md +++ b/docs/security-hardening/focused-implementation-plan.md @@ -208,11 +208,22 @@ Non-blocking shortfalls found while implementing F1; none blocks R1. ### FP-6 — S3 queue bound and string policy (C3, C4) -- [ ] `pkg/xtcp/destinations_s3parquet.go`: explicit `-s3QueueDepth` and `-s3MaxBufferedBytes`; drop-oldest with `xtcp2_s3_dropped_total{reason}`; `xtcp2_s3_buffered_bytes` gauge; document the memory budget (≈ 2–3× flush threshold) next to `GOMEMLIMIT` guidance. -- [ ] New `pkg/xtcp/strpolicy.go`: `clean(s, max)` — truncate at a rune boundary, replace invalid UTF-8 with U+FFFD, replace C0/C1 controls. Applied once where the enrichment values enter the record (container name/image/label, LLDP chassis names, congestion-control name), not per row in `rowFromProto`, so the cost is per container or per interface, not per socket. -- [ ] Log call sites that print these strings use the same function. +- [x] `pkg/xtcp/destinations_s3parquet.go`: explicit `-s3QueueDepth` and `-s3MaxBufferedBytes`; drop-oldest with `xtcp2_s3_dropped_total{reason}`; `xtcp2_s3_buffered_bytes` gauge; document the memory budget (≈ 2–3× flush threshold) next to `GOMEMLIMIT` guidance. *Done as:* `envelopeQueue` (`pkg/xtcp/s3queue.go`), a mutex ring with a byte count; `Send` never blocks (it used to block the poller once 16 envelopes were queued). `XtcpConfig` fields 116 (`s3_queue_depth`, default 16, max 4096) and 117 (`s3_max_buffered_bytes`, flag default 64 MiB, `0` = depth bound only); env `S3_QUEUE_DEPTH`, `S3_MAX_BUFFERED_BYTES`; kept across `ConfigService.Set` by `preserveCaps`. Metrics in the package's `xtcp` subsystem: `xtcp_s3_dropped_total{reason="queue_depth"|"buffered_bytes"}`, `xtcp_s3_dropped_bytes_total{reason}`, `xtcp_s3_buffered_bytes{stage="queue"|"builder"}`; `xtcp_counts{function="destS3Parquet",variable="queueFull"}` now counts drops. No row counts: a dropped envelope is never unmarshaled. Budget in `design-jitter-and-backoff.md` and `focused-container-hardening.md`. +- [x] New `pkg/xtcp/strpolicy.go`: `clean(s, max)` — truncate at a rune boundary, replace invalid UTF-8 with U+FFFD, replace C0/C1 controls. Applied once where the enrichment values enter the record (container name/image/label, LLDP chassis names, congestion-control name), not per row in `rowFromProto`, so the cost is per container or per interface, not per socket. *Done as:* `pkg/strpolicy.Clean` (its own package so dockermeta, localnet, ipasn and xtcpnl can import it), which also replaces DEL. Limits: 256 bytes, 16 for interface and congestion names. Applied in `dockermeta.inspect` (name, image), `buildUplinkStamp` (NIC strings, LLDP fields, uplink ifnames), `localnet.BuildSnapshot` (the tenant's ifnames), the ipasn load (network owner), reconcile (netns name) and `DeserializeCongInfoXTCP`. The congestion name is per socket, so its decoder folds the printable check into its NUL scan and calls `Clean` only otherwise. Container labels are not collected. +- [x] Log call sites that print these strings use the same function. *Done as:* every value is cleaned where it enters, so the logs that print them (netns names, containers) print cleaned values. -**Tests:** FC `s3_queue_full`, `long_container_name`, `invalid_utf8`; benchmark guard on the enrichment path. +**Tests:** FC `s3_queue_full`, `long_container_name`, `invalid_utf8`; benchmark guard on the enrichment path. *Done as:* `TestClean`, `FuzzClean`, `TestEnvelopeQueue` (+ concurrent), `TestS3ParquetDest_queueBounds`, the reworked `corner_queueFull` and one source test per call site, plus flag, env, validation and `preserveCaps` rows. + +| benchmark | result | verdict | +|---|---|---| +| `BenchmarkClean` ascii16 / ascii256 / dirty256 | 12 ns / 149 ns / 1.05 µs; 0 / 0 / 1 allocs | ASCII fast path allocation-free ✓ | +| `BenchmarkDeserializeCongInfoXTCP` vs main, 10 interleaved runs | 21.0 → 23.3 ns median (+2 ns), allocs unchanged (1) | accepted: a whole record decodes in µs. Calling `Clean` after `IndexByte` cost +7–10 ns; folding the check into one scan brought it to +2 | +| `BenchmarkS3Send` queue vs the old channel send | 52 ns vs 118 ns | faster ✓ | +| `microvm-x86_64-lifecycle-s3parquet` Check 14c: Garage stopped for 160 s (4× wait scale), `-s3QueueDepth 2` | polls kept running (343), 22 envelopes dropped, RSS 50.8 → 55.8 MB, a new object after restart | pass ✓ | + +**Status:** open as #308. + +**FP-6 TODOs:** `max_len` in protovalidate for the operator strings (`label`, `tag`, `location`, `hostname`), which are trusted and not cleaned; spill the S3 queue to disk during long outages (out of scope: memory bound only). ### FP-7 — fuzzers (D4) diff --git a/gen/cpp/xtcp_config/v1/xtcp_config.pb.cc b/gen/cpp/xtcp_config/v1/xtcp_config.pb.cc index a6c10608b..90a109047 100644 --- a/gen/cpp/xtcp_config/v1/xtcp_config.pb.cc +++ b/gen/cpp/xtcp_config/v1/xtcp_config.pb.cc @@ -2550,7 +2550,7 @@ constexpr XtcpConfig::ParseTableT_ XtcpConfig::InternalGenerateParseTable_(const offsetof(ParseTableT_, field_lookup_table), 3758096895, // skipmap offsetof(ParseTableT_, field_entries), - 92, // num_field_entries + 94, // num_field_entries 16, // num_aux_entries offsetof(ParseTableT_, aux_entries), class_data, @@ -2664,16 +2664,16 @@ constexpr XtcpConfig::ParseTableT_ XtcpConfig::InternalGenerateParseTable_(const 64527, 31, 61695, 37, 4095, 41, - 61472, 45, - 33791, 56, - 16383, 61, - 63736, 63, - 65475, 69, - 65535, 73, - 58360, 73, - 49039, 79, - 255, 83, - 65531, 91, + 49184, 45, + 33791, 58, + 16383, 63, + 63736, 65, + 65475, 71, + 65535, 75, + 58360, 75, + 49039, 81, + 255, 85, + 65531, 93, 65535, 65535 }}, {{ // uint64 nl_timeout_milliseconds = 10 [json_name = "nlTimeoutMilliseconds", (.buf.validate.field) = { @@ -2788,6 +2788,10 @@ constexpr XtcpConfig::ParseTableT_ XtcpConfig::InternalGenerateParseTable_(const {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.s3_upload_max_attempts_), _Internal::kHasBitsOffset + 77, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, // .google.protobuf.Duration s3_upload_backoff_cap = 115 [json_name = "s3UploadBackoffCap", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.s3_upload_backoff_cap_), _Internal::kHasBitsOffset + 54, 7, (0 | ::_fl::kFcOptional | ::_fl::kMessage | ::_fl::kTvTable)}, + // uint32 s3_queue_depth = 116 [json_name = "s3QueueDepth", (.buf.validate.field) = { + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.s3_queue_depth_), _Internal::kHasBitsOffset + 78, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, + // uint64 s3_max_buffered_bytes = 117 [json_name = "s3MaxBufferedBytes", (.buf.validate.field) = { + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.s3_max_buffered_bytes_), _Internal::kHasBitsOffset + 79, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt64)}, // string hostname = 130 [json_name = "hostname", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.hostname_), _Internal::kHasBitsOffset + 37, 0, (0 | ::_fl::kFcOptional | ::_fl::kUtf8String | ::_fl::kRepAString)}, // string location = 131 [json_name = "location", (.buf.validate.field) = { @@ -2799,9 +2803,9 @@ constexpr XtcpConfig::ParseTableT_ XtcpConfig::InternalGenerateParseTable_(const // string daemon_version = 134 [json_name = "daemonVersion", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.daemon_version_), _Internal::kHasBitsOffset + 41, 0, (0 | ::_fl::kFcOptional | ::_fl::kUtf8String | ::_fl::kRepAString)}, // uint32 ipv4_ttl = 150 [json_name = "ipv4Ttl", (.buf.validate.field) = { - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.ipv4_ttl_), _Internal::kHasBitsOffset + 78, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.ipv4_ttl_), _Internal::kHasBitsOffset + 80, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, // uint32 ipv6_hop_limit = 151 [json_name = "ipv6HopLimit", (.buf.validate.field) = { - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.ipv6_hop_limit_), _Internal::kHasBitsOffset + 79, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.ipv6_hop_limit_), _Internal::kHasBitsOffset + 81, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, // .xtcp_config.v1.ListenerAuth listener_auth = 152 [json_name = "listenerAuth", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.listener_auth_), _Internal::kHasBitsOffset + 55, 8, (0 | ::_fl::kFcOptional | ::_fl::kMessage | ::_fl::kTvTable)}, // .xtcp_config.v1.ListenerEndpoint prometheus_listener = 153 [json_name = "prometheusListener", (.buf.validate.field) = { @@ -2809,7 +2813,7 @@ constexpr XtcpConfig::ParseTableT_ XtcpConfig::InternalGenerateParseTable_(const // .xtcp_config.v1.ListenerAuth admin_listener_auth = 154 [json_name = "adminListenerAuth", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.admin_listener_auth_), _Internal::kHasBitsOffset + 57, 10, (0 | ::_fl::kFcOptional | ::_fl::kMessage | ::_fl::kTvTable)}, // uint32 grpc_port = 160 [json_name = "grpcPort", (.buf.validate.field) = { - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.grpc_port_), _Internal::kHasBitsOffset + 80, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.grpc_port_), _Internal::kHasBitsOffset + 82, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, // .xtcp_config.v1.ListenerEndpoint grpc_listener = 161 [json_name = "grpcListener", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.grpc_listener_), _Internal::kHasBitsOffset + 58, 11, (0 | ::_fl::kFcOptional | ::_fl::kMessage | ::_fl::kTvTable)}, // repeated .xtcp_config.v1.Listener listeners = 162 [json_name = "listeners", (.buf.validate.field) = { @@ -2819,31 +2823,31 @@ constexpr XtcpConfig::ParseTableT_ XtcpConfig::InternalGenerateParseTable_(const // string pyroscope_app_name = 171 [json_name = "pyroscopeAppName", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.pyroscope_app_name_), _Internal::kHasBitsOffset + 43, 0, (0 | ::_fl::kFcOptional | ::_fl::kUtf8String | ::_fl::kRepAString)}, // uint32 pyroscope_sample_hz = 172 [json_name = "pyroscopeSampleHz", (.buf.validate.field) = { - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.pyroscope_sample_hz_), _Internal::kHasBitsOffset + 81, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.pyroscope_sample_hz_), _Internal::kHasBitsOffset + 83, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, // uint32 pyroscope_upload_interval_sec = 173 [json_name = "pyroscopeUploadIntervalSec", (.buf.validate.field) = { - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.pyroscope_upload_interval_sec_), _Internal::kHasBitsOffset + 82, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.pyroscope_upload_interval_sec_), _Internal::kHasBitsOffset + 84, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, // bool resolve_container_id = 200 [json_name = "resolveContainerId", (.buf.validate.field) = { - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.resolve_container_id_), _Internal::kHasBitsOffset + 83, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.resolve_container_id_), _Internal::kHasBitsOffset + 85, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, // bool enrich_container_enable = 201 [json_name = "enrichContainerEnable"]; - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_container_enable_), _Internal::kHasBitsOffset + 84, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_container_enable_), _Internal::kHasBitsOffset + 86, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, // string docker_socket_path = 202 [json_name = "dockerSocketPath", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.docker_socket_path_), _Internal::kHasBitsOffset + 44, 0, (0 | ::_fl::kFcOptional | ::_fl::kUtf8String | ::_fl::kRepAString)}, // bool enrich_lldp_enable = 210 [json_name = "enrichLldpEnable"]; - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_lldp_enable_), _Internal::kHasBitsOffset + 85, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_lldp_enable_), _Internal::kHasBitsOffset + 87, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, // string lldpd_socket_path = 211 [json_name = "lldpdSocketPath", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.lldpd_socket_path_), _Internal::kHasBitsOffset + 45, 0, (0 | ::_fl::kFcOptional | ::_fl::kUtf8String | ::_fl::kRepAString)}, // string lldpd_version_hint = 212 [json_name = "lldpdVersionHint", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.lldpd_version_hint_), _Internal::kHasBitsOffset + 46, 0, (0 | ::_fl::kFcOptional | ::_fl::kUtf8String | ::_fl::kRepAString)}, // bool enrich_nic_enable = 220 [json_name = "enrichNicEnable"]; - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_nic_enable_), _Internal::kHasBitsOffset + 86, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_nic_enable_), _Internal::kHasBitsOffset + 88, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, // uint32 uplink_count = 221 [json_name = "uplinkCount", (.buf.validate.field) = { - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.uplink_count_), _Internal::kHasBitsOffset + 87, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.uplink_count_), _Internal::kHasBitsOffset + 89, 0, (0 | ::_fl::kFcOptional | ::_fl::kUInt32)}, // repeated string uplink_interfaces = 222 [json_name = "uplinkInterfaces", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.uplink_interfaces_), _Internal::kHasBitsOffset + 23, 0, (0 | ::_fl::kFcRepeated | ::_fl::kUtf8String | ::_fl::kRepSString)}, // bool populate_nsid = 230 [json_name = "populateNsid"]; - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.populate_nsid_), _Internal::kHasBitsOffset + 88, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.populate_nsid_), _Internal::kHasBitsOffset + 90, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, // bool enrich_asn_enable = 240 [json_name = "enrichAsnEnable"]; - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_asn_enable_), _Internal::kHasBitsOffset + 89, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_asn_enable_), _Internal::kHasBitsOffset + 91, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, // string asn_db_path = 241 [json_name = "asnDbPath", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.asn_db_path_), _Internal::kHasBitsOffset + 47, 0, (0 | ::_fl::kFcOptional | ::_fl::kUtf8String | ::_fl::kRepAString)}, // .google.protobuf.Duration asn_refresh_interval = 242 [json_name = "asnRefreshInterval"]; @@ -2853,11 +2857,11 @@ constexpr XtcpConfig::ParseTableT_ XtcpConfig::InternalGenerateParseTable_(const // string ipmeta_cache_path = 244 [json_name = "ipmetaCachePath", (.buf.validate.field) = { {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.ipmeta_cache_path_), _Internal::kHasBitsOffset + 49, 0, (0 | ::_fl::kFcOptional | ::_fl::kUtf8String | ::_fl::kRepAString)}, // bool enrich_locality_enable = 245 [json_name = "enrichLocalityEnable"]; - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_locality_enable_), _Internal::kHasBitsOffset + 90, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.enrich_locality_enable_), _Internal::kHasBitsOffset + 92, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, // .google.protobuf.Duration locality_refresh_interval = 246 [json_name = "localityRefreshInterval"]; {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.locality_refresh_interval_), _Internal::kHasBitsOffset + 60, 14, (0 | ::_fl::kFcOptional | ::_fl::kMessage | ::_fl::kTvTable)}, // bool locality_events_enable = 247 [json_name = "localityEventsEnable"]; - {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.locality_events_enable_), _Internal::kHasBitsOffset + 91, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, + {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.locality_events_enable_), _Internal::kHasBitsOffset + 93, 0, (0 | ::_fl::kFcOptional | ::_fl::kBool)}, // .xtcp_config.v1.LinkMonitorConfig link_monitor = 250 [json_name = "linkMonitor"]; {PROTOBUF_FIELD_OFFSET(XtcpConfig, _impl_.link_monitor_), _Internal::kHasBitsOffset + 61, 15, (0 | ::_fl::kFcOptional | ::_fl::kMessage | ::_fl::kTvTable)}, }}, @@ -2944,7 +2948,7 @@ constexpr XtcpConfig::ParseTableT_ XtcpConfig::InternalGenerateParseTable_(const #endif }}, {{ - "\31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\14\0\0\0\4\12\13\17\0\0\5\20\0\21\13\11\11\11\15\15\0\0\12\0\0\0\0\0\0\10\10\5\3\16\0\0\0\0\0\0\0\0\15\22\0\0\0\0\22\0\21\22\0\0\21\0\0\13\0\25\21\0\0\0\0\0\0\0" + "\31\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\14\0\0\0\4\12\13\17\0\0\5\20\0\21\13\11\11\11\15\15\0\0\12\0\0\0\0\0\0\0\0\10\10\5\3\16\0\0\0\0\0\0\0\0\15\22\0\0\0\0\22\0\21\22\0\0\21\0\0\13\0\25\21\0\0\0\0\0" "xtcp_config.v1.XtcpConfig" "capture_path" "dest" @@ -3125,6 +3129,8 @@ inline constexpr XtcpConfig::Impl_::Impl_( s3_flush_jitter_pct_{0u}, s3_flush_threshold_jitter_pct_{0u}, s3_upload_max_attempts_{0u}, + s3_queue_depth_{0u}, + s3_max_buffered_bytes_{::uint64_t{0u}}, ipv4_ttl_{0u}, ipv6_hop_limit_{0u}, grpc_port_{0u}, @@ -4295,7 +4301,7 @@ const ::uint32_t 22, 0x081, // bitmap PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_._has_bits_), - 95, // hasbit index offset + 97, // hasbit index offset PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.nl_timeout_milliseconds_), PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.poll_frequency_), PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.poll_timeout_), @@ -4352,6 +4358,8 @@ const ::uint32_t PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.s3_flush_threshold_jitter_pct_), PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.s3_upload_max_attempts_), PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.s3_upload_backoff_cap_), + PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.s3_queue_depth_), + PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.s3_max_buffered_bytes_), PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.hostname_), PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.location_), PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::XtcpConfig, _impl_.label_), @@ -4444,41 +4452,43 @@ const ::uint32_t 76, 77, 54, + 78, + 79, 37, 38, 39, 40, 41, - 78, - 79, + 80, + 81, 55, 56, 57, - 80, + 82, 58, 22, 42, 43, - 81, - 82, 83, 84, - 44, 85, - 45, - 46, 86, + 44, 87, - 23, + 45, + 46, 88, 89, + 23, + 90, + 91, 47, 59, 48, 49, - 90, + 92, 60, - 91, + 93, 61, 0x081, // bitmap PROTOBUF_FIELD_OFFSET(::xtcp_config::v1::EnabledDeserializers_EnabledEntry_DoNotUse, _impl_._has_bits_), @@ -4515,8 +4525,8 @@ static const ::_pbi::MigrationSchema {104, sizeof(::xtcp_config::v1::Listener)}, {117, sizeof(::xtcp_config::v1::LinkMonitorConfig)}, {176, sizeof(::xtcp_config::v1::XtcpConfig)}, - {363, sizeof(::xtcp_config::v1::EnabledDeserializers_EnabledEntry_DoNotUse)}, - {370, sizeof(::xtcp_config::v1::EnabledDeserializers)}, + {367, sizeof(::xtcp_config::v1::EnabledDeserializers_EnabledEntry_DoNotUse)}, + {374, sizeof(::xtcp_config::v1::EnabledDeserializers)}, }; static const ::_pbi::MessageGlobalsBase* PROTOBUF_NONNULL const file_message_globals[] = { @@ -4675,7 +4685,7 @@ const char descriptor_table_protodef_xtcp_5fconfig_2fv1_2fxtcp_5fconfig_2eproto[ "2\031.google.protobuf.DurationB\016\272H\013\252\001\010\"\004\010\200\243" "\0052\000R\rlearnDeadlineB\020\n\016_error_classesB\025\n\023" "_metrics_categoriesB\032\n\030_metrics_interfac" - "e_pathsB\023\n\021_metrics_all_path\"\327*\n\nXtcpCon" + "e_pathsB\023\n\021_metrics_all_path\"\305+\n\nXtcpCon" "fig\022F\n\027nl_timeout_milliseconds\030\n \001(\004B\016\272H" "\0132\006\030\240\215\006(\000\310\001\001R\025nlTimeoutMilliseconds\022S\n\016p" "oll_frequency\030\013 \001(\0132\031.google.protobuf.Du" @@ -4758,106 +4768,109 @@ const char descriptor_table_protodef_xtcp_5fconfig_2fv1_2fxtcp_5fconfig_2eproto[ "mpts\030r \001(\rB\014\272H\t*\004\030d(\001\310\001\000R\023s3UploadMaxAtt" "empts\022Y\n\025s3_upload_backoff_cap\030s \001(\0132\031.g" "oogle.protobuf.DurationB\013\272H\010\252\001\0022\000\310\001\000R\022s3" - "UploadBackoffCap\022(\n\010hostname\030\202\001 \001(\tB\013\272H\010" - "r\003\030\375\001\310\001\000R\010hostname\022(\n\010location\030\203\001 \001(\tB\013\272" - "H\010r\003\030\375\001\310\001\000R\010location\022!\n\005label\030\204\001 \001(\tB\n\272H" - "\007r\002\030(\310\001\000R\005label\022\035\n\003tag\030\205\001 \001(\tB\n\272H\007r\002\030(\310\001" - "\000R\003tag\0223\n\016daemon_version\030\206\001 \001(\tB\013\272H\010r\003\030\375" - "\001\310\001\000R\rdaemonVersion\022\'\n\010ipv4_ttl\030\226\001 \001(\rB\013" - "\272H\010*\003\030\377\001\310\001\000R\007ipv4Ttl\0222\n\016ipv6_hop_limit\030\227" - "\001 \001(\rB\013\272H\010*\003\030\377\001\310\001\000R\014ipv6HopLimit\022J\n\rlist" - "ener_auth\030\230\001 \001(\0132\034.xtcp_config.v1.Listen" - "erAuthB\006\272H\003\310\001\000R\014listenerAuth\022Z\n\023promethe" - "us_listener\030\231\001 \001(\0132 .xtcp_config.v1.List" - "enerEndpointB\006\272H\003\310\001\000R\022prometheusListener" - "\022U\n\023admin_listener_auth\030\232\001 \001(\0132\034.xtcp_co" - "nfig.v1.ListenerAuthB\006\272H\003\310\001\000R\021adminListe" - "nerAuth\022\'\n\tgrpc_port\030\240\001 \001(\rB\t\272H\006*\004\030\377\377\003R\010" - "grpcPort\022N\n\rgrpc_listener\030\241\001 \001(\0132 .xtcp_" - "config.v1.ListenerEndpointB\006\272H\003\310\001\000R\014grpc" - "Listener\022A\n\tlisteners\030\242\001 \003(\0132\030.xtcp_conf" - "ig.v1.ListenerB\010\272H\005\222\001\002\020\020R\tlisteners\022,\n\rp" - "yroscope_url\030\252\001 \001(\tB\006\272H\003\310\001\000R\014pyroscopeUr" - "l\0225\n\022pyroscope_app_name\030\253\001 \001(\tB\006\272H\003\310\001\000R\020" - "pyroscopeAppName\0227\n\023pyroscope_sample_hz\030" - "\254\001 \001(\rB\006\272H\003\310\001\000R\021pyroscopeSampleHz\022J\n\035pyr" - "oscope_upload_interval_sec\030\255\001 \001(\rB\006\272H\003\310\001" - "\000R\032pyroscopeUploadIntervalSec\0229\n\024resolve" - "_container_id\030\310\001 \001(\010B\006\272H\003\310\001\000R\022resolveCon" - "tainerId\0227\n\027enrich_container_enable\030\311\001 \001" - "(\010R\025enrichContainerEnable\0227\n\022docker_sock" - "et_path\030\312\001 \001(\tB\010\272H\005r\003\030\377\001R\020dockerSocketPa" - "th\022-\n\022enrich_lldp_enable\030\322\001 \001(\010R\020enrichL" - "ldpEnable\0225\n\021lldpd_socket_path\030\323\001 \001(\tB\010\272" - "H\005r\003\030\377\001R\017lldpdSocketPath\0226\n\022lldpd_versio" - "n_hint\030\324\001 \001(\tB\007\272H\004r\002\030\020R\020lldpdVersionHint" - "\022+\n\021enrich_nic_enable\030\334\001 \001(\010R\017enrichNicE" - "nable\022+\n\014uplink_count\030\335\001 \001(\rB\007\272H\004*\002\030\002R\013u" - "plinkCount\0226\n\021uplink_interfaces\030\336\001 \003(\tB\010" - "\272H\005\222\001\002\020\002R\020uplinkInterfaces\022$\n\rpopulate_n" - "sid\030\346\001 \001(\010R\014populateNsid\022+\n\021enrich_asn_e" - "nable\030\360\001 \001(\010R\017enrichAsnEnable\022)\n\013asn_db_" - "path\030\361\001 \001(\tB\010\272H\005r\003\030\377\001R\tasnDbPath\022L\n\024asn_" - "refresh_interval\030\362\001 \001(\0132\031.google.protobu" - "f.DurationR\022asnRefreshInterval\022=\n\025ipmeta" - "_bootstrap_path\030\363\001 \001(\tB\010\272H\005r\003\030\377\001R\023ipmeta" - "BootstrapPath\0225\n\021ipmeta_cache_path\030\364\001 \001(" - "\tB\010\272H\005r\003\030\377\001R\017ipmetaCachePath\0225\n\026enrich_l" - "ocality_enable\030\365\001 \001(\010R\024enrichLocalityEna" - "ble\022V\n\031locality_refresh_interval\030\366\001 \001(\0132" - "\031.google.protobuf.DurationR\027localityRefr" - "eshInterval\0225\n\026locality_events_enable\030\367\001" - " \001(\010R\024localityEventsEnable\022E\n\014link_monit" - "or\030\372\001 \001(\0132!.xtcp_config.v1.LinkMonitorCo" - "nfigR\013linkMonitor:s\272Hp\032n\n\017XtcpConfig.pol" - "l\0222Poll timeout must be less than poll p" - "oll_frequency\032\'this.poll_frequency > thi" - "s.poll_timeout\"\237\001\n\024EnabledDeserializers\022" - "K\n\007enabled\030\001 \003(\01321.xtcp_config.v1.Enable" - "dDeserializers.EnabledEntryR\007enabled\032:\n\014" - "EnabledEntry\022\020\n\003key\030\001 \001(\tR\003key\022\024\n\005value\030" - "\002 \001(\010R\005value:\0028\001*h\n\017ListenerNetwork\022 \n\034L" - "ISTENER_NETWORK_UNSPECIFIED\020\000\022\030\n\024LISTENE" - "R_NETWORK_TCP\020\001\022\031\n\025LISTENER_NETWORK_UNIX" - "\020\002*\241\001\n\020ListenerAuthMode\022\"\n\036LISTENER_AUTH" - "_MODE_UNSPECIFIED\020\000\022\037\n\033LISTENER_AUTH_MOD" - "E_DISABLED\020\001\022 \n\034LISTENER_AUTH_MODE_RAW_T" - "OKEN\020\002\022&\n\"LISTENER_AUTH_MODE_HMAC_UTC_MI" - "NUTE\020\003*\367\001\n\rEndpointClass\022\036\n\032ENDPOINT_CLA" - "SS_UNSPECIFIED\020\000\022\031\n\025ENDPOINT_CLASS_HEALT" - "H\020\001\022\032\n\026ENDPOINT_CLASS_METRICS\020\002\022!\n\035ENDPO" - "INT_CLASS_METRICS_DETAIL\020\003\022\025\n\021ENDPOINT_C" - "LASS_IP\020\004\022\030\n\024ENDPOINT_CLASS_DEBUG\020\005\022\034\n\030E" - "NDPOINT_CLASS_GRPC_READ\020\006\022\035\n\031ENDPOINT_CL" - "ASS_GRPC_ADMIN\020\0072\245\007\n\rConfigService\022a\n\003Ge" - "t\022\032.xtcp_config.v1.GetRequest\032\033.xtcp_con" - "fig.v1.GetResponse\"!\310\363\030\001\202\323\344\223\002\027\032\022/ConfigS" - "ervice/Get:\001*\022a\n\003Set\022\032.xtcp_config.v1.Se" - "tRequest\032\033.xtcp_config.v1.SetResponse\"!\310" - "\363\030\002\202\323\344\223\002\027\032\022/ConfigService/Set:\001*\022\225\001\n\020Set" - "PollFrequency\022\'.xtcp_config.v1.SetPollFr" - "equencyRequest\032(.xtcp_config.v1.SetPollF" - "requencyResponse\".\310\363\030\002\202\323\344\223\002$\032\037/ConfigSer" - "vice/SetPollFrequency:\001*\022\201\001\n\013TriggerPoll" - "\022\".xtcp_config.v1.TriggerPollRequest\032#.x" - "tcp_config.v1.TriggerPollResponse\")\310\363\030\002\202" - "\323\344\223\002\037\032\032/ConfigService/TriggerPoll:\001*\022\225\001\n" - "\020TriggerPollBurst\022\'.xtcp_config.v1.Trigg" - "erPollBurstRequest\032(.xtcp_config.v1.Trig" - "gerPollBurstResponse\".\310\363\030\002\202\323\344\223\002$\032\037/Confi" - "gService/TriggerPollBurst:\001*\022\201\001\n\013SetS3Up" - "load\022\".xtcp_config.v1.SetS3UploadRequest" - "\032#.xtcp_config.v1.SetS3UploadResponse\")\310" - "\363\030\002\202\323\344\223\002\037\032\032/ConfigService/SetS3Upload:\001*" - "\022\225\001\n\020SetEnvelopeFlush\022\'.xtcp_config.v1.S" - "etEnvelopeFlushRequest\032(.xtcp_config.v1." - "SetEnvelopeFlushResponse\".\310\363\030\002\202\323\344\223\002$\032\037/C" - "onfigService/SetEnvelopeFlush:\001*B\257\001\n\022com" - ".xtcp_config.v1B\017XtcpConfigProtoP\001Z3gith" - "ub.com/randomizedcoder/xtcp2/gen/go/xtcp" - "_config\242\002\003XXX\252\002\rXtcpConfig.V1\312\002\rXtcpConf" - "ig\\V1\342\002\031XtcpConfig\\V1\\GPBMetadata\352\002\016Xtcp" - "Config::V1b\006proto3" + "UploadBackoffCap\0221\n\016s3_queue_depth\030t \001(\r" + "B\013\272H\010*\003\030\200 \310\001\000R\014s3QueueDepth\0229\n\025s3_max_bu" + "ffered_bytes\030u \001(\004B\006\272H\003\310\001\000R\022s3MaxBuffere" + "dBytes\022(\n\010hostname\030\202\001 \001(\tB\013\272H\010r\003\030\375\001\310\001\000R\010" + "hostname\022(\n\010location\030\203\001 \001(\tB\013\272H\010r\003\030\375\001\310\001\000" + "R\010location\022!\n\005label\030\204\001 \001(\tB\n\272H\007r\002\030(\310\001\000R\005" + "label\022\035\n\003tag\030\205\001 \001(\tB\n\272H\007r\002\030(\310\001\000R\003tag\0223\n\016" + "daemon_version\030\206\001 \001(\tB\013\272H\010r\003\030\375\001\310\001\000R\rdaem" + "onVersion\022\'\n\010ipv4_ttl\030\226\001 \001(\rB\013\272H\010*\003\030\377\001\310\001" + "\000R\007ipv4Ttl\0222\n\016ipv6_hop_limit\030\227\001 \001(\rB\013\272H\010" + "*\003\030\377\001\310\001\000R\014ipv6HopLimit\022J\n\rlistener_auth\030" + "\230\001 \001(\0132\034.xtcp_config.v1.ListenerAuthB\006\272H" + "\003\310\001\000R\014listenerAuth\022Z\n\023prometheus_listene" + "r\030\231\001 \001(\0132 .xtcp_config.v1.ListenerEndpoi" + "ntB\006\272H\003\310\001\000R\022prometheusListener\022U\n\023admin_" + "listener_auth\030\232\001 \001(\0132\034.xtcp_config.v1.Li" + "stenerAuthB\006\272H\003\310\001\000R\021adminListenerAuth\022\'\n" + "\tgrpc_port\030\240\001 \001(\rB\t\272H\006*\004\030\377\377\003R\010grpcPort\022N" + "\n\rgrpc_listener\030\241\001 \001(\0132 .xtcp_config.v1." + "ListenerEndpointB\006\272H\003\310\001\000R\014grpcListener\022A" + "\n\tlisteners\030\242\001 \003(\0132\030.xtcp_config.v1.List" + "enerB\010\272H\005\222\001\002\020\020R\tlisteners\022,\n\rpyroscope_u" + "rl\030\252\001 \001(\tB\006\272H\003\310\001\000R\014pyroscopeUrl\0225\n\022pyros" + "cope_app_name\030\253\001 \001(\tB\006\272H\003\310\001\000R\020pyroscopeA" + "ppName\0227\n\023pyroscope_sample_hz\030\254\001 \001(\rB\006\272H" + "\003\310\001\000R\021pyroscopeSampleHz\022J\n\035pyroscope_upl" + "oad_interval_sec\030\255\001 \001(\rB\006\272H\003\310\001\000R\032pyrosco" + "peUploadIntervalSec\0229\n\024resolve_container" + "_id\030\310\001 \001(\010B\006\272H\003\310\001\000R\022resolveContainerId\0227" + "\n\027enrich_container_enable\030\311\001 \001(\010R\025enrich" + "ContainerEnable\0227\n\022docker_socket_path\030\312\001" + " \001(\tB\010\272H\005r\003\030\377\001R\020dockerSocketPath\022-\n\022enri" + "ch_lldp_enable\030\322\001 \001(\010R\020enrichLldpEnable\022" + "5\n\021lldpd_socket_path\030\323\001 \001(\tB\010\272H\005r\003\030\377\001R\017l" + "ldpdSocketPath\0226\n\022lldpd_version_hint\030\324\001 " + "\001(\tB\007\272H\004r\002\030\020R\020lldpdVersionHint\022+\n\021enrich" + "_nic_enable\030\334\001 \001(\010R\017enrichNicEnable\022+\n\014u" + "plink_count\030\335\001 \001(\rB\007\272H\004*\002\030\002R\013uplinkCount" + "\0226\n\021uplink_interfaces\030\336\001 \003(\tB\010\272H\005\222\001\002\020\002R\020" + "uplinkInterfaces\022$\n\rpopulate_nsid\030\346\001 \001(\010" + "R\014populateNsid\022+\n\021enrich_asn_enable\030\360\001 \001" + "(\010R\017enrichAsnEnable\022)\n\013asn_db_path\030\361\001 \001(" + "\tB\010\272H\005r\003\030\377\001R\tasnDbPath\022L\n\024asn_refresh_in" + "terval\030\362\001 \001(\0132\031.google.protobuf.Duration" + "R\022asnRefreshInterval\022=\n\025ipmeta_bootstrap" + "_path\030\363\001 \001(\tB\010\272H\005r\003\030\377\001R\023ipmetaBootstrapP" + "ath\0225\n\021ipmeta_cache_path\030\364\001 \001(\tB\010\272H\005r\003\030\377" + "\001R\017ipmetaCachePath\0225\n\026enrich_locality_en" + "able\030\365\001 \001(\010R\024enrichLocalityEnable\022V\n\031loc" + "ality_refresh_interval\030\366\001 \001(\0132\031.google.p" + "rotobuf.DurationR\027localityRefreshInterva" + "l\0225\n\026locality_events_enable\030\367\001 \001(\010R\024loca" + "lityEventsEnable\022E\n\014link_monitor\030\372\001 \001(\0132" + "!.xtcp_config.v1.LinkMonitorConfigR\013link" + "Monitor:s\272Hp\032n\n\017XtcpConfig.poll\0222Poll ti" + "meout must be less than poll poll_freque" + "ncy\032\'this.poll_frequency > this.poll_tim" + "eout\"\237\001\n\024EnabledDeserializers\022K\n\007enabled" + "\030\001 \003(\01321.xtcp_config.v1.EnabledDeseriali" + "zers.EnabledEntryR\007enabled\032:\n\014EnabledEnt" + "ry\022\020\n\003key\030\001 \001(\tR\003key\022\024\n\005value\030\002 \001(\010R\005val" + "ue:\0028\001*h\n\017ListenerNetwork\022 \n\034LISTENER_NE" + "TWORK_UNSPECIFIED\020\000\022\030\n\024LISTENER_NETWORK_" + "TCP\020\001\022\031\n\025LISTENER_NETWORK_UNIX\020\002*\241\001\n\020Lis" + "tenerAuthMode\022\"\n\036LISTENER_AUTH_MODE_UNSP" + "ECIFIED\020\000\022\037\n\033LISTENER_AUTH_MODE_DISABLED" + "\020\001\022 \n\034LISTENER_AUTH_MODE_RAW_TOKEN\020\002\022&\n\"" + "LISTENER_AUTH_MODE_HMAC_UTC_MINUTE\020\003*\367\001\n" + "\rEndpointClass\022\036\n\032ENDPOINT_CLASS_UNSPECI" + "FIED\020\000\022\031\n\025ENDPOINT_CLASS_HEALTH\020\001\022\032\n\026END" + "POINT_CLASS_METRICS\020\002\022!\n\035ENDPOINT_CLASS_" + "METRICS_DETAIL\020\003\022\025\n\021ENDPOINT_CLASS_IP\020\004\022" + "\030\n\024ENDPOINT_CLASS_DEBUG\020\005\022\034\n\030ENDPOINT_CL" + "ASS_GRPC_READ\020\006\022\035\n\031ENDPOINT_CLASS_GRPC_A" + "DMIN\020\0072\245\007\n\rConfigService\022a\n\003Get\022\032.xtcp_c" + "onfig.v1.GetRequest\032\033.xtcp_config.v1.Get" + "Response\"!\310\363\030\001\202\323\344\223\002\027\032\022/ConfigService/Get" + ":\001*\022a\n\003Set\022\032.xtcp_config.v1.SetRequest\032\033" + ".xtcp_config.v1.SetResponse\"!\310\363\030\002\202\323\344\223\002\027\032" + "\022/ConfigService/Set:\001*\022\225\001\n\020SetPollFreque" + "ncy\022\'.xtcp_config.v1.SetPollFrequencyReq" + "uest\032(.xtcp_config.v1.SetPollFrequencyRe" + "sponse\".\310\363\030\002\202\323\344\223\002$\032\037/ConfigService/SetPo" + "llFrequency:\001*\022\201\001\n\013TriggerPoll\022\".xtcp_co" + "nfig.v1.TriggerPollRequest\032#.xtcp_config" + ".v1.TriggerPollResponse\")\310\363\030\002\202\323\344\223\002\037\032\032/Co" + "nfigService/TriggerPoll:\001*\022\225\001\n\020TriggerPo" + "llBurst\022\'.xtcp_config.v1.TriggerPollBurs" + "tRequest\032(.xtcp_config.v1.TriggerPollBur" + "stResponse\".\310\363\030\002\202\323\344\223\002$\032\037/ConfigService/T" + "riggerPollBurst:\001*\022\201\001\n\013SetS3Upload\022\".xtc" + "p_config.v1.SetS3UploadRequest\032#.xtcp_co" + "nfig.v1.SetS3UploadResponse\")\310\363\030\002\202\323\344\223\002\037\032" + "\032/ConfigService/SetS3Upload:\001*\022\225\001\n\020SetEn" + "velopeFlush\022\'.xtcp_config.v1.SetEnvelope" + "FlushRequest\032(.xtcp_config.v1.SetEnvelop" + "eFlushResponse\".\310\363\030\002\202\323\344\223\002$\032\037/ConfigServi" + "ce/SetEnvelopeFlush:\001*B\257\001\n\022com.xtcp_conf" + "ig.v1B\017XtcpConfigProtoP\001Z3github.com/ran" + "domizedcoder/xtcp2/gen/go/xtcp_config\242\002\003" + "XXX\252\002\rXtcpConfig.V1\312\002\rXtcpConfig\\V1\342\002\031Xt" + "cpConfig\\V1\\GPBMetadata\352\002\016XtcpConfig::V1" + "b\006proto3" }; static const ::_pbi::DescriptorTable* PROTOBUF_NONNULL const descriptor_table_xtcp_5fconfig_2fv1_2fxtcp_5fconfig_2eproto_deps[4] = { @@ -4870,7 +4883,7 @@ static ::absl::once_flag descriptor_table_xtcp_5fconfig_2fv1_2fxtcp_5fconfig_2ep PROTOBUF_CONSTINIT const ::_pbi::DescriptorTable descriptor_table_xtcp_5fconfig_2fv1_2fxtcp_5fconfig_2eproto = { false, false, - 12538, + 12648, descriptor_table_protodef_xtcp_5fconfig_2fv1_2fxtcp_5fconfig_2eproto, "xtcp_config/v1/xtcp_config.proto", &descriptor_table_xtcp_5fconfig_2fv1_2fxtcp_5fconfig_2eproto_once, @@ -10306,22 +10319,22 @@ PROTOBUF_NOINLINE void XtcpConfig::Clear() { if (BatchCheckHasBit(cached_has_bits, 0x0000ff00U)) { ::memset(&this_._impl_.s3_skip_bucket_probe_, 0, static_cast<::size_t>( - reinterpret_cast(&this_._impl_.ipv6_hop_limit_) - + reinterpret_cast(&this_._impl_.s3_max_buffered_bytes_) - reinterpret_cast(&this_._impl_.s3_skip_bucket_probe_)) + - sizeof(_impl_.ipv6_hop_limit_)); + sizeof(_impl_.s3_max_buffered_bytes_)); } if (BatchCheckHasBit(cached_has_bits, 0x00ff0000U)) { - ::memset(&this_._impl_.grpc_port_, 0, + ::memset(&this_._impl_.ipv4_ttl_, 0, static_cast<::size_t>( - reinterpret_cast(&this_._impl_.uplink_count_) - - reinterpret_cast(&this_._impl_.grpc_port_)) + - sizeof(_impl_.uplink_count_)); + reinterpret_cast(&this_._impl_.enrich_lldp_enable_) - + reinterpret_cast(&this_._impl_.ipv4_ttl_)) + + sizeof(_impl_.enrich_lldp_enable_)); } - if (BatchCheckHasBit(cached_has_bits, 0x0f000000U)) { - ::memset(&this_._impl_.populate_nsid_, 0, + if (BatchCheckHasBit(cached_has_bits, 0x3f000000U)) { + ::memset(&this_._impl_.enrich_nic_enable_, 0, static_cast<::size_t>( reinterpret_cast(&this_._impl_.locality_events_enable_) - - reinterpret_cast(&this_._impl_.populate_nsid_)) + + reinterpret_cast(&this_._impl_.enrich_nic_enable_)) + sizeof(_impl_.locality_events_enable_)); } _impl_._has_bits_.Clear(); @@ -10868,6 +10881,26 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( stream); } + cached_has_bits = this_._impl_._has_bits_[2]; + // uint32 s3_queue_depth = 116 [json_name = "s3QueueDepth", (.buf.validate.field) = { + if (CheckHasBit(cached_has_bits, 0x00004000U)) { + if (this_._internal_s3_queue_depth() != 0) { + target = stream->EnsureSpace(target); + target = ::_pbi::WireFormatLite::WriteUInt32ToArray( + 116, this_._internal_s3_queue_depth(), target); + } + } + + // uint64 s3_max_buffered_bytes = 117 [json_name = "s3MaxBufferedBytes", (.buf.validate.field) = { + if (CheckHasBit(cached_has_bits, 0x00008000U)) { + if (this_._internal_s3_max_buffered_bytes() != 0) { + target = stream->EnsureSpace(target); + target = ::_pbi::WireFormatLite::WriteUInt64ToArray( + 117, this_._internal_s3_max_buffered_bytes(), target); + } + } + + cached_has_bits = this_._impl_._has_bits_[1]; // string hostname = 130 [json_name = "hostname", (.buf.validate.field) = { if (CheckHasBit(cached_has_bits, 0x00000020U)) { if (!this_._internal_hostname().empty()) { @@ -10920,7 +10953,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( cached_has_bits = this_._impl_._has_bits_[2]; // uint32 ipv4_ttl = 150 [json_name = "ipv4Ttl", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00004000U)) { + if (CheckHasBit(cached_has_bits, 0x00010000U)) { if (this_._internal_ipv4_ttl() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteUInt32ToArray( @@ -10929,7 +10962,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( } // uint32 ipv6_hop_limit = 151 [json_name = "ipv6HopLimit", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00008000U)) { + if (CheckHasBit(cached_has_bits, 0x00020000U)) { if (this_._internal_ipv6_hop_limit() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteUInt32ToArray( @@ -10961,7 +10994,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( cached_has_bits = this_._impl_._has_bits_[2]; // uint32 grpc_port = 160 [json_name = "grpcPort", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00010000U)) { + if (CheckHasBit(cached_has_bits, 0x00040000U)) { if (this_._internal_grpc_port() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteUInt32ToArray( @@ -11013,7 +11046,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( cached_has_bits = this_._impl_._has_bits_[2]; // uint32 pyroscope_sample_hz = 172 [json_name = "pyroscopeSampleHz", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00020000U)) { + if (CheckHasBit(cached_has_bits, 0x00080000U)) { if (this_._internal_pyroscope_sample_hz() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteUInt32ToArray( @@ -11022,7 +11055,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( } // uint32 pyroscope_upload_interval_sec = 173 [json_name = "pyroscopeUploadIntervalSec", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00040000U)) { + if (CheckHasBit(cached_has_bits, 0x00100000U)) { if (this_._internal_pyroscope_upload_interval_sec() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteUInt32ToArray( @@ -11031,7 +11064,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( } // bool resolve_container_id = 200 [json_name = "resolveContainerId", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00080000U)) { + if (CheckHasBit(cached_has_bits, 0x00200000U)) { if (this_._internal_resolve_container_id() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteBoolToArray( @@ -11040,7 +11073,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( } // bool enrich_container_enable = 201 [json_name = "enrichContainerEnable"]; - if (CheckHasBit(cached_has_bits, 0x00100000U)) { + if (CheckHasBit(cached_has_bits, 0x00400000U)) { if (this_._internal_enrich_container_enable() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteBoolToArray( @@ -11061,7 +11094,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( cached_has_bits = this_._impl_._has_bits_[2]; // bool enrich_lldp_enable = 210 [json_name = "enrichLldpEnable"]; - if (CheckHasBit(cached_has_bits, 0x00200000U)) { + if (CheckHasBit(cached_has_bits, 0x00800000U)) { if (this_._internal_enrich_lldp_enable() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteBoolToArray( @@ -11092,7 +11125,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( cached_has_bits = this_._impl_._has_bits_[2]; // bool enrich_nic_enable = 220 [json_name = "enrichNicEnable"]; - if (CheckHasBit(cached_has_bits, 0x00400000U)) { + if (CheckHasBit(cached_has_bits, 0x01000000U)) { if (this_._internal_enrich_nic_enable() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteBoolToArray( @@ -11101,7 +11134,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( } // uint32 uplink_count = 221 [json_name = "uplinkCount", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00800000U)) { + if (CheckHasBit(cached_has_bits, 0x02000000U)) { if (this_._internal_uplink_count() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteUInt32ToArray( @@ -11122,7 +11155,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( cached_has_bits = this_._impl_._has_bits_[2]; // bool populate_nsid = 230 [json_name = "populateNsid"]; - if (CheckHasBit(cached_has_bits, 0x01000000U)) { + if (CheckHasBit(cached_has_bits, 0x04000000U)) { if (this_._internal_populate_nsid() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteBoolToArray( @@ -11131,7 +11164,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( } // bool enrich_asn_enable = 240 [json_name = "enrichAsnEnable"]; - if (CheckHasBit(cached_has_bits, 0x02000000U)) { + if (CheckHasBit(cached_has_bits, 0x08000000U)) { if (this_._internal_enrich_asn_enable() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteBoolToArray( @@ -11179,7 +11212,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( cached_has_bits = this_._impl_._has_bits_[2]; // bool enrich_locality_enable = 245 [json_name = "enrichLocalityEnable"]; - if (CheckHasBit(cached_has_bits, 0x04000000U)) { + if (CheckHasBit(cached_has_bits, 0x10000000U)) { if (this_._internal_enrich_locality_enable() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteBoolToArray( @@ -11197,7 +11230,7 @@ ::uint8_t* PROTOBUF_NONNULL XtcpConfig::_InternalSerialize( cached_has_bits = this_._impl_._has_bits_[2]; // bool locality_events_enable = 247 [json_name = "localityEventsEnable"]; - if (CheckHasBit(cached_has_bits, 0x08000000U)) { + if (CheckHasBit(cached_has_bits, 0x20000000U)) { if (this_._internal_locality_events_enable() != 0) { target = stream->EnsureSpace(target); target = ::_pbi::WireFormatLite::WriteBoolToArray( @@ -11770,96 +11803,110 @@ ::size_t XtcpConfig::ByteSizeLong() const { this_._internal_s3_upload_max_attempts()); } } - // uint32 ipv4_ttl = 150 [json_name = "ipv4Ttl", (.buf.validate.field) = { + // uint32 s3_queue_depth = 116 [json_name = "s3QueueDepth", (.buf.validate.field) = { if (CheckHasBit(cached_has_bits, 0x00004000U)) { + if (this_._internal_s3_queue_depth() != 0) { + total_size += 2 + ::_pbi::WireFormatLite::UInt32Size( + this_._internal_s3_queue_depth()); + } + } + // uint64 s3_max_buffered_bytes = 117 [json_name = "s3MaxBufferedBytes", (.buf.validate.field) = { + if (CheckHasBit(cached_has_bits, 0x00008000U)) { + if (this_._internal_s3_max_buffered_bytes() != 0) { + total_size += 2 + ::_pbi::WireFormatLite::UInt64Size( + this_._internal_s3_max_buffered_bytes()); + } + } + } + if (BatchCheckHasBit(cached_has_bits, 0x00ff0000U)) { + // uint32 ipv4_ttl = 150 [json_name = "ipv4Ttl", (.buf.validate.field) = { + if (CheckHasBit(cached_has_bits, 0x00010000U)) { if (this_._internal_ipv4_ttl() != 0) { total_size += 2 + ::_pbi::WireFormatLite::UInt32Size( this_._internal_ipv4_ttl()); } } // uint32 ipv6_hop_limit = 151 [json_name = "ipv6HopLimit", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00008000U)) { + if (CheckHasBit(cached_has_bits, 0x00020000U)) { if (this_._internal_ipv6_hop_limit() != 0) { total_size += 2 + ::_pbi::WireFormatLite::UInt32Size( this_._internal_ipv6_hop_limit()); } } - } - if (BatchCheckHasBit(cached_has_bits, 0x00ff0000U)) { // uint32 grpc_port = 160 [json_name = "grpcPort", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00010000U)) { + if (CheckHasBit(cached_has_bits, 0x00040000U)) { if (this_._internal_grpc_port() != 0) { total_size += 2 + ::_pbi::WireFormatLite::UInt32Size( this_._internal_grpc_port()); } } // uint32 pyroscope_sample_hz = 172 [json_name = "pyroscopeSampleHz", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00020000U)) { + if (CheckHasBit(cached_has_bits, 0x00080000U)) { if (this_._internal_pyroscope_sample_hz() != 0) { total_size += 2 + ::_pbi::WireFormatLite::UInt32Size( this_._internal_pyroscope_sample_hz()); } } // uint32 pyroscope_upload_interval_sec = 173 [json_name = "pyroscopeUploadIntervalSec", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00040000U)) { + if (CheckHasBit(cached_has_bits, 0x00100000U)) { if (this_._internal_pyroscope_upload_interval_sec() != 0) { total_size += 2 + ::_pbi::WireFormatLite::UInt32Size( this_._internal_pyroscope_upload_interval_sec()); } } // bool resolve_container_id = 200 [json_name = "resolveContainerId", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00080000U)) { + if (CheckHasBit(cached_has_bits, 0x00200000U)) { if (this_._internal_resolve_container_id() != 0) { total_size += 3; } } // bool enrich_container_enable = 201 [json_name = "enrichContainerEnable"]; - if (CheckHasBit(cached_has_bits, 0x00100000U)) { + if (CheckHasBit(cached_has_bits, 0x00400000U)) { if (this_._internal_enrich_container_enable() != 0) { total_size += 3; } } // bool enrich_lldp_enable = 210 [json_name = "enrichLldpEnable"]; - if (CheckHasBit(cached_has_bits, 0x00200000U)) { + if (CheckHasBit(cached_has_bits, 0x00800000U)) { if (this_._internal_enrich_lldp_enable() != 0) { total_size += 3; } } + } + if (BatchCheckHasBit(cached_has_bits, 0x3f000000U)) { // bool enrich_nic_enable = 220 [json_name = "enrichNicEnable"]; - if (CheckHasBit(cached_has_bits, 0x00400000U)) { + if (CheckHasBit(cached_has_bits, 0x01000000U)) { if (this_._internal_enrich_nic_enable() != 0) { total_size += 3; } } // uint32 uplink_count = 221 [json_name = "uplinkCount", (.buf.validate.field) = { - if (CheckHasBit(cached_has_bits, 0x00800000U)) { + if (CheckHasBit(cached_has_bits, 0x02000000U)) { if (this_._internal_uplink_count() != 0) { total_size += 2 + ::_pbi::WireFormatLite::UInt32Size( this_._internal_uplink_count()); } } - } - if (BatchCheckHasBit(cached_has_bits, 0x0f000000U)) { // bool populate_nsid = 230 [json_name = "populateNsid"]; - if (CheckHasBit(cached_has_bits, 0x01000000U)) { + if (CheckHasBit(cached_has_bits, 0x04000000U)) { if (this_._internal_populate_nsid() != 0) { total_size += 3; } } // bool enrich_asn_enable = 240 [json_name = "enrichAsnEnable"]; - if (CheckHasBit(cached_has_bits, 0x02000000U)) { + if (CheckHasBit(cached_has_bits, 0x08000000U)) { if (this_._internal_enrich_asn_enable() != 0) { total_size += 3; } } // bool enrich_locality_enable = 245 [json_name = "enrichLocalityEnable"]; - if (CheckHasBit(cached_has_bits, 0x04000000U)) { + if (CheckHasBit(cached_has_bits, 0x10000000U)) { if (this_._internal_enrich_locality_enable() != 0) { total_size += 3; } } // bool locality_events_enable = 247 [json_name = "localityEventsEnable"]; - if (CheckHasBit(cached_has_bits, 0x08000000U)) { + if (CheckHasBit(cached_has_bits, 0x20000000U)) { if (this_._internal_locality_events_enable() != 0) { total_size += 3; } @@ -12452,75 +12499,85 @@ void XtcpConfig::MergeImpl(::google::protobuf::MessageLite& to_msg, } } if (CheckHasBit(cached_has_bits, 0x00004000U)) { - if (from._internal_ipv4_ttl() != 0) { - _this->_impl_.ipv4_ttl_ = from._impl_.ipv4_ttl_; + if (from._internal_s3_queue_depth() != 0) { + _this->_impl_.s3_queue_depth_ = from._impl_.s3_queue_depth_; } } if (CheckHasBit(cached_has_bits, 0x00008000U)) { - if (from._internal_ipv6_hop_limit() != 0) { - _this->_impl_.ipv6_hop_limit_ = from._impl_.ipv6_hop_limit_; + if (from._internal_s3_max_buffered_bytes() != 0) { + _this->_impl_.s3_max_buffered_bytes_ = from._impl_.s3_max_buffered_bytes_; } } } if (BatchCheckHasBit(cached_has_bits, 0x00ff0000U)) { if (CheckHasBit(cached_has_bits, 0x00010000U)) { + if (from._internal_ipv4_ttl() != 0) { + _this->_impl_.ipv4_ttl_ = from._impl_.ipv4_ttl_; + } + } + if (CheckHasBit(cached_has_bits, 0x00020000U)) { + if (from._internal_ipv6_hop_limit() != 0) { + _this->_impl_.ipv6_hop_limit_ = from._impl_.ipv6_hop_limit_; + } + } + if (CheckHasBit(cached_has_bits, 0x00040000U)) { if (from._internal_grpc_port() != 0) { _this->_impl_.grpc_port_ = from._impl_.grpc_port_; } } - if (CheckHasBit(cached_has_bits, 0x00020000U)) { + if (CheckHasBit(cached_has_bits, 0x00080000U)) { if (from._internal_pyroscope_sample_hz() != 0) { _this->_impl_.pyroscope_sample_hz_ = from._impl_.pyroscope_sample_hz_; } } - if (CheckHasBit(cached_has_bits, 0x00040000U)) { + if (CheckHasBit(cached_has_bits, 0x00100000U)) { if (from._internal_pyroscope_upload_interval_sec() != 0) { _this->_impl_.pyroscope_upload_interval_sec_ = from._impl_.pyroscope_upload_interval_sec_; } } - if (CheckHasBit(cached_has_bits, 0x00080000U)) { + if (CheckHasBit(cached_has_bits, 0x00200000U)) { if (from._internal_resolve_container_id() != 0) { _this->_impl_.resolve_container_id_ = from._impl_.resolve_container_id_; } } - if (CheckHasBit(cached_has_bits, 0x00100000U)) { + if (CheckHasBit(cached_has_bits, 0x00400000U)) { if (from._internal_enrich_container_enable() != 0) { _this->_impl_.enrich_container_enable_ = from._impl_.enrich_container_enable_; } } - if (CheckHasBit(cached_has_bits, 0x00200000U)) { + if (CheckHasBit(cached_has_bits, 0x00800000U)) { if (from._internal_enrich_lldp_enable() != 0) { _this->_impl_.enrich_lldp_enable_ = from._impl_.enrich_lldp_enable_; } } - if (CheckHasBit(cached_has_bits, 0x00400000U)) { + } + if (BatchCheckHasBit(cached_has_bits, 0x3f000000U)) { + if (CheckHasBit(cached_has_bits, 0x01000000U)) { if (from._internal_enrich_nic_enable() != 0) { _this->_impl_.enrich_nic_enable_ = from._impl_.enrich_nic_enable_; } } - if (CheckHasBit(cached_has_bits, 0x00800000U)) { + if (CheckHasBit(cached_has_bits, 0x02000000U)) { if (from._internal_uplink_count() != 0) { _this->_impl_.uplink_count_ = from._impl_.uplink_count_; } } - } - if (BatchCheckHasBit(cached_has_bits, 0x0f000000U)) { - if (CheckHasBit(cached_has_bits, 0x01000000U)) { + if (CheckHasBit(cached_has_bits, 0x04000000U)) { if (from._internal_populate_nsid() != 0) { _this->_impl_.populate_nsid_ = from._impl_.populate_nsid_; } } - if (CheckHasBit(cached_has_bits, 0x02000000U)) { + if (CheckHasBit(cached_has_bits, 0x08000000U)) { if (from._internal_enrich_asn_enable() != 0) { _this->_impl_.enrich_asn_enable_ = from._impl_.enrich_asn_enable_; } } - if (CheckHasBit(cached_has_bits, 0x04000000U)) { + if (CheckHasBit(cached_has_bits, 0x10000000U)) { if (from._internal_enrich_locality_enable() != 0) { _this->_impl_.enrich_locality_enable_ = from._impl_.enrich_locality_enable_; } } - if (CheckHasBit(cached_has_bits, 0x08000000U)) { + if (CheckHasBit(cached_has_bits, 0x20000000U)) { if (from._internal_locality_events_enable() != 0) { _this->_impl_.locality_events_enable_ = from._impl_.locality_events_enable_; } diff --git a/gen/cpp/xtcp_config/v1/xtcp_config.pb.h b/gen/cpp/xtcp_config/v1/xtcp_config.pb.h index 403b1e0a2..c0121ddb3 100644 --- a/gen/cpp/xtcp_config/v1/xtcp_config.pb.h +++ b/gen/cpp/xtcp_config/v1/xtcp_config.pb.h @@ -3900,6 +3900,8 @@ class PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED XtcpConfig final : public ::google: kS3FlushJitterPctFieldNumber = 112, kS3FlushThresholdJitterPctFieldNumber = 113, kS3UploadMaxAttemptsFieldNumber = 114, + kS3QueueDepthFieldNumber = 116, + kS3MaxBufferedBytesFieldNumber = 117, kIpv4TtlFieldNumber = 150, kIpv6HopLimitFieldNumber = 151, kGrpcPortFieldNumber = 160, @@ -4935,6 +4937,26 @@ class PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED XtcpConfig final : public ::google: ::uint32_t _internal_s3_upload_max_attempts() const; void _internal_set_s3_upload_max_attempts(::uint32_t value); + public: + // uint32 s3_queue_depth = 116 [json_name = "s3QueueDepth", (.buf.validate.field) = { + void clear_s3_queue_depth() ; + [[nodiscard]] ::uint32_t s3_queue_depth() const; + void set_s3_queue_depth(::uint32_t value); + + private: + ::uint32_t _internal_s3_queue_depth() const; + void _internal_set_s3_queue_depth(::uint32_t value); + + public: + // uint64 s3_max_buffered_bytes = 117 [json_name = "s3MaxBufferedBytes", (.buf.validate.field) = { + void clear_s3_max_buffered_bytes() ; + [[nodiscard]] ::uint64_t s3_max_buffered_bytes() const; + void set_s3_max_buffered_bytes(::uint64_t value); + + private: + ::uint64_t _internal_s3_max_buffered_bytes() const; + void _internal_set_s3_max_buffered_bytes(::uint64_t value); + public: // uint32 ipv4_ttl = 150 [json_name = "ipv4Ttl", (.buf.validate.field) = { void clear_ipv4_ttl() ; @@ -5080,7 +5102,7 @@ class PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED XtcpConfig final : public ::google: private: class _Internal; using ParseTableT_ = - ::google::protobuf::internal::TcParseTable<5, 92, + ::google::protobuf::internal::TcParseTable<5, 94, 16, 474, 33>; static constexpr ParseTableT_ InternalGenerateParseTable_( @@ -5187,6 +5209,8 @@ class PROTOBUF_FUTURE_ADD_EARLY_WARN_UNUSED XtcpConfig final : public ::google: ::uint32_t s3_flush_jitter_pct_; ::uint32_t s3_flush_threshold_jitter_pct_; ::uint32_t s3_upload_max_attempts_; + ::uint32_t s3_queue_depth_; + ::uint64_t s3_max_buffered_bytes_; ::uint32_t ipv4_ttl_; ::uint32_t ipv6_hop_limit_; ::uint32_t grpc_port_; @@ -13222,6 +13246,54 @@ inline void XtcpConfig::set_allocated_s3_upload_backoff_cap(::google::protobuf:: // @@protoc_insertion_point(field_set_allocated:xtcp_config.v1.XtcpConfig.s3_upload_backoff_cap) } +// uint32 s3_queue_depth = 116 [json_name = "s3QueueDepth", (.buf.validate.field) = { +inline void XtcpConfig::clear_s3_queue_depth() { + ::google::protobuf::internal::TSanWrite(&_impl_); + _impl_.s3_queue_depth_ = 0u; + ClearHasBit(_impl_._has_bits_[2], 0x00004000U); +} +inline ::uint32_t XtcpConfig::s3_queue_depth() const { + // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.s3_queue_depth) + return _internal_s3_queue_depth(); +} +inline void XtcpConfig::set_s3_queue_depth(::uint32_t value) { + _internal_set_s3_queue_depth(value); + SetHasBit(_impl_._has_bits_[2], 0x00004000U); + // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.s3_queue_depth) +} +inline ::uint32_t XtcpConfig::_internal_s3_queue_depth() const { + ::google::protobuf::internal::TSanRead(&_impl_); + return _impl_.s3_queue_depth_; +} +inline void XtcpConfig::_internal_set_s3_queue_depth(::uint32_t value) { + ::google::protobuf::internal::TSanWrite(&_impl_); + _impl_.s3_queue_depth_ = value; +} + +// uint64 s3_max_buffered_bytes = 117 [json_name = "s3MaxBufferedBytes", (.buf.validate.field) = { +inline void XtcpConfig::clear_s3_max_buffered_bytes() { + ::google::protobuf::internal::TSanWrite(&_impl_); + _impl_.s3_max_buffered_bytes_ = ::uint64_t{0u}; + ClearHasBit(_impl_._has_bits_[2], 0x00008000U); +} +inline ::uint64_t XtcpConfig::s3_max_buffered_bytes() const { + // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.s3_max_buffered_bytes) + return _internal_s3_max_buffered_bytes(); +} +inline void XtcpConfig::set_s3_max_buffered_bytes(::uint64_t value) { + _internal_set_s3_max_buffered_bytes(value); + SetHasBit(_impl_._has_bits_[2], 0x00008000U); + // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.s3_max_buffered_bytes) +} +inline ::uint64_t XtcpConfig::_internal_s3_max_buffered_bytes() const { + ::google::protobuf::internal::TSanRead(&_impl_); + return _impl_.s3_max_buffered_bytes_; +} +inline void XtcpConfig::_internal_set_s3_max_buffered_bytes(::uint64_t value) { + ::google::protobuf::internal::TSanWrite(&_impl_); + _impl_.s3_max_buffered_bytes_ = value; +} + // string hostname = 130 [json_name = "hostname", (.buf.validate.field) = { inline void XtcpConfig::clear_hostname() { ::google::protobuf::internal::TSanWrite(&_impl_); @@ -13546,7 +13618,7 @@ inline void XtcpConfig::set_allocated_daemon_version(::std::string* PROTOBUF_NUL inline void XtcpConfig::clear_ipv4_ttl() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.ipv4_ttl_ = 0u; - ClearHasBit(_impl_._has_bits_[2], 0x00004000U); + ClearHasBit(_impl_._has_bits_[2], 0x00010000U); } inline ::uint32_t XtcpConfig::ipv4_ttl() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.ipv4_ttl) @@ -13554,7 +13626,7 @@ inline ::uint32_t XtcpConfig::ipv4_ttl() const { } inline void XtcpConfig::set_ipv4_ttl(::uint32_t value) { _internal_set_ipv4_ttl(value); - SetHasBit(_impl_._has_bits_[2], 0x00004000U); + SetHasBit(_impl_._has_bits_[2], 0x00010000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.ipv4_ttl) } inline ::uint32_t XtcpConfig::_internal_ipv4_ttl() const { @@ -13570,7 +13642,7 @@ inline void XtcpConfig::_internal_set_ipv4_ttl(::uint32_t value) { inline void XtcpConfig::clear_ipv6_hop_limit() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.ipv6_hop_limit_ = 0u; - ClearHasBit(_impl_._has_bits_[2], 0x00008000U); + ClearHasBit(_impl_._has_bits_[2], 0x00020000U); } inline ::uint32_t XtcpConfig::ipv6_hop_limit() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.ipv6_hop_limit) @@ -13578,7 +13650,7 @@ inline ::uint32_t XtcpConfig::ipv6_hop_limit() const { } inline void XtcpConfig::set_ipv6_hop_limit(::uint32_t value) { _internal_set_ipv6_hop_limit(value); - SetHasBit(_impl_._has_bits_[2], 0x00008000U); + SetHasBit(_impl_._has_bits_[2], 0x00020000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.ipv6_hop_limit) } inline ::uint32_t XtcpConfig::_internal_ipv6_hop_limit() const { @@ -13888,7 +13960,7 @@ inline void XtcpConfig::set_allocated_admin_listener_auth(::xtcp_config::v1::Lis inline void XtcpConfig::clear_grpc_port() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.grpc_port_ = 0u; - ClearHasBit(_impl_._has_bits_[2], 0x00010000U); + ClearHasBit(_impl_._has_bits_[2], 0x00040000U); } inline ::uint32_t XtcpConfig::grpc_port() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.grpc_port) @@ -13896,7 +13968,7 @@ inline ::uint32_t XtcpConfig::grpc_port() const { } inline void XtcpConfig::set_grpc_port(::uint32_t value) { _internal_set_grpc_port(value); - SetHasBit(_impl_._has_bits_[2], 0x00010000U); + SetHasBit(_impl_._has_bits_[2], 0x00040000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.grpc_port) } inline ::uint32_t XtcpConfig::_internal_grpc_port() const { @@ -14193,7 +14265,7 @@ inline void XtcpConfig::set_allocated_pyroscope_app_name(::std::string* PROTOBUF inline void XtcpConfig::clear_pyroscope_sample_hz() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.pyroscope_sample_hz_ = 0u; - ClearHasBit(_impl_._has_bits_[2], 0x00020000U); + ClearHasBit(_impl_._has_bits_[2], 0x00080000U); } inline ::uint32_t XtcpConfig::pyroscope_sample_hz() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.pyroscope_sample_hz) @@ -14201,7 +14273,7 @@ inline ::uint32_t XtcpConfig::pyroscope_sample_hz() const { } inline void XtcpConfig::set_pyroscope_sample_hz(::uint32_t value) { _internal_set_pyroscope_sample_hz(value); - SetHasBit(_impl_._has_bits_[2], 0x00020000U); + SetHasBit(_impl_._has_bits_[2], 0x00080000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.pyroscope_sample_hz) } inline ::uint32_t XtcpConfig::_internal_pyroscope_sample_hz() const { @@ -14217,7 +14289,7 @@ inline void XtcpConfig::_internal_set_pyroscope_sample_hz(::uint32_t value) { inline void XtcpConfig::clear_pyroscope_upload_interval_sec() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.pyroscope_upload_interval_sec_ = 0u; - ClearHasBit(_impl_._has_bits_[2], 0x00040000U); + ClearHasBit(_impl_._has_bits_[2], 0x00100000U); } inline ::uint32_t XtcpConfig::pyroscope_upload_interval_sec() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.pyroscope_upload_interval_sec) @@ -14225,7 +14297,7 @@ inline ::uint32_t XtcpConfig::pyroscope_upload_interval_sec() const { } inline void XtcpConfig::set_pyroscope_upload_interval_sec(::uint32_t value) { _internal_set_pyroscope_upload_interval_sec(value); - SetHasBit(_impl_._has_bits_[2], 0x00040000U); + SetHasBit(_impl_._has_bits_[2], 0x00100000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.pyroscope_upload_interval_sec) } inline ::uint32_t XtcpConfig::_internal_pyroscope_upload_interval_sec() const { @@ -14241,7 +14313,7 @@ inline void XtcpConfig::_internal_set_pyroscope_upload_interval_sec(::uint32_t v inline void XtcpConfig::clear_resolve_container_id() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.resolve_container_id_ = false; - ClearHasBit(_impl_._has_bits_[2], 0x00080000U); + ClearHasBit(_impl_._has_bits_[2], 0x00200000U); } inline bool XtcpConfig::resolve_container_id() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.resolve_container_id) @@ -14249,7 +14321,7 @@ inline bool XtcpConfig::resolve_container_id() const { } inline void XtcpConfig::set_resolve_container_id(bool value) { _internal_set_resolve_container_id(value); - SetHasBit(_impl_._has_bits_[2], 0x00080000U); + SetHasBit(_impl_._has_bits_[2], 0x00200000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.resolve_container_id) } inline bool XtcpConfig::_internal_resolve_container_id() const { @@ -14265,7 +14337,7 @@ inline void XtcpConfig::_internal_set_resolve_container_id(bool value) { inline void XtcpConfig::clear_enrich_container_enable() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.enrich_container_enable_ = false; - ClearHasBit(_impl_._has_bits_[2], 0x00100000U); + ClearHasBit(_impl_._has_bits_[2], 0x00400000U); } inline bool XtcpConfig::enrich_container_enable() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.enrich_container_enable) @@ -14273,7 +14345,7 @@ inline bool XtcpConfig::enrich_container_enable() const { } inline void XtcpConfig::set_enrich_container_enable(bool value) { _internal_set_enrich_container_enable(value); - SetHasBit(_impl_._has_bits_[2], 0x00100000U); + SetHasBit(_impl_._has_bits_[2], 0x00400000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.enrich_container_enable) } inline bool XtcpConfig::_internal_enrich_container_enable() const { @@ -14353,7 +14425,7 @@ inline void XtcpConfig::set_allocated_docker_socket_path(::std::string* PROTOBUF inline void XtcpConfig::clear_enrich_lldp_enable() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.enrich_lldp_enable_ = false; - ClearHasBit(_impl_._has_bits_[2], 0x00200000U); + ClearHasBit(_impl_._has_bits_[2], 0x00800000U); } inline bool XtcpConfig::enrich_lldp_enable() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.enrich_lldp_enable) @@ -14361,7 +14433,7 @@ inline bool XtcpConfig::enrich_lldp_enable() const { } inline void XtcpConfig::set_enrich_lldp_enable(bool value) { _internal_set_enrich_lldp_enable(value); - SetHasBit(_impl_._has_bits_[2], 0x00200000U); + SetHasBit(_impl_._has_bits_[2], 0x00800000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.enrich_lldp_enable) } inline bool XtcpConfig::_internal_enrich_lldp_enable() const { @@ -14505,7 +14577,7 @@ inline void XtcpConfig::set_allocated_lldpd_version_hint(::std::string* PROTOBUF inline void XtcpConfig::clear_enrich_nic_enable() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.enrich_nic_enable_ = false; - ClearHasBit(_impl_._has_bits_[2], 0x00400000U); + ClearHasBit(_impl_._has_bits_[2], 0x01000000U); } inline bool XtcpConfig::enrich_nic_enable() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.enrich_nic_enable) @@ -14513,7 +14585,7 @@ inline bool XtcpConfig::enrich_nic_enable() const { } inline void XtcpConfig::set_enrich_nic_enable(bool value) { _internal_set_enrich_nic_enable(value); - SetHasBit(_impl_._has_bits_[2], 0x00400000U); + SetHasBit(_impl_._has_bits_[2], 0x01000000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.enrich_nic_enable) } inline bool XtcpConfig::_internal_enrich_nic_enable() const { @@ -14529,7 +14601,7 @@ inline void XtcpConfig::_internal_set_enrich_nic_enable(bool value) { inline void XtcpConfig::clear_uplink_count() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.uplink_count_ = 0u; - ClearHasBit(_impl_._has_bits_[2], 0x00800000U); + ClearHasBit(_impl_._has_bits_[2], 0x02000000U); } inline ::uint32_t XtcpConfig::uplink_count() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.uplink_count) @@ -14537,7 +14609,7 @@ inline ::uint32_t XtcpConfig::uplink_count() const { } inline void XtcpConfig::set_uplink_count(::uint32_t value) { _internal_set_uplink_count(value); - SetHasBit(_impl_._has_bits_[2], 0x00800000U); + SetHasBit(_impl_._has_bits_[2], 0x02000000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.uplink_count) } inline ::uint32_t XtcpConfig::_internal_uplink_count() const { @@ -14625,7 +14697,7 @@ XtcpConfig::_internal_mutable_uplink_interfaces() { inline void XtcpConfig::clear_populate_nsid() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.populate_nsid_ = false; - ClearHasBit(_impl_._has_bits_[2], 0x01000000U); + ClearHasBit(_impl_._has_bits_[2], 0x04000000U); } inline bool XtcpConfig::populate_nsid() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.populate_nsid) @@ -14633,7 +14705,7 @@ inline bool XtcpConfig::populate_nsid() const { } inline void XtcpConfig::set_populate_nsid(bool value) { _internal_set_populate_nsid(value); - SetHasBit(_impl_._has_bits_[2], 0x01000000U); + SetHasBit(_impl_._has_bits_[2], 0x04000000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.populate_nsid) } inline bool XtcpConfig::_internal_populate_nsid() const { @@ -14649,7 +14721,7 @@ inline void XtcpConfig::_internal_set_populate_nsid(bool value) { inline void XtcpConfig::clear_enrich_asn_enable() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.enrich_asn_enable_ = false; - ClearHasBit(_impl_._has_bits_[2], 0x02000000U); + ClearHasBit(_impl_._has_bits_[2], 0x08000000U); } inline bool XtcpConfig::enrich_asn_enable() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.enrich_asn_enable) @@ -14657,7 +14729,7 @@ inline bool XtcpConfig::enrich_asn_enable() const { } inline void XtcpConfig::set_enrich_asn_enable(bool value) { _internal_set_enrich_asn_enable(value); - SetHasBit(_impl_._has_bits_[2], 0x02000000U); + SetHasBit(_impl_._has_bits_[2], 0x08000000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.enrich_asn_enable) } inline bool XtcpConfig::_internal_enrich_asn_enable() const { @@ -14958,7 +15030,7 @@ inline void XtcpConfig::set_allocated_ipmeta_cache_path(::std::string* PROTOBUF_ inline void XtcpConfig::clear_enrich_locality_enable() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.enrich_locality_enable_ = false; - ClearHasBit(_impl_._has_bits_[2], 0x04000000U); + ClearHasBit(_impl_._has_bits_[2], 0x10000000U); } inline bool XtcpConfig::enrich_locality_enable() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.enrich_locality_enable) @@ -14966,7 +15038,7 @@ inline bool XtcpConfig::enrich_locality_enable() const { } inline void XtcpConfig::set_enrich_locality_enable(bool value) { _internal_set_enrich_locality_enable(value); - SetHasBit(_impl_._has_bits_[2], 0x04000000U); + SetHasBit(_impl_._has_bits_[2], 0x10000000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.enrich_locality_enable) } inline bool XtcpConfig::_internal_enrich_locality_enable() const { @@ -15075,7 +15147,7 @@ inline void XtcpConfig::set_allocated_locality_refresh_interval(::google::protob inline void XtcpConfig::clear_locality_events_enable() { ::google::protobuf::internal::TSanWrite(&_impl_); _impl_.locality_events_enable_ = false; - ClearHasBit(_impl_._has_bits_[2], 0x08000000U); + ClearHasBit(_impl_._has_bits_[2], 0x20000000U); } inline bool XtcpConfig::locality_events_enable() const { // @@protoc_insertion_point(field_get:xtcp_config.v1.XtcpConfig.locality_events_enable) @@ -15083,7 +15155,7 @@ inline bool XtcpConfig::locality_events_enable() const { } inline void XtcpConfig::set_locality_events_enable(bool value) { _internal_set_locality_events_enable(value); - SetHasBit(_impl_._has_bits_[2], 0x08000000U); + SetHasBit(_impl_._has_bits_[2], 0x20000000U); // @@protoc_insertion_point(field_set:xtcp_config.v1.XtcpConfig.locality_events_enable) } inline bool XtcpConfig::_internal_locality_events_enable() const { diff --git a/gen/dart/xtcp_config/v1/xtcp_config.pb.dart b/gen/dart/xtcp_config/v1/xtcp_config.pb.dart index bfdab47f6..110d7aab5 100644 --- a/gen/dart/xtcp_config/v1/xtcp_config.pb.dart +++ b/gen/dart/xtcp_config/v1/xtcp_config.pb.dart @@ -1863,6 +1863,8 @@ class XtcpConfig extends $pb.GeneratedMessage { $core.int? s3FlushThresholdJitterPct, $core.int? s3UploadMaxAttempts, $1.Duration? s3UploadBackoffCap, + $core.int? s3QueueDepth, + $fixnum.Int64? s3MaxBufferedBytes, $core.String? hostname, $core.String? location, $core.String? label, @@ -1975,6 +1977,9 @@ class XtcpConfig extends $pb.GeneratedMessage { result.s3UploadMaxAttempts = s3UploadMaxAttempts; if (s3UploadBackoffCap != null) result.s3UploadBackoffCap = s3UploadBackoffCap; + if (s3QueueDepth != null) result.s3QueueDepth = s3QueueDepth; + if (s3MaxBufferedBytes != null) + result.s3MaxBufferedBytes = s3MaxBufferedBytes; if (hostname != null) result.hostname = hostname; if (location != null) result.location = location; if (label != null) result.label = label; @@ -2135,6 +2140,11 @@ class XtcpConfig extends $pb.GeneratedMessage { fieldType: $pb.PbFieldType.OU3) ..aOM<$1.Duration>(115, _omitFieldNames ? '' : 's3UploadBackoffCap', subBuilder: $1.Duration.$_createMessage) + ..aI(116, _omitFieldNames ? '' : 's3QueueDepth', + fieldType: $pb.PbFieldType.OU3) + ..a<$fixnum.Int64>( + 117, _omitFieldNames ? '' : 's3MaxBufferedBytes', $pb.PbFieldType.OU6, + defaultOrMaker: $fixnum.Int64.ZERO) ..aOS(130, _omitFieldNames ? '' : 'hostname') ..aOS(131, _omitFieldNames ? '' : 'location') ..aOS(132, _omitFieldNames ? '' : 'label') @@ -2947,17 +2957,41 @@ class XtcpConfig extends $pb.GeneratedMessage { @$pb.TagNumber(115) $1.Duration ensureS3UploadBackoffCap() => $_ensure(55); + /// Envelopes queued between the poller and the S3 worker. Past this depth + /// the oldest is dropped (xtcp_s3_dropped_total{reason="queue_depth"}), so + /// an S3 outage never blocks polling. 0 = 16. + @$pb.TagNumber(116) + $core.int get s3QueueDepth => $_getIZ(56); + @$pb.TagNumber(116) + set s3QueueDepth($core.int value) => $_setUnsignedInt32(56, value); + @$pb.TagNumber(116) + $core.bool hasS3QueueDepth() => $_has(56); + @$pb.TagNumber(116) + void clearS3QueueDepth() => $_clearField(116); + + /// Marshaled envelope bytes the S3 queue may hold. Past this the oldest + /// envelope is dropped (reason="buffered_bytes"); the newest is always + /// kept. 0 = no byte bound (the depth bound alone). Flag default 64 MiB. + @$pb.TagNumber(117) + $fixnum.Int64 get s3MaxBufferedBytes => $_getI64(57); + @$pb.TagNumber(117) + set s3MaxBufferedBytes($fixnum.Int64 value) => $_setInt64(57, value); + @$pb.TagNumber(117) + $core.bool hasS3MaxBufferedBytes() => $_has(57); + @$pb.TagNumber(117) + void clearS3MaxBufferedBytes() => $_clearField(117); + /// Hostname override. When empty the daemon uses os.Hostname(); set this to /// stamp an explicit hostname on records — required in containers, where /// os.Hostname() returns the container id, not the host. Set via -hostname /// flag or XTCP_HOSTNAME env (NOT HOSTNAME, which Docker sets to the /// container id). @$pb.TagNumber(130) - $core.String get hostname => $_getSZ(56); + $core.String get hostname => $_getSZ(58); @$pb.TagNumber(130) - set hostname($core.String value) => $_setString(56, value); + set hostname($core.String value) => $_setString(58, value); @$pb.TagNumber(130) - $core.bool hasHostname() => $_has(56); + $core.bool hasHostname() => $_has(58); @$pb.TagNumber(130) void clearHostname() => $_clearField(130); @@ -2965,31 +2999,31 @@ class XtcpConfig extends $pb.GeneratedMessage { /// region, site, …). Generic; stamped on every record's `location` field. /// Set via -location flag or LOCATION env. @$pb.TagNumber(131) - $core.String get location => $_getSZ(57); + $core.String get location => $_getSZ(59); @$pb.TagNumber(131) - set location($core.String value) => $_setString(57, value); + set location($core.String value) => $_setString(59, value); @$pb.TagNumber(131) - $core.bool hasLocation() => $_has(57); + $core.bool hasLocation() => $_has(59); @$pb.TagNumber(131) void clearLocation() => $_clearField(131); /// Label applied to the protobuf @$pb.TagNumber(132) - $core.String get label => $_getSZ(58); + $core.String get label => $_getSZ(60); @$pb.TagNumber(132) - set label($core.String value) => $_setString(58, value); + set label($core.String value) => $_setString(60, value); @$pb.TagNumber(132) - $core.bool hasLabel() => $_has(58); + $core.bool hasLabel() => $_has(60); @$pb.TagNumber(132) void clearLabel() => $_clearField(132); /// Tag applied to the protobuf @$pb.TagNumber(133) - $core.String get tag => $_getSZ(59); + $core.String get tag => $_getSZ(61); @$pb.TagNumber(133) - set tag($core.String value) => $_setString(59, value); + set tag($core.String value) => $_setString(61, value); @$pb.TagNumber(133) - $core.bool hasTag() => $_has(59); + $core.bool hasTag() => $_has(61); @$pb.TagNumber(133) void clearTag() => $_clearField(133); @@ -2998,11 +3032,11 @@ class XtcpConfig extends $pb.GeneratedMessage { /// vars, not a user flag; informational only (debugging which binary produced a /// row). See XtcpFlatRecord.daemon_version. @$pb.TagNumber(134) - $core.String get daemonVersion => $_getSZ(60); + $core.String get daemonVersion => $_getSZ(62); @$pb.TagNumber(134) - set daemonVersion($core.String value) => $_setString(60, value); + set daemonVersion($core.String value) => $_setString(62, value); @$pb.TagNumber(134) - $core.bool hasDaemonVersion() => $_has(60); + $core.bool hasDaemonVersion() => $_has(62); @$pb.TagNumber(134) void clearDaemonVersion() => $_clearField(134); @@ -3012,22 +3046,22 @@ class XtcpConfig extends $pb.GeneratedMessage { /// analogue of the host nftables TTL clamp. Set via -ipv4Ttl / IPV4_TTL. /// (cf. prometheus/exporter-toolkit#396.) @$pb.TagNumber(150) - $core.int get ipv4Ttl => $_getIZ(61); + $core.int get ipv4Ttl => $_getIZ(63); @$pb.TagNumber(150) - set ipv4Ttl($core.int value) => $_setUnsignedInt32(61, value); + set ipv4Ttl($core.int value) => $_setUnsignedInt32(63, value); @$pb.TagNumber(150) - $core.bool hasIpv4Ttl() => $_has(61); + $core.bool hasIpv4Ttl() => $_has(63); @$pb.TagNumber(150) void clearIpv4Ttl() => $_clearField(150); /// Outgoing IPv6 unicast hop limit for xtcp2's own TCP listeners. 0 = kernel /// default. Same intent as ipv4_ttl. Set via -ipv6HopLimit / IPV6_HOP_LIMIT. @$pb.TagNumber(151) - $core.int get ipv6HopLimit => $_getIZ(62); + $core.int get ipv6HopLimit => $_getIZ(64); @$pb.TagNumber(151) - set ipv6HopLimit($core.int value) => $_setUnsignedInt32(62, value); + set ipv6HopLimit($core.int value) => $_setUnsignedInt32(64, value); @$pb.TagNumber(151) - $core.bool hasIpv6HopLimit() => $_has(62); + $core.bool hasIpv6HopLimit() => $_has(64); @$pb.TagNumber(151) void clearIpv6HopLimit() => $_clearField(151); @@ -3037,28 +3071,28 @@ class XtcpConfig extends $pb.GeneratedMessage { /// the read credential: it reaches only methods marked ACCESS_READ. /// Secrets are redacted from every ConfigService response. @$pb.TagNumber(152) - ListenerAuth get listenerAuth => $_getN(63); + ListenerAuth get listenerAuth => $_getN(65); @$pb.TagNumber(152) set listenerAuth(ListenerAuth value) => $_setField(152, value); @$pb.TagNumber(152) - $core.bool hasListenerAuth() => $_has(63); + $core.bool hasListenerAuth() => $_has(65); @$pb.TagNumber(152) void clearListenerAuth() => $_clearField(152); @$pb.TagNumber(152) - ListenerAuth ensureListenerAuth() => $_ensure(63); + ListenerAuth ensureListenerAuth() => $_ensure(65); /// Prometheus / pprof / health listener endpoint. When unset, the daemon /// keeps the legacy TCP behavior from -promListen / PROM_LISTEN. @$pb.TagNumber(153) - ListenerEndpoint get prometheusListener => $_getN(64); + ListenerEndpoint get prometheusListener => $_getN(66); @$pb.TagNumber(153) set prometheusListener(ListenerEndpoint value) => $_setField(153, value); @$pb.TagNumber(153) - $core.bool hasPrometheusListener() => $_has(64); + $core.bool hasPrometheusListener() => $_has(66); @$pb.TagNumber(153) void clearPrometheusListener() => $_clearField(153); @$pb.TagNumber(153) - ListenerEndpoint ensurePrometheusListener() => $_ensure(64); + ListenerEndpoint ensurePrometheusListener() => $_ensure(66); /// Admin credential. When set, ACCESS_ADMIN methods (ConfigService.Set, the /// runtime triggers, link-monitor rebaseline, /debug/pprof) require it, and @@ -3066,47 +3100,47 @@ class XtcpConfig extends $pb.GeneratedMessage { /// listener_auth behaviour. Secrets are redacted from ConfigService /// responses. @$pb.TagNumber(154) - ListenerAuth get adminListenerAuth => $_getN(65); + ListenerAuth get adminListenerAuth => $_getN(67); @$pb.TagNumber(154) set adminListenerAuth(ListenerAuth value) => $_setField(154, value); @$pb.TagNumber(154) - $core.bool hasAdminListenerAuth() => $_has(65); + $core.bool hasAdminListenerAuth() => $_has(67); @$pb.TagNumber(154) void clearAdminListenerAuth() => $_clearField(154); @$pb.TagNumber(154) - ListenerAuth ensureAdminListenerAuth() => $_ensure(65); + ListenerAuth ensureAdminListenerAuth() => $_ensure(67); /// GRPC listening port /// Legacy: the gRPC port when -grpcPort (or GRPC_PORT) expands into a /// listener. 0 when the listener model configures gRPC. @$pb.TagNumber(160) - $core.int get grpcPort => $_getIZ(66); + $core.int get grpcPort => $_getIZ(68); @$pb.TagNumber(160) - set grpcPort($core.int value) => $_setUnsignedInt32(66, value); + set grpcPort($core.int value) => $_setUnsignedInt32(68, value); @$pb.TagNumber(160) - $core.bool hasGrpcPort() => $_has(66); + $core.bool hasGrpcPort() => $_has(68); @$pb.TagNumber(160) void clearGrpcPort() => $_clearField(160); /// gRPC listener endpoint. When unset, the daemon keeps the legacy TCP /// behavior derived from grpc_port. @$pb.TagNumber(161) - ListenerEndpoint get grpcListener => $_getN(67); + ListenerEndpoint get grpcListener => $_getN(69); @$pb.TagNumber(161) set grpcListener(ListenerEndpoint value) => $_setField(161, value); @$pb.TagNumber(161) - $core.bool hasGrpcListener() => $_has(67); + $core.bool hasGrpcListener() => $_has(69); @$pb.TagNumber(161) void clearGrpcListener() => $_clearField(161); @$pb.TagNumber(161) - ListenerEndpoint ensureGrpcListener() => $_ensure(67); + ListenerEndpoint ensureGrpcListener() => $_ensure(69); /// Every listener the daemon opens (-listener, LISTENERS). Empty means the /// default: one UDS, /run/xtcp2/metrics.sock mode 0600, raw bearer auth, /// serving health, metrics and metrics-detail. The legacy -promListen*, /// -grpc* and -linkMonitorIPListen flags expand into entries here. @$pb.TagNumber(162) - $pb.PbList get listeners => $_getList(68); + $pb.PbList get listeners => $_getList(70); /// Pyroscope continuous-profiling server URL (e.g. /// http://127.0.0.1:4040). When set, the daemon streams CPU, @@ -3116,11 +3150,11 @@ class XtcpConfig extends $pb.GeneratedMessage { /// Grafana Cloud Pyroscope) and point xtcp2 at it for live profile /// data without restarts. @$pb.TagNumber(170) - $core.String get pyroscopeUrl => $_getSZ(69); + $core.String get pyroscopeUrl => $_getSZ(71); @$pb.TagNumber(170) - set pyroscopeUrl($core.String value) => $_setString(69, value); + set pyroscopeUrl($core.String value) => $_setString(71, value); @$pb.TagNumber(170) - $core.bool hasPyroscopeUrl() => $_has(69); + $core.bool hasPyroscopeUrl() => $_has(71); @$pb.TagNumber(170) void clearPyroscopeUrl() => $_clearField(170); @@ -3129,34 +3163,34 @@ class XtcpConfig extends $pb.GeneratedMessage { /// Set per fleet/role for multi-host environments /// (e.g. "xtcp2.prod.iad", "xtcp2.staging.fra"). @$pb.TagNumber(171) - $core.String get pyroscopeAppName => $_getSZ(70); + $core.String get pyroscopeAppName => $_getSZ(72); @$pb.TagNumber(171) - set pyroscopeAppName($core.String value) => $_setString(70, value); + set pyroscopeAppName($core.String value) => $_setString(72, value); @$pb.TagNumber(171) - $core.bool hasPyroscopeAppName() => $_has(70); + $core.bool hasPyroscopeAppName() => $_has(72); @$pb.TagNumber(171) void clearPyroscopeAppName() => $_clearField(171); /// CPU profile sampling rate in Hz. Default 100. The Pyroscope /// agent uses this to call runtime.SetCPUProfileRate at startup. @$pb.TagNumber(172) - $core.int get pyroscopeSampleHz => $_getIZ(71); + $core.int get pyroscopeSampleHz => $_getIZ(73); @$pb.TagNumber(172) - set pyroscopeSampleHz($core.int value) => $_setUnsignedInt32(71, value); + set pyroscopeSampleHz($core.int value) => $_setUnsignedInt32(73, value); @$pb.TagNumber(172) - $core.bool hasPyroscopeSampleHz() => $_has(71); + $core.bool hasPyroscopeSampleHz() => $_has(73); @$pb.TagNumber(172) void clearPyroscopeSampleHz() => $_clearField(172); /// Profile upload interval (seconds between batched profile /// pushes). Default 15 s. @$pb.TagNumber(173) - $core.int get pyroscopeUploadIntervalSec => $_getIZ(72); + $core.int get pyroscopeUploadIntervalSec => $_getIZ(74); @$pb.TagNumber(173) set pyroscopeUploadIntervalSec($core.int value) => - $_setUnsignedInt32(72, value); + $_setUnsignedInt32(74, value); @$pb.TagNumber(173) - $core.bool hasPyroscopeUploadIntervalSec() => $_has(72); + $core.bool hasPyroscopeUploadIntervalSec() => $_has(74); @$pb.TagNumber(173) void clearPyroscopeUploadIntervalSec() => $_clearField(173); @@ -3167,11 +3201,11 @@ class XtcpConfig extends $pb.GeneratedMessage { /// CONTAINER_ID_RESOLVE env. Needs /sys/fs/cgroup readable (mount it and run /// --cgroupns=host in a container). @$pb.TagNumber(200) - $core.bool get resolveContainerId => $_getBF(73); + $core.bool get resolveContainerId => $_getBF(75); @$pb.TagNumber(200) - set resolveContainerId($core.bool value) => $_setBool(73, value); + set resolveContainerId($core.bool value) => $_setBool(75, value); @$pb.TagNumber(200) - $core.bool hasResolveContainerId() => $_has(73); + $core.bool hasResolveContainerId() => $_has(75); @$pb.TagNumber(200) void clearResolveContainerId() => $_clearField(200); @@ -3179,21 +3213,21 @@ class XtcpConfig extends $pb.GeneratedMessage { /// by joining the socket's owning netns inode against the Docker Engine API /// index over docker_socket_path. Default false. @$pb.TagNumber(201) - $core.bool get enrichContainerEnable => $_getBF(74); + $core.bool get enrichContainerEnable => $_getBF(76); @$pb.TagNumber(201) - set enrichContainerEnable($core.bool value) => $_setBool(74, value); + set enrichContainerEnable($core.bool value) => $_setBool(76, value); @$pb.TagNumber(201) - $core.bool hasEnrichContainerEnable() => $_has(74); + $core.bool hasEnrichContainerEnable() => $_has(76); @$pb.TagNumber(201) void clearEnrichContainerEnable() => $_clearField(201); /// Docker Engine API unix socket. Default "/run/docker.sock". @$pb.TagNumber(202) - $core.String get dockerSocketPath => $_getSZ(75); + $core.String get dockerSocketPath => $_getSZ(77); @$pb.TagNumber(202) - set dockerSocketPath($core.String value) => $_setString(75, value); + set dockerSocketPath($core.String value) => $_setString(77, value); @$pb.TagNumber(202) - $core.bool hasDockerSocketPath() => $_has(75); + $core.bool hasDockerSocketPath() => $_has(77); @$pb.TagNumber(202) void clearDockerSocketPath() => $_clearField(202); @@ -3201,32 +3235,32 @@ class XtcpConfig extends $pb.GeneratedMessage { /// Enrich per-uplink LLDP neighbor labels by reading the lldpd control socket /// (lldpd_socket_path) once at startup. Default false. @$pb.TagNumber(210) - $core.bool get enrichLldpEnable => $_getBF(76); + $core.bool get enrichLldpEnable => $_getBF(78); @$pb.TagNumber(210) - set enrichLldpEnable($core.bool value) => $_setBool(76, value); + set enrichLldpEnable($core.bool value) => $_setBool(78, value); @$pb.TagNumber(210) - $core.bool hasEnrichLldpEnable() => $_has(76); + $core.bool hasEnrichLldpEnable() => $_has(78); @$pb.TagNumber(210) void clearEnrichLldpEnable() => $_clearField(210); /// lldpd control socket. Default "/run/lldpd.socket". @$pb.TagNumber(211) - $core.String get lldpdSocketPath => $_getSZ(77); + $core.String get lldpdSocketPath => $_getSZ(79); @$pb.TagNumber(211) - set lldpdSocketPath($core.String value) => $_setString(77, value); + set lldpdSocketPath($core.String value) => $_setString(79, value); @$pb.TagNumber(211) - $core.bool hasLldpdSocketPath() => $_has(77); + $core.bool hasLldpdSocketPath() => $_has(79); @$pb.TagNumber(211) void clearLldpdSocketPath() => $_clearField(211); /// Optional lldpd version hint ("1.0.13"/"1.0.18") selecting the struct-layout /// descriptor for the wire parser. Empty = auto-detect. Default "". @$pb.TagNumber(212) - $core.String get lldpdVersionHint => $_getSZ(78); + $core.String get lldpdVersionHint => $_getSZ(80); @$pb.TagNumber(212) - set lldpdVersionHint($core.String value) => $_setString(78, value); + set lldpdVersionHint($core.String value) => $_setString(80, value); @$pb.TagNumber(212) - $core.bool hasLldpdVersionHint() => $_has(78); + $core.bool hasLldpdVersionHint() => $_has(80); @$pb.TagNumber(212) void clearLldpdVersionHint() => $_clearField(212); @@ -3234,38 +3268,38 @@ class XtcpConfig extends $pb.GeneratedMessage { /// Enrich per-uplink NIC labels (driver/model/pci/speed/firmware) from sysfs + /// the ethtool ioctl once at startup. Default false. @$pb.TagNumber(220) - $core.bool get enrichNicEnable => $_getBF(79); + $core.bool get enrichNicEnable => $_getBF(81); @$pb.TagNumber(220) - set enrichNicEnable($core.bool value) => $_setBool(79, value); + set enrichNicEnable($core.bool value) => $_setBool(81, value); @$pb.TagNumber(220) - $core.bool hasEnrichNicEnable() => $_has(79); + $core.bool hasEnrichNicEnable() => $_has(81); @$pb.TagNumber(220) void clearEnrichNicEnable() => $_clearField(220); /// Number of host uplink slots to populate (dual-homed hosts = 2). Default 2. @$pb.TagNumber(221) - $core.int get uplinkCount => $_getIZ(80); + $core.int get uplinkCount => $_getIZ(82); @$pb.TagNumber(221) - set uplinkCount($core.int value) => $_setUnsignedInt32(80, value); + set uplinkCount($core.int value) => $_setUnsignedInt32(82, value); @$pb.TagNumber(221) - $core.bool hasUplinkCount() => $_has(80); + $core.bool hasUplinkCount() => $_has(82); @$pb.TagNumber(221) void clearUplinkCount() => $_clearField(221); /// Explicit uplink interface names, slot order. Empty = auto-detect from the /// default IPv4/IPv6 routes. @$pb.TagNumber(222) - $pb.PbList<$core.String> get uplinkInterfaces => $_getList(81); + $pb.PbList<$core.String> get uplinkInterfaces => $_getList(83); /// -- nsid (230-239) /// Populate nsid (record field 32) best-effort via RTM_GETNSID. Usually 0 for /// Docker/containerd namespaces. Default false. @$pb.TagNumber(230) - $core.bool get populateNsid => $_getBF(82); + $core.bool get populateNsid => $_getBF(84); @$pb.TagNumber(230) - set populateNsid($core.bool value) => $_setBool(82, value); + set populateNsid($core.bool value) => $_setBool(84, value); @$pb.TagNumber(230) - $core.bool hasPopulateNsid() => $_has(82); + $core.bool hasPopulateNsid() => $_has(84); @$pb.TagNumber(230) void clearPopulateNsid() => $_clearField(230); @@ -3276,47 +3310,47 @@ class XtcpConfig extends $pb.GeneratedMessage { /// enabled but asn_db_path is missing/unreadable, xtcp2 logs, bumps a counter, /// and leaves both columns empty. Default false. @$pb.TagNumber(240) - $core.bool get enrichAsnEnable => $_getBF(83); + $core.bool get enrichAsnEnable => $_getBF(85); @$pb.TagNumber(240) - set enrichAsnEnable($core.bool value) => $_setBool(83, value); + set enrichAsnEnable($core.bool value) => $_setBool(85, value); @$pb.TagNumber(240) - $core.bool hasEnrichAsnEnable() => $_has(83); + $core.bool hasEnrichAsnEnable() => $_has(85); @$pb.TagNumber(240) void clearEnrichAsnEnable() => $_clearField(240); /// Path to the ipfeed-collector Parquet artifact (prefix -> {asn, /// network_owner}). Default "". @$pb.TagNumber(241) - $core.String get asnDbPath => $_getSZ(84); + $core.String get asnDbPath => $_getSZ(86); @$pb.TagNumber(241) - set asnDbPath($core.String value) => $_setString(84, value); + set asnDbPath($core.String value) => $_setString(86, value); @$pb.TagNumber(241) - $core.bool hasAsnDbPath() => $_has(84); + $core.bool hasAsnDbPath() => $_has(86); @$pb.TagNumber(241) void clearAsnDbPath() => $_clearField(241); /// How often to reload asn_db_path in the background so a refreshed artifact /// is picked up without a restart. 0 = load once at startup, never reload. @$pb.TagNumber(242) - $1.Duration get asnRefreshInterval => $_getN(85); + $1.Duration get asnRefreshInterval => $_getN(87); @$pb.TagNumber(242) set asnRefreshInterval($1.Duration value) => $_setField(242, value); @$pb.TagNumber(242) - $core.bool hasAsnRefreshInterval() => $_has(85); + $core.bool hasAsnRefreshInterval() => $_has(87); @$pb.TagNumber(242) void clearAsnRefreshInterval() => $_clearField(242); @$pb.TagNumber(242) - $1.Duration ensureAsnRefreshInterval() => $_ensure(85); + $1.Duration ensureAsnRefreshInterval() => $_ensure(87); /// Optional zstd-compressed lookup artifact baked into the image and used at /// startup when no fresher writable cache is available. Default is supplied /// by the daemon CLI as /share/xtcp2/ipmeta/bootstrap.lookup.parquet.zst. @$pb.TagNumber(243) - $core.String get ipmetaBootstrapPath => $_getSZ(86); + $core.String get ipmetaBootstrapPath => $_getSZ(88); @$pb.TagNumber(243) - set ipmetaBootstrapPath($core.String value) => $_setString(86, value); + set ipmetaBootstrapPath($core.String value) => $_setString(88, value); @$pb.TagNumber(243) - $core.bool hasIpmetaBootstrapPath() => $_has(86); + $core.bool hasIpmetaBootstrapPath() => $_has(88); @$pb.TagNumber(243) void clearIpmetaBootstrapPath() => $_clearField(243); @@ -3325,11 +3359,11 @@ class XtcpConfig extends $pb.GeneratedMessage { /// The companion previous.lookup.parquet.zst in the same directory is kept /// as a recovery fallback. @$pb.TagNumber(244) - $core.String get ipmetaCachePath => $_getSZ(87); + $core.String get ipmetaCachePath => $_getSZ(89); @$pb.TagNumber(244) - set ipmetaCachePath($core.String value) => $_setString(87, value); + set ipmetaCachePath($core.String value) => $_setString(89, value); @$pb.TagNumber(244) - $core.bool hasIpmetaCachePath() => $_has(87); + $core.bool hasIpmetaCachePath() => $_has(89); @$pb.TagNumber(244) void clearIpmetaCachePath() => $_clearField(244); @@ -3342,11 +3376,11 @@ class XtcpConfig extends $pb.GeneratedMessage { /// Non-fatal: a per-namespace discovery failure just leaves that namespace's /// sockets unclassified (and is retried with backoff). Default false. @$pb.TagNumber(245) - $core.bool get enrichLocalityEnable => $_getBF(88); + $core.bool get enrichLocalityEnable => $_getBF(90); @$pb.TagNumber(245) - set enrichLocalityEnable($core.bool value) => $_setBool(88, value); + set enrichLocalityEnable($core.bool value) => $_setBool(90, value); @$pb.TagNumber(245) - $core.bool hasEnrichLocalityEnable() => $_has(88); + $core.bool hasEnrichLocalityEnable() => $_has(90); @$pb.TagNumber(245) void clearEnrichLocalityEnable() => $_clearField(245); @@ -3355,15 +3389,15 @@ class XtcpConfig extends $pb.GeneratedMessage { /// namespaces are always snapshotted on the next reconcile regardless. 0 = /// discover once per namespace, never refresh. Daemon default 60s. @$pb.TagNumber(246) - $1.Duration get localityRefreshInterval => $_getN(89); + $1.Duration get localityRefreshInterval => $_getN(91); @$pb.TagNumber(246) set localityRefreshInterval($1.Duration value) => $_setField(246, value); @$pb.TagNumber(246) - $core.bool hasLocalityRefreshInterval() => $_has(89); + $core.bool hasLocalityRefreshInterval() => $_has(91); @$pb.TagNumber(246) void clearLocalityRefreshInterval() => $_clearField(246); @$pb.TagNumber(246) - $1.Duration ensureLocalityRefreshInterval() => $_ensure(89); + $1.Duration ensureLocalityRefreshInterval() => $_ensure(91); /// Keep each namespace's locality snapshot current from rtnetlink /// notifications (link, address and route changes) instead of waiting for @@ -3371,25 +3405,25 @@ class XtcpConfig extends $pb.GeneratedMessage { /// locality_refresh_interval then becomes the backstop, a resync taken /// after that long. Needs enrich_locality_enable. Default false. @$pb.TagNumber(247) - $core.bool get localityEventsEnable => $_getBF(90); + $core.bool get localityEventsEnable => $_getBF(92); @$pb.TagNumber(247) - set localityEventsEnable($core.bool value) => $_setBool(90, value); + set localityEventsEnable($core.bool value) => $_setBool(92, value); @$pb.TagNumber(247) - $core.bool hasLocalityEventsEnable() => $_has(90); + $core.bool hasLocalityEventsEnable() => $_has(92); @$pb.TagNumber(247) void clearLocalityEventsEnable() => $_clearField(247); /// Physical link monitor; see LinkMonitorConfig. @$pb.TagNumber(250) - LinkMonitorConfig get linkMonitor => $_getN(91); + LinkMonitorConfig get linkMonitor => $_getN(93); @$pb.TagNumber(250) set linkMonitor(LinkMonitorConfig value) => $_setField(250, value); @$pb.TagNumber(250) - $core.bool hasLinkMonitor() => $_has(91); + $core.bool hasLinkMonitor() => $_has(93); @$pb.TagNumber(250) void clearLinkMonitor() => $_clearField(250); @$pb.TagNumber(250) - LinkMonitorConfig ensureLinkMonitor() => $_ensure(91); + LinkMonitorConfig ensureLinkMonitor() => $_ensure(93); } class EnabledDeserializers extends $pb.GeneratedMessage { diff --git a/gen/dart/xtcp_config/v1/xtcp_config.pbjson.dart b/gen/dart/xtcp_config/v1/xtcp_config.pbjson.dart index 8bff2e73a..f23288a70 100644 --- a/gen/dart/xtcp_config/v1/xtcp_config.pbjson.dart +++ b/gen/dart/xtcp_config/v1/xtcp_config.pbjson.dart @@ -1171,6 +1171,22 @@ const XtcpConfig$json = { '8': {}, '10': 's3UploadBackoffCap' }, + { + '1': 's3_queue_depth', + '3': 116, + '4': 1, + '5': 13, + '8': {}, + '10': 's3QueueDepth' + }, + { + '1': 's3_max_buffered_bytes', + '3': 117, + '4': 1, + '5': 4, + '8': {}, + '10': 's3MaxBufferedBytes' + }, {'1': 'hostname', '3': 130, '4': 1, '5': 9, '8': {}, '10': 'hostname'}, {'1': 'location', '3': 131, '4': 1, '5': 9, '8': {}, '10': 'location'}, {'1': 'label', '3': 132, '4': 1, '5': 9, '8': {}, '10': 'label'}, @@ -1467,46 +1483,48 @@ final $typed_data.Uint8List xtcpConfigDescriptor = $convert.base64Decode( 'Qgq6SAfIAQAqAhhkUhlzM0ZsdXNoVGhyZXNob2xkSml0dGVyUGN0EkEKFnMzX3VwbG9hZF9tYX' 'hfYXR0ZW1wdHMYciABKA1CDLpICcgBACoEGGQoAVITczNVcGxvYWRNYXhBdHRlbXB0cxJZChVz' 'M191cGxvYWRfYmFja29mZl9jYXAYcyABKAsyGS5nb29nbGUucHJvdG9idWYuRHVyYXRpb25CC7' - 'pICMgBAKoBAjIAUhJzM1VwbG9hZEJhY2tvZmZDYXASKAoIaG9zdG5hbWUYggEgASgJQgu6SAjI' - 'AQByAxj9AVIIaG9zdG5hbWUSKAoIbG9jYXRpb24YgwEgASgJQgu6SAjIAQByAxj9AVIIbG9jYX' - 'Rpb24SIQoFbGFiZWwYhAEgASgJQgq6SAfIAQByAhgoUgVsYWJlbBIdCgN0YWcYhQEgASgJQgq6' - 'SAfIAQByAhgoUgN0YWcSMwoOZGFlbW9uX3ZlcnNpb24YhgEgASgJQgu6SAjIAQByAxj9AVINZG' - 'FlbW9uVmVyc2lvbhInCghpcHY0X3R0bBiWASABKA1CC7pICMgBACoDGP8BUgdpcHY0VHRsEjIK' - 'DmlwdjZfaG9wX2xpbWl0GJcBIAEoDUILukgIyAEAKgMY/wFSDGlwdjZIb3BMaW1pdBJKCg1saX' - 'N0ZW5lcl9hdXRoGJgBIAEoCzIcLnh0Y3BfY29uZmlnLnYxLkxpc3RlbmVyQXV0aEIGukgDyAEA' - 'UgxsaXN0ZW5lckF1dGgSWgoTcHJvbWV0aGV1c19saXN0ZW5lchiZASABKAsyIC54dGNwX2Nvbm' - 'ZpZy52MS5MaXN0ZW5lckVuZHBvaW50Qga6SAPIAQBSEnByb21ldGhldXNMaXN0ZW5lchJVChNh' - 'ZG1pbl9saXN0ZW5lcl9hdXRoGJoBIAEoCzIcLnh0Y3BfY29uZmlnLnYxLkxpc3RlbmVyQXV0aE' - 'IGukgDyAEAUhFhZG1pbkxpc3RlbmVyQXV0aBInCglncnBjX3BvcnQYoAEgASgNQgm6SAYqBBj/' - '/wNSCGdycGNQb3J0Ek4KDWdycGNfbGlzdGVuZXIYoQEgASgLMiAueHRjcF9jb25maWcudjEuTG' - 'lzdGVuZXJFbmRwb2ludEIGukgDyAEAUgxncnBjTGlzdGVuZXISQQoJbGlzdGVuZXJzGKIBIAMo' - 'CzIYLnh0Y3BfY29uZmlnLnYxLkxpc3RlbmVyQgi6SAWSAQIQEFIJbGlzdGVuZXJzEiwKDXB5cm' - '9zY29wZV91cmwYqgEgASgJQga6SAPIAQBSDHB5cm9zY29wZVVybBI1ChJweXJvc2NvcGVfYXBw' - 'X25hbWUYqwEgASgJQga6SAPIAQBSEHB5cm9zY29wZUFwcE5hbWUSNwoTcHlyb3Njb3BlX3NhbX' - 'BsZV9oehisASABKA1CBrpIA8gBAFIRcHlyb3Njb3BlU2FtcGxlSHoSSgodcHlyb3Njb3BlX3Vw' - 'bG9hZF9pbnRlcnZhbF9zZWMYrQEgASgNQga6SAPIAQBSGnB5cm9zY29wZVVwbG9hZEludGVydm' - 'FsU2VjEjkKFHJlc29sdmVfY29udGFpbmVyX2lkGMgBIAEoCEIGukgDyAEAUhJyZXNvbHZlQ29u' - 'dGFpbmVySWQSNwoXZW5yaWNoX2NvbnRhaW5lcl9lbmFibGUYyQEgASgIUhVlbnJpY2hDb250YW' - 'luZXJFbmFibGUSNwoSZG9ja2VyX3NvY2tldF9wYXRoGMoBIAEoCUIIukgFcgMY/wFSEGRvY2tl' - 'clNvY2tldFBhdGgSLQoSZW5yaWNoX2xsZHBfZW5hYmxlGNIBIAEoCFIQZW5yaWNoTGxkcEVuYW' - 'JsZRI1ChFsbGRwZF9zb2NrZXRfcGF0aBjTASABKAlCCLpIBXIDGP8BUg9sbGRwZFNvY2tldFBh' - 'dGgSNgoSbGxkcGRfdmVyc2lvbl9oaW50GNQBIAEoCUIHukgEcgIYEFIQbGxkcGRWZXJzaW9uSG' - 'ludBIrChFlbnJpY2hfbmljX2VuYWJsZRjcASABKAhSD2VucmljaE5pY0VuYWJsZRIrCgx1cGxp' - 'bmtfY291bnQY3QEgASgNQge6SAQqAhgCUgt1cGxpbmtDb3VudBI2ChF1cGxpbmtfaW50ZXJmYW' - 'NlcxjeASADKAlCCLpIBZIBAhACUhB1cGxpbmtJbnRlcmZhY2VzEiQKDXBvcHVsYXRlX25zaWQY' - '5gEgASgIUgxwb3B1bGF0ZU5zaWQSKwoRZW5yaWNoX2Fzbl9lbmFibGUY8AEgASgIUg9lbnJpY2' - 'hBc25FbmFibGUSKQoLYXNuX2RiX3BhdGgY8QEgASgJQgi6SAVyAxj/AVIJYXNuRGJQYXRoEkwK' - 'FGFzbl9yZWZyZXNoX2ludGVydmFsGPIBIAEoCzIZLmdvb2dsZS5wcm90b2J1Zi5EdXJhdGlvbl' - 'ISYXNuUmVmcmVzaEludGVydmFsEj0KFWlwbWV0YV9ib290c3RyYXBfcGF0aBjzASABKAlCCLpI' - 'BXIDGP8BUhNpcG1ldGFCb290c3RyYXBQYXRoEjUKEWlwbWV0YV9jYWNoZV9wYXRoGPQBIAEoCU' - 'IIukgFcgMY/wFSD2lwbWV0YUNhY2hlUGF0aBI1ChZlbnJpY2hfbG9jYWxpdHlfZW5hYmxlGPUB' - 'IAEoCFIUZW5yaWNoTG9jYWxpdHlFbmFibGUSVgoZbG9jYWxpdHlfcmVmcmVzaF9pbnRlcnZhbB' - 'j2ASABKAsyGS5nb29nbGUucHJvdG9idWYuRHVyYXRpb25SF2xvY2FsaXR5UmVmcmVzaEludGVy' - 'dmFsEjUKFmxvY2FsaXR5X2V2ZW50c19lbmFibGUY9wEgASgIUhRsb2NhbGl0eUV2ZW50c0VuYW' - 'JsZRJFCgxsaW5rX21vbml0b3IY+gEgASgLMiEueHRjcF9jb25maWcudjEuTGlua01vbml0b3JD' - 'b25maWdSC2xpbmtNb25pdG9yOnO6SHAabgoPWHRjcENvbmZpZy5wb2xsEjJQb2xsIHRpbWVvdX' - 'QgbXVzdCBiZSBsZXNzIHRoYW4gcG9sbCBwb2xsX2ZyZXF1ZW5jeRondGhpcy5wb2xsX2ZyZXF1' - 'ZW5jeSA+IHRoaXMucG9sbF90aW1lb3V0'); + 'pICMgBAKoBAjIAUhJzM1VwbG9hZEJhY2tvZmZDYXASMQoOczNfcXVldWVfZGVwdGgYdCABKA1C' + 'C7pICMgBACoDGIAgUgxzM1F1ZXVlRGVwdGgSOQoVczNfbWF4X2J1ZmZlcmVkX2J5dGVzGHUgAS' + 'gEQga6SAPIAQBSEnMzTWF4QnVmZmVyZWRCeXRlcxIoCghob3N0bmFtZRiCASABKAlCC7pICMgB' + 'AHIDGP0BUghob3N0bmFtZRIoCghsb2NhdGlvbhiDASABKAlCC7pICMgBAHIDGP0BUghsb2NhdG' + 'lvbhIhCgVsYWJlbBiEASABKAlCCrpIB8gBAHICGChSBWxhYmVsEh0KA3RhZxiFASABKAlCCrpI' + 'B8gBAHICGChSA3RhZxIzCg5kYWVtb25fdmVyc2lvbhiGASABKAlCC7pICMgBAHIDGP0BUg1kYW' + 'Vtb25WZXJzaW9uEicKCGlwdjRfdHRsGJYBIAEoDUILukgIyAEAKgMY/wFSB2lwdjRUdGwSMgoO' + 'aXB2Nl9ob3BfbGltaXQYlwEgASgNQgu6SAjIAQAqAxj/AVIMaXB2NkhvcExpbWl0EkoKDWxpc3' + 'RlbmVyX2F1dGgYmAEgASgLMhwueHRjcF9jb25maWcudjEuTGlzdGVuZXJBdXRoQga6SAPIAQBS' + 'DGxpc3RlbmVyQXV0aBJaChNwcm9tZXRoZXVzX2xpc3RlbmVyGJkBIAEoCzIgLnh0Y3BfY29uZm' + 'lnLnYxLkxpc3RlbmVyRW5kcG9pbnRCBrpIA8gBAFIScHJvbWV0aGV1c0xpc3RlbmVyElUKE2Fk' + 'bWluX2xpc3RlbmVyX2F1dGgYmgEgASgLMhwueHRjcF9jb25maWcudjEuTGlzdGVuZXJBdXRoQg' + 'a6SAPIAQBSEWFkbWluTGlzdGVuZXJBdXRoEicKCWdycGNfcG9ydBigASABKA1CCbpIBioEGP//' + 'A1IIZ3JwY1BvcnQSTgoNZ3JwY19saXN0ZW5lchihASABKAsyIC54dGNwX2NvbmZpZy52MS5MaX' + 'N0ZW5lckVuZHBvaW50Qga6SAPIAQBSDGdycGNMaXN0ZW5lchJBCglsaXN0ZW5lcnMYogEgAygL' + 'MhgueHRjcF9jb25maWcudjEuTGlzdGVuZXJCCLpIBZIBAhAQUglsaXN0ZW5lcnMSLAoNcHlyb3' + 'Njb3BlX3VybBiqASABKAlCBrpIA8gBAFIMcHlyb3Njb3BlVXJsEjUKEnB5cm9zY29wZV9hcHBf' + 'bmFtZRirASABKAlCBrpIA8gBAFIQcHlyb3Njb3BlQXBwTmFtZRI3ChNweXJvc2NvcGVfc2FtcG' + 'xlX2h6GKwBIAEoDUIGukgDyAEAUhFweXJvc2NvcGVTYW1wbGVIehJKCh1weXJvc2NvcGVfdXBs' + 'b2FkX2ludGVydmFsX3NlYxitASABKA1CBrpIA8gBAFIacHlyb3Njb3BlVXBsb2FkSW50ZXJ2YW' + 'xTZWMSOQoUcmVzb2x2ZV9jb250YWluZXJfaWQYyAEgASgIQga6SAPIAQBSEnJlc29sdmVDb250' + 'YWluZXJJZBI3ChdlbnJpY2hfY29udGFpbmVyX2VuYWJsZRjJASABKAhSFWVucmljaENvbnRhaW' + '5lckVuYWJsZRI3ChJkb2NrZXJfc29ja2V0X3BhdGgYygEgASgJQgi6SAVyAxj/AVIQZG9ja2Vy' + 'U29ja2V0UGF0aBItChJlbnJpY2hfbGxkcF9lbmFibGUY0gEgASgIUhBlbnJpY2hMbGRwRW5hYm' + 'xlEjUKEWxsZHBkX3NvY2tldF9wYXRoGNMBIAEoCUIIukgFcgMY/wFSD2xsZHBkU29ja2V0UGF0' + 'aBI2ChJsbGRwZF92ZXJzaW9uX2hpbnQY1AEgASgJQge6SARyAhgQUhBsbGRwZFZlcnNpb25IaW' + '50EisKEWVucmljaF9uaWNfZW5hYmxlGNwBIAEoCFIPZW5yaWNoTmljRW5hYmxlEisKDHVwbGlu' + 'a19jb3VudBjdASABKA1CB7pIBCoCGAJSC3VwbGlua0NvdW50EjYKEXVwbGlua19pbnRlcmZhY2' + 'VzGN4BIAMoCUIIukgFkgECEAJSEHVwbGlua0ludGVyZmFjZXMSJAoNcG9wdWxhdGVfbnNpZBjm' + 'ASABKAhSDHBvcHVsYXRlTnNpZBIrChFlbnJpY2hfYXNuX2VuYWJsZRjwASABKAhSD2VucmljaE' + 'FzbkVuYWJsZRIpCgthc25fZGJfcGF0aBjxASABKAlCCLpIBXIDGP8BUglhc25EYlBhdGgSTAoU' + 'YXNuX3JlZnJlc2hfaW50ZXJ2YWwY8gEgASgLMhkuZ29vZ2xlLnByb3RvYnVmLkR1cmF0aW9uUh' + 'Jhc25SZWZyZXNoSW50ZXJ2YWwSPQoVaXBtZXRhX2Jvb3RzdHJhcF9wYXRoGPMBIAEoCUIIukgF' + 'cgMY/wFSE2lwbWV0YUJvb3RzdHJhcFBhdGgSNQoRaXBtZXRhX2NhY2hlX3BhdGgY9AEgASgJQg' + 'i6SAVyAxj/AVIPaXBtZXRhQ2FjaGVQYXRoEjUKFmVucmljaF9sb2NhbGl0eV9lbmFibGUY9QEg' + 'ASgIUhRlbnJpY2hMb2NhbGl0eUVuYWJsZRJWChlsb2NhbGl0eV9yZWZyZXNoX2ludGVydmFsGP' + 'YBIAEoCzIZLmdvb2dsZS5wcm90b2J1Zi5EdXJhdGlvblIXbG9jYWxpdHlSZWZyZXNoSW50ZXJ2' + 'YWwSNQoWbG9jYWxpdHlfZXZlbnRzX2VuYWJsZRj3ASABKAhSFGxvY2FsaXR5RXZlbnRzRW5hYm' + 'xlEkUKDGxpbmtfbW9uaXRvchj6ASABKAsyIS54dGNwX2NvbmZpZy52MS5MaW5rTW9uaXRvckNv' + 'bmZpZ1ILbGlua01vbml0b3I6c7pIcBpuCg9YdGNwQ29uZmlnLnBvbGwSMlBvbGwgdGltZW91dC' + 'BtdXN0IGJlIGxlc3MgdGhhbiBwb2xsIHBvbGxfZnJlcXVlbmN5Gid0aGlzLnBvbGxfZnJlcXVl' + 'bmN5ID4gdGhpcy5wb2xsX3RpbWVvdXQ='); @$core.Deprecated('Use enabledDeserializersDescriptor instead') const EnabledDeserializers$json = { diff --git a/gen/go/xtcp_config/xtcp_config.pb.go b/gen/go/xtcp_config/xtcp_config.pb.go index e03c6be81..aaaafa9a7 100644 --- a/gen/go/xtcp_config/xtcp_config.pb.go +++ b/gen/go/xtcp_config/xtcp_config.pb.go @@ -1762,6 +1762,14 @@ type XtcpConfig struct { // [0, window], window grows exponentially up to this cap). 0 = derive as // clamp(poll_frequency/10, 1s, 1h). S3UploadBackoffCap *durationpb.Duration `protobuf:"bytes,115,opt,name=s3_upload_backoff_cap,json=s3UploadBackoffCap,proto3" json:"s3_upload_backoff_cap,omitempty"` + // Envelopes queued between the poller and the S3 worker. Past this depth + // the oldest is dropped (xtcp_s3_dropped_total{reason="queue_depth"}), so + // an S3 outage never blocks polling. 0 = 16. + S3QueueDepth uint32 `protobuf:"varint,116,opt,name=s3_queue_depth,json=s3QueueDepth,proto3" json:"s3_queue_depth,omitempty"` + // Marshaled envelope bytes the S3 queue may hold. Past this the oldest + // envelope is dropped (reason="buffered_bytes"); the newest is always + // kept. 0 = no byte bound (the depth bound alone). Flag default 64 MiB. + S3MaxBufferedBytes uint64 `protobuf:"varint,117,opt,name=s3_max_buffered_bytes,json=s3MaxBufferedBytes,proto3" json:"s3_max_buffered_bytes,omitempty"` // Hostname override. When empty the daemon uses os.Hostname(); set this to // stamp an explicit hostname on records — required in containers, where // os.Hostname() returns the container id, not the host. Set via -hostname @@ -2341,6 +2349,20 @@ func (x *XtcpConfig) GetS3UploadBackoffCap() *durationpb.Duration { return nil } +func (x *XtcpConfig) GetS3QueueDepth() uint32 { + if x != nil { + return x.S3QueueDepth + } + return 0 +} + +func (x *XtcpConfig) GetS3MaxBufferedBytes() uint64 { + if x != nil { + return x.S3MaxBufferedBytes + } + return 0 +} + func (x *XtcpConfig) GetHostname() string { if x != nil { return x.Hostname @@ -2745,7 +2767,7 @@ const file_xtcp_config_v1_xtcp_config_proto_rawDesc = "" + "\x0e_error_classesB\x15\n" + "\x13_metrics_categoriesB\x1a\n" + "\x18_metrics_interface_pathsB\x13\n" + - "\x11_metrics_all_path\"\xd7*\n" + + "\x11_metrics_all_path\"\xc5+\n" + "\n" + "XtcpConfig\x12F\n" + "\x17nl_timeout_milliseconds\x18\n" + @@ -2825,7 +2847,9 @@ const file_xtcp_config_v1_xtcp_config_proto_rawDesc = "" + "\x1ds3_flush_threshold_jitter_pct\x18q \x01(\rB\n" + "\xbaH\a\xc8\x01\x00*\x02\x18dR\x19s3FlushThresholdJitterPct\x12A\n" + "\x16s3_upload_max_attempts\x18r \x01(\rB\f\xbaH\t\xc8\x01\x00*\x04\x18d(\x01R\x13s3UploadMaxAttempts\x12Y\n" + - "\x15s3_upload_backoff_cap\x18s \x01(\v2\x19.google.protobuf.DurationB\v\xbaH\b\xc8\x01\x00\xaa\x01\x022\x00R\x12s3UploadBackoffCap\x12(\n" + + "\x15s3_upload_backoff_cap\x18s \x01(\v2\x19.google.protobuf.DurationB\v\xbaH\b\xc8\x01\x00\xaa\x01\x022\x00R\x12s3UploadBackoffCap\x121\n" + + "\x0es3_queue_depth\x18t \x01(\rB\v\xbaH\b\xc8\x01\x00*\x03\x18\x80 R\fs3QueueDepth\x129\n" + + "\x15s3_max_buffered_bytes\x18u \x01(\x04B\x06\xbaH\x03\xc8\x01\x00R\x12s3MaxBufferedBytes\x12(\n" + "\bhostname\x18\x82\x01 \x01(\tB\v\xbaH\b\xc8\x01\x00r\x03\x18\xfd\x01R\bhostname\x12(\n" + "\blocation\x18\x83\x01 \x01(\tB\v\xbaH\b\xc8\x01\x00r\x03\x18\xfd\x01R\blocation\x12!\n" + "\x05label\x18\x84\x01 \x01(\tB\n" + diff --git a/gen/go/xtcp_config/xtcp_config_vtproto.pb.go b/gen/go/xtcp_config/xtcp_config_vtproto.pb.go index 85510a9a4..42427ae5d 100644 --- a/gen/go/xtcp_config/xtcp_config_vtproto.pb.go +++ b/gen/go/xtcp_config/xtcp_config_vtproto.pb.go @@ -1577,6 +1577,20 @@ func (m *XtcpConfig) MarshalToSizedBufferVT(dAtA []byte) (int, error) { i-- dAtA[i] = 0x92 } + if m.S3MaxBufferedBytes != 0 { + i = protohelpers.EncodeVarint(dAtA, i, uint64(m.S3MaxBufferedBytes)) + i-- + dAtA[i] = 0x7 + i-- + dAtA[i] = 0xa8 + } + if m.S3QueueDepth != 0 { + i = protohelpers.EncodeVarint(dAtA, i, uint64(m.S3QueueDepth)) + i-- + dAtA[i] = 0x7 + i-- + dAtA[i] = 0xa0 + } if m.S3UploadBackoffCap != nil { size, err := (*durationpb.Duration)(m.S3UploadBackoffCap).MarshalToSizedBufferVT(dAtA[:i]) if err != nil { @@ -2730,6 +2744,12 @@ func (m *XtcpConfig) SizeVT() (n int) { l = (*durationpb.Duration)(m.S3UploadBackoffCap).SizeVT() n += 2 + l + protohelpers.SizeOfVarint(uint64(l)) } + if m.S3QueueDepth != 0 { + n += 2 + protohelpers.SizeOfVarint(uint64(m.S3QueueDepth)) + } + if m.S3MaxBufferedBytes != 0 { + n += 2 + protohelpers.SizeOfVarint(uint64(m.S3MaxBufferedBytes)) + } l = len(m.Hostname) if l > 0 { n += 2 + l + protohelpers.SizeOfVarint(uint64(l)) @@ -7198,6 +7218,44 @@ func (m *XtcpConfig) UnmarshalVT(dAtA []byte) error { return err } iNdEx = postIndex + case 116: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field S3QueueDepth", wireType) + } + m.S3QueueDepth = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return protohelpers.ErrIntOverflow + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.S3QueueDepth |= uint32(b&0x7F) << shift + if b < 0x80 { + break + } + } + case 117: + if wireType != 0 { + return fmt.Errorf("proto: wrong wireType = %d for field S3MaxBufferedBytes", wireType) + } + m.S3MaxBufferedBytes = 0 + for shift := uint(0); ; shift += 7 { + if shift >= 64 { + return protohelpers.ErrIntOverflow + } + if iNdEx >= l { + return io.ErrUnexpectedEOF + } + b := dAtA[iNdEx] + iNdEx++ + m.S3MaxBufferedBytes |= uint64(b&0x7F) << shift + if b < 0x80 { + break + } + } case 130: if wireType != 2 { return fmt.Errorf("proto: wrong wireType = %d for field Hostname", wireType) diff --git a/gen/openapi/xtcp_config/v1/xtcp_config.swagger.json b/gen/openapi/xtcp_config/v1/xtcp_config.swagger.json index 8f2a66a15..414066777 100644 --- a/gen/openapi/xtcp_config/v1/xtcp_config.swagger.json +++ b/gen/openapi/xtcp_config/v1/xtcp_config.swagger.json @@ -911,6 +911,16 @@ "type": "string", "description": "Cap on a single upload retry's backoff window (full jitter draws in\n[0, window], window grows exponentially up to this cap). 0 = derive as\nclamp(poll_frequency/10, 1s, 1h)." }, + "s3QueueDepth": { + "type": "integer", + "format": "int64", + "description": "Envelopes queued between the poller and the S3 worker. Past this depth\nthe oldest is dropped (xtcp_s3_dropped_total{reason=\"queue_depth\"}), so\nan S3 outage never blocks polling. 0 = 16." + }, + "s3MaxBufferedBytes": { + "type": "string", + "format": "uint64", + "description": "Marshaled envelope bytes the S3 queue may hold. Past this the oldest\nenvelope is dropped (reason=\"buffered_bytes\"); the newest is always\nkept. 0 = no byte bound (the depth bound alone). Flag default 64 MiB." + }, "hostname": { "type": "string", "description": "Hostname override. When empty the daemon uses os.Hostname(); set this to\nstamp an explicit hostname on records — required in containers, where\nos.Hostname() returns the container id, not the host. Set via -hostname\nflag or XTCP_HOSTNAME env (NOT HOSTNAME, which Docker sets to the\ncontainer id)." diff --git a/gen/python/xtcp_config/v1/xtcp_config_pb2.py b/gen/python/xtcp_config/v1/xtcp_config_pb2.py index 42b79395b..185294da7 100644 --- a/gen/python/xtcp_config/v1/xtcp_config_pb2.py +++ b/gen/python/xtcp_config/v1/xtcp_config_pb2.py @@ -28,7 +28,7 @@ from buf.validate import validate_pb2 as buf_dot_validate_dot_validate__pb2 -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n xtcp_config/v1/xtcp_config.proto\x12\x0extcp_config.v1\x1a\x1egoogle/protobuf/duration.proto\x1a\x1cgoogle/api/annotations.proto\x1a\x1cxtcp_auth/v1/xtcp_auth.proto\x1a\x1b\x62uf/validate/validate.proto\"\x0c\n\nGetRequest\"A\n\x0bGetResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"@\n\nSetRequest\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"A\n\x0bSetResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"\xb4\x02\n\x17SetPollFrequencyRequest\x12S\n\x0epoll_frequency\x18\x14 \x01(\x0b\x32\x19.google.protobuf.DurationB\x11\xbaH\x0e\xaa\x01\x08\"\x04\x08\x80\xf5$2\x00\xc8\x01\x01R\rpollFrequency\x12O\n\x0cpoll_timeout\x18\x1e \x01(\x0b\x32\x19.google.protobuf.DurationB\x11\xbaH\x0e\xaa\x01\x08\"\x04\x08\x80\xf5$2\x00\xc8\x01\x01R\x0bpollTimeout:s\xbaHp\x1an\n\x0fXtcpConfig.poll\x12\x32Poll timeout must be less than poll poll_frequency\x1a\'this.poll_timeout < this.poll_frequency\"N\n\x18SetPollFrequencyResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"\x14\n\x12TriggerPollRequest\"\x15\n\x13TriggerPollResponse\"\x89\x01\n\x17TriggerPollBurstRequest\x12#\n\x05\x63ount\x18\n \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x01\xc8\x01\x01R\x05\x63ount\x12I\n\x08interval\x18\x14 \x01(\x0b\x32\x19.google.protobuf.DurationB\x12\xbaH\x0f\xaa\x01\t\"\x03\x08\x90\x1c\x32\x02\x08\x01\xc8\x01\x01R\x08interval\"g\n\x18TriggerPollBurstResponse\x12\x14\n\x05\x63ount\x18\n \x01(\rR\x05\x63ount\x12\x35\n\x08interval\x18\x14 \x01(\x0b\x32\x19.google.protobuf.DurationR\x08interval\"\xe3\x02\n\x12SetS3UploadRequest\x12R\n\x11s3_flush_interval\x18\n \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x0fs3FlushInterval\x12N\n s3_parquet_flush_threshold_bytes\x18\x14 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1cs3ParquetFlushThresholdBytes:\xa8\x01\xbaH\xa4\x01\x1a\xa1\x01\n\x16SetS3Upload.atLeastOne\x12=set s3_flush_interval and/or s3_parquet_flush_threshold_bytes\x1aHhas(this.s3_flush_interval) || this.s3_parquet_flush_threshold_bytes > 0\"I\n\x13SetS3UploadResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"\xf4\x02\n\x17SetEnvelopeFlushRequest\x12K\n\x1e\x65nvelope_flush_threshold_bytes\x18\n \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1b\x65nvelopeFlushThresholdBytes\x12I\n\x1d\x65nvelope_flush_threshold_rows\x18\x14 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1a\x65nvelopeFlushThresholdRows:\xc0\x01\xbaH\xbc\x01\x1a\xb9\x01\n\x1bSetEnvelopeFlush.atLeastOne\x12Gset envelope_flush_threshold_bytes and/or envelope_flush_threshold_rows\x1aQthis.envelope_flush_threshold_bytes > 0 || this.envelope_flush_threshold_rows > 0\"N\n\x18SetEnvelopeFlushResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"\x8a\x03\n\x10ListenerEndpoint\x12\x43\n\x07network\x18\x01 \x01(\x0e\x32\x1f.xtcp_config.v1.ListenerNetworkB\x08\xbaH\x05\x82\x01\x02\x10\x01R\x07network\x12\"\n\x07\x61\x64\x64ress\x18\x02 \x01(\tB\x08\xbaH\x05r\x03\x18\x80\x04R\x07\x61\x64\x64ress\x12\x32\n\x10unix_socket_mode\x18\x03 \x01(\rB\x08\xbaH\x05*\x03\x18\xff\x03R\x0eunixSocketMode\x12\x37\n\x18unlink_stale_unix_socket\x18\x04 \x01(\x08R\x15unlinkStaleUnixSocket\x12\x32\n\x0fmax_connections\x18\x05 \x01(\rB\t\xbaH\x06*\x04\x18\xa0\x8d\x06R\x0emaxConnections\x12>\n\x16\x61\x63\x63\x65pt_rate_per_second\x18\x06 \x01(\rB\t\xbaH\x06*\x04\x18\xa0\x8d\x06R\x13\x61\x63\x63\x65ptRatePerSecond\x12,\n\x0c\x61\x63\x63\x65pt_burst\x18\x07 \x01(\rB\t\xbaH\x06*\x04\x18\xa0\x8d\x06R\x0b\x61\x63\x63\x65ptBurst\"\x83\x06\n\x0cListenerAuth\x12>\n\x04mode\x18\x01 \x01(\x0e\x32 .xtcp_config.v1.ListenerAuthModeB\x08\xbaH\x05\x82\x01\x02\x10\x01R\x04mode\x12%\n\traw_token\x18\x02 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x08rawToken\x12\x30\n\x0fhmac_shared_key\x18\x03 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\rhmacSharedKey\x12G\n\x19signed_token_skew_minutes\x18\x04 \x01(\rB\x07\xbaH\x04*\x02\x18\x05H\x00R\x16signedTokenSkewMinutes\x88\x01\x01\x12Q\n\x12\x66\x61ilure_jitter_min\x18\x05 \x01(\x0b\x32\x19.google.protobuf.DurationB\x08\xbaH\x05\xaa\x01\x02\x32\x00R\x10\x66\x61ilureJitterMin\x12Q\n\x12\x66\x61ilure_jitter_max\x18\x06 \x01(\x0b\x32\x19.google.protobuf.DurationB\x08\xbaH\x05\xaa\x01\x02\x32\x00R\x10\x66\x61ilureJitterMax\x12.\n\x0eraw_token_file\x18\x07 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0crawTokenFile\x12\x39\n\x14hmac_shared_key_file\x18\x08 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x11hmacSharedKeyFile:\xe1\x01\xbaH\xdd\x01\x1a\xda\x01\n\x1aListenerAuth.failureJitter\x12\x46\x66\x61ilure_jitter_max must be greater than or equal to failure_jitter_min\x1at!has(this.failure_jitter_min) || !has(this.failure_jitter_max) || this.failure_jitter_max >= this.failure_jitter_minB\x1c\n\x1a_signed_token_skew_minutes\"\xbf\x02\n\x08Listener\x12\x30\n\x04name\x18\x01 \x01(\tB\x1c\xbaH\x19r\x17\x10\x01\x18 2\x11^[a-z][a-z0-9-]*$R\x04name\x12\x44\n\x08\x65ndpoint\x18\x02 \x01(\x0b\x32 .xtcp_config.v1.ListenerEndpointB\x06\xbaH\x03\xc8\x01\x01R\x08\x65ndpoint\x12\x30\n\x04\x61uth\x18\x03 \x01(\x0b\x32\x1c.xtcp_config.v1.ListenerAuthR\x04\x61uth\x12;\n\nadmin_auth\x18\x04 \x01(\x0b\x32\x1c.xtcp_config.v1.ListenerAuthR\tadminAuth\x12L\n\x06serves\x18\x05 \x03(\x0e\x32\x1d.xtcp_config.v1.EndpointClassB\x15\xbaH\x12\x92\x01\x0f\x08\x01\x10\x07\x18\x01\"\x07\x82\x01\x04\x10\x01 \x00R\x06serves\"\x9a\x0e\n\x11LinkMonitorConfig\x12\x16\n\x06\x65nable\x18\x01 \x01(\x08R\x06\x65nable\x12-\n\rbaseline_file\x18\x02 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0c\x62\x61selineFile\x12\x41\n\x06settle\x18\x03 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05\x32\x00R\x06settle\x12\x41\n\x06resync\x18\x04 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xf5$*\x00R\x06resync\x12P\n\x0estats_interval\x18\x05 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05*\x00R\rstatsInterval\x12-\n\rstats_include\x18\x06 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0cstatsInclude\x12-\n\rstats_exclude\x18\x07 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0cstatsExclude\x12/\n\x0enetstat_fields\x18\x08 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\rnetstatFields\x12\x44\n\x14max_speed_exceptions\x18\t \x03(\tB\x12\xbaH\x0f\x92\x01\x0c\x10\x80\x02\"\x07r\x05\x10\x01\x18\x80\x01R\x12maxSpeedExceptions\x12:\n\x14metrics_core_include\x18\n \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x12metricsCoreInclude\x12:\n\x14metrics_core_exclude\x18\x0b \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x12metricsCoreExclude\x12\x1f\n\x0bmetrics_all\x18\x0c \x01(\x08R\nmetricsAll\x12\x41\n\x0f\x63ollection_mode\x18\r \x01(\tB\x18\xbaH\x15r\x13R\x00R\x04pollR\ton-demandR\x0e\x63ollectionMode\x12P\n\x0ewatch_interval\x18\x0e \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05*\x00R\rwatchInterval\x12N\n\rslow_interval\x18\x0f \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05*\x00R\x0cslowInterval\x12M\n\ron_demand_ttl\x18\x10 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05\x32\x00R\x0bonDemandTtl\x12M\n\x0eon_demand_wait\x18\x11 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0c\xbaH\t\xaa\x01\x06\x1a\x02\x08\n*\x00R\x0conDemandWait\x12L\n\x0c\x65rror_window\x18\x12 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xf5$2\x00R\x0b\x65rrorWindow\x12J\n\x0b\x66lap_window\x18\x13 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xf5$2\x00R\nflapWindow\x12\\\n\rerror_classes\x18\x14 \x01(\tB2\xbaH/r-\x18@2)^((link|rdma|drop)(,(link|rdma|drop))*)?$H\x00R\x0c\x65rrorClasses\x88\x01\x01\x12-\n\rerror_exclude\x18\x15 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0c\x65rrorExclude\x12<\n\x12metrics_categories\x18\x16 \x01(\tB\x08\xbaH\x05r\x03\x18\x80\x02H\x01R\x11metricsCategories\x88\x01\x01\x12;\n\x17metrics_interface_paths\x18\x17 \x01(\x08H\x02R\x15metricsInterfacePaths\x88\x01\x01\x12\x37\n\x10metrics_all_path\x18\x18 \x01(\tB\x08\xbaH\x05r\x03\x18\x80\x02H\x03R\x0emetricsAllPath\x88\x01\x01\x12\x19\n\x08ip_paths\x18\x19 \x01(\x08R\x07ipPaths\x12-\n\x0cip_max_bytes\x18\x1a \x01(\rB\x0b\xbaH\x08*\x06\x18\x80\x80\x80\x80\x04R\nipMaxBytes\x12$\n\tip_listen\x18\x1b \x01(\tB\x07\xbaH\x04r\x02\x18@R\x08ipListen\x12P\n\x0elearn_deadline\x18\x1c \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05\x32\x00R\rlearnDeadlineB\x10\n\x0e_error_classesB\x15\n\x13_metrics_categoriesB\x1a\n\x18_metrics_interface_pathsB\x13\n\x11_metrics_all_path\"\xd7*\n\nXtcpConfig\x12\x46\n\x17nl_timeout_milliseconds\x18\n \x01(\x04\x42\x0e\xbaH\x0b\x32\x06\x18\xa0\x8d\x06(\x00\xc8\x01\x01R\x15nlTimeoutMilliseconds\x12S\n\x0epoll_frequency\x18\x0b \x01(\x0b\x32\x19.google.protobuf.DurationB\x11\xbaH\x0e\xaa\x01\x08\"\x04\x08\x80\xf5$*\x00\xc8\x01\x01R\rpollFrequency\x12O\n\x0cpoll_timeout\x18\x0c \x01(\x0b\x32\x19.google.protobuf.DurationB\x11\xbaH\x0e\xaa\x01\x08\"\x04\x08\x80\xf5$*\x00\xc8\x01\x01R\x0bpollTimeout\x12\x32\n\x0fpoll_jitter_pct\x18\r \x01(\rB\n\xbaH\x07*\x02\x18\x64\xc8\x01\x00R\rpollJitterPct\x12+\n\tmax_loops\x18\x0e \x01(\x04\x42\x0e\xbaH\x0b\x32\x06\x18\xa0\x8d\x06(\x00\xc8\x01\x00R\x08maxLoops\x12,\n\nnetlinkers\x18\x0f \x01(\rB\x0c\xbaH\t*\x04\x18\x64(\x01\xc8\x01\x01R\nnetlinkers\x12H\n\x19netlinkers_done_chan_size\x18\x10 \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x01\xc8\x01\x01R\x16netlinkersDoneChanSize\x12*\n\tnlmsg_seq\x18\x11 \x01(\rB\r\xbaH\n*\x05\x18\x90N(\x00\xc8\x01\x01R\x08nlmsgSeq\x12/\n\x0bpacket_size\x18\x12 \x01(\x04\x42\x0e\xbaH\x0b\x32\x06\x18\xc0\x84=(\x00\xc8\x01\x00R\npacketSize\x12\x36\n\x10packet_size_mply\x18\x13 \x01(\rB\x0c\xbaH\t*\x04\x18\x64(\x00\xc8\x01\x00R\x0epacketSizeMply\x12(\n\x07modulus\x18\x14 \x01(\x04\x42\x0e\xbaH\x0b\x32\x06\x18\xc0\x84=(\x01\xc8\x01\x01R\x07modulus\x12\x61\n\x15\x65nabled_deserializers\x18\x15 \x01(\x0b\x32$.xtcp_config.v1.EnabledDeserializersB\x06\xbaH\x03\xc8\x01\x00R\x14\x65nabledDeserializers\x12!\n\x08io_uring\x18\x16 \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x07ioUring\x12\x45\n\x18io_uring_recv_batch_size\x18\x17 \x01(\rB\r\xbaH\n*\x05\x18\x80 (\x01\xc8\x01\x00R\x14ioUringRecvBatchSize\x12\x43\n\x17io_uring_cqe_batch_size\x18\x18 \x01(\rB\r\xbaH\n*\x05\x18\x80 (\x01\xc8\x01\x00R\x13ioUringCqeBatchSize\x12,\n\x0emax_dump_bytes\x18\x19 \x01(\x04\x42\x06\xbaH\x03\xc8\x01\x00R\x0cmaxDumpBytes\x12\x39\n\x15max_records_per_netns\x18\x1a \x01(\x04\x42\x06\xbaH\x03\xc8\x01\x00R\x12maxRecordsPerNetns\x12\x37\n\x14max_records_per_poll\x18\x1b \x01(\x04\x42\x06\xbaH\x03\xc8\x01\x00R\x11maxRecordsPerPoll\x12K\n\x1cio_uring_sockstat_batch_size\x18\x1c \x01(\rB\x0b\xbaH\x08*\x03\x18\x80\x08\xc8\x01\x00R\x18ioUringSockstatBatchSize\x12\x45\n\x19io_uring_sockstat_workers\x18\x1d \x01(\rB\n\xbaH\x07*\x02\x18@\xc8\x01\x00R\x16ioUringSockstatWorkers\x12W\n\x13reconcile_frequency\x18( \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x12reconcileFrequency\x12\x32\n\x15reconcile_before_poll\x18) \x01(\x08R\x13reconcileBeforePoll\x12#\n\tmax_netns\x18* \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x08maxNetns\x12S\n\x11reconcile_timeout\x18+ \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x10reconcileTimeout\x12\x35\n\x13max_netns_per_owner\x18, \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x10maxNetnsPerOwner\x12\x46\n\x1cmax_netns_adds_per_reconcile\x18- \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x18maxNetnsAddsPerReconcile\x12.\n\x0bwrite_files\x18\x32 \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x00\xc8\x01\x00R\nwriteFiles\x12/\n\x0c\x63\x61pture_path\x18\x33 \x01(\tB\x0c\xbaH\tr\x04\x10\x01\x18P\xc8\x01\x00R\x0b\x63\x61pturePath\x12\x37\n\x10\x64\x65st_write_files\x18\x34 \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x00\xc8\x01\x00R\x0e\x64\x65stWriteFiles\x12.\n\x0b\x64\x65\x62ug_level\x18\x35 \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x00\xc8\x01\x01R\ndebugLevel\x12\x31\n\x10\x64\x65\x62ug_prometheus\x18\x36 \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x0f\x64\x65\x62ugPrometheus\x12!\n\x04\x64\x65st\x18< \x01(\tB\r\xbaH\nr\x05\x10\x04\x18\x80\x04\xc8\x01\x01R\x04\x64\x65st\x12+\n\nmarshal_to\x18= \x01(\tB\x0c\xbaH\tr\x04\x10\x03\x18(\xc8\x01\x01R\tmarshalTo\x12\'\n\x0b\x63sv_columns\x18> \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\ncsvColumns\x12\x34\n\x0fxtcp_proto_file\x18? \x01(\tB\x0c\xbaH\tr\x04\x10\x01\x18P\xc8\x01\x00R\rxtcpProtoFile\x12K\n\x1e\x65nvelope_flush_threshold_bytes\x18@ \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1b\x65nvelopeFlushThresholdBytes\x12I\n\x1d\x65nvelope_flush_threshold_rows\x18\x41 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1a\x65nvelopeFlushThresholdRows\x12\"\n\x05topic\x18P \x01(\tB\x0c\xbaH\tr\x04\x10\x01\x18(\xc8\x01\x00R\x05topic\x12\x36\n\x10kafka_schema_url\x18Q \x01(\tB\x0c\xbaH\tr\x04\x10\x01\x18<\xc8\x01\x00R\x0ekafkaSchemaUrl\x12_\n\x15kafka_produce_timeout\x18R \x01(\x0b\x32\x19.google.protobuf.DurationB\x10\xbaH\r\xaa\x01\x07\"\x03\x08\xd8\x04\x32\x00\xc8\x01\x00R\x13kafkaProduceTimeout\x12\x33\n\x11kafka_compression\x18S \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x10kafkaCompression\x12\'\n\x0bs3_endpoint\x18\x64 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\ns3Endpoint\x12#\n\ts3_region\x18\x65 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x08s3Region\x12#\n\ts3_bucket\x18\x66 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x08s3Bucket\x12#\n\ts3_prefix\x18g \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x08s3Prefix\x12*\n\rs3_access_key\x18h \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x0bs3AccessKey\x12*\n\rs3_secret_key\x18i \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x0bs3SecretKey\x12\x37\n\x14s3_skip_bucket_probe\x18j \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x11s3SkipBucketProbe\x12\x32\n\x11s3_allow_insecure\x18k \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x0fs3AllowInsecure\x12)\n\ns3_ca_file\x18l \x01(\tB\x0b\xbaH\x08r\x03\x18\x80 \xc8\x01\x00R\x08s3CaFile\x12N\n s3_parquet_flush_threshold_bytes\x18n \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1cs3ParquetFlushThresholdBytes\x12R\n\x11s3_flush_interval\x18o \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x0fs3FlushInterval\x12\x39\n\x13s3_flush_jitter_pct\x18p \x01(\rB\n\xbaH\x07*\x02\x18\x64\xc8\x01\x00R\x10s3FlushJitterPct\x12L\n\x1ds3_flush_threshold_jitter_pct\x18q \x01(\rB\n\xbaH\x07*\x02\x18\x64\xc8\x01\x00R\x19s3FlushThresholdJitterPct\x12\x41\n\x16s3_upload_max_attempts\x18r \x01(\rB\x0c\xbaH\t*\x04\x18\x64(\x01\xc8\x01\x00R\x13s3UploadMaxAttempts\x12Y\n\x15s3_upload_backoff_cap\x18s \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x12s3UploadBackoffCap\x12(\n\x08hostname\x18\x82\x01 \x01(\tB\x0b\xbaH\x08r\x03\x18\xfd\x01\xc8\x01\x00R\x08hostname\x12(\n\x08location\x18\x83\x01 \x01(\tB\x0b\xbaH\x08r\x03\x18\xfd\x01\xc8\x01\x00R\x08location\x12!\n\x05label\x18\x84\x01 \x01(\tB\n\xbaH\x07r\x02\x18(\xc8\x01\x00R\x05label\x12\x1d\n\x03tag\x18\x85\x01 \x01(\tB\n\xbaH\x07r\x02\x18(\xc8\x01\x00R\x03tag\x12\x33\n\x0e\x64\x61\x65mon_version\x18\x86\x01 \x01(\tB\x0b\xbaH\x08r\x03\x18\xfd\x01\xc8\x01\x00R\rdaemonVersion\x12\'\n\x08ipv4_ttl\x18\x96\x01 \x01(\rB\x0b\xbaH\x08*\x03\x18\xff\x01\xc8\x01\x00R\x07ipv4Ttl\x12\x32\n\x0eipv6_hop_limit\x18\x97\x01 \x01(\rB\x0b\xbaH\x08*\x03\x18\xff\x01\xc8\x01\x00R\x0cipv6HopLimit\x12J\n\rlistener_auth\x18\x98\x01 \x01(\x0b\x32\x1c.xtcp_config.v1.ListenerAuthB\x06\xbaH\x03\xc8\x01\x00R\x0clistenerAuth\x12Z\n\x13prometheus_listener\x18\x99\x01 \x01(\x0b\x32 .xtcp_config.v1.ListenerEndpointB\x06\xbaH\x03\xc8\x01\x00R\x12prometheusListener\x12U\n\x13\x61\x64min_listener_auth\x18\x9a\x01 \x01(\x0b\x32\x1c.xtcp_config.v1.ListenerAuthB\x06\xbaH\x03\xc8\x01\x00R\x11\x61\x64minListenerAuth\x12\'\n\tgrpc_port\x18\xa0\x01 \x01(\rB\t\xbaH\x06*\x04\x18\xff\xff\x03R\x08grpcPort\x12N\n\rgrpc_listener\x18\xa1\x01 \x01(\x0b\x32 .xtcp_config.v1.ListenerEndpointB\x06\xbaH\x03\xc8\x01\x00R\x0cgrpcListener\x12\x41\n\tlisteners\x18\xa2\x01 \x03(\x0b\x32\x18.xtcp_config.v1.ListenerB\x08\xbaH\x05\x92\x01\x02\x10\x10R\tlisteners\x12,\n\rpyroscope_url\x18\xaa\x01 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x0cpyroscopeUrl\x12\x35\n\x12pyroscope_app_name\x18\xab\x01 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x10pyroscopeAppName\x12\x37\n\x13pyroscope_sample_hz\x18\xac\x01 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x11pyroscopeSampleHz\x12J\n\x1dpyroscope_upload_interval_sec\x18\xad\x01 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1apyroscopeUploadIntervalSec\x12\x39\n\x14resolve_container_id\x18\xc8\x01 \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x12resolveContainerId\x12\x37\n\x17\x65nrich_container_enable\x18\xc9\x01 \x01(\x08R\x15\x65nrichContainerEnable\x12\x37\n\x12\x64ocker_socket_path\x18\xca\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\x10\x64ockerSocketPath\x12-\n\x12\x65nrich_lldp_enable\x18\xd2\x01 \x01(\x08R\x10\x65nrichLldpEnable\x12\x35\n\x11lldpd_socket_path\x18\xd3\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\x0flldpdSocketPath\x12\x36\n\x12lldpd_version_hint\x18\xd4\x01 \x01(\tB\x07\xbaH\x04r\x02\x18\x10R\x10lldpdVersionHint\x12+\n\x11\x65nrich_nic_enable\x18\xdc\x01 \x01(\x08R\x0f\x65nrichNicEnable\x12+\n\x0cuplink_count\x18\xdd\x01 \x01(\rB\x07\xbaH\x04*\x02\x18\x02R\x0buplinkCount\x12\x36\n\x11uplink_interfaces\x18\xde\x01 \x03(\tB\x08\xbaH\x05\x92\x01\x02\x10\x02R\x10uplinkInterfaces\x12$\n\rpopulate_nsid\x18\xe6\x01 \x01(\x08R\x0cpopulateNsid\x12+\n\x11\x65nrich_asn_enable\x18\xf0\x01 \x01(\x08R\x0f\x65nrichAsnEnable\x12)\n\x0b\x61sn_db_path\x18\xf1\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\tasnDbPath\x12L\n\x14\x61sn_refresh_interval\x18\xf2\x01 \x01(\x0b\x32\x19.google.protobuf.DurationR\x12\x61snRefreshInterval\x12=\n\x15ipmeta_bootstrap_path\x18\xf3\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\x13ipmetaBootstrapPath\x12\x35\n\x11ipmeta_cache_path\x18\xf4\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\x0fipmetaCachePath\x12\x35\n\x16\x65nrich_locality_enable\x18\xf5\x01 \x01(\x08R\x14\x65nrichLocalityEnable\x12V\n\x19locality_refresh_interval\x18\xf6\x01 \x01(\x0b\x32\x19.google.protobuf.DurationR\x17localityRefreshInterval\x12\x35\n\x16locality_events_enable\x18\xf7\x01 \x01(\x08R\x14localityEventsEnable\x12\x45\n\x0clink_monitor\x18\xfa\x01 \x01(\x0b\x32!.xtcp_config.v1.LinkMonitorConfigR\x0blinkMonitor:s\xbaHp\x1an\n\x0fXtcpConfig.poll\x12\x32Poll timeout must be less than poll poll_frequency\x1a\'this.poll_frequency > this.poll_timeout\"\x9f\x01\n\x14\x45nabledDeserializers\x12K\n\x07\x65nabled\x18\x01 \x03(\x0b\x32\x31.xtcp_config.v1.EnabledDeserializers.EnabledEntryR\x07\x65nabled\x1a:\n\x0c\x45nabledEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\x08R\x05value:\x02\x38\x01*h\n\x0fListenerNetwork\x12 \n\x1cLISTENER_NETWORK_UNSPECIFIED\x10\x00\x12\x18\n\x14LISTENER_NETWORK_TCP\x10\x01\x12\x19\n\x15LISTENER_NETWORK_UNIX\x10\x02*\xa1\x01\n\x10ListenerAuthMode\x12\"\n\x1eLISTENER_AUTH_MODE_UNSPECIFIED\x10\x00\x12\x1f\n\x1bLISTENER_AUTH_MODE_DISABLED\x10\x01\x12 \n\x1cLISTENER_AUTH_MODE_RAW_TOKEN\x10\x02\x12&\n\"LISTENER_AUTH_MODE_HMAC_UTC_MINUTE\x10\x03*\xf7\x01\n\rEndpointClass\x12\x1e\n\x1a\x45NDPOINT_CLASS_UNSPECIFIED\x10\x00\x12\x19\n\x15\x45NDPOINT_CLASS_HEALTH\x10\x01\x12\x1a\n\x16\x45NDPOINT_CLASS_METRICS\x10\x02\x12!\n\x1d\x45NDPOINT_CLASS_METRICS_DETAIL\x10\x03\x12\x15\n\x11\x45NDPOINT_CLASS_IP\x10\x04\x12\x18\n\x14\x45NDPOINT_CLASS_DEBUG\x10\x05\x12\x1c\n\x18\x45NDPOINT_CLASS_GRPC_READ\x10\x06\x12\x1d\n\x19\x45NDPOINT_CLASS_GRPC_ADMIN\x10\x07\x32\xa5\x07\n\rConfigService\x12\x61\n\x03Get\x12\x1a.xtcp_config.v1.GetRequest\x1a\x1b.xtcp_config.v1.GetResponse\"!\xc8\xf3\x18\x01\x82\xd3\xe4\x93\x02\x17\x1a\x12/ConfigService/Get:\x01*\x12\x61\n\x03Set\x12\x1a.xtcp_config.v1.SetRequest\x1a\x1b.xtcp_config.v1.SetResponse\"!\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02\x17\x1a\x12/ConfigService/Set:\x01*\x12\x95\x01\n\x10SetPollFrequency\x12\'.xtcp_config.v1.SetPollFrequencyRequest\x1a(.xtcp_config.v1.SetPollFrequencyResponse\".\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02$\x1a\x1f/ConfigService/SetPollFrequency:\x01*\x12\x81\x01\n\x0bTriggerPoll\x12\".xtcp_config.v1.TriggerPollRequest\x1a#.xtcp_config.v1.TriggerPollResponse\")\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02\x1f\x1a\x1a/ConfigService/TriggerPoll:\x01*\x12\x95\x01\n\x10TriggerPollBurst\x12\'.xtcp_config.v1.TriggerPollBurstRequest\x1a(.xtcp_config.v1.TriggerPollBurstResponse\".\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02$\x1a\x1f/ConfigService/TriggerPollBurst:\x01*\x12\x81\x01\n\x0bSetS3Upload\x12\".xtcp_config.v1.SetS3UploadRequest\x1a#.xtcp_config.v1.SetS3UploadResponse\")\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02\x1f\x1a\x1a/ConfigService/SetS3Upload:\x01*\x12\x95\x01\n\x10SetEnvelopeFlush\x12\'.xtcp_config.v1.SetEnvelopeFlushRequest\x1a(.xtcp_config.v1.SetEnvelopeFlushResponse\".\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02$\x1a\x1f/ConfigService/SetEnvelopeFlush:\x01*B\xaf\x01\n\x12\x63om.xtcp_config.v1B\x0fXtcpConfigProtoP\x01Z3github.com/randomizedcoder/xtcp2/gen/go/xtcp_config\xa2\x02\x03XXX\xaa\x02\rXtcpConfig.V1\xca\x02\rXtcpConfig\\V1\xe2\x02\x19XtcpConfig\\V1\\GPBMetadata\xea\x02\x0eXtcpConfig::V1b\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n xtcp_config/v1/xtcp_config.proto\x12\x0extcp_config.v1\x1a\x1egoogle/protobuf/duration.proto\x1a\x1cgoogle/api/annotations.proto\x1a\x1cxtcp_auth/v1/xtcp_auth.proto\x1a\x1b\x62uf/validate/validate.proto\"\x0c\n\nGetRequest\"A\n\x0bGetResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"@\n\nSetRequest\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"A\n\x0bSetResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"\xb4\x02\n\x17SetPollFrequencyRequest\x12S\n\x0epoll_frequency\x18\x14 \x01(\x0b\x32\x19.google.protobuf.DurationB\x11\xbaH\x0e\xaa\x01\x08\"\x04\x08\x80\xf5$2\x00\xc8\x01\x01R\rpollFrequency\x12O\n\x0cpoll_timeout\x18\x1e \x01(\x0b\x32\x19.google.protobuf.DurationB\x11\xbaH\x0e\xaa\x01\x08\"\x04\x08\x80\xf5$2\x00\xc8\x01\x01R\x0bpollTimeout:s\xbaHp\x1an\n\x0fXtcpConfig.poll\x12\x32Poll timeout must be less than poll poll_frequency\x1a\'this.poll_timeout < this.poll_frequency\"N\n\x18SetPollFrequencyResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"\x14\n\x12TriggerPollRequest\"\x15\n\x13TriggerPollResponse\"\x89\x01\n\x17TriggerPollBurstRequest\x12#\n\x05\x63ount\x18\n \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x01\xc8\x01\x01R\x05\x63ount\x12I\n\x08interval\x18\x14 \x01(\x0b\x32\x19.google.protobuf.DurationB\x12\xbaH\x0f\xaa\x01\t\"\x03\x08\x90\x1c\x32\x02\x08\x01\xc8\x01\x01R\x08interval\"g\n\x18TriggerPollBurstResponse\x12\x14\n\x05\x63ount\x18\n \x01(\rR\x05\x63ount\x12\x35\n\x08interval\x18\x14 \x01(\x0b\x32\x19.google.protobuf.DurationR\x08interval\"\xe3\x02\n\x12SetS3UploadRequest\x12R\n\x11s3_flush_interval\x18\n \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x0fs3FlushInterval\x12N\n s3_parquet_flush_threshold_bytes\x18\x14 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1cs3ParquetFlushThresholdBytes:\xa8\x01\xbaH\xa4\x01\x1a\xa1\x01\n\x16SetS3Upload.atLeastOne\x12=set s3_flush_interval and/or s3_parquet_flush_threshold_bytes\x1aHhas(this.s3_flush_interval) || this.s3_parquet_flush_threshold_bytes > 0\"I\n\x13SetS3UploadResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"\xf4\x02\n\x17SetEnvelopeFlushRequest\x12K\n\x1e\x65nvelope_flush_threshold_bytes\x18\n \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1b\x65nvelopeFlushThresholdBytes\x12I\n\x1d\x65nvelope_flush_threshold_rows\x18\x14 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1a\x65nvelopeFlushThresholdRows:\xc0\x01\xbaH\xbc\x01\x1a\xb9\x01\n\x1bSetEnvelopeFlush.atLeastOne\x12Gset envelope_flush_threshold_bytes and/or envelope_flush_threshold_rows\x1aQthis.envelope_flush_threshold_bytes > 0 || this.envelope_flush_threshold_rows > 0\"N\n\x18SetEnvelopeFlushResponse\x12\x32\n\x06\x63onfig\x18\x01 \x01(\x0b\x32\x1a.xtcp_config.v1.XtcpConfigR\x06\x63onfig\"\x8a\x03\n\x10ListenerEndpoint\x12\x43\n\x07network\x18\x01 \x01(\x0e\x32\x1f.xtcp_config.v1.ListenerNetworkB\x08\xbaH\x05\x82\x01\x02\x10\x01R\x07network\x12\"\n\x07\x61\x64\x64ress\x18\x02 \x01(\tB\x08\xbaH\x05r\x03\x18\x80\x04R\x07\x61\x64\x64ress\x12\x32\n\x10unix_socket_mode\x18\x03 \x01(\rB\x08\xbaH\x05*\x03\x18\xff\x03R\x0eunixSocketMode\x12\x37\n\x18unlink_stale_unix_socket\x18\x04 \x01(\x08R\x15unlinkStaleUnixSocket\x12\x32\n\x0fmax_connections\x18\x05 \x01(\rB\t\xbaH\x06*\x04\x18\xa0\x8d\x06R\x0emaxConnections\x12>\n\x16\x61\x63\x63\x65pt_rate_per_second\x18\x06 \x01(\rB\t\xbaH\x06*\x04\x18\xa0\x8d\x06R\x13\x61\x63\x63\x65ptRatePerSecond\x12,\n\x0c\x61\x63\x63\x65pt_burst\x18\x07 \x01(\rB\t\xbaH\x06*\x04\x18\xa0\x8d\x06R\x0b\x61\x63\x63\x65ptBurst\"\x83\x06\n\x0cListenerAuth\x12>\n\x04mode\x18\x01 \x01(\x0e\x32 .xtcp_config.v1.ListenerAuthModeB\x08\xbaH\x05\x82\x01\x02\x10\x01R\x04mode\x12%\n\traw_token\x18\x02 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x08rawToken\x12\x30\n\x0fhmac_shared_key\x18\x03 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\rhmacSharedKey\x12G\n\x19signed_token_skew_minutes\x18\x04 \x01(\rB\x07\xbaH\x04*\x02\x18\x05H\x00R\x16signedTokenSkewMinutes\x88\x01\x01\x12Q\n\x12\x66\x61ilure_jitter_min\x18\x05 \x01(\x0b\x32\x19.google.protobuf.DurationB\x08\xbaH\x05\xaa\x01\x02\x32\x00R\x10\x66\x61ilureJitterMin\x12Q\n\x12\x66\x61ilure_jitter_max\x18\x06 \x01(\x0b\x32\x19.google.protobuf.DurationB\x08\xbaH\x05\xaa\x01\x02\x32\x00R\x10\x66\x61ilureJitterMax\x12.\n\x0eraw_token_file\x18\x07 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0crawTokenFile\x12\x39\n\x14hmac_shared_key_file\x18\x08 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x11hmacSharedKeyFile:\xe1\x01\xbaH\xdd\x01\x1a\xda\x01\n\x1aListenerAuth.failureJitter\x12\x46\x66\x61ilure_jitter_max must be greater than or equal to failure_jitter_min\x1at!has(this.failure_jitter_min) || !has(this.failure_jitter_max) || this.failure_jitter_max >= this.failure_jitter_minB\x1c\n\x1a_signed_token_skew_minutes\"\xbf\x02\n\x08Listener\x12\x30\n\x04name\x18\x01 \x01(\tB\x1c\xbaH\x19r\x17\x10\x01\x18 2\x11^[a-z][a-z0-9-]*$R\x04name\x12\x44\n\x08\x65ndpoint\x18\x02 \x01(\x0b\x32 .xtcp_config.v1.ListenerEndpointB\x06\xbaH\x03\xc8\x01\x01R\x08\x65ndpoint\x12\x30\n\x04\x61uth\x18\x03 \x01(\x0b\x32\x1c.xtcp_config.v1.ListenerAuthR\x04\x61uth\x12;\n\nadmin_auth\x18\x04 \x01(\x0b\x32\x1c.xtcp_config.v1.ListenerAuthR\tadminAuth\x12L\n\x06serves\x18\x05 \x03(\x0e\x32\x1d.xtcp_config.v1.EndpointClassB\x15\xbaH\x12\x92\x01\x0f\x08\x01\x10\x07\x18\x01\"\x07\x82\x01\x04\x10\x01 \x00R\x06serves\"\x9a\x0e\n\x11LinkMonitorConfig\x12\x16\n\x06\x65nable\x18\x01 \x01(\x08R\x06\x65nable\x12-\n\rbaseline_file\x18\x02 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0c\x62\x61selineFile\x12\x41\n\x06settle\x18\x03 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05\x32\x00R\x06settle\x12\x41\n\x06resync\x18\x04 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xf5$*\x00R\x06resync\x12P\n\x0estats_interval\x18\x05 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05*\x00R\rstatsInterval\x12-\n\rstats_include\x18\x06 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0cstatsInclude\x12-\n\rstats_exclude\x18\x07 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0cstatsExclude\x12/\n\x0enetstat_fields\x18\x08 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\rnetstatFields\x12\x44\n\x14max_speed_exceptions\x18\t \x03(\tB\x12\xbaH\x0f\x92\x01\x0c\x10\x80\x02\"\x07r\x05\x10\x01\x18\x80\x01R\x12maxSpeedExceptions\x12:\n\x14metrics_core_include\x18\n \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x12metricsCoreInclude\x12:\n\x14metrics_core_exclude\x18\x0b \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x12metricsCoreExclude\x12\x1f\n\x0bmetrics_all\x18\x0c \x01(\x08R\nmetricsAll\x12\x41\n\x0f\x63ollection_mode\x18\r \x01(\tB\x18\xbaH\x15r\x13R\x00R\x04pollR\ton-demandR\x0e\x63ollectionMode\x12P\n\x0ewatch_interval\x18\x0e \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05*\x00R\rwatchInterval\x12N\n\rslow_interval\x18\x0f \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05*\x00R\x0cslowInterval\x12M\n\ron_demand_ttl\x18\x10 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05\x32\x00R\x0bonDemandTtl\x12M\n\x0eon_demand_wait\x18\x11 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0c\xbaH\t\xaa\x01\x06\x1a\x02\x08\n*\x00R\x0conDemandWait\x12L\n\x0c\x65rror_window\x18\x12 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xf5$2\x00R\x0b\x65rrorWindow\x12J\n\x0b\x66lap_window\x18\x13 \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xf5$2\x00R\nflapWindow\x12\\\n\rerror_classes\x18\x14 \x01(\tB2\xbaH/r-\x18@2)^((link|rdma|drop)(,(link|rdma|drop))*)?$H\x00R\x0c\x65rrorClasses\x88\x01\x01\x12-\n\rerror_exclude\x18\x15 \x01(\tB\x08\xbaH\x05r\x03\x18\x80 R\x0c\x65rrorExclude\x12<\n\x12metrics_categories\x18\x16 \x01(\tB\x08\xbaH\x05r\x03\x18\x80\x02H\x01R\x11metricsCategories\x88\x01\x01\x12;\n\x17metrics_interface_paths\x18\x17 \x01(\x08H\x02R\x15metricsInterfacePaths\x88\x01\x01\x12\x37\n\x10metrics_all_path\x18\x18 \x01(\tB\x08\xbaH\x05r\x03\x18\x80\x02H\x03R\x0emetricsAllPath\x88\x01\x01\x12\x19\n\x08ip_paths\x18\x19 \x01(\x08R\x07ipPaths\x12-\n\x0cip_max_bytes\x18\x1a \x01(\rB\x0b\xbaH\x08*\x06\x18\x80\x80\x80\x80\x04R\nipMaxBytes\x12$\n\tip_listen\x18\x1b \x01(\tB\x07\xbaH\x04r\x02\x18@R\x08ipListen\x12P\n\x0elearn_deadline\x18\x1c \x01(\x0b\x32\x19.google.protobuf.DurationB\x0e\xbaH\x0b\xaa\x01\x08\"\x04\x08\x80\xa3\x05\x32\x00R\rlearnDeadlineB\x10\n\x0e_error_classesB\x15\n\x13_metrics_categoriesB\x1a\n\x18_metrics_interface_pathsB\x13\n\x11_metrics_all_path\"\xc5+\n\nXtcpConfig\x12\x46\n\x17nl_timeout_milliseconds\x18\n \x01(\x04\x42\x0e\xbaH\x0b\x32\x06\x18\xa0\x8d\x06(\x00\xc8\x01\x01R\x15nlTimeoutMilliseconds\x12S\n\x0epoll_frequency\x18\x0b \x01(\x0b\x32\x19.google.protobuf.DurationB\x11\xbaH\x0e\xaa\x01\x08\"\x04\x08\x80\xf5$*\x00\xc8\x01\x01R\rpollFrequency\x12O\n\x0cpoll_timeout\x18\x0c \x01(\x0b\x32\x19.google.protobuf.DurationB\x11\xbaH\x0e\xaa\x01\x08\"\x04\x08\x80\xf5$*\x00\xc8\x01\x01R\x0bpollTimeout\x12\x32\n\x0fpoll_jitter_pct\x18\r \x01(\rB\n\xbaH\x07*\x02\x18\x64\xc8\x01\x00R\rpollJitterPct\x12+\n\tmax_loops\x18\x0e \x01(\x04\x42\x0e\xbaH\x0b\x32\x06\x18\xa0\x8d\x06(\x00\xc8\x01\x00R\x08maxLoops\x12,\n\nnetlinkers\x18\x0f \x01(\rB\x0c\xbaH\t*\x04\x18\x64(\x01\xc8\x01\x01R\nnetlinkers\x12H\n\x19netlinkers_done_chan_size\x18\x10 \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x01\xc8\x01\x01R\x16netlinkersDoneChanSize\x12*\n\tnlmsg_seq\x18\x11 \x01(\rB\r\xbaH\n*\x05\x18\x90N(\x00\xc8\x01\x01R\x08nlmsgSeq\x12/\n\x0bpacket_size\x18\x12 \x01(\x04\x42\x0e\xbaH\x0b\x32\x06\x18\xc0\x84=(\x00\xc8\x01\x00R\npacketSize\x12\x36\n\x10packet_size_mply\x18\x13 \x01(\rB\x0c\xbaH\t*\x04\x18\x64(\x00\xc8\x01\x00R\x0epacketSizeMply\x12(\n\x07modulus\x18\x14 \x01(\x04\x42\x0e\xbaH\x0b\x32\x06\x18\xc0\x84=(\x01\xc8\x01\x01R\x07modulus\x12\x61\n\x15\x65nabled_deserializers\x18\x15 \x01(\x0b\x32$.xtcp_config.v1.EnabledDeserializersB\x06\xbaH\x03\xc8\x01\x00R\x14\x65nabledDeserializers\x12!\n\x08io_uring\x18\x16 \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x07ioUring\x12\x45\n\x18io_uring_recv_batch_size\x18\x17 \x01(\rB\r\xbaH\n*\x05\x18\x80 (\x01\xc8\x01\x00R\x14ioUringRecvBatchSize\x12\x43\n\x17io_uring_cqe_batch_size\x18\x18 \x01(\rB\r\xbaH\n*\x05\x18\x80 (\x01\xc8\x01\x00R\x13ioUringCqeBatchSize\x12,\n\x0emax_dump_bytes\x18\x19 \x01(\x04\x42\x06\xbaH\x03\xc8\x01\x00R\x0cmaxDumpBytes\x12\x39\n\x15max_records_per_netns\x18\x1a \x01(\x04\x42\x06\xbaH\x03\xc8\x01\x00R\x12maxRecordsPerNetns\x12\x37\n\x14max_records_per_poll\x18\x1b \x01(\x04\x42\x06\xbaH\x03\xc8\x01\x00R\x11maxRecordsPerPoll\x12K\n\x1cio_uring_sockstat_batch_size\x18\x1c \x01(\rB\x0b\xbaH\x08*\x03\x18\x80\x08\xc8\x01\x00R\x18ioUringSockstatBatchSize\x12\x45\n\x19io_uring_sockstat_workers\x18\x1d \x01(\rB\n\xbaH\x07*\x02\x18@\xc8\x01\x00R\x16ioUringSockstatWorkers\x12W\n\x13reconcile_frequency\x18( \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x12reconcileFrequency\x12\x32\n\x15reconcile_before_poll\x18) \x01(\x08R\x13reconcileBeforePoll\x12#\n\tmax_netns\x18* \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x08maxNetns\x12S\n\x11reconcile_timeout\x18+ \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x10reconcileTimeout\x12\x35\n\x13max_netns_per_owner\x18, \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x10maxNetnsPerOwner\x12\x46\n\x1cmax_netns_adds_per_reconcile\x18- \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x18maxNetnsAddsPerReconcile\x12.\n\x0bwrite_files\x18\x32 \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x00\xc8\x01\x00R\nwriteFiles\x12/\n\x0c\x63\x61pture_path\x18\x33 \x01(\tB\x0c\xbaH\tr\x04\x10\x01\x18P\xc8\x01\x00R\x0b\x63\x61pturePath\x12\x37\n\x10\x64\x65st_write_files\x18\x34 \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x00\xc8\x01\x00R\x0e\x64\x65stWriteFiles\x12.\n\x0b\x64\x65\x62ug_level\x18\x35 \x01(\rB\r\xbaH\n*\x05\x18\xe8\x07(\x00\xc8\x01\x01R\ndebugLevel\x12\x31\n\x10\x64\x65\x62ug_prometheus\x18\x36 \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x0f\x64\x65\x62ugPrometheus\x12!\n\x04\x64\x65st\x18< \x01(\tB\r\xbaH\nr\x05\x10\x04\x18\x80\x04\xc8\x01\x01R\x04\x64\x65st\x12+\n\nmarshal_to\x18= \x01(\tB\x0c\xbaH\tr\x04\x10\x03\x18(\xc8\x01\x01R\tmarshalTo\x12\'\n\x0b\x63sv_columns\x18> \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\ncsvColumns\x12\x34\n\x0fxtcp_proto_file\x18? \x01(\tB\x0c\xbaH\tr\x04\x10\x01\x18P\xc8\x01\x00R\rxtcpProtoFile\x12K\n\x1e\x65nvelope_flush_threshold_bytes\x18@ \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1b\x65nvelopeFlushThresholdBytes\x12I\n\x1d\x65nvelope_flush_threshold_rows\x18\x41 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1a\x65nvelopeFlushThresholdRows\x12\"\n\x05topic\x18P \x01(\tB\x0c\xbaH\tr\x04\x10\x01\x18(\xc8\x01\x00R\x05topic\x12\x36\n\x10kafka_schema_url\x18Q \x01(\tB\x0c\xbaH\tr\x04\x10\x01\x18<\xc8\x01\x00R\x0ekafkaSchemaUrl\x12_\n\x15kafka_produce_timeout\x18R \x01(\x0b\x32\x19.google.protobuf.DurationB\x10\xbaH\r\xaa\x01\x07\"\x03\x08\xd8\x04\x32\x00\xc8\x01\x00R\x13kafkaProduceTimeout\x12\x33\n\x11kafka_compression\x18S \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x10kafkaCompression\x12\'\n\x0bs3_endpoint\x18\x64 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\ns3Endpoint\x12#\n\ts3_region\x18\x65 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x08s3Region\x12#\n\ts3_bucket\x18\x66 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x08s3Bucket\x12#\n\ts3_prefix\x18g \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x08s3Prefix\x12*\n\rs3_access_key\x18h \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x0bs3AccessKey\x12*\n\rs3_secret_key\x18i \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x0bs3SecretKey\x12\x37\n\x14s3_skip_bucket_probe\x18j \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x11s3SkipBucketProbe\x12\x32\n\x11s3_allow_insecure\x18k \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x0fs3AllowInsecure\x12)\n\ns3_ca_file\x18l \x01(\tB\x0b\xbaH\x08r\x03\x18\x80 \xc8\x01\x00R\x08s3CaFile\x12N\n s3_parquet_flush_threshold_bytes\x18n \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1cs3ParquetFlushThresholdBytes\x12R\n\x11s3_flush_interval\x18o \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x0fs3FlushInterval\x12\x39\n\x13s3_flush_jitter_pct\x18p \x01(\rB\n\xbaH\x07*\x02\x18\x64\xc8\x01\x00R\x10s3FlushJitterPct\x12L\n\x1ds3_flush_threshold_jitter_pct\x18q \x01(\rB\n\xbaH\x07*\x02\x18\x64\xc8\x01\x00R\x19s3FlushThresholdJitterPct\x12\x41\n\x16s3_upload_max_attempts\x18r \x01(\rB\x0c\xbaH\t*\x04\x18\x64(\x01\xc8\x01\x00R\x13s3UploadMaxAttempts\x12Y\n\x15s3_upload_backoff_cap\x18s \x01(\x0b\x32\x19.google.protobuf.DurationB\x0b\xbaH\x08\xaa\x01\x02\x32\x00\xc8\x01\x00R\x12s3UploadBackoffCap\x12\x31\n\x0es3_queue_depth\x18t \x01(\rB\x0b\xbaH\x08*\x03\x18\x80 \xc8\x01\x00R\x0cs3QueueDepth\x12\x39\n\x15s3_max_buffered_bytes\x18u \x01(\x04\x42\x06\xbaH\x03\xc8\x01\x00R\x12s3MaxBufferedBytes\x12(\n\x08hostname\x18\x82\x01 \x01(\tB\x0b\xbaH\x08r\x03\x18\xfd\x01\xc8\x01\x00R\x08hostname\x12(\n\x08location\x18\x83\x01 \x01(\tB\x0b\xbaH\x08r\x03\x18\xfd\x01\xc8\x01\x00R\x08location\x12!\n\x05label\x18\x84\x01 \x01(\tB\n\xbaH\x07r\x02\x18(\xc8\x01\x00R\x05label\x12\x1d\n\x03tag\x18\x85\x01 \x01(\tB\n\xbaH\x07r\x02\x18(\xc8\x01\x00R\x03tag\x12\x33\n\x0e\x64\x61\x65mon_version\x18\x86\x01 \x01(\tB\x0b\xbaH\x08r\x03\x18\xfd\x01\xc8\x01\x00R\rdaemonVersion\x12\'\n\x08ipv4_ttl\x18\x96\x01 \x01(\rB\x0b\xbaH\x08*\x03\x18\xff\x01\xc8\x01\x00R\x07ipv4Ttl\x12\x32\n\x0eipv6_hop_limit\x18\x97\x01 \x01(\rB\x0b\xbaH\x08*\x03\x18\xff\x01\xc8\x01\x00R\x0cipv6HopLimit\x12J\n\rlistener_auth\x18\x98\x01 \x01(\x0b\x32\x1c.xtcp_config.v1.ListenerAuthB\x06\xbaH\x03\xc8\x01\x00R\x0clistenerAuth\x12Z\n\x13prometheus_listener\x18\x99\x01 \x01(\x0b\x32 .xtcp_config.v1.ListenerEndpointB\x06\xbaH\x03\xc8\x01\x00R\x12prometheusListener\x12U\n\x13\x61\x64min_listener_auth\x18\x9a\x01 \x01(\x0b\x32\x1c.xtcp_config.v1.ListenerAuthB\x06\xbaH\x03\xc8\x01\x00R\x11\x61\x64minListenerAuth\x12\'\n\tgrpc_port\x18\xa0\x01 \x01(\rB\t\xbaH\x06*\x04\x18\xff\xff\x03R\x08grpcPort\x12N\n\rgrpc_listener\x18\xa1\x01 \x01(\x0b\x32 .xtcp_config.v1.ListenerEndpointB\x06\xbaH\x03\xc8\x01\x00R\x0cgrpcListener\x12\x41\n\tlisteners\x18\xa2\x01 \x03(\x0b\x32\x18.xtcp_config.v1.ListenerB\x08\xbaH\x05\x92\x01\x02\x10\x10R\tlisteners\x12,\n\rpyroscope_url\x18\xaa\x01 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x0cpyroscopeUrl\x12\x35\n\x12pyroscope_app_name\x18\xab\x01 \x01(\tB\x06\xbaH\x03\xc8\x01\x00R\x10pyroscopeAppName\x12\x37\n\x13pyroscope_sample_hz\x18\xac\x01 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x11pyroscopeSampleHz\x12J\n\x1dpyroscope_upload_interval_sec\x18\xad\x01 \x01(\rB\x06\xbaH\x03\xc8\x01\x00R\x1apyroscopeUploadIntervalSec\x12\x39\n\x14resolve_container_id\x18\xc8\x01 \x01(\x08\x42\x06\xbaH\x03\xc8\x01\x00R\x12resolveContainerId\x12\x37\n\x17\x65nrich_container_enable\x18\xc9\x01 \x01(\x08R\x15\x65nrichContainerEnable\x12\x37\n\x12\x64ocker_socket_path\x18\xca\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\x10\x64ockerSocketPath\x12-\n\x12\x65nrich_lldp_enable\x18\xd2\x01 \x01(\x08R\x10\x65nrichLldpEnable\x12\x35\n\x11lldpd_socket_path\x18\xd3\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\x0flldpdSocketPath\x12\x36\n\x12lldpd_version_hint\x18\xd4\x01 \x01(\tB\x07\xbaH\x04r\x02\x18\x10R\x10lldpdVersionHint\x12+\n\x11\x65nrich_nic_enable\x18\xdc\x01 \x01(\x08R\x0f\x65nrichNicEnable\x12+\n\x0cuplink_count\x18\xdd\x01 \x01(\rB\x07\xbaH\x04*\x02\x18\x02R\x0buplinkCount\x12\x36\n\x11uplink_interfaces\x18\xde\x01 \x03(\tB\x08\xbaH\x05\x92\x01\x02\x10\x02R\x10uplinkInterfaces\x12$\n\rpopulate_nsid\x18\xe6\x01 \x01(\x08R\x0cpopulateNsid\x12+\n\x11\x65nrich_asn_enable\x18\xf0\x01 \x01(\x08R\x0f\x65nrichAsnEnable\x12)\n\x0b\x61sn_db_path\x18\xf1\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\tasnDbPath\x12L\n\x14\x61sn_refresh_interval\x18\xf2\x01 \x01(\x0b\x32\x19.google.protobuf.DurationR\x12\x61snRefreshInterval\x12=\n\x15ipmeta_bootstrap_path\x18\xf3\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\x13ipmetaBootstrapPath\x12\x35\n\x11ipmeta_cache_path\x18\xf4\x01 \x01(\tB\x08\xbaH\x05r\x03\x18\xff\x01R\x0fipmetaCachePath\x12\x35\n\x16\x65nrich_locality_enable\x18\xf5\x01 \x01(\x08R\x14\x65nrichLocalityEnable\x12V\n\x19locality_refresh_interval\x18\xf6\x01 \x01(\x0b\x32\x19.google.protobuf.DurationR\x17localityRefreshInterval\x12\x35\n\x16locality_events_enable\x18\xf7\x01 \x01(\x08R\x14localityEventsEnable\x12\x45\n\x0clink_monitor\x18\xfa\x01 \x01(\x0b\x32!.xtcp_config.v1.LinkMonitorConfigR\x0blinkMonitor:s\xbaHp\x1an\n\x0fXtcpConfig.poll\x12\x32Poll timeout must be less than poll poll_frequency\x1a\'this.poll_frequency > this.poll_timeout\"\x9f\x01\n\x14\x45nabledDeserializers\x12K\n\x07\x65nabled\x18\x01 \x03(\x0b\x32\x31.xtcp_config.v1.EnabledDeserializers.EnabledEntryR\x07\x65nabled\x1a:\n\x0c\x45nabledEntry\x12\x10\n\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n\x05value\x18\x02 \x01(\x08R\x05value:\x02\x38\x01*h\n\x0fListenerNetwork\x12 \n\x1cLISTENER_NETWORK_UNSPECIFIED\x10\x00\x12\x18\n\x14LISTENER_NETWORK_TCP\x10\x01\x12\x19\n\x15LISTENER_NETWORK_UNIX\x10\x02*\xa1\x01\n\x10ListenerAuthMode\x12\"\n\x1eLISTENER_AUTH_MODE_UNSPECIFIED\x10\x00\x12\x1f\n\x1bLISTENER_AUTH_MODE_DISABLED\x10\x01\x12 \n\x1cLISTENER_AUTH_MODE_RAW_TOKEN\x10\x02\x12&\n\"LISTENER_AUTH_MODE_HMAC_UTC_MINUTE\x10\x03*\xf7\x01\n\rEndpointClass\x12\x1e\n\x1a\x45NDPOINT_CLASS_UNSPECIFIED\x10\x00\x12\x19\n\x15\x45NDPOINT_CLASS_HEALTH\x10\x01\x12\x1a\n\x16\x45NDPOINT_CLASS_METRICS\x10\x02\x12!\n\x1d\x45NDPOINT_CLASS_METRICS_DETAIL\x10\x03\x12\x15\n\x11\x45NDPOINT_CLASS_IP\x10\x04\x12\x18\n\x14\x45NDPOINT_CLASS_DEBUG\x10\x05\x12\x1c\n\x18\x45NDPOINT_CLASS_GRPC_READ\x10\x06\x12\x1d\n\x19\x45NDPOINT_CLASS_GRPC_ADMIN\x10\x07\x32\xa5\x07\n\rConfigService\x12\x61\n\x03Get\x12\x1a.xtcp_config.v1.GetRequest\x1a\x1b.xtcp_config.v1.GetResponse\"!\xc8\xf3\x18\x01\x82\xd3\xe4\x93\x02\x17\x1a\x12/ConfigService/Get:\x01*\x12\x61\n\x03Set\x12\x1a.xtcp_config.v1.SetRequest\x1a\x1b.xtcp_config.v1.SetResponse\"!\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02\x17\x1a\x12/ConfigService/Set:\x01*\x12\x95\x01\n\x10SetPollFrequency\x12\'.xtcp_config.v1.SetPollFrequencyRequest\x1a(.xtcp_config.v1.SetPollFrequencyResponse\".\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02$\x1a\x1f/ConfigService/SetPollFrequency:\x01*\x12\x81\x01\n\x0bTriggerPoll\x12\".xtcp_config.v1.TriggerPollRequest\x1a#.xtcp_config.v1.TriggerPollResponse\")\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02\x1f\x1a\x1a/ConfigService/TriggerPoll:\x01*\x12\x95\x01\n\x10TriggerPollBurst\x12\'.xtcp_config.v1.TriggerPollBurstRequest\x1a(.xtcp_config.v1.TriggerPollBurstResponse\".\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02$\x1a\x1f/ConfigService/TriggerPollBurst:\x01*\x12\x81\x01\n\x0bSetS3Upload\x12\".xtcp_config.v1.SetS3UploadRequest\x1a#.xtcp_config.v1.SetS3UploadResponse\")\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02\x1f\x1a\x1a/ConfigService/SetS3Upload:\x01*\x12\x95\x01\n\x10SetEnvelopeFlush\x12\'.xtcp_config.v1.SetEnvelopeFlushRequest\x1a(.xtcp_config.v1.SetEnvelopeFlushResponse\".\xc8\xf3\x18\x02\x82\xd3\xe4\x93\x02$\x1a\x1f/ConfigService/SetEnvelopeFlush:\x01*B\xaf\x01\n\x12\x63om.xtcp_config.v1B\x0fXtcpConfigProtoP\x01Z3github.com/randomizedcoder/xtcp2/gen/go/xtcp_config\xa2\x02\x03XXX\xaa\x02\rXtcpConfig.V1\xca\x02\rXtcpConfig\\V1\xe2\x02\x19XtcpConfig\\V1\\GPBMetadata\xea\x02\x0eXtcpConfig::V1b\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -252,6 +252,10 @@ _globals['_XTCPCONFIG'].fields_by_name['s3_upload_max_attempts']._serialized_options = b'\272H\t*\004\030d(\001\310\001\000' _globals['_XTCPCONFIG'].fields_by_name['s3_upload_backoff_cap']._loaded_options = None _globals['_XTCPCONFIG'].fields_by_name['s3_upload_backoff_cap']._serialized_options = b'\272H\010\252\001\0022\000\310\001\000' + _globals['_XTCPCONFIG'].fields_by_name['s3_queue_depth']._loaded_options = None + _globals['_XTCPCONFIG'].fields_by_name['s3_queue_depth']._serialized_options = b'\272H\010*\003\030\200 \310\001\000' + _globals['_XTCPCONFIG'].fields_by_name['s3_max_buffered_bytes']._loaded_options = None + _globals['_XTCPCONFIG'].fields_by_name['s3_max_buffered_bytes']._serialized_options = b'\272H\003\310\001\000' _globals['_XTCPCONFIG'].fields_by_name['hostname']._loaded_options = None _globals['_XTCPCONFIG'].fields_by_name['hostname']._serialized_options = b'\272H\010r\003\030\375\001\310\001\000' _globals['_XTCPCONFIG'].fields_by_name['location']._loaded_options = None @@ -322,12 +326,12 @@ _globals['_CONFIGSERVICE'].methods_by_name['SetS3Upload']._serialized_options = b'\310\363\030\002\202\323\344\223\002\037\032\032/ConfigService/SetS3Upload:\001*' _globals['_CONFIGSERVICE'].methods_by_name['SetEnvelopeFlush']._loaded_options = None _globals['_CONFIGSERVICE'].methods_by_name['SetEnvelopeFlush']._serialized_options = b'\310\363\030\002\202\323\344\223\002$\032\037/ConfigService/SetEnvelopeFlush:\001*' - _globals['_LISTENERNETWORK']._serialized_start=10898 - _globals['_LISTENERNETWORK']._serialized_end=11002 - _globals['_LISTENERAUTHMODE']._serialized_start=11005 - _globals['_LISTENERAUTHMODE']._serialized_end=11166 - _globals['_ENDPOINTCLASS']._serialized_start=11169 - _globals['_ENDPOINTCLASS']._serialized_end=11416 + _globals['_LISTENERNETWORK']._serialized_start=11008 + _globals['_LISTENERNETWORK']._serialized_end=11112 + _globals['_LISTENERAUTHMODE']._serialized_start=11115 + _globals['_LISTENERAUTHMODE']._serialized_end=11276 + _globals['_ENDPOINTCLASS']._serialized_start=11279 + _globals['_ENDPOINTCLASS']._serialized_end=11526 _globals['_GETREQUEST']._serialized_start=173 _globals['_GETREQUEST']._serialized_end=185 _globals['_GETRESPONSE']._serialized_start=187 @@ -365,11 +369,11 @@ _globals['_LINKMONITORCONFIG']._serialized_start=3450 _globals['_LINKMONITORCONFIG']._serialized_end=5268 _globals['_XTCPCONFIG']._serialized_start=5271 - _globals['_XTCPCONFIG']._serialized_end=10734 - _globals['_ENABLEDDESERIALIZERS']._serialized_start=10737 - _globals['_ENABLEDDESERIALIZERS']._serialized_end=10896 - _globals['_ENABLEDDESERIALIZERS_ENABLEDENTRY']._serialized_start=10838 - _globals['_ENABLEDDESERIALIZERS_ENABLEDENTRY']._serialized_end=10896 - _globals['_CONFIGSERVICE']._serialized_start=11419 - _globals['_CONFIGSERVICE']._serialized_end=12352 + _globals['_XTCPCONFIG']._serialized_end=10844 + _globals['_ENABLEDDESERIALIZERS']._serialized_start=10847 + _globals['_ENABLEDDESERIALIZERS']._serialized_end=11006 + _globals['_ENABLEDDESERIALIZERS_ENABLEDENTRY']._serialized_start=10948 + _globals['_ENABLEDDESERIALIZERS_ENABLEDENTRY']._serialized_end=11006 + _globals['_CONFIGSERVICE']._serialized_start=11529 + _globals['_CONFIGSERVICE']._serialized_end=12462 # @@protoc_insertion_point(module_scope) diff --git a/gen/python/xtcp_config/v1/xtcp_config_pb2.pyi b/gen/python/xtcp_config/v1/xtcp_config_pb2.pyi index 00eca9537..a24ffa4db 100644 --- a/gen/python/xtcp_config/v1/xtcp_config_pb2.pyi +++ b/gen/python/xtcp_config/v1/xtcp_config_pb2.pyi @@ -253,7 +253,7 @@ class LinkMonitorConfig(_message.Message): def __init__(self, enable: _Optional[bool] = ..., baseline_file: _Optional[str] = ..., settle: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., resync: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., stats_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., stats_include: _Optional[str] = ..., stats_exclude: _Optional[str] = ..., netstat_fields: _Optional[str] = ..., max_speed_exceptions: _Optional[_Iterable[str]] = ..., metrics_core_include: _Optional[str] = ..., metrics_core_exclude: _Optional[str] = ..., metrics_all: _Optional[bool] = ..., collection_mode: _Optional[str] = ..., watch_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., slow_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., on_demand_ttl: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., on_demand_wait: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., error_window: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., flap_window: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., error_classes: _Optional[str] = ..., error_exclude: _Optional[str] = ..., metrics_categories: _Optional[str] = ..., metrics_interface_paths: _Optional[bool] = ..., metrics_all_path: _Optional[str] = ..., ip_paths: _Optional[bool] = ..., ip_max_bytes: _Optional[int] = ..., ip_listen: _Optional[str] = ..., learn_deadline: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ...) -> None: ... class XtcpConfig(_message.Message): - __slots__ = ("nl_timeout_milliseconds", "poll_frequency", "poll_timeout", "poll_jitter_pct", "max_loops", "netlinkers", "netlinkers_done_chan_size", "nlmsg_seq", "packet_size", "packet_size_mply", "modulus", "enabled_deserializers", "io_uring", "io_uring_recv_batch_size", "io_uring_cqe_batch_size", "max_dump_bytes", "max_records_per_netns", "max_records_per_poll", "io_uring_sockstat_batch_size", "io_uring_sockstat_workers", "reconcile_frequency", "reconcile_before_poll", "max_netns", "reconcile_timeout", "max_netns_per_owner", "max_netns_adds_per_reconcile", "write_files", "capture_path", "dest_write_files", "debug_level", "debug_prometheus", "dest", "marshal_to", "csv_columns", "xtcp_proto_file", "envelope_flush_threshold_bytes", "envelope_flush_threshold_rows", "topic", "kafka_schema_url", "kafka_produce_timeout", "kafka_compression", "s3_endpoint", "s3_region", "s3_bucket", "s3_prefix", "s3_access_key", "s3_secret_key", "s3_skip_bucket_probe", "s3_allow_insecure", "s3_ca_file", "s3_parquet_flush_threshold_bytes", "s3_flush_interval", "s3_flush_jitter_pct", "s3_flush_threshold_jitter_pct", "s3_upload_max_attempts", "s3_upload_backoff_cap", "hostname", "location", "label", "tag", "daemon_version", "ipv4_ttl", "ipv6_hop_limit", "listener_auth", "prometheus_listener", "admin_listener_auth", "grpc_port", "grpc_listener", "listeners", "pyroscope_url", "pyroscope_app_name", "pyroscope_sample_hz", "pyroscope_upload_interval_sec", "resolve_container_id", "enrich_container_enable", "docker_socket_path", "enrich_lldp_enable", "lldpd_socket_path", "lldpd_version_hint", "enrich_nic_enable", "uplink_count", "uplink_interfaces", "populate_nsid", "enrich_asn_enable", "asn_db_path", "asn_refresh_interval", "ipmeta_bootstrap_path", "ipmeta_cache_path", "enrich_locality_enable", "locality_refresh_interval", "locality_events_enable", "link_monitor") + __slots__ = ("nl_timeout_milliseconds", "poll_frequency", "poll_timeout", "poll_jitter_pct", "max_loops", "netlinkers", "netlinkers_done_chan_size", "nlmsg_seq", "packet_size", "packet_size_mply", "modulus", "enabled_deserializers", "io_uring", "io_uring_recv_batch_size", "io_uring_cqe_batch_size", "max_dump_bytes", "max_records_per_netns", "max_records_per_poll", "io_uring_sockstat_batch_size", "io_uring_sockstat_workers", "reconcile_frequency", "reconcile_before_poll", "max_netns", "reconcile_timeout", "max_netns_per_owner", "max_netns_adds_per_reconcile", "write_files", "capture_path", "dest_write_files", "debug_level", "debug_prometheus", "dest", "marshal_to", "csv_columns", "xtcp_proto_file", "envelope_flush_threshold_bytes", "envelope_flush_threshold_rows", "topic", "kafka_schema_url", "kafka_produce_timeout", "kafka_compression", "s3_endpoint", "s3_region", "s3_bucket", "s3_prefix", "s3_access_key", "s3_secret_key", "s3_skip_bucket_probe", "s3_allow_insecure", "s3_ca_file", "s3_parquet_flush_threshold_bytes", "s3_flush_interval", "s3_flush_jitter_pct", "s3_flush_threshold_jitter_pct", "s3_upload_max_attempts", "s3_upload_backoff_cap", "s3_queue_depth", "s3_max_buffered_bytes", "hostname", "location", "label", "tag", "daemon_version", "ipv4_ttl", "ipv6_hop_limit", "listener_auth", "prometheus_listener", "admin_listener_auth", "grpc_port", "grpc_listener", "listeners", "pyroscope_url", "pyroscope_app_name", "pyroscope_sample_hz", "pyroscope_upload_interval_sec", "resolve_container_id", "enrich_container_enable", "docker_socket_path", "enrich_lldp_enable", "lldpd_socket_path", "lldpd_version_hint", "enrich_nic_enable", "uplink_count", "uplink_interfaces", "populate_nsid", "enrich_asn_enable", "asn_db_path", "asn_refresh_interval", "ipmeta_bootstrap_path", "ipmeta_cache_path", "enrich_locality_enable", "locality_refresh_interval", "locality_events_enable", "link_monitor") NL_TIMEOUT_MILLISECONDS_FIELD_NUMBER: _ClassVar[int] POLL_FREQUENCY_FIELD_NUMBER: _ClassVar[int] POLL_TIMEOUT_FIELD_NUMBER: _ClassVar[int] @@ -310,6 +310,8 @@ class XtcpConfig(_message.Message): S3_FLUSH_THRESHOLD_JITTER_PCT_FIELD_NUMBER: _ClassVar[int] S3_UPLOAD_MAX_ATTEMPTS_FIELD_NUMBER: _ClassVar[int] S3_UPLOAD_BACKOFF_CAP_FIELD_NUMBER: _ClassVar[int] + S3_QUEUE_DEPTH_FIELD_NUMBER: _ClassVar[int] + S3_MAX_BUFFERED_BYTES_FIELD_NUMBER: _ClassVar[int] HOSTNAME_FIELD_NUMBER: _ClassVar[int] LOCATION_FIELD_NUMBER: _ClassVar[int] LABEL_FIELD_NUMBER: _ClassVar[int] @@ -402,6 +404,8 @@ class XtcpConfig(_message.Message): s3_flush_threshold_jitter_pct: int s3_upload_max_attempts: int s3_upload_backoff_cap: _duration_pb2.Duration + s3_queue_depth: int + s3_max_buffered_bytes: int hostname: str location: str label: str @@ -438,7 +442,7 @@ class XtcpConfig(_message.Message): locality_refresh_interval: _duration_pb2.Duration locality_events_enable: bool link_monitor: LinkMonitorConfig - def __init__(self, nl_timeout_milliseconds: _Optional[int] = ..., poll_frequency: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., poll_timeout: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., poll_jitter_pct: _Optional[int] = ..., max_loops: _Optional[int] = ..., netlinkers: _Optional[int] = ..., netlinkers_done_chan_size: _Optional[int] = ..., nlmsg_seq: _Optional[int] = ..., packet_size: _Optional[int] = ..., packet_size_mply: _Optional[int] = ..., modulus: _Optional[int] = ..., enabled_deserializers: _Optional[_Union[EnabledDeserializers, _Mapping]] = ..., io_uring: _Optional[bool] = ..., io_uring_recv_batch_size: _Optional[int] = ..., io_uring_cqe_batch_size: _Optional[int] = ..., max_dump_bytes: _Optional[int] = ..., max_records_per_netns: _Optional[int] = ..., max_records_per_poll: _Optional[int] = ..., io_uring_sockstat_batch_size: _Optional[int] = ..., io_uring_sockstat_workers: _Optional[int] = ..., reconcile_frequency: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., reconcile_before_poll: _Optional[bool] = ..., max_netns: _Optional[int] = ..., reconcile_timeout: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., max_netns_per_owner: _Optional[int] = ..., max_netns_adds_per_reconcile: _Optional[int] = ..., write_files: _Optional[int] = ..., capture_path: _Optional[str] = ..., dest_write_files: _Optional[int] = ..., debug_level: _Optional[int] = ..., debug_prometheus: _Optional[bool] = ..., dest: _Optional[str] = ..., marshal_to: _Optional[str] = ..., csv_columns: _Optional[str] = ..., xtcp_proto_file: _Optional[str] = ..., envelope_flush_threshold_bytes: _Optional[int] = ..., envelope_flush_threshold_rows: _Optional[int] = ..., topic: _Optional[str] = ..., kafka_schema_url: _Optional[str] = ..., kafka_produce_timeout: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., kafka_compression: _Optional[str] = ..., s3_endpoint: _Optional[str] = ..., s3_region: _Optional[str] = ..., s3_bucket: _Optional[str] = ..., s3_prefix: _Optional[str] = ..., s3_access_key: _Optional[str] = ..., s3_secret_key: _Optional[str] = ..., s3_skip_bucket_probe: _Optional[bool] = ..., s3_allow_insecure: _Optional[bool] = ..., s3_ca_file: _Optional[str] = ..., s3_parquet_flush_threshold_bytes: _Optional[int] = ..., s3_flush_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., s3_flush_jitter_pct: _Optional[int] = ..., s3_flush_threshold_jitter_pct: _Optional[int] = ..., s3_upload_max_attempts: _Optional[int] = ..., s3_upload_backoff_cap: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., hostname: _Optional[str] = ..., location: _Optional[str] = ..., label: _Optional[str] = ..., tag: _Optional[str] = ..., daemon_version: _Optional[str] = ..., ipv4_ttl: _Optional[int] = ..., ipv6_hop_limit: _Optional[int] = ..., listener_auth: _Optional[_Union[ListenerAuth, _Mapping]] = ..., prometheus_listener: _Optional[_Union[ListenerEndpoint, _Mapping]] = ..., admin_listener_auth: _Optional[_Union[ListenerAuth, _Mapping]] = ..., grpc_port: _Optional[int] = ..., grpc_listener: _Optional[_Union[ListenerEndpoint, _Mapping]] = ..., listeners: _Optional[_Iterable[_Union[Listener, _Mapping]]] = ..., pyroscope_url: _Optional[str] = ..., pyroscope_app_name: _Optional[str] = ..., pyroscope_sample_hz: _Optional[int] = ..., pyroscope_upload_interval_sec: _Optional[int] = ..., resolve_container_id: _Optional[bool] = ..., enrich_container_enable: _Optional[bool] = ..., docker_socket_path: _Optional[str] = ..., enrich_lldp_enable: _Optional[bool] = ..., lldpd_socket_path: _Optional[str] = ..., lldpd_version_hint: _Optional[str] = ..., enrich_nic_enable: _Optional[bool] = ..., uplink_count: _Optional[int] = ..., uplink_interfaces: _Optional[_Iterable[str]] = ..., populate_nsid: _Optional[bool] = ..., enrich_asn_enable: _Optional[bool] = ..., asn_db_path: _Optional[str] = ..., asn_refresh_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., ipmeta_bootstrap_path: _Optional[str] = ..., ipmeta_cache_path: _Optional[str] = ..., enrich_locality_enable: _Optional[bool] = ..., locality_refresh_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., locality_events_enable: _Optional[bool] = ..., link_monitor: _Optional[_Union[LinkMonitorConfig, _Mapping]] = ...) -> None: ... + def __init__(self, nl_timeout_milliseconds: _Optional[int] = ..., poll_frequency: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., poll_timeout: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., poll_jitter_pct: _Optional[int] = ..., max_loops: _Optional[int] = ..., netlinkers: _Optional[int] = ..., netlinkers_done_chan_size: _Optional[int] = ..., nlmsg_seq: _Optional[int] = ..., packet_size: _Optional[int] = ..., packet_size_mply: _Optional[int] = ..., modulus: _Optional[int] = ..., enabled_deserializers: _Optional[_Union[EnabledDeserializers, _Mapping]] = ..., io_uring: _Optional[bool] = ..., io_uring_recv_batch_size: _Optional[int] = ..., io_uring_cqe_batch_size: _Optional[int] = ..., max_dump_bytes: _Optional[int] = ..., max_records_per_netns: _Optional[int] = ..., max_records_per_poll: _Optional[int] = ..., io_uring_sockstat_batch_size: _Optional[int] = ..., io_uring_sockstat_workers: _Optional[int] = ..., reconcile_frequency: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., reconcile_before_poll: _Optional[bool] = ..., max_netns: _Optional[int] = ..., reconcile_timeout: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., max_netns_per_owner: _Optional[int] = ..., max_netns_adds_per_reconcile: _Optional[int] = ..., write_files: _Optional[int] = ..., capture_path: _Optional[str] = ..., dest_write_files: _Optional[int] = ..., debug_level: _Optional[int] = ..., debug_prometheus: _Optional[bool] = ..., dest: _Optional[str] = ..., marshal_to: _Optional[str] = ..., csv_columns: _Optional[str] = ..., xtcp_proto_file: _Optional[str] = ..., envelope_flush_threshold_bytes: _Optional[int] = ..., envelope_flush_threshold_rows: _Optional[int] = ..., topic: _Optional[str] = ..., kafka_schema_url: _Optional[str] = ..., kafka_produce_timeout: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., kafka_compression: _Optional[str] = ..., s3_endpoint: _Optional[str] = ..., s3_region: _Optional[str] = ..., s3_bucket: _Optional[str] = ..., s3_prefix: _Optional[str] = ..., s3_access_key: _Optional[str] = ..., s3_secret_key: _Optional[str] = ..., s3_skip_bucket_probe: _Optional[bool] = ..., s3_allow_insecure: _Optional[bool] = ..., s3_ca_file: _Optional[str] = ..., s3_parquet_flush_threshold_bytes: _Optional[int] = ..., s3_flush_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., s3_flush_jitter_pct: _Optional[int] = ..., s3_flush_threshold_jitter_pct: _Optional[int] = ..., s3_upload_max_attempts: _Optional[int] = ..., s3_upload_backoff_cap: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., s3_queue_depth: _Optional[int] = ..., s3_max_buffered_bytes: _Optional[int] = ..., hostname: _Optional[str] = ..., location: _Optional[str] = ..., label: _Optional[str] = ..., tag: _Optional[str] = ..., daemon_version: _Optional[str] = ..., ipv4_ttl: _Optional[int] = ..., ipv6_hop_limit: _Optional[int] = ..., listener_auth: _Optional[_Union[ListenerAuth, _Mapping]] = ..., prometheus_listener: _Optional[_Union[ListenerEndpoint, _Mapping]] = ..., admin_listener_auth: _Optional[_Union[ListenerAuth, _Mapping]] = ..., grpc_port: _Optional[int] = ..., grpc_listener: _Optional[_Union[ListenerEndpoint, _Mapping]] = ..., listeners: _Optional[_Iterable[_Union[Listener, _Mapping]]] = ..., pyroscope_url: _Optional[str] = ..., pyroscope_app_name: _Optional[str] = ..., pyroscope_sample_hz: _Optional[int] = ..., pyroscope_upload_interval_sec: _Optional[int] = ..., resolve_container_id: _Optional[bool] = ..., enrich_container_enable: _Optional[bool] = ..., docker_socket_path: _Optional[str] = ..., enrich_lldp_enable: _Optional[bool] = ..., lldpd_socket_path: _Optional[str] = ..., lldpd_version_hint: _Optional[str] = ..., enrich_nic_enable: _Optional[bool] = ..., uplink_count: _Optional[int] = ..., uplink_interfaces: _Optional[_Iterable[str]] = ..., populate_nsid: _Optional[bool] = ..., enrich_asn_enable: _Optional[bool] = ..., asn_db_path: _Optional[str] = ..., asn_refresh_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., ipmeta_bootstrap_path: _Optional[str] = ..., ipmeta_cache_path: _Optional[str] = ..., enrich_locality_enable: _Optional[bool] = ..., locality_refresh_interval: _Optional[_Union[datetime.timedelta, _duration_pb2.Duration, _Mapping]] = ..., locality_events_enable: _Optional[bool] = ..., link_monitor: _Optional[_Union[LinkMonitorConfig, _Mapping]] = ...) -> None: ... class EnabledDeserializers(_message.Message): __slots__ = ("enabled",) diff --git a/nix/microvms/default.nix b/nix/microvms/default.nix index 843a64807..2692b3b9c 100644 --- a/nix/microvms/default.nix +++ b/nix/microvms/default.nix @@ -889,6 +889,7 @@ let "S3PARQUET_FILES" "S3PARQUET_ROWS" "S3PARQUET_EVENTDATE" + "S3PARQUET_OUTAGE" ]; }; }); diff --git a/nix/microvms/mkVm.nix b/nix/microvms/mkVm.nix index f545fc729..cb17e916d 100644 --- a/nix/microvms/mkVm.nix +++ b/nix/microvms/mkVm.nix @@ -2587,6 +2587,10 @@ let "xtcp2testsecretkey" "-s3ParquetFlushBytes" "1048576" + # A shallow upload queue, so the self-test's S3 outage (Check 14c) + # overflows it within a few polls. + "-s3QueueDepth" + "2" ]; # The three broker flavors (valkey/nats/nsq) share one arg shape: publish diff --git a/nix/microvms/self-test.nix b/nix/microvms/self-test.nix index d767e441d..3c6abc93d 100644 --- a/nix/microvms/self-test.nix +++ b/nix/microvms/self-test.nix @@ -84,6 +84,10 @@ # XTCP2_SELF_TEST_S3PARQUET_ROWS_{PASS,FAIL} (s3parquet only) # duckdb decodes the file and # returns ≥1 row +# XTCP2_SELF_TEST_S3PARQUET_OUTAGE_{PASS,FAIL} (s3parquet only) +# Garage stopped: polls continue, +# xtcp_s3_dropped_total rises, +# uploads resume after restart # XTCP2_SELF_TEST_VALKEY_CONSUME_{PASS,FAIL} (valkey only) # ≥1 record published to a real # in-VM Valkey server AND consumed @@ -1667,6 +1671,49 @@ pkgs.writeShellApplication { echo "XTCP2_SELF_TEST_S3PARQUET_EVENTDATE_FAIL (no parquet file to test)" fi if [ "$check14b" -ne 0 ]; then overall_ok=0; fi + + # ─── Check 14c: S3 outage — polling continues, drops are counted ── + # Stop Garage for a while. The upload worker blocks in its retries, + # the bounded queue (-s3QueueDepth 2 in this flavor) drops the oldest + # envelopes, and Send never blocks the poller: polls keep their + # cadence, xtcp_s3_dropped_total rises and RSS stays bounded. After + # Garage restarts, a new object lands. + echo "--- check 14c: s3parquet — S3 outage drops, never stalls polling ---" + check14c=1 + xtcp2_rss_kb() { + awk '/^VmRSS:/ {print $2}' "/proc/$(systemctl show -p MainPID --value xtcp2)/status" 2>/dev/null || echo 0 + } + objects_before=$(rclone lsf -R --files-only --include '*.parquet' s3:${s3Bucket} 2>/dev/null | wc -l) + polls_before=$(metric_value "xtcp_counts" 'function="Poller"' 'variable="pollingLoops"') + drops_before=$(metric_value "xtcp_s3_dropped_total" "" "") + rss_before=$(xtcp2_rss_kb) + outage_s=$(scaled 40) + systemctl stop garage + sleep "$outage_s" + polls_during=$(( $(metric_value "xtcp_counts" 'function="Poller"' 'variable="pollingLoops"') - polls_before )) + drops_during=$(( $(metric_value "xtcp_s3_dropped_total" "" "") - drops_before )) + rss_after=$(xtcp2_rss_kb) + systemctl start garage + # -frequency 2s: expect at least half the polls the outage had room for. + min_polls=$(( outage_s / 4 )) + objects_after=$objects_before + for _ in $(seq 1 "$(scaled 90)"); do + objects_after=$(rclone lsf -R --files-only --include '*.parquet' s3:${s3Bucket} 2>/dev/null | wc -l) + if [ "$objects_after" -gt "$objects_before" ]; then + break + fi + sleep 1 + done + rss_growth_kb=$(( rss_after - rss_before )) + detail="polls=$polls_during (min $min_polls), dropped=$drops_during, rss_kb=$rss_before->$rss_after, objects=$objects_before->$objects_after" + if [ "$polls_during" -ge "$min_polls" ] && [ "$drops_during" -gt 0 ] \ + && [ "$rss_growth_kb" -lt 131072 ] && [ "$objects_after" -gt "$objects_before" ]; then + echo "XTCP2_SELF_TEST_S3PARQUET_OUTAGE_PASS ($detail)" + check14c=0 + else + echo "XTCP2_SELF_TEST_S3PARQUET_OUTAGE_FAIL ($detail)" + fi + if [ "$check14c" -ne 0 ]; then overall_ok=0; fi ''} ${lib.optionalString runValkeyCheck '' diff --git a/pkg/dockermeta/dockermeta.go b/pkg/dockermeta/dockermeta.go index 7763dad46..c3c5209d1 100644 --- a/pkg/dockermeta/dockermeta.go +++ b/pkg/dockermeta/dockermeta.go @@ -31,6 +31,8 @@ import ( "time" "golang.org/x/sys/unix" + + "github.com/randomizedcoder/xtcp2/pkg/strpolicy" ) // DefaultSocketPath is the standard Docker Engine API unix socket. @@ -282,10 +284,12 @@ func (ix *Index) inspect(ctx context.Context, id string) (Container, bool) { if len(name) > 0 && name[0] == '/' { name = name[1:] } + // The name and image are the tenant's choice: clean them once here, + // before they reach records and logs. return Container{ ID: in.ID, - Name: name, - Image: image, + Name: strpolicy.Clean(name, strpolicy.MaxLen), + Image: strpolicy.Clean(image, strpolicy.MaxLen), Pid: in.State.Pid, SandboxKey: in.NetworkSettings.SandboxKey, NetnsInode: ino, diff --git a/pkg/dockermeta/dockermeta_test.go b/pkg/dockermeta/dockermeta_test.go index 6aff3f3bf..6abb390cc 100644 --- a/pkg/dockermeta/dockermeta_test.go +++ b/pkg/dockermeta/dockermeta_test.go @@ -509,3 +509,46 @@ func TestURLQueryEscape(t *testing.T) { } } } + +// TestInspectCleansTenantStrings: a container's name and image are the +// tenant's choice; inspect bounds and cleans them once. +func TestInspectCleansTenantStrings(t *testing.T) { + const id = "aaaa111111111111111111111111111111111111111111111111111111111111" + tests := []struct { + name string + category string + description string + cName string // JSON string body + cImage string // JSON string body + expectedName string + expectedImage string + }{ + {name: "plain", category: "P", description: "ordinary name and image pass unchanged", + cName: "/web", cImage: "nginx:1.27", expectedName: "web", expectedImage: "nginx:1.27"}, + {name: "long_container_name", category: "N", description: "a 10 kB name is cut to 256 bytes", + cName: "/" + strings.Repeat("a", 10000), cImage: "nginx", expectedName: strings.Repeat("a", 256), expectedImage: "nginx"}, + {name: "escape_image", category: "N", description: "a terminal escape in the image is replaced", + cName: "/web", cImage: `nginx\u001b[2J`, expectedName: "web", expectedImage: "nginx�[2J"}, + {name: "exact_max_name", category: "B", description: "a 256-byte name is kept whole", + cName: "/" + strings.Repeat("b", 256), cImage: "x", expectedName: strings.Repeat("b", 256), expectedImage: "x"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + fs, srv := newFixtureServer(t) + fs.inspects[id] = fmt.Sprintf(`{"Id":%q,"Name":"%s","Config":{"Image":"%s"},"State":{"Pid":1},`+ + `"NetworkSettings":{"SandboxKey":"/var/run/docker/netns/aaaahash"}}`, id, tc.cName, tc.cImage) + ix, err := newIndexFromClient(context.Background(), srv.Client(), srv.URL, fakeStat) + if err != nil { + t.Fatalf("newIndexFromClient: %v", err) + } + defer ix.Close() + c, ok := ix.Lookup(4026531001) + if !ok { + t.Fatalf("%s: container not indexed", tc.description) + } + if c.Name != tc.expectedName || c.Image != tc.expectedImage { + t.Errorf("%s: got name %q image %q, expected %q %q", tc.description, c.Name, c.Image, tc.expectedName, tc.expectedImage) + } + }) + } +} diff --git a/pkg/ipasn/ipasn.go b/pkg/ipasn/ipasn.go index 2d7af948f..d294ca8e8 100644 --- a/pkg/ipasn/ipasn.go +++ b/pkg/ipasn/ipasn.go @@ -32,6 +32,8 @@ import ( "github.com/gaissmai/bart" "github.com/klauspost/compress/zstd" "github.com/parquet-go/parquet-go" + + "github.com/randomizedcoder/xtcp2/pkg/strpolicy" ) // ErrNoPrefixes is returned when an artifact opens and reads cleanly but yields @@ -307,11 +309,12 @@ func buildFromReaderAt(name string, r io.ReaderAt, size int64) (*bart.Table[Attr if perr != nil { continue // artifact is collector-canonicalized; skip any stray row } - t.Insert(pfx.Masked(), Attr{ASN: buf[i].ASN, NetworkOwner: buf[i].NetworkOwner}) + owner := strpolicy.Clean(buf[i].NetworkOwner, strpolicy.MaxLen) // feed data, not ours + t.Insert(pfx.Masked(), Attr{ASN: buf[i].ASN, NetworkOwner: owner}) rows = append(rows, row{ Prefix: pfx.Masked().String(), ASN: buf[i].ASN, - NetworkOwner: buf[i].NetworkOwner, + NetworkOwner: owner, }) } if err != nil { diff --git a/pkg/ipasn/ipasn_test.go b/pkg/ipasn/ipasn_test.go index e4a205b79..cc5a65244 100644 --- a/pkg/ipasn/ipasn_test.go +++ b/pkg/ipasn/ipasn_test.go @@ -1175,3 +1175,36 @@ func TestWriteRowsZstdCloseJoins_table(t *testing.T) { }) } } + +// TestLoadCleansOwner: the network owner is feed data; the load bounds and +// cleans it once per row. +func TestLoadCleansOwner(t *testing.T) { + tests := []struct { + name string + category string + description string + owner string + expected string + }{ + {name: "plain", category: "P", description: "an ordinary owner is unchanged", owner: "cloudflare", expected: "cloudflare"}, + {name: "asn_owner_long", category: "N", description: "a 1 kB owner is cut to 256 bytes", + owner: strings.Repeat("o", 1024), expected: strings.Repeat("o", 256)}, + {name: "escape", category: "N", description: "an escape sequence is replaced", owner: "evil\x1b]0;x\x07", expected: "evil�]0;x�"}, + {name: "empty", category: "B", description: "an empty owner stays empty", owner: "", expected: ""}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + ix, err := New(writeArtifact(t, []row{{Prefix: "1.1.1.0/24", ASN: 13335, NetworkOwner: tc.owner}})) + if err != nil { + t.Fatalf("New: %v", err) + } + a, ok := ix.Lookup(netip.MustParseAddr("1.1.1.1")) + if !ok { + t.Fatalf("%s: no match", tc.description) + } + if a.NetworkOwner != tc.expected { + t.Errorf("%s: owner %q, expected %q", tc.description, a.NetworkOwner, tc.expected) + } + }) + } +} diff --git a/pkg/localnet/localnet.go b/pkg/localnet/localnet.go index f88f2214c..1dcd1ba11 100644 --- a/pkg/localnet/localnet.go +++ b/pkg/localnet/localnet.go @@ -20,6 +20,7 @@ import ( "github.com/gaissmai/bart" "golang.org/x/sys/unix" + "github.com/randomizedcoder/xtcp2/pkg/strpolicy" "github.com/randomizedcoder/xtcp2/pkg/xtcpnl" ) @@ -201,7 +202,9 @@ func BuildSnapshot(addrs []xtcpnl.AddrInfo, routes []xtcpnl.RouteInfo, links map ifnames := make(map[uint32]string, len(links)) for idx, name := range links { - ifnames[idx] = name + // A tenant names the interfaces in its own netns, and the kernel + // allows control bytes and invalid UTF-8 up to IFNAMSIZ. + ifnames[idx] = strpolicy.Clean(name, strpolicy.MaxIfname) } nonLoopbackSelf := false diff --git a/pkg/localnet/localnet_test.go b/pkg/localnet/localnet_test.go index f8e105a54..4bfd007b0 100644 --- a/pkg/localnet/localnet_test.go +++ b/pkg/localnet/localnet_test.go @@ -2,6 +2,7 @@ package localnet import ( "net/netip" + "strings" "testing" "golang.org/x/sys/unix" @@ -898,3 +899,29 @@ func TestHasNonLoopbackSelf(t *testing.T) { }) } } + +// TestIfNameStrPolicy: a tenant names the interfaces in its own netns, so +// the snapshot cleans each name once. +func TestIfNameStrPolicy(t *testing.T) { + tests := []struct { + name string + category string + description string + in string + expected string + }{ + {name: "plain", category: "P", description: "an ordinary name is unchanged", in: "eth0", expected: "eth0"}, + {name: "tenant_ifname", category: "N", description: "ESC in the name is replaced", in: "e\x1bth", expected: "e�th"}, + {name: "ifnamsiz", category: "B", description: "a 15-byte name (IFNAMSIZ-1) is kept", in: "abcdefghijklmno", expected: "abcdefghijklmno"}, + {name: "invalid_utf8_full", category: "C", description: "15 invalid bytes fit five U+FFFD in 16 bytes", + in: strings.Repeat("\xff", 15), expected: strings.Repeat("�", 5)}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + snap := BuildSnapshot(nil, nil, map[uint32]string{7: tc.in}) + if got := snap.IfName(7); got != tc.expected { + t.Errorf("%s: IfName = %q, expected %q", tc.description, got, tc.expected) + } + }) + } +} diff --git a/pkg/strpolicy/strpolicy.go b/pkg/strpolicy/strpolicy.go new file mode 100644 index 000000000..7c2413f4f --- /dev/null +++ b/pkg/strpolicy/strpolicy.go @@ -0,0 +1,66 @@ +// Package strpolicy cleans strings that come from tenants or the network +// before they reach records, objects or logs: interface names set inside a +// tenant's netns, container names and images, LLDP fields from the link +// peer, the ASN feed's network owner, netns names and congestion-control +// names. +// +// Clean is applied once where each value enters (per container, interface, +// snapshot or namespace), not per row. +package strpolicy + +import ( + "strings" + "unicode/utf8" +) + +const ( + // MaxLen bounds container names and images, LLDP fields, the network + // owner and netns names. + MaxLen = 256 + // MaxIfname is IFNAMSIZ, the kernel's interface name buffer. + MaxIfname = 16 + // MaxCong is TCP_CA_NAME_MAX, the kernel's congestion name buffer. + MaxCong = 16 +) + +// Clean returns s with invalid UTF-8, C0 controls, DEL and C1 controls +// replaced by U+FFFD, truncated at a rune boundary to at most max bytes. +// A printable ASCII string that already fits is returned as is, without +// allocating. +func Clean(s string, maxBytes int) string { + if maxBytes <= 0 { + return "" + } + if len(s) <= maxBytes && printableASCII(s) { + return s + } + var b strings.Builder + b.Grow(min(len(s), maxBytes)) + for i := 0; i < len(s); { + r, size := utf8.DecodeRuneInString(s[i:]) + i += size + if r == utf8.RuneError || isControl(r) { + r = utf8.RuneError + } + if b.Len()+utf8.RuneLen(r) > maxBytes { + break + } + b.WriteRune(r) + } + return b.String() +} + +// printableASCII reports whether every byte of s is in 0x20-0x7e. +func printableASCII(s string) bool { + for i := 0; i < len(s); i++ { + if c := s[i]; c < 0x20 || c > 0x7e { + return false + } + } + return true +} + +// isControl reports whether r is a C0 control, DEL or a C1 control. +func isControl(r rune) bool { + return r < 0x20 || (r >= 0x7f && r <= 0x9f) +} diff --git a/pkg/strpolicy/strpolicy_test.go b/pkg/strpolicy/strpolicy_test.go new file mode 100644 index 000000000..20ee321e6 --- /dev/null +++ b/pkg/strpolicy/strpolicy_test.go @@ -0,0 +1,112 @@ +package strpolicy + +import ( + "strings" + "testing" + "unicode/utf8" +) + +func TestClean(t *testing.T) { + ascii256 := strings.Repeat("a", 256) + tests := []struct { + name string + description string + in string + max int + expected string + }{ + {"ascii_short", "positive: short printable ASCII is returned as is", "web-1", MaxLen, "web-1"}, + {"exact_max", "boundary: 256 ASCII bytes at max 256 are unchanged", ascii256, MaxLen, ascii256}, + {"max_plus_one", "boundary: 257 ASCII bytes are cut to the first 256", ascii256 + "b", MaxLen, ascii256}, + {"empty", "boundary: the empty string stays empty", "", MaxLen, ""}, + {"max_zero", "boundary: max 0 yields the empty string", "abc", 0, ""}, + {"max_negative", "corner: a negative max yields the empty string", "abc", -1, ""}, + {"utf8_valid", "positive: valid non-ASCII UTF-8 is unchanged", "café-ü", MaxLen, "café-ü"}, + {"truncate_mid_rune", "corner: a 2-byte rune straddling max is dropped whole", "abcé", 4, "abc"}, + {"invalid_utf8", "negative: an invalid byte becomes U+FFFD", "a\xffb", MaxLen, "a�b"}, + {"c0_controls", "negative: ESC and LF become U+FFFD", "a\x1b[2Jb\n", MaxLen, "a�[2Jb�"}, + {"nul", "negative: NUL becomes U+FFFD", "a\x00b", MaxLen, "a�b"}, + {"del", "negative: DEL becomes U+FFFD", "\x7f", MaxLen, "�"}, + {"c1_controls", "negative: U+0085 and U+009B become U+FFFD", "a\u0085b\u009bc", MaxLen, "a�b�c"}, + {"nbsp_kept", "boundary: U+00A0, just past C1, is kept", "a b", MaxLen, "a b"}, + {"replacement_grows", "corner: 16 invalid bytes at max 16 fit five U+FFFD (15 bytes)", strings.Repeat("\xff", 16), 16, strings.Repeat("�", 5)}, + {"ifname_15", "positive: a 15-byte tenant ifname with \\x01 stays within IFNAMSIZ", "eth\x01abcdefghijk", MaxIfname, "eth�abcdefghij"}, + {"replacement_in_input", "corner: a literal U+FFFD in the input is kept", "a�b", MaxLen, "a�b"}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := Clean(tc.in, tc.max) + if got != tc.expected { + t.Fatalf("%s: Clean(%q, %d) = %q, want %q", tc.description, tc.in, tc.max, got, tc.expected) + } + if !utf8.ValidString(got) { + t.Fatalf("%s: result %q is not valid UTF-8", tc.description, got) + } + if tc.max > 0 && len(got) > tc.max { + t.Fatalf("%s: result is %d bytes, over max %d", tc.description, len(got), tc.max) + } + }) + } +} + +func TestCleanNoAlloc(t *testing.T) { + tests := []struct { + name string + description string + in string + max int + expected float64 + }{ + {"ascii16", "positive: a fitting ASCII ifname does not allocate", "eth0", MaxIfname, 0}, + {"ascii256", "boundary: 256 ASCII bytes at max 256 do not allocate", strings.Repeat("x", 256), MaxLen, 0}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + got := testing.AllocsPerRun(100, func() { _ = Clean(tc.in, tc.max) }) + if got != tc.expected { + t.Fatalf("%s: %v allocs, want %v", tc.description, got, tc.expected) + } + }) + } +} + +func FuzzClean(f *testing.F) { + f.Add("web-1", 256) + f.Add("a\xffb\x1b", 3) + f.Fuzz(func(t *testing.T, s string, maxBytes int) { + maxBytes %= 512 + got := Clean(s, maxBytes) + if !utf8.ValidString(got) { + t.Fatalf("Clean(%q, %d) = %q: invalid UTF-8", s, maxBytes, got) + } + if len(got) > max(maxBytes, 0) { + t.Fatalf("Clean(%q, %d) = %q: over max", s, maxBytes, got) + } + for _, r := range got { + if isControl(r) { + t.Fatalf("Clean(%q, %d) = %q: control rune %U kept", s, maxBytes, got, r) + } + } + }) +} + +func BenchmarkClean(b *testing.B) { + dirty := strings.Repeat("a\x1b\xff", 86)[:256] + cases := []struct { + name string + in string + max int + }{ + {"ascii16", "eth0-tenant-abcd", MaxIfname}, + {"ascii256", strings.Repeat("x", 256), MaxLen}, + {"dirty256", dirty, MaxLen}, + } + for _, c := range cases { + b.Run(c.name, func(b *testing.B) { + b.ReportAllocs() + for b.Loop() { + _ = Clean(c.in, c.max) + } + }) + } +} diff --git a/pkg/xtcp/caps_test.go b/pkg/xtcp/caps_test.go index 12bbbd4f3..9341813fa 100644 --- a/pkg/xtcp/caps_test.go +++ b/pkg/xtcp/caps_test.go @@ -476,6 +476,7 @@ func TestSetPreservesCaps(t *testing.T) { MaxDumpBytes: 100, MaxRecordsPerNetns: 200, MaxNetns: 300, MaxRecordsPerPoll: 400, IoUringSockstatBatchSize: 16, IoUringSockstatWorkers: 2, MaxNetnsPerOwner: 5, MaxNetnsAddsPerReconcile: 6, + S3QueueDepth: 32, S3MaxBufferedBytes: 64 << 20, ReconcileTimeout: durationpb.New(700 * time.Millisecond), } } @@ -493,7 +494,8 @@ func TestSetPreservesCaps(t *testing.T) { running: newRunning(), in: &xtcp_config.XtcpConfig{MaxDumpBytes: 1 << 40, MaxRecordsPerNetns: 1 << 40, MaxNetns: 1 << 20, MaxRecordsPerPoll: 1 << 40, IoUringSockstatBatchSize: 1024, IoUringSockstatWorkers: 64, - MaxNetnsPerOwner: 1 << 20, MaxNetnsAddsPerReconcile: 1 << 20, ReconcileTimeout: durationpb.New(time.Hour)}, + MaxNetnsPerOwner: 1 << 20, MaxNetnsAddsPerReconcile: 1 << 20, ReconcileTimeout: durationpb.New(time.Hour), + S3QueueDepth: 4096, S3MaxBufferedBytes: 0}, expected: newRunning()}, {name: "set_same_caps", category: "P", description: "a Get→Set round trip", running: newRunning(), in: newRunning(), expected: newRunning()}, diff --git a/pkg/xtcp/destinations_s3parquet.go b/pkg/xtcp/destinations_s3parquet.go index 55fc7c735..fe21458f8 100644 --- a/pkg/xtcp/destinations_s3parquet.go +++ b/pkg/xtcp/destinations_s3parquet.go @@ -13,6 +13,7 @@ import ( "fmt" "io" "log" + "math" "net/http" "os" "path" @@ -41,11 +42,6 @@ const S3ParquetFlushThresholdBytesCst = 63 * 1024 * 1024 // string across the many same-day rows in a file. const nsPerDayCst = 24 * 60 * 60 * 1_000_000_000 -// s3ParquetDestQueueCapacity bounds the in-flight backlog between -// Send() and the worker. Full queue → Send blocks; queueFull counter -// bumps so operators can spot back-pressure. -const s3ParquetDestQueueCapacity = 16 - // s3ParquetWorkerDrainTimeout caps how long Close() will wait for the // worker to flush its final partial Parquet to S3 before giving up. const s3ParquetWorkerDrainTimeout = 30 * time.Second @@ -113,11 +109,11 @@ type s3ParquetDest struct { sleep func(context.Context, time.Duration) bool newTimer func(time.Duration) (<-chan time.Time, func() bool) - // queueCh carries marshaled envelopes from Send to the worker. - // IMPORTANT: never closed by Close (sending on a closed channel - // panics, and Close races with concurrent Sends). The worker exits - // via closedCh instead, draining queueCh's residual items first. - queueCh chan envelopeBytes + // queue carries marshaled envelopes from Send to the worker, bounded + // by depth and bytes (see envelopeQueue). The worker exits via + // closedCh, draining the queue's residual items first. + queue *envelopeQueue + metrics *s3QueueMetrics // closedCh is closed by Close exactly once. Send checks it before // each channel-send and bails with errSendOnClosed if closed. @@ -281,7 +277,8 @@ func newS3ParquetDest(ctx context.Context, x *XTCP) (Destination, error) { jitterInt: misc.JitterIntN, sleep: misc.SleepCtx, newTimer: func(dur time.Duration) (<-chan time.Time, func() bool) { t := time.NewTimer(dur); return t.C, t.Stop }, - queueCh: make(chan envelopeBytes, s3ParquetDestQueueCapacity), + queue: newEnvelopeQueue(resolveS3QueueDepth(x.config), resolveS3MaxBufferedBytes(x.config)), + metrics: newS3QueueMetrics(x.registry), closedCh: make(chan struct{}), workerDone: make(chan struct{}), } @@ -289,6 +286,21 @@ func newS3ParquetDest(ctx context.Context, x *XTCP) (Destination, error) { return d, nil } +// resolveS3QueueDepth returns the envelope depth bound: s3_queue_depth, or +// s3QueueDepthDefaultCst when unset (0), capped at s3QueueDepthMaxCst. +func resolveS3QueueDepth(c *xtcp_config.XtcpConfig) int { + if c.S3QueueDepth == 0 { + return s3QueueDepthDefaultCst + } + return int(min(c.S3QueueDepth, s3QueueDepthMaxCst)) +} + +// resolveS3MaxBufferedBytes returns the queue's byte bound: 0 (none) when +// s3_max_buffered_bytes is unset, saturating at the largest int. +func resolveS3MaxBufferedBytes(c *xtcp_config.XtcpConfig) int { + return int(min(c.S3MaxBufferedBytes, uint64(math.MaxInt))) +} + // resolveFlushInterval returns the staleness-ceiling flush interval: the // configured s3_flush_interval, or when unset (0) the derived // max(PollFrequency, 30m) floor (see s3FlushIntervalFloorCst). @@ -319,49 +331,55 @@ func resolveBackoffCap(c *xtcp_config.XtcpConfig) time.Duration { return capDur } -// Send enqueues the marshaled envelope for the background worker. The -// fast path is a non-blocking channel send (queue has slack); if the -// worker is behind (e.g. mid-upload), Send falls back to a blocking -// send and bumps queueFull so operators can spot the back-pressure. +// Send enqueues the marshaled envelope for the background worker and never +// blocks: flushEnvelope runs mid-poll, so a slow or unreachable S3 must not +// stall polling. Past the queue's depth or byte bound the oldest envelopes +// are dropped, counted (s3_dropped_total, s3_dropped_bytes_total and the +// queueFull counter) and their buffers returned to destBytesPool. // -// closedCh is checked in every select so Send never tries to write to a -// closed-and-replaced queueCh (which would panic). Sends arriving after -// Close return errSendOnClosed and refund the buffer to destBytesPool -// so the upstream pool stays warm. +// Sends arriving after Close return errSendOnClosed and refund the buffer +// to destBytesPool so the upstream pool stays warm. // // Returns (1, nil) on enqueue to mirror the per-record accounting the // caller (flushEnvelope in poller.go) expects. -func (d *s3ParquetDest) Send(ctx context.Context, b *[]byte) (int, error) { - // Closed-first fast check so Sends arriving after Close exit cheaply. +func (d *s3ParquetDest) Send(_ context.Context, b *[]byte) (int, error) { select { case <-d.closedCh: d.refundOnReject(b) return 0, errSendOnClosed default: } - // Non-blocking enqueue when queue has slack. - select { - case d.queueCh <- envelopeBytes{buf: b}: - return 1, nil - case <-d.closedCh: - d.refundOnReject(b) - return 0, errSendOnClosed - default: + for _, drop := range d.queue.push(envelopeBytes{buf: b}) { + d.countDrop(drop) + d.returnBuf(drop.item.buf) } - // Queue full → blocking path. Bump counter so back-pressure shows up - // in dashboards. + d.observeQueue() + return 1, nil +} + +// countDrop records one envelope the queue evicted. +func (d *s3ParquetDest) countDrop(drop droppedEnvelope) { if d.x.pC != nil { d.x.pC.WithLabelValues("destS3Parquet", "queueFull", "error").Inc() } - select { - case d.queueCh <- envelopeBytes{buf: b}: - return 1, nil - case <-d.closedCh: - d.refundOnReject(b) - return 0, errSendOnClosed - case <-ctx.Done(): - d.refundOnReject(b) - return 0, ctx.Err() + if d.metrics != nil { + d.metrics.dropped.WithLabelValues(drop.reason).Inc() + d.metrics.droppedBytes.WithLabelValues(drop.reason).Add(float64(len(*drop.item.buf))) + } +} + +// observeQueue sets the queue stage of the buffered-bytes gauge. +func (d *s3ParquetDest) observeQueue() { + if d.metrics != nil { + _, n := d.queue.stats() + d.metrics.buffered.WithLabelValues(s3StageQueue).Set(float64(n)) + } +} + +// observeBuilder sets the builder stage of the buffered-bytes gauge. +func (d *s3ParquetDest) observeBuilder(n int) { + if d.metrics != nil { + d.metrics.buffered.WithLabelValues(s3StageBuilder).Set(float64(n)) } } @@ -379,9 +397,8 @@ func (d *s3ParquetDest) refundOnReject(b *[]byte) { // if the worker doesn't finish in time, but the daemon shuts down // regardless (closeDestination is best-effort during teardown). // -// Closes closedCh only — never closes queueCh, since concurrent Sends -// would panic on a send-to-closed channel. The worker drains queueCh -// via its own select on closedCh. +// Closes closedCh only; the worker drains the queue via its own select +// on closedCh. func (d *s3ParquetDest) Close() error { var err error d.closeOnce.Do(func() { @@ -396,7 +413,7 @@ func (d *s3ParquetDest) Close() error { } // worker is the only goroutine that touches the Parquet builder. -// Receives marshaled envelopes from queueCh, decodes them, appends each +// Receives marshaled envelopes from the queue, decodes them, appends each // row to the in-memory writer, and finalizes + uploads when the // accumulated byte threshold is reached. On queue close (Close was // called) finalizes whatever's left and exits. @@ -427,6 +444,7 @@ func (d *s3ParquetDest) worker(ctx context.Context) { // fresh random target in [threshold*(1-pct/100), threshold], so // identically-loaded hosts don't cross the byte cap in lockstep. effectiveThreshold = jitteredThreshold(d.threshold, d.thresholdJitterPct, d.jitterInt) + d.observeBuilder(0) } startBuilder() @@ -488,6 +506,17 @@ func (d *s3ParquetDest) worker(ctx context.Context) { finalize() } } + d.observeBuilder(accumBytes) + } + drain := func() { + for { + item, ok := d.queue.pop() + if !ok { + return + } + d.observeQueue() + processItem(item) + } } // Staleness-ceiling flush timer. Fires on a jittered interval so the @@ -502,19 +531,12 @@ func (d *s3ParquetDest) worker(ctx context.Context) { select { case <-d.closedCh: // Drain any items already enqueued (a Send that won the - // race against closedCh and got onto the channel before - // the close), then exit. - for { - select { - case item := <-d.queueCh: - processItem(item) - default: - finalize() - return - } - } - case item := <-d.queueCh: - processItem(item) + // race against closedCh), then exit. + drain() + finalize() + return + case <-d.queue.notify: + drain() case <-flushCh: finalize() // no-op when the builder is empty flushStop() diff --git a/pkg/xtcp/destinations_s3parquet_jitter_test.go b/pkg/xtcp/destinations_s3parquet_jitter_test.go index 5b1ecd10a..619d4357e 100644 --- a/pkg/xtcp/destinations_s3parquet_jitter_test.go +++ b/pkg/xtcp/destinations_s3parquet_jitter_test.go @@ -163,7 +163,11 @@ func TestS3ParquetDest_timeFlush(t *testing.T) { } flushCh <- time.Now() // fire the staleness ceiling - time.Sleep(30 * time.Millisecond) + // Poll: finalizing a Parquet object under -race takes well over a + // fixed short sleep. + for deadline := time.Now().Add(10 * time.Second); len(upl.Calls()) == 0 && time.Now().Before(deadline); { + time.Sleep(5 * time.Millisecond) + } if got := len(upl.Calls()); got != 1 { t.Fatalf("upload after timer fire = %d, want 1", got) } diff --git a/pkg/xtcp/destinations_s3parquet_test.go b/pkg/xtcp/destinations_s3parquet_test.go index d7764a239..70070952b 100644 --- a/pkg/xtcp/destinations_s3parquet_test.go +++ b/pkg/xtcp/destinations_s3parquet_test.go @@ -120,7 +120,8 @@ func newS3ParquetFixtureCustom(t *testing.T, threshold int, injectErr func(int) jitterInt: func(int) int { return 0 }, sleep: func(context.Context, time.Duration) bool { return true }, newTimer: func(time.Duration) (<-chan time.Time, func() bool) { return neverFire, func() bool { return true } }, - queueCh: make(chan envelopeBytes, s3ParquetDestQueueCapacity), + queue: newEnvelopeQueue(s3QueueDepthDefaultCst, 0), + metrics: newS3QueueMetrics(reg), closedCh: make(chan struct{}), workerDone: make(chan struct{}), } @@ -372,48 +373,39 @@ func TestS3ParquetDest_corner_sendAfterClose(t *testing.T) { } func TestS3ParquetDest_corner_queueFull(t *testing.T) { - // Hold the worker by injecting a slow uploader. + // Hold the worker in its first upload; Sends past the queue's depth + // must return at once, dropping the oldest and ticking queueFull. hold := make(chan struct{}) + uploading := make(chan struct{}, 1) d, _, x := newS3ParquetFixture(t, 1, func(_ int) error { - <-hold // block forever in the worker - return nil - }) - - // Fill the queue: capacity + 1 sends so the (cap+1)th has to fall - // through to the blocking path, ticking the queueFull counter. - bufs := make([]*[]byte, s3ParquetDestQueueCapacity+1) - for i := range bufs { - bufs[i] = marshalEnvelopeBuf(t, x, mkEnvelope(1)) - } - // Send the first N+1; the (N+1)th blocks. Use a goroutine + timeout. - doneCh := make(chan struct{}) - go func() { - for _, b := range bufs { - _, _ = d.Send(context.Background(), b) - } - close(doneCh) - }() - - // Wait for the queueFull counter to tick. The loop breaks the instant - // the counter reaches 1, so a passing run finishes in milliseconds; the - // deadline only bounds the genuine-failure case. Keep it generous so a - // loaded CI box (full `go test ./...`, esp. under -race) can't trip a - // false negative just because the sender goroutine scheduled late. - deadline := time.After(30 * time.Second) - for { select { - case <-deadline: - t.Fatal("queueFull counter never ticked") + case uploading <- struct{}{}: default: } - v := promCounterValue(t, x, "destS3Parquet", "queueFull", "error") - if v >= 1 { - break + <-hold + return nil + }) + if _, err := d.Send(context.Background(), marshalEnvelopeBuf(t, x, mkEnvelope(1))); err != nil { + t.Fatalf("Send: %v", err) + } + <-uploading // the worker holds the first envelope in its upload + for range s3QueueDepthDefaultCst + 1 { + buf := marshalEnvelopeBuf(t, x, mkEnvelope(1)) + done := make(chan struct{}) + go func() { + _, _ = d.Send(context.Background(), buf) + close(done) + }() + select { + case <-done: + case <-time.After(10 * time.Second): + t.Fatal("Send blocked on a full queue") } - time.Sleep(5 * time.Millisecond) } - close(hold) // release worker so Close can drain - <-doneCh + if v := promCounterValue(t, x, "destS3Parquet", "queueFull", "error"); v != 1 { + t.Errorf("queueFull = %v, want 1", v) + } + close(hold) _ = d.Close() } diff --git a/pkg/xtcp/enrich.go b/pkg/xtcp/enrich.go index e75b35987..d098470d3 100644 --- a/pkg/xtcp/enrich.go +++ b/pkg/xtcp/enrich.go @@ -14,6 +14,7 @@ import ( "github.com/randomizedcoder/xtcp2/pkg/lldp" "github.com/randomizedcoder/xtcp2/pkg/nicinfo" "github.com/randomizedcoder/xtcp2/pkg/nsdiscover" + "github.com/randomizedcoder/xtcp2/pkg/strpolicy" ) // enrichDialTimeout bounds the one-shot startup connections to the docker and @@ -209,22 +210,24 @@ func buildUplinkStamp(ifnames []string, nics []nicinfo.NIC, neigh map[string]lld for i := 0; i < len(ifnames) && i < maxUplinks; i++ { name := ifnames[i] col := &s.slots[i] - col.ifname = name + col.ifname = strpolicy.Clean(name, strpolicy.MaxIfname) + // NIC strings come from the driver and firmware, LLDP ones from the + // link peer: clean each once per refresh, not per row. if n, ok := nicByName[name]; ok { - col.nicDriver = n.Driver - col.nicModel = n.Model - col.nicBusInfo = n.BusInfo - col.nicFwVersion = n.FwVersion + col.nicDriver = strpolicy.Clean(n.Driver, strpolicy.MaxLen) + col.nicModel = strpolicy.Clean(n.Model, strpolicy.MaxLen) + col.nicBusInfo = strpolicy.Clean(n.BusInfo, strpolicy.MaxLen) + col.nicFwVersion = strpolicy.Clean(n.FwVersion, strpolicy.MaxLen) col.nicPCIVendor = n.PCIVendor col.nicPCIDevice = n.PCIDevice col.nicSpeedMbps = n.SpeedMbps } if nb, ok := neigh[name]; ok { - col.lldpChassisName = nb.ChassisName - col.lldpChassisID = nb.ChassisID - col.lldpMgmtIP = nb.MgmtIP - col.lldpPortID = nb.PortID - col.lldpPortDescr = nb.PortDescr + col.lldpChassisName = strpolicy.Clean(nb.ChassisName, strpolicy.MaxLen) + col.lldpChassisID = strpolicy.Clean(nb.ChassisID, strpolicy.MaxLen) + col.lldpMgmtIP = strpolicy.Clean(nb.MgmtIP, strpolicy.MaxLen) + col.lldpPortID = strpolicy.Clean(nb.PortID, strpolicy.MaxLen) + col.lldpPortDescr = strpolicy.Clean(nb.PortDescr, strpolicy.MaxLen) } s.enabled = true } diff --git a/pkg/xtcp/enrich_test.go b/pkg/xtcp/enrich_test.go index 3b8260c95..6f54eab4c 100644 --- a/pkg/xtcp/enrich_test.go +++ b/pkg/xtcp/enrich_test.go @@ -1,6 +1,7 @@ package xtcp import ( + "strings" "testing" "github.com/randomizedcoder/xtcp2/gen/go/xtcp_flat_record" @@ -159,3 +160,35 @@ func TestUplinkStamp_apply(t *testing.T) { t.Fatalf("disabled apply must be a no-op, got %q", r2.Uplink1Ifname) } } + +// TestBuildUplinkStamp_strPolicy: LLDP fields come from the link peer and +// NIC strings from the driver; the stamp cleans them once per refresh. +func TestBuildUplinkStamp_strPolicy(t *testing.T) { + tests := []struct { + name string + category string + description string + neigh lldp.Neighbor + nic nicinfo.NIC + expected [2]string // lldpChassisName, nicModel + }{ + {name: "plain", category: "P", description: "ordinary values are unchanged", + neigh: lldp.Neighbor{ChassisName: "sw-a"}, nic: nicinfo.NIC{Ifname: "eth0", Model: "CX6"}, + expected: [2]string{"sw-a", "CX6"}}, + {name: "lldp_escape", category: "N", description: "ESC in the chassis name is replaced", + neigh: lldp.Neighbor{ChassisName: "sw\x1b[31m"}, nic: nicinfo.NIC{Ifname: "eth0"}, + expected: [2]string{"sw�[31m", ""}}, + {name: "lldp_long", category: "N", description: "a 1 kB chassis name is cut to 256 bytes", + neigh: lldp.Neighbor{ChassisName: strings.Repeat("c", 1024)}, nic: nicinfo.NIC{Ifname: "eth0", Model: "m\xff"}, + expected: [2]string{strings.Repeat("c", 256), "m�"}}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + s := buildUplinkStamp([]string{"eth0"}, []nicinfo.NIC{tc.nic}, map[string]lldp.Neighbor{"eth0": tc.neigh}) + got := [2]string{s.slots[0].lldpChassisName, s.slots[0].nicModel} + if got != tc.expected { + t.Errorf("%s: got %q, expected %q", tc.description, got, tc.expected) + } + }) + } +} diff --git a/pkg/xtcp/grpc_configService.go b/pkg/xtcp/grpc_configService.go index 4d87e1965..c4546c6e2 100644 --- a/pkg/xtcp/grpc_configService.go +++ b/pkg/xtcp/grpc_configService.go @@ -566,8 +566,8 @@ func (c *xtcpConfigService) SetEnvelopeFlush( // preserveCaps keeps the running daemon's tenant caps (-maxDumpBytes, // -maxRecordsPerNetns, -maxNetns, -maxRecordsPerPoll, -maxNetnsPerOwner, -// -maxNetnsAddsPerReconcile, -reconcileTimeout and the sockstat ring sizing) -// across a Set. They bound what a workload can cost the collector, so they +// -maxNetnsAddsPerReconcile, -reconcileTimeout, the sockstat ring sizing and +// the S3 queue bounds -s3QueueDepth and -s3MaxBufferedBytes) across a Set. They bound what a workload can cost the collector, so they // are startup-only: a Set that omitted them would otherwise turn them off (0), // and one that raised them would undo the bound from the admin plane. Change // them with flags or env and a restart. @@ -580,6 +580,8 @@ func preserveCaps(in, running *xtcp_config.XtcpConfig) { in.IoUringSockstatWorkers = running.GetIoUringSockstatWorkers() in.MaxNetnsPerOwner = running.GetMaxNetnsPerOwner() in.MaxNetnsAddsPerReconcile = running.GetMaxNetnsAddsPerReconcile() + in.S3QueueDepth = running.GetS3QueueDepth() + in.S3MaxBufferedBytes = running.GetS3MaxBufferedBytes() in.ReconcileTimeout = nil if rt := running.GetReconcileTimeout(); rt != nil { in.ReconcileTimeout = durationpb.New(rt.AsDuration()) diff --git a/pkg/xtcp/ns_reconcile.go b/pkg/xtcp/ns_reconcile.go index 4dd52d6da..88ebca701 100644 --- a/pkg/xtcp/ns_reconcile.go +++ b/pkg/xtcp/ns_reconcile.go @@ -8,6 +8,7 @@ import ( "time" "github.com/randomizedcoder/xtcp2/pkg/nsdiscover" + "github.com/randomizedcoder/xtcp2/pkg/strpolicy" ) // mapReconciler is a ticking loop around reconcile, which converges xtcp's set @@ -263,7 +264,7 @@ func (x *XTCP) addNew(ctx context.Context, live map[uint64]nsIdentity, stop nsdi } } for _, id := range admit { - id.name = x.nsResolver.Name(id.inode, id.pid) + id.name = strpolicy.Clean(x.nsResolver.Name(id.inode, id.pid), strpolicy.MaxLen) x.nsAdd(ctx, id) stores++ } diff --git a/pkg/xtcp/s3queue.go b/pkg/xtcp/s3queue.go new file mode 100644 index 000000000..d8dd8174e --- /dev/null +++ b/pkg/xtcp/s3queue.go @@ -0,0 +1,155 @@ +//go:build dest_s3parquet + +package xtcp + +import ( + "sync" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/promauto" +) + +// s3QueueDepthDefaultCst is the envelope depth bound when s3_queue_depth is +// unset (0); s3QueueDepthMaxCst is the largest the config accepts. +const ( + s3QueueDepthDefaultCst = 16 + s3QueueDepthMaxCst = 4096 +) + +// s3MaxBufferedBytesDefaultCst is the documented default for +// s3_max_buffered_bytes (the flag's default; 0 in the config means no byte +// bound, the depth bound alone). +const s3MaxBufferedBytesDefaultCst = 64 * 1024 * 1024 + +// Drop reasons, the reason label of the s3 drop counters. +const ( + s3DropReasonDepth = "queue_depth" + s3DropReasonBytes = "buffered_bytes" +) + +// envelopeQueue is the bounded FIFO between Send and the S3 worker. push +// never blocks: past maxDepth envelopes, or past maxBytes buffered bytes, +// it drops the oldest, so an S3 outage costs data rather than stalling the +// poller. The newest envelope is always kept, even when it alone is over +// maxBytes. notify (capacity 1) wakes the worker; it drains with pop. +type envelopeQueue struct { + mu sync.Mutex + items []envelopeBytes // ring of cap maxDepth+1 + head int + depth int + bytes int + maxDepth int + maxBytes int // 0 = no byte bound + notify chan struct{} +} + +// droppedEnvelope is an envelope push evicted and why. +type droppedEnvelope struct { + item envelopeBytes + reason string +} + +func newEnvelopeQueue(maxDepth, maxBytes int) *envelopeQueue { + if maxDepth <= 0 { + maxDepth = s3QueueDepthDefaultCst + } + return &envelopeQueue{ + items: make([]envelopeBytes, maxDepth+1), + maxDepth: maxDepth, + maxBytes: max(maxBytes, 0), + notify: make(chan struct{}, 1), + } +} + +// push appends item, evicts the oldest envelopes past either bound and +// wakes the worker. The evicted envelopes are returned for the caller to +// count and recycle outside the lock. +func (q *envelopeQueue) push(item envelopeBytes) []droppedEnvelope { + var dropped []droppedEnvelope + q.mu.Lock() + q.items[(q.head+q.depth)%len(q.items)] = item + q.depth++ + q.bytes += len(*item.buf) + for { + var reason string + switch { + case q.depth > q.maxDepth: + reason = s3DropReasonDepth + case q.maxBytes > 0 && q.bytes > q.maxBytes && q.depth > 1: + reason = s3DropReasonBytes + } + if reason == "" { + break + } + dropped = append(dropped, droppedEnvelope{item: q.popLocked(), reason: reason}) + } + q.mu.Unlock() + select { + case q.notify <- struct{}{}: + default: + } + return dropped +} + +// pop removes the oldest envelope; ok is false when the queue is empty. +func (q *envelopeQueue) pop() (item envelopeBytes, ok bool) { + q.mu.Lock() + defer q.mu.Unlock() + if q.depth == 0 { + return envelopeBytes{}, false + } + return q.popLocked(), true +} + +func (q *envelopeQueue) popLocked() envelopeBytes { + item := q.items[q.head] + q.items[q.head] = envelopeBytes{} + q.head = (q.head + 1) % len(q.items) + q.depth-- + q.bytes -= len(*item.buf) + return item +} + +// stats returns the queued envelope count and bytes. +func (q *envelopeQueue) stats() (depth, bytes int) { + q.mu.Lock() + defer q.mu.Unlock() + return q.depth, q.bytes +} + +// s3QueueMetrics are the bounded queue's drop counters and buffer gauge. +type s3QueueMetrics struct { + dropped *prometheus.CounterVec // envelopes, by reason + droppedBytes *prometheus.CounterVec // bytes, by reason + buffered *prometheus.GaugeVec // bytes held, by stage +} + +func newS3QueueMetrics(reg prometheus.Registerer) *s3QueueMetrics { + if reg == nil { + reg = prometheus.DefaultRegisterer + } + f := promauto.With(reg) + return &s3QueueMetrics{ + dropped: f.NewCounterVec(prometheus.CounterOpts{ + Subsystem: promSubsystemXTCP, + Name: "s3_dropped_total", + Help: "Envelopes the s3parquet queue dropped, oldest first: reason=queue_depth (-s3QueueDepth), buffered_bytes (-s3MaxBufferedBytes).", + }, []string{promLabelReasonCst}), + droppedBytes: f.NewCounterVec(prometheus.CounterOpts{ + Subsystem: promSubsystemXTCP, + Name: "s3_dropped_bytes_total", + Help: "Marshaled envelope bytes the s3parquet queue dropped, by the same reasons as s3_dropped_total.", + }, []string{promLabelReasonCst}), + buffered: f.NewGaugeVec(prometheus.GaugeOpts{ + Subsystem: promSubsystemXTCP, + Name: "s3_buffered_bytes", + Help: "Bytes the s3parquet destination holds: stage=queue (marshaled envelopes awaiting the worker), builder (approximate row bytes in the Parquet object being built).", + }, []string{"stage"}), + } +} + +// Stages, the stage label of the buffered-bytes gauge. +const ( + s3StageQueue = "queue" + s3StageBuilder = "builder" +) diff --git a/pkg/xtcp/s3queue_test.go b/pkg/xtcp/s3queue_test.go new file mode 100644 index 000000000..efa654ac7 --- /dev/null +++ b/pkg/xtcp/s3queue_test.go @@ -0,0 +1,336 @@ +//go:build dest_s3parquet + +package xtcp + +import ( + "context" + "slices" + "sync" + "testing" + "time" + + "github.com/prometheus/client_golang/prometheus" + dto "github.com/prometheus/client_model/go" +) + +// qItem is a queue item whose buffer is n bytes, tagged by its first byte. +func qItem(tag byte, n int) envelopeBytes { + b := make([]byte, n) + if n > 0 { + b[0] = tag + } + return envelopeBytes{buf: &b} +} + +func qTag(item envelopeBytes) byte { + if len(*item.buf) == 0 { + return 0 + } + return (*item.buf)[0] +} + +func TestEnvelopeQueue(t *testing.T) { + type push struct { + tag byte + n int + } + tests := []struct { + name string + category string + description string + maxDepth int + maxBytes int + pushes []push + expected []byte // tags left in the queue, oldest first + expectedDrop []string // reason of each drop, in order + }{ + {name: "fifo", category: "P", description: "push 3, pop 3 keeps the order", + maxDepth: 4, pushes: []push{{1, 1}, {2, 1}, {3, 1}}, expected: []byte{1, 2, 3}}, + {name: "depth_drop_oldest", category: "N", description: "depth 2, push 3 drops the oldest", + maxDepth: 2, pushes: []push{{1, 1}, {2, 1}, {3, 1}}, expected: []byte{2, 3}, + expectedDrop: []string{s3DropReasonDepth}}, + {name: "bytes_drop_oldest", category: "N", description: "maxBytes 10, push 6+6 drops the first", + maxDepth: 4, maxBytes: 10, pushes: []push{{1, 6}, {2, 6}}, expected: []byte{2}, + expectedDrop: []string{s3DropReasonBytes}}, + {name: "bytes_drop_several", category: "N", description: "a large push evicts several older items", + maxDepth: 8, maxBytes: 10, pushes: []push{{1, 3}, {2, 3}, {3, 3}, {4, 8}}, expected: []byte{4}, + expectedDrop: []string{s3DropReasonBytes, s3DropReasonBytes, s3DropReasonBytes}}, + {name: "single_over_bytes", category: "C", description: "one 20-byte item over maxBytes 10 is kept", + maxDepth: 4, maxBytes: 10, pushes: []push{{1, 20}}, expected: []byte{1}}, + {name: "bytes_zero_unlimited", category: "P", description: "maxBytes 0: only depth drops", + maxDepth: 2, pushes: []push{{1, 1 << 20}, {2, 1 << 20}, {3, 1 << 20}}, expected: []byte{2, 3}, + expectedDrop: []string{s3DropReasonDepth}}, + {name: "exact_bytes", category: "B", description: "a sum equal to maxBytes drops nothing", + maxDepth: 4, maxBytes: 10, pushes: []push{{1, 5}, {2, 5}}, expected: []byte{1, 2}}, + {name: "depth_default", category: "B", description: "maxDepth 0 means the default depth", + maxDepth: 0, pushes: make([]push, s3QueueDepthDefaultCst+1), expected: make([]byte, s3QueueDepthDefaultCst), + expectedDrop: []string{s3DropReasonDepth}}, + {name: "pop_empty", category: "B", description: "an empty queue pops nothing", + maxDepth: 4}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + q := newEnvelopeQueue(tc.maxDepth, tc.maxBytes) + var drops []string + for _, p := range tc.pushes { + for _, d := range q.push(qItem(p.tag, p.n)) { + drops = append(drops, d.reason) + } + } + if !slices.Equal(drops, tc.expectedDrop) { + t.Errorf("%s: drops %v, expected %v", tc.description, drops, tc.expectedDrop) + } + var got []byte + for { + item, ok := q.pop() + if !ok { + break + } + got = append(got, qTag(item)) + } + if !slices.Equal(got, tc.expected) { + t.Errorf("%s: queue %v, expected %v", tc.description, got, tc.expected) + } + if depth, bytes := q.stats(); depth != 0 || bytes != 0 { + t.Errorf("%s: drained queue reports depth %d bytes %d", tc.description, depth, bytes) + } + }) + } +} + +// TestEnvelopeQueue_concurrent: 8 pushers and one popper; every item is +// popped or dropped exactly once and the byte count returns to zero. +func TestEnvelopeQueue_concurrent(t *testing.T) { + const pushers, perPusher = 8, 2000 + q := newEnvelopeQueue(4, 64) + var mu sync.Mutex + dropped := 0 + var wg sync.WaitGroup + for range pushers { + wg.Go(func() { + for i := range perPusher { + n := len(q.push(qItem(1, 1+i%32))) + mu.Lock() + dropped += n + mu.Unlock() + } + }) + } + stop := make(chan struct{}) + popped := 0 + popDone := make(chan struct{}) + go func() { + defer close(popDone) + for { + if _, ok := q.pop(); ok { + popped++ + continue + } + select { + case <-q.notify: + case <-stop: + for { + if _, ok := q.pop(); !ok { + return + } + popped++ + } + } + } + }() + wg.Wait() + close(stop) + <-popDone + if popped+dropped != pushers*perPusher { + t.Errorf("popped %d + dropped %d != pushed %d", popped, dropped, pushers*perPusher) + } + if depth, bytes := q.stats(); depth != 0 || bytes != 0 { + t.Errorf("drained queue reports depth %d bytes %d", depth, bytes) + } +} + +func s3GaugeValue(t *testing.T, g prometheus.Gauge) float64 { + t.Helper() + var m dto.Metric + if err := g.Write(&m); err != nil { + t.Fatalf("gauge Write: %v", err) + } + return m.GetGauge().GetValue() +} + +func s3CounterValue(t *testing.T, c prometheus.Counter) float64 { + t.Helper() + var m dto.Metric + if err := c.Write(&m); err != nil { + t.Fatalf("counter Write: %v", err) + } + return m.GetCounter().GetValue() +} + +// TestS3ParquetDest_queueBounds holds the worker in an upload and checks +// that Send never blocks, what it drops and that the gauges settle. +func TestS3ParquetDest_queueBounds(t *testing.T) { + tests := []struct { + name string + category string + description string + depth int + maxBytesEnvs int // byte bound in envelopes; 0 = none + sends int + expectedQueue int + expectedDrops map[string]int + }{ + {name: "s3_queue_full_no_block", category: "N", description: "100 Sends behind a blocked upload drop all but depth", + depth: 16, sends: 100, expectedQueue: 16, expectedDrops: map[string]int{s3DropReasonDepth: 84}}, + {name: "bytes_bound", category: "N", description: "a byte bound of 3 envelopes keeps 3", + depth: 16, maxBytesEnvs: 3, sends: 10, expectedQueue: 3, expectedDrops: map[string]int{s3DropReasonBytes: 7}}, + {name: "under_bounds", category: "P", description: "fewer Sends than the depth drop nothing", + depth: 16, sends: 5, expectedQueue: 5, expectedDrops: map[string]int{}}, + {name: "depth_one", category: "B", description: "depth 1 keeps only the newest", + depth: 1, sends: 4, expectedQueue: 1, expectedDrops: map[string]int{s3DropReasonDepth: 3}}, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + hold := make(chan struct{}) + uploading := make(chan struct{}, 1) + var envBytes int + d, upl, x := newS3ParquetFixtureCustom(t, 1, func(_ int) error { + select { + case uploading <- struct{}{}: + default: + } + <-hold + return nil + }, func(d *s3ParquetDest) { + d.queue = newEnvelopeQueue(tc.depth, 0) + }) + first := marshalEnvelopeBuf(t, x, mkEnvelope(1)) + envBytes = len(*first) + if tc.maxBytesEnvs > 0 { + // Every envelope here marshals to the same size, so set + // the bound now; the worker hasn't seen another push. + d.queue.mu.Lock() + d.queue.maxBytes = tc.maxBytesEnvs * envBytes + d.queue.mu.Unlock() + } + if _, err := d.Send(context.Background(), first); err != nil { + t.Fatalf("Send: %v", err) + } + <-uploading + for range tc.sends { + buf := marshalEnvelopeBuf(t, x, mkEnvelope(1)) + start := time.Now() + if _, err := d.Send(context.Background(), buf); err != nil { + t.Fatalf("Send: %v", err) + } + if el := time.Since(start); el > time.Second { + t.Fatalf("%s: Send took %v", tc.description, el) + } + } + depth, qBytes := d.queue.stats() + if depth != tc.expectedQueue { + t.Errorf("%s: queue depth %d, expected %d", tc.description, depth, tc.expectedQueue) + } + if g := s3GaugeValue(t, d.metrics.buffered.WithLabelValues(s3StageQueue)); int(g) != qBytes { + t.Errorf("%s: queue gauge %v, queue holds %d bytes", tc.description, g, qBytes) + } + if tc.maxBytesEnvs > 0 && qBytes > tc.maxBytesEnvs*envBytes { + t.Errorf("%s: queue holds %d bytes, over the bound %d", tc.description, qBytes, tc.maxBytesEnvs*envBytes) + } + for _, reason := range []string{s3DropReasonDepth, s3DropReasonBytes} { + got := int(s3CounterValue(t, d.metrics.dropped.WithLabelValues(reason))) + if got != tc.expectedDrops[reason] { + t.Errorf("%s: dropped{%s} = %d, expected %d", tc.description, reason, got, tc.expectedDrops[reason]) + } + gotBytes := int(s3CounterValue(t, d.metrics.droppedBytes.WithLabelValues(reason))) + if gotBytes != tc.expectedDrops[reason]*envBytes { + t.Errorf("%s: dropped_bytes{%s} = %d, expected %d", tc.description, reason, gotBytes, tc.expectedDrops[reason]*envBytes) + } + } + close(hold) + if err := d.Close(); err != nil { + t.Fatalf("Close: %v", err) + } + // close_drains: the held envelope and every queued one upload. + if got := len(upl.Calls()); got != 1+tc.expectedQueue { + t.Errorf("%s: %d uploads, expected %d", tc.description, got, 1+tc.expectedQueue) + } + // gauge_returns_zero + if g := s3GaugeValue(t, d.metrics.buffered.WithLabelValues(s3StageQueue)); g != 0 { + t.Errorf("%s: queue gauge %v after drain, expected 0", tc.description, g) + } + if g := s3GaugeValue(t, d.metrics.buffered.WithLabelValues(s3StageBuilder)); g != 0 { + t.Errorf("%s: builder gauge %v after drain, expected 0", tc.description, g) + } + }) + } +} + +// TestS3ParquetDest_sendAfterCloseRefunds: a Send after Close returns +// errSendOnClosed and leaves the queue untouched. +func TestS3ParquetDest_sendAfterCloseRefunds(t *testing.T) { + d, _, x := newS3ParquetFixture(t, 1<<20, nil) + if err := d.Close(); err != nil { + t.Fatalf("Close: %v", err) + } + buf := marshalEnvelopeBuf(t, x, mkEnvelope(1)) + n, err := d.Send(context.Background(), buf) + if err != errSendOnClosed || n != 0 { + t.Fatalf("Send after Close = (%d, %v), expected (0, errSendOnClosed)", n, err) + } + if len(*buf) != 0 { + t.Errorf("refunded buffer not reset: len %d", len(*buf)) + } + if depth, _ := d.queue.stats(); depth != 0 { + t.Errorf("queue depth %d after a rejected Send", depth) + } +} + +// BenchmarkS3Send is the Send hot path, push and notify, with a popper +// draining so the queue stays shallow; chan_baseline is the buffered +// channel send it replaced. +func BenchmarkS3Send(b *testing.B) { + buf := make([]byte, 768<<10) + item := envelopeBytes{buf: &buf} + b.Run("queue", func(b *testing.B) { + q := newEnvelopeQueue(s3QueueDepthDefaultCst, s3MaxBufferedBytesDefaultCst) + done := make(chan struct{}) + go func() { + for { + select { + case <-q.notify: + for { + if _, ok := q.pop(); !ok { + break + } + } + case <-done: + return + } + } + }() + defer close(done) + b.ReportAllocs() + for b.Loop() { + _ = q.push(item) + } + }) + b.Run("chan_baseline", func(b *testing.B) { + ch := make(chan envelopeBytes, s3QueueDepthDefaultCst) + done := make(chan struct{}) + go func() { + for { + select { + case <-ch: + case <-done: + return + } + } + }() + defer close(done) + b.ReportAllocs() + for b.Loop() { + ch <- item + } + }) +} diff --git a/pkg/xtcpnl/xtcpnl_inet_diag_conginfo.go b/pkg/xtcpnl/xtcpnl_inet_diag_conginfo.go index 4711c2f3d..e31845f43 100644 --- a/pkg/xtcpnl/xtcpnl_inet_diag_conginfo.go +++ b/pkg/xtcpnl/xtcpnl_inet_diag_conginfo.go @@ -1,10 +1,10 @@ package xtcpnl import ( - "bytes" "errors" "github.com/randomizedcoder/xtcp2/gen/go/xtcp_flat_record" + "github.com/randomizedcoder/xtcp2/pkg/strpolicy" ) // https://github.com/torvalds/linux/blob/master/include/uapi/linux/inet_diag.h#L134 @@ -82,14 +82,6 @@ func DeserializeCongInfoXTCP(data []byte, x *xtcp_flat_record.XtcpFlatRecord) (e return ErrCongInfoSmall } - // Preserve the actual kernel name, including algorithms absent from our - // enum (for example Reno). Controller exposure is based on this field. - // Copy out of the reused netlink buffer and exclude the NUL terminator. - name := data - if end := bytes.IndexByte(name, 0); end >= 0 { - name = name[:end] - } - x.InetDiagCong = string(name) x.InetDiagCongEnum = xtcp_flat_record.XtcpFlatRecord_CONGESTION_ALGORITHM_UNSPECIFIED // Match on the first 3 bytes — the kernel attribute is a null-terminated @@ -119,5 +111,25 @@ func DeserializeCongInfoXTCP(data []byte, x *xtcp_flat_record.XtcpFlatRecord) (e x.InetDiagCongEnum = xtcp_flat_record.XtcpFlatRecord_CONGESTION_ALGORITHM_VEGAS } + // Preserve the actual kernel name, including algorithms absent from our + // enum (for example Reno). Controller exposure is based on this field. + // Copy out of the reused netlink buffer and exclude the NUL terminator. + // One pass finds the NUL and checks the name is printable ASCII, the + // string policy's fast path; anything else goes through Clean. + name, printable := data, true + for i, c := range data { + if c == 0 { + name = data[:i] + break + } + if c < 0x20 || c > 0x7e { + printable = false + } + } + if printable && len(name) <= strpolicy.MaxCong { + x.InetDiagCong = string(name) + } else { + x.InetDiagCong = strpolicy.Clean(string(name), strpolicy.MaxCong) + } return nil } diff --git a/pkg/xtcpnl/xtcpnl_inet_diag_conginfo_test.go b/pkg/xtcpnl/xtcpnl_inet_diag_conginfo_test.go index 5b19216a3..ebf6cd82c 100644 --- a/pkg/xtcpnl/xtcpnl_inet_diag_conginfo_test.go +++ b/pkg/xtcpnl/xtcpnl_inet_diag_conginfo_test.go @@ -36,6 +36,10 @@ func TestDeserializeCongInfoXTCPName(t *testing.T) { {"bbr\x00", "bbr", xtcp_flat_record.XtcpFlatRecord_CONGESTION_ALGORITHM_BBR1}, {"bbr2\x00", "bbr2", xtcp_flat_record.XtcpFlatRecord_CONGESTION_ALGORITHM_BBR2}, {"bbr3\x00", "bbr3", xtcp_flat_record.XtcpFlatRecord_CONGESTION_ALGORITHM_BBR3}, + // String policy: a control byte is replaced and the name stays + // within TCP_CA_NAME_MAX. + {"ab\x01cdefghijklmn\x00", "ab\uFFFDcdefghijklm", xtcp_flat_record.XtcpFlatRecord_CONGESTION_ALGORITHM_UNSPECIFIED}, + {"0123456789abcdefXYZ", "0123456789abcdef", xtcp_flat_record.XtcpFlatRecord_CONGESTION_ALGORITHM_UNSPECIFIED}, } { t.Run(tc.name, func(t *testing.T) { data := []byte(tc.wire) @@ -202,3 +206,16 @@ func DeserializeCongInfoBoth(b *testing.B, fn func(data []byte, ci *CongInfo) (n // b.Errorf("Test %s resultMI.Fmem:%d != test.mi.Fmem:%d", test.description, resultMI.Fmem, test.mi.Fmem) // } } + +// BenchmarkDeserializeCongInfoXTCP is the per-socket congestion decode the +// poller runs, including the string policy's ASCII fast path. +func BenchmarkDeserializeCongInfoXTCP(b *testing.B) { + data := []byte("cubic\x00\x00\x00") + record := new(xtcp_flat_record.XtcpFlatRecord) + b.ReportAllocs() + for b.Loop() { + if err := DeserializeCongInfoXTCP(data, record); err != nil { + b.Fatal(err) + } + } +} diff --git a/proto/xtcp_config/v1/xtcp_config.proto b/proto/xtcp_config/v1/xtcp_config.proto index 970a8b30d..afc9d9d46 100644 --- a/proto/xtcp_config/v1/xtcp_config.proto +++ b/proto/xtcp_config/v1/xtcp_config.proto @@ -1188,6 +1188,22 @@ message XtcpConfig { gte: { seconds: 0 } }]; + // Envelopes queued between the poller and the S3 worker. Past this depth + // the oldest is dropped (xtcp_s3_dropped_total{reason="queue_depth"}), so + // an S3 outage never blocks polling. 0 = 16. + uint32 s3_queue_depth = 116 [ + (buf.validate.field).required = false, + (buf.validate.field).uint32 = { + lte: 4096 + }]; + + // Marshaled envelope bytes the S3 queue may hold. Past this the oldest + // envelope is dropped (reason="buffered_bytes"); the newest is always + // kept. 0 = no byte bound (the depth bound alone). Flag default 64 MiB. + uint64 s3_max_buffered_bytes = 117 [ + (buf.validate.field).required = false + ]; + // ---- identity & labels stamped on every record (130-149) ----------------- // Hostname override. When empty the daemon uses os.Hostname(); set this to