Coroot Node Agent: "error encoding and sending metric family: write tcp ... broken pipe"

[somasundar-espina]

While running the Coroot Node Agent in our Kubernetes cluster, we are observing repeated broken pipe errors in the node agent logs when it tries to encode and send metric families.

The errors appear continuously and seem to indicate that the TCP connection to the receiving endpoint is being closed unexpectedly while metrics are being transmitted.

Logs
E0306 12:19:19.614375 435883 main.go:193] error encoding and sending metric family: write tcp 10.244.2.225:80->10.244.0.63:47976: write: broken pipe
E0306 12:19:19.614375 435883 main.go:193] error encoding and sending metric family: write tcp 10.244.2.225:80->10.244.0.63:47976: write: broken pipe
E0306 12:19:19.614375 435883 main.go:193] error encoding and sending metric family: write tcp 10.244.2.225:80->10.244.0.63:47976: write: broken pipe
E0306 12:19:19.614375 435883 main.go:193] error encoding and sending metric family: write tcp 10.244.2.225:80->10.244.0.63:47976: write: broken pipe
E0306 12:19:19.920647 435883 main.go:193] error encoding and sending metric family: write tcp 10.244.2.225:80->10.244.0.63:46878: write: broken pipe
E0306 12:19:19.920647 435883 main.go:193] error encoding and sending metric family: write tcp 10.244.2.225:80->10.244.0.63:46878: write: broken pipe
E0306 12:19:19.920647 435883 main.go:193] error encoding and sending metric family: write tcp 10.244.2.225:80->10.244.0.63:46878: write: broken pipe

Observations
The issue occurs repeatedly in the logs.
It appears when the node agent is sending metric families.
The error suggests the remote side closes the connection before the write completes.

Questions
Is this expected behavior in some scenarios (e.g., connection reuse or client disconnect)?
Could this indicate a configuration or networking issue between the node agent and the receiver?
Please let me know if any additional logs or debugging information would help investigate this further.

[somasundar-espina]

@def @apollo13 can anyone help me on this

[def]

It’s definitely unexpected behavior. You didn’t mention what is receiving the metrics.

[somasundar-espina]

The coroot node-agent is trying to send the metrics to Prometheus server. The Prometheus server was deployed by the coroot

[def]

If you deployed Coroot via the Coroot Operator, agents send telemetry to Coroot (not Prometheus), so check the Coroot logs for any errors.

[somasundar-espina]

@def We have deployed coorot using helm chart community edition, we have disabled the clickhouse as we are not using it, and the Prometheus is deployed by the coroot itself. So it was working good till last week the errors started persisting from last week

coroot -> node-agent -> Prometheus

From Prometheus we are accessing the data in out application

[somasundar-espina]

In the mentioned error this line 10.244.2.225:80->10.244.0.63:46878. 10.244.0.63 This ip is the Prometheus pod but it is sending on wrong port prometheus is running on 80 port