Metrics, actor metrics, cluster metrics etc.

[rogeralsing]

Cluster module is in good shape now and I think it's time to start thinking about what metrics we would like to provide out of the box.

e.g.
a standardized package for Promethius and other tools.
What metrics do we want to track?

Cluster

• Number of activations per kind per node
• Messages sent per kind per node
• Messages received per kind per node
• Response times for Cluster RequestAsync

Actor

• Supervisor Failures

Remote

• Endpoint failures

others?

[rogeralsing]

Relevant

https://sematext.com/blog/akka-monitoring/

https://developer.lightbend.com/docs/console/current/

[rogeralsing]

PoC #610

[alexeyzimarev]

For me, the biggest question is how to do it. Shall we use System.Diagnostics with event listeners and activities? Everything that happens within the actor and uses libraries with activities implemented (like HTTP client or SQL client) will work automatically in this case. I expected the gRPC client to be already instrumented but it doesn't seem to be the case.

Here's how the HttpClient is instrumented https://github.com/vancem/corefx/blob/master/src/System.Net.Http/src/HttpDiagnosticsGuide.md

We also use it in many production components.

[cupen]

Cluster

• Number of members per node.
• Number of cluster topology changes per node.

[rogeralsing]

There are some different ways forward here.

• Move all metric related bits into the core, the core is tightly coupled to these metric interfaces (but decoupled from implementations)
• Keep metrics outside as a separate module. then provide ways to augment the standard modules. e.g. allow modifying props for system actors

Coupled for simplicity, decoupled for flexibility ?

[rogeralsing]

https://twitter.com/jbogard/status/1336683978431074305

[rogeralsing]

Adding cluster heartbeats to the list

[rogeralsing]

Adding Futures to the list.
Might be good to see if the number of futures grow due to retries timing out