Resolution: Batching processor and/or connector still needed

[jmacd]

Component(s)

No response

Is your feature request related to a problem? Please describe.

The batch processor deprecation formally never happened, yet it still has well-known defects (mainly error propagation, concurrency), and yet the exporterhelper still does not entirely replace it for the metadata_keys feature was not finished.

We have debated whether to modernize it #13582.

We have tried to remove references to it #13766.

There are two obstacles to the proposed solution in #13583:

1. Metrics are named incorrectly: exporterhelper is used to implement the "modernized" batch processor, its internal metrics show.
2. Many configurations will gain "double batching" by default. We cannot let the OTel helm chart by default begin double batching.

Describe the solution you'd like

To address stated problem (2) in #13583, consider renaming the component: inlinebatchprocessor would be appropriate, to keep the existing component alive. As we know, there are a few processors that specifically advertise they are better when used with a batch processor before them, and we could say "use the inline batch processor".

However, we can also expand the scope. In open-telemetry/opentelemetry-collector-contrib#37787 I proposed a "pipeline processor" component, which is effectively all of the exporterhelper features bundled into an accessory for introducing a queue (maybe persistent), a batcher, a retry, and a timeout anywhere you need one. This component does plumbing inclusive of the batch processor feature set, but then it would ideally use configuration closer to exporterhelper, not the legacy batchprocessor.

Considering both concerns, I personally prefer the pipelineprocessor direction: it would be like #13583 except:

• Configuration uses struct { Timeout; Retry; QueueBatch } i.e., exactly/precisely the exporterhelper feature set including the storage extension to enable mid-pipeline persistence
• Component is named pipelineprocessor

Then, batchprocessor is deprecated. As in #13766, we remove all references to the batch processor except the few places where it continues to make sense, e.g., groupbyattrsprocessor, where we'll say "use the pipeline processor with batch settings".

Describe alternatives you've considered

A similar proposal to "pipeline processor" was made, namely "queue processor":
open-telemetry/opentelemetry-collector-contrib#35803

The reason why someone might want a queue, has to do with failoverconnector, see:
open-telemetry/opentelemetry-collector-contrib#33007

The failoverconnector eventually added QueueBatch configuration. This has come full circle!

Untested! This means you can use a failover connector as an in-line batching process, and that the "modern" batch processor is already available by configuring a single failover connection. That is, if you're willing to (a) modify batchprocessor to exporterhelper configuration, (b) insert a connector for this purpose.

We might argue for "pipeline connector", then, which is exporterhelper's feature set in a connector, like "pipeline processor" is exporterhelper's feature set in a processor. Both appear to be reasonable ideas.

Additional context

I believe the Collector has power because mostly users do not have to think about connectors. The model of multiple receivers fanning-in (parallel), multiple processors (serial), multiple exporters fanning-out (parallel) is part of its success, its concise mental model. I would not want to recommend using a connector where a processor will do.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[atoulme]

I think an "accumulation" connector makes sense, rather than a queue, which implies we keep ordering.

[andrzej-stencel]

I think we should NOT deprecate the Batch processor. We should instead implement the improvements proposed by @jmacd and change the docs to discourage using of the Batch processor in favor of exporter helper, describing the specific situations where the Batch processor can be useful.

Having a connector might be useful in some scenarios, but not having a processor and having to use a connector instead, splitting the pipelines and obfuscating the config in effect would be bad user experience in my opinion.

[johnmichaelll238-cloud]

Hello, please can anyone put me up to speed on this issue, I'm a newcomer here and would really like to help out, even if it's just a sub issue. Thank you!

[lahsivjar]

I think we should NOT deprecate the Batch processor. We should instead implement the improvements proposed by @jmacd and change the docs to discourage using of the Batch processor in favor of exporter helper, describing the specific situations where the Batch processor can be useful.

+1 on this. There are use-cases where having batching outside the scope of exporter is required for performance. In the current architecture, multiple instances of exporter are required if different batching configuration need to be supported by the collector and the overhead of running multiple exporter instances can be very high depending on the exporter in question - for example for kafkaexporter, each instances maintains a cache of the topic metadata.

[dmitryax]

the exporterhelper still does not entirely replace it for the metadata_keys feature was not finished.

It is finished, #14139. We now has full parity.

I like the idea of having an optional queue+batch processor built on top of the exporter helper. I'd like to discuss the naming though. Maybe buffer or queue_batch processor?

[axw]

I like the idea of having an optional queue+batch processor built on top of the exporter helper.

👍

Maybe buffer or queue_batch processor?

Either of those sounds fine to me, as long as the scope of the processor is essentially the sending_queue config.

I also like queueprocessor for this scope, as proposed in open-telemetry/opentelemetry-collector-contrib#35803

If the scope also includes timeout and retry_on_failure, then a more abstract pipelineprocessor seems more appropriate. Although this is pragmatic, I personally find the name less intuitive, and it feels like a bit of a grab-bag. Another option would be to eventually introduce multiple processors: timeoutprocessor, retryprocessor, queueprocessor.