Skip to content

Project Proposal: Audit Logging SIG #2409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mlenkeit wants to merge 25 commits into
base: main
Choose a base branch
from
+213 −0
Open

Project Proposal: Audit Logging SIG #2409

Changes from 8 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
5094fb1
docs(auditlogs): add audit logging sig proposal
mlenkeit Oct 24, 2024
9337b7f
docs(auditlogs): re-number requirements
mlenkeit Oct 24, 2024
75f2c57
docs(auditlogs): remove template instructions
mlenkeit Oct 24, 2024
f81c2f4
docs(auditlogs): use OTel over OTEL
mlenkeit Nov 19, 2024
65ae32e
docs(auditlogs): list @reyang as first sponsor
mlenkeit Nov 19, 2024
776b821
docs(auditlogs): add Microsoft to interested vendors
mlenkeit Nov 19, 2024
6dd519d
docs(auditlogs): add contacts to vendor list
mlenkeit Nov 19, 2024
2ec002d
docs(auditlogs): use consistent punctuation for requirement list
mlenkeit Nov 19, 2024
d7e265f
docs(auditlogs): minor word change in Challenges chapter
mlenkeit Nov 20, 2024
405ddb5
docs(auditlogs): describe guarantee of delivery in appendix
mlenkeit Nov 21, 2024
0adb8e5
docs(auditlogs): add sample audit logs to appendix
mlenkeit Nov 21, 2024
a5ef343
docs(auditlogs): add links to sample audit logs
mlenkeit Nov 21, 2024
711dc46
docs(auditlogs): add links to appendix A
mlenkeit Nov 21, 2024
087865c
docs(auditlogs): use GitHub handle only in staffing list
mlenkeit Nov 21, 2024
3876a31
docs(auditlogs): add svrnm as GC liaison
mlenkeit Nov 21, 2024
8b38626
Merge branch 'audit-logging-sig-project-proposal' of github.com:apeir…
mlenkeit Nov 21, 2024
066501b
docs(auditlogs): minor changes in wording
mlenkeit Nov 22, 2024
70cbac4
docs(auditlogs): shorten requirement ids to pass spell check
mlenkeit Nov 22, 2024
a6b34f1
Merge branch 'main' into audit-logging-sig-project-proposal
mlenkeit Nov 25, 2024
6bc9a5e
docs(auditlogs): propose project lead
mlenkeit Mar 27, 2025
e03dcd8
Merge branch 'main' into audit-logging-sig-project-proposal
mlenkeit Mar 27, 2025
86bd77a
Merge remote-tracking branch 'upstream/main' into audit-logging-sig-p…
mlenkeit May 20, 2025
41329f3
* docs(auditlogs): describe phases approach
mlenkeit May 20, 2025
03d1de1
Add files via upload
hilmarf Sep 19, 2025
2f9813f
Revert "Add files via upload"
hilmarf Sep 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions projects/audit-logging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Audit Logging

## Background and description

Audit logging describes the capability of capturing audit-trail relevant events of a system to meet compliance requirements. Such events may originate from the infrastructure (e.g. a Kubernetes cluster) up to the application-level. It is a capability that is particularly relevant for providers of enterprise software.

Unlike regular application logs, audit logs are usually subject to long retention periods and software providers must guarantee their completeness (i.e. guarantee of delivery).

@reyang reyang Oct 31, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good points! In addition, these are something we might want to consider:

  1. Audit logs might be considered as a critical part of the business, which could result in a different API design strategy - for example, audit logging might require a different API behavior, if the information provided by the caller is invalid, the API might throw exception instead of failing silently and move on.
  2. Audit log might require some sensitive information without redaction due to the regulation requirements (e.g. user identity and client IP address).
  3. The data path could require higher level of access control or privilege.

@mlenkeit mlenkeit Nov 19, 2024 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reyang thanks for mentioning these points.

Especially the API behavior is something that we had thought about initially. However, when we first pitched audit logging on Slack, we received the following comment from Ted Young:

As a rule, the OpenTelemetry API never throws an exception. I understand why you might want this, though it is not present in many audit logging systems, which use regular loggers. So a strong case would have to be made on this particular point.

Based on this initial feedback, we decided to file this SIG proposal without proposing such API changes.


Examples of audit logs include:
- permission changes (e.g. of a service account or application user)
- modification of data
- accessing sensitive information
- failed login attempts

### Current challenges

Audit Logging is currently not within the scope of OpenTelemetry
Comment thread
mlenkeit marked this conversation as resolved.
Outdated

- no semantic conventions for audit logs in OTel

@svrnm svrnm Nov 19, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- no semantic conventions for audit logs in OTel
- no semantic conventions for audit logs in OTel

Can you provide some examples of what would be part of such semantic conventions? My knowledge on audit logs is very limited, so it would help to understand the problem much better.

@mlenkeit mlenkeit Nov 21, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@svrnm our experience has shown that in order to analyze audit logs at scale, it is important to define an (extensible) event catalog. The event catalog standardizes audit log events across workloads/produces. For example, our internal event catalog currently consists of 50+ such events. Ideally, such a catalog would be part of semantic conventions.

To make this more tangible, I've added some examples to the appendix of the document:
https://github.com/open-telemetry/community/pull/2409/files#diff-736e6b0ae9ae655b78d9ba007d08592071abb6cc1ef64d7893ff81642c8ec734R115-R192

@renewelches renewelches Nov 21, 2024

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another examples from the security world is https://github.com/ocsf.

@svrnm svrnm Nov 22, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @mlenkeit. Makes it much clearer

The metadata looks like attributes that would be covered by other semantic conventions (e.g. there is a log.record.id for the metadata.id, the timestamp of course and some of the other ones (e.g. for k8sCluster we have k8s.cluster.name. So I would assume here it is more about re-using and extending certain other domains that are not unique to "audit logs"

For the event and data examples you gave, I would argue that they are not "semantic conventions for audit logs" but "semantic conventions for log types that typically require the strict requirements of auditing". What do I mean by that: if we talk about "semantic conventions for audit logs" I think about a namespace called audit. that holds attributes that are specific to the business logic of audit logging, like a signature that helps to tamperproof the log line, or maybe even meta information under which regulation this log is required to be an "audit log"
In contrast "semantic conventions for log types that typically require the strict requirements of auditing" are their own namespaces like the "UserLoginFailure" example would fall into a "authentication" or "auth" namespace, with "auth.login.method" or "auth.login.failureReason" as potential attributes, event.name being set to auth.login.failure or something.

I am just making those things up to exemplify the difference, they will probably take a different form or shape eventually, so to make a long story short, here is a suggestion to rephrase:

Suggested change
- no semantic conventions for audit logs in OTel
- no semantic conventions for audit logs in OTel
- no semantic conventions for log types that typically rquire the strict requirements of auditing, like authentication, authorization and data changes

@renewelches thanks for calling out OCSF, if I remember correctly there were conversations in the past between OTel and OCSF, cc @lmolkova

@mlenkeit mlenkeit Nov 22, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding metadata, I fully agree: Most of these attributes are already covered by semconv. We may identify additional attributes in SIG meetings though, depending on the experience/requirement of other contributors/companies.

I understand how "semantic conventions for audit logs" can be misleading. To me, the suggestion that you made has a notion of particularly describe logs that are "already there" (e.g. events emitted by a K8s cluster) and can be considered relevant for audit purposes. Especially in enterprise software, it's common that applications produce logs that are specifically mean to be audit logs (and nothing else). To me, it' s important that we find wording that covers these two types that we do have.

How about the following?

Suggested change
- no semantic conventions for audit logs in OTel
- no semantic conventions for representing and identifying audit trail-relevant events in OTel (like authentication, authorization or modification of

@renewelches renewelches Nov 22, 2024

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned in another comment, this all depends on what attributes are changeable or must be immutable. As of my understanding an attribute could be altered by a processor in the collector. Which is something we would want to avoid or want to prevent in cases of audit logs. If we conclude that we can or should only guarantee immutability for the log itself then we must live with replication/doublication. Otherwise we might have to add the constrain that also certain attributes must be immutable.

@tigrannajaryan tigrannajaryan Dec 3, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 to looking into OCSF for security events and borrowing relevant semantic conventions from there.

@trisch-me trisch-me Jan 22, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mlenkeit security semconv SIG is also working on introducing security related attributes to the semconv, we have currently discussion around user authentication. I feel that this SIG and security SIG might have some common tasks/discussions

- OTel APIs/SDKs do not provide feedback to the application level whether data (in particular logs) have been successfully delivered to a remote endpoint. To guarantee delivery, either the SDK has to give those guarantees, or provide feedback to the application so that it can take care of guaranteed delivery itself.
- OTel collectors may lose audit logs in transit (i.e. no guarantee of delivery)
Comment thread
svrnm marked this conversation as resolved.
Outdated

@reyang reyang Nov 21, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does "OTel collectors" mean the "OpenTelemetry Collector" (https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification#project-naming) or "any collectors that can handle OpenTelemetry data (whether OTLP or something else)"?

@mlenkeit mlenkeit Nov 22, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@reyang This is indeed supposed to refer to the OpenTelemetry Collector. I've rephrased this particular sentence to "OTel Collector instances may lose audit logs in transit" and adjusted other occurrences of OTel collectors as well. 066501b


### Goals, objectives, and requirements

The goal of this project is to make OTel fit for audit logging purposes that meet compliance requirements of enterprise software providers, in particular:

- REQ-CONV-01: Semantic conventions for application-level audit logs are defined
- REQ-CONV-02: Semantic conventions for infrastructure-level audit logs are defined
- REQ-APPL-01: Guaranteed delivery of audit logs exported via OpenTelemetry SDK
- REQ-PIPE-01: OTel collector must provide guaranteed delivery of audit logs, including when its process is interrupted

## Deliverables

- semantic convention for audit logs
- extend OTel APIs/SDKs for audit logging purposes (in collaboration with the respective SIG)
- extend OTel collector for audit logging purposes (in collaboration with the respective SIG)

## Staffing / Help Wanted

The following vendors are interested in improving this area:
- SAP (@mlenkeit, @FWinkler79)
- Microsoft (@reyang)

Other vendors are invited to join the discussion.

### Required staffing

* Project lead: SAP (name tbd)
* Sponsors:
- Reiley Yang (@reyang)
- tbd
* GC liaison: tbd
Comment thread
mlenkeit marked this conversation as resolved.
Outdated
* Engineers:
* SAP will provide a prototype in two languages (tbd; likely two of Java, JavaScript, Go)

@reyang reyang Oct 31, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need prototype in two parts:

  1. API/SDK - this is where we need three programming languages IIRC.
  2. OTel Collector - higher guarantee on data delivery (completeness, integrity, latency, etc.), data path security.

@mlenkeit mlenkeit Nov 19, 2024 •
edited
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out! It's clear to us, but I'll work on making this clearer in the doc...

* Maintainers/approvers: tbd

## Timeline

TBD based on community involvement.

## Labels

- audit-logging (tbc)

## Project Board

TODO: add link

## SIG Meetings and Other Info

TODO: add information