Clean re-implementation of PR #92: relative date filters, export improvements, and duplicate suppression fixes by Copilot · Pull Request #121 · rsyslog/loganalyzer

Copilot · 2026-05-04T14:16:49Z

Fixed index.php:356: changed $tsDiff < $DuplicateRecordMaxTsDistance to $tsDiff <= $DuplicateRecordMaxTsDistance so records exactly at the configured distance are also suppressed

Original prompt

Summary

This is a clean re-implementation of the features proposed in PR #92 (by @rusak47), which has grown stale and has merge conflicts. All 7 bugs identified by automated review must be fixed in this implementation. Do not cherry-pick the old PR code directly — re-implement each feature cleanly against the current master.

Features to Implement

1. Relative Date/Time Syntax for `datefrom` / `dateto` Filters

File: src/include/functions_filters.php → GetTimeStampFromTimeString()
File: src/include/functions_common.php → GetEventTime()

Add support for compact relative date expressions in addition to existing absolute formats:

Input example	Meaning
`T00:00:00`	Today at midnight
`T12:30`	Today at 12:30
`\-1T` or `-1T`	Yesterday (same time)
`\-2T01:30` or `-2T01:30`	2 days ago at 01:30

Implementation notes:

Strip any leading backslash escaping from the input string at the start of both functions (to handle URL-encoded filter values).
Add a new else if branch using regex: /(-?[0-9]{1,2})?T([0-9]{0,2}):?([0-9]{0,2}):?([0-9]{0,2})/
Parse hours/minutes/seconds (default 0 if absent), call mktime($hh, $mm, $ss), then apply the day offset via strtotime("$days days", $szTime) if the day component is a negative integer.
This must not break any existing absolute date format parsing.

2. Export All Matched Pages (not just current page)

File: src/export.php
File: src/include/config.sample.php

Add a new config setting $CFG['ExportAllMatchPages'] (default 0).

When enabled, the export loop must iterate through all matching records, not just the current page's worth. When disabled, it keeps the existing behavior (current page only).

Implementation notes:

Read the config at the top of export.php: $content['ExportAllMatchPages'] = GetConfigSetting("ExportAllMatchPages", 0, CFGLEVEL_USER) == 1;
In the main export loop, after incrementing $counter, handle the ERROR_MSG_SKIPMESSAGE skip loop, then break only if !$content['ExportAllMatchPages'] && $counter >= $content['CurrentViewEntriesPerPage']. Otherwise continue until $ret != SUCCESS.
The loop condition must become while ($ret == SUCCESS) (remove the counter-based limit from the while condition).
Document the new config in src/include/config.sample.php with a comment.

3. Timestamped Export Filenames

File: src/export.php

Replace the static filename ExportMessages with one that includes the actual time range of exported records:

ExportMessages_20230101T120000-20230101T130000

Implementation notes:

Track $content['period_start_ts'] (first record's timestamp) and $content['period_end_ts'] (last record's timestamp) during the export loop. Only set period_start_ts once (on the first record).
After the loop, build the filename: "ExportMessages_" . date('Ymd\THis', $content['period_start_ts']) . "-" . date('Ymd\THis', $content['period_end_ts']).
Guard with isset($content['period_start_ts']) — fall back to ExportMessages if no records were exported.
The timereported field must be guarded with isset($logArray['timereported']) before accessing [EVTIME_TIMESTAMP] — this applies to all timestamp accesses in the export loop.

4. Separate Export Duplicate Suppression from View Suppression

File: src/export.php
File: src/include/config.sample.php

Add $CFG['ExportSuppressDuplicatedMessages'] (default 0) — distinct from the existing $CFG['SuppressDuplicatedMessages'] (which controls the view).

Implementation notes:

In export.php, read: $content['SuppressDuplicatedMessages'] = GetConfigSetting("ExportSuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1;
This decouples export deduplication from view deduplication.
Document in config.sample.php.

5. Fix Duplicate Message Suppression in View (`src/index.php`)

File: src/index.php

The existing "Suppress duplicated messages" logic is broken. Re-implement it cleanly, fixing all 7 known bugs:

Bug fixes required:

timereported not guarded — Before accessing $logArray["timereported"][EVTIME_TIMESTAMP], always check isset($logArray["timereported"]). If absent, treat the timestamp as 0 (no timestamp-based distance check).
Pending duplicate count dropped at boundary — After the main do...while loop exits, if $duplicateCount > 0 (there are unseen pending duplicates from a trailing run), inject the suppression summary row before the loop ends. This flush must happen outside the loop body.
Off-by-one in suppression threshold — Change if ($duplicateCount > 1) to if ($duplicateCount > 0) — a single duplicate should still be reported/counted.
Initialization — Initialize $szLastMessage = "", $szLastMessageTimestamp = 0, $duplicateCount = 0, $duplicateCountTotal = 0 before the loop.
DuplicateRecordMaxTsDistance config — Rea...

This pull request was created from Copilot chat.

Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/95914bfd-8533-44ae-8d55-d98bb7e9195e Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

…esult.cache to .gitignore Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/95914bfd-8533-44ae-8d55-d98bb7e9195e Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

alorbach

Looks good

cubic-dev-ai

5 issues found across 8 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/include/functions_filters.php">

<violation number="1" location="src/include/functions_filters.php:350">
P2: Positive day offsets are accepted by the regex but never applied, so accepted inputs like `1T12:00` produce an incorrect timestamp.</violation>
</file>

<file name="src/export.php">

<violation number="1" location="src/export.php:237">
P2: Update `period_end_ts` before the duplicate `continue`; otherwise trailing suppressed duplicates can produce an incorrect export end timestamp in the filename.</violation>

<violation number="2" location="src/export.php:261">
P2: Check `timereported[EVTIME_TIMESTAMP]` before using it; the current guard only checks the parent key and can produce undefined-index warnings.</violation>
</file>

<file name="src/index.php">

<violation number="1" location="src/index.php:353">
P2: The timestamp guard is incomplete: it checks `timereported` but not `timereported[EVTIME_TIMESTAMP]`, which can trigger runtime warnings on records missing that nested key.</violation>
</file>

<file name="src/include/functions_common.php">

<violation number="1" location="src/include/functions_common.php:1337">
P2: Positive relative day offsets are parsed but ignored. If the input is positive, it silently evaluates to today instead of the intended future date.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}

…ds, period_end_ts in duplicate path Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/3fe9f4cb-f857-4b60-a17f-9756aa9d478e Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

alorbach · 2026-05-04T18:53:07Z

@gemini review
@cubic-dev-ai review

cubic-dev-ai · 2026-05-04T18:53:17Z

@gemini review
@cubic-dev-ai review

@alorbach I have started the AI code review. It will take a few minutes to complete.

gemini-code-assist

Code Review

This pull request enhances duplicate message suppression for both the main display and export functionality, introducing a configurable maximum timestamp distance for identifying duplicates. It also adds support for relative date syntax in time filters, updates export filenames to include date ranges, and includes UI updates to show the count of suppressed records. Review feedback focuses on improving the consistency and safety of array access when retrieving event timestamps to prevent potential PHP notices and improve readability.

gemini-code-assist · 2026-05-04T18:56:01Z

+				if ( GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1 && isset($logArray[SYSLOG_MESSAGE]) )
 				{
+					$szCurrentMessage = $logArray[SYSLOG_MESSAGE];
+					$szCurrentTs = (isset($logArray['timereported']) && is_array($logArray['timereported']) && isset($logArray['timereported'][EVTIME_TIMESTAMP])) ? $logArray['timereported'][EVTIME_TIMESTAMP] : 0;


The variable $szCurrentTs is calculated using a ternary operator that checks is_array($logArray['timereported']). This check is redundant if isset($logArray['timereported']) is already true, as the subsequent access ['timereported'][EVTIME_TIMESTAMP] would fail if it were not an array. Simplify this to improve readability.

$szCurrentTs = (isset($logArray['timereported'][EVTIME_TIMESTAMP])) ? $logArray['timereported'][EVTIME_TIMESTAMP] : 0;

gemini-code-assist · 2026-05-04T18:56:01Z

+							if ( isset($logArray['timereported'][EVTIME_TIMESTAMP]) )
+								$content['period_end_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];


The nested key check isset($logArray['timereported'][EVTIME_TIMESTAMP]) is correct, but for consistency with other parts of the codebase and to avoid potential notices if timereported is not an array, ensure is_array($logArray['timereported']) is also checked.

if ( isset($logArray['timereported']) && is_array($logArray['timereported']) && isset($logArray['timereported'][EVTIME_TIMESTAMP]) ) $content['period_end_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];

gemini-code-assist · 2026-05-04T18:56:01Z

+					if ( !isset($content['period_start_ts']) && isset($logArray['timereported'][EVTIME_TIMESTAMP]) )
+						$content['period_start_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];
+					if ( isset($logArray['timereported'][EVTIME_TIMESTAMP]) )
+						$content['period_end_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];


Similar to the duplicate suppression logic, the timestamp tracking should explicitly verify that timereported is an array before accessing the nested key to prevent potential PHP notices.

if ( !isset($content['period_start_ts']) && isset($logArray['timereported']) && is_array($logArray['timereported']) && isset($logArray['timereported'][EVTIME_TIMESTAMP]) ) $content['period_start_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP]; if ( isset($logArray['timereported']) && is_array($logArray['timereported']) && isset($logArray['timereported'][EVTIME_TIMESTAMP]) ) $content['period_end_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];

cubic-dev-ai

No issues found across 8 files

Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/2cf4027a-cbdb-407e-8a06-bf092eaf68f5 Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

alorbach · 2026-05-04T19:03:28Z

@gemini review
@cubic-dev-ai review

cubic-dev-ai · 2026-05-04T19:03:38Z

@gemini review
@cubic-dev-ai review

@alorbach I have started the AI code review. It will take a few minutes to complete.

gemini-code-assist · 2026-05-04T19:07:44Z

Warning

Gemini is experiencing higher than usual traffic and was unable to create the review. Please try again in a few hours by commenting /gemini review.

cubic-dev-ai

1 issue found across 8 files

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/index.php">

<violation number="1" location="src/index.php:356">
P2: The duplicate timestamp window uses `<` instead of `<=`, so records exactly at `DuplicateRecordMaxTsDistance` are not suppressed.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.}

… distance check Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/1e048796-a278-4c79-ac25-855036f80f54 Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

alorbach · 2026-05-04T19:14:44Z

@gemini review
@cubic-dev-ai review

cubic-dev-ai · 2026-05-04T19:14:53Z

@gemini review
@cubic-dev-ai review

@alorbach I have started the AI code review. It will take a few minutes to complete.

gemini-code-assist

Code Review

This pull request introduces several enhancements to the log viewer, including a more robust duplicate message suppression system for both the main view and export functionality. It adds a new configuration setting, DuplicateRecordMaxTsDistance, to control the time window for duplicates and updates the UI to display the count of suppressed records. Additionally, the PR implements support for relative date syntax (e.g., 'T00:00:00', '-1T') in time filters and improves export file naming by including the period's start and end timestamps. Feedback focuses on optimizing configuration lookups within loops and ensuring the new timestamp distance logic is applied consistently across both the main view and the export process.

gemini-code-assist · 2026-05-04T19:17:24Z

+			$DuplicateRecordMaxTsDistance = GetConfigSetting("DuplicateRecordMaxTsDistance", PHP_INT_MAX, CFGLEVEL_USER);
+			// ---
+
+			// --- Init duplicate suppression state
+			$szLastMessage = "";
+			$szLastMessageTimestamp = 0;
+			$duplicateCount = 0;
+			$duplicateCountTotal = 0;
 			// ---

 			//Loop through the messages!
 			do
 			{
 				// --- Extra stuff for suppressing messages
-				if (
-						GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1 
-						&&
-						isset($logArray[SYSLOG_MESSAGE])
-					)
+				if ( GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1 && isset($logArray[SYSLOG_MESSAGE]) )


The call to GetConfigSetting for SuppressDuplicatedMessages inside the loop is inefficient as it performs multiple checks (session, user config, global config) on every iteration. It should be cached in a local variable outside the loop, similar to how $DuplicateRecordMaxTsDistance is handled at line 336.

$DuplicateRecordMaxTsDistance = GetConfigSetting("DuplicateRecordMaxTsDistance", PHP_INT_MAX, CFGLEVEL_USER); $SuppressDuplicatedMessages = GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1; // --- // --- Init duplicate suppression state $szLastMessage = ""; $szLastMessageTimestamp = 0; $duplicateCount = 0; $duplicateCountTotal = 0; // --- //Loop through the messages! do { // --- Extra stuff for suppressing messages if ( $SuppressDuplicatedMessages && isset($logArray[SYSLOG_MESSAGE]) )

gemini-code-assist · 2026-05-04T19:17:24Z

-
-						if ( !isset($szLastMessage) ) // Only set lastmgr
-							$szLastMessage = $logArray[SYSLOG_MESSAGE];
+						if ( $szLastMessage !== "" && $szLastMessage == $logArray[SYSLOG_MESSAGE] )


The duplicate suppression logic in the export loop is missing the timestamp distance check (DuplicateRecordMaxTsDistance) that was implemented for the main view. To ensure consistency and adhere to the documentation in config.sample.php (line 108), the export suppression should also respect this distance. This requires reading the DuplicateRecordMaxTsDistance setting outside the loop and tracking the timestamp of the last non-suppressed message (szLastMessageTimestamp) to calculate the difference.

cubic-dev-ai

No issues found across 8 files

Initial plan

c832e45

Copilot AI assigned Copilot and alorbach May 4, 2026

Copilot started work on behalf of alorbach May 4, 2026 14:16 View session

Copilot AI and others added 3 commits May 4, 2026 14:21

Initial plan: clean re-implementation of PR #92 features

d0df95b

Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/95914bfd-8533-44ae-8d55-d98bb7e9195e Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

Implement all 7 features from PR #92 clean re-implementation

2213e14

Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/95914bfd-8533-44ae-8d55-d98bb7e9195e Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

Fix unanchored regex in GetTimeStampFromTimeString and add .phpunit.r…

7bb2a92

…esult.cache to .gitignore Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/95914bfd-8533-44ae-8d55-d98bb7e9195e Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Re-implement date/time filters with relative syntax~~ Clean re-implementation of PR #92: relative date filters, export improvements, and duplicate suppression fixes May 4, 2026

Copilot AI requested a review from alorbach May 4, 2026 14:29

Copilot finished work on behalf of alorbach May 4, 2026 14:29

alorbach marked this pull request as ready for review May 4, 2026 14:30

alorbach approved these changes May 4, 2026

View reviewed changes

cubic-dev-ai Bot reviewed May 4, 2026

View reviewed changes

Comment thread src/include/functions_filters.php Outdated

Comment thread src/export.php

Comment thread src/export.php Outdated

Comment thread src/index.php Outdated

Comment thread src/include/functions_common.php Outdated

Copilot started work on behalf of alorbach May 4, 2026 16:09 View session

Fix 5 issues from cubic-dev-ai review: day offsets, timereported guar…

8001597

…ds, period_end_ts in duplicate path Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/3fe9f4cb-f857-4b60-a17f-9756aa9d478e Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

Copilot AI requested a review from alorbach May 4, 2026 16:17

Copilot finished work on behalf of alorbach May 4, 2026 16:17

gemini-code-assist Bot reviewed May 4, 2026

View reviewed changes

cubic-dev-ai Bot reviewed May 4, 2026

View reviewed changes

Copilot started work on behalf of alorbach May 4, 2026 18:59 View session

Simplify redundant is_array() guard in index.php timereported check

decae94

Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/2cf4027a-cbdb-407e-8a06-bf092eaf68f5 Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

Copilot finished work on behalf of alorbach May 4, 2026 19:01

cubic-dev-ai Bot reviewed May 4, 2026

View reviewed changes

Comment thread src/index.php Outdated

Copilot started work on behalf of alorbach May 4, 2026 19:10 View session

Fix duplicate suppression boundary: use <= instead of < for timestamp…

3d9b587

… distance check Agent-Logs-Url: https://github.com/rsyslog/loganalyzer/sessions/1e048796-a278-4c79-ac25-855036f80f54 Co-authored-by: alorbach <1675286+alorbach@users.noreply.github.com>

Copilot finished work on behalf of alorbach May 4, 2026 19:12

gemini-code-assist Bot reviewed May 4, 2026

View reviewed changes

cubic-dev-ai Bot reviewed May 4, 2026

View reviewed changes

Copilot started work on behalf of alorbach May 4, 2026 19:21 View session

Copilot finished work on behalf of alorbach May 4, 2026 19:22

alorbach merged commit feb6180 into master May 4, 2026
7 checks passed

alorbach mentioned this pull request May 4, 2026

Export all pages, simplified datefrom, and suppress fix #92

Closed

alorbach deleted the copilot/reimplement-date-time-filters branch May 4, 2026 19:23

alorbach mentioned this pull request May 5, 2026

Incorrect date display #97

Closed

		if ( isset($logArray['timereported'][EVTIME_TIMESTAMP]) )
		$content['period_end_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];

Uh oh!

Conversation

Copilot AI commented May 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Features to Implement

1. Relative Date/Time Syntax for datefrom / dateto Filters

2. Export All Matched Pages (not just current page)

3. Timestamped Export Filenames

4. Separate Export Duplicate Suppression from View Suppression

5. Fix Duplicate Message Suppression in View (src/index.php)

Uh oh!

alorbach left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alorbach commented May 4, 2026

Uh oh!

cubic-dev-ai Bot commented May 4, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 4, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 4, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 4, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

alorbach commented May 4, 2026

Uh oh!

cubic-dev-ai Bot commented May 4, 2026

Uh oh!

gemini-code-assist Bot commented May 4, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alorbach commented May 4, 2026

Uh oh!

cubic-dev-ai Bot commented May 4, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 4, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 4, 2026

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented May 4, 2026 •

edited

Loading

1. Relative Date/Time Syntax for `datefrom` / `dateto` Filters

5. Fix Duplicate Message Suppression in View (`src/index.php`)