Skip to content

Clean re-implementation of PR #92: relative date filters, export improvements, and duplicate suppression fixes #121

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alorbach merged 7 commits into from
May 4, 2026
Merged

Clean re-implementation of PR #92: relative date filters, export improvements, and duplicate suppression fixes #121

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c832e45
Initial plan
Copilot May 4, 2026
d0df95b
Initial plan: clean re-implementation of PR #92 features
Copilot May 4, 2026
2213e14
Implement all 7 features from PR #92 clean re-implementation
Copilot May 4, 2026
7bb2a92
Fix unanchored regex in GetTimeStampFromTimeString and add .phpunit.r…
Copilot May 4, 2026
8001597
Fix 5 issues from cubic-dev-ai review: day offsets, timereported guar…
Copilot May 4, 2026
decae94
Simplify redundant is_array() guard in index.php timereported check
Copilot May 4, 2026
3d9b587
Fix duplicate suppression boundary: use <= instead of < for timestamp…
Copilot May 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@ src/config.php
/doc-site/docs/user-guide/chapters/

src/index.html
.phpunit.result.cache
95 changes: 75 additions & 20 deletions src/export.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,11 @@
// No limit - export all matching records
$maxExportRecords = PHP_INT_MAX;
}

// --- Read export-specific config settings
$content['ExportAllMatchPages'] = GetConfigSetting("ExportAllMatchPages", 0, CFGLEVEL_USER) == 1;
$content['SuppressDuplicatedMessages'] = GetConfigSetting("ExportSuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1;
// ---

// Copy current used columns here!
$content['Columns'] = $content['Views'][$currentViewID]['Columns'];
Expand Down Expand Up @@ -209,33 +214,55 @@
// We found matching records, so continue
if ( $ret == SUCCESS )
{
// --- Init duplicate suppression state
$szLastMessage = "";
$duplicateCount = 0;
// ---

//Loop through the messages!
do
{
// --- Extra stuff for suppressing messages
if (
GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1
&&
isset($logArray[SYSLOG_MESSAGE])
)
if ( $content['SuppressDuplicatedMessages'] && isset($logArray[SYSLOG_MESSAGE]) )
{

if ( !isset($szLastMessage) ) // Only set lastmgr
$szLastMessage = $logArray[SYSLOG_MESSAGE];
if ( $szLastMessage !== "" && $szLastMessage == $logArray[SYSLOG_MESSAGE] )

@gemini-code-assist gemini-code-assist Bot May 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The duplicate suppression logic in the export loop is missing the timestamp distance check (DuplicateRecordMaxTsDistance) that was implemented for the main view. To ensure consistency and adhere to the documentation in config.sample.php (line 108), the export suppression should also respect this distance. This requires reading the DuplicateRecordMaxTsDistance setting outside the loop and tracking the timestamp of the last non-suppressed message (szLastMessageTimestamp) to calculate the difference.

{
// It's a duplicate — count and skip
$duplicateCount++;

// Read next entry
do {
$ret = $stream->ReadNext($uID, $logArray);
} while ( $ret == ERROR_MSG_SKIPMESSAGE );
continue;
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
}
else
{
// Skip if same msg
if ( $szLastMessage == $logArray[SYSLOG_MESSAGE] )
// Different message — flush any pending duplicate summary
if ( $duplicateCount > 0 )
{
// Set last mgr
$szLastMessage = $logArray[SYSLOG_MESSAGE];

// Skip entry
continue;
foreach ( $content['Columns'] as $mycolkey )
{
$content['syslogmessages'][$counter][$mycolkey]['FieldColumn'] = $mycolkey;
$content['syslogmessages'][$counter][$mycolkey]['uid'] = '';
$content['syslogmessages'][$counter][$mycolkey]['fieldvalue'] = '';
}
if ( isset($content['fields'][SYSLOG_MESSAGE]) )
$content['syslogmessages'][$counter][SYSLOG_MESSAGE]['fieldvalue'] = "... suppressed $duplicateCount duplicate(s)...";
$counter++;
$duplicateCount = 0;
}
$szLastMessage = $logArray[SYSLOG_MESSAGE];
}
}
// ---
// ---

// --- Track period timestamps for the export filename
if ( !isset($content['period_start_ts']) && isset($logArray['timereported']) )
$content['period_start_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];
if ( isset($logArray['timereported']) )
$content['period_end_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Outdated
Comment on lines +265 to +268

@gemini-code-assist gemini-code-assist Bot May 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similar to the duplicate suppression logic, the timestamp tracking should explicitly verify that timereported is an array before accessing the nested key to prevent potential PHP notices.

if ( !isset($content['period_start_ts']) && isset($logArray['timereported']) && is_array($logArray['timereported']) && isset($logArray['timereported'][EVTIME_TIMESTAMP]) )
						$content['period_start_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];
					if ( isset($logArray['timereported']) && is_array($logArray['timereported']) && isset($logArray['timereported'][EVTIME_TIMESTAMP]) )
						$content['period_end_ts'] = $logArray['timereported'][EVTIME_TIMESTAMP];

// ---

// --- Now we populate the values array!
foreach($content['Columns'] as $mycolkey)
Expand Down Expand Up @@ -293,15 +320,41 @@

// Increment Counter
$counter++;
} while ($counter < $maxExportRecords && ($ret = $stream->ReadNext($uID, $logArray)) == SUCCESS);

// Safety limit — stop if we've hit the maximum export record count
if ( $counter >= $maxExportRecords )
break;

// Read next entry, skipping filtered-out entries
do {
$ret = $stream->ReadNext($uID, $logArray);
} while ( $ret == ERROR_MSG_SKIPMESSAGE );

// If not exporting all pages, stop after the current page size
if ( !$content['ExportAllMatchPages'] && $counter >= $content['CurrentViewEntriesPerPage'] )
break;

} while ($ret == SUCCESS);

// Flush any trailing duplicate summary row
if ( $content['SuppressDuplicatedMessages'] && $duplicateCount > 0 )
{
foreach ( $content['Columns'] as $mycolkey )
{
$content['syslogmessages'][$counter][$mycolkey]['FieldColumn'] = $mycolkey;
$content['syslogmessages'][$counter][$mycolkey]['uid'] = '';
$content['syslogmessages'][$counter][$mycolkey]['fieldvalue'] = '';
}
if ( isset($content['fields'][SYSLOG_MESSAGE]) )
$content['syslogmessages'][$counter][SYSLOG_MESSAGE]['fieldvalue'] = "... suppressed $duplicateCount duplicate(s)...";
$counter++;
}

if ( $content['read_direction'] == EnumReadDirection::Forward )
{
// Back Button was clicked, so we need to flip the array
$content['syslogmessages'] = array_reverse ( $content['syslogmessages'] );
}
// DEBUG
//print_r ( $content['syslogmessages'] );
}
}
else
Expand Down Expand Up @@ -338,7 +391,9 @@
$szOutputMimeType = "text/plain";
$szOutputCharset = "";

$szOutputFileName = "ExportMessages";
$szOutputFileName = isset($content['period_start_ts'])
? "ExportMessages_" . date('Ymd\THis', $content['period_start_ts']) . "-" . date('Ymd\THis', $content['period_end_ts'])
: "ExportMessages";
$szOutputFileExtension = ".txt";
$szOPFieldSeparator = " ";
$szOPFirstLineFieldNames = true;
Expand Down
3 changes: 3 additions & 0 deletions src/include/config.sample.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@
$CFG['EnableContextLinks'] = 1; // if enabled, context links within the messages will automatically be created and added. Set this to 0 to disable all context links.
$CFG['EnableIPAddressResolve'] = 1; // If enabled, IP Addresses inline messages are automatically resolved and the result is added in brackets {} behind the IP Address
$CFG['SuppressDuplicatedMessages'] = 0; // If enabled, duplicated messages will be suppressed in the main display.
$CFG['ExportSuppressDuplicatedMessages'] = 0; // If enabled, export will suppress duplicate records (see also DuplicateRecordMaxTsDistance)
$CFG['ExportAllMatchPages'] = 0; // If enabled, export includes all matched records, not just the current page
$CFG['DuplicateRecordMaxTsDistance'] = PHP_INT_MAX; // Max timestamp delta (seconds) between two matching records to be considered duplicates
$CFG['TreatNotFoundFiltersAsTrue'] = 0; // If you filter / search for messages, and the fields you are filtering for is not found, the filter result is treaten as TRUE!
$CFG['PopupMenuTimeout'] = 3000; // This variable defines the default timeout value for popup menus in milliseconds. (those menus which popup when you click on the value of a field.
$CFG['PhplogconLogoUrl'] = ""; // Put an Url to a custom toplogo you want to use.
Expand Down
20 changes: 19 additions & 1 deletion src/include/functions_common.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1322,8 +1322,26 @@ function RedirectResult( $szMsg, $newpage )
*/
function GetEventTime($szTimStr)
{
// Strip optional leading backslash (URL-encoded filter values may include it)
if ( strlen($szTimStr) > 0 && $szTimStr[0] === '\\' )
$szTimStr = substr($szTimStr, 1);

// Relative date syntax, samples: T00:00:00, T12:30, -1T, -2T01:30
if ( preg_match("/^(-?[0-9]{1,2})?T([0-9]{0,2}):?([0-9]{0,2}):?([0-9]{0,2})$/", $szTimStr, $out ) )
{
$days = (isset($out[1]) && strlen($out[1]) > 0) ? intval($out[1]) : 0;
$hh = strlen($out[2]) > 0 ? intval($out[2]) : 0;
$mm = strlen($out[3]) > 0 ? intval($out[3]) : 0;
$ss = strlen($out[4]) > 0 ? intval($out[4]) : 0;
$szTime = mktime($hh, $mm, $ss);
if ( $days < 0 )
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Outdated
$szTime = strtotime("$days days", $szTime);
$eventtime[EVTIME_TIMESTAMP] = $szTime;
$eventtime[EVTIME_TIMEZONE] = date('O');
$eventtime[EVTIME_MICROSECONDS] = 0;
}
// Sample: Mar 10 14:45:44
if ( preg_match("/(...) ([0-9]{1,2}) ([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/", $szTimStr, $out ) )
else if ( preg_match("/(...) ([0-9]{1,2}) ([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})/", $szTimStr, $out ) )
{
// RFC 3164 typical timestamp
$eventtime[EVTIME_TIMESTAMP] = mktime($out[3], $out[4], $out[5], GetMonthFromString($out[1]), $out[2], date("Y") );
Expand Down
16 changes: 16 additions & 0 deletions src/include/functions_filters.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,6 +323,10 @@ function GetMessageTypeDisplayName( $nMsgTypeID )

function GetTimeStampFromTimeString($szTimeString)
{
// Strip optional leading backslash (URL-encoded filter values may include it)
if ( strlen($szTimeString) > 0 && $szTimeString[0] === '\\' )
$szTimeString = substr($szTimeString, 1);

//Sample: 2008-4-1T00:00:00
if ( preg_match("/([0-9]{4,4})-([0-9]{1,2})-([0-9]{1,2})T([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2})$/", $szTimeString, $out) )
{
Expand All @@ -335,6 +339,18 @@ function GetTimeStampFromTimeString($szTimeString)
// return new timestamp
return mktime(0,0,0, $out[2], $out[3], $out[1]);
}
// Relative date syntax, samples: T00:00:00, T12:30, -1T, -2T01:30
else if ( preg_match("/^(-?[0-9]{1,2})?T([0-9]{0,2}):?([0-9]{0,2}):?([0-9]{0,2})$/", $szTimeString, $out) )
{
$days = (isset($out[1]) && strlen($out[1]) > 0) ? intval($out[1]) : 0;
$hh = strlen($out[2]) > 0 ? intval($out[2]) : 0;
$mm = strlen($out[3]) > 0 ? intval($out[3]) : 0;
$ss = strlen($out[4]) > 0 ? intval($out[4]) : 0;
$szTime = mktime($hh, $mm, $ss);
if ( $days < 0 )
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Outdated
$szTime = strtotime("$days days", $szTime);
return $szTime;
}
else
{
OutputDebugMessage("Unparseable Time in GetTimeStampFromTimeString - '" . $szTimeString . "'", DEBUG_WARN);
Expand Down
99 changes: 78 additions & 21 deletions src/index.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -333,39 +333,70 @@
$myMsgCharLimit = GetConfigSetting("ViewMessageCharacterLimit", 80, CFGLEVEL_USER);
$myStrCharLimit = GetConfigSetting("ViewStringCharacterLimit", 30, CFGLEVEL_USER);
$ViewColoredCells = GetConfigSetting("ViewColoredCells", 0, CFGLEVEL_USER);
$DuplicateRecordMaxTsDistance = GetConfigSetting("DuplicateRecordMaxTsDistance", PHP_INT_MAX, CFGLEVEL_USER);
// ---

// --- Init duplicate suppression state
$szLastMessage = "";
$szLastMessageTimestamp = 0;
$duplicateCount = 0;
$duplicateCountTotal = 0;
// ---

//Loop through the messages!
do
{
// --- Extra stuff for suppressing messages
if (
GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1
&&
isset($logArray[SYSLOG_MESSAGE])
)
if ( GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1 && isset($logArray[SYSLOG_MESSAGE]) )
Comment on lines +336 to +350

@gemini-code-assist gemini-code-assist Bot May 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The call to GetConfigSetting for SuppressDuplicatedMessages inside the loop is inefficient as it performs multiple checks (session, user config, global config) on every iteration. It should be cached in a local variable outside the loop, similar to how $DuplicateRecordMaxTsDistance is handled at line 336.

            $DuplicateRecordMaxTsDistance = GetConfigSetting("DuplicateRecordMaxTsDistance", PHP_INT_MAX, CFGLEVEL_USER);
            $SuppressDuplicatedMessages = GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1;
            // ---

            // --- Init duplicate suppression state
            $szLastMessage = "";
            $szLastMessageTimestamp = 0;
            $duplicateCount = 0;
            $duplicateCountTotal = 0;
            // ---

            //Loop through the messages!
            do
            {
                // --- Extra stuff for suppressing messages
                if ( $SuppressDuplicatedMessages && isset($logArray[SYSLOG_MESSAGE]) )

{
$szCurrentMessage = $logArray[SYSLOG_MESSAGE];
$szCurrentTs = isset($logArray['timereported']) ? $logArray['timereported'][EVTIME_TIMESTAMP] : 0;
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Outdated
$tsDiff = ($szLastMessageTimestamp > 0 && $szCurrentTs > 0) ? abs($szCurrentTs - $szLastMessageTimestamp) : 0;

if ( !isset($szLastMessage) ) // Only set lastmgr
$szLastMessage = $logArray[SYSLOG_MESSAGE];
else
if ( $szLastMessage !== "" && $szLastMessage == $szCurrentMessage && $tsDiff < $DuplicateRecordMaxTsDistance )
Comment thread
cubic-dev-ai[bot] marked this conversation as resolved.
Outdated
{
// Skip if same msg
if ( $szLastMessage == $logArray[SYSLOG_MESSAGE] )
// It's a duplicate
$duplicateCount++;
$duplicateCountTotal++;
$szLastMessageTimestamp = $szCurrentTs;

// --- Extra Loop to get the next entry!
do
{
// Set last mgr
$szLastMessage = $logArray[SYSLOG_MESSAGE];
$ret = $stream->ReadNext($uID, $logArray);
} while ( $ret == ERROR_MSG_SKIPMESSAGE );
// ---

// --- Extra Loop to get the next entry!
do
// Skip entry
continue;
}
else
{
// Different message — flush any pending duplicate summary row
if ( $duplicateCount > 0 )
{
$content['syslogmessages'][$counter]['cssclass'] = "line1";
$content['syslogmessages'][$counter]['MiscShowDebugGridCounter'] = $content['MiscShowDebugGridCounter'];
foreach ( $content['Columns'] as $mycolkey )
{
$ret = $stream->ReadNext($uID, $logArray);
} while ( $ret == ERROR_MSG_SKIPMESSAGE );
// ---

// Skip entry
continue;
$content['syslogmessages'][$counter]['values'][$mycolkey]['FieldColumn'] = $mycolkey;
$content['syslogmessages'][$counter]['values'][$mycolkey]['uid'] = '';
$content['syslogmessages'][$counter]['values'][$mycolkey]['FieldAlign'] = isset($fields[$mycolkey]) ? $fields[$mycolkey]['FieldAlign'] : 'left';
$content['syslogmessages'][$counter]['values'][$mycolkey]['fieldcssclass'] = "line1";
$content['syslogmessages'][$counter]['values'][$mycolkey]['fieldbgcolor'] = "";
$content['syslogmessages'][$counter]['values'][$mycolkey]['isnowrap'] = "nowrap";
$content['syslogmessages'][$counter]['values'][$mycolkey]['hasdetails'] = "false";
$content['syslogmessages'][$counter]['values'][$mycolkey]['detailimagealign'] = "TOP";
$content['syslogmessages'][$counter]['values'][$mycolkey]['detaillink'] = "#";
$content['syslogmessages'][$counter]['values'][$mycolkey]['fieldvalue'] = '';
}
if ( isset($content['syslogmessages'][$counter]['values'][SYSLOG_MESSAGE]) )
$content['syslogmessages'][$counter]['values'][SYSLOG_MESSAGE]['fieldvalue'] = "... suppressed $duplicateCount duplicate(s)...";
$counter++;
$duplicateCount = 0;
}
$szLastMessage = $szCurrentMessage;
$szLastMessageTimestamp = $szCurrentTs;
}
}
// ---
Expand Down Expand Up @@ -689,7 +720,33 @@
} while ( $ret == ERROR_MSG_SKIPMESSAGE );
// ---
} while ( $counter < $content['CurrentViewEntriesPerPage'] && ($ret == SUCCESS) );
//print_r ( $content['syslogmessages'] );

// Flush any trailing duplicate summary row that was pending when the loop ended
if ( GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1 && $duplicateCount > 0 )
{
$content['syslogmessages'][$counter]['cssclass'] = "line1";
$content['syslogmessages'][$counter]['MiscShowDebugGridCounter'] = $content['MiscShowDebugGridCounter'];
foreach ( $content['Columns'] as $mycolkey )
{
$content['syslogmessages'][$counter]['values'][$mycolkey]['FieldColumn'] = $mycolkey;
$content['syslogmessages'][$counter]['values'][$mycolkey]['uid'] = '';
$content['syslogmessages'][$counter]['values'][$mycolkey]['FieldAlign'] = isset($fields[$mycolkey]) ? $fields[$mycolkey]['FieldAlign'] : 'left';
$content['syslogmessages'][$counter]['values'][$mycolkey]['fieldcssclass'] = "line1";
$content['syslogmessages'][$counter]['values'][$mycolkey]['fieldbgcolor'] = "";
$content['syslogmessages'][$counter]['values'][$mycolkey]['isnowrap'] = "nowrap";
$content['syslogmessages'][$counter]['values'][$mycolkey]['hasdetails'] = "false";
$content['syslogmessages'][$counter]['values'][$mycolkey]['detailimagealign'] = "TOP";
$content['syslogmessages'][$counter]['values'][$mycolkey]['detaillink'] = "#";
$content['syslogmessages'][$counter]['values'][$mycolkey]['fieldvalue'] = '';
}
if ( isset($content['syslogmessages'][$counter]['values'][SYSLOG_MESSAGE]) )
$content['syslogmessages'][$counter]['values'][SYSLOG_MESSAGE]['fieldvalue'] = "... suppressed $duplicateCount duplicate(s)...";
$counter++;
}

// Expose suppressed record count and flag for the UI
$content['main_suppressed_recordcount'] = $duplicateCountTotal;
$content['SUPPRESS_ENABLED'] = GetConfigSetting("SuppressDuplicatedMessages", 0, CFGLEVEL_USER) == 1 ? "true" : "false";

// Move below processing - Read First and LAST UID's before start reading the stream!
// $content['uid_last'] = $stream->GetLastPageUID();
Expand Down
1 change: 1 addition & 0 deletions src/lang/en/main.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
$content['LN_GEN_NEXTPAGE'] = "Next Page";
$content['LN_GEN_PREVIOUSPAGE'] = "Previous Page";
$content['LN_GEN_RECORDCOUNT'] = "Total records found";
$content['LN_GEN_SUPPRESSEDRECORDCOUNT'] = "Suppressed";
$content['LN_GEN_PAGERSIZE'] = "Records per page";
$content['LN_GEN_PAGE'] = "Page";
$content['LN_GEN_PREDEFINEDSEARCHES'] = "Predefined Searches";
Expand Down
5 changes: 5 additions & 0 deletions src/templates/index.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,11 @@
<td nowrap class="line2" STYLE="padding: 0px 15px 0px 0px;"><B>{main_recordcount}</B></td>
<!-- ENDIF main_recordcount_found="true" -->

<!-- IF SUPPRESS_ENABLED="true" -->
<td nowrap class="cellmenu2">{LN_GEN_SUPPRESSEDRECORDCOUNT}:</td>
<td nowrap class="line2" STYLE="padding: 0px 15px 0px 0px;"><B>{main_suppressed_recordcount}</B></td>
<!-- ENDIF SUPPRESS_ENABLED="true" -->

<!-- IF main_pagerenabled="true" -->
<td nowrap class="cellmenu2">{LN_GEN_PAGERSIZE}:</td>
<td nowrap class="line2">
Expand Down
Loading