Skip to content

TimeVary 2026 summer - #1690

Open
eshandilcsumb wants to merge 99 commits into
OpenEnergyDashboard:timeVaryfrom
eshandilcsumb:timeVary2026Summer
Open

TimeVary 2026 summer#1690
eshandilcsumb wants to merge 99 commits into
OpenEnergyDashboard:timeVaryfrom
eshandilcsumb:timeVary2026Summer

Conversation

@eshandilcsumb

@eshandilcsumb eshandilcsumb commented Aug 2, 2026

Copy link
Copy Markdown

Description

This pull request introduces a TimescaleDB-based reading aggregation implementation for fresh database installations.

The goal of this work is to replace the legacy PostgreSQL materialized-view aggregation approach with a scalable TimescaleDB design that better supports large reading datasets, time-varying unit conversions, and efficient meter and group graph queries.

The new implementation adds:

  • A TimescaleDB hypertable that splits raw readings at hour boundaries and time-varying conversion boundaries.
  • Meter hourly and daily continuous aggregates.
  • Group hourly and daily continuous aggregates using physical dependency caches.
  • Revision tracking for stale split data and group cache data.
  • Refresh and rebuild orchestration with advisory locking and bounded refresh support.
  • Batched reading and conversion writes.
  • Preloaded conversion metadata during time-varying conversion generation.
  • TimescaleDB-backed line, bar, compare, radar, and 3-D reading queries.
  • Supporting indexes for reading access, aggregation queries, group relationships, conversion overlap, and logs.

This change also updates fresh database schema creation to install the TimescaleDB objects in dependency order.

The motivation for this work is to improve aggregation performance and provide a foundation for moving OED reading analytics from legacy materialized views to TimescaleDB. The implementation is integrated for fresh databases, while migration support for existing installations remains future work.

Contributors:

Type of change

  • Note merging this changes the database configuration.
  • This change requires a documentation update

Checklist

  • I have followed the OED pull request ideas
  • I have removed text in ( ) from the issue request
  • I acknowledge that every person contributing to this work has signed the OED Contributing License Agreement and each author is listed in the Description section.

Limitations

  • A migration for upgrading existing databases has not yet been implemented or registered. Fresh database creation uses the new TimescaleDB schema, but existing deployments cannot yet be upgraded automatically.
  • Legacy PostgreSQL reading aggregation code and SQL definitions remain and need cleanup after migration support is complete.
  • Reading split maintenance currently uses row-level triggers. Statement-level or bulk split generation has not yet been implemented.
  • TimescaleDB operational tuning has not yet been completed, including chunk sizing, compression/columnstore settings, retention policies, and continuous aggregate refresh policies.
  • Production-scale benchmarking has not yet been completed.
  • Group dependency caches are currently rebuilt globally when invalidated rather than incrementally updated.
  • The daily meter aggregate currently reads directly from the split hypertable instead of using a hierarchical daily rollup.
  • Full database and web test suites have not been rerun as part of this update. Focused validation has been performed for TimescaleDB schema creation, conversion segment integration, reverse bidirectional conversion behavior, new indexes, and set-based group queries.

minhtriet359 and others added 30 commits June 29, 2025 18:28
…ine function that uses view, also all of Wendy's changes
…dings_unit, altered cik table to reflect previous groups work with time varying conversion, added 3rd version of meter line function based on previous groups implementaion of time varying conversions for testing purposes
Currently read-only. Uses fake data, as the Weeks API is not yet implemented.
…_readings_unit so that it no that it no longer has to apply the conversions it self.
* CRUD operations working with back-end
* client-side validation
* delete confirmation
* Spanish and French translations
Omit `id` from the default week values model
docker-compose should not be ignored and should not have to be altered for our time-varying conversion work
eshandilcsumb and others added 29 commits July 6, 2026 20:09
…ile as this was create file locks when running tests
…er_line_readings_unit structure and comment. create_hourly_readings and create_prerequisities have been updated to give proper credit to JoBarn for proof-of-concept and foundational research
…removed the TimescaleDB view drop for every restart and TimescaleDB vs materialized view value compare from unit tests
…guideline. Adjusted meter_daily_readings_unit TimeScaleDB view to be the base of group_daily_readings_units
…and removed PostgreSQL materialized views. Resolved migration issue to pg17 and up.
… proper repository in OED, timeVary2026SummerHandover.md
… reading bulk inserts and removed procedural query codes
…daries and synchronizing aggregate rebuilds.
@huss

huss commented Aug 4, 2026

Copy link
Copy Markdown
Member

@eshandilcsumb & @MartinCSUMB Thank you for this contribution. The PR description states the CLA has been signed by all contributors. However, OED records do not have one for @MartinCSUMB . Thus, I would like you to sign the CLA via the link in the description or let me know that you think OED records are off.

Once this is done, this PR can be formally reviewed. Please let me know if you have any questions/thoughts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants