diff --git a/README.md b/README.md index 16dd50a..b0d4405 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ Note OED is transitioning to mostly public design documents. The private DevDocs - [MQTT.md](./MQTT.md): Looking into integrating for meter data acquisition. - [baseline/baseline.md](./baseline/baseline.md): Add the ability for an admin to add baselines to meters and for users to display on a graphic. - [enhancementToGithubAction.md](./githubAction/enhancementToGithubAction.md): securing GitHub action information. -- [timescaleDB/timescaleDB.md](./timescaleDB/timescaleDB.md): Information on efforts to investigate TimescaleDB usage in OED. +- [timescaleDB/Handover.md](./timescaleDB/Handover.md): Information on migration to TimescaleDB in OED and possible improvement opportunities. - [infisicalIntegration/infisicalIntegration.md](./infisicalIntegration/infisicalIntegration.md): Instructions for how to utilize Infisical secrets management to store database passwords. ## Information diff --git a/timescaleDB/Handover.md b/timescaleDB/Handover.md new file mode 100644 index 0000000..4a711b3 --- /dev/null +++ b/timescaleDB/Handover.md @@ -0,0 +1,576 @@ +# TimescaleDB Meter Reading Aggregation Handover + +## Document status + +This document describes the implementation on branch `timeVary2026Summer` at +commit `1c08fb95c` on 2026-08-02. Treat the code and SQL linked below as the +source of truth if the branch advances after this handover. + +The TimescaleDB implementation is the active schema and query path for fresh +databases. It includes raw-reading ingestion, an incrementally maintained split +hypertable, four continuous aggregates, group dependency caches, bounded +refreshes, revision-driven rebuilds, and Timescale-backed line, bar, compare, +radar, and 3-D queries. + +The implementation is not deployment-complete for existing databases. The +current migration registry does not register the TimescaleDB migration work. +Do not assume that running `npm run migratedb` upgrades an existing production +database to the schema described here. + +The development database image is pinned in +[`containers/database/Dockerfile`](containers/database/Dockerfile) to: + +```text +timescale/timescaledb:2.27.2-pg17 +``` + +## Current status + +Implemented: + +- TimescaleDB hypertable preprocessing for raw readings. +- Splitting at both hour boundaries and time-varying conversion boundaries. +- Hourly and daily meter continuous aggregates. +- Hourly and daily group continuous aggregates. +- Physical group membership and graphic-unit compatibility caches. +- Batched raw-reading inserts and upserts. +- Bounded refreshes for CSV and eGauge imports. +- Advisory locking around aggregate refresh and rebuild operations. +- Revision counters for stale split data and stale group caches. +- Set-based meter line queries and multi-meter reading counts. +- Set-based group listing and group-child queries. +- Preloaded conversion metadata during `cik_vary` regeneration. +- Supporting indexes for reading bounds, group relationships, group caches, + CAGG access, conversion overlap, and log retrieval. + +Not complete: + +- A registered migration for installing or upgrading these objects in an + existing database. +- Removal of retained legacy PostgreSQL materialized-view code and SQL. +- Statement-level split-table maintenance for bulk reading writes. +- TimescaleDB chunk, compression/columnstore, retention, and refresh policies. +- Production-scale benchmarks for the current implementation. +- A hierarchical daily meter aggregate. +- Incremental group-cache recomputation. + +## Architecture + +### Meter data flow + +```text +readings + | + | AFTER INSERT/UPDATE/DELETE, FOR EACH ROW + v +hypertable_hourly_split + | + +-------------------------------+ + | | + v v +meter_hourly_readings_unit_cagg meter_daily_readings_unit_cagg +``` + +Both meter aggregates read `hypertable_hourly_split` directly. The daily meter +aggregate does not roll up the hourly meter aggregate. + +Each split row is bounded by: + +- The source reading interval. +- An hour boundary. +- The active `cik_vary` conversion interval. + +The split table copies the unit representation, seconds-in-rate, slope, +intercept, and destination graphic unit needed by downstream aggregation. + +### Group data flow + +```text +groups_immediate_children groups_immediate_meters + | | + +-------------+-------------+ + | + v + groups_deep_meters_cache + | + v + group_graphic_units_cache + +meter_hourly_readings_unit_cagg + dependency caches + | + v + group_hourly_readings_unit_cagg + +meter_daily_readings_unit_cagg + dependency caches + | + v + group_daily_readings_unit_cagg +``` + +The physical caches exist because TimescaleDB continuous aggregates cannot use +the recursive and dynamic dependency logic that previously resolved nested +groups and compatible graphic units at query time. + +## Core database objects + +| Object | Type | Purpose | +|---|---|---| +| `readings` | PostgreSQL table | Authoritative raw meter readings | +| `hypertable_hourly_split` | TimescaleDB hypertable | Hour- and conversion-bounded reading contributions with copied conversion metadata | +| `meter_hourly_readings_unit_cagg` | Continuous aggregate | Duration-weighted meter rates by hour and graphic unit | +| `meter_daily_readings_unit_cagg` | Continuous aggregate | Duration-weighted meter rates by day and graphic unit | +| `groups_deep_meters_cache` | PostgreSQL table | Flattened group-to-meter membership | +| `group_graphic_units_cache` | PostgreSQL table | Graphic units compatible with every source unit in a group | +| `group_hourly_readings_unit_cagg` | Continuous aggregate | Summed group rates by hour and graphic unit | +| `group_daily_readings_unit_cagg` | Continuous aggregate | Summed group rates by day and graphic unit | +| `reading_aggregate_state` | PostgreSQL table | Split-rebuild and group-cache revision counters | + +Primary definitions: + +- [`create_prerequisites.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/create_prerequisites.sql) +- [`create_hourly_readings.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/create_hourly_readings.sql) +- [`create_daily_readings.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/create_daily_readings.sql) +- [`create_group_dependencies.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/create_group_dependencies.sql) +- [`create_group_hourly_readings.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/create_group_hourly_readings.sql) +- [`create_group_daily_readings.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/create_group_daily_readings.sql) + +## Reading ingestion + +### Application batching + +[`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/Reading.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/Reading.js) writes readings +in transactional batches of 1,000 through `jsonb_to_recordset`. + +Current conflict behavior: + +- `insertAll()` fails on an existing `(meter_id, start_timestamp)` key. +- `insertOrIgnoreAll()` retains the existing row. +- `insertOrUpdateAll()` updates only `reading`. +- Duplicate upsert keys in one input retain the first end timestamp and final + reading value, matching the former sequential behavior. + +Batching reduces application/database round trips. It does not remove the +per-reading trigger cost. + +### Split-table trigger + +`trigger_readings_update_hourly_hypertable` calls +`update_hourly_hypertable()` once per changed reading. + +- `INSERT` generates all hour/conversion overlap slices. +- `UPDATE` deletes slices for the old reading interval and regenerates them. +- `DELETE` deletes slices belonging to the removed reading interval. + +The trigger joins the meter to its unit, finds overlapping `cik_vary` rows, +generates touched hours, intersects all boundaries, and inserts only positive +duration slices. + +Because conversion and unit metadata are copied, source metadata changes can +make existing split rows stale. Revision-driven rebuilding addresses this. + +## Time-varying conversion lifecycle + +[`redoCik.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/graph/redoCik.js) performs: + +1. Build the unit conversion graph. +2. Process suffix units. +3. Generate time-aligned conversion paths. +4. Replace `cik_vary` set-wise. +5. Rebuild the non-time-varying `cik` pairs from `cik_vary`. +6. Increment split-rebuild and group-cache revisions. + +Conversion regeneration now loads units, conversions, and all conversion +segments once before traversing source/destination paths. It performs five +metadata queries regardless of path count instead of querying conversion +segments inside every path edge. Relevant files: + +- [`createConversionArrays.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/graph/createConversionArrays.js) +- [`timeVaryingPathConversion.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/graph/timeVaryingPathConversion.js) +- [`ConversionSegment.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/ConversionSegment.js) +- [`CikVary.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/CikVary.js) + +`CikVary.insert()` marks derived data stale but does not rebuild it itself. A +later `refreshAllReadingViews()` detects the revision and rebuilds. The normal +client conversion workflow requests both conversion regeneration and reading +refresh, but an API caller can request them separately and leave a rebuild +pending intentionally. + +## Continuous aggregates + +### Meter hourly + +`meter_hourly_readings_unit_cagg` groups split rows by meter, graphic unit, +hour, unit representation, and seconds-in-rate. It computes duration-weighted +`reading_rate`, `min_rate`, and `max_rate`. + +### Meter daily + +`meter_daily_readings_unit_cagg` groups the split hypertable directly into day +buckets. It is a sibling of the hourly meter aggregate, not a child of it. + +Do not replace the weighted calculation with an average of hourly averages. A +correct hierarchical design must carry weighted sums and durations as well as +minimum and maximum state, including partial-hour behavior. + +### Group hourly and daily + +The group aggregates join the matching meter resolution to +`groups_deep_meters_cache` and `group_graphic_units_cache`, then sum compatible +meter rates: + +- Group hourly reads meter hourly. +- Group daily reads meter daily. + +Relational cache changes are not native continuous-aggregate invalidations, +so the application refreshes dirty caches before refreshing group aggregates. + +### Real-time mode + +All four continuous aggregates currently use: + +```sql +timescaledb.materialized_only = false +``` + +Queries can include recent unmaterialized source data. This improves freshness +but adds query-time work. Do not change to materialized-only mode until every +write path has reliable refresh coverage. + +## Refresh and rebuild orchestration + +The main entry point is +[`refreshAllReadingViews.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/refreshAllReadingViews.js). +It holds PostgreSQL advisory lock `724536221` so application refreshers do not +refresh the same aggregates concurrently. + +### Normal refresh + +When no rebuild is pending: + +1. Refresh meter hourly. +2. Refresh meter daily. +3. Refresh group caches only if their revision is dirty. +4. Refresh group hourly. +5. Refresh group daily. + +Bounded refreshes expand timestamps to full UTC bucket boundaries: + +- Meter/group hourly use hour boundaries. +- Meter/group daily use day boundaries. + +Passing neither bound refreshes all materialized ranges. Passing only one +bound throws an error. + +### Full rebuild + +A rebuild runs when either: + +- The caller passes `{ rebuild: true }`. +- `rebuild_revision > completed_rebuild_revision`. + +`rebuild_hourly_hypertable_split()` deletes and regenerates the entire split +hypertable from raw readings and current conversion/unit metadata. All four +continuous aggregates are then refreshed without bounds. + +The refresher records only the revision observed before work began. A newer +concurrent source change remains pending. + +### Rebuild invalidation + +| Source change | Split rebuild | Group-cache refresh | +|---|---:|---:| +| `meters.unit_id` changes | Yes | Yes | +| `units.unit_represent` changes | Yes | No | +| `units.sec_in_rate` changes | Yes | No | +| `cik`/`cik_vary` replacement | Yes | Yes | +| Group hierarchy changes | No | Yes | +| Direct group-meter changes | No | Yes | + +### Import behavior + +- CSV reading upload supplies its accepted range when refresh is requested. +- eGauge combines successful meter ranges and performs one bounded refresh. +- MAMAC acquisition does not refresh directly; deployment cron scripts run an + independent, currently unbounded refresh. +- Ordinary raw-reading imports do not recompute group dependency caches unless + their revision is dirty. + +## Commands and state inspection + +```bash +# Refresh CAGGs; a pending revision still causes an automatic full rebuild. +npm run refreshAllReadingViews + +# Force split regeneration and refresh every aggregate. +npm run rebuildAllReadingViews + +# Recalculate cik/cik_vary, then rebuild dependent reading data. +npm run updateCikAndViews +``` + +Inspect pending work: + +```sql +SELECT + rebuild_revision, + completed_rebuild_revision, + group_cache_revision, + completed_group_cache_revision +FROM reading_aggregate_state +WHERE id = 1; +``` + +Pending conditions: + +```text +rebuild_revision > completed_rebuild_revision +group_cache_revision > completed_group_cache_revision +``` + +Do not manually advance completed revisions unless repairing a verified state +problem. + +## Query integration and current performance work + +Active graph functions are installed from: + +- [`update_meter_line_readings_unit.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/update_meter_line_readings_unit.sql) +- [`update_group_line_readings_unit.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/update_group_line_readings_unit.sql) +- [`update_meter_group_bar.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/update_meter_group_bar.sql) +- [`update_function_get_compare_readings.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/update_function_get_compare_readings.sql) +- [`update_function_get_3d_readings.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/update_function_get_3d_readings.sql) + +Implemented optimizations include: + +- Set-based requested-meter processing in the meter line function. +- Direct bucket predicates for index use and chunk pruning. +- Indexed first-start and last-end lookups instead of full-history scans. +- Set-based multi-meter count queries. +- Batched raw-reading and `cik_vary` writes. +- Preloaded conversion metadata during conversion-path generation. +- Set-based `/api/groups` retrieval of all groups and deep meters. +- Independent aggregation of immediate group and meter children, avoiding a + child-meter by child-group Cartesian intermediate result. +- Reverse-key relationship indexes for parent lookups, recursive maintenance, + and foreign-key checks. +- `EXISTS` rather than `COUNT(*)` for group cycle detection. +- Time-first log indexing for bounded, ordered log retrieval. + +A synthetic comparison of the immediate-child query with 100 groups, each +having 100 meter and 100 group children, reduced execution from approximately +510 ms to 9.6 ms in the local container. This is development evidence, not a +production benchmark. + +## Important indexes + +| Index | Purpose | +|---|---| +| `readings` primary key `(meter_id, start_timestamp)` | Writes, raw range access, first-reading lookup | +| `readings_meter_end_timestamp_idx` | Latest reading-end lookup | +| `hypertable_hourly_split_meter_graphic_time_idx` | Meter/graphic-unit/time access | +| `hypertable_hourly_split_meter_time_idx` | Split uniqueness and maintenance | +| `cik_vary_source_time_idx` | Trigger lookup by source and overlapping interval | +| `groups_deep_meters_cache` primary key | Group-to-meter cache access | +| `groups_deep_meters_cache_meter_group_idx` | Meter-to-group aggregate joins | +| `groups_immediate_children_child_parent_idx` | Reverse group relationship lookup | +| `meters_immediate_children_child_parent_idx` | Reverse meter relationship lookup | +| `groups_immediate_meters_meter_group_idx` | Reverse group-meter lookup | +| Meter CAGG meter/graphic/bucket indexes | Meter graph range scans | +| Group CAGG group/graphic/bucket indexes | Group graph range scans | + +Check representative `EXPLAIN (ANALYZE, BUFFERS)` plans before adding more +indexes; each additional index increases write and maintenance cost. + +## Schema creation and deployment gap + +Fresh-schema creation in +[`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/database.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/database.js) installs +objects in this order: + +1. Shared reading helper functions. +2. Split hypertable, indexes, state, and triggers. +3. Group dependency caches. +4. Meter hourly CAGG. +5. Meter daily CAGG. +6. Group hourly CAGG. +7. Group daily CAGG. +8. Meter and group line functions. +9. Bar functions. +10. Compare functions. +11. 3-D functions. + +`CREATE ... IF NOT EXISTS` makes parts of fresh setup rerunnable, but it does +not replace an existing continuous aggregate definition. + +Critical deployment issue: + +- [`registerMigration.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/migrations/registerMigration.js) + currently requires a nonexistent `1.0.0-1.1.0` migration, so loading the + migration registry is expected to fail before migration begins. +- The repository contains `1.0.0-2.0.0` and `2.0.0-3.0.0` directories, but + neither is registered. +- A `2.0.0-3.0.0` directory exists but is not registered and does not install + the complete current TimescaleDB schema. +- `package.json` still reports application version `1.0.0`. + +Before production rollout, define the supported source version, create and +register a migration that installs or replaces every required object in a safe +dependency order, test it on a production-like copy, and document rollback or +recovery behavior. + +## Testing and validation + +Relevant coverage exists in: + +- [`readingTests.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/db/readingTests.js) +- [`unitReadingsTests.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/db/unitReadingsTests.js) +- [`compareTests.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/db/compareTests.js) +- [`cikVaryTests.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/db/cikVaryTests.js) +- [`groupTests.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/db/groupTests.js) +- Reading API suites under [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/web`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/web) + +Example focused commands: + +```bash +docker compose exec -T web npm run testsome -- \ + --timeout 30000 https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/db/cikVaryTests.js + +docker compose exec -T web npm run testsome -- \ + --timeout 30000 https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/test/web/readingsLineGroupQuantity.js +``` + +Do not carry forward old pass counts or benchmark ratios as current evidence. +The current review confirmed the focused `cik_vary` segment integration case, +reverse bidirectional conversion behavior, schema creation for the new indexes, +and the set-based group model query. The full database and web suites were not +revalidated as part of this document update. + +## Legacy and possible obsolete code + +The active fresh-schema path no longer creates legacy reading materialized +views, but compatibility code remains: + +- Deprecated helpers in [`Reading.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/Reading.js). +- Commented legacy setup and refresh calls. +- [`drop_legacy_reading_views.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB/drop_legacy_reading_views.sql), + whose schema-creation call remains disabled. +- Older SQL definitions retained for migration history and compatibility. + +Recent TODO markers identify code that appears unused, duplicated, superseded, +or broken but may have external consumers. These include the `currentDB` +accessor, an unreferenced `CikVary` point-in-time lookup, obsolete group null +cleanup, duplicate group SQL, and unused single-row CIK SQL. Research external +and downstream use before removal. + +## Known risks and next priorities + +### 1. Build the deployment migration + +This is the highest-priority release blocker. Fresh database success does not +prove an existing installation can be upgraded safely. + +### 2. Replace row-level split maintenance + +Bulk inserts still invoke conversion joins and `generate_series()` once per +reading. Evaluate statement-level transition-table triggers or an explicit +bulk split-generation path for INSERT, UPDATE, and DELETE. + +### 3. Fix suffix-unit asynchronous cleanup + +`removeAdditionalConversionsAndUnits()` uses `forEach(async ...)`, so the +function can return before conversion deletion and destination-unit updates +finish. Research and replace it with awaited set-based work or an awaited loop +before relying on conversion refresh ordering. + +### 4. Reduce conversion graph CPU work + +Database N+1 queries have been removed, but shortest-path search is still run +for each meter-unit/destination pair. One traversal per source or cached path +trees may reduce CPU for large unit graphs. + +### 5. Tune group cache maintenance + +Any dirty group revision currently recomputes the desired contents of both +group caches globally. Track affected groups if production group graphs make +this expensive. + +### 6. Evaluate a hierarchical daily aggregate + +Daily currently scans split rows directly. Any hierarchical replacement must +preserve weighted sums, durations, minimums, maximums, conversion boundaries, +and partial-hour behavior. + +### 7. Tune remaining graph functions + +Use production-like plans to evaluate: + +- Set-based 3-D meter processing. +- First/last bucket lookup in 3-D and bar helpers. +- Combined current/previous compare scans. +- Direct bucketing in place of compatible `generate_series()` joins. + +### 8. Configure TimescaleDB operational policies + +No explicit chunk interval, compression/columnstore policy, retention policy, +or CAGG refresh policy is installed. Base these settings on real ingestion +volume, late data, query windows, storage limits, and TimescaleDB version. + +### 9. Limit import concurrency + +Meter polling uses parallel network requests and database writes. A large fleet +can pressure the connection pool and per-row split trigger. Evaluate a +configurable concurrency limit. + +## Safe maintenance guidance + +- Prefer bounded refreshes after ordinary reading imports. +- Force a rebuild only for stale copied metadata, explicit recovery, or a + verified need. +- Preserve meter-before-group refresh ordering. +- Refresh dirty group caches before group aggregates. +- Do not average already averaged rates without carrying their weights. +- Preserve conversion-boundary splitting when changing ingestion. +- Do not manually advance revision counters casually. +- Test inserts, upserts, deletes, backfills, conversion changes, partial + buckets, empty groups, nested groups, repeated meters, and missing readings. +- Test migrations on a production-like database rather than only fresh schema + creation. + +## File map + +### Orchestration and models + +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/database.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/database.js): schema creation order. +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/TimeScaleDB/Reading.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/TimeScaleDB/Reading.js): CAGG creation and bounded refresh methods. +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/refreshAllReadingViews.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/refreshAllReadingViews.js): advisory lock and rebuild selection. +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/Reading.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/Reading.js): batched writes and graph-query wrappers. +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/CikVary.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/CikVary.js): set-based conversion replacement and revision invalidation. +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/graph`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/graph): graph creation and time-varying path generation. +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/Group.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/models/Group.js): group/cache queries. + +### Import paths + +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/routes/csv.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/routes/csv.js): optional bounded refresh after CSV upload. +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/eGauge/updateEgaugeMeters.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/eGauge/updateEgaugeMeters.js): combined bounded refresh. +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/updateMamacMeters.js`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/services/updateMamacMeters.js): acquisition without direct refresh. + +### SQL + +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/create_readings_table.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/create_readings_table.sql) +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/reading/TimeScaleDB) +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/group/create_groups_tables.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/group/create_groups_tables.sql) +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/group/get_all_children.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/group/get_all_children.sql) +- [`https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/group/get_all_groups_with_deep_meters.sql`](https://github.com/OpenEnergyDashboard/OED/tree/timeVary/src/server/sql/group/get_all_groups_with_deep_meters.sql) + +## Summary + +The branch has an integrated TimescaleDB implementation for fresh databases: +raw readings feed an hour- and conversion-aware split hypertable, four +real-time continuous aggregates serve meter and group queries, revision +counters coordinate derived-data maintenance, and recent work removed several +high-cardinality database round trips. + +The primary handoff concern is release engineering, not basic architecture: +the current migration registry cannot install this complete schema into an +existing deployment. Address migration coverage, then benchmark the per-row +split trigger and operational TimescaleDB settings with production-like data. diff --git a/timescaleDB/postgres17.md b/timescaleDB/History/postgres17.md similarity index 100% rename from timescaleDB/postgres17.md rename to timescaleDB/History/postgres17.md diff --git a/timescaleDB/History/timeVary2026SummerHandover.md b/timescaleDB/History/timeVary2026SummerHandover.md new file mode 100644 index 0000000..bfd325a --- /dev/null +++ b/timescaleDB/History/timeVary2026SummerHandover.md @@ -0,0 +1,1182 @@ +# Technical Handover Document + +## TimescaleDB Migration for Meter Reading Aggregation + +--- + +### Executive Summary + +This document provides a technical handover for the TimescaleDB migration completed as part of the meter reading aggregation optimization project. + +The objective of the project was to replace the existing PostgreSQL materialized-view-based reporting architecture with a TimescaleDB implementation using hypertables and continuous aggregates. + +The migration was driven by the increasing cost of refreshing PostgreSQL materialized views as historical meter data continued to grow. The previous implementation recalculated all of historical data during every refresh, resulting in long execution times and some unnecessary joins repeated across hourly and daily aggregations. + +The new implementation introduces: + +- A TimescaleDB hypertable that stores precomputed hourly reading slices. +- Continuous aggregates for incremental hourly and daily aggregation. +- Cache tables that replace recursive views and runtime functions required for group aggregation. +- Updated application initialization and refresh workflows. +- Benchmarking and validation tools to verify correctness against the legacy implementation. + +The migration successfully preserves analytical correctness while reducing aggregate refresh times by more than two orders of magnitude, even when the amount of historical data is modest. Extensive benchmarking demonstrated approximately 250× faster hourly refreshes and approximately 340× faster daily refreshes compared to the previous implementation. + +### Benchmark + +#### Configuration + +| config_key | config_value | description +|--------------------------|---------------------------------------|-------------------------------------------------------------------- +| baseline_end_date | 2021-12-31 23:59:59 | End of baseline dataset +| baseline_start_date | 2020-01-01 00:00:00 | Beginning of baseline dataset +| benchmark_meter_id | 32 | Meter ID used for benchmark execution +| daily_cagg | meter_daily_readings_unit_cagg | TimescaleDB daily continuous aggregate +| daily_materialized_view | meter_daily_readings_unit | Legacy PostgreSQL daily materialized view +| generate_random_seed | false | Deprecated: benchmark readings are deterministic for repeatability +| hourly_cagg | meter_hourly_readings_unit_cagg | TimescaleDB hourly continuous aggregate +| hourly_materialized_view | meter_hourly_readings_unit | Legacy PostgreSQL hourly materialized view +| hourly_split_table | hypertable_hourly_split | TimescaleDB hourly split hypertable +| hourly_split_trigger | trg_readings_update_hourly_hypertable | Trigger responsible for hourly split generation +| refresh_concurrent | false | Use concurrent materialized view refresh if supported +| run_query_tests | true | Enable query performance testing +| run_storage_tests | true | Enable storage measurements +| source_table | readings | Source readings table + +#### Baseline + +| object_name | object_type | row_count | min_timestamp | max_timestamp +|---------------------------------|------------------------|-----------|---------------------|--------------------- +| readings | SOURCE_TABLE | 70176 | 2020-01-01 00:00:00 | 2022-01-01 00:00:00 +| hypertable_hourly_split | TIMESCALE_HOURLY_SPLIT | 631584 | 2020-01-01 00:00:00 | 2022-01-01 00:00:00 +| meter_hourly_readings_unit_cagg | TIMESCALE_HOURLY_CAGG | 157896 | | +| meter_daily_readings_unit_cagg | TIMESCALE_DAILY_CAGG | 6579 | | + + Since group is calculated from meter hourly and daily, they are in milliseconds and comparing with legacy views seemed unnecessary. + +#### Parameters + +| config_key | config_value +|--------------------------|--------------------------------------- +| baseline_end_date | 2021-12-31 23:59:59 +| baseline_start_date | 2020-01-01 00:00:00 +| benchmark_meter_id | 32 +| daily_cagg | meter_daily_readings_unit_cagg +| daily_materialized_view | meter_daily_readings_unit +| generate_random_seed | false +| hourly_cagg | meter_hourly_readings_unit_cagg +| hourly_materialized_view | meter_hourly_readings_unit +| hourly_split_table | hypertable_hourly_split +| hourly_split_trigger | trg_readings_update_hourly_hypertable +| refresh_concurrent | false +| run_query_tests | true +| run_storage_tests | true +| source_table | readings + +#### Test Data + +| location_code | situation_description | size_code | scenario_name | start_time | end_time +|--------------------|--------------------------------------|-----------|----------------------------|---------------------|--------------------- +| AFTER_LAST_DATE | Append data after existing history | DAY | AFTER_LAST_DATE_DAY | 2022-01-01 00:00:00 | 2022-01-02 00:00:00 +| AFTER_LAST_DATE | Append data after existing history | MONTH | AFTER_LAST_DATE_MONTH | 2022-01-01 00:00:00 | 2022-02-01 00:00:00 +| AFTER_LAST_DATE | Append data after existing history | WEEK | AFTER_LAST_DATE_WEEK | 2022-01-01 00:00:00 | 2022-01-08 00:00:00 +| AFTER_LAST_DATE | Append data after existing history | YEAR | AFTER_LAST_DATE_YEAR | 2022-01-01 00:00:00 | 2023-01-01 00:00:00 +| BEFORE_FIRST_DATE | Insert data before existing history | DAY | BEFORE_FIRST_DATE_DAY | 2019-01-01 00:00:00 | 2019-01-02 00:00:00 +| BEFORE_FIRST_DATE | Insert data before existing history | MONTH | BEFORE_FIRST_DATE_MONTH | 2019-01-01 00:00:00 | 2019-02-01 00:00:00 +| BEFORE_FIRST_DATE | Insert data before existing history | WEEK | BEFORE_FIRST_DATE_WEEK | 2019-01-01 00:00:00 | 2019-01-08 00:00:00 +| BEFORE_FIRST_DATE | Insert data before existing history | YEAR | BEFORE_FIRST_DATE_YEAR | 2019-01-01 00:00:00 | 2020-01-01 00:00:00 +| MIDDLE_REPLACEMENT | Replace data inside existing history | MONTH | MIDDLE_REPLACEMENT_ MONTH | 2020-06-01 00:00:00 | 2020-07-01 00:00:00 + +| scenario_name | situation | date_range | source_rows +|----------------------------|--------------------|------------|------------- +| AFTER_LAST_DATE_ DAY | AFTER_LAST_DATE | DAY | 24 +| AFTER_LAST_DATE_MONTH | AFTER_LAST_DATE | MONTH | 744 +| AFTER_LAST_DATE_WEEK | AFTER_LAST_DATE | WEEK | 168 +| AFTER_LAST_DATE_YEAR | AFTER_LAST_DATE | YEAR | 8760 +| BEFORE_FIRST_DATE_DAY | BEFORE_FIRST_DATE | DAY | 24 +| BEFORE_FIRST_DATE_MONTH | BEFORE_FIRST_DATE | MONTH | 744 +| BEFORE_FIRST_DATE_WEEK | BEFORE_FIRST_DATE | WEEK | 168 +| BEFORE_FIRST_DATE_YEAR | BEFORE_FIRST_DATE | YEAR | 8760 +| MIDDLE_REPLACEMENT_MONTH | MIDDLE_REPLACEMENT | MONTH | 720 + +#### Insert + +| scenario_name | situation | date_range | implementation | source_rows | hourly_split_rows | duration_ms | rows_per_second +|---------------------------|-------------------|------------|----------------|-------------|-------------------|-------------|----------------- +| AFTER_LAST_DATE_DAY | AFTER_LAST_DATE | DAY | LEGACY | 24 | 0 | 2.587 | 9277.155 +| AFTER_LAST_DATE_DAY | AFTER_LAST_DATE | DAY | TIMESCALE | 24 | 216 | 31.047 | 773.022 +| AFTER_LAST_DATE_WEEK | AFTER_LAST_DATE | WEEK | LEGACY | 168 | 0 | 6.562 | 25601.951 +| AFTER_LAST_DATE_WEEK | AFTER_LAST_DATE | WEEK | TIMESCALE | 168 | 1512 | 69.547 | 2415.633 +| AFTER_LAST_DATE_MONTH | AFTER_LAST_DATE | MONTH | LEGACY | 744 | 0 | 24.533 | 30326.499 +| AFTER_LAST_DATE_MONTH | AFTER_LAST_DATE | MONTH | TIMESCALE | 744 | 6696 | 298.165 | 2495.263 +| AFTER_LAST_DATE_YEAR | AFTER_LAST_DATE | YEAR | LEGACY | 8760 | 0 | 303.218 | 28890.105 +| AFTER_LAST_DATE_YEAR | AFTER_LAST_DATE | YEAR | TIMESCALE | 8760 | 78840 | 3671.501 | 2385.945 +| BEFORE_FIRST_DATE_DAY | BEFORE_FIRST_DATE | DAY | LEGACY | 24 | 0 | 2.294 | 10462.075 +| BEFORE_FIRST_DATE_DAY | BEFORE_FIRST_DATE | DAY | TIMESCALE | 24 | 216 | 14.191 | 1691.213 +| BEFORE_FIRST_DATE_WEEK | BEFORE_FIRST_DATE | WEEK | LEGACY | 168 | 0 | 6.791 | 24738.625 +| BEFORE_FIRST_DATE_WEEK | BEFORE_FIRST_DATE | WEEK | TIMESCALE | 168 | 1512 | 65.123 | 2579.734 +| BEFORE_FIRST_DATE_MONTH | BEFORE_FIRST_DATE | MONTH | LEGACY | 744 | 0 | 23.744 | 31334.232 +| BEFORE_FIRST_DATE_MONTH | BEFORE_FIRST_DATE | MONTH | TIMESCALE | 744 | 6696 | 306.256 | 2429.340 +| BEFORE_FIRST_DATE_YEAR | BEFORE_FIRST_DATE | YEAR | LEGACY | 8760 | 0 | 288.164 | 30399.356 +| BEFORE_FIRST_DATE_YEAR | BEFORE_FIRST_DATE | YEAR | TIMESCALE | 8760 | 78840 | 3787.205 | 2313.051 + +![After Last Date Chart](./Images/after_insert_meter_duration.png) ![Before First Date Chart](./Images/before_insert_meter_duration.png) + +The numbers before the labels on the graph has no value other than to align the graph better. The alignment would not be correct without the numbers prefixed. Example the spelling of Week starts with "W" and the spelling of Month starts with "M", if ordered by letters, month will be displayed before week, therefore, the numbers appear before the duration. + +#### Refresh: + +| scenario_name | situation | date_range | aggregation_level | implementation | rows_affected | duration_ms | rows_per_second +|---------------------------|-------------------|------------|-------------------|----------------|---------------|-------------|----------------- +| AFTER_LAST_DATE_DAY | AFTER_LAST_DATE | DAY | DAILY | LEGACY | 9 | 5094.777 | 1.767 +| AFTER_LAST_DATE_DAY | AFTER_LAST_DATE | DAY | DAILY | TIMESCALE | 9 | 15.242 | 590.474 +| AFTER_LAST_DATE_WEEK | AFTER_LAST_DATE | WEEK | DAILY | LEGACY | 63 | 4686.750 | 13.442 +| AFTER_LAST_DATE_WEEK | AFTER_LAST_DATE | WEEK | DAILY | TIMESCALE | 63 | 11.969 | 5263.598 +| AFTER_LAST_DATE_MONTH | AFTER_LAST_DATE | MONTH | DAILY | LEGACY | 279 | 4940.009 | 56.478 +| AFTER_LAST_DATE_MONTH | AFTER_LAST_DATE | MONTH | DAILY | TIMESCALE | 279 | 22.093 | 12628.434 +| AFTER_LAST_DATE_YEAR | AFTER_LAST_DATE | YEAR | DAILY | LEGACY | 3285 | 5158.685 | 636.790 +| AFTER_LAST_DATE_YEAR | AFTER_LAST_DATE | YEAR | DAILY | TIMESCALE | 3285 | 88.837 | 36977.836 +| BEFORE_FIRST_DATE_DAY | BEFORE_FIRST_DATE | DAY | DAILY | LEGACY | 9 | 4699.987 | 1.915 +| BEFORE_FIRST_DATE_DAY | BEFORE_FIRST_DATE | DAY | DAILY | TIMESCALE | 9 | 38.082 | 236.332 +| BEFORE_FIRST_DATE_WEEK | BEFORE_FIRST_DATE | WEEK | DAILY | LEGACY | 63 | 4746.455 | 13.273 +| BEFORE_FIRST_DATE_WEEK | BEFORE_FIRST_DATE | WEEK | DAILY | TIMESCALE | 63 | 28.580 | 2204.339 +| BEFORE_FIRST_DATE_MONTH | BEFORE_FIRST_DATE | MONTH | DAILY | LEGACY | 279 | 4952.453 | 56.336 +| BEFORE_FIRST_DATE_MONTH | BEFORE_FIRST_DATE | MONTH | DAILY | TIMESCALE | 279 | 19.033 | 14658.751 +| BEFORE_FIRST_DATE_YEAR | BEFORE_FIRST_DATE | YEAR | DAILY | LEGACY | 3285 | 5028.000 | 653.341 +| BEFORE_FIRST_DATE_YEAR | BEFORE_FIRST_DATE | YEAR | DAILY | TIMESCALE | 3285 | 71.498 | 45945.341 +| AFTER_LAST_DATE_DAY | AFTER_LAST_DATE | DAY | HOURLY | LEGACY | 216 | 8427.066 | 25.632 +| AFTER_LAST_DATE_DAY | AFTER_LAST_DATE | DAY | HOURLY | TIMESCALE | 216 | 33.005 | 6544.463 +| AFTER_LAST_DATE_WEEK | AFTER_LAST_DATE | WEEK | HOURLY | LEGACY | 1512 | 8232.224 | 183.668 +| AFTER_LAST_DATE_WEEK | AFTER_LAST_DATE | WEEK | HOURLY | TIMESCALE | 1512 | 59.630 | 25356.364 +| AFTER_LAST_DATE_MONTH | AFTER_LAST_DATE | MONTH | HOURLY | LEGACY | 6696 | 8443.237 | 793.061 +| AFTER_LAST_DATE_MONTH | AFTER_LAST_DATE | MONTH | HOURLY | TIMESCALE | 6696 | 145.218 | 46109.986 +| AFTER_LAST_DATE_YEAR | AFTER_LAST_DATE | YEAR | HOURLY | LEGACY | 78840 | 8688.681 | 9073.874 +| AFTER_LAST_DATE_YEAR | AFTER_LAST_DATE | YEAR | HOURLY | TIMESCALE | 78840 | 1020.274 | 77273.360 +| BEFORE_FIRST_DATE_DAY | BEFORE_FIRST_DATE | DAY | HOURLY | LEGACY | 216 | 8076.936 | 26.743 +| BEFORE_FIRST_DATE_DAY | BEFORE_FIRST_DATE | DAY | HOURLY | TIMESCALE | 216 | 67.495 | 3200.237 +| BEFORE_FIRST_DATE_WEEK | BEFORE_FIRST_DATE | WEEK | HOURLY | LEGACY | 1512 | 8114.325 | 186.337 +| BEFORE_FIRST_DATE_WEEK | BEFORE_FIRST_DATE | WEEK | HOURLY | TIMESCALE | 1512 | 62.265 | 24283.305 +| BEFORE_FIRST_DATE_MONTH | BEFORE_FIRST_DATE | MONTH | HOURLY | LEGACY | 6696 | 8491.706 | 788.534 +| BEFORE_FIRST_DATE_MONTH | BEFORE_FIRST_DATE | MONTH | HOURLY | TIMESCALE | 6696 | 137.100 | 48840.263 +| BEFORE_FIRST_DATE_YEAR | BEFORE_FIRST_DATE | YEAR | HOURLY | LEGACY | 78840 | 8327.288 | 9467.668 +| BEFORE_FIRST_DATE_YEAR | BEFORE_FIRST_DATE | YEAR | HOURLY | TIMESCALE | 78840 | 1140.075 | 69153.345 + +![After Last Date Chart](./Images/after_refresh_meter_hourly.png) ![After Last Date Chart](./Images/after_refresh_meter_daily.png) +![Before First Date Chart](./Images/before_refresh_meter_hourly.png) ![Before First Date Chart](./Images/before_refresh_meter_daily.png) + +The numbers before the labels on the graph has no value other than to align the graph better. The alignment would not be correct without the numbers prefixed. Example the spelling of Week starts with "W" and the spelling of Month starts with "M", if ordered by letters, month will be displayed before week, therefore, the numbers appear before the duration. + +#### Mismatch: + +| aggregation_level | legacy_rows | timescale_rows | mismatch_rows | tolerance | passed +|-------------------|-------------|----------------|---------------|-----------|-------- +| DAILY | 6579 | 6579 | 0 | 1e-11 | t +| HOURLY | 157896 | 157896 | 0 | 1e-11 | t + + +#### Fair Comparison +At this point it is unfair to just compare the refresh data. Since TimeScaleDB uses significant time to insert data, and overall comparison seems fair. + +![After Last Date Chart](./Images/after_insert_refresh_meter_hourly.png) ![Before Last Date Chart](./Images/before_insert_refresh_meter_hourly.png) +![After Last Date Chart](./Images/after_insert_refresh_meter_daily.png) ![Before Last Date Chart](./Images/before_insert_refresh_meter_daily.png) + +The numbers before the labels on the graph has no value other than to align the graph better. The alignment would not be correct without the numbers prefixed. Example the spelling of Week starts with "W" and the spelling of Month starts with "M", if ordered by letters, month will be displayed before week, therefore, the numbers appear before the duration. + +The implementation is functionally complete and ready for continued development. + +--- + +# 1. Background + +## Existing Architecture + +Prior to this project, reporting was performed entirely using PostgreSQL materialized views. Below the readings were a regular table and the rest were materialized views. Note this is based on work done in the timeVary branch so the development branch did not have the group views, dealing with conversions in views and some other changes. + +``` +readings + │ + ▼ +meter_hourly_readings_unit + │ + ▼ +meter_daily_readings_unit + │ + ▼ +group_hourly_readings_unit + │ + ▼ +group_daily_readings_unit +``` + +These materialized views were responsible for: + +- Splitting readings across hourly boundaries +- Applying unit conversions +- Handling time-varying conversion factors +- Aggregating readings into hourly values +- Rolling hourly values into daily values +- Aggregating meters into groups + +Although functionally correct, the design had several limitations. + +### Expensive Refreshes + +Materialized views refreshed by recomputing all of historical data. + +As the database grew, refresh times increased proportionally. + +### Duplicate Work + +Before the previous time-varying work, where this was originally addressed, hourly and daily materialized views independently repeated much of the same aggregation logic. + +### Runtime Conversion Overhead + +Every refresh repeatedly joined against conversion tables and recalculated overlap durations. + +### Group Aggregation Limitations + +Group aggregation depended on recursive views and runtime helper functions that are incompatible with TimescaleDB continuous aggregates. + +# 2. Project Objectives + +The migration was designed around four primary objectives. + +## Performance + +Replace expensive full refreshes with TimescaleDB's incremental aggregation model. + +## Correctness + +Maintain identical analytical results compared with the existing PostgreSQL implementation. + +## Scalability + +Support significantly larger datasets without proportional increases in refresh time. + +## Maintainability + +Move expensive calculations into predictable preprocessing and refresh stages rather than executing them repeatedly during aggregation. + +# 3. Final Architecture + +The implemented architecture is shown below. + +``` + readings + │ + Trigger + │ + ▼ + hypertable_hourly_split + │ + ┌──────────────┴──────────────┐ + ▼ ▼ +meter_hourly_readings_unit_cagg meter_daily_readings_unit_cagg + │ │ + ▼ ▼ +group_hourly_readings_unit_cagg group_daily_readings_unit_cagg +``` + +The intended aggregation hierarchy is: + +1. Convert raw readings into hourly slices. +2. Aggregate hourly slices into meter hourly values. +3. Roll hourly values into daily values. +4. Aggregate meter values into group values. + +This layered design minimizes repeated calculations and enables TimescaleDB to perform incremental refreshes efficiently. + +# 4. Database Components + +## 4.1 hypertable_hourly_split + +File: + +``` +TimeScaleDB/create_prerequisites.sql +``` + +### Purpose + +The hypertable stores hourly reading slices derived from the raw readings table. + +Each row contains: + +- hourly overlap duration +- weighted reading contribution +- unit metadata +- conversion metadata +- graphic unit information + +Previously this information was calculated every time aggregation occurred. + +The new implementation performs the calculation once during ingestion and stores the results for reuse. + +The hypertable includes sec_in_rate and unit_represents columns. Since this code was based on the work of previous cohorts, these columns have been kept in place. If these two columns are not used in future, they must be removed from create statement and the trigger function must be adjusted accordingly. + + +### Benefits + +- Eliminates repeated hourly splitting. +- Eliminates repeated conversion joins. +- Provides a stable source for continuous aggregates. +- Improves refresh performance. + +--- + +## 4.2 Meter Hourly Continuous Aggregate + +File: + +``` +TimeScaleDB/create_hourly_readings.sql +``` + +Created object: + +``` +meter_hourly_readings_unit_cagg +``` + +This replaces: + +``` +meter_hourly_readings_unit +``` + +Responsibilities include: + +- hourly bucketing +- weighted aggregation +- minimum values +- maximum values +- unit conversion + +Because the hourly split hypertable already contains precomputed overlap information, expensive calculations are not repeated. + +## 4.3 Meter Daily Continuous Aggregate + +File: + +``` +TimeScaleDB/create_daily_readings.sql +``` + +Created object: + +``` +meter_daily_readings_unit_cagg +``` + +The current implementation aggregates directly from: + +``` +hypertable_hourly_split +``` + +rather than from the hourly continuous aggregate. + +While this maintains correctness, it does not yet realize the full benefits of hierarchical aggregation. + +A future implementation should expose rollup state from the hourly aggregate (weighted sums, durations, minimums, and maximums) so that daily aggregates can be computed by summing intermediate states rather than reprocessing raw hourly slices. Since the daily meter values are used by daily group view and the buckets are hourly, the bucket could be lost and TimeScaleDB need the underlying hourly bucket on the hypertable to track changes, tehrefore this design was chosen. + +Care must be taken not to average hourly averages, as this would produce incorrect weighted results. There was an instance where not having a better understanding of how daily rates were calculated for KWh, incorrect result was discovered during materialized view and TimeScaleDB. This may nhot be a problem, but if you find the values are different, this is a good starting point. + +--- + +## 4.4 Group Aggregation + +### Challenge + +The legacy implementation depended on: + +- recursive views +- PL/pgSQL helper functions + +Examples include: + +- groups_deep_meters +- get_graphic_unit() + +Continuous aggregates cannot depend on these objects. + +### Solution + +The project introduced cache tables that are refreshed before group aggregate refreshes. + +``` +groups_immediate_children + │ + ▼ +groups_deep_children + │ + ▼ +groups_deep_meters_cache + │ + ▼ +group_graphic_units_cache + │ + ▼ +group continuous aggregates +``` + +Implemented components include: + +- cache tables +- cache refresh procedures +- hourly group continuous aggregates +- daily group continuous aggregates + +This removes runtime recursion while remaining compatible with TimescaleDB. + +# 5. Application Integration + +The application initialization workflow was updated to create and maintain the new TimescaleDB objects. + +Implemented in: + +``` +TimeScaleDB/Reading.js +``` + +Functions added include: + +- createPrerequisites() +- createGroupDependencies() +- createHourlyReadings() +- createDailyReadings() +- createGroupHourlyReadings() +- createGroupDailyReadings() + +Existing query functions were updated to reference the new continuous aggregates. + +Meter queries now use: + +- meter_hourly_readings_unit_cagg +- meter_daily_readings_unit_cagg + +Group queries now use: + +- group_hourly_readings_unit_cagg +- group_daily_readings_unit_cagg + +--- + +# 6. Refresh Workflow + +The implemented refresh order is: + +1. Rebuild hypertable_hourly_split (when required, changes to cik and cik_vary data). +2. Refresh hourly continuous aggregate. +3. Refresh daily continuous aggregate. +4. Refresh group dependency caches. +5. Refresh hourly group aggregate. +6. Refresh daily group aggregate. + +This dependency order must be preserved because each stage depends on results produced by previous stages. + +# 7. Testing and Validation + +A comprehensive validation suite was created to compare the TimescaleDB implementation against the last time-vary version that had already modified/added views and was moderately tested to verify it is correct. + +Comparison scripts were created for: + +- meter hourly +- meter daily +- group hourly +- group daily + +Validation confirmed: + +| Aggregate | Legacy | TimescaleDB | Mismatches | +|-----------|---------|-------------|------------| +| Hourly | 157,896 | 157,896 | 0 | +| Daily | 6,579 | 6,579 | 0 | + +Comparison tolerance: 10e-11 + +The following values were validated: + +- reading_rate +- minimum +- maximum +- timestamps +- missing rows + +No analytical differences were detected. + +``` +/* + * 1. Compare reading_rate values. + * + * This compares the primary hourly aggregation result between: + * + * Original: + * meter_hourly_readings_unit + * + * TimescaleDB: + * meter_hourly_readings_unit_cagg + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.meter_id, + LOWER(mv.time_interval) AS mv_time, + mv.reading_rate AS mv_reading_rate, + cagg.bucket AS cagg_time, + cagg.reading_rate AS cagg_reading_rate, + (COALESCE(mv.reading_rate, 0) - COALESCE(cagg.reading_rate)) AS difference +FROM meter_hourly_readings_unit AS mv INNER JOIN + meter_hourly_readings_unit_cagg AS cagg + ON mv.meter_id = cagg.meter_id + AND lower(mv.time_interval) = cagg.bucket + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + ABS(COALESCE(mv.reading_rate) - COALESCE(cagg.reading_rate)) DESC +LIMIT 20; + + +/* + * 2. Compare max_rate and min_rate values. + * + * This verifies that the continuous aggregate preserves the same minimum and + * maximum hourly rates as the original materialized view. + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.meter_id, + LOWER(mv.time_interval) AS mv_time, + mv.max_rate AS mv_max_rate, + cagg.max_rate AS cagg_max_rate, + (COALESCE(mv.max_rate) - COALESCE(cagg.max_rate)) AS max_difference, + mv.min_rate AS mv_min_rate, + cagg.min_rate AS cagg_min_rate, + (COALESCE(mv.min_rate) - COALESCE(cagg.min_rate)) AS min_difference +FROM meter_hourly_readings_unit AS mv INNER JOIN + meter_hourly_readings_unit_cagg AS cagg + ON mv.meter_id = cagg.meter_id + AND lower(mv.time_interval) = cagg.bucket + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + GREATEST( + ABS(COALESCE(mv.max_rate, 0) - COALESCE(cagg.max_rate, 0)), + ABS(COALESCE(mv.min_rate, 0) - COALESCE(cagg.min_rate, 0)) + ) DESC +LIMIT 20; + +/* + * 3. Compare reading_rate values. + * + * This compares the primary hourly aggregation result between: + * + * Original: + * meter_daily_readings_unit + * + * TimescaleDB: + * meter_daily_readings_unit_cagg + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.meter_id, + LOWER(mv.time_interval) AS mv_time, + mv.reading_rate AS mv_reading_rate, + LOWER(cagg.time_interval) AS cagg_time, + cagg.reading_rate AS cagg_reading_rate, + (COALESCE(mv.reading_rate, 0) - COALESCE(cagg.reading_rate, 0)) AS difference +FROM meter_daily_readings_unit AS mv LEFT JOIN + meter_daily_readings_unit_cagg AS cagg + ON mv.meter_id = cagg.meter_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + ABS(COALESCE(mv.reading_rate, 0) - COALESCE(cagg.reading_rate, 0)) DESC +LIMIT 20; + + +/* + * 4. Compare max_rate and min_rate values. + * + * This verifies that the continuous aggregate preserves the same minimum and + * maximum hourly rates as the original materialized view. + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.meter_id, + LOWER(mv.time_interval) AS mv_time_start, + LOWER(cagg.time_interval) AS cagg_time_start, + UPPER(mv.time_interval) AS mv_time_end, + UPPER(cagg.time_interval) AS cagg_time_end, + mv.max_rate AS mv_max_rate, + cagg.max_rate AS cagg_max_rate, + (COALESCE(mv.max_rate, 0) - COALESCE(cagg.max_rate, 0)) AS max_difference, + mv.min_rate AS mv_min_rate, + cagg.min_rate AS cagg_min_rate, + (COALESCE(mv.min_rate, 0) - COALESCE(cagg.min_rate, 0)) AS min_difference +FROM meter_daily_readings_unit AS mv LEFT JOIN + meter_daily_readings_unit_cagg AS cagg + ON mv.meter_id = cagg.meter_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + GREATEST( + ABS(COALESCE(mv.max_rate, 0) - COALESCE(cagg.max_rate, 0)), + ABS(COALESCE(mv.min_rate, 0) - COALESCE(cagg.min_rate, 0)) + ) DESC +LIMIT 20; + +/* + * 5. Compare reading_rate values. + * + * This compares the primary hourly aggregation result between: + * + * Original: + * group_hourly_readings_unit + * + * TimescaleDB: + * group_hourly_readings_unit_cagg + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.group_id, + LOWER(mv.time_interval) AS mv_time, + mv.reading_rate AS mv_reading_rate, + LOWER(cagg.time_interval) AS cagg_time, + cagg.reading_rate AS cagg_reading_rate, + ( + COALESCE(mv.reading_rate, 0) + - + COALESCE(cagg.reading_rate, 0) + ) AS difference +FROM group_hourly_readings_unit AS mv INNER JOIN + group_hourly_readings_unit_cagg AS cagg + ON mv.group_id = cagg.group_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + ABS( + COALESCE(mv.reading_rate, 0) + - + COALESCE(cagg.reading_rate, 0) + ) DESC +LIMIT 20; + +/* + * 6. Identify rows missing from the TimescaleDB continuous aggregate. + * + * This catches cases where the continuous aggregate does not contain a + * corresponding group/hour/unit combination. + */ +SELECT + mv.group_id, + mv.time_interval, + mv.graphic_unit_id, + mv.reading_rate +FROM group_hourly_readings_unit AS mv LEFT JOIN + group_hourly_readings_unit_cagg AS cagg + ON mv.group_id = cagg.group_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id +WHERE cagg.group_id IS NULL +ORDER BY mv.group_id, mv.time_interval +LIMIT 20; + +/* + * 1. Compare reading_rate values. + * + * This compares the primary daily aggregation result between: + * + * Original: + * group_daily_readings_unit + * + * TimescaleDB: + * group_daily_readings_unit_cagg + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.group_id, + LOWER(mv.time_interval) AS mv_time, + LOWER(cagg.time_interval) AS cagg_time, + mv.reading_rate AS mv_reading_rate, + cagg.reading_rate AS cagg_reading_rate, + ( + COALESCE(mv.reading_rate, 0) + - + COALESCE(cagg.reading_rate, 0) + ) AS difference +FROM group_daily_readings_unit AS mv LEFT JOIN + group_daily_readings_unit_cagg AS cagg + ON mv.group_id = cagg.group_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + ABS( + COALESCE(mv.reading_rate, 0) + - + COALESCE(cagg.reading_rate, 0) + ) DESC +LIMIT 20; + +/* + * 2. Compare row counts. + * + * This identifies missing or additional daily buckets between the two + * implementations. + */ +SELECT + COUNT(*) AS total_rows, + COUNT(cagg.group_id) AS matching_cagg_rows, + COUNT(*) - COUNT(cagg.group_id) AS missing_cagg_rows +FROM group_daily_readings_unit AS mv LEFT JOIN + group_daily_readings_unit_cagg AS cagg + ON mv.group_id = cagg.group_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id; + +/* + * 3. Display missing TimescaleDB rows. + * + * These rows exist in the original materialized view but not in the + * continuous aggregate. + */ +SELECT + mv.group_id, + LOWER(mv.time_interval) AS mv_time, + UPPER(mv.time_interval) AS mv_time_end, + mv.graphic_unit_id, + mv.reading_rate +FROM group_daily_readings_unit AS mv LEFT JOIN + group_daily_readings_unit_cagg AS cagg + ON mv.group_id = cagg.group_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id +WHERE cagg.group_id IS NULL +ORDER BY + mv.group_id, + mv.time_interval +LIMIT 20; +``` + +# 8. Benchmark Results + +## Hourly Refresh + +Legacy: ~8.4 seconds + +TimescaleDB: ~33 milliseconds + +Approximately **250× faster**. Review benchmark details above for details. + +## Daily Refresh + +Legacy: ~5.1 seconds + +TimescaleDB: ~15 milliseconds + +Approximately **340× faster**. Review benchmark details above for details. + +# 9. Storage Considerations + +|object_name | object_type | size_bytes | size_pretty +|----------------------------------------------------------|-------------------------|------------|------------- +| hypertable_hourly_split | HOURLY_SPLIT_HYPERTABLE | 1360068608 | 1297 MB +| public.hypertable_hourly_split chunks | TIMESCALE_CHUNKS | 1360052224 | 1297 MB +| meter_hourly_readings_unit_cagg | TIMESCALE_HOURLY_CAGG | 229097472 | 218 MB +| _timescaledb_internal._materialized_hypertable_12 chunks | TIMESCALE_CHUNKS | 229056512 | 218 MB +| meter_hourly_readings_unit | LEGACY_HOURLY_MV | 140754944 | 134 MB +| readings | SOURCE_TABLE | 34250752 | 33 MB +| meter_daily_readings_unit_cagg | TIMESCALE_DAILY_CAGG | 13082624 | 12 MB +| _timescaledb_internal._materialized_hypertable_13 chunks | TIMESCALE_CHUNKS | 13041664 | 12 MB +| meter_daily_readings_unit | LEGACY_DAILY_MV | 5611520 | 5480 kB + +The storage increase is primarily due to the hourly split hypertable. + +This is expected because each reading is decomposed into hourly slices before aggregation. Since some of the meter readings are in minutes, hourly buckets are expected to be lesser, therefore, requiring lesser storage. But the meta data, and other checks - such as change in bucket and tracking dirty bits per bucket - the storage size increases. + +The additional storage represents an intentional trade-off that enables: + +- dramatically faster refreshes +- reduced runtime computation +- improved scalability + +--- + +# 10. Files Added and Modified + +## SQL + +- [create_prerequisites.sql](./sqlScripts/implementation/create_prerequisites.sql) +- [create_group_dependencies.sql](./sqlScripts/implementation/create_group_dependencies.sql) +- [create_hourly_readings.sql](./sqlScripts/implementation/create_hourly_readings.sql) +- [create_daily_readings.sql](./sqlScripts/implementation/create_daily_readings.sql) +- [create_group_hourly_readings.sql](./sqlScripts/implementation/create_group_hourly_readings.sql) +- [create_group_daily_readings.sql](./sqlScripts/implementation/create_group_daily_readings.sql) +- [update_function_get_3d_readings.sql](./sqlScripts/implementation/update_function_get_3d_readings.sql) +- [update_function_get_compare_readings.sql](./sqlScripts/implementation/update_function_get_compare_readings.sql) +- [update_group_line_readings_unit.sql](./sqlScripts/implementation/update_group_line_readings_unit.sql) +- [update_meter_group_bar.sql](./sqlScripts/implementation/update_meter_group_bar.sql) +- [update_meter_line_readings_unit.sql](./sqlScripts/implementation/update_meter_line_readings_unit.sql) +- [update_function_gupdate_reading_viewset_3d_readings.sql](./sqlScripts/implementation/update_reading_views.sql) +- [drop_legacy_reading_views.sql](./sqlScripts/implementation/drop_legacy_reading_views.sql) +- [CompareHourlyReadings.sql](./sqlScripts/compare/CompareHourlyReadings.sql) +- [CompareDailyReadings.sql](./sqlScripts/compare/CompareDailyReadings.sql) +- [CompareGroupHourlyReadings.sql](./sqlScripts/compare/CompareGroupHourlyReadings.sql) +- [CompareGroupDailyReadings.sql](./sqlScripts/compare/CompareGroupDailyReadings.sql) + +## Application + +``` +TimeScaleDB/Reading.js +``` + +Responsible for: + +- object creation +- refresh workflow +- rebuild workflow +- integration with database setup + +# 11. Known Limitations + +The migration is complete but several areas remain for future optimization. + +The current daily aggregate still processes the hourly split hypertable directly rather than rolling up hourly aggregate state. + +Full hypertable rebuilds are still used more frequently than necessary. + +Group cache tables are refreshed during every refresh cycle regardless of whether metadata has changed. + +No explicit chunk interval has been configured. + +Historical data compression has not yet been evaluated. + +# 12. Future Work and Optimization Opportunities + +## 12.1 Batch Reading Ingestion + +The current implementation inserts readings sequentially. + +Each inserted row triggers the hourly splitting logic independently. + +The trigger performs joins against conversion tables and generates hourly slices using `generate_series()`. + +Future work should investigate: + +- multi-row INSERT statements +- COPY ingestion +- statement-level triggers using transition tables +- set-based generation of hourly slices + +These approaches would significantly reduce ingestion overhead. + +## 12.2 Avoid Full Hypertable Rebuilds + +The refresh workflow currently rebuilds the entire split hypertable whenever no refresh window is supplied. + +This process: + +- deletes every split row +- regenerates every split row +- refreshes every continuous aggregate + +Normal reading imports already maintain the hypertable through triggers. + +Full rebuilds should therefore be reserved for: + +- conversion changes +- meter-unit changes + +Ordinary refreshes should instead use bounded refresh windows or TimescaleDB refresh policies. I initially thought that making unbounded refreshes was inexpensive, but after carefully reviewing the TimeScaleDB document, as the buckets count increases, the surface area of checks increases. This is an opportunity that could be further investigated. For more details, links to useful resources are in the "Recommended References and Further Reading" section below. + +## 12.3 Build Daily Aggregates from Hourly Aggregates + +The current daily aggregate still processes the hourly split hypertable directly. + +A true hierarchical implementation would expose rollup state from the hourly aggregate including: + +- weighted sums +- durations +- minimum values +- maximum values + +Daily aggregation could then process significantly fewer rows while maintaining correctness. For more information see section 4.3 above, under hypertable_hourly_split. + +## 12.4 Improve Time Predicate Efficiency + +Several graph functions create temporary `tsrange` values when filtering buckets. + +Direct comparisons against the bucket timestamp are more likely to enable: + +- chunk pruning +- index range scans + +Some queries also call `time_bucket()` on values that are already bucketed. + +Removing this unnecessary computation should improve query performance. + +## 12.5 Refresh Group Dependency Caches Only When Required + +Group cache tables are refreshed during every aggregate refresh. + +Normal reading imports do not modify (this needs verification and further research): + +- group membership +- conversion compatibility +- meter metadata + +Cache refreshes should therefore occur only when metadata changes. + +The cache refresh procedures also recompute identical recursive queries multiple times. + +Materializing these intermediate results once would reduce unnecessary work. + +Additional indexes such as: + +``` +(meter_id, group_id) +``` + +should also be benchmarked. + +## 12.6 Optimize Conversion Lookups + +The ingestion trigger repeatedly searches the conversion table using overlapping time ranges. + +The existing primary key is not optimized for this access pattern. + +Future benchmarking should evaluate: + +- B-tree indexes +- GiST indexes +- range indexes + +to improve ingestion performance. + +## 12.7 Tune Chunk Size and Historical Storage + +The split hypertable currently uses TimescaleDB's default chunk interval. + +Future work should determine an optimal chunk size based on: + +- ingestion rate +- available memory +- workload characteristics + +Historical chunks may also benefit from TimescaleDB columnstore compression once frequent rebuilds are eliminated. + +## 12.8 Additional Benchmarking + +Several optimizations remain worth investigating. + +These include: + +- composite continuous aggregate indexes +- materialized_only=true +- precomputed weighted values +- precomputed durations +- larger datasets +- higher-frequency readings +- additional meters +- production-scale workloads + +# 13. Lessons Learned + +## Continuous Aggregates Require Careful Dependency Planning + +Objects used by continuous aggregates cannot depend upon: + +- recursive views +- runtime helper functions +- dynamic calculations + +Required metadata should be materialized before aggregation. + +--- + +## Hierarchical Aggregation Improves Scalability + +Building: + +``` +hypertable_hourly_split + │ +Hourly + │ +Daily +``` + +is substantially more efficient than repeatedly aggregating directly from raw data since some expensive calculations are done when the meter reading data is inserted in the readings table. + +--- + +## Correctness Must Be Verified + +Performance improvements are only valuable if analytical correctness is preserved. + +Every aggregate created during this project was validated against the legacy implementation before benchmarking. + +# 14. Acknowledgements + +@MartinCSUMB contributed the initial work integrating the group views and provided an important foundation for the final group aggregation implementation. + +@huss provided valuable guidance throughout testing, validation, benchmarking, and verification of the implementation. + +Their support along with all the previous contributors to OED helped ensure both analytical correctness and significant performance improvements. + +# 15. Current Status + +The TimescaleDB migration has been: + +- implemented +- benchmarked +- validated +- integrated into the application + +Current status: + +- Correctness: Passed +- Performance: Significantly improved +- Testing: Complete +- Integration: Complete + +The project is considered feature complete and ready for future development. + +Future contributors should use this document as the primary technical reference when extending, optimizing, or maintaining the TimescaleDB aggregation workflow. + +# Recommended References and Further Reading + +The following TimescaleDB documentation references provide additional background and guidance for future contributors working on optimization, maintenance, and further development of the aggregation architecture. + +## Data Ingestion Optimization + +TimescaleDB recommends using multi-row inserts or bulk loading methods such as COPY instead of inserting rows individually. Batch ingestion reduces transaction overhead and improves write performance, particularly for high-volume time-series workloads. + +Reference: + +https://www.tigerdata.com/docs/build/data-management/write-data/insert + +Caption: + +TimescaleDB documentation describing recommended approaches for efficient data ingestion, including multi-row INSERT operations and COPY-based loading. + +## Continuous Aggregate Refresh Policies + +Continuous aggregates should generally use bounded refresh windows or scheduled refresh policies rather than repeatedly performing open-ended full refreshes. + +This is particularly relevant to this project because full rebuilds currently regenerate the entire hourly split hypertable and refresh all dependent continuous aggregates. + +Reference: + +https://www.tigerdata.com/docs/build/continuous-aggregates/refresh-policies + +Caption: + +TimescaleDB documentation explaining continuous aggregate refresh policies and recommended approaches for managing incremental materialization. + +## Hypertable Query Performance and Chunk Pruning + +Efficient time filtering is important for TimescaleDB hypertables because queries should allow TimescaleDB to exclude unnecessary chunks. + +Future query optimization should ensure that time predicates directly constrain the hypertable time column whenever possible. + +Reference: + +https://www.tigerdata.com/docs/build/performance-optimization/secondary-indexes + +Caption: + +TimescaleDB documentation covering query performance optimization, indexing strategies, and efficient access patterns for hypertables. + +## Hierarchical Continuous Aggregates + +The current architecture uses layered aggregation. Future improvements should consider building daily aggregates from hourly aggregate states rather than recalculating from hourly split data. + +When implementing hierarchical continuous aggregates, aggregate states must be designed carefully. For example, averages cannot simply be averaged again because this can produce incorrect results. + +Reference: + +https://www.tigerdata.com/docs/learn/continuous-aggregates/hierarchical-continuous-aggregates + +Caption: + +TimescaleDB documentation describing hierarchical continuous aggregates and techniques for building multi-level aggregation pipelines. + +## Continuous Aggregate Limitations with Joined Tables + +Group aggregation relies on cached dependency tables because continuous aggregates have limitations when tracking changes in joined tables. + +Future contributors should consider these limitations when modifying group membership, metadata dependencies, or refresh workflows. + +Reference: + +https://www.tigerdata.com/docs/learn/continuous-aggregates + +Caption: + +TimescaleDB documentation explaining continuous aggregate behaviour, limitations, and considerations when using joins. + +## Hypertable Chunk Sizing + +The current implementation relies on TimescaleDB default chunk sizing. + +Future optimization should evaluate explicit chunk intervals based on actual ingestion volume, memory availability, and query patterns. + +Reference: + +https://www.tigerdata.com/docs/learn/hypertables/understand-hypertables + +Caption: + +TimescaleDB documentation explaining hypertables, chunk sizing, and storage management considerations. + +## Continuous Aggregate Indexing + +Future benchmarking should evaluate additional composite indexes on continuous aggregates. + +Potential candidates include: + +- (meter_id, graphic_unit_id, bucket) +- (group_id, graphic_unit_id, bucket) + +The optimal indexing strategy should be determined through workload benchmarking. + +Reference: + +https://www.tigerdata.com/docs/build/continuous-aggregates/create-index + +Caption: + +TimescaleDB documentation describing indexing strategies for continuous aggregates. + +--- + +## Real-Time Aggregates + +All continuous aggregates currently use real-time aggregation behaviour. + +Future benchmarking should compare real-time aggregates against materialized-only queries to determine whether disabling real-time aggregation improves predictability and performance. + +Reference: + +https://www.tigerdata.com/docs/learn/continuous-aggregates/real-time-aggregates + +Caption: + +TimescaleDB documentation explaining real-time continuous aggregates and the relationship between materialized data and recent raw data. + +--- + +## Summary + +These references should be considered starting points for future optimization work. The current migration has established the TimescaleDB architecture, but additional improvements may be achieved through: + +- Optimized batch ingestion. +- Improved refresh strategies. +- Better chunk and index tuning. +- Hierarchical aggregate design. +- Reduced unnecessary rebuild operations. +- Improved monitoring of continuous aggregate performance. diff --git a/timescaleDB/timescaleDB.md b/timescaleDB/History/timescaleDB.md similarity index 100% rename from timescaleDB/timescaleDB.md rename to timescaleDB/History/timescaleDB.md diff --git a/timescaleDB/Images/after_insert_meter_duration.png b/timescaleDB/Images/after_insert_meter_duration.png new file mode 100644 index 0000000..d31f11c Binary files /dev/null and b/timescaleDB/Images/after_insert_meter_duration.png differ diff --git a/timescaleDB/Images/after_insert_refresh_meter_daily.png b/timescaleDB/Images/after_insert_refresh_meter_daily.png new file mode 100644 index 0000000..9d9559f Binary files /dev/null and b/timescaleDB/Images/after_insert_refresh_meter_daily.png differ diff --git a/timescaleDB/Images/after_insert_refresh_meter_hourly.png b/timescaleDB/Images/after_insert_refresh_meter_hourly.png new file mode 100644 index 0000000..58c29cf Binary files /dev/null and b/timescaleDB/Images/after_insert_refresh_meter_hourly.png differ diff --git a/timescaleDB/Images/after_refresh_meter_daily.png b/timescaleDB/Images/after_refresh_meter_daily.png new file mode 100644 index 0000000..5358445 Binary files /dev/null and b/timescaleDB/Images/after_refresh_meter_daily.png differ diff --git a/timescaleDB/Images/after_refresh_meter_hourly.png b/timescaleDB/Images/after_refresh_meter_hourly.png new file mode 100644 index 0000000..3e52b9b Binary files /dev/null and b/timescaleDB/Images/after_refresh_meter_hourly.png differ diff --git a/timescaleDB/Images/before_insert_meter_duration.png b/timescaleDB/Images/before_insert_meter_duration.png new file mode 100644 index 0000000..6486d46 Binary files /dev/null and b/timescaleDB/Images/before_insert_meter_duration.png differ diff --git a/timescaleDB/Images/before_insert_refresh_meter_daily.png b/timescaleDB/Images/before_insert_refresh_meter_daily.png new file mode 100644 index 0000000..181751e Binary files /dev/null and b/timescaleDB/Images/before_insert_refresh_meter_daily.png differ diff --git a/timescaleDB/Images/before_insert_refresh_meter_hourly.png b/timescaleDB/Images/before_insert_refresh_meter_hourly.png new file mode 100644 index 0000000..ea03d56 Binary files /dev/null and b/timescaleDB/Images/before_insert_refresh_meter_hourly.png differ diff --git a/timescaleDB/Images/before_refresh_meter_daily.png b/timescaleDB/Images/before_refresh_meter_daily.png new file mode 100644 index 0000000..9efd680 Binary files /dev/null and b/timescaleDB/Images/before_refresh_meter_daily.png differ diff --git a/timescaleDB/Images/before_refresh_meter_hourly.png b/timescaleDB/Images/before_refresh_meter_hourly.png new file mode 100644 index 0000000..7863de2 Binary files /dev/null and b/timescaleDB/Images/before_refresh_meter_hourly.png differ diff --git a/timescaleDB/sqlScripts/compare/CompareDailyReadings.sql b/timescaleDB/sqlScripts/compare/CompareDailyReadings.sql new file mode 100644 index 0000000..7c7f9a2 --- /dev/null +++ b/timescaleDB/sqlScripts/compare/CompareDailyReadings.sql @@ -0,0 +1,91 @@ +/* + * CompareDailyReadings.sql + * + * Purpose: + * + * Compare results between the existing PostgreSQL materialized view + * + * meter_daily_readings_unit + * + * and the TimescaleDB continuous aggregate + * + * meter_daily_readings_unit_cagg + * + * + * The purpose of this script is to validate that the TimescaleDB + * implementation produces equivalent results to the existing materialized + * view before considering migration. + * + * Expected result: + * + * - reading_rate differences should be zero or within floating-point + * rounding tolerance. + * - max_rate and min_rate differences should be zero or within + * floating-point rounding tolerance. + * + * Results are ordered by the largest differences first to make discrepancies + * easier to identify. + */ + + +/* + * 1. Compare reading_rate values. + * + * This compares the primary hourly aggregation result between: + * + * Original: + * meter_daily_readings_unit + * + * TimescaleDB: + * meter_daily_readings_unit_cagg + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.meter_id, + LOWER(mv.time_interval) AS mv_time, + mv.reading_rate AS mv_reading_rate, + LOWER(cagg.time_interval) AS cagg_time, + cagg.reading_rate AS cagg_reading_rate, + (COALESCE(mv.reading_rate, 0) - COALESCE(cagg.reading_rate, 0)) AS difference +FROM meter_daily_readings_unit AS mv LEFT JOIN + meter_daily_readings_unit_cagg AS cagg + ON mv.meter_id = cagg.meter_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + ABS(COALESCE(mv.reading_rate, 0) - COALESCE(cagg.reading_rate, 0)) DESC +LIMIT 20; + + +/* + * 2. Compare max_rate and min_rate values. + * + * This verifies that the continuous aggregate preserves the same minimum and + * maximum hourly rates as the original materialized view. + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.meter_id, + LOWER(mv.time_interval) AS mv_time_start, + LOWER(cagg.time_interval) AS cagg_time_start, + UPPER(mv.time_interval) AS mv_time_end, + UPPER(cagg.time_interval) AS cagg_time_end, + mv.max_rate AS mv_max_rate, + cagg.max_rate AS cagg_max_rate, + (COALESCE(mv.max_rate, 0) - COALESCE(cagg.max_rate, 0)) AS max_difference, + mv.min_rate AS mv_min_rate, + cagg.min_rate AS cagg_min_rate, + (COALESCE(mv.min_rate, 0) - COALESCE(cagg.min_rate, 0)) AS min_difference +FROM meter_daily_readings_unit AS mv LEFT JOIN + meter_daily_readings_unit_cagg AS cagg + ON mv.meter_id = cagg.meter_id + AND mv.time_interval = cagg.time_interval + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + GREATEST( + ABS(COALESCE(mv.max_rate, 0) - COALESCE(cagg.max_rate, 0)), + ABS(COALESCE(mv.min_rate, 0) - COALESCE(cagg.min_rate, 0)) + ) DESC +LIMIT 20; \ No newline at end of file diff --git a/timescaleDB/sqlScripts/compare/CompareGroupDailyReadings.sql b/timescaleDB/sqlScripts/compare/CompareGroupDailyReadings.sql new file mode 100644 index 0000000..addf1c2 --- /dev/null +++ b/timescaleDB/sqlScripts/compare/CompareGroupDailyReadings.sql @@ -0,0 +1,158 @@ +/* + * CompareGroupDailyReadings.sql + * + * Purpose: + * + * Compare results between the existing PostgreSQL materialized view + * + * group_daily_readings_unit + * + * and the TimescaleDB continuous aggregate + * + * group_daily_readings_unit_cagg + * + * + * The purpose of this script is to validate that the TimescaleDB + * implementation produces equivalent results to the existing materialized + * view before considering migration. + * + * + * Expected result: + * + * - reading_rate differences should be zero or within floating-point + * rounding tolerance. + * + * - Missing rows should be investigated. + * + * Results are ordered by the largest differences first to make discrepancies + * easier to identify. + */ + + +/* + * 1. Compare reading_rate values. + * + * This compares the primary daily aggregation result between: + * + * Original: + * group_daily_readings_unit + * + * TimescaleDB: + * group_daily_readings_unit_cagg + * + * Any non-zero differences should be investigated. + */ +SELECT + + mv.group_id, + + LOWER(mv.time_interval) AS mv_time, + + LOWER(cagg.time_interval) AS cagg_time, + + mv.reading_rate AS mv_reading_rate, + + cagg.reading_rate AS cagg_reading_rate, + + ( + COALESCE(mv.reading_rate, 0) + - + COALESCE(cagg.reading_rate, 0) + ) AS difference + + +FROM group_daily_readings_unit AS mv + + +LEFT JOIN group_daily_readings_unit_cagg AS cagg + + ON mv.group_id = cagg.group_id + + AND mv.time_interval = cagg.time_interval + + AND mv.graphic_unit_id = cagg.graphic_unit_id + + +ORDER BY + + ABS( + COALESCE(mv.reading_rate, 0) + - + COALESCE(cagg.reading_rate, 0) + ) DESC + + +LIMIT 20; + + + +/* + * 2. Compare row counts. + * + * This identifies missing or additional daily buckets between the two + * implementations. + */ +SELECT + + COUNT(*) AS total_rows, + + COUNT(cagg.group_id) AS matching_cagg_rows, + + COUNT(*) - COUNT(cagg.group_id) AS missing_cagg_rows + + +FROM group_daily_readings_unit AS mv + + +LEFT JOIN group_daily_readings_unit_cagg AS cagg + + ON mv.group_id = cagg.group_id + + AND mv.time_interval = cagg.time_interval + + AND mv.graphic_unit_id = cagg.graphic_unit_id; + + + +/* + * 3. Display missing TimescaleDB rows. + * + * These rows exist in the original materialized view but not in the + * continuous aggregate. + */ +SELECT + + mv.group_id, + + LOWER(mv.time_interval) AS mv_time, + + UPPER(mv.time_interval) AS mv_time_end, + + mv.graphic_unit_id, + + mv.reading_rate + + +FROM group_daily_readings_unit AS mv + + +LEFT JOIN group_daily_readings_unit_cagg AS cagg + + ON mv.group_id = cagg.group_id + + AND mv.time_interval = cagg.time_interval + + AND mv.graphic_unit_id = cagg.graphic_unit_id + + +WHERE cagg.group_id IS NULL + + +ORDER BY + + mv.group_id, + + mv.time_interval + + +LIMIT 20; diff --git a/timescaleDB/sqlScripts/compare/CompareGroupHourlyReadings.sql b/timescaleDB/sqlScripts/compare/CompareGroupHourlyReadings.sql new file mode 100644 index 0000000..437227b --- /dev/null +++ b/timescaleDB/sqlScripts/compare/CompareGroupHourlyReadings.sql @@ -0,0 +1,123 @@ +/* + * CompareGroupHourlyReadings.sql + * + * Purpose: + * + * Compare results between the existing PostgreSQL materialized view + * + * group_hourly_readings_unit + * + * and the TimescaleDB continuous aggregate + * + * group_hourly_readings_unit_cagg + * + * + * The purpose of this script is to validate that the TimescaleDB + * implementation produces equivalent results to the existing materialized + * view before considering migration. + * + * + * Expected result: + * + * - reading_rate differences should be zero or within floating-point + * rounding tolerance. + * + * Results are ordered by the largest differences first to make discrepancies + * easier to identify. + */ + + +/* + * 1. Compare reading_rate values. + * + * This compares the primary hourly aggregation result between: + * + * Original: + * group_hourly_readings_unit + * + * TimescaleDB: + * group_hourly_readings_unit_cagg + * + * Any non-zero differences should be investigated. + */ +SELECT + + mv.group_id, + + LOWER(mv.time_interval) AS mv_time, + + mv.reading_rate AS mv_reading_rate, + + LOWER(cagg.time_interval) AS cagg_time, + + cagg.reading_rate AS cagg_reading_rate, + + ( + COALESCE(mv.reading_rate, 0) + - + COALESCE(cagg.reading_rate, 0) + ) AS difference + + +FROM group_hourly_readings_unit AS mv + +INNER JOIN group_hourly_readings_unit_cagg AS cagg + + ON mv.group_id = cagg.group_id + + AND mv.time_interval = cagg.time_interval + + AND mv.graphic_unit_id = cagg.graphic_unit_id + + +ORDER BY + + ABS( + COALESCE(mv.reading_rate, 0) + - + COALESCE(cagg.reading_rate, 0) + ) DESC + + +LIMIT 20; + + + +/* + * 2. Identify rows missing from the TimescaleDB continuous aggregate. + * + * This catches cases where the continuous aggregate does not contain a + * corresponding group/hour/unit combination. + */ +SELECT + + mv.group_id, + + mv.time_interval, + + mv.graphic_unit_id, + + mv.reading_rate + +FROM group_hourly_readings_unit AS mv + +LEFT JOIN group_hourly_readings_unit_cagg AS cagg + + ON mv.group_id = cagg.group_id + + AND mv.time_interval = cagg.time_interval + + AND mv.graphic_unit_id = cagg.graphic_unit_id + + +WHERE cagg.group_id IS NULL + + +ORDER BY + + mv.group_id, + + mv.time_interval + + +LIMIT 20; \ No newline at end of file diff --git a/timescaleDB/sqlScripts/compare/CompareHourlyReadings.sql b/timescaleDB/sqlScripts/compare/CompareHourlyReadings.sql new file mode 100644 index 0000000..feffba8 --- /dev/null +++ b/timescaleDB/sqlScripts/compare/CompareHourlyReadings.sql @@ -0,0 +1,88 @@ +/* + * CompareHourlyReadings.sql + * + * Purpose: + * + * Compare results between the existing PostgreSQL materialized view + * + * meter_hourly_readings_unit + * + * and the TimescaleDB continuous aggregate + * + * meter_hourly_readings_unit_cagg + * + * + * The purpose of this script is to validate that the TimescaleDB + * implementation produces equivalent results to the existing materialized + * view before considering migration. + * + * Expected result: + * + * - reading_rate differences should be zero or within floating-point + * rounding tolerance. + * - max_rate and min_rate differences should be zero or within + * floating-point rounding tolerance. + * + * Results are ordered by the largest differences first to make discrepancies + * easier to identify. + */ + + +/* + * 1. Compare reading_rate values. + * + * This compares the primary hourly aggregation result between: + * + * Original: + * meter_hourly_readings_unit + * + * TimescaleDB: + * meter_hourly_readings_unit_cagg + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.meter_id, + LOWER(mv.time_interval) AS mv_time, + mv.reading_rate AS mv_reading_rate, + cagg.bucket AS cagg_time, + cagg.reading_rate AS cagg_reading_rate, + (COALESCE(mv.reading_rate, 0) - COALESCE(cagg.reading_rate)) AS difference +FROM meter_hourly_readings_unit AS mv INNER JOIN + meter_hourly_readings_unit_cagg AS cagg + ON mv.meter_id = cagg.meter_id + AND lower(mv.time_interval) = cagg.bucket + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + ABS(COALESCE(mv.reading_rate) - COALESCE(cagg.reading_rate)) DESC +LIMIT 20; + + +/* + * 2. Compare max_rate and min_rate values. + * + * This verifies that the continuous aggregate preserves the same minimum and + * maximum hourly rates as the original materialized view. + * + * Any non-zero differences should be investigated. + */ +SELECT + mv.meter_id, + LOWER(mv.time_interval) AS mv_time, + mv.max_rate AS mv_max_rate, + cagg.max_rate AS cagg_max_rate, + (COALESCE(mv.max_rate) - COALESCE(cagg.max_rate)) AS max_difference, + mv.min_rate AS mv_min_rate, + cagg.min_rate AS cagg_min_rate, + (COALESCE(mv.min_rate) - COALESCE(cagg.min_rate)) AS min_difference +FROM meter_hourly_readings_unit AS mv INNER JOIN + meter_hourly_readings_unit_cagg AS cagg + ON mv.meter_id = cagg.meter_id + AND lower(mv.time_interval) = cagg.bucket + AND mv.graphic_unit_id = cagg.graphic_unit_id +ORDER BY + GREATEST( + ABS(COALESCE(mv.max_rate, 0) - COALESCE(cagg.max_rate, 0)), + ABS(COALESCE(mv.min_rate, 0) - COALESCE(cagg.min_rate, 0)) + ) DESC +LIMIT 20; \ No newline at end of file diff --git a/timescaleDB/sqlScripts/implementation/create_daily_readings.sql b/timescaleDB/sqlScripts/implementation/create_daily_readings.sql new file mode 100644 index 0000000..e6fc63f --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/create_daily_readings.sql @@ -0,0 +1,72 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + +/* + * Aggregating directly from hypertable_hourly_split, it will be used by by group_daily_readings_unit_cagg. + * Therefore it is necessaary to retain the bucket to allow for proper grouping in the next level of + * aggregation. + * + * Data flow: + * + * readings + * | + * trigger + * | + * v + * hypertable_hourly_split + * | + * v + * meter_daily_readings_unit_cagg + * + * + * Daily aggregation: + * + * Each row in hypertable_hourly_split represents the aggregated a single + * meter, graphic unit, and hour. + * + * The daily continuous aggregate groups those hourly rows into one-day + * buckets and computes: + * + * - Average hourly reading rate for the day. + * - Minimum hourly reading rate observed during the day. + * - Maximum hourly reading rate observed during the day. + * + * + * Time interval: + * + * Exposes the bucket timestamp directly + * + */ +CREATE MATERIALIZED VIEW IF NOT EXISTS meter_daily_readings_unit_cagg +WITH (timescaledb.continuous) +AS +SELECT + meter_id, + sum((reading / extract(EPOCH FROM (end_timestamp - start_timestamp)) * slope + intercept) * extract(EPOCH FROM (end_timestamp - start_timestamp))) / sum(extract(EPOCH FROM (end_timestamp - start_timestamp))) AS reading_rate, + max(reading / extract(EPOCH FROM (end_timestamp - start_timestamp)) * slope + intercept) AS max_rate, + min(reading / extract(EPOCH FROM (end_timestamp - start_timestamp)) * slope + intercept) AS min_rate, + tsrange( + time_bucket('1 day', start_timestamp), + time_bucket('1 day', start_timestamp) + INTERVAL '1 day', + '()' + ) AS time_interval, + graphic_unit_id, + time_bucket('1 day', start_timestamp) AS bucket +FROM hypertable_hourly_split +GROUP BY meter_id, time_bucket('1 day', start_timestamp), graphic_unit_id +WITH NO DATA; + +-- This should improve meter continuous-aggregate refreshes +CREATE INDEX IF NOT EXISTS meter_daily_cagg_meter_graphic_bucket_idx +ON meter_daily_readings_unit_cagg + (meter_id, graphic_unit_id, bucket); + +/* + * Allow queries to include recent data that has not yet been materialized. + */ +ALTER MATERIALIZED VIEW meter_daily_readings_unit_cagg +SET ( + timescaledb.materialized_only = false +); diff --git a/timescaleDB/sqlScripts/implementation/create_group_daily_readings.sql b/timescaleDB/sqlScripts/implementation/create_group_daily_readings.sql new file mode 100644 index 0000000..31bf5f4 --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/create_group_daily_readings.sql @@ -0,0 +1,71 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + +/* + * Purpose: + * + * This materialized view rolls up daily meter readings into daily group + * readings. + * + * + * Data flow: + * + * readings + * | + * trigger + * | + * v + * hypertable_hourly_split + * | + * v + * meter_hourly_readings_unit_cagg + * | + * v + * group_daily_readings_unit_cagg + * + * + * The view combines meter-level daily readings into group-level daily + * readings by summing all meters belonging to the group. + */ +CREATE MATERIALIZED VIEW IF NOT EXISTS group_daily_readings_unit_cagg +WITH (timescaledb.continuous) +AS +SELECT + gdm.group_id, + SUM(dr.reading_rate) AS reading_rate, + tsrange( + time_bucket('1 day', dr.bucket), + time_bucket('1 day', dr.bucket) + INTERVAL '1 day', + '()' + ) AS time_interval, + time_bucket('1 day', dr.bucket) AS bucket, + dr.graphic_unit_id +FROM meter_daily_readings_unit_cagg dr JOIN + groups_deep_meters_cache gdm ON dr.meter_id = gdm.meter_id JOIN + group_graphic_units_cache gu ON gu.group_id = gdm.group_id AND dr.graphic_unit_id = gu.graphic_unit_id +GROUP BY + gdm.group_id, + time_bucket('1 day', dr.bucket), + dr.graphic_unit_id +WITH NO DATA; + +-- This should improve group continuous-aggregate refreshes +CREATE INDEX IF NOT EXISTS group_daily_cagg_group_graphic_bucket_idx +ON group_daily_readings_unit_cagg + (group_id, graphic_unit_id, bucket); + +/* + * Allow queries to include recent data that has not yet been materialized. + */ +ALTER MATERIALIZED VIEW group_daily_readings_unit_cagg +SET ( + timescaledb.materialized_only = false +); + +/* + * Preserve existing database ownership. + */ +ALTER TABLE group_daily_readings_unit_cagg +OWNER TO oed; diff --git a/timescaleDB/sqlScripts/implementation/create_group_dependencies.sql b/timescaleDB/sqlScripts/implementation/create_group_dependencies.sql new file mode 100644 index 0000000..71a027b --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/create_group_dependencies.sql @@ -0,0 +1,189 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + +/* + * Purpose: + * + * Create and maintain the non-time-series cache tables required by the + * group continuous aggregates. + * + * + * Background: + * + * TimescaleDB continuous aggregates cannot depend on: + * + * - recursive views + * - PL/pgSQL functions + * - dynamic table-returning logic + * + * The previous group aggregation workflow depended on: + * + * groups_deep_meters_cache table + * legacy graphic-unit compatibility function + * + * These objects are replaced with cache tables that are refreshed before + * refreshing the group continuous aggregates. + * + * + * Data flow: + * + * groups_immediate_children + * | + * v + * groups_deep_children + * | + * v + * groups_deep_meters_cache + * | + * v + * group_graphic_units_cache + * | + * v + * group_hourly_readings_unit_cagg + * group_daily_readings_unit_cagg + * + * + * Refresh: + * + * The following functions should be called before refreshing group + * continuous aggregates: + * + * update_groups_deep_meters_cache() + * update_group_graphic_units_cache() + * + */ + + +/* + * 1. Cache all group to meter relationships. + * + * This replaces the previous groups_deep_meters_cache materialized view. + * + * Each row represents a meter that contributes readings to a group, + * including meters inherited through child groups. + */ + +CREATE TABLE IF NOT EXISTS groups_deep_meters_cache ( + group_id INTEGER NOT NULL REFERENCES groups(id), + meter_id INTEGER NOT NULL REFERENCES meters(id), + PRIMARY KEY(group_id, meter_id) +); + +-- This should improve group continuous-aggregate refreshes +CREATE INDEX IF NOT EXISTS groups_deep_meters_cache_meter_group_idx +ON groups_deep_meters_cache (meter_id, group_id); + + +/* + * Refresh groups_deep_meters_cache. + * + * Calculates the desired rows once, removes stale rows, and inserts new rows. + */ +CREATE OR REPLACE FUNCTION update_groups_deep_meters_cache() +RETURNS void +AS $$ +BEGIN + WITH desired AS MATERIALIZED ( + WITH all_deep_meters(group_id, meter_id) AS ( + SELECT DISTINCT gdc.parent_id AS group_id, gim.meter_id AS meter_id + FROM groups_immediate_meters gim INNER JOIN + groups_deep_children gdc ON gdc.child_id = gim.group_id + UNION + SELECT gim.group_id, gim.meter_id + FROM groups_immediate_meters gim + ) + SELECT group_id, meter_id + FROM all_deep_meters + ), + removed AS ( + DELETE FROM groups_deep_meters_cache target + WHERE NOT EXISTS ( + SELECT 1 + FROM desired + WHERE desired.group_id = target.group_id + AND desired.meter_id = target.meter_id + ) + RETURNING target.group_id + ) + INSERT INTO groups_deep_meters_cache(group_id, meter_id) + SELECT desired.group_id, desired.meter_id + FROM desired + ON CONFLICT (group_id, meter_id) DO NOTHING; +END; +$$ LANGUAGE plpgsql; + +/* + * 2. Cache compatible graphic units for groups. + * + * This replaces the legacy graphic-unit compatibility function. + * + * Each row represents a graphic unit that can display all meters belonging + * to the group. + */ +CREATE TABLE IF NOT EXISTS group_graphic_units_cache ( + group_id INTEGER NOT NULL REFERENCES groups(id), + graphic_unit_id INTEGER NOT NULL REFERENCES units(id), + PRIMARY KEY(group_id, graphic_unit_id) +); + + +/* + * Refresh group graphic unit compatibility. + * + * A graphic unit is valid for a group only when every meter unit contained + * in that group has a conversion path to that graphic unit. + */ +CREATE OR REPLACE FUNCTION update_group_graphic_units_cache() +RETURNS void +AS $$ +BEGIN + + WITH desired AS MATERIALIZED ( + WITH group_source_units AS ( + SELECT gdm.group_id, array_agg(DISTINCT m.unit_id) AS source_units + FROM groups_deep_meters_cache gdm INNER JOIN + meters m ON m.id = gdm.meter_id + GROUP BY gdm.group_id + ), + compatible_units AS ( + SELECT gsu.group_id, c.destination_id AS graphic_unit_id + FROM group_source_units gsu INNER JOIN + cik c ON c.source_id = ANY(gsu.source_units) + GROUP BY gsu.group_id, c.destination_id, gsu.source_units + HAVING array_agg(DISTINCT c.source_id) @> gsu.source_units + ) + SELECT group_id, graphic_unit_id + FROM compatible_units + ), + removed AS ( + DELETE FROM group_graphic_units_cache target + WHERE NOT EXISTS ( + SELECT 1 + FROM desired + WHERE desired.group_id = target.group_id + AND desired.graphic_unit_id = target.graphic_unit_id + ) + RETURNING target.group_id + ) + INSERT INTO group_graphic_units_cache(group_id, graphic_unit_id) + SELECT desired.group_id, desired.graphic_unit_id + FROM desired + ON CONFLICT (group_id, graphic_unit_id) DO NOTHING; + +END; +$$ LANGUAGE plpgsql; + +/* + * Initial population. + * + * This allows a fresh installation to immediately support group CAGGs. + * Subsequent updates should call these functions before refreshing CAGGs. + */ +DO $$ +BEGIN + PERFORM update_groups_deep_meters_cache(); + PERFORM update_group_graphic_units_cache(); +END +$$; diff --git a/timescaleDB/sqlScripts/implementation/create_group_hourly_readings.sql b/timescaleDB/sqlScripts/implementation/create_group_hourly_readings.sql new file mode 100644 index 0000000..4942854 --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/create_group_hourly_readings.sql @@ -0,0 +1,92 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + + /* + * Purpose: + * + * This materialized view rolls up hourly meter readings into hourly group + * readings. + * + * + * Data flow: + * + * readings + * | + * trigger + * | + * v + * hypertable_hourly_split + * | + * v + * meter_hourly_readings_unit_cagg + * | + * v + * group_hourly_readings_unit_cagg + * + * + * The aggregate combines meter-level hourly readings into group-level hourly + * readings by summing all meters belonging to the group. + * + * + * The group hourly aggregate applies group membership and graphic unit + * filtering while preserving the hourly reporting interval format used by the + * existing reporting layer. + */ + +/* + * Create the group materialized view over the TimescaleDB meter aggregate. + * + * Source: + * + * meter_hourly_readings_unit_cagg + * + * The meter-level hourly aggregate already contains: + * + * - Hourly time bucketing + * - Unit conversion + * - Time-varying conversion handling + * + * This aggregate only performs the group-level rollup. + */ +CREATE MATERIALIZED VIEW IF NOT EXISTS group_hourly_readings_unit_cagg +WITH (timescaledb.continuous) +AS +SELECT + gdm.group_id, + SUM(hr.reading_rate) AS reading_rate, + tsrange( + time_bucket('1 hour', hr.bucket), + time_bucket('1 hour', hr.bucket) + INTERVAL '1 hour', + '()' + ) AS time_interval, + time_bucket('1 hour', hr.bucket) AS bucket, + hr.graphic_unit_id +FROM meter_hourly_readings_unit_cagg hr INNER JOIN + groups_deep_meters_cache gdm ON hr.meter_id = gdm.meter_id INNER JOIN + group_graphic_units_cache gu ON gu.group_id = gdm.group_id AND hr.graphic_unit_id = gu.graphic_unit_id +GROUP BY + gdm.group_id, + time_bucket('1 hour', hr.bucket), + hr.graphic_unit_id +WITH NO DATA; + +-- This should improve group continuous-aggregate refreshes +CREATE INDEX IF NOT EXISTS group_hourly_cagg_group_graphic_bucket_idx +ON group_hourly_readings_unit_cagg + (group_id, graphic_unit_id, bucket); + +/* + * Allow queries to include recent data that has not yet been materialized. + */ +ALTER MATERIALIZED VIEW group_hourly_readings_unit_cagg +SET ( + timescaledb.materialized_only = false +); + +/* + * Preserve existing database ownership. + */ +ALTER TABLE group_hourly_readings_unit_cagg +OWNER TO oed; \ No newline at end of file diff --git a/timescaleDB/sqlScripts/implementation/create_hourly_readings.sql b/timescaleDB/sqlScripts/implementation/create_hourly_readings.sql new file mode 100644 index 0000000..e1afd2f --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/create_hourly_readings.sql @@ -0,0 +1,117 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + + /* + * Prefrace: + * This script continues the work introduced in PR#1546, which established the + * benchmark for migrating hourly meter reading queries from PostgreSQL + * materialized views to TimescaleDB hypertables and continuous aggregates. + * + * Only the database objects required from PR#1546 were carried forward and + * adapted to integrate TimescaleDB continuous aggregates with the existing + * hourly meter reading workflow in the timeVary branch. + * + * Purpose: + * + * Create the TimescaleDB continuous aggregate used for hourly meter + * readings. + * + * Data flow: + * + * readings + * | + * trigger + * │ + * ▼ + * hypertable_hourly_split + * │ + * ▼ + * meter_hourly_readings_unit_cagg + * + * Notes: + * + * - The continuous aggregate is built from hypertable_hourly_split. + * - Hourly reading slices are maintained by the trigger created in + * create_prerequisites.sql. + * - Refreshing the aggregate is performed separately using + * refresh_continuous_aggregate() or a refresh policy. + */ + +/* + * 1. Create continuous aggregate for hourly meter readings. + * + * This continuous aggregate replaces the existing + * meter_hourly_readings_unit materialized view using TimescaleDB's + * incremental aggregation engine. + * + * Data flow: + * + * hypertable_hourly_split + * | + * v + * meter_hourly_readings_unit_cagg + * + * + * The hourly split table already contains: + * + * - hourly overlap calculations + * - cik_vary conversion parameters + * - unit metadata + * + * Therefore, the continuous aggregate does not need to join against + * cik_vary or other lookup tables during query execution. + * + * + * Reading calculation: + * + * hypertable_hourly_split stores reading contributions scaled by the + * duration overlap within each hourly slice. + * + * To calculate the final hourly reading rate: + * + * 1. Convert the stored contribution back into a rate. + * 2. Apply the cik_vary conversion: + * + * converted_value = rate * slope + intercept + * + * 3. Weight the converted rate by the duration of the slice. + * 4. Divide by the total duration to produce the weighted average. + * + * This reproduces the calculation performed by the original + * meter_hourly_readings_unit materialized view. + */ +CREATE MATERIALIZED VIEW IF NOT EXISTS meter_hourly_readings_unit_cagg +WITH (timescaledb.continuous) +AS +SELECT + meter_id, + graphic_unit_id, + time_bucket('1 hour', start_timestamp) AS bucket, + sum(( reading / extract( EPOCH FROM (end_timestamp - start_timestamp) ) * slope + intercept ) * extract(EPOCH FROM (end_timestamp - start_timestamp))) / sum(extract(EPOCH FROM (end_timestamp - start_timestamp))) AS reading_rate, + max(reading / extract(EPOCH FROM (end_timestamp - start_timestamp)) * slope + intercept) AS max_rate, + min(reading / extract(EPOCH FROM (end_timestamp - start_timestamp)) * slope + intercept) AS min_rate, + unit_represent, + sec_in_rate +FROM hypertable_hourly_split +GROUP BY + meter_id, + graphic_unit_id, + time_bucket('1 hour', start_timestamp), + unit_represent, + sec_in_rate +WITH NO DATA; + +-- This should improve meter continuous-aggregate refreshes +CREATE INDEX IF NOT EXISTS meter_hourly_cagg_meter_graphic_bucket_idx +ON meter_hourly_readings_unit_cagg + (meter_id, graphic_unit_id, bucket); + +/* + * Allow queries to include recent data that has not yet been materialized. + */ +ALTER MATERIALIZED VIEW meter_hourly_readings_unit_cagg +SET ( + timescaledb.materialized_only = false +); diff --git a/timescaleDB/sqlScripts/implementation/create_prerequisites.sql b/timescaleDB/sqlScripts/implementation/create_prerequisites.sql new file mode 100644 index 0000000..82ff701 --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/create_prerequisites.sql @@ -0,0 +1,394 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + + /* + * Prefrace: + * This script continues the work introduced in PR#1546, which established the + * benchmark for migrating hourly meter reading queries from PostgreSQL + * materialized views to TimescaleDB hypertables and continuous aggregates. + * + * Only the database objects required from PR#1546 were carried forward and + * adapted to integrate TimescaleDB continuous aggregates with the existing + * hourly meter reading workflow in the timeVary branch. + * + * Purpose: + * + * Create the TimescaleDB infrastructure required by the hourly and daily + * continuous aggregates. + * + * This script creates: + * + * 1. hypertable_hourly_split + * 2. TimescaleDB hypertable + * 3. Supporting indexes + * 4. Trigger function to maintain the hypertable + * 5. Trigger on readings + * 6. Rebuild function + * + * Data flow: + * + * readings + * │ + * ▼ + * hypertable_hourly_split + * + * Notes: + * + * - This script does not create any continuous aggregates. + * - The trigger maintains the hypertable as readings are inserted, + * updated, or deleted. + * - The rebuild function allows the hypertable to be regenerated from + * readings when required (for example after rebuilding cik_vary). + */ + + /* + * 1. Create hypertable_hourly_split. + * + * Stores readings after they have been split into hourly intervals. + * A single row in readings may produce multiple rows in this table when the + * reading crosses one or more hour boundaries. + * + * Conversion information from cik_vary is stored with each hourly slice so + * downstream aggregation does not need to join back to cik_vary. + * + * Columns: + * + * reading: + * Reading contribution scaled by the overlap duration within this hour. + * + * slope/intercept/graphic_unit_id: + * Conversion parameters required to convert the reading into the target + * graphic unit that was valid at the time of the reading. + * + * unit_represent/sec_in_rate: + * Original unit metadata required to correctly aggregate quantity, flow, + * and raw readings. + */ +CREATE TABLE IF NOT EXISTS hypertable_hourly_split ( + meter_id INTEGER NOT NULL, + reading FLOAT NOT NULL, + start_timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL, + end_timestamp TIMESTAMP WITHOUT TIME ZONE NOT NULL, + unit_represent unit_represent_type NOT NULL, + sec_in_rate FLOAT NOT NULL, + slope FLOAT NOT NULL, + intercept FLOAT NOT NULL, + graphic_unit_id INTEGER NOT NULL +); + +-- All four aggregates use materialized_only=false, meaning queries can reach unmaterialized split rows. +CREATE INDEX IF NOT EXISTS hypertable_hourly_split_meter_graphic_time_idx +ON hypertable_hourly_split + (meter_id, graphic_unit_id, start_timestamp DESC); + +-- row trigger searches cik_vary by source_id and an overlapping time range +CREATE INDEX IF NOT EXISTS cik_vary_source_time_idx +ON cik_vary (source_id, start_time, end_time); + + +/* + * 2. Convert hypertable_hourly_split into a TimescaleDB hypertable. + * + * start_timestamp is used as the time dimension because hourly slices are + * primarily queried and aggregated by time range. + */ +SELECT create_hypertable( + 'hypertable_hourly_split', + by_range('start_timestamp'), + if_not_exists => TRUE +); + + +/* + * 3. Enforce uniqueness of hourly slices. + * + * The unique index prevents duplicate hourly slices for the same meter and + * time range and supports efficient maintenance operations. + * This index also improves lookup performance when synchronizing changed + * readings. + */ +CREATE UNIQUE INDEX IF NOT EXISTS hypertable_hourly_split_meter_time_idx +ON hypertable_hourly_split +( + meter_id, + start_timestamp, + end_timestamp, + graphic_unit_id, + slope, + intercept +); + +/* + * Track source changes that require rebuilding hypertable_hourly_split. + * + * A revision counter is used instead of a boolean so a source change that + * commits while a rebuild is running cannot be accidentally cleared. The + * refresher only records the revision that it actually rebuilt. + */ +CREATE TABLE IF NOT EXISTS reading_aggregate_state ( + id SMALLINT PRIMARY KEY CHECK (id = 1), + rebuild_revision BIGINT NOT NULL DEFAULT 0, + completed_rebuild_revision BIGINT NOT NULL DEFAULT 0, + group_cache_revision BIGINT NOT NULL DEFAULT 0, + completed_group_cache_revision BIGINT NOT NULL DEFAULT 0 +); + +ALTER TABLE reading_aggregate_state +ADD COLUMN IF NOT EXISTS group_cache_revision BIGINT NOT NULL DEFAULT 0; + +ALTER TABLE reading_aggregate_state +ADD COLUMN IF NOT EXISTS completed_group_cache_revision BIGINT NOT NULL DEFAULT 0; + +INSERT INTO reading_aggregate_state (id) +VALUES (1) +ON CONFLICT (id) DO NOTHING; + +/* + * Unit metadata is copied into hypertable_hourly_split. Mark existing split + * rows stale when a meter changes units or relevant unit metadata changes. + */ +CREATE OR REPLACE FUNCTION mark_reading_aggregate_rebuild_required() +RETURNS trigger +AS $$ +BEGIN + UPDATE reading_aggregate_state + SET rebuild_revision = rebuild_revision + 1 + WHERE id = 1; + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS trigger_meter_unit_requires_reading_rebuild +ON meters; + +CREATE TRIGGER trigger_meter_unit_requires_reading_rebuild +AFTER UPDATE OF unit_id +ON meters +FOR EACH ROW +WHEN (OLD.unit_id IS DISTINCT FROM NEW.unit_id) +EXECUTE FUNCTION mark_reading_aggregate_rebuild_required(); + +/* + * Group membership, meter units, and conversion paths determine the contents + * of the two group dependency caches. Track those changes independently from + * reading imports so normal aggregate refreshes can skip cache maintenance. + */ +CREATE OR REPLACE FUNCTION mark_group_reading_cache_refresh_required() +RETURNS trigger +AS $$ +BEGIN + UPDATE reading_aggregate_state + SET group_cache_revision = group_cache_revision + 1 + WHERE id = 1; + + RETURN NULL; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS trigger_meter_unit_requires_group_cache_refresh +ON meters; + +CREATE TRIGGER trigger_meter_unit_requires_group_cache_refresh +AFTER UPDATE OF unit_id +ON meters +FOR EACH ROW +WHEN (OLD.unit_id IS DISTINCT FROM NEW.unit_id) +EXECUTE FUNCTION mark_group_reading_cache_refresh_required(); + +DROP TRIGGER IF EXISTS trigger_group_children_require_group_cache_refresh +ON groups_immediate_children; + +CREATE TRIGGER trigger_group_children_require_group_cache_refresh +AFTER INSERT OR UPDATE OR DELETE +ON groups_immediate_children +FOR EACH STATEMENT +EXECUTE FUNCTION mark_group_reading_cache_refresh_required(); + +DROP TRIGGER IF EXISTS trigger_group_meters_require_group_cache_refresh +ON groups_immediate_meters; + +CREATE TRIGGER trigger_group_meters_require_group_cache_refresh +AFTER INSERT OR UPDATE OR DELETE +ON groups_immediate_meters +FOR EACH STATEMENT +EXECUTE FUNCTION mark_group_reading_cache_refresh_required(); + +DROP TRIGGER IF EXISTS trigger_unit_metadata_requires_reading_rebuild +ON units; + +CREATE TRIGGER trigger_unit_metadata_requires_reading_rebuild +AFTER UPDATE OF unit_represent, sec_in_rate +ON units +FOR EACH ROW +WHEN ( + OLD.unit_represent IS DISTINCT FROM NEW.unit_represent + OR OLD.sec_in_rate IS DISTINCT FROM NEW.sec_in_rate +) +EXECUTE FUNCTION mark_reading_aggregate_rebuild_required(); + + +/* + * 4. Maintain hypertable_hourly_split from changes in readings. + * + * This trigger function maintains only the hourly slices associated with the + * reading affected by the trigger event. + * + * INSERT: + * Generate hourly slices for the new reading. + * + * UPDATE: + * Recalculate hourly slices for the updated reading. + * + * DELETE: + * Remove hourly slices generated from the deleted reading. + */ +CREATE OR REPLACE FUNCTION update_hourly_hypertable() +RETURNS trigger +AS $$ +BEGIN + + /* + * DELETE removes all hourly slices generated from the deleted reading. + */ + IF TG_OP = 'DELETE' THEN + + DELETE FROM hypertable_hourly_split + WHERE meter_id = OLD.meter_id + AND start_timestamp >= OLD.start_timestamp + AND end_timestamp <= OLD.end_timestamp; + + RETURN OLD; + + END IF; + + + /* + * UPDATE may change the reading duration or hour boundaries. + * Remove the hourly slices generated from the previous version + * of the reading before rebuilding them from NEW. + */ + IF TG_OP = 'UPDATE' THEN + + DELETE FROM hypertable_hourly_split + WHERE meter_id = OLD.meter_id + AND start_timestamp >= OLD.start_timestamp + AND end_timestamp <= OLD.end_timestamp; + + END IF; + + INSERT INTO hypertable_hourly_split(meter_id, reading, start_timestamp, end_timestamp, unit_represent, sec_in_rate, slope, intercept, graphic_unit_id) + SELECT + NEW.meter_id, + CASE + WHEN u.unit_represent = 'quantity'::unit_represent_type THEN + (NEW.reading * 3600 / extract(EPOCH FROM (NEW.end_timestamp - NEW.start_timestamp))) * extract(EPOCH FROM (least(NEW.end_timestamp, gen.interval_start + INTERVAL '1 hour') - greatest(NEW.start_timestamp, gen.interval_start))) + WHEN u.unit_represent IN ('flow'::unit_represent_type, 'raw'::unit_represent_type ) THEN + (NEW.reading * 3600 / u.sec_in_rate) * extract(EPOCH FROM(least(NEW.end_timestamp, gen.interval_start + INTERVAL '1 hour') - greatest(NEW.start_timestamp,gen.interval_start)) + ) END AS reading, + greatest(NEW.start_timestamp, gen.interval_start) AS start_timestamp, + least(NEW.end_timestamp, gen.interval_start + INTERVAL '1 hour') AS end_timestamp, + u.unit_represent, + u.sec_in_rate, + c.slope, + c.intercept, + c.destination_id AS graphic_unit_id + FROM meters m INNER JOIN + units u ON m.unit_id = u.id INNER JOIN + cik_vary c ON c.source_id = m.unit_id AND /*tsrange(c.start_time, c.end_time, '()') && tsrange(NEW.start_timestamp, NEW.end_timestamp, '[]')*/ + c.start_time < NEW.end_timestamp AND c.end_time > NEW.start_timestamp CROSS JOIN + LATERAL generate_series(date_trunc('hour', NEW.start_timestamp), date_trunc_up('hour', NEW.end_timestamp) - INTERVAL '1 hour', INTERVAL '1 hour') gen(interval_start) + WHERE m.id = NEW.meter_id; + RETURN NEW; + +END; +$$ LANGUAGE plpgsql; + +/* + * 5. Recreate the readings trigger. + * + * PostgreSQL does not support CREATE TRIGGER IF NOT EXISTS, so the existing + * trigger is removed first to make this script safe to rerun during development + * and deployment. + */ +DROP TRIGGER IF EXISTS trigger_readings_update_hourly_hypertable +ON readings; + + +/* + * 6. Create trigger to maintain hypertable_hourly_split. + * + * The trigger fires after changes are committed to readings so the trigger + * function can query the updated source row when generating hourly slices. + * + * This trigger maintains the source data used by TimescaleDB continuous + * aggregates. + * + * TimescaleDB continuous aggregates do not refresh immediately from this + * trigger. They track affected time ranges through invalidation and are + * refreshed separately using: + * + * refresh_continuous_aggregate() + * + * or a continuous aggregate refresh policy. + * + * Events: + * + * INSERT: + * Creates hourly split records for the new reading. + * + * UPDATE: + * Removes old hourly split records and recreates them from the + * modified reading. + * + * DELETE: + * Removes hourly split records generated from the deleted reading. + */ +CREATE TRIGGER trigger_readings_update_hourly_hypertable +AFTER INSERT OR UPDATE OR DELETE +ON readings +FOR EACH ROW +EXECUTE FUNCTION update_hourly_hypertable(); + + +/* + * Rebuild hypertable_hourly_split from the current readings and cik_vary rows. + * + * This is needed after cik_vary is regenerated because the trigger stores the + * conversion metadata that exists at reading insert time. If readings were + * inserted before redoCikVary ran, or if conversion segments changed, the split + * rows must be rebuilt from the current conversion table before the continuous + * aggregate is refreshed. + */ +CREATE OR REPLACE FUNCTION rebuild_hourly_hypertable_split() +RETURNS void +AS $$ +BEGIN + DELETE FROM hypertable_hourly_split; + + INSERT INTO hypertable_hourly_split(meter_id, reading, start_timestamp, end_timestamp, unit_represent, sec_in_rate, slope, intercept, graphic_unit_id) + SELECT + r.meter_id, + CASE + WHEN u.unit_represent = 'quantity'::unit_represent_type THEN + (r.reading * 3600 / extract(EPOCH FROM(r.end_timestamp - r.start_timestamp))) * extract(EPOCH FROM (least(r.end_timestamp, gen.interval_start + INTERVAL '1 hour') - greatest(r.start_timestamp, gen.interval_start))) + WHEN u.unit_represent IN('flow'::unit_represent_type, 'raw'::unit_represent_type) THEN + (r.reading * 3600 / u.sec_in_rate) * extract(EPOCH FROM(least(r.end_timestamp, gen.interval_start + INTERVAL '1 hour') - greatest(r.start_timestamp, gen.interval_start))) + END AS reading, + greatest(r.start_timestamp, gen.interval_start) AS start_timestamp, + least(r.end_timestamp, gen.interval_start + INTERVAL '1 hour') AS end_timestamp, + u.unit_represent, + u.sec_in_rate, + c.slope, + c.intercept, + c.destination_id AS graphic_unit_id + FROM readings r INNER JOIN + meters m ON r.meter_id = m.id INNER JOIN + units u ON m.unit_id = u.id INNER JOIN + cik_vary c ON c.source_id = m.unit_id AND /*tsrange(c.start_time, c.end_time, '()') && tsrange(r.start_timestamp, r.end_timestamp, '[]')*/ + c.start_time < r.end_timestamp AND c.end_time > r.start_timestamp CROSS JOIN + LATERAL generate_series(date_trunc('hour', r.start_timestamp), date_trunc_up('hour', r.end_timestamp) - INTERVAL '1 hour', INTERVAL '1 hour') gen(interval_start); +END; +$$ LANGUAGE plpgsql; diff --git a/timescaleDB/sqlScripts/implementation/drop_legacy_reading_views.sql b/timescaleDB/sqlScripts/implementation/drop_legacy_reading_views.sql new file mode 100644 index 0000000..a72148c --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/drop_legacy_reading_views.sql @@ -0,0 +1,17 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + + /* +TO DO: can be removed after the next release, when all users have upgraded to a version +that uses TimescaleDB continuous aggregates instead of PostgreSQL materialized views. +Remove PostgreSQL materialized reading views replaced by TimescaleDB +continuous aggregates. Group views must be dropped before the meter views +they depend on. + */ +DROP MATERIALIZED VIEW IF EXISTS group_daily_readings_unit; +DROP MATERIALIZED VIEW IF EXISTS group_hourly_readings_unit; +DROP MATERIALIZED VIEW IF EXISTS meter_daily_readings_unit; +DROP MATERIALIZED VIEW IF EXISTS meter_hourly_readings_unit; +DROP MATERIALIZED VIEW IF EXISTS groups_deep_meters; diff --git a/timescaleDB/sqlScripts/implementation/update_function_get_3d_readings.sql b/timescaleDB/sqlScripts/implementation/update_function_get_3d_readings.sql new file mode 100644 index 0000000..6d75696 --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/update_function_get_3d_readings.sql @@ -0,0 +1,264 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + + /* +This takes tsrange_to_shrink which is the requested time range to plot and makes sure it does +not exceed the start/end times for the readings in the supplied meter. This can be an issue, in particular, +because infinity is used to indicate to graph all readings. This version does it to the nearest +day by using the day reading view and is used by 3D readings which only allow days and a single meter. + */ +CREATE OR REPLACE FUNCTION shrink_tsrange_to_meter_readings_by_day(tsrange_to_shrink TSRANGE, meter_id_desired INTEGER) + RETURNS TSRANGE +AS $$ +DECLARE + readings_max_tsrange TSRANGE; +BEGIN + SELECT tsrange(min(bucket), max(bucket + INTERVAL '1 day')) INTO readings_max_tsrange + FROM meter_daily_readings_unit_cagg + where meter_id = meter_id_desired; + RETURN tsrange_to_shrink * readings_max_tsrange; +END; +$$ LANGUAGE 'plpgsql'; + +/* Similar to meter version but for a group */ +CREATE OR REPLACE FUNCTION shrink_tsrange_to_group_readings_by_day(tsrange_to_shrink TSRANGE, group_id_desired INTEGER) + RETURNS TSRANGE +AS $$ +DECLARE + readings_max_tsrange TSRANGE; +BEGIN + SELECT tsrange(min(bucket), max(bucket + INTERVAL '1 day')) INTO readings_max_tsrange + FROM group_daily_readings_unit_cagg + where group_id = group_id_desired; + RETURN tsrange_to_shrink * readings_max_tsrange; +END; +$$ LANGUAGE 'plpgsql'; + +-- Determines the spacing between 3D points. It uses the lowest valid spacing +-- for all requested meters. +CREATE OR REPLACE FUNCTION reading_interval_3d ( + IN meter_ids_requested INTEGER[], + IN reading_length_hours INTEGER, + OUT reading_length_hours_use INTEGER, + OUT reading_length_interval INTERVAL +) +AS $$ +DECLARE + meter_frequency INTERVAL; + meter_frequency_hour_up INTEGER; + max_frequency INTEGER; +BEGIN + SELECT min(reading_frequency) INTO meter_frequency + FROM meters m + INNER JOIN unnest(meter_ids_requested) meters(id) ON m.id = meters.id; + + meter_frequency_hour_up := CEIL(EXTRACT(EPOCH FROM meter_frequency) / 3600); + max_frequency := GREATEST(meter_frequency_hour_up, reading_length_hours); + + IF (max_frequency = 5) THEN + reading_length_hours_use := 6; + ELSIF (max_frequency = 7) THEN + reading_length_hours_use := 8; + ELSIF (max_frequency > 8 AND max_frequency < 12) THEN + reading_length_hours_use := 12; + ELSE + reading_length_hours_use := max_frequency; + END IF; + + reading_length_interval := (reading_length_hours_use::TEXT || ' hour')::INTERVAL; +END; +$$ LANGUAGE 'plpgsql'; + +-- Gets meters graphing data for 3D graphic by returning points that span the requested +-- length of time over the days requested. +-- New meter_3d_readings_unit function that uses new meter_hourly_readings_unit_cagg view. +CREATE OR REPLACE FUNCTION meter_3d_readings_unit ( + -- The desired meter ids. It is normally a single value for a 3D graphic. + -- TODO Should the array be changed to a single value as with group? Need to be sure client never asks for multiple. + meter_ids_requested INTEGER[], + -- The desired graphic unit of the returned data + graphic_unit_id_requested INTEGER, + -- The start and end time for the data to return + start_stamp TIMESTAMP, + end_stamp TIMESTAMP, + -- The number of hours in each reading requested + reading_length_hours INTEGER +) + RETURNS TABLE(meter_id INTEGER, reading_rate FLOAT, start_timestamp TIMESTAMP, end_timestamp TIMESTAMP) +AS $$ +DECLARE + -- Holds the range of dates for returned data that fits the actual data. + requested_range TSRANGE; + -- The number of hours in each reading determined as an interval + reading_length_interval INTERVAL; + -- Which index of the meter_id array you are currently working on. + current_meter_index INTEGER := 1; + -- The id of the meter index working on + current_meter_id INTEGER; + -- The meter frequency from all meters. + meter_frequency INTERVAL; + -- The meter frequency rounded up to a whole number of hours. + meter_frequency_hour_up INTEGER; + -- The larger of the meter value and the argument sent. + max_frequency INTEGER; + -- The number of hours in each reading determined + reading_length_hours_use INTEGER; +BEGIN + -- Find the correct number of hours per reading returned. + SELECT * from reading_interval_3d(meter_ids_requested, reading_length_hours) into reading_length_hours_use, reading_length_interval; + + -- Loop over all meters. + WHILE current_meter_index <= cardinality(meter_ids_requested) LOOP + -- ID of the current meter in loop + current_meter_id := meter_ids_requested[current_meter_index]; + + -- Get the range of days requested by calling shrink_tsrange_to_meter_readings_by_day. + -- First make requested range only be full days by dropping any partial days at start/end. + requested_range := shrink_tsrange_to_meter_readings_by_day(tsrange(date_trunc_up('day', start_stamp), date_trunc('day', end_stamp)), current_meter_id); + + IF (reading_length_hours_use <= 12) THEN + -- Need to generate_series to group the desired hours together + RETURN QUERY + -- The readings are rates in the hourly table so want to average not sum so + -- work for quantity, flow & raw. + -- The time starts at the time of the generated sequence and ends at the length + -- of each block later. This is the same as the start time of the next value + -- in the sequence (except last one). + SELECT + -- Modified to retrieve converted hourly readings from the materialized view. + mhr.meter_id as meter_id, + AVG(mhr.reading_rate) as reading_rate, + hours.hour AS start_timestamp, + hours.hour + reading_length_interval AS end_timestamp + -- This is the series that starts at the beginning of the desired days, + -- ends at the end of the desired days and steps by the desired interval. + -- You need to subtract from the last interval for the end since generate_series + -- is inclusive. + FROM ( + SELECT hour + FROM generate_series( + lower(requested_range), + upper(requested_range) - reading_length_interval, + reading_length_interval + ) hours(hour) + ) hours(hour), + -- Also need the values in the meter hourly table. + meter_hourly_readings_unit_cagg mhr + -- Only want the desired meter + WHERE mhr.meter_id = current_meter_id + -- Only want the desired graphing unit + AND mhr.graphic_unit_id = graphic_unit_id_requested + -- Only want readings that lie within this slice of the desired data + AND mhr.bucket >= hours.hour + AND mhr.bucket <= hours.hour + reading_length_interval - INTERVAL '1 hour' + -- Group by the start time of the generated series since all points in + -- the desired slice have the same start time for the series. + -- Also group by the meter_id since Postgres wants and desired for graphing + GROUP BY hours.hour, mhr.meter_id + -- Time sort by the meter and start time for graphing. + ORDER BY mhr.meter_id, hours.hour + ; + ELSE + -- The reading rate is more than 12 so return a single row with dummy values that easy to detect. + -- The end time differs from the start time by the meter reading frequency or min one for groups. + -- This means the meter reading frequency is too long for a 3D graphic. + RETURN QUERY + SELECT -999, -999::FLOAT, '1900-01-01 00:00:00'::TIMESTAMP, '1900-01-01 00:00:00'::TIMESTAMP + reading_length_interval + ; + END IF; + + -- Go to the next meter + current_meter_index := current_meter_index + 1; + END LOOP; +END; +$$ LANGUAGE plpgsql; + +/* Gets group graphing data for 3D graphic by returning points that span the requested + length of time over the days requested. +*/ +CREATE OR REPLACE FUNCTION group_3d_readings_unit ( + --Desire group ID + --For 3D graphics, users will only be able to select 1 group to graph. + group_id_requested INTEGER, + -- The desired graphic unit of the returned data + graphic_unit_id_requested INTEGER, + -- The start and end time for the data to return + start_stamp TIMESTAMP, + end_stamp TIMESTAMP, + -- The number of hours in each reading requested + reading_length_hours INTEGER +) + RETURNS TABLE(reading_rate FLOAT, start_timestamp TIMESTAMP, end_timestamp TIMESTAMP) +AS $$ +DECLARE + -- Holds the range of dates for returned data that fits the actual data. + requested_range TSRANGE; + --Holds the desired meter IDs in order to call meter_3d_readings_unit in the query below. + meter_ids INTEGER[]; + -- The number of hours in each reading determined + reading_length_hours_use INTEGER; + -- The number of hours in each reading determined as an interval + reading_length_interval INTERVAL; +BEGIN + --Get all the meter IDS that will be included in the group being requested. + SELECT array_agg(DISTINCT gdm.meter_id) INTO meter_ids + FROM groups_deep_meters_cache gdm + WHERE group_id = group_id_requested; + + -- Find the correct number of hours per reading returned. + SELECT * from reading_interval_3d(meter_ids, reading_length_hours) into reading_length_hours_use, reading_length_interval; + -- Get the range of days requested by calling shrink_tsrange_to_group_readings_by_day. + -- First make requested range only be full days by dropping any partial days at start/end. + requested_range := shrink_tsrange_to_group_readings_by_day(tsrange(date_trunc_up('day', start_stamp), date_trunc('day', end_stamp)), group_id_requested); + + IF (reading_length_hours_use <= 12) THEN + -- Need to generate_series to group the desired hours together + RETURN QUERY + -- The readings are rates in the hourly table so want to average not sum so + -- work for quantity, flow & raw. + -- The time starts at the time of the generated sequence and ends at the length + -- of each block later. This is the same as the start time of the next value + -- in the sequence (except last one). + SELECT + AVG(ghr.reading_rate) as reading_rate, + hours.hour AS start_timestamp, + hours.hour + reading_length_interval AS end_timestamp + -- This is the series that starts at the beginning of the desired days, + -- ends at the end of the desired days and steps by the desired interval. + -- You need to subtract from the last interval for the end since generate_series + -- is inclusive. + FROM ( + SELECT hour + FROM generate_series( + lower(requested_range), + upper(requested_range) - reading_length_interval, + reading_length_interval + ) hours(hour) + ) hours(hour), + -- Also need the values in the group hourly table. + group_hourly_readings_unit_cagg ghr + -- Only want the desired meter + WHERE ghr.group_id = group_id_requested + -- Only want the desired graphing unit + AND ghr.graphic_unit_id = graphic_unit_id_requested + -- Only want readings that lie within this slice of the desired data + AND ghr.bucket >= hours.hour + AND ghr.bucket <= hours.hour + reading_length_interval - INTERVAL '1 hour' + -- Group by the start time of the generated series since all points in + -- the desired slice have the same start time for the series. + GROUP BY hours.hour + -- Time sort by the start time for graphing. + ORDER BY hours.hour + ; + ELSE + -- The reading rate is more than 12 so return a single row with dummy values that easy to detect. + -- The end time differs from the start time by the meter reading frequency or min one for groups. + -- This means the meter reading frequency is too long for a 3D graphic. + RETURN QUERY + SELECT -999::FLOAT, '1900-01-01 00:00:00'::TIMESTAMP, '1900-01-01 00:00:00'::TIMESTAMP + reading_length_interval + ; + END IF; +END; +$$ LANGUAGE 'plpgsql'; diff --git a/timescaleDB/sqlScripts/implementation/update_function_get_compare_readings.sql b/timescaleDB/sqlScripts/implementation/update_function_get_compare_readings.sql new file mode 100644 index 0000000..116d74b --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/update_function_get_compare_readings.sql @@ -0,0 +1,171 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ + +/* +This shouldn't ever be looking at more than a few weeks of data, so we don't need to deal with compression. + */ + +/* +TODO This function can probably be improved for two reasons: +1) While it was noted that you don't look at too many days, it does limit its usage to modest time lengths. +There has been thought to allowing comparisons, for example, or a whole year. It also assumes that the +frequency of readings is not too high so the number of readings looked at could be large even over +modest time frames. +2) Readings are only included if they are within the current time period. This means that if you have +a reading that crosses the timeframe they are excluded. This can be viewed as either a good thing or +a bad thing. However, if the frequency of readings is high, then large segments of time can be excluded. +For example, monthly readings would not be includes in either day, week or four week comparisons that +OED currently does. Also note that the daily and hourly views do include readings that span a time frame +so including them would be consistent. + +We need to think about how best to deal with this but one option is to use the daily and hourly tables +to get the reading values as in done with bar graphs. This needs to be a little different since partial +days can be involved. However, getting the full days from the daily table and then the hours from the +hourly table to be combined would solve this. (One could get the subhour from the raw readings but it is +unclear users would want that.) Another consider is that the current system, as is the case for bar graphs, +does not take into account missing times in readings which can lead to lower than expected values. The +daily and hourly readings would help fix this. +*/ + +/* +The following function returns data for plotting compare graphs. It works on meters. +It should not be used on raw readings. +This only works when the curr start/end times are on the hour as it uses the hourly reading view. +It will truncate to lose partial hours if they are given. The current client code will only +send full hours so this should be fine. +It is the new version of compare_readings that works with units. It takes these parameters: +meter_ids: A array of meter ids to query. +graphic_unit_id: The unit id of the unit to use for the graph. +curr_start: When the current/this time period begins for the compare. +curr_end: When the current/this time period ends for the compare. +shift: How far back in time to shift the curr_start and curr_end date/time to get the previous + times to compare. + */ +-- New version of the meter_compare_readings function that uses the meter_hourly_readings_unit_cagg view. +CREATE OR REPLACE FUNCTION meter_compare_readings_unit ( + meter_ids INTEGER[], + -- This is the graphic unit id, changed from graphic_unit_id to avoid confusion with the graphic unit id in the view. + passed_graphic_unit_id INTEGER, + curr_start TIMESTAMP, + curr_end TIMESTAMP, + shift INTERVAL +) + RETURNS TABLE(meter_id INTEGER, curr_use FLOAT, prev_use FLOAT) +AS $$ +DECLARE + curr_tsrange TSRANGE; + prev_tsrange TSRANGE; +BEGIN + curr_tsrange := tsrange(curr_start, curr_end); + prev_tsrange := tsrange(curr_start - shift, curr_end - shift); + -- Modified to retrieve converted hourly readings from the materialized view. + RETURN QUERY + WITH + curr_period AS ( + SELECT + hourly.meter_id AS meter_id, + -- It is okay to sum the flow units because hourly readings has a rate of per hour so + -- to convert to a quantity would multiply by 1 so it is the same formula. + SUM(hourly.reading_rate) AS reading + FROM meter_hourly_readings_unit_cagg hourly + -- This is getting the conversion for the meter and unit to graph. + WHERE + -- The range requested must be completely within the hour so partial hours are not included. + hourly.bucket >= lower(curr_tsrange) AND + hourly.bucket <= upper(curr_tsrange) - INTERVAL '1 hour' AND + hourly.graphic_unit_id = passed_graphic_unit_id AND + hourly.meter_id = ANY(meter_ids) + GROUP BY hourly.meter_id + ), + prev_period AS ( + SELECT + hourly.meter_id AS meter_id, + SUM(hourly.reading_rate) AS reading + FROM meter_hourly_readings_unit_cagg hourly + -- This is getting the conversion for the meter and unit to graph. + -- The slope and intercept are used above the transform the reading to the desired unit. + WHERE + -- The range requested must be completely within the hour so partial hours are not included. + hourly.bucket >= lower(prev_tsrange) AND + hourly.bucket <= upper(prev_tsrange) - INTERVAL '1 hour' AND + hourly.graphic_unit_id = passed_graphic_unit_id AND + hourly.meter_id = ANY(meter_ids) + GROUP BY hourly.meter_id + ) + SELECT + meters.id AS meter_id, + curr_period.reading::FLOAT AS curr_use, + prev_period.reading::FLOAT AS prev_use + FROM + unnest(meter_ids) meters(id) + -- Left joins here so we get nulls instead of missing rows if readings don't exist for some time intervals + LEFT JOIN prev_period ON meters.id = prev_period.meter_id + LEFT JOIN curr_period ON meters.id = curr_period.meter_id; +END; +$$ LANGUAGE 'plpgsql'; + + +/* +The following function returns data for plotting compare graphs. It works on groups. +It should not be used on raw readings. +See meter version for needing start/end on the hour. +It is the new version of group_compare_readings that works with units. It takes these parameters: +group_ids: A array of group ids to query. +graphic_unit_id: The unit id of the unit to use for the graph. +curr_start: When the current/this time period begins for the compare. +curr_end: When the current/this time period ends for the compare. +shift: How far back in time to shift the curr_start and curr_end date/time to get the previous + times to compare. + */ +CREATE OR REPLACE FUNCTION group_compare_readings_unit ( + group_ids INTEGER[], + requested_graphic_unit_id INTEGER, + curr_start TIMESTAMP, + curr_end TIMESTAMP, + shift INTERVAL +) + RETURNS TABLE(group_id INTEGER, curr_use FLOAT, prev_use FLOAT) +AS $$ +DECLARE + curr_tsrange TSRANGE; + prev_tsrange TSRANGE; +BEGIN + curr_tsrange := tsrange(curr_start, curr_end); + prev_tsrange := tsrange(curr_start - shift, curr_end - shift); + + RETURN QUERY + WITH + curr_period AS ( + SELECT + hourly.group_id, + SUM(hourly.reading_rate) AS reading + FROM group_hourly_readings_unit_cagg hourly + WHERE hourly.bucket >= lower(curr_tsrange) + AND hourly.bucket <= upper(curr_tsrange) - INTERVAL '1 hour' + AND requested_graphic_unit_id = hourly.graphic_unit_id + AND hourly.group_id = ANY(group_ids) + GROUP BY hourly.group_id + ), + prev_period AS ( + SELECT + hourly.group_id, + SUM(hourly.reading_rate) AS reading + FROM group_hourly_readings_unit_cagg hourly + WHERE hourly.bucket >= lower(prev_tsrange) + AND hourly.bucket <= upper(prev_tsrange) - INTERVAL '1 hour' + AND requested_graphic_unit_id = hourly.graphic_unit_id + AND hourly.group_id = ANY(group_ids) + GROUP BY hourly.group_id + ) + SELECT + gids.id AS group_id, + curr_period.reading::FLOAT AS curr_use, + prev_period.reading::FLOAT AS prev_use + FROM + unnest(group_ids) gids(id) + -- Left joins here so we get nulls instead of missing rows if readings don't exist for some time intervals + LEFT JOIN prev_period ON gids.id = prev_period.group_id + LEFT JOIN curr_period ON gids.id = curr_period.group_id; +END; +$$ LANGUAGE 'plpgsql'; diff --git a/timescaleDB/sqlScripts/implementation/update_group_line_readings_unit.sql b/timescaleDB/sqlScripts/implementation/update_group_line_readings_unit.sql new file mode 100644 index 0000000..cdfa590 --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/update_group_line_readings_unit.sql @@ -0,0 +1,124 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + +/* +The following function determines the correct duration view to query from, and returns averaged readings from it. +It is designed to return data for plotting line graphs. It works on groups. +It is the new version of compressed_group_readings_2 that works with units. It takes these parameters: +group_ids: A array of group ids to query. +graphic_unit_id: The unit id of the unit to use for the graph. +start_timestamp: The start timestamp of the data to return. +end_timestamp: The end timestamp of the data to return. +point_accuracy: Tells how decisions should be made on which types of points to return. 'auto' if automatic. +max_hour_points: The maximum number of data points to return if using the hour view. Only used if 'auto'/'raw' for point_accuracy. +Details on how this function works can be found in the devDocs in the resource generalization document and above +in the meter function that is equivalent. + */ +CREATE OR REPLACE FUNCTION group_line_readings_unit ( + group_ids INTEGER[], + requested_graphic_unit_id INTEGER, + start_stamp TIMESTAMP, + end_stamp TIMESTAMP, + point_accuracy reading_line_accuracy, + max_hour_points INTEGER +) + RETURNS TABLE(group_id INTEGER, reading_rate FLOAT, start_timestamp TIMESTAMP, end_timestamp TIMESTAMP) +AS $$ +DECLARE + meter_ids INTEGER[]; + requested_range TSRANGE; + requested_interval INTERVAL; + requested_interval_seconds INTEGER; + meters_min_frequency INTERVAL; + +BEGIN + -- First get all the meter ids that will be included in one or more groups being queried. + -- In case meter is repeated, make this distinct. + SELECT array_agg(DISTINCT gdm.meter_id) INTO meter_ids + FROM groups_deep_meters_cache gdm + INNER JOIN unnest(group_ids) gids(id) ON gdm.group_id = gids.id; + + -- Calculate point accuracy if request (auto) or if raw since that is not allowed for groups. + IF (point_accuracy = 'auto'::reading_line_accuracy OR point_accuracy = 'raw'::reading_line_accuracy) THEN + -- The request needs automatic calculation of the points returned. + + -- Make sure the time range is within the reading values for meters in this group. + requested_range := shrink_tsrange_to_real_readings(tsrange(start_stamp, end_stamp, '[]'), meter_ids); + -- The request_range will still be infinity if there is no meter data. This causes the + -- auto calculation to fail because you cannot subtract them. + -- Just check the upper range since simpler. + IF (upper(requested_range) = 'infinity') THEN + -- We know there is no data but easier to just let a query happen since fast. + -- Do daily since that should be the fastest due to the least data in most cases. + point_accuracy := 'daily'::reading_line_accuracy; + ELSE + -- The interval of time for the requested_range. + requested_interval := upper(requested_range) - lower(requested_range); + -- Get the seconds in the interval. + -- Wanted to use the INTO syntax used above but could not get it to work so using the set syntax. + requested_interval_seconds := (SELECT * FROM EXTRACT(EPOCH FROM requested_interval)); + -- Make sure that the number of hour points is no more than maximum hourly readings. + -- Thus, check if no more than interval in seconds / (60 seconds/minute * 60 minutes/hour) = # hours in interval. + IF (requested_interval_seconds / 3600 <= max_hour_points) THEN + -- Return hourly reading data. + point_accuracy := 'hourly'::reading_line_accuracy; + ELSE + -- Return daily reading data. + point_accuracy := 'daily'::reading_line_accuracy; + END IF; + + -- Groups can require reading interpolation because of multiple meters. For example, if one meter + -- is 30 day reading frequency then it will interpolate to hourly or daily depending other + -- meters (if exist). However, to limit this effect, if hourly has been selected automatically, + -- check if shortest meter reading frequency for this group is more than an hour and then + -- choose daily instead. + IF (point_accuracy = 'hourly'::reading_line_accuracy) THEN + -- Find the min reading frequency for all meters in the group. + SELECT min(reading_frequency) INTO meters_min_frequency + FROM (meters m + INNER JOIN unnest(meter_ids) meters(id) ON m.id = meters.id); + IF (EXTRACT(EPOCH FROM meters_min_frequency) > 3600) THEN + -- The smallest meter frequency is greater than 1 hour (3600 seconds) so use daily instead. + point_accuracy = 'daily'::reading_line_accuracy; + END IF; + END IF; + END IF; + END IF; + -- point_accuracy should either be daily or hourly at this point. + + IF (point_accuracy = 'daily'::reading_line_accuracy) THEN + RETURN QUERY + SELECT + readings.group_id, + readings.reading_rate, + readings.bucket AS start_timestamp, + readings.bucket + INTERVAL '1 day' AS end_timestamp + FROM group_daily_readings_unit_cagg readings + INNER JOIN unnest(group_ids) gids(id) ON readings.group_id = gids.id + WHERE readings.graphic_unit_id = requested_graphic_unit_id + -- Undefined API bounds arrive as NULL. Convert them to PostgreSQL + -- infinities while keeping bucket directly usable by its B-tree index. + AND readings.bucket >= COALESCE(start_stamp, '-infinity'::TIMESTAMP) + AND readings.bucket <= COALESCE(end_stamp, 'infinity'::TIMESTAMP) - INTERVAL '1 day' + ORDER BY readings.bucket ASC; + + ELSIF (point_accuracy = 'hourly'::reading_line_accuracy) THEN + RETURN QUERY + SELECT + readings.group_id AS group_id, + readings.reading_rate AS reading_rate, + readings.bucket AS start_timestamp, + readings.bucket + INTERVAL '1 hour' AS end_timestamp + FROM group_hourly_readings_unit_cagg readings + INNER JOIN unnest(group_ids) gids(id) ON readings.group_id = gids.id + WHERE readings.graphic_unit_id = requested_graphic_unit_id + -- Undefined API bounds arrive as NULL. Convert them to PostgreSQL + -- infinities while keeping bucket directly usable by its B-tree index. + AND readings.bucket >= COALESCE(start_stamp, '-infinity'::TIMESTAMP) + AND readings.bucket <= COALESCE(end_stamp, 'infinity'::TIMESTAMP) - INTERVAL '1 hour' + ORDER BY readings.bucket ASC; + END IF; +END; +$$ LANGUAGE 'plpgsql'; diff --git a/timescaleDB/sqlScripts/implementation/update_meter_group_bar.sql b/timescaleDB/sqlScripts/implementation/update_meter_group_bar.sql new file mode 100644 index 0000000..5a9df98 --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/update_meter_group_bar.sql @@ -0,0 +1,161 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + +/* +The following function returns data for plotting bar graphs. It works on meters. +It should not be used on raw readings. +It is the new version of compressed_barchart_readings_2 that works with units. It takes these parameters: +meter_ids: A array of meter ids to query. +graphic_unit_id: The unit id of the unit to use for the graph. +bar_width_days: The number of days to use for the bar width. +start_timestamp: The start timestamp of the data to return. +end_timestamp: The end timestamp of the data to return. + */ +-- New version of meter_bar_readings_unit that uses the new meter_daily_readings_unit_cagg view. +CREATE OR REPLACE FUNCTION meter_bar_readings_unit ( + meter_ids INTEGER[], + -- This is the graphic unit id, changed from graphic_unit_id to avoid confusion with the graphic unit id in the view. + passed_graphic_unit_id INTEGER, + bar_width_days INTEGER, + start_stamp TIMESTAMP, + end_stamp TIMESTAMP +) + RETURNS TABLE(meter_id INTEGER, reading FLOAT, start_timestamp TIMESTAMP, end_timestamp TIMESTAMP) +AS $$ +DECLARE + bar_width INTERVAL; + real_tsrange TSRANGE; + real_start_stamp TIMESTAMP; + real_end_stamp TIMESTAMP; + num_bars INTEGER; +BEGIN + -- This is how wide (time interval) for each bar. + bar_width := INTERVAL '1 day' * bar_width_days; + /* + This rounds to the day for the start and end times requested. It then shrinks in case the actual readings span + less time than the request. This can commonly happen when you get +/-infinity for all readings available. + It uses the day reading view because that is faster than using all the readings. + This has an issue associated with it: + + 1) If the readings at the start/end have a partial day then it shows up as a day. The original code did: + real_tsrange := shrink_tsrange_to_real_readings(tsrange(date_trunc_up('day', start_stamp), date_trunc('day', end_stamp))); + and did not have this issue since it used the readings and then truncated up/down. + A more general solution would be to change the daily (and hourly) view so it does not include partial ones at start/end. + This would fix this case and also impact other uses in what seems a positive way. + Note this does not address that missing days in a bar width get no value so the bar will likely read low. + */ + real_tsrange := shrink_tsrange_to_meters_by_day(tsrange(start_stamp, end_stamp), meter_ids); + -- Get the actual start/end time rounded to the nearest day from the range. + real_start_stamp := lower(real_tsrange); + real_end_stamp := upper(real_tsrange); + -- This gives the number of whole bars that will fit within the real start/end times. For example, if the number of days + -- between start and end is 14 days and the bar width is 3 days then you get 4. + num_bars := floor(extract(EPOCH FROM real_end_stamp - real_start_stamp) / extract(EPOCH FROM bar_width)); + -- This makes the full bars go from the end time to as far back in time as possible. + -- This means that if some time was dropped to get full bars it is at the start of the interval. + -- It was felt that the most recent readings are the most important so drop older ones. + -- It also helps with maps since they use the latest bar for their value. + real_start_stamp := real_end_stamp - (num_bars * bar_width); + -- Since the inner join on the generate_series adds the bar_width, we need to back up the + -- end timestamp by that amount so it stops at the desired end timestamp. + real_end_stamp := real_end_stamp - bar_width; + + RETURN QUERY + SELECT + -- Modified to retrieve converted daily readings from the materialized view. + mdr.meter_id AS meter_id, + sum(mdr.reading_rate * 24) AS reading, + bars.interval_start AS start_timestamp, + bars.interval_start + bar_width AS end_timestamp + + FROM meter_daily_readings_unit_cagg mdr + INNER JOIN generate_series(real_start_stamp, real_end_stamp, bar_width) bars(interval_start) + ON mdr.bucket >= bars.interval_start + AND mdr.bucket <= bars.interval_start + bar_width - INTERVAL '1 day' + INNER JOIN unnest(meter_ids) meters(id) ON mdr.meter_id = meters.id + INNER JOIN meters m ON m.id = meters.id + INNER JOIN units u ON m.unit_id = u.id AND u.unit_represent != 'raw'::unit_represent_type + WHERE mdr.graphic_unit_id = passed_graphic_unit_id + GROUP BY mdr.meter_id, bars.interval_start + ORDER BY mdr.meter_id, bars.interval_start; + +END; +$$ LANGUAGE 'plpgsql'; + + +/* +The following function returns data for plotting bar graphs. It works on groups. +It should not be used on raw readings. +It is the new version of compressed_barchart_group_readings_2 that works with units. It takes these parameters: +group_ids: A array of group ids to query. +graphic_unit_id: The unit id of the unit to use for the graph. +bar_width_days: The number of days to use for the bar width. +start_timestamp: The start timestamp of the data to return. +end_timestamp: The end timestamp of the data to return. + */ +CREATE OR REPLACE FUNCTION group_bar_readings_unit ( + group_ids INTEGER[], + requested_graphic_unit_id INTEGER, + bar_width_days INTEGER, + start_stamp TIMESTAMP, + end_stamp TIMESTAMP +) + RETURNS TABLE(group_id INTEGER, reading FLOAT, start_timestamp TIMESTAMP, end_timestamp TIMESTAMP) +AS $$ +DECLARE + bar_width INTERVAL; + real_tsrange TSRANGE; + real_start_stamp TIMESTAMP; + real_end_stamp TIMESTAMP; + num_bars INTEGER; + readings_max_tsrange TSRANGE; +BEGIN + bar_width := INTERVAL '1 day' * bar_width_days; + + SELECT tsrange(min(bucket), max(bucket + INTERVAL '1 day')) INTO readings_max_tsrange + FROM group_daily_readings_unit_cagg dr + -- Get all the group ids passed in. + INNER JOIN unnest(group_ids) gids(id) ON dr.group_id = gids.id; + + real_tsrange := tsrange(date_trunc_up('day', start_stamp), date_trunc('day', end_stamp)) * readings_max_tsrange; + -- Get the actual start/end time rounded to the nearest day from the range. + real_start_stamp := lower(real_tsrange); + real_end_stamp := upper(real_tsrange); + -- This gives the number of whole bars that will fit within the real start/end times. For example, if the number of days + -- between start and end is 14 days and the bar width is 3 days then you get 4. + num_bars := floor(extract(EPOCH FROM real_end_stamp - real_start_stamp) / extract(EPOCH FROM bar_width)); + -- This makes the full bars go from the end time to as far back in time as possible. + -- This means that if some time was dropped to get full bars it is at the start of the interval. + -- It was felt that the most recent readings are the most important so drop older ones. + -- It also helps with maps since they use the latest bar for their value. + real_start_stamp := real_end_stamp - (num_bars * bar_width); + -- Since the inner join on the generate_series adds the bar_width, we need to back up the + -- end timestamp by that amount so it stops at the desired end timestamp. + real_end_stamp := real_end_stamp - bar_width; + + RETURN QUERY + SELECT + -- readings.reading_rate is the weighted average reading rate per hour over the day. + -- Convert to a quantity by multiplying by the time in hours which is 24 since daily values. + -- reading is the sum of all readings within one bar. + readings.group_id AS group_id, + SUM(readings.reading_rate * 24) AS reading, + bars.interval_start AS start_timestamp, + bars.interval_start + bar_width AS end_timestamp + + FROM (((group_daily_readings_unit_cagg readings + INNER JOIN generate_series(real_start_stamp, real_end_stamp, bar_width) bars(interval_start) + ON readings.bucket >= bars.interval_start + AND readings.bucket <= bars.interval_start + bar_width - INTERVAL '1 day') + -- Don't return bar data if raw since cannot sum. + INNER JOIN units u ON readings.graphic_unit_id = u.id AND u.unit_represent != 'raw'::unit_represent_type) + INNER JOIN unnest(group_ids) gids(id) ON readings.group_id = gids.id) + -- Use the readings in the passed in graphic unit + WHERE readings.graphic_unit_id = requested_graphic_unit_id + + GROUP BY readings.group_id, bars.interval_start + ORDER BY readings.group_id, bars.interval_start; +END; +$$ LANGUAGE 'plpgsql'; diff --git a/timescaleDB/sqlScripts/implementation/update_meter_line_readings_unit.sql b/timescaleDB/sqlScripts/implementation/update_meter_line_readings_unit.sql new file mode 100644 index 0000000..c018b17 --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/update_meter_line_readings_unit.sql @@ -0,0 +1,371 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + +/* + * Function: meter_line_readings_unit + * + * Purpose: + * + * Retrieve meter readings optimized for rendering line graphs. This function + * determines the most appropriate data resolution (raw, hourly, or daily) + * based on the requested time range, meter reading frequency, and configured + * point limits. + * + * This function is the unit-aware replacement for compressed_readings_2. + * It supports graphic unit conversion and time-varying unit conversions. + * + * + * Data sources: + * + * Raw: + * Reads directly from the readings table and applies cik_vary conversion + * during query execution. + * + * Hourly: + * Uses the TimescaleDB continuous aggregate: + * + * meter_hourly_readings_unit_cagg + * + * The continuous aggregate is built on top of: + * + * hypertable_hourly_split + * + * Hourly splitting and time-varying conversion metadata are applied + * before aggregation, avoiding runtime joins to cik_vary. + * + * Daily: + * Uses the TimescaleDB continuous aggregate: + * + * meter_daily_readings_unit_cagg + * + * The daily continuous aggregate is built on top of: + * + * meter_hourly_readings_unit_cagg + * + * Daily aggregation reuses precomputed hourly aggregate values instead + * of recalculating from the hourly split hypertable. + * + * + * Resolution selection: + * + * When point_accuracy = 'auto', the function selects the highest resolution + * possible while staying within the requested point limits. + * + * Selection order: + * + * 1. Raw readings + * 2. Hourly continuous aggregate + * 3. Daily continuous aggregate + * + * + * Notes: + * + * - Each meter is processed independently because meters may have different + * reading frequencies and available data ranges. + * + * - The hourly data path previously queried the PostgreSQL materialized view: + * + * meter_hourly_readings_unit + * + * and now uses the TimescaleDB continuous aggregate: + * + * meter_hourly_readings_unit_cagg + * + * - The daily data path previously queried the PostgreSQL materialized view: + * + * meter_daily_readings_unit + * + * and now uses the TimescaleDB continuous aggregate: + * + * meter_daily_readings_unit_cagg + * + * - Design details: + * + * meter_line_readings_unit + * | + * +----------------+----------------+ + * | | | + * v v v + * readings hourly continuous daily continuous + * aggregate aggregate + * | | + * v v + * meter_hourly_readings_unit_cagg + * | + * v + * meter_daily_readings_unit_cagg + */ + + +/* + * DAILY READINGS + * + * Uses TimescaleDB daily continuous aggregate. + * + * Data source: + * + * meter_daily_readings_unit_cagg + * + * The daily continuous aggregate rolls up hourly aggregate values into + * one-day intervals. + * + * The time interval is stored as a PostgreSQL tsrange: + * + * ("2021-06-01 00:00:00","2021-06-02 00:00:00") + * + * The lower and upper bounds are converted back into timestamps so the + * result matches the function return type. + */ + + +/* +The following function determines the correct duration view to query from, and returns averaged or raw reading from it. +It is designed to return data for plotting line graphs. It works on meters. +It is the new version of compressed_readings_2 that works with units. It takes these parameters: +meter_ids: A array of meter ids to query. +graphic_unit_id: The unit id of the unit to use for the graphic. +start_timestamp: The start timestamp of the data to return. +end_timestamp: The end timestamp of the data to return. +point_accuracy: Tells how decisions should be made on which types of points to return. 'auto' if automatic. +max_raw_points: The maximum number of data points to return if using the raw points for a meter. Only used if 'auto' for point_accuracy. +max_hour_points: The maximum number of data points to return if using the hour view. Only used if 'auto' for point_accuracy. +Details on how this function works can be found in the devDocs in the resource generalization document. + */ +-- New version of meter_line_readings_unit that uses the new views. +CREATE OR REPLACE FUNCTION meter_line_readings_unit ( + meter_ids INTEGER[], + -- This is the graphic unit id, changed from graphic_unit_id to avoid confusion with the graphic unit id in the view. + passed_graphic_unit_id INTEGER, + start_stamp TIMESTAMP, + end_stamp TIMESTAMP, + point_accuracy reading_line_accuracy, + max_raw_points INTEGER, + max_hour_points INTEGER +) + RETURNS TABLE(meter_id INTEGER, reading_rate FLOAT, min_rate FLOAT, max_rate FLOAT, start_timestamp TIMESTAMP, end_timestamp TIMESTAMP) +AS $$ +BEGIN + RETURN QUERY + /* + * Process all requested meters as one set. WITH ORDINALITY preserves the + * caller's meter order in the final result. + */ + WITH requested_meters AS ( + SELECT requested.id AS meter_id, requested.request_order + FROM unnest(meter_ids) WITH ORDINALITY requested(id, request_order) + ), + /* + * Use the readings indexes to find the first start and last end for every + * requested meter without aggregating its complete reading history. + */ + reading_bounds AS ( + SELECT + requested.meter_id, + first_reading.start_timestamp AS min_start_timestamp, + last_reading.end_timestamp AS max_end_timestamp + FROM (SELECT DISTINCT rm.meter_id FROM requested_meters rm) requested + LEFT JOIN LATERAL ( + SELECT r.start_timestamp + FROM readings r + WHERE r.meter_id = requested.meter_id + ORDER BY r.start_timestamp + LIMIT 1 + ) first_reading ON TRUE + LEFT JOIN LATERAL ( + SELECT r.end_timestamp + FROM readings r + WHERE r.meter_id = requested.meter_id + ORDER BY r.end_timestamp DESC + LIMIT 1 + ) last_reading ON TRUE + ), + /* + * Restrict the requested range to the readings available for each meter. + * An absent reading bound produces an unbounded range, preserving the + * behavior of shrink_tsrange_to_real_readings(). + */ + meter_ranges AS ( + SELECT + rm.meter_id, + rm.request_order, + m.unit_id, + m.reading_frequency, + tsrange(start_stamp, end_stamp, '[]') + * tsrange(bounds.min_start_timestamp, bounds.max_end_timestamp) AS requested_range + FROM requested_meters rm + INNER JOIN meters m ON m.id = rm.meter_id + LEFT JOIN reading_bounds bounds ON bounds.meter_id = rm.meter_id + ), + /* + * Select raw, hourly, or daily resolution independently for each meter. + * + * Raw is selected when the estimated number of readings does not exceed + * max_raw_points. A frequency of at least one day also stays raw because + * hourly or daily aggregation would interpolate additional points. + * + * Hourly is selected when the requested number of hours does not exceed + * max_hour_points and the meter frequency is no greater than one hour. + * All remaining meters use daily data. + */ + meter_resolutions AS ( + SELECT + mr.*, + CASE + WHEN point_accuracy <> 'auto'::reading_line_accuracy THEN point_accuracy + WHEN upper(mr.requested_range) = 'infinity'::TIMESTAMP THEN 'daily'::reading_line_accuracy + WHEN ( + EXTRACT(EPOCH FROM (upper(mr.requested_range) - lower(mr.requested_range))) + / EXTRACT(EPOCH FROM mr.reading_frequency) <= max_raw_points + OR EXTRACT(EPOCH FROM mr.reading_frequency) >= 86400 + ) THEN 'raw'::reading_line_accuracy + WHEN ( + EXTRACT(EPOCH FROM (upper(mr.requested_range) - lower(mr.requested_range))) / 3600 + <= max_hour_points + AND EXTRACT(EPOCH FROM mr.reading_frequency) <= 3600 + ) THEN 'hourly'::reading_line_accuracy + ELSE 'daily'::reading_line_accuracy + END AS selected_accuracy + FROM meter_ranges mr + ), + /* + * RAW READINGS + * + * Apply time-varying conversions directly to raw readings. Multiple + * cik_vary segments can overlap one reading, so each converted value is + * weighted by the duration of its overlap with that reading. + */ + raw_results AS ( + SELECT + selected.request_order, + r.meter_id, + CASE + WHEN u.unit_represent = 'quantity'::unit_represent_type THEN + /* + * Quantity readings are normalized to a per-hour rate before + * applying the conversion. + */ + SUM( + (EXTRACT(EPOCH FROM ( + upper(tsrange(c.start_time, c.end_time, '()') * tsrange(r.start_timestamp, r.end_timestamp, '[]')) + - lower(tsrange(c.start_time, c.end_time, '()') * tsrange(r.start_timestamp, r.end_timestamp, '[]')) + )) / 3600) + * (c.slope * (r.reading / (EXTRACT(EPOCH FROM (r.end_timestamp - r.start_timestamp)) / 3600)) + c.intercept) + ) / (EXTRACT(EPOCH FROM (r.end_timestamp - r.start_timestamp)) / 3600) + WHEN u.unit_represent IN ('flow'::unit_represent_type, 'raw'::unit_represent_type) THEN + /* + * Flow and raw readings are already rates. Normalize them to + * an hourly rate before applying the conversion. + */ + SUM( + (EXTRACT(EPOCH FROM ( + upper(tsrange(c.start_time, c.end_time, '()') * tsrange(r.start_timestamp, r.end_timestamp, '[]')) + - lower(tsrange(c.start_time, c.end_time, '()') * tsrange(r.start_timestamp, r.end_timestamp, '[]')) + )) / 3600) + * (c.slope * (r.reading * 3600 / u.sec_in_rate) + c.intercept) + ) / (EXTRACT(EPOCH FROM (r.end_timestamp - r.start_timestamp)) / 3600) + END AS reading_rate, + /* + * Raw meter data has no min/max range. NaN is converted to null by + * the route when the result is stored in Redux state. + */ + 'NaN'::DOUBLE PRECISION AS min_rate, + 'NaN'::DOUBLE PRECISION AS max_rate, + r.start_timestamp, + r.end_timestamp + FROM meter_resolutions selected + INNER JOIN readings r ON r.meter_id = selected.meter_id + INNER JOIN units u ON u.id = selected.unit_id + INNER JOIN cik_vary c + ON c.source_id = selected.unit_id + AND c.destination_id = passed_graphic_unit_id + /* + * Allow multiple time-varying conversion segments to contribute + * when they overlap a reading. + */ + AND c.start_time < r.end_timestamp + AND c.end_time > r.start_timestamp + WHERE selected.selected_accuracy = 'raw'::reading_line_accuracy + AND r.start_timestamp >= lower(selected.requested_range) + AND r.end_timestamp <= upper(selected.requested_range) + /* + * unit_represent is stable for a meter, but PostgreSQL requires it in + * the GROUP BY because it controls the CASE expression above. + */ + GROUP BY + selected.request_order, + r.meter_id, + r.start_timestamp, + r.end_timestamp, + u.unit_represent + ), + /* + * HOURLY READINGS + * + * Use the TimescaleDB hourly continuous aggregate. Direct bucket bounds + * allow the meter/graphic-unit/bucket index to constrain the time range. + */ + hourly_results AS ( + SELECT + selected.request_order, + hourly.meter_id, + hourly.reading_rate, + hourly.min_rate, + hourly.max_rate, + hourly.bucket AS start_timestamp, + hourly.bucket + INTERVAL '1 hour' AS end_timestamp + FROM meter_resolutions selected + INNER JOIN meter_hourly_readings_unit_cagg hourly + ON hourly.meter_id = selected.meter_id + AND hourly.graphic_unit_id = passed_graphic_unit_id + AND hourly.bucket >= lower(selected.requested_range) + AND hourly.bucket <= upper(selected.requested_range) - INTERVAL '1 hour' + WHERE selected.selected_accuracy = 'hourly'::reading_line_accuracy + ), + /* + * DAILY READINGS + * + * Use the TimescaleDB daily continuous aggregate with the same indexable + * complete-bucket bounds. + */ + daily_results AS ( + SELECT + selected.request_order, + daily.meter_id, + daily.reading_rate, + daily.min_rate, + daily.max_rate, + daily.bucket AS start_timestamp, + daily.bucket + INTERVAL '1 day' AS end_timestamp + FROM meter_resolutions selected + INNER JOIN meter_daily_readings_unit_cagg daily + ON daily.meter_id = selected.meter_id + AND daily.graphic_unit_id = passed_graphic_unit_id + AND daily.bucket >= lower(selected.requested_range) + AND daily.bucket <= upper(selected.requested_range) - INTERVAL '1 day' + WHERE selected.selected_accuracy = 'daily'::reading_line_accuracy + ), + /* + * Each meter appears in exactly one resolution branch. UNION ALL avoids + * unnecessary duplicate elimination. + */ + results AS ( + SELECT * FROM raw_results + UNION ALL + SELECT * FROM hourly_results + UNION ALL + SELECT * FROM daily_results + ) + SELECT + results.meter_id, + results.reading_rate, + results.min_rate, + results.max_rate, + results.start_timestamp, + results.end_timestamp + FROM results + -- Preserve the original per-meter chronological result ordering. + ORDER BY results.request_order, results.start_timestamp; +END; +$$ LANGUAGE 'plpgsql'; diff --git a/timescaleDB/sqlScripts/implementation/update_reading_views.sql b/timescaleDB/sqlScripts/implementation/update_reading_views.sql new file mode 100644 index 0000000..1a6173f --- /dev/null +++ b/timescaleDB/sqlScripts/implementation/update_reading_views.sql @@ -0,0 +1,93 @@ +/* This Source Code Form is subject to the terms of the Mozilla Public + * License, v. 2.0. If a copy of the MPL was not distributed with this + * file, You can obtain one at http://mozilla.org/MPL/2.0/. + */ + +/* +Rounds a timestamp up to the next interval. + */ +CREATE OR REPLACE FUNCTION date_trunc_up(interval_precision TEXT, ts TIMESTAMP) + RETURNS TIMESTAMP LANGUAGE SQL +IMMUTABLE +AS $$ +SELECT CASE + WHEN ts = date_trunc(interval_precision, ts) THEN ts + ELSE date_trunc(interval_precision, ts + ('1 ' || interval_precision)::INTERVAL) + END +$$; + +/* +Restricts a requested range to the available raw readings for the supplied +meters. + */ +CREATE OR REPLACE FUNCTION shrink_tsrange_to_real_readings(tsrange_to_shrink TSRANGE, meter_ids INTEGER[]) + RETURNS TSRANGE +AS $$ +DECLARE + readings_max_tsrange TSRANGE; +BEGIN + SELECT tsrange(min(start_timestamp), max(end_timestamp)) INTO readings_max_tsrange + FROM readings r + INNER JOIN unnest(meter_ids) meters(id) ON r.meter_id = meters.id; + RETURN tsrange_to_shrink * readings_max_tsrange; +END; +$$ LANGUAGE 'plpgsql'; + +-- TODO: Remove this retained legacy function once the hypertable +-- implementation is finalized. group_graphic_units_cache replaces it. +-- CREATE OR REPLACE FUNCTION get_graphic_unit(requested_group_id INTEGER) +-- RETURNS INTEGER[] AS $$ +-- DECLARE +-- src_ids INTEGER[]; +-- dest_ids INTEGER[]; +-- child_meters_unit_ids INTEGER[]; +-- unit_ids_compatible INTEGER[] := '{}'; +-- unit_id INTEGER; +-- BEGIN +-- SELECT array_agg(DISTINCT m.unit_id) INTO child_meters_unit_ids +-- FROM groups_deep_meters_cache gdm +-- JOIN meters m ON m.id = gdm.meter_id +-- WHERE gdm.group_id = requested_group_id; +-- +-- SELECT array_agg(u.id) INTO dest_ids +-- FROM units u +-- JOIN cik c ON u.id = c.destination_id; +-- +-- FOREACH unit_id IN ARRAY dest_ids +-- LOOP +-- SELECT array_agg(source_id) INTO src_ids +-- FROM cik +-- WHERE destination_id = unit_id; +-- +-- IF src_ids @> child_meters_unit_ids +-- AND NOT (unit_id = ANY (unit_ids_compatible)) +-- THEN +-- unit_ids_compatible := array_append(unit_ids_compatible, unit_id); +-- END IF; +-- END LOOP; +-- +-- RETURN unit_ids_compatible; +-- END; +-- $$ LANGUAGE 'plpgsql'; + +/* +This takes tsrange_to_shrink which is the requested time range to plot and makes sure it does +not exceed the start/end times for all the readings. This can be an issue, in particular, +because infinity is used to indicate to graph all readings. This version does it to the nearest +day by using the day reading view since bars use to the nearest day and this should be faster. +This should be fine since bar uses the same view to get data. + */ +CREATE OR REPLACE FUNCTION shrink_tsrange_to_meters_by_day(tsrange_to_shrink TSRANGE, meter_ids INTEGER[]) + RETURNS TSRANGE +AS $$ +DECLARE + readings_max_tsrange TSRANGE; +BEGIN + SELECT tsrange(min(bucket), max(bucket + INTERVAL '1 day')) INTO readings_max_tsrange + FROM meter_daily_readings_unit_cagg dr + -- Get all the meter_ids in the passed array of meters. + INNER JOIN unnest(meter_ids) meters(id) ON dr.meter_id = meters.id; + -- Make the original range be to the day by dropping parts of days at start/end. + RETURN tsrange(date_trunc_up('day', lower(tsrange_to_shrink)), date_trunc('day', upper(tsrange_to_shrink))) * readings_max_tsrange; +END; +$$ LANGUAGE 'plpgsql';