Repository navigation
feat(traffic): add city analytics with coordinates - #51
Merged
Merged
Conversation
5 of 10 tasks
Strip request URLs from download errors to protect license keys, apply size limits to raw GeoIP databases, and drop out-of-range coordinates. Add regression tests for redaction, database limits, and coordinate bounds.
Suppress and delete city breakdowns when TRAFFIC_IP_MODE is off. Cache GeoIP lookups by database generation and prefer subdivision names. Serialize GeoIP refreshes, clean partial downloads, reuse older DB-IP caches, and add retry backoff with shutdown cancellation. Upgrade maxminddb to 0.32. Optimize breakdown indexes and coordinate queries, update stored coordinates, prune unused coordinates, and amortize aggregation pruning. Match benchmark city ranking to the API and add regression coverage.
Mark optional latitude and longitude fields as non-nullable in OpenAPI. Use local proxy peers and public visitor IPs in Cloudflare benchmark logs, and test that enrichment trusts their country headers. Advance daily compaction by two days so benchmark queries include daily rollups.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
citybreakdown dimension to traffic analytics, with coordinates, so Coolify can draw a city map.GEOIP_MAXMIND_EDITIONdefaults toGeoLite2-City.citydimension. Values areCC|subdivision|City(for exampleDE|BE|Berlin).GET .../traffic/breakdown/cityand the dashboards returnlat/lonon city entries. Other dimensions serialize as before.traffic_city_coordstable, written in the same transaction as the minute flush.Behaviour changes
cityrow, as an unknown country makes nocountryrow. A country-only database gives country data only.countrydimension still prefers a trustedCF-IPCountryheader, so the two can differ for one request.geoip.meta.jsonrecords the file, the source and the HTTP validators). With none, the lookup starts empty. The download runs in the background, and is retried with backoff while no database is loaded (15 minutes, doubled after each failure, up to 24 hours). On the first start, requests have no country or city until the download completes.TRAFFIC_IP_MODE=offrecords no city. At start, Sentinel deletes the stored city rows in this mode.traffic_city_coords. Coordinates that nocityrow uses are deleted, and the coordinates of a city update when the database changes.Resource usage
Benchmark (
sentinel-bench analytics --require-geoip --cf-header-pct 0 --events 1000000 --minutes 60, one machine):The benchmark now uses a pool of 2048 client addresses that resolve to cities, and reports city counts, coordinate rows and the GeoIP load time.
BENCHMARK.mddescribes this.How to test
cargo test --workspace --features trafficcargo test -p traffic -- --ignored geoip(uses the network)TRAFFIC_ENABLED=trueand an empty database directory. The API answers at once, and the log shows "geoip database loaded" a moment later./api/traffic/breakdown/city?from=...&to=.... City entries havelatandlon, and no value ends with||.