Repository navigation
fix and reordering of process to improve matching - #37
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the school standardization/matching pipeline to use Google “identity” (Place ID + current Google place name) earlier in the workflow to better bridge school renames, while also deprecating legacy DQA helpers in favor of the newer final_format_utils.R implementations.
Changes:
- Adds early Google identity resolution (
resolve_school_place_ids(),add_google_name_variants()) and usesplace_id+ Google-current-name as additional matching signals. - Refactors repeated multi-pass matching into a shared
run_matching_cascade()helper (including an exact-join “Pass 0” forplace_id). - Marks legacy DQA functions / wrappers as deprecated with updated roxygen + generated man pages; updates Places status lookup to also cache/return
google_place_name.
Reviewed changes
Copilot reviewed 21 out of 40 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/testthat/test-school_identity_utils.R | Adds unit tests for the new identity/rename matching behavior (no live API calls). |
| R/standardize_schools.R | Reorders pipeline to resolve Google identity earlier and writes rename QC output sooner. |
| R/school_identity_utils.R | New utilities to resolve place_id and attach standardized Google current names; adds rename report writer. |
| R/preprocessing_utils.R | Extracts shared name/county/level/district standardization into .standardize_school_source(). |
| R/matching_schools_utils.R | Introduces .run_exact_join_pass() and exported run_matching_cascade() for multi-pass matching. |
| R/googleapi_utils.R | Improves cache keying, expands Places lookup to include name, and propagates google_place_name through caching. |
| R/run_dqa_checks.R | Updates documentation to reflect deprecation of legacy DQA orchestration. |
| R/generate_dqa_summary.R | Updates documentation to reflect deprecation of legacy DQA summary generation. |
| R/dqa_checks.R | Adds file-level and per-function deprecation documentation mapping to new DQA checks. |
| R/clean_state_data.R | Documents deprecated status and why it remains separate from newer formatting utilities. |
| NAMESPACE | Exports new identity/matching helpers. |
| man/*.Rd (multiple) | Adds/updates generated documentation for new APIs and deprecations. |
| data/state_zip_prefix.csv | Adds CSV companion to existing dataset for ZIP-prefix filtering used in get_geo_info(). |
Files not reviewed (15)
- man/add_google_name_variants.Rd: Generated file
- man/augment_with_google_name_variant.Rd: Generated file
- man/build_reference_key.Rd: Generated file
- man/check_coverage_outliers.Rd: Generated file
- man/check_duplicates.Rd: Generated file
- man/check_enrollment_deviation.Rd: Generated file
- man/check_exceeding_enrollment_values.Rd: Generated file
- man/check_extreme_outliers.Rd: Generated file
- man/check_negative_values.Rd: Generated file
- man/check_vaccination_deviation.Rd: Generated file
- man/clean_state_data.Rd: Generated file
- man/dot-run_exact_join_pass.Rd: Generated file
- man/dot-standardize_school_source.Rd: Generated file
- man/generate_detailed_dqa_report.Rd: Generated file
- man/generate_dqa_summary.Rd: Generated file
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…aude-written, so error checking might be needed
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements and clarifications to data quality assurance (DQA) and Google Places API integration in the codebase. The most significant changes include deprecating legacy DQA functions in favor of newer, dplyr-based implementations, and enhancing the Google Places API utilities to better handle school renames and cache key collisions. Below is a summary of the most important changes:
Deprecation and Documentation of Legacy DQA Functions:
R/dqa_checks.RandR/generate_dqa_summary.Ras deprecated, with clear documentation pointing to their modern replacements infinal_format_utils.R. This helps guide developers to use the current, supported pipeline and clarifies the status of legacy code. [1] [2] [3] [4] [5] [6] [7] [8] [9]clean_state_data()to indicate its deprecated status and explain differences from the preferred utility functions.Google Places API and Caching Improvements:
load_and_merge_cache()(inR/googleapi_utils.R) to use bothschool_nameandcounty_stdwhen available, preventing collisions for schools with the same name in different counties. [1] [2] [3]get_business_status()toget_place_details(), which now returns both the business status and the current Google name for a place. This supports more robust detection of school renames and maintains backward compatibility with a deprecated wrapper. [1] [2]google_place_namefield, handling older cache files gracefully and ensuring all relevant columns are present in the merged result. [1] [2] [3]NAMESPACE and Export Updates:
These changes collectively improve code clarity, maintainability, and robustness, especially around DQA and geocoding workflows.