Efficiency improvements - #120
Open
khoroshevskyi wants to merge 6 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes:
get_stats()with a TTL to avoid running uncached COUNT queries on the bed table on every request to hot API paths (stats, neighbours, list, search)get_neighbours()by fetching all neighbour metadata in a single batched query (with annotations eager-loaded) instead of one query per neighbour; stale Qdrant points are now skipped rather than raisingBedAgentBedSet.get_ids_list(): it was refetching each bedset by id and lazy-loading its full bedfile membership just to build the list page. Now builds results directly from the paginated query;bed_idsis left unpopulated on list results (useget(identifier)for a single bedset's member ids)get_detailed_stats()no longer reuses aSessionafter itswithblock has closed (was forcing 3 extra connection checkouts for_stats_comments/_stats_geo_status/_get_geo_stats); all queries now share one session/transactionbed_files_info()call insideget_detailed_stats()with a targeted 3-column query, avoiding a full-tableFileInfoPydantic construction (with per-row try/except) for every bed record just to extractnumber_of_regions/mean_region_width/file_sizefor histogram binningBedAgentBedFile.get_ids_list()(backs/bed/list) had noorder_by()on its paginated query, so row order across pages was undefined -- rows could be duplicated or skipped between requests. Now orders byBed.id.get_detailed_stats()crashed with a pydanticValidationErrorwheneverbed_compliance,data_format,genome_alias,species_name,assay, orcell_linehad NULL rows: theGROUP BYqueries included the NULL group, producing aNonedict key, whichFileStats'sdict[str, int]fields reject. All six queries now filter out NULLs before grouping.bedfile_countcolumn tobedsets, exposed asBedSetMetadata.bedfile_count. Set once at bedset creation time (membership is write-once;add_bedfile/delete_bedfileare unimplemented), so reads never need to touchbedfile_bedset_relationto know a bedset's size. Requires a DB migration -- seescripts/migrations/2026_07_31_add_bedset_bedfile_count.sqlTODO:
__version__.pyfile