CNDB-17543: fix flaky cleanup timing assertion in FlushFailingOnNotificationSubscriberTest#2316
Open
CNDB-17543: fix flaky cleanup timing assertion in FlushFailingOnNotificationSubscriberTest#2316
Conversation
…clustering columns in descending order - port CASSANDRA-20295
The fix for CNDB-13591 was developed and only committed to the March 25 release branch commit e3dfead (HEAD -> cndb-13591-main, origin/cndb-13591-main) Author: Jacek Lewandowski <lewandowski.jacek@gmail.com> Date: Fri Apr 4 16:14:24 2025 +0200 CNDB-13591: Fix unit tests (missed commit in previous hotfix) (cherry picked from commit 7c51303) commit b1f732e Author: Enrico Olivelli <enrico.olivelli@datastax.com> Date: Fri Apr 4 14:18:30 2025 +0200 CNDB-13591: Error while loading chunkcache: java.lang.IllegalArgumentException: Negative position (#1677) Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com> (cherry picked from commit 09748bd) commit 4ff4879 Author: Branimir Lambov <branimir.lambov@datastax.com> Date: Fri Apr 4 14:42:26 2025 +0300 CNDB-13591: Go back to storing byte buffers in the cache (cherry picked from commit 7624a19) --------- Co-authored-by: Branimir Lambov <branimir.lambov@datastax.com> Co-authored-by: Jacek Lewandowski <lewandowski.jacek@gmail.com>
…yzed expression (#1620) Memoize the analyzed tokens of the right operand of an analyzed expression. Also add some refactoring and simplifications around how analysis is dealt with in `RowFilter` and `Operator`.
…iew (#1700) Fixes: riptano/cndb#13693 The new design is to: 1. Make sai `View` reference-able so that it holds references to the underlying `SSTableIndex`s. When the `view` is released the final time, it releases the indexes. This moves all of the complexity of grabbing references to sstable update time and out of the query path, which seems like a generally good improvement. 2. Observe that `SSTableIndex` holds a reference to its associated sstable reader. Note also that we need to create the memtable index on deletions in order to make the index-first solution work. Also note that sstables indexes are added before they are removed on flush, so this change in design is safe for flush.
Renames confusing numRows arguments to totalPointCount in methods propagating the value to BKDWriter, which expects the total point count. Fixes to provide total point counts instead of row counts. In the most of tests the point count and the row count are the equivalent. One unused test is removed.
Fixes some IntelliJ warnings in the affected files.
This implements one of the possible fixes for riptano/cndb#12407. This PR skips loading the PrimaryKey's token in the case where the sstables/memtables do not overlap, which is particularly helpful as datasets become more compacted (especially after major compaction). It is implemented via a subtle change to the `PrimaryKeyWithSource` class that only loads the token info when the token is needed. We avoid this check by first checking to see if the sstable ranges overlap or if the key is contained in the sstable range. If it is not contained, we can short circuit the logic and avoid loading the primary key from disk. This results in a significant optimization for SAI hybrid queries that search-then-sort. CNDB PR with passing tests: riptano/cndb#12444 - [x] Make sure there is a PR in the CNDB project updating the Converged Cassandra version - [x] Use `NoSpamLogger` for log lines that may appear frequently in the logs - [x] Verify test results on Butler - [x] Test coverage for new/modified code is > 80% - [x] Proper code formatting - [x] Proper title for each commit staring with the project-issue number, like CNDB-1234 - [x] Each commit has a meaningful description - [x] Each commit is not very long and contains related changes - [x] Renames, moves and reformatting are in distinct commits
HCD requires custom authenticators enabled via a cqlsh plugin. Backports [CASSANDRA-16456](https://issues.apache.org/jira/browse/CASSANDRA-16456) to add support for `cqlsh` plugins. Adds `datastax_db_*-VERSION.zip` to the list of automatically loaded plugins (used by HCD only). --------- Co-authored-by: Bhouse99 <bhouse99@protonmail.com> Co-authored-by: Stefan Miklosovic <smiklosovic@apache.org>
Fix MAX_SEGMENT_SIZE < chunkSize in MmappedRegions::updateState Opportunistically, fixes some leaks in a test.
…hreshold or expiration period. Default disabled. (#1724)
… skip slowest replica if configured (#1734)
…t with different address (#1666) - add `IEndpointSnitch#filterByAffinityForWrite` and rename `filterByAffinity` to `filterByAffinityForReads`
- **Add failing test** - **CNDB-14153: Fix SAI updates (non-null solution)** ### What is the issue Fixes riptano/cndb#14153 ### What does this PR fix and why was it fixed This is meant as an alternative to #1749. It fixes riptano/cndb#14153 by never returning `null` from the `UpsertTransformer`. #1749 is a more memory efficient solution, but has additional complexity, which is why I am proposing this as an alternative.
### What is the issue Fixes riptano/cndb#14171 ### What does this PR fix and why was it fixed #1200 introduced a bug for SAI indexes version AA that have clustering columns. As the tests show, updates incorrectly removed rows from the index. We need the update logic for later versions of SAI, so it is key to keep the update feature, but AA does not support those features precisely because it only indexes the partition key, so this is a safe update.
Fixes: riptano/cndb#14160 The loop is supposed to loop until the deadline, not after the deadline. The test fails without the change.
This commit changes the approximate calculation of the average length of documents with queried terms to the calculation of the average length of all documents in a segment. The average length uses number of rows/documents and total number of terms in the documents. This PR changes obtaining this numbers from calculating them per query execution to calculating them during flushing and compacting and storing them in metadata. Thus the disk format is updated to new version 8 and `ED`. As the result new average length is applied from version `ED`, while older versions use the previous way of calculating it. The tests are added. This commit reduces code duplication related to BM25 sorting in TrieMemtableIndex. In the affected code explicit types replace vars in declarations, since they are prohibited in Apache Cassandra, see CASSANDRA-20389. Also few IDEA warnings are fixed in affected files: typos in comments and code simplification by removing unnecessary string builder.
…ytesCounted (#1735) Modify CQL counters calculation of rows data size to include cell deletions even after the row has been purged. That way, the counters will count the same size in bytes for filtered and unfiltered base partition/row iterators. This solves a bug where byte-based paging is wrongly considering replicas as exhausted if their responses contain tombstones. This leads to queries using byte-based paging returning fewer rows than expected. It includes all aggregation queries, where byte-based paging is always used internally. The problem is that replicas apply counters to unfiltered iterators, whereas the coordinator controls paging by applying counters to reconciled and filtered iterators. Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com> Co-authored-by: shunsaker <shunsaker@users.noreply.github.com>
…ore we run tests. The patch can be tested on InvertedIndexSearcherTest, for example.
… unloaded after tenant unassignment (#1766) When schema is unloaded after tenant unassignment, compaction task might finishes without corresponding index files, making index non-queryable. Replace `isValid` with `isDropped` and `isUnloaded`. If index is dropped, compaction task or index build can proceed without the index, same behavior as before. If index is unloaded, compaction task or index build will be aborted to avoid completing without index files. --- #1754 was reverted because[ CNDB PR](riptano/cndb#14179) compiled failed with wrong hash. Re-merge it again.
…ize (#1755) The data inserted into the trie in `TrieMemoryIndex` is `encodedTerm`, which is built from `term` and only on the "available" bytes (between `term.position()` and `term.limit()`). But the check that decides to use the (more efficient) recursive path or not uses `term.limit()` to assess the size of `encodedTerm`. If `term.position()` is not 0, this is incorrect, and can lead to using the less optimal pass completely unecessarily. This has been shown to happen when investigating riptano/cndb#14153: the non-recursive path was taken even for boolean values (because the nodes were using `offheap_buffers`; with `offheap_objects`, the buffers getting to `TriMemoryIndex` are 0-positioned). See riptano/cndb#14184.
Fixes: riptano/cndb#14167 We upgraded to jvector 4 too soon. We need to use jvector 2 for a release cycle and then when we upgrade next, we can go to jvector 4. We needed a two phase release. CNDB test PR: riptano/cndb#14196 Co-authored-by: Michael Marshall <michael.marshall@datastax.com>
…1763) - **CNDB-14210: Fix analyzed sai index on compound partition key column** ### What is the issue Fixes: riptano/cndb#14210 ### What does this PR fix and why was it fixed Fixes some queries that were broken by the march and may release. In #1434, we introduced some logic to help make the eq behavior better, and it incorrectly handled compound partition keys. This fixes that. The central fix is to use the `EQRestriction` any time we have a primary key column. This is necessary to ensure we can write and read data. The tests cover the relevant cases. I also fixed the error message returned when attempting to use `:` on a clustering column index. Please review the text of the error message.
Few indexes were created with execute method, which doesn't check if an index is ready. Changing it to createIndex fixes the observed flakiness.
…ion observer for composite compaction (#1767) CNDB: riptano/cndb#14203
Currently returning a null from an Upserter may corrupt the trie state which may end up in a serious problem. It is better to crash instead.
CNDB-13770 Separate timeout for aggregation queries In CC, aggregate user queries use the same range read timeout. In DSE, we had a separate timeout that defaulted to 120s. We would like to retain that functionality. This PR adds a separate 120s timeout for aggregate queries. The timeout is configurable with aggregation_request_timeout_in_ms Config parameter
PartitionAwarePrimaryKeyMap implements overcomplicated `ceiling` method calling `exactRowIdOrInvertedCeiling`. This commit Simplifies PartitionAwarePrimaryKeyMap.ceiling to use the corresponding correct method from the reader directly. This can be seen as a follow up to https://github.com/datastax/cassandra/pull/1096/files#diff-c5011580ab9b0d99d9e504570c4cccb152221d3dbe62c8a956e83fce9070b380 Rebase notes: - commit 48b55fb was dropped because it only contains lint/refactor changes and should instead be upstreamed
The rationale behind this upgrade is to remediate CVE-2025-67735. The BoringSSL has been upgraded as well for consistency with 4.1.130.Final. CNDB PR: riptano/cndb#16481
The current version of `lz4-java` (1.8.0) has known vulnerabilities, namely [CVE-2025-12183](https://www.cve.org/CVERecord?id=CVE-2025-12183) [CVE-2025-66566](https://www.cve.org/CVERecord?id=CVE-2025-66566) This patch updates the lz4-java library from 1.8.0 to 1.10.2 As of lz4-java 1.8.1, the artifact coordinates changed from `org.lz4` to `at.yawk.lz4`, so this patch also updates the artifact coordinates
All version of jackson-dataformat-msgpack before 0.9.11 contain a critical vulnerability - CVE-2026-21452. The jackson-dataformat-msgpack has been upgraded to 0.9.11. Given the mentioned library is built against jackson 2.18.4, to ensure the full compatibility jackson has been upgraded as well. CNDB PR: riptano/cndb#16480
…HADED_IO_NETTY_TRANSPORT_NONATIVE fixes DTest jar builds
``` CNDB-16350: Optimize ChronicleMap access, iteration to reduce serde cost ### What is the issue Fixes: riptano/cndb#16350 ### What does this PR fix and why was it fixed ChronicleMap gives us several ways to use lower level APIs to avoid deserializing keys/values and the associated allocation that comes with them. The first key thing to mention is that iteration is very expensive as these maps get big, so we want to avoid it if possible. The second is that if we use the typical map iteration methods, they deserialize the key and the value eagerly. Since the key is typically a high dimensional vector, it is valuable to avoid such deserialization. This change: * Removes unnecessary iteration leveraging the fact that compaction is additive * Replaces `forEach` with `forEachEntry`, which gives better semantics * Updates the `maybeAddVector` method to avoid serializing the vector key twice by using the `searchContext`. The `ChronicleMap#put` method uses this pattern internally I added two sets of benchmarks, however the `VectorCompactionBench` doesn't seem to register the benefit of the ChronicleMap. I am leaving `VectorCompactionBench` in place since it is still useful. Likely, this is because ChronicleMap's cost isn't as expensive as graph construction. Here are some of the benchmark results. They show between a 50x and 100x improvement. The improvement seems to increase as we build larger graphs. benchmark results before change: ``` [java] Benchmark (dimension) (numVectors) Mode Cnt Score Error Units [java] V5VectorPostingsWriterBench.createGenericIdentityMapping 768 100000 avgt 5 271.569 ± 3.473 ms/op [java] V5VectorPostingsWriterBench.createGenericIdentityMapping 768 1000000 avgt 5 5452.393 ± 227.905 ms/op [java] V5VectorPostingsWriterBench.createGenericIdentityMapping 1536 100000 avgt 5 1392.607 ± 30.388 ms/op [java] V5VectorPostingsWriterBench.createGenericIdentityMapping 1536 1000000 avgt 5 11496.696 ± 345.886 ms/op [java] V5VectorPostingsWriterBench.describeForCompactionOneToMany 768 100000 avgt 5 242.049 ± 20.708 ms/op [java] V5VectorPostingsWriterBench.describeForCompactionOneToMany 768 1000000 avgt 5 2365.691 ± 84.173 ms/op [java] V5VectorPostingsWriterBench.describeForCompactionOneToMany 1536 100000 avgt 5 265.395 ± 4.167 ms/op [java] V5VectorPostingsWriterBench.describeForCompactionOneToMany 1536 1000000 avgt 5 3641.557 ± 130.649 ms/op ``` after change: ``` [java] Benchmark (dimension) (numVectors) Mode Cnt Score Error Units [java] V5VectorPostingsWriterBench.createGenericIdentityMapping 768 100000 avgt 5 5.721 ± 1.727 ms/op [java] V5VectorPostingsWriterBench.createGenericIdentityMapping 768 1000000 avgt 5 124.536 ± 22.464 ms/op [java] V5VectorPostingsWriterBench.createGenericIdentityMapping 1536 100000 avgt 5 5.662 ± 0.610 ms/op [java] V5VectorPostingsWriterBench.createGenericIdentityMapping 1536 1000000 avgt 5 122.671 ± 3.343 ms/op [java] V5VectorPostingsWriterBench.describeForCompactionOneToMany 768 100000 avgt 5 5.364 ± 1.194 ms/op [java] V5VectorPostingsWriterBench.describeForCompactionOneToMany 768 1000000 avgt 5 119.449 ± 4.809 ms/op [java] V5VectorPostingsWriterBench.describeForCompactionOneToMany 1536 100000 avgt 5 5.379 ± 0.552 ms/op [java] V5VectorPostingsWriterBench.describeForCompactionOneToMany 1536 1000000 avgt 5 121.293 ± 3.040 ms/op ``` Co-authored-by: Michael Marshall <michael.marshall@datastax.com>
…ricsRegistry (#2221) ### What is the issue Fixes #13868 Tables cannot be created in UCS if the name contains the name of a UCS metric. ### What does this PR fix and why was it fixed Adds a microbenchmark to quantify the performance impact and removes the restriction.
### What is the issue RandomSchemaTest has a fixed seed ### What does this PR fix and why was it fixed Makes the test random again by removing the seed
…ows/tombstones (#2169) riptano/cndb#16180 Port into main-5.0 commit 4fc5df6 ``` CNDB-15946: Revamp SAI metrics for fetched/returned keys/partitions/rows/tombstones (#2132) The metrics partitionsRead, rowsFiltered, rowsPreFiltered and shadowedPrimaryKeyCount, which don't always work as expected, are replaced by these metrics: * keysFetched: Number of partition/row keys that will be used to fetch rows from the base table. * partitionsFetched: Number of live partitions fetched from the base table, before post-filtering and sorting. Note that currently ANN fetches one partition per index key, without any grouping of same-partition clusterings. * partitionsReturned: Number of live partitions returned to the coordinator, after post-filtering and sorting. * partitionTombstonesFetched: Number of deleted partitions found when reading the base table. * rowsFetched: Number of live rows fetched from the base table, before post-filtering and sorting. * rowsReturned: Number of live rows returned to the coordinator, after post-filtering and sorting. * rowTombstonesFetched: Number deleted rows or row ranges found when reading the base table. StorageAttachedIndexSearcher is modified to use the command timestamp, rather than getting the current time every time a query timestamp is needed, which was possibly buggy and inefficient. ``` Snapshot SAI metrics when the query is done --------- Co-authored-by: Andrés de la Peña <adelapena@users.noreply.github.com> Co-authored-by: Andrés de la Peña <a.penya.garcia@gmail.com>
…2230) ### What is the issue CorruptedSSTablesCompactionsTest.testCorruptedSSTablesWithUnifiedCompactionStrategy-cdc is failing ### What does this PR fix and why was it fixed Adds VIntOutOfRangeException to caught exceptions in SortedStringTableCursor
### What is the issue The pycode compliance test is failing due to trailing whitespace ### What does this PR fix and why was it fixed Removes the whitespace
### What is the issue When reading SSTables containing dropped columns with tuple types (or UDTs containing tuples), the column ordering is being corrupted during bitmap deserialization. ### What does this PR fix and why was it fixed Fixes dropped column handling with User-Defined Types (UDTs). Column ordering depends on isComplex(), which depends on the type's isMultiCell(). When a dropped column's type had a different isMultiCell() value in the schema vs the SSTable, column ordering became incorrect, causing bitmap decode errors and data corruption.
### What is the issue Fixes #16793 CC5 doesn't understand CC4 system tables and generates a new host_id on upgrade ### What does this PR fix and why was it fixed Reads CC4's file-based node metadata stored in MessagePack format and converts them to CC5 LocalInfo and PeerInfo objects on first boot.
…void JUnit test timeouts. (#2121) ### What is the issue CNDB-14023, ForceRepairTest fails sometimes in CI with a JUnit test timeout. ### What does this PR fix and why was it fixed Adds a timeout while waiting for node to be marked down instead of waiting indefinitely and raising a JUnit timeout.
### What is the issue UCS settings files are not dropped after the table gets dropped. Instead they are supposed to be cleared after the node restart. The cleanup is faulty though and it prevents the node from startup. Root Cause: The cleanupControllerConfig() method in CompactionManager attempts to verify if a table exists by calling getColumnFamilyStore(). When the table is dropped, this method throws IllegalArgumentException, which was not being caught. The existing catch block only handled NullPointerException (for missing keyspace). ### What does this PR fix and why was it fixed Extended the exception handler to catch both NullPointerException and IllegalArgumentException, allowing orphaned controller-config.JSON files to be properly identified and deleted during node restart. 5.0 counterpart of #2145.
Adds storage-compatibility guards for pre-5.0 mode around auth/role and system schema behavior, defers incompatible changes to NONE
### What is the issue This file is missing causing the loadCommitLogAndSSTablesWithDroppedColumnTestCC50 test to fail ### What does this PR fix and why was it fixed Adds the missing file.
### What is the issue CIDR authz is a 5.0 feature ### What does this PR fix and why was it fixed Gates CIDR setup by storage compatibility mode
### What is the issue TrieMemtableMetricsTest's byteman rule has the wrong target method ### What does this PR fix and why was it fixed Correctly uses the 'apply' method
### What does this PR fix and why was it fixed Adds a new artifact alongside the existing one.
…2261) ### What is the issue CNDB-17010 ### What does this PR fix and why was it fixed CC4 stored the memtable column in system_schema.tables as frozen<map<text, text>>, while CC5 uses text. During upgrades, binary-serialized map data is misinterpreted as UTF-8 text, causing memtable configurations to fall back to defaults.
### What is the issue CVE-2024-12798, CVE-2024-12801, CVE-2025-11226, CVE-2026-1225 ### What does this PR fix and why was it fixed Upgrades logback to 1.5.25
### What is the issue There are security advisories present for jackson 2.18.4 ### What does this PR fix and why was it fixed Upgrades jackson-core, jackson-databind, and jackson-annotations to 2.18.6
CNDB-17333: Create separate physical artifacts for db-all Maven coordinate
Update build.xml to create separate physical artifacts for db-all Maven
coordinate (com.datastax.db:db-all) alongside existing dse-db-all artifacts.
### What is the issue
CNDB-17333: Support publishing both db-all and dse-db-all artifacts
### What does this PR fix and why was it fixed
Creates separate physical `db-all` artifacts alongside existing `dse-db-all` artifacts to support gradual migration to new Maven coordinates.
**Changes to build.xml:**
1. **artifacts target** - Creates separate db-all artifacts:
- Copies `db-all` JARs and POMs from `dse-db` artifacts
- Creates separate `db-all-{version}-bin.tar.gz` from `${dist.dir}`
- Creates separate `db-all-{version}-src.tar.gz` from `${basedir}`
- Generates SHA-256 and SHA-512 checksums for all db-all tarballs
2. **publish target** - Signs both artifact sets:
- Signs `dse-db-all` tarballs (existing)
- Signs `db-all` tarballs (new)
**Artifacts created (example for version 5.0.4.0):**
- `dse-db-all-5.0.4.0.jar` + POM + sources
- `dse-db-all-5.0.4.0-bin.tar.gz` + checksums
- `dse-db-all-5.0.4.0-src.tar.gz` + checksums
- `db-all-5.0.4.0.jar` + POM + sources (copied from dse-db)
- `db-all-5.0.4.0-bin.tar.gz` + checksums (separate physical file)
- `db-all-5.0.4.0-src.tar.gz` + checksums (separate physical file)
**Deployment:**
Works with jenkins-pipeline-lib PR #254 which deploys artifacts under both Maven coordinates:
- `com.datastax.dse:dse-db-all:{version}`
- `com.datastax.db:db-all:{version}`
**Related PR:** riptano/jenkins-pipeline-lib#254
…icationSubscriberTest Reset lastTimePoolNeededCleaning after successful flushes so the cleanup gap measurement doesn't span flush boundaries.
Checklist before you submit for review
|
|
❌ Build ds-cassandra-pr-gate/PR-2316 rejected by Butler4 regressions found Found 4 new test failures
Found 2 known test failures |
michaelsembwever
force-pushed
the
main-5.0
branch
from
8f66239 to
c00cfa8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR fix and why was it fixed
Reset lastTimePoolNeededCleaning after successful flushes so the cleanup gap measurement doesn't span flush boundaries.