fix raft_repl_dev UT by JacksonYao287 · Pull Request #756 · eBay/HomeStore

JacksonYao287 · 2025-06-24T06:14:34Z

1 enable RaftReplDev test and RaftReplDevDynamic test , which are essential gates for raft_repl_dev.
2 add code to avoid generating duplicated test req , which will fail raft_repl_dev test.
3 fix bug when stopping repl_service_timers

codecov-commenter · 2025-06-24T06:53:01Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 92.59259% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 66.68%. Comparing base (1a0cef8) to head (7627091).
⚠️ Report is 272 commits behind head on master.

Files with missing lines	Patch %	Lines
src/lib/blkalloc/append_blk_allocator.cpp	0.00%	1 Missing ⚠️
src/lib/replication/repl_dev/raft_repl_dev.cpp	0.00%	1 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #756       +/-   ##
===========================================
+ Coverage   56.51%   66.68%   +10.17%     
===========================================
  Files         108      110        +2     
  Lines       10300    13040     +2740     
  Branches     1402     1892      +490     
===========================================
+ Hits         5821     8696     +2875     
+ Misses       3894     3339      -555     
- Partials      585     1005      +420

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

yuwmao

LGTM, thanks for the fix.

JacksonYao287 · 2025-07-10T04:07:55Z

@sanebay can you help review this?

JacksonYao287 · 2025-10-13T09:19:52Z


-void RaftReplService::stop_reaper_thread() {
-    iomanager.run_on_wait(m_reaper_fiber, [] { iomanager.stop_io_loop(); });
+void RaftReplService::stop_repl_service_timers() {


we need make sure all the timers that may access repl_dev is stopped before we stop repl_dev, unless these timer might access a destroyed repl_dev and lead to crash

JacksonYao287 · 2025-10-24T13:52:10Z

ssl test will hit a grpc/openssl issue, which will result in the sanitizer check failure。
see this openssl/openssl#24892

fixed in this PR openssl/openssl@77e7a47

=================================================================
==4747==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 3616 byte(s) in 4 object(s) allocated from:
    #0 0x7f1a0b8b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x55c03bf8cf0d in CRYPTO_malloc crypto/mem.c:202
    #2 0x55c03bf8cf93 in CRYPTO_zalloc crypto/mem.c:222
    #3 0x55c03bf55174 in OSSL_ERR_STATE_new crypto/err/err_save.c:23
    #4 0x55c03bf53124 in ossl_err_get_state_int crypto/err/err.c:691
    #5 0x55c03bf52518 in ERR_clear_error crypto/err/err.c:339
    #6 0x55c03b68369a in grpc_core::DoSslWrite(ssl_st*, unsigned char*, unsigned long) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/tsi/ssl_transport_security_utils.cc:68
    #7 0x55c03b683b95 in grpc_core::SslProtectorProtectFlush(unsigned long&, unsigned char*, ssl_st*, bio_st*, unsigned char*, unsigned long*, unsigned long*) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/tsi/ssl_transport_security_utils.cc:184
    #8 0x55c03b67f7ee in ssl_protector_protect_flush /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/tsi/ssl_transport_security.cc:1073
    #9 0x55c03b684115 in tsi_frame_protector_protect_flush(tsi_frame_protector*, unsigned char*, unsigned long*, unsigned long*) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/tsi/transport_security.cc:114
    #10 0x55c03b956710 in endpoint_write /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/security/transport/secure_endpoint.cc:463
    #11 0x55c03b557292 in grpc_endpoint_write(grpc_endpoint*, grpc_slice_buffer*, grpc_closure*, void*, int) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/endpoint.cc:32
    #12 0x55c03b498b9c in write_action /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:999
    #13 0x55c03b4989da in write_action_begin_locked /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:964
    #14 0x55c03b556bb6 in grpc_combiner_continue_exec_ctx() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/combiner.cc:247
    #15 0x55c03b559894 in grpc_core::ExecCtx::Flush() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/exec_ctx.cc:75
    #16 0x55c03b23584c in grpc_core::ExecCtx::~ExecCtx() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/exec_ctx.h:117
    #17 0x55c03b4a2138 in operator() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2670
    #18 0x55c03b4a6b7c in __invoke_impl<void, finish_bdp_ping_locked(void*, grpc_error_handle)::<lambda()>&> /usr/local/include/c++/11.4.0/bits/invoke.h:61
    #19 0x55c03b4a6aaa in __invoke<finish_bdp_ping_locked(void*, grpc_error_handle)::<lambda()>&> /usr/local/include/c++/11.4.0/bits/invoke.h:96
    #20 0x55c03b4a6968 in invoke<finish_bdp_ping_locked(void*, grpc_error_handle)::<lambda()>&> /usr/local/include/c++/11.4.0/functional:97
    #21 0x55c03b4a66f9 in InvokeR<void, finish_bdp_ping_locked(void*, grpc_error_handle)::<lambda()>&> /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/abseil/20230802.1/_/_/package/958c3b440426d3de5c0972e7f507dbde401ac1a5/include/absl/functional/internal/any_invocable.h:132
    #22 0x55c03b4a6369 in LocalInvoker<false, void, finish_bdp_ping_locked(void*, grpc_error_handle)::<lambda()>&> /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/abseil/20230802.1/_/_/package/958c3b440426d3de5c0972e7f507dbde401ac1a5/include/absl/functional/internal/any_invocable.h:310
    #23 0x55c03b5253e7 in absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/abseil/20230802.1/_/_/package/958c3b440426d3de5c0972e7f507dbde401ac1a5/include/absl/functional/internal/any_invocable.h:868
    #24 0x55c03b52504f in grpc_event_engine::experimental::PosixEventEngine::ClosureData::Run() (/home/runner/.conan/data/homestore/6.20.31/_/_/build/d5f6cfc94ddb76ce7f01ea3f35c5fa8bac881a19/build/Sanitized/src/tests/test_raft_repl_dev+0xd38b04f)
    #25 0x55c03b542420 in operator() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/event_engine/thread_pool.cc:166
    #26 0x55c03b543152 in __invoke_impl<void, grpc_event_engine::experimental::ThreadPool::Run(grpc_event_engine::experimental::EventEngine::Closure*)::<lambda()>&> /usr/local/include/c++/11.4.0/bits/invoke.h:61
    #27 0x55c03b54312f in __invoke<grpc_event_engine::experimental::ThreadPool::Run(grpc_event_engine::experimental::EventEngine::Closure*)::<lambda()>&> /usr/local/include/c++/11.4.0/bits/invoke.h:96
    #28 0x55c03b54310c in invoke<grpc_event_engine::experimental::ThreadPool::Run(grpc_event_engine::experimental::EventEngine::Closure*)::<lambda()>&> /usr/local/include/c++/11.4.0/functional:97
    #29 0x55c03b5430cd in InvokeR<void, grpc_event_engine::experimental::ThreadPool::Run(grpc_event_engine::experimental::EventEngine::Closure*)::<lambda()>&> /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/abseil/20230802.1/_/_/package/958c3b440426d3de5c0972e7f507dbde401ac1a5/include/absl/functional/internal/any_invocable.h:132

Direct leak of 1808 byte(s) in 2 object(s) allocated from:
    #0 0x7f1a0b8b4887 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x55c03bf8cf0d in CRYPTO_malloc crypto/mem.c:202
    #2 0x55c03bf8cf93 in CRYPTO_zalloc crypto/mem.c:222
    #3 0x55c03bf55174 in OSSL_ERR_STATE_new crypto/err/err_save.c:23
    #4 0x55c03bf53124 in ossl_err_get_state_int crypto/err/err.c:691
    #5 0x55c03bf52518 in ERR_clear_error crypto/err/err.c:339
    #6 0x55c03b68369a in grpc_core::DoSslWrite(ssl_st*, unsigned char*, unsigned long) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/tsi/ssl_transport_security_utils.cc:68
    #7 0x55c03b683b95 in grpc_core::SslProtectorProtectFlush(unsigned long&, unsigned char*, ssl_st*, bio_st*, unsigned char*, unsigned long*, unsigned long*) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/tsi/ssl_transport_security_utils.cc:184
    #8 0x55c03b67f7ee in ssl_protector_protect_flush /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/tsi/ssl_transport_security.cc:1073
    #9 0x55c03b684115 in tsi_frame_protector_protect_flush(tsi_frame_protector*, unsigned char*, unsigned long*, unsigned long*) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/tsi/transport_security.cc:114
    #10 0x55c03b956710 in endpoint_write /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/security/transport/secure_endpoint.cc:463
    #11 0x55c03b557292 in grpc_endpoint_write(grpc_endpoint*, grpc_slice_buffer*, grpc_closure*, void*, int) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/endpoint.cc:32
    #12 0x55c03b498b9c in write_action /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:999
    #13 0x55c03b4989da in write_action_begin_locked /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:964
    #14 0x55c03b556bb6 in grpc_combiner_continue_exec_ctx() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/combiner.cc:247
    #15 0x55c03b559894 in grpc_core::ExecCtx::Flush() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/exec_ctx.cc:75
    #16 0x55c03b55a347 in grpc_core::Executor::RunClosures(char const*, grpc_closure_list) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/executor.cc:130
    #17 0x55c03b55ad4a in grpc_core::Executor::ThreadMain(void*) /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/iomgr/executor.cc:246
    #18 0x55c03ba4f60e in operator() /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/gprpp/posix/thd.cc:145
    #19 0x55c03ba4f643 in _FUN /home/conan/workspace/prod-v1/bsr/112691/dafef/.conan/data/grpc/1.54.3/_/_/build/62d231655e163ca40d6b92436e3e76c3872c41c7/src/src/core/lib/gprpp/posix/thd.cc:150
    #20 0x7f1a0a894ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)

SUMMARY: AddressSanitizer: 5424 byte(s) leaked in 6 allocation(s).

to fix this issue, we need to upgrade openssl to at least 3.4.0. will use a separate PR to upgrade openssl version and then enable this UT

xiaoxichen · 2025-10-27T09:28:47Z

+               // for now , we only support append blk allocator for replication service
+               // TODO: let raft_repl_dev supports other blk allocator type after we support deserializing multiple
+               // multiblkid from log entry, see
+               // https://github.com/JacksonYao287/HomeStore/blob/727699625a4699efb222ebed9d6bc2887875aa42/src/lib/replication/repl_dev/raft_state_machine.cpp#L89


fix this link please

JacksonYao287 requested review from sanebay and yuwmao and removed request for sanebay June 24, 2025 06:23

JacksonYao287 force-pushed the fix-repl-dev-UT branch from 7264580 to ff5219d Compare June 26, 2025 02:57

yuwmao previously approved these changes Jun 26, 2025

View reviewed changes

JacksonYao287 dismissed yuwmao’s stale review via 77a9741 June 26, 2025 06:44

JacksonYao287 force-pushed the fix-repl-dev-UT branch 2 times, most recently from 77a9741 to 61e0671 Compare June 26, 2025 06:46

JacksonYao287 requested a review from yuwmao June 26, 2025 06:49

JacksonYao287 force-pushed the fix-repl-dev-UT branch from 4d53f48 to 0d84836 Compare June 27, 2025 01:11

JacksonYao287 force-pushed the fix-repl-dev-UT branch from 0d84836 to 86829a4 Compare July 10, 2025 03:05

JacksonYao287 force-pushed the fix-repl-dev-UT branch 5 times, most recently from 8692b3c to 5178ae1 Compare September 29, 2025 14:11

JacksonYao287 force-pushed the fix-repl-dev-UT branch 7 times, most recently from b02b9da to 1d7c80f Compare October 13, 2025 09:16

JacksonYao287 commented Oct 13, 2025

View reviewed changes

JacksonYao287 force-pushed the fix-repl-dev-UT branch from 1d7c80f to e70f9b2 Compare October 16, 2025 00:00

JacksonYao287 force-pushed the fix-repl-dev-UT branch 2 times, most recently from 0c77ce1 to 0efff18 Compare October 23, 2025 03:44

JacksonYao287 force-pushed the fix-repl-dev-UT branch 5 times, most recently from 200838c to 00d3c07 Compare October 24, 2025 07:44

JacksonYao287 force-pushed the fix-repl-dev-UT branch 9 times, most recently from 610ed7c to d5f170c Compare October 27, 2025 07:37

fix raft_repl_dev UT

9c84cc1

JacksonYao287 force-pushed the fix-repl-dev-UT branch from d5f170c to 9c84cc1 Compare October 27, 2025 07:55

xiaoxichen reviewed Oct 27, 2025

View reviewed changes

fix link

7627091

JacksonYao287 requested a review from xiaoxichen October 28, 2025 01:52

xiaoxichen approved these changes Oct 28, 2025

View reviewed changes

xiaoxichen merged commit dd993a8 into eBay:master Oct 28, 2025
21 checks passed

JacksonYao287 deleted the fix-repl-dev-UT branch October 28, 2025 06:00

JacksonYao287 mentioned this pull request Oct 31, 2025

bump openssl version and enable WriteWithSLL UT for raft repl dev #825

Closed

Navigation Menu

Repository navigation

fix raft_repl_dev UT - #756

fix raft_repl_dev UT#756
xiaoxichen merged 2 commits into
eBay:masterfrom
JacksonYao287:fix-repl-dev-UT

JacksonYao287 commented Jun 24, 2025 •

edited

Loading

Uh oh!

codecov-commenter commented Jun 24, 2025 •

edited

Loading

Uh oh!

yuwmao left a comment

Uh oh!

JacksonYao287 commented Jul 10, 2025

Uh oh!

JacksonYao287 Oct 13, 2025 •

edited

Loading

Uh oh!

JacksonYao287 commented Oct 24, 2025 •

edited

Loading

Uh oh!

xiaoxichen Oct 27, 2025

Uh oh!

JacksonYao287 Oct 27, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

JacksonYao287 commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yuwmao left a comment

Choose a reason for hiding this comment

Uh oh!

JacksonYao287 commented Jul 10, 2025

Uh oh!

JacksonYao287 Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JacksonYao287 commented Oct 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xiaoxichen Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

JacksonYao287 Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JacksonYao287 commented Jun 24, 2025 •

edited

Loading

codecov-commenter commented Jun 24, 2025 •

edited

Loading

JacksonYao287 Oct 13, 2025 •

edited

Loading

JacksonYao287 commented Oct 24, 2025 •

edited

Loading