Skip to content

Reject zero benchmark resamples on the command line #3218

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
r4venrudy wants to merge 1 commit into
base: devel
Choose a base branch
from
Open

Reject zero benchmark resamples on the command line #3218

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/command-line.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -498,10 +498,10 @@ Per sample a clock resolution dependent number of iterations of the user code is

After the measurements are performed, statistical [bootstrapping] is performed
on the samples. The number of resamples for that bootstrapping is configurable
but defaults to 100000. Due to the bootstrapping it is possible to give
estimates for the mean and standard deviation. The estimates come with a lower
bound and an upper bound, and the confidence interval (which is configurable but
defaults to 95%).
but defaults to 100000 and must be greater than zero. Due to the bootstrapping
it is possible to give estimates for the mean and standard deviation. The
estimates come with a lower bound and an upper bound, and the confidence
interval (which is configurable but defaults to 95%).

[bootstrapping]: http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29

Expand Down
15 changes: 14 additions & 1 deletion src/catch2/internal/catch_commandline.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,19 @@ namespace Catch {
config.benchmarkSamples = *parsedSamples;
return ParserResult::ok( ParseResultType::Matched );
};
auto const setBenchmarkResamples = [&]( std::string const& resamples ) {
auto parsedResamples = parseUInt( resamples );
if ( !parsedResamples ) {
return ParserResult::runtimeError(
"Could not parse '" + resamples + "' as benchmark resamples" );
}
if ( *parsedResamples == 0 ) {
return ParserResult::runtimeError(
"Benchmark resamples must be greater than 0" );
}
config.benchmarkResamples = *parsedResamples;
return ParserResult::ok( ParseResultType::Matched );
};

auto const setShardIndex = [&](std::string const& shardIndex) {
auto parsedIndex = parseUInt( shardIndex );
Expand Down Expand Up @@ -346,7 +359,7 @@ namespace Catch {
| Opt( setBenchmarkSamples, "samples" )
["--benchmark-samples"]
( "number of samples to collect (default: 100)" )
| Opt( config.benchmarkResamples, "resamples" )
| Opt( setBenchmarkResamples, "resamples" )
["--benchmark-resamples"]
( "number of resamples for the bootstrap (default: 100000)" )
| Opt( config.benchmarkConfidenceInterval, "confidence interval" )
Expand Down
8 changes: 7 additions & 1 deletion tests/SelfTest/Baselines/compact.sw.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1614,6 +1614,12 @@ CmdLine.tests.cpp:<line number>: passed: !(result) for: !{?}
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Could not parse 'abc' as benchmark samples") for: "Could not parse 'abc' as benchmark samples" contains: "Could not parse 'abc' as benchmark samples"
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "--benchmark-resamples=20000" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.benchmarkResamples == 20000 for: 20000 (0x<hex digits>) == 20000 (0x<hex digits>)
CmdLine.tests.cpp:<line number>: passed: !(result) for: !{?}
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Benchmark resamples must be greater than 0") for: "Benchmark resamples must be greater than 0" contains: "Benchmark resamples must be greater than 0"
CmdLine.tests.cpp:<line number>: passed: config.benchmarkResamples == 100000 for: 100000 (0x<hex digits>) == 100000 (0x<hex digits>)
CmdLine.tests.cpp:<line number>: passed: !(result) for: !{?}
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Could not parse 'abc' as benchmark resamples") for: "Could not parse 'abc' as benchmark resamples" contains: "Could not parse 'abc' as benchmark resamples"
CmdLine.tests.cpp:<line number>: passed: config.benchmarkResamples == 100000 for: 100000 (0x<hex digits>) == 100000 (0x<hex digits>)
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "--benchmark-confidence-interval=0.99" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.benchmarkConfidenceInterval == Catch::Approx(0.99) for: 0.98999999999999999
==
Expand Down Expand Up @@ -3031,6 +3037,6 @@ InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 455 | 335 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2438 | 2237 passed | 158 failed | 43 failed as expected
assertions: 2444 | 2243 passed | 158 failed | 43 failed as expected


8 changes: 7 additions & 1 deletion tests/SelfTest/Baselines/compact.sw.multi.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1612,6 +1612,12 @@ CmdLine.tests.cpp:<line number>: passed: !(result) for: !{?}
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Could not parse 'abc' as benchmark samples") for: "Could not parse 'abc' as benchmark samples" contains: "Could not parse 'abc' as benchmark samples"
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "--benchmark-resamples=20000" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.benchmarkResamples == 20000 for: 20000 (0x<hex digits>) == 20000 (0x<hex digits>)
CmdLine.tests.cpp:<line number>: passed: !(result) for: !{?}
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Benchmark resamples must be greater than 0") for: "Benchmark resamples must be greater than 0" contains: "Benchmark resamples must be greater than 0"
CmdLine.tests.cpp:<line number>: passed: config.benchmarkResamples == 100000 for: 100000 (0x<hex digits>) == 100000 (0x<hex digits>)
CmdLine.tests.cpp:<line number>: passed: !(result) for: !{?}
CmdLine.tests.cpp:<line number>: passed: result.errorMessage(), ContainsSubstring("Could not parse 'abc' as benchmark resamples") for: "Could not parse 'abc' as benchmark resamples" contains: "Could not parse 'abc' as benchmark resamples"
CmdLine.tests.cpp:<line number>: passed: config.benchmarkResamples == 100000 for: 100000 (0x<hex digits>) == 100000 (0x<hex digits>)
CmdLine.tests.cpp:<line number>: passed: cli.parse({ "test", "--benchmark-confidence-interval=0.99" }) for: {?}
CmdLine.tests.cpp:<line number>: passed: config.benchmarkConfidenceInterval == Catch::Approx(0.99) for: 0.98999999999999999
==
Expand Down Expand Up @@ -3020,6 +3026,6 @@ InternalBenchmark.tests.cpp:<line number>: passed: q3 == 23. for: 23.0 == 23.0
Misc.tests.cpp:<line number>: passed:
Misc.tests.cpp:<line number>: passed:
test cases: 455 | 335 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2438 | 2237 passed | 158 failed | 43 failed as expected
assertions: 2444 | 2243 passed | 158 failed | 43 failed as expected


2 changes: 1 addition & 1 deletion tests/SelfTest/Baselines/console.std.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1744,5 +1744,5 @@ due to unexpected exception with message:

===============================================================================
test cases: 455 | 353 passed | 76 failed | 7 skipped | 19 failed as expected
assertions: 2416 | 2237 passed | 136 failed | 43 failed as expected
assertions: 2422 | 2243 passed | 136 failed | 43 failed as expected

50 changes: 49 additions & 1 deletion tests/SelfTest/Baselines/console.sw.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10714,6 +10714,54 @@ CmdLine.tests.cpp:<line number>: PASSED:
with expansion:
20000 (0x<hex digits>) == 20000 (0x<hex digits>)

-------------------------------------------------------------------------------
Process can be configured on command line
Benchmark options
resamples must be greater than zero
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................

CmdLine.tests.cpp:<line number>: PASSED:
CHECK_FALSE( result )
with expansion:
!{?}

CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("Benchmark resamples must be greater than 0") )
with expansion:
"Benchmark resamples must be greater than 0" contains: "Benchmark resamples
must be greater than 0"

CmdLine.tests.cpp:<line number>: PASSED:
CHECK( config.benchmarkResamples == 100000 )
with expansion:
100000 (0x<hex digits>) == 100000 (0x<hex digits>)

-------------------------------------------------------------------------------
Process can be configured on command line
Benchmark options
resamples must be parseable
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................

CmdLine.tests.cpp:<line number>: PASSED:
CHECK_FALSE( result )
with expansion:
!{?}

CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("Could not parse 'abc' as benchmark resamples") )
with expansion:
"Could not parse 'abc' as benchmark resamples" contains: "Could not parse
'abc' as benchmark resamples"

CmdLine.tests.cpp:<line number>: PASSED:
CHECK( config.benchmarkResamples == 100000 )
with expansion:
100000 (0x<hex digits>) == 100000 (0x<hex digits>)

-------------------------------------------------------------------------------
Process can be configured on command line
Benchmark options
Expand Down Expand Up @@ -20328,5 +20376,5 @@ Misc.tests.cpp:<line number>: PASSED:

===============================================================================
test cases: 455 | 335 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2438 | 2237 passed | 158 failed | 43 failed as expected
assertions: 2444 | 2243 passed | 158 failed | 43 failed as expected

50 changes: 49 additions & 1 deletion tests/SelfTest/Baselines/console.sw.multi.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10712,6 +10712,54 @@ CmdLine.tests.cpp:<line number>: PASSED:
with expansion:
20000 (0x<hex digits>) == 20000 (0x<hex digits>)

-------------------------------------------------------------------------------
Process can be configured on command line
Benchmark options
resamples must be greater than zero
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................

CmdLine.tests.cpp:<line number>: PASSED:
CHECK_FALSE( result )
with expansion:
!{?}

CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("Benchmark resamples must be greater than 0") )
with expansion:
"Benchmark resamples must be greater than 0" contains: "Benchmark resamples
must be greater than 0"

CmdLine.tests.cpp:<line number>: PASSED:
CHECK( config.benchmarkResamples == 100000 )
with expansion:
100000 (0x<hex digits>) == 100000 (0x<hex digits>)

-------------------------------------------------------------------------------
Process can be configured on command line
Benchmark options
resamples must be parseable
-------------------------------------------------------------------------------
CmdLine.tests.cpp:<line number>
...............................................................................

CmdLine.tests.cpp:<line number>: PASSED:
CHECK_FALSE( result )
with expansion:
!{?}

CmdLine.tests.cpp:<line number>: PASSED:
REQUIRE_THAT( result.errorMessage(), ContainsSubstring("Could not parse 'abc' as benchmark resamples") )
with expansion:
"Could not parse 'abc' as benchmark resamples" contains: "Could not parse
'abc' as benchmark resamples"

CmdLine.tests.cpp:<line number>: PASSED:
CHECK( config.benchmarkResamples == 100000 )
with expansion:
100000 (0x<hex digits>) == 100000 (0x<hex digits>)

-------------------------------------------------------------------------------
Process can be configured on command line
Benchmark options
Expand Down Expand Up @@ -20317,5 +20365,5 @@ Misc.tests.cpp:<line number>: PASSED:

===============================================================================
test cases: 455 | 335 passed | 96 failed | 6 skipped | 18 failed as expected
assertions: 2438 | 2237 passed | 158 failed | 43 failed as expected
assertions: 2444 | 2243 passed | 158 failed | 43 failed as expected

4 changes: 3 additions & 1 deletion tests/SelfTest/Baselines/junit.sw.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuitesloose text artifact
>
<testsuite name="<exe-name>" errors="17" failures="141" skipped="12" tests="2450" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<testsuite name="<exe-name>" errors="17" failures="141" skipped="12" tests="2456" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties>
<property name="random-seed" value="1"/>
<property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/>
Expand Down Expand Up @@ -984,6 +984,8 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/samples must be greater than zero" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/samples must be parseable" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/resamples" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/resamples must be greater than zero" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/resamples must be parseable" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/confidence-interval" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/no-analysis" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/warmup-time" time="{duration}" status="run"/>
Expand Down
4 changes: 3 additions & 1 deletion tests/SelfTest/Baselines/junit.sw.multi.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
<testsuite name="<exe-name>" errors="17" failures="141" skipped="12" tests="2450" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<testsuite name="<exe-name>" errors="17" failures="141" skipped="12" tests="2456" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
<properties>
<property name="random-seed" value="1"/>
<property name="filters" value="&quot;*&quot; ~[!nonportable] ~[!benchmark] ~[approvals]"/>
Expand Down Expand Up @@ -983,6 +983,8 @@ at Message.tests.cpp:<line number>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/samples must be greater than zero" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/samples must be parseable" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/resamples" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/resamples must be greater than zero" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/resamples must be parseable" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/confidence-interval" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/no-analysis" time="{duration}" status="run"/>
<testcase classname="<exe-name>.global" name="Process can be configured on command line/Benchmark options/warmup-time" time="{duration}" status="run"/>
Expand Down
2 changes: 2 additions & 0 deletions tests/SelfTest/Baselines/sonarqube.sw.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,8 @@ at AssertionHandler.tests.cpp:<line number>
<testCase name="Process can be configured on command line/Benchmark options/samples must be greater than zero" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/samples must be parseable" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/resamples" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/resamples must be greater than zero" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/resamples must be parseable" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/confidence-interval" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/no-analysis" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/warmup-time" duration="{duration}"/>
Expand Down
2 changes: 2 additions & 0 deletions tests/SelfTest/Baselines/sonarqube.sw.multi.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,6 +132,8 @@ at AssertionHandler.tests.cpp:<line number>
<testCase name="Process can be configured on command line/Benchmark options/samples must be greater than zero" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/samples must be parseable" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/resamples" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/resamples must be greater than zero" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/resamples must be parseable" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/confidence-interval" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/no-analysis" duration="{duration}"/>
<testCase name="Process can be configured on command line/Benchmark options/warmup-time" duration="{duration}"/>
Expand Down
14 changes: 13 additions & 1 deletion tests/SelfTest/Baselines/tap.sw.approved.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2681,6 +2681,18 @@ ok {test-number} - cli.parse({ "test", "--benchmark-resamples=20000" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.benchmarkResamples == 20000 for: 20000 (0x<hex digits>) == 20000 (0x<hex digits>)
# Process can be configured on command line
ok {test-number} - !(result) for: !{?}
# Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("Benchmark resamples must be greater than 0") for: "Benchmark resamples must be greater than 0" contains: "Benchmark resamples must be greater than 0"
# Process can be configured on command line
ok {test-number} - config.benchmarkResamples == 100000 for: 100000 (0x<hex digits>) == 100000 (0x<hex digits>)
# Process can be configured on command line
ok {test-number} - !(result) for: !{?}
# Process can be configured on command line
ok {test-number} - result.errorMessage(), ContainsSubstring("Could not parse 'abc' as benchmark resamples") for: "Could not parse 'abc' as benchmark resamples" contains: "Could not parse 'abc' as benchmark resamples"
# Process can be configured on command line
ok {test-number} - config.benchmarkResamples == 100000 for: 100000 (0x<hex digits>) == 100000 (0x<hex digits>)
# Process can be configured on command line
ok {test-number} - cli.parse({ "test", "--benchmark-confidence-interval=0.99" }) for: {?}
# Process can be configured on command line
ok {test-number} - config.benchmarkConfidenceInterval == Catch::Approx(0.99) for: 0.98999999999999999 == Approx( 0.98999999999999999 )
Expand Down Expand Up @@ -4895,5 +4907,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
ok {test-number} -
# xmlentitycheck
ok {test-number} -
1..2450
1..2456

Loading