Skip to content

Use Codspeed for testing #24

Description

@bifurcation

Codspeed is a performance testing tool designed to produce highly consistent performance measurements. It also integrates very cleanly with criterion.rs.

I prototyped this out in bifurcation#1. (Unfortunately, that also includes a bunch of noise due to main branch divergence. The main commit is bifurcation@30a1c59.) In addition to those code changes, an admin for the RustCrypto org would have to enable Codspeed for the repo and provision the required secret.

I'm having a little trouble having confidence in the measurements, though. Looking at bifurcation#3 and its Codspeed report, it is measurably slower on my local machine, but Codspeed reports marginal performance improvements. The execution times are also off by about an order of magnitude.

keygen                  time:   [37.873 µs 37.899 µs 37.935 µs]
                        change: [+1.6945% +2.8309% +3.4566%] (p = 0.00 < 0.05)
                        Performance has regressed.

encapsulate             time:   [36.971 µs 37.129 µs 37.401 µs]
                        change: [+1.4643% +1.7390% +2.1480%] (p = 0.00 < 0.05)
                        Performance has regressed.

decapsulate             time:   [45.140 µs 45.368 µs 45.625 µs]
                        change: [+0.5023% +0.9468% +1.4248%] (p = 0.00 < 0.05)
                        Change within noise threshold.

round_trip              time:   [109.86 µs 110.04 µs 110.27 µs]
                        change: [+1.3724% +1.8421% +2.2354%] (p = 0.00 < 0.05)
                        Performance has regressed.

The version in that prototype is pretty slow, because it builds everything afresh, as @newpavlov noted in the accidental PR #20. If we're going to do this, we should do something like @newpavlov suggests to make it faster.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions