Conversation
Add a repeatable local performance harness for Cryfa compression and decompression runs. The harness supports configurable t=1/4/8 thread sweeps, default and stop-shuffle modes, round-trip verification, CSV/Markdown summaries, and before/after comparison reports. Wire it into the existing benchmark orchestrator and keep generated results out of git.
Add command-line options, environment overrides, interactive confirmation, and stale CMake cache handling to the Unix and Windows install scripts. Configure from the repository root, validate parallel job settings, and copy built executables back to the project root.
Add a clang-format configuration and normalize the C++ source layout. Refresh source headers, update diagnostic string construction to use std::format, clean up helper formatting code, and bump the displayed version to 26.05.
…environment variables
…ived state management and caching
…report output in run_local_perf.sh
…move author details Co-authored-by: Copilot <copilot@github.com>
There was a problem hiding this comment.
Pull request overview
This PR modernizes Cryfa’s build/runtime tooling and refactors the FASTA/FASTQ processing path to stream packed plaintext directly into encryption/decryption, while also introducing generated version metadata and a new local performance harness. In the broader codebase, it shifts core data handling away from temp-file-based packing/depacking toward ordered in-memory pipelines and updates install/benchmark documentation/scripts around that.
Changes:
- Refactor FASTA/FASTQ compression/decompression to use streaming encryption/decryption plus new ordered pipeline/plaintext stream helpers.
- Add generated version metadata, broader formatting/cleanup updates, and refresh install/build/CI-related scripts.
- Introduce a local performance harness and document new benchmarking workflow.
Reviewed changes
Copilot reviewed 33 out of 34 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/security.hpp |
Adds cached derived crypto state and streaming encrypt/decrypt APIs. |
src/security.cpp |
Implements streaming encryption/decryption, state caching, and unshuffle caching. |
src/parser.hpp |
Updates CLI parsing/help text and error formatting. |
src/keygen.cpp |
Minor cleanup/formatting updates in key generation flow. |
src/include/time.hpp |
Reworks time formatting helpers using std::format. |
src/include/string.hpp |
Refactors string/terminal formatting helpers. |
src/include/plaintext_stream.hpp |
New buffered plaintext handoff utility for streaming decrypt/decompress. |
src/include/ordered_pipeline.hpp |
New ordered worker pipeline for chunked processing. |
src/include/numeric.hpp |
Minor formatting cleanup in numeric helpers. |
src/include/file.hpp |
Updates file helper error formatting. |
src/include/assert.hpp |
Refines assertion/error helper messaging and formatting. |
src/fastq.hpp |
Minor declaration cleanup for FASTQ structures/APIs. |
src/fastq.cpp |
Replaces temp-file workflow with chunked ordered streaming FASTQ pack/unpack. |
src/fasta.hpp |
Minor declaration cleanup for FASTA structures/APIs. |
src/fasta.cpp |
Replaces temp-file workflow with chunked ordered streaming FASTA pack/unpack. |
src/endecrypto.hpp |
Cleans declarations and keeps shared packing/unpacking interfaces aligned. |
src/endecrypto.cpp |
Refactors packing lookups, chunk sizing, and temp-file join/shuffle helpers. |
src/def.hpp |
Introduces generated version header and splits I/O buffer vs chunk sizing constants. |
src/cryfa.cpp |
Refreshes main-file headers/comments only. |
src/application.hpp |
Header cleanup only. |
src/application.cpp |
Switches decrypt dispatch to peek decrypted type before selecting handler. |
scripts/runtime/run_local_perf.sh |
Adds new local benchmark/correctness harness and report generation. |
scripts/orchestration/benchmark_orchestrator.sh |
Hooks the new local perf harness into orchestration. |
scripts/config/par.sh |
Renames benchmark output directory from result to results. |
install.sh |
Reworks Unix install flow with prompts, cache reset handling, and copies instead of moves. |
install.ps1 |
Reworks Windows install flow with prompts and stale-cache handling. |
cmake/version.hpp.in |
Adds template for generated version header. |
bench_cryfa.sh |
Adds configuration knobs for the local performance harness. |
README.md |
Refreshes headings/badges and documents the new local performance harness. |
Dockerfile |
Minor cleanup of multi-stage Docker build definition. |
CMakeLists.txt |
Adds version generation and generated include path wiring. |
.gitignore |
Ignores generated results/ output. |
.github/workflows/ci.yml |
Narrows CI triggers to master branches. |
.clang-format |
Adds repository formatting config. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+285
to
+289
| DATASET="$dataset_dir/${base_name}_${TARGET_MB}mb_x${copies}.${ext}" | ||
|
|
||
| if [[ -f $DATASET && -f $DATASET.meta ]] && | ||
| grep -qx "copies=$copies" "$DATASET.meta"; then | ||
| log "Reusing cached generated dataset: $DATASET ($(format_mib "$(file_size_bytes "$DATASET")") MiB)" |
Comment on lines
+303
to
+323
| if (( copies == 1 )); then | ||
| cp "$INPUT" "$tmp" | ||
| else | ||
| perl -e ' | ||
| use strict; | ||
| use warnings; | ||
|
|
||
| my ($input, $copies, $output) = @ARGV; | ||
| open my $in, "<", $input or die "Unable to open $input: $!"; | ||
| binmode $in; | ||
| local $/; | ||
| my $chunk = <$in>; | ||
| close $in; | ||
|
|
||
| open my $out, ">", $output or die "Unable to open $output: $!"; | ||
| binmode $out; | ||
| for (1 .. $copies) { | ||
| print {$out} $chunk; | ||
| } | ||
| close $out or die "Unable to write $output: $!"; | ||
| ' "$INPUT" "$copies" "$tmp" |
Comment on lines
+157
to
+159
| Write-Log "Copying executables to the repository root" | ||
| Copy-Item "$BUILD\cryfa.exe" -Destination $Root -Force | ||
| Copy-Item "$BUILD\keygen.exe" -Destination $Root -Force |
Comment on lines
+180
to
+182
| log "Copying executables to the repository root" | ||
| cp "$BUILD/cryfa" "$ROOT_DIR/cryfa" | ||
| cp "$BUILD/keygen" "$ROOT_DIR/keygen" |
| For quick local performance and correctness checks, use the local harness: | ||
|
|
||
| ```sh | ||
| sh scripts/runtime/run_local_perf.sh --label local-check --input example/in.fq --target-mb 200 --threads "1 4 8" --runs 1 --modes both --no-prompt |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.