Repository navigation
Refactor CPU and GPU cache for faster memory allocation - #253
Conversation
The host-side `apply_smoothing!` from Trixi.jl reads the interface and mortar connectivity element by element, which only works with the CPU cache. Move the smoothing to the GPU, where `neighbor_ids` already lives: - The host indicator now only computes the raw values. - `calc_indicator_gpu!` copies them into a preallocated device buffer and runs the smoothing kernels, so the HG volume integral no longer allocates `alpha` on every RHS call. - The kernels use an atomic `max` and read only from the frozen `alpha_tmp`, so the result matches Trixi.jl's serial loop independent of thread order. The HG volume integral now passes `cache_gpu` to the indicator, so it no longer needs the CPU cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhLr3TjfMCfrH8o5zgUDc1
…yout The benchmark scripts of the target revision also measure the baseline revision, so they must work whether or not the semidiscretization still has a CPU cache: - Time the `semi`-level `rhs_hyperbolic!`, whose signature is the same in both revisions, instead of unpacking the caches. - Build the host element data for the initial input with Trixi directly. The node coordinates are the same as before, so the input hashes do not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhLr3TjfMCfrH8o5zgUDc1
…ample The example modified a host copy of the initial state and never wrote it back, so `ode.u0` on the GPU kept the default initial condition. Copy the modified state back to `ode.u0`, and read the node coordinates from the GPU cache instead of the CPU cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhLr3TjfMCfrH8o5zgUDc1
Nothing reads the CPU cache anymore: the HG indicator now works on the GPU cache, and `digest_boundary_conditions` does not read the cache for tree meshes. Drop it completely: - Remove the `cache_cpu` field, the `CacheCPU` type parameter, and the `initial_cache_cpu` keyword from `SemidiscretizationHyperbolicGPU`, and stop building Trixi's full CPU cache in the constructor. - Pass `cache_gpu` to `digest_boundary_conditions`. - Drop the `cache_cpu` argument from `calc_volume_integral!` and `rhs_hyperbolic!` and update their docstrings. - Update the tests and profiling scripts accordingly. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhLr3TjfMCfrH8o5zgUDc1
With the CPU cache gone there is only one cache, so follow Trixi.jl's naming: - `semi.cache_gpu` becomes `semi.cache`, and the `initial_cache_gpu` keyword becomes `initial_cache`. - The cache argument of `calc_volume_integral!`, `rhs_hyperbolic!`, and the other solver functions is now called `cache`, matching Trixi.jl's signatures. Tests and profiling scripts keep their local `cache_gpu` variable, since `cache` there already refers to Trixi.jl's CPU semidiscretization. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhLr3TjfMCfrH8o5zgUDc1
GPU RHS Benchmarks
Full benchmark report |
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
It would be better if benchmark cache initialization before / after could be given before merge. |
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Remove the empty line at the start of the function body, which JuliaFormatter flags in the style check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhLr3TjfMCfrH8o5zgUDc1
The RHS no longer wraps its arrays, since TrixiCUDA keeps the solution in the DG layout. Clean up what is no longer needed: - Allocate the initial state directly on the GPU in the DG layout instead of allocating a flat array, wrapping it, and copying it again with `CuArray`. - Drop the wrapping in `calc_error_norms`, which receives the integrator state in the DG layout. - Remove the GPU `wrap_array_native` method. Trixi.jl's save callbacks copy GPU states to the host first, so only the `Array` method is reached. - Remove the commented-out FDSBP `wrap_array` stub and outdated comments, and explain why the GPU `wrap_array` method is kept (Trixi.jl's step-size and analysis callbacks call it). - Update the `rhs_hyperbolic!` docstring to the `du, u` names and layout. - Pass `u` and `du` to the RHS in the benchmark instead of flat views, which the RHS no longer reshapes. Both compared revisions accept the DG layout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhLr3TjfMCfrH8o5zgUDc1
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The documentation cleanup in 211ce57 also removed the methods of `calc_volume_integral!`, `calc_interface_flux!`, `calc_boundary_flux!`, and `calc_mortar_flux!` that select `have_nonconservative_terms(equations)` themselves. `rhs_hyperbolic!` and the tests still call these methods, so every RHS call failed with a `MethodError`, starting at the volume integral. Restore the four methods without their docstrings, so the documentation cleanup stays as it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhLr3TjfMCfrH8o5zgUDc1
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
The benchmark results for before / after this PR. We mainly benchmarked the CPU and GPU memory allocation and runtime for three processes: 1. For simplicity, we only did this benchmark on basic linear advection cases (since the memory allocation is obviously improved in theory). |
|
The benchmark check in CI failed because the new benchmark suite does not match the original source code, and it is normal. Since we have already given out the benchmark results below, we could safely merge this PR by passing |
Closes #239.
Replaces the separate CPU and GPU caches with one GPU cache,
semi.cache, and removes redundant allocations and copies.cache_cpu,initial_cache_cpuandinitial_cache, and renamescache_gputocache. The RHS functions now takecachelike Trixi.jl.alphabuffers, so the indicator no longer needs the CPU cache.copyto!instead of allocating them twice, makes the structs immutable, keepscell_idson the host, and fixes the element type of the mortar arrays.const.wrap_array_native.