build: migrate from Conan + vendored deps to vcpkg manifest mode - #7487
Merged
Conversation
3 tasks
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
4 times, most recently
from
April 17, 2026 18:20
4131e76 to
43d6379
Compare
Contributor
|
can only say kudos :) I've been on/off working with vcpkg over quite some time as well and in 2026 it's definitely worth migrating. so much better than conan for most scenarios. I didn't look: it's still optional right and fall back to system dependencies? in case the docker optimizations are more scattered around, maybe it'd make sense to do that as a separate PR? |
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
5 times, most recently
from
April 20, 2026 09:45
c1aaf8d to
2b67c3f
Compare
Collaborator
|
does the switch have an impact on static linking of the binaries? |
Contributor
Author
|
Yes, mostly in a good direction. Linux / macOS ( Boost, TBB, zlib, bz2, expat, lzma, zstd, lua, libxml2, osmium, protozero, flatbuffers runtime, abseil — all statically linked into the binary. This is actually more static than before: pre-vcpkg, TBB came from the system as Windows ( |
Collaborator
|
Fully static binaries would be preferred. Tbb is a bit of a borderline case which could be static or dynamic. |
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
from
April 20, 2026 10:49
2b67c3f to
077e7bc
Compare
Contributor
Author
|
Switched Windows to TBB stays static (vcpkg builds it static on all our triplets now). Force-pushed, CI will tell us if anything breaks on Windows with the static linkage. |
Collaborator
|
Gave the PR a first round of scrutiny. Generally speaking I think this looks good. I will post a couple of comments later on a number of smaller touch ups and changes. Looks like we should be able to get this merged before the next monthly release. |
Contributor
Author
Sorry @nilsnolde I let claude code anwser and your second paragraph slipped through. Yes indeed the vcpkg integration is always optional. By default it will try to pick up the system packages. You may need to tweak a bit the cmake presets but it should be no problem |
3 tasks
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
from
April 20, 2026 16:39
077e7bc to
06a10df
Compare
DennisOSRM
reviewed
Apr 20, 2026
DennisOSRM
left a comment
DennisOSRM
left a comment
Collaborator
There was a problem hiding this comment.
left a first round of comments. I think this is already pretty close to being mergeable. It seems the major remaining piece is back porting the lua5.5 compatibility patch. Pinning old versions of sol2 and lua seems subpar. The other comments should be pretty easy to fix.
All in all, good work! We are close.
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
4 times, most recently
from
April 21, 2026 09:45
ca78f9a to
bb57f60
Compare
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
2 times, most recently
from
April 21, 2026 12:02
c25686f to
8611320
Compare
Collaborator
|
I gave this a test run on macOS 15 (X64) today using AppleClang 17. A couple of things seem to be not fully stable yet. The cmake bootstrapping runs at glacial pace. What used to be seconds is now a minute or more. Also, the bootstrapping does not complete. These issue block moving this PR further right now: |
Collaborator
|
The glacial pace issue seemed unrelated, and I was able to fix this. The RapidJSON issue remains. It could be something about platform triples, but not 100% sure. |
Contributor
Author
|
Hi @DennisOSRM I don't have a mac with me to test but I suspect you may not have enabled the vcpkg integration since RapidJSON should be comming straight from the vcpkg install and the mac os runners of github work. Can I ask of a few things:
This should hopefully clear things out. |
Collaborator
|
I was able to resolve the issue. In the end, it came down to a broken CMakeUserPresets.json in directory. Removed it, ran bootstrap using cmake —preset release, again, and then as able to build everything as expected. I had to remove locally installed boost, tho. There's a minor annoyance when building on macOS (arm as well as X64): This should not block us from moving forward, tho. The branch needs a final rebase/merge to fix the merge conflict, and then we are ready to go. 🚀 |
DennisOSRM
approved these changes
Apr 27, 2026
DennisOSRM
left a comment
DennisOSRM
left a comment
Collaborator
There was a problem hiding this comment.
good job. This PR is not an insignificant achievement. 👍 Thanks for the contribution.
Collaborator
|
There's a small issue on GCC 15.2.0 (Ubuntu 26.04) where LTO doesn't work in one of the unit tests. I had to circumvent this with the following patch: This should be the final change after resolving the merge conflict. |
Introduce vcpkg manifest mode as the single dependency management strategy. The manifest pins all dependencies with a baseline commit and a single version override for flatbuffers (25.9.23) to match the committed generated headers. sol2 comes from an overlay port, and Lua stays at the baseline — Lua 5.5 compat lives in the sol2 overlay as a patch. CMakePresets.json provides cross-platform build presets (release, debug, asan, CI variants for Linux/macOS/Windows) that wire up the vcpkg toolchain file automatically. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the dual ENABLE_CONAN/system-packages dependency resolution with a single vcpkg-based path. All find_package() calls now consume vcpkg CONFIG-mode packages or standard CMake modules. Key changes: - Remove ENABLE_CONAN option and entire if/else dependency block - Remove flatbuffers add_subdirectory (now a vcpkg package) - Remove vendored include_directories for libs moved to vcpkg - Add cmake/FindOsmium.cmake (extracted from third_party/libosmium) with patched protozero detection for vcpkg's header-only port - Wire up modern imported targets (Boost::*, TBB::tbb, etc.) - Harvest INTERFACE_INCLUDE_DIRECTORIES from header-only targets for legacy code that reads Boost/TBB include path variables - Update Node.js CMakeLists.txt to use TBB::tbb imported target Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Delete third_party/{flatbuffers,fmt,libosmium,protozero,rapidjson,sol2}
— all are now consumed as vcpkg packages. The remaining third_party/
contents (microtar, vtzero) stay vendored as they have no vcpkg port.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Delete conanfile.py — Conan is fully replaced by vcpkg manifest mode. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace all Conan install steps with lukka/run-vcpkg@v11 - Remove manual Boost, TBB, and system library install steps - Add vcpkg binary cache via GitHub Actions cache backend - Rename matrix entries from conan-* to vcpkg-* - Add vcpkg-smoke.yml for non-gating manifest resolution check - Update windows-build.bat to use cmake --preset ci-windows - Update UBSan suppressions for vcpkg include paths Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace apt-get/apk system dependency installation with vcpkg bootstrap in the builder stage. Key optimizations: - Split vcpkg install into its own layer (cached when manifest unchanged, source-only changes skip the expensive dep build) - BuildKit cache mounts for vcpkg archives, downloads, and buildtrees persist across docker build invocations on the host - Switch from Make to Ninja for lower scheduling overhead - Add ccache with a persistent cache mount for C++ compilation, dramatically speeding up rebuilds with small source changes - Keep VCPKG_ROOT=/vcpkg (outside /opt) so the runstage COPY doesn't drag the ~5 GB vcpkg tree into the final image Image size remains ~331 MB (debian) with only TBB shared libs copied to the runstage. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace Conan and system-package build instructions with vcpkg workflow in README and Windows dependency documentation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three vcpkg-related fixes bundled into the sol2 overlay port: 1. sol2 3.3.1 has noexcept specifiers on call() templates whose addresses are taken as lua_CFunction (int (*)(lua_State*)). On ARM64 clang-18 this causes "address of overloaded function does not match required type" errors. Patch them out and fix broken operator() calls that referenced the template without explicit arguments. 2. Port the Lua 5.5 compatibility shim from Project-OSRM#7404 into the overlay as lua55-compat.patch: widen the LUA_ERRGCMM removal shim to trigger on LUA_VERSION_NUM >= 504 (Lua 5.5 also lacks it), bump the supported- version gate from 504 to 505, and wrap lua_newstate so the extra Lua 5.5 seed argument is threaded through sol::state's constructor. Lets vcpkg pick current Lua 5.5 without the 5.4.8 pin. 3. file(GENERATE) for libosrm.pc fails on multi-config generators (MSVC) because $<TARGET_LINKER_FILE:...> produces different values per config. Skip pkgconfig generation for multi-config generators since it is not used on Windows. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two separate caching gaps were making every CI run rebuild all 113 ports from scratch: 1. The x-gha binary caching backend has been removed in the vcpkg version we pin (c3867e714). VCPKG_BINARY_SOURCES=clear;x-gha,readwrite was a no-op, vcpkg printed a warning and built everything from source. Switch to the files backend pointing at a workspace directory, and persist that directory across runs with actions/cache@v5. Cache key includes hashes of vcpkg.json, vcpkg-configuration.json, and the overlay ports tree so port changes invalidate. 2. Docker builds used BuildKit --mount=type=cache, which only persists on the same runner — on ephemeral GHA runners that means no cache hits. Replace raw 'docker build' with docker/build-push-action@v6 + buildx configured with cache-to/cache-from=type=gha. Scoped per base image so debian and alpine don't collide. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bookworm ships cmake 3.25, but vcpkg's lua 5.5.0 port now requires cmake_minimum_required(VERSION 3.31), so `docker build` fails at vcpkg's configure step for lua. Trixie ships cmake 3.31.6 and is the current stable Debian, which is the distro we support. This is a side-effect of dropping the lua 5.4.8 override in the vcpkg manifest — previously we pinned to a version whose port tolerated older cmake, so the Dockerfile's apt cmake (3.25) was enough. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Works around a GCC 15 false positive that breaks LTO for util-tests. Patch suggested by @DennisOSRM in PR review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
from
April 27, 2026 13:47
8611320 to
84fb898
Compare
Contributor
Author
|
Hey @DennisOSRM thanks for the feedback. I pushed another commit My employee is planning to use this more extensive so why not contribute back 😄 as a way to pay back for all the effort made into this |
4 tasks
3 tasks
tete17
added a commit
to tete17/osrm-backend
that referenced
this pull request
May 4, 2026
The script managed git-subtree updates for vendored dependencies under third_party/. After Project-OSRM#7487 (vcpkg migration), Project-OSRM#7488 (flatbuffers from vcpkg), Project-OSRM#7495 (microtar -> libarchive), and Project-OSRM#7508 (vtzero unvendored via custom vcpkg overlay), third_party/ is empty and every entry the script knew how to update is now provided by vcpkg. Nothing left to manage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrate osrm-backend's dependency management from the current three-path system (Conan 2.x + vendored
third_party/sources + system packages) to a single vcpkg manifest-mode setup. One dependency path on every platform, vendored copies deleted, CMakeLists simplified.Why vcpkg over the status quo
The current setup has three parallel dependency paths that have diverged over time:
ENABLE_CONANgating a separate CMakeLists branchapt-get,apk) for Linux CI and Docker, with custom Find modulesthird_party/(flatbuffers, fmt, sol2, rapidjson, protozero, libosmium) as a fallbackThis creates maintenance burden: dependency version bumps require touching 3+ places, the
if(ENABLE_CONAN)/else()block in CMakeLists.txt has subtle differences (e.g., Lua 5.2 vs 5.4), and the vendored copies are perpetually stale.vcpkg manifest mode collapses this into:
vcpkg.json— single source of truth for all dependency versionsvcpkg-configuration.json— pinned baseline for reproducibilityCMakePresets.json— cross-platform build configurationsfind_package()path in CMakeLists.txt — no moreENABLE_CONANbranchingWhy vcpkg specifically
x-gha,readwrite) — CI doesn't rebuild boost from source on every runfind_package(), no wrapper scripts neededWhat changes
build: add vcpkg manifest...vcpkg.json+vcpkg-configuration.json+CMakePresets.jsonbuild: rewrite CMakeLists.txt...ENABLE_CONAN, addcmake/FindOsmium.cmakebuild: remove vendored third_party...third_party/{flatbuffers,fmt,libosmium,protozero,rapidjson,sol2}(−360k lines)build: remove Conanconanfile.pyci: migrate CI workflows...lukka/run-vcpkg@v11, addvcpkg-smoke.ymlbuild(docker): rewrite Dockerfiles...docs: update build instructionsVersion pins (overrides in vcpkg.json)
usertype_containerwithunordered_map<string,bool>LUA_ERRGCMM, incompatible with sol2 3.3.1generated/are pinned to this versionDocker build optimizations
COPY vcpkg.json→vcpkg install→COPY . /srcso source-only changes skip the expensive dep layerWhat stays vendored
third_party/microtar/— single .c OBJECT library, no vcpkg portthird_party/vtzero/— not in the vcpkg baseline usedTest plan
--helpsmoke test)osrm-routed --helpexits 0)npm install && npm test)🤖 Generated with Claude Code