Skip to content

Build packages without installing them - #214

Merged
Lukas-Heiligenbrunner merged 13 commits into
Lukas-Heiligenbrunner:masterfrom
gyscos:build_not_install
Apr 24, 2026
Merged

Build packages without installing them#214
Lukas-Heiligenbrunner merged 13 commits into
Lukas-Heiligenbrunner:masterfrom
gyscos:build_not_install

Conversation

@gyscos

@gyscos gyscos commented Jun 28, 2025

Copy link
Copy Markdown
Contributor

This is a draft PR/request for comment to switch from installing packages with paru -S to simply building them with paru -B.

Note that this is currently entirely untested. I'm not entirely sure how to prepare an image for testing :^S

Here is the main idea: instead of installing a package in one step with paru -S, download the PKGBUILD and sources with paru -G, then build that pkgbuild with paru -B, and finally move the resulting packages to the destination folder.

To avoid moving/copying the package archives across filesystem/mount-point boundaries, I build the entire package in the mounted build dir, but only select files that look like archives from there (instead of picking everything).

Fixes #206

A few questions arise from these changes:

  • We no longer need a custom PKGDEST in the makepkg.conf. This now only contains the -j$(nproc) build flag. Should this instead be in the base builder image, removing the need for a makepkg.conf entirely?
  • Building a package has much fewer turning knobs than installing a package. As such, there's really nothing that could be added to the -B --noconfirm --noprogressbar --color never. Any custom flags instead add a potential failure mode. Maybe we could remove custom build flags entirely? Do you have examples of how they are currently used?
  • The previous build flags included -Syu {package_name}, which was updating every package installed on the build node. paru -B by itself doesn't change anything installed on the build node, so it doesn't update anything. I added an explicit paru -Syu before that to match the current behaviour, but it's an open question: do we need to update everything everytime?

Comment thread backend/aurcache/src/builder/build.rs Outdated
Comment thread backend/aurcache/src/builder/build.rs Outdated
.to_str()
.ok_or(anyhow!("Failed to get string from filename"))?
.to_string();
if !archive_name.ends_with(".pkg.tar.zst") {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This relies on the extension being .zst (currently the case, but could potentially change with a different makepkg.conf). We could use a fancier check, possibly including the architecture... Or leave it as-is for now.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I even built the feature to support .xz container into the pacman-repo-utils.
We probably want to add a check for .tar.xz too. Are there also different containers supported?
https://github.com/Lukas-Heiligenbrunner/AURCache/blob/master/backend/pacman-repo-utils/src/repo_add.rs#L29
Here I check only for .zst or .xz...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compression can also be entirely disabled. Looking at the end of makepkg.conf you can see it implies potential support for .gz, .bz2, .lrz, .lzo, .z, .lz4, .lz. Not sure we have to support all of them but we could just accept .tar*.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes true.
I was confused and meant the extension of the repo database (repo.db.tar.gz & repo.files.tar.gz), sry.
Yeah, probably a wildcard is the easiest solution for now. In the future we could make this configureable.

Comment thread backend/aurcache/src/builder/docker.rs Outdated
let build_flags = self.package_model.build_flags.get()?.split(";").join(" ");
// create new docker container for current build
let build_dir_base = "/var/cache/makepkg/pkg";
let build_dir_base = "/build";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actual build location doesn't really matter, but /var/cache/makepkg/pkg is a bit obscure when we just want some working directory. It's pretty common to use top-level directories like /build, /app, ... But we can obviously use something else if you prefer.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This path gets configured by the makepkg.conf file (which we create later per build)
And i think this is just the default path where makepkg builds pkgs, thats why I left it there.
But you're right, its probably cleaner if we move this to top level.

Comment on lines +2 to +3
_name: String,
_build_dir_base: &str,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just temporary while we decide if we want this function at all in this state.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PKGDEST is not required since we use

paru -G {name}
paru {build_flags} {name}

To checkout the pkgbuild and then build the subfolder with -B flag right? Or am I missing something?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, we don't rely on any specific PKGDEST, we just look at the resulting archives in the current directory (like makepkg would do).

Comment thread backend/aurcache-builder/src/docker.rs
Comment thread backend/aurcache/src/builder/docker.rs Outdated
Comment thread backend/aurcache/src/builder/docker.rs Outdated
Comment on lines +183 to +187
let init_cmd = "
sudo pacman-key --init
sudo pacman-key --populate archlinux
paru -Syu
";

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arbitrary split between a constant "initialization" part, and the build part that depends on the package. I could merge the two in a single one if desired.

Comment thread backend/aurcache/src/builder/docker.rs Outdated

let build_id = self.build_model.id.get()?;
let container_name = format!("aurcache_build_{filtered_name}_{build_id}");
let auto_remove = !cfg!(debug_assertions);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Negating a constant results in the weird-looking !cfg!. Maybe it'd be better to do cfg!(not(debug_assertions)) here.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, feel free to edit.

@gyscos

gyscos commented Jun 29, 2025

Copy link
Copy Markdown
Contributor Author

Unfortunately, while it can build packages just like before, it still fails with conflicting packages. I think it's an upstream bug, and I filed Morganamilo/paru#1376.

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

Some of your questions:

So the per-package arguments were required for packages that require pgp signing:
#65
#64
This helped to make package builds at least a bit more configureable and more pkgs should be buildable

I use Rustrover for backend and Intellij for frontend dev and use the podman/docker socket from my system to build packages. When you start the backend on your pc and start the frontend it should just work to try stuff out. (Remember, you can build flutter for native linux, so then it behaves just like a native app and works great.) When you need to try out docker specific stuff you can use the docker compose files in the root of the project docker-compose.dindmode.dev.yaml and docker-compose.hostmode.dev.yaml depending on what build mode you want to try out.
You can build the buillder image locally with sth like: docker build -f docker/builder.Dockerfile .

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

The main idea why packages are built in docker containers is because of isolation (and concurrency), after a build the container gets deleted automatically. So there can't be any leftover build artifacts which might clutter the main aurcache container. Your idea, sounds good, but we have to be careful to delete all the leftover build artifacts. Since there is just this one folder it should be fine, but just as a note.

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

I'll review this tomorrow too.
And I don't think we need to -Syu update before each build since the image is auto-built once a week, so this should be fine.

@gyscos

gyscos commented Jun 29, 2025

Copy link
Copy Markdown
Contributor Author

So the per-package arguments were required for packages that require pgp signing:

Ah looks like --mflags --skippgpcheck is still available for paru -B. Though maybe something changed, but I'm now able to compile freeipa without this flag.

Though it revealed a problem with the current PR: it assumes that paru -G {package_name} will result in a {package_name} folder created; but the folder created is that of the base package. In the case of split packages (like freeipa-client-common), it can differ from the requested package name.

This brings up a question: how to support split packages? I see a few options:

  • Should aurcache only work with base packages (like freeipa), which will build (among others) the freeipa-client-common package archive?
    This would be my personal preference. We could have the UI indicate that when we add/install a split package, it will actually add the parent package for minimal surprise. Might also be nice to have
  • If instead we want to accept adding "sub-packages" like freeipa-client-common, then should we build the base package but only keep the requested package (and discard the rest)? One downside is that if we want all splits of a base package, we'd now need to build the entire base independently in each of the split packages.
  • If instead we keep everything we built, we essentially have a base package named like the split-package. It's not much better than the first "base packages only" solution, and instead risks having overlapping packages if we add two packages that share the same base. Overlapping packages sounds like something we want to avoid for obvious reasons.
    This is a bad option all around, just here for completeness.

I'm still trying to understand the current situation. It pulls everything from makepkg's PKGDEST. It looks like any dependency that's an AUR package itself (and needs to be built) will end up there, and gets registered as part of the main package target list of files. This means:

  • Any AUR dependency is now available for users, so they don't have to build anything themselves.
  • If different packages depend on the same AUR package, they will all try to build and include that dependency, resulting in overlapping packages or version mismatch/inconsistencies.

I think it would be best to either entirely ignore the dependencies when registering archive files, or, better, to automatically properly register them as separate packages in AURCache. This second option is the best long-term but includes a large work:

  • Compute the dependency tree when adding the package in AURCache itself, not in the builder.
  • Build the dependency package before building the downstream target (need to synchronize and order the builds).
  • Inject AURCache's repo in the builder's mirrorlist (I thought this was mentioned but I don't know if this is currently done?)

In the meantime, ignoring dependency archives might be the best short-term way to avoid overlapping packages.

Sorry for the wall of text - I realize this investigation sent me in all directions, and this would probably deserve dedicated issues.

@gyscos
gyscos force-pushed the build_not_install branch 2 times, most recently from 705f01d to c3e0e59 Compare June 30, 2025 18:23

@Lukas-Heiligenbrunner Lukas-Heiligenbrunner left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rest looks fine. Thanks again!

You're right we can bake the makepkg.conf into the image now. I'd just add it to the docker subfolder and ADD it in the Dockerfile.
I'd leave the custom build flags in for now until we are sure it is really not required.

The current situation is:
When a user builds a package which depends on other AUR packages those are getting built too and end up in the output folder. Aurcache picks them up and adds them to the repo. Since pacman doesn't allow different versions of the same package in one repo (and archlinux isn't really built for that) I thought this should be fine if they are just in the repo. All the archives which belong to one package are registered in a db table. (One could build a fancy tree out of that maybe) When a package is deleted it is checked if no other package requires this dependency package if not, it gets deleted too, if yes its kept.
I don't think an AUR package is able to request a specific version for a aur dependency, right?

Split packages are a pain. I'm aware of them but I didn't really care/had time to tackle them properly... But I guess the easiest would be to add just the base package, as you prefered. When user tries to add a split package, maybe give him a warning and add just the base package. Further steps might be to add a fancy list in the package overview of which split pakcage this pkgbuild consists.

Edit:
text wall is fine, thanks for your efforts!

Comment thread backend/aurcache/src/builder/build.rs Outdated
Comment thread backend/aurcache/src/builder/build.rs Outdated
.to_str()
.ok_or(anyhow!("Failed to get string from filename"))?
.to_string();
if !archive_name.ends_with(".pkg.tar.zst") {

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I even built the feature to support .xz container into the pacman-repo-utils.
We probably want to add a check for .tar.xz too. Are there also different containers supported?
https://github.com/Lukas-Heiligenbrunner/AURCache/blob/master/backend/pacman-repo-utils/src/repo_add.rs#L29
Here I check only for .zst or .xz...

Comment thread backend/aurcache/src/builder/docker.rs Outdated
let build_flags = self.package_model.build_flags.get()?.split(";").join(" ");
// create new docker container for current build
let build_dir_base = "/var/cache/makepkg/pkg";
let build_dir_base = "/build";

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This path gets configured by the makepkg.conf file (which we create later per build)
And i think this is just the default path where makepkg builds pkgs, thats why I left it there.
But you're right, its probably cleaner if we move this to top level.

Comment thread backend/aurcache/src/builder/docker.rs Outdated
let init_cmd = "
sudo pacman-key --init
sudo pacman-key --populate archlinux
paru -Syu --noconfirm

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not required anymore since builder image gets auto built once a week.

Comment thread backend/aurcache/src/builder/docker.rs Outdated
Comment thread backend/aurcache/src/builder/docker.rs Outdated

let build_id = self.build_model.id.get()?;
let container_name = format!("aurcache_build_{filtered_name}_{build_id}");
let auto_remove = !cfg!(debug_assertions);

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, feel free to edit.

Comment on lines +2 to +3
_name: String,
_build_dir_base: &str,

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PKGDEST is not required since we use

paru -G {name}
paru {build_flags} {name}

To checkout the pkgbuild and then build the subfolder with -B flag right? Or am I missing something?

@gyscos

gyscos commented Jun 30, 2025

Copy link
Copy Markdown
Contributor Author

I don't think an AUR package is able to request a specific version for a aur dependency, right?

It's possible, but rarely used and rarely a good idea. In a PKGBUILD there's no difference between dependencies in the aur or in a repo (and indeed, a AURCache or similar repo would blur the distinction anyway). It's possible to add version constraints, mostly to prevent updating or installing the wrong version. It can't be used to fetch the newest supported version in a semver fashion.

When a package is deleted it is checked if no other package requires this dependency package if not, it gets deleted too, if yes its kept.

Ah I didn't realize that - in that case overlapping packages are not as terrible. They just likely cause duplicated compute to build from multiple places.

But I guess the easiest would be to add just the base package, as you prefered

One problem is that the "base package" is not always the name of an actual package. For example, here is the freeipa base package. Note the url includes pkgbase, not package. There is no freeipa package, only freeipa-client-common (and a couple others). So currently freeipa doesn't even appear in the dropdown list. I didn't check yet but there may be an option to look for these as well, and mark them specially in the UI if a base package contains more than 1 package.

But in any case it shouldn't be blocking for this PR, and could come later as an improved support for split packages.

When a user builds a package which depends on other AUR packages those are getting built too and end up in the output folder. Aurcache picks them up and adds them to the repo.

This is the part that is changing in this PR: currently, I only pick up what is defined by this PKGBUILD, not any dependency that got built along the way. Though I suppose I could bring back PKGDEST and keep both - all built dependencies, and the built-but-not-installed target package.

@gyscos
gyscos force-pushed the build_not_install branch from c3e0e59 to 252eb3e Compare June 30, 2025 20:52
@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

It's possible, but rarely used and rarely a good idea. In a PKGBUILD there's no difference between dependencies in the aur or in a repo (and indeed, a AURCache or similar repo would blur the distinction anyway). It's possible to add version constraints, mostly to prevent updating or installing the wrong version. It can't be used to fetch the newest supported version in a semver fashion.

Ok, good to know! But I think we can ignore this edge case for now and leave it as is.

Ah I didn't realize that - in that case overlapping packages are not as terrible. They just likely cause duplicated compute to build from multiple places.

Yeah. But I think the duplicate compute is a price we easily want pay instead of all the dependencies are auto-added to the package list.

One problem is that the "base package" is not always the name of an actual package. For example, here is the freeipa base package. Note the url includes pkgbase, not package. There is no freeipa package, only freeipa-client-common (and a couple others). So currently freeipa doesn't even appear in the dropdown list. I didn't check yet but there may be an option to look for these as well, and mark them specially in the UI if a base package contains more than 1 package.
But in any case it shouldn't be blocking for this PR, and could come later as an improved support for split packages.

I just noticed, the link to the aur repo is broken within split packages...
The current behaviour when building a split package is building and addint all the subpackages Here example log:
log.txt
Does this PR break this behaviour?

I'll create a issue for split packages and we handle this in another PR.

that got built along the way. Though I suppose I could bring back PKGDEST and keep both - all built dependencies, and the built-but-not-installed target package.

In my opinion for usability it would be better as it is now. I think a user doesn't really care about the packages dependencies. They just want to add the package they want to build, install it from the aurcache repo, and it should just work out of the box.

@gyscos

gyscos commented Jul 3, 2025

Copy link
Copy Markdown
Contributor Author

In the current state, this PR now preserves the same behaviour as the current master branch (includes the archives of all dependencies and other split packages). It still relies on a custom makepkg.conf, but the path doesn't actually depend on the package name, so in the future we may be able to move this to the builder base image.

The main issues remaining are:

An alternative I haven't investigated yet is the chroot-build feature of paru. It looks like it creates a chroot, build the package there, then add it to some "local repo" rather than installing it, which sounds very similar to what we want here.
We could run the chroot-build in the docker container, as an extra layer of isolation (and with the least changes from the current situation).
EDIT: looks like it won't be trivial to use chroot inside docker. Even when running privileged, some stuff aren't working.

We could also consider replacing docker entirely with the chroot build, but it'd need more investigation to confirm we can run parallel chroot builds (we don't want to lose parallel builds). Also it may be a ton of changes, might not be worth it. Keeping docker gets us closer to possibly one day running the builder on kubernetes for a proper distributed build farm.

@Lukas-Heiligenbrunner

Lukas-Heiligenbrunner commented Jul 4, 2025

Copy link
Copy Markdown
Owner

Great!

All existing build flags will need to be updated in a migration, since all the -Syu need to become -B. In addition, any additional custom flag like --skippgpcheck might need to become --mflags --skippgpcheck. This sounds like a painful migration, but it looks like at least we can write the migration in rust (and not just in SQL)?

Yes, we should create a new db migration. For this just create a new migration file (for example here and add it to the mod.rs. In every build_flags field you replace the -Syu with rust code...
I think we can ignore the --skippgpcheck since not so many people should use it. We just add a breaking change into the next releases changelogs.

Until paru -B --noconfirm complains about conflicts for the package it's not installing Morganamilo/paru#1376 is fixed there's really no benefit to this PR yet.

Thanks! But I think just building and not installing the package should be a cleaner method as before. 👍

An alternative I haven't investigated yet is the chroot-build feature of paru.

I've already seen that chroot stuff and tried it (just very quickly) out like one year ago. I couldn't get it to work then and thats why I sticked with the way it works now. Important thing is that we don't have to give the spawned docker containers more permissions than now for getting chroot to work.
Just saw your edit, yeah, I'm not sure if chrooting would give as major benefits anyway.

We could also consider replacing docker entirely with the chroot build.

Disagree with that, docker gives us major advantages in isolation. Eg. limiting cpu cores, memory, networking, multiple parallel builds... Most of this stuff would be much more pain without docker.
And yes, kubernets would be cool one day. :)

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

Hey, @gyscos, can I help you somehow with this pr? Should we wait until the paru issue is resolved?

@gyscos

gyscos commented Jul 24, 2025

Copy link
Copy Markdown
Contributor Author

Sorry, I've been busy these couple weeks. I may have a bit more time next week to come back to that.

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

Sorry, I've been busy these couple weeks. I may have a bit more time next week to come back to that.

All right, no stress. Feel free to ping me anytime!

@Lukas-Heiligenbrunner
Lukas-Heiligenbrunner marked this pull request as draft November 5, 2025 18:24
@systemofapwne

Copy link
Copy Markdown

I feel like I am running into issues for some of my packages due to paru trying to install the packages. This PR seems to be the right spot to report, as it seems to solve the issue.

I maintain a meta-package repository, that is a fork of manjaro-meta.
It fails to install due to mutual exclusive packages like manjaro-pipewire conflicts with manjaro-pulse and vice versa.

Build log
Pulling image: ghcr.io/lukas-heiligenbrunner/aurcache-builder:latest:: Synchronizing package databases...
 core downloading...
 extra downloading...
 multilib downloading...
==> Appending keys from archlinux.gpg...

gpg: next trustdb check due at 2026-01-30
:: Generating .SRCINFO for ./tmp...
:: Generating .SRCINFO for ./tmp...
:: Resolving dependencies...

:: There are 2 providers available for resolvconf:
:: Repository core:
    1) systemd-resolvconf  
:: Repository extra:
    2) openresolv  
Enter a number (default=1): 
:: There are 2 providers available for jack:
:: Repository extra:
    1) jack2  2) pipewire-jack  
Enter a number (default=1): :: Calculating conflicts...
:: Calculating inner conflicts...

:: Inner conflicts found:
    manjaro-pipewire: manjaro-pulse (manjaro-pipewire)  
    manjaro-pulse: manjaro-pipewire (manjaro-pulse)  
    pipewire-pulse: pulseaudio (pipewire-pulse)  manjaro-pulse (pipewire-pulse)  
    pulseaudio: pipewire-pulse (pulseaudio)  

:: Conflicting packages will have to be confirmed manually
error: can not install conflicting packages with --noconfirm
failed to build packageDocker container wait error: 

This issue arise from paru trying to install the packages and might be solved by this PR.

@systemofapwne

systemofapwne commented Dec 3, 2025

Copy link
Copy Markdown

Another note: I just figured, that I was already using paru -B (as this PR introduces) since I used the new "intall from git" feature for my meta package. It still failed so you might want to consider adding this feature (or giving options to avoid build issues then).

My first try to solve this: Separating the conflicting package to another git repo
This is almost already working. At least for the manjaro-meta, it will work by simply removing the manjaro-pulse meta-package.

However, building all my own meta-packages still failed. This time, paru somehow installed jack2 on its own and then complained that jack2 conflicts with pipewire-jack (both provide jack). The reason was the following build order in my PKGBUILD and some funny makedepends issues.

- my-gstreamer --depends--> gst-libav      --makedepends-->  jack2 (provides jack)
- my-pipewire  --depends--> pipewire-jack (provides jack)

The make dependency of gst-libav caused the conflict. Switchin the order of

pkgname=(
  'my-gstreamer'
  'my-pipewire'
  'my-pulse'
)

to

pkgname=(
  'my-pipewire'
  'my-gstreamer'
  'my-pulse'
)

solved this for me (Reason: pipewire-jack now provides the required makedeps for gst-libav as it gets built before)

However: Since I am assembling several meta-packages here that might have unsolveable cross-depencies in the future (no reordering will help here), I sinply turned off all dependency checks.

First, paru's internal dependency checks must be forwarded to pacman via the --useask flag (solves the inner conflicts). Pacman then sees no issue for my-pipewire and my-pulse being mutually exclusive.
The conflict of jack2 vs pipewire-jack can then finally solved by adding the ultimate nodepends flag: -dd.

With all of this, I was able to build my meta-package.

You can find a strapped down example meta package, that produces the errors here: https://github.com/systemofapwne/pkg-my-meta-failing.git

@xXPerditorXx

Copy link
Copy Markdown

Why not just :

git clone https://aur.archlinux.org/<pkgname>
makepkg -s

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

Why not just :

git clone https://aur.archlinux.org/<pkgname>
makepkg -s

Hey, the problem with makepkg is it doesn't build transitive dependencies if they are also from the AUR.
paru or yay resolves the dependencies properly and builds dependent AUR packages before the main one.
AURCache then picks up all built package archives and puts them in the repo.

@xXPerditorXx

Copy link
Copy Markdown

Why not just :

git clone https://aur.archlinux.org/<pkgname>
makepkg -s

Hey, the problem with makepkg is it doesn't build transitive dependencies if they are also from the AUR. paru or yay resolves the dependencies properly and builds dependent AUR packages before the main one. AURCache then picks up all built package archives and puts them in the repo.

Okay, yes, that makes sense. And what if we do it like this:

paru -G pkgname
paru -B pkgname
mv pkgname/*.pkg.tar.zst /app/builds

@gyscos
gyscos force-pushed the build_not_install branch from d9b78fb to 72fe274 Compare April 9, 2026 21:35
@gyscos

gyscos commented Apr 9, 2026

Copy link
Copy Markdown
Contributor Author

That's pretty much what this branch does.

Finally got some time to rebase and cleanup this PR. Some work went into preparing an end-to-end test script (script/test-e2e.sh), which ended up being a bit more involved than I hoped.

  • I wanted to be able to test a local builder image, so the test sets up a local registry and pushes the image there. It also deploys aurcache itself, with an env var to control the builder image. Since the registry is accessed with plain HTTP, that required some config tweaks on the aurcache container. Still, easier than getting HTTPS working for the test.
  • Requesting a new package build is currently done with curl calls. If we were to add a CLI http client we could use that instead.
  • For validation, I am trying to replicate the actual end use-case of installing the built package. For that, I spin up another container, adds the aurcache container as a repo, and installs the package. There's some boilerplate around making sure the container is up-to-date enough to install the package.

The current test uses "host mode", where the docker socket is mounted in the aurcache container to spin up the builder. I could try the "dind"/podman mode too if that's desired.

The test script ended around 200 lines. Not sure it can be made significantly smaller while staying readable, but I'm open to suggestions.

Regarding the rest of the changes: I tried to get the e2e test passing for a package like paru or paru-git. I had to fix a few things not technically related to get there:

  • I use paru-git in the builder, since the fix to properly handling conflicts with -B hasn't made it into a release yet.
  • The builder was failing when installing some makedepends for paru-git, because it couldn't import the GPG keys. Turns out that was because the builder was run with a volume mount shadowing /etc/pacman.d (to get the mirrorlist), but that also includes /etc/pacman.d/gnupg. This was meant the builder did not have an up-to-date keyring, and was messing up adding more keys to that keyring. Instead, I just mount the mirrorlist itself (we can mount single files in docker, doesn't have to be directories). For now I did not change the way the mirrorlist is given (the env var is still assumed to point to a directory), only what we mount into the builder. I tested it in host mode, but will need to further test in dind.

I'm trying to keep this PR minimal but things keep creeping in... :( Might bring the test and the mirrorlist changes to separate PRs if needed, and this PR would depend on both of them.

@Lukas-Heiligenbrunner
Lukas-Heiligenbrunner marked this pull request as ready for review April 19, 2026 14:08
@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

Hi thanks.
paru-git is fine for me.
Do you think integrating your e2e test in a CI pipeline makes sense?
I think we can merge this in and if in dind mode regressions occur, we fix them in follow-up PRs, right?

@gyscos
gyscos force-pushed the build_not_install branch from f110d96 to 4d79714 Compare April 19, 2026 16:05
@gyscos

gyscos commented Apr 19, 2026

Copy link
Copy Markdown
Contributor Author

Looks like CI is currently failing with:

ERROR: failed to build: failed to solve: failed to push ghcr.io/lukas-heiligenbrunner/aurcache:pr-214: denied: installation not allowed to Write organization package

Do you think integrating your e2e test in a CI pipeline makes sense?

I was not sure we could run the docker machinery there, but apparently it's fine. I removed the privileged mode (only required for dind, but we're doing host mode in the test), and added a job to the ci workflow. It does add some stress on the mirror & AUR, but it's not terrible for the current rate of PR in this repo. We can consider smarter caching if we need to.

I think we can merge this in and if in dind mode regressions occur, we fix them in follow-up PRs, right?

Sure. I'll try to get a similar dind test script, I think there may be a few issues left with some path/mounts.

I ended up reverting back to paru non-git, apparently it fails on paru-git but works with regular paru, even though the fix should only be in paru-git? 🙃 Nope, nevermind, that wasn't it. I have a PR to fix it (Morganamilo/paru#1536), but seeing the current activity on that repo I have no idea how long until (if?) it gets merged.

An alternative solution could be to not install the paru package, but just cargo install paru (once the dependencies are installed). This would let us:

  • Chose where to install paru from (crates.io, the upstream git repo, a fork, ...)
  • Not have paru/paru-bin/paru-git installed at all on the builder, so even without the fix (building from crates.io) it would still not conflict with paru-git and paru, the main use-cases for conflict-free building.

@gyscos

gyscos commented Apr 23, 2026

Copy link
Copy Markdown
Contributor Author

Somehow even with these "foolproof" test scripts I must have mixed up my tests, because neither paru not paru-git was actually still able to build both paru and paru-git.

As mentioned, I posted a PR to paru with a fix; in the meantime, I updated this branch to do both workarounds I mentioned:

  • It now builds paru directly using cargo install, no package involved. It actually has no dependency (apart from pacman and git, both are already installed).
  • Currently use my fork (with my patch) to solve the overzealous conflict check.

I'm not sure these two decisions are ideal long term. I like the idea of having paru itself "updatable" like other packages (though maybe the builder doesn't actually need to update too much and is instead built from scratch regularly?). I also don't like using a fork (even my own), and hopefully it'll get merged soon enough and we can use regular upstream again.

Finally, finally, I actually deployed this on my truenas instance and am not seeing any issue so far, even though it's using the not-quite-as-tested dind mode. 🤞

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

Haha, yeah happens i guess... :)
Finally the CI passes after changing the PR trigger, we can now simply try this out with image tag :pr-214
I'll try this out on my own aurcache instance (also in Truenas container) and report back if everything works as expected.
If it works fine I see no problem in merging this in.

I think building paru from source is no problem at all. The builder image gets auto built, I think, once a week, so it should always be up to date.
The only problem I see is when they have an unstable regression/bug on their master branch it gets built directly into the builder image. So maybe pinning their newest tag when your PR gets into a tagged version could make sense?
Something like:
git checkout $(git describe --tags --abbrev=0)

@gyscos

gyscos commented Apr 23, 2026

Copy link
Copy Markdown
Contributor Author

Yes if we end up building from git, pinning the commit makes sense in the absence of a stable branch. Maybe cargo can take the branch/commit directly in the command.
Ideally we'd build from crates.io, if it gets updated often enough.

We can also consider "vendoring" paru as a git submodule, and build from that (mounted) path when building the builder. Pretty similar to got pinning, only the choice is done "outside" of add-aur sh. Not sure which is cleaner.

(Looks like my PR in paru got some attention, so it might get merged sooner than I feared)

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

Works fine in my environment, great! Thanks.
Did you also try to build the builder image for all supported platforms (amd64, arm32, arm64, riscv)? This was always a pain point when changing stuff around in the paru add script^^

https://crates.io/crates/paru --> The crates.io paru version is 3years old
I'd advise against vendoring as a submodule. In my experience git submodules are always a pain to work with and to avoid at all cost. I think we can come up with a better solution ;)

@gyscos

gyscos commented Apr 23, 2026

Copy link
Copy Markdown
Contributor Author

Did you also try to build the builder image for all supported platforms (amd64, arm32, arm64, riscv)? This was always a pain point when changing stuff around in the paru add script^^

No, I haven't yet. The goal is to prepare a $ARCH docker image that runs on $ARCH and produces $ARCH packages (no cross compiling)? Do you test in qemu, or on real hardware?

@Lukas-Heiligenbrunner

Lukas-Heiligenbrunner commented Apr 23, 2026

Copy link
Copy Markdown
Owner

The only two platforms AURCache supports to run itself on is amd64 and arm64. But on those platforms (with the help of qemu-binfmt) you can cross build packages for those 4 platforms. For this to work we need to build the builder image for all 4 platforms in the CI Pipeline. The pipeline uses docker buildx (which uses qemu)
But on your local machine might be some slight differences than in the CI worker (Landlock and sandboxing)
I'll try this out if they still build properly, but i think everything should be fine.

Edit:
Mostly I tested this with qemu-binfmt on my PC or sometimes on a Raspberry Pi.
But unfortunatelly its not tested very well, since I'm not using this myself very often and its a pain to setup correctly inside a LXC or Incus container...

@Lukas-Heiligenbrunner

Copy link
Copy Markdown
Owner

Looks good so far!
Please open a followup PR when Morganamilo/paru#1536 gets merged to use the original paru again.
Thanks!
Hope hearing from you with more PRs. :)

@Lukas-Heiligenbrunner
Lukas-Heiligenbrunner merged commit f83dfb1 into Lukas-Heiligenbrunner:master Apr 24, 2026
5 checks passed
@gyscos

gyscos commented Apr 24, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! I think I found a potential issue with some files lingering in the build folder, might come up with a fix PR shortly :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot build paru package (conflicts with paru-bin)

4 participants