feat: update cookiecutter template to support new package managers and asset bundlers

[Samk13]

❤️ Thank you for your contribution!

Description

For:
inveniosoftware/invenio-cli#385
inveniosoftware/invenio-cli#384

Needs:
inveniosoftware/invenio-cli#392

Checklist

Ticks in all boxes and 🟢 on all GitHub actions status checks are required to merge:

• I'm aware of the code of conduct.
• I've created logical separate commits and followed the commit message format.
• I've added relevant test cases.
• I've added relevant documentation.
• I've marked translation strings.
• I've identified the copyright holder(s) and updated copyright headers for touched files (>15 lines contributions).
• I've NOT included third-party code (copy/pasted source code or new dependencies).
  • If you have added third-party code (copy/pasted or new dependencies), please reach out to an architect on Discord.

Frontend

• I've followed the CSS/JS and React guidelines.
• I've followed the web accessibility guidelines.
• I've followed the user interface guidelines.

Reminder

By using GitHub, you have already agreed to the GitHub’s Terms of Service including that:

1. You license your contribution under the same terms as the current repository’s license.
2. You agree that you have the right to license your contribution under the current repository’s license.

[max-moser]

the Dockerfile still needs a few tweaks for uv and/or pnpm.
for reference, this is our current variant, with both enabled:

# Dockerfile that builds a fully functional image of your app.

FROM ghcr.io/astral-sh/uv:alpine AS builder

# the server name is just there to satisfy the strict startup sanity check
# it's not really used during the build step, so it can be set to anything
ARG INVENIO_SERVER_NAME=localhost
ARG INVENIO_INSTANCE_PATH=/var/instance

# set language/locale
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
ENV PATH="${INVENIO_INSTANCE_PATH}/.venv/bin:${PATH}"

# create the instance dir and set it as working directory
RUN mkdir -p "${INVENIO_INSTANCE_PATH}"
WORKDIR ${INVENIO_INSTANCE_PATH}

# install build dependencies
RUN apk update && \
    apk add cairo gcc git linux-headers musl-dev nodejs npm py3-setuptools python3 python3-dev && \
    npm install --global --ignore-scripts pnpm

# install the python dependencies system-wide
COPY pyproject.toml uv.lock ./
RUN uv sync --locked --no-progress --compile-bytecode && \
    uv clean

# copy the relevant files from the local project directory
COPY ./docker/uwsgi/ ${INVENIO_INSTANCE_PATH}/uwsgi/
COPY ./invenio.cfg ${INVENIO_INSTANCE_PATH}/
COPY ./app_data/ ${INVENIO_INSTANCE_PATH}/app_data/
COPY ./assets/ /tmp/assets/
COPY ./static/ /tmp/static/
COPY ./templates/ /tmp/templates/
COPY ./translations/ /tmp/translations/

# collect & build the frontend, and clean up unnecessary source files
# local overrides are copied over before/after the build step so they don't get lost during the build
ENV INVENIO_WEBPACKEXT_NPM_PKG_CLS=pynpm:PNPMPackage
RUN invenio collect --verbose && \
    mkdir assets templates translations && \
    cp -r /tmp/assets/ ${INVENIO_INSTANCE_PATH}/ && \
    invenio webpack buildall && \
    cp -r /tmp/static/ ${INVENIO_INSTANCE_PATH}/ && \
    cp -r /tmp/templates/ ${INVENIO_INSTANCE_PATH}/ && \
    cp -r /tmp/translations/ ${INVENIO_INSTANCE_PATH}/ && \
    rm -rf ${INVENIO_INSTANCE_PATH}/assets/node_modules && \
    pnpm cache delete


# the actual invenio app image
FROM ghcr.io/astral-sh/uv:alpine

ARG INVENIO_INSTANCE_PATH=/var/instance
WORKDIR ${INVENIO_INSTANCE_PATH}

# set language/locale
ENV LANG=en_US.UTF-8
ENV LANGUAGE=en_US:en
ENV LC_ALL=en_US.UTF-8
ENV INVENIO_INSTANCE_PATH=/var/instance
ENV PATH="${INVENIO_INSTANCE_PATH}/.venv/bin:${PATH}"

# install the runtime dependencies
RUN apk update && \
    apk add cairo font-dejavu imagemagick py3-setuptools python3 && \
    apk cache clean

# copy over the built application
COPY --from=builder "${INVENIO_INSTANCE_PATH}" "${INVENIO_INSTANCE_PATH}"
RUN chmod g+w "${INVENIO_INSTANCE_PATH}"

ENTRYPOINT ["sh", "-c"]

[Samk13]

the Dockerfile still needs a few tweaks for uv and/or pnpm. for reference, this is our current variant, with both enabled:

Thanks for sharing your image @max-moser it was really helpful! 😊
Here’s what I ended up creating:

I noticed there were some unnecessary packages being installed during both the build and runtime stages. For example, installing Python separately isn't needed if you're already starting from a Python base image.
Also, I saw a command like uv clean, which doesn't seem to exist. I think you meant uv cache clean.

Regarding the copy steps you were copying into /tmp and then moving multiple folders. I felt it was simpler to use a single COPY . . command and manage exclusions through the .dockerignore file instead as the build time is fast I ignored the layers caching that we can benefit from multiple copy commands. Also for Node.js wasn't version-controlled, so I opted to copy it from a Node base image so we can control the version in the future.

For the runtime stage, I could have started from a Python Alpine base image, which would’ve reduced the image size to around 1.20 GB. However, I chose to keep uv as base so I can manage packages directly on the pod if needed. With this setup, the final image size is 1.24 GB, and the build time stays under 4 minutes.

This is the updated version let me know if we miss something here:

Here is also an example for the pyproject.yaml file for both the instance and site package

ARG JS_PACKAGE_MANAGER=pnpm@10.8.1
ARG NODE_IMAGE=node:22-alpine
ARG PYTHON_BASE_IMAGE=ghcr.io/astral-sh/uv:0.6-python3.12-alpine

ARG WORKING_DIR=/opt/invenio
ARG INVENIO_INSTANCE_PATH=${WORKING_DIR}/var/instance

# --- NODE.JS STAGE ---
FROM ${NODE_IMAGE} AS node
ARG JS_PACKAGE_MANAGER
ENV JS_PACKAGE_MANAGER=${JS_PACKAGE_MANAGER}
RUN corepack enable && corepack prepare ${JS_PACKAGE_MANAGER} --activate

# --- BASE SETUP STAGE ---
FROM ${PYTHON_BASE_IMAGE} AS python_base
ARG INVENIO_INSTANCE_PATH
ENV INVENIO_INSTANCE_PATH=${INVENIO_INSTANCE_PATH} \
LANG=en_US.UTF-8 \
LANGUAGE=en_US:en \
LC_ALL=en_US.UTF-8 \
# Compile Python files to .pyc bytecode files
UV_COMPILE_BYTECODE=1 \
# Copy Python files from cache mount, resolving symlink issues
UV_LINK_MODE=copy
ENV PATH="${INVENIO_INSTANCE_PATH}/.venv/bin:${PATH}"
RUN apk update && \
    apk add --no-cache \
    bash cairo \
    imagemagick util-linux

# --- BUILD APP STAGE ---
FROM python_base AS builder
WORKDIR ${INVENIO_INSTANCE_PATH}
RUN apk add --no-cache \
        gcc \
        musl-dev \
        linux-headers

# Copy Node.js runtime libraries and binaries
COPY --from=node /usr/lib /usr/lib
COPY --from=node /usr/local/bin /usr/local/bin
COPY --from=node /usr/local/lib /usr/local/lib
COPY --from=node /usr/local/include /usr/local/include
COPY --from=node /usr/local/share /usr/local/share

# Count on .dockerignore to exclude files
COPY . .

# Sync Python dependencies
RUN uv sync --locked && \
    uv cache clean

# --- FRONTEND BUILD ---
ENV INVENIO_WEBPACKEXT_NPM_PKG_CLS=pynpm:PNPMPackage
RUN uv run invenio collect --verbose && \
    mkdir -p assets templates translations site data archive && \
    uv run invenio webpack buildall && \
    rm -rf assets/node_modules && \
    # Experimental command!
    # https://pnpm.io/cli/cache-delete
    pnpm cache delete && \
    rm -rf "$(pnpm store path)" && \
    # Uwsgi config expected to be on instance level
    cp -a docker/uwsgi/. .

# --- RUNTIME STAGE ---
FROM python_base AS runtime

RUN addgroup -S invenio && \
    adduser -S -G invenio invenio

COPY --from=builder --chown=invenio:invenio \
    "${INVENIO_INSTANCE_PATH}" \
    "${INVENIO_INSTANCE_PATH}"

USER invenio
WORKDIR ${INVENIO_INSTANCE_PATH}

ENTRYPOINT ["bash", "-c"]

[Samk13]

After discussing with @slint and others during the teleconference call, here’s a summary of the Cookie-cutter strategy as I understand it:

• v13: Keep pipenv + npm as the defaults, but strip out most other options to keep the template opinionated and beginner-friendly.

• v14 / master branch: Switch defaults to uv + pnpm, introduce the new base Dockerfile, and drop pipenv entirely.

In short, we'll stick to one option for the Dockerfile and avoid offering multiple choices.

[OliverGeneser]

@Samk13 Hi 👋 What's the current status on this pr?

[Samk13]

@Samk13 Hi 👋 What's the current status on this pr?

Hi @OliverGeneser 👋

The only open point is the Dockerfile base image.
The current template uses AlmaLinux (multi-GB), while this PR switches to an Alpine-based image. There are also downstream maintainers using different base images, and we’re still waiting for alignment from the CERN side on which direction to standardize on.

Apart from that, everything is ready. Once this is clarified, I can resolve the conflicts and squash the commits.

[max-moser]

Something to keep in mind is that Alpine uses musl over glibc, which can introduce some niche issues.

For instance, some Python packages provide pre-built wheels for glibc but not for musl, requiring extra build steps.
IIRC we added matplotlib as a dependency, which increased our build times by several minutes because of that.

Some Go programs also didn't execute or even compile properly because of some incompatibility with musl?
It's been a while, I can unfortunately not remember the precise error...
Our use case here was either siegfried for file format detection, or Flask-Minify.

We've switched over to Wolfi as a base, which is very similar to Alpine but e.g. uses glibc instead.
This fixed our problems.

Here's a few more insights about glibc vs. musl, some of which I can confirm from experience: https://edu.chainguard.dev/chainguard/chainguard-images/about/images-compiled-programs/glibc-vs-musl/

[Samk13]

Thanks for sharing @max-moser, that’s a very good point 👍
Although we haven’t hit any issues yet in our instance, the musl vs glibc concerns you mention are exactly the kind of edge cases we want to avoid.
Wolfi sounds like an interesting middle ground that I wasn’t aware of before.

We should bring this up again in the upcoming maintainers meeting and decide on the base image there. I’ll add it as an agenda point.

[Samk13]

Update from maintainer summary on this change last week:

• Support multiple Docker base images, not just one.
• AlmaLinux stays the default (CERN-maintained).
• Debian/Alpine/Wolfi only if clearly maintained and proven to work.
• Maintenance follows ownership: whoever provides an image maintains it.
• Cookiecutter stays opinionated: fewer options, strong defaults, no setup wizard menus.
• Tooling: move to fixed defaults (uv, pnpm), reduce toggles.

Alignment with the maintainer call agreement ( AI analysis)

✅ Cookiecutter simplification achieved
Database and search choices were removed, defaults are enforced (PostgreSQL + OpenSearch), and build tooling is standardized on uv/pnpm. This matches the “fewer prompts, strong defaults” direction.

✅ Tooling defaults implemented
pipenv is fully removed in favor of uv, scripts and docs updated accordingly. This aligns with the agreed tooling direction.

⚠️ Docker direction partially aligned
The Dockerfile documents that multiple base-image flavors live in docker-invenio, but the template currently ships a single Alpine-based image as the Dockerfile currently combines build and runtime logic. This is intended to be moved to invenio-docker once the base-image structure is finalized.

[max-moser]

Question: Should we add named volumes in the docker-*.yml files?

In my experience, that's always something of the first things that I add on a fresh setup, to make the setup less brittle.
E.g. have data survive across container rebuilds, which is a major point of annoyance for me personally whenever setting up a new instance.

[Samk13]

Updated here added named volumes for persistence across rebuilds.

Do you think we should also move the CHANGE_ME values in docker-services to env vars and add a .env.example?

[max-moser]

From my perspective, I would very much welcome fewer hard-coded secrets, be it through .env (which I generally use) or other mechanisms!

Would you suggest adding a .env file as part of the template, filled out with some default values?
That sounds good to me; the only thing that I could imagine is the files starting with . being hidden from ls by default.
That's why I've come to change the order in the naming, usually to example.env, e.g. with some explainer text; cf. KSTU setup

Perhaps we could find some inbetween way, like instantiating both .env and example.env, and adding .env to .gitignore?
What do you think?

I'm open for alternatives!

[Samk13]

I intentionally avoided adding broader app-level variables to example.env at this stage, except INVENIO_SECRET_KEY, because otherwise we start mixing two concerns service/container and general Invenio application.
WDYT @fenekku @max-moser?
c9537ab (this PR)

[max-moser]

Personally, I'm fine with adding all kinds of configuration in .env; at TUW we've been doing that for a good while already with the env_file: .env property.
Admittedly it's true that our .env files are quite messy with all sorts of config in there, but so far this hasn't been a huge issue for me...
At least for us, the benefit of simplicity outweighs the downside of mixed concerns.

If splitting service config from app config is a concern, would it be feasible to have two different config files (that are excluded from version control), e.g. .invenio-env and .containers-env, or perhaps even an invenio-private.cfg (a Python file which gets loaded by invenio.cfg if it exists [1]), along with a .env for services?

[1] Should of course not be built into the container images upon build but only mounted; otherwise anybody with access to the image gets to see the secrets. Getting access to images is of course generally much easier than getting access to the running containers.

[Samk13]

I think I'd still lean towards a single .env eventually, but since this PR is primarily about upgrading the cookiecutter, I'd prefer to keep the current separation for now.

Introducing a new configuration approach (single .env, multiple env files, loading additional config files, etc.) is a broader change that deserves its own discussion and PR.
That would let us evaluate the trade-offs independently of the cookiecutter upgrade and keep the scope of this PR "focused".
We already have enough changes here that make the PR intimidating enough to review as it is 😅.

As there's interest, I'd revert this change and open a follow-up PR to consolidate the configuration and move the remaining CHANGE_ME values into a single .env if that's okay with you.

[Samk13]

I moved the .env changes here: #333
We need to merge this one first.

[max-moser]

Agreed, splitting that part out of this PR is probably best 👍

[fenekku]

Thank you for all these updates! There are obviously work outside of this PR that needs to be done before it can be used. I really appreciate how it cleans up a lot of cruft and drops technologies not part of the guaranteed happy path.

[fenekku]

Let's keep setuptools for now I would say since this is what is used by every other Invenio(RDM) package. We can have a maintainer meeting to discuss build systems if people have strong opinions.

[Samk13]

I added Hatchling mainly to align with the Zenodo setup, which has already moved to.
That said, I don't have a strong opinion on the build backend itself.

[fenekku]

I thought the zenodo switch was a test-it-out situation. Rather we keep with setuptools but it's not a blocker for me then. I'd be curious to hear the arguments for hatchling.

[fenekku]

This would change defaults. I don't have a strong opinion, but worth making it clear to other reviewers.

[Samk13]

Yeah, for other reviewers, this intentionally changes the old default.

This cookiecutter is mostly used for development and demos, where you typically want a running instance fast without long time indexing. I found myself selecting yes way more often than not, and occasionally missing the prompt and having to start over, which is why I changed this default.

[fenekku]

✔️

[Samk13]

eq is numeric comparison in POSIX shell. this variable is a string

[fenekku]

Thanks for the updates! This is good by me. Just need the public docker images out. I will see if I can do anything about it this week.

[fenekku]

With OR2026 I haven't had the time to look in docker-invenio hosting, but will do so next week ( 🤞 )

[fenekku]

I didn't have time to investigate the docker hosting on the docker-invenio repo directly and won't until next week, but this LGTM and is better merged in to get more people to test it out. So merging :) .