Skip to content

[PULP-658] Update usage of storage settings on minio profile #188

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gerrod3 merged 2 commits into from
Aug 4, 2025
Merged

[PULP-658] Update usage of storage settings on minio profile #188

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ebcc0ab
Update usage of storage settings on minio profile
pedro-psb Jul 22, 2025
ebedf33
Remove version from compose files
pedro-psb Jul 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .github/assets/profiles/_test/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
version: "3.7"

services:
test:
build:
context: "{OCI_ENV_DIR}/profiles/_test/"
dockerfile: "Dockerfile"
ports:
- "{SECOND_SERVICE_PORT}:80"
- "{SECOND_SERVICE_PORT}:80"
2 changes: 0 additions & 2 deletions base/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
---
version: "3.7"

services:
_base:
build:
Expand Down
5 changes: 3 additions & 2 deletions docs/dev/reference/profiles/pminio.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ instance is launched as a separate service and can be accessed inside the main c
`http://pminio:9000`. The Minio Client CLI,`mc`, is also installed and configured inside the main
container to talk to the Minio service.

Do not forget to add a new entry to `/etc/hosts` on the host machine for the `pminio` alias, like
so:
If you want to interact with Pulp distributions from your host,
add a new entry to `/etc/hosts` on the host machine for the `pminio` alias, like so:

```
127.0.0.1 localhost localhost4 pminio
::1 localhost localhost6 pminio
Expand Down
2 changes: 0 additions & 2 deletions profiles/galaxy_base/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
version: "3.7"

services:
_base:
build:
Expand Down
2 changes: 0 additions & 2 deletions profiles/galaxy_ui/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
version: "3.7"

services:
ui:
build:
Expand Down
4 changes: 1 addition & 3 deletions profiles/kafka/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
version: "3.7"

services:
kafka:
hostname: kafka
Expand All @@ -19,4 +17,4 @@ services:

volumes:
oci_kafka_data:
name: "{COMPOSE_PROJECT_NAME}_kafka"
name: "{COMPOSE_PROJECT_NAME}_kafka"
2 changes: 0 additions & 2 deletions profiles/local_fixtures/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
version: "3.7"

services:
local_fixtures:
image: pulp/pulp-fixtures
Expand Down
2 changes: 0 additions & 2 deletions profiles/opentelemetry_dev/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
version: "3.7"

services:

jaeger:
Expand Down
16 changes: 10 additions & 6 deletions profiles/pminio/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,22 @@
version: "3.7"

services:
pminio:
image: minio/minio
image: quay.io/minio/minio
environment:
- "MINIO_ACCESS_KEY={S3_ACCESS_KEY}"
- "MINIO_SECRET_KEY={S3_SECRET_KEY}"
- "MINIO_ROOT_USER={S3_ACCESS_KEY}"
- "MINIO_ROOT_PASSWORD={S3_SECRET_KEY}"
ports:
- "{MINIO_PORT}:9000"
- "{MINIO_CONSOLE_PORT}:9091"
volumes:
- "oci_pminio:/data"
depends_on:
- pulp
- _base
healthcheck:
test: curl -I http://pminio:9000/minio/health/live || exit 1
interval: 20s
timeout: 5s
retries: 5
start_interval: 5s
command: "server /data --console-address ':9091'"

volumes:
Expand Down
35 changes: 23 additions & 12 deletions profiles/pminio/init.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,37 @@

set -e

MC_DIR="$HOME/minio-binaries"
MC_BINARY="$MC_DIR/mc"

if command -v "$MC_BINARY" &> /dev/null; then
echo "pminio/init.sh script already run."
exit 0
fi

echo "Installing S3 backend"
pip3 install --prefix /usr/local/ --no-cache-dir django-storages[boto3]

echo "Installing Minio client"
curl https://dl.min.io/client/mc/release/linux-amd64/mc \
--create-dirs \
-o $HOME/minio-binaries/mc

chmod +x $HOME/minio-binaries/mc
export PATH=$PATH:$HOME/minio-binaries/

echo "export PATH=$PATH:$HOME/minio-binaries/" >> /etc/profile.d/minioclient.sh
-o "$MC_BINARY"
chmod +x "$MC_BINARY"
export PATH=$PATH:$MC_DIR
echo "export PATH=$PATH:$MC_DIR" >> /etc/profile.d/minioclient.sh

echo "Setting up Minio backend"
mc config host add s3 ${PULP_AWS_S3_ENDPOINT_URL} ${PULP_AWS_ACCESS_KEY_ID} ${PULP_AWS_SECRET_ACCESS_KEY} --api S3v4
mc config host rm local
ENDPOINT_URL="${PULP_STORAGES__default__OPTIONS__endpoint_url}"
Copy link
Copy Markdown
Contributor

@gerrod3 gerrod3 Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need to install the django-storages[boto3] anymore, it's included in the base container for a while now. Also, looking at the logs this script might get ran a second time if the Pulp service restarts, so can we update the logic to handle running a second time?

Copy link
Copy Markdown
Member Author

@pedro-psb pedro-psb Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've removed and it looks like it's not available. The base container oci-env uses it ghcr.io/pulp/pulp-ci-centos9:latest and it has just a few packages installed.

$ podman pull ghcr.io/pulp/pulp-ci-centos9:latest
$ podman run --rm -ti ghcr.io/pulp/pulp-ci-centos9:latest pip list
Package         Version
--------------- ----------
cffi            1.14.5
cryptography    36.0.1
dbus-python     1.2.18
gpg             1.15.1
libcomps        0.1.18
Mako            1.1.4.dev0
Markdown        3.3.4
MarkupSafe      1.1.1
pip             21.3.1
ply             3.11
psycopg2        2.8.6
pycparser       2.20
python-dateutil 2.8.1
rpm             4.16.1.3
setuptools      53.0.0
six             1.15.0
systemd-python  234

This doesn't feel the right place to install it, do you have a better idea on where to put it? Or may I keep it here?

Copy link
Copy Markdown
Contributor

@gerrod3 gerrod3 Jul 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's in the all-in-one container, not the base container. In the single container we do pip install pulpcore[s3,azure,google]. There doesn't seem to be a good place to put it here.

ACCESS_KEY="${PULP_STORAGES__default__OPTIONS__access_key}"
SECRET_KEY="${PULP_STORAGES__default__OPTIONS__secret_key}"
BUCKET_NAME="${PULP_STORAGES__default__OPTIONS__bucket_name}"
mc alias set s3 ${ENDPOINT_URL} ${ACCESS_KEY} ${SECRET_KEY} --api S3v4
mc alias rm local
mc admin info s3

if $(mc find s3/${PULP_AWS_STORAGE_BUCKET_NAME}) ; then
echo "Minio ${PULP_AWS_STORAGE_BUCKET_NAME} bucket already created"
if mc find s3/${BUCKET_NAME} &> /dev/null; then
echo "Minio ${BUCKET_NAME} bucket already created"
else
echo "Creating Minio bucket ${PULP_AWS_STORAGE_BUCKET_NAME}"
mc mb s3/${PULP_AWS_STORAGE_BUCKET_NAME}
echo "Creating Minio bucket ${BUCKET_NAME}"
mc mb s3/${BUCKET_NAME}
fi
14 changes: 7 additions & 7 deletions profiles/pminio/pulp_config.env
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
PULP_AWS_ACCESS_KEY_ID={S3_ACCESS_KEY}
PULP_AWS_SECRET_ACCESS_KEY={S3_SECRET_KEY}
PULP_AWS_STORAGE_BUCKET_NAME=pulp
PULP_AWS_S3_SIGNATURE_VERSION=s3v4
PULP_AWS_S3_ADDRESSING_STYLE=path
PULP_AWS_S3_ENDPOINT_URL={S3_ENDPOINT_URL}
PULP_DEFAULT_FILE_STORAGE=storages.backends.s3boto3.S3Boto3Storage
PULP_STORAGES__default__OPTIONS__access_key={S3_ACCESS_KEY}
PULP_STORAGES__default__OPTIONS__secret_key={S3_SECRET_KEY}
PULP_STORAGES__default__OPTIONS__bucket_name=pulp
PULP_STORAGES__default__OPTIONS__signature_version=s3v4
PULP_STORAGES__default__OPTIONS__addressing_style=path
PULP_STORAGES__default__OPTIONS__endpoint_url={S3_ENDPOINT_URL}
PULP_STORAGES__default__BACKEND=storages.backends.s3boto3.S3Boto3Storage
2 changes: 0 additions & 2 deletions profiles/pulp_ui/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
version: "3.7"

services:
ui:
build:
Expand Down