Missing encryption key to decrypt file with. Ask your team for your master key and write it to /rails/config/master.key or put it in the ENV['RAILS_MASTER_KEY']

[jasonfb]

Testing using MRSK 0.11.0

Please bear with me as I am working through numerous MRSK setup issues and I am new to MRSK and relatively new to Docker.

Initially started building this app using the default config/deploy.yml file, making small tweeks and re-testing each time as I worked through various issues.

Somewhere along the way, I uncommented this code in the deploy.yml file:

(1)

env:
  secret:
    - RAILS_MASTER_KEY

Note that I just uncommented the code, leaving the characters RAILS_MASTER_KEY as just a string. It is not clear to me reading the instructions and default of the file whether it should have been replaced by my real master key (or not?).

I was confused by the comment in the default file that reads: "(secrets come from .env)" which made me think that the string RAILS_MASTER_KEY would get replaced by the secret in the .env file.

Somewhere along the way, someone on SLACK suggested this as a way to bring your Ruby version, bundler version, and secret key into the MRSK setup:

(2)

builder:
  args:
    RUBY_VERSION: "<%= Pathname.pwd.join('.ruby-version').read.strip.delete_prefix('ruby-') %>"
    BUNDLER_VERSION: "<%= Bundler::LockfileParser.new(Pathname.pwd.join('Gemfile.lock').read).bundler_version %>"
    RAILS_MASTER_KEY: "<%= ENV['RAILS_MASTER_KEY'] %>"

(Notice that these are builder args, not env secret)

Should I also do this same syntax for env secret configured in config/deploy.yml?

If I do this:
(3)

env:
  secret:
    - "<%= ENV['RAILS_MASTER_KEY'] %>"

then I can no longer do anything with MRSK at all --it's like I'm borked now --- mrsk lock release and mrsk deploy both now tell me:

ERROR (KeyError): key not found: "f0e5e49fXXXXXXXXXXXXXXXXXXXXXX"

(rest of key has been obscured for privacy but I confirmed it matches my real secret key. )

If I use the original syntax (#1), then the lock mechanism works as expected to release the lock, but when I try to deploy, I get:

 > [linux/arm64 build 10/10] RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile:
#0 0.977 Missing encryption key to decrypt file with. Ask your team for your master key and write it to /rails/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].
------
Dockerfile:53
--------------------
  51 |     
  52 |     # Precompiling assets for production without requiring secret RAILS_MASTER_KEY
  53 | >>> RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile
  54 |     
  55 |     

Note that in production.rb this line has been UNCOMMENTED:

  config.require_master_key = true

as well, my app contains config/master.key (but it is not being copied into the container by Dockerfile), and my app also has the RAILS_MASTER_KEY set as an .env variable.

if I locally run RAILS_ENV=production rails s, I get these result:

3.1.2 :005 > Rails.application.credentials.key_file_contents
 => nil 
3.1.2 :006 > Rails.application.credentials.env_key
 => "RAILS_MASTER_KEY" 
3.1.2 :007 > Rails.application.credentials.key_file_contents
 => nil 
3.1.2 :008 > ENV['RAILS_MASTER_KEY']
 => nil 

if I instead run RAILS_ENV=development rails s, I get this:

3.1.2 :002 > Rails.application.credentials.env_key
 => "RAILS_MASTER_KEY" 
3.1.2 :003 > Rails.application.credentials.key_file_contents
 => nil 
3.1.2 :001 > ENV['RAILS_MASTER_KEY']
 => "f0e5e49fXXXXXXXXXXXXXXXXXXXXXXXXX" 

The Dockerfile is the same Dockerfile found on https://www.driftingruby.com/episodes/deploying-with-mrsk and the line that it fails on is:

# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile

[jasonfb]

https://github.com/bkeepers/dotenv explicitly discusses that the original intention of the Gem is to be used on development + test, but not in production. It seems it CAN be used in production, but that wans't the original intention.

MRSK makes dotenv a specific dependency for all environments, changing for most of us what was once a dependency used only in development+test into dev+test+prod

What is confusing is generally, it is advised against to put your master key into source control -- this is the whole point of having environment variables separated from the source code.

So, what do I do? I could see two ways forward:

(1) put my rails master key into .env.production -- a no-no based on security, or

(2) if using a config/master.key file instead, should my Dockerfile be copying that file into the Docker image during the build steps?

[jasonfb]

cc: @driftingruby — I'm not sure if this problem is related to MRSK or the Dockerfile which I had copied from your site https://www.driftingruby.com/episodes/deploying-with-mrsk

[jasonfb]

I tried adding this but it didn't work as expected

COPY --link config/master.key ./

ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref iu1slb3t4migzukxu8utkd59t::wo21gohuj7e7t4qunsw6t9nfs: "/config/master.key": not found

am I going down the wrong path here?

[jasonfb]

@driftingruby says the Dockerfile which I had copied into my Rails 7.0.4 app originally came from here in Rails 7.1

https://github.com/rails/rails/blob/10285e977b7be785e34a107811aec7e363d5621a/railties/lib/rails/generators/rails/app/templates/Dockerfile.tt#L2

[jasonfb]

if I have config.require_master_key = true enabled in production.rb, the error when running mrsk deploy is:

#25 0.815 Missing encryption key to decrypt file with. Ask your team for your master key and write it to /rails/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

if I COMMENT OUT `config.require_master_key = true in production.rb, the error is instead this:

[4:17](https://rubyonrails-link.slack.com/archives/C3EEPTJHG/p1683058659069799)
#0 0.876 ArgumentError: Missing `secret_key_base` for 'production' environment, set this string with `bin/rails credentials:edit`

Either way, if I run RAILS_ENV=production rails c, there's no env setting for RAILS_MASTER_KEY:

% RAILS_ENV=production rails c
Loading production environment (Rails 7.0.4.3)
irb: warn: can't alias context from irb_context.
3.1.2 :001 > ENV['RAILS_MASTER_KEY']
 => nil 

[jasonfb]

tested on MRSK 0.12.0, same results

[jasonfb]

next step:

• Tobias suggests :
  "I'd add it at bin/rails credentials:edit --environment production"

[justinallenmarsh]

Did you ever resolve this issue? I'm running into the same problem.

[jasonfb]

@jmarsh24 - working on it again today, will update below

[mikker]

I managed to work around this by supplying my master key during build like so

RUN --mount=type=secret,id=RAILS_MASTER_KEY \
  RAILS_MASTER_KEY=$(cat /run/secrets/RAILS_MASTER_KEY) ./bin/rails assets:precompile

And deploy.yml

builder:
  secrets:
    - RAILS_MASTER_KEY

There are likely some security concerns with this so looking forward to Rails 7.1 but until then, this works for me.

[justinallenmarsh]

@mikker I don't think is a good idea because it exposes your secrets in the build. I was asking around on slack regarding why this is happening and it has to do with having encrypted credentials in our initializers and this build step is trying to run the app to compile the assets but requires these credentials. They aren't necessary for rails to compile the assets though...

So the solution is here under Access to Environment Variables at Built Time...

https://fly.io/docs/rails/getting-started/existing/

This checks to see if the credentials are available before running the initializer. The dummy key line allows rails to boot up without needing the RAILS_MASTER_KEY, compiles your assets, and then lets you continue with the build process. You shouldn't need your RAILS_MASTER_KEY in the dockerfile.

[justinallenmarsh]

@dhh

[jasonfb]

when you say "exposes your secrets in the build" — what attack vector are you concerned with, specifically?
For me, the only attack vector I've been concerned with is NOT exposing the Rails master key in source control, I haven't really thought about "exposing the secrets in the build" -- do you mean if someone has hacked my computer (doing the build), they'll be able to see my master key while I run mrsk deploy?
because if they hacked my computer they could see the master key anyway.

[justinallenmarsh]

I'm not familiar with the intricacies of a docker build, it's possible that the key could be in the logs. I don't know why the RAILS_MASTER_KEY isn't provided in the default dockerfile. I figured that it's for security reasons.

[jasonfb]

@mikker 's solution for Rails 7.0 gives me

 => ERROR [stage-0 7/9] RUN --mount=type=secret,id=RAILS_MASTER_KEY   RAILS_MASTER_KEY=$(cat /run/secrets/RAILS_MASTER_KEY) ./bin/rails a  0.2s
------
 > [stage-0 7/9] RUN --mount=type=secret,id=RAILS_MASTER_KEY   RAILS_MASTER_KEY=$(cat /run/secrets/RAILS_MASTER_KEY) ./bin/rails assets:precompile:
#0 0.153 cat: /run/secrets/RAILS_MASTER_KEY: No such file or directory
#0 0.154 /bin/sh: 1: ./bin/rails: not found
------
Dockerfile:18
--------------------
  17 |     
  18 | >>> RUN --mount=type=secret,id=RAILS_MASTER_KEY \
  19 | >>>   RAILS_MASTER_KEY=$(cat /run/secrets/RAILS_MASTER_KEY) ./bin/rails assets:precompile
  20 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c RAILS_MASTER_KEY=$(cat /run/secrets/RAILS_MASTER_KEY) ./bin/rails assets:precompile" did not complete successfully: exit code: 127

[jasonfb]

this was happening because I was attempting to use SECRET_KEY_BASE_DUMMY in a Rails 7.0 app but that feature isn't introduced until Rails 7.1

it has to do with having encrypted credentials in our initializers and this build step is trying to run the app to compile the assets but requires these credentials. They aren't necessary for rails to compile the assets though

I think the SECRET_KEY_BASE_DUMMY in Rails 7.1 alleviates the problem

You shouldn't need your RAILS_MASTER_KEY in the dockerfile.

If I'm setting this up on Rails 7.0 (which I am), someone on stack overflow showed me a way to pass the master key into the Dockerfile as an builder argument... I think that's the solution for Rails 7.0 and for Rails 7.1 the solution is SECRET_KEY_BASE_DUMMY

will test again this morning and update here.

Closing for now.

[justinallenmarsh]

If you do not use encrypted credentials in your initializers this will alleviate the problem. you don't need to put your rails_master_key in the build processes.

This still brings up an issue if you are converting a rails app that has any initializer using encrypted credentials (the recommended rails way) you will run into an issue because it will search for the RAILS_MASTER_KEY to access those ENV variables but won't be able to find it.

if you write it in an if statement that prevents the initialization of that variable then you prevent it.

Other than on fly.io, I don't think it's discussed. I think that instructions should be added to the readme.

[jasonfb]

I will test and get back to you

[jasonfb]

@jmarsh24 -- The solution below makes it so you can pass the RAILS_MASTER_KEY into the build using Docker without having to commit it into source control.

This still brings up an issue if you are converting a rails app that has any initializer using encrypted credentials (the recommended rails way)

I think fundamentally you are correct about this, except that some of us do not agree that this is "the recommended way". In fact, although it is a common pitfall and problem, most people recommend that you do NOT refer to encrypted settings during the build process. (Heroku support and I have been back and forth about this very issue with regard to the Heroku pipeline which AFAIK requires you to remove all credentials from the build process to use it... essentially, for the very same root cause).

I do agree with you that this is tricky, but from what I can tell, the solution is straightforward (in my mind at least):

1. if you DON'T require credentials in your Rails precompile, you're already ahead of the game as this is the preferred strategy moving forward. For that, you can either use the Docker arg for Rails 7.0 (explained below) or use the bypass mechanism SECRET_KEY_BASE_DUMMY=1 starting with Rails 7.1

2. If you DO have a legacy app that requires credentials during precompile, then you should work towards removing that dependency in your app. If that's not possible for you at the moment, you won't be able to use SECRET_KEY_BASE_DUMMY=1 even after upgrading to Rails 7.1, and you're gonna need to fallback to the other strategy of putting the master key in as a builder arg.

I'm pretty sure that means there isn't a bug or issue, just some clarification which I hope this issue serves to do. I see @dhh had closed your other Issue with the same title, but from the perspective of my issue that originally brought me here, I feel satisfied this is not a bug but just my misunderstanding.

[jasonfb]

Solution

For Rails 7.0.x

Add this to Dockerfile any place before the line RUN bundle install --deployment

ARG RAILS_MASTER_KEY
ENV RAILS_MASTER_KEY=$RAILS_MASTER_KEY

~The precompile line in Dockerfile should be ~

RUN ./bin/rails assets:precompile

This will create the ENV based on the arg MRSK passes when running the container.

For Rails 7.1

Use the newly introduced SECRET_KEY_BASE_DUMMY=1 which tells the precompile step to bypass the secret key base while precompiling.

cross-posted to SO
https://stackoverflow.com/questions/76157925/rails-7-validate-secret-key-base-missing-secret-key-base-for-production-e/76190212

[jasonfb]

I still can't seem to get this working in Rails 7.0 actually 😞

[justinallenmarsh]

@jasonfb This is true brand new rails apps, but if somebody is on rails 7.1 and has initializers in the transitional way of requiring credentials without a if guard statement to prevent the initialization if RAILS_MASTER_KEY isn't present they will run into the same error.

[jasonfb]

Ok back at this after a break. Strangly none of these solutions work for me, including my own.

Re-opening discussion as this is still not resolved (going down the Rails 7.0 rabbit hole, not yet tested against Rails 7.1)

if I enableconfig.require_master_key = true in production.rb, the error changes to this instead:

#0 1.623 Missing encryption key to decrypt file with. Ask your team for your master key and write it to /app/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

I have the master key of course my question is how do I get it into into the Docker container

[jasonfb]

Incidentally so far I am able to get this working this by upgrading to rails 7.1.0.alpha and using SECRET_KEY_BASE_DUMMY=1 on the precompile step in Dockefile.

seems like there should be a way to do it for Rails 7.0 though, given that is still the officially latest version of Rails .

[jasonfb]

SOLUTION

Rails 7.0
SECRET_KEY_BASE=DUMMY

Rails 7.1
SECRET_KEY_BASE_DUMMY=1

cc: @jmarsh24

[yorsant]

What about previous versions of Rails? (6.0, 6.1)

[yorsant]

Where can I read more about the internals of this?

[gamesover]

I tried both methods, but neither works in Rails 6.0.

[jasonfb]

https://jasonfleetwoodboldt.com/courses/rails-7-crash-course/mrsk-on-amazon-ec2/

[SebSwan]

Your link get a error 404

[vokshirg]

Would it be good solution?

#config/environments/staging.rb
config.require_master_key = !ENV["SECRET_KEY_BASE_DUMMY"].present?

[victorhazbun]

Why would assets:precompile attempt to connect to the DB?

RUN --mount=type=secret,id=RAILS_MASTER_KEY \
    RAILS_MASTER_KEY=$(cat /run/secrets/RAILS_MASTER_KEY) SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile

In this case, I'm on Rails 8 using Kamal 2. No longer able to deploy.

[oleingemann]

I did a deep dive into my credentials setup and turned every stone. Regardless of what I did, I ended up with this:

#25 0.815 Missing encryption key to decrypt file with. Ask your team for your master key and write it to /rails/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

Turned out that this message was returned regardless if my keys were fine. The real problem was rails assets:precompile, not the keys.

Ones those were fixed, kamal was fine.