Swarming: Patches Auth default credentials at startup

[IvanBM18]

Fuzzing tasks in swarming have an intermittent error in which startup of any gcp library or module fails with:

  File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/datastore/ndb_init.py", line 39, in _client
    _ndb_client = ndb.Client(project=utils.get_application_id())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/client.py", line 145, in __init__
    super(Client, self).__init__(
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/client/__init__.py", line 320, in __init__
    _ClientProjectMixin.__init__(self, project=project, credentials=credentials)
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/client/__init__.py", line 268, in __init__
    project = self._determine_default(project)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/client.py", line 249, in _determine_default
    return _determine_default_project(project)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/client.py", line 70, in _determine_default_project
    project = _helpers._determine_default_project(project=project)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/_helpers/__init__.py", line 152, in _determine_default_project
    _, project = google.auth.default()
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/scratch0/clusterfuzz/src/third_party/google/auth/_default.py", line 692, in default
    raise exceptions.DefaultCredentialsError(_CLOUD_SDK_MISSING_CREDENTIALS)
google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

This happens because the timeout window for calling the default credentials is very short, and since we are using a luci-auth server the requests tend to take a little bit more time than the server in GCE bots.

To fix this, we are manually patching the google.auth.default module to return some credentials that do come from the swarming bot.

Tests performed

This changes have been present in the dev env since April 15th, which allowed us to perform some fuzzing sessions in swarming like this one

[IvanBM18]

Note: We have the args and kwargs in the method signature because otherwise if we call the default() method with any args an exception would be thrown because the patch didn't matched the expected signature.

[jardondiego]

Changing the function signature to this should fix it without having this pylint directive.

def patched_default_credentials(*_args, **_kwargs) -> tuple[compute_engine.Credentials, str] | None:

[IvanBM18]

This will be logged into swarming only, not GCP, since when this executes we still haven't setup GCP logging

[letitz]

I'm confused and need some more details. I was assuming that luci-auth would set up some credentials on disk somewhere, that GCP libs would then pick up. You're saying that it runs a local server that GCP libs talk to instead?

In general, do you have a bug link for this that we can discuss more on?

[IvanBM18]

I'm confused and need some more details. I was assuming that luci-auth would set up some credentials on disk somewhere, that GCP libs would then pick up. You're saying that it runs a local server that GCP libs talk to instead?

In general, do you have a bug link for this that we can discuss more on?

Sure heres the bug im using:
b/502273525

Regarding luci-auth....
This doc explains in more detail how service account and auth works in a swarming task:
go/swarming-service-accounts
TL;DR
It setups a "mock" metadata server in the swarming host(thats why we need to pass a bunch of env vars and setup the --network=host at the container) said server then uses the callers account(that being me or the appengine account) to impersonate the service account, and then it generates short lived Oauth tokens.

You can also think of it like its a kind of a proxy running in your swarming bot.