Skip to content

sam local start-api fails to move /var/rapid/aws-lambda-rie-x86_64 in docker image for serverless function #3891

Closed
Closed
sam local start-api fails to move /var/rapid/aws-lambda-rie-x86_64 in docker image for serverless function#3891
Labels
area/local/start-apisam local start-api commandmaintainer/need-followupstage/bug-reproThe issue/bug needs to be reproducedstage/needs-investigationRequires a deeper investigationtype/bug

Description

@llacroix
llacroix
opened on May 19, 2022

Description:

Accessing a lambda using a custom docker image with python awslambdaric fails with:

samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

Steps to reproduce:

Create a serverless function with a custom docker image that uses awslambdaric try to access it through an HttpApi locally.

Observed result:

Invoking Container created from httphandler:latest
Image was not found.
Removing rapid images for repo httphandler
Building image..............
Failed to build Docker Image
NoneType: None
Exception on / [GET]
Traceback (most recent call last):
  File "flask/app.py", line 2447, in wsgi_app
  File "flask/app.py", line 1952, in full_dispatch_request
  File "flask/app.py", line 1821, in handle_user_exception
  File "flask/_compat.py", line 39, in reraise
  File "flask/app.py", line 1950, in full_dispatch_request
  File "flask/app.py", line 1936, in dispatch_request
  File "samcli/local/apigw/local_apigw_service.py", line 357, in _request_handler
  File "samcli/commands/local/lib/local_lambda.py", line 144, in invoke
  File "samcli/lib/telemetry/metric.py", line 230, in wrapped_func
  File "samcli/local/lambdafn/runtime.py", line 177, in invoke
  File "samcli/local/lambdafn/runtime.py", line 88, in create
  File "samcli/local/docker/lambda_container.py", line 94, in __init__
  File "samcli/local/docker/lambda_container.py", line 236, in _get_image
  File "samcli/local/docker/lambda_image.py", line 164, in build
  File "samcli/local/docker/lambda_image.py", line 278, in _build_image
samcli.commands.local.cli_common.user_exceptions.ImageBuildException: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1
2022-05-19 01:01:55 127.0.0.1 - - [19/May/2022 01:01:55] "GET / HTTP/1.1" 502 -

Expected result:

Should just work or tell how to fix the issue with moving rapid/aws-lambda-rie

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

  1. OS: Fedora
  2. sam --version: 1.50.0
  3. AWS region: us-east-1

Activity

  1. torresxb1 commented on May 20, 2022

    @torresxb1
    torresxb1
    on May 20, 2022
    Contributor

    Thanks for bringing this to our attention! Could you include a basic sample template, handler, and Dockerfile (and anything else necessary) so that we could reproduce this issue more easily and accurately?

  2. torresxb1
    added
    type/bug
    blocked/more-info-neededMore info is needed from the requester. If no response in 14 days, it will become stale.
    area/local/start-apisam local start-api command
    stage/bug-reproThe issue/bug needs to be reproduced
    on May 20, 2022

  3. llacroix commented on May 23, 2022

    @llacroix
    llacroix
    on May 23, 2022
    Author

    Yes, I'll give it a look when I'll have time

  4. aws-sam-cli-bot
    added
    maintainer/need-followup
    on May 26, 2022
  5. moelasmar
    removed
    maintainer/need-followup
    on Jun 1, 2022
  6. aws-sam-cli-bot
    added
    maintainer/need-followup
    on Jun 2, 2022

  7. mjvirt commented on Sep 6, 2022

    @mjvirt
    mjvirt
    on Sep 6, 2022

    I have just run into this issue and it affects sam local invoke etc.

    TL;DR: Lambda functions default to Linux/x86_64 architecture even when run locally with SAM. On Linux for arm/aarch(64) only the Docker Engine/CE is available to install. Unlike for Docker Desktop, CE doesn't include QEMU emulation configuration for other platforms (specifically Linux/x86_64). Third-party binfmt is needed. This on very light testing appears to resolve the issue for SAM.

    Longer version:

    docker ps -a and info shows the command that fails:

    "mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie"

    Additional environment details:

    OS: Fedora Linux 36 (ARM64) - Parallels VM on Macbook M1
sam --version: SAM CLI, version 1.56.0
AWS region: eu-central-1

    It is fairly easy to reproduce. Just create the hello-world sam app via $ sam init in an empty directory:
    1 - AWS Quick Start Templates
    1 - Hello World Example

    $ sam local invoke HelloWorldFunction
Invoking app.lambda_handler (python3.9)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-python3.9
Building image...................
Failed to build Docker Image
NoneType: None
Error: Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

    The problem is due to a platform mismatch - attempting to run x86_64 images on arm64 architecture. Ordinarily with Docker Desktop on x86_64 it's possible to emulate other architectures such as arm64.

    Per the AWS SAM document here https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install-linux.html it states:

    On Linux, to build and run Lambda functions with a different instruction set architecture than your host machine, you must take additional steps to configure Docker. For example, to run arm64 functions on an x86_64 machine, you can run the following command to configure the Docker daemon: docker run --rm --privileged multiarch/qemu-user-static --reset -p yes.

    However, multiarch/qemu-user-static only supports the x86_64 host architecture per https://github.com/multiarch/qemu-user-static/.

    I did find this and have managed to get it working for SAM: https://github.com/tonistiigi/binfmt.

  8. aws-sam-cli-bot
    added
    maintainer/need-followup
    and removed
    maintainer/need-followup
    on Sep 7, 2022
  9. torresxb1
    added
    stage/needs-investigationRequires a deeper investigation
    and removed
    blocked/more-info-neededMore info is needed from the requester. If no response in 14 days, it will become stale.
    on Dec 5, 2022

  10. wangzhe3224 commented on Jan 24, 2023

    @wangzhe3224
    wangzhe3224
    on Jan 24, 2023

    Hi I am running into same issue, any update on the thread please?

  11. 3 remaining items

  12. chrisrhymes commented on Jan 30, 2023

    @chrisrhymes
    chrisrhymes
    on Jan 30, 2023

    Hi @mjvirt I'm also having this same issue when using an M1 Mac running macOS 13.1.

    I changed the template.yml Architecture to arm64, reran sam build and then sam local invoke --no-event worked for me.

    Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: hello-world/
      Handler: app.lambdaHandler
      Runtime: nodejs18.x
      Architectures:
        - arm64 # Update this line for it to work locally

  13. mjvirt commented on Jan 30, 2023

    @mjvirt
    mjvirt
    on Jan 30, 2023

    Hi @chrisrhymes and others, maybe I wasn't clear in my original message above, but this last line works:

    I did find this and have managed to get it working for SAM: https://github.com/tonistiigi/binfmt (other CPU architecture emulation on arm64/M1)

    In other words you can emulate x86_64 on arm64 - and it works fine with SAM/CDK - at least insofar as I've tested it (just use docker run --privileged --rm tonistiigi/binfmt --install all to install the available platform emulators; or you can be more picky...).

  14. chrisrhymes commented on Jan 30, 2023

    @chrisrhymes
    chrisrhymes
    on Jan 30, 2023

    Thanks, I should have said before, but I tried running the command, but only for arm64 --install arm64 and it still didn't work.

    I just tried the docker run --privileged --rm tonistiigi/binfmt --install all command which all looks ok, ran sam build and then sam local start-api and I get the same error as before:

    Error building docker image: The command '/bin/sh -c mv /var/rapid/aws-lambda-rie-x86_64 /var/rapid/aws-lambda-rie && chmod +x /var/rapid/aws-lambda-rie' returned a non-zero code: 1

  15. ixolt commented on Jan 30, 2023

    @ixolt
    ixolt
    on Jan 30, 2023

    @chrisrhymes I managed to fix this by executing the following commands

    $ brew uninstall aws-sam-cli
$ brew install rust
$ brew install aws/tap/aws-sam-cli

  16. pub-technology commented on Jan 31, 2023

    @pub-technology
    pub-technology
    on Jan 31, 2023

    @chrisrhymes I managed to fix this by executing the following commands

    $ brew uninstall aws-sam-cli
$ brew install rust
$ brew install aws/tap/aws-sam-cli

    Got an error from the last command brew install aws/tap/aws-sam-cli

    Already downloaded: /Users/haithai/Library/Caches/Homebrew/downloads/563186ab187d77b0c27b2db3c0f3fc5d25300867da5d18083dd099e13ce4431b--aws-aws-sam-cli-v1.71.0-0-gc681067.tar.gz
Error: aws-sam-cli: SHA256 mismatch
Expected: c27333fb067fd0ffef723c48ed9f26272061d6f3299f5b00ee3d421cc607e0c5
  Actual: e05774257d3bbf0080d7d0f0114991bfaf44a9d5bc464cc1427ccc98a4c99cd6
    File: /Users/haithai/Library/Caches/Homebrew/downloads/563186ab187d77b0c27b2db3c0f3fc5d25300867da5d18083dd099e13ce4431b--aws-aws-sam-cli-v1.71.0-0-gc681067.tar.gz
To retry an incomplete download, remove the file above

    I tried to remove and run install again - The error still happening @ixolt

  17. pub-technology commented on Jan 31, 2023

    @pub-technology
    pub-technology
    on Jan 31, 2023

    Thanks @ixolt

    The new version of AWS SAM CLI - (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ The problem Fixed

    $ brew uninstall aws-sam-cli
$ brew install rust
$ brew tap aws/tap
$ brew install aws-sam-cli

  18. garyaparker commented on Feb 1, 2023

    @garyaparker
    garyaparker
    on Feb 1, 2023

    I'm having the same issue on SAM 1.71.0 (latest) and the emulator install is not resolving it.

  19. sbilello commented on Feb 3, 2023

    @sbilello
    sbilello
    on Feb 3, 2023

    I solved it by

    $ rm -rf /usr/local/bin/sam

    Installing sam via pkg file from https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html
    MacBook Pro M1

  20. luisdelarosa commented on Feb 4, 2023

    @luisdelarosa
    luisdelarosa
    on Feb 4, 2023

    For an M1 Mac, you can pass the architecture flag -a to sam init like: sam init -a arm64.

    Or you could change your Architectures line in your template.yaml like @chrisrhymes described above.

    I wrote about it a bit more in this Stack Overflow answer: https://stackoverflow.com/questions/75216548/aws-sam-cli-throws-error-error-building-docker-image/75342572#75342572

    I think the ideal solution would be for SAM to detect your architecture and act accordingly when creating a Docker image for emulating Lambda locally, while allowing for the Architectures to specify the deployment architecture of the Lambda function. Currently it looks SAM uses the Architectures line for both.

  21. sriram-mv commented on Feb 13, 2023

    @sriram-mv
    sriram-mv
    on Feb 13, 2023
    Contributor

    Moving this issue to discussions: https://github.com/aws/aws-sam-cli/discussions/4705

  22. sriram-mv
    closed this as completedon Feb 13, 2023

  23. github-actions commented on Feb 13, 2023

    @github-actions
    github-actionsBot
    on Feb 13, 2023
    Contributor

    ⚠️COMMENT VISIBILITY WARNING⚠️

    Comments on closed issues are hard for our team to see.
    If you need more assistance, please either tag a team member or open a new issue that references this one.
    If you wish to keep having a conversation with other community members under this issue feel free to do so.

  24. sriram-mv
    reopened this on Feb 13, 2023
  25. aws
    locked and limited conversation to collaborators on Feb 13, 2023
  26. sriram-mv
    converted this issue into a discussion #4708 on Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/local/start-apisam local start-api commandmaintainer/need-followupstage/bug-reproThe issue/bug needs to be reproducedstage/needs-investigationRequires a deeper investigationtype/bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions