Lambda ARM (Apple M1): AWS SAM Golang lambda debugging does not work

Description:

After creating a plain vanilla AWS SAM application with sam init for golang, and then using the AWS Toolkit provided code lens in VS Code, to create an AWS SAM template file based debug configuration, I get the following error when trying to debug the lambda code.

Running the code without debugging works.

Note:
I had raised this issue in aws/aws-toolkit-vscode as aws/aws-toolkit-vscode#2020. There was a suggestion in that thread that I should raise an issue here.

Steps to reproduce:

Create a new AWS SAM application using Go with sam init.
Select the Hello world template.
In the main.go file, create a debug configuration from the code lens provided by the AWS Toolkit and associated with the handler.
Set a breakpoint and start debugging.

Observed result:

2021-09-11 10:55:02 [INFO]: The Delve repo was not found in your GOPATH. Downloading in a temporary directory...
2021-09-11 10:55:03 [INFO]: Preparing to debug locally: Lambda "hello-world"
2021-09-11 10:55:03 [INFO]: Building SAM application...
2021-09-11 10:55:03 [INFO]: Running command: (not started) [/opt/homebrew/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctkeYZ6BB/output --template /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world/app___vsctk___template.yaml --base-dir /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world]
2021-09-11 10:55:03 [INFO]: Building codeuri: /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world runtime: go1.x metadata: {} functions: ['helloworld']

2021-09-11 10:55:03 [INFO]: Running GoModulesBuilder:Build

2021-09-11 10:55:03 [INFO]: 
Build Succeeded

2021-09-11 10:55:03 [INFO]: 
Built Artifacts  : ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output
Built Template   : ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml
[*] Deploy: sam deploy --guided --template-file ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml
    

2021-09-11 10:55:04 [INFO]: Build complete.
2021-09-11 10:55:04 [INFO]: Starting SAM application locally
2021-09-11 10:55:05 [INFO]: AWS.running.command
Invoking hello-world (go1.x)
Requested to skip pulling images ...

Mounting /tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/helloworld as /var/task:ro,delegated inside runtime container
START RequestId: 7c546e9e-8558-4fde-8dd6-fc6946034850 Version: $LATEST
API server listening at: [::]:5858
2021-09-11T00:55:07Z info layer=debugger launching process with args: [/var/task/hello-world]
2021-09-11 10:55:07 [INFO]: Waiting for SAM application to start...
could not launch process: fork/exec /var/task/hello-world: function not implemented
2021/09/11 00:55:07 exit status 1
time="2021-09-11T00:55:07.765" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2021-09-11T00:55:07.773" level=error msg="INIT DONE failed: Runtime.ExitError"
2021-09-11 10:55:08 [INFO]: Attaching debugger to SAM application...

Command stopped: "sam local invoke"

From the output listed above, it seems to me that when using the debugger, it thinks that the function is not implemented.
2021-09-11 10:55:07 [INFO]: Waiting for SAM application to start...
could not launch process: fork/exec /var/task/hello-world: function not implemented

Other considerations

Go debugging works for vanilla go apps (non lambdas) on M1 based Apple Macs.
Go lambdas run successfully with sam local invoke on M1 based Apple Macs.
Go lambdas can be successfully debugged on x86 based Apple Macs.

Expected result:

Lambda debugging working with breakpoints being hit.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: macOS Big Sur 11.6 (M1) / Darwin arm64 20.6.0
sam --version: 1.31.0
AWS region: any (currently working in ap-south-1)
VS Code version: 1.16
AWS Toolkit extension version: 1.29.0
Go version: 1.17.1
dlv and dlv-dap version: 1.7.1 Build: $Id: 3bde2354aafb5a4043fd59838842c4cd4a8b6f0b $
Docker version: 20.10.8, build 3967b7d

Hello, just checking in to see if there is any update on this. My self interest here is to be able to work with Go based Lambdas on my M1 Mac and not rely on an intel based system, which is what I am doing now for just those Lambdas we use Go for. At the least it would be good to get some understanding regarding what is the cause for this issue from the aws-sam-cli team point of view?

@ashprao

Hey @ashprao, sorry for the late response. I confirmed that Go lambdas can be successfully debugged on x86 based Apple Macs. But I personally don't have a M1 Mac, I'll ask a team member to help with this.

@hawflau

Hello @hawflau, Thanks for getting to this. Would love to be able fully switch to my M1 based Mac for all of my development. :-)

To add more context. I did try debugging after updating, AWS CLI, AWS SAM, VS Code, VS Code AWS Toolkit to the latest versions and I am still facing the issue.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: macOS Big Sur 11.6 (M1) / Darwin arm64 20.6.0
aws --version: aws-cli/2.2.45 Python/3.8.8 Darwin/20.6.0 exe/x86_64 prompt/off
sam --version: SAM CLI, version 1.33.0
AWS region: any (currently working in ap-south-1)
VS Code version: 1.61.0 (Universal)
AWS Toolkit extension version: 1.31.0
Go version: 1.17.1
dlv and dlv-dap version: 1.7.1 Build: $Id: 3bde2354aafb5a4043fd59838842c4cd4a8b6f0b $
Docker version: 20.10.8, build 3967b7d

@hawflau

Hello @hawflau, just checking in to see if your colleagues have had any success in reproducing this, and if so, whether there is a resolution or workaround that can be made available in the near term.

@hawflau

Hi @hawflau, apologies for the second ping on this. Any luck with having this reproduced? If this is an issue with my setup only, then I would be happy to get some pointers to figure out how to resolve it? But I am hoping that this is a bug that would have an ETA for a fix. Thanks in advance.

Thank you for raising this issue. It seems that the issue is related to the fact that Go Lambda functions can only run on x86_64 architecture (https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html), and that Docker Desktop does not support running containers using this architecture (see these issues docker/for-mac#5191 (comment), aws/aws-toolkit-vscode#2020).

As a workaround, you can use Colima instead of Docker Desktop allows you to start the container as x86_64 architecture. I tried this workaround and worked fine. These are the steps:

You need first to uninstall the docker desktop
/Applications/Docker.app/Contents/MacOS/uninstall
Install Colima, and docker client

brew install colima
brew install docker

start colima using the x86_64 architecture

colima start --arch x86_64

build a dlv binary on an Intel mac and copy it to your M1 Mac
GOARCH=amd64 GOOS=linux go build -o x86_64/dlv github.com/go-delve/delve/cmd/dlv
in vscode, update the launch.json file, and add the following configuration:

{
   "configurations": [
       {
           "name": "Connect to server",
           "type": "go",
           "request": "attach",
           "mode": "remote",
           "remotePath": "",
           "port": 9999,
           "host": "127.0.0.1"
       }
   ]
 }

build your project using sam
sam build
run sam local invoke in debug mode
sam local invoke -d 9999 --debugger-path x86_64/ --debug-args "-delveAPI=2"
wait till you see in the logs that the server is listening to :9999, start the debugging

I hope this information helps, and please let me know if there is anything else I can assist with.

@moelasmar

Thanks @moelasmar! This is great info. I will try it out when I get a chance. Much appreciate the update.

In the meantime I am wondering if you or anybody else in the forum has insights into if and when Go lambdas will be supported on arm64 architectures.

Also, I did briefly try to use the provided.al2 runtime as described here - Migrating AWS Lambda functions to Amazon Linux 2. I was not successful in getting this working, but I did not really persevere, I must admit, and this was a while back.

We have continued development with local debugging rather than running the Go code as lambdas in debugging mode as a stop gap, which is largely okay unless you are trying to debug issues relating to the code running in the Lambda context.

But it would be great to get Go Lambdas debuggable in Macs with M1 architectures out of the box as we do in Intel systems. My intel based Mac is nearing 10 years and I am not planning on getting a new Intel based Mac moving forward. :-)

Hi,
I also observe the same problem with dotnet. Don't know if this information helps narrow it down if it's a cli problem or go.
I create a simple solution using AWS Toolkit solution, both dotnet and Go have the same problem.
I am on M1 and I tried to ask my colleague who run the MacOS but with intel processor, it works for him.

@dkosasih

Hi @dkosasih,
Both Lambda, and AWS SAM CLI supports dotnet 6 for ARM architecture. Could you please provide more details on the specific issue you are encountering?

@moelasmar

Thanks for the quick reply @moelasmar.
Rather than replicating the behavior, this thread definitely sums it up better than I could, and it also point to this discussion from Aws-Toolkit-VsCode repo.

@dkosasih

Thank you @dkosasih .. I will check it later

@moelasmar

Thank you for raising this issue. It seems that the issue is related to the fact that Go Lambda functions can only run on x86_64 architecture (https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html), and that Docker Desktop does not support running containers using this architecture (see these issues docker/for-mac#5191 (comment), aws/aws-toolkit-vscode#2020).

As a workaround, you can use Colima instead of Docker Desktop allows you to start the container as x86_64 architecture. I tried this workaround and worked fine. These are the steps:
* You need first to uninstall the docker desktop
  `/Applications/Docker.app/Contents/MacOS/uninstall`

* Install Colima, and docker client
brew install colima
brew install docker 
* start colima using the x86_64 architecture
colima start --arch x86_64
* build a dlv binary on an Intel mac and copy it to your M1 Mac
  `GOARCH=amd64 GOOS=linux go build -o x86_64/dlv github.com/go-delve/delve/cmd/dlv`

* in vscode, update the launch.json file, and add the following configuration:
{
   "configurations": [
       {
           "name": "Connect to server",
           "type": "go",
           "request": "attach",
           "mode": "remote",
           "remotePath": "",
           "port": 9999,
           "host": "127.0.0.1"
       }
   ]
 }
* build your project using sam
  ` sam build`

* run sam local invoke in debug mode
  `sam local invoke -d 9999 --debugger-path x86_64/ --debug-args "-delveAPI=2"`

* wait till you see in the logs that the server is listening to :9999, start the debugging
I hope this information helps, and please let me know if there is anything else I can assist with.

Hi @moelasmar, thanks for the reply. I've tried following your instructions but receive the following still:

START RequestId: d632e759-34fa-41b8-b21c-57fcb6814c13 Version: $LATEST
API server listening at: [::]:9999
2023-02-13T17:21:56Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2023-02-13T17:21:56Z info layer=debugger launching process with args: [/var/task/main]
could not launch process: fork/exec /var/task/main: function not implemented
2023/02/13 17:21:56 exit status 1

A cursory glance at Google indicates that this is a problem with delve running on arm architecture. I followed all previous steps including compiling for amd64, I also tried compiling natively on a Ryzen powered system I have next to me and had no luck with this either.

At this point we're largely engineered our way around being dependant on local sam debugging by having extensive unit tests but it would be nice to be able to debug from the context of a Lambda.

@benanderton

Hello @benanderton, I appreciate your feedback. I encountered a similar issue when trying to utilize Delve. When I used the M1 Mac compiled version on Colima or the Intel Mac compiled version on Docker Desktop, I received the same error. However, I did not have any problems when I used the Intel Mac compiled version on Colima. Unfortunately, I do not know why it is not working for you.

We are moving this issue to a discussion thread because it is related to the development setup and aligning it with the architecture of the Lambda function, rather than being directly related to SAM CLI.

Navigation Menu

Repository navigation

Lambda ARM (Apple M1): AWS SAM Golang lambda debugging does not work #3283

Description:

Steps to reproduce:

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

46 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Lambda ARM (Apple M1): AWS SAM Golang lambda debugging does not work #3283

Description

Description:

Steps to reproduce:

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Activity

ashprao commented on Oct 3, 2021

hawflau commented on Oct 14, 2021

ashprao commented on Oct 15, 2021

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

ashprao commented on Oct 28, 2021

ashprao commented on Nov 15, 2021

46 remaining items

moelasmar commented on Feb 12, 2023

ashprao commented on Feb 12, 2023

dkosasih commented on Feb 12, 2023

moelasmar commented on Feb 13, 2023

dkosasih commented on Feb 13, 2023

moelasmar commented on Feb 13, 2023

benanderton commented on Feb 13, 2023

moelasmar commented on Feb 13, 2023

moelasmar commented on Feb 13, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions