Lambda ARM (Apple M1): AWS SAM Golang lambda debugging does not work

Description:

After creating a plain vanilla AWS SAM application with sam init for golang, and then using the AWS Toolkit provided code lens in VS Code, to create an AWS SAM template file based debug configuration, I get the following error when trying to debug the lambda code.

Running the code without debugging works.

Note:
I had raised this issue in aws/aws-toolkit-vscode as aws/aws-toolkit-vscode#2020. There was a suggestion in that thread that I should raise an issue here.

Steps to reproduce:

Create a new AWS SAM application using Go with sam init.
Select the Hello world template.
In the main.go file, create a debug configuration from the code lens provided by the AWS Toolkit and associated with the handler.
Set a breakpoint and start debugging.

Observed result:

2021-09-11 10:55:02 [INFO]: The Delve repo was not found in your GOPATH. Downloading in a temporary directory...
2021-09-11 10:55:03 [INFO]: Preparing to debug locally: Lambda "hello-world"
2021-09-11 10:55:03 [INFO]: Building SAM application...
2021-09-11 10:55:03 [INFO]: Running command: (not started) [/opt/homebrew/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctkeYZ6BB/output --template /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world/app___vsctk___template.yaml --base-dir /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world]
2021-09-11 10:55:03 [INFO]: Building codeuri: /Users/user1/dev/godev/aws/lambdas/hello-world/hello-world runtime: go1.x metadata: {} functions: ['helloworld']

2021-09-11 10:55:03 [INFO]: Running GoModulesBuilder:Build

2021-09-11 10:55:03 [INFO]: 
Build Succeeded

2021-09-11 10:55:03 [INFO]: 
Built Artifacts  : ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output
Built Template   : ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml
[*] Deploy: sam deploy --guided --template-file ../../../../../../../private/tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/template.yaml
    

2021-09-11 10:55:04 [INFO]: Build complete.
2021-09-11 10:55:04 [INFO]: Starting SAM application locally
2021-09-11 10:55:05 [INFO]: AWS.running.command
Invoking hello-world (go1.x)
Requested to skip pulling images ...

Mounting /tmp/aws-toolkit-vscode/vsctkeYZ6BB/output/helloworld as /var/task:ro,delegated inside runtime container
START RequestId: 7c546e9e-8558-4fde-8dd6-fc6946034850 Version: $LATEST
API server listening at: [::]:5858
2021-09-11T00:55:07Z info layer=debugger launching process with args: [/var/task/hello-world]
2021-09-11 10:55:07 [INFO]: Waiting for SAM application to start...
could not launch process: fork/exec /var/task/hello-world: function not implemented
2021/09/11 00:55:07 exit status 1
time="2021-09-11T00:55:07.765" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2021-09-11T00:55:07.773" level=error msg="INIT DONE failed: Runtime.ExitError"
2021-09-11 10:55:08 [INFO]: Attaching debugger to SAM application...

Command stopped: "sam local invoke"

From the output listed above, it seems to me that when using the debugger, it thinks that the function is not implemented.
2021-09-11 10:55:07 [INFO]: Waiting for SAM application to start...
could not launch process: fork/exec /var/task/hello-world: function not implemented

Other considerations

Go debugging works for vanilla go apps (non lambdas) on M1 based Apple Macs.
Go lambdas run successfully with sam local invoke on M1 based Apple Macs.
Go lambdas can be successfully debugged on x86 based Apple Macs.

Expected result:

Lambda debugging working with breakpoints being hit.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: macOS Big Sur 11.6 (M1) / Darwin arm64 20.6.0
sam --version: 1.31.0
AWS region: any (currently working in ap-south-1)
VS Code version: 1.16
AWS Toolkit extension version: 1.29.0
Go version: 1.17.1
dlv and dlv-dap version: 1.7.1 Build: $Id: 3bde2354aafb5a4043fd59838842c4cd4a8b6f0b $
Docker version: 20.10.8, build 3967b7d

@ashprao

Any update to this? I seem to be able to run this without debugging just fine as @ashprao

@jfuss

bump @jfuss

@hawflau

Any update on this @hawflau? I am also facing the issue with my M1 Pro.

@aahung

@aahung can you help with this

I am having the same issue.

Model Name:	MacBook Pro
Model Identifier: MacBookPro17,1
Chip: Apple M1
Total Number of Cores:	8 (4 performance and 4 efficiency)
Memory:	16 GB
System Firmware Version: 7429.61.2
OS Loader Version:	7429.61.2

Same issue here: debug run fails on an M1 MacBook.

Just checked again with version 1.38.0. Problem persists.

Invoking handler (go1.x)
Image was not found.
Removing rapid images for repo public.ecr.aws/sam/emulation-go1.x
Building image................................
Skip pulling image and use local one: public.ecr.aws/sam/emulation-go1.x:rapid-1.38.0-x86_64.

Mounting /Users/######/GolandProjects/#######/Development/#########/.aws-sam/build/Function as /var/task:ro,delegated inside runtime container
START RequestId: 0c52d50b-####-####-acdc-d864ae3b4862 Version: $LATEST
API server listening at: [::]:65486
2022-02-16T20:11:03Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2022-02-16T20:11:03Z info layer=debugger launching process with args: [/var/task/handler]
could not launch process: fork/exec /var/task/handler: function not implemented
2022/02/16 20:11:03 exit status 1
16 Feb 2022 20:11:03,501 [ERROR] (rapid) Init failed error=Runtime exited with error: exit status 1 InvokeID=
API server listening at: [::]:65486
2022-02-16T20:11:04Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2022-02-16T20:11:04Z info layer=debugger launching process with args: [/var/task/handler]
could not launch process: fork/exec /var/task/handler: function not implemented
2022/02/16 20:11:04 exit status 1
END RequestId: 5c7b6ef8-####-####-aa70-20440b6d1ff8
REPORT RequestId: 5c7b6ef8-####-####-aa70-20440b6d1ff8	Init Duration: 1.59 ms	Duration: 1732.04 ms	Billed Duration: 1733 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

@jackmcguire1

@jackmcguire1 @aahung
Is there anything we can do to assist with this fix?

any updates?

Any updates here?

Just for reference, I wanted to drop the info that the above issue is experienced even when using sam-cli locally in debug mode, as opposed to using VS Code.

❯ sam local invoke -d 5858 --debugger-path $GOPATH/bin/linux_amd64/
Invoking hello-world (go1.x)
Skip pulling image and use local one: public.ecr.aws/sam/emulation-go1.x:rapid-1.53.0-x86_64.

Mounting /Users/varungawande/playground/goLambdaDebug/sunflowers/.aws-sam/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
START RequestId: 21a7a6c7-c9ee-4bee-8c77-5be8cdc3a9c6 Version: $LATEST
API server listening at: [::]:5858
2022-08-05T04:16:43Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2022-08-05T04:16:43Z info layer=debugger launching process with args: [/var/task/hello-world]
could not launch process: fork/exec /var/task/hello-world: function not implemented
2022/08/05 04:16:43 exit status 1
05 Aug 2022 04:16:43,434 [ERROR] (rapid) Init failed error=Runtime exited with error: exit status 1 InvokeID=
API server listening at: [::]:5858
2022-08-05T04:16:43Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2022-08-05T04:16:43Z info layer=debugger launching process with args: [/var/task/hello-world]
could not launch process: fork/exec /var/task/hello-world: function not implemented
2022/08/05 04:16:43 exit status 1
END RequestId: 04bbf54c-1711-4a3c-bd9f-668866f340ff
REPORT RequestId: 04bbf54c-1711-4a3c-bd9f-668866f340ff	Init Duration: 1.51 ms	Duration: 1294.62 msBilled Duration: 1295 ms	Memory Size: 128 MB	Max Memory Used: 128 MB

@hawflau

@hawflau Is there any way we can help reproduce this?

Is this on a roadmap anywhere?

i think it's related to this, the problem being docker emulation for apple soc.

@georgedimac

i think it's related to this, the problem being docker emulation for apple soc.

@georgedimac Unfortunately, that's related to this.
Only after SAM CLI supports M1, can others like VSCode-Toolkit use that feature.

Any update on this issue?

Having a similar issue on M2

Dockerfile:

FROM --platform=linux/arm64 public.ecr.aws/lambda/provided:al2 as build-image
# FROM public.ecr.aws/bitnami/golang:1.19 as build-image
RUN yum install -y golang
RUN go env -w GOPROXY=direct
WORKDIR /go/src
COPY . .
RUN go build -o /main

FROM public.ecr.aws/lambda/provided:al2
WORKDIR /var/task
COPY --from=build-image /main /main
#TODO: shouldn't add my local AWS creds here
# COPY .aws /root/.aws
# Command can be overwritten by providing a different command in the template directly.
ENTRYPOINT ["/main"]

Logs when running debugger via vscode:

2022-12-28 20:40:01 [INFO]: The Delve repo was not found in your GOPATH. Downloading in a temporary directory...
2022-12-28 20:40:01 [INFO]: Preparing to debug locally: Lambda "ParseCSVFunction"
2022-12-28 20:40:01 [INFO]: Building SAM application...
2022-12-28 20:40:01 [ERROR]: SamLaunchRequestError: Failed to build SAM application [BuildFailure]
2022-12-28 20:40:01 [INFO]: SAM CLI location: [object Object]
2022-12-28 20:40:01 [INFO]: Command: (not started) [/opt/homebrew/bin/sam build --build-dir /tmp/aws-toolkit-vscode/vsctkHg6ywY/output --template /Users/reigun/work/repos/parsley/template.yaml --use-container]
2022-12-28 20:40:01 [INFO]: SAM CLI not configured, using SAM found at: '/opt/homebrew/bin/sam'
2022-12-28 20:43:38 [INFO]: telemetry: sent batch (size=3)

Thank you for raising this issue. It seems that the issue is related to the fact that Go Lambda functions can only run on x86_64 architecture (https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html), and that Docker Desktop does not support running containers using this architecture (see these issues docker/for-mac#5191 (comment), aws/aws-toolkit-vscode#2020).

As a workaround, you can use Colima instead of Docker Desktop allows you to start the container as x86_64 architecture. I tried this workaround and worked fine. These are the steps:

You need first to uninstall the docker desktop
/Applications/Docker.app/Contents/MacOS/uninstall
Install Colima, and docker client

brew install colima
brew install docker

start colima using the x86_64 architecture

colima start --arch x86_64

build a dlv binary on an Intel mac and copy it to your M1 Mac
GOARCH=amd64 GOOS=linux go build -o x86_64/dlv github.com/go-delve/delve/cmd/dlv
in vscode, update the launch.json file, and add the following configuration:

{
   "configurations": [
       {
           "name": "Connect to server",
           "type": "go",
           "request": "attach",
           "mode": "remote",
           "remotePath": "",
           "port": 9999,
           "host": "127.0.0.1"
       }
   ]
 }

build your project using sam
sam build
run sam local invoke in debug mode
sam local invoke -d 9999 --debugger-path x86_64/ --debug-args "-delveAPI=2"
wait till you see in the logs that the server is listening to :9999, start the debugging

I hope this information helps, and please let me know if there is anything else I can assist with.

@moelasmar

Thanks @moelasmar! This is great info. I will try it out when I get a chance. Much appreciate the update.

In the meantime I am wondering if you or anybody else in the forum has insights into if and when Go lambdas will be supported on arm64 architectures.

Also, I did briefly try to use the provided.al2 runtime as described here - Migrating AWS Lambda functions to Amazon Linux 2. I was not successful in getting this working, but I did not really persevere, I must admit, and this was a while back.

We have continued development with local debugging rather than running the Go code as lambdas in debugging mode as a stop gap, which is largely okay unless you are trying to debug issues relating to the code running in the Lambda context.

But it would be great to get Go Lambdas debuggable in Macs with M1 architectures out of the box as we do in Intel systems. My intel based Mac is nearing 10 years and I am not planning on getting a new Intel based Mac moving forward. :-)

Hi,
I also observe the same problem with dotnet. Don't know if this information helps narrow it down if it's a cli problem or go.
I create a simple solution using AWS Toolkit solution, both dotnet and Go have the same problem.
I am on M1 and I tried to ask my colleague who run the MacOS but with intel processor, it works for him.

@dkosasih

Hi @dkosasih,
Both Lambda, and AWS SAM CLI supports dotnet 6 for ARM architecture. Could you please provide more details on the specific issue you are encountering?

@moelasmar

Thanks for the quick reply @moelasmar.
Rather than replicating the behavior, this thread definitely sums it up better than I could, and it also point to this discussion from Aws-Toolkit-VsCode repo.

@dkosasih

Thank you @dkosasih .. I will check it later

@moelasmar

Thank you for raising this issue. It seems that the issue is related to the fact that Go Lambda functions can only run on x86_64 architecture (https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html), and that Docker Desktop does not support running containers using this architecture (see these issues docker/for-mac#5191 (comment), aws/aws-toolkit-vscode#2020).

As a workaround, you can use Colima instead of Docker Desktop allows you to start the container as x86_64 architecture. I tried this workaround and worked fine. These are the steps:
* You need first to uninstall the docker desktop
  `/Applications/Docker.app/Contents/MacOS/uninstall`

* Install Colima, and docker client
brew install colima
brew install docker 
* start colima using the x86_64 architecture
colima start --arch x86_64
* build a dlv binary on an Intel mac and copy it to your M1 Mac
  `GOARCH=amd64 GOOS=linux go build -o x86_64/dlv github.com/go-delve/delve/cmd/dlv`

* in vscode, update the launch.json file, and add the following configuration:
{
   "configurations": [
       {
           "name": "Connect to server",
           "type": "go",
           "request": "attach",
           "mode": "remote",
           "remotePath": "",
           "port": 9999,
           "host": "127.0.0.1"
       }
   ]
 }
* build your project using sam
  ` sam build`

* run sam local invoke in debug mode
  `sam local invoke -d 9999 --debugger-path x86_64/ --debug-args "-delveAPI=2"`

* wait till you see in the logs that the server is listening to :9999, start the debugging
I hope this information helps, and please let me know if there is anything else I can assist with.

Hi @moelasmar, thanks for the reply. I've tried following your instructions but receive the following still:

START RequestId: d632e759-34fa-41b8-b21c-57fcb6814c13 Version: $LATEST
API server listening at: [::]:9999
2023-02-13T17:21:56Z warning layer=rpc Listening for remote connections (connections are not authenticated nor encrypted)
2023-02-13T17:21:56Z info layer=debugger launching process with args: [/var/task/main]
could not launch process: fork/exec /var/task/main: function not implemented
2023/02/13 17:21:56 exit status 1

A cursory glance at Google indicates that this is a problem with delve running on arm architecture. I followed all previous steps including compiling for amd64, I also tried compiling natively on a Ryzen powered system I have next to me and had no luck with this either.

At this point we're largely engineered our way around being dependant on local sam debugging by having extensive unit tests but it would be nice to be able to debug from the context of a Lambda.

@benanderton

Hello @benanderton, I appreciate your feedback. I encountered a similar issue when trying to utilize Delve. When I used the M1 Mac compiled version on Colima or the Intel Mac compiled version on Docker Desktop, I received the same error. However, I did not have any problems when I used the Intel Mac compiled version on Colima. Unfortunately, I do not know why it is not working for you.

We are moving this issue to a discussion thread because it is related to the development setup and aligning it with the architecture of the Lambda function, rather than being directly related to SAM CLI.

Navigation Menu

Repository navigation

Lambda ARM (Apple M1): AWS SAM Golang lambda debugging does not work #3283

Description:

Steps to reproduce:

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Lambda ARM (Apple M1): AWS SAM Golang lambda debugging does not work #3283

Description

Description:

Steps to reproduce:

Observed result:

Expected result:

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Activity

casimcdaniels commented on Nov 30, 2021

jackmcguire1 commented on Dec 22, 2021

rush51 commented on Jan 17, 2022

jackmcguire1 commented on Jan 17, 2022

jeremyforan commented on Jan 27, 2022

flythebluesky commented on Feb 13, 2022

jeremyforan commented on Feb 16, 2022

rush51 commented on Feb 21, 2022

wsherm commented on May 9, 2022

V4G4X commented on Aug 5, 2022

V4G4X commented on Aug 5, 2022

V4G4X commented on Sep 5, 2022

benanderton commented on Sep 9, 2022

georgedimac commented on Sep 27, 2022

V4G4X commented on Oct 4, 2022

ravitejaavv commented on Nov 21, 2022

rei-gun commented on Dec 28, 2022

moelasmar commented on Feb 12, 2023

ashprao commented on Feb 12, 2023

dkosasih commented on Feb 12, 2023

moelasmar commented on Feb 13, 2023

dkosasih commented on Feb 13, 2023

moelasmar commented on Feb 13, 2023

benanderton commented on Feb 13, 2023

moelasmar commented on Feb 13, 2023

moelasmar commented on Feb 13, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions