possible to provide golang binding for this?

[kolinfluence]

possible to provide golang binding for this?

thx in advance!

[simonhf]

Thanks for the request!

Unfortunately Golang is not a good language for using with SharedHashFile. Why not?

Reason #1: Context switching:

Ideally when using SharedHashFile there should be as little context switching as possible for processes running on Linux threads. Why? Because SharedHashFile uses spin locks in user land [1]. So if a context switch occurs while a spin lock is held then there could be a bigger delay before the spin lock is unlocked again. With other languages then it's possible to master then number of Linux threads being used at any time and always ensure there are no more than the number of CPUs, meaning that no context switching will occur. However, Golang has taken a route as a language where it's difficult to put a limit on the number of Go-routines, and therefore, it's difficult to guarantee that Golang will not do its own "context switching" on its own Go-routines, causing a similar issue with the spin locks.

Theoretically it would be possible to create a Golang program which didn't suffer from this issue, but you'd likely not be using many 3rd party Golang modules (which could create extra Go-routines), and you'd be an expert at wrangling your spawned Go-routines so that they never get context switched (read: "pre-empted" in Golang terminology; a newer feature in Golang since 2017 which interestingly can be switched off!).

Reason #2: Thread local storage:

AKA TLS, it's a simple and powerful mechanism for keeping data separate without using expensive locks in multi-threaded C/C++ programs. SharedHashFile uses TLS extensively. However, the designers of Golang have actively avoided the use of TLS in Golang, and also tried to enforce that other developers cannot easily introduce mechanism to use it in the future. If you Google then you can find some big long discussions about whether TLS should be part of Golang or not, and some attempts to introduce it officially or unofficially into the language.

The problem with Golang is that a particular Go-routine can be moved from any Linux OS thread to any other OS thread without the Go-routines knowledge. And the TLS is specific to a particular OS thread.

IMO if Golang introduced a feature to pin Go-routines to particular OS thread, and then for this special case, allow those pinned Go-routines to have their own version of TLS -- let's call it GRLS -- then there might be a chance of this working :-) But I think it's unlikely that such a new feature will come along based upon the historical avoidance of TLS by authors of Golang.

So what other options are there to get SharedHashFile working with Golang?

One option is to move most of SharedHashFile into a kernel module. Kernel modules can share memory with user land, and SharedHashFile is built around shared memory anyway. In a kernel module then context switching can be disabled, so the spin lock issue will no longer be an issue. And the TLS issue can also be worked around because there would be a new interface to access the kernel module. However, this would be quite a lot of difficult work and I'm not sure success is guaranteed. But it would be an interesting project :-)

[1] https://github.com/simonhf/sharedhashfile/blob/master/src/shf.lock.h

[kolinfluence]

1. everything u mentioned i already have the solution in fact it's pinned to per thread in golang etc., that's why i'm looking at sharedhashfile for performance.

2. so, possible to provide golang binding for this please? with reference to my 1st answer.

can i pls buy u a few coffees for this? can give u credit or backlink to your site / blog too.
hope to have the golang binding done. i can do the binding but will probably take 10 days to figure everything out.

[ouvaa]

@simonhf just realised this issue is related to my question. seems like there is no workarounds for golang binding.
#27

[simonhf]

seems like there is no workarounds for golang binding

@ouvaa yes, things don't look good for a Golang binding due to the reasons outlined above.

I think a possible workaround would be to change the way that SHF works. Thinking out loud: In theory SHF could be moved into its own Linux kernel module. This would in theory workaround the reasons outlined above. Due to the spin locks, SHF is currently designed to work with software designed to context switch a minimum. However, moving SHF into a kernel module would also make SHF more robust in general for environments where much context switching is happening in user land. Why? In a kernel module then execution cannot be pre-empted, i.e. there is no context switching. The question is: What would the performance be like as a kernel module?