TODOs before someguy's General Availability

[lidel]

TODOs for delegated-ipfs.dev/routing/v1

Minimum we need to switch delegated-ipfs.dev/routing/v1 from Kubo to someguy:

• ask both cid.contact and Amino DHT
• allow adjusting config via env variables instead of CLI params (prior art in )
• Docker image build & publish the same way as https://github.com/ipfs/rainbow/

cc @ns4plabs @cewood lmk if there are any additional asks from bifrost infra side that would make deployment/configuration easier – my idea is to follow similar conventions as rainbow / bifrost-gateway

TODOs for General Availability

Things to do before we start recommending someguy to our users for use in their dev/prod:

• defaults: change default port to something unique (NOT 8080) (Change default port to avoid conflict with gateway #47)
• Default Delegated Routing V1 Endpoints #63
• README with usage examples (cli, docker – prior art)
• tag v1.0.0
• docs
  • docs/environment-variables.md (prior art)
  • docs/kubo.md showing how to use someguy as delegated routing for Kubo specifically (including Kubo config) – this could be linked from kubo/docs/config.md too
  • mention at docs.ipfs.tech (cc Document delegated routing concept ipfs-docs#1752)

cc @2color for visibility, feel free to flag any dx/ux pain points we should fix before GA

[aschmahmann]

defaults: in addition to DHT, should we ask both delegated-ipfs.dev and cid.contact, or only delegated-ipfs.dev?

Looks like this is being done in #50 where both are asked, but trying to understand why a little more. Is this about future proofing?

At the moment there are two widely deployed routing "systems" used by IPFS implementations:

• Amino DHT
• IPNI (albeit until federation is landed along with expectations around sync-times it's mostly considered the cid.contact service)

delgated-ipfs.dev currently just proxies those two systems. So by default hitting it in addition to the others just means the same query is happening twice, just with maybe some caching happening on delegated-ipfs.dev. What's the goal here?

---

Setting the default for someguy ask to delegated-ipfs.dev, or (re)building a tool like ipget to default to delegated-ipfs.dev seems reasonable though.

[lidel]

@aschmahmann I raised similar questions in #50 (comment) before reading your comment.

In my mind the goals of including delegated-ipfs.dev are:

• include an endpoint that follows latest specs and is actively maintained by the same team, allowing us to dogfood more and give people example of system where there is more than a single endpoint.
• cid.contact and IPNI as it is today does not support /routing/v1/peers or /routing/v1/ipns routing endpoints. delegated-ipfs.dev supports all routing types.
• cid.contact does not support streaming (Accept: application/x-ndjson), delegated-ipfs.dev does.

My main question is do we even keep cid.contact , if delegated-ipfs.dev proxies to it already.

We could say we trade the additional request for resiliency of having different hostnames with separate HTTP caches, which I am fine enough as a reason enough to keep it. Thoughts?

[willscott]

cid.contact does not support streaming (Accept: application/x-ndjson)

It does support streaming afaik? - is there a subtle difference in the spec vs what we support that we should update?

[masih]

cid.contact does not support streaming (Accept: application/x-ndjson)

This is inaccurate. cid.contact does support streaming result.

[aschmahmann]

cid.contact does not support streaming (Accept: application/x-ndjson)

This is inaccurate. cid.contact does support streaming result.

@masih are you sure?

❯ curl -H "accept: application/json" https://cid.contact/routing/v1/providers/bafybeif6f27eonqanzvltpfhaf2fgmwz6n5e7j6fksuc6jrs5payvufyha
{"Providers":[{"Addrs":["/ip4/76.219.232.45/tcp/24001"],"ID":"12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys","Protocols":["transport-graphsync-filecoinv1"],"Schema":"peer","transport-graphsync-filecoinv1":"kBKjaFBpZWNlQ0lE2CpYKAABgeIDkiAgmiTyU31IQA81ZyvKUkNR5cW/orcsg79MUwG2wmtwthVsVmVyaWZpZWREZWFs9W1GYXN0UmV0cmlldmFs9Q=="},{"Addrs":["/ip4/76.219.232.45/tcp/24888"],"ID":"12D3KooWSoSgVaUvoguDQZu1doytze9RgnnANwJoiLw7KUcAXq8i","Protocols":["transport-bitswap"],"Schema":"peer","transport-bitswap":"gBI="},{"Addrs":["/dns/cesginc.com/tcp/443/https"],"ID":"12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys","Protocols":["transport-ipfs-gateway-http"],"Schema":"peer","transport-ipfs-gateway-http":"oBIA"}]}
❯ curl -H "accept: application/x-ndjson" https://cid.contact/routing/v1/providers/bafybeif6f27eonqanzvltpfhaf2fgmwz6n5e7j6fksuc6jrs5payvufyha
{"Providers":[{"Addrs":["/ip4/76.219.232.45/tcp/24001"],"ID":"12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys","Protocols":["transport-graphsync-filecoinv1"],"Schema":"peer","transport-graphsync-filecoinv1":"kBKjaFBpZWNlQ0lE2CpYKAABgeIDkiAgmiTyU31IQA81ZyvKUkNR5cW/orcsg79MUwG2wmtwthVsVmVyaWZpZWREZWFs9W1GYXN0UmV0cmlldmFs9Q=="},{"Addrs":["/ip4/76.219.232.45/tcp/24888"],"ID":"12D3KooWSoSgVaUvoguDQZu1doytze9RgnnANwJoiLw7KUcAXq8i","Protocols":["transport-bitswap"],"Schema":"peer","transport-bitswap":"gBI="},{"Addrs":["/dns/cesginc.com/tcp/443/https"],"ID":"12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys","Protocols":["transport-ipfs-gateway-http"],"Schema":"peer","transport-ipfs-gateway-http":"oBIA"}]}
❯ curl -H "accept: application/json" https://delegated-ipfs.dev/routing/v1/providers/bafybeif6f27eonqanzvltpfhaf2fgmwz6n5e7j6fksuc6jrs5payvufyha
{"Providers":[{"Addrs":["/ip4/76.219.232.45/tcp/24001"],"ID":"12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys","Protocols":["transport-graphsync-filecoinv1"],"Schema":"peer","transport-graphsync-filecoinv1":"kBKjaFBpZWNlQ0lE2CpYKAABgeIDkiAgmiTyU31IQA81ZyvKUkNR5cW/orcsg79MUwG2wmtwthVsVmVyaWZpZWREZWFs9W1GYXN0UmV0cmlldmFs9Q=="},{"Addrs":["/ip4/76.219.232.45/tcp/24888"],"ID":"12D3KooWSoSgVaUvoguDQZu1doytze9RgnnANwJoiLw7KUcAXq8i","Protocols":["transport-bitswap"],"Schema":"peer","transport-bitswap":"gBI="},{"Addrs":["/dns/cesginc.com/tcp/443/https"],"ID":"12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys","Protocols":["transport-ipfs-gateway-http"],"Schema":"peer","transport-ipfs-gateway-http":"oBIA"}]}
❯ curl -H "accept: application/x-ndjson" https://delegated-ipfs.dev/routing/v1/providers/bafybeif6f27eonqanzvltpfhaf2fgmwz6n5e7j6fksuc6jrs5payvufyha
{"Addrs":["/ip4/76.219.232.45/tcp/24001"],"ID":"12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys","Protocols":["transport-graphsync-filecoinv1"],"Schema":"peer","transport-graphsync-filecoinv1":"kBKjaFBpZWNlQ0lE2CpYKAABgeIDkiAgmiTyU31IQA81ZyvKUkNR5cW/orcsg79MUwG2wmtwthVsVmVyaWZpZWREZWFs9W1GYXN0UmV0cmlldmFs9Q=="}
{"Addrs":["/ip4/76.219.232.45/tcp/24888"],"ID":"12D3KooWSoSgVaUvoguDQZu1doytze9RgnnANwJoiLw7KUcAXq8i","Protocols":["transport-bitswap"],"Schema":"peer","transport-bitswap":"gBI="}
{"Addrs":["/dns/cesginc.com/tcp/443/https"],"ID":"12D3KooWPNbkEgjdBNeaCGpsgCrPRETe4uBZf1ShFXStobdN18ys","Protocols":["transport-ipfs-gateway-http"],"Schema":"peer","transport-ipfs-gateway-http":"oBIA"}

It does support streaming afaik? - is there a subtle difference in the spec vs what we support that we should update?

It's doing json rather than ndjson, i.e. it's not streaming and relying on this line from the spec https://specs.ipfs.tech/routing/http-routing-v1/#streaming

Legacy server MAY respond with non-streaming application/json response even if the client requested streaming. It is up to the client to inspect the Content-Type header before parsing the response.

Showing this explicitly:

❯ curl -I -H "accept: application/x-ndjson" https://cid.contact/routing/v1/providers/bafybeif6f27eonqanzvltpfhaf2fgmwz6n5e7j6fksuc6jrs5payvufyha
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 715
Connection: keep-alive
Date: Tue, 03 Sep 2024 11:12:49 GMT
Access-Control-Allow-Methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=15724800; includeSubDomains
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization
Access-Control-Max-Age: 1728000
X-Cache: Hit from cloudfront
Via: 1.1 521101b4b5baafcfa7548a73a3442cea.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: BOS50-P1
X-Amz-Cf-Id: jtfTprleZVwJJ9CQ0LeoOJfXHyBCElEXZuZgTjtLBWZLUqVlgg2Ctw==
Age: 681

[willscott]

cid.contact will stream https://github.com/ipni/indexstar/blob/main/find_ndjson.go#L280-L282

but i think it maybe doesn't do it on the '/routing/v1/providers' endpoint, but rather on the '/mh/' find endpoint that lassie / saturn was using

[masih]

Thanks for pointing that out @aschmahmann; you are right. Captured this issue to get it sorted.

[willscott]

Note on spec:
https://specs.ipfs.tech/routing/http-routing-v1/#streaming

• the types of the lines of streaming aren't defined
• the main response is an 'IPNI response' record, but the lines are publisher records and that semantic that we're getting rid of the top level object could probably get spelled out a bit better.

There's a PR to add support to indexer delegated routing endpoint for streaming here ipni/indexstar#200

[willscott]

Also, am I right that delegated routing drops the support for encrypted / double-hashed query/responses? is that something we want to add back in?