Fix: prevent duplicate payment enqueuing if all celery workers are stuck

[Luquitasjeffrey]

This PR introduces LNPayment.Status.QUEUED to prevent a payment from being re-enqueued multiple times via follow_send_payment.delay when workers are stuck.

[KoalaSat]

Thanks for the PR! Can you add some test to it? that way we ensure it won´t happen again and also we can run them separately to confirm it fails on main

[KoalaSat]

I just checked the workflow and just realized one important fact: while in the context of AMP invoices multiple payments are possible. The code already depends on hash-level dedup, explicitly:

api/lightning/lnd.py:653-718 catches exactly the two errors LND's payment control raises when you re-submit the same payment hash:

elif "payment is in transition" in str(e):   # LND ErrPaymentInFlight
    ... TrackPaymentV2(payment_hash) ... handle_response(response, was_in_transit=True)
elif "invoice is already paid" in str(e):    # LND ErrAlreadyPaid
    ... TrackPaymentV2(payment_hash) ... handle_response(response)

So a second concurrent SendPaymentV2 for the same hash never becomes a second payment, it degenerates into a tracker of the first one

[Luquitasjeffrey]

I just checked the workflow and just realized one important fact: while in the context of AMP invoices multiple payments are possible. The code already depends on hash-level dedup, explicitly:

api/lightning/lnd.py:653-718 catches exactly the two errors LND's payment control raises when you re-submit the same payment hash:

elif "payment is in transition" in str(e):   # LND ErrPaymentInFlight
    ... TrackPaymentV2(payment_hash) ... handle_response(response, was_in_transit=True)
elif "invoice is already paid" in str(e):    # LND ErrAlreadyPaid
    ... TrackPaymentV2(payment_hash) ... handle_response(response)

So a second concurrent SendPaymentV2 for the same hash never becomes a second payment, it degenerates into a tracker of the first one

That is not the exploitable vulnerability. If the first 3 attempts of the payment fails, then the order transitions to status FAI, and the code assumes there are no payments in flight for that order, but... with my bug an attacker could enqueue a payment leys say, 10 times, so, the first 3 fail, and then it will call update_invoice at the same time it is paying the foruth payment. It is not easily exploitable unless there is high latency in the connection with LND. The attacker would be able to attach a new invoice when a previous payment is queued