How to type function composition function compose?

[Dutcho]

I'm trying to type function composition, and get stuck.

While the general case for n functions is more difficult, for two functions this should be easy with Python 3.10's ParamSpec:

from typing import *

PS = ParamSpec('PS')  # arguments of inner function is arguments of composite function
IT = TypeVar('IT')    # result of inner function needs to be compatible with single argument of outer function
RT = TypeVar('RT')    # result of outer function is result of composite function


def compose(outer: Callable[[IT], RT], inner: Callable[PS, IT]) -> Callable[PS, RT]:
    """Return composite function `outer` after `inner`.
    """
    def h(*args: PS.args, **kwargs: PS.kwargs) -> RT:
        """Return result of  composite function `outer` after `inner` on `args` and `kwargs`.
        """
        return outer(inner(*args, **kwargs))

    return h

This works sometimes:

assert_type((compose(len, str.capitalize)('test')), int)  # succeeds

But not always:

assert_type((compose(abs, len)('test')), int)             # fails in MyPy 0.971 and Python 3.11.0rc1

I get mypy errors:

compose.py:20: error: Expression is of type "Any", not "int"
compose.py:20: error: Cannot infer type argument 1 of "compose"
compose.py:20: error: Argument 2 to "compose" has incompatible type "Callable[[Sized], int]"; expected "Callable[[VarArg(Any), KwArg(Any)], Any]"
Found 3 errors in 1 file (checked 1 source file)

What am I doing wrong? Any ideas?

[sobolevn]

You can have two major cases:

1. With pre-defined number of parameters, like this one: https://github.com/dry-python/returns/blob/119b0817e0d4690efd106dd33ed2fd807dfc7064/returns/functions.py#L34-L51 It is quite easy.

def compose(
    first: Callable[[_FirstType], _SecondType],
    second: Callable[[_SecondType], _ThirdType],
) -> Callable[[_FirstType], _ThirdType]: ...

2. Or you can have a function that composes arbitary number of other functions (like *args). In this case, it is impossible to express this with python's type system. So, we use a mypy plugin:

• Definition: https://github.com/dry-python/returns/blob/master/returns/_internal/pipeline/pipe.py and https://github.com/dry-python/returns/blob/master/returns/_internal/pipeline/pipe.pyi
• Plugin: https://github.com/dry-python/returns/blob/master/returns/contrib/mypy/_features/pipe.py

3. You can also create multiple overloads for a big number of possible arguments. For example, something like 50. Nobody is going to compose more functions in a single call! This is something we used to have: https://github.com/dry-python/returns/blob/0.10.0/returns/generated/pipe.pyi

Or you can just use https://github.com/dry-python/returns 🙂
We have function composition covered!

[Dutcho]

Thanks @sobolevn

Option 1 is similar to the one I mentioned, only I did

• 'outer after first' instead of 'first and then second' (order of args in compose)
• ParamSpec instead of TypeVar for inner or first function (multiple args in innermost == first applied and composite functions)

In both of these, equivalents 'abs after len' and 'len and then abs' (I know the example doesn't make sense) fail on MyPy's not grasping that int matches SupportsAbs.

And yes, option 3 is chaining many together. But first I want the minimal case work correctly, i.e. ascertain match between expected arg type and produced return type.
Any more suggestions would be welcome.

[hmc-cs-mdrissi]

Does len have overloads? I recall paramspec with functions with overloads is buggy. I’m unsure any type checker handles that case today. I think mypy already has a bug report for that.

Edit: https://github.com/python/typeshed/blob/ef6bacab926bd16f8e336f71b5360e37be03b82e/stdlib/builtins.pyi#L1344 Len appears to not have overloads so it’s not that.

I tested your examples in pyright and they both work. So this is a bug in mypy to report.

[Dutcho]

Thanks @hmc-cs-mdrissi
Then it's likely this bug