Exported data (`PyAPI_DATA` rather than `PyAPI_FUNC`)

[encukou]

Exporting data from shared libraries is problematic, and we should instead export functions that fetch the data.
This issue didn't make it to PEP 733 (aside from a passing reference to “non-function symbols”).

I don't recall the reason behind this, and I'd like to have it written down so we avoid cargo-culting. @zooba, could you fill it in here? (Or in the guidelines draft?)

[steve-s]

In HPy we also chose to avoid exporting data, the reason is that function call is more flexible (can be switched to lazy initialization or on demand computation, for example). On top of that I am not sure how well are exported data and (re-)exporting data supported in non-C languages.

[zooba]

Yeah, @steve-s has basically nailed it. My proposed text (but feel free to rewrite/amend):

Data exports (``PyAPI_DATA``) should no longer be used to make global variables
available to embedders and extensions. Instead, use a function (``PyAPI_FUNC``)
that returns the value and an error code (which may just be a ``NULL`` return).
Any allocation, copying, or reference counting should be performed within the
function. This protects against future changes, such as to thread safety or
refcounting, allows the addition of deprecation warnings or failure conditions,
and better supports non-C callers.

The C API Working Group may approve exceptions to any of these guidelines.

[bschoenmaeckers]

I’m working on implementing the limited API for RustPython. This API has advanced significantly in concealing CPython’s implementation details and is becoming a viable alternative to HPy. However, I’m encountering a problem. Currently, built-in types are exported using static data exports which are fundamentally incompatible with how RustPython constructs these types.

Therefore, I’m seeking an alternative API design. Would it be beneficial to move away from static type exports and instead expose getter functions for all built-in types?

xref RustPython/RustPython#7562

[bschoenmaeckers]

I'm looking for something like Py_GetConstant, but for build-in types.

[encukou]

I'd support that change (but can't currently champion it).

[zooba]

PyObject_GetAttr(PyEval_GetBuiltins(), "name")?

[encukou]

PyEval_GetBuiltins() is good for most uses, but it's not a proper C-level solution since users can replace/modify a frame's builtins dict.

[bschoenmaeckers]

Please see the discusses at https://discuss.python.org/t/c-api-add-py-getbuildintype-and-py-getbuildintypeborrowed/107010

[zooba]

PyEval_GetBuiltins() is good for most uses, but it's not a proper C-level solution since users can replace/modify a frame's builtins dict.

I would argue that the limited API should respect those modifications anyway, since it's supposed to be operating at a Python level. Use the CPython API if you want to do CPython-level stuff.

[encukou]

“At a Python level” to a point, but not in the current execution frame scope (by default at least).
Dividing by zero should raise PyExc_ZeroDivisionError, not builtins.ZeroDivisionError.

[zooba]

Dividing by zero should raise PyExc_ZeroDivisionError, not builtins.ZeroDivisionError.

I don't think you can justify that from a language design perspective - we only do it for performance and safety reasons (and because it's easier for native code to access a native struct than a Python one). Consider:

def div(a, b):
    if b == 0:
        raise ZeroDivisionError()
    return a / b

(Also consider the ability to catch the exception, which is only possible if we raise the one that Python code is going to expect.)

Either Python code is the model of the Python language and our C implementation tries to satisfy it, or our C implementation is the model of the Python language and the actual Python language adapts to our reference implementation. Right now, we can justify either, though it makes no sense to not at least try to make the implementation match the language.

As the limited API is supposed/hoped to be the set that any Python implementation can provide, it should behave like the Python language, not the C implementation.

[encukou]

PyEval_GetBuiltins is a reflection function that's inspects the current Python execution frame.

In Python, functions don't use their caller's builtins:

def div(a, b):
    if b == 0:
        raise ZeroDivisionError()
    return a / b
    
try:
    eval("div(1, 0)", globals={'__builtins__': {}, 'div': div})
except ZeroDivisionError:
    print("caught!")

That won't work if div is a C function that uses PyEval_GetBuiltins to get the exception -- unless the C code somehow sets up a frame for itself, but we don't even have API for that yet.

[zooba]

I think we're probably just looking at an edge case interaction between how we handle frames here, because reassigning __builtins__ members works differently from overriding it:

def div(a, b):
    if b == 0:
        raise ZeroDivisionError()
    return a / b

orig_ZeroDivisionError = ZeroDivisionError
try:
    __builtins__.ZeroDivisionError = OverflowError
    eval("div(1, 0)", globals={'div': div})
except orig_ZeroDivisionError:
    print("not caught!")
except ZeroDivisionError:
    print("caught!")

Accessing PyExc_ZeroDivisionError is not going to account for this - it'll still raise the original exception.

So which behaviour is Python? And which is "we didn't manage to get the state machine perfect"? And which behaviour do we want to double-down on by embedding it in more official APIs, thereby making it the de facto specification, even if it wasn't intentional?

These questions are probably above our pay grade though, so we might want to call in a SC consult.