LazyProxy can't be encoded by Flask JSONEncoder

[ramnes]

from flask import Flask, jsonify
from lazify import LazyProxy

app = Flask(__name__)


@app.route("/")
def test():
    return jsonify(LazyProxy(lambda: {"test": "test"}))


app.run()

This test case raises:

[2018-04-16 09:59:48,336] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/test/.venv/lib/python3.4/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/test/.venv/lib/python3.4/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/test/.venv/lib/python3.4/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/test/.venv/lib/python3.4/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/test/.venv/lib/python3.4/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/test/.venv/lib/python3.4/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "toto.py", line 9, in test
    return jsonify(LazyProxy(lambda: {"test": "test"}))
  File "/test/.venv/lib/python3.4/site-packages/flask/json.py", line 263, in jsonify
    (dumps(data, indent=indent, separators=separators), '\n'),
  File "/test/.venv/lib/python3.4/site-packages/flask/json.py", line 123, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "/usr/lib64/python3.4/json/__init__.py", line 237, in dumps
    **kw).encode(obj)
  File "/usr/lib64/python3.4/json/encoder.py", line 194, in encode
    chunks = list(chunks)
  File "/usr/lib64/python3.4/json/encoder.py", line 429, in _iterencode
    o = _default(o)
  File "/test/.venv/lib/python3.4/site-packages/flask/json.py", line 80, in default
    return _json.JSONEncoder.default(self, o)
  File "/usr/lib64/python3.4/json/encoder.py", line 173, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <lazify.LazyProxy object at 0x7f6ea61340a0> is not JSON serializable
127.0.0.1 - - [16/Apr/2018 09:59:48] "GET / HTTP/1.1" 500 

Versions in use:

• Lazify==0.3.1
• Flask==0.12.2
• Werkzeug==0.14.1

[ramnes]

Good news, bad news:

• caf8acb fixes this problem with Flask<=0.12.2. 👍
• Sadly, something in Flask==1.0.2 still avoid the proxy to be properly recognized and — strange thing — it only happens when Flask's debug mode is off! 👎

Another thing to note is that the issue doesn't seem linked to the json library installed, nor its version, nor Werkzeug version. Just upgrading from Flask==0.12.2 to Flask==1.0.2 without touching any other dependency makes it happen.

ping @trubesv, FYI

[ramnes]

I've just made a bisect and it broke between Flask==0.12.5 and Flask==1.0.

[ramnes]

Hey, looks like we found a bug in Python.

This works:

>>> dumps(LazyProxy(dict), indent=0)
'{}'

But this does not:

>>> dumps(LazyProxy(dict), indent=None)
... stacktrace ...
TypeError: <lazify.LazyProxy object at 0x7f71ee8e73b8> is not JSON serializable

Before 1.0, Flask used to prettify JSON by default (with indent=2), so the bug was not easily visible.

Therefore, one workaround is to set JSONIFY_PRETTYPRINT_REGULAR to True.

[matrixise]

strange, because the default value for indent is None in the code of >= Python 3.7
https://github.com/python/cpython/blob/master/Lib/json/__init__.py#L184

[ramnes]

@matrixise I was just adding indent=None to make it explicit. indent default value is not the point here; the point is that json.dumps doesn't do the same type checking with and without an indent value. It might be that the C version of it is used when indent=None, and that it behaves differently than the Python version.

[matrixise]

yep maybe, fill your issue on b.p.o and we will check it

[ramnes]

I forgot to update the issue here, but: it's already in Python's bug tracker.

IRC copy-pasta from months ago:

14h19 berker ramnes: I think it's a similar issue, if I change line 1498 to PyObject_IsInstance(obj, (PyObject*)&PyDict_Type), it works
14h39 berker ramnes: I'd suggest adding a "I'm working on this" comment and include a small reproducer in the issue, so someone else won't work on it at the same time
14h41 berker ramnes: here's my minimal reproducer: https://gist.github.com/berkerpeksag/3f06b99e7e590ed7ca24b152ff01669e you probably can make it even more smaller since you know lazify.py than myself :)