Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion crates/ty_python_core/src/place.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ use crate::{Db, PossiblyNarrowedPlaces};
use ruff_db::parsed::ParsedModuleRef;
use ruff_index::IndexVec;
use ruff_python_ast as ast;
use ruff_python_ast::name::Name;
use smallvec::SmallVec;
use std::hash::Hash;
use std::iter::FusedIterator;
Expand Down Expand Up @@ -45,11 +46,16 @@ pub enum PlaceExpr {
}

impl PlaceExpr {
/// Create a symbol place from a name, without requiring an AST occurrence.
pub fn from_name(name: Name) -> Self {
Self::Symbol(Symbol::new(name))
}

/// Create a new `PlaceExpr` from a name.
///
/// This always returns a `PlaceExpr::Symbol` with empty flags and `name`.
pub fn from_expr_name(name: &ast::ExprName) -> Self {
PlaceExpr::Symbol(Symbol::new(name.id.clone()))
Self::from_name(name.id.clone())
}

/// Tries to create a `PlaceExpr` from an expression.
Expand Down
241 changes: 240 additions & 1 deletion crates/ty_python_semantic/resources/mdtest/annotations/invalid.md
Original file line number Diff line number Diff line change
Expand Up @@ -1240,7 +1240,7 @@ python-version = "3.12"
class C:
list = 42

# TODO: `visible_ancestor_scopes` skips the class through nested annotation scopes,
# TODO: The builtin lookup skips the class through nested annotation scopes,
# so we incorrectly offer a fix that resolves `list` to `C.list`.
type Alias[T] = [int] # snapshot: invalid-type-form
```
Expand All @@ -1262,6 +1262,220 @@ help: Replace with `list[...]`
note: This is an unsafe fix and may change runtime behavior
```

#### Collection literal fixes with aliases to builtins

An import or assignment that binds a collection name to the corresponding builtin still permits the
fix.

```py
import builtins
from builtins import list

items: [int] # snapshot: invalid-type-form

tuple = builtins.tuple
pair: (int, str) # snapshot: invalid-type-form
```

```snapshot
error[invalid-type-form]: List literals are not allowed in this context in a type expression
--> src/mdtest_snippet.py:4:8
|
4 | items: [int] # snapshot: invalid-type-form
| ^^^^^ Did you mean `list[int]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
help: Replace with `list[...]`
|
3 |
- items: [int] # snapshot: invalid-type-form
4 + items: list[int] # snapshot: invalid-type-form
5 |
|
note: This is an unsafe fix and may change runtime behavior


error[invalid-type-form]: Tuple literals are not allowed in this context in a type expression
--> src/mdtest_snippet.py:7:7
|
7 | pair: (int, str) # snapshot: invalid-type-form
| ^^^^^^^^^^ Did you mean `tuple[int, str]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
help: Replace with `tuple[...]`
|
6 | tuple = builtins.tuple
- pair: (int, str) # snapshot: invalid-type-form
7 + pair: tuple[int, str] # snapshot: invalid-type-form
|
note: This is an unsafe fix and may change runtime behavior
```

#### Collection literal fixes with unreachable shadowing

An assignment in an unreachable module-level branch does not shadow the builtin.

```py
if False:
list = 42

items: [int] # snapshot: invalid-type-form
```

```snapshot
error[invalid-type-form]: List literals are not allowed in this context in a type expression
--> src/mdtest_snippet.py:4:8
|
4 | items: [int] # snapshot: invalid-type-form
| ^^^^^ Did you mean `list[int]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
help: Replace with `list[...]`
|
3 |
- items: [int] # snapshot: invalid-type-form
4 + items: list[int] # snapshot: invalid-type-form
|
note: This is an unsafe fix and may change runtime behavior
```

#### Collection literal fixes with global declarations

A `global` declaration skips enclosing function bindings and resolves to the builtin imported at
module scope.

```py
from builtins import list

def outer():
list = 42

def inner():
global list
items: [int] # snapshot: invalid-type-form
```

```snapshot
error[invalid-type-form]: List literals are not allowed in this context in a type expression
--> src/mdtest_snippet.py:8:16
|
8 | items: [int] # snapshot: invalid-type-form
| ^^^^^ Did you mean `list[int]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
help: Replace with `list[...]`
|
7 | global list
- items: [int] # snapshot: invalid-type-form
8 + items: list[int] # snapshot: invalid-type-form
|
note: This is an unsafe fix and may change runtime behavior
```

#### Collection literal fixes with nonlocal declarations

Following `nonlocal` declarations through nested functions can resolve a name to an alias of the
builtin.

```py
def outer():
from builtins import set

def middle():
nonlocal set

def inner():
nonlocal set
items: {int} # snapshot: invalid-type-form
```

```snapshot
error[invalid-type-form]: Set literals are not allowed in type expressions
--> src/mdtest_snippet.py:9:20
|
9 | items: {int} # snapshot: invalid-type-form
| ^^^^^ Did you mean `set[int]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
help: Replace with `set[...]`
|
8 | nonlocal set
- items: {int} # snapshot: invalid-type-form
9 + items: set[int] # snapshot: invalid-type-form
|
note: This is an unsafe fix and may change runtime behavior
```

#### Collection literal fixes with ambiguous bindings

A name that might refer to a different class is not a suitable replacement, even if one of its
possible values is the expected builtin.

```py
import builtins

def check(flag: bool):
list = builtins.list if flag else builtins.set

def inner():
items: [int] # snapshot: invalid-type-form
```

```snapshot
error[invalid-type-form]: List literals are not allowed in this context in a type expression
--> src/mdtest_snippet.py:7:16
|
7 | items: [int] # snapshot: invalid-type-form
| ^^^^^ Did you mean `list[int]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
```

#### Collection literal fixes with unbound local names

An unreachable local assignment still makes the name local to the function. It prevents lookup from
falling back to the builtin.

```py
def check():
if False:
list = 42
items: [int] # snapshot: invalid-type-form
```

```snapshot
error[invalid-type-form]: List literals are not allowed in this context in a type expression
--> src/mdtest_snippet.py:4:12
|
4 | items: [int] # snapshot: invalid-type-form
| ^^^^^ Did you mean `list[int]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
```

#### Collection literal fixes before a local alias is bound

An assignment of the builtin to a local name does not make that name available earlier in the
function.

```py
import builtins

def check():
items: [int] # snapshot: invalid-type-form
list = builtins.list
Comment on lines +1464 to +1466

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about when the items is in an inner class definition given that class variables are evaluated first?

import builtins

def check():
    class C:
        items: [int]
    list = builtins.list

This currently suggests to replace it with list[...] but it raises a NameError on 3.13 but not from 3.14 due to deferred annotations.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried fixing this, but it quickly started spiralling in complexity, and codex kept finding more and more issues with each review pass. I ended up asking codex to reconsider this approach in its entirety and here's its conclusion (I gave it a leading question, so I probably biased it, but still):


I recommend abandoning this implementation and retaining the conservative helper.
Sharing name-resolution logic was a reasonable starting point, but the branch has exposed two different requirements:

  1. Ordinary inference tolerates assumptions that autofix eligibility cannot. For example, the class-body compatibility rule deliberately promotes possibly-bound values to definitely bound. Reusing that result does not establish that introducing a name is safe.
  2. My last change crossed an inappropriate boundary. Changing shared closure snapshots to answer an autofix question affected ordinary inference and discarded useful information. That was a mistake. Insufficient evidence for an autofix should have meant withholding the fix, not changing the semantic index’s behavior.

I don’t think robust reuse is impossible. It would need a clearly separate contract: reuse whatever lookup information is available, but reject cases whose safety depends on inference assumptions or missing execution-order information. Ordinary inference should remain unchanged. However, building and maintaining that distinction would need a worthwhile benefit.

Here, the gains are mostly additional fixes involving aliases, unreachable shadowing, and similar cases. Given that missing those fixes is acceptable, I don’t see enough benefit to justify the additional machinery and coupling. The existing helper’s simple rule—reject visible shadowing and project-level builtin overrides—is much easier to reason about.

I would retain any useful regression tests that pass with the conservative implementation and drop the implementation changes. I haven’t changed the branch.


So I think it is far from trivial to simply "reuse our existing machinery" for name lookup when it comes to autofixes, unfortunately, @MichaReiser. The problem is that our existing machinery wasn't designed for autofix safety in mind, and would have to be significantly redesigned to capture a lot more information if we wanted to reuse it for that purpose. I don't think putting that much effort in is justified at this stage.

The status quo on main is fine for now: we offer autofixes in the common case, and we refrain from offering them in edge cases where we can't be confident that we'd give a good fix.

```

```snapshot
error[invalid-type-form]: List literals are not allowed in this context in a type expression
--> src/mdtest_snippet.py:4:12
|
4 | items: [int] # snapshot: invalid-type-form
| ^^^^^ Did you mean `list[int]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
```

#### Collection literal fixes with project-level builtin overrides

A project-level `__builtins__.pyi` can replace `list` while leaving the standard `set` builtin
Expand Down Expand Up @@ -1304,6 +1518,31 @@ note: This is an unsafe fix and may change runtime behavior
list: object
```

#### Collection literal fixes with project-level replacement classes

A replacement class in `__builtins__.pyi` is not the standard collection class, even when it has the
same name.

```py
items: [int] # snapshot: invalid-type-form
```

```snapshot
error[invalid-type-form]: List literals are not allowed in this context in a type expression
--> src/mdtest_snippet.py:1:8
|
1 | items: [int] # snapshot: invalid-type-form
| ^^^^^ Did you mean `list[int]`?
info: See the following page for a reference on valid type expressions:
info: https://typing.python.org/en/latest/spec/annotations.html#type-and-annotation-expressions
```

`__builtins__.pyi`:

```pyi
class list: ...
```

#### Collection literal fixes are omitted in string annotations

Collection literals parsed from quoted annotations do not have source ranges that can be rewritten
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,31 @@ error[unresolved-reference]: Name `List` used when not defined
| ^^^^ Did you mean `list`?
```

### Builtin replacement imported explicitly

Importing `list` from `builtins` still permits replacing the unresolved `List` with `list`.

```py
from builtins import list

items: List[int] # snapshot: unresolved-reference
```

```snapshot
error[unresolved-reference]: Name `List` used when not defined
--> src/mdtest_snippet.py:3:8
|
3 | items: List[int] # snapshot: unresolved-reference
| ^^^^ Did you mean `list`?
help: Replace with `list`
|
2 |
- items: List[int] # snapshot: unresolved-reference
3 + items: list[int] # snapshot: unresolved-reference
|
note: This is an unsafe fix and may change runtime behavior
```

### Info not present before Python 3.9

<!-- snapshot-diagnostics -->
Expand Down
Loading
Loading