RFC: Table Comprehension

[InfraredGodYT]

Rendered

This RFC proposes the feature of table comprehension, similar to languages like Python & MoonScript. This is my first RFC so it's probably pretty rough.

[InfraredGodYT]

Marked it as a draft because it will probably require several changes before it's ready.

[Cooldude2606]

Just an early comment for consideration as you flesh it out.

What is the reasoning for why comprehension should not be combined with standard lua fields? It is possible that it could follow the same logic as if the comprehension produces duplicate keys.

Would be nice to see an extended alternative to "helper functions" of including them in the standard lib. For example having table.map would enable fast calling and iteration on the C side. Which could see better performance than a lua only helper function, although likely slower than a for loop or comprehension given table.foreach is deprecated.

[vegorov-rbx]

Fastcall functions can not call user code, yield, fail, or reallocate stack, so if table.map was a library function it would not get special treatment on the C side.

[Bottersnike]

Does this RFC address multiple loops and conditions within a single comprehension? [j for i in array for j in i] in Python is a very useful way to flatten a 2D array into a 1D one.

If not, something to consider is what order the fors would evaluate. Python for example does [expression] [ordered as if you were writing normal for loops] [conditions] which can be quite confusing to read.

With Luau in its current state I personally would prefer comprehensions over table.map simply due to the cumbersome nature of writing lambdas right now; table.map(foo, function (x) return x + 1 end) is quite a mouthful, and the actual "what's going on" (x + 1) is pretty nested in there.

Something else to consider is other types of for loops. For example, (again using python as an example) you can write [i + 1 for i in range(10)] in python, to get an array of 1..10. Luau doesn't use a magic range iterator, instead doing for i=1,10 do, so it would be interesting to be able to write {i+1 for i=1,10} or similar (to in this case get 2..11).

Something else to also consider is allowing mixing for and if statements, rather than exclusively fors followed by ifs. For example, Python allows

>>> a = [[1, 2, 3], None, [4, 5, 6]]
>>> [
...  i
...  for j in a
...  if j
...  for i in j
... ]
[1, 2, 3, 4, 5, 6]

These suggestions all depend on how "complex" you want to allow comprehensions to be. Python's can be very complex, and therefore very powerful, but the trade-off comes from potential readability. Formatting them properly aids a lot with this, but that complexity may still be undesirable. As is, too, there's nothing in the proposal that would prevent future expansion to any of the above, so it may be preferable to have first a minimal implementation, followed by more complex extensions later.

Edit: Just as a reference, given I've mentioned Python a lot in this comment, https://docs.python.org/3/reference/expressions.html#grammar-token-python-grammar-comprehension and https://docs.python.org/3/reference/expressions.html#grammar-token-python-grammar-dict_comprehension are the explained grammar for how Python handles comprehensions.

[InfraredGodYT]

Just an early comment for consideration as you flesh it out.

What is the reasoning for why comprehension should not be combined with standard lua fields? It is possible that it could follow the same logic as if the comprehension produces duplicate keys.

Would be nice to see an extended alternative to "helper functions" of including them in the standard lib. For example having table.map would enable fast calling and iteration on the C side. Which could see better performance than a lua only helper function, although likely slower than a for loop or comprehension given table.foreach is deprecated.

I added standard library helpers as an alternative in the RFC.

[InfraredGodYT]

Does this RFC address multiple loops and conditions within a single comprehension? [j for i in array for j in i] in Python is a very useful way to flatten a 2D array into a 1D one.

If not, something to consider is what order the fors would evaluate. Python for example does [expression] [ordered as if you were writing normal for loops] [conditions] which can be quite confusing to read.

With Luau in its current state I personally would prefer comprehensions over table.map simply due to the cumbersome nature of writing lambdas right now; table.map(foo, function (x) return x + 1 end) is quite a mouthful, and the actual "what's going on" (x + 1) is pretty nested in there.

Something else to consider is other types of for loops. For example, (again using python as an example) you can write [i + 1 for i in range(10)] in python, to get an array of 1..10. Luau doesn't use a magic range iterator, instead doing for i=1,10 do, so it would be interesting to be able to write {i+1 for i=1,10} or similar (to in this case get 2..11).

Something else to also consider is allowing mixing for and if statements, rather than exclusively fors followed by ifs. For example, Python allows

>>> a = [[1, 2, 3], None, [4, 5, 6]]
>>> [
...  i
...  for j in a
...  if j
...  for i in j
... ]
[1, 2, 3, 4, 5, 6]

These suggestions all depend on how "complex" you want to allow comprehensions to be. Python's can be very complex, and therefore very powerful, but the trade-off comes from potential readability. Formatting them properly aids a lot with this, but that complexity may still be undesirable. As is, too, there's nothing in the proposal that would prevent future expansion to any of the above, so it may be preferable to have first a minimal implementation, followed by more complex extensions later.

Edit: Just as a reference, given I've mentioned Python a lot in this comment, https://docs.python.org/3/reference/expressions.html#grammar-token-python-grammar-comprehension and https://docs.python.org/3/reference/expressions.html#grammar-token-python-grammar-dict_comprehension are the explained grammar for how Python handles comprehensions.

This RFC currently only targets a minimal form of table comprehension. At the moment, the proposal only supports a single for clause and an optional trailing if condition. More complex forms like multiple for clauses or interleaved conditions aren't included for now to keep the feature simple and readable.

Do you think starting with this minimal subset makes sense?

[Bottersnike]

Do you think starting with this minimal subset makes sense?

I'm honestly unsure! As-is, it doesn't restrict their addition in the future, which is really good.

I really couldn't say if it makes more sense to have one uber-fleshed-out RFC or one that's a starting point and a second that expands it. The benefit of just starting with a simple RFC is it's potentially more likely to get merged, but the risk of hoping to get a second merged too is if that second doesn't you can end up with a half-baked feature.

[InfraredGodYT]

I really couldn't say if it makes more sense to have one uber-fleshed-out RFC or one that's a starting point and a second that expands it. The benefit of just starting with a simple RFC is it's potentially more likely to get merged, but the risk of hoping to get a second merged too is if that second doesn't you can end up with a half-baked feature.

That makes sense. I think it’s safer to establish a simple baseline first and then explore more complex extensions separately if there’s interest.

For now, I’m planning to keep the RFC minimal to keep the feature easy to understand and reason about. That said, I’m open to expanding the scope if there’s strong feedback that the minimal version would be too limited.

[deviaze]

I always found the [ x for y in whatever ] syntax quite confusing once you got into nested comprehensions, so I recommend a syntax that aligns with regular Luau's if expressions (in Python, you have x if y else z but in Luau you have if y then x else z) and preserves readability with nested comprehensions:

-- simple
local luau_files: { string } = { 
    for _, filename in fs.listdir("./some/path") do
        if string.match(filename, "%.luau$") then
            filename
        else -- like with regular if expressions outside table comprehensions, every if expression must have an else clause
            continue
}

This makes nested comprehensions much more readable (even if they're more verbose)

-- nested
local results = {} :: { [string]: { string } }
local elements_with_meow = {
    for name, list in results do
        for _, element in list do
            if string.match(element, "meow") then
                element
            else
                continue     
}

I actually wrote a whole table comprehensions RFC on this a few years ago (that also clearly describes the semantics of break and continue within comprehensions) but I can't seem to find it anymore :(

As a nit, I recommend removing the uses of pairs and ipairs in the draft since they're *basically* deprecated in favor of generalized iteration (no iterator function needed) in Luau.

[InfraredGodYT]

I think that kind of design could be worth exploring, but it would likely be better suited as a separate RFC rather than being combined into this one. For this RFC, I’m intentionally keeping the design minimal, while leaving room for more complex forms like nested comprehensions to be explored as future extensions if there’s interest.

I'll also remove the use of pairs and ipairs in the next commit.

[Brawldude2]

What does it do other than bloating syntax tree and ruining the intuitive design principle. It doesn't take 2 lines to do the same thing without additional syntax

[InfraredGodYT]

What does it do other than bloating syntax tree and ruining the intuitive design principle. It doesn't take 2 lines to do the same thing without additional syntax

While this RFC doesn't introduce new capabilities, the goal of it is to provide a more concise way to represent common table transformation patterns. These transformations can be written within a few lines today, but usually involve boilerplate that can obscure the intent of the code (e.g. table initialization, mutation, and conditional insertion). Using less than 2 lines for said common transformations will make the code unnecessarily more difficult to read, and this proposal focuses on making those patterns easier to express inline rather than expanding what the language can do.