Modalities expects JSONL to have a trailing \n for the last JSON document (best practice but not guaranteed by JSONL file format)

[le1nux]

System Info

current master version (cb096c2) and probably all previous versions.

🐛 Describe the bug

Modalities currently expects that each JSONL file ends with a trailing \n.

While a trailing \nafter all JSONs is best practice, it is not enforced by the JSONL format that also the last JSON ends with \n.

https://jsonlines.org/

Including a line terminator after the last JSON value in a file is strongly recommended but not required.

In Modalities, this can lead to e.g., two documents getting concatenated when shuffling (last document + some non-last document). For large files, this is probably negligible ... but for consistency, we should fix this issue.

[le1nux]

Even if it is not enforced by the JSONL standard, I think we should enforce it in Modalities for consistency reasons.