Challenges posed by ZIM files with a huge number of entries

[veloman-yunkan]

Recently there have been a couple of issues spawned by the attempts to convert (geographic) digital maps to ZIM format:

• Seeking advise regarding creation of millions of redirects #1045
• fix: clamp pread() size to avoid EINVAL on macOS for large reads #1053

The problem boils down to the fact that the count of entries in such a ZIM file created from full-world data exceeds the count of entries in our thus far largest ZIM files by an order of magnitude (hundreds of millions vs tens of million in wikipedia_en_all_maxi). Such a leap pushes the current implementation of libzim if not the ZIM file format spec itself beyond limits implicitly assumed during design.

The huge number of entries leads to the following issues/problems/challenges:

1. ZIM file size (inefficient usage of ZIM file space by the entry listing which has always been assumed to be much smaller than the space taken by the entry data).
2. Memory consumption (both during ZIM file creation and/or consumption)
3. Performance (both during ZIM file creation and/or consumption)

[veloman-yunkan]

@rgaudin @benoit74 @kelson42

The first question is whether the said issue with huge number of entries in ZIM files deserves a general treatment or it is peculiar to maps alone (and will stay such in the near future)?

[benoit74]

Thank you for bringing this discussion to the table.

Short answer: I don't know.

Another question which I don't know how to answer is whether we should solve this issue or consider creating so many entries in a ZIM is a non-sense and we need to find another alternative.

This is of particular relevance for maps for two reasons.

First, when implementing maps I followed the approach to create hundreds of millions of entries because it looked more appealing at first sight. It is important to note that it should also be possible to use another data format where we have one single (giant) entry and we do range requests on that file. I was quite worried about how HTTP range requests on a giant file would be supported by the various readers.

Second (this is maybe more the workaround we will follow for the time being), we could decide to create maps ZIMs without the most detailed level of zoom (at least for big areas), which will probably decrease number of files by an order of magnitude, I do not have exact numbers but each zoom level contains 4 times more tiles than the previous one. And hundreds of millions of tiles is also the worst case for the whole world, which we might decide not to build.

At least it could help to find "time" in the sense that we will release ZIMs now with known workarounds and working well everywhere and have time to correctly identify strategies to build the ZIMs we want to have in the future.

[kelson42]

ZIM file size (inefficient usage of ZIM file space by the entry listing which has always been assumed to be much smaller than the space taken by the entry data).
Memory consumption (both during ZIM file creation and/or consumption)
Performace (both during ZIM file creation and/or consumption)

From my understanding, none of these points is a blocker for 1.0 release of these maps, right?

For the rest, these maps have a lots of tiles and there nothing much we can do to reduce the number of entries (beside using aliases). I guess, we will just have to coop with this kind of amount of entries in the future.

At this stage we should focus on optims on both ZIM and libzim.

[benoit74]

From my understanding, none of these points is a blocker for 1.0 release of these maps, right?

I consider that using GBs of memory is an issue (see #1053, but issue is probably general on all readers, we just observed visible consequences on kiwix-apple).

For the rest, these maps have a lots of tiles and there nothing much we can do to reduce the number of entries (beside using aliases). I guess, we will just have to coop with this kind of amount of entries in the future.

Implementing aliases will not reduce number of entries AFAIK.

As mentioned above, to reduce number of entries we can at least "refuse" to create certain ZIMs (e.g. whole world up to zoom 14).

Another "marginal" improvement might be possible with openzim/maps#78

[kelson42]

From my understanding, none of these points is a blocker for 1.0 release of these maps, right?

I consider that using GBs of memory is an issue (see #1053, but issue is probably general on all readers, we just observed visible consequences on kiwix-apple).

Yes, this is a problem if it happens at reader time, but not a blocker at this stage.

A few improvements have been done at both libzim and scraper. We need now to make the worldwide map, see openzim/zim-requests#1745 and make precise measurements.

[kelson42]

We have now a worldwide map at https://farm.openzim.org/recipes/maps_en_all.

We should now get the hard facts about memory consumption to be able to move the issue forward.

[benoit74]

Zimfarm task says it used the 64GB I've entitled: https://farm.openzim.org/pipeline/ebac5f07-9a05-449b-a20c-fe1e8ad32773/debug

In general this means we were probably quite close to OOM.