[Vamsi-klu]

Closes #10584

Change Description

Background

table_extractor.lua already checks hidden segments only below _lakefs_tables/, and Spark does the same when it lists a table (HadoopFSUtils.shouldFilterOutPath looks only at components below the base path). The Hive partition pager did not.

Bug Fix

1. Problem - Symlink export of a Hive table at a path like _staging/ev writes one empty ev//symlink.txt and the run reports success. The same table at staging/ev exports one symlink.txt per partition.
2. Root cause - extract_partition_pager in pkg/actions/lua/lakefs/catalogexport/hive.lua calls pathlib.is_hidden(entry.path) on the full repository path. is_hidden is true when any segment starts with _, so every object under _staging/ev is skipped and the pager returns a single empty partition.
3. Solution - Normalize the table path to a /-terminated prefix once, and run is_hidden only on the part of each path below it. An object exactly at the table path counts as "". Paths outside the prefix (for example a sibling _staging/ev2/... returned by the prefix listing) keep the full-path check. _SUCCESS, _temporary/ and other _ entries below the table path are still skipped. IsHidden is unchanged. Only hive.lua changes; the symlink and Glue exporters use it as is.

New Feature

N/A

Testing Details

New Lua fixture catalogexport_hive_partition_pager_hidden_base in TestLuaRunTable. Its cases:

• base _staging/ev with and without a trailing slash: two partitions, with _SUCCESS, _tmp/ and _temporary/ below the base, a .crc file, and a sibling _staging/ev2
• base ""
• an unpartitioned table at _staging/raw, including an object exactly at the table path
• partition value a=_foo
• partition column _a
• an empty table

On master every _ base yields one empty partition. With the fix the expected partitions and files come back. The existing catalogexport_hive_partition_pager golden file is unchanged.

go test -count=1 ./pkg/actions/ -run 'TestLuaRunTable/catalogexport_hive_partition_pager_hidden_base' -v
go test -count=1 -race ./pkg/actions/...
go test -count=1 ./pkg/actions/lua/path/ -run TestIsHidden
go vet ./pkg/actions/...
go tool golangci-lint run ./pkg/actions/...

Manual: I ran the issue's repro on lakefs run --quickstart built from this branch. The hook's writer also PUTs through the lakeFS S3 gateway, and the data includes _SUCCESS and _temporary/0/part-9.parquet below the table path.

• Before: one 0-byte ev//symlink.txt.
• After: ev/dt=2024-01-01/symlink.txt with one physical address and ev/dt=2024-01-02/symlink.txt with two. The hidden files are not listed.

Breaking Change?

No. Only tables whose path has a segment starting with _ behave differently, and those exported nothing before. For other tables the check below the base gives the same result as the old full-path check.

Additional info

Out of scope, unchanged:

• Partition column names starting with _ (for example _a=1) are still skipped, as in Spark.
• A table with no exportable objects still writes one empty <name>//symlink.txt, and the run still reports success. Whether that should fail is a separate decision.
• examples/hooks/s3_hive_manifest_exporter.lua (lines 115 and 167) has the same full-path is_hidden check. It is an example hook, not the shipped exporter, so it is not changed here. I can include it if you prefer.

Contact Details

Comments on this PR.

Drafted-by: Claude Opus 5.5; reviewed by @Vamsi-klu before posting

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-01T20:33:04.882418Z	ecb1498	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.