Skip to content

[#10788] improvement(deps): upgrade Hadoop from 2.10.2 to 3.3.6 and remove hadoop2 dependency line#10789

Open
geyanggang wants to merge 3 commits intoapache:mainfrom
geyanggang:upgrade-hadoop-version
Open

[#10788] improvement(deps): upgrade Hadoop from 2.10.2 to 3.3.6 and remove hadoop2 dependency line#10789
geyanggang wants to merge 3 commits intoapache:mainfrom
geyanggang:upgrade-hadoop-version

Conversation

@geyanggang
Copy link
Copy Markdown
Collaborator

@geyanggang geyanggang commented Apr 15, 2026

What changes were proposed in this pull request?

  • Remove hadoop2 version (2.10.2) and its 6 library definitions from libs.versions.toml
  • Bump hadoop3 from 3.3.1 to 3.3.6 (including hadoop3-abs, hadoop3-aliyun, hadoop-minikdc)
  • Replace all libs.hadoop2.* references with libs.hadoop3.* in 8 build files
  • Add Hadoop 3.x runtime dependencies stripped by exclude("*"): hadoop-shaded-guava, hadoop-shaded-protobuf, commons-configuration2, re2j,hadoop-hdfs-client
  • Remove outdated hadoop2/hadoop3 conflict comment in catalog-hive

Why are the changes needed?

  • The hadoop2 (2.10.2) dependency line brings in outdated transitive dependencies (Jackson 2.6.x, Netty 3.x, Jetty 9.3.x) that are no longer maintained
  • hive-metastore3-libs was incorrectly configured with hadoop2 despite Hive 3.1.3 natively depending on Hadoop 3.1.0
  • Unifying on a single Hadoop version eliminates classpath conflicts and simplifies dependency management

Fix: #10788

Does this PR introduce any user-facing change?

No.

How was this patch tested?

  • Full compilation (./gradlew build -x test) — passed
  • Verified hadoop-common-3.3.6 resolves in all affected modules, no Hadoop 2.x JARs in distribution
  • All unit tests passed across 12 key modules including catalog-hive, hive-metastore-common, catalog-lakehouse-hudi, catalog-fileset, filesystem-hadoop3, hadoop-common, authorization-ranger, spark-connector, flink-connector, core
  • Integration tests pending CI validation

@github-actions
Copy link
Copy Markdown

github-actions bot commented Apr 15, 2026

Code Coverage Report

Overall Project 65.15% 🟢
Files changed No Java source files changed -

Module Coverage
aliyun 1.73% 🔴
api 47.09% 🟢
authorization-common 85.96% 🟢
aws 1.1% 🔴
azure 2.6% 🔴
catalog-common 10.2% 🔴
catalog-fileset 80.02% 🟢
catalog-glue 75.36% 🟢
catalog-hive 81.83% 🟢
catalog-jdbc-clickhouse 79.06% 🟢
catalog-jdbc-common 42.89% 🟢
catalog-jdbc-doris 80.28% 🟢
catalog-jdbc-hologres 54.03% 🟢
catalog-jdbc-mysql 79.23% 🟢
catalog-jdbc-oceanbase 78.38% 🟢
catalog-jdbc-postgresql 82.05% 🟢
catalog-jdbc-starrocks 78.27% 🟢
catalog-kafka 77.01% 🟢
catalog-lakehouse-generic 45.07% 🟢
catalog-lakehouse-hudi 79.1% 🟢
catalog-lakehouse-iceberg 87.27% 🟢
catalog-lakehouse-paimon 77.71% 🟢
catalog-model 77.72% 🟢
cli 44.51% 🟢
client-java 77.63% 🟢
common 48.97% 🟢
core 81.41% 🟢
filesystem-hadoop3 76.97% 🟢
flink 40.55% 🟢
flink-runtime 0.0% 🔴
gcp 14.2% 🔴
hadoop-common 10.39% 🔴
hive-metastore-common 46.14% 🟢
iceberg-common 50.73% 🟢
iceberg-rest-server 65.93% 🟢
integration-test-common 0.0% 🔴
jobs 66.17% 🟢
lance-common 23.88% 🔴
lance-rest-server 57.84% 🟢
lineage 53.02% 🟢
optimizer 82.95% 🟢
optimizer-api 21.95% 🔴
server 85.75% 🟢
server-common 69.52% 🟢
spark 32.79% 🔴
spark-common 39.09% 🔴
trino-connector 33.83% 🔴

@geyanggang geyanggang force-pushed the upgrade-hadoop-version branch from 09b0aa7 to 3d49584 Compare April 16, 2026 03:53
@geyanggang geyanggang requested a review from diqiu50 April 16, 2026 08:41
@geyanggang geyanggang self-assigned this Apr 16, 2026
@geyanggang geyanggang requested review from mchades and yuqi1129 April 16, 2026 08:42
@geyanggang geyanggang force-pushed the upgrade-hadoop-version branch from 3d49584 to 524d69d Compare April 16, 2026 14:34
@geyanggang geyanggang force-pushed the upgrade-hadoop-version branch from 524d69d to c7b3584 Compare April 17, 2026 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Upgrade Hadoop from 2.10.2 to 3.3.6 and unify on a single Hadoop 3.x dependency line

1 participant