Flyte 2 WIP by EngHabu · Pull Request #6583 · flyteorg/flyte

EngHabu · 2025-08-22T06:38:23Z

Add flyteidl and minimal publishing workflows

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

added: For new features.
changed: For changes in existing functionality.
deprecated: For soon-to-be-removed features.
removed: For features being removed.
fixed: For any bug fixed.
security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

I updated the documentation accordingly.
All new and existing tests passed.
All commits are signed-off.

Related PRs

Docs link

main
- Flyte 2 WIP 👈

* Add flyteidl and minimal publishing workflows Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * pretend version for buf python Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

flyte-bot · 2025-08-22T06:38:58Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* python version fix and use OIDC to authenticate crates Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix python versioning Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Set version before setting up env Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * echo version Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Write to GITHUB_ENV Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Use env: Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf generate Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

flyte-bot · 2025-08-22T18:38:23Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

flyte-bot · 2025-08-25T20:39:37Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Mark go package as /v2 to be able to import both at the same time Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Upgrade to go 1.24 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Rename python, rust and npm to flyteidl2 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

flyte-bot · 2025-08-26T01:51:47Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

…6610) * trying idl2 Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update trace def Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename idl2 to flyteidl2 Signed-off-by: machichima <nary12321@gmail.com> * generate -> buf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor: rename idl2 to flyteidl2 Signed-off-by: machichima <nary12321@gmail.com> * build: re-gen Signed-off-by: machichima <nary12321@gmail.com> * fix: exclude python venv Signed-off-by: machichima <nary12321@gmail.com> * Add translator service Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Regen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf gen Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * remove node execution service proto Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Breakup task service into a separate package Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Help strings for make file targets Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update TaskLog with read bool Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Trying the updated protovalidate plugin Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * protovalidate with semver Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try claude :shrug Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add buf Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update deps Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * build: remove generate validate for python Signed-off-by: machichima <nary12321@gmail.com> * build: use managed mode on buf.gen.yaml Signed-off-by: machichima <nary12321@gmail.com> * Update validate go plugin Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update imagebuild Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Fix oneof buf lint Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Add deps to generated python code Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Be explicit about the index Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * updated idl Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Catch up to main Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Generate mocks for workflow package Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * generate mocks for other packages Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * buf improvements Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * updates Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * feat: generate mock for common Signed-off-by: machichima <nary12321@gmail.com> * use make gen in GH Actions Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Do not setup python env in generation CI Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * setup uv Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * set working directory Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Try to reference files by path Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Explicitly move .venv to root Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * try exclude in buf.yaml Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * set UV_PROJECT_ENVIRONMENT Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * download tooling Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * add which mockery Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> Signed-off-by: machichima <nary12321@gmail.com> Co-authored-by: machichima <nary12321@gmail.com>

flyte-bot · 2025-09-22T20:23:04Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at eduardo@union.ai.

* Add Trigger APIs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * update triggers Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * wip Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * refactor Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

bito-code-review · 2025-09-30T18:23:19Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Fix cargo build and add CI check Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * lint lib.rs Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * Update Readme and contributing guides Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * signed? Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix build-crate target Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> * fix included ts files in npm Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com> --------- Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

bito-code-review · 2025-09-30T22:29:13Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

bito-code-review · 2025-10-03T18:55:43Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Add device class to `GpuAccelerator` Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * make gen Signed-off-by: Jeev B <jeevb@users.noreply.github.com> * Add HABANA_GAUDI as supported device class Signed-off-by: Jeev B <jeevb@users.noreply.github.com> --------- Signed-off-by: Jeev B <jeevb@users.noreply.github.com>

bito-code-review · 2025-10-04T00:01:40Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

bito-code-review · 2025-10-04T04:21:41Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

bito-code-review · 2025-10-07T18:11:43Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

bito-code-review · 2025-10-08T19:51:43Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

* Clean up triggers api Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * make gen Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> --------- Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

bito-code-review · 2025-10-09T18:11:16Z

Bito Automatic Review Skipped - Branch Excluded

Bito didn't auto-review because the source or target branch is excluded from automatic reviews.
No action is needed if you didn't intend for the agent to review it. Otherwise, to manually trigger a review, type /review in a comment and save.
You can change the branch exclusion settings here, or contact your Bito workspace admin at haytham@union.ai.

…rgo check (#7168)

…ffs (#7185) * Add .gitattributes to collapse generated files and mocks in Github diffs Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * regen mocks Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * nvm Signed-off-by: Katrina Rogan <katroganGH@gmail.com> --------- Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

* feat: add CreateDownloadLink dataproxy service Signed-off-by: machichima <nary12321@gmail.com> * feat: implement dataproxy CreateDownloadLink Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>

* feat: add trigger model and service Signed-off-by: machichima <nary12321@gmail.com> * fix: DB migration & run service get trigger when CreateRun Signed-off-by: machichima <nary12321@gmail.com> * feat: add scheduler Signed-off-by: machichima <nary12321@gmail.com> * fix: add kickoff args to CreateRun request Signed-off-by: machichima <nary12321@gmail.com> * fix: parse taskspec and store to run DetailedInfo in service layer Signed-off-by: machichima <nary12321@gmail.com> * fix: do not catch up on inactive schedule Signed-off-by: machichima <nary12321@gmail.com> * feat: add fixed rate support Signed-off-by: machichima <nary12321@gmail.com> * feat: update runName hash Signed-off-by: machichima <nary12321@gmail.com> * build: make gen Signed-off-by: machichima <nary12321@gmail.com> * feat: update TriggeredAt when create action Signed-off-by: machichima <nary12321@gmail.com> * feat: store trigger info in action model & return in GetRunDetail & WatchAction Signed-off-by: machichima <nary12321@gmail.com> * refactor: set deployedBy to empty string rather than mock-XXX Signed-off-by: machichima <nary12321@gmail.com> * feat: trigger from gorm to sqlx Signed-off-by: machichima <nary12321@gmail.com> * feat: CreateTask call upsert trigger Signed-off-by: machichima <nary12321@gmail.com> * feat: task key to struct Signed-off-by: machichima <nary12321@gmail.com> * fix: write trigger info in CreateAction Signed-off-by: machichima <nary12321@gmail.com> * fix: fixed utc for correct schedule time Signed-off-by: machichima <nary12321@gmail.com> * feat: trigger scheduler enable by default Signed-off-by: machichima <nary12321@gmail.com> * fix: distinguish not found and internal err Signed-off-by: machichima <nary12321@gmail.com> * fix: create action set task name Signed-off-by: machichima <nary12321@gmail.com> * nit: use root action name constant Signed-off-by: machichima <nary12321@gmail.com> * feat: list task filter with version Signed-off-by: machichima <nary12321@gmail.com> * fix: do not catch up when reactivate Signed-off-by: machichima <nary12321@gmail.com> * fix: show trigger type in list task Signed-off-by: machichima <nary12321@gmail.com> * feat: prune triggers that's not link to the task anymore Signed-off-by: machichima <nary12321@gmail.com> --------- Signed-off-by: machichima <nary12321@gmail.com>

* Add cluster service and SelectCluster method Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * regen Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * revert mocks Signed-off-by: Katrina Rogan <katroganGH@gmail.com> --------- Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

This reverts commit 82ae70b.

* initial proto Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * fix Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * regen Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * revert Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * fix tests Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * regen mocks Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * consistent Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * regen Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * restore service changes Signed-off-by: Katrina Rogan <katroganGH@gmail.com> * unit tests, fixes Signed-off-by: Katrina Rogan <katroganGH@gmail.com> --------- Signed-off-by: Katrina Rogan <katroganGH@gmail.com>

Signed-off-by: Alex Wu <c.alexwu@gmail.com>

* fix(runs): persist model CreatedAt in CreateAction for correct trace ordering Trace actions set CreatedAt to the span's start_time so that parents (which start before children but are recorded later) sort earlier in ListActions ORDER BY created_at ASC. The INSERT was omitting created_at, causing the DB to use CURRENT_TIMESTAMP (insert time) and reversing the parent/child order — leading to "parent node not found" errors in WatchActions when a child action was processed before its parent. Non-trace actions leave CreatedAt as the zero value and fall back to time.Now(), preserving existing behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: Alex Wu <c.alexwu@gmail.com> * add end time Signed-off-by: Alex Wu <c.alexwu@gmail.com> * fix sql sentence Signed-off-by: Alex Wu <c.alexwu@gmail.com> --------- Signed-off-by: Alex Wu <c.alexwu@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Add Abort APIs (#6590)

82a41b6

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Vendor in python buf dependencies (#6651)

1993238

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

EngHabu added 5 commits October 3, 2025 21:20

wip

7bb0db0

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

queue service implementation

075f324

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Support sqllite

987d536

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Runs service

aa24751

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Connect RunService to QueueService

3ef9544

Signed-off-by: Haytham Abuelfutuh <haytham@afutuh.com>

Triggers API adjustments (#6655)

83acecf

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

RunService.CreateRun from_automation flags (#6653)

cda285a

Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

Cleanup triggers api (#6666)

a480286

* Clean up triggers api Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> * make gen Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai> --------- Signed-off-by: Iaroslav Ciupin <iaroslav@union.ai>

github-actions bot mentioned this pull request Apr 8, 2026

[V2] Rename flyte-sandbox to flyte-demo and simplify pod names #7170

Merged

4 tasks

[V2] Speed up check-generate CI with parallelization, caching, and ca…

4f65725

…rgo check (#7168)

pingsutw and others added 2 commits April 9, 2026 09:37

[V2] Rename flyte-sandbox to flyte-demo and simplify pod names (#7170)

9cb3b78

machichima added 2 commits April 9, 2026 21:50

github-actions bot mentioned this pull request Apr 10, 2026

Move TailLogs endpoint to dataproxy #7194

Open

3 tasks

This was referenced Apr 10, 2026

[V2] List run accept filter #7192

Open

Revert "[V2] Dataproxy support CreateDownloadLink (#7191)" #7195

Merged

Revert "[V2] Dataproxy support CreateDownloadLink (#7191)" (#7195)

04515a5

This reverts commit 82ae70b.

github-actions bot mentioned this pull request Apr 10, 2026

feat: path based ingress routing for Flyte apps via Traefik + Knative #7197

Open

3 tasks

github-actions bot mentioned this pull request Apr 13, 2026

[V2] Dataproxy support CreateDownloadLink #7199

Open

3 tasks

popojk and others added 2 commits April 13, 2026 17:11

fix (#7181)

668913c

Signed-off-by: Alex Wu <c.alexwu@gmail.com>

Conversation

EngHabu commented Aug 22, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tracking issue

Why are the changes needed?

What changes were proposed in this pull request?

How was this patch tested?

Labels

Setup process

Screenshots

Check all the applicable boxes

Related PRs

Docs link

Uh oh!

flyte-bot commented Aug 22, 2025

Uh oh!

flyte-bot commented Aug 22, 2025

Uh oh!

flyte-bot commented Aug 25, 2025

Uh oh!

flyte-bot commented Aug 26, 2025

Uh oh!

flyte-bot commented Sep 22, 2025

Uh oh!

bito-code-review bot commented Sep 30, 2025

Uh oh!

bito-code-review bot commented Sep 30, 2025

Uh oh!

bito-code-review bot commented Oct 3, 2025

Uh oh!

bito-code-review bot commented Oct 4, 2025

Uh oh!

bito-code-review bot commented Oct 4, 2025

Uh oh!

bito-code-review bot commented Oct 7, 2025

Uh oh!

bito-code-review bot commented Oct 8, 2025

Uh oh!

bito-code-review bot commented Oct 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

EngHabu commented Aug 22, 2025 •

edited by github-actions bot

Loading