-
Notifications
You must be signed in to change notification settings - Fork 1.3k
feat(asset-gen): AI Asset Generation — 3D gen/import + 2D image (BYO-key) #1218
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 24 commits
e838276
d0a6ef0
d6604a5
b7652ca
e728c35
9de38c6
ca5f961
3e51289
c5e3d85
8513ff8
57d2523
4886307
71887b9
1316b6d
86cd4ee
6dcdda1
61b9b0f
7755a5f
125c97d
e243e30
2efb786
5ad4e5e
4b4b7bd
39eb562
60ce630
77163d4
a884ada
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| --- | ||
| name: blender-to-unity | ||
| description: Hand off a model from Blender (via BlenderMCP) into Unity (via MCP for Unity) — export the current Blender model, import it through import_model_file, and place it in the open scene. Use when the user has BlenderMCP and MCP for Unity both connected and wants to bring a Blender model into Unity. Does NOT drive Blender's own generators; BlenderMCP owns how the model got into Blender. | ||
| --- | ||
|
|
||
| # Blender → Unity Model Handoff | ||
|
|
||
| Bring whatever model is currently in Blender into the open Unity scene. The seam is the | ||
| local filesystem: Blender exports a file, Unity imports it. The two servers never talk | ||
| directly. | ||
|
|
||
| ## Preconditions | ||
| - Both `mcp__blender__*` tools and MCP for Unity tools are connected. | ||
| - A model exists in the Blender scene (confirm with `mcp__blender__get_scene_info` / | ||
| `mcp__blender__get_object_info`). If empty, stop and tell the user — this skill does not generate models. | ||
|
|
||
| ## Steps | ||
| 1. **Resolve the Unity project path.** Read `mcpforunity://editor/state` for the project | ||
| root (the editor dataPath's parent). Decide the export format: **FBX by default** | ||
| (built-in importer, zero extra dependencies). Use glTF/.glb only if glTFast is installed | ||
| and PBR fidelity matters. | ||
| 2. **Export from Blender to a temp path** via `mcp__blender__execute_blender_code`: | ||
| ```python | ||
| import bpy, os, tempfile | ||
| out = os.path.join(tempfile.gettempdir(), "blender_to_unity.fbx") | ||
| # Export the selection if any, else the whole scene: | ||
| bpy.ops.export_scene.fbx(filepath=out, use_selection=bool(bpy.context.selected_objects), | ||
| apply_unit_scale=True, bake_space_transform=True) | ||
| print(out) | ||
| ``` | ||
| (glTF branch: `bpy.ops.export_scene.gltf(filepath=out_glb, export_format='GLB')`.) | ||
| 3. **Import into Unity** with `import_model_file`: | ||
| `import_model_file(source_path=<temp path>, name=<asset name>, target_size=<final size in meters>)`. | ||
| It returns `{ asset_path, asset_guid }`. Pass `target_size` as the intended final size, but treat | ||
| it only as a hint: it rescales at import solely when the project's **Auto-normalize** pref is on, | ||
| and even then is unreliable for Blender FBX (see the **Scale** note). Step 4 does the reliable | ||
| normalization. | ||
| 4. **Place it in the scene, normalized to size.** Ensure the scene has a camera + directional | ||
| light (`manage_scene` / `manage_gameobject`). Instantiate the model at the chosen position via | ||
| `manage_gameobject(action="create", prefab_path=<asset_path>, name=<asset name>, position=[x,y,z])`. | ||
| Then normalize its size deterministically — Blender FBX commonly imports ~100× too large — by | ||
| measuring the placed model's world bounds and scaling so its largest dimension equals your target | ||
| size. Run via `execute_code` (substitute your object name and target meters): | ||
| ```csharp | ||
| var go = GameObject.Find("<asset name>"); | ||
| var rs = go.GetComponentsInChildren<Renderer>(); | ||
| var b = rs[0].bounds; for (int i = 1; i < rs.Length; i++) b.Encapsulate(rs[i].bounds); | ||
| float maxDim = Mathf.Max(b.size.x, Mathf.Max(b.size.y, b.size.z)); | ||
| float target = 2f; // intended size in meters | ||
| if (maxDim > 0.0001f) go.transform.localScale *= target / maxDim; | ||
| ``` | ||
| 5. **Verify** with `manage_camera(action="screenshot", include_image=true)` and report the | ||
| asset path + a screenshot. | ||
|
|
||
| ## Notes | ||
| - **Scale.** Models from Blender almost always arrive at the wrong scale — its FBX unit handling | ||
| makes them land ~100× too large in Unity. `import_model_file`'s `target_size` only rescales at | ||
| import when the project's Auto-normalize pref is enabled, and its importer-level normalization is | ||
| unreliable for Blender FBX (it can over- or under-shoot, e.g. a `target_size=2` model measured 200 m). | ||
| The robust fix is the Step 4 measure-bounds-then-set-`localScale` routine, which hits the target | ||
| size deterministically regardless of the import scale or the Auto-normalize pref. | ||
| - FBX is the default because glTFast is optional in MCP for Unity. If the import errors with | ||
| "GLB import requires glTFast", re-export as FBX (or install glTFast from the Dependencies tab). | ||
| - Keep one model per handoff; for batches, repeat the loop with distinct names. | ||
| - This skill never sends API keys or file bytes over the MCP bridge — Unity reads the file from disk. | ||
| - `import_model_file` copies only the single source file; for multi-file exports (a text `.gltf` with an external `.bin`, or an `.obj` with a sibling `.mtl`/textures), zip them first and pass the `.zip` — a bare `.gltf`/`.obj` will lose its sidecars. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Manual Verify — blender-to-unity | ||
|
|
||
| Run against a live Blender (BlenderMCP) + Unity (MCP for Unity) pair. | ||
|
|
||
| - [ ] A cube/model exists in Blender (`get_scene_info` shows it). | ||
| - [ ] FBX export writes a non-empty file to the temp path (printed path exists, size > 0). | ||
| - [ ] `import_model_file` returns `success: true` with `asset_path` under `Assets/` and a non-empty `asset_guid`. | ||
| - [ ] The imported model appears in the Project window at `asset_path`. | ||
| - [ ] The model is instantiated in the open scene and visible in a `manage_camera` screenshot. | ||
| - [ ] Scale: after the Step 4 measure-bounds-then-`localScale` routine, the placed model's largest | ||
| world dimension ≈ the target size (Blender FBX imports ~100× too large until normalized). | ||
| - [ ] glTF path: with glTFast installed, a `.glb` export imports successfully; without it, | ||
| the error names glTFast/the Dependencies tab (and FBX still works). | ||
| - [ ] No API keys or file bytes appear in any bridge payload (handoff is filesystem-only). |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| using System; | ||
| using System.IO; | ||
| using UnityEngine; | ||
|
|
||
| namespace MCPForUnity.Editor.Helpers | ||
| { | ||
| /// <summary> | ||
| /// Project-path conversions shared by the asset-gen import/write code: project-relative | ||
| /// ("Assets/...") ↔ absolute on-disk paths, with forward-slash normalization for | ||
| /// cross-platform consistency. | ||
| /// </summary> | ||
| public static class AssetGenPaths | ||
| { | ||
| /// <summary>Resolve a project-relative ("Assets/...") path to an absolute, forward-slashed path.</summary> | ||
| public static string ToAbsolute(string projectRelative) | ||
| { | ||
| if (string.IsNullOrWhiteSpace(projectRelative)) return projectRelative; | ||
| string p = projectRelative.Replace('\\', '/'); | ||
| string abs = Path.IsPathRooted(p) ? p : Path.Combine(ProjectRoot(), p); | ||
| return Path.GetFullPath(abs).Replace('\\', '/'); | ||
| } | ||
|
|
||
| /// <summary>Convert an absolute (or already-relative) path to a project-relative ("Assets/...") path.</summary> | ||
| public static string ToProjectRelative(string path) | ||
| { | ||
| if (TryGetAssetsRelativePath(path, out string rel)) return rel; | ||
| return path?.Replace('\\', '/'); | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// Normalize an absolute or "Assets/..." path and verify it resolves inside the project's | ||
| /// Assets directory. Rejects traversal such as "Assets/../ProjectSettings". | ||
| /// </summary> | ||
| public static bool TryGetAssetsRelativePath(string path, out string projectRelative) | ||
| { | ||
| projectRelative = null; | ||
| if (string.IsNullOrWhiteSpace(path)) return false; | ||
|
|
||
| try | ||
| { | ||
| string p = path.Replace('\\', '/'); | ||
| string abs; | ||
| if (p == "Assets" || p.StartsWith("Assets/", StringComparison.Ordinal)) | ||
| { | ||
| abs = Path.Combine(ProjectRoot(), p); | ||
| } | ||
| else if (Path.IsPathRooted(p)) | ||
| { | ||
| abs = p; | ||
| } | ||
| else | ||
| { | ||
| return false; | ||
| } | ||
|
|
||
| string full = Path.GetFullPath(abs).Replace('\\', '/'); | ||
| string dataPath = Path.GetFullPath(Application.dataPath).Replace('\\', '/').TrimEnd('/'); | ||
| if (string.Equals(full, dataPath, StringComparison.Ordinal)) | ||
| { | ||
| projectRelative = "Assets"; | ||
| return true; | ||
| } | ||
|
|
||
| string prefix = dataPath + "/"; | ||
| if (!full.StartsWith(prefix, StringComparison.Ordinal)) return false; | ||
|
|
||
| projectRelative = "Assets/" + full.Substring(prefix.Length); | ||
| return true; | ||
| } | ||
| catch | ||
| { | ||
| return false; | ||
| } | ||
| } | ||
|
|
||
| /// <summary>Normalize an Assets folder path, trimming trailing slashes.</summary> | ||
| public static bool TryGetAssetsFolder(string path, out string projectRelative) | ||
| { | ||
| if (!TryGetAssetsRelativePath(path, out projectRelative)) return false; | ||
| projectRelative = projectRelative.TrimEnd('/'); | ||
| return true; | ||
| } | ||
|
|
||
| private static string ProjectRoot() | ||
| { | ||
| string dataPath = Path.GetFullPath(Application.dataPath).Replace('\\', '/'); | ||
| return dataPath.Substring(0, dataPath.Length - "Assets".Length); | ||
| } | ||
| } | ||
| } | ||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| using MCPForUnity.Editor.Constants; | ||
| using UnityEditor; | ||
|
|
||
| namespace MCPForUnity.Editor.Helpers | ||
| { | ||
| /// <summary> | ||
| /// Per-user, NON-SECRET configuration for the AI Asset Generation feature. | ||
| /// | ||
| /// Provider API keys are never stored here — they live in the OS secure store | ||
| /// (<see cref="MCPForUnity.Editor.Security.SecureKeyStore"/>). Only UI/behavior | ||
| /// preferences (selected provider, default format, output folder, normalize toggle, | ||
| /// per-provider enable flags) are kept in EditorPrefs. | ||
| /// </summary> | ||
| public static class AssetGenPrefs | ||
| { | ||
| public const string DefaultOutputRoot = "Assets/Generated"; | ||
| public const string DefaultModelProvider = "tripo"; | ||
| public const string DefaultImageProvider = "fal"; | ||
| public const string DefaultFormatValue = "glb"; | ||
|
|
||
| public static string ModelProvider | ||
| { | ||
| get => EditorPrefs.GetString(EditorPrefKeys.AssetGenSelectedModelProvider, DefaultModelProvider); | ||
| set => SetOrDelete(EditorPrefKeys.AssetGenSelectedModelProvider, value); | ||
| } | ||
|
|
||
| public static string ImageProvider | ||
| { | ||
| get => EditorPrefs.GetString(EditorPrefKeys.AssetGenSelectedImageProvider, DefaultImageProvider); | ||
| set => SetOrDelete(EditorPrefKeys.AssetGenSelectedImageProvider, value); | ||
| } | ||
|
|
||
| public static string DefaultFormat | ||
| { | ||
| get => EditorPrefs.GetString(EditorPrefKeys.AssetGenDefaultFormat, DefaultFormatValue); | ||
| set => SetOrDelete(EditorPrefKeys.AssetGenDefaultFormat, value); | ||
| } | ||
|
|
||
| /// <summary>Project-relative root under which generated assets are written. Defaults to Assets/Generated.</summary> | ||
| public static string OutputRoot | ||
| { | ||
| get | ||
| { | ||
| string v = EditorPrefs.GetString(EditorPrefKeys.AssetGenOutputRoot, string.Empty); | ||
| return string.IsNullOrWhiteSpace(v) ? DefaultOutputRoot : v; | ||
| } | ||
| set => SetOrDelete(EditorPrefKeys.AssetGenOutputRoot, value); | ||
| } | ||
|
|
||
| /// <summary>When true, imported models are uniformly scaled to a target size on import.</summary> | ||
| public static bool AutoNormalize | ||
| { | ||
| get => EditorPrefs.GetBool(EditorPrefKeys.AssetGenAutoNormalize, true); | ||
| set => EditorPrefs.SetBool(EditorPrefKeys.AssetGenAutoNormalize, value); | ||
| } | ||
|
|
||
| public static bool IsProviderEnabled(string providerId) => | ||
| !string.IsNullOrEmpty(providerId) | ||
| && EditorPrefs.GetBool(EditorPrefKeys.AssetGenProviderEnabledPrefix + providerId, false); | ||
|
|
||
| public static void SetProviderEnabled(string providerId, bool enabled) | ||
| { | ||
| if (string.IsNullOrEmpty(providerId)) return; | ||
| EditorPrefs.SetBool(EditorPrefKeys.AssetGenProviderEnabledPrefix + providerId, enabled); | ||
| } | ||
|
|
||
| private static void SetOrDelete(string key, string value) | ||
| { | ||
| if (string.IsNullOrWhiteSpace(value)) EditorPrefs.DeleteKey(key); | ||
| else EditorPrefs.SetString(key, value.Trim()); | ||
| } | ||
| } | ||
| } |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.IO; | ||
| using UnityEngine; | ||
|
|
||
| namespace MCPForUnity.Editor.Helpers | ||
| { | ||
| /// <summary> | ||
| /// Best-effort detection of a locally installed Blender application, for the Asset Gen tab's | ||
| /// "Blender → Unity handoff" hint. This finds the Blender APP only — it cannot tell whether the | ||
| /// BlenderMCP server is configured in the user's AI client (that lives outside Unity). | ||
| /// </summary> | ||
| internal static class BlenderDetection | ||
| { | ||
| /// <summary>True if a Blender executable is found in a well-known location or on PATH.</summary> | ||
| public static bool IsInstalled() | ||
| { | ||
| try { return DetectIn(CandidatePaths(), File.Exists); } | ||
| catch { return false; } | ||
| } | ||
|
|
||
| /// <summary>Pure core: true if <paramref name="exists"/> reports any candidate present. Testable.</summary> | ||
| internal static bool DetectIn(IEnumerable<string> candidates, Func<string, bool> exists) | ||
| { | ||
| if (candidates == null || exists == null) return false; | ||
| foreach (string c in candidates) | ||
| if (!string.IsNullOrEmpty(c) && exists(c)) return true; | ||
| return false; | ||
| } | ||
|
|
||
| /// <summary>Well-known Blender executable paths for the current platform, plus PATH entries.</summary> | ||
| internal static IEnumerable<string> CandidatePaths() | ||
| { | ||
| var list = new List<string>(); | ||
| bool win = Application.platform == RuntimePlatform.WindowsEditor; | ||
| string exeName = win ? "blender.exe" : "blender"; | ||
|
|
||
| // PATH entries: <dir>/blender(.exe) | ||
| string pathVar = Environment.GetEnvironmentVariable("PATH") ?? string.Empty; | ||
| foreach (string dir in pathVar.Split(win ? ';' : ':')) | ||
| if (!string.IsNullOrWhiteSpace(dir)) list.Add(Path.Combine(dir.Trim(), exeName)); | ||
|
|
||
| switch (Application.platform) | ||
| { | ||
| case RuntimePlatform.OSXEditor: | ||
| list.Add("/Applications/Blender.app/Contents/MacOS/Blender"); | ||
| string home = Environment.GetFolderPath(Environment.SpecialFolder.UserProfile); | ||
| if (!string.IsNullOrEmpty(home)) | ||
| list.Add(Path.Combine(home, "Applications/Blender.app/Contents/MacOS/Blender")); | ||
| break; | ||
| case RuntimePlatform.WindowsEditor: | ||
| foreach (string pf in new[] | ||
| { | ||
| Environment.GetEnvironmentVariable("ProgramFiles"), | ||
| Environment.GetEnvironmentVariable("ProgramFiles(x86)") | ||
| }) | ||
| { | ||
| if (string.IsNullOrEmpty(pf)) continue; | ||
| string foundation = Path.Combine(pf, "Blender Foundation"); | ||
| // Blender installs under a version subdir (Blender X.Y); enumerate them. | ||
| try | ||
| { | ||
| if (Directory.Exists(foundation)) | ||
| foreach (string d in Directory.GetDirectories(foundation)) | ||
| list.Add(Path.Combine(d, "blender.exe")); | ||
| } | ||
| catch { /* unreadable dir; ignore */ } | ||
| } | ||
| break; | ||
| case RuntimePlatform.LinuxEditor: | ||
| list.Add("/usr/bin/blender"); | ||
| list.Add("/usr/local/bin/blender"); | ||
| list.Add("/snap/bin/blender"); | ||
| list.Add("/var/lib/flatpak/exports/bin/org.blender.Blender"); | ||
| break; | ||
| } | ||
| return list; | ||
| } | ||
| } | ||
| } |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Reject non-
Assets/...inputs in the conversion helpers.ToAbsolutecurrently resolves any rooted path (and evenAssets/../...) without re-checking containment, whileToProjectRelativefalls back to returning non-project paths unchanged. That breaks the helper’s documentedAssets-only contract and weakens the traversal protection this layer is supposed to provide for generated imports/outputs.Proposed fix
public static string ToAbsolute(string projectRelative) { - if (string.IsNullOrWhiteSpace(projectRelative)) return projectRelative; - string p = projectRelative.Replace('\\', '/'); - string abs = Path.IsPathRooted(p) ? p : Path.Combine(ProjectRoot(), p); - return Path.GetFullPath(abs).Replace('\\', '/'); + if (!TryGetAssetsRelativePath(projectRelative, out string rel)) + throw new ArgumentException("Path must resolve inside the project's Assets folder.", nameof(projectRelative)); + + return Path.GetFullPath(Path.Combine(ProjectRoot(), rel)).Replace('\\', '/'); } /// <summary>Convert an absolute (or already-relative) path to a project-relative ("Assets/...") path.</summary> public static string ToProjectRelative(string path) { - if (TryGetAssetsRelativePath(path, out string rel)) return rel; - return path?.Replace('\\', '/'); + if (!TryGetAssetsRelativePath(path, out string rel)) + throw new ArgumentException("Path must resolve inside the project's Assets folder.", nameof(path)); + + return rel; }📝 Committable suggestion
🤖 Prompt for AI Agents