Repository files navigation

Create stunning AI-generated skybox assets withinUnity for use as skyboxes, HDRI style lighting, and world meshes. Speed up your prototyping or use in your existing creative workflow!

In order to use this package you need to provide an API key from Blockade Labs in the API section.Get one athttps://www.skyboxai.net/plug-in-api.

Unity 2020.3 LTS or higher.

• Add this package to your assets from theUnity Asset Store
• Go toWindow > Package Manager
• Change the project scope toMy Assets
• FindBlockade Labs SDK and clickInstall

• CLI:
  • openupm add com.blockadelabs.sdk
• Registry:
  • Name:OpenUPM
  • URL:https://package.openupm.com
  • Scope:com.blockadelabs

• Go toWindow > Package Manager
• + > Add package from git URL...
• Enterhttps://github.com/Blockade-Games/BlockadeLabs-SDK-Unity.git

The Blockade SDK can be used standalone or optionally together with a Pusher websockets package. If installed, the Pusher package will use websockets to listen for any changes in the Asset Generation Process. This changes the API request update strategy from Polling to Callbacks which will reduce the chances of being rate limited by the API.

You can learn more about the Pusher packagehere.

The Pusher library requires .NET Framework runtime:

• SetEdit > Project Settings > Player > Api Compatibility Level to.NET Framework.

Option 1: Install the Pusher package with git:

• Go toWindow > Package Manager
• + > Add package from git URL...
• Enterhttps://github.com/pusher/pusher-websocket-unity.git#upm

Option 2: Install the Pusher package with OpenUPM:

• CLI:
  • openupm add com.pusher.pusherwebsocketunity
• Registry:
  • Add scope:com.pusher.pusherwebsocketunity

Assembly 'Packages/com.pusher.pusherwebsocketunity/Packages/PusherClient.2.1.0/lib/net472/PusherClient.dll' will not be loaded due to errors: PusherClient references strong named Newtonsoft.Json Assembly references: 12.0.0.0 Found in project: 13.0.0.0.

To resolve the issue go toEdit > Project Settings > Player > Other Settings > Configuration > Assembly Version Validation and disableVersion Validation.

Refer to the changelog filehere.

• Getting Started
• Authentication
• Generating Skyboxes in Editor
  • Using Skyboxes in your scene
  • 3D Mesh Creator
• Generating Skyboxes at Runtime
  • BlockadeLabs API Proxy
  • Skyboxes API
    • Get Skybox Styles
    • Get Skybox Style Families
    • Get Skybox Export Options
    • Generate Skybox
    • Get Skybox by Id
    • Request Skybox Export
    • Delete Skybox by Id
    • Get Skybox History
    • Cancel Skybox Generation
    • Cancel All Pending Skybox Generations

After importing the package an onboarding window should pop up with directions to open the SkyboxAI scene.

If the onboarding window does not open for you, you can access the SkyboxAI scene by the following:

• menu:Tools > Blockade Labs > Open SkyboxAI Scene
• open:Packages/Blockade Labs SDK/Scenes/SkyboxScene

The scene contains two notable gameObjects:

• Blockade Labs Skybox Generator generates skybox textures and materials.
• Blockade Labs Skybox Mesh generates and configures a mesh which combines the skybox with a generated depth map.

There are 4 ways to provide your API keys, in order of precedence:

1. Pass keys directly with constructor⚠️
2. Unity Scriptable Object⚠️
3. Load key from configuration file
4. Use System Environment Variables

varapi=newBlockadeLabsClient("yourApiKey");

Or create aBlockadeLabsAuthentication object manually

varapi=newBlockadeLabsClient(newBlockadeLabsAuthentication("yourApiKey"));

You can save the key directly into a scriptable object that is located in theAssets/Resources folder.

You can create a new one by using the context menu of the project pane and creating a newBlockadeLabsConfiguration scriptable object.

Create > BlockadeLabs > BlockadeLabsConfiguration

Attempts to load api keys from a configuration file, by default.blockadelabs in the current directory, optionally traversing up the directory tree or in the user's home directory.

To create a configuration file, create a new text file named.blockadelabs and containing the line:

{"apiKey":"yourApiKey",}

You can also load the file directly with known path by calling a static method in Authentication:

varapi=newBlockadeLabsClient(newBlockadeLabsAuthentication().LoadFromDirectory("your/path/to/.blockadelabs"));;

Use your system's environment variables specify an api key to use.

• UseBLOCKADELABS_API_KEY for your api key.

varapi=newBlockadeLabsClient(newBlockadeLabsAuthentication().LoadFromEnvironment());

You can generate a new skybox that will replace the existing one by following these steps.

1. Select theBlockade Labs Skybox Generator gameObject.
2. Add your Blockade Labs'API key in the designated inspector field.
3. Click theApply button.
4. After the plugin is successfully initialized, some additional fields will become available.
5. Select the desired style.
6. Fill in thePrompt field.
7. Click theGenerate Skybox Button.
8. Generated textures and materials are placed in theAssets/Blockade Labs SDK for use in your project.
9. You should see your new skybox in the game view. You can also clickMove Scene Camera to Skybox to see the skybox in the scene view.

For detailed information on how skyboxes work in Unity, seeusing skyboxes.

1. Go toWindow > Rendering > Lighting.
2. Go to theEnvironment tab.
3. Drag the generatedskybox material into the theSkybox Material field.
4. If you want to use the skybox as background lighting in your scene, ensureEnvironment Lighting Source is set toSkybox, then clickGenerate Lighting.

1. Add a global volume to your scene:GameObject > Volume > Global Volume.
2. Drag the generatedHDRP volume profile in to theProfile field.

If you are building a 'Fully Immersive' visionOS app, then the steps are the same as the render pipeline which you are using.

If you are building a 'PolySpatial Mixed Reality' app, then the scene sky is not rendered. However, if you useMesh Creator to create skybox mesh, then you can render it in mixed reality. Since PolySpatial only supports Shader Graph materials, you need to change the type of the generated depth material:

1. Install theShader Graph package from the Unity Package Manager.
2. Go to the "skybox depth material" you want to use and duplicate it to create a material for visionOS.
3. Change the material type fromBlockade Skybox Depth toBlock Skybox Depth Shader Graph.

If you want theBlockade Labs Skybox Generator component to always generate shader graph materials, update theDepth Material field.

TheBlockade Labs Skybox Generator component generates a color texture and a depth texture, which are assigned to the skybox material.

TheBlockade Labs Skybox Mesh component generates a Tetrahedron mesh to apply this material. You can configure theMesh Density andDepth Scale fields. The mesh and material will be configured to apply the generated depth map to deform the mesh.

Try zooming in and out with the scroll wheel in play mode to see the effect of the depth scale!

If you want to use the generated mesh in your own scene, clickSave Prefab, then drag the new prefab into your scene.

If you're interested in having runtime generated content generated on demand, you can use the public API surface directly.

Using either theBlockadeLabs-SDK-DotNet orBlockadeLabs-SDK-Unity packages directly in your front-end app may expose your API keys and other sensitive information. To mitigate this risk, it is recommended to set up an intermediate API that makes requests to BlockadeLabs on behalf of your front-end app. This library can be utilized for both front-end and intermediary host configurations, ensuring secure communication with the BlockadeLabs API.

In the front end example, you will need to securely authenticate your users using your preferred OAuth provider. Once the user is authenticated, exchange your custom auth token with your API key on the backend.

Follow these steps:

1. Setup a new project using either theBlockadeLabs-SDK-DotNet orBlockadeLabs-SDK-Unity packages.
2. Authenticate users with your OAuth provider.
3. After successful authentication, create a newBlockadeLabsAuthentication object and pass in the custom token as your apiKey.
4. Create a newBlockadeLabsClientSettings object and specify the domain where your intermediate API is located.
5. Pass your newauth andsettings objects to theBlockadeLabsClient constructor when you create the client instance.

Here's an example of how to set up the front end:

varauthToken=awaitLoginAsync();varauth=newBlockadeLabsAuthentication(authToken);varsettings=newBlockadeLabsClientSettings(domain:"api.your-custom-domain.com");usingvarapi=newBlockadeLabsClient(auth,settings);

This setup allows your front end application to securely communicate with your backend that will be using the BlockadeLabs-SDK-DotNet-Proxy, which then forwards requests to the BlockadeLabs API. This ensures that your BlockadeLabs API keys and other sensitive information remain secure throughout the process.

In this example, we demonstrate how to set up and useBlockadeLabsProxy in a new ASP.NET Core web app. The proxy server will handle authentication and forward requests to the BlockadeLabs API, ensuring that your API keys and other sensitive information remain secure.

1. Create a newASP.NET Core minimal web API project.
2. Add the BlockadeLabs-SDK-DotNet nuget package to your project.
   • Powershell install:Install-Package BlockadeLabs-SDK-DotNet-Proxy
   • dotnet:dotnet add package BlockadeLabs-SDK-DotNet-Proxy
   • Manually editing .csproj:<PackageReference Include="BlockadeLabs-SDK-DotNet-Proxy" />
3. Create a new class that inherits fromAbstractAuthenticationFilter and override theValidateAuthenticationAsync method. This will implement theIAuthenticationFilter that you will use to check user session token against your internal server.
4. InProgram.cs, create a new proxy web application by callingBlockadeLabsProxy.CreateWebApplication method, passing your customAuthenticationFilter as a type argument.
5. CreateBlockadeLabsAuthentication as you would normally and load your API key from environment variable.

usingBlockadeLabsSDK;usingBlockadeLabsSDK.Proxy;usingSystem.Security.Authentication;publicpartialclassProgram{privateclassAuthenticationFilter:AbstractAuthenticationFilter{publicoverrideasyncTaskValidateAuthenticationAsync(IHeaderDictionaryrequest){awaitTask.CompletedTask;// remote resource call// You will need to implement your own class to properly test// custom issued tokens you've setup for your end users.if(!request["x-api-key"].ToString().Contains(TestUserToken)){thrownewAuthenticationException("User is not authorized");}}}publicstaticvoidMain(string[]args){varauth=BlockadeLabsAuthentication.LoadFromEnvironment();usingvarblockadeLabsClient=newBlockadeLabsClient(auth);BlockadeLabsProxy.CreateWebApplication<AuthenticationFilter>(args,blockadeLabsClient).Run();}}

Once you have set up your proxy server, your end users can now make authenticated requests to your proxy api instead of directly to the BlockadeLabs API. The proxy server will handle authentication and forward requests to the BlockadeLabs API, ensuring that your API keys and other sensitive information remain secure.

Returns the list of predefined styles that can influence the overall aesthetic of your skybox generation.

varapi=newBlockadeLabsClient();varskyboxStyles=awaitapi.SkyboxEndpoint.GetSkyboxStylesAsync(SkyboxModel.Model3);foreach(varskyboxStyleinskyboxStyles){Debug.Log($"{skyboxStyle.Name}");}

Returns the list of predefined styles that can influence the overall aesthetic of your skybox generation, sorted by style family. This route can be used in order to build a menu of styles sorted by family.

varapi=newBlockadeLabsClient();varskyboxFamilyStyles=awaitBlockadeLabsClient.SkyboxEndpoint.GetSkyboxStyleFamiliesAsync(SkyboxModel.Model3);foreach(varskyboxStyleinskyboxFamilyStyles){Debug.Log($"{skyboxStyle.Name}");}

Returns the list of all available export types.

varapi=newBlockadeLabsClient();varexportOptions=awaitapi.SkyboxEndpoint.GetAllSkyboxExportOptionsAsync();foreach(varexportOptioninexportOptions){Debug.Log($"{exportOption.Id}:{exportOption.Name} |{exportOption.Key}");}varrequest=newSkyboxRequest("mars",enhancePrompt:true);// Generates ALL export options for the skyboxvarskyboxInfo=awaitapi.SkyboxEndpoint.GenerateSkyboxAsync(request,exportOptions);Debug.Log($"Successfully created skybox:{skyboxInfo.Id}");

Generate a skybox.

varapi=newBlockadeLabsClient();varskyboxStyles=awaitBlockadeLabsClient.SkyboxEndpoint.GetSkyboxStylesAsync(SkyboxModel.Model3);varrequest=newSkyboxRequest(skyboxStyles.First(),"mars",enhancePrompt:true);// You can also get progress callbacks when the generation progress has changed/updatedvarprogress=newProgress<SkyboxInfo>(async progress=>{Debug.Log(progress);});varskyboxInfo=awaitapi.SkyboxEndpoint.GenerateSkyboxAsync(request,progressCallback:progress);Debug.Log($"Successfully created skybox:{skyboxInfo.Id}");if(skyboxInfo.TryGetAsset<Texture2D>(SkyboxExportOption.Equirectangular_PNG,outvartexture)){skyboxMaterial.mainTexture=texture;}

Returns the skybox metadata for the given skybox id.

varskyboxId=42;varapi=newBlockadeLabsClient();varskyboxInfo=awaitapi.SkyboxEndpoint.GetSkyboxInfoAsync(skyboxId);Debug.Log($"Skybox:{result.Id}");// Note: If you wish to use the returned skybox info textures you'll need to additionally call await SkyboxInfo.LoadAssetsAsync(); before you can assign them to a material property.awaitskyboxInfo.LoadAssetsAsync();if(skyboxInfo.TryGetAsset<Texture2D>(SkyboxExportOption.Equirectangular_PNG,outvartexture)){skyboxMaterial.mainTexture=texture;}

Exports the skybox with the requested export type.

varskyboxId=42;varapi=newBlockadeLabsClient();varskyboxInfo=awaitapi.SkyboxEndpoint.GetSkyboxInfoAsync(skyboxId);skyboxInfo=awaitapi.SkyboxEndpoint.ExportSkyboxAsync(skyboxInfo,DefaultExportOptions.DepthMap_PNG);awaitskyboxInfo.LoadAssetsAsync();if(skyboxInfo.TryGetAsset<Texture2D>(SkyboxExportOption.DepthMap_PNG,outvartexture)){skyboxMaterial.depthTexture=texture;}

Deletes a skybox by id.

varskyboxId=42;varresult=awaitapi.SkyboxEndpoint.DeleteSkyboxAsync(skybox);// result == true

Gets the previously generated skyboxes.

varhistory=awaitapi.SkyboxEndpoint.GetSkyboxHistoryAsync();Debug.Log($"Found{history.TotalCount} skyboxes");foreach(varskyboxinhistory.Skyboxes){Debug.Log($"{skybox.Id}{skybox.Title} status:{skybox.Status}");}

Cancels a pending skybox generation request by id.

varskyboxId=42;varresult=awaitCancelSkyboxGenerationAsync(skyboxId);// result == true

Cancels ALL pending skybox generation requests.

varresult=awaitapi.SkyboxEndpoint.CancelAllPendingSkyboxGenerationsAsync();Debug.Log(result?"All pending generations successfully cancelled":"No pending generations");