Unity / Editor tools / Shaders

Procedural cave terrain masks

From environmental prop positions to a texture the terrain shader can sample.

Top and side terrain views beside red and green projected masks, the final RGB mask, and the mask combined with a height map. — Generated mask breakdown Projected channels, a packed RGB texture, and the mask combined with a height map.

Wide Unity editor view of rocky cave terrain, environmental props, platforms, and structures. — Large rocky terrain Wider Unity editor view of the terrain and surrounding props.

Ice terrain shader example

The masking approach can be used with different terrain materials. This example shows it with an ice shader in the Unity editor.

Ice shader example Cropped Unity editor capture.

I created a Unity editor tool to generate masks around selected environmental props for a procedural cave terrain shader. Props could be collected automatically or added manually to a list in the editor. The tool then baked their projected masks into an RGB texture.

My contribution: C# editor tool, custom HLSL functions, and terrain shader
Tools: Unity / Amplify Shader Editor
Collaboration: Under the supervision of the lead technical programmer

01 / The challenge

Generate masks around a large number of props

The levels contained more than 1,000 props. Evaluating all of their positions in a shader loop raised performance concerns, so I used an editor tool to bake projected masks into a texture for the shader to sample.

My contribution

Under the supervision of the lead technical programmer, I developed the C# editor tool, custom HLSL functions for use in Amplify Shader Editor, and the main terrain shader from scratch. The development involved testing different approaches and discarding those that did not meet the project's needs.

02 / The workflow

Collect props, bake projections, sample the mask

Collect the environmental props
Find GameObjects by name, identify objects with a specific script component, or add objects manually to a list in the editor.
Bake the projected masks
Use the collected positions to draw masks into separate RGB channels, with one channel for each projection plane.
Sample in world space
Use custom HLSL functions to calculate texture coordinates from world positions and the selected world bounds, then sample each projection in the shader.

One RGB texture, three projections
Channel	Projection plane	Axes
Red	XZ	X and Z
Green	XY	X and Y
Blue	YZ	Y and Z

Separate mask management from texture drawing

The editor tool used two scripts: a terrain mask manager and a texture helper with static methods. The helper drew soft-edged circular masks into a Texture2D; the manager used these methods to build the projected masks.

Scatter soft masks around each prop

Around each collected point in world space, the editor tool scattered circles and blended them into a mask. Adjusting their number, size, and softness produced different mask patterns around the props.

The editor provided controls for brush and scatter radii, the number of circles, and edge softness. The XZ examples show the resulting patterns. The final texture packed the top, side, and front projections into RGB channels.

Unity editor controls for scatter radius, the number of circles, edge softness, brush size, and world bounds. — Editor settings Brush and scatter controls, followed by generated XZ mask examples.

Black-and-white texture examples showing soft mask clusters, small scattered shapes, and blended circular patterns. — Editor settings Brush and scatter controls, followed by generated XZ mask examples.

Packed RGB terrain mask texture with red, green, and blue patterns overlapping into cyan, magenta, yellow, and white. — Packed RGB texture Top, side, and front projections combined in one texture.

View the C# mask-generation excerpts

Static C# texture helper method accepting a radius, fade power, and channel mask for gradient-circle drawing. — Texture helper Radius, softness, and channel selection for mask drawing.

C# calls drawing masks at XZ, XY, and YZ coordinates into red, green, and blue texture channels. — C# editor tool Drawing the projected masks into the texture.

Explore the manager and helper methods

Editor code structure showing the manager's mask-drawing, prop-matching, path, and preview methods. — Mask manager Collection, drawing, and supporting editor methods.

Texture helper code structure listing mask drawing, coordinate conversion, texture creation, and asset-saving methods. — Texture helper Reusable drawing and texture utilities.

03 / Shader integration

Combine the projections on the cave terrain

I combined the sampled channels with custom math to create a black-and-white mask on the procedural cave terrain. The shader used world-space projections to apply the mask to the changing terrain geometry.

Amplify Shader Editor graph connecting world-position projection functions to RGB mask sampling and blending operations. — Amplify Shader Editor World-space projection and channel blending.

View the world-space HLSL functions

HLSL functions normalizing world positions against minimum and maximum bounds for projected texture coordinates. — Custom HLSL functions Coordinate conversion for world-space sampling.

04 / Final results

One tool, different terrain themes

I used the tool to generate masks for different terrain themes and custom shaders. In procedurally generated caves, the masks could define lava veins or create brighter, glowing areas in ice around nodes and enemy spawn points.

Rocky cave walls with thin orange lava veins and warm light. — 01 / Lava terrain Terrain material examples in procedurally generated caves.

Bright, branching lava patterns across dark rocky cave surfaces. — 01 / Lava terrain Terrain material examples in procedurally generated caves.

Back to Tech Art & Scripting