Controlling Pose and Composition with ControlNet

Text prompts describe desired content but do not precisely place every body part, object, edge, or depth relationship. ControlNet adds a structural reference that influences the generation. It can guide pose, depth, outlines, line art, scribbles, or segmented regions while the positive prompt controls the new visual content.

ControlNet is applied to conditioning, not to the latent image. The Apply ControlNet node sits between the positive prompt encoder and the KSampler:

The negative prompt remains connected to the KSampler's Negative input.

A ControlNet Loader supplies the ControlNet model. ControlNet Union supports the range of controls used here instead of requiring a separate ControlNet model for every type. It is approximately 2.5 GB and must be installed through Model Manager, followed by Refresh.

The reference image must be converted into the appropriate control representation. The ArtVenture custom-node package supplies an AV ControlNet Preprocessor node with multiple preprocessors in one interface. The input path is:

Configure the preprocessor for the model family, such as SDXL, and for the selected control type. Set its resolution to match the intended generation scale. A Preview Image node can be attached to the preprocessor output to inspect the actual control image before it influences the final generation.

Available control types include:

OpenPose: Converts a person into a pose skeleton so generated characters follow the body arrangement.

Depth: Creates a depth map that can preserve spatial relationships, object placement, and relative distance.

Canny: Reduces an image to prominent edges so a new generation follows its major shapes.

Line Art: Uses drawn outlines as structural guidance.

Anime Line Art: A line-art option optimized for anime characters, including pose and clothing outlines.

Scribble: Uses simple drawn lines as a loose structural guide.

Segmentation: Divides the image into regions so complex scenes with multiple people and objects can preserve their approximate locations.

Strength controls the influence of the processed reference. A value of 0 means ControlNet has no effect. A value of 1 applies maximum influence. The pose example starts at 0.8; increasing it to 1 makes the final image follow the pose more completely.

ControlNet separates visual content from spatial control. A prompt can request “princess, arctic tundra, snowing,” while an OpenPose reference determines the character's body position. A prompt can request “mountains, sunset,” while a Canny reference preserves the outline of a different mountain scene.

This is particularly important in complex images with multiple characters or objects. Segmentation and depth references can guide where elements appear, reducing dependence on the prompt's ability to describe exact placement.

1. Begin with the standard text-to-image graph.
2. Install ControlNet Union through Model Manager and refresh.
3. Install the ArtVenture custom nodes and restart ComfyUI.
4. Disconnect the positive prompt from the KSampler.
5. Add Apply ControlNet.
6. Connect the positive prompt encoder to Apply ControlNet.
7. Connect Apply ControlNet's conditioning output to the KSampler's Positive input.
8. Add ControlNet Loader and select the installed ControlNet Union model.
9. Add Load Image and upload a photograph containing the desired pose.
10. Add AV ControlNet Preprocessor.
11. Select SDXL, OpenPose, and a resolution such as 768 for the faster example.
12. Set the generated image dimensions to the same scale.
13. Connect the preprocessed image to Apply ControlNet.
14. Set strength to 0.8.
15. Preview the preprocessed pose image to confirm that it captured the intended skeleton.
16. Enter a new positive prompt and queue the workflow.
17. Increase strength toward 1 when the generated pose should follow the reference more strictly.

For a landscape transformation, upload a mountain image, choose Canny, inspect the edge map, and prompt for mountains at sunset. The resulting image follows the reference mountain shapes to some degree while changing the lighting and style.

ControlNet guides rather than guarantees. The generated image follows the reference to an extent determined by the preprocessor, strength, model, and prompt.

Maximum strength gives the reference more control but reduces freedom. Zero strength disables the structural guidance entirely.

The correct preprocessor depends on the intended constraint. OpenPose is suitable for body arrangement, Canny and line art for outlines, depth for spatial structure, and segmentation for region placement. Selecting a preprocessor that does not represent the desired information weakens the result.

The workflow adds several nodes and connections, making layout and labeling more important. The first use of a preprocessor may also trigger an additional model download.

Using 768 rather than SDXL's preferred 1024 scale improves speed but compromises the preferred generation scale.

ControlNet adds structural conditioning before sampling. Choose the preprocessor that captures the feature you need to preserve, inspect its output, and use strength to balance reference control against generative freedom.