Overview ControlNet is applied to conditioning, not to the latent image. The Apply ControlNet node sits between the positive prompt encoder and the KSampler: