I had the same issue, it turned out the Lora I was using was trained for Z image turbo and not Z image base! Z Image base loras work with Turbo, but not the other way around...
This Klein 9B workflow will let you easily change the clothing on an image based on a reference. Will work will any type of clothing if the reference is laid out properly and at a high enough resolution! Sub version includes multi reference input for more customization.
@Enrique S Same! when I load the workflow it has a VAE named "smallflux2.safetensors" I have tried googling it, but could not find one... and if I try normal flux vae, it gives a grainy output