This workflow is adapted to change very small parts of the image, and still get good results in terms of the details and the composite of the new pixels in the existing image
Here is used:
ControlNet
Segment Anything
IP-Adapter
Image Composite masked
masks
and more..