sd.cpp-webui_inpainting_with_sam

Introduction

This is a short guide explaining the changes i made to sd.cpp webui and how the img2img generation pipeline works with my modifications. The repository can be cloned from https://github.com/vito-turco/sd.cpp-webui_inpainting_with_sam.

Inpainting and image-to-image (img2img) pipelines empower generative AI models —driven by underlying UNet or transformer architectures— to modify existing visuals rather than generating them from scratch. \ Standard img2img introduces noise to an input image and denoises it under the guidance of a text prompt to alter overall styles or compositions. In particular, specialized models like InstructPix2Pix allow direct instruction-based transformations (e.g., “turn the day scene into night”) while preserving the core layout.

But what if i want to restrain the editing to specific zones of the image? Here’s where masking comes into play.
Masking serves as the cornerstone of precision editing, enabling users to isolate specific pixel zones. Its primary applications include:

• Object Insertion & Addition: Seamlessly blending new elements into a targeted zone using surrounding context.
• Object Removal & Clean-up: Erasing unwanted blemishes, watermarks, or background figures so the model hallucinates a natural fill.
• Outpainting: Extending canvas borders outward to expand an image’s framing.
• Targeted Retouching: Fixing distorted features (like hands or eyes) or specific textures without risking changes to the pristine parts of the image.

By combining text prompts with precise masks, creators retain exact artistic control over local modifications while protecting the rest of the visual composition.

Returning to the sd.cpp web UI—an interface I’ve always favored over others, like Automatic111, since it uses sd.cpp as backend instead of utilizing PyTorch as its core deep learning framework. However, I found it lacked the ability to leverage segmentation models for precise masking, forcing users to create masks externally and then upload them.

To address this, I decided to implement a dedicated section for generating masks using Meta’s SAM3, while also adding the option to refine these masks—or create them from scratch—using a brush tool. I’ve also kept the original interface, which allows for separate uploads of the input image and the mask, giving users the flexibility to choose their preferred workflow. In both pipelines, it is possible to proceed without loading any mask at all, enabling inpainting across the entire image—useful, for instance, when applying a style change.

Let’s dive into the new components and explain how them work, before proceeding to the examples.

Case 1: Targeted Masking When modifying specific regions of the input image, the workflow provides two masking paths:

• Option A: SAM Auto-Segmentation Upload Input Image: Load the target image into the workplace canvas.

  Click “Generate SAM Masks”: Run the SAM model to automatically segment objects and structural boundaries within the image.

  Enter Text Prompt: Provide a prompt describing the desired addition, replacement, or modification.

  Select & Edit Masks:

  Review the generated candidate masks and select which ones to keep.

  Refine or extend the selected SAM masks using the manual brush tool.

  Invert Mask (Optional): Toggle the invert option if you wish to apply edits to everything except the selected area (e.g., background replacement while keeping the foreground subject untouched).

  Click “Extract Final Mask” to merge all selected/edited selections into a unified binary mask.

• Option B: Manual Brush Masking Upload Input Image: Load the target image onto the canvas.

  Draw Mask: Directly paint over the target region using the manual brush tool.

  Mask Post-Processing & Canvas Setup (Both Masking Paths)

  Invert Mask (Optional): Toggle the invert option if you wish to apply edits to everything except the selected area (e.g., background replacement while keeping the foreground subject untouched).

Set Dimensions & Click “Resize and Show”: Define the output aspect ratio/resolution and render a canvas preview displaying the final image framing and active mask placement.

Generate: Click Generate to run the diffusion model (inpainting) and produce the final edited image.

Case 2: Full-Image Transformation (Standard Img2Img) If no localized masking is required:

Upload Input Image: Load the target image into the editor.

Bypass Masking: Skip drawing a manual mask and do not trigger “Generate SAM Masks.”

Set Dimensions & Click “Resize and Show”: Set target resolution/dimensions and generate the preview bounding box over the full input image.

Generate: Click Generate to apply the text prompt and denoise across the entire image using standard img2img.

Examples

This section demonstrates how combining Segment Anything Model (SAM) auto-segmentation, manual brush refinement, and prompt-guided inpainting enables precise local modifications. In all the examples i used Realistic Vision V5.1 Inpainting, with safetensors downlaoded from HF, for generation.

---

Subject Replacement (Tabby Cat to Black Cat)

This example shows how to isolate a foreground subject using SAM, refine the boundaries, and replace the animal while keeping the surrounding environment intact.

Workflow Steps:

1. Upload the input image of the tabby cat.
2. Click Generate SAM Masks and select the cat’s full body.
3. Enter the prompt: a black cat, click Resize and Show, and hit Generate.

Background Replacement via Inverse Masking (Cat at the Beach)

By utilizing mask inversion, the subject is safely protected while the entire background environment is transformed.

Workflow Steps:

1. Load the original cat image.
2. Generate and extract the SAM mask for the cat.
3. Toggle Invert Mask so the background becomes active for editing and the cat is masked out.
4. Enter the prompt: a sunny beach with sand and ocean, resize, and Generate.

Object Removal (License Plate and Selfie Cleanup)

Inpainting can also be used for clean-up tasks, removing unwanted text/objects or background distractions.

Workflow Steps:

1. For the car: Generate and extract the SAM mask for the license plate and prompt clean car bumper.
2. For the selfie: Generate SAM masks to select the background figures, correct the mask with brush, merge them via Extract Final Mask, and prompt blurred background, a park with a path.

Localized Accessory Modification (Glasses to Sunglasses)

Targeting micro-regions (like eyewear) allows seamless wardrobe or accessory adjustments on portraits.

Workflow Steps:

1. Upload the selfie image.
2. Use SAM to isolate the eye/glasses area.
3. Enter the prompt: sunglasses, adjust sizing, and click Generate.

Conclusions

The integration of interactive masking frameworks like Segment Anything Model (SAM) with generative diffusion backbones highlights the value of combining automated segmentation with targeted image-to-image pipeline operations.

Key takeaways from this workflow include:

• Precision and Control: SAM drastically lowers the friction of accurate region selection, enabling granular modifications—such as subject swapping, accessory editing, or photobomber removal—without disturbing pristine surrounding context.
• Flexibility via Post-Processing: Features such as mask inversion (Invert Mask) effectively double the capabilities of a single mask extraction, allowing seamless transitions between localized subject editing and global background replacement.
• Versatility Across Scales: Whether applied to subtle retouching tasks (e.g., swapping eyeglasses for sunglasses) or structural canvas shifts (outpainting and full-background replacement), combining prompt guidance with precise spatial constraints provides an elastic and effective approach for AI-assisted image editing.

Future development

• Multiclass masking with SAM3: introduce the possibility to add instances of different classes to the final binary mask.
• Introduction of SAM3.cpp backend for consistency with the C++ backend used in the original repo, and to leverage C++’s inference speed for mask generation as well.