How to edit 3D models with VoxHammer and prompts

TL;DR

VoxHammer lets you microedit 3D models by targeting specific parts with a mask and applying edits via text prompts or reference images. Compass improves spatial accuracy for image generation by influencing object placement. New tools like USO and VibeVoice enable style transfer and long, expressive TTS respectively, expanding AI video and image workflows.

Transcript

AI never sleeps and this week has been absolutely insane. We have a new video generator by bite dance. Alibaba releases a new version of one which can generate video with audio and I will not be the person you settle for just because you cannot have her. This AI can reconstruct 3D objects that are physically accurate. We have a really good deep fak... Read More

Key Insights

• VoxHammer is a 3D model editor that edits specific parts while keeping the rest of the model intact.
• Edits can be driven by text prompts or by reference images to guide the change.
• Part-aware editing segments the model into meaningful parts before applying changes.
• Compass acts as a spatial guidance layer for image generators to place objects correctly.
• Compass is compatible with multiple open source generators including Flux and Stable Diffusion.
• Compass improves spatial accuracy metrics across benchmarks, enhancing generation consistency.
• USO is a free open source tool for character style transfer and image generation, supporting online demos and local execution.
• USO demonstrates strong performance against existing methods in both deep fake style transfer and image style transfer.
• More videos from this channel:
  • How does Pusa open source video model compare to rivals?
  • How to Run MiniMax H3 Locally in ComfyUI
  • Kim K2 Thinking Open Source AI Review and Demos
• More from this channel: AI Search

Questions & Answers

Q: A real search query question about VoxHammer, what is VoxHammer and how does it work?

VoxHammer is described as a microedit 3D model editor that lets you specify a region of a model with a mask and then apply edits using either a text prompt or a reference image. The key idea is to change only the targeted parts while preserving the rest of the object, enabling precise modifications such as turning a bowl of bread into bread inside the bowl or replacing a sword with a rose. It requires a GPU for running locally and ships with a GitHub repository containing instructions for download and setup.

Q: How does Compass improve image generation compared to using a base model alone?

Compass is presented as a fine-tune style tool that improves the spatial arrangement of objects in generated images. By adding Compass on top of existing image generators like Flux or Stable Diffusion, users can reliably place items where they want, such as a bird below a skateboard or a bear to the right of a truck. Benchmarks show substantial improvements in spatial accuracy, making prompts adhere more closely to intended layouts.

Q: What is USO in the context of this video and what can it do?

USO is described as a free and open source AI for character and style transfer in image generation. It supports transferring a character or blending styles between images, with online demos on HuggingFace where users can upload a reference image and a style image, then generate new images in that style or with that character in different contexts. It also offers a local run option and a demo interface for ease of use.

Q: What is VibeVoice and what capabilities does it offer for multi speaker text to speech?

VibeVoice is a Microsoft released text-to-speech generator that can take transcripts from multiple speakers, support up to four distinct speakers, and generate very long outputs up to 90 minutes. It automatically applies emotions and expressions based on the transcript, producing natural sounding voice generations and even enabling spontaneous singing in some cases as demonstrated by example transcripts.

Q: Which hardware considerations are mentioned for VoxHammer, and what is recommended?

The description notes that VoxHammer code and models are released and that running it locally requires an Nvidia GPU with at least 40 GB of VRAM, with 80 GB strongly recommended. This indicates that the tool is resource intensive and benefits from high GPU memory to handle complex 3D models and edits efficiently.

Q: What is the relationship between VoxHammer and part-aware editing in practice?

VoxHammer uses what it calls part-aware or part-based editing where the model is segmented into meaningful parts. Edits are applied only to the specified parts, which helps preserve the rest of the model and maintain consistency across the edited object. This approach enables precise, localized changes without unintended alterations to adjacent regions.

Q: What examples are given to illustrate VoxHammer edits in the video?

Several examples show VoxHammer changing parts of models, such as turning a bowl of apples into oranges, adding bread inside a bowl, transforming a shell into stone, or turning a character into a panda. These demonstrations illustrate how prompt or image guided edits can yield targeted modifications while keeping the overall structure intact.

Q: How does the video describe the accessibility of Compass for different image generators?

The video notes that Compass is available for different open source image generators, including Flux and various versions of Stable Diffusion. It emphasizes that Compass can be integrated into existing workflows and provides links to a GitHub repository with instructions for downloading and running locally, along with model variants for different generators.

Summary & Key Takeaways

• VoxHammer enables targeted 3D edits by masking regions and applying changes with prompts or example images, preserving the rest of the model.

• Compass enhances image generation by reliably enforcing spatial relationships across open source models, improving layout accuracy.

• USO provides free, open source character and style transfer for image generation, with online demos and local run options.