New AI Tools for 3D Scenes, Video Editing & VFX

TL;DR

MIDI turns a single photo into a realistic 3D scene in as little as 40 seconds by segmenting each object and reconstructing it separately. Tight Inversion edits images while preserving original facial details, and Trajectory Crafter changes the camera movement of any existing video, letting one camera mimic multi-angle shoots.

Transcript

AI never sleeps and this week has been absolutely insane we have an AI that can turn a photo into a super realistic 3D scene we have another AI that can generate long videos with multiple consistent scenes Nvidia releases a new image model that can generate images in less than a second we have another AI that's really good at editing images while k... Read More

Key Insights

• MIDI stands for multi-instance diffusion for single image to 3D scene generation, and it creates an accurate 3D scene from one image by segmenting each object and generating a 3D model for each, then arranging them realistically.
• MIDI can process an image and generate a 3D scene in as little as 40 seconds, and its code is already released on a GitHub repo for running locally.
• MIDI outperforms competitors like SSR and gen 3D SR on scenes such as kitchen tables and patio chairs, though anime bedroom inputs still produce artifacts like a stray beige patch and missing wall pictures.
• Tight Inversion edits images while keeping original details intact, so prompting a portrait as a clown or vampire retains the original person's face instead of generating a completely different face.
• Tight Inversion is model agnostic, working with flux plus PuLID or sdxl with IP adapter, and it uses the original image as a condition to align the inversion and reconstruction paths for higher fidelity.
• Trajectory Crafter changes the camera movements of an existing video, supporting pan, orbit, and zoom, and it can generate unseen areas like the back of a kitchen on the fly.
• Trajectory Crafter can freeze a scene to create bullet-time videos and requires a CUDA GPU of at least 28 GB, so a 24 GB RTX 4090 cannot run it locally.
• Alibaba VACE is an all-in-one video editor and generator that can drive a character's motion from a reference video or replace a subject with a reference character or object.

Questions & Answers

Q: What is MIDI and how does it create a 3D scene from one image?

MIDI stands for multi-instance diffusion for single image to 3D scene generation. It creates a super accurate 3D scene from just a single image by analyzing the photo, segmenting each individual object, and then generating a 3D object for each one before arranging them realistically. Examples show plants, dogs, books, and leaves reconstructed accurately, even generating back-facing data not present in the original input photo.

Q: How fast is MIDI at generating a 3D scene?

MIDI is described as blazing fast. According to the page, it can process an image and generate a 3D scene in as little as 40 seconds. This speed, combined with its accuracy, makes it stand out against competitors. The code is already released on a GitHub repo containing all the instructions on how to download and run it locally on your own computer.

Q: How does MIDI compare to competitors like SSR and gen 3D SR?

When compared with competitors such as SSR and gen 3D SR, MIDI is not even close in quality. For a photo of a kitchen table, MIDI's generation is much more accurate. On a tricky kitchen scene, MIDI reconstructs it very accurately while the other two have many flaws. The same advantage appears on an example with two patio chairs, showing MIDI's superior reconstruction.

Q: What is Tight Inversion and what problem does it solve?

Tight Inversion is an AI that edits images while keeping the original details intact. Normally, modifying an image requires inverting it into a latent space and reconstructing it, but the inversion and reconstruction paths are often not aligned, causing distortion and lower fidelity. Tight Inversion uses the original image as a condition to guide editing, aligning the paths so the edited image keeps original details at higher quality.

Q: Can Tight Inversion be used with different image generators?

Yes, Tight Inversion is model agnostic. Instead of using flux with PuLID, you can also use another image generator like sdxl with IP adapter, which is also used for face swapping. Examples show it keeping the original pose and facial features while adding a beard, or keeping a restaurant setting while adding robots. They released hugging face demos for both flux and sdxl that are free and open source.

Q: How does Trajectory Crafter change camera movement in videos?

Trajectory Crafter is an AI that changes the camera movements of an existing video. You input any video and it generates the same scene with different camera movements and positions. Controls include pan left, right, up, and down, orbit left, right, up, and down, and zoom in and out. You can also specify the center point of an orbit, and it generates parts of the scene not in the original video on the fly.

Q: What hardware do you need to run Trajectory Crafter locally?

Because Trajectory Crafter involves video generation, you cannot run its hugging face space on the free plan and must pay for GPU credits, or run it locally with a powerful enough GPU. The GitHub repo notes it requires a CUDA GPU of at least 28 GB, so even a 4090 with 24 GB of VRAM cannot run it. The hope is the open-source community will create a more quantized, lower-memory version.

Q: What can Alibaba VACE do?

Alibaba unveiled VACE, an extremely powerful all-in-one video editor and generator that can do many things at once. You can generate a video from a prompt, then plug in a reference video of a moving rectangle to make a boy move accordingly. You can also plug in an existing video and a reference character to replace the subject, or supply reference images of a character and an object, which is useful for product commercials.

Summary & Key Takeaways

• MIDI generates a super accurate 3D scene from a single image by analyzing the photo, segmenting each individual object, generating a 3D model for each, and arranging them realistically. It runs in as little as 40 seconds, beats competitors like SSR and gen 3D SR, and has released code on GitHub.

• Tight Inversion edits images while keeping the original details intact, retaining a person's face when prompted to add a beard, hat, or transform them into a clown or vampire. It is model agnostic, works with flux or sdxl, and offers free hugging face demos to clone and run locally.

• Trajectory Crafter changes the camera movement of any existing video, offering pan, orbit, and zoom controls and even bullet-time freezes. It generates unseen parts of scenes on the fly, replacing multi-camera shoots, but needs a 28 GB CUDA GPU. Alibaba VACE is a powerful all-in-one video editor and generator.