2 mins read

Meet Boximator, Bytedance’s Video Manipulation AI Tool

Workers across Europe interacting with Artificial Intelligence Source
Meet Boximator, Bytedance's Video Manipulation AI Tool Source

Meet Boximator, Bytedance’s Video Manipulation AI Tool – Key Notes

  • Boximator introduces fine-grained motion control for video synthesis.
  • Developed by ByteDance Research team, led by Jiawei Wang.
  • Offers two constraint types for motion control: hard boxes and soft boxes.
  • Preserves base model knowledge by freezing original weights during training.
  • Employs a self-tracking technique to simplify learning box-object correlations.
  • Achieves state-of-the-art video quality and robust motion controllability.
  • Capable of creating complex video manipulations, like changing object motions.

Boximator – Video Manipulator from Tiktok’s owner

Synthesizing videos with rich and controllable motions has always been a crucial challenge in the field of computer vision.

Today, we delve into a novel solution in AI technology, that aims to tackle this issue head-on – Boximator. This approach promises fine-grained motion control, taking video synthesis to new heights – developer and owner company of Tiktok.

The Concept Behind Boximator

Boximator is a new tool developed by a team of researchers led by Jiawei Wang. The team, affiliated with ByteDance Research, has unveiled Boximator as a plug-in for existing video diffusion models.

It introduces two constraint types: hard boxes and soft boxes.

“Boximator introduces two constraint types: hard box and soft box. Users select objects in

the conditional frame using hard boxes and then use either type of boxes to roughly or rigorously define the object’s position, shape, or motion path in future frames.”

How Boximator Works – Boximator’s Performance

Boximator's performance <a href="https://boximator.github.io/" rel="nofollow">Source</a>
Boximator’s performance Source

Boximator’s training process preserves the base model’s knowledge by freezing the original weights and training only the control module.

To address training challenges, it introduces a novel self-tracking technique that greatly simplifies the learning of box-object correlations.

Empirically, Boximator achieves state-of-the-art video quality (FVD) scores, improving on two base models, and further enhanced after incorporating box constraints. Its robust motion controllability is validated by drastic increases in the bounding box alignment metric.

Boximator In Action

Boximator’s effectiveness can be visualized with examples.

For instance, it can manipulate a “A young woman is turning her head, revealing her face in profile” to “stand and then walk”, or have an “adorable pika turn to the camera”.

The wind can even “blow a woman’s umbrella away” on a rainy day.

Trying Out Boximator

Boximator is currently under development and will be available in the next 2-3 months. However, for users looking for an early experience, the team has prepared an experience channel. Users can try Boximator by emailing the team.

Frequently Asked Questions

  1. What is Boximator?
    • Boximator is a novel tool developed for enhancing video synthesis with precise motion control.
  2. Who developed Boximator?
    • Boximator was developed by a team led by Jiawei Wang, affiliated with ByteDance Research.
  3. What unique features does Boximator offer?
    • Boximator offers hard and soft boxes for fine-grained motion control in videos.
  4. How does Boximator improve video synthesis?
    • It preserves base model knowledge and introduces self-tracking to simplify box-object correlations, enhancing video quality and motion control.
  5. What kind of video manipulations can Boximator perform?
    • Boximator can manipulate objects in videos to change their motion or position in highly detailed ways.

Juhasz "the Mage" Gabor

As a fervent tech and AI enthusiast, I blend my passion for the latest in technology with a flair for writing, illuminating the fascinating world of Artificial Intelligence and its endless possibilities.

Google Empowering Europe with AI Opportunity Initiative Source
Previous Story

5 Million EUR Funding from Google.org: Empowering Europe with AI Opportunity Initiative

Next Story

Studies Show AI Chatbots Provide Inconsistent Accuracy for Musculoskeletal Health Information

Latest from Blog

Go toTop