deforum/sd-webui-deforum

Use Inpainting for Consistency on Single Prompt Animations

Open

#120 opened on Nov 29, 2022

View on GitHubContributor guide
 (3 comments) (0 reactions) (0 assignees)Python (408 forks)auto 404
2D3Dduplicateenhancementhelp wanted

Repository metrics

Stars
 (2,854 stars)
PR merge metrics
 (PR metrics pending)

Description

I believe that Deforum videos of a single prompt would be more consistent if there was a mode that instead of generating a new frame, inpainted the missing details. For example for a 2D camera that is zooming in what it would do is keep all of the image the exact same (maybe with noise scheduling settings) except for the center which it would inpaint at a strength proportional to the zooming speed, so that the zoomed area would be in proper resolution. For a 3D rotating camera as another example, it would find the areas that MiDaS shows weren't in view last frame.

Contributor guide

Research direction
Understand the current frame generation pipeline in Deforum, then implement inpainting for single prompt animations by reusing previous frame content and inpainting only the regions that change due to camera motion (zoom or rotation). Use MiDaS depth estimation for 3D rotations to identify new areas.
Tech stack
python
Domain
ai
Issue type
Feature
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
3
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-2 days
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Active
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Clear
Prerequisites
PythonStable DiffusionDeforum
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
40