My first Binary dream
Blog post description.
BLOG
5/28/20236 min read
Experience
When I was a kid, I loved Adove After effects! So I have some experience with keyframing and camera movement within a 3d space. I have also studied art and illustration for a couple of years, so I am hopeing that when I am comfortable using this tool, I can use that experience to make something cool and unique!
Techincal info
I am using Google colab. [Main reason is to save my poor machine some stress] I am using Colab pro to get enough ressrouces and runtime Generating these kinds of videos is though for the machine! I ran on way to high settings to begin with (100 steps on each image). This wasted too many ressources. (Link to the colab document in references!)
Every animation is only in 720p because of hardware and lime limitations. When I have a good template, I will try to generate a bigger project! (And maybe try AI upscalers)
Idea
Since the the video over this post triggered me to start this project, I am going for the same idea. My goal is to make a music video for one of my faviorite songs. "We all lift together" by Keith Power. I took the lyrics and manually wrote them into a chain of prompts and keyframes. The lyrcis is clear has plenty of verbs, so this could be interesting!
text_prompts = { 0:["Theme is sad and tired cyberpunk. In the heart of a dystopian landscape, a community of hardy souls works together to keep the darkness at bay by greg rutkowski and thomas kinkade, Trending on artstation.", "deep blue color scheme"],1380:["Theme is sad and tired cyberpunk. Cold: the air and water flowing, by greg rutkowski and thomas kinkade, Trending on artstation.", "cyan color scheme"],1500:["Theme is uplifting and tired cyberpunk. Hard: the land we call our home, by greg rutkowski and thomas kinkade, Trending on artstation.", "cyan color scheme"],1590:["Theme is uplifting cyberpunk. Push to keep the dark from coming, by greg rutkowski and thomas kinkade, Trending on artstation.", "cyan color scheme"],1710:["Theme is uplifting cyberpunk. of what we owe, by greg rutkowski and thomas kinkade, Trending on artstation.", "cyan color scheme"],
1830:["Theme is uplifting cyberpunk. This: the song of sons and daughters, by greg rutkowski and thomas kinkade, Trending on artstation.", "dark purple and cyan color scheme"],1920:["Theme is uplifting cyberpunk. Hide the heart of who we are, by greg rutkowski and thomas kinkade, Trending on artstation.", "dark purple and cyan color scheme"],2040:["Theme is uplifting cyberpunk. Making peace to build our future, by greg rutkowski and thomas kinkade, Trending on artstation.", "dark purple and cyan color scheme"],2130:["Theme is strong and empowering cyberpunk. Strong, united, working 'till we fall, by greg rutkowski and thomas kinkade, Trending on artstation.", "dark purple and cyan color scheme"],2250:["Theme is sad and tired cyberpunk. Cold: the air and water flowing, by greg rutkowski and thomas kinkade, Trending on artstation.", "cyan color scheme"],2340:["Theme is uplifting and tired cyberpunk. Hard: the land we call our home, by greg rutkowski and thomas kinkade, Trending on artstation.", "cyan color scheme"],2460:["Theme is uplifting cyberpunk. Push to keep the dark from coming, by greg rutkowski and thomas kinkade, Trending on artstation.", "cyan color scheme"],2580:["Theme is uplifting cyberpunk. of what we owe, by greg rutkowski and thomas kinkade, Trending on artstation.", "cyan color scheme"],2670:["Theme is strong and empowering cyberpunk. This: the song of sons and daughters, by greg rutkowski and thomas kinkade, Trending on artstation.", "light pink and cyan color scheme"],2790:["Theme is strong and empowering cyberpunk. Hide the heart of who we are, by greg rutkowski and thomas kinkade, Trending on artstation.", "light pink and cyan color scheme"],2880:["Theme is strong and empowering cyberpunk. Making peace to build our future, by greg rutkowski and thomas kinkade, Trending on artstation.", "light pink and cyan color scheme"],3000:["Theme is very strong and empowering cyberpunk. Strong, united, working 'till we fall, by greg rutkowski and thomas kinkade, Trending on artstation.", "light pink and cyan color scheme"],3150:["Theme is strong and empowering cyberpunk. And we all lift, and we're all adrift together, together, by greg rutkowski and thomas kinkade, Trending on artstation.", "light pink and cyan color scheme"],3540:["Theme is strong and empowering cyberpunk. Through the cold mist, 'till we're lifeless together, together, by greg rutkowski and thomas kinkade, Trending on artstation.", "light pink and cyan color scheme"],3990:["Theme is uplifting and tired cyberpunk. In the heart of a dystopian landscape, a community of hardy souls works together to keep the darkness at bay by greg rutkowski and thomas kinkade, Trending on artstation.", "deep blue color scheme"],4260:["Darkness cyberpunk", "black color scheme"]}
What is a binary dream?
To understand binary dreaming, it is important to understand what a "typical" human dream is. A dream is when someone manages to come up with content out of nowhere. The content is often based on experiences or input the person have seen or heard through its time, or maybe a thought that has been on the persons mind for a while? A dream is much more than the content, such as feelings and sensations. But we will focus on the "content" part for this post.
A binary dream is not much different from "Human dream". It is content based on "memories" and its "interpretation" of what it has seen. By giving a machine input, we can tell it to "dream" about it by creating content in a dataflow. (Note that AI cannot dream, its just a metaphor!) This flow is set together in an animation which can look very abstract and hard to understand, yet intriguing and mesmerizing.. Like a "human" dream! :]
An example of a well put together Binary dream created by the amazing DoodleChaos, which is also the inspiration for this project:
Making my first(ish) Binary dream!
Main prompt
[Rendered in 14 hours]
I learned that Overfitting is haaaard to fix.
Speed is often much higher than you think.
For slow songs, zoom at "0: (1.07)" is way to fast, I took it down to "0: (1.02)" in the next attempt. (It is so fast that it is hard to see all the details) And 30 fps is to many frames for such a speed. I changed the zoom settings from "0: (1)" to "0:( 0.40)". (I learned later that I should have changed z_position instead..) and "10: (1.10)" is turned to "10: (0.45)".
skip_steps at 10 with init_scale at 1000 is not enough for up to 1 min of content. The color almost allways becomes a solid color in the end because of the lack of randomness. Decreased it to 2 and the scale to 1100. (I later learned that I should have done the opposite! This made the next animation overshift way earlier.)
Increased frames_scale to make sure that the animation can recover if it gets stuck in solid color (Makes it pop up more random stuff more often.) Decreased from 1500 to 100. I dont think this did much..
1st attempt
[Rendered in 7 hours]
The speed is way to fast again! I reduced it alot, but not enough.
The rotation is also way to rough, making me a little dizzy.
Ended up with overfitting again..
Tried "saving" a potential overfitting by changing camera movement.. Did not work..
[Rendered in 8 hours]
After chatting with GPT, it teached me that I shoul increase the sip_steps.. Not decrease! (Facepalm), I try skip_steps at 400 instead of 10 to make it not follow the prompt too much. It worked well!
The animation is way to slow with not much happening. This makes it boring to watch.
With these settings it never gets a overfitting problem! This is because it is slow enough to allways create something new AND has enough freedom to spawn something out of nowhere!
This animation only ran 25 steps on each image. This is seen in the quality. Done to save time.
It looks sooooo cool though!!!
2nd attempt
3rd attempt
4th and 5th attempts
[Rendered in 20 hours]
What worked
It is 4530 frames.
No overfitting!
Very cool style.
The speed works with the song.
What could improve
I had 25 steps for each image in this render. I would like it to have at least 75, but that takes to long.
It was too slow.
I restricted its creativity too much. It made cool things, but at a limited pace. This also made it hard to see that the images are based on unique promps.
A few "lag spikes" where I tried to move the camera to the music.. Need to work on that.
4k without sound
1080p with music
Conclution
I learned a lot doing this! I learned that overfitting is a serious issue that can lead to hours of rendering going to waste. It is hard to fight it because of all the randomness happening with the AI, but slow speed and proper camera movement helps alot! (I think reducing its creativity actually hurt the final product in the end.)
I cannot wait to make more videos like this! It was fun! :D
https://www.reddit.com/r/DiscoDiffusion/search/?q=%20solid%20color
https://www.reddit.com/r/DiscoDiffusion/comments/vrbppr/animation_solid_color_after_first_prompt_can/
https://www.reddit.com/r/DiscoDiffusion/comments/wsqrro/preventing_solid_color_blocks_in_3d_animation_mode/
https://www.youtube.com/watch?v=0fDJXmqdN-A
https://colab.research.google.com/github/alembics/disco-diffusion/blob/main/Disco_Diffusion.ipynb
References
