This week in multimodal ai art (30/Apr - 06/May)

Follow on Twitter, come hang on Discord, consider supporting on Patreon. Try MindsEye, our image generation GUI

* code released

Text-to-Image synthesizers:

- CLIP GEN small model released (Colab, GitHub, Paper)

by HFAiLab

Last week we reported that HFAiLab released the code for CLIP-GEN. This week they released pre-trained models. I've assembled a Colab for it.

CLIP-GEN model is out!
The model is small, so it doesn't compete with state of the art models neither in speed or quality, but still is an interesting approach to keep an 👀

GitHub: https://t.co/ZoGaoMHgbr
Quick Colab I've assembled for you to try out: https://t.co/PWHluhY3Si pic.twitter.com/7YLuJhBWUO
— multimodal ai art (@multimodalart) May 5, 2022

- Dall-E Mega mid-training checkpoint (Hugging Face Spaces)

by Boris Dayma

We reported two weeks ago that DALL-E Mega (the DALL-E replication) had started training, last week a 15% training model was out. And now a ~50% in-training model was released

Updated sample predictions: https://t.co/F089SJ5hrj pic.twitter.com/ABo2bqtKsX
— Boris Dayma 🥑 (@borisdayma) May 6, 2022

- Centipede Diffusion Inpainting Upgrade (colab)

by Zalring

We reported Centipede Diffusion two weeks ago here, it generates images with Latent Diffusion and then upscales with Disco Diffusion. It got an update with both Inpainting for both the Latent Diffusion and the Disco Diffusion bits, check it out!

Centipede Diffusion V2 is out! Now with inpainting for Latent Diffusion and Disco Diffusion, and some other improvements.
Link : https://t.co/nugZLSx0Kr pic.twitter.com/c0txjnu601
— Zalring (@ZalringTW) May 4, 2022

- DALL-E Flow released (Colab,GitHub)

by JINA AI

Like Centipede Diffusion it is a mix of models: It generates images with DALL-E Mega, refines and creates variations with Latent Diffusion, ranks the best with CLIP and upscales the results with SwinR. The backend runs on Jina's servers, which can be convenient for not needing Colab's VM, but also can be overloaded

adding some of my creations. prompt = 'A scientist comparing apples and oranges, by Norman Rockwell.' pic.twitter.com/tSTsJXgs9J
— Han Xiao (@hxiao) May 7, 2022

New CLIP and CLIP like models:

OpenCLIP ViT-B/16+ (GitHub)

by mlfoundations

Four weeks ago we reported on the LAION 400M trained OpenCLIP ViT-B/32. Last week we reported ViT-B/16. Now they released a ViT-B/16+ beefed up version, that overcomes OpenAI's ViT-B/16 performance. I am not aware of any notebook that uses that yet, but I will soon add all OpenCLIP models as options on MindsEye

OpenCLIP concluded an amazing cycle of releases of LAION-400M trained CLIP models releasing a beefed up version of CLIP ViT-B/16, thanks for all this work @wightmanr and team

"The game Among Us in a favela" LAION 400M CLIP ViT-B 16+ Guided Diffusion https://t.co/pIzN8v7joE pic.twitter.com/eatyf3E6el
— multimodal ai art (@multimodalart) May 7, 2022

Text-to-3D synthesizers:

CLIP-Forge code and models released (GitHub, Paper)

by AutodeskAILab

Generate 3D shapes from text!

Excited to share Clip-Forge (CVPR 2022) with everyone. This is a very simple but effective method which tackles the problem of generating shapes from text when paired text and shape data are not available. https://t.co/TF35GPb9pN pic.twitter.com/YZ5tIw0Bm2
— Aditya Sanghi (@sanghiad) May 2, 2022

Learning AI Art:

AIAIArt course (GitHub, Discord)

AIArt is a free and open source AI art course by John Whitaker. There are synchronous classes for the next few Saturdays 4 PM UTC on Twitch. All previous classes stay recorded and available on Google Colabs on the GitHub link