This week in multimodal AI art! (07/Jun- 13/Jun)

This week in multimodal ai art (07/Jun - 13/Jun)

Follow on Twitter, come hang on Discord, consider supporting on Patreon. Try MindsEye, our image generation GUI

4 new exciting text-to-3D notebooks, fine tuning CLIP on unlabeled images, Dall-E mini went viral, a new way to control a StyleGAN from CLIP and more!

* code released

Text-to-Image synthesizers:

- DALL-E Mini went viral (Hugging Face Spaces)

by Boris Dayma

DALL-E Mini, a text-to-image model that tries to replicate and amplify DALL-E 1 went viral on its Hugging Face Spaces for meme generation. It is very exciting to see a text-to-image model with full public unrestricted access, in a perspective of understanding its capacities and limitations through play and exploration as opposed to gatekeeping.

pic.twitter.com/t0PDxBACse
— Weird Dall-E Generations (@weirddalle) June 12, 2022

In a bit it will feel weird we couldn't simply conjure our memes before
— multimodal ai art (@multimodalart) June 11, 2022

- ContraCLIP - Interpretable GAN generation driven by pairs of contrasting sentences, released (Paper, GitHub)

by Christos Tzelepis

ContraCLIP is a way to manipulate the latent space of an image generation GAN (such as StyleGAN) with two contrastive sentences (using CLIP!) to create a cool interpolation between each image.

Text-to-3D updates

- CLIP-Mesh SMPLX released (GitHub)

by Nasir Khalid

4 different Google Colabs were released by the author: text-to-character, text-to-facial-expression, text-to-pose, text-to-mesh (similar to the previously reported CLIPMatrix) - but faster and less memory intensive

CLIPMesh SMPLX (needs a nickname asap!) is a new text-to-3D collection of techniques and colab notebooks that can do character creation, facial expression, posing (dab!) and manipulating mashes in a faster and more memory efficient way! https://t.co/KPCRCjF1Vt pic.twitter.com/s8RViioUJP
— multimodal ai art (@multimodalart) June 14, 2022

New CLIP and CLIP-like models:

MUST - Masked Unsupervised Self-training for Zero-shot Image Classification - CLIP fine-tuning on unlabeled images (GitHub)

by Salesforce

MUST is a technique to fine-tune models like CLIP on unlabeled images, this means that you can make models like CLIP perform better on domain-specific tasks and even generally by fine-tuning it on unlabeled images.

Masked Unsupervised Self-training for Zero-shot Image Classification
abs: https://t.co/gAsXXVj1AD

MUST achieves a zero-shot top-1 accuracy of 77.7% on ImageNet using ViT-B, +9.4% higher than CLIP pic.twitter.com/vbexsyGvCU
— AK (@ak92501) June 8, 2022

Learning AI Art:

AIAIArt course (GitHub, Discord)

AIArt is a free and open source AI art course by John Whitaker. There are synchronous classes for the next few Saturdays 4 PM UTC on Twitch. All previous classes stay recorded and available on Google Colabs on the GitHub link