Megaman's Jockstrap posted:

Stable Diffusion

This is a huge topic and you could literally write a book about it. I won't. We'll try to make this short(ish) and sweet.

At first glance, Stable Diffusion is not as good as MidJourney. Until SDXL (an upcoming model) it had a bug that meant everything in it was literally trained wrong, so all of it's produced images will generally looked more washed out and lower-contrast than Midjourney (this is somewhat fixable). It's base models have less cohesion and more artifacting than Midjourney. Given the same prompt, it will generally produce a weaker and less interesting image.

Well, why use it then? The answer is easy: it's very customizable and you can have a lot of control over it. Forgive the facile analogy, but if Midjourney is Windows then Stable Diffusion is Unix/Linux - the analogy even holds a little because Stable Diffusion is open source and Midjourney is closed and proprietary. The basic expectation here is that you are going to take the time to learn the in and outs and dig under the hood of Stable Diffusion to learn it. If not, stop right now. Use Midjourney.

We'll go over the four things that are good about Stable Diffusion that Midjourney doesn't offer:

1) You can run it on your local box

Yes, if your local box has an NVIDIA graphics card made in the last 10 years it can run Stable Diffusion. Obviously the beefier and faster the card the better this works, but I was getting perfectly acceptable (if slow) results on a decade-old i5 with a GTX 970. You can also run it on AMD stuff but I don't know how that works at all. I know it's a bit trickier and requires some special config work but people do it.

The de-facto interface for it is a web front-end called Automatic1111. There's a ton of tutorials out there on how to install Automatic1111 and get the SD models so I'll leave that to you and Google.

Because you host it you don't have to pay credits or setup an account or anything. You have control over it. You can just generate whatever images you want. It's FREEEEE (unless you're an insane person and buy a new graphics card for SD! don't do that!)

2) Customized models

Stable Diffusion released their models completely for free. Open source and all that. Because it's open source, about 4 minutes later somebody made a way to continue to train models. We're not going to get into how models work but just know that it's entirely possible to extend these models. For example, if a model doesn't know what a Porsche is, you could give it a bunch of pictures of a Porsche and say "that's a Porsche" and then it "knows" and can gen images of that. So lots of people have been extending the two biggest Stable Diffusion models (referred to by their version numbers, 1.5 and 2.1 - but mostly 1.5) with new stuff, or refining old stuff.

Because these models use Stable Diffusion as a base (so that they can continue to "understand" styles and concepts that the creators didn't specifically train them on) they're usually a decent size, 6 gigs or more. But they CAN be highly useful. I say CAN because some of them are very good, but some are WAY overtrained. If you use only a few images for a concept, or a bunch of very similar images, the model will not have any variance and keep generating things that look the same. For example, if I made a model and the only pictures of men I uploaded were of 1973's Mr. Universe Arnold Swartzenegger, then every time I asked for a picture of a man I would get some variation of Arnie (probably flexing). Well, people do this all the time. In fact I'm sorry to say that most people just train them on big tiddy anime waifus. No, I am not joking. The model training community are a bunch of terminal horndogs.

At the time of writing, Civitai.com hosts Stable Diffusion models. Should you choose to visit this site: NSFW NSFW NSFW the front page will instantly poison your mind and make you hate AI. Push past it. Search for RPG to find some decent stuff (and a loving ton of samey anime girls). The "RPG_v4" model (soon to be v5) is pretty good and great for generating portraits and weapons for your games.

3) LORAs

So let's say you have a model that you really like except for one small problem: it doesn't know what a BattleTech Marauder 2C looks like. Criminal! But who wants to train a model and generate gigs of data just to add a Marauder? Well you don't have to, luckily.

Think of a LORA as a post-it-note that you stick to a model that contains exactly one style or concept that you want it to use. It requires incredibly less computer power than adding to a model, and less work too. It's entirely feasible for you (yes you!) to train a LORA, the amount of images needed is shockingly small (5 - 7 gets you decent results, more is better of course). You can use these to do things that would be nearly impossible for Midjourney, such as outputting art in the style of RIFTs. You can get real powerful results here that are simply not obtainable with other methods using custom models and LORA styles.


A picture from a little known RIFTS sourcebook from 1992, RIFTS: Robot Boyz

Remember earlier when I said you could mitigate some of the aspects of Stable Diffusion's bad training bug? Well that's a LORA. https://civitai.com/models/13941/epinoiseoffset

4) Controlnet

The best way to think of Controlnet is a way to apply traits from one image to another, like telling an artist "I want this person to be doing this pose" or whatever. So if you like the general composition of an image, or the pose a person is holding, but you want other things to change completely, that's Controlnet. If you want to apply some aspect of an image to another image, that's Controlnet.

You might have noticed earlier that people are still using SD 1.5 even though 2.1 is out. Well, that's because ControlNet only works with it. That fact alone has kept it going.

When it comes to RPG applications it's really good for getting the facial expressions you want. Here's a rakish rogue with the prompt wanting him smiling:


That's just not good enough! I want a real big smile and a particular pose, let's go ahead and grab this picture that's pretty close to what I want and put it into Controlnet, then tell Controlnet to apply that facial expression to my guy:



It doesn't matter that it's a cartoon. If it's got a face, Controlnet can work with it. Here's the final output:


So we didn't get the Dreamworks Smirk (and there's a small artifact and some small teeth issues that I just didn't bother to fix) but he does look a lot happier! Also note that his head position and pose is exactly like the cartoon picture (although his eyes aren't looking the same way). And if my player cared enough we could try another picture or whatever, but this is good enough as an example I did in 5 minutes.

Here's another 5 minute example. I wanted to make a flower sword where the blade is coming out of petals near the hilt. A quick sketch in GIMP (not even using a tablet) and sending it to Controlnet on the RPG_V4 model with a very basic "a photograph of a flower sword, highly detailed blade" prompt goes from this:

to this:


Again, it's in the exact orientation I wanted, proportions what I asked for, no guessing for a magic prompt. It's not perfect but for 5 minutes it's a great player handout.

It also does full body poses, fingers and hands, architecture, edge detection, A LOT. Controlnet is magic. Controlnet is life. Covering what Controlnet can do is waaaaay beyond the scope of this post so instead here's a mega big tutorial on it: https://stable-diffusion-art.com/controlnet/

Final Thoughts

Stable Diffusion is huge and I've barely scratched the surface here (I did not mention image-to-image, interrogating, inpainting, Krita integration, hypernetworks, Dreambooth, etc), but sufficed to say you're not locked in to giving money to Midjourney or trying to find the "magic phrase" if you don't want to. You can host and use your own image diffusion system and get a really cool way to make player portraits, player handouts, maps, and anything else you could want. A very useful tool for upping the professionalism of your game.

KwegiboHB posted:

Amazing writeup! Thank you for this!

I just want to add there have been advances and an NVIDIA card isn't strictly required anymore. There is AMD, Intel, CPU, hell even iPhone support now. Yeah you can run this on your phone.
Here is a list of quick installers, no messing with git or PATH required.

- Automatic1111 - NVIDIA GPU - By far the most popular webui with an incredible array of options and extensions. https://github.com/EmpireMediaScience/A1111-Web-UI-Installer
- InvokeAI - AMD or Intel GPU - This uses DirectML instead of CUDA. https://invoke-ai.github.io/InvokeAI/installation/010_INSTALL_AUTOMATED/
- NMKD - NVIDIA GPU - Executable windows GUI not webui. https://nmkd.itch.io/t2i-gui
- ComfyUI - NVIDIA GPU - Node based with downloadable premade workflows. https://github.com/comfyanonymous/ComfyUI/releases
- Stable Horde - NO GPU - Crowdsourced donated compute free for those without other means. https://stablehorde.net/ links to a client interface with no installation required https://dbzer0.itch.io/lucid-creations
- Mac and iPhones - I don't know anything about Mac, here's a link anyways https://apps.apple.com/us/app/draw-things-ai-generation/id6444050820
- Runpod - 'The Cloud' - Just means someone elses computer. Rent someone elses GPU for money. Method of last resort. https://blog.runpod.io/stable-diffusion-ui-on-runpod/
- OpenVino - CPU mode - This takes forever but is still doable on literally a toaster, hell yeah stable toast. https://github.com/bes-dev/stable_diffusion.openvino

BrainDance posted:

Here's a quick thing I wrote up about running local LLMs for the next OP.

Open-source local LLMs are currently going through their Stable Diffusion moment. Before March open source LLMs were much weaker than GPT3 and mostly would not run on consumer hardware. We were limited to running GPT-Neo or, at best, GPT-J slowly. Training custom models on them was slow, hard, and poorly documented.
Then in March two things happened at about the same time. Meta released (it “leaked”) their LLaMA models, trained differently from previous open-source models with more training data to compensate for a lower parameter size making a 60B parameter model perform about as well as a 120B parameter model. Then the hardware requirements for running these larger models was drastically reduced through “quantization”, shaving off some bits and leaving the model a fraction of the size. A group at Stanford then trained a model on top of LLaMA in basically the same way as ChatGPT making the first actual open-source equivalent to ChatGPT.
Soon after LLaMA was released methods to run LLaMA and other models at an actually decent speed on a CPU were released.
Just like with Stable Diffusion, LoRA (Low Rank Adaptation) finetuning allowed LLMs to be finetuned with much weaker hardware than before, people could replicate exactly what Stanford had done on their own with a gaming GPU.

Now we’re at the point where we have new models that outperform the last by a huge jump almost every week. LLaMA was released in four sizes, 7B, 13B, 33B, and 65B. The 65B models are a thing most people still cant run, but the 33B models are, in many tasks, almost indistinguishable from GPT3.5.

What do I need to run these models?
7B models with 4-bit quantization require 6GB of vram to run with a GPU, or 3.9GB ram to run with a CPU
13B 10GB vram or 7.8GB ram
30B 20GB vram or 19.5GB ram
65B 40GB vram or 38.5GB ram
The low ram requirements don’t mean you can realistically run a 30B model on any computer with enough ram. You technically can, but it will be slow. Still, running 7B and 13B models on modern CPUs is probably faster than you think it’s going to be.
There are other kinds of quantization which used to make more sense when it was more difficult to use 4-bit models but that’s not all that relevant anymore.

How do I run these models?
There are so many frontends at this point but the two big names are oobabooga’s text-generation-webui and llama.cpp

Oobabooga’s text-generation-webui - The LLM equivalent to Automatic1111’s webui for Stable Diffusion. Download the one-click installer and the rest is pretty obvious. Allows you to download models from within it by just pasting in the huggingface user and model name, has extensions, built in powerful support for finetuning, works with CPUs and GPUs. It now supports 4-bit models by default. The one thing that does require a little bit more setup is using LoRA’s with 4-bit models. For that you need the monkey patch, and to start the webui with the –monkey-patch flag (stick it after CMD_FLAGS = in webui.py, this recently changed so some documentation tells you otherwise) instructions are here

Llama.cpp - started as a way to run 4-bit models on macbooks (which works surprisingly well) and is now basically the forefront of running LLMs on your CPU. Getting a lot of development, like currently the big thing is using the CPU but offloading what you can to the GPU to get a big speedup, sometimes outperforming GPU only models.
I don’t use it though, because I spent way too much on getting 24gb of vram

Which models do I use?
There are two main quantized model formats right now. Things are a little chaotic so who knows how long this will stay true, and one of the formats has even gone through a big update making all previous models obsolete (but just needing conversion) so things can change. Generally though:

GPTQ Models – For running on a GPU
GGML Models – For running on a CPU
Generally you’re going to want to have the original LLaMA models to apply LoRAs to.
Otherwise, almost all models get quantized by one guy right now, TheBloke https://huggingface.co/TheBloke
Right now, good general models are the Vicuna models, Wizard Vicuna uncensored, and for larger models (30B) Guanaco, though I don’t think an uncensored version of this exists yet. Even censored models here though are usually a lot less censored than ChatGPT. You can find all these models in different parameter sizes on TheBloke’s huggingface.
There are a lot of models that have a more niche purpose though. Like Samantha, a model not trained to be a generally helpful model but instead trained to believe she is actually sentient.

How do I finetune models?
This is really where the local models become incredibly useful. It’s not easy, but it’s a lot easier than it was a few months ago. The most flexible and powerful way to finetune a larger model is to finetune a LoRA in the oobabooga webui. This actually has good documentation here
The trickiest thing is formatting all of your training data. Most people are using the alpaca standard right now where this is a field for an instruction and then a field for an output. The AI learns when it gets an instruction it’s then supposed to generate an output like in its training data.
The other, easier way is to just give it a kind of trigger word. You have examples of the kind of data you want it to output all following a word you made up so it learns “when I see this word, complete for it with this kind of text”

This is an area though that gets incredibly complicated, there are too many different ways to do it and too many steps to just write down here. This though, is probably where open source models become most relevant to traditional games, models trained to be aware of specific lores or to respond to questions in a way that fits that world or who knows what else (I’ve had an idea for training a model on questions and answers from DnD masters before for the longest time, just as an experiment to see how much it learns the rules and how much it just hallucinates)

Another option for the hard way to finetune a whole model that is probably going to require renting an A100 somewhere is mallorbc’s finetuning repo, which started before LLaMA as a method for finetuning GPT-J. This is where I started out, before the oobabooga webui existed.

There is a lot more. There are ways to give an AI a massive memory from a larger database, experiments in much larger or even infinite context sizes are popping up, people trying out new formats and new ways to quantize models but, this post can only be so long.

Humbug Scoolbus posted:

For the maps, midjourney kicks out really good battlemaps actually since v4.

Viking village, top down battlemap, detailed, intricate

Humbug Scoolbus posted:

Bonus MJ cyberpunk battlemaps...
top down battle map, cyberpunk alley ,night time, intricate,detailed

Fuzz posted:

Just posting here so I can link said post in another thread and not get probated yet again:


Playing around with Foundry as a VN style for narrative games:

Elysium in Chicago, 1862:


Price Maxwell has entered to address the coterie!

Megazver posted:

Doctor Zero posted:

Another friend's BadAss barbarian.

feverish and oversexed posted:

I'm using a combination of different AIs to run ironsworn. I've been obsessed with getting AI to be my DM since chatgpt4 was released, and I've had varying levels of success. All my testing led me to solo ttrpg which I hadn't heard of before, and I just wanted to talk to other peeps doing similar things.

I'm still testing out exactly how to play, but currently I'm experimenting withing using chatgpt4 as my main "DM" or "oracle", various types of other chats with both chatgpt3 or 4 depending (I have a plus subscription). I use bing to clarifying rules for me, or discuss plot points, and just recently I've been experimenting with using Character.ai to run my secondary character (I was primarily using chatgpt3 for this but I wanted to explore other websites)

I'm using a combination of paper and googlesheets to keep progress, and I'm having a blast tbh

quote:

Spell Name: Castle Keeper's Touch

Spell Level: 4th

Spell School: Transmutation

Casting Time: 1 action

Duration: Instantaneous

Material Components: A small feather duster, a piece of soap, and a vial of cleaning solution

Spell Description: This spell cleans a castle or other large structure within range. The spell removes all dirt, grime, and debris from the structure, leaving it sparkling clean. The spell also removes any stains or marks from the walls, floors, and surfaces, making them look like new. The spell does not affect any creatures or objects within the castle, only the structure itself.

Flavor Text: With a wave of your hand and a whispered incantation, you touch the walls of the castle, imbuing them with the power of cleanliness. In an instant, all the dirt, grime, and debris that has accumulated over the years is whisked away, leaving the castle sparkling clean. The floors are polished, the windows are washed, and the surfaces are gleaming, making the castle look like new once again.

Skios posted:

And some NPC illustrations:



These were all generated by appending a basic description with the following prompt:

quote:

high detail, high resolution, ink on paper, noir comic book style, sharp lines, black and white, high contrast