简介
ControlNet 使 Stable Diffusion 有了一层额外的控制，官方的实现中可以从深度、边缘线、OpenPose 等几个维度控制生成的图像。不过ControlNet的Brightness模型在国内被玩出花来了，这是上次在SD上AI生成光影图像的教程：点击此处。那么ControlNet Brightness如何训练的？我们可以通过亮度（brightness / grayscale）控制生图，从而实现老照片还原彩色、对现有图像重新着色等需求。
下面将记录和介绍使用 HuggingFace Diffusers 训练 Brightness ControlNet 的过程。
数据集准备
数据源：
- LAION-Aesthetics V1（LAION 美学评分大于 7 的子集）
- COYO-700M（包含 aesthetic_score_laion_v2 评分）
下载数据：
from img2dataset import download
import shutil
import multiprocessing
def main():
download(
processes_count=16,
thread_count=64,
url_list="laion2B-en-aesthetic",
resize_mode="center_crop",
image_size=512,
output_folder="../laion-en-aesthetic",
output_format="files",
input_format="parquet",
skip_reencode=True,
save_additional_columns=["similarity","hash","punsafe","pwatermark","aesthetic"],
url_col="URL",
caption_col="TEXT",
distributor="multiprocessing",
)
if __name__ == "__main__":
multiprocessing.freeze_support()
main()构建 HuggingFace Datasets，保存本地并推至 Hub：
import os
from datasets import Dataset
from pathlib import Path
from PIL import Image
data_dir = Path(r"H:\DataScience\laion-en-aesthetic")
def entry_for_id(image_folder, filename):
img = Image.open(image_folder / filename)
gray_img = img.convert('L')
caption_filename = filename.replace('.jpg', '.txt')
with open(image_folder / caption_filename) as f:
caption = f.read()
return {
"image": img,
"grayscale_image": gray_img,
"caption": caption,
}
max_images = 1000000
def generate_entries():
index = 0
# cc3m 的所有子文件夹
image_folders = [f.path for f in os.scandir(data_dir) if f.is_dir()]
for image_folder in image_folders:
image_folder = Path(image_folder)
print(image_folder)
# cc3m 子文件夹的所有文件
for filename in os.listdir(image_folder):
if not filename.endswith('.jpg'):
continue
yield entry_for_id(image_folder, filename)
index += 1
if index >= max_images:
break
if index >= max_images:
break
ds = Dataset.from_generator(generate_entries, cache_dir="./.cache")
ds.save_to_disk("./grayscale_image_aesthetic_1M")
ds.push_to_hub('ioclab/grayscale_image_aesthetic_1M', private=True)训练过程
使用 ControlNet training example 脚本训练，具体参数如下：
accelerate launch train_controlnet_local.py \
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" \
--output_dir="./output_v1a2u" \
--dataset_name="./grayscale_image_aesthetic_100k" \
--resolution=512 \
--learning_rate=1e-5 \
--image_column=image \
--caption_column=caption \
--conditioning_image_column=grayscale_image \
--train_batch_size=16 \
--gradient_accumulation_steps=4 \
--num_train_epochs=2 \
--tracker_project_name="control_v1a2u_sd15_brightness" \
--enable_xformers_memory_efficient_attention \
--checkpointing_steps=5000 \
--hub_model_id="ioclab/grayscale_controlnet" \
--report_to wandb \
--push_to_hubwandb 后台数据：
A6000 GPU 训练时长：13h，sample_count：100k，epoch：1，batch_size：16，gradient_accumulation_steps：1。
TPU v4-8 GPU 训练时长：25h，sample_count：3m，epoch：1，batch_size：2，gradient_accumulation_steps：25。
训练报告：
Google 提供的 TPU v4-8 的机器，配置了 240 核 480 线程 CPU、400GB 内存、128GB TPU 内存、2000Mbps 带宽、3TB 磁盘。
粗浅计算，TPU v4-8 bf16 较单块 A6000 fp32 有 15 倍的速度提升。
ControlNet 论文中提到的 ”Sudden Convergence“ 现象：
效果
Can you be more specific about the content of your article? After reading it, I still have some doubts. Hope you can help me.
okok email：gorgegao@gmail.com
Can you be more specific about the content of your article? After reading it, I still have some doubts. Hope you can help me.
Thanks for sharing. I read many of your blog posts, cool, your blog is very good.
Piece of writing writing is also a fun, if you be acquainted with after that you can write if not it is complicated to write.
At this time it sounds like Expression Engine is the best blogging platform available right now. (from what I’ve read) Is that what you’re using on your blog?
Excellent article. Keep posting such kind of information on your blog. Im really impressed by it.
Right now it sounds like WordPress is the preferred blogging platform available right now. (from what I’ve read) Is that what you are using on your blog?
At this time it seems like Expression Engine is the top blogging platform out there right now. (from what I’ve read) Is that what you’re using on your blog?
Aw, this was a really nice post. Spending some time and actual effort to make a top notch article… but what can I say… I put things off a whole lot and never manage to get nearly anything done.
Currently it sounds like Expression Engine is the preferred blogging platform out there right now. (from what I’ve read) Is that what you are using on your blog?
Right now it looks like Drupal is the top blogging platform available right now. (from what I’ve read) Is that what you’re using on your blog?
Hello there, You have done an incredible job. I’ll certainly digg it and for my part recommend to my friends. I’m sure they will be benefited from this website.
Aw, this was an incredibly nice post. Spending some time and actual effort to generate a good article… but what can I say… I put things off a whole lot and don’t seem to get nearly anything done.
Wow that was odd. I just wrote an really long comment but after I clicked submit my comment didn’t appear. Grrrr… well I’m not writing all that over again. Anyway, just wanted to say great blog!
May I just say what a comfort to find someone who truly knows what they’re discussing over the internet. You actually know how to bring an issue to light and make it important. More and more people have to look at this and understand this side of the story. I can’t believe you are not more popular given that you surely have the gift.
Aw, this was an incredibly nice post. Taking the time and actual effort to produce a good article… but what can I say… I hesitate a lot and never seem to get anything done.
Can I just say what a relief to uncover a person that truly knows what they are talking about online. You certainly know how to bring a problem to light and make it important. More people should read this and understand this side of the story. I was surprised you’re not more popular since you certainly have the gift.
Aw, this was a really good post. Taking the time and actual effort to create a superb article… but what can I say… I put things off a whole lot and don’t manage to get anything done.
Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.
Currently it sounds like Drupal is the preferred blogging platform available right now. (from what I’ve read) Is that what you are using on your blog?
Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.
Your point of view caught my eye and was very interesting. Thanks. I have a question for you. https://accounts.binance.bh/en-ZA/register-person?ref=B4EPR6J0
Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.
At this time it looks like Movable Type is the preferred blogging platform available right now. (from what I’ve read) Is that what you are using on your blog?
Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.
At this time it sounds like Expression Engine is the best blogging platform out there right now. (from what I’ve read) Is that what you’re using on your blog?
Right now it looks like BlogEngine is the top blogging platform available right now. (from what I’ve read) Is that what you are using on your blog?
Pretty section of content. I just stumbled upon your blog and in accession capital to assert that I get in fact enjoyed account your blog posts. Any way I will be subscribing to your augment and even I achievement you access consistently quickly.
At this time it seems like BlogEngine is the best blogging platform available right now. (from what I’ve read) Is that what you’re using on your blog?
Wow that was unusual. I just wrote an incredibly long comment but after I clicked submit my comment didn’t appear. Grrrr… well I’m not writing all that over again. Anyhow, just wanted to say excellent blog!
Currently it sounds like WordPress is the best blogging platform available right now. (from what I’ve read) Is that what you are using on your blog?
Aw, this was an incredibly good post. Spending some time and actual effort to make a great article… but what can I say… I put things off a lot and never seem to get anything done.
Hello there, You’ve done an excellent job. I’ll certainly digg it and individually suggest to my friends. I am sure they will be benefited from this web site.
Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.
Aw, this was an exceptionally good post. Taking a few minutes and actual effort to generate a great article… but what can I say… I procrastinate a whole lot and never manage to get anything done.
Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.
Currently it looks like Expression Engine is the preferred blogging platform out there right now. (from what I’ve read) Is that what you are using on your blog?
Wow that was strange. I just wrote an incredibly long comment but after I clicked submit my comment didn’t appear. Grrrr… well I’m not writing all that over again. Anyways, just wanted to say excellent blog!
Can I simply just say what a comfort to discover an individual who truly understands what they are discussing on the web. You actually understand how to bring an issue to light and make it important. A lot more people have to look at this and understand this side of your story. It’s surprising you aren’t more popular because you definitely have the gift.
Aw, this was an incredibly good post. Taking the time and actual effort to create a good article… but what can I say… I hesitate a whole lot and don’t manage to get anything done.
Excellent pieces. Keep writing such kind of info on your page. Im really impressed by your blog.
Wow that was strange. I just wrote an really long comment but after I clicked submit my comment didn’t show up. Grrrr… well I’m not writing all that over again. Anyway, just wanted to say wonderful blog!
Aw, this was a really good post. Spending some time and actual effort to produce a really good article… but what can I say… I procrastinate a lot and never seem to get anything done.
Currently it looks like Movable Type is the top blogging platform available right now. (from what I’ve read) Is that what you’re using on your blog?
Greate post. Keep writing such kind of information on your site. Im really impressed by your blog.
Wow that was odd. I just wrote an very long comment but after I clicked submit my comment didn’t show up. Grrrr… well I’m not writing all that over again. Anyhow, just wanted to say wonderful blog!
Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.
Thanks for another great article. The place else may just anyone get that type of information in such a perfect approach of writing? I have a presentation subsequent week, and I am on the look for such information.
Hello there, You’ve done a great job. I will definitely digg it and in my view recommend to my friends. I am confident they’ll be benefited from this web site.
Excellent article. Keep posting such kind of information on your page. Im really impressed by it.
At this time it seems like BlogEngine is the best blogging platform out there right now. (from what I’ve read) Is that what you’re using on your blog?
Hey I know this is off topic but I was wondering if you knew of any widgets I could add to my blog that automatically tweet my newest twitter updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe you would have some experience with something like this. Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.
Aw, this was an extremely nice post. Taking a few minutes and actual effort to generate a great article… but what can I say… I hesitate a whole lot and don’t manage to get anything done.
Right now it appears like WordPress is the preferred blogging platform out there right now. (from what I’ve read) Is that what you’re using on your blog?
May I just say what a comfort to find somebody who really knows what they’re discussing on the net. You actually understand how to bring a problem to light and make it important. A lot more people should look at this and understand this side of the story. It’s surprising you’re not more popular since you surely possess the gift.
Hi there, You’ve performed an incredible job. I’ll definitely digg it and for my part suggest to my friends. I am sure they’ll be benefited from this website.
Hey there, You’ve done a great job. I’ll certainly digg it and in my view suggest to my friends. I am confident they will be benefited from this website.
Can I just say what a relief to uncover a person that truly understands what they are talking about on the net. You certainly understand how to bring an issue to light and make it important. A lot more people need to look at this and understand this side of your story. I was surprised that you’re not more popular because you certainly possess the gift.