Overall Guidlines for training StyleGAN (+ADA) in PyTorch StudioGAN

• Choose backbone btw ["stylegan2", "stylegan3"]

• If backbone is stylegan3, choose stylegan3_cfg btw ["stylegan3-t", "stylegan3-r"] (Discriminator backbone is always stylegan2, stylegan3-t : translatino equiv., stylegan3-r : translation & rotation equiv.).

• adv_loss used in paper is logistic.

• g_cond_mtd can be chosen btw ["W/O", "cAdaIN"].

• d_cond_mtd can be chosen btw ["W/O", "AC", "PD", "MH", "MD", "2C", "D2DCE", "SPD"].

• z_dim, w_dim should be fixed to 512 regardless of image size.

• apply_g_ema should be true for stable results.

• apply_r1_reg should be true.

• d_reg_interval is fixed to 16 regardless of image size.

• r1_place could be chosen btw ["outside_loop", "inside_loop"], paper setting is "outside_loop".

• For optimizer use Adam with beta1 0, beta2 0.99 and keep d_first False for paper setting.

• You can use ADA (adaptive discriminator augmentation) with apply_ada option.

• Most general ada_aug_type to choose is "bgc".

• For most cases keep ada_initial_augment_p as 0, ada_kimg as 500, ada_interval as 4, and ada_target as 0.6 but you may need to adjust ada_target based on model backbone and other settings.

• Reference StyleGAN_v2, StyleGAN_v3 settings is given below. Choose appropriate total_step, batch_size, d_epilogue_mbstd_group_size, g/d_lr, r1_lambda, g_ema_kimg, g_ema_rampup, mapping_network from below.

• Just in case... the batch_size below corresponds to (OPTIMIZATION.batch_size * OPTIMIZATION.acml_steps) in StudioGAN

• StyleGAN_v3 reproducibility checked using configs on AFHQ_V2. (AFHQ_V2-StyleGAN2 follows setup from StyleGAN3 paper whereas other StyleGAN2 configs come from StyleGAN2-ada)

StyleGAN_v2 official implementation settings.

Slightly more detailed configurations can be found at https://github.com/NVlabs/stylegan2-ada-pytorch.

• style_mixing_p should be 0.9 for all settings except for CIFAR10&100 (0 for CIFAR10&100).

• pl_reg is disabled only for CIFAR10&100, for the rest it should be true with scale of 2.

• d_architecture is 'orig' for CIFAR10&100, for the rest it should be 'resnet'.

• g_reg_interval is fixed to 4 regardless of image size.

• 'paper256': total_steps=390625, batch_size=64, d_epilogue_mbstd_group_size=8, g/d_lr=0.0025, r1_lambda=1, g_ema_kimg=20, g_ema_rampup="N/A", mapping_network=8

• 'paper512': total_steps=390625, batch_size=64, d_epilogue_mbstd_group_size=8, g/d_lr=0.0025, r1_lambda=0.5, g_ema_kimg=20, g_ema_rampup="N/A", mapping_network=8

• 'paper1024':total_steps=781250, batch_size=32, d_epilogue_mbstd_group_size=4, g/d_lr=0.002, r1_lambda=2, g_ema_kimg=10, g_ema_rampup="N/A", mapping_network=8

• 'cifar': total_steps=1562500, batch_size=64, d_epilogue_mbstd_group_size=32, g/d_lr=0.0025, r1_lambda=0.01, g_ema_kimg=500, g_ema_rampup=0.05, mapping_network=2

StyleGAN_v3 official implementation settings.

Highly detailed configurations can be found at https://github.com/NVlabs/stylegan3/blob/main/docs/configs.md. r1_lambda is the key hyperparameter. Below configurations generally apply to both stylegan3-t and stylegan3-r but more specific dataset-optimized configurations can be found in the official link.

• style_mixing_p should be 0 for all settings.

• pl_reg is disabled for all settings (apply_pl_reg=False, pl_weight=0).

• d_architecture is 'resnet' for all settings.

• g_reg_interval is fixed to 1 since there's no regularization for generator.

• set blur_init_sigma for stylegan3-r only with value 10.

• '128': total_steps= 781250, batch_size=32, d_epilogue_mbstd_group_size=4, g/d_lr=0.0025/0.002, r1_lambda=0.5, g_ema_kimg=10, g_ema_rampup="N/A", mapping_network=2

• '256': total_steps= 781250, batch_size=32, d_epilogue_mbstd_group_size=4, g/d_lr=0.0025/0.002, r1_lambda=2, g_ema_kimg=10, g_ema_rampup="N/A", mapping_network=2

• '512': total_steps= 781250, batch_size=32, d_epilogue_mbstd_group_size=4, g/d_lr=0.0025/0.002, r1_lambda=8, g_ema_kimg=10, g_ema_rampup="N/A", mapping_network=2

• '1024': total_steps= 781250, batch_size=32, d_epilogue_mbstd_group_size=4, g/d_lr=0.0025/0.002, r1_lambda=32, g_ema_kimg=10, g_ema_rampup="N/A", mapping_network=2