Wasserstein GAN

Traditional GANs lean on Jensen-Shannon divergence and can stall, oscillate, or collapse. The Earth Mover (Wasserstein) distance offers a smoother, more meaningful gradient landscape that tracks how mass must move to align two distributions.

The Wasserstein distance, also known as the Earth Mover's distance, compares distributions by the cost of optimal transport. Given two probability distributions $P$ and $Q$ on metric space $(\mathcal{X}, d)$:

For $p=1$, the dual (KR) form is

The KR dual form highlights the connection between optimal transport and Lipschitz functions and gives a tractable way to optimize Wasserstein distance in practice.

The reformulation of the Wasserstein distance as a maximization over 1-Lipschitz functions simplifies the optimization problem in several ways:

1. The optimization problem becomes convex: The class of 1-Lipschitz functions is a convex set, and the supremum over this set is a convex function. Therefore, the optimization problem becomes convex, which makes it easier to solve using standard techniques from convex optimization.
2. The Lipschitz constraint is easy to enforce: The Lipschitz constraint on the optimizing function can be enforced using techniques like weight clipping or gradient penalty, which are simple to implement and have been shown to work well in practice.
3. The objective function is more stable: Unlike traditional GANs, which suffer from issues like mode collapse and instability due to the non-convexity of their objective function, Wasserstein GANs have a more stable objective function that encourages convergence to a global optimum.
4. The Wasserstein distance is a more meaningful measure of distance between probability distributions: Unlike other metrics like total variation distance or Kullback-Leibler divergence, the Wasserstein distance takes into account the geometry of the underlying space and provides a more meaningful measure of distance between probability distributions.

Which statement would break the KR-dual-based WGAN story?

The paper by Gulrajani et al. (2017) begins by discussing the limitations of traditional GAN training methodologies, particularly in terms of mode collapse and training instability. It highlights how the Wasserstein distance, also known as the Earth-Mover (EM) distance, provides a more meaningful and smooth gradient flow for training GANs.

A key innovation described is the use of a gradient penalty term added to the Wasserstein loss function, which enforces a Lipschitz constraint for the critic (discriminator). This approach, unlike previous methods that clip weights (see the original WGAN algorithm below), allows for more effective training dynamics and avoids common pitfalls such as gradient vanishing or exploding.

The authors present extensive experiments comparing their method against traditional GAN training frameworks. The results demonstrate significant improvements in the stability of GAN training, reduction of mode collapse, and enhancement in the quality of generated images. Metrics used for evaluation include the Inception Score and qualitative assessments by human evaluators.

• If $v$ is a unit vector, then show that $v$ and $(I - vv^T)v$ are orthogonal.
• If $v$ is a unit
• If $v$ is a unit vector, then for any vector $x,$ show that the following holds: $$ \| x \|^2 = (x^T v)^2 - \| (x - x^T v)v \|^2. $$

The Wasserstein GAN algorithm proceeds as follows:

• The algorithm initializes with default values for the learning rate $(\alpha)$, clipping parameter $(c)$, batch size $(m)$, and the number of critic iterations per generator iteration $(n_{\text{critic}})$. Initial parameters for the critic $(w_0)$ and the generator $(\theta_0)$ are also set.
• The training loop continues until the generator's parameters converge.
• Within the training loop, the following steps occur for a specified number of critic iterations $(n_{\text{critic}})$:
  • A batch of real data $\{x^{(i)}\}_{i=1}^m$ is sampled from the real data distribution.
  • A batch of prior samples $\{z^{(i)}\}_{i=1}^m$ is drawn from the latent space's prior distribution.
  • The critic's stochastic gradient is computed based on the difference in evaluations between the real data and the generated data from the generator.
  • The critic's parameters are updated using the RMSProp optimizer with the computed gradient and the learning rate.
  • The critic's weights are then clipped to fall within the range defined by the clipping parameter to maintain the Lipschitz constraint.
• After completing the critic updates, a new batch of prior samples is drawn.
• The generator's stochastic gradient is then computed based on the critic's evaluations of the generated data.
• The generator's parameters are updated using the RMSProp optimizer with the negative of the computed gradient, effectively performing gradient ascent.
• These steps are repeated until the generator's parameters have sufficiently converged, indicating the end of training.

Require: α learning rate, c clipping parameter, m batch size, n_critic critic iterations per generator iteration
Require: w₀ critic params, θ₀ generator params
while θ not converged:
    for t = 0 … n_critic:
        sample {x^(i)}₁ᵐ ~ P_r
        sample {z^(i)}₁ᵐ ~ p(z)
        g_w ← ∇_w [ 1/m Σ f_w(x^(i)) - 1/m Σ f_w(g_θ(z^(i))) ]
        w ← w + α · RMSProp(w, g_w)
        w ← clip(w, -c, c)
    sample {z^(i)}₁ᵐ ~ p(z)
    g_θ ← -∇_θ 1/m Σ f_w(g_θ(z^(i)))
    θ ← θ - α · RMSProp(θ, g_θ)
                

Click a chip for a stabilization lever.