图像风格迁移
最后要生成的图片是怎样的是难以想象的，所以朴素的监督学习方法可能不会生效，
Content Loss
根据输入图片和输出图片的像素差别可以比较损失
\(l_{content} = \frac{1}{2}\sum (C_c-T_c)^2\)
Style Loss
从中间提取多个特征层来衡量损失。
利用\(Gram\) \(Matrix\)（格拉姆矩阵）可以衡量风格的相关性，对于一个实矩阵\(X\)，矩阵\(XX^T\)是\(X\)的行向量的格拉姆矩阵
\(l_{style}=\sum wi(Ts-Ss)^2\)
总的损失函数
\(L_{total(S,C,T)}=\alpha l_{content}(C,T)+\beta L_{style}(S,T)\)
代码
from PIL import Image
import matplotlib.pyplot as plt
import numpy as npimport torch
import torch.optim as optim
from torchvision import transforms, modelsvgg = models.vgg19(pretrained=True).features#使用预训练的VGG19，features表示只提取不包括全连接层的部分for i in vgg.parameters():
i.requires_grad_(False)#不要求训练VGG的参数
定义一个显示图片的函数
def load_img(path, max_size=400,shape=None):
img = Image.open(path).convert('RGB') if(max(img.size)) > max_size:#规定图像的最大尺寸
size = max_size
else:
size = max(img.size) if shape is not None:
size = shape
transform = transforms.Compose([
transforms.Resize(size),
transforms.ToTensor(),
transforms.Normalize((0.485, 0.456, 0.406),
(0.229, 0.224, 0.225))
])
'''删除alpha通道(jpg)， 转为png，补足另一个维度-batch'''
img = transform(img)[:3,:,:].unsqueeze(0)
return img
载入图像
content = load_img('./images/turtle.jpg')
style = load_img('./images/wave.jpg', shape=content.shape[-2:])#让两张图尺寸一样'''转换为plt可以画出来的形式'''
def im_convert(tensor):
img = tensor.clone().detach()
img = img.numpy().squeeze()
img = img.transpose(1,2,0)
img = img * np.array((0.229, 0.224, 0.225)) + np.array((0.485, 0.456, 0.406))
img = img.clip(0,1)
return img
使用的图像为（左边为Content Image，右边为Style Image）：
定义几个待会要用到的函数
def get_features(img, model, layers=None):
'''获取特征层'''
if layers is None:
layers = {
'0':'conv1_1',
'5':'conv2_1',
'10':'conv3_1',
'19':'conv4_1',
'21':'conv4_2', #content层
'28':'conv5_1'
} features = {}
x = img
for name, layer in model._modules.items():
x = layer(x)
if name in layers:
features[layers[name]] = x return featuresdef gram_matrix(tensor):
'''计算Gram matrix'''
_, d, h, w = tensor.size() #第一个是batch_size tensor = tensor.view(d, h*w) gram = torch.mm(tensor, tensor.t()) return gram content_features = get_features(content, vgg)
style_features = get_features(style, vgg)style_grams = {layer:gram_matrix(style_features[layer]) for layer in style_features}target = content.clone().requires_grad_(True)'''定义不同层的权重'''
style_weights = {
'conv1_1': 1,
'conv2_1': 0.8,
'conv3_1': 0.5,
'conv4_1': 0.3,
'conv5_1': 0.1,
}
'''定义2种损失对应的权重'''
content_weight = 1
style_weight = 1e6
训练过程
show_every = 400
optimizer = optim.Adam([target], lr=0.003)
steps = 2000for ii in range(steps):
target_features = get_features(target, vgg) content_loss = torch.mean((target_features['conv4_2'] - content_features['conv4_2'])**2)
style_loss = 0
'''加上每一层的gram_matrix矩阵的损失'''
for layer in style_weights:
target_feature = target_features[layer]
target_gram = gram_matrix(target_feature)
_, d, h, w = target_feature.shape
style_gram = style_grams[layer]
layer_style_loss = style_weights[layer] * torch.mean((target_gram - style_gram)**2)
style_loss += layer_style_loss/(d*h*w) #加到总的style_loss里，除以大小 total_loss = content_weight * content_loss + style_weight * style_loss optimizer.zero_grad()
total_loss.backward()
optimizer.step() if ii % show_every == 0 :
print('Total Loss:',total_loss.item())
plt.imshow(im_convert(target))
plt.show()
将输入的图像和最后得到的混合图作比较：
没有达到最好的效果，还有可以优化的空间√
参考：
- Image Style Transfer Using Convolutional Neural Networks论文
- Udacity——PyTorch Scholarship Challenge
