Backprop实施问题

Question

我该怎么做.我有一个黑白图像(100x100px):

我应该用这个图像训练一个反向传播神经网络.输入是图像的x,y坐标(从0到99),输出为1(白色)或0(黑色).

一旦网络学会了,我希望它能够根据其权重重现图像,并获得与原始图像最接近的图像.

这是我的backprop实现:

import os
import math
import Image
import random
from random import sample

#------------------------------ class definitions

class Weight:
    def __init__(self, fromNeuron, toNeuron):
        self.value = random.uniform(-0.5, 0.5)
        self.fromNeuron = fromNeuron
        self.toNeuron = toNeuron
        fromNeuron.outputWeights.append(self)
        toNeuron.inputWeights.append(self)
        self.delta = 0.0 # delta value, this will accumulate and after each training cycle used to adjust the weight value

    def calculateDelta(self, network):
        self.delta += self.fromNeuron.value * self.toNeuron.error

class Neuron:
    def __init__(self):
        self.value = 0.0        # the output
        self.idealValue = 0.0   # the ideal output
        self.error = 0.0        # error between output and ideal output
        self.inputWeights = []
        self.outputWeights = []

    def activate(self, network):
        x = 0.0;
        for weight in self.inputWeights:
            x += weight.value * weight.fromNeuron.value
        # sigmoid function
        if x < -320:
            self.value = 0
        elif x > 320:
            self.value = 1
        else:
            self.value = 1 / (1 + math.exp(-x))

class Layer:
    def __init__(self, neurons):
        self.neurons = neurons

    def activate(self, network):
        for neuron in self.neurons:
            neuron.activate(network)

class Network:
    def __init__(self, layers, learningRate):
        self.layers = layers
        self.learningRate = learningRate # the rate at which the network learns
        self.weights = []
        for hiddenNeuron in self.layers[1].neurons:
            for inputNeuron in self.layers[0].neurons:
                self.weights.append(Weight(inputNeuron, hiddenNeuron))
            for outputNeuron in self.layers[2].neurons:
                self.weights.append(Weight(hiddenNeuron, outputNeuron))

    def setInputs(self, inputs):
        self.layers[0].neurons[0].value = float(inputs[0])
        self.layers[0].neurons[1].value = float(inputs[1])

    def setExpectedOutputs(self, expectedOutputs):
        self.layers[2].neurons[0].idealValue = expectedOutputs[0]

    def calculateOutputs(self, expectedOutputs):
        self.setExpectedOutputs(expectedOutputs)
        self.layers[1].activate(self) # activation function for hidden layer
        self.layers[2].activate(self) # activation function for output layer        

    def calculateOutputErrors(self):
        for neuron in self.layers[2].neurons:
            neuron.error = (neuron.idealValue - neuron.value) * neuron.value * (1 - neuron.value)

    def calculateHiddenErrors(self):
        for neuron in self.layers[1].neurons:
            error = 0.0
            for weight in neuron.outputWeights:
                error += weight.toNeuron.error * weight.value
            neuron.error = error * neuron.value * (1 - neuron.value)

    def calculateDeltas(self):
        for weight in self.weights:
            weight.calculateDelta(self)

    def train(self, inputs, expectedOutputs):
        self.setInputs(inputs)
        self.calculateOutputs(expectedOutputs)
        self.calculateOutputErrors()
        self.calculateHiddenErrors()
        self.calculateDeltas()

    def learn(self):
        for weight in self.weights:
            weight.value += self.learningRate * weight.delta

    def calculateSingleOutput(self, inputs):
        self.setInputs(inputs)
        self.layers[1].activate(self)
        self.layers[2].activate(self)
        #return round(self.layers[2].neurons[0].value, 0)
        return self.layers[2].neurons[0].value


#------------------------------ initialize objects etc

inputLayer = Layer([Neuron() for n in range(2)])
hiddenLayer = Layer([Neuron() for n in range(10)])
outputLayer = Layer([Neuron() for n in range(1)])

learningRate = 0.4

network = Network([inputLayer, hiddenLayer, outputLayer], learningRate)


# let's get the training set
os.chdir("D:/stuff")
image = Image.open("backprop-input.gif")
pixels = image.load()
bbox = image.getbbox()
width = 5#bbox[2] # image width
height = 5#bbox[3] # image height

trainingInputs = []
trainingOutputs = []
b = w = 0
for x in range(0, width):
    for y in range(0, height):
        if (0, 0, 0, 255) == pixels[x, y]:
            color = 0
            b += 1
        elif (255, 255, 255, 255) == pixels[x, y]:
            color = 1
            w += 1
        trainingInputs.append([float(x), float(y)])
        trainingOutputs.append([float(color)])

print "\nOriginal image ... Black:"+str(b)+" White:"+str(w)+"\n"

#------------------------------ let's train

for i in range(500):
    for j in range(len(trainingOutputs)):
        network.train(trainingInputs[j], trainingOutputs[j])
        network.learn()
    for w in network.weights:
        w.delta = 0.0

#------------------------------ let's check

b = w = 0
for x in range(0, width):
    for y in range(0, height):
        out = network.calculateSingleOutput([float(x), float(y)])
        if 0.0 == round(out):
            color = (0, 0, 0, 255)
            b += 1
        elif 1.0 == round(out):
            color = (255, 255, 255, 255)
            w += 1
        pixels[x, y] = color
        #print out

print "\nAfter learning the network thinks ... Black:"+str(b)+" White:"+str(w)+"\n"

显然,我的实施存在一些问题.上面的代码返回:

原始图像...黑色:21白色:4

学习网络后认为...黑色:25白色:0

如果我尝试使用更大的训练集(我只是测试上面图像中的25个像素用于测试目的),它也会做同样的事情.它返回所有像素在学习后应为黑色.

现在,如果我使用这样的手动训练集:

trainingInputs = [
    [0.0,0.0],
    [1.0,0.0],
    [2.0,0.0],
    [0.0,1.0],
    [1.0,1.0],
    [2.0,1.0],
    [0.0,2.0],
    [1.0,2.0],
    [2.0,2.0]
]
trainingOutputs = [
    [0.0],
    [1.0],
    [1.0],
    [0.0],
    [1.0],
    [0.0],
    [0.0],
    [0.0],
    [1.0]
]

#------------------------------ let's train

for i in range(500):
    for j in range(len(trainingOutputs)):
        network.train(trainingInputs[j], trainingOutputs[j])
        network.learn()
    for w in network.weights:
        w.delta = 0.0

#------------------------------ let's check

for inputs in trainingInputs:
    print network.calculateSingleOutput(inputs)

输出例如:

0.0330125791296   # this should be 0, OK
0.953539182136    # this should be 1, OK
0.971854575477    # this should be 1, OK
0.00046146137467  # this should be 0, OK
0.896699762781    # this should be 1, OK
0.112909223162    # this should be 0, OK
0.00034058462280  # this should be 0, OK
0.0929886299643   # this should be 0, OK
0.940489647869    # this should be 1, OK

换句话说,网络猜测所有像素都是正确的(黑色和白色).如果我使用图像中的实际像素而不是如上所述的硬编码训练集,为什么所有像素都应该是黑色？

我尝试改变隐藏层(最多100个神经元)中的神经元数量,但没有成功.

这是一个功课.

这也是我之前关于backprop的问题的延续.

Answer 1

已经有一段时间了,但我确实获得了这个学位的学位,所以我希望其中一些已经卡住了.

据我所知,你用输入集过度沉重中层神经元.也就是说,您的输入集包含10,000个离散输入值(100像素x 100像素); 你试图将这10,000个值编码成10个神经元.这种级别的编码很难(我怀疑这是可能的,但肯定很难); 至少,你需要大量的培训(超过500次运行)才能让它合理地重现.即使中间层有100个神经元,你也会看到一个相对密集的压缩级别(100像素到1个神经元).

至于如何处理这些问题; 好吧,这很棘手.你可以大大增加中间神经元的数量,你会得到合理的效果,但当然需要很长时间训练.但是,我认为可能会有不同的解决方案; 如果可能,您可以考虑使用极坐标而不是笛卡尔坐标作为输入; 输入模式的快速眼球表示高水平的对称性,并且有效地你会看到沿着角坐标具有重复可预测变形的线性模式,它似乎可以很好地编码在少数中间层神经元中.

这个东西很棘手; 寻找模式编码的一般解决方案(正如您的原始解决方案所做的那样)非常复杂,并且通常(即使有大量的中间层神经元)也需要大量的训练过程; 另一方面,一些先进的启发式任务分解和一些问题重新定义(即从笛卡尔坐标转换到极坐标)可以为明确定义的问题集提供良好的解决方案.当然,这是永久的磨擦; 一般的解决方案很难得到,但稍微更具体的解决方案确实非常好.

有趣的东西,无论如何!