如何扩展Tensorflow变量

Question

有没有办法让Tensorflow变量更大？就像,我想说我想在训练过程中将神经元添加到神经网络的一层.我该怎么做呢？在回答这个问题告诉我如何改变变量的形状,将其展开,以适应权重的另一行,但我不知道如何初始化这些新的权重.

我认为另一种解决方法可能涉及组合变量,如在第二个变量中首先初始化权重,然后将其作为第一个变量的新行或列添加,但我找不到任何可以让我做的事情那个.

Answer 1

有多种方法可以实现这一目标.

1)该帖子中的第二个答案(/sf/answers/2356387631/)解释了如何通过使用validate_shape = False调用'assign'来更改变量的形状.例如,你可以做类似的事情

# Assume var is [m, n] 
# Add the new 'data' of shape [1, n] with new values
new_neuron = tf.constant(...)  

# If concatenating to add a row, concat on the first dimension.
# If new_neuron was [m, 1], you would concat on the second dimension.
new_variable_data = tf.concat(0, [var, new_neuron])  # [m+1, n]

resize_var = tf.assign(var, new_variable_data, validate_shape=False)

然后,当您运行resize_var时,'var'指向的数据现在将具有更新的数据.

2)您还可以创建一个大的初始变量,并在训练进行时在变量的不同区域调用tf.slice,因为您可以动态更改切片的'begin'和'size'属性.

Answer 2

弄清楚了。这是一种迂回的过程，但这是我能说的唯一真正起作用的过程。您需要首先解压变量，然后将新变量附加到末尾，然后将它们重新打包在一起。

如果沿着第一个维度扩展，它会相当短：只有 7 行实际代码。

#the first variable is 5x3
v1 = tf.Variable(tf.zeros([5, 3], dtype=tf.float32), "1")

#the second variable is 1x3
v2 = tf.Variable(tf.zeros([1, 3], dtype=tf.float32), "2")

#unpack the first variable into a list of size 3 tensors
#there should be 5 tensors in the list
change_shape = tf.unpack(v1)

#unpack the second variable into a list of size 3 tensors
#there should be 1 tensor in this list
change_shape_2 = tf.unpack(v2)

#for each tensor in the second list, append it to the first list
for i in range(len(change_shape_2)):
  change_shape.append(change_shape_2[i])

#repack the list of tensors into a single tensor
#the shape of this resultant tensor should be [6, 3]
final = tf.pack(change_shape)

如果你想沿着第二维扩展，它会变得更长一些。

#First variable, 5x3
v3 = tf.Variable(tf.zeros([5, 3], dtype=tf.float32))

#second variable, 5x1
v4 = tf.Variable(tf.zeros([5, 1], dtype=tf.float32))

#unpack tensors into lists of size 3 tensors and size 1 tensors, respectively
#both lists will hold 5 tensors
change = tf.unpack(v3)
change2 = tf.unpack(v4)

#for each tensor in the first list, unpack it into its own list
#this should make a 2d array of size 1 tensors, array will be 5x3
changestep2 = []
for i in range(len(change)):
  changestep2.append(tf.unpack(change[i]))

#do the same thing for the second tensor
#2d array of size 1 tensors, array will be 5x1
change2step2 = []
for i in range(len(change2)):
  change2step2.append(tf.unpack(change2[i]))

  #for each tensor in the array, append it onto the corresponding array in the first list
  for j in range(len(change2step2[i])):
    changestep2[i].append(change2step2[i][j])

  #pack the lists in the array back into tensors
  changestep2[i] = tf.pack(changestep2[i])

#pack the list of tensors into a single tensor
#the shape of this resultant tensor should be [5, 4]
final2 = tf.pack(changestep2)

我不知道是否有更有效的方法来做到这一点，但就目前而言，这是可行的。根据需要，更改更多维度将需要更多层列表。