在Dataflow Worker上使用SSH密钥提取私有库

Question

我正在设置一个数据流作业，为此作业，工作人员需要访问私有的bitbucket存储库以安装库来处理数据。为了授予对数据流工作者的访问权限，我设置了一对SSH密钥（公共和私有）。我设法将私钥添加到我的数据流工作者上。当尝试通过git + ssh pip安装软件包时，出现错误Host key verification failed。

我试图.ssh/known_hosts在dataflow worker上查找文件，但是在普通VM上并不那么简单。

另外，我可以通过以下命令自行设置它，但效果不佳：

mkdir -p ~/.ssh
chmod 0700 ~/.ssh
ssh-keyscan bitbucket.org > ~/.ssh/known_hosts

我仍然收到Host key verification failed错误。

建议的替代解决方案是运行此问题，ssh-keygen -R bitbucket.org但随后出现以下错误： mkstemp: No such file or directory

对于Dataflow Python SDK，您需要使用来打包代码setup.py。工人启动时要执行的所有命令都用编写subprocess.Popen。命令列表如下：

CUSTOM_COMMANDS = [
    # decrypt key encrypted key in repository via gcloud kms
    ['gcloud', '-v'],
    ['gcloud', 'kms', 'decrypt', '--location', 'global', '--keyring',
     'bitbucketpackages', '--key', 'package', '--plaintext-file',
     'bb_package_key_decrypted', '--ciphertext-file', 'bb_package_key'],
    ['chmod', '700', 'bb_package_key_decrypted'],
    # install git & ssh
    ['apt-get', 'update'],
    ['apt-get', 'install', '-y', 'openssh-server'],
    ['apt-get', 'install', '-y', 'git'],
    # add bitbucket.org as known host
    ['mkdir', '-p', '~/.ssh'],
    ['chmod', '0700', '~/.ssh'],
    ['ssh-keyscan', 'bitbucket.org', '>', '~/.ssh/known_hosts'],
    # other attempts to fix it
    # ['ssh-keygen', '-R', 'bitbucket.org']
    # pip install
    ['sh', '-c', 'GIT_SSH_COMMAND="ssh -i ./bb_package_key_decrypted" pip install git+ssh://git@bitbucket.org/team/repo.git'],
] 

Answer 1

尝试更新ssh-keyscan以写入一些临时路径，然后将已知的主机文件位置作为的一部分传递GIT_SSH_COMMAND。例如，我将您的脚本更新为：

CUSTOM_COMMANDS = [
    # decrypt key encrypted key in repository via gcloud kms
    ['gcloud', '-v'],
    ['gcloud', 'kms', 'decrypt', '--location', 'global', '--keyring',
     'bitbucketpackages', '--key', 'package', '--plaintext-file',
     'bb_package_key_decrypted', '--ciphertext-file', 'bb_package_key'],
    ['chmod', '700', 'bb_package_key_decrypted'],
    # install git & ssh
    ['apt-get', 'update'],
    ['apt-get', 'install', '-y', 'openssh-server'],
    ['apt-get', 'install', '-y', 'git'],
    # add bitbucket.org as known host
    ['mkdir', '-p', '~/.ssh'],
    ['chmod', '0700', '~/.ssh'],
    ['ssh-keyscan', 'bitbucket.org', '>', '/tmp/bit_bucket_known_hosts'],
    # other attempts to fix it
    # ['ssh-keygen', '-R', 'bitbucket.org']
    # pip install
    ['sh', '-c', 'GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/tmp/bit_bucket_known_hosts -i ./bb_package_key_decrypted" pip install git+ssh://git@bitbucket.org/team/repo.git'],
]