如何在Python中的AWS Lambda函数中正确解密PGP加密文件？

Question

如何在Python中的AWS Lambda函数中正确解密PGP加密文件？

Eri*_*tin 2 gnupg pgp python-3.x aws-lambda python-gnupgp

我正在尝试用 python 创建一个 AWS Lambda：

从 S3 存储桶下载压缩和加密的文件
使用解密文件python-gnupg
将解密的压缩内容存储在另一个S3存储桶中

python-gnupg这是在 Lambda 层中使用 python 3.8 和包。

我已验证 PGP 密钥是否正确，它是否已正确加载到密钥环中，并且加密文件是否已正确下载。但是，当我尝试运行时，gnupg.decrypt_file我得到的输出看起来好像已经成功，但解密状态显示not ok并且解密的文件不存在。

如何在 Lambda 中进行 PGP 解密？

以下是从 lambda 函数中提取的相关代码：

import gnupg
from pathlib import Path

# ...

gpg = gnupg.GPG(gnupghome='/tmp')

# ...

encrypted_path = '/tmp/encrypted.zip'
decrypted_path = '/tmp/decrypted.zip'

# ...

# this works as expected
status = gpg.import_keys(MY_KEY_DATA)

# ...

print('Performing Decryption of', encrypted_path)
print(encrypted_path, "exists :", Path(encrypted_path).exists())

with open(encrypted_path, 'rb') as f:
    status = gpg.decrypt_file(f, output=decrypted_path, always_trust = True)

print('decrypt ok =', status.ok)
print('decrypt status = ', status.status)
print('decrypt stderr = ', status.status)
print('decrypt stderr = ', status.stderr)
print(decrypted_path, "exists :", Path(decrypted_path).exists())

Run Code Online (Sandbox Code Playgroud)

期望在 CloudWatch 中获得类似于以下内容的输出：

2022-11-08T10:24:43.939-05:00 Performing Decryption of /tmp/encrypted.zip
2022-11-08T10:24:44.018-05:00 /tmp/encrypted.txt exists : True
2022-11-08T10:24:44.018-05:00 decrypt ok = True
2022-11-08T10:24:44.018-05:00 decrypt status = [SOME OUTPUT FROM GPG BINARY]
2022-11-08T10:24:44.018-05:00 decrypt stderr = ""
2022-11-08T10:24:44.214-05:00 /tmp/decrypted.txt exists : True

Run Code Online (Sandbox Code Playgroud)

相反，我得到的是：

2022-11-08T10:24:43.939-05:00 Performing Decryption of /tmp/encrypted.zip
2022-11-08T10:24:44.018-05:00 /tmp/encrypted.txt exists : True
2022-11-08T10:24:44.018-05:00 decrypt ok = False
2022-11-08T10:24:44.018-05:00 decrypt status = good passphrase
2022-11-08T10:24:44.018-05:00 decrypt stderr = [GNUPG:] ENC_TO XXXXXX 1 0
2022-11-08T10:24:44.214-05:00 /tmp/decrypted.txt exists : False

Run Code Online (Sandbox Code Playgroud)

看起来好像解密过程开始工作，但有什么东西杀死了它，或者可能是gpg二进制文件可能正在等待一些 TTY 输入并停止？

我尝试gpg使用 cli 在本地运行解密，它按预期工作，尽管我使用的是 GnuPG 版本 2.3.1，不确定 Lambda 上存在什么版本。

Answer 1

Eri*_*tin 7

经过大量挖掘后，我设法使其正常工作。我不能 100% 确定原因是否是GnuPG默认安装在 Lambda 映像上的旧二进制文件，但为了确定，我决定为 lambda 构建一个 GnuPG 2.3.1 层，我确认它在 Docker 容器中按预期工作。

我使用https://github.com/skeeto/lean-static-gpg/blob/master/build.sh作为在 Docker 中编译二进制文件的基础，但对其进行了更新以包括压缩，这是此用例所需的。

以下是我使用的更新后的build.sh脚本，针对 Lambda 构建进行了优化：

#!/bin/sh

set -e

MUSL_VERSION=1.2.2
GNUPG_VERSION=2.3.1
LIBASSUAN_VERSION=2.5.5
LIBGCRYPT_VERSION=1.9.2
LIBGPGERROR_VERSION=1.42
LIBKSBA_VERSION=1.5.1
NPTH_VERSION=1.6
PINENTRY_VERSION=1.1.1
BZIP_VERSION=1.0.6-g10
ZLIB_VERSION=1.2.12

DESTDIR=""
PREFIX="/opt"
WORK="$PWD/work"
PATH="$PWD/work/deps/bin:$PATH"
NJOBS=$(nproc)

clean() {
    rm -rf "$WORK"
}

distclean() {
    clean
    rm -rf download
}

download() {
    gnupgweb=https://gnupg.org/ftp/gcrypt
    mkdir -p download
    (
        cd download/
        xargs -n1 curl -O <<EOF
https://www.musl-libc.org/releases/musl-$MUSL_VERSION.tar.gz
$gnupgweb/gnupg/gnupg-$GNUPG_VERSION.tar.bz2
$gnupgweb/libassuan/libassuan-$LIBASSUAN_VERSION.tar.bz2
$gnupgweb/libgcrypt/libgcrypt-$LIBGCRYPT_VERSION.tar.bz2
$gnupgweb/libgpg-error/libgpg-error-$LIBGPGERROR_VERSION.tar.bz2
$gnupgweb/libksba/libksba-$LIBKSBA_VERSION.tar.bz2
$gnupgweb/npth/npth-$NPTH_VERSION.tar.bz2
$gnupgweb/pinentry/pinentry-$PINENTRY_VERSION.tar.bz2
$gnupgweb/bzip2/bzip2-$BZIP_VERSION.tar.gz
$gnupgweb/zlib/zlib-$ZLIB_VERSION.tar.gz
EOF
    )
}

clean

if [ ! -d download/ ]; then
    download
fi

mkdir -p "$DESTDIR$PREFIX" "$WORK/deps"

tar -C "$WORK" -xzf download/musl-$MUSL_VERSION.tar.gz
(
    mkdir -p "$WORK/musl"
    cd "$WORK/musl"
    ../musl-$MUSL_VERSION/configure \
        --prefix="$WORK/deps" \
        --enable-wrapper=gcc \
        --syslibdir="$WORK/deps/lib"
    make -kj$NJOBS
    make install
    make clean
)

tar -C "$WORK" -xzf download/zlib-$ZLIB_VERSION.tar.gz
(
    mkdir -p "$WORK/zlib"
    cd "$WORK/zlib"
    ../zlib-$ZLIB_VERSION/configure \
        --prefix="$WORK/deps"
    make -kj$NJOBS
    make install
    make clean
)

tar -C "$WORK" -xzf download/bzip2-$BZIP_VERSION.tar.gz
(
    export CFLAGS="-fPIC"
    cd "$WORK/bzip2-$BZIP_VERSION"
    make install PREFIX="$WORK/deps"
    make clean
)

tar -C "$WORK" -xjf download/npth-$NPTH_VERSION.tar.bz2
(
    mkdir -p "$WORK/npth"
    cd "$WORK/npth"
    ../npth-$NPTH_VERSION/configure \
        CC="$WORK/deps/bin/musl-gcc" \
        --prefix="$WORK/deps" \
        --enable-shared=no \
        --enable-static=yes
    make -kj$NJOBS
    make install
)

tar -C "$WORK" -xjf download/libgpg-error-$LIBGPGERROR_VERSION.tar.bz2
(
    mkdir -p "$WORK/libgpg-error"
    cd "$WORK/libgpg-error"
    ../libgpg-error-$LIBGPGERROR_VERSION/configure \
        CC="$WORK/deps/bin/musl-gcc" \
        --prefix="$WORK/deps" \
        --enable-shared=no \
        --enable-static=yes \
        --disable-nls \
        --disable-doc \
        --disable-languages
    make -kj$NJOBS
    make install
)

tar -C "$WORK" -xjf download/libassuan-$LIBASSUAN_VERSION.tar.bz2
(
    mkdir -p "$WORK/libassuan"
    cd "$WORK/libassuan"
    ../libassuan-$LIBASSUAN_VERSION/configure \
        CC="$WORK/deps/bin/musl-gcc" \
        --prefix="$WORK/deps" \
        --enable-shared=no \
        --enable-static=yes \
        --with-libgpg-error-prefix="$WORK/deps"
    make -kj$NJOBS
    make install
)

tar -C "$WORK" -xjf download/libgcrypt-$LIBGCRYPT_VERSION.tar.bz2
(
    mkdir -p "$WORK/libgcrypt"
    cd "$WORK/libgcrypt"
    ../libgcrypt-$LIBGCRYPT_VERSION/configure \
        CC="$WORK/deps/bin/musl-gcc" \
        --prefix="$WORK/deps" \
        --enable-shared=no \
        --enable-static=yes \
        --disable-doc \
        --with-libgpg-error-prefix="$WORK/deps"
    make -kj$NJOBS
    make install
)

tar -C "$WORK" -xjf download/libksba-$LIBKSBA_VERSION.tar.bz2
(
    mkdir -p "$WORK/libksba"
    cd "$WORK/libksba"
    ../libksba-$LIBKSBA_VERSION/configure \
        CC="$WORK/deps/bin/musl-gcc" \
        --prefix="$WORK/deps" \
        --enable-shared=no \
        --enable-static=yes \
        --with-libgpg-error-prefix="$WORK/deps"
    make -kj$NJOBS
    make install
)

tar -C "$WORK" -xjf download/gnupg-$GNUPG_VERSION.tar.bz2
(
    mkdir -p "$WORK/gnupg"
    cd "$WORK/gnupg"
    ../gnupg-$GNUPG_VERSION/configure \
        CC="$WORK/deps/bin/musl-gcc" \
        LDFLAGS="-static -s" \
        --prefix="$PREFIX" \
        --with-libgpg-error-prefix="$WORK/deps" \
        --with-libgcrypt-prefix="$WORK/deps" \
        --with-libassuan-prefix="$WORK/deps" \
        --with-ksba-prefix="$WORK/deps" \
        --with-npth-prefix="$WORK/deps" \
        --with-agent-pgm="$PREFIX/bin/gpg-agent" \
        --with-pinentry-pgm="$PREFIX/bin/pinentry" \
        --enable-zip \
        --enable-bzip2 \
        --disable-card-support \
        --disable-ccid-driver \
        --disable-dirmngr \
        --disable-gnutls \
        --disable-gpg-blowfish \
        --disable-gpg-cast5 \
        --disable-gpg-idea \
        --disable-gpg-md5 \
        --disable-gpg-rmd160 \
        --disable-gpgtar \
        --disable-ldap \
        --disable-libdns \
        --disable-nls \
        --disable-ntbtls \
        --disable-photo-viewers \
        --disable-scdaemon \
        --disable-sqlite \
        --disable-wks-tools
    make -kj$NJOBS
    make install DESTDIR="$DESTDIR"
    rm "$DESTDIR$PREFIX/bin/gpgscm"
)

tar -C "$WORK" -xjf download/pinentry-$PINENTRY_VERSION.tar.bz2
(
    mkdir -p "$WORK/pinentry"
    cd "$WORK/pinentry"
    ../pinentry-$PINENTRY_VERSION/configure \
        CC="$WORK/deps/bin/musl-gcc" \
        LDFLAGS="-static -s" \
        --prefix="$PREFIX" \
        --with-libgpg-error-prefix="$WORK/deps" \
        --with-libassuan-prefix="$WORK/deps" \
        --disable-ncurses \
        --disable-libsecret \
        --enable-pinentry-tty \
        --disable-pinentry-curses \
        --disable-pinentry-emacs \
        --disable-inside-emacs \
        --disable-pinentry-gtk2 \
        --disable-pinentry-gnome3 \
        --disable-pinentry-qt \
        --disable-pinentry-tqt \
        --disable-pinentry-fltk
    make -kj$NJOBS
    make install DESTDIR="$DESTDIR"
)

rm -rf "$DESTDIR$PREFIX/sbin"
rm -rf "$DESTDIR$PREFIX/share/doc"
rm -rf "$DESTDIR$PREFIX/share/info"
# cleanup
distclean

Run Code Online (Sandbox Code Playgroud)

以下是用于构建该层的 Dockerfile：

FROM public.ecr.aws/lambda/python:3.8

# the output volume to extract the build contents
VOLUME ["/opt/bin"]

RUN yum -y groupinstall 'Development Tools'
RUN yum -y install tar gzip zlib bzip2 file hostname
WORKDIR /opt
# copy the build script
COPY static-gnupg-build.sh .
# run the build script
RUN bash ./static-gnupg-build.sh
# when run output the version
ENTRYPOINT [ "/opt/bin/gpg", "--version" ]

Run Code Online (Sandbox Code Playgroud)

在图像中编译代码后，我将其复制到本地目录，压缩并发布图层： docker cp MY_DOCKER_ID:/opt/bin ./gnupg

cd ./gnupg && zip -r gnupg-layer.zip bin

发布图层：

FROM public.ecr.aws/lambda/python:3.8

# the output volume to extract the build contents
VOLUME ["/opt/bin"]

RUN yum -y groupinstall 'Development Tools'
RUN yum -y install tar gzip zlib bzip2 file hostname
WORKDIR /opt
# copy the build script
COPY static-gnupg-build.sh .
# run the build script
RUN bash ./static-gnupg-build.sh
# when run output the version
ENTRYPOINT [ "/opt/bin/gpg", "--version" ]

Run Code Online (Sandbox Code Playgroud)

我决定不使用该python-gnupg包来更好地控制确切的 GnuPG 二进制标志，因此我添加了自己的二进制包装函数：

aws lambda publish-layer-version \
    --layer-name gnupg \
    --zip-file fileb://layer-gpg2.3.zip \
    --compatible-architectures python3.8

Run Code Online (Sandbox Code Playgroud)

然后添加了导入密钥和解码功能：

def gpg_run(flags: list, subprocess_kwargs: dict):
    gpg_bin_args = [
        '/opt/bin/gpg',
        '--no-tty',
        '--yes', # don't prompt for input
        '--always-trust',   # always trust
        '--status-fd', '1', # return status to stdout
        '--homedir', '/tmp'
    ]
    gpg_bin_args.extend(flags)
    print('running cmd', ' '.join(gpg_bin_args))
    result = subprocess.run(gpg_bin_args, 
                            stdout=subprocess.PIPE, 
                            stderr=subprocess.PIPE, 
                            **subprocess_kwargs)
    return result.returncode, \
           result.stdout.decode('utf-8').split('/n'), \
           result.stderr.decode('utf-8').split('/n')

Run Code Online (Sandbox Code Playgroud)

并更新了相关的 Lambda 代码：


# ...
encrypted_path = '/tmp/encrypted.zip'
decrypted_path = '/tmp/decrypted.zip'
#...

# TODO: import the keys only needs to run once per instance 
# ideally would be moved to a singleton 
code, stdout, stderr = gpg_import_keys(bytes(MY_KEY_DATA, 'utf-8'))
if code > 0:
    raise Exception(f'gpg_import_keys failed with code {code}: {stdout} {stderr}')
print('import_keys stdout =', stdout)
print('import_keys stderr =', stderr)

# Perform decryption.
print('Performing Decryption of', encrypted_path)

code, stdout, stderr = gpg_decrypt(encrypted_path, output=decrypted_path)

if code > 0:
    raise Exception(f'gpg_decrypt failed with code {code}: {stderr}')

print('decrypt stdout =', stdout)
print('decrypt stderr =', stderr)
print('Status: OK')
print(decrypted_path, "exists :", Path(decrypted_path).exists())

Run Code Online (Sandbox Code Playgroud)

现在 Cloudwatch 日志输出符合预期，并且我已经确认解码的文件是正确的！

...
2022-11-17T09:25:22.732-06:00   running cmd:['/opt/bin/gpg', '--no-tty', '--batch', '--yes', '--always-trust', '--status-fd', '1', '--homedir', '/tmp', '--import']
2022-11-17T09:25:22.769-06:00   import_keys ok = True
2022-11-17T09:25:22.769-06:00   import_keys stdout = ['[GNUPG:] IMPORT_OK 0 XXX', '[GNUPG:] KEY_CONSIDERED XXX 0', '[GNUPG:] IMPORT_OK 16 XXX', '[GNUPG:] IMPORT_RES 1 0 0 0 1 0 0 0 0 1 0 1 0 0 0', '']
2022-11-17T09:25:22.769-06:00   import_keys stderr = ['']
2022-11-17T09:25:22.769-06:00   Performing Decryption of /tmp/test.txt.gpg
2022-11-17T09:25:22.769-06:00   running cmd: /opt/bin/gpg --no-tty --yes --always-trust --status-fd 1 --homedir /tmp --output /tmp/decrypted.zip --decrypt /tmp/encrypted.zip
2022-11-17T09:25:22.850-06:00   decrypt stdout = ['[GNUPG:] ENC_TO XXX 1 0', '[GNUPG:] KEY_CONSIDERED XXX 0', '[GNUPG:] DECRYPTION_KEY XXX -', '[GNUPG:] BEGIN_DECRYPTION', '[GNUPG:] DECRYPTION_INFO 0 9 2', '[GNUPG:] PLAINTEXT 62 1667796554 encrypted.zip', '[GNUPG:] PLAINTEXT_LENGTH 428', '[GNUPG:] DECRYPTION_OKAY', '[GNUPG:] GOODMDC', '[GNUPG:] END_DECRYPTION', '']
2022-11-17T09:25:22.850-06:00   decrypt stderr = ['gpg: encrypted with rsa2048 key, ID XXX, created 2022-11-07', ' "XXX"', '']
2022-11-17T09:25:22.850-06:00   /tmp/decrypted.zip exists: True
...

Run Code Online (Sandbox Code Playgroud)

归档时间：	2 年，9 月前
查看次数：	3105 次
最近记录：	2 年，9 月前