尝试使用此代码在TLS上运行TLS时,为什么会出现握手失败？

Question

我尝试使用twisted.protocols.tls一个使用内存BIO的OpenSSL接口来实现一个可以通过TLS运行TLS的协议.

我实现此作为协议封装器的是大多看起来像一个普通的TCP传输,但其具有startTLS与stopTLS用于添加和去除分别TLS的层的方法.这适用于第一层TLS.如果我在"原生"Twisted TLS传输上运行它也可以正常工作.但是,如果我尝试使用startTLS此包装器提供的方法添加第二个TLS层,则会立即出现握手错误,并且连接最终会处于某种未知的不可用状态.

包装器和让它工作的两个帮助器看起来像这样:

from twisted.python.components import proxyForInterface
from twisted.internet.error import ConnectionDone
from twisted.internet.interfaces import ITCPTransport, IProtocol
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
from twisted.protocols.policies import ProtocolWrapper, WrappingFactory

class TransportWithoutDisconnection(proxyForInterface(ITCPTransport)):
    """
    A proxy for a normal transport that disables actually closing the connection.
    This is necessary so that when TLSMemoryBIOProtocol notices the SSL EOF it
    doesn't actually close the underlying connection.

    All methods except loseConnection are proxied directly to the real transport.
    """
    def loseConnection(self):
        pass


class ProtocolWithoutConnectionLost(proxyForInterface(IProtocol)):
    """
    A proxy for a normal protocol which captures clean connection shutdown
    notification and sends it to the TLS stacking code instead of the protocol.
    When TLS is shutdown cleanly, this notification will arrive.  Instead of telling
    the protocol that the entire connection is gone, the notification is used to
    unstack the TLS code in OnionProtocol and hidden from the wrapped protocol.  Any
    other kind of connection shutdown (SSL handshake error, network hiccups, etc) are
    treated as real problems and propagated to the wrapped protocol.
    """
    def connectionLost(self, reason):
        if reason.check(ConnectionDone):
            self.onion._stopped()
        else:
            super(ProtocolWithoutConnectionLost, self).connectionLost(reason)


class OnionProtocol(ProtocolWrapper):
    """
    OnionProtocol is both a transport and a protocol.  As a protocol, it can run over
    any other ITransport.  As a transport, it implements stackable TLS.  That is,
    whatever application traffic is generated by the protocol running on top of
    OnionProtocol can be encapsulated in a TLS conversation.  Or, that TLS conversation
    can be encapsulated in another TLS conversation.  Or **that** TLS conversation can
    be encapsulated in yet *another* TLS conversation.

    Each layer of TLS can use different connection parameters, such as keys, ciphers,
    certificate requirements, etc.  At the remote end of this connection, each has to
    be decrypted separately, starting at the outermost and working in.  OnionProtocol
    can do this itself, of course, just as it can encrypt each layer starting with the
    innermost.
    """
    def makeConnection(self, transport):
        self._tlsStack = []
        ProtocolWrapper.makeConnection(self, transport)


    def startTLS(self, contextFactory, client, bytes=None):
        """
        Add a layer of TLS, with SSL parameters defined by the given contextFactory.

        If *client* is True, this side of the connection will be an SSL client.
        Otherwise it will be an SSL server.

        If extra bytes which may be (or almost certainly are) part of the SSL handshake
        were received by the protocol running on top of OnionProtocol, they must be
        passed here as the **bytes** parameter.
        """
        # First, create a wrapper around the application-level protocol
        # (wrappedProtocol) which can catch connectionLost and tell this OnionProtocol 
        # about it.  This is necessary to pop from _tlsStack when the outermost TLS
        # layer stops.
        connLost = ProtocolWithoutConnectionLost(self.wrappedProtocol)
        connLost.onion = self
        # Construct a new TLS layer, delivering events and application data to the
        # wrapper just created.
        tlsProtocol = TLSMemoryBIOProtocol(None, connLost, False)
        tlsProtocol.factory = TLSMemoryBIOFactory(contextFactory, client, None)

        # Push the previous transport and protocol onto the stack so they can be
        # retrieved when this new TLS layer stops.
        self._tlsStack.append((self.transport, self.wrappedProtocol))

        # Create a transport for the new TLS layer to talk to.  This is a passthrough
        # to the OnionProtocol's current transport, except for capturing loseConnection
        # to avoid really closing the underlying connection.
        transport = TransportWithoutDisconnection(self.transport)

        # Make the new TLS layer the current protocol and transport.
        self.wrappedProtocol = self.transport = tlsProtocol

        # And connect the new TLS layer to the previous outermost transport.
        self.transport.makeConnection(transport)

        # If the application accidentally got some bytes from the TLS handshake, deliver
        # them to the new TLS layer.
        if bytes is not None:
            self.wrappedProtocol.dataReceived(bytes)


    def stopTLS(self):
        """
        Remove a layer of TLS.
        """
        # Just tell the current TLS layer to shut down.  When it has done so, we'll get
        # notification in *_stopped*.
        self.transport.loseConnection()


    def _stopped(self):
        # A TLS layer has completely shut down.  Throw it away and move back to the
        # TLS layer it was wrapping (or possibly back to the original non-TLS
        # transport).
        self.transport, self.wrappedProtocol = self._tlsStack.pop()

我有简单的客户端和服务器程序来执行此操作,可从launchpad(bzr branch lp:~exarkun/+junk/onion)获得.当我用它来调用startTLS上面的方法两次,没有干预调用时stopTLS,会出现这个OpenSSL错误:

OpenSSL.SSL.Error: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'unknown protocol')]

为什么出问题？

Answer 1

至少有两个问题OnionProtocol:

1. 的最内 TLSMemoryBIOProtocol成为wrappedProtocol,当它应该是最外面的 ;
2. ProtocolWithoutConnectionLost不会弹出任何TLSMemoryBIOProtocols OnionProtocol的堆栈,因为connectionLost只有在FileDescriptors doRead或doWrite方法返回断开原因后才会调用它.

我们无法在不改变OnionProtocol管理堆栈的方式的情况下解决第一个问题,在找出新的堆栈实现之前我们无法解决第二个问题.不出所料,正确的设计是Twisted中数据流动方式的直接结果,因此我们将从一些数据流分析开始.

Twisted表示与任一个twisted.internet.tcp.Server或的实例建立的连接twisted.internet.tcp.Client.由于我们程序中唯一的交互发生在stoptls_client,我们只考虑Client进出实例的数据流.

让我们用一个最小的LineReceiver客户端进行预热,该客户端回送从端口9999上的本地服务器接收的线路:

from twisted.protocols import basic
from twisted.internet import defer, endpoints, protocol, task

class LineReceiver(basic.LineReceiver):
    def lineReceived(self, line):
        self.sendLine(line)

def main(reactor):
    clientEndpoint = endpoints.clientFromString(
        reactor, "tcp:localhost:9999")
    connected = clientEndpoint.connect(
        protocol.ClientFactory.forProtocol(LineReceiver))
    def waitForever(_):
        return defer.Deferred()
    return connected.addCallback(waitForever)

task.react(main)

建立连接后,a Client成为我们LineReceiver协议的传输并调解输入和输出:

从服务器的新数据使反应器调用Client的doRead方法,这反过来又通过什么它收到LineReceiver的dataReceived方法.最后,当至少有一行可用时LineReceiver.dataReceived调用LineReceiver.lineReceived.

我们的应用程序通过调用将一行数据发送回服务器LineReceiver.sendLine.这将调用write绑定到协议实例的传输,该Client实例与处理传入数据的实例相同. Client.write安排数据由反应堆Client.doWrite发送,同时实际通过套接字发送数据.

我们已经准备好了解一个OnionClient从未调用过的行为startTLS:

OnionClients被包裹在OnionProtocols中,这是我们尝试嵌套TLS的关键.作为一个子类twisted.internet.policies.ProtocolWrapper,一个实例OnionProtocol是一种协议传输三明治; 它将自身呈现为较低级别传输的协议,并作为协议的传输,它通过在连接时建立的伪装来包装WrappingFactory.

现在,Client.doRead调用OnionProtocol.dataReceived代理数据到的OnionClient.作为OnionClient运输,OnionProtocol.write接受从其发送OnionClient.sendLine和代理它们的线路Client,它自己的运输.这是a ProtocolWrapper,它的包装协议和它自己的传输之间的正常交互,因此数据自然地流入和流出每个数据而没有任何麻烦.

OnionProtocol.startTLS有所不同.它试图在已建立的协议传输对之间插入一个新的ProtocolWrapper- 恰好是TLSMemoryBIOProtocol- .这似乎很容易:a 将上层协议存储为其属性,将代理和其他属性存储到其自己的传输中. 应该能够通过修补自己的实例并将以下实例注入到连接中的新包装:ProtocolWrapperwrappedProtocolwritestartTLSTLSMemoryBIOProtocolOnionClientwrappedProtocoltransport

def startTLS(self):
    ...
    connLost = ProtocolWithoutConnectionLost(self.wrappedProtocol)
    connLost.onion = self
    # Construct a new TLS layer, delivering events and application data to the
    # wrapper just created.
    tlsProtocol = TLSMemoryBIOProtocol(None, connLost, False)

    # Push the previous transport and protocol onto the stack so they can be
    # retrieved when this new TLS layer stops.
    self._tlsStack.append((self.transport, self.wrappedProtocol))
    ...
    # Make the new TLS layer the current protocol and transport.
    self.wrappedProtocol = self.transport = tlsProtocol

这是第一次调用后的数据流startTLS:

正如预期的那样,传送到的新数据OnionProtocol.dataReceived被路由到TLSMemoryBIOProtocol存储在其上的数据_tlsStack,该数据将解密的明文传递给OnionClient.dataReceived. OnionClient.sendLine还将其数据传递给TLSMemoryBIOProtocol.write,对其进行加密并将得到的密文发送给OnionProtocol.write然后Client.write.

不幸的是,这个方案在第二次调用后失败了startTLS.根本原因是这一行:

    self.wrappedProtocol = self.transport = tlsProtocol

每次调用startTLS替换wrappedProtocol用的最里面 TLSMemoryBIOProtocol,即使通过接收到的数据Client.doRead被加密的最外层:

在transportS,然而,正确的嵌套. OnionClient.sendLine只能调用它的传输write- 也就是说OnionProtocol.write- 所以OnionProtocol应该用transport最内层替换它TLSMemoryBIOProtocol以确保写入连续嵌套在附加的加密层中.

该溶液中,然后,是确保数据流经第一 TLSMemoryBIOProtocol上_tlsStack到下依次一个,从而使加密每一层在它施加相反的顺序被剥离:

_tlsStack鉴于这一新要求,代表列表似乎不那么自然.幸运的是,线性表示传入的数据流表明了一种新的数据结构:

输入数据的错误和正确流程都类似于单链接列表,wrappedProtocol作为ProtocolWrapper下一个链接并protocol充当Clients.该列表应该从下降OnionProtocol并始终以OnionClient.发生该错误是因为违反了排序不变量.

一个单链表可以很好地将协议推送到堆栈上但是很难将它们弹出,因为它需要从头部向下移动到节点才能删除.当然,每次收到数据时都会发生这种遍历,因此关注的是额外遍历所隐含的复杂性,而不是最坏情况下的时间复杂度.幸运的是,该列表实际上是双重关联的:

该transport属性将每个嵌套协议与其前任协议相链接,以便transport.write在最终通过网络发送数据之前,可以连续降低加密级别.我们有两个哨兵来帮助管理清单:Client必须始终位于顶部,并且OnionClient必须始终位于底部.

把两者放在一起,我们最终得到这个:

from twisted.python.components import proxyForInterface
from twisted.internet.interfaces import ITCPTransport
from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
from twisted.protocols.policies import ProtocolWrapper, WrappingFactory


class PopOnDisconnectTransport(proxyForInterface(ITCPTransport)):
    """
    L{TLSMemoryBIOProtocol.loseConnection} shuts down the TLS session
    and calls its own transport's C{loseConnection}.  A zero-length
    read also calls the transport's C{loseConnection}.  This proxy
    uses that behavior to invoke a C{pop} callback when a session has
    ended.  The callback is invoked exactly once because
    C{loseConnection} must be idempotent.
    """
    def __init__(self, pop, **kwargs):
        super(PopOnDisconnectTransport, self).__init__(**kwargs)
        self._pop = pop

    def loseConnection(self):
        self._pop()
        self._pop = lambda: None


class OnionProtocol(ProtocolWrapper):
    """
    OnionProtocol is both a transport and a protocol.  As a protocol,
    it can run over any other ITransport.  As a transport, it
    implements stackable TLS.  That is, whatever application traffic
    is generated by the protocol running on top of OnionProtocol can
    be encapsulated in a TLS conversation.  Or, that TLS conversation
    can be encapsulated in another TLS conversation.  Or **that** TLS
    conversation can be encapsulated in yet *another* TLS
    conversation.

    Each layer of TLS can use different connection parameters, such as
    keys, ciphers, certificate requirements, etc.  At the remote end
    of this connection, each has to be decrypted separately, starting
    at the outermost and working in.  OnionProtocol can do this
    itself, of course, just as it can encrypt each layer starting with
    the innermost.
    """

    def __init__(self, *args, **kwargs):
        ProtocolWrapper.__init__(self, *args, **kwargs)
        # The application level protocol is the sentinel at the tail
        # of the linked list stack of protocol wrappers.  The stack
        # begins at this sentinel.
        self._tailProtocol = self._currentProtocol = self.wrappedProtocol


    def startTLS(self, contextFactory, client, bytes=None):
        """
        Add a layer of TLS, with SSL parameters defined by the given
        contextFactory.

        If *client* is True, this side of the connection will be an
        SSL client.  Otherwise it will be an SSL server.

        If extra bytes which may be (or almost certainly are) part of
        the SSL handshake were received by the protocol running on top
        of OnionProtocol, they must be passed here as the **bytes**
        parameter.
        """
        # The newest TLS session is spliced in between the previous
        # and the application protocol at the tail end of the list.
        tlsProtocol = TLSMemoryBIOProtocol(None, self._tailProtocol, False)
        tlsProtocol.factory = TLSMemoryBIOFactory(contextFactory, client, None)

        if self._currentProtocol is self._tailProtocol:
            # This is the first and thus outermost TLS session.  The
            # transport is the immutable sentinel that no startTLS or
            # stopTLS call will move within the linked list stack.
            # The wrappedProtocol will remain this outermost session
            # until it's terminated.
            self.wrappedProtocol = tlsProtocol
            nextTransport = PopOnDisconnectTransport(
                original=self.transport,
                pop=self._pop
            )
            # Store the proxied transport as the list's head sentinel
            # to enable an easy identity check in _pop.
            self._headTransport = nextTransport
        else:
            # This a later TLS session within the stack.  The previous
            # TLS session becomes its transport.
            nextTransport = PopOnDisconnectTransport(
                original=self._currentProtocol,
                pop=self._pop
            )

        # Splice the new TLS session into the linked list stack.
        # wrappedProtocol serves as the link, so the protocol at the
        # current position takes our new TLS session as its
        # wrappedProtocol.
        self._currentProtocol.wrappedProtocol = tlsProtocol
        # Move down one position in the linked list.
        self._currentProtocol = tlsProtocol
        # Expose the new, innermost TLS session as the transport to
        # the application protocol.
        self.transport = self._currentProtocol
        # Connect the new TLS session to the previous transport.  The
        # transport attribute also serves as the previous link.
        tlsProtocol.makeConnection(nextTransport)

        # Left over bytes are part of the latest handshake.  Pass them
        # on to the innermost TLS session.
        if bytes is not None:
            tlsProtocol.dataReceived(bytes)


    def stopTLS(self):
        self.transport.loseConnection()


    def _pop(self):
        pop = self._currentProtocol
        previous = pop.transport
        # If the previous link is the head sentinel, we've run out of
        # linked list.  Ensure that the application protocol, stored
        # as the tail sentinel, becomes the wrappedProtocol, and the
        # head sentinel, which is the underlying transport, becomes
        # the transport.
        if previous is self._headTransport:
            self._currentProtocol = self.wrappedProtocol = self._tailProtocol
            self.transport = previous
        else:
            # Splice out a protocol from the linked list stack.  The
            # previous transport is a PopOnDisconnectTransport proxy,
            # so first retrieve proxied object off its original
            # attribute.
            previousProtocol = previous.original
            # The previous protocol's next link becomes the popped
            # protocol's next link
            previousProtocol.wrappedProtocol = pop.wrappedProtocol
            # Move up one position in the linked list.
            self._currentProtocol = previousProtocol
            # Expose the new, innermost TLS session as the transport
            # to the application protocol.
            self.transport = self._currentProtocol



class OnionFactory(WrappingFactory):
    """
    A L{WrappingFactory} that overrides
    L{WrappingFactory.registerProtocol} and
    L{WrappingFactory.unregisterProtocol}.  These methods store in and
    remove from a dictionary L{ProtocolWrapper} instances.  The
    C{transport} patching done as part of the linked-list management
    above causes the instances' hash to change, because the
    C{__hash__} is proxied through to the wrapped transport.  They're
    not essential to this program, so the easiest solution is to make
    them do nothing.
    """
    protocol = OnionProtocol

    def registerProtocol(self, protocol):
        pass


    def unregisterProtocol(self, protocol):
        pass

(这也可以在GitHub上找到.)

第二个问题的解决方案在于PopOnDisconnectTransport.原始代码尝试从堆栈中弹出TLS会话connectionLost,但由于只connectionLost调用了已关闭的文件描述符,因此无法删除未关闭底层套接字的已停止的TLS会话.

At the time of this writing, TLSMemoryBIOProtocol calls its transport's loseConnection in exactly two places: _shutdownTLS and _tlsShutdownFinished. _shutdownTLS is called on active closes (loseConnection, abortConnection, unregisterProducer and after loseConnection and all pending writes have been flushed), while _tlsShutdownFinished is called on passive closes (handshake failures, empty reads, read errors, and write errors). This all means that both sides of a closed connection can pop stopped TLS sessions off the stack during loseConnection. PopOnDisconnectTransport does this idempotently because loseConnection is generally idempotent, and TLSMemoryBIOProtocol certainly expects it to be.

放置堆栈管理逻辑的缺点loseConnection是它取决于实现的细节TLSMemoryBIOProtocol.通用解决方案需要跨越多个Twisted级别的新API.

在那之前,我们仍然坚持另一个Hyrum定律的例子.

Answer 2

您可能需要在启动之前通知远程设备您希望启动一个环境并为第二层分配资源（如果该设备具有此功能）。