小编jse*_*ars的帖子

我已经安装了HTTPS来终止我的AWS ELB上的外部HTTPS连接.我现在正尝试使用带有自签名证书的HTTPS来保护我在EL2和我的后端NGINX服务器之间的连接.我已按照文档进行操作,但通过HTTPS访问服务器会导致408 HTTP超时.我似乎无法获得任何调试信息来确定失败的地方.

• 我已经确认安全组允许ELB上的ELB和NGINX之间的连接.
• 我已经确认VPC允许在ELB和EC2节点之间路由流量(同样HTTP工作正常).
• 我已经确认EC2节点上的HTTPS侦听器正在运行(我可以直接点击它而不去ELB.
• 我创建了一个类型为PublicKeyPolicyType的ELB策略,并关联了我的公钥.
• 我创建了一个tyep BackendServerAuthenticationPolicyType的ELB策略,并将其与PublicKeyPolicyType相关联.
• 我已将BackendServerAuthenticationPolicyType与ELB相关联.
• 我确保SSLNegotiationPolicyType支持我在NGINX配置中指定的算法和密码.
• 我在NGINX访问日志中看到了HTTP请求,但没有看到HTTPS请求.

有什么方法可以获得任何其他诊断信息来测试这个吗？

这是我的ELB配置:

$ aws elb describe-load-balancers --load-balancer-name <MY-ELB-NAME>

{
    "LoadBalancerDescriptions": [
        {
            "Subnets": [
                "<REDACTED>",
                "<REDACTED>",
                "<REDACTED>"
            ],
            "CanonicalHostedZoneNameID": "<REDACTED>",
            "VPCId": "<REDACTED>",
            "ListenerDescriptions": [
                {
                    "Listener": {
                        "InstancePort": 80,
                        "LoadBalancerPort": 80,
                        "Protocol": "HTTP",
                        "InstanceProtocol": "HTTP"
                    },
                    "PolicyNames": []
                },
                {
                    "Listener": {
                        "InstancePort": 443,
                        "SSLCertificateId": "<REDACTED>",
                        "LoadBalancerPort": 443,
                        "Protocol": "HTTPS",
                        "InstanceProtocol": "HTTPS"
                    },
                    "PolicyNames": [
                        "ELBSecurityPolicy-2015-05"
                    ]
                }
            ],
            "HealthCheck": {
                "HealthyThreshold": 2,
                "Interval": 30,
                "Target": …

有没有办法在使用代理服务器的企业网络上使用ez_setup.py安装Python的easy_install？目前,我收到连接超时:

Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
Traceback (most recent call last):
  File "C:\jsears\python\ez_setup.py", line 278, in <module>
    main(sys.argv[1:])
  File "C:\jsears\python\ez_setup.py", line 210, in main
    egg = download_setuptools(version, delay=0)
  File "C:\jsears\python\ez_setup.py", line 158, in download_setuptools
    src = urllib2.urlopen(url)
  File "C:\jsears\Python27\lib\urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "C:\jsears\Python27\lib\urllib2.py", line 400, in open
    response = self._open(req, data)
  File "C:\jsears\Python27\lib\urllib2.py", line 418, in _open
    '_open', req)
  File "C:\jsears\Python27\lib\urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "C:\jsears\Python27\lib\urllib2.py", line 1207, in http_open
    return self.do_open(httplib.HTTPConnection, req) …

Apache Spark pyspark.RDDAPI文档提到groupByKey()效率低下.相反,它是推荐使用reduceByKey(),aggregateByKey(),combineByKey(),或foldByKey()代替.这将导致在shuffle之前在worker中进行一些聚合,从而减少跨工作人员的数据混乱.

给定以下数据集和groupByKey()表达式,什么是等效且有效的实现(减少的跨工作者数据混洗),它不使用groupByKey(),但提供相同的结果？

dataset = [("a", 7), ("b", 3), ("a", 8)]
rdd = (sc.parallelize(dataset)
       .groupByKey())
print sorted(rdd.mapValues(list).collect())

输出:

[('a', [7, 8]), ('b', [3])]