小编Hel*_*lad的帖子

我想使用pylab的散点图函数

x = [1,2,3,4,5]
y = [2,1,3,6,7]

在这5个点中有两个聚类,索引1-2(聚类1)和索引2-4(聚类2).集群1中的点应使用标记'^',而集群2中的点应使用标记's'.所以

cluster = ['^','^','^','s','s']

我试过了

fig, ax = pl.subplots()
ax.scatter(x,y,marker=cluster)
pl.show()

这是一个玩具示例,真实数据有10000多个样本

使用pyspark在Spark集群上编程,数据很大且分片,因此无法加载到内存中或轻松检查数据的完整性

基本上它看起来像

af.b Current%20events 1 996
af.b Kategorie:Musiek 1 4468
af.b Spesiaal:RecentChangesLinked/Gebruikerbespreking:Freakazoid 1 5209
af.b Spesiaal:RecentChangesLinked/Sir_Arthur_Conan_Doyle 1 5214

维基百科数据:

我从aws S3读取它,然后尝试在pyspark intepreter中使用以下python代码构造spark Dataframe:

parts = data.map(lambda l: l.split())
wikis = parts.map(lambda p: (p[0], p[1],p[2],p[3]))


fields = [StructField("project", StringType(), True),
StructField("title", StringType(), True),
StructField("count", IntegerType(), True),
StructField("byte_size", StringType(), True)] 

schema = StructType(fields) 

df = sqlContext.createDataFrame(wikis, schema)

一切都很好,只有createDataFrame给我错误

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/spark/python/pyspark/sql/context.py", line 404, in   createDataFrame
rdd, schema = self._createFromRDD(data, schema, samplingRatio)
File …

有没有办法在原始表上添加/更改表列编码而不创建新表并从旧表中选择所有内容到新表？

使用以下命令将数据传输到S3时,普通AWS CLI是否默认使用SSL？

aws s3 cp source to destination

我在Unix上.我安装了postgresql-9.3.
当我想使用pg_ctl或启动服务器时postgres,终端给我:

The program 'postgres' is currently not installed. You can install it by typing:
sudo apt-get install postgres-xc

没有这个postgres-xc,我不能启动服务器吗？

我使用无服务器框架将python函数部署到aws lambda上

我的配置文件serverless.yml如下

frameworkVersion: "=1.27.3"

service: recipes

provider:
  name: aws
  endpointType: REGIONAL
  runtime: python3.6
  stage: dev
  region: eu-central-1
  memorySize: 512
  deploymentBucket:
    name: dfki-meta
  versionFunctions: false
  stackTags:
    Project: DFKIAPP
  # Allows updates to all resources except deleting/replacing EC2 instances
  stackPolicy:
    - Effect: Allow
      Principal: "*"
      Action: "Update:*"
      Resource: "*"
    - Effect: Deny
      Principal: "*"
      Action:
        - Update: Replace
        - Update: Delete
      Resource: "*"
      Condition:
        StringEquals:
          ResourceType:
            - AWS::EC2::Instance
  # Access to RDS and S3 Bucket
  iamRoleStatements:
    -  Effect: "Allow"
       Action: "s3:ListBucket" …

我正在使用包上传压缩文件

frameworkVersion: "=1.27.3"

service: recipes

provider:
  name: aws
  endpointType: REGIONAL
  runtime: python3.6
  stage: dev
  region: eu-central-1
  memorySize: 512
  deploymentBucket:
    name: dfki-meta
  versionFunctions: false
  stackTags:
    Project: DFKIAPP
  # Allows updates to all resources except deleting/replacing EC2 instances
  stackPolicy:
    - Effect: Allow
      Principal: "*"
      Action: "Update:*"
      Resource: "*"
    - Effect: Deny
      Principal: "*"
      Action:
        - Update: Replace
        - Update: Delete
      Resource: "*"
      Condition:
        StringEquals:
          ResourceType:
            - AWS::EC2::Instance
  # Access to RDS and S3 Bucket
  iamRoleStatements:
    -  Effect: "Allow"
       Action: "s3:ListBucket"
       Resource: "*" …

这样做很明显

 data.groupby(['A','B']).mean()

我们得到了一个多级索引,一级是"A"和"B",一列是每组的平均值

我怎么能同时拥有count(),std()？

所以结果在数据框中看起来像

A   B    mean   count   std

SNS允许订户具有以下类型的协议

HTTP/S
Lambda
SQS
Email/JSON
Application

不确定应用程序协议引用了什么

以及如何填写端点

(a example is arn:aws:sns:us-east-1:5555555555:endpoint/ADM/application-name/uuid)

我有一个大约100GB的postgres备份,并希望将其加载到欧盟法兰克福的S3并在云数据库中恢复它.

我无法访问AWS Import/Export服务.在Ubuntu笔记本电脑上

我尝试过的策略

1) management console upload, at least 2 weeks needed
2) bucket explore multi-upload, task failed due to java memory error every time
3) SDK multi-upload(boto, boto3, java SDK), do not show the progress bar. can not estimate how long it needs
4) other windows explore, do not have Linux version

将此加载到S3的最快方法是什么？或python或java中的代码片段.非常感谢