小编Abh*_*his的帖子

你好，我正在尝试=C2*A3^B3在 python 数据框中复制这个 excel 计算。但无法将前一行的计算字段放入我的计算中。

输入数据 ：

 ftax        stax
 1.000000000     0   
 0.999959316     3   
 0.999959316     1   
 0.999959316     3   
 0.999959316     3   
 0.999959316     1

如果我将值放入 excel 并应用此公式，我会得到以下结果并希望在 python 中复制相同的结果

=C2*A3^B3在单元格 C3 和 =A2^B2单元格 C2 中

期望的输出：

ftax         stax   cal_field
1.000000000  0      1.000000000
0.999959316  3      0.999877952
0.999959316  1      0.999837272
0.999959316  3      0.999715244
0.999959316  3      0.999593230
0.999959316  1      0.999552562

我尝试过的示例代码

nav_df = pd.DataFrame(nav_cal)
nav_df[cal_filed_1] = nav_df[cal_filed].shift(1).fillna("1")

print (nav_df[cal_filed]*(nav_df[cal_filed_1])**nav_df[9]) #I think this is not the right way

我得到以下输出：

      ftax     stax cal_field
0 …

我们试图在Spark shell中执行一个简单的Scala代码来从Hbase中检索数据.Hadoop环境启用了Kerberos,我们确保执行kinit.

调用Spark Shell的步骤:

MASTER=yarn-client

DRIVER_CLASSPATH="/opt/cloudera/parcels/CDH/lib/hbase/lib/*"
DRIVER_LIBRARY_PATH="/opt/cloudera/parcels/CDH/lib/hadoop/lib/native"

spark-shell --driver-class-path "$DRIVER_CLASSPATH" --driver-library-path "$DRIVER_LIBRARY_PATH" --driver-memory 10G --executor-memory 15G --executor-cores 8 --num-executors 3 --master $MASTER

码:

import org.apache.hadoop.fs._
import org.apache.hadoop.hbase._
import org.apache.hadoop.hbase.client._
import org.apache.hadoop.hbase.io._
import org.apache.hadoop.hbase.mapreduce._
import org.apache.hadoop.hbase.util._
import org.apache.spark._

val hc = HBaseConfiguration.create
hc.addResource(new Path("file:///opt/cloudera/parcels/CDH/lib/hbase/conf/hbase-site.xml"))

hc.addResource(new Path("file:///opt/cloudera/parcels/CDH/lib/hbase/conf/core-site.xml"))

hc.set(TableInputFormat.INPUT_TABLE, "poc-customers")
val rdd = sc.newAPIHadoopRDD(hc, classOf[TableInputFormat], classOf[ImmutableBytesWritable], classOf[Result])

rdd.count

以下是ERROR

org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the location
        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:308)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:149)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:293)
        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:268)
        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:140)
        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:135)
        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:888)
        at …

我试图从文本文件下面的15间隔之后读取第6,第7,第8和第13行.

REPORT ID    : XZ-12                     ABC SERVICES COMPANY                     RUN DATE: 08/05/2014
PROGRAM      : XYZ                            SYSTEM                              PAGE: 0001
BUSINESS DAY : 08/04/14                 MONTHLY  REPORT                           TIME: 06:28:35

NAME          : FUND11
SRC NUMBER  : 3978
JOB CODE      : 80

    ABC                   DEF               RISK                AZ FUND             ASZ            ADS
    FEE                  FEE                FEE                 FEE                 FEE             FEE
================== ================== ================== ================== ================== ==================
              0.00               0.00               0.00               0.00               0.00               141.00

DATE: 08/05/2014                              

REPORT ID    : XZ-12                    ABC SERVICES COMPANY                      RUN DATE: 08/05/2014
PROGRAM      : XYZ                          SYSTEM                                PAGE: 0002
BUSINESS …