我正在尝试为我的表创建分区以便更新值.
这是我的样本数据
1,Anne,Admin,50000,A
2,Gokul,Admin,50000,B
3,Janet,Sales,60000,A
Run Code Online (Sandbox Code Playgroud)
我想更新Janet's系乙.
因此,为此,我创建了一个以Department作为分区的表.
创建外部表跟踪(EmployeeID Int,FirstName String,Designation String,Salary Int)PARTITIONED BY(Department String)行格式分隔的字段以","location"/ user/sreeveni/HIVE"终止;
但在做上述命令的同时.没有数据插入到跟踪表中.
hive>select * from trail;
OK
Time taken: 0.193 seconds
hive>desc trail;
OK
employeeid int None
firstname string None
designation string None
salary int None
department string None
# Partition Information
# col_name data_type comment
department string None
Run Code Online (Sandbox Code Playgroud)
我做错了吗?
UPDATE
正如建议我尝试将数据插入到我的表中
加载数据在路径'/ user/aibladmin/HIVE'覆盖到表跟踪分区(部门);
但它正在显示
FAILED:SemanticException [错误10096]:动态分区严格模式至少需要一个静态分区列.要关闭它,请设置hive.exec.dynamic.partition.mode = nonstrict
设置后set hive.exec.dynamic.partition.mode=nonstrict也没有工作正常.
还有别的事可做.
如何计算 Hive 中的统计模式?
让我们说在 hive 表中查找列的模式。
我们是否有任何用于计算Mode 的内置函数。
如何从命令行运行以下代码?
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.*;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.util.*;
public class MyHBase {
public static void main(String[] args) throws Exception {
Configuration conf = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(conf);
try {
HTable table = new HTable(conf, "test-table");
Put put = new Put(Bytes.toBytes("test-key"));
put.add(Bytes.toBytes("cf"), Bytes.toBytes("q"), Bytes.toBytes("value"));
table.put(put);
} finally {
admin.close();
}
}
}
Run Code Online (Sandbox Code Playgroud)
如何设置我的hbase类路径?我在类路径中得到一个巨大的字符串.
UPDATE
root# vi MyHBase.java
hbase-0.92.2 root# java -classpath `hbase classpath`:./ /var/root/MyHBase
-sh: hbase: command not found
Exception in thread "main" java.lang.NoClassDefFoundError: /var/root/MyHBase
Caused by: java.lang.ClassNotFoundException: …Run Code Online (Sandbox Code Playgroud) 我正在尝试实现reduce side join,并使用mapfile reader查找分布式缓存,但是在stderr中检查时它没有查找值,它显示以下错误,lookupfile文件已经存在于hdfs中,并且似乎正确加载进入缓存,如stdout中所示.
java.lang.IllegalArgumentException:Wrong FS:file:/ app/hadoop/tmp/mapred/local/taskTracker/distcache/-8118663285704962921_-1196516983_170706299/localhost/input/delivery_status/DeliveryStatusCodes/data,expected:hdfs:// localhost:9000 org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:390)位于org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus的org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:140) (DistributedFileSystem.java:554)atg.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:816)org.apache.hadoop.io.SequenceFile $ Reader.(SequenceFile.java:1479)org.apache .hadoop.io.SequenceFile $ Reader.(SequenceFile.java:1474)org.apache.hadoop.io.MapFile $ Reader.createDataFileReader(MapFile.java:302)at org.apache.hadoop.io.MapFile $ Reader.打开(MapFile.java:284)org.apache.hadoop.io.MapFile $ Reader.(MapFile.java:273)org.apache.hadoop.io.MapFile $ Reader.(MapFile.java:260)at org .apache.hadoop.io.MapFile $读卡器(MapFile.java:253)在mr_poc.reducerrsj.initializeDepartmentsMap(reducerrsj.java:59)mr_poc.reducerrsj.setup(reducerrsj.java:42)atg.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174)at org.位于org.apache.hadoop.mapred.Child $ 4.run的org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)中的apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) .java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:416)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java) :1190)org.apache.hadoop.mapred.Child.main(Child.java:249)mr_poc.reducerrsj.reduce(reducerrsj.java)mr_poc.reducerrsj.buildOutputValue(reducerrsj.java:83)中的java.lang.NullPointerException :127)位于org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.)的org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:177)中的mr_poc.reducerrsj.reduce(reducerrsj.java:1). java:649)at org.apache.hadoop.mapred.ReduceTask.run(Reduc)eTask.java:418)atg.apache.hadoop.mapred.Child $ 4.run(Child.java:255)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject .java:416)在org.apache.hadoop.security.
这是我的驱动程序代码,
package mr_poc;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.filecache.DistributedCache;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class driverrsj extends Configured implements Tool{
@Override …Run Code Online (Sandbox Code Playgroud) 我正在尝试使用vmware工作站在我的笔记本电脑中配置一个主机和两个从机的hadoop-1.0.3多节点集群.
当我从master 运行start-all.sh时,在主节点(namenode,datanode,tasktracker,jobtracker,secondarynamenode)中运行的守护进程,但是Datanode和tasktracker没有在slave节点上启动.密码少ssh已启用,我可以从没有pwd的masternode为主服务器和从服务器执行ssh.
请帮我解决这个问题.
我需要在Flutter 中使用Auth0,但 Auth0 站点中没有这样的 SDK。
Auth0致力于为 Flutter创建这样的SDK。
有没有人在 Flutter 中使用过 Auth0 或者你有什么建议?
我试图使用hadoop找到任何给定点的总和,我遇到的问题是从单个reducer中获取给定键的所有值.看起来像这样.
减速器:
public static class Reduce extends MapReduceBase implements
Reducer<Text, IntWritable, Text, DoubleWritable> {
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, DoubleWritable> output, Reporter reporter)
throws IOException {
Text word = new Text();
Iterator<IntWritable> tr = values;
IntWritable v;
while (tr.hasNext()) {
v = tr.next();
Iterator<IntWritable> td = values;
while (td.hasNext()) {
IntWritable u = td.next();
double sum = u+v;
word.set( u + " + " + v);
output.collect(word, new DoubleWritable(sum));
}
}
}
}
Run Code Online (Sandbox Code Playgroud)
我试图创建Iterator变量的两个副本,以便我可以通过第二个迭代器的所有值,而我从前一个迭代器获取单个值(两个while循环),但两个迭代器保持相同的值所有时间.
我不确定这是否是正确的方法,任何帮助都非常感谢.
谢谢,
Tsegay
我正在尝试检查文件内容是否为空。我有一个内容为空的源文件。我尝试了不同的选择。但没有什么对我有用。
这是我的代码:
Path in = new Path(source);
/*
* Check if source is empty
*/
BufferedReader br = null;
try {
br = new BufferedReader(new InputStreamReader(fs.open(in)));
} catch (IOException e) {
e.printStackTrace();
}
try {
if (br.readLine().length() == 0) {
/*
* Empty file
*/
System.out.println("In empty");
System.exit(0);
}
else{
System.out.println("not empty");
}
} catch (IOException e) {
e.printStackTrace();
}
Run Code Online (Sandbox Code Playgroud)
我试过使用 -
1. br.readLine().length() == 0
2. br.readLine() == null
3. br.readLine().isEmpty()
Run Code Online (Sandbox Code Playgroud)
以上所有内容都不是空的。我需要使用 -
BufferedReader br = null;
try { …Run Code Online (Sandbox Code Playgroud) double[][] data_array
SimpleMatrix dataMatrix = new SimpleMatrix(data_array);
SimpleMatrix omegaMatrix = new SimpleMatrix(omega);
SimpleMatrix cMatrix = dataMatrix.mult(omegaMatrix);
System.out.println("Multiplied");
cMatrix.print();
Run Code Online (Sandbox Code Playgroud)
我正在使用EJML库进行矩阵运算。
1.How to convert a simple matrix back to double[][].
Run Code Online (Sandbox Code Playgroud)
上面的结果是一个 1 x 1 矩阵。
2. Can we store this value in a double variable?
Run Code Online (Sandbox Code Playgroud) 当我在mapreduce模式下运行pig时,我收到ConnectionRefused错误.
详细信息:
我已经从tarball(pig-0.14)安装了Pig,并在bashrc中导出了类路径.
我已经启动并运行了所有Hadoop(hadoop-2.5)守护进程(由JPS确认).
[root@localhost sbin]# jps
2272 Jps
2130 DataNode
2022 NameNode
2073 SecondaryNameNode
2238 NodeManager
2190 ResourceManager
Run Code Online (Sandbox Code Playgroud)
我在mapreduce模式下运行pig:
[root@localhost sbin]# pig
grunt> file = LOAD '/input/pig_input.csv' USING PigStorage(',') AS (col1,col2,col3);
grunt> dump file;
Run Code Online (Sandbox Code Playgroud)
然后我收到错误:
java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getTaskReports(HadoopShims.java:231)
at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:352)
at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:233)
at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:165)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:360)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
at …Run Code Online (Sandbox Code Playgroud)