大家好我在大型桌子上运行凤凰计数查询时收到以下错误消息.
0: jdbc:phoenix:hadoopm1:2181> select Count(*) from PJM_DATASET;
+------------+
| COUNT(1) |
+------------+
java.lang.RuntimeException: org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Jan 09 02:18:10 CST 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=62365: row '' on table 'PJM_DATASET' at region=PJM_DATASET,,1420633295836.4394a3aa2721f87f3e6216d20ebeec44., hostname=hadoopctrl,60020,1420790733247, seqNum=27753
at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440)
at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074)
at sqlline.SqlLine.print(SqlLine.java:1735)
at sqlline.SqlLine$Commands.execute(SqlLine.java:3683)
at sqlline.SqlLine$Commands.sql(SqlLine.java:3584)
at sqlline.SqlLine.dispatch(SqlLine.java:821)
at sqlline.SqlLine.begin(SqlLine.java:699)
at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441)
at sqlline.SqlLine.main(SqlLine.java:424)
0: jdbc:phoenix:hadoopm1:2181>
请帮忙.
问题
我正在尝试与Phoenix建立二级索引.索引创建需要几个小时.这似乎是由于HBase扫描速度慢,因为我注意到以下性能:
使用HBase shell和Java扫描程序.
注意:GET(通过rowkey)操作实现了良好的性能(约0.5秒).
上下文
故障排除
基于HBase书籍(http://hbase.apache.org/book.html#performance),这是我已经检查过的内容:
1)硬件
2)JVM
3)数据
4)HBase配置
大多数配置保持不变.
一些对我来说看起来不错的params
尝试以下配置更改而不会对性能产生任何影响
5)日志没有任何用处
cat hbase-hbase-master-cox.log …
我可以让Phoenix在独立的Apache Hbase上工作
(注意,这一切都适用于RHEL6.5上的Hbase 1.0.0)
对于Hbase的Cloudera风味,但是如果没有它抛出异常,我永远不会让它工作.(甚至尝试过RHEL7 minimal as as OS)
对于Hbase 1.0,Phoenix 4.4也会发生同样的事情.
hbase(main):001:0> version
1.0.0-cdh5.4.4, rUnknown, Mon Jul 6 16:59:55 PDT 2015
堆栈跟踪:
[ec2-user@ip-172-31-60-109 phoenix-4.5.0-HBase-1.0-bin]$ bin/sqlline.py localhost:2181:/hbase
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:localhost:2181:/hbase none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:localhost:2181:/hbase
15/08/06 03:10:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/08/06 03:10:26 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
15/08/06 03:10:27 WARN ipc.CoprocessorRpcChannel: Call failed on IOException
org.apache.hadoop.hbase.DoNotRetryIOException: org.apache.hadoop.hbase.DoNotRetryIOException: SYSTEM.CATALOG: org.apache.hadoop.hbase.client.Scan.setRaw(Z)Lorg/apache/hadoop/hbase/client/Scan;
at …我是大数据的新手,我正在尝试了解持久化和检索数据的各种方法.我知道Parquet和HBase都是面向列的存储格式,但Parquet是面向文件的存储,而不是像HBase那样的数据库.我的问题是:
我对HBase和Phoenix很新.
但有没有办法可以将数据转储/导出到文本文件？如果我可以指定字段终止符,例如',','|>'等,那将是非常值得注意的.
谢谢.
我想在apache phoenix中导出现有表的模式.是否有一些命令或工具可以像show create table TABLE_NAME在mysql中那样做？
谢谢
我有一个程序,在服务器内的docker容器中将新患者插入HBase.一切正常,直到我尝试将连接IP更改为凤凰查询服务器以运行JUnit测试.我在属性文件中设置URL如下:
java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
at [Source: java.io.StringReader@1105b1f; line: 1, column: 2]
...
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: java.io.StringReader@1105b1f; line: 1, column: 2]
我不知道为什么我收到一个json解析异常.由于外部apis,调试没有帮助.我甚至不知道异常是否与我的属性文件中的URL格式,患者(以xml格式提供)或甚至与凤凰客户端相关.
我已经启动了凤凰查询服务器,看起来它可以连接到它,因为它之前抛出了"连接拒绝"异常,现在它没有.
我在我的pom.xml中添加了phoenix,如下所示:
<dependency>
<groupId>org.apache.phoenix</groupId>
<artifactId>phoenix-server</artifactId>
<version>4.7.0-HBase-1.1</version>
</dependency>
但我不知道是否必须为凤凰瘦客户端添加其他内容.如果必须的话,我找不到那种依赖关系,所以我只是假设它被包括在内.
任何帮助表示赞赏!
编辑:
这是服务器抛出的异常:
> > 2016-05-05 08:52:11,979 WARN org.eclipse.jetty.server.HttpChannel: / java.lang.RuntimeException:
> org.apache.calcite.avatica.com.google.protobuf.InvalidProtocolBufferException:
> …我正在尝试将Dataframe写入Phoenix表,但我得到了例外.
这是我的代码:
df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(collection.immutable.Map(
"zkUrl" -> "localhost:2181/hbase-unsecure",
"table" -> "TEST")).save();
例外是:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 3.0 (TID 411, ip-xxxxx-xx-xxx.ap-southeast-1.compute.internal):
java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost:2181:/hbase-unsecure;
at org.apache.phoenix.mapreduce.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:58)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1$anonfun$12.apply(PairRDDFunctions.scala:1030)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1$anonfun$12.apply(PairRDDFunctions.scala:1014)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
我已经加入phoenix-spark和phoenix-core罐子到我的pom.xml
我面临一个问题:过程工厂的数据库.采样率为50 ms时,最多有50,000个传感器.所有测量值都需要存储至少3年,并且必须支持实时查询(即用户可以查看延迟小于1秒的历史数据).我最近阅读了一篇关于时间序列数据库的文章,现有很多选项:OpenTSDB,KairosDB,InfluxDB,......
我很困惑哪一个适合这个目的？任何人都知道这个请帮助我!
更新15.06.25
今天我运行一个基于OpenTSDB的测试.我使用Virtual Box创建了一个由3个CentOS x64 VM组成的集群(1个主服务器,2个从服务器).主机配置为8 GB RAM,核心i5.主VM配置为3 GB RAM,从站配置为1.5 GB RAM.我编写了一个python程序来向OpenTSDB发送数据,如下所示:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("192.168.10.55", 4242))
start_time = time.time()
start_epoch = 1434192418;
for x in range(0, 1000000):
curr_epoch = start_epoch + x
tag1 = "put TAG_1 %d 12.9 stt=good\n" % (curr_epoch)
tag2 = "put TAG_2 %d 12.9 stt=good\n" % (curr_epoch)
tag3 = "put TAG_3 %d 12.9 stt=good\n" % (curr_epoch)
tag4 = "put TAG_4 %d 12.9 stt=good\n" % (curr_epoch)
tag5 = "put TAG_5 %d 12.9 …我是Saiku的新手.我想把saiku和凤凰融为一体.凤凰实习生与HBase联系.我创建了一个模式,当Saiku尝试加载phoenix schema xml时,Am触发了以下错误.我正在努力想弄明白.任何人都可以建议我错过了什么,做了什么.
我使用的是以下版本的s/ws:
Phoenix 4.4 HBase 1.1.2 Saiku 3.8
感谢您的支持.
请在下面找到异常跟踪:
java.lang.IllegalArgumentException: Unable to PTableType enum for value of 'MATERIALIZED VIEW'
at org.apache.phoenix.schema.PTableType.fromValue(PTableType.java:88)
at org.apache.phoenix.jdbc.PhoenixDatabaseMetaData.getTables(PhoenixDatabaseMetaData.java:1057)
at org.apache.commons.dbcp.DelegatingDatabaseMetaData.getTables(DelegatingDatabaseMetaData.java:604)
at mondrian.rolap.aggmatcher.JdbcSchema.loadTablesOfType(JdbcSchema.java:1205)
at mondrian.rolap.aggmatcher.JdbcSchema.loadTables(JdbcSchema.java:1162)
at mondrian.rolap.aggmatcher.JdbcSchema.load(JdbcSchema.java:1042)
at mondrian.rolap.RolapSchema$PhysSchema.<init>(RolapSchema.java:976)
at mondrian.rolap.RolapSchemaLoader.createSyntheticPhysicalSchema(RolapSchemaLoader.java:1479)
at mondrian.rolap.RolapSchemaLoader.validatePhysicalSchema(RolapSchemaLoader.java:684)
at mondrian.rolap.RolapSchemaLoader.loadStage2(RolapSchemaLoader.java:376)
at mondrian.rolap.RolapSchemaLoader.loadStage1(RolapSchemaLoader.java:336)
at mondrian.rolap.RolapSchemaLoader.loadStage0(RolapSchemaLoader.java:272)
at mondrian.rolap.RolapSchemaLoader.createSchema(RolapSchemaLoader.java:4304)
at mondrian.rolap.RolapSchemaPool.get(RolapSchemaPool.java:210)
at mondrian.rolap.RolapSchemaPool.get(RolapSchemaPool.java:62)
at mondrian.rolap.RolapConnection.<init>(RolapConnection.java:160)
at mondrian.rolap.RolapConnection.<init>(RolapConnection.java:84)
at mondrian.olap.DriverManager.getConnection(DriverManager.java:112)
at mondrian.olap.DriverManager.getConnection(DriverManager.java:68)
at mondrian.olap4j.MondrianOlap4jConnection.<init>(MondrianOlap4jConnection.java:153)
at mondrian.olap4j.FactoryJdbc4Plus$AbstractConnection.<init>(FactoryJdbc4Plus.java:323)
at mondrian.olap4j.FactoryJdbc41Impl$MondrianOlap4jConnectionJdbc41.<init>(FactoryJdbc41Impl.java:118)
at mondrian.olap4j.FactoryJdbc41Impl.newConnection(FactoryJdbc41Impl.java:32)
at mondrian.olap4j.MondrianOlap4jDriver.connect(MondrianOlap4jDriver.java:139)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.saiku.datasources.connection.SaikuOlapConnection.connect(SaikuOlapConnection.java:111)
at org.saiku.datasources.connection.SaikuOlapConnection.connect(SaikuOlapConnection.java:57)
at org.saiku.datasources.connection.SaikuConnectionFactory.getConnection(SaikuConnectionFactory.java:29) …