我想在Hadoop YARN集群模式下运行我的spark Job ,我使用以下命令:
spark-submit --master yarn-cluster
--driver-memory 1g
--executor-memory 1g
--executor-cores 1
--class com.dc.analysis.jobs.AggregationJob
sparkanalitic.jar param1 param2 param3
我收到错误,请提出错误,命令是否正确.我正在使用CDH 5.3.1.
Diagnostics: Application application_1424284032717_0066 failed 2 times due
to AM Container for appattempt_1424284032717_0066_000002 exited with
exitCode: 15 due to: Exception from container-launch.
Container id: container_1424284032717_0066_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit …以下是我的语法
List synchronizedpubliesdList = Collections.synchronizedList(publiesdList);
我收到的语法错误:
List is a raw type. References to generic type List<E> should be parameterized.
请建议解决方案.
我正在使用以下代码但是gettting错误消息:
javax.mail.SendFailedException: Invalid Addresses;
nested exception is:
com.sun.mail.smtp.SMTPAddressFailedException: 530 5.7.0 : Recipient address rejected: Authentication Required
at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1607)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:887)
at javax.mail.Transport.send0(Transport.java:191)
at javax.mail.Transport.send(Transport.java:120)
at MailClient.sendMail(MailClient.java:55)
at MailClient.main(MailClient.java:94)
Caused by: com.sun.mail.smtp.SMTPAddressFailedException: 530 5.7.0 : Recipient address rejected: Authentication Required
at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1505)
... 5 more
/**
*
* @author sachin
*/
import javax.mail.*;
import javax.mail.internet.*;
import javax.activation.*;
import java.io.*;
import java.util.Properties;
public class MailClient
{
public void sendMail(String mailServer, String from, String to,
String subject, String messageBody
) throws MessagingException, AddressException …我在我的ArrayList中有一些数据>我们可以将这些数据写在镶木地板文件中,如果是,那么如何,请推荐和Java程序,在此先感谢.
请注意我没有使用任何Spark,Kafka ......
我想使用shell脚本在"yyyy-mm-dd hh:mm:ss"格式中传递时间戳,我的意思是从shell脚本我将调用我的jar文件并将参数作为参数传递,所以我做的就像 -
DATE=`date +%Y-%m-%d" "%H:%M:%S`
spark-submit --jars ./mylibs/log4j-1.2.17.jar --class com.mycom.test.jobtest.agg myApp-.0.1.jar $DATE
如果我接受DATE的回应它将给出正确的日期作为预期的格式"yyyy-mm-dd hh:mm:ss"但是在java中解析时它正在考虑空间并且只考虑日期部分,并且在下面给出错误 -
Exception in thread "main" java.text.ParseException: Unparseable date: "2015-03-08"
请建议传递时间戳作为参数的最佳方法,