小编roy*_*roy的帖子

我们使用Cloudera Manager启用Namenode High Availability

Cloudera Manager >> HDFS >> Action> Enable High Availability >> Selected Stand By Namenode&Journal Nodes然后nameservice1

完成整个过程后,部署客户端配置.

通过列出HDFS目录(hadoop fs -ls /)从Client Machine进行测试,然后手动故障转移到备用namenode并再次列出HDFS目录(hadoop fs -ls /).这项测试完美无缺.

但是当我使用以下命令运行hadoop睡眠作业时它失败了

$ hadoop jar /opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop-0.20-mapreduce/hadoop-examples.jar sleep -m 1 -r 0
java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:448)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:410)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:128)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2308)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:980)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:974)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:974) …

在80%已满的YARN集群中,我们看到一些纱线节点管理器被标记为不健康.在挖掘日志之后我发现了它,因为数据目录的磁盘空间已满90%.有以下错误

2015-02-21 08:33:51,590 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Node hdp009.abc.com:8041 reported UNHEALTHY with details: 4/4 local-dirs are bad: /data3/yarn/nm,/data2/yarn/nm,/data4/yarn/nm,/data1/yarn/nm;
2015-02-21 08:33:51,590 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: hdp009.abc.com:8041 Node Transitioned from RUNNING to UNHEALTHY

我试图了解纱线如何标记节点不健康&有没有办法改变门槛？

谢谢

我有 pgAdmin ( dpage/pgadmin4:4.29) 容器在 Kubernetes 中运行。作为主用户，我添加了数据库连接并共享连接。作为主用户，我可以在没有密码的情况下断开/重新连接到数据库。

但对于我创建的其他用户，即使他们在连接时选择了“保存密码”选项，这些用户的密码也不会被保存。连接数据库时，pgAdmin 不断要求输入密码。

我的设置中缺少什么。

我们在Vertica中有数据集,Tableau正在从vertica查询仪表板的数据(40亿条记录),如下所示:

所有列表和图形都是表格中的单独工作表,并使用与Vertica DB相同的连接.每个列表都是DB中的一列,列表是数据集各自列中项目数的降序.图表也与列表相同,但计算方式略有不同.开始日期和结束日期是数据要查询的日期范围,如数据连接过滤器,它将查询限制为过去一周,上个月等的固定数据量示例.

但是我得到了这个错误:

Vertica] [VerticaDSII](10)查询准备期间发生错误:在同一连接上无法激活多个命令.在启动另一个命令之前,请考虑增加ResultBufferSize或获取所有结果.

是这个问题的解决方法还是更好的方法来做到这一点

我想通过Nginx将S3的可下载文件提供给内部人员。

我试过了rufuspollock/s3-bucket-listing，效果很好。但这需要公开。我不想公开存储桶。

还有其他方法可以做到这一点吗？

我试图使用PowerShell从JSON对象中提取值,我有以下JSON:

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
     "clusterName": {
      "value": "hbasehd35"
    },
     "location": {
      "value": "East US 2"
    },
     "clusterType": {
      "value": "hbase"
    },
     "clusterVersion": {
      "value": "3.5"
    },
     "clusterWorkerNodeCount": {
      "value": 5
    },
     "subnetName": {
      "value": "hbase-subnet"
    },
     "headNodeSize": {
      "value": "Standard_D12_v2"
    },
     "workerNodeSize": {
      "value": "Standard_D12_v2"
    },
     "zookeeperSize": {
      "value": "Large"
    },
     "clusterStorageAccountName": {
      "value": "hbasestorage"
    },
     "storageAccountType": {
      "value": "Standard_GRS"
    },
     "Environment": {
      "value": "test"
    }
  }
}

在这里,我想clusterStorageAccountName使用powershell从此文件中提取,并将其分配给变量.

有人知道怎么做吗 ？

我正在尝试使用rest api 通过livy 提交spark 作业。但是，如果我多次运行相同的脚本，它会运行具有不同作业 ID 的作业的多个实例。我正在寻找一种方法，在开始新作业之前终止以相同名称运行的 Spark/Yarn 作业。Livy文档说（https://github.com/cloudera/livy#batch）删除批处理作业，但livy会话不返回应用程序名称，只返回应用程序id。

还有另一种方法可以做到这一点吗？

我试图在mac mini上启动jenkins slave跟随 /Library/LaunchDaemons/com.jenkins.ci.plist

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
        <key>Label</key>
        <string>com.jenkins.ci</string>
        <key>UserName</key>
        <string>jenkins</string>
        <key>SessionCreate</key>
        <true/>
        <key>ProgramArguments</key>
        <array>
            <string>/usr/bin/java</string>
            <string>-Djava.awt.headless=true</string>
            <string>-jar</string>
            <string>/Users/jenkins/agent.jar</string>
            <string>-jnlpUrl</string>
            <string>http://jenkins2.domain.net:8080/computer/jenkins-mac/slave-agent.jnlp</string>
            <string>-secret</string>
            <string>23erft6yhujnhyujnbftyujbvcdrtyhbvcxswedaw</string>
            <string>-workDir</string>
            <string>"/Users/jenkins/jenkins_slave/"</string>
        </array>
        <key>RunAtLoad</key>
        <true/>
        <key>KeepAlive</key>
        <true/>
        <key>StandardErrorPath</key>
        <string>/Users/jenkins/error.log</string>
        <key>StandardOutPath</key>
        <string>/Users/jenkins/stdout.log</string>
    </dict>
    </plist>

然后 sudo launchctl load /Library/LaunchDaemons/com.jenkins.ci.plist

但是在 /Users/jenkins/error.log

我知道了

    Exception in thread "main" java.io.IOException: The specified working directory should be fully accessible to the remoting executable (RWX): "/Users/jenkins/jenkins_slave/"
        at org.jenkinsci.remoting.engine.WorkDirManager.verifyDirectory(WorkDirManager.java:249)
        at org.jenkinsci.remoting.engine.WorkDirManager.initializeWorkDir(WorkDirManager.java:202)
        at hudson.remoting.Launcher.run(Launcher.java:300)
        at …

我正在尝试通过jenkins管道（Jenkinsfile）中的以下内容访问存储在Jenkins中的AWS凭证

steps {
            withCredentials([usernamePassword(credentialsId: 'eb1092d1-0f06-4bf9-93c7-32e5f7b9ef76', accessKeyVariable: 'AWS_ACCESS_KEY_ID', secretKeyVariable: 'AWS_SECRET_ACCESS_KEY')]) {
                sh 'echo $AWS_ACCESS_KEY_ID'
                sh 'echo $AWS_SECRET_ACCESS_KEY'
            }
      }

其错误

org.jenkinsci.plugins.credentialsbinding.impl.CredentialNotFoundException: Could not find credentials entry with ID 'eb1092d1-0f06-4bf9-93c7-32e5f7b9ef76'
at org.jenkinsci.plugins.credentialsbinding.MultiBinding.getCredentials(MultiBinding.java:153)
at org.jenkinsci.plugins.credentialsbinding.impl.UsernamePasswordMultiBinding.bind(UsernamePasswordMultiBinding.java:76)
at org.jenkinsci.plugins.credentialsbinding.impl.BindingStep$Execution.start(BindingStep.java:114)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:270)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:178)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
at sun.reflect.GeneratedMethodAccessor3000.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
at WorkflowScript.run(WorkflowScript:16)

与 …

我正在尝试按照此文档https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-access-logs.html为 AWS 网络 ELB 配置访问日志

我指定了存储桶名称和前缀，然后选择了 Create this location for me

以下存储桶策略是为存储桶自动生成的

{
    "Version": "2012-10-17",
    "Id": "AWSConsole-AccessLogs-Policy-xxxxxxxxxxxxx",
    "Statement": [
        {
            "Sid": "AWSConsoleStmt-xxxxxxxxxxxxx",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::bucket_name/prefix/AWSLogs/123456789012/*"
        },
        {
            "Sid": "AWSLogDeliveryWrite",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::bucket_name/prefix/AWSLogs/123456789012/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        },
        {
            "Sid": "AWSLogDeliveryAclCheck",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::bucket_name"
        }
    ]
}

存储桶已创建，但在AWSLogs/xxxxxxxx下面没有生成任何日志文件。我在这里缺少什么吗？