小编roy*_*roy的帖子

我正在尝试从我的docker文件创建一个docker镜像,其中包含以下内容

FROM ubuntu:14.04.4
RUN echo 'deb http://private-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.4.2.0 HDP main' >> /etc/apt/sources.list.d/HDP.list
RUN echo 'deb http://private-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/ubuntu14 HDP-UTILS main'  >> /etc/apt/sources.list.d/HDP.list
RUN echo 'deb [arch=amd64] https://apt-mo.trafficmanager.net/repos/azurecore/ trusty main' >> /etc/apt/sources.list.d/azure-public-trusty.list
RUN gpg --keyserver pgp.mit.edu --recv-keys B9733A7A07513CAD
RUN gpg -a --export 07513CAD | apt-key add -
RUN gpg --keyserver pgp.mit.edu --recv-keys B02C46DF417A0893
RUN gpg -a --export 417A0893 | apt-key add -
RUN apt-get update

哪个失败,出现以下错误

root@sbd-docker:~/ubuntu# docker build -t hdinsight .
Sending build context to Docker daemon 3.072 kB
Step 1 : FROM …

我有以下docker命令来运行容器

docker run -d --name test -v /etc/hadoop/conf:/etc/hadoop/conf -v /usr/lib/python2.7/dist-packages/hdinsight_common:/usr/lib/python2.7/dist-packages/hdinsight_common -v /etc/hive/conf/:/etc/hive/conf/ -v /etc/tez/conf/:/etc/tez/conf/ -v /usr/hdp/2.4.2.0-258/sqoop/lib/:/usr/hdp/2.4.2.0-258/sqoop/lib/ -i -t hdinsight /bin/bash

这很复杂,所以我试图创建像这样的docker-compose文件

version: '2'
services:
  hdinsight:
    image: hdinsight
    container_name: ABC
    volumes:
     - /etc/hadoop/conf:/etc/hadoop/conf
     - /usr/lib/python2.7/dist-packages/hdinsight_common:/usr/lib/python2.7/dist-packages/hdinsight_common
     - /etc/hive/conf/:/etc/hive/conf/
     - /etc/tez/conf/:/etc/tez/conf/
     - /usr/hdp/2.4.2.0-258/sqoop/lib/:/usr/hdp/2.4.2.0-258/sqoop/lib/
    entrypoint:
     - bash
    labels:
     - "HDInsight client VM"

但我不知道该到哪里-d,-i和-t从我原来的码头工人,运行命令flages

我正在运行像这样的码头组合

docker-compose -f docker-compose.yml run hdinsight

任何人都可以指出我正确的方向吗？

第一次回答后更新

我试着跑 docker-compose up -d

root@abc-docker:~/ubuntu# docker-compose up -d
Creating ABC
root@sbd-docker:~/ubuntu# docker ps -a
CONTAINER ID        IMAGE …

我正在尝试使用以下命令将数据加载到AWS redshift中

copy venue from 's3://mybucket/venue'
credentials 'aws_access_key_id=<access-key-id>;aws_secret_access_key=<secret-access-key>'
delimiter '\t';

但数据加载失败,当我检查查询部分的特定加载时,我发现它失败了,原因是"错误的UTF8十六进制序列:a4(错误3)"

有没有办法将数据加载中的错误记录跳转到redshift？

我正在 Github-Action 中运行 docker-compose。docker-compose.yml有以下服务定义postgres

  postgres:
    container_name: postgres
    image: postgres:12
    restart: always
    volumes:
      - ./test/data/init.sql:/docker-entrypoint-initdb.d/init.sql
    environment:
      POSTGRES_DB: "pgdb"
      POSTGRES_USER: "pguser"
      POSTGRES_PASSWORD: "fr2Yitl4BgX"
    ports:
      - "${POSTGRES_PORT:-5432}:5432"
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - "local-api"

但是当容器在农奴托管的 Github-Action 运行器上启动时，我看到以下内容

postgres    | 2021-12-02 19:48:33.537 UTC [414] FATAL:  role "root" does not exist
postgres    | 2021-12-02 19:48:43.984 UTC [424] FATAL:  role "root" does not exist
postgres    | 2021-12-02 19:48:54.265 UTC [433] FATAL:  role "root" does not …

我正在寻找一种通过ARM模板在Azure blob和文件共享存储中创建容器的方法.

目前我有ARM模板来配置存储帐户,但我想在ARM中创建容器.

{
    "name": "[parameters('storageAccountName')]",
    "type": "Microsoft.Storage/storageAccounts",
    "location": "[resourceGroup().location]",
    "apiVersion": "[variables('storageApiVersion')]",
    "sku": {
        "name": "[variables('storageAccountType')]"
    },
    "dependsOn": [ ],
    "tags": {
      "Environment": "[parameters('Environment')]",
      "Project": "[parameters('ProjectName')]",
      "Contact": "[parameters('ContactName')]"
    },
    "kind": "Storage",
    "properties": {
      "encryption": {
        "keySource": "Microsoft.Storage",
        "services": {
              "blob": {
                "enabled": true
              }
        }
      }
    }
  }

我已经安装了Python 2.6.6

[17:50:21 root@data.dev:~]# which python
/usr/local/bin/python

还有Python 2.7.6

[17:51:12 root@data.dev:~]# which python2.7
/usr/local/bin/python2.7

但缺少libpython2.7.so

[17:48:52 root@data.dev:~]# locate libpython2.6.so
/usr/lib64/libpython2.6.so
/usr/lib64/libpython2.6.so.1.0
/usr/lib64/python2.6/config/libpython2.6.so
[17:48:56 root@data.dev:~]# locate libpython2.7.so
[17:49:02 root@data.dev:~]#

不知道如何解决这个问题.

有人可以帮忙吗？

我想删除超过7天的文件和文件夹,所以我尝试了

[17:07:14 root@client01.abc.com:~]# find /tmp/ -mindepth 1 -maxdepth 1 -ctime +7 -exec ls -l {} \;

因此,当我运行find /tmp/ -mindepth 1 -maxdepth 1 -ctime +7 -exec ls -l {} \;它不显示任何目录,但因为find /tmp/ -mindepth 1 -maxdepth 2 -ctime +7 -exec ls -l {} \;它确实显示在subdir中的几个文件.

什么是在一个特定目录中删除超过7天的文件/文件夹的正确方法？

paths: 'frontend/**'在单个存储库中，我想为不同的路径 [和]创建多个触发器，paths: 'backend/**'并为（相同的）构建作业使用不同的参数。以下是两个工作流程。

name: Trigger Jenkins Build [ Build-Portal ]
on:
  push:
    branches: [ develop ]
    paths: 'frontend/**'
    types: [closed]
jobs:
  build:
    name: Triggering Jenkins Build [ Build-Portal ]
    runs-on: ubuntu-latest
    if: github.event.pull_request.merged == true
    steps:
    - name: Trigger Build-Portal
      uses: actions/trigger-jenkins@develop
      with:
        ...
        job_name: "Build-Portal"
        job_params: '{"FRESH_BUILD":"True", "UI":"True", "BUILD_BRANCH":"develop", "DEPLOY_DEV":"True"}'
        ...

和

name: Trigger Jenkins Build [ Build-Portal ]
on:
  push:
    branches: [ develop ]
    paths: 'backend/**'
    types: [closed]
jobs:
  build:
    name: Triggering Jenkins Build …

/usr/bin/npm" "install" "--no-optional 在 ubuntu 16.04 上给我以下错误

npm ERR! Linux 4.4.0-1098-aws
npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "install" "--no-optional"
npm ERR! node v4.2.6
npm ERR! npm  v3.5.2
npm ERR! code EMISSINGARG

npm ERR! typeerror Error: Missing required argument #1
npm ERR! typeerror     at andLogAndFinish (/usr/share/npm/lib/fetch-package-metadata.js:31:3)
npm ERR! typeerror     at fetchPackageMetadata (/usr/share/npm/lib/fetch-package-metadata.js:51:22)
npm ERR! typeerror     at resolveWithNewModule (/usr/share/npm/lib/install/deps.js:456:12)
npm ERR! typeerror     at /usr/share/npm/lib/install/deps.js:457:7
npm ERR! typeerror     at /usr/share/npm/node_modules/iferr/index.js:13:50
npm ERR! typeerror     at /usr/share/npm/lib/fetch-package-metadata.js:37:12
npm ERR! typeerror     at addRequestedAndFinish (/usr/share/npm/lib/fetch-package-metadata.js:82:5)
npm ERR! typeerror …

我们使用Cloudera Manager启用Namenode High Availability

Cloudera Manager >> HDFS >> Action> Enable High Availability >> Selected Stand By Namenode&Journal Nodes然后nameservice1

完成整个过程后,部署客户端配置.

通过列出HDFS目录(hadoop fs -ls /)从Client Machine进行测试,然后手动故障转移到备用namenode并再次列出HDFS目录(hadoop fs -ls /).这项测试完美无缺.

但是当我使用以下命令运行hadoop睡眠作业时它失败了

$ hadoop jar /opt/cloudera/parcels/CDH-4.6.0-1.cdh4.6.0.p0.26/lib/hadoop-0.20-mapreduce/hadoop-examples.jar sleep -m 1 -r 0
java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:414)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:164)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:129)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:448)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:410)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:128)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2308)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:103)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:980)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:974)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:974) …