直接从 Python 提交 Google Cloud ML Engine 作业

Question

我有一个 Keras .h5 模型，我一直在本地训练它，但是现在希望通过 Google Cloud ML-Engine 自动化整个过程。

我已将所有 GCloud Storage 存储桶设置为可从应用程序访问，并且我已阅读有关配置作业以提交Keras模型以在 GCloud ML-Engine 中进行训练的信息。但是，所有这些教程（包括 google cloud ml-engine 上的文档）都声明要运行作业最好从命令行运行gcloud ml-engine jobs submit training。

不过，我知道 Google Cloud 有一个 Python 客户端库，但它的文档似乎有点不透明。

有人知道我是否可以完全从 python 文件本身提交模型的训练（通过直接 API 调用或通过 Google 客户端库）？我问是因为我希望把它变成一个完全自动化的、托管的 Flask 网络应用程序，用于模型训练，所以它需要尽可能地放手。

Answer 1

There is indeed a way of submitting jobs to Cloud ML Engine from a Python script. You can use the Google Python API Client Library for that purpose, and in the link I shared you can have a look at a close explanation of how the API calls. There's a command-by-command explanation, and at the end an example of how to put everything together. In order to work with the library, you will have to install it first, as explained in this other page.

Then, the method you are interested in (for submitting jobs) is cloudml.projects.jobs.create(), and you can find detailed information on how to call it in the developers page. I think you might be interested in playing around with the REST API first, in order to get familiar with how it works; you can do so through the APIs Explorer. Below there's an example of a body used to make the API call:

training_inputs = {'scaleTier': 'CUSTOM',
    'masterType': 'complex_model_m',
    'workerType': 'complex_model_m',
    'parameterServerType': 'large_model',
    'workerCount': 9,
    'parameterServerCount': 3,
    'packageUris': ['gs://<YOUR_TRAINER_PATH>/package-0.0.0.tar.gz'],
    'pythonModule': 'trainer.task',
    'args': ['--arg1', 'value1', '--arg2', 'value2'],
    'region': '<REGION>',
    'jobDir': 'gs://<YOUR_TRAINING_PATH>',
    'runtimeVersion': '1.4'}

job_spec = {'jobId': my_job_name, 'trainingInput': training_inputs}

You should, adapt it to the specifications of your model. Once it is ready, you can have a look at this page explaining how to submit a training job using Python, but in short, it should be something like this:

from oauth2client.client import GoogleCredentials
from googleapiclient import discovery
from googleapiclient import errors

project_name = 'my_project_name'
project_id = 'projects/{}'.format(project_name)

credentials = GoogleCredentials.get_application_default()

cloudml = discovery.build('ml', 'v1', credentials=credentials)

request = cloudml.projects().jobs().create(body=job_spec, parent=project_id)

try:
    response = request.execute()
    # Handle a successful request

except errors.HttpError, err:
    logging.error('There was an error creating the training job.'
                  ' Check the details:')
    logging.error(err._get_reason())

You should be able to run this code in order to submit a Cloud ML Engine job through a Python script.

我希望这会有所帮助并减轻您提到的文档的不透明度。