我不断收到 UnexpectedStatusException: AWS sagemaker 中的 HyperParameterTuning 作业错误

Question

正如问题中提到的，我不断收到 UnexpectedStatusException：HyperParameterTuning 作业 xgboost-211***-1631 错误：失败。原因：尝试5次后仍未成功。有关更多详细信息，请通过列出超参数调整作业的训练作业来查看训练作业失败情况。

我根据https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html研究了参数范围研究了参数范围，以确保范围良好并且看起来没问题。数据绝对是好的，因为我可以训练模型但无法调整它。

这是我正在使用的代码：

import sagemaker
import boto3
import numpy as np                                # For matrix operations and numerical processing
import pandas as pd                               # For munging tabular data
import os 
from sagemaker.tuner import IntegerParameter, CategoricalParameter, ContinuousParameter, HyperparameterTuner
from sagemaker.session import TrainingInput
from sagemaker.debugger import Rule, rule_configs
 
region = boto3.Session().region_name    
smclient = boto3.Session().client('sagemaker')

role = sagemaker.get_execution_role()



s3_output_location='s3://{}/{}/{}'.format(bucket, prefix, 'output')
container=sagemaker.image_uris.retrieve("xgboost", region, "latest")


xgb_model=sagemaker.estimator.Estimator(
    image_uri=container,
    role=role,
    instance_count=1,
    instance_type='ml.m4.xlarge',
    volume_size=5,
    output_path=s3_output_location,
    sagemaker_session=sagemaker.Session(),
    rules=[Rule.sagemaker(rule_configs.create_xgboost_report())]
)

xgb_model.set_hyperparameters(
    max_depth = 5,
    eta = 0.2,
    gamma = 4,
    min_child_weight = 6,
    subsample = 0.7,
    objective = "binary:logistic",
    num_round = 10
)

hyperparameter_ranges = {'eta': ContinuousParameter(0.1, 0.5),
                        'min_child_weight': ContinuousParameter(1, 10),
                        'alpha': ContinuousParameter(0, 3),
                        'max_depth': IntegerParameter(0, 4)}
objective_metric_name = 'validation:auc'


tuner = HyperparameterTuner(xgb_model,
                            objective_metric_name,
                            hyperparameter_ranges,
                            max_jobs=60,
                            max_parallel_jobs=6)



train_input = TrainingInput(
    "s3://{}/{}/{}".format(bucket, prefix, "train/train.csv"), content_type="csv"
)
validation_input = TrainingInput(
    "s3://{}/{}/{}".format(bucket, prefix, "validate/validation.csv"), content_type="csv"
)

tuner.fit({"train": train_input, "validation": validation_input}, include_cls_metadata=False)

这是我得到的错误

> --------------------------------------------------------------------------- UnexpectedStatusException                 Traceback (most recent call
> last) <ipython-input-2-7824ad80a8bb> in <module>
>      62 )
>      63 
> ---> 64 tuner.fit({"train": train_input, "validation": validation_input}, include_cls_metadata=False)
> 
> ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py
> in fit(self, inputs, job_name, include_cls_metadata, estimator_kwargs,
> wait, **kwargs)
>     449 
>     450         if wait:
> --> 451             self.latest_tuning_job.wait()
>     452 
>     453     def _fit_with_estimator(self, inputs, job_name, include_cls_metadata, **kwargs):
> 
> ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/tuner.py
> in wait(self)    1595     def wait(self):    1596        
> """Placeholder docstring."""
> -> 1597         self.sagemaker_session.wait_for_tuning_job(self.name)    1598     1599 
> 
> ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py
> in wait_for_tuning_job(self, job, poll)    3253         """    3254   
> desc = _wait_until(lambda: _tuning_job_status(self.sagemaker_client,
> job), poll)
> -> 3255         self._check_job_status(job, desc, "HyperParameterTuningJobStatus")    3256         return desc    3257 
> 
> ~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py
> in _check_job_status(self, job, desc, status_key_name)    3336        
> ),    3337                 allowed_statuses=["Completed", "Stopped"],
> -> 3338                 actual_status=status,    3339             )    3340 
> 
> UnexpectedStatusException: Error for HyperParameterTuning job
> xgboost-211XXX-1641: Failed. Reason: No training job succeeded after 5
> attempts. For additional details, please take a look at the training
> job failures by listing training jobs for the hyperparameter tuning
> job.

先感谢您，

山姆