小编zde*_*ulo的帖子

我正在努力正确设置 Vertex AI 管道，该管道执行以下操作：

1. 从 API 读取数据并存储到 GCS 并作为批量预测的输入。
2. 获取现有模型（Vertex AI 上的视频分类）
3. 使用第 1 点的输入创建批量预测作业
   。正如所见，我对 Vertex Pipelines/Kubeflow 没有太多经验，因此我寻求帮助/建议，希望这只是一些初学者的错误。这是我用作管道的代码的要点

from google_cloud_pipeline_components import aiplatform as gcc_aip
from kfp.v2 import dsl

from kfp.v2.dsl import component
from kfp.v2.dsl import (
    Output,
    Artifact,
    Model,
)

PROJECT_ID = 'my-gcp-project'
BUCKET_NAME = "mybucket"
PIPELINE_ROOT = "{}/pipeline_root".format(BUCKET_NAME)


@component
def get_input_data() -> str:
    # getting data from API, save to Cloud Storage
    # return GS URI
    gcs_batch_input_path = 'gs://somebucket/file'
    return gcs_batch_input_path


@component(
    base_image="python:3.9",
    packages_to_install=['google-cloud-aiplatform==1.8.0']
)
def load_ml_model(project_id: str, …

我有Scrapy(版本1.0.3)蜘蛛,其中我从网页中提取了一些数据,我也下载了文件,像这样(简化):

def extract_data(self, response):
    title = response.xpath('//html/head/title/text()').extract()[0].strip()
    my_item = MyItem()
    my_item['title'] = title    

    file_url = response.xpath('...get url of file...')
    file_urls = [file_url]  # here there can be more urls, so I'm storing like a list
    fi = FileItem()
    fi['file_urls'] = file_urls 
    yield my_item
    yield fi

在pipelines.py中我只是重写FilePipeline来更改文件的名称:

from scrapy.pipelines.files import FilesPipeline

class CustomFilesPipeline(FilesPipeline):
    def file_path(self, request, response=None, info=None):
        filename = format_filename(request.url)
        return filename

在items.py我有:

class MyItem(scrapy.Item):
    title = scrapy.Field()

class FileItem(scrapy.Item):
    file_urls = scrapy.Field()
    files = scrapy.Field()

在settings.py我有:

ITEM_PIPELINES = { …