使用 python、AutoGPT 和 ChatGPT 从下载的 HTML 页面中提取数据

Question

使用 python、AutoGPT 和 ChatGPT 从下载的 HTML 页面中提取数据

Flo*_*Flo 1 python openai-api chatgpt-api autogpt

注意：如果您投反对票，至少请分享原因。我花了很多精力来写这个问题，首先分享了我的代码并做了我自己的研究，所以不确定我还可以添加什么。

我已经使用Scrapy成功抓取网站。我使用 CSS 选择器从网页中提取特定数据。然而，设置起来非常耗时并且容易出错。我希望能够将原始 HTML 传递给 chatGPT 并提出如下问题

“以 JSON 对象格式提供该对象的价格、照片数组、描述、主要功能、街道地址和邮政编码”

下面是所需的输出。为了便于阅读，我截断了描述、主要功能和照片。

{
"price":"$945,000",
"photos":"https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/1500x1500/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542874?w=3840&q=75;https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/1500x1500/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542875?w=3840&q=75;https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/1500x1500/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542876?w=3840&q=75",
"description":"<div>This spacious 2 bedroom 1 bath home easily converts to 3 bedrooms. Featuring a BRIGHT and quiet southern exposure, the expansive great room (with 9ft ceilings) is what sets (...)",
"key features":"Center island;Central air;Dining in living room;Dishwasher",
"street address":"170 West 89th Street, 2D",
"zipcode":"NY 10024",
}

Run Code Online (Sandbox Code Playgroud)

现在我遇到的最大聊天长度为 4096 个字符。所以我决定分块发送页面。然而，即使有一个简单的问题，例如“这个物品的价格是多少？” 我预计答案是“945,000 美元”，但我只收到一大堆文字。我想知道我做错了什么。我听说 AutoGPT 提供了新的灵活性，所以我也想知道这是否可以作为一个解决方案。

我的代码：

import requests
from bs4 import BeautifulSoup, Comment
import openai
import json

# Set up your OpenAI API key
openai.api_key = "MYKEY"

# Fetch the HTML from the page
url = "https://www.corcoran.com/listing/for-sale/170-west-89th-street-2d-manhattan-ny-10024/22053660/regionId/1"
response = requests.get(url)

# Parse and clean the HTML
soup = BeautifulSoup(response.text, "html.parser")

# Remove unnecessary tags, comments, and scripts
for script in soup(["script", "style"]):
    script.extract()

# for comment in soup.find_all(text=lambda text: isinstance(text, Comment)):
#     comment.extract()

text = soup.get_text(strip=True)

# Divide the cleaned text into chunks of 4096 characters
def chunk_text(text, chunk_size=4096):
    chunks = []
    for i in range(0, len(text), chunk_size):
        chunks.append(text[i:i+chunk_size])
    return chunks

print(text)

text_chunks = chunk_text(text)

# Send text chunks to ChatGPT API and ask for the price
def get_price_from_gpt(text_chunks, question):
    for chunk in text_chunks:
        prompt = f"{question}\n\n{chunk}"
        response = openai.Completion.create(
            engine="text-davinci-002",
            prompt=prompt,
            max_tokens=50,
            n=1,
            stop=None,
            temperature=0.5,
        )

        answer = response.choices[0].text.strip()
        if answer.lower() != "unknown" and len(answer) > 0:
            return answer

    return "Price not found"

question = "What is the price of this object?"
price = get_price_from_gpt(text_chunks, question)
print(price)

Run Code Online (Sandbox Code Playgroud)

Answer 1

Lif*_*lex 5

更新答案 2023 年 6 月 28 日

\n

你的问题很有趣，所以我想尝试改进我之前你已经接受的答案。

\n

我注意到我之前的答案查询 .05 美分左右OpenAI api。这些成本与文本分块功能和在for loop. 我已经删除了文本分块功能，for loop因为我能够将标记缩小到压缩大小。

\n

降低成本所需的核心项目之一是文本清理，这是一个标准的 NLP 和数据科学问题。我添加了更多代码以从 SOUP 对象中删除其他不需要的文本。这样做会对性能造成影响，但还不足以导致失眠。

\n

还需要完善查询提示，以便在单个请求中提交所有内容。这样做可以降低查询成本。

\n

下面的代码还可以进一步细化。目前，使用text-davinci-003. 提示需要重新设计才能使用text-davinci-002，比便宜一点text-davinci-003。

\n

以下代码的 API 查询时间可能会超过 15 秒。OpenAI 的社区论坛上有很多关于查询性能的讨论。根据我的研究，没有可靠的技术来提高查询性能。

\n

import json\nimport spacy\nimport openai\nimport requests\nimport re as regex\nfrom bs4 import BeautifulSoup\n\nopenai.api_key = \'my_key\'\n\n\n# this code can be refined. \ndef remove_unwanted_tags(soup):\n    for disclaimer_tag in soup.select(\'div[class*="Disclaimer__TextContainer"]\'):\n        disclaimer_tag.decompose()\n\n    for footer_tag in soup.select(\'div[class*="GlobalFooter__"]\'):\n        footer_tag.decompose()\n\n    for hidden_tag in soup.select(\'p[class*="visually-hidden"]\'):\n        hidden_tag.decompose()\n\n    for agent_contact_card_tag in soup.select(\'div[class*="AgentContactCard__"]\'):\n        agent_contact_card_tag.decompose()\n\n    for listing_agent_tag in soup.select(\'div[class*="DetailsAndAgents__ListingFirmName"]\'):\n        listing_agent_tag.decompose()\n\n    for property_sale_history_tag in soup.select(\'section[class*="SalesHistory__"]\'):\n        property_sale_history_tag.decompose()\n\n    for data in soup([\'style\', \'script\', \'iframe\', \'footer\', \'h2\', \'a\']):\n        data.decompose()\n\n    dirty_soup = \' \'.join(soup.find_all(string=True))\n    remove_allcaps_words = regex.sub(r\'\\b[A-Z]+\\b\', \'\', dirty_soup)\n    clean_soup = regex.sub("\\s\\s+", " ", str(remove_allcaps_words))\n    return  clean_soup\n\n\ndef tokenize_text(text):\n    nlp = spacy.load("en_core_web_sm")\n    sentences = nlp(text)\n    return [sentence.text for sentence in sentences.sents]\n\n\ndef get_property_details_from_gpt(text, question):\n        prompt = f"{question}\\n\\n{text}"\n        response = openai.Completion.create(\n            engine="text-davinci-003",\n            prompt=prompt,\n            max_tokens=200,\n            n=1,\n            stop=None,\n            temperature=0.7,\n            frequency_penalty=0,\n            presence_penalty=0.6\n        )\n\n        answer = response.choices[0].text.strip()\n        if answer.lower() != "unknown" and len(answer) > 0:\n            return answer\n\nurl = "https://www.corcoran.com/listing/for-sale/170-west-89th-street-2d-manhattan-ny-10024/22053660/regionId/1"\nresponse = requests.get(url)\nsoup = BeautifulSoup(response.content, "html.parser")\nproperty_photos = [element.find(\'img\').attrs[\'src\'] for element in soup.select(\'div[class*="carousel-item"]\')]\n\nstrained_soup = remove_unwanted_tags(soup)\ntext = u\'\'.join(tokenize_text(strained_soup))\n\nquestion = """Please extract the following details from the provided items:\n{"price": "(1) Exact selling price of this property with dollar sign and thousands formatting.",\n"street_address": "(2) Exact street address of this property without state or zip code.",\n"description": "(3) Short description of this property.",\n"key_features": "(4) Key features of this property. Provide these items in a semicolon delimiter string.",\n"state": "(5) State abbreviation for this property.",\n"zipcode": "(6) Zip code for this property, if available.",\n"photos": ""},\n(8) Provide this data back in a python dictionary that can be processed by json.loads"""\n\nquery_results = get_property_details_from_gpt(text, question)\nquery_data = query_results.replace(\'Answer:\', \'\')\ncorcoran_properties = json.loads(query_data)\ncorcoran_properties["photos"] = f"{property_photos}"\ncorcoran_json = json.dumps(corcoran_properties, indent=4)\nprint(corcoran_json)\n\n

Run Code Online (Sandbox Code Playgroud)\n

这是输出：

\n

{\n    "price": "$945,000",\n    "street_address": "170 West 89th Street",\n    "description": "This spacious 2 bedroom 1 bath home easily converts to 3 bedrooms. Featuring a and quiet southern exposure, the expansive great room (with 9ft ceilings) is what sets this home apart from others. Paired with a renovated open kitchen, new bathroom, and washer/dryer, this is an Upper West Side gem.",\n    "key_features": "Center island; Central air; Dining in living room; Dishwasher; En suite; Excellent light light; Hardwood floors; High ceilings; Modern kitchen; New windows; Open kitchen; Pet friendly; Prewar detail; Storage space; Washer/dryer; Window /",\n    "state": "NY",\n    "zipcode": "10024",\n    "photos": "[\'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542874?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542875?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542876?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542877?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542878?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542879?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542880?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542881?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543843?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543845?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543846?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543847?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543848?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543849?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543850?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543851?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543852?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543853?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543854?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543855?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542882?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542883?w=3840&q=75\', \'https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/PropertyAPI/NewTaxi/5415/mediarouting.vestahub.com/Media/111409705?w=3840&q=75\']"\n}\n\n

Run Code Online (Sandbox Code Playgroud)\n

更新答案 2023 年 6 月 26 日

\n

我正在尝试完善这个答案。我决定在将数据发送到 API 之前稍微清理一下数据。这样做让我得到了一些更清晰的答案。我从之前的答案中删除了代码，但我留下了我的笔记，我认为这对于任何尝试做类似事情的人都很重要。

\n

我发现它text-davinci-003比给出了更精确的问题答案text-davinci-002，但使用成本更高text-davinci-003。

\n

更新的代码：

\n

import json\nimport spacy\nimport openai\nimport requests\nimport re as regex\nfrom bs4 import BeautifulSoup\n\n#####################################################\n# The NLP package en_core_web_sm has to be download \n# once to your environment. Use the following command \n# to accomplish this. \n# \n# spacy.cli.download("en_core_web_sm")\n#\n######################################################\n\nopenai.api_key = \'my_key\'\n\n# I needed to deep clean the soup by removing some excess text. Doing this allowed for better responses for OpenAI. \ndef remove_unwanted_tags(soup):\n    for disclaimer_tag in soup.select(\'div[class*="Disclaimer__Text"]\'):\n        disclaimer_tag.decompose()\n\n    for footer_tag in soup.select(\'div[class*="GlobalFooter__Footer"]\'):\n        footer_tag.decompose()\n\n    for hidden_tag in soup.select(\'p[class*="visually-hidden"]\'):\n        hidden_tag.decompose()\n\n    for data in soup([\'style\', \'script\', \'iframe\']):\n        data.decompose()\n\n    dirty_soup = \' \'.join(soup.find_all(string=True))\n    \n    # I removed all the UPPER case words in the soup, \n    # because they provided no value. \n    clean_soup = regex.sub(r\'\\b[A-Z]+\\b\', \'\', dirty_soup)\n    return clean_soup\n\ndef tokenize_text(text):\n    nlp = spacy.load("en_core_web_sm")\n    sentences = nlp(text)\n    return [sentence.text for sentence in sentences.sents]\n\ndef get_text_chunks(text, max_tokens_per_chunk=3000):\n    chunks = []\n    current_chunk = []\n    current_token_count = 0\n    sentences = tokenize_text(text)\n    for sentence in sentences:\n        current_chunk.append(sentence)\n        current_token_count += len(sentence.split(" "))\n\n        if current_token_count >= max_tokens_per_chunk:\n            chunks.append(current_chunk)\n            current_chunk = []\n            current_token_count = 0\n\n    if current_chunk:\n        chunks.append(current_chunk)\n    return chunks\n\ndef get_property_details_from_gpt(text_chunks, question):\n    for chunk in text_chunks:\n        prompt = f"{question}\\n\\n{chunk}"\n        response = openai.Completion.create(\n            engine="text-davinci-003",\n            prompt=prompt,\n            max_tokens=400,\n            n=1,\n            stop=None,\n            temperature=0.5,\n        )\n\n        answer = response.choices[0].text.strip()\n        if answer.lower() != "unknown" and len(answer) > 0:\n            return answer\n\n\nurl = "https://www.corcoran.com/listing/for-sale/170-west-89th-street-2d-manhattan-ny-10024/22053660/regionId/1"\nresponse = requests.get(url)\nsoup = BeautifulSoup(response.content, "html.parser")\n\nquestions = {\'price\': \'Provide only the price the for this property\',\n             \'description\': \'Provide a short description about this property?\\nOnly provide complete \'\n                            \'sentences. Please correct the spelling for each word provide.\',\n             \'key features\':\'What are the key features of this property?\\nProvide these items in a semicolon delimiter string.\',\n             \'street address\': \'What is the exact street address of this propery?\\nOnly provide the street address \'\n                               \'and nothing additional.\',\n             \'zipcode\': \'What is the state abbreviation and zip code for this property?\\nOnly provide the state abbreviation and zipcode.\'}\n\ncorcoran_properties = {}\n\nstrained_soup = remove_unwanted_tags(soup)\ntext_chunks = get_text_chunks(strained_soup)\nfor json_key, question in questions.items():\n    question_response = get_property_details_from_gpt(text_chunks, question)\n    corcoran_properties[json_key] = question_response\n\nproperty_photos = [element.find(\'img\').attrs[\'src\'] for element in soup.select(\'div[class*="carousel-item"]\')]\ncorcoran_properties[\'photos\'] = property_photos\n\ncorcoran_json = json.dumps(corcoran_properties, indent=4)\nprint(corcoran_json)\n\n\n

Run Code Online (Sandbox Code Playgroud)\n

这是上面代码的输出：

\n

{\n    "price": "$945,000",\n    "description": "This property is a 2 bedroom, 1 bathroom Co-op located at 170 West 89th Street in the Upper West Side of Manhattan, New York. Built in 1910, this spacious home features 9ft ceilings, a renovated open kitchen, new bathroom, and washer/dryer. Building amenities include a storage unit, stroller parking and bicycle storage. It has excellent natural light and Southern exposure. The neighborhood is full of iconic architecture, cultural institutions, and historical sites. It is close to Central Park and Riverside Park.",\n    "key features": "9ft ceilings; 2 beds; 1 bath; Southern exposure; Renovated open kitchen; New bathroom; Washer/Dryer; Storage unit; Convenient stroller parking; Bicycle storage; Center island; Central air; Dining in living room; Dishwasher; En suite; Excellent light; Hardwood floors; High ceilings; Modern kitchen; New windows; Open kitchen; Pet friendly; Prewar detail; Storage space; Washer/Dryer; Window/Listing agent; Iconic architecture; City-defining structures; Cultural institutions; Historical sites; Central Park; Riverside Park.",\n    "street address": "170 West 89th Street",\n    "zipcode": "NY, 10024",\n    "photos": [\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542874?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542875?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542876?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542877?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542878?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542879?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542880?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542881?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543843?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543845?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543846?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543847?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543848?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543849?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543850?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543851?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543852?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543853?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543854?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134543855?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542882?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/ListingFullAPI/NewTaxi/7625191/mediarouting.vestahub.com/Media/134542883?w=3840&q=75",\n        "https://media-cloud.corcoranlabs.com/filters:format(webp)/fit-in/768x768/PropertyAPI/NewTaxi/5415/mediarouting.vestahub.com/Media/111409705?w=3840&q=75"\n    ]\n}\n

Run Code Online (Sandbox Code Playgroud)\n

原始答案 06.24.2023（为了可读性删除了代码）

\n

我注意到代码中的核心问题之一是这一行：

\n

\n
文本 = soup.get_text(strip=True)
\n

\n

该行删除了一些OpenAI chatGPT处理所需的空间。

\n

\n
内置 19102 个床位 1 个浴室 $945,000 维护/普通费用：$1,68010%
\n

\n

这样做可以留出空间：

\n

\n
文本 = u\' \'.join(soup.find_all(string=True))
\n
建于 1910 年 2 床 1 浴室 $945,000 维护/常见费用：
\n

\n

此外，apiOpenAI处理的是标记而不是字符，因此您的分块代码需要替换为处理标记化的代码。

\n

我不确定这个答案的可扩展性，因为您肯定需要考虑与您的数据源相关的所有适用问题。

\n

例如：

\n

该房产的地址是什么？ 该酒店的地址为 170 West 89th Street #2D, New York, NY 10024。

\n

该房产是哪一年建造的？ 该物业建于1910年。

\n

该房产的维护费是多少？ 该房产的维护费为 1,680 美元。

\n

** 有一些物业设施吗？**有一些物业设施，包括储藏室、婴儿车停车场和自行车存放处。

\n

使用的核心问题之一OpenAI api是从提供的文本中提取清晰的描述。

\n

该行text = \' \'.join(soup.find_all(string=True))产生以下文本：

\n

170 West 89th Street #2D, New York, NY 10024 Property for sale Skip to main content Sign In Agent Sign in Preferences Open visitors preferences modal BUY RENT SELL NEW DEVELOPMENTS COMMERCIAL SEARCH ALL COMMERCIAL WEXLER HEALTHCARE PROPERTIES AGENTS SEARCH LOCAL AGENTS BROWSE ALL AGENTS BECOME AN AGENT OFFICES SEARCH LOCAL OFFICES SEARCH ALL OFFICES ABOUT US ABOUT CORCORAN CORCORAN LEADERSHIP CORCORAN BRAND BROWSE AFFILIATES BECOME AN AFFILIATE EXPLORE MARKET REPORTS NEIGHBORHOOD GUIDES INHABIT BLOG MEDIA COVERAGE EXCLUSIVE RENTAL BUILDINGS Save share Contact BUY SEARCH in contract WEB ID:    22053660 170 West 89th Street, 2D   Upper West Side,  Manhattan, NY 10024 Upper West Side,  Manhattan, NY 10024 in contract |    Co-op |    Built in 1910     2 beds     1 bath $945,000 Maintenance/Common Charges:     $1,680 10   % Down:     $94

归档时间：	2 年，6 月前
查看次数：	2343 次
最近记录：	2 年，4 月前