获取维基百科文章的第一段(并且仅文本)返回不期望的结果

Question

我试图在这个例子中检索维基百科,UNIX的文章的第一段文本,但它返回了一个非期望的输出.

对于我在维基百科api和StackOverflow上阅读的内容,这是进行调用的请求URL:

http://en.wikipedia.org/w/api.php?format=php&action=query&titles=unix&redirects=1&prop=revisions&rvprop=content&rvsection=0&rvlimit=1

我的预期输出将是:

Unix(官方注册为UNIX,有时也称为小型大写的Unix)是一种多任务,多用户计算机操作系统,最初由贝尔实验室的一组AT&T员工于1969年开发,包括Ken Thompson,Dennis Ritchie,Brian Kernighan, Douglas McIlroy,Michael Lesk和Joe Ossanna.[1] Unix操作系统最初是用汇编语言开发的,但到1973年几乎完全用C语言编写,极大地促进了它的进一步开发和移植到其他硬件.今天的Unix系统演变分为各个分支,由AT&T以及各种商业供应商,大学(如加州大学伯克利分校的BSD)和非营利组织随时间开发.

我目前的结果:

{{Use dmy dates|date=August 2012}}
{{Infobox OS
|name               = Unix
|logo               = 
|screenshot         = [[File:Unix history-simple.svg|250px]]
|caption            = Evolution of Unix and Unix-like systems
|website            = [http://www.unix.org unix.org]
|developer          = [[Ken Thompson (computer programmer)|Ken Thompson]], [[Dennis Ritchie]], [[Brian Kernighan]], [[Douglas McIlroy]], and [[Joe Ossanna]] at [[Bell Labs]]
|source_model       = Historically [[Closed source software|closed source]], now some Unix projects ([[Berkeley Software Distribution|BSD]] family and [[Illumos]]) are [[open source]]d.
|frequently_updated = yes <!-- Release version update? Don't edit this page, just click on the version number! -->
|programmed_in      = [[C (programming language)|C]] 
|kernel_type        = [[Monolithic Kernel|Monolithic]]
|ui                 = [[Command-line interface]] & [[Graphical user interface|Graphical]] ([[X Window System]])
|language           = English 
|family             = Unix
|released           = {{start date and age|df=yes|1969}}
|license            = [[Proprietary software|Proprietary]]
|working_state      = Current 
}}

'''Unix''' (officially trademarked as '''UNIX''', sometimes also written as '''<span style="font-variant: small-caps;">Unix</span>''' in small caps) is a [[Computer multitasking|multitasking]], [[multi-user]] computer [[operating system]] originally developed in 1969 by a group of [[American Telephone & Telegraph|AT&T]] employees at [[Bell Labs]], including [[Ken Thompson]], [[Dennis Ritchie]], [[Brian Kernighan]], [[Douglas McIlroy]], [[Michael Lesk]] and [[Joe Ossanna]].<ref name=" Ritchie">{{cite journal
  | last = Ritchie
  | first = D.M.
  | authorlink = 
  | coauthors = Thompson, K.
  | title = The UNIX Time-Sharing System
  | journal = Bell System Tech. J.
  | volume = 57
  | issue = 6
  | pages = 1905-1929
  | publisher = American Tel. & Tel.
  | location = USA
  | date = July 1978
  | url = http://www.alcatel-lucent.com/bstj/vol57-1978/articles/bstj57-6-1905.pdf
  | issn = 
  | doi = 
  | id = 
  | accessdate = December 9, 2012}}</ref>  The Unix operating system was first developed in [[assembly language]], but by 1973 had been almost entirely recoded in [[C (programming language)|C]], greatly facilitating its further development and [[Software portability|porting]] to other hardware. Today's Unix system evolution is split into various branches, developed over time by AT&T as well as various commercial vendors, universities (such as [[University of California, Berkeley]]'s [[BSD]]), and [[non-profit]] organizations.

[[The Open Group]], an industry standards consortium, owns the UNIX trademark. Only systems fully compliant with and certified according to the [[Single UNIX Specification]] are qualified to use the trademark; others might be called ''Unix system-like'' or ''[[Unix-like]]'', although the Open Group disapproves<ref>[http://www.unix.org/questions_answers/faq.html#7a  What is a "Unix-like" operating system?] Unix.org FAQ</ref> of this term.  However, the term ''Unix'' is often used informally to denote any operating system that closely resembles the trademarked system.

During the late 1970s and early 1980s, the influence of Unix in academic circles led to large-scale adoption of Unix (particularly of the [[Berkeley Software Distribution|BSD]] variant, originating from the [[University of California, Berkeley]]) by commercial startups, the most notable of which are [[Solaris (operating system)|Solaris]], [[HP-UX]], [[Sequent Computer Systems|Sequent]], and [[AIX operating system|AIX]], as well as [[Darwin (operating system)|Darwin]], which forms the core set of components upon which [[Apple Inc.|Apple]]'s [[OS X]], [[Apple TV]], and [[IOS (Apple)|iOS]] are based.<ref>{{cite web|url=http://marketshare.hitslink.com/operating-system-market-share.aspx?qprid=8&qpcustomd=0 |title=Operating system market share |publisher=Marketshare.hitslink.com |date= |accessdate=2012-08-22}}</ref><ref>{{cite web|url=http://developer.apple.com/library/mac/#documentation/MacOSX/Conceptual/OSX_Technology_Overview/SystemTechnology/SystemTechnology.html#//apple_ref/doc/uid/TP40001067-CH207-BCICAIFJ |title=Loading |publisher=Developer.apple.com |date= |accessdate=2012-08-22}}</ref> Today, in addition to certified Unix systems such as those already mentioned, [[Unix-like]] operating systems such as [[MINIX]], [[Linux]], and [[BSD]] descendants ([[FreeBSD]], [[NetBSD]], [[OpenBSD]], and [[DragonFly BSD]]) are commonly encountered. The term ''traditional Unix'' may be used to describe an operating system that has the characteristics of either [[Version 7 Unix]] or [[UNIX System V]]."

检索文章的正确方法是什么？

提前致谢!

Answer 1

如果只需要纯文本，请使用TextExtracts：http : //en.wikipedia.org/w/api.php? format=json&action=query&prop=extracts&explaintext=1&titles=Unix

这将产生： Unix is a multitasking, multi-user computer operating system that exists in many variants. The original Unix was developed at AT&T's Bell Labs research center by Ken Thompson, Dennis Ritchie, and others. From the power user's or programmer's perspective, Unix systems are characterized by a modular design that is sometimes called the "Unix philosophy," meaning the OS provides a set of simple tools that each perform a limited, well-defined function, with a unified filesystem as the main means of communication and a shell scripting and command language to combine the tools to perform complex workflows.

Answer 2

我一直在用Python处理这个问题。第一个任务是获取所需的文本；之后，您需要解析 HTML 并删除所有无关信息。

以下函数将为您提供第 n 个文本部分（n=0 返回摘要）：

import requests

def getWikiSection(topic, n):
    url = 'http://en.wikipedia.org/w/api.php?action=parse&page=%s&format=json&prop=text&section=%s' % (topic, str(n))
    json_response = requests.get(url).json().items()
    if len(json_response) > 1 and json_response[1][0] == u'error':
        print json_response[1][1][u'info']
        return None
    return stripTags(json_response[0][1][u'text'][u'*'])

快速演练：首先，我们为给定主题创建 URL；然后，我们获取 JSON 响应；如果我们查询了无效的部分或主题（即，该主题不存在页面或超出了页面的长度），我们会打印错误；否则，我们清理响应。

清理响应由最后一行的“stripTags”函数处理，该函数会删除 HTML 标签。这里是：

from HTMLParser import HTMLParser

class MLStripper(HTMLParser):
    def __init__(self):
        self.reset()
        self.fed = []

    def handle_data(self, d):
        self.fed.append(d)

    def get_data(self):
        return ''.join(self.fed)

def stripTags(html):
    s = MLStripper()
    s.feed(html)
    return s.get_data()

当然，这可以扩展为按照您喜欢的方式解析文本。例如，我删除了如下引用：

import re

def removeReferences(s):
    return re.sub(r'\[[0-9]+\]', '', s)

希望这可以帮助。