Yahoo Finance URL无效

Question

我一直在使用以下网址从雅虎财经获取历史数据已经有一段时间了,但截至昨天它已停止工作.

https://ichart.finance.yahoo.com/table.csv?s=SPY

浏览本网站时说:

马上回来...

感谢您的耐心等待.

我们的工程师正在迅速解决这个问题.

但是,由于这个问题自昨天起仍然存在,我开始认为他们已经停止了这项服务？

我的SO搜索只指向了这个主题,它与https有关...

还有其他人遇到过这个问题吗？我该如何解决这个问题？他们是否提供对历史数据的不同访问权限？

Answer 1

看起来他们已经开始添加必需的cookie,但您可以相当容易地检索它,例如:

GET https://uk.finance.yahoo.com/quote/AAPL/history

响应表单中的标题:

set-cookie:B=xxxxxxxx&b=3&s=qf; expires=Fri, 18-May-2018 00:00:00 GMT; path=/; domain=.yahoo.com

您应该能够阅读此内容并将其附加到您的.csv请求中:

GET https://query1.finance.yahoo.com/v7/finance/download/AAPL?period1=1492524105&period2=1495116105&interval=1d&events=history&crumb=tO1hNZoUQeQ
cookie: B=xxxxxxxx&b=3&s=qf;

注意crumb查询参数,这似乎与您cookie的某些方式相对应.您最好的选择是scrape从HTML响应到您的初始GET请求.在该响应中,您可以执行正则表达式搜索:"CrumbStore":\{"crumb":"(?<crumb>[^"]+)"\}并提取crumb匹配组.

它看起来就像你有了这个crumb价值,虽然你可以cookie在明年的任何符号/代码上使用它,这意味着你不必scrape太频繁地做.

要获取当前报价,请加载:

https://query1.finance.yahoo.com/v8/finance/chart/AAPL?interval=2m

附:

• AAPL替换为您的股票代码
• 间隔之一 [1m, 2m, 5m, 15m, 30m, 60m, 90m, 1h, 1d, 5d, 1wk, 1mo, 3mo]
• period1带有您的纪元范围开始日期的可选查询参数,例如period1=1510340760
• period2带有您的纪元范围结束日期的可选查询参数,例如period2=1510663712

Answer 2

Yahoo已进入Reactjs前端,这意味着如果您分析从客户端到后端的请求标头,您可以获得用于填充客户端存储的实际JSON.

主持人:

• query1.finance.yahoo.com HTTP/1.0
• query2.finance.yahoo.comHTTP/1.1(HTTP/1.0和HTTP/1.1之间的区别)

如果您打算使用代理或持久连接使用query2.finance.yahoo.com.但是出于本文的目的,用于示例URL的主机并不意味着暗示它正在使用的路径.

基本数据

• /v10/finance/quoteSummary/AAPL?modules= (以下模块的完整列表)

(用你的符号代替:AAPL)

?modules=查询的输入:

• modules = [ 'assetProfile', 'incomeStatementHistory', 'incomeStatementHistoryQuarterly', 'balanceSheetHistory', 'balanceSheetHistoryQuarterly', 'cashflowStatementHistory', 'cashflowStatementHistoryQuarterly', 'defaultKeyStatistics', 'financialData', 'calendarEvents', 'secFilings', 'recommendationTrend', 'upgradeDowngradeHistory', 'institutionOwnership', 'fundOwnership', 'majorDirectHolders', 'majorHoldersBreakdown', 'insiderTransactions', 'insiderHolders', 'netSharePurchaseActivity', 'earnings', 'earningsHistory', 'earningsTrend', 'industryTrend', 'indexTrend', 'sectorTrend' ]

示例网址:

• https://query1.finance.yahoo.com/v10/finance/quoteSummary/AAPL?modules=assetProfile%2CearningsHistory

查询:assetProfile和earningsHistory

的%2C是的十六进制表示,,需要每次请求模块之间插入.有关十六进制编码位的详细信息(如果您关心)

期权合约

• /v7/finance/options/AAPL (当前到期)
• /v7/finance/options/AAPL?date=1579219200 (2020年1月17日到期)

示例网址:

• https://query2.yahoo.finance.com/v7/finance/options/AAPL (当前到期)
• https://query2.yahoo.finance.com/v7/finance/options/AAPL?date=1579219200 (2020年1月17日到期)

可以在?date=查询中使用表示为UNIX时间戳的任何有效的未来到期.如果查询当前过期,JSON响应将包含可在?date=查询中使用的所有有效过期的列表.(这是一篇解释在Python中将人类可读日期转换为unix时间戳的帖子)

价钱

• /v8/finance/chart/AAPL?symbol=AAPL&period1=0&period2=9999999999&interval=3mo

间隔:

• &interval=3mo 3个月,回到初始交易日期.
• &interval=1d 1天,回到最初的交易日期.
• &interval=5m 5分钟,回到80(ish)天.
• &interval=1m 1分钟,回去4-5天.

每个间隔你可以走多远,有点混乱,似乎不一致.我的假设是内部雅虎在交易日计算,而我的天真方法并不是假期.虽然这是一个猜测和YMMV.

period1=:unix时间戳表示您希望开始的日期.低于初始交易日的价值将四舍五入至初始交易日.

period2=:unix时间戳表示您希望结束的日期.大于上次交易日期的值将向下舍入到可用的最新时间戳.

注意: 如果您使用period1=(开始日期)在您选择的时间间隔内查询过多,则3mo无论您请求的间隔是多少,yahoo都会在该时间间隔内返回价格.

添加前后市场数据

&includePrePost=true

增加股息和分割

&events=div%2Csplit

示例网址:

• https://query1.finance.yahoo.com/v8/finance/chart/AAPL?symbol=AAPL&period1=0&period2=9999999999&interval=1d&includePrePost=true&events=div%2Csplit

上述请求将在1天的时间间隔内返回股票代码AAPL的所有价格数据,包括市场前和市场后数据以及股息和拆分.

注意: 价格示例网址中用于period1=&的值用于period2=演示每个输入的相应舍入行为.

Answer 3

我设法找到一个.NET类来从Yahoo Finance获得有效的令牌(cookie和crumb)

有关从新的Yahoo Finance获取历史数据的完整API库,您可以访问Github中的YahooFinanceAPI

这是抓住cookie和面包屑的类

Token.cs

using System;
using System.Diagnostics;
using System.Net;
using System.IO;
using System.Text.RegularExpressions;

namespace YahooFinanceAPI
{
    /// <summary>
    /// Class for fetching token (cookie and crumb) from Yahoo Finance
    /// Copyright Dennis Lee
    /// 19 May 2017
    /// 
    /// </summary>
    public class Token
    {
        public static string Cookie { get; set; }
        public static string Crumb { get; set; }

        private static Regex regex_crumb;
        /// <summary>
        /// Refresh cookie and crumb value Yahoo Fianance
        /// </summary>
        /// <param name="symbol">Stock ticker symbol</param>
        /// <returns></returns>
        public static bool Refresh(string symbol = "SPY")
        {

            try
            {
                Token.Cookie = "";
                Token.Crumb = "";

                string url_scrape = "https://finance.yahoo.com/quote/{0}?p={0}";
                //url_scrape = "https://finance.yahoo.com/quote/{0}/history"

                string url = string.Format(url_scrape, symbol);

                HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);

                request.CookieContainer = new CookieContainer();
                request.Method = "GET";

                using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                {

                    string cookie = response.GetResponseHeader("Set-Cookie").Split(';')[0];

                    string html = "";

                    using (Stream stream = response.GetResponseStream())
                    {
                        html = new StreamReader(stream).ReadToEnd();
                    }

                    if (html.Length < 5000)
                        return false;
                    string crumb = getCrumb(html);
                    html = "";

                    if (crumb != null)
                    {
                        Token.Cookie = cookie;
                        Token.Crumb = crumb;
                        Debug.Print("Crumb: '{0}', Cookie: '{1}'", crumb, cookie);
                        return true;
                    }

                }

            }
            catch (Exception ex)
            {
                Debug.Print(ex.Message);
            }

            return false;

        }

        /// <summary>
        /// Get crumb value from HTML
        /// </summary>
        /// <param name="html">HTML code</param>
        /// <returns></returns>
        private static string getCrumb(string html)
        {

            string crumb = null;

            try
            {
                //initialize on first time use
                if (regex_crumb == null)
                    regex_crumb = new Regex("CrumbStore\":{\"crumb\":\"(?<crumb>.+?)\"}", 
                RegexOptions.CultureInvariant | RegexOptions.Compiled, TimeSpan.FromSeconds(5));

                MatchCollection matches = regex_crumb.Matches(html);

                if (matches.Count > 0)
                {
                    crumb = matches[0].Groups["crumb"].Value;
                }
                else
                {
                    Debug.Print("Regex no match");
                }

                //prevent regex memory leak
                matches = null;

            }
            catch (Exception ex)
            {
                Debug.Print(ex.Message);
            }

            GC.Collect();
            return crumb;

        }

    }
}

已更新1月17日
积分@ Ed0906
修改crumb正则表达式模式到Regex("CrumbStore\":{\"crumb\":\"(?<crumb>.+?)\"}"

Answer 4

在这个论坛:https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503/page/3

尼克松说:

大家好 - 此功能已被财务团队终止,他们不会重新引入该功能.

Answer 5

下载历史数据的URL现在是这样的:

https://query1.finance.yahoo.com/v7/finance/download/SPY?period1=1492449771&period2=1495041771&interval=1d&events=history&crumb=9GaimFhz.WU

请注意,上述网址不适合您或其他任何人.你会得到这样的东西:

{
    "finance": {
        "error": {
            "code": "Unauthorized",
            "description": "Invalid cookie"
        }
    }
}

雅虎似乎正在使用一些哈希来阻止人们像你一样访问数据.网址因每个会话而异,因此您很可能不再使用固定网址执行此操作.

您需要进行一些报废才能从主页面获取正确的URL,例如:

https://finance.yahoo.com/quote/SPY/history?p=SPY

Answer 6

对于那里的python爱好者,我已经在tradingWithPython库中更新了yahooFinance.py.

还有一个基于Ed0906提示的笔记本示例,演示了如何逐步获取数据.看到它

Answer 7

我找到了另一个不需要cookie的雅虎网站,但生成了jason输出:https://query1.finance.yahoo.com/v7/finance/chart/YHOO ？ range = 2y & interval = 1d & indicators = quote & includeTimestamps = true

从这里指出:https://www.stock-data-solutions.com/kb/how-to-load-historical-prices-from-yahoo-finance-to-excel.htm

事实证明,它们似乎支持'perod1'和'period2'(在unix时间内)参数,可以用来代替'interval'.

String quoteSite = "https://query1.finance.yahoo.com/v7/finance/chart/"
                   + symbolName + "?"
                   + "period1=" + period1
                   + "&period2=" + period2
                   + "&interval=1d&indicators=quote&includeTimestamps=true";

以下为我解析杰森:

JSONObject topObj = new JSONObject(inp);
Object error = topObj.getJSONObject("chart").get("error");
if (!error.toString().equals("null")) {
    System.err.prinltn(error.toString());
    return null;
}
JSONArray results = topObj.getJSONObject("chart").getJSONArray("result");
if (results == null || results.length() != 1) {
    return null;
}
JSONObject result = results.getJSONObject(0);
JSONArray timestamps = result.getJSONArray("timestamp");
JSONObject indicators = result.getJSONObject("indicators");
JSONArray quotes = indicators.getJSONArray("quote");
if (quotes == null || quotes.length() != 1) {
    return null;
}
JSONObject quote = quotes.getJSONObject(0);
JSONArray adjcloses = indicators.getJSONArray("adjclose");
if (adjcloses == null || adjcloses.length() != 1) {
   return null;
}
JSONArray adjclose = adjcloses.getJSONObject(0).getJSONArray("adjclose");
JSONArray open = quote.getJSONArray("open");
JSONArray close = quote.getJSONArray("close");
JSONArray high = quote.getJSONArray("high");
JSONArray low = quote.getJSONArray("low");
JSONArray volume = quote.getJSONArray("volume");

Answer 8

我在同一条船上.慢慢到达那里.历史价格页面上的下载链接仍然有效.所以我将导出cookie扩展添加到firefox,登录到yahoo,转储cookie.使用交互式会话中的crumb值,我能够检索值.这是一个有效的测试perl脚本的一部分.

use Time::Local;

# create unix time variables for start and end date values: 1/1/2014 thru 12/31/2017
$p1= timelocal(0,0,0,1,0,114);
$p2= timelocal(0,0,0,31,11,117);

$symbol = 'AAPL';

# create variable for string to be executed as a system command
# cookies.txt exported from firefox
# crumb variable retrieved from yahoo download data link
$task = "wget --load-cookies cookies.txt --no-check-certificate -T 30 -O          $symbol.csv \"https://query1.finance.yahoo.com/v7/finance/download/$symbol?period1=$p1&period2=$p2&interval=1d&events=history&crumb=7WhHVu5N4e3\" ";

#show what we're executing
print $task;

# execute system command using backticks
`$task`;

#output is AAPL.csv

我需要一段时间来自动完成我的工作.希望雅虎能够简化或提供一些指导,如果他们真的打算让人们使用它.

Answer 9

对于java爱好者.

您可以通过这种方式从URLConnection访问您的cookie.

 //  "https://finance.yahoo.com/quote/SPY";
 URLConnection con = url.openConnection();
 ...  
 for (Map.Entry<String, List<String>> entry : con.getHeaderFields().entrySet()) {
        if (entry.getKey() == null 
            || !entry.getKey().equals("Set-Cookie"))
            continue;
        for (String s : entry.getValue()) {
           // store your cookie
           ...
        }
 }

现在你可以在雅虎网站上搜索crumb:

String crumb = null;
InputStream inStream = con.getInputStream();
InputStreamReader irdr = new InputStreamReader(inStream);
BufferedReader rsv = new BufferedReader(irdr);

Pattern crumbPattern = Pattern.compile(".*\"CrumbStore\":\\{\"crumb\":\"([^\"]+)\"\\}.*");

String line = null;
while (crumb == null && (line = rsv.readLine()) != null) {
    Matcher matcher = crumbPattern.matcher(line);
    if (matcher.matches()) 
        crumb = matcher.group(1);
}
rsv.close();

最后,设置cookie

String quoteUrl = "https://query1.finance.yahoo.com/v7/finance/download/IBM?period1=1493425217&period2=1496017217&interval=1d&events=history&crumb="
                           + crumb
...
List<String> cookies = cookieStore.get(key);
if (cookies != null) {
    for (String c: cookies) 
        con.setRequestProperty("Cookie", c);
}
...
con.connect();

Answer 10

完整工作的PHP示例,基于此帖和相关来源:

function readYahoo($symbol, $tsStart, $tsEnd) {
  preg_match('"CrumbStore\":{\"crumb\":\"(?<crumb>.+?)\"}"',
    file_get_contents('https://uk.finance.yahoo.com/quote/' . $symbol),
    $crumb);  // can contain \uXXXX chars
  if (!isset($crumb['crumb'])) return 'Crumb not found.';
  $crumb = json_decode('"' . $crumb['crumb'] . '"');  // \uXXXX to UTF-8
  foreach ($http_response_header as $header) {
    if (0 !== stripos($header, 'Set-Cookie: ')) continue;
    $cookie = substr($header, 14, strpos($header, ';') - 14);  // after 'B='
  }  // cookie looks like "fkjfom9cj65jo&b=3&s=sg"
  if (!isset($cookie)) return 'Cookie not found.';
  $fp = fopen('https://query1.finance.yahoo.com/v7/finance/download/' . $symbol
    . '?period1=' . $tsStart . '&period2=' . $tsEnd . '&interval=1d'
    . '&events=history&crumb=' . $crumb, 'rb', FALSE,
    stream_context_create(array('http' => array('method' => 'GET',
      'header' => 'Cookie: B=' . $cookie))));
  if (FALSE === $fp) return 'Can not open data.';
  $buffer = '';
  while (!feof($fp)) $buffer .= implode(',', fgetcsv($fp, 5000)) . PHP_EOL;
  fclose($fp);
  return $buffer;
}

用法:

$csv = readYahoo('AAPL', mktime(0, 0, 0, 6, 2, 2017), mktime(0, 0, 0, 6, 3, 2017));

Answer 11

Python

我使用此代码来获取 cookie（从fix-yahoo-finance复制）：

def get_yahoo_crumb_cookie():
    """Get Yahoo crumb cookie value."""
    res = requests.get('https://finance.yahoo.com/quote/SPY/history')
    yahoo_cookie = res.cookies['B']
    yahoo_crumb = None
    pattern = re.compile('.*"CrumbStore":\{"crumb":"(?P<crumb>[^"]+)"\}')
    for line in res.text.splitlines():
        m = pattern.match(line)
        if m is not None:
            yahoo_crumb = m.groupdict()['crumb']
    return yahoo_cookie, yahoo_crumb

然后这段代码来获取响应：

cookie, crumb = get_yahoo_crumb_cookie()
params = {
    'symbol': stock.symbol,
    'period1': 0,
    'period2': int(time.time()),
    'interval': '1d',
    'crumb': crumb,
}
url_price = 'https://query1.finance.yahoo.com/v7/finance/download/{symbol}'

response = requests.get(url_price, params=params, cookies={'B': cookie})

这看起来也不错http://blog.bradlucas.com/posts/2017-06-03-yahoo-finance-quote-download-python/