Click here to Login





                                                   Downloaders

  0

0
David
2020-04-29 02:04:06


I appear to be having an issue with the downloaders, that seems to be based on the GMT of the exchange where the data is coming from.
I only pull down data from the NZX and ASX (NZ and Australia) and also monitor some key US indices like the S&P500 DJI etc

Recently with my ASX symbols I needed to know the days of the week.
Used DayOfWeek() and noticed some days were listed as 0 and some 6 i.e. Sunday and Saturday.
Thinking there was an error in the data, I downloaded the entire database again with the "Historical Stock Market Data" downloader,
again it shows up this inconsistency occasionally with Sunday and Saturday listed as trading days, also tried with "Yahoo EOD Quotes" and same thing.

Created a new account with just ^NZ50,^AORD,^GSPC
Used Yahoo EOD quotes to download from April 1st to April the 24th 2020
For the NZ50 16 bars are downloaded with dates from the 31st March to the 23rd of April , but with the data from the 1st of April to the 24th of April
For the AORD It starts at the 31st of March and ends at the 24th, also 16 bars. Pricing after the 6th of April is correct, before is following day pricing.
Also it jumps from the 2nd of April to the 6th of April, Friday the 3rd is missing.
For the ^GSPC it is perfect.

NZ is GMT + 12, AU is GMT + 10, US-New York is GMT -4

Had a look at the Downloaders URL script, and tried adjusting the P1 and P2 values with UNIX time offsets, but not much luck.

Mucked around a bit with the generated URL from yahoo EOD Quotes (AORD below) and cant seem to alter the outcome.

Putting the below into a browser like Firefox will get the data nicely formatted in JSON, and you can check the first timestamp is 31st Mar 2020

https://query2.finance.yahoo.com/v8/finance/chart/^aord?formatted=true&crumb=&lang=en-US&region=US&period1=1585699200&period2=1587735173&interval=1d&events=div%7Csplit&corsDomain=finance.yahoo.com

Also had a look at indices closer to the GMT like a number of the European ones and these seemed correct

So just a "too far from everything" issue? Easy fix?

Cheers




David
2020-05-08 21:37:33

  0

Best Answer
Ok, been trying alternate downloaders like YLoader etc to get around this date issue, but found other small quirks using these packages.

Had a closer look at the Data Yahoo was presenting for download via say "Yahoo EOD" and it seems the timestamp has been adjusted for the exchanges GMT offset.

So all I have done is altered the "Pre Script" to add the exchanges GMT offset to the timestamps and this appears to have fixed the problem.

All my ASX and NZX data is now correct, i.e. no Sunday or Saturday trading days. Also checked the S&P500/DJI and also appears correct. (note, was not wrong previously, but this didn't break it)

Not 100% sure this will work with every exchange, hence not uploading, but for GMT > 8ish exchanges like Australia and New Zealand. the data and dates, line up correctly now.

Pre Script below , altered lines have //*

System.Threading.Thread.CurrentThread.CurrentCulture = System.Globalization.CultureInfo.InvariantCulture;
for(int i=0;i<Content.Rows.Length;i++)
{
Content.Rows[i].IsIgnoreLine = true;
}

string content = Content.GetContent();
JsonData j = Content.Net.ReadJsonData(content);

//Content.AddRow(j["chart"]["result"][0]["indicators"]["quote"][0]["close"].ToString());

DateTime unix = new DateTime(1970, 1, 1);
JsonData r = j["chart"]["result"][0];
JsonData q = r["indicators"]["quote"][0];
JsonData t = r["timestamp"];
JsonData c = q["close"];
JsonData o = q["open"];
JsonData h = q["high"];
JsonData l = q["low"];
JsonData v = q["volume"];
JsonData gmt = r["meta"]["gmtoffset"]; //*
for(int i=0;i<c.Count;i++)
{
int tt,os; //*
if(int.TryParse(t[i].ToString(), out tt))
{
try{
string open = (o[i] == null) ? "" : o[i].ToString();
string high = (h[i] == null) ? "" : h[i].ToString();
string low = (l[i] == null) ? "" : l[i].ToString();
string close = (c[i] == null) ? "" : c[i].ToString();
string volume = (v[i] == null) ? "" : v[i].ToString();
int.TryParse(gmt.ToString(), out os); //*
tt += os; //*
Content.AddRow(unix.AddSeconds(tt).ToString("MM/dd/yyyy", System.Globalization.CultureInfo.InvariantCulture), open, high, low, close, volume);
}
catch
{
}
}
}




No more messages
0




Reply:

No html code. URLs turn into links automatically.

Type in the trading objects you want to include: - Add Objects
To add a trading object in your message, type in the object name, select it and then click on "Add Objects"










QuantShare

Trading Items
BSE stock quotes
Rank Industries based on the average Relative Strength Index
Israel Stock Market
Update the Industry Name of your U.S. Stocks
Price to Book Ratio Historical Data - Market to Book

How-to Lessons
How to quickly select stocks based on the last value of a databas...
How to download EOD quotes for active and valid stocks only
How to quickly download the most recent EOD data for your stocks
How to download a trading item from the Sharing Server
How to save and restore charts

Related Forum Threads
downloaders using google trends data
Daily and Historical Downloaders
Script to remove Symbols from an Index
Symbol Information
Reference Data for European ETF?

Blog Posts
Creating a download item: Initial Jobless Claims
Charting, Backtesting and Trading using Fundamental Data
How to Download Trading Data for Certain Securities Only
Running QuantShare on Amazon's EC2 Cloud & Automate Strategies Si...
German Stock Exchanges: 5 free ways to download historical EOD/In...









QuantShare
Product
QuantShare
Features
Create an account
Affiliate Program
Support
Contact Us
Trading Forum
How-to Lessons
Manual
Company
About Us
Privacy
Terms of Use

Copyright 2020 QuantShare.com
Social Media
Follow us on Facebook
Twitter Follow us on Twitter
Google+
Follow us on Google+
RSS Trading Items



Trading financial instruments, including foreign exchange on margin, carries a high level of risk and is not suitable for all investors. The high degree of leverage can work against you as well as for you. Before deciding to invest in financial instruments or foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with trading and seek advice from an independent financial advisor if you have any doubts.