Click here to Login





                                                   downloaders using google trends data

Back to threads - Tags: --
  0

0
Seeker
2013-10-27 12:19:56


I am not able to work successfully with those downloaders which aim to pull data from google trends. I have set my google email and password for these downloaders but seems to be not able to download data. As its not accessing the links, i am not able to go to parser and see the contents too. This includes downloaders like
Rosenberg Sentiment Index, Permabear Sentiment Index,etc.



QuantShare
2013-10-28 13:38:51

  0

These items were shared by one of our members. I have checked them and it looks like Google no longer allows exporting trends data to CSV.



Seeker
2013-10-28 15:49:15

  0

Oh! Thats very disappointing :)


Zoidberg
2013-11-04 21:53:46

  0

After having snooped around on the internet and existing downloaders, it seems it is possible.
In particular, it is possible to download CSV files manually as long as you are logged in.
If you are in for it, please try the following attempt at solution, based on Tom Huggens Permabear downloader Permabear Sentiment Index and other sources from which I have shamelesly copied :-). Main credits maybe to Tom Huggens and http://stackoverflow.com/questions/4986758/oauth-google-trends-download-csv-file.
The proposed changes could certainly need some quality control and refinement...

Some points to bear in mind:
1. Zero-values are no longer removed in the pre-script, just because I want them there. Feel free to change back.
2. I removed the mapping of " " (Space) to hex-code for "+" in the pre-script. With original mapping results seemed to be wrong With current Yahoo trends ("+" == or-operation?). Can anyone verify this?
3. You can successfully run the download only a few times a day, it seems. Google trends now have a strict quota (try to Google "Google trends quota limit"). This does not seem to apply for manual downloads, so somehow it should be possible to avoid, but seems difficult.
4. The modified downloader suggested below let you change the terms to search for (e.g. "Roubini,PeterSchiff...") without changing scripts. A comment in the web page mentioned above suggests at most 5 terms can be used together. I haven't tested this yet.
5. Even though the pre-script sets the row's IsIgnoreLine=true when it doesn't recognise a date at the start of the row, you still need to enable "Continue parsing even if there are errors", and there Will be errors. I don't understand why, if someone can explain, please do.
6. The close date/time will be the start of the week for which the data applies. Could make more sense to use the end of the week?
7. The URL-script communicates the symbol name to the pre-script through a global variable. This might give problems if running several instances of the downloader.

***********************
Steps to make updated downloader:

1. Download Permabear downloader Permabear Sentiment Index

2. Add 3 more fields, to a total of 5: email, password, symbol,query and work_dir. The two first are described in the permabear downloader.
You may want to make a special email account at Google for this since the password is sent in cleartext on the internet.
symbol=^GOOGLE_TRENDS_PERMABEAR (or anything else you want)
query=Roubini,Peter Schiff,Marc Faber,Nassim Taleb (or anything else you want, though may be limited to 5 terms by Google)
work_dir =C:\ (or anything else you want. CSV-files will remain here).

3. Click on Settings/Create URL-script, replace contents With URL-script given below. Compile. Press OK.

4. Press Parser, OK, Next, and you should be in the CSV settings window.
Change Separator to "Space". Turn on "continue parsing even if there are errors" (I don't understand why I get errors, but I do).
You may also delete the default value of Column 3, no longer needed.

5. Press "Pre Script" and change contents With pre-script provided below. Compile. Press OK.

6. Press next, etc, and check that values are OK.

Remember that you can only run the downloader successfully a few times every day, due to the quota of Google. For testing it can be useful to keep a downloaded .csv-file (could be downloaded manually) but rename it, then comment out the downloading part and change the filename in the URL-script.
If you are over the quota, the .csv file will be a small file containing "...You have reached Your quota limit...".



####################### Updated URL-script below #################################
// Lots of code from http://stackoverflow.com/questions/4986758/oauth-google-trends-download-csv-file;
// as well as original PermaBear downloader (etc?).

// google limits 5 sets of terms per request?
using System.Net;
using (System.Net.WebClient client = new WebClient()) {
string symbol = Functions.GetValues("symbol");
Global.SetVariable("google_trends_symbol", symbol); // Save symbol name for pre-script
string filename = System.String.Format("{0}\\report_{1}.csv", Functions.GetValues("work_dir"), symbol);
string keyword = Functions.GetValues("query");
// Replacing " " with "+" as in original seems wrong now...
//keyword = keyword.Replace(" ", "%2B").Replace(",", "%2C");
keyword = keyword.Replace(",", "%2C");

string username = Functions.GetValues("email");
string password = Functions.GetValues("password");
string response = client.DownloadString(string.Format("https://www.google.com/accounts/ClientLogin?accountType=GOOGLE&Email={0}&Passwd={1}&service=trendspro&source=test-test-v1", username, password));
// The SID is the first line in the response
// The Auth line
string auth = response.Split('\n')[2];
client.Headers.Add("Authorization", "GoogleLogin " + auth);
byte[] csv = client.DownloadData(string.Format("http://www.google.com/trends/trendsReport?hl=en-US&q={0}&cmpt=q&content=1&export=1", keyword));
System.IO.File.WriteAllBytes(filename, csv);

Functions.AddURL(filename, filename);
}

####################### Updated pre-script below #################################
string symbol = (string)Global.GetVariable("google_trends_symbol"); // Saved by URL-script
for(int i=0;i





Permabear Sentiment Index (by Tom Huggens, uploaded several months ago)
No notes

Rate an item Rate an item Rate an item Rate an item Rate an item Number of downloads Notes Report an item

Seeker
2013-11-06 20:18:38

  0

What a brilliant effort! I can understand the soul/sole searching behind this...congrats and a big thank you. The URL script gives me a compile error for L15 C65 and L20 C58 saying ")"expected. Sorry for being so code-impaired to be not able to fix such a small issue myself. Waiting eagerly to proceed to the pre-script and see it working :)


QuantShare
2013-11-07 12:08:19

  0

Please copy the code again


Zoidberg
2013-11-07 14:42:48

  0

Hi,
the problems seem to be caused by the conversion to HTML performed when added to this forum as well as cutting/pasting HTML (I got different results when cutting from Internet Explorer and pasting into notepad and WordPad!).

I have put the code into two files (with some updates to at least the pre-code), available at https://www.dropbox.com/sh/5qsf0ewk0wa90ab/cE9hZhpmtP

I appreciate comments/suggestions/updates to the code.

QuantShare (or anyone else):
Is it possible to access the downloader fields from the pre-script, so we can avoid the use of global variable which can be problematic if several instances of the downloader run simultaneously?
Any comments on why there are errors on lines even though row.IsIgnoreLine is set to true? Any misunderstanding in the way IsIgnoreLine is used?



QuantShare
2013-11-07 18:57:04

  0

Hi,

1/ The only way to get these fields without Global variables is to set the file name to something like:
report_[Field1]_[Field2]...

Then read and parse the file name using the following function:
Content.GetURLORFileName()

2/ You shouldn't get any error if "IsIgnoreLine" is set to TRUE. The issue is probably related to something else.




Seeker
2013-11-09 00:06:56

  0

Yes, the notepad pasting has proceeded without any compile error. Thanks for that. Are these numbers in some multiples? Because, the highest number is shown for a particular week during Oct.'08 with a number of 157. It cant be only 157 searches right? Should be some multiple, i guess. Once again a big thank for the wonderful work from your side.


Zoidberg
2013-11-09 00:49:13

  0

Hi,

QuantShare, thanks for the good tip to question 1.
I just updated the files available at https://www.dropbox.com/sh/5qsf0ewk0wa90ab/cE9hZhpmtP accordingly.

Seeker:
Thanks for Your kind words. Glad that it worked!
I recommend you use the files available from the link mentioned above, if you didn't already.
I did not get correct results using cut/past with notepad I think...
Btw. I get 156, not 157 as the highest, but could be because of changed handling of "empty" results from earlier.

Regarding "multiples": Yes, I think there is a rescaling done. For interactive chart on Google Trends this is described here:
https://support.google.com/trends/topic/13975?hl=en&ref_topic=13762
See in particular "How is the data scaled".
The CSV file contents are not necessarily exactly the same as interactive graphs, at a minimum the CSV files have week-resolution and the interactive graphs have month-resolution.
My best guess is that the scaling is more or less the same for CSV and interactive graphs. In both cases it might seem that when using multiple search terms ("Roubini", "Peter Schiff" and so on) in one search, all terms are scaled by by the same value, so only the most popular term(s) will reach 100.
If this is correct, I guess the Permabear (for example) should have a theoretical range of 0 to 400, since (for the permabear) we add 4 terms, but 400 would only be reached if all the terms have the same maximum popularity.
Something like that, anyway...
Furthermore, if a search term reaches new heights in popularity at a future time, this Will likely change the calculated values backwards in time.
Not ideal maybe... But don't know any easy way of fixing it without Google changing their data.



Seeker
2013-11-09 12:47:48

  0

Yes. I have pasted it from your link only which was perfect without any compiler error.


Zoidberg
2014-01-30 22:58:35

  0

Best Answer
Made a couple of changes to the pre-script to
a) get rid of loads of error messages (hopefully the old results were still correct)
b) timestamp = the last day in the interval (earlier: first)
Download from https://www.dropbox.com/s/u7ob2jg2q616g04/pre2.cs



QuantShare
2014-01-31 22:47:03

  0

Thanks. Can you upload the new downloader to the sharing server?



Zoidberg
2014-02-01 05:55:00

  0

Hi,
I just uploaded it. Since it is based on Tom Huggens' Permabear downloader etc, which I hope I have documented enough, I got the message
"This Object was already uploaded.
You must not upload objects created by other users
Are you sure you want to upload this item?".

If I am breaking any rules here, or "overwrite" Tom's original downloader item, please delete the new item.

- Zoidberg



QuantShare
2014-02-01 11:33:53

  0

It is ok, thank you.



No more messages
0




Reply:

No html code. URLs turn into links automatically.

Type in the trading objects you want to include: - Add Objects
To add a trading object in your message, type in the object name, select it and then click on "Add Objects"










QuantShare

Trading Items
Google domestic trends data
Google domestic trends indexes
Google Trends Sentiment Indices
News data for US companies from Google RSS feed
Toronto Stock Exchange - historical data

How-to Lessons
How to download historical EOD data for penny stocks
How to speed up watchlist and screener plug-ins when working with...
How to quickly download the most recent EOD data for your stocks
How to download and use U.S. stocks earnings data
How to create trading rules based on Put and Call volume data

Related Forum Threads
Download data from google
Using the same data in Amibroker and Quantshare
Plotting 3D Surface Plots in Excel using Optimization Data
How to Debug a Strategy using data variables values by day by Tic...
Removing freak data from the database

Blog Posts
Download quotes, news, sentiment, fundamental... data using Quant...
Backtesting trading strategies using intraday data
Charting, Backtesting and Trading using Fundamental Data
Download Trading Data using the Post-Script
Creating Stock & Market Short Interest Ratios using Historical Sh...









QuantShare
Product
QuantShare
Features
Create an account
Affiliate Program
Support
Contact Us
Trading Forum
How-to Lessons
Manual
Company
About Us
Privacy
Terms of Use

Copyright 2021 QuantShare.com
Social Media
Follow us on Facebook
Twitter Follow us on Twitter
Google+
Follow us on Google+
RSS Trading Items



Trading financial instruments, including foreign exchange on margin, carries a high level of risk and is not suitable for all investors. The high degree of leverage can work against you as well as for you. Before deciding to invest in financial instruments or foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with trading and seek advice from an independent financial advisor if you have any doubts.