I am not able to work successfully with those downloaders which aim to pull data from google trends. I have set my google email and password for these downloaders but seems to be not able to download data. As its not accessing the links, i am not able to go to parser and see the contents too. This includes downloaders like
Rosenberg Sentiment Index, Permabear Sentiment Index,etc.
After having snooped around on the internet and existing downloaders, it seems it is possible.
In particular, it is possible to download CSV files manually as long as you are logged in.
If you are in for it, please try the following attempt at solution, based on Tom Huggens Permabear downloader Permabear Sentiment Index and other sources from which I have shamelesly copied :-). Main credits maybe to Tom Huggens and http://stackoverflow.com/questions/4986758/oauth-google-trends-download-csv-file.
The proposed changes could certainly need some quality control and refinement...
Some points to bear in mind:
1. Zero-values are no longer removed in the pre-script, just because I want them there. Feel free to change back.
2. I removed the mapping of " " (Space) to hex-code for "+" in the pre-script. With original mapping results seemed to be wrong With current Yahoo trends ("+" == or-operation?). Can anyone verify this?
3. You can successfully run the download only a few times a day, it seems. Google trends now have a strict quota (try to Google "Google trends quota limit"). This does not seem to apply for manual downloads, so somehow it should be possible to avoid, but seems difficult.
4. The modified downloader suggested below let you change the terms to search for (e.g. "Roubini,PeterSchiff...") without changing scripts. A comment in the web page mentioned above suggests at most 5 terms can be used together. I haven't tested this yet.
5. Even though the pre-script sets the row's IsIgnoreLine=true when it doesn't recognise a date at the start of the row, you still need to enable "Continue parsing even if there are errors", and there Will be errors. I don't understand why, if someone can explain, please do.
6. The close date/time will be the start of the week for which the data applies. Could make more sense to use the end of the week?
7. The URL-script communicates the symbol name to the pre-script through a global variable. This might give problems if running several instances of the downloader.
***********************
Steps to make updated downloader:
2. Add 3 more fields, to a total of 5: email, password, symbol,query and work_dir. The two first are described in the permabear downloader.
You may want to make a special email account at Google for this since the password is sent in cleartext on the internet.
symbol=^GOOGLE_TRENDS_PERMABEAR (or anything else you want)
query=Roubini,Peter Schiff,Marc Faber,Nassim Taleb (or anything else you want, though may be limited to 5 terms by Google)
work_dir =C:\ (or anything else you want. CSV-files will remain here).
3. Click on Settings/Create URL-script, replace contents With URL-script given below. Compile. Press OK.
4. Press Parser, OK, Next, and you should be in the CSV settings window.
Change Separator to "Space". Turn on "continue parsing even if there are errors" (I don't understand why I get errors, but I do).
You may also delete the default value of Column 3, no longer needed.
5. Press "Pre Script" and change contents With pre-script provided below. Compile. Press OK.
6. Press next, etc, and check that values are OK.
Remember that you can only run the downloader successfully a few times every day, due to the quota of Google. For testing it can be useful to keep a downloaded .csv-file (could be downloaded manually) but rename it, then comment out the downloading part and change the filename in the URL-script.
If you are over the quota, the .csv file will be a small file containing "...You have reached Your quota limit...".
// google limits 5 sets of terms per request?
using System.Net;
using (System.Net.WebClient client = new WebClient()) {
string symbol = Functions.GetValues("symbol");
Global.SetVariable("google_trends_symbol", symbol); // Save symbol name for pre-script
string filename = System.String.Format("{0}\\report_{1}.csv", Functions.GetValues("work_dir"), symbol);
string keyword = Functions.GetValues("query");
// Replacing " " with "+" as in original seems wrong now...
//keyword = keyword.Replace(" ", "%2B").Replace(",", "%2C");
keyword = keyword.Replace(",", "%2C");
string username = Functions.GetValues("email");
string password = Functions.GetValues("password");
string response = client.DownloadString(string.Format("https://www.google.com/accounts/ClientLogin?accountType=GOOGLE&Email={0}&Passwd={1}&service=trendspro&source=test-test-v1", username, password));
// The SID is the first line in the response
// The Auth line
string auth = response.Split('\n')[2];
client.Headers.Add("Authorization", "GoogleLogin " + auth);
byte[] csv = client.DownloadData(string.Format("http://www.google.com/trends/trendsReport?hl=en-US&q={0}&cmpt=q&content=1&export=1", keyword));
System.IO.File.WriteAllBytes(filename, csv);
Functions.AddURL(filename, filename);
}
####################### Updated pre-script below #################################
string symbol = (string)Global.GetVariable("google_trends_symbol"); // Saved by URL-script
for(int i=0;i
What a brilliant effort! I can understand the soul/sole searching behind this...congrats and a big thank you. The URL script gives me a compile error for L15 C65 and L20 C58 saying ")"expected. Sorry for being so code-impaired to be not able to fix such a small issue myself. Waiting eagerly to proceed to the pre-script and see it working :)
Hi,
the problems seem to be caused by the conversion to HTML performed when added to this forum as well as cutting/pasting HTML (I got different results when cutting from Internet Explorer and pasting into notepad and WordPad!).
I have put the code into two files (with some updates to at least the pre-code), available at https://www.dropbox.com/sh/5qsf0ewk0wa90ab/cE9hZhpmtP
I appreciate comments/suggestions/updates to the code.
QuantShare (or anyone else):
Is it possible to access the downloader fields from the pre-script, so we can avoid the use of global variable which can be problematic if several instances of the downloader run simultaneously?
Any comments on why there are errors on lines even though row.IsIgnoreLine is set to true? Any misunderstanding in the way IsIgnoreLine is used?
Yes, the notepad pasting has proceeded without any compile error. Thanks for that. Are these numbers in some multiples? Because, the highest number is shown for a particular week during Oct.'08 with a number of 157. It cant be only 157 searches right? Should be some multiple, i guess. Once again a big thank for the wonderful work from your side.
QuantShare, thanks for the good tip to question 1.
I just updated the files available at https://www.dropbox.com/sh/5qsf0ewk0wa90ab/cE9hZhpmtP accordingly.
Seeker:
Thanks for Your kind words. Glad that it worked!
I recommend you use the files available from the link mentioned above, if you didn't already.
I did not get correct results using cut/past with notepad I think...
Btw. I get 156, not 157 as the highest, but could be because of changed handling of "empty" results from earlier.
Regarding "multiples": Yes, I think there is a rescaling done. For interactive chart on Google Trends this is described here:
https://support.google.com/trends/topic/13975?hl=en&ref_topic=13762
See in particular "How is the data scaled".
The CSV file contents are not necessarily exactly the same as interactive graphs, at a minimum the CSV files have week-resolution and the interactive graphs have month-resolution.
My best guess is that the scaling is more or less the same for CSV and interactive graphs. In both cases it might seem that when using multiple search terms ("Roubini", "Peter Schiff" and so on) in one search, all terms are scaled by by the same value, so only the most popular term(s) will reach 100.
If this is correct, I guess the Permabear (for example) should have a theoretical range of 0 to 400, since (for the permabear) we add 4 terms, but 400 would only be reached if all the terms have the same maximum popularity.
Something like that, anyway...
Furthermore, if a search term reaches new heights in popularity at a future time, this Will likely change the calculated values backwards in time.
Not ideal maybe... But don't know any easy way of fixing it without Google changing their data.
Made a couple of changes to the pre-script to
a) get rid of loads of error messages (hopefully the old results were still correct)
b) timestamp = the last day in the interval (earlier: first)
Download from https://www.dropbox.com/s/u7ob2jg2q616g04/pre2.cs
Hi,
I just uploaded it. Since it is based on Tom Huggens' Permabear downloader etc, which I hope I have documented enough, I got the message
"This Object was already uploaded.
You must not upload objects created by other users
Are you sure you want to upload this item?".
If I am breaking any rules here, or "overwrite" Tom's original downloader item, please delete the new item.
Trading financial instruments, including foreign exchange on margin, carries a high level of risk and is not suitable for all investors. The high degree of leverage can work against you as well as for you. Before deciding to invest in financial instruments or foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with trading and seek advice from an independent financial advisor if you have any doubts.
0
