The US stock exchanges list more than 7000 companies. This huge number of stocks make it time consuming to download quotes, news or any other data. As an example, if you want to download news data for US symbols, the "yahoo news downloader" (Yahoo News) will have to send as many requests as the number of symbols that exist in your account; that is, +7000 in the case you are trading US stocks.
For the rest of this article, we will take the "yahoo news downloader" as an example.
One way to speed up the downloading process is by removing the stocks or symbols that do not have any news items. These symbols may not be recognized by Yahoo or may be too small to have news reported in the Yahoo website.
In order to remove these stocks from the list of symbols used by the downloader, we first need to search for these tickers. This can be done using the screener plug-in (Note that prior to this, you will have to execute the downloader for a few days to collect some data).
- Select "Analysis->Screener" in the menu to open the screener plug-in
- Select "Bar" and keep "n" at zero.
- We will use the "GetDataCount" function to get the number of news items for each bar. -- GetDataCount("yahoo_news", "title") --
This function will return the number of news items for each symbol, but only for the last bar. However, because we need to know whether a stock has at least one news item in all its history, we should add the "sum" function. The new formula will be: Sum(GetDataCount("yahoo_news", "title")) > 0;
- Start the screener to display the symbols that should not be removed
- Copy these symbols by clicking on the "Copy Symbols to Clipboard" button, then close the screener form.
- Select the "downloader manager" form , then open the "yahoo news" item.
- In the "Symbols" tab and under the "Name" column, select "list"
- Click on "Custom Symbols" (under the "Values" column)
- In the "Choose symbols" form, press on CTRL+V to paste the symbols or right click using your mouse then select "Paste"; Click on "OK".
- That is it; in my computer I have reduced the number of symbols from +7000 symbols to approximately 5000 symbols, which will speed up the downloading process by nearly 28%.
If you are interested in liquid stocks, you can further decrease the number of symbols by modifying the rule used in the Screener plug-in.