Click here to Login





                                                   Speed of script execution for symbols with many bars

  1

0
Kyra Packens
2020-10-24 14:31:16


Hi there,

I'm running a script which processes the S&P 500 stocks on an intraday basis. Below is a small snippet to give you an idea:

>>>>
ParserTool ptool = new ParserTool();
Parser p = ptool.CreateParser(state._filterFormula);
p.UseDataOutsideRegularHours = false;
Report report = p.Parse(symbolName, state._barLength, state._barInterval == "day" ? true : false);
<<<<

What I'm trying to do is speed this script up so that I can generate trading signals in a timely fashion. The issue I'm running into is that
for these symbols I have hundreds of thousands of intraday bars of historical data, and the p.Parse line just runs too slowly no matter how much I cut
down the formula that it's executing.

From my understanding, what QS is doing in that p.Parse line is executing the formula over every bar from the symbol, and because there are so many
bars it cannot help but take a while. However, for preduction use I really only need the formula executed on the last couple thousand bars or so,
and I think if I could tell QS to do this, the Parse would be MUCH faster.

So, my question is, is there any way to do this? What I would love to see is another parameter on the Parse function that specifies what bars to execute
the formula on. Do you know of a way to do this that I may have missed, or if a feature like this has been added to a recent version of QS?

Thank you,
KP



Josh McCormick
2020-10-28 16:13:46

  0

You have an interesting problem. And now I question a decision I made just a couple of hours ago to increase my intraday data retention from 90 days to an unlimited retention policy. :/

I personally don't have a great answer here, but it made me wonder if QuantShare is capable of executing multiple scripts simultaneously. If so, it would be a pain, but you could spread the work among multiple scripts to grab more processing power. Still, this is just theory which could be completely bogus. If you're interested, the easiest way to test it out is to divide your list of symbols in half and perform an experiment with two scripts which launch (more or less) at the same time. If it works, then you could figure out how you further divide and improve on the process. I don't envy your problem.

Cheers,
Josh McCormick



Kyra Packens
2020-10-29 00:01:43

  0

Hey, thanks for the feedback!

I've found that QS can indeed execute multiple scripts simultaneously; if you go to your toolbar, right click, and 'add shortcut' to a script you can execute several in parallel. I do a lot of scripting, so I've found this to be very handy over the years, though it would be nice if script editor included the functionality directly.

As to my problem, breaking the task into numCpuCores scripts doesn't provide enough of a speedup. Was thinking along those lines too, so I went ahead and multithreaded the part of the script that's doing the heavy lifting (the lines above). It helps a bit, but nothing on the order of what reducing the bars processed from like 500k -> 5k would do. I'm considering creating alternate symbols for the stocks I am interested in that I only load the last few thousand bars of data into. I think that would fix it, tbh, but I'd hate to take on the extra maintenance overhead. Was hoping QS had an option on the Parse function to only load the most recent N bars, or that one could be added.

Thanks,
KP



QuantShare
2020-10-29 10:14:21

  0

You are correct. QS is executing the formula on all bars but data is vectorized so if we are going to execute formula on fewer bars, you will certainly see speed increase but probably not in the order of magnitude that you imagine.
Unfortunately currently this is not possible as the formula needs to be execute on all data.

Using Intraday data, you can try to instruct QuantShare to load less data using "Accounts -> Intraday Settings -> Number of days worth of intraday data to load"
If you do not need all the historical data, you can also delete it (specific period) from your database using "Data -> Edit databases -> Manage Databases -> Delete databases data"



Josh McCormick
2020-10-30 13:40:06

  0

QuantShare Support,

Besides reducing the number of day she retains data (or manually culling older data from the database), do you have any additional tips for Kyra? Or are there any enhancements that you might be able to perform on your side of things?


Kyra,

Good tip on the script shortcut. I need to do that, thanks!

I too was thinking of alternate symbols with reduced data, but I didn't want to recommend that... I feared that it would create too many unintended side-effects and extra work for you to deal with. (Also thought of putting it in a second user account, but that's likely to be an incredible pain just to .) It sounds like you're totally on top of the performance issue from a system/hardware perspective, or I would have dived into configuration issues and if you were on an SSD (which may or may not help).

I see that he thinks that reducing the data isn't going to be as helpful as you think, but you know what? It isn't going to be difficult to run a real-life performance test and see how much it's going to bring (before you commit to anything). In the past, I've backed up my QuantShare directory, ran a (destructive) experiment, and restored the files back with no harm done. Your situation may be different, but that might be worth a shot to see how much performance boost you get by culling your local database.

Just spitballing ideas here, I hope I 'm not getting in the way. I'm hoping to do some real-time intraday scanning and execution myself, so this topic interests me!

Thanks,
Josh McCormick




Kyra Packens
2020-11-01 13:30:44

  0

Quantshare:

Thank you for the suggestion on intraday days to load! I had high hopes for this when I read it as it sounds like it would do just what I am wanting, so I tested execution speeds with this set to 100 days vs unlimited. Unfortunately, as you suggested, it didn't have any effect on report generation speed in script via parser.Parse or in the GUI backtester. It DID limit data loaded on charts, though, which can be useful if you have a very heavy formula attached to a huge intraday chart (maybe also for watchlists?). Will keep that in mind.

I also tested reducing number of intraday bars given to the parser by running my script against 500 symbols of 5000 bars and comparing that to 500 symbols of 100000 bars. This resulted in around a 20x speedup, which is what I was expecting (O(n) in number of bars) and in fact what I need for this execution to be viable in production.

I would love to have an option to the parser.Parse function (and in the report GUI tool) that truly limits the number of bars (say, to a given date range) that the parser operates on. Not sure of the internals of QS so I've no idea if this is a big ask or not. Seems like it would be straightforward, but I can imagine there may be some bar synchronization issues...


Josh:

Your ideas are welcome!. As you can see, I did go ahead and run the performance comparison, which sadly turned out as we expected it would.

It'd be great if QS develops a way to limit the number of bars processed by scripts / the report GUI tool. In the meantime I will probably go ahead with creating a "historical intraday" version of my symbols with the complete intraday data and a "production intraday" version where I keep the last N bars that I need to enable my production scripts to execute in a timely manner. Kind of a pain, and with some potential pitfalls as you mentioned, but not something that I can't manage. I just would *prefer* not to :D


Thanks,
-- KP





No more messages
0




Reply:

No html code. URLs turn into links automatically.

Type in the trading objects you want to include: - Add Objects
To add a trading object in your message, type in the object name, select it and then click on "Add Objects"










QuantShare

Trading Items
Consecutive Up and Down Bars - Works With Any Trading Indicator
Stop Loss for the N-first Trading Bars only
BarsSince with support for Nth Prior or Future Occurrence
Historical Stock Quotes Data for more than 50 Exchanges
Minimum Variance Optimization with Solver V1.1- AMM

How-to Lessons
How to get stocks for a particular index using the global script
How to screen for stocks having a high correlation with the Dow J...
How to speed up watchlist and screener plug-ins when working with...
How to add future bars to plan for market scenarios
How to use QuantShare with Metastock data

Related Forum Threads
Intraday Data for NSE Symbols
Symbols with 0..9 in name
Script to remove Symbols from an Index
Help With Daily Data Import Script
Ranking with symbols not "date aligned"

Blog Posts
Different Ticker Symbols for Each Data Source
QS Trading Software: Global Script
How to Select the Best Market Indicator for your Trading System
How to speed up quotes and news downloads
How to Backtest Your Trading System for Each Industry









QuantShare
Product
QuantShare
Features
Create an account
Affiliate Program
Support
Contact Us
Trading Forum
How-to Lessons
Manual
Company
About Us
Privacy
Terms of Use

Copyright 2020 QuantShare.com
Social Media
Follow us on Facebook
Twitter Follow us on Twitter
Google+
Follow us on Google+
RSS Trading Items



Trading financial instruments, including foreign exchange on margin, carries a high level of risk and is not suitable for all investors. The high degree of leverage can work against you as well as for you. Before deciding to invest in financial instruments or foreign exchange you should carefully consider your investment objectives, level of experience, and risk appetite. The possibility exists that you could sustain a loss of some or all of your initial investment and therefore you should not invest money that you cannot afford to lose. You should be aware of all the risks associated with trading and seek advice from an independent financial advisor if you have any doubts.