Let me ask this question - how many matches one needs to analyse to say with high confidence that the trigger brings profit? 1000? 10000? less? more?
I don't think one can say with 100% confidence how many markets are needed to test your betting strategy. It depends on so many factors, e.g. how frequently does your trigger bet, whether you have had many losing or winning streaks, how your bank has changed over the time, whether you take high risks, etc.
1) From my observation I can see that matches are probably loaded from a giant file (tm.db) to the memory - note that about 35 football matches take over 1GB of disk space. In comparison, the history files that one can download from the official betfair website take about 5-6GB for the whole month worth of data.
You are probably referring to the
Historical Data service. If we are talking about their free data, then the frequency is 1 min, whereas in Time Machine the frequency is 0.5 sec. This evidently explains the difference in the file size. The number of betting strategies you can test at 1 min is very limited.
I completely agree with all the points you have listed. We are trying to find a solution, but can't see it at the moment (BetFair is struggling with this amount of data too, see my message above). On the one hand, customers want the ability to test bets based on price fluctuations or in-play markets where prices change dramatically with each second, so 1-minute frequency is not an option.
On the other hand, everyone want to have the flexibility of loading a month's data and running their strategies within an instant.
We still need to find a technical balance between the two (very valid) customer expectations.
Among the options I can see here is the ability to run a cloud testing service where customers would load their triggers to our powerful server, specify the markets they want to test them against and choose the notification method (i.e. by email, Telegram, logs, etc.). This would obviously limit their ability to place bets manually or control the flow of Time Machine, but at the same time this would speed up the process several times. We are considering this route, but we need to study the demand for such service, as it will be pricey and will require a big investment of time and effort. Do you think such service would make sense?