Some Details On How I Did The Recent Stock Market Analysis
One of the few “regulars” on this blog asked me a question in regards to the recent stock market analysis I did. The essential question was:
“Could you say something about the process of collecting and analyzing the data? Where did you get it, what software did you use, how long did it take you to do this analysis etc.?”
To elaborate on the brief explanation outlined in this article about the Very Large Dataset Project, I made a fairly complex Visual Basic for Applications program in Microsoft Excel that would “scrape” information for each ticker symbol and place it into a spreadsheet. After having done this scrape periodically, I had a sizeable quantity of data from which to work with. To do the actual analysis, I spent several days arranging the data in different ways, sorting different fundamental stock statistics according to their yields and making pictures of the graphs that summarized the different things I thought were useful.
If there is enough interest in the Excel code, I may have the time to publish that on this blog for its readers, but it’s not an “out of the box” kind of application—it would require some tutorial to use. The actual gathering of the data takes many hours, and has to be run in smaller pieces (I run 8 separate files, each with about 1000 ticker symbols) because the program slows down considerably when trying to pull all the data all at once (in other words, as the spreadsheet grows in size, it also runs less quickly).
Thanks for the clarification.
The reason I am so interested in your analysis process is because I wish to compare the effort involved with the results, and more importantly (for me), what kind of effort I would have to put in to achieve the same results.
I too have been using Excel but have become increasingly frustrated with its inefficiency, ie. all the scrolling and clicking. I can see that your programming experience has probably helped you a lot with automating things.
Recently I stumbled upon an open source statistical analysis software/programming language called R, which is supposedly what the professional scientific community uses. It seems to be much more efficient than excel, but with a steeper learning curve. Perhaps you will find it easier than I have as you have some programming experience. Here is the link to the official website:
http://www.r-project.org/index.html
In addition, a google search for “econometrics in R” will yield some introductory texts, that I think are very helpful.