System for data scraping, analysis and prediction

bjardnin — Fri, 30 Oct 2009 20:33:30 +0000

In this article a system is designed, that collects data from publicly available databases from the internet, performs data analysis using General Systems Theory and applying suitable methods for time series prediction based on the obtained data.

Data collection is made by parsing HTML pages that are transformed to XML using software Web-harvest. It uses language XSLT for parsing HTML and extraction in XML. The files in the XML format posses uniform structure. After the files are saved, they are stored in database technology JiBX that allows to manipulate the data in XML as objects.

The systems of variables are decomponed and then connected in order to figure out, how the decomposition approximates the original system. System analysis is performed using standard statistic test, the test.

Prediction is performed using the methodology of General Systems Theory, another possibility is the method Support Vector Machines (SVM) for time series forecasting, see [1] ,[2]. Finally the predicted values are tested using t-test and F-test.

[1]: Lijuan Cao: Support vector machines experts for time series forecasting, Neurocomputing 51: 321-339, 2003.
[2]: Alex J. Smola and Bernhard Schölkopf: A Tutorial on Support Vector Regression, Statistics and Computing, 1998.

bj's Blog

System for data scraping, analysis and prediction