There are many ways for Extracting or fetching desired data or content from an html pages. During my work, i had to build a Miner to get news from multiple sites & update a wordpress blog with the latest news, daily.I had many choices to accomplish this task, few of them are
- Regular expressions
- PHP string functions.
- Using DOM Parsing
- Using Xpath.

First of all i used regular expressions to extract data from a static html page.Which was really a nightmare for me.As there is not proper structure of page, it was really hard to extract news from specific tags.Some of them were in Div,some were in TD & others in Li tags. After a long effort i couldn’t extract data using regular expressions.
I thought to use some html parser, some of them are available on internet.I used it to get data of my choice, even it couldn’t get desired data, becuase of complexity of structure of page.After that i tried to use PHP string functions, like str_replace,Trim etc, to replace specific tags with known id, so that parsing can get easy. But still, to get desired content, it was really a mess to get to them.

Finally i used firebug to analyze the content of the page, to see the content holders, during which, i cam to find exact XPath location of desired content holder.
XPath (XML Path Language) is a language for selecting nodes from an XML document. In addition, XPath may be used to compute values (strings, numbers, or boolean values) from the content of an XML document. but i wasn’t aware that Xpath can also be used with html pages, to parse the DOM tree of page.

finally, i got the exact path of Content holder, which had the latest news, daily updated. And i grabbed all the content from it.

I came to conclusion that in case of mining data from web, Xpath proved to be the best & most efficient.

In case of updates, we don’t need anything, but only the content holder Xpath tree structure of the content & we can get the content in single shot.Regular expression & other ways were very inefficient & time consuming.

Thanks to Farhan for sharing his valuable expertise with us on Fahd Murtaza’s Blog.