Sunday, March 08, 2009

Yahoo Pipes

Pipes - Documentation

Here is an excellent idea - Yahoo! Pipes is rewiring the web. It is a service that allows for fetching and processing the data from the Internet and then presenting the output of that process. To simplify, I could process a web page of the weather forecast service and pick up only the current temperature. Then, that number could be presented on another page, aggregating the current temperatures in a few selected cities.
Of course, all of this can be done if your web server supports processing. Then, simply writing the same thing in ASP.Net or PHP or Python would allow for the same thing. But, Yahoo! Pipes allows the same thing for web sites that are built with pure HTML (and JavaScript, to be precise).
Since I'm still using the above technologies, I prefer to offload that work to a service like Yahoo! Pipes and keep the pages in pure HTML.

The example above is only one possible scenario. The data that can be processed includes RSS, JSON, HTML and other formats available on the Internet.
I'm off to studying the documentation. And, since I already have a few ideas on how to use the pipes, I'll be off to creating some pipes that concentrate the economic data. One of the first things could be the current official interest rate aggregator. (Update: the current interest rates are now available at my Forex page - here)

There is JSON2HTML tool available here, which can be used to visualize the output of a pipe.
RSS2HTML project (link) does the same with RSS.
Yahoo also provides Yahoo Query Language (YQL). The console is here. It is possible to query any data source it supports.
Dapper is another html scraping tool that can be used instead of Pipes, in case only html scraping is needed.

No comments: