Using Yahoo Pipes to turn XML feeds into JSON APIs
While I've known of Yahoo Pipes for awhile, I never really thought to use it until now. Pipes is a neat tool that Yahoo has put together to allow you to mashup feeds, filter feeds and create a completely new feed.
Commonly, I've seen people take feeds from a number of different sources on a particular subject and combine them into a master feed. For example, they want a master CakePHP feed but want to grab only posts that actually mention CakePHP.
However, my goal was a little different. For a little fundraising effort, I decided to use the Chipin service. The problem is that they only offer a Flash widget. Flash doesn't work everywhere (like on the iPhone). With no API to be found, I noticed that the widget pulls in an XML file. Cross domain restrictions prevent me from really doing anything with it unless I want to use server-side code. (Which you could definitely do and I could stop this article right here…)
Being a JavaScript fiend, I decided to check out Pipes and get that XML feed turned into a JSON object.
Setting up Yahoo Pipes
For Chipin, this process was fairly straightforward. Within Pipes, grab an object from the left and drag it onto the canvas. In this case, "Fetch Data" will allow you to grab an XML or JSON document (which is exactly what we want!). Plug the Chipin URL in and then link the window to the Pipes Output window.
What's the Chipin URL, you ask? I traced out the call from Flash and it's very straightforward. When you embed the script, the widget points to:
http://widget.chipin.com/widget/id/a90ee12345b4d634Take that ID and attach it to the end of the XML call, like so:
http://widget.chipin.com/widgetfeed/id/a90ee12345b4d634Save the pipe and then click on the Run Pipes link at the top of the page. You should now see the pipe with a number of links such as Get as RSS and Get as JSON. Since we're interested in the JSON, copy the URL from the Get as JSON link. We're almost there. With the JSON URL in hand, add &_callback=myfunction to the end of it. When embedded as a script tag in your page, it'll run myfunction and pass the JSON object as its first parameter.
Here's a quick code example that can determine how much money has been raised from a campaign:
<script>
function myfunction(o) {
alert(o.value.items[0].collectedAmount);
}
</script>
<script src="http://pipes.yahoo.com/[...]&_render=json&_callback=myfunction"></script>Yahoo Pipes automatically wraps the Chipin result with its own metadata. As a result, you need to dig down through the object to get at what you need. With data in hand, modify the DOM to do your dastardly deeds.
You can take a look at the example to see how everything came together. If you'd like to see what the example looks like in the final implementation, you may check out my Adoption Fundraising campaign.
While I haven't given Yahoo Pipes much interest up until now, it looks like I have plenty of reason to consider it now.
Conversation
The twitter search plugin on my blog uses Pipes. It's definitely cool.
I like to use YQL instead of Pipes. Keeps the code all in one place. See my Github Activity Badge for an example
Yeah, if you're using JSON for your own consumption, YQL is a bit more programmery, less gimmicky.
Pipes is definitely good for producing consumable RSS, though.
Wow, YQL is more programmery but seemed a heck of a lot easier than trying to hack up a URL to do the callback. The YQL interface has it right there. I'll have to dig in a little deeper.
Interesting...The first comment in this thread (by Chris) refers to the Twitter Search WordPress plugin that I wrote using Yahoo Pipes. I have contemplated whether or not Pipes was a good service to use for this thing or just native PHP calls (via fopen, curl, etc) to search.twitter.com.
Now, YQL is brought up and seems promising. I have a question for the masses. Would you stick with Yahoo Pipes or YQL for a Wordpress Twitter plugin or just talk directly to search.twitter.com via PHP?
YQL++
The documentation isn't great but the language is pretty powerful. I wrote up my experiences consuming it from Javascript here:
http://www.kelvinluck.com/2009/02/data-scraping-with-yql-and-jquery/
@Rogie - I guess the question is does pipes or YQL make your job much easier than talking direct to twitter? If so then use them, if not then don't bother!
This guy has got Flash working on the iPhone: http://www.youtube.com/watch?v=321ueOiBw7s
David: the guy has Flash Video working on the iPhone. Sort of. All it's doing is converting Flash Video into a Quicktime format. That's different. Even more, there's only certain portals that are supported. In other words, it would do little to allow me to use a Flash widget on the iPhone.
Nice example, Jonathan. I like Pipes quite a bit; it's possible to use it to gather up material from many different sites and create a common API layer over everything. (See link to Blog Juice, above, for an example.)
In addition to RSS, Pipes also does data calls. It's an excellent secondary layer to smear over fragile APIs like Twitter; being whitelisted, it never runs out of queries, and being cached, it isn't nearly as hard on the target as individual calls.
In case you YQL fans missed it, there's a YQL module inside Pipes now, so you can write your command line and pipe it straight through, for the best of both worlds.
YQL is more programmery but seemed a heck of a lot easier than trying to hack up a URL to do the callback.
Really cool post Jonathan!
I've already created two pipes to convert the XML data from both BrightKite and Last.FM to JSON. Now I won't have to worry if I want to do something with the data from a service and they only offer XML.
I haven't looked into YQL, but I really enjoy Yahoo Pipes. Right now, I am using it to combine all of my friends' blogfeeds (atom and rss) into a single rss feed. It's nice cause I don't have to do all the caching for each of the feeds on my server. I also like the feature where you can fetch a feed by the main site's URL. It just finds the first feed on the site and reads that one. Saved me from having to find and input those manually.
The twitter search plugin on my blog uses Pipes. It's definitely awesome...
It looks like a great concept as it's the first time I've heard of Yahoo Pipes. I want to try it out.
if you're looking for an easy way to convert a yahoo pipe into html check out the jquery plug-in json2html .. http://plugins.jquery.com/plugin-tags/json2html .. all you have to do is write a simple transform and presto html!