Live data from Hacker News

Yahoo Pipes

en.wikipedia.org

71–80 of 97 posts

Re: Yahoo Pipes

#71

I was a big user of Yahoo Pipes for preparing custom RSS feeds. I have now found that https://feedity.com/ does a good enough job scraping pages to produce feeds. I then run that through https://fivefilters.org/content-only/ to fetch the full content of the page and read it in https://www.inoreader.com/ , my favorite Google Reader clone. Some people are amazed that I am always on top of new content they produce.

Hey Dan, Feedity's founder here. Thank you for using our feeds service! Just so you know, besides HTML webpages you can also create feeds for dynamic webpages with client-scripts, as well as social media profiles.

Re: Yahoo Pipes

#72

I was at the O'Reilly Emerging Tech conference session that functioned as (what I remember to have been) the Pipes launch party. Everyone involved with the project seemed young and full of optimism, and while they didn't know exactly how Pipes would make money, I remember being as impressed with the UI as I was the forward-thinking ideas that motivated the project. Keep in mind that in 2006, IE 6 was still omnipresen…

> UX trailblazers like Flickr and DabbleDB.

And Meebo! They build an entire windowing system in the browser...and supported IE6.

Re: Yahoo Pipes

#73

I feel like it's similar to Airflow https://airflow.apache.org/

It is, but it provided a ton of web-centric APIs in an era where people published useful RSS feeds. Half the value was in publishing modules that spoke RSS in some form, and the other half was a number of online query systems that spoke HTTP and responded with RSS.

There were many technical problems with pipes and one major business problem. The technical problems as usual, were solvable:

- branching. the system really didn't have a clean way to cause branching. if you wanted to only apply a regex on certain items, that would fail hard. I feel like some part of the anime fansub system I built must have been stressing the system because I'd occasionally find it disabled or key components removed from Pipes. (Deluge can consume RSS in the background, but often the fansub RSS feeds have duplicate items for different qualities, or are missing the enclosure field the system keys off of and other field normalization problems)

- types. the system really operated on RSS as a medium of data exchange. Any operation that worked on whole RSS feeds was great. So the typical 'put a ton of blogs into a pipe and union them all into one big feed' hello world app worked great. As soon as you wanted to operate on a specific line of input items and do subprocessing, you were in a sort of hell. It wasn't clear which modules worked on which types, or why.

- composition. a general engineering principle is to build things out of small parts. That's the fundamental reason pipes works, but you never had the ability to treat a pipe you built as a small part itself. You couldn't build pipes that accepted RSS in. The subpipe command they eventually added only allowed RSS out, without even any parameterization. Judging by the comments from support engineers here, the reason is mostly obvious: they couldn't prevent people from building pipes that called themselves indirectly.

But the business problem was most fundamental. The most useful APIs were often monetized via web ads. Yea, someone was collecting amazon referral revenue, but beyond that, no money was changing hands. That was a fundamental challenge to further adoption. Bloggers started publishing truncated or headline blog feeds, and what sort of evolved was a cat and mouse game with content creators: people invent new ways to generate RSS feeds for sites that don't have them while the sites fuzz their UI to prevent it.

Re: Yahoo Pipes

#74

As a fresh Yahoo! Developer Network employee I was the person who blabbed about the top-secret _callback parameter the weekend Pipes launched, which caused the service to be taken offline for capacity upgrade. I had an interesting series of voicemails, ranging from "dude, WTF?" to "okay, we need to talk to you right away" to "we're pretty sure you're fired" but ending with "hey, Tim O'Reilly picked it up and ALL IS F…

What were the implications of knowing the _callback parameter?

Re: Yahoo Pipes

#76
post #3

Pipes was such a good product and sorely missed. Is there anything equivalent to it today?

I'm in the middle of pushing out v1.0 of Simple scraper[0] this week which will recreate some of the functionality.

Having a lot of fun building the product but with so many of these types of companies shutting down it has me wondering what it forebodes :/

[0] https://simplescraper.io

Re: Yahoo Pipes

#77

As a fresh Yahoo! Developer Network employee I was the person who blabbed about the top-secret _callback parameter the weekend Pipes launched, which caused the service to be taken offline for capacity upgrade. I had an interesting series of voicemails, ranging from "dude, WTF?" to "okay, we need to talk to you right away" to "we're pretty sure you're fired" but ending with "hey, Tim O'Reilly picked it up and ALL IS F…

What were the implications of knowing the _callback parameter?

You could invoke pipes from any webpage without worrying about cross-domain access.

If you look at some janky JS code[1] I had on my website at the time, you can see what you can do with it.

That would basically source

://pipes.yahoo.com/pipes/pipe.run?_id=7CTRtbtL3BGX_AHbjknRlg&_render=json&_callback=load_daily_show

as a static asset in my page, which would load the JS and call load_daily_show() as a result.

Now, EVERYONE who hits my page is invoking the pipe as a backend API call, with no caching and unfortunately with the entire Y/T cookies intact.

[1] - https://web.archive.org/web/20081007043923/http://t3.dotgnu....

Re: Yahoo Pipes

#78
post #16

Thanks for posting and the nice comments. Glad to have built something that people found useful...

Sorry, one last note. I was a broke college student at the time. I made tens of thousands of dollars with Pipes doing ethical arbitrage. I still miss it, where's our Kickstarter?

arbitrage of what? Sounds interesting

Re: Yahoo Pipes

#79
Surprised to see no mention of Apache NiFi, Quartz Composer, or Simulink:

https://en.wikipedia.org/wiki/Apache_NiFi

https://en.wikipedia.org/wiki/Quartz_Composer

https://en.wikipedia.org/wiki/Simulink

I feel like every few years GUI based workflow tools make a revival, but then slowly die out. In the 1990s the selling point was that your "business logic" could be written by "the business" instead of those expensive programmers, and we all know how that panned out.

There is obviously something here, however, especially when you consider what people are doing with Simulink. But why do these tools remain niche? Is there some threshold of complexity past which these tools don't scale well, but source code does? Considering these tools are effectively representations of a program's call graph, could there be a future where I toggle between text and graphical versions of my program depending on what works best?

Re: Yahoo Pipes

#80

As a fresh Yahoo! Developer Network employee I was the person who blabbed about the top-secret _callback parameter the weekend Pipes launched, which caused the service to be taken offline for capacity upgrade. I had an interesting series of voicemails, ranging from "dude, WTF?" to "okay, we need to talk to you right away" to "we're pretty sure you're fired" but ending with "hey, Tim O'Reilly picked it up and ALL IS F…

And now, Pipes can be mobilized and viewed on any mobile phone using SMS and WAP! Simply get the RSS feed of the Pipe and mobilize it using the 411Sync Mobile API. Within seconds, view it on your cell phone

Oh wow. There is so much in this paragraph that was best left forgotten.

Post reply on HN