I was a big user of Yahoo Pipes for preparing custom RSS feeds. I have now found that https://feedity.com/ does a good enough job scraping pages to produce feeds. I then run that through https://fivefilters.org/content-only/ to fetch the full content of the page and read it in https://www.inoreader.com/ , my favorite Google Reader clone. Some people are amazed that I am always on top of new content they produce.
Yahoo Pipes
71–80 of 97 posts
Re: Yahoo Pipes
#72I was at the O'Reilly Emerging Tech conference session that functioned as (what I remember to have been) the Pipes launch party. Everyone involved with the project seemed young and full of optimism, and while they didn't know exactly how Pipes would make money, I remember being as impressed with the UI as I was the forward-thinking ideas that motivated the project. Keep in mind that in 2006, IE 6 was still omnipresen…
And Meebo! They build an entire windowing system in the browser...and supported IE6.
Re: Yahoo Pipes
#73I feel like it's similar to Airflow https://airflow.apache.org/
There were many technical problems with pipes and one major business problem. The technical problems as usual, were solvable:
- branching. the system really didn't have a clean way to cause branching. if you wanted to only apply a regex on certain items, that would fail hard. I feel like some part of the anime fansub system I built must have been stressing the system because I'd occasionally find it disabled or key components removed from Pipes. (Deluge can consume RSS in the background, but often the fansub RSS feeds have duplicate items for different qualities, or are missing the enclosure field the system keys off of and other field normalization problems)
- types. the system really operated on RSS as a medium of data exchange. Any operation that worked on whole RSS feeds was great. So the typical 'put a ton of blogs into a pipe and union them all into one big feed' hello world app worked great. As soon as you wanted to operate on a specific line of input items and do subprocessing, you were in a sort of hell. It wasn't clear which modules worked on which types, or why.
- composition. a general engineering principle is to build things out of small parts. That's the fundamental reason pipes works, but you never had the ability to treat a pipe you built as a small part itself. You couldn't build pipes that accepted RSS in. The subpipe command they eventually added only allowed RSS out, without even any parameterization. Judging by the comments from support engineers here, the reason is mostly obvious: they couldn't prevent people from building pipes that called themselves indirectly.
But the business problem was most fundamental. The most useful APIs were often monetized via web ads. Yea, someone was collecting amazon referral revenue, but beyond that, no money was changing hands. That was a fundamental challenge to further adoption. Bloggers started publishing truncated or headline blog feeds, and what sort of evolved was a cat and mouse game with content creators: people invent new ways to generate RSS feeds for sites that don't have them while the sites fuzz their UI to prevent it.
Re: Yahoo Pipes
#74As a fresh Yahoo! Developer Network employee I was the person who blabbed about the top-secret _callback parameter the weekend Pipes launched, which caused the service to be taken offline for capacity upgrade. I had an interesting series of voicemails, ranging from "dude, WTF?" to "okay, we need to talk to you right away" to "we're pretty sure you're fired" but ending with "hey, Tim O'Reilly picked it up and ALL IS F…
Re: Yahoo Pipes
#75Re: Yahoo Pipes
#76Pipes was such a good product and sorely missed. Is there anything equivalent to it today?
Having a lot of fun building the product but with so many of these types of companies shutting down it has me wondering what it forebodes :/
Re: Yahoo Pipes
#77As a fresh Yahoo! Developer Network employee I was the person who blabbed about the top-secret _callback parameter the weekend Pipes launched, which caused the service to be taken offline for capacity upgrade. I had an interesting series of voicemails, ranging from "dude, WTF?" to "okay, we need to talk to you right away" to "we're pretty sure you're fired" but ending with "hey, Tim O'Reilly picked it up and ALL IS F…
What were the implications of knowing the _callback parameter?
If you look at some janky JS code[1] I had on my website at the time, you can see what you can do with it.
That would basically source
://pipes.yahoo.com/pipes/pipe.run?_id=7CTRtbtL3BGX_AHbjknRlg&_render=json&_callback=load_daily_show
as a static asset in my page, which would load the JS and call load_daily_show() as a result.
Now, EVERYONE who hits my page is invoking the pipe as a backend API call, with no caching and unfortunately with the entire Y/T cookies intact.
[1] - https://web.archive.org/web/20081007043923/http://t3.dotgnu....
Re: Yahoo Pipes
#78Thanks for posting and the nice comments. Glad to have built something that people found useful...
Sorry, one last note. I was a broke college student at the time. I made tens of thousands of dollars with Pipes doing ethical arbitrage. I still miss it, where's our Kickstarter?
Re: Yahoo Pipes
#79https://en.wikipedia.org/wiki/Apache_NiFi
https://en.wikipedia.org/wiki/Quartz_Composer
https://en.wikipedia.org/wiki/Simulink
I feel like every few years GUI based workflow tools make a revival, but then slowly die out. In the 1990s the selling point was that your "business logic" could be written by "the business" instead of those expensive programmers, and we all know how that panned out.
There is obviously something here, however, especially when you consider what people are doing with Simulink. But why do these tools remain niche? Is there some threshold of complexity past which these tools don't scale well, but source code does? Considering these tools are effectively representations of a program's call graph, could there be a future where I toggle between text and graphical versions of my program depending on what works best?
Re: Yahoo Pipes
#80As a fresh Yahoo! Developer Network employee I was the person who blabbed about the top-secret _callback parameter the weekend Pipes launched, which caused the service to be taken offline for capacity upgrade. I had an interesting series of voicemails, ranging from "dude, WTF?" to "okay, we need to talk to you right away" to "we're pretty sure you're fired" but ending with "hey, Tim O'Reilly picked it up and ALL IS F…
Oh wow. There is so much in this paragraph that was best left forgotten.