I’ve been interested in P2P approaches to syndication for a while, and FeedTree is a research project offering one solution to the problem. This poster provides a good overview of how the system works. A more detailed technical description is available here.
In a nutshell, an HTTP proxy on each reader’s machine becomes a node in a Pastry overlay network. The node then joins a multicast tree for each subscribed feed, and updates are pushed to all subscribers using the Scribe group messaging protocol. Feeds are either directly published into the network by FeedTree-aware publishers, or are polled by some subset of interested nodes on behalf of all the others. A digital signature can be added to the feed by the publisher to prevent spoofing. Configuring your feed reader to use the proxy is quite simple, in most cases, and then you’re set. Publishing takes a bit more work to set up, at present, but I don’t see why you couldn’t eventually wrap a simple GUI/wizard around this process to make it painless.
I had every intention of giving FeedTree a go before posting this, but here’s the rub: “Step 2: Be sure that your computer is accessible from the Internet on port 29690.” Well, I could have done so, but it would have required re-configuring my entire home network first. I couldn’t justify the effort just to try out one piece of software. I suspect most software end-users either wouldn’t have the permissions necessary to meet this requirement; wouldn’t know how to go about it; or, like me, wouldn’t be sufficiently motivated to take the trouble. Alas, methinks this is a showstopper if FeedTree is to become anything more than a research project. That’s a shame.
The “not even on the napkin” approach I had in my head was based on JXTA, of course (a JXTA-based solution would remove the need to hack your network, making it end-user friendly, for one thing.) Rather than pushing updates to subscribers, I envisioned peers using the Discovery service to advertise and search for feeds. To wit: rather than polling feed A every hour, the peer would first try to discover a copy of the feed less than one-hour old. If none could be found, only then would it poll feed A’s source, and publish an appropriately time-stamped advertisement for other peers to discover. The FeedTree “push” approach provides for more timely updates than this if the publisher injects the feed directly into the FeedTree network. Also, I can see the potential in my scheme to end up with “too many” peers polling for a feed, if their desired polling frequencies are too much out of synch. This requires considerably more thought. FeedTree also allows multiple “volunteers” to poll a feed in order to maintain a certain level of timeliness, but the overall polling frequency is better controlled.
Anyway, I’m glad to see someone doing real work on alternatives to uncontrolled HTTP polling for feeds. Ultimately, though, I don’t see anything gaining wide usage that isn’t dead-simple to install and use, without regard to network topology. (Imagine that you did get FeedTree installed on your laptop at home. As soon as you take the machine to the office, or to school or wherever…you’re hosed. *sigh*) And without mass adoption and seamless mobile usage, no solution will be able to make any real dent in the scaling problem.
Thanks for taking a hard look at FeedTree! I’m always looking for thoughtful feedback from users and researchers.
It looks like JXTA supports seamless behind-the-NAT operation by using dedicated relay nodes (I’m not a JXTA expert by any measure, so I might have gotten this wrong). We could probably add something like that to FreePastry if there’s great enough demand, but such a scheme will have an impact on the ultimate scalability of the network (e.g. with many users relying on a few relays, we’re no longer adding resources to the network proportional to the demand on the network).
I agree that port forwarding is highly unpleasant; it would be nice to do away with it entirely. However, plenty of home Internet users have figured out how to get multiplayer games and other p2p apps running behind their firewalls (either by fiddling with ports or just by checking the “DMZ” box on their router configuration); FeedTree’s setup cost is no greater.
As for the mobility issue, at the moment you can restart your proxy when you change locations and things should work fine. (We could also try to detect this change in location and perform the reboot seamlessly.) Another approach for mobility would be to establish a persistent proxy (e.g. on your desktop), and use that proxy when travelling. [This is the solution I currently use, but it won’t work for everyone.]
Oh, and, yes, the publisher needs a GUI. Working on it!
Thanks for writing, Dan. You’re right that JXTA is a super-peer network, so it doesn’t scale quite as well as a pure peer one. However, one thing you might be able to do to distribute the load is to have any non-NATed peers self-promote themselves to be JXTA relays. That way you’re not dependent on a few dedicated relays. Your resources will grow along with the number of users, but not at the same rate. I am working on implementing a scheme like this myself, so I can’t say for sure yet how well it works or how difficult it is to do. It’s just a thought.