I only use the mailing lists and I don't see your 've set up my google mail account in such a way that all mails from a specific list get the list's name as label attached and that they don't go through the normal mail entry. In Thunderbird these labels appear as IMAP folders. So no clutter here.
Additionally I use threaded view and as long as no one('s client) messes up the list-ID that also works without problems. How do you think Gmane manages to keep track of the structure? It also uses information contained in the mails...

Regards,
Sven

I started using Gmane long ago, because back then neither Google or Yahoo, IIRC, neither of them had the email rules feature (like, filter by sender address and move to a folder by name of the mailing list. ) Some free and non-free email providers still might not have that feature, even in this day and age...

Also email providers can have quite tough quota rules, for message size and/or count.

Thunderbird, well yes, it does download same data from NNTP server as you would via IMAP. (and keeps it on your disk)
But you're in control here too - it only downloads whole article IFF you wish to read it (click its header) or otherwise request it to be downloaded while syncing the newsgroup. (IMAP synchronization of folders is not on demand: you either sync whole messages (can not just only download headers), or have the 'keep the messages for this account on this computer' checkbox ticked, so you keep nothing locally)

And Thunderbird can search the nntp archive on the server too;
something that is quite much more efficient than searching in mailing list archives.

Also it enabled you to be able to read the mailing list, without going through subscription process too; (I've no problem to subscribe if I want to post somewhere but for just reading, gmane nntp was infinitely more convenient than e.g. the mailman web interface before HyperKitty :J )

I wish you also could have seen the thread.gmane.org web interface, and the way it searched.... however they've offlined it already.

Gmane will be sorely missed if the nntp gateway ever happens to go away...

-L.

I'm sure that if one dug there would be plenty of prewritten mail distribution programs, probably in Perl on unix (and if somebody doesn't like that, he might as well stop reading here).

The key to managing anything like this is usually to use a different email ID for each mailing list, e.g. something like for this one. Then either make sure you
deal with an ISP who is prepared to accept either all mail for your domain or a significant number of predefined names (i.e. rather than just a single one) or set up your own email server (Sendmail+Cyrus or whatever). Then use Thunderbird etc. rules to distribute traffic to folders by recipient.

Otherwise it's not particularly difficult to pipe messages from Sendmail into a discussion group server, although I'd suggest that anybody who chooses INN for this use v1 rather than anything later. It's also not too difficult to redirect anything sent locally to an external mailing list, but there is a number of "gotchas" that one has to be aware of lest one sets up a loop.

there was even a pre-cursor to (probably) every known program like that, fetchmail... up to the interested reader to find what was it written in ;) and by who. (as in, I remember fetchmail from ~15-20 years ago, it had quite /specific/ approach to config files)

Yes. However it requires that filter/sorter be running somewhere all the time, to sort the messages. So you end up having a 24/7 computer in the house, which isn't to everybody's taste... or hosting it to run in a vm somewhere for free/non-free. And, well if you happen to think, that an DD/OpenWRT'd home ADSL modem(*) with some storage is enough, try to think of a good answer to 'Why do you need that hard drive attached to the modem all the time?' question, in 2 minutes ... ;)

(*) (of which workable ones are not as easy to come by by the way either)

Or for the matter, if the project happens to run a mailing list, to set up their own mail2news gateway with a nntp server of their own. (however it increases maintenance which nobody ever has time for...) (and you still might need to deal with takedown notices, as Gmane admin had to)

-L.

Fetchmail's OK if you can rely on an ISP running a mail server (MTA, Sendmail or equivalent) which will accept all IDs for a given domain in a compatible format. Some usages aren't compatible, and I've had to write a local equivalent.

This is basically stuff I'm working on at the moment, to the detriment of other things like testing FPC on outrageous platforms. Let's consider the two sides of the problem separately.

First, if you want to receive all possible emails (i.e. with user names mis-spelt etc.) you'll usually have to set up your own server somewhere. But these days a Raspberry Pi can use a 64Gb SDCard instead of a conventional hard disc, and that is more than adequate to buffer email traffic. In our case we've got Sendmail as an MTA, greylisting, and Cyrus as storage... there's a few problems in there related to duplicated messages but it works. Client systems need to know what user IDs they're collecting mail for, and periodically anything uncollected will need to be reviewed.

Second, connection etc. A dedicated email server will usually require a routable IP4 address (i.e. not 192.168.x.x etc.) and these have been increasingly difficult to get over the last few years. Routable IP6 addresses are easier, but ADSL services that support IP6 are still comparatively rare.

A solution to this is to use a non-routable IP address allocated by DHCP and served over ADSL or 3G/4G mobile. Over that you set up an L2TP tunnel, and at your end you have a mail server listening on a routable IP6 address (and IP4 if you can get one). You possibly also have a very low-priority IP4 server run by your ISP, but this will need very heavy spam filtering since it will get a lot of undesirable stuff.

L2TP in this mode is still rare, we're beta-testing it with our ISP but since they also manufacture the Firebrick appliances used by a fair number of small ISPs expect it to be rolled out gradually. http://aa.net.uk/broadband-l2tp.html

We've historically used Draytek router/modems connected via multiple carriers to AAISP (link above). Over the years we've seen reliability gradually improve as carriers have gradually introduced proper failover when e.g. taking a core router out overnight (we run 24x7. They don't). We're currently experimenting with diversified connections, and one of the things I've got on my plate is to integrate an inference engine with our firewall to handle traffic control (i.e. choosing which connection to use, irrespective of routing).

The inference engine is from NASA, and written in Perl. Sorry.