My friend Nicolai Langfeldt (author of the DNS HOWTO) works at a large ISP, where he recently needed to ping somewhere between 10,000 and 100,000 hosts every ten seconds and feed the up/down data to Nagios. He had started hacking on fping to do this, but I thought it could be done better by a smaller program written specifically for this purpose.
I wrote most of github.com/amenonsen/heaping on a flight home. It takes a list of IP addresses on the command line, writes ICMP echo requests for each address to a raw socket every ten seconds, and creates another process to print the response time (or "unreachable") for each response. It does nothing else, and does not need to allocate any memory after startup.
Nicolai's preliminary tests show that it can send out the 10k pings in under 100ms, which far exceeds the performance I was hoping for (though his networking colleagues may yet frown upon sending that many pings so fast). With that kind of performance, monitoring 100k hosts isn't out of the question, even if the sender has to be slowed down a bit.
The code is on github in case anyone else finds it useful. Feedback is welcome.
While looking for something that could be adapted for this use, I found the source code of the original ping.c on the ISOC web site. I decided against modifying it because of several misleading comments, type abuse (e.g. allocating a struct sockaddr and casting its address to struct sockaddr_in *), and other K&R/BSD oddities. But it's good code overall, and it was instructive to look through it.
Since Arnt already announced snippets on rant.gulbrandsen last month, I thought I'd write a quick introduction to it here as well.
Snippets is a Jabber bot that periodically asks its subscribers what they're doing, and provides access to the answers through a simple web interface. It is meant primarily to help teams of developers to communicate with each other.
See toroid.org/snippets for more, including links to the source code and installation instructions.
Feedback and feature requests are welcome.
Hassath wanted to download a couple of gigabytes worth of mail from her GMail account for offline storage, but was frustrated by the IMAP server closing the connection during large transfers, which made Thunderbird go into a sulk, and forced her to restart the download by hand every time. Using POP was not an option, since it had no way to preserve the labels she had assigned to the messages (which show up as mailboxes via IMAP).
I tried some other programs, but they didn't work the way I wanted. Some of them provided little or no progress information. Some would fetch all of the messages into memory before writing anything to disk. Some would fetch messages one by one, with separate command-response cycles. Most importantly, none of them coped well with the frequent disconnections.
Having exhausted my patience, and being in need of a weekend hack, I set out to write a program to download the mail. I tried the IMAP modules on CPAN, but found them lacking in one way or another. IMAP::Client parses IMAP responses in a fundamentally broken way (scanning literal data for things which look like OK responses). Net::IMAP::Client doesn't provide debugging information or allow MSN fetches. Net::IMAP doesn't know about IMAP+SSL. Most importantly, none of the modules would allow me to stream messages to disk sensibly. So I wrote one from scratch.
My program connects to imap.gmail.com and downloads messages from each mailbox (except special ones like "All Mail" and "Spam") into local Maildir-format directories. It doesn't waste bandwidth, and saves data to disk as it is received, so that it can use partial fetches to resume after an interrupted download. It is safe to stop or restart it at any time. It never deletes any mail. When it completes, you will have an offline mirror of all your GMail messages.
Hassath felt I should make the program available, especially since a friend of ours with an even more flaky network connection also wants to use it.
Download gmail-mirror (from github.com/amenonsen/gmail-mirror) and place it somewhere in your PATH. (Use, modification, and distribution of this code is allowed without any limitations. There is no warranty, express or implied.)
You will need to have the IO::Socket::SSL module installed
(Ubuntu users can apt-get install libio-socket-ssl-perl).
There are no other module dependencies.
Run "gmail-mirror.pl username" to download all your mail, or
specify a list of mailboxes explicitly: "gmail-mirror.pl username
INBOX work personal". (Note: GMail doesn't really have mailboxes.
An IMAP mailbox named x is the set of messages labelled x.) In either
case, mail will be stored locally in a Maildir-format directory named
"gmail/mailboxname/somenumber" in your current directory. You can run
"gmail-mirror.pl -v username" for more verbose progress
information. It is safe to interrupt (^C) and restart the
program at any time.
Write to me if you need help or have any suggestions. If you find this program useful, I'd love to hear about it.
I started using Blosxom a few weeks ago. I liked the basic idea (that each post is one file) very much, but I found the code hard to read and poorly documented. Plugins varied widely in quality, and I had to jump through hoops to implement some of the things I wanted. By the time I got everything working the way I liked it, I ended up with a rewrite: Loathsxome.
The code is well-documented, supports tagging and pagination, generates an RSS feed, works nicely with Git, and lets me publish URLs like http://toroid.org/ams/etc/200908 and /ams/etc/birds+language. (Disadvantages: it isn't quite plugin-compatible with Blosxom, and it doesn't support static generation.)
I'm sure I could have used a more modern, featureful program to achieve the same effect, but I'm glad I didn't have to. I was happier to spend a couple of hours writing something worse than a few days trying to fit something better, like Wordpress, into my head.
I wrote this code for my own use with no intention of releasing it, but a few people have expressed an interest in using it, so here it is.