The Advisory Boar

By Abhijit Menon-Sen <ams@toroid.org>

heaping ICMP echo requests on unsuspecting hosts

2011-06-29

My friend Nicolai Langfeldt (author of the DNS HOWTO) works at a large ISP, where he recently needed to ping somewhere between 10,000 and 100,000 hosts every ten seconds and feed the up/down data to Nagios. He had started hacking on fping to do this, but I thought it could be done better by a smaller program written specifically for this purpose.

I wrote most of github.com/amenonsen/heaping on a flight home. It takes a list of IP addresses on the command line, writes ICMP echo requests for each address to a raw socket every ten seconds, and creates another process to print the response time (or "unreachable") for each response. It does nothing else, and does not need to allocate any memory after startup.

Nicolai's preliminary tests show that it can send out the 10k pings in under 100ms, which far exceeds the performance I was hoping for (though his networking colleagues may yet frown upon sending that many pings so fast). With that kind of performance, monitoring 100k hosts isn't out of the question, even if the sender has to be slowed down a bit.

The code is on github in case anyone else finds it useful. Feedback is welcome.

ping.c

While looking for something that could be adapted for this use, I found the source code of the original ping.c on the ISOC web site. I decided against modifying it because of several misleading comments, type abuse (e.g. allocating a struct sockaddr and casting its address to struct sockaddr_in *), and other K&R/BSD oddities. But it's good code overall, and it was instructive to look through it.

Introducing Snippets

2011-05-08

Since Arnt already announced snippets on rant.gulbrandsen last month, I thought I'd write a quick introduction to it here as well.

Snippets is a Jabber bot that periodically asks its subscribers what they're doing, and provides access to the answers through a simple web interface. It is meant primarily to help teams of developers to communicate with each other.

Snippets is a Perl program that depends on a few CPAN modules, and needs access to a Postgres database. You can get the source from the github page, and installation instructions are given in the README.

Feedback and feature requests are welcome. Please send them to ams@toroid.org.

Mirroring GMail mailboxes with IMAP

2010-04-11

Hassath wanted to download a couple of gigabytes worth of mail from her GMail account for offline storage, but was frustrated by the IMAP server closing the connection during large transfers, which made Thunderbird go into a sulk, and forced her to restart the download by hand every time. Using POP was not an option, since it had no way to preserve the labels she had assigned to the messages (which show up as mailboxes via IMAP).

I tried some other programs, but they didn't work the way I wanted. Some of them provided little or no progress information. Some would fetch all of the messages into memory before writing anything to disk. Some would fetch messages one by one, with separate command-response cycles. Most importantly, none of them coped well with the frequent disconnections.

Having exhausted my patience, and being in need of a weekend hack, I set out to write a program to download the mail. I tried the IMAP modules on CPAN, but found them lacking in one way or another. IMAP::Client parses IMAP responses in a fundamentally broken way (scanning literal data for things which look like OK responses). Net::IMAP::Client doesn't provide debugging information or allow MSN fetches. Net::IMAP doesn't know about IMAP+SSL. Most importantly, none of the modules would allow me to stream messages to disk sensibly. So I wrote one from scratch.

Read more…

Introducing Loathsxome: minimalist weblog software

2009-08-10

I started using Blosxom a few weeks ago. I liked the basic idea (that each post is one file) very much, but I found the code hard to read and poorly documented. Plugins varied widely in quality, and I had to jump through hoops to implement some of the things I wanted. By the time I got everything working the way I liked it, I ended up with a complete rewrite.

The code is well-documented, supports tagging and pagination, generates an RSS feed, works nicely with Git, and lets me publish URLs like http://toroid.org/etc/200908 and /etc/birds+language. (Disadvantages: it isn't quite plugin-compatible with Blosxom, and it doesn't support static generation.)

I'm sure I could have used a more modern, featureful program to achieve the same effect, but I'm glad I didn't have to. I was happier to spend a couple of hours writing something worse than a few days trying to fit something better, like Wordpress, into my head.

I wrote this code for my own use with no intention of releasing it, but a few people have expressed an interest in using it, so here it is.

No longer maintained

Update (November 2013): Loathsxome never had more than a handful of users, and only two who regularly contributed code (Arnt Gulbrandsen and myself). Arnt wrote plusxome, and I wrote something else using Mojolicious without even pretending to maintain backwards compatibility.

Loathsxome is still here. It works as well as it ever did, and none of what's written below is untrue, but nobody uses it any more.

Read more…

Inspector34

2009-03-20

Inspector34 is a transparent web proxy that records requests and responses for later playback and comparison. It is meant to help with regression testing.

It consists of three programs: i34-record is the proxy server, which keeps a journal of HTTP requests and responses; and i34-replay recreates the original requests from this journal. i34-diff compares responses from the original and replayed sessions.

Inspector34 is written in Perl, and is distributed under a BSD-style open source license. Feedback is welcome.

Read more…

pemtrans: Convert OpenSSL RSA private keys to Cryptlib keysets

2008-06-27

OpenSSL stores private keys in an undocumented PEM format (the key data is DER-encoded and the result is ASCII-armoured), which Cryptlib does not support.

pemtrans reads an OpenSSL RSA private key and the corresponding signed public key certificate, and writes a PKCS #15 keyset that Cryptlib can use. The included manual page explains how to use the program.

Key usage

Cryptlib requires (and respects) the KEYUSAGE attribute on certificates. Some certificates do not contain this attribute, and pemtrans issues a warning when it encounters them, because Cryptlib will probably reject it when you try to use it for something, e.g. in a TLS server.

OpenSSL, by default, does not set KEYUSAGE. This can be fixed by adding a line like the following to openssl.cnf before generating the key and certificate (the attributes specified here are enough for the result to be used in a TLS server):

keyUsage = cRLSign, digitalSignature, keyEncipherment, keyCertSign

Download

Download pemtrans.tar.gz (from github.com/amenonsen/pemtrans)

Use, modification, and distribution of pemtrans is allowed without any limitations. There is no warranty, express or implied.

Please send questions and comments to ams@toroid.org.

Better time management through profiling

2008-05-07

This web page is about the most potent charm in my long-standing quest for better time management: a program to help me keep track of exactly where and how my time is spent.

Of course, I can't tell you how best to manage your own time, but just as profiling is the best guide to code optimisation, I found it useful (and sometimes very surprising) to begin by knowing exactly where time goes. I've been very happy with the results, and I know of others who have evolved similar approaches to the problem.

Instrumentation

The idea is that you run a program (ts) every time you change what you are doing. This sounds intrusive, and it does take a while to get used to; but ts goes out of its way to make it easy, and it's almost second nature to me now. I find the benefits worth this extra investment.

To run ts, you will need to have Perl installed along with the DBI and DBD::Pg modules. You'll also need to provide a PostgreSQL database for it to use, and you'll have to create the activities table from ts.sql.

The profiling data is stored in this very simple SQL table:

    create table activities (
        id          serial,
        category    text not NULL,
        comment     text,
        started     timestamp(0) not NULL default current_timestamp,
        stopped     timestamp(0)
                    check (stopped is NULL or stopped > started)
    );

A row in this table describes a period of time spent on some activity. The category is usually something like "work/mail" or "work/src", the latter perhaps with a comment like "Fixing Jamfile". I find it useful to organise my activities into a hierarchy, but you can decide for yourself how you want to categorise things.

When you start a new activity ("ts work/src Fixing bug 42"), its started time is set, and the stopped time of any activity that doesn't have one (any activity that is in progress, in other words) is also set. You can also specify the start and end times explicitly ("ts -s 12:00 -e 12:30 lunch").

My .zshrc contains "compctl -/ -W ~/.categories ts", and I've created directories that reflect my categorisation under ~/.categories. Now I can do things like "ts w<TAB>e<TAB>" instead of having to type the category name ("work/email") in full.

Running ts without arguments explains how to use it.

Visualisation

Once the data is available, it's up to you to figure out how to use it to your advantage ("How much time did I spend on the phone last week?" "Let me generate a time sheet for work/some-client/code/" ...). Just playing with the data in psql is often instructive.

The tsc program generates sc spreadsheets from the profiling data. It presents, for a given time period (the last week by default), a list of the categories matching some pattern (everything by default), and a table (with totals) of how much time has been spent on them on each of the days in the period under consideration. (You need a patched version of sc to use tsc.)

I plan to write something that generates nicely-formatted time sheets from this data, but I haven't gotten around to it yet. If you have an interesting visualisation idea, please let me know.

Download

Here's a tarball of the source code. It contains the ts and tsc programs, the SQL table definition, a patch required for sc to understand the tsc output, and some documentation. (You can also download a patched version of sc.)

Please write to ams@toroid.org if you have any questions, comments, or suggestions. If this program is useful to you in some way, I'd love to hear about it.

A photo album with nice URLs

2007-11-14

I've always uploaded my photographs into an "img" or a "misc" directory, and handed out URLs like http://toroid.org/ams/misc/foo.jpeg to people. This has worked pretty well.

Occasionally, however, I would upload a group of related photographs, and, tiring quickly of cutting and pasting many URLs, end up writing a little HTML page with links to the images in question.

Eventually, I tired of that too.

Update 2015-10-31: I don't actually use this code any more. It's been replaced by something that works very similarly, but this version is no longer maintained.

Read more…

mod_leech: Per-user download limits for Apache 1.3.x

2007-09-29

mod_leech is an Apache 1.3.x module which uses a MySQL database to enforce per-user download and concurrent-login quotas.

Read more…