The Advisory Boar

By Abhijit Menon-Sen <>

Mojolicious: getting rid of waypoints

I have been less active than usual for several weeks due to poor health, and it came as a surprise to discover that Mojolicious 2.82 deprecated waypoints, a feature that I used in more than one app. I couldn't find any explanation of the reasons for the change (the commit message says only "deprecated waypoint support in Mojolicious"), hence this note.

Waypoints were earlier considered for removal because they were poorly understood, but were retained because sensible uses of them were found in the wild at the time. This time, they were deprecated because their special-case handling was in conflict with providing more control over format detection, and because their lack could be worked around easily.

Unfortunately, subsequent changes (after the deprecation) seem to have broken waypoints, which I discovered when going from 2.6x to 2.9x in a single step. I chose to get rid of my waypoints rather than investigate the problem more closely (because I suspected that the "fix" would break other things).

It's easy to rewrite code that uses waypoints. Here's an example from one of my apps:


The above code can be written as:

my $logs = $internal->route('/logs');

Depending on the situation, other alternatives may be available. For example, you could just as easily declare one route for /logs, and another for /logs/tail.

It's an easy workaround, with only minor disadvantages. If you use the nested routes, you need a new temporary variable ($logs). If you declare separate routes, you have to repeat the prefix unnecessarily. But that's a small price to pay.

I was worried about the breakage at first, but Sebastian points out that nobody else has complained in the month since the change. Given that few people ever understood or used waypoints, perhaps nobody was affected.

Mojolicious and HTTP streaming

In response to a number of recent requests for examples of how to implement "tail -f" functionality in a Mojolicious application, this post dissects a log-file viewer I wrote. It's not difficult, but it's not obvious how to put the many moving parts together.

The nicest way to do this would be to stream lines from the logfile to the client over a websocket, or as DOM updates using server-sent events. Sadly, widespread and usable browser support for either method is still months away, and in their absence we must pick from a collection of unsatisfying hacks to implement "server push".

I wanted something easy to implement that would work on Firefox, Chrome, and IE8. I also wanted to minimise wastage of resources, more because of my parsimonious nature than based on any measured need (so forcing the page to reload every n seconds was right out). The Wikipedia entry for Comet describes the available options. It's not easy to find complete, non-trivial examples of their use, but I eventually settled on the "forever iframe" as the best fit for my needs. (But my code could be adapted to fit long-polling, another common strategy.)

The idea is to serve a page with a hidden iframe pointing to a URL that keeps a connection open forever and sends HTTP-chunked <script> tags to the client to display each new line in the log. The logfile doesn't need to keep being reopened and re-read, and multiple clients reading the logfile concurrently are supported cleanly (each one will have a dedicated connection, and will never need to tell the server which line in the file they last saw).

Read more…

Serving static files with Mojolicious

I use Hypnotoad behind nginx to run my Mojolicious applications. nginx is better at handling TLS (than the flaky IO::Socket::SSL module), and it can be configured to cache static resources. My first attempt to set up caching was to set Expires in an after_static_dispatch hook, but it took me a while to figure out the right combination of hooks to do what I wanted.

This post describes how to use after_dispatch and after_static_dispatch hooks to make sure static resources (i.e. public/*) are served with an expiry date based on the content type (and without cookies), and also to prevent some dynamic responses from being cached.

At first, I thought that after_static_dispatch would be called when the static dispatcher served a response, and after_dispatch would be called otherwise. But in fact, both hooks are always executed, and you can't rely on one being called before the other. Instead, we must check if a response code is already set when after_static_dispatch is called. If so, we know the response came from the static dispatcher, and we can add an Expires header based on the content type. If that header is not present when after_dispatch is called, we know it's a dynamic response, and can add a Cache-control header.

The static dispatcher does not set the content type when serving a "304 Not modified" response (to clients that used If-Modified-Since), so we can't rely on the type being available in after_static_dispatch.

Read more…

Spreadsheet::XLSX and humongous spreadsheets

I've been using Spreadsheet::Read backed by Spreadsheet::ParseExcel and Spreadsheet::XLSX to process Excel spreadsheets, and the combination works reasonably well (despite various frustrating problems in retrieving dates).

Today, I wrote code with lots of diagnostics to parse a large spreadsheet I'd never seen before. The code ran properly for a while, and then complained of various inconsistencies in the data. After much digging, I realised that Spreadsheet::XLSX wasn't parsing columns beyond ZZ. LibreOffice showed me data that my Perl code wasn't seeing; but it, too, has an arbitrary 1024 column limit. I had to unzip the XLSX file and read the XML source to find the real number of columns: 1059.

Until today, I'd never imagined a spreadsheet that big either.

The fault lies with the following code in Spreadsheet/, which assumes that the column name is two letters followed by a number:

foreach ($member_sheet -> contents =~ /(\<.*?\/?\>|.*?(?=\<))/g) {
    if (/^\<c r=\"([A-Z])([A-Z]?)(\d+)\"/) {
        $col = ord ($1) - 65;
        if ($2) {
            $col *= 26;
            $col += (ord ($2) - 65);

Fortunately, it's not hard to extend the code to handle longer names:

foreach ($member_sheet -> contents =~ /(\<.*?\/?\>|.*?(?=\<))/g) {
    if (/^\<c r=\"([A-Z])([A-Z]*)(\d+)\"/) {
    $col = ord ($1) - 65;

    my $rest = $2;
    while (my $l = substr ($rest, 0, 1, '')) {
        $col *= 26;
        $col += (ord ($l) - 65);

I sent the patch to Dmitry Ovsyenko, author of Spreadsheet::XLSX, but received no response.

Back to procmail

When I first started receiving more email than I could deal with in my inbox, I reluctantly cobbled together a .procmailrc from bits of other people's configuration files. procmail was a mystery, but my needs were simple, and although I never graduated to liking its syntax, I learned to get along with it reasonably well over time.

Then I read Simon Cozens' article about Mail filtering with Mail::Audit, and I was delighted by the thought of writing a clear and understandable Perl program (ha!) to sort email. I had switched away from procmail to a simple Mail::Audit script almost before I had reached the end of the article.

My little program was convenient. Rather than having a separate recipe for each mailing list, the code would derive the mailbox name from the List-Id or Mailing-List header field. (For all I know, procmail may be able to do this too, but none of the examples show how to do it, and I never tried very hard to figure it out.) That code kept me happy for almost ten years.

Yesterday, I installed Ubuntu 11.04, which ships with Perl 5.10.1; by coincidence, the official end of support for Perl 5.10 was announced a day earlier, along with the release of Perl 5.14. I decided to leave the system Perl alone, and use perlbrew to maintain my own Perl installation. But I knew that would take hours with my reduced bandwidth, and I didn't want to wait until then to read my mail.

So I had the bright idea of installing Mail::Audit in ~/perl just to be able to run my mail filter. I downloaded Mail-Audit-2.225.tar.gz from a CPAN mirror and ran Makefile.PL, only to be warned that File::HomeDir, File::Tempdir, and MIME::Entity were missing. I tried to install using the system, only to find still more dependencies: Test::Tester, File::Which, Test::NoWarnings, Test::Script, Probe::Perl, MIME-tools, IPC::Run3, and Test::Deep. Something amongst that collection failed to install, and I was left looking at several screenfuls of failing tests.

Do I understand that the use of File::HomeDir and IPC::Run3 probably allows users of Windows and VMS to use Mail::Audit? Yes. Am I glad that all of these modules have comprehensive test suites? Yes. Could I have fixed the problem? Sure, if I had spent some time investigating. But my newly-installed gnome-terminal wouldn't scroll back far enough, and I suddenly remembered that procmail, inscrutable as always, was already installed.

Ten minutes and some regex-surgery on my filter script later, I had cobbled together enough of a .procmailrc to begin reading my mail.

(I'm not overly fond of CPAN dependencies, but this post is about my mail filter growing into something that demanded more attention than procmail, not about Mail::Audit having too many dependencies per se. For comparison, I am a reasonably happy user of Email::Stuff in some of my apps, and that has even more dependencies.)

Mojolicious and DB handles

"But… what about the model?" is a very common question from newcomers to Mojolicious, and not everyone is happy with the simple answer: "You can do whatever you want, Mojolicious doesn't care". What this really means (especially for people coming from Catalyst) is that you can keep using your DBIx::Class model with "use My::Model", or use whatever other method you prefer to interact with your database.

I don't get along with DBIx::Class myself, but enough people ask about using straight DBI that I felt it worthwhile to share the way I do it. There's not much to it: my code boils down to calling DBI->connect and keeping the handle for later use. Mojo's package attributes with delayed initialisation make it easy to do this.

I put the following code in my startup function (i.e., sub startup in lib/

sub startup {
    my $app = shift;

    # …read configuration and do other stuff…

    (ref $app)->attr(
        db => sub {
            DBI->connect("dbi:Pg:database=$db", $user, $pass)

This adds a "db" attribute to my application class (represented here by the $app object), which can be accessed as "$app->db". The first time it is accessed, the function I provided is called, and its return value is cached and returned as the value of the attribute thereafter. In this case, my sub is a closure that uses the $db, $user, and $pass variables read from the configuration file, and connects to a Postgres database using DBI/DBD::Pg, but those are just details. You could do the same thing with DBIx::Connector or a similar module.

(Aside: don't just create a database handle in startup. It will break if you run your app under a preforking server like Hypnotoad. Your startup will be called once before forking, but the DBI handle can't be shared between the server processes. The delayed attribute initialisation is perfect in this case, because each server will have its own handle.)

With this code, the database connection is created only the first time it is needed, and it is kept around as long as the application is alive (how long that is depends on which server your application runs under). Controllers can just do "$self->app->db" to fetch a db handle at any time without worrying about any of this.

Marcus Ramberg pointed out on IRC that I could also use "$app->helper(db => sub {...})" to make the handle accessible from controllers (as "$self->db") and from epl templates (as "db"). I don't need to do that for database handles, but someone else might find it useful.

(This is an old post that I forgot to submit when it was written.)

Cryptographically secure randomness in Perl

I can usually sidestep the need for cryptographically secure randomness, but I decided to investigate the available options to generate CSRF protection tokens.

On Linux, /dev/random is a good, cryptographically secure random number generator, but the entropy available without special hardware support is severely limited. Reads from /dev/random block for however long it takes (even minutes) to gather enough entropy to satisfy the request. Entropy is in high demand throughout the system, so this is not a resource that can be indiscriminately drawn upon.

One standard solution to this problem is to use the output of a randomly keyed block cipher running in counter (CTR) mode. This provides only as much entropy as the random key, but if /dev/random is used to generate the key, the result is suitable for many purposes. Andrew Main's Data::Entropy module implements this strategy in Perl.

Read more…

Exceptions during Mojolicious testing

I've been using Test::Mojo to write tests for my Mojolicious applications, and it's great.

my $t = Test::Mojo->new(app => "Wigeon");
$t->post_form_ok('/login', {__login => $login, __passwd => $password})
    ->text_is('html body p', 'First paragraph');

When something breaks (i.e., an unexpected 404 or 500 response), however, Mojolicious renders the built-in not_found or exception template, and I see the whole HTML/CSS page in my test results. Usually many times, as one failure precipitates others.

I fixed this by loading Mojolicious::Controller early in my application class and overriding its render_not_found and render_exception methods.

use Mojolicious::Controller;


    no warnings 'redefine';
    package Mojolicious::Controller;

    sub render_not_found {
        my $self = shift;
            status => 404, format => 'txt',
            text => "Not found: " . $self->req->url->path

    sub render_exception {
        my ($self, $e) = @_;
        return if $self->stash->{'mojo.exception'};
            status => 500, format => 'txt',
            handler => undef, layout => undef, extends => undef,
            text => Mojo::Exception->new($e)->message,
            "mojo.exception" => 1

Soon after I grumbled about having to do this on IRC, Sebastian Riedel added experimental support for mode-specific exception templates. One can now create stripped-down not_found.testing and exception.testing templates, and they will be automatically detected and used.

Returning text/plain responses in Mojolicious

The Mojolicious documentation gave me the impression that the way to send a text/plain response was just:

$self->render_text("Plain text content");

I wrote a content_type_is test for such a response the other day, and its failure made me notice that the type was actually text/html. I asked about it on #mojo, and learned that render_text is just the opposite of render_data, and the function name has nothing to do with the content type of the response.

The incorrect examples at the very beginning of the rendering guide have been fixed, and the description of render_text has been changed. It used to say "Render the given content as plain text, note that text will be encoded" and now it says "Render the given content as Perl characters, which will be encoded to bytes".

Given the misleading examples present earlier, I wish the description had also said something like "The default content type of the response is text/html, and this function does not change it unless you specify a different format". Anyway, the correct way to send a text/plain response is to specify the format explicitly as text:

$self->render_text("Plain text content", format => 'txt');

Mojolicious and X-Powered-By

A few days ago, I discovered that Mojolicious::Plugin::PoweredBy is enabled by default, and adds an "X-Powered-By: Mojolicious (Perl)" header to all HTTP responses. Although I can change the value of the header, there is no way to suppress it (e.g. by setting the value to undef). Since X-headers are meant for private experiments, and should not be exposed to the Internet (never mind how many people do it anyway), I thought this was poor behaviour.

The solution was to bundle an empty PoweredBy plugin with my app:

package Mojolicious::Plugin::PoweredBy;

use base 'Mojolicious::Plugin';

sub register {


I brought this up in #mojo on IRC, suggesting tactfully that the plugin should either not be enabled by default, or be easier to disable (when what I really wanted was to suggest it be removed entirely). The Mojo author disagreed, and said that nobody would enable it if it wasn't enabled by default (which is quite true, and to me suggests that it should not exist at all), and that it was "advertising" for Mojo.

It may be advertising, but I'm not sure it sends the right message.