Dada Mail v9 Released

The latest major release of Dada Mail has been released. Version Nine! 

Dada Mail is an email mailing list manager, written in Perl. Usually in my posts, I go over all the cool things I was able to get accomplished, because of some neat Perl module or technique.

The last major release focuses on redesigning the backend of Dada Mail, and adding support to run it under PSGI/Plack, while also maintaining the ability to run it under CGI. For that, I moved over to CGI::Application.

This release focuses almost explicitly on the front end design and layout -  and for that, I utilized the Zurb Foundation Framework.

I couldn't have imagined how much work moving to Foundation would have taken, when I started. Sling a few <div>'s here and there - right?

Well, wrong - and it wasn't Foundation's fault. Although there is a learning curve to working with their flavor of HTML markup, and a slightly tougher grade to learn their SCSS workflow (and setting up the environment is a little... obtuse) - the real culprit is that the Dada Mail project really pretty large.

Dada Mail has over 300 separate template files that make up all its views, and the project itself was started in 1999.  Let's say you need to make a complicated layout of form fields and such. What do you use?

Well, in 1999 you use tables. And if it ain't broke, why fix it? So tables it was for a lot of the forms that the UI used for all those preference screens. That's fine and good, until you want your app to work well on a mobile phone's browser. Now, everything is legitimately broken, because a lot of the design has fixed widths and isn't very responsive.

To fix a lot of the UI problems meant replacing the table-based layout with Foundation's Grid System.  A lot of my forms had secondary and even tertiary form elements, that needed to be shown underneith other form elements, so that the user knows they're somehow related. After a bit if wrangling, Foundation made this realistic to do.

Here's an example of a somewhat complicated part of form for changing the template of your mailing lists message,

It's pretty phenomenal that this can now be done, with a having to revert to using a table-based layout. The diff file for this release was over 100,000 lines long, which goes to show you just how much HTML I've been working with for the last few months. I have long since separated the HTML out of Dada Mail, so almost nothing had to be modified in the Perl side of things at all, to get things runnin'. For templates, I've been using HTML::Template (and friends) since forever, and probably will keep on chugging - with the most obvious successor being Template-Toolkit, if the need ever arises. I'm not sure if that makes sense for me now, but we'll see what the future holds.

Dada Mail Screencasts!

Dada Mail v8.4.0 was recently released (hurray!) One of the ways I've been experimenting with showing people all the cool features of Dada Mail is by doing screencasts - below are some of the screencasts I've recently put together.

Dada Mail v8 Quick Install Guide: 

This screencast is meant to showcase just how easy it is to install Dada Mail, using the web-based installer. Dada Mail does have a CLI installer, as well - that's the way to go, if you want to automate an install/upgrade, as well as to take advantage of Dada Mail's PSGI/Plack support.

Importing Addresses into Dada Mail: 

Many people using Dada Mail - especially for discussion lists, are moving from a previous system (like Mailman) - this screencast was created to show how easy it is to import an already-created list over.

Magic User Templates

This screencast shows how to change the layout of Dada Mail to match your own website - it using HTML::Tree under the hood, which I wrote about, recently. 

Dada Mail Scheduled Mailings: 

This screencast showcases a few mature features in Dada Mail, including Sending a Webpage (whole or in part), as well as setting up a recurring scheduled mass mailing, that only sends when the source content has changed. Neat stuff.

Give Dada Mail a spin, if you haven't already - it's a long running, and fairly popular Perl web app. I would say one of the most popular Perl-based web app targeted for regular folks (as opposed to Perl hackers)

Using HTML::Tree to create magic

Dada Mail v8.3 was recently released. Dada Mail itself is a mailing list manager written in Perl. It's  a project I've been working on since 1999(!) - lots of blood/sweat/tears. It's somewhat of a mix between Mailman (or Majordomo, if you remember that app, which I initially went out to inadvertently replace!), and one of the big email marketing services, like Constant Contact or MailChimp.

Difference to those services, is that you run it on your own server/hosting account. The system requirements are  minimal (really), so a cheap cPanel-based shared hosting on Bluehost or whatever will work just fine.

It has support to hook up with Amazon SES for mail sending, giving you a very competitive platform to send mass mailings to your announcement list or discussion list at a really good price. If you want more performance, you can run it as a PSGI app - full support to run it under Plack/PSGI, or running as a FastCGI script without Plack (mod_fcgi?), or as a plain CGI script, even. Dada Mail tries to be flexible. You won't need root access to install - it comes with its own web-based installer. Just upload the distro and a helper script, and away you go. 

One of the newer features of the app is called, Magic User Templates. This feature allows someone with very little experience with manipulating HTML code to integrate Dada Mail into their already existing site. It does this by basing it's own layout and design off of an already created page - say: the homepage to your own website.

Magic User Templates fetch this via a URL, and then places its own content between a tag, it finds via a css selector that you specify. Pretty easy for the user, and solves the problem I've been having of people wanting to brand the app to match their site, but not wanting to dive into their site's HTML  the HTML of the Dada Mail's templates, and then learning how the templating language Dada Mail works  (HTML::Template, basically). That's a big barrier to entry, when you just want something branded.

"Integration with Wordpress" is maybe my #1 feature request, but it's a nonsensical thing to me - what exactly does that mean? If it means: "I want Dada Mail to look like the rest of my site", then Magic Templates are the answer.

To do all this manipulation, I've taken advantage of the excellent HTML::Tree CPAN distribution, which can read in the HTML pretty well, and give a tree-based interface to the HTML the page is made up of. Then you can fairly successfully output the HTML back.

It does have some shortcomings, as it has a hard time with HTML5 tags. There's a bug opened up about it (which, I now can't track down - believe it has to do with HTML::Parser?), but there's not been a lot of work to fix it, as the maintainer wants a correct fix, and not a hack.

One solution is to use the HTML::TreeBuilder::LibXML module instead of HTML::TreeBuilder, but it comes with its own silent dragons - the docs are just so minimal, it's difficult to gauge what the heck it actually to does and what methods are now unsupported.

I gather the lack of docs is in direct inverse to the brilliance of the module author, and that's a weakness of being just a regular ol' joe (like me). Regardless, that will be the future of where I go with this, but for now, HTML::Tree is working great.

Here's the code I came up with.  Hopefully the code is very straightforward, readable, and holds no tricks whatsoever.
 
My coding is fairly straightforward for many reasons: I like no-nonsense stuff, and this is actively maintained code - it's a living project for the last 15 years! One demerit I'll give it though, is that it is a long, long sub. Someone much cleverer than I could take this down a bit.

HTML::Tree (and friends) provide just the right amount of API to do the job and do it well (although it took a little bit to figure out the right incantation). To work around some of the HTML 5 weaknesses of HTML::TreeBuilder, I call it with the following arguments:

my $root = HTML::TreeBuilder->new(
                ignore_unknown      => 0,
                no_space_compacting => 1,
                store_comments      => 1,
            ); 

For the majority of the cases, that works well.  You may notice that the code is also written very defensively. Since Dada Mail is released as an app you're supposed to simply upload and run (Basically), I can't promise that HTML::Tree and friends are going to be available, so this entire feature has to be optional, and not cause the entire app to break.  Rather, I give directions on how to install the missing CPAN modules, so that a motivated user can do that part of everything themselves.

I've also gotten into the habit of returning such defensive subs  with 3 values: a status of if it worked, a hashref of errors to describe what didn't work, and the returned string, which is also set to undef if things didn't pan out right. With that, I can then fallback to something else, if the sub didn't return what I need. The idea is that something needs to work, since a template has to be returned to be used, and it's not acceptable to have something like this break, which would break the entire app. This is a great example of why: since we're relying on an outside resource (the URL to the webpage), that resource could be gone, or the details of it (the css selector in the HTML) could be changed, rendering our configuration outdated.

What do you do? Throw an error and break the app, or return what the problem is, and try to describe would could be wrong, so that you can reapply the Magic Template in a more correct manner, later? For me, the correct answer is to do the latter. This is a consumer focused app, and people just want to get the job done, without having the app break in surprising ways.

Dada Mail Eight Released

Dada Mail v8 was recently released  - Dada Mail is a mailing list manager that works with announcement-only and discussion lists, public and private lists and a ton of things in between. It's written in Perl and runs on practically anything you can throw at it (as long is it's Unix-like)

The big ta-do with v8 is that Dada Mail supports being run under FastCGI and Plack/PSGI, when before it was only able to run under CGI. That's a big level up. Thankfully, because of the Perl community,  porting a project that began in 1999  and has been growing quite organically since then wasn't terribly impossible. For the rest of this post, I'll be going into some of approaches I used, to make this happen, with the minimal amount of work maximum amount of laziness, impatience and hubris.

CGI::Application

Fortunately or unfortunately, one of the constraints of adding support for Plack/PSGI is that CGI support still has to be kept: meaning you can throw up the app via FTP, run an install script, give it some parameters like database creds. and be off to the races.

The vast majority of installs of Dada Mail are done on pretty simple, cPanel-based (also a Perl company) shared hosting accounts. Small business use Dada Mail to keep in contact with their customers. Churches and private groups (think doctors and lawyers that want to talk shop) run off of it. With Dada Mail's support for third party email services like Amazon SES, this is actually a pretty realistic situation. Shared host companies usually have pretty terrible shared host mail servers, especially if you're running a very big/busy list; Amazon SES (and friends) work much better.

This means using a framework like Catalyst or Mojolicious is currently not looking so hot. The minimum Perl version being targeted is v5.10.1 (and even that isn't available in many shared hosts - tragic!)

But, the venerable CGI::Application framework looks great! Since it already walks the line of easily supporting both CGI apps (which it was built to do!) and PSGI/Plack apps (which it has since been given support for!).

The main difference with writing a CGI app for CGI::Application, is that you return a string for the body of your page in each method, and set parameters for any headers you would also like to send. CGI::App does the actual printing to (say) STDOUT for you. This small amount of encapsulation helps keep your app a bit tidier than without it, since you're not print()ing whilly-nilly everywhere.

So, the first step, if you were to migrate to CGI::App, is to find all those print statements, and get rid of them, only returning the data you want to show.  That's an easy step, but can take a little while to find every instance of this happening.

Run Modes

 CGI::App has the idea of run modes - run modes can be thought of each screen of your app. You probably have a default run mode. For Dada Mail there's a list page run mode, and admin login run mode, a send out a mass mailing run mode, etc.

Thankfully, the previous version of Dada Mail had a similar idea, with code right out of the Perl Cookbook:

sub run {

    #external (mostly..) functions called from the web browser)
    # a few things this program  can do.... :)
    my %Mode = (
        'default'                                     => \&default,
        'subscribe'                                   => \&subscribe,
        # a whole bunch of other subs
    );

    if ($flavor) {
        if ( exists( $Mode{$flavor} ) ) {
            $Mode{$flavor}->();    #call the correct subroutine
        }
        else {
            &default;
        }
    }
    else {
        &default;
    }
}

 Porting this to CGI was simple enough, in its setup() method. In App.pm:

sub setup {

    my $self = shift;

    $self->start_mode('default');
    $self->mode_param('flavor');
    $self->error_mode('yikes');

    $self->run_modes(
        'default'                                     => \&default,
        'subscribe'                                   => \&subscribe,
        # Many, many more, 
        );

}

CGI::App does have CGI::App::Dispatch to help with mapping PATH_INFO information to various resources served up by your app. I wasn't keen on porting Dada Mail's dispatch-ish stuff to this - yet. It seemed like a great way to break current ways we're doing things, AND the current way of doing things is 15 years of messiness.

Instead, I modified the parameters you pass to CGI::App, before it's given the param()-compatible object (in our case: CGI.pm). in the harness script (in CGI::App parlance: app.cgi):

 
use CGI;

# DADA::App is our CGI::App
use DADA::App;

# DADA::App::Dispatch changes around paramaters, and puts things

# form the PATH_INFO env. var. into param()s:  

use DADA::App::Dispatch;

my $d = DADA::App::Dispatch->new;
my $q = new CGI;
   $q = $d->prepare_cgi_obj($q);

my $dadamail = new DADA::App( QUERY => $q, );
   $dadamail->run();

Here, prepare_cgi_obj() just takes the CGI.pm object, reads the PATH_INFO and stuffs values into various parameters, much like CGI::APP::Dispatch, but without the need for me to rewrite everything - we just encapsulate it into its own method.  When I'm ready, I could move over to CGI::App::Dispatch (or not! Thus the beauty of CGI::App).

Once that's all figured out, I'm still left with a CGI script, rather than a Plack/PSGI script. My solution is to ship with multiple version of this harness script, and run a different harness script, depending on the environment I want to run under. The PSGI/Plack harness script looks like this:

use CGI::PSGI;

use DADA::App;
use DADA::App::Dispatch;
 
my $handler  = sub {
    my $env    = shift; 
    require Data::Dumper; 
    my $d      = DADA::App::Dispatch->new;
    my $q      = CGI::PSGI->new($env); 
    my $q      = $d->prepare_cgi_obj($q);
    my $webapp = DADA::App->new({ QUERY => $q });
       $webapp->run_as_psgi;
};

Dada Mail already comes with a web-based installer, which works great for installing Dada Mail as a CGI script, but would need a little more work to work under Plack/PSGI. It didn't seem really worth it to add support for that, so I instead gave Dada Mail's current installer support for being run via the command line,  so you can install/configure Dada Mail for Plack/PSGI, right before you run  the, plackup command, which is a much nicer workflow.

In a nutshell, that's all it took to take a very mature codebase and have it run under Plack/PSGI, as well as still keep support for CGI. There's some details I've skipped, like making sure database connections are persistent (I used a singleton module to keep the connection cached and persistent)

Here's some more meta on this dramatic change in the back end of things:

Dada Mail is on Github, so you can see the difference between the pre and post CGI::App-based Dada Mail. There's not too many new features, but we're hoping all this new organization helps with keeping the app growing well into the future:

Showing 281 changed files, with 38,717 additions and 31,998 deletions.

Here's how the installer works for web-based (for CGI installs), as well as on the command line (for PSGI/Plack)

More on PSGI/Plack support with Dada Mail

Please give Dada Mail a spin, the next time you have the need to set up a mailing list.