Tuesday, April 8, 2014

Dada Mail Seven Released! Rejoice!

I've just shipped v7 of Dada Mail! Dada Mail is a web-based mailing list manager, written in Perl. It's a project I started way back in 1999 - one of those, "started in my dorm room" types of projects.

I'm currently catching my breath from the big push to finally Ship, but it feels good to get it all done.

One of the more interesting (to me) aspects of this release was to finally ship a PDF manual. I write most of the docs in Plain Old Documentation (with a little HTML sprinkled in), which is then turned into HTML, which is then! then turned into a PDF  book. I've been wrestling with how best to do with, but it turned out just using HTMLDoc directly was the best way to go about things. A little (minor) futzing with my POD/HTML to make the table of contents pretty is all I needed to do.  I already and still have an HTML version of the manual online for people to utilize, but now people can download it and read it on their favorite device, which a lot of my users have been clamoring for.

I think I surprised myself to know that the book surprised 300 pages by a wide margin! That's a lot of writing! It gives me a lot of confidence to write another book, since I can't tell myself, "Well, books are long! And there's no way you'll ever write that much!" . I can now tell myself, "Whoops! You somewhat mistakenly already have! DO IT AGAIN!"


Wednesday, October 16, 2013

Email::Valid Peculiarities

Over here at Dada Mail, we do a lot of verification of email addresses when confirming subscriptions. Is the form of the email correct? Has this address been used to try to subscribe with before? Has the user themselves confirmed their subscription? What happens when there's an error? There's a lot to it (sadly) and if something in one of these steps goes wrong, a subscription doesn't happen, or worse yet, a mailing list is tarnished with not-so-valid addresses. This little tale is about invalid in form addresses getting on people's mailing lists.

A user posted on the Dada Mail support forums: "There's an address with a space in it, that's entered my list! It's causing problems with delivery of the ENTIRE list? How did that happen?"

A lot of the time developing a very complicated piece of software is supporting it (I think, anyways). When a question like this comes down the tubes, I have no great answer, except it must have been an outside thing. So I ask, "Was the address added directly into the backend?" That's something that happens from time to time, although there are many pitfalls: importing data can be done incorrectly - and even if it's done correctly once, the design of the backend can change, and then it's not done correctly anymore.

That's why, if it's an option, it's a good idea to use the API that presented to the developer, rather than try to retinker it, yourself. 

When the user replied back with, "Nope!" I knew I was really stuck.  "But I have tests!"  I protested to my Macbook Pro. "Coverage!" "I didn't roll my own email validator!" "I did everything you're supposed to do!"

 So, current tests be-damned, I had to start from square one: Attempt to recreate the problem, in the simplest way I could. So, I tried to join a list, with an address with a space in it, like this:

myaddress@example .com

And, wouldn't you know it, the app took the address like nothing was wrong. Now,  totally believing my user, but being in total disbelief myself, it was time to hit the code.  We use Email::Valid in Dada Mail, to test the validity of an email address, like so:



#!/usr/bin/perl 

use Email::Valid; 

my $address = 'someone@example.com'; 

if(Email::Valid->address($address)){ 
 print "Valid!"; 
}
else { 
 print "NOPE!"; 
} 
# prints, Valid! 


This is wrong.  Observe:


#!/usr/bin/perl 

use Email::Valid; 

my $address = 'someone@example .com'; # notice the space! 

if(Email::Valid->address($address)){ 
 print "Valid!"; 
}
else { 
 print "NOPE!"; 
}
# Prints, Valid!

And, there's my problem, in a pretty simple case.

So, what's going on? Basically my code had assumed that Email::Valid's address method returned a boolean value. It does not. From Email::Valid's docs (emphasis is mine):
This is the primary method which determines whether an email address is valid. It's behavior is modified by the values of mxcheck(), tldcheck(), local_rules(), fqdn(), and fudge(). If the address passes all checks, the (possibly modified) address is returned as a string. Otherwise, the undefined value is returned. In a list context, the method also returns an instance of the Mail::Address class representing the email address.
Big difference! So let's try that code snippet, again:

 
#!/usr/bin/perl 

use Email::Valid; 

my $address = 'someone@example .com'; # notice the space! 

my $validated_address = undef; 
if($validated_address = Email::Valid->address($address)){ 
 print "Valid!: $validated_address"; 
}
else { 
 print "NOPE!"; 
}
# prints, Valid!: someone@example.com

Ah-ha: Email::Valid will take it upon itself to modify the email address for you - but only in certain circumstances - spaces in certain places of the address. So, this still returns undef:


#!/usr/bin/perl 

use Email::Valid; 

my $address = 'some one@example.com'; # notice the space! 

my $validated_address = undef; 
if($validated_address = Email::Valid->address($address)){ 
 print "Valid!: $validated_address"; 
}
else { 
 print "NOPE!"; 
}
# prints, NOPE!

This certainly makes things confusing. The reason that the email address gets modified, is that under the hood, Email::Valid turns the address you give it, into a Mail::Address object (it'll also just accept an Mail::Address object, and Mail::Address itself will take out that pesky space in the domain part of our email address, when it's own address() method returns a string:


#!/usr/bin/perl 

use Mail::Address;

my    $address  = 'someone@example .com'; # notice the space!
my    $modified = ( Mail::Address->parse($address) )[0]->address;
print $modified;

# prints,  someone@example.com

This makes it difficult to patch up Email::Valid, since then I'd have to delve into Mail::Address - and perhaps Mail::Address has a perfectly good reason to do this. Since both Email::Valid and Mail::Address have been around for so long, and many programs have been written with their API the way it is, it's a pretty good bet they'll stay this way.

So, what to do? Since my app's validation system is a little more than, "is the form correct?" it's difficult in the  workflow to take in account of the changes Email::Valid makes to the address I pass - I'm not sure if I even like the idea: getting a boolean back in a validation method seems a better idea - most of my own valiation methods in the app have the pattern of, return a boolean status, plus a hashref saying what may be wrong,

 
# In a perfect world: 
my ($status, $errors) = $validater->check_out_this_address($address); 
if($status == 1){ 
    # Good to go!
}
else { 
    print "Dude, PROBLEMS:\n"; 
    for (keys %$errors){ 
        print '* ' . $_ . "\n"; 
    }
}


So, for example, say I'm importing a whole address book of addresses - thousands of addresses! And before I do that, I sort out the duplicate addresses, as the validation steps are many, and cpu consuming. Using Email::Valid the correct way could introduce subtle bugs into the system. Simplifying things:

 
#!/usr/bin/perl 

use Email::Valid;

# Say, I've already sorted out dupes: 
my @check_these = 
(
'one@foobar.com',
'another@foobar.com',
'another@foobar .com',
'two@foobar.com',
'blech'
); 

my @validated = (); 
my @invalid   = (); 

for(@check_these){ 
    my $address; 
    if($address = Email::Valid->address($_)) { 
        push(@validated, $address); 
    }
    else { 
        push(@invalid, $_); 
    } 
}

print "Good! Addresses:\n";
print "* $_\n" for @validated;

print "BAD! Addresses:\n";
print "* $_\n" for @invalid;
This prints out,
 
Good! Addresses:
* one@foobar.com
* another@foobar.com
* another@foobar.com
* two@foobar.com
BAD! Addresses:
* blech


As you can see, some weird things happen:
  • The number of addresses total that are returned isn't the same as what gets passed
  • All the invalid address are not reported
  • my once unique list now isn't so unique anymore. 
 
I'm not sure yet what the best thing to do is. For now, sadly, I have to do something hacky in my own email (in form) validator - I just look for a space, if there's a space, it's automatically invalid:



#!/usr/bin/perl 

use Email::Valid; 

my $address = 'someone@example. com'; # notice the space! 

if(is_valid_email($address)){ 
 print "Valid!"; 
}
else { 
 print "Nope!"; 
}

sub is_valid_email { 
 my $a = shift;
 
 if($a =~ m/\s/){ 
  return 0; 
 } 
 if(Email::Valid->address($a)){ 
  return 1; 
 }
 else { 
  return 0; 
 }
}

# prints, Nope!

So many miscalculations on my part, which! means much to learn from my own dumb self:

Firstly, I couldn't believe it was a bug, since the way things have worked hadn't been changed in eons.

Next: way back when I first wrote put this validation stuff in the app yarns ago, I was thinking that Email::Valid was nothing but a nice wrapper around a very, very complex regex created by Jeffrey Friedl back in the day for a book on Regular Expressions that he wrote. It looks a little like this. It's a doozy.

Another miscalculation I had made was, "I have tests! How could this fail!" and I made the assumption the user was in fact on the wrong. So, I looked at my tests ,and whatayaknow: 
 

TODO: {
    local $TODO = 'This test fails.';
    ok(valid_email_sub('test@example .com')           == 0); 
};

So I actually knew about this problem before, and forgot - there's actually a link to a bug tracker in the actual test, which means I had handed off the problem to someone else to fix. Lazy!

Things to learn: 

  • Read the docs! This very peculiarity is documented in Email::Valid's own docs. Don't assume!

Let's see an example of how the address may be modified:
$addr = Email::Valid->address('Alfred Neuman ');
print "$addr\n"; # prints Neuman@foo.bar

  • Just because you think everything's working, doesn't mean there aren't weird little edge cases -  it's probably a given.
  • Tests are good! Skipping tests that don't pass doesn't always help you. 
  • Reporting bugs in other modules is good! Trying to fix them? Probably better. 

My thanks to rjbs for being patient as I stumble around a module he has been nice enough to adopt, warts and all.
 

Tuesday, October 1, 2013

Dada Mail v6.7.0 Released! Save as Draft, Enhanced Attachments, jQuery Plugin

Dada Mail, v6.7.0 has been released! Hurrah! You can read the official announcement here, view the changelog, as well as download/install it yourself. If you're looking for a simple to use mailing list manager (written in Perl), give it a go! It's one of the few Perl apps that's available as a one-click install in many services that you'll see in hosting control panels, like cPanel - safe to say, it's installed a ton!

Feel free to explore away Dada Mail Lane - but that's not really what I wanted to focus on this blog post. What I really wanted to focus on, is how Perl and  CPAN has helped me ship the new features of this app in a matter of weeks. So let's do it!

Feature! Save as Draft


The Save as Draft feature in Dada Mail allows you to start writing a message, then allow you to save it as you go, so you don't have to finish creating a message in one sitting. It's very similar to the auto-save feature in the blog software I'm using right now to author this post!

When I create a new feature, there's a few things I like to make sure I'm doing right: I like to make things as simple as possible, but have room for future growth. I can't predict the future, but I still can try to think of features that I would like to work on, in the future.

For Save as Draft, the backend is super simple: the SQL table (Dada Mail supports MySQL, PostgreSQL and SQLite) looks a little bit like this:

 
CREATE TABLE IF NOT EXISTS dada_message_drafts (
id INT4 NOT NULL PRIMARY KEY AUTO_INCREMENT,
list varchar(16),
created_timestamp TIMESTAMP DEFAULT NOW(),
last_modified_timestamp TIMESTAMP,
name varchar(80), 
screen varchar(80),
role varchar(80),
draft mediumtext
);

id is your primary key,

list is used to distinguish what list the draft messages belong to.

created_timestamp and last_modified_timestamp give some file status-type information on your draft, since you can think of a draft as a sort of document. It's nice for example, to be able to sort your drafts by their last_modified_timestamp, so we can always load up the most recent draft, when someone visits the screen that we use to edit a message.

Speaking of the two timestamp cols in thw same table,  there seems to be many different opinions on how exactly to use them together, as some databases don't allow you to have >1 cols with a default timestamp values. Again, for simplicity sake, I decided to make INSERTS set the created_timestamp automatically, and setting the last_modified_timestamp explicitly:

INSERT INTO dada_message_drafts (list, screen, role, draft, last_modified_timestamp) 
VALUES (?,?,?,?, NOW());

and then UPDATES would again just explicitly update the last_modified_timestamp

UPDATE dada_message_drafts SET screen = ?, role = ?, draft = ?, last_modified_timestamp = NOW() WHERE list = ? AND id = ?;

Again: simple, simple, simple.

A bigger problem I faced was basically how to save what is going to be an ever-changing set of form fields. I didn't want the Save as Draft feature to dictate what I'd be presenting the user in terms of widgets and things that they can utilize.  

Maybe one day, I want to set up how the Partial List Sending feature works differently in the next version, but not have to worry about having a column in this table for drafts now useless, or worse: shoehorned for a feature it wasn't meant to be -

 I didn't want this:

 
CREATE TABLE IF NOT EXISTS dada_message_drafts (
id INT4 NOT NULL PRIMARY KEY AUTO_INCREMENT,
list varchar(16),
created_timestamp TIMESTAMP DEFAULT NOW(),
last_modified_timestamp TIMESTAMP,
name varchar(80), 
screen varchar(80),
role varchar(80),
header_subject,
custom_header_1,
message_plaintext,
message_html mediumtext, 
attachment_1 mediumtext, 
attachment_2 mediumtext, 
attachment_3 mediumtext, 
custom_widget_value_1 mediumtext
);

Blech. Rather, it made more sense to encode all the form values together, and have an easy way to decode them, which runs right into the next problem:

Now that I have the values to fill in my form - well, how do I do that, easily - and more importantly with the flexibility I want. It's a big-time drain problem - sure: filling out a text field is easy, but how about a check box array, or a popup form? Forget it!

So, of course there's a solution in Perl/CPAN! And the one I decided upon was HTML::FIllinForm::Lite. Lite, because this app is shipped with any required modules (the app is easy to install for people who know nothing of cpan/cpanm/perlbrew/so one and so forth) and that also means no XS modules, outside the core.

HTML::FillinForm::Lite requires a datasource to use to fill in the HTML form fields (that you also supply) and one of the data sources it supports is an object with a param() method, like CGI.pm (yes! I still use CGI.pm - Dada Mail requires only Perl v5.8.1 and a CGI env. to work!) CGI.pm itself has a way to "save the state of the form", which sounds goofy, but is exactly what we need to do! The examples in its own docs only show how to write to a filehandle, and it's a little easier to passa scalar to a database handle, so you can do something like this very Perlish trick, to get the data into a string form:

 
sub stringify_cgi_params {

    my $self = shift;
    my ($args) = @_;

    if ( !exists( $args->{-cgi_obj} ) ) {
        croak "You MUST pass a, '-cgi_obj' parameter!";
    }

    my $q = $args->{-cgi_obj};
    $q = $self->remove_unwanted_params(
        {
            -cgi_obj => $args->{-cgi_obj},
        }
    );

    my $buffer = "";
    open my $fh, ">", \$buffer or die 'blarg!' . $!;
    $q->save($fh);
    return $buffer;
}



The old, "open filehandles directly to Perl scalars" trick (open my $fh, ">", \$buffer or die 'blarg!' . $!;)

Once we want to repopulate the form, it's easy enough to decode those form fields, again, like so:


sub decode_draft {
    my $self  = shift;
    my $saved = shift;
    open my $fh, '<', \$saved || die $!;
    require CGI;
    my $q = CGI->new($fh);
    return $q;
}


$q is now a CGI object, that we can pass right to HTML::FillinForm::Lite: 

require HTML::FillInForm::Lite;
        my $h       = HTML::FillInForm::Lite->new();
        $str = $h->fill( \$html_str, $q );



Couldn't be any simpler.

Features! Attachments



Dada Mail has supported sending a message out with attachments for quite some time - I can't remember I time it didn't, but it only presented the user with a file upload widget, for the user to pick a local file. Not very flexible, and not something that was going to work, as it's not really possible to save the state of the file upload widget without really jumping through some flaming hopes - so let's not.

What I really needed was something a lot more flexible: I wanted to allow the user to pick a file from their local computer, as well as have available files that have already been uploaded. I also wanted it easy to add an attachment, but also remove one, if they decide against it. Finally, and most importantly, it had to all work with the new Drafts feature - so, if we're using a different form field - something HTML::FillinForm::Lite can fill in, that would be ideal.

Dada Mail already ships with a a web file browser called, KCFinder, which itself is an alternative to a somewhat expensive add-on to CKEditor called, CKFinder. Luckily, KCFinder works flippin' great as a image uploader for Dada Mail's HTML message text box. Could I use it to simply fill in the value of a form field?

I wasn't looking forward to reverse engineering how KCFinder talks to CKEditor, but to my surprise, there was an already-working example on KCFinder's site, that shows you how to open a new KCFinder window, and then capture the name of the file selected, for you to do anything you want with - say fill in a text field. Perfect! With a few flurishes, that feature was stamped out, in record time.

Incidentally, KCFinder has a semblance of security by using a sessioning system, so that only the right people can have access to its file upload/browsing feature. You don't want just anyone to be able to upload file type X to your server. Nightmarish!

Dastardly, KCFinder's server backend is in PHP, so it uses PHP-style-sessioning. Not the happiest place to be, if you have a Perl app. Thankfully, there's PHP::Session to bridge the PHP-style, Perl-style gap. Saved again!

 Feature! jQuery Plugin


Perhaps some will remember the Bad Old Days, when one's JavaScript was a mish-mash of ill-begotten weird lines of noise - back before their was a framework of any way/shape/form to 'rastle all this stuff together. Dada Mail harkens back to the winter of 1999, and it's gone through many iterations of its JavaScript functionality - from minor flourishes, to utilizing prototype.js and Scriptaculous, and of course porting all that to jQuery.

With learning jQuery, one wants to make a jQuery plugin, and one learns sort of a dirty little secret: there's no one way to do so! It's filled with compromises (I mean - check it! It even states: Remember, it's a compromise!), and metric ton of boilerplate code. You can't really think of a plugin as having a namespace, as plugins kinda only support one, "method". It makes Perl's OO system look exceptionally attractive. What a train wreck. Sigh.

So, I guess(?) you just sort of have to except that this a very warty part of jQuery, and JavaScript is sort of a very warty thing in of itself, but there's no alternative. My only real constraint that I wanted was to have > 1 method available. For this plugin: I wanted to be able to create a subscription form, that ties into a Dada Mail, mailing list, and I also wanted a method to control an already existed form, so that a designer can have full control on the presentation of the form, if they don't like what gets created by the plugin itself.

I used this boilerplate as a basis for my own plugin (179 lines! Of boilerplate!).  and created the two, "methods" I wanted, and once you get your brain around how things work, it was, eh, OK. Once the plugin takes control of a form, either one it creates, or one it, "takes control of", I wanted to simply show the results of a subscription request, in a modal window.  Like-a-dis:



Luckily, Dada Mail already takes advantage of another jQuery plugin, called Colorbox, so it was easy enough to leverage that for our modal window. The trick really was having jQuery talk to Dada Mail, and for that we use Dada Mail's sparkling new RESTful interface, that jQuery can talk to via JSONP, which itself is a weird hackity hack, to get over a security thing-a-mabob. Great!

Like a ton of things in Perl, working with JSON is pretty darn easy. Here's how we take a request from jQuery:

$.ajax({
 url: $("#" + targetForm).attr("action") + '/json/subscribe',
 type: "POST",
 dataType: "jsonp",
 cache: false,
 data: JSON.stringify(
  { 
   list:  $("#" + targetForm + " :input[name='list']").val(),
   email: $("#" + targetForm + " :input[name='email']").val(),
   fields: fields
   }
 ),
    contentType: "application/json; charset=UTF-8",
 success: function(data) {
  /* Do something, after success! */
  $.colorbox({
   html: '<h1>
Hey, that worked!</h1>
',
   opacity: 0.50
  });
  
 },
 error: function(xhr, ajaxOptions, thrownError) {
  console.log('status: ' + xhr.status);
  console.log('thrownError:' + thrownError);
  $.colorbox({
   html: '<h1>Apologies,</h1>An error occured while processing your request. Please try again in a few minutes.',
   opacity: 0.50
  });
 }
});


And here's how we could handle the request:

    require JSON::PP;
    my $json = JSON::PP->new->allow_nonref;
    if ( !$q->content_type || $q->content_type =~ m/text\/html/ ) {
        # Perhaps, die here? Show the API docs? 
    }
    elsif ( $q->content_type =~ m/application\/json/ ) {
        # well. OK, then.
    }
    else {
        die '425';
    }
    my $post_data = $q->param('POSTDATA');

    my $data = undef;
    try {
        $data = $json->decode($post_data);
    }
    catch {
        die '400';
    };

    # Now, we can work with $data! 


the $data hashref now contains the parameters that the jQuery stuff passes to us (list to subscribe to, email to subscribe with, etc). I did a ton of housekeeping in Dada Mail's own system to make it dead easy to go through the subscription process, and have the method return a quite the complex data structure - with which you can gleam all sorts of information from: was it successful? No? What was the problem? What should I do now - show a message? redirect to a different URL?

A complex data structure is fine, but it's a Perl complex data structure. Thankfully, turning a data structure into JSON for jQuery to understand is again, dead simple:


    my $fancy_data = { 
         email => 'you@yours.com', 
         list  => 'somelist', 
         status => 1, 
         fancy_message => 'Hey, that worked! Hurrah!', 
    }; 
    require JSON::PP;
    my $json = JSON::PP->new->allow_nonref;
    my $data_back = $json->pretty->encode($fancy_data);


$data_back will now be the following JSON:
 
{
   "email" : "you@yours.com",
   "status" : 1,
   "fancy_message" : "Hey, that worked! Hurrah!",
   "list" : "somelist"
}


I'm actually really glad I had to figure out how to make things work by creating this JSON response, as it truly forced me to make the Perl code clear, pragmatic, and a whole less complex - which surprisingly made things a lot more flexible. Kind of like being in the middle of a solitaire game you know you're going to win, there's more simplification I can do to really drill down and make things way easier to read. Again: no real tricks, just elbow grease and going in there and making small changes, which add up!

Here's the docs to the jQuery plugin, as well as the RESTful interface (which is still very much in a moving-and-shaking phase of things - I still have a ton to learn!), that'll give you a better idea on how this all fits together - it's complete with full examples of using the interface in both a Perl app, as well as your own jQuery app. The jQuery plugin for this release can be viewed here.


Friday, September 6, 2013

Dada Mail, v6.6.0 Released!

I've just put out a new release of Dada Mail - v6.6.0. Its  a web-based mailing list manager, written in Perl. You can check out/fork/push/pull the source code here. Some of its main goals is to be pretty easy to install by mere mortals, easy to work with, yet pretty darn powerful and flexible, to fill a ton of different needs - it's not really trying to be a replacement to the creaky Majordomo, or Mailman. 

I'm heading up to my 14th year of developing the application. It's currently free to download from its own support site as well as from various installer services that have popped up in the last few years (Mojo Marketplace, Installatron, Softaculous, etc) Follow along with the project on Twitter, or hey - we have a mailing list too!

There's lots of things a like about working on it, but the application has become quite monolithic, and a little less flexible in its underlying code, than I'd like. In the next few months, I'm hoping to alleviate that, without falling into the problem of totally rewriting the app (it would be almost impossible to do so, with a one-dude team).

My initial idea is to move from basically no framework, to working with CGI::Application. That should help me gain some sort of framework, and a way to break the overgrowth of parts of this app, into more manageable pieces. It'll also hopefully give me a bridge to be able to have Dada Mail run as a CGI script (how people are used to running Dada Mail), as well as a PSGI app (a great alternative, that has a TON of benefits, one being SPEED!).

From there, we'll see how it works, but it may be time to move on to some more whiz-bang things that a framework like Mojolicious would provide. The bummer with trying to move to Mojolicious is that the Perl version required is rarely met on the shared hosting accounts that the majority of people who run Dada Mail are on. Saying, "hey, just install perlbrew!" is a far cry on what lots of people are capable of. They're used to a one-click install. Sigh.

Maybe cPanel, being a Perl-based shop, knows of a better way to have available a more up-to-date Perl version? A how-to, so not to fudge this up, would be super-helpful and somewhat break huge shackles from those that ship apps for this types of hosting situations. (maybe you know?)

Without further ado, here's the changelog from this last release:


Friday, November 9, 2012

Dada Mail v6.0.0 Alpha 1 Released

This is the 6th MAJOR release of this app I've made since December of 1999. I think that's pretty wild. Github tells me that, compared to the last stable version of Dada Mail (v5.2.1) there has been: 353 changed files with 18,740 additions and 16,697 deletions.  Goodness.

One of the major focuses on this release has really nothing to do with Perl, but  JavaScript - I managed, somehow to port all the Prototype/Scriptaculous code to jQuery. Sadly, the Prototype project seems to be dead. But gladly, the jQuery code that's now part of the app is pretty easy to read.

To those interested, here's a rundown of the differences between the v5 and v6, as well as the direct link to install,






Dada Mail 5 to Dada Mail 6 Upgrade Guide


Focus

The focus of this major release of Dada Mail is to make room for growth of the app, by creating a better foundation for future features.

JavaScript UI/UX Breathing/Growing Room

Dada Mail's UI/UX has been migrated to jQuery, from Prototype, where development has all but ceased.
The vast majority of Dada Mail's extra functionality through JavaScript is done in an unobtrusive way. A great example of new features allowed with this JS library move is the rewritten and redesigned Tracker plugin.

Towards a new web framework

Dada Mail is written with the thinnest of veneer of a web framework, and the next major version of Dada Mail may be a migration to a more robust web framework such as CGI::Application, Catalyst, or Mojolicious. v6 of Dada Mail moves the various components that make up Dada Mail (Perl code, JavaScript code, templates, images, stylesheets) into consolidated bundles of files and directories that are more sensible than previous versions.

Database Schema

There are no changes in any of the database schemas between Dada Mail 5 and Dada Mail 6.
Upgrading is fairly straightforward, with no need for any Dada Mail 5 to Dada Mail 6 SQL upgrade scripts.
When using the Dada Mail Installer (which you should use, when installing or upgrading), you will definitely want to create a new .dada_config file, when asked.
There is one new config variable, $SUPPORT_FILES that needs to be set up correctly. If you do not make a new .dada_config file, or install manually, without putting in this new variable, the upgrade will not work.

Perl Version Requirements

This is currently no change in the Perl Version requirement of v5.8.1. This may change, before a stable release of v6 is released, most likely to v5.10.x of Perl. We're still working on that.

.dada_config file

The .dada_config file that's generated by the included installer has undergone a few changes.

Removed variables: $FILES, $TEMPLATES, $TMP, $BACKUPS, $ARCHIVES, $LOGS

These variables were added to the .dada_config file, but are almost kept the same as the default, ala:
        $FILES                    = $DIR . '/.lists';
        $TEMPLATES                = $DIR . '/.templates';
        $TMP                      = $DIR . '/.tmp';
        $BACKUPS                  = $DIR . '/.backups';
        $ARCHIVES                 = $DIR . '/.archives';
        $LOGS                     = $DIR . '/.logs';
Instead of taking up extra space and line noise, these files are simply added to the dada/DADA/Config.pm file as the default. You may still add these variables in the .dada_config file to change their default, but it's not really recommended. The, $DIR variable has been added to the dada/DADA/Config.pm as a true config variable, so you do not want to have the first line read,
        my $DIR         = '/home/account/.dada_files';
Simply,
        $DIR         = '/home/account/.dada_files';
Will be fine. Keeping, my will probably break things.

Added variable: $BACKEND_DB_TYPE; Removed variables: $SUBSCRIBER_DB_TYPE, $ARCHIVE_DB_TYPE, $SETTINGS_DB_TYPE, $SESSION_DB_TYPE, $BOUNCE_SCORECARD_DB_TYPE, $CLICKTHROUGH_DB_TYPE

For the SQL backend, all these variables are already set the same, ala:
        $SUBSCRIBER_DB_TYPE       = 'SQL'; 
        $ARCHIVE_DB_TYPE          = 'SQL'; 
        $SETTINGS_DB_TYPE         = 'SQL'; 
        $SESSION_DB_TYPE          = 'SQL'; 
        $BOUNCE_SCORECARD_DB_TYPE = 'SQL';
        $CLICKTHROUGH_DB_TYPE     = 'SQL';
For the Default backend, they're set like this:
        $SUBSCRIBER_DB_TYPE       = 'PlainText'; 
        $ARCHIVE_DB_TYPE          = 'Db'; 
        $SETTINGS_DB_TYPE         = 'Db'; 
        $SESSION_DB_TYPE          = 'Db';
        $BOUNCE_SCORECARD_DB_TYPE = 'Db';
        $CLICKTHROUGH_DB_TYPE     = 'Db';
To reduce line noise, $BACKEND_DB_TYPE is being introduced. If set to, SQL, it'll set the other variables accordingly. Same with setting this variable to, Default.

%SQL_PARAMS: removed table names

Remove to further lesson the line noise.

Installer

Database Connection, Bounce Handler POP3 Connection Testers

The installer now has inline testers for both your Database Connection, and the Bounce Handler's POP3 Connection, to help you make sure these work, before committing to an installation.

Installer can now grab information from the previous installation

When using the installer to upgrade a current Dada Mail, it'll able to grab some of the configuration information from your previous installation: Program URL, Support Files Directory, Dada Mail Root Password, Backend Options, Plugin/Extensions that were previously installed, and your WYSIWYG options.
(If you added additional configuration variables after an install has completed, you will still have to re-add these manually)
This can help save time during an upgrade and ensure that everything goes smoothly. You can also re-enable the installer, after a successful initial installation, to change your Dada Mail configuration - for example: add/remove a plugin.

Template Changes

"templates" directory move

In v5, template files lived in the directory:
dada/DADA/Template/templates
They are now located in:
dada/templates
It's hoped that this directory move will make it easier for people to find the templates directory. This directory is still in the, dada directory, instead of, say, the, static directory since template files are themselves little, simply programs, that need to be interpreted, and aren't themselves actually static.

Plugins: Templated out

There was a voluminous amount of inline HTML in the plugins/extensions that are shipped with Dada Mail. This HTML is now saved in separate template files, which you can find in,
dada/templates/plugins
Each plugin has its own directory for its templates, eg: dada/templates/plugins/tracker, etc.
The Scheduled Mailings plugin still has program generated and inline HTML. It's a big mess.

JavaScript, images and css files removed from templates directory, relocated in, "static" directory

Many static files that used to reside in the, templates directory have been moved to the, static directory, to be served directly from the webserver. These include JavaScript, image and cascading stylesheet files.
This change should help performance of the app. These files are also not cached in Dada Mail's Screen Cache, as they're going to be served faster by the webserver simply as static files.

Inline JavaScript removed from template files

The vast majority of JavaScript that may have been found inline in the template files has been removed. You will now find that code in the, static/javascripts/dada_mail.js file.

JavaScript

Migration from Prototype/Scriptaculous to jQuery/jQueryUI

Most of the JavaScript that relies on the Prototype/Scriptaculous libraries has been migrated to jQuery/jQueryUI. Some remains, and jQuery has been configured in, NoConflict mode. We hope to move all Prototype/Scriptaculous code to jQuery eventually.

Static Files directory location, $SUPPORT_FILES config variable, Installation and Configuration

Dada Mail now separates out its static files needed for the app, from the dynamic template files. These static files ship with the app under the directory,
dada/static
When you install Dada Mail using the included Installer, this entire directory will be copied to the publicly accessable directory that you specify, so that they may be served directly by the webserver. JavaScript, images and cascading style sheets will be served from this directory.
In previous versions of Dada Mail, these files were served via Dada Mail, creating a new running instance of the app per request, which is slow, awkward and confusing.
Both the server path, and URL to this directory will be saved in your .dada_config file, under the varible, $SUPPORT_FILES. This variable is required to be correctly configured, for Dada Mail to work.
This does create an additional thing that needs to be configured for Dada Mail to be successfully installed, but we feel this extra step is worth it, for performance and for future features of the app.

Clickthrough/Message Opens, etc tracking support dropped for Default Backend

The Tracker plugin and all of its functionality now requires the use of one of the SQL backends.

List Control Panel Screens

Mass Mailing >> Send a Message

Options for Mass Mailing, including File Attachments, Archive Options, etc have been moved to a tabbed interface.

Membership >> View

Breakdown by Domain Graph

The Breakdown by Domain graph, that used to be in the Tracker plugin has been moved to the Membership - View screen. Clicking on any of the pie slices will perform a search on the domain you have selected.

Search Autocomplete

The Subscriber search form has autocomplete capabilities, which searches and matches on email addresses

Membership >> Recent Activity

Subscription Trends graph

The Recent Activity screen now has a graph labeled, Subscription Trends. This graph shows daily, as well as cumulative data on Subscriptions and Unsubscriptions on your mailing list.

WYSIWYG editors and Template Tags

In version < 6 of Dada Mail, when template tags like,
        <!-- tmpl_var list_settings.list_name -->
are entered in the rich text (rather than source view) of the WYSIWYG editor, the "<", and, ">" characters are converted into their HTML entities, so the source of your message now looks like this:
        &lt;!-- tmpl_var list_settings.list_name --&gt;
Which breaks the tag, and you will see the original template tag, which will not be converted to its value.
In v6, Dada Mail will look for these types of conversions, and convert them back, fixing the template tag, to be parsed correctly.

Data Cache

Dada Mail now has a data cache, for things like generated JSON files. This data cache can be managed with the new, DADA::App::DataCache module and can be enabled/disabled using the $DADA_CACHE config variable. Old cached data will be removed periodically (1 hour), automatically. The data cache is located in the,
.dada_files/.tmp/_dada_cache
directory. The files, as well as the directory itself, may be safely removed at any time.

Plugins

Tracker

The Tracker plugin has been essentially rewritten, and its features have been expanded. All charts, graphs and maps generated are now interactive and mousing over various parts of the chart/graph/map will reveal additional information. The UI itself has been redesigned, as well.
The new charts/graphs/maps are powered by the Google Visualization API. Data, in the JSON format is created for Dada Mail using the Perl CPAN module, JSON (http://search.cpan.org/~makamaka/JSON/). This module will need to be installed on your hosting platform for these charts/graphs/maps to be created.
Although not part of the standard Perl Library, the module proves to be so useful, that it's usually already installed on many hosting platforms (like LWP, CGI, etc).

Expanded location-based information

As well as country-specific information, Tracker now supports showing information on a city-specific level.

City Reports by IP Address

Reports are now generated per country, breaking down then by city, and then by individual IP address that may have caused an event, (a message open, clickthrough, forward, archive view) and listed in chronological order.

Included City Geo IP Data

Dada Mail now comes with both a country Geo IP database, as well as a city-level Geo IP database. The city-level database is fairly large - ~20 megs., but it's utility is worth its size. Geo IP data is provided by http://maxmind.com, who also provide more accurate country/city data, on a subscription.
The distro of Dada Mail is much larger - around 20 megs. compressed, rather than ~7.5 megs for v5 of Dada Mail. This additional size is mostly from the bundled city-level Geo IP database.
Both the city, and country-level Geo IP databases can be found in the,
dada/data
directory.

Bounce Handler

Filename change: dada_bounce_handler.pl to bounce_handler.cgi

Make sure to update any cronjobs, links, etc.

Rules

The Bounce Handler Rules, used to match the type of bounced message that gets sent to the bounce handler, has been moved out of the code and into its own file, which you may find at,
dada/data/bounce_handler_rules.pl
You may also copy the file, and place it in your,
.dada_files/.configs
directory, make your own changes/customizations, and the Bounce Handler will use your copy, instead of what ships with Dada Mail.

Bridge

Filename change: dada_bridge.pl to bridge.cgi

Make sure to update any cronjobs, links, etc.

List Email Mail Forwarding Support

You may now set up a List Email as a Mail Forward, that pipes a message directory to Dada Bridge, rather than as a POP3 email account, that is checked on a schedule via a cronjob. This should greatly help any lag between when the message is sent to the List Email address, and when the message is checked by Bridge, and sent out to the entire mailing list.

Clean Up Replies Filter - Disabled

This experimental feature doesn't work very well and has been disabled in v6.

Scheduled Mailings - DEPRECATED

The current scheduled_mailings.pl plugin, as it currently stands, needs a rewrite and isn't currently benefitting from the advances of code style, best practices and techniques offered by the rest of Dada Mail.
Unless rewritten, it will be removed in a future version of Dada Mail.

Extensions

dada_digest.pl - REMOVED


Cheers, 

Monday, July 30, 2012

Dada Mail v5.2.0 Released



Hello Everyone,

Dada Mail v5.2.0 has been released - install/upgrade/download it from:

http://dadamailproject.com/support/documentation-5_2_0/install_dada_mail.pod.html

The changelog is below:



Summary 5.2.0

Dada Mail v5.2.0 release focuses on WYSIWYG editors, which allow you to author your HTML email messages in rich text, right from within the List Control Panel.

Dada Mail now comes bundled with three different WYSIWYG editors: CKEditor, Tiny MCE and FCKeditor. KCfinder, an image/file browser/uploader is also bundled. Dragging and Dropping images to be used in your email messages is also supported in Dada Mail. All these utilities can be installed easily using the included Dada Mail Installer.

We've streamlined the Send a Message and Send a Webpage screens to better utilize and more cleanly present these editors, while still allowing you full flexibility on how you author your messages: We've moved to a tabbed interface for the mailing list message, so that you can easily toggle between your HTML Version and your PlainText version, without unnecessary scrolling.

If upgrading from any previous version of Dada Mail, we highly suggest Creating a new .dada_config file, when using the included Dada Mail installer, as many items in the starter .dada_config file have now changed.



Features 5.2.0

WYSIWYG/File Browsers


Dada Mail now supports and comes bundled with the following WYSIWYG editors:


  • CKEditor 
  • Tiny MCE 
  • FCKEditor 


Dada Mail also comes bundled with the file browser/file upload utility called, KCFinder. This utility allows file file browsing and file (image) uploading for all three included WYSIWYG editor.

Dada Mail Installer Support

The Dada Mail Installer now has the ability to configure and install all three WYSIWYG editors, as well as the file browser. Installing all these utilities is optional, but much easier than doing it manually. The editors and browser are bundled with Dada Mail in the, dada/extras/packages directory.

Added Support for Tiny MCE

Along with Support for CKeditor and FCKeditor, Dada Mail now supports the Tiny MCE WYSIWYG HTML editor in its Send a Message and Send a Webpage screens.


Support for Drag and Drop images (in some browsers)

Instead of having to go through the File Browser to upload an image, you can simply drag an image into one of the WYSIWYG editors. The image will then be saved in your file browser's file uploads directory and will be able to be used again for a future mass mailing.

This doesn't work for all browsers, but is confirmed to be support in Firefox 14+.



Discussion Lists: Experimental Support for Cleaning up long quoted replies of mailing list openings/signatures

Look for the option labeled, Attempt to clean up replies in Dada Bridge.



Tabbed Interface for PlainText/HTML Mailing List Messages

Dada Mail now allows you to toggle between the HTML, and PlainText versions of your mailing list message with a tabbed interface, instead of showing/hiding each type.



Subscription/Unsubscription Admin notices may now go to the entire mailing list

It's now an option to send the Subscription and Unsubscription Notices that usually go to the List Owner, to your entire list. This could be useful for a discussion list.

This feature was commissioned by David Smith for the Adytum Builders discussion list http://adytumbuilders.net/



Bugfixes 5.2.0

First Archive message shows incorrect link

https://github.com/justingit/dada-mail/issues/303



Lack of "message_body" tag in Mailing List Message Template leads to completely blank mailing list message

https://github.com/justingit/dada-mail/issues/304



Sending Preferences: Amazon SES has strange, blank blue box with nothing in it.

https://github.com/justingit/dada-mail/issues/305



Dada Bridge does not run the mailing monitor, when done checking messages

https://github.com/justingit/dada-mail/issues/306



Impossible to delete invalid-in-form email address in "Membership >> View" screen

https://github.com/justingit/dada-mail/issues/307

Changes 5.2.0

WYSIWYG Editors


Dada Bridge: Mailing List Message Template Tags for Discussion Lists Will be, "de-Personalized"

By default, the Mailing List Messages that ship with Dada Mail have template tags and links that point to various resources for subscribers of a mailing list. This includes changing the status of a subscription and logging into a subscriber's Profile. These links are personalized for the subscriber, so to avoid any additional entering of information/clicks.

These links cause havoc, though, on a discussion list, where messages are replied to and the original message is often quoted, and copied with the reply, as well as forwarded to people, off the list. The once-personal links now can be clicked by a third party, initializing such things as unsubscriptions. You may edit your Mailing List Message Templates to removed this personalization, but it is another step to set up a successful mailing list.

Starting with v5.2.0 of Dada Mail, these links will be, "de-Personalized" automatically for discussion lists. For example, a link that goes goes to a specific addresses' profile (or a form to log into a profile), that has their email address already embedded within the link, to pass automatically to Dada Mail:

ttp://example.com/cgi-bin/dada/mail.cgi/profile_login/user/example.com

will have the embedded email address removed:

http://example.com/cgi-bin/dada/mail.cgi/profile_login/

Similarily, Subscription and Unsubscription links:

http://example.com/cgi-bin/dada/mail.cgi/u/listname/user/example.com

will be changed to,

http://example.com/cgi-bin/dada/mail.cgi/u/listname/

To avoid problems.



FCKeditor listed as "Deprecated"


FCKeditor has now been deprecated in Dada Mail for many reasons: Internet Explorer 9 does not work well with FCKeditor, the included File Browser in FCKeditor is very buggy and has unfixed security issues and CKeditor is its natural replacement. We do not suggest using FCKeditor, but it is bundled with Dada Mail currently for backwards compatibility. We do bundle CKEditor and Tiny MCE, either of which will work better the FCKeditor.



$FCKEDITOR_URL and $CKEDITOR_URL Config variables - removed


Both the $FCKEDITOR_URL and $CKEDITOR_URL variables will not function as before - use the new $WYSIWYG_OPTIONS variable to configure these editors. Having these variables in your .dada_config file will not cause Dada Mail to error, yet.



"By default, reveal: PlainText Version|HTML Version" option removed


The Send a Message/Webpage screens now have a tabbed interface for HTML/PlainText messages, so this option has been removed. In Beatitude, both versions will be shown, by default.

Monday, September 27, 2010

On making a Simple Web-based installer in Perl

Recently and after 10 years of avoiding the idea, I made a web-based installer for Dada Mail.

My original thought was, something like this isn't necessary, that the installation process had been twiddled down to changing, oh, 3 variables in a heavily documented config file and additional "power user" fancy things to do, if You Knew What You Were Doing. What could possibly be easier?

Oh, how wrong I was - and how I never knew how wrong I was, until I've gotten some feedback on the alternative of having an installer.

You can get a gist of how the installer works in this video:




Basically, you pop up a tar.gz distro and a helper script that just gets things all set, you run the helper script and fill in a few things in a form - most of which are pre-filled with Best Guesses. I am not breaking new ground, here.

BUT what I am doing is giving my users the patented PHP Xperience - and that's what they want:

You throw up some files, you visit some stuff via your browser, you set some params and you go zoom.

The problem with, "But, there's all these CPAN dependencies!!!" was solved a long time ago in this project by only using Pure Perl modules (gets limiting in some instances) and shipping the app with an already pre-filled app-specific perllib. This also is a huge headache and the shipped-perllib is always behind, but it's better than nothing. The cpan deps service has been heaven-sent for roll-your-own-manually folks. The, "But, there's Cool Things in CPAN to do Cool Things in your App!" wish is solved by just making these cools things optional. cPanel, which is what a lot of Very Cheap Web Hosts use as their platform has a web-based cpan installer, so it's even within a mortal's reach to use CPAN to do fun things.

Anyways: here is, after a month of having the installer, some reflections:

The support boards aren't devoid of installation questions, but they sure are a lot quieter.

There are users who are intimidated by the installer. I don't know how else I can make things easier and these users would also be intimidated by doing things manually (which is still an option). I used to be intimidated with oiling the chain on my bicycle, but it takes... I dunno, 5 minutes to learn what you need to do. I don't ever say, "Uh, Google-fu the base level how-to to, (for example) FTP a file", but I don't quite know what I need to say.

But, the other amount of users that were intimidated doing things manually, but can also figure out the installer is the majority. Which is great - solved the problem. Now, the questions are simply bizarre edge cases dealing with weird MySQL setups on questionable hosting accounts. Happy to help.

Since more people are having an easier time installing the program, I get the feeling that more people are installing the program, successfully, which is great. I don't have any hard facts for that one, sadly. There are a small amount of people that tell me they like the Hard Way of doing things, and Why would I ever change? You're ruining my flawed work flow!, and I don't really know what to tell those folks, except to try the new way, that there is a manual way and it's progress - babe. And also, the previous version is still available if you want Hell, again. I had shipped the installer as the only new feature of the current version, so there's nothing that was missing.

Another hunch I have is that my own services to install the program are dramatically down. I can certainly graph how many installs I do per day/month/whatever, but these numbers are affected by a number of things: the economy, other competing programs, if Burning Man was particularly amazing and how long it takes someone after Labor Day to decompress, etc. This is also fine for me, because I loathe having to install the program for people: it takes a lot of time. I would rather they do it, themselves.

When I do install the program? Guess what, I use the web-based installer too. It's that much better/faster/spiffier. I really wish I created the installer earlier, as the time it took to make the installer would have paid back sooner for the time it's saving me number. I would have like 10 years of that time back, instead of just a month, but whatchagonna do.

But I've now raised the bar on Easiness...es and have a new problem: people who found the installer easy to use, want to upgrade just as easy. 10 years of development of a program, especially when Year 0 I knew not of the ways of Perl and Year 10 I'm still very much an amateur with a Right Hemispherical controlled mind.... things are messy. The config file format sucks, for example. And I have ten years of versions people want to upgrade from.

Should be interesting.

Some notes on the design of the installer:

I basically decided to make the installer as separate from the main program as possible, meaning it has its own library files, it's own template files, it's own testing suite (coming soon, I promise!). It does not use a framework, but uses the modulino approach. Things are fairly tidy.

It's written in a procedural design, because installations and configurations have steps. See? I just listed two of them. I made wrappers around things that deal with system calls - copying/removing files/directories, for example. I was thinking there could/would/should be edge cases for different OS's, but I really haven't found any, but I guess it still gives breathing room for someone much smarter than me to move in and make a better version of whatever I've botched. I did start with just using back ticks to system calls and replace those with perlish alternatives.


So. I'm not sure what stops all of us from making similar web-based installers for our web-apps. I am coming from a field (Art) and a preference for visual things, so I never really gotten a handle on system-admin type of tasks. The trend, it seems, in software is less, Give me as many options to roll my own doo-dad as I can have" and more, "Just freakin' work. And On my phone. When my 2-y/o uses it." I'm a little embarrassed at how much of the code I have is simply crap boilerplate to look up boring things about someone's environment to install. It seems that can all be sweeped under the floor and get some shared code. But then again, there's the rub, huh? If its shared code it's on CPAN, and then, how do you get the CPAN module, without a major compromise? And someone will be brilliant and write it in Moose (which is also, sincerely, brilliant) and there goes the baby, with the bathwater.