I've just put out a new release of Dada Mail - v6.6.0. Its a web-based mailing list manager, written in Perl. You can check out/fork/push/pull the source code here. Some of its main goals is to be pretty easy to install by mere mortals, easy to work with, yet pretty darn powerful and flexible, to fill a ton of different needs - it's not really trying to be a replacement to the creaky Majordomo, or Mailman.
I'm heading up to my 14th year of developing the application. It's currently free to download from its own support site as well as from various installer services that have popped up in the last few years (Mojo Marketplace, Installatron, Softaculous, etc) Follow along with the project on Twitter, or hey - we have a mailing list too!
There's lots of things a like about working on it, but the application has become quite monolithic, and a little less flexible in its underlying code, than I'd like. In the next few months, I'm hoping to alleviate that, without falling into the problem of totally rewriting the app (it would be almost impossible to do so, with a one-dude team).
My initial idea is to move from basically no framework, to working with CGI::Application. That should help me gain some sort of framework, and a way to break the overgrowth of parts of this app, into more manageable pieces. It'll also hopefully give me a bridge to be able to have Dada Mail run as a CGI script (how people are used to running Dada Mail), as well as a PSGI app (a great alternative, that has a TON of benefits, one being SPEED!).
From there, we'll see how it works, but it may be time to move on to some more whiz-bang things that a framework like Mojolicious would provide. The bummer with trying to move to Mojolicious is that the Perl version required is rarely met on the shared hosting accounts that the majority of people who run Dada Mail are on. Saying, "hey, just install perlbrew!" is a far cry on what lots of people are capable of. They're used to a one-click install. Sigh.
Maybe cPanel, being a Perl-based shop, knows of a better way to have available a more up-to-date Perl version? A how-to, so not to fudge this up, would be super-helpful and somewhat break huge shackles from those that ship apps for this types of hosting situations. (maybe you know?)
Without further ado, here's the changelog from this last release:
Friday, September 6, 2013
Friday, November 9, 2012
Dada Mail v6.0.0 Alpha 1 Released
This is the 6th MAJOR release of this app I've made since December of 1999. I think that's pretty wild. Github tells me that, compared to the last stable version of Dada Mail (v5.2.1) there has been: 353 changed files
with 18,740 additions
and 16,697 deletions. Goodness.
One of the major focuses on this release has really nothing to do with Perl, but JavaScript - I managed, somehow to port all the Prototype/Scriptaculous code to jQuery. Sadly, the Prototype project seems to be dead. But gladly, the jQuery code that's now part of the app is pretty easy to read.
To those interested, here's a rundown of the differences between the v5 and v6, as well as the direct link to install,
The vast majority of Dada Mail's extra functionality through JavaScript is done in an unobtrusive way. A great example of new features allowed with this JS library move is the rewritten and redesigned Tracker plugin.
Upgrading is fairly straightforward, with no need for any Dada Mail 5 to Dada Mail 6 SQL upgrade scripts.
When using the Dada Mail Installer (which you should use, when installing or upgrading), you will definitely want to create a new
There is one new config variable,
(If you added additional configuration variables after an install has completed, you will still have to re-add these manually)
This can help save time during an upgrade and ensure that everything goes smoothly. You can also re-enable the installer, after a successful initial installation, to change your Dada Mail configuration - for example: add/remove a plugin.
dada/DADA/Template/templates
They are now located in:
dada/templates
It's hoped that this directory move will make it easier for people to find the
dada/templates/plugins
Each plugin has its own directory for its templates, eg: dada/templates/plugins/tracker, etc.
The Scheduled Mailings plugin still has program generated and inline HTML. It's a big mess.
This change should help performance of the app. These files are also not cached in Dada Mail's Screen Cache, as they're going to be served faster by the webserver simply as static files.
dada/static
When you install Dada Mail using the included Installer, this entire directory will be copied to the publicly accessable directory that you specify, so that they may be served directly by the webserver. JavaScript, images and cascading style sheets will be served from this directory.
In previous versions of Dada Mail, these files were served via Dada Mail, creating a new running instance of the app per request, which is slow, awkward and confusing.
Both the server path, and URL to this directory will be saved in your
This does create an additional thing that needs to be configured for Dada Mail to be successfully installed, but we feel this extra step is worth it, for performance and for future features of the app.
In v6, Dada Mail will look for these types of conversions, and convert them back, fixing the template tag, to be parsed correctly.
.dada_files/.tmp/_dada_cache
directory. The files, as well as the directory itself, may be safely removed at any time.
The new charts/graphs/maps are powered by the Google Visualization API. Data, in the JSON format is created for Dada Mail using the Perl CPAN module, JSON (http://search.cpan.org/~makamaka/JSON/). This module will need to be installed on your hosting platform for these charts/graphs/maps to be created.
Although not part of the standard Perl Library, the module proves to be so useful, that it's usually already installed on many hosting platforms (like LWP, CGI, etc).
The distro of Dada Mail is much larger - around 20 megs. compressed, rather than ~7.5 megs for v5 of Dada Mail. This additional size is mostly from the bundled city-level Geo IP database.
Both the city, and country-level Geo IP databases can be found in the,
dada/data
directory.
dada/data/bounce_handler_rules.pl
You may also copy the file, and place it in your,
.dada_files/.configs
directory, make your own changes/customizations, and the Bounce Handler will use your copy, instead of what ships with Dada Mail.
Unless rewritten, it will be removed in a future version of Dada Mail.
Cheers,
One of the major focuses on this release has really nothing to do with Perl, but JavaScript - I managed, somehow to port all the Prototype/Scriptaculous code to jQuery. Sadly, the Prototype project seems to be dead. But gladly, the jQuery code that's now part of the app is pretty easy to read.
To those interested, here's a rundown of the differences between the v5 and v6, as well as the direct link to install,
Dada Mail 5 to Dada Mail 6 Upgrade Guide
Focus
The focus of this major release of Dada Mail is to make room for growth of the app, by creating a better foundation for future features.JavaScript UI/UX Breathing/Growing Room
Dada Mail's UI/UX has been migrated to jQuery, from Prototype, where development has all but ceased.The vast majority of Dada Mail's extra functionality through JavaScript is done in an unobtrusive way. A great example of new features allowed with this JS library move is the rewritten and redesigned Tracker plugin.
Towards a new web framework
Dada Mail is written with the thinnest of veneer of a web framework, and the next major version of Dada Mail may be a migration to a more robust web framework such as CGI::Application, Catalyst, or Mojolicious. v6 of Dada Mail moves the various components that make up Dada Mail (Perl code, JavaScript code, templates, images, stylesheets) into consolidated bundles of files and directories that are more sensible than previous versions.Database Schema
There are no changes in any of the database schemas between Dada Mail 5 and Dada Mail 6.Upgrading is fairly straightforward, with no need for any Dada Mail 5 to Dada Mail 6 SQL upgrade scripts.
When using the Dada Mail Installer (which you should use, when installing or upgrading), you will definitely want to create a new
.dada_config file, when asked.There is one new config variable,
$SUPPORT_FILES that needs to be set up correctly. If you do not make a new .dada_config file, or install manually, without putting in this new variable, the upgrade will not work.Perl Version Requirements
This is currently no change in the Perl Version requirement of v5.8.1. This may change, before a stable release of v6 is released, most likely to v5.10.x of Perl. We're still working on that..dada_config file
The.dada_config file that's generated by the included installer has undergone a few changes.Removed variables: $FILES, $TEMPLATES, $TMP, $BACKUPS, $ARCHIVES, $LOGS
These variables were added to the.dada_config file, but are almost kept the same as the default, ala: $FILES = $DIR . '/.lists';
$TEMPLATES = $DIR . '/.templates';
$TMP = $DIR . '/.tmp';
$BACKUPS = $DIR . '/.backups';
$ARCHIVES = $DIR . '/.archives';
$LOGS = $DIR . '/.logs';
Instead of taking up extra space and line noise, these files are simply added to the dada/DADA/Config.pm file as the default. You may still add these variables in the .dada_config file to change their default, but it's not really recommended. The, $DIR variable has been added to the dada/DADA/Config.pm as a true config variable, so you do not want to have the first line read,my $DIR = '/home/account/.dada_files';Simply,
$DIR = '/home/account/.dada_files';Will be fine. Keeping,
my will probably break things.Added variable: $BACKEND_DB_TYPE; Removed variables: $SUBSCRIBER_DB_TYPE, $ARCHIVE_DB_TYPE, $SETTINGS_DB_TYPE, $SESSION_DB_TYPE, $BOUNCE_SCORECARD_DB_TYPE, $CLICKTHROUGH_DB_TYPE
For the SQL backend, all these variables are already set the same, ala: $SUBSCRIBER_DB_TYPE = 'SQL';
$ARCHIVE_DB_TYPE = 'SQL';
$SETTINGS_DB_TYPE = 'SQL';
$SESSION_DB_TYPE = 'SQL';
$BOUNCE_SCORECARD_DB_TYPE = 'SQL';
$CLICKTHROUGH_DB_TYPE = 'SQL';
For the Default backend, they're set like this: $SUBSCRIBER_DB_TYPE = 'PlainText';
$ARCHIVE_DB_TYPE = 'Db';
$SETTINGS_DB_TYPE = 'Db';
$SESSION_DB_TYPE = 'Db';
$BOUNCE_SCORECARD_DB_TYPE = 'Db';
$CLICKTHROUGH_DB_TYPE = 'Db';
To reduce line noise, $BACKEND_DB_TYPE is being introduced. If set to, SQL,
it'll set the other variables accordingly. Same with setting this variable to, Default.%SQL_PARAMS: removed table names
Remove to further lesson the line noise.Installer
Database Connection, Bounce Handler POP3 Connection Testers
The installer now has inline testers for both your Database Connection, and the Bounce Handler's POP3 Connection, to help you make sure these work, before committing to an installation.Installer can now grab information from the previous installation
When using the installer to upgrade a current Dada Mail, it'll able to grab some of the configuration information from your previous installation: Program URL, Support Files Directory, Dada Mail Root Password, Backend Options, Plugin/Extensions that were previously installed, and your WYSIWYG options.(If you added additional configuration variables after an install has completed, you will still have to re-add these manually)
This can help save time during an upgrade and ensure that everything goes smoothly. You can also re-enable the installer, after a successful initial installation, to change your Dada Mail configuration - for example: add/remove a plugin.
Template Changes
"templates" directory move
In v5, template files lived in the directory:dada/DADA/Template/templates
They are now located in:
dada/templates
It's hoped that this directory move will make it easier for people to find the
templates directory. This directory is still in the, dada directory, instead of, say, the, static directory since template files are themselves little, simply programs, that need to be interpreted, and aren't themselves actually static.Plugins: Templated out
There was a voluminous amount of inline HTML in the plugins/extensions that are shipped with Dada Mail. This HTML is now saved in separate template files, which you can find in,dada/templates/plugins
Each plugin has its own directory for its templates, eg: dada/templates/plugins/tracker, etc.
The Scheduled Mailings plugin still has program generated and inline HTML. It's a big mess.
JavaScript, images and css files removed from templates directory, relocated in, "static" directory
Many static files that used to reside in the, templates directory have been moved to the, static directory, to be served directly from the webserver. These include JavaScript, image and cascading stylesheet files.This change should help performance of the app. These files are also not cached in Dada Mail's Screen Cache, as they're going to be served faster by the webserver simply as static files.
Inline JavaScript removed from template files
The vast majority of JavaScript that may have been found inline in the template files has been removed. You will now find that code in the, static/javascripts/dada_mail.js file.JavaScript
Migration from Prototype/Scriptaculous to jQuery/jQueryUI
Most of the JavaScript that relies on the Prototype/Scriptaculous libraries has been migrated to jQuery/jQueryUI. Some remains, and jQuery has been configured in,NoConflict mode. We hope to move all Prototype/Scriptaculous code to jQuery eventually.Static Files directory location, $SUPPORT_FILES config variable, Installation and Configuration
Dada Mail now separates out its static files needed for the app, from the dynamic template files. These static files ship with the app under the directory,dada/static
When you install Dada Mail using the included Installer, this entire directory will be copied to the publicly accessable directory that you specify, so that they may be served directly by the webserver. JavaScript, images and cascading style sheets will be served from this directory.
In previous versions of Dada Mail, these files were served via Dada Mail, creating a new running instance of the app per request, which is slow, awkward and confusing.
Both the server path, and URL to this directory will be saved in your
.dada_config file, under the varible, $SUPPORT_FILES. This variable is required to be correctly configured, for Dada Mail to work.This does create an additional thing that needs to be configured for Dada Mail to be successfully installed, but we feel this extra step is worth it, for performance and for future features of the app.
Clickthrough/Message Opens, etc tracking support dropped for Default Backend
The Tracker plugin and all of its functionality now requires the use of one of the SQL backends.List Control Panel Screens
Mass Mailing >> Send a Message
Options for Mass Mailing, including File Attachments, Archive Options, etc have been moved to a tabbed interface.Membership >> View
Breakdown by Domain Graph
The Breakdown by Domain graph, that used to be in the Tracker plugin has been moved to the Membership - View screen. Clicking on any of the pie slices will perform a search on the domain you have selected.Search Autocomplete
The Subscriber search form has autocomplete capabilities, which searches and matches on email addressesMembership >> Recent Activity
Subscription Trends graph
The Recent Activity screen now has a graph labeled, Subscription Trends. This graph shows daily, as well as cumulative data on Subscriptions and Unsubscriptions on your mailing list.WYSIWYG editors and Template Tags
In version < 6 of Dada Mail, when template tags like,<!-- tmpl_var list_settings.list_name -->are entered in the rich text (rather than source view) of the WYSIWYG editor, the "<", and, ">" characters are converted into their HTML entities, so the source of your message now looks like this:
<!-- tmpl_var list_settings.list_name -->Which breaks the tag, and you will see the original template tag, which will not be converted to its value.
In v6, Dada Mail will look for these types of conversions, and convert them back, fixing the template tag, to be parsed correctly.
Data Cache
Dada Mail now has a data cache, for things like generated JSON files. This data cache can be managed with the new,DADA::App::DataCache module and can be enabled/disabled using the $DADA_CACHE config variable. Old cached data will be removed periodically (1 hour), automatically. The data cache is located in the,.dada_files/.tmp/_dada_cache
directory. The files, as well as the directory itself, may be safely removed at any time.
Plugins
Tracker
The Tracker plugin has been essentially rewritten, and its features have been expanded. All charts, graphs and maps generated are now interactive and mousing over various parts of the chart/graph/map will reveal additional information. The UI itself has been redesigned, as well.The new charts/graphs/maps are powered by the Google Visualization API. Data, in the JSON format is created for Dada Mail using the Perl CPAN module, JSON (http://search.cpan.org/~makamaka/JSON/). This module will need to be installed on your hosting platform for these charts/graphs/maps to be created.
Although not part of the standard Perl Library, the module proves to be so useful, that it's usually already installed on many hosting platforms (like LWP, CGI, etc).
Expanded location-based information
As well as country-specific information, Tracker now supports showing information on a city-specific level.City Reports by IP Address
Reports are now generated per country, breaking down then by city, and then by individual IP address that may have caused an event, (a message open, clickthrough, forward, archive view) and listed in chronological order.Included City Geo IP Data
Dada Mail now comes with both a country Geo IP database, as well as a city-level Geo IP database. The city-level database is fairly large - ~20 megs., but it's utility is worth its size. Geo IP data is provided by http://maxmind.com, who also provide more accurate country/city data, on a subscription.The distro of Dada Mail is much larger - around 20 megs. compressed, rather than ~7.5 megs for v5 of Dada Mail. This additional size is mostly from the bundled city-level Geo IP database.
Both the city, and country-level Geo IP databases can be found in the,
dada/data
directory.
Bounce Handler
Filename change: dada_bounce_handler.pl to bounce_handler.cgi
Make sure to update any cronjobs, links, etc.Rules
The Bounce Handler Rules, used to match the type of bounced message that gets sent to the bounce handler, has been moved out of the code and into its own file, which you may find at,dada/data/bounce_handler_rules.pl
You may also copy the file, and place it in your,
.dada_files/.configs
directory, make your own changes/customizations, and the Bounce Handler will use your copy, instead of what ships with Dada Mail.
Bridge
Filename change: dada_bridge.pl to bridge.cgi
Make sure to update any cronjobs, links, etc.List Email Mail Forwarding Support
You may now set up a List Email as a Mail Forward, that pipes a message directory to Dada Bridge, rather than as a POP3 email account, that is checked on a schedule via a cronjob. This should greatly help any lag between when the message is sent to the List Email address, and when the message is checked by Bridge, and sent out to the entire mailing list.Clean Up Replies Filter - Disabled
This experimental feature doesn't work very well and has been disabled in v6.Scheduled Mailings - DEPRECATED
The current scheduled_mailings.pl plugin, as it currently stands, needs a rewrite and isn't currently benefitting from the advances of code style, best practices and techniques offered by the rest of Dada Mail.Unless rewritten, it will be removed in a future version of Dada Mail.
Extensions
dada_digest.pl - REMOVED
Cheers,
Monday, July 30, 2012
Dada Mail v5.2.0 Released
Hello Everyone,
Dada Mail v5.2.0 has been released - install/upgrade/download it from:
http://dadamailproject.com/support/documentation-5_2_0/install_dada_mail.pod.html
The changelog is below:
Summary 5.2.0
Dada Mail v5.2.0 release focuses on WYSIWYG editors, which allow you to author your HTML email messages in rich text, right from within the List Control Panel.
Dada Mail now comes bundled with three different WYSIWYG editors: CKEditor, Tiny MCE and FCKeditor. KCfinder, an image/file browser/uploader is also bundled. Dragging and Dropping images to be used in your email messages is also supported in Dada Mail. All these utilities can be installed easily using the included Dada Mail Installer.
We've streamlined the Send a Message and Send a Webpage screens to better utilize and more cleanly present these editors, while still allowing you full flexibility on how you author your messages: We've moved to a tabbed interface for the mailing list message, so that you can easily toggle between your HTML Version and your PlainText version, without unnecessary scrolling.
If upgrading from any previous version of Dada Mail, we highly suggest Creating a new .dada_config file, when using the included Dada Mail installer, as many items in the starter .dada_config file have now changed.
Features 5.2.0
WYSIWYG/File Browsers
Dada Mail now supports and comes bundled with the following WYSIWYG editors:
- CKEditor
- Tiny MCE
- FCKEditor
Dada Mail also comes bundled with the file browser/file upload utility called, KCFinder. This utility allows file file browsing and file (image) uploading for all three included WYSIWYG editor.
Dada Mail Installer Support
The Dada Mail Installer now has the ability to configure and install all three WYSIWYG editors, as well as the file browser. Installing all these utilities is optional, but much easier than doing it manually. The editors and browser are bundled with Dada Mail in the, dada/extras/packages directory.
Added Support for Tiny MCE
Along with Support for CKeditor and FCKeditor, Dada Mail now supports the Tiny MCE WYSIWYG HTML editor in its Send a Message and Send a Webpage screens.
Support for Drag and Drop images (in some browsers)
Instead of having to go through the File Browser to upload an image, you can simply drag an image into one of the WYSIWYG editors. The image will then be saved in your file browser's file uploads directory and will be able to be used again for a future mass mailing.
This doesn't work for all browsers, but is confirmed to be support in Firefox 14+.
Discussion Lists: Experimental Support for Cleaning up long quoted replies of mailing list openings/signatures
Look for the option labeled, Attempt to clean up replies in Dada Bridge.
Tabbed Interface for PlainText/HTML Mailing List Messages
Dada Mail now allows you to toggle between the HTML, and PlainText versions of your mailing list message with a tabbed interface, instead of showing/hiding each type.
Subscription/Unsubscription Admin notices may now go to the entire mailing list
It's now an option to send the Subscription and Unsubscription Notices that usually go to the List Owner, to your entire list. This could be useful for a discussion list.
This feature was commissioned by David Smith for the Adytum Builders discussion list http://adytumbuilders.net/
Bugfixes 5.2.0
First Archive message shows incorrect linkhttps://github.com/justingit/dada-mail/issues/303
Lack of "message_body" tag in Mailing List Message Template leads to completely blank mailing list message
https://github.com/justingit/dada-mail/issues/304
Sending Preferences: Amazon SES has strange, blank blue box with nothing in it.
https://github.com/justingit/dada-mail/issues/305
Dada Bridge does not run the mailing monitor, when done checking messages
https://github.com/justingit/dada-mail/issues/306
Impossible to delete invalid-in-form email address in "Membership >> View" screen
https://github.com/justingit/dada-mail/issues/307
Changes 5.2.0
WYSIWYG Editors
Dada Bridge: Mailing List Message Template Tags for Discussion Lists Will be, "de-Personalized"
By default, the Mailing List Messages that ship with Dada Mail have template tags and links that point to various resources for subscribers of a mailing list. This includes changing the status of a subscription and logging into a subscriber's Profile. These links are personalized for the subscriber, so to avoid any additional entering of information/clicks.
These links cause havoc, though, on a discussion list, where messages are replied to and the original message is often quoted, and copied with the reply, as well as forwarded to people, off the list. The once-personal links now can be clicked by a third party, initializing such things as unsubscriptions. You may edit your Mailing List Message Templates to removed this personalization, but it is another step to set up a successful mailing list.
Starting with v5.2.0 of Dada Mail, these links will be, "de-Personalized" automatically for discussion lists. For example, a link that goes goes to a specific addresses' profile (or a form to log into a profile), that has their email address already embedded within the link, to pass automatically to Dada Mail:
ttp://example.com/cgi-bin/dada/mail.cgi/profile_login/user/example.com
will have the embedded email address removed:
http://example.com/cgi-bin/dada/mail.cgi/profile_login/
Similarily, Subscription and Unsubscription links:
http://example.com/cgi-bin/dada/mail.cgi/u/listname/user/example.com
will be changed to,
http://example.com/cgi-bin/dada/mail.cgi/u/listname/
To avoid problems.
FCKeditor listed as "Deprecated"
FCKeditor has now been deprecated in Dada Mail for many reasons: Internet Explorer 9 does not work well with FCKeditor, the included File Browser in FCKeditor is very buggy and has unfixed security issues and CKeditor is its natural replacement. We do not suggest using FCKeditor, but it is bundled with Dada Mail currently for backwards compatibility. We do bundle CKEditor and Tiny MCE, either of which will work better the FCKeditor.
$FCKEDITOR_URL and $CKEDITOR_URL Config variables - removed
Both the $FCKEDITOR_URL and $CKEDITOR_URL variables will not function as before - use the new $WYSIWYG_OPTIONS variable to configure these editors. Having these variables in your .dada_config file will not cause Dada Mail to error, yet.
"By default, reveal: PlainText Version|HTML Version" option removed
The Send a Message/Webpage screens now have a tabbed interface for HTML/PlainText messages, so this option has been removed. In Beatitude, both versions will be shown, by default.
Monday, September 27, 2010
On making a Simple Web-based installer in Perl
Recently and after 10 years of avoiding the idea, I made a web-based installer for Dada Mail.
My original thought was, something like this isn't necessary, that the installation process had been twiddled down to changing, oh, 3 variables in a heavily documented config file and additional "power user" fancy things to do, if You Knew What You Were Doing. What could possibly be easier?
Oh, how wrong I was - and how I never knew how wrong I was, until I've gotten some feedback on the alternative of having an installer.
You can get a gist of how the installer works in this video:
Basically, you pop up a tar.gz distro and a helper script that just gets things all set, you run the helper script and fill in a few things in a form - most of which are pre-filled with Best Guesses. I am not breaking new ground, here.
BUT what I am doing is giving my users the patented PHP Xperience - and that's what they want:
You throw up some files, you visit some stuff via your browser, you set some params and you go zoom.
The problem with, "But, there's all these CPAN dependencies!!!" was solved a long time ago in this project by only using Pure Perl modules (gets limiting in some instances) and shipping the app with an already pre-filled app-specific perllib. This also is a huge headache and the shipped-perllib is always behind, but it's better than nothing. The cpan deps service has been heaven-sent for roll-your-own-manually folks. The, "But, there's Cool Things in CPAN to do Cool Things in your App!" wish is solved by just making these cools things optional. cPanel, which is what a lot of Very Cheap Web Hosts use as their platform has a web-based cpan installer, so it's even within a mortal's reach to use CPAN to do fun things.
Anyways: here is, after a month of having the installer, some reflections:
The support boards aren't devoid of installation questions, but they sure are a lot quieter.
There are users who are intimidated by the installer. I don't know how else I can make things easier and these users would also be intimidated by doing things manually (which is still an option). I used to be intimidated with oiling the chain on my bicycle, but it takes... I dunno, 5 minutes to learn what you need to do. I don't ever say, "Uh, Google-fu the base level how-to to, (for example) FTP a file", but I don't quite know what I need to say.
But, the other amount of users that were intimidated doing things manually, but can also figure out the installer is the majority. Which is great - solved the problem. Now, the questions are simply bizarre edge cases dealing with weird MySQL setups on questionable hosting accounts. Happy to help.
Since more people are having an easier time installing the program, I get the feeling that more people are installing the program, successfully, which is great. I don't have any hard facts for that one, sadly. There are a small amount of people that tell me they like the Hard Way of doing things, and Why would I ever change? You're ruining my flawed work flow!, and I don't really know what to tell those folks, except to try the new way, that there is a manual way and it's progress - babe. And also, the previous version is still available if you want Hell, again. I had shipped the installer as the only new feature of the current version, so there's nothing that was missing.
Another hunch I have is that my own services to install the program are dramatically down. I can certainly graph how many installs I do per day/month/whatever, but these numbers are affected by a number of things: the economy, other competing programs, if Burning Man was particularly amazing and how long it takes someone after Labor Day to decompress, etc. This is also fine for me, because I loathe having to install the program for people: it takes a lot of time. I would rather they do it, themselves.
When I do install the program? Guess what, I use the web-based installer too. It's that much better/faster/spiffier. I really wish I created the installer earlier, as the time it took to make the installer would have paid back sooner for the time it's saving me number. I would have like 10 years of that time back, instead of just a month, but whatchagonna do.
But I've now raised the bar on Easiness...es and have a new problem: people who found the installer easy to use, want to upgrade just as easy. 10 years of development of a program, especially when Year 0 I knew not of the ways of Perl and Year 10 I'm still very much an amateur with a Right Hemispherical controlled mind.... things are messy. The config file format sucks, for example. And I have ten years of versions people want to upgrade from.
Should be interesting.
Some notes on the design of the installer:
I basically decided to make the installer as separate from the main program as possible, meaning it has its own library files, it's own template files, it's owntesting suite (coming soon, I promise!). It does not use a framework, but uses the modulino approach. Things are fairly tidy.
It's written in a procedural design, because installations and configurations have steps. See? I just listed two of them. I made wrappers around things that deal with system calls - copying/removing files/directories, for example. I was thinking there could/would/should be edge cases for different OS's, but I really haven't found any, but I guess it still gives breathing room for someone much smarter than me to move in and make a better version of whatever I've botched. I did start with just using back ticks to system calls and replace those with perlish alternatives.
So. I'm not sure what stops all of us from making similar web-based installers for our web-apps. I am coming from a field (Art) and a preference for visual things, so I never really gotten a handle on system-admin type of tasks. The trend, it seems, in software is less, Give me as many options to roll my own doo-dad as I can have" and more, "Just freakin' work. And On my phone. When my 2-y/o uses it." I'm a little embarrassed at how much of the code I have is simply crap boilerplate to look up boring things about someone's environment to install. It seems that can all be sweeped under the floor and get some shared code. But then again, there's the rub, huh? If its shared code it's on CPAN, and then, how do you get the CPAN module, without a major compromise? And someone will be brilliant and write it in Moose (which is also, sincerely, brilliant) and there goes the baby, with the bathwater.
My original thought was, something like this isn't necessary, that the installation process had been twiddled down to changing, oh, 3 variables in a heavily documented config file and additional "power user" fancy things to do, if You Knew What You Were Doing. What could possibly be easier?
Oh, how wrong I was - and how I never knew how wrong I was, until I've gotten some feedback on the alternative of having an installer.
You can get a gist of how the installer works in this video:
Basically, you pop up a tar.gz distro and a helper script that just gets things all set, you run the helper script and fill in a few things in a form - most of which are pre-filled with Best Guesses. I am not breaking new ground, here.
BUT what I am doing is giving my users the patented PHP Xperience - and that's what they want:
You throw up some files, you visit some stuff via your browser, you set some params and you go zoom.
The problem with, "But, there's all these CPAN dependencies!!!" was solved a long time ago in this project by only using Pure Perl modules (gets limiting in some instances) and shipping the app with an already pre-filled app-specific perllib. This also is a huge headache and the shipped-perllib is always behind, but it's better than nothing. The cpan deps service has been heaven-sent for roll-your-own-manually folks. The, "But, there's Cool Things in CPAN to do Cool Things in your App!" wish is solved by just making these cools things optional. cPanel, which is what a lot of Very Cheap Web Hosts use as their platform has a web-based cpan installer, so it's even within a mortal's reach to use CPAN to do fun things.
Anyways: here is, after a month of having the installer, some reflections:
The support boards aren't devoid of installation questions, but they sure are a lot quieter.
There are users who are intimidated by the installer. I don't know how else I can make things easier and these users would also be intimidated by doing things manually (which is still an option). I used to be intimidated with oiling the chain on my bicycle, but it takes... I dunno, 5 minutes to learn what you need to do. I don't ever say, "Uh, Google-fu the base level how-to to, (for example) FTP a file", but I don't quite know what I need to say.
But, the other amount of users that were intimidated doing things manually, but can also figure out the installer is the majority. Which is great - solved the problem. Now, the questions are simply bizarre edge cases dealing with weird MySQL setups on questionable hosting accounts. Happy to help.
Since more people are having an easier time installing the program, I get the feeling that more people are installing the program, successfully, which is great. I don't have any hard facts for that one, sadly. There are a small amount of people that tell me they like the Hard Way of doing things, and Why would I ever change? You're ruining my flawed work flow!, and I don't really know what to tell those folks, except to try the new way, that there is a manual way and it's progress - babe. And also, the previous version is still available if you want Hell, again. I had shipped the installer as the only new feature of the current version, so there's nothing that was missing.
Another hunch I have is that my own services to install the program are dramatically down. I can certainly graph how many installs I do per day/month/whatever, but these numbers are affected by a number of things: the economy, other competing programs, if Burning Man was particularly amazing and how long it takes someone after Labor Day to decompress, etc. This is also fine for me, because I loathe having to install the program for people: it takes a lot of time. I would rather they do it, themselves.
When I do install the program? Guess what, I use the web-based installer too. It's that much better/faster/spiffier. I really wish I created the installer earlier, as the time it took to make the installer would have paid back sooner for the time it's saving me number. I would have like 10 years of that time back, instead of just a month, but whatchagonna do.
But I've now raised the bar on Easiness...es and have a new problem: people who found the installer easy to use, want to upgrade just as easy. 10 years of development of a program, especially when Year 0 I knew not of the ways of Perl and Year 10 I'm still very much an amateur with a Right Hemispherical controlled mind.... things are messy. The config file format sucks, for example. And I have ten years of versions people want to upgrade from.
Should be interesting.
Some notes on the design of the installer:
I basically decided to make the installer as separate from the main program as possible, meaning it has its own library files, it's own template files, it's own
It's written in a procedural design, because installations and configurations have steps. See? I just listed two of them. I made wrappers around things that deal with system calls - copying/removing files/directories, for example. I was thinking there could/would/should be edge cases for different OS's, but I really haven't found any, but I guess it still gives breathing room for someone much smarter than me to move in and make a better version of whatever I've botched. I did start with just using back ticks to system calls and replace those with perlish alternatives.
So. I'm not sure what stops all of us from making similar web-based installers for our web-apps. I am coming from a field (Art) and a preference for visual things, so I never really gotten a handle on system-admin type of tasks. The trend, it seems, in software is less, Give me as many options to roll my own doo-dad as I can have" and more, "Just freakin' work. And On my phone. When my 2-y/o uses it." I'm a little embarrassed at how much of the code I have is simply crap boilerplate to look up boring things about someone's environment to install. It seems that can all be sweeped under the floor and get some shared code. But then again, there's the rub, huh? If its shared code it's on CPAN, and then, how do you get the CPAN module, without a major compromise? And someone will be brilliant and write it in Moose (which is also, sincerely, brilliant) and there goes the baby, with the bathwater.
Sunday, March 7, 2010
Dada Mail Four Point Zero Point Three Released. Super Duper UTF-8/Unicode Support
Download:
What's New
We've been working really really hard to get the UTF-8/Unicode support working well in Dada Mail and this release, we really - no foolin' this time, think we've nailed it.
If you require character set support that's a little more than what's usually found in Latin-based languages, well, this is the release for you. We're going to build localization/internationalization support into Dada Mail, starting with this release. That means, we're going to start translating Dada Mail into multiple languages.
This release also has some pleasant bug-fixes. We couldn't have done it without your feedback.
Pro Dada Four - Ever: $44
There's only one more week to take advantage of this deal. After that? Gone.
Purchase Pro Dada for a special price of Forty-Four Dollars and your subscription to download Pro Dada and the Dada Mail Manual lasts Forever.
Pro Dada subscriptions are usually for a year. This offer extends your subscription for the entire life of the Dada Mail Project.
Purchase at:
http://dadamailproject.com/purchase/pro.html
This irrational offer lasts until pi day (3/14/2010)
Pro Dada Four Installed - $88
Get Pro Dada installed by us or get any current Dada Mail upgraded to Pro Dada Mail Four, for $88 (regularly $100). We'll keep upgrading it at your request for a year, but you'll have access to the Pro Dada Download and Dada Mail Manual for the life of the Dada Mail project.
Request an Install or Upgrade:
http://dadamailproject.com/installation/request.html
This irrational offer also only lasts until pi day (3/14/2010) - you have week left to submit that installation request.
Dada Mail Turns Ten in 2010
The Dada Mail Project started ten years ago in December of 1999 as a small curiosity and has gradually evolved and developed into an extremely popular programming and conceptual art project. Happy Birthday, Dada Mail.
Dada Mail Four is our latest release. Thanks for everyone's thoughtful feedback in this year-long development effort.
We couldn't have done it without you.
We're looking forward to receiving your feedback on Dada Mail Four.
Good luck!
Justin J
Lead Dadaist
http://dadamailproject.com
Dada Mail Change Log for version 4.0.3
Unicode/UTF-8 Work
We have worked very, very hard to get Dada Mail working with UTF-8/Unicode.
We think we did a pretty good job and you'll have a most amazing experience when comparing this version to any previous version of Dada Mail (ever), but there may be tiny things still to work out.
We need to know about them, don't be shy!
SQL table schema changes!
People who upgrade to 4.0.3 (and any version afterwards, until things change!) should note that the MySQL and PostgreSQL Table Schemas have changed!
You may need to update your own tables, to support UTF-8 (if they aren't already in that encoding).
See Also:
If you're upgrading, please read over the updated UTF-8/Unicode FAQ:
http://dadamailproject.com/support/documentation/features-UTF-8.pod.html
If you're doing a new install, there's nothing you need to know, Dada Mail should work well out of the box in re: to UTF-8/Unicode stuff.
Changes to Default List Settings
We've changed a few of the default list settings, hopefully so that everyone has a more pleasant experience, right off the bat:
Activate Black List
We've enabled the setting to active the Black List, by default.
We're also enabling the below settings:
- Move Unsubscribed Subscribers Automatically to the Black List
- Continue to Allow Subscriptions From Subscribers of Black Listed Addresses
You still have the option to change new lists to the previous behavior and already created lists will have their previous behavior, if Black List Settings have already been edited.
Print List-Specific Headers option Removed
The option, Print List-Specific Headers has been removed from, Mail Sending -Advanced Sending Preferences has been removed, but the functionality has not. All mailing list messages will have these headers.
Send Unsubscription Confirmation Emails (Closed-Loop Opt-Out) - disabled by default
Send Unsubscription Confirmation Emails (Closed-Loop Opt-Out) has been disabled by default (you can still enable it)
This option, when enabled, requires that when someone wants to unsubscribe, they have to confirm this unsubscription by clicking on the unsubscription confirmation link in a URL sent their subscribed address.
When disabled (the new default), they simply have to fill out the subscribe/unsubscribe form.
Subscription and Unsubscription links now include an Email Address
When available, both the Subscription and Unsubscription links will have the potential subscriber's (or unsubscriber's) email address in the link itself, so that the user does not have to do the two-step of first following the link and then typing in their email address.
These links are created per-subscriber (or potential sub/unsub), when you use the:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/s/dada_announce/example/example.com/
or,
http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dada_announce/example/example.com/
tags. Previously, these tags only provided a link to the subscription/unsubscription form, without the email address embedded within the link itself. There is no way to revert this behaviour, but you can still roll your own links, like this:
Subscription Link:
/s/
Unsubscription Link:
/u/
Unsubscription Links Now Mandatory for Mass Mailing Messages Dada Mail will now do a quick check to make sure that there is a Dada Mail Unsubscription link in your mass mailing messages, before sending them out.
If one is not found, one will be automatically appended to the end of your message.
It will not be very fancy.
We suggest that you make sure that you have a real, valid, Dada Mail unsubscription link in your Mailing List Messages.
Bug Fixes 4.0.3
- Send newest archived message may have outdated header information
http://github.com/justingit/dada-mail/issues/issue/30
- pop3 username/password not saved when "Save, Then Test..." button pressed in Sending Preferences
http://github.com/justingit/dada-mail/issues/issue/29
- Beatitude: Months are listed out of order
http://github.com/justingit/dada-mail/issues/issue/28
- profile field names can contain more than just ascii letters, numbers and underscores
http://github.com/justingit/dada-mail/issues/issue/27
- list short names can contain more than just ascii letters, numbers and underscores
http://github.com/justingit/dada-mail/issues/issue/26
- Beatitude: Scheduled List Not in Any Useable Order?
http://github.com/justingit/dada-mail/issues/issue/16
- Dada Bridge: Spam Assassin Level Picker isn't available
http://github.com/justingit/dada-mail/issues/issue/21
- Sending Preferences don't correctly state if you can use Use Secure Sockets Layer (SSL) for POP-before-SMTP
http://github.com/justingit/dada-mail/issues/issue/24
- Double Subscriptions when using List Invitation
http://github.com/justingit/dada-mail/issues/issue/23
- Archived messages not templated out in publicly displayed archives
http://github.com/justingit/dada-mail/issues/issue/20
*Link to edit subscriber information broken when using the search
http://github.com/justingit/dada-mail/issues/issue/19
- Unsubsciption Notice to List Owner doesn't have subscriber (profile) fields
http://github.com/justingit/dada-mail/issues/issue/18
- Disabled Menu items return server error when using the, "Classic" session type
Tuesday, February 23, 2010
Experimental Dada Mail w/unicode ¡Support! Released
(this is a repost from here, 'cause I'm pretty stoked on it)
This is the first step in the localization project, since we can't very well translate Dada Mail if Dada Mail can't use the translations available.
I have to let this project rest for a little bit (and collect my wits - it was a very difficult step!) but any and all feedback is welcome, if you'd like to give this a spin - bug reports/problems of any kind are very much appreciated.
This version of Dada Mail should basically be able to support any language that can in the unicode characters set and UTF-8 encoding. Which, should be, well, a lot of them. It doesn't (Dada Mail), but where it fails? I don't know - but it's a good time to test and see where it's wrong.
For simple Euro-centric stuff, like this:
Je peux manger du verre, ça ne me fait pas mal.
It should be fine. For something a little more wild:
أنا قادر على أكل الزجاج و هذا لا يؤلمني.
(which should be Arabic)
Well, I can only go on if something visually looks correct :) Even this email is sort of a test - I don't know if it's going to work, or not - so, fingers crossed! If it does - we're on a good track, since Dada Bridge taking a random email, having it go through the system that's mostly tested using a very specific way of creating emails and coming out readable on the other side is a great big step - not even talking about the online archive, rss/atom feeds, twitter thingie, etc, etc, etc.
Here's the download to the version I'm now running at the Dada Mail support site:
http://github.com/downloads/justingit/dada-mail/dada-4_0_2-unicode.zip
http://github.com/downloads/justingit/dada-mail/dada-4_0_2-unicode.tar.gz
If you want to check it out via github, the branch is at:
http://github.com/justingit/dada-mail/tree/charset_work
To grab it with git, you have to do this:
git clone git://github.com/justingit/dada-mail.git
cd dada-mail
git fetch
git checkout --track -b your_local_branch_name origin/charset_work
Here's the explanation of all that:
http://groups.google.com/group/github/browse_thread/thread/71f944b925467ab6
There's a guide of what to expect with Dada Mail and unicode/UTF-8 you can read here:
http://dadamailproject.com/support/documentation-4_0_2-unicode/features-UTF-8.pod.html
Which I'll paste the contents of at the end of this message - but you may also want to compare it to the version of this doc for 4.0.2 STABLE:
http://dadamailproject.com/support/documentation-4_0_2/features-UTF-8.pod.html
(Long story short: 4.0.2 UTF-8/Unicode Support: "uhh...")
And, that's about it. This was a hard part of the project, since this is a 10+ y/o codebase - it very much pre-dates even unicode/UTF-8 support in Perl itself, so there's a reason, I guess, why the program was in such bad shape when it came to support it. Many,
many many bugs showed themselves, once this feature was asked for. I think a great majority of them have been solved.
Give it a spin if this interests you and if I can help out with anything, let me know,
--
Introduction
Dada Mail can speak UTF-8 and almost expects that everything else around it does, too.
That means:
• It treats everything it handles as UTF-8
• Everything it returns is in UTF-8
How To Have a Pleasant Experience
If you're installing Dada Mail for the first time, there's nothing you'll need to do, but below are some great guidelines on how to keep your lists configured, so you continue to have a good experience.
If you're upgrading, make sure your configuration reflects the advice below.
It's heavily advised to keep everything in Dada Mail speaking UTF-8 without any real exceptions.
Config Variable: $HTML_CHARSET
By default, the config variable, $HTML_CHARSET is set to, UTF-8
Keep it that way, same case (UTF-8) - same everything.
Dada Mail is only tested with the charset set this way.
Advanced Sending Preferences
Default Character Set
Set this as, UTF-8 UTF-8
Default Plain Text/HTML Message Encoding
There's really only a few choices recommended for Dada Mail.
• 8bit
Should work.
• quoted-printable
If you have any trouble with 8bit, try quoted-printable. Because of the amount of time that Dada Mail creates, tweaks, formats and templates out email messages, the encoding can potentially get mucked up.
This potential mucking-up is mitigated when Dada Mail uses quoted-printable encoding internally. This should be the default for email messages.
Encode Message Headers
Have this option checked.
SQL Backends
Database
PostgreSQL
Encoding for PostgreSQL databases is done when the database is created - make sure to create your database with a, UTF-8 encoding, like so:
CREATE DATABASE dadamail WITH ENCODING 'UTF-8'
MySQL
Nothing you'll have to do.
SQLite
Nothing you'll have to do.
DBM Files
DBM Files have no encoding support, but Dada Mail knows this and compensates.
Schema
MySQL
The MySQL schemas are set to create tables with an encoding of, UTF-8
PostgreSQL
Nothing has changed.
SQLite
Nothing has changed.
Drivers
The current support SQL backends, mysql (MySQL), Pg (PostgreSQL) and SQLite all have different ways to somewhat, "enable" their UTF-8 support.
• MySQL
add,
mysql_enable_utf8 => 1,
has been added to the $DBI_PARAMS hashref.
• PostgreSQL
add,
pg_enable_utf8 => 1,
has been added to the $DBI_PARAMS hashref.
• SQLite
add,
sqlite_unicode => 1
has been added to the $DBI_PARAMS hashref.
No explicit encoding/decoding is done in Dada Mail when saving/retrieving data. Hopefully, the drivers are UTF-8-aware enough.
Plugins/Extensions
The Plugins and Extensions that come with Dada Mail have not been as thoroughly tested as the main program. There's still warts.
Dada Bridge
Dada Bridge has a unique position needing to handle a lot of different stuff thown at it and deal with it gracefully. Dada Mail does, in fact, handle, any realistic character set/encoding you throw at it, but Dada Mail will convert messages it receives to its internal format, before it resends it out to your list.
This means the encoding of your choice (8bit or quoted-printable) and the charset of your choice (as long as your charset is, UTF-8)
Upgrading
You are potentially going to have problems.
Its possible that, since List Settings were never decoded/encoded correctly in past versions, they'll show up the program (once you've upgrade) incorrectly. The easiest thing to do is to edit the mistakes and resave the information. For most of the program, you're going to have to manually export the information and re-import it with the correct encoding, sadly. Dada Mail will probably fail gracefully with old information, but it's possible that you'll see squiggly charaters, instead of what you want to see. There's nothing in Dada Mail that will stop this from happening. If you experience it (from old information), we're not going to count it as a bug, but rather a known issue.
Problems?
Please let us know via the Support Boards:
http://dadamailproject.com/support/boards/
Or the developer mailing list:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev/
Thanks!
See Also:
• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
http://www.joelonsoftware.com/articles/Unicode.html
• perlunitut - Perl Unicode Tutorial
http://perldoc.perl.org/perlunitut.html
• perlunifaq - Perl Unicode FAQ
http://perldoc.perl.org/perlunifaq.html
--
Post:
mailto:dadadev@dadamailproject.com
Unsubscribe:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/
List Information:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev
Archive:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev
Developer Info:
http://dev.dadamailproject.com
This is the first step in the localization project, since we can't very well translate Dada Mail if Dada Mail can't use the translations available.
I have to let this project rest for a little bit (and collect my wits - it was a very difficult step!) but any and all feedback is welcome, if you'd like to give this a spin - bug reports/problems of any kind are very much appreciated.
This version of Dada Mail should basically be able to support any language that can in the unicode characters set and UTF-8 encoding. Which, should be, well, a lot of them. It doesn't (Dada Mail), but where it fails? I don't know - but it's a good time to test and see where it's wrong.
For simple Euro-centric stuff, like this:
Je peux manger du verre, ça ne me fait pas mal.
It should be fine. For something a little more wild:
أنا قادر على أكل الزجاج و هذا لا يؤلمني.
(which should be Arabic)
Well, I can only go on if something visually looks correct :) Even this email is sort of a test - I don't know if it's going to work, or not - so, fingers crossed! If it does - we're on a good track, since Dada Bridge taking a random email, having it go through the system that's mostly tested using a very specific way of creating emails and coming out readable on the other side is a great big step - not even talking about the online archive, rss/atom feeds, twitter thingie, etc, etc, etc.
Here's the download to the version I'm now running at the Dada Mail support site:
http://github.com/downloads/justingit/dada-mail/dada-4_0_2-unicode.zip
http://github.com/downloads/justingit/dada-mail/dada-4_0_2-unicode.tar.gz
If you want to check it out via github, the branch is at:
http://github.com/justingit/dada-mail/tree/charset_work
To grab it with git, you have to do this:
git clone git://github.com/justingit/dada-mail.git
cd dada-mail
git fetch
git checkout --track -b your_local_branch_name origin/charset_work
Here's the explanation of all that:
http://groups.google.com/group/github/browse_thread/thread/71f944b925467ab6
There's a guide of what to expect with Dada Mail and unicode/UTF-8 you can read here:
http://dadamailproject.com/support/documentation-4_0_2-unicode/features-UTF-8.pod.html
Which I'll paste the contents of at the end of this message - but you may also want to compare it to the version of this doc for 4.0.2 STABLE:
http://dadamailproject.com/support/documentation-4_0_2/features-UTF-8.pod.html
(Long story short: 4.0.2 UTF-8/Unicode Support: "uhh...")
And, that's about it. This was a hard part of the project, since this is a 10+ y/o codebase - it very much pre-dates even unicode/UTF-8 support in Perl itself, so there's a reason, I guess, why the program was in such bad shape when it came to support it. Many,
many many bugs showed themselves, once this feature was asked for. I think a great majority of them have been solved.
Give it a spin if this interests you and if I can help out with anything, let me know,
--
Introduction
Dada Mail can speak UTF-8 and almost expects that everything else around it does, too.
That means:
• It treats everything it handles as UTF-8
• Everything it returns is in UTF-8
How To Have a Pleasant Experience
If you're installing Dada Mail for the first time, there's nothing you'll need to do, but below are some great guidelines on how to keep your lists configured, so you continue to have a good experience.
If you're upgrading, make sure your configuration reflects the advice below.
It's heavily advised to keep everything in Dada Mail speaking UTF-8 without any real exceptions.
Config Variable: $HTML_CHARSET
By default, the config variable, $HTML_CHARSET is set to, UTF-8
Keep it that way, same case (UTF-8) - same everything.
Dada Mail is only tested with the charset set this way.
Advanced Sending Preferences
Default Character Set
Set this as, UTF-8 UTF-8
Default Plain Text/HTML Message Encoding
There's really only a few choices recommended for Dada Mail.
• 8bit
Should work.
• quoted-printable
If you have any trouble with 8bit, try quoted-printable. Because of the amount of time that Dada Mail creates, tweaks, formats and templates out email messages, the encoding can potentially get mucked up.
This potential mucking-up is mitigated when Dada Mail uses quoted-printable encoding internally. This should be the default for email messages.
Encode Message Headers
Have this option checked.
SQL Backends
Database
PostgreSQL
Encoding for PostgreSQL databases is done when the database is created - make sure to create your database with a, UTF-8 encoding, like so:
CREATE DATABASE dadamail WITH ENCODING 'UTF-8'
MySQL
Nothing you'll have to do.
SQLite
Nothing you'll have to do.
DBM Files
DBM Files have no encoding support, but Dada Mail knows this and compensates.
Schema
MySQL
The MySQL schemas are set to create tables with an encoding of, UTF-8
PostgreSQL
Nothing has changed.
SQLite
Nothing has changed.
Drivers
The current support SQL backends, mysql (MySQL), Pg (PostgreSQL) and SQLite all have different ways to somewhat, "enable" their UTF-8 support.
• MySQL
add,
mysql_enable_utf8 => 1,
has been added to the $DBI_PARAMS hashref.
• PostgreSQL
add,
pg_enable_utf8 => 1,
has been added to the $DBI_PARAMS hashref.
• SQLite
add,
sqlite_unicode => 1
has been added to the $DBI_PARAMS hashref.
No explicit encoding/decoding is done in Dada Mail when saving/retrieving data. Hopefully, the drivers are UTF-8-aware enough.
Plugins/Extensions
The Plugins and Extensions that come with Dada Mail have not been as thoroughly tested as the main program. There's still warts.
Dada Bridge
Dada Bridge has a unique position needing to handle a lot of different stuff thown at it and deal with it gracefully. Dada Mail does, in fact, handle, any realistic character set/encoding you throw at it, but Dada Mail will convert messages it receives to its internal format, before it resends it out to your list.
This means the encoding of your choice (8bit or quoted-printable) and the charset of your choice (as long as your charset is, UTF-8)
Upgrading
You are potentially going to have problems.
Its possible that, since List Settings were never decoded/encoded correctly in past versions, they'll show up the program (once you've upgrade) incorrectly. The easiest thing to do is to edit the mistakes and resave the information. For most of the program, you're going to have to manually export the information and re-import it with the correct encoding, sadly. Dada Mail will probably fail gracefully with old information, but it's possible that you'll see squiggly charaters, instead of what you want to see. There's nothing in Dada Mail that will stop this from happening. If you experience it (from old information), we're not going to count it as a bug, but rather a known issue.
Problems?
Please let us know via the Support Boards:
http://dadamailproject.com/support/boards/
Or the developer mailing list:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev/
Thanks!
See Also:
• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
http://www.joelonsoftware.com/articles/Unicode.html
• perlunitut - Perl Unicode Tutorial
http://perldoc.perl.org/perlunitut.html
• perlunifaq - Perl Unicode FAQ
http://perldoc.perl.org/perlunifaq.html
--
Post:
mailto:dadadev@dadamailproject.com
Unsubscribe:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/u/dadadev/
List Information:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/list/dadadev
Archive:
http://dadamailproject.com/cgi-bin/dada/mail.cgi/archive/dadadev
Developer Info:
http://dev.dadamailproject.com
Saturday, February 20, 2010
Perl HTML::Template and UTF-8 Unicode
HTML::Template does not support file encoding:
prints, ¡™£¢∞§¶•ªº (or something like that!)
In the example above, this makes sense, since we're printing on an open filehandle (even if it's only to our magical, DATA) that we didn't put a file layer filter thingy to. That's easy to fix:
This also works if we want to just pass a reference to a scalar to HTML::Template:
This doesn't work, if we want to just give it a name of a template file. This is really useful, since HTML::Template has a feature to allow you to search through a file structure (or at least an array of directories, looking for the file).
And this is where encoding madness begins.
Cause I know what you're thinking, just treat HTML::Template's output like information that's coming from outside your program (since, if you're using a template *file*, it kinda is).
So, all you need to do is decode (this is the WRONG WAY to solve the problem, but let's just make that mistake...) the return value of ->output, like this:
prints, ¡™£¢∞§¶•ªº. Yes.
But... what if you have a variable (it is a templating system) and the variable in the param() you pass has UTF-8 strings? MUAHAHA!
Cannot decode string with wide characters at /System/Library/Perl/5.10.0/darwin-thread-multi-2level/Encode.pm line 162.
Bahahaha!
Take those decode/encode lines (I know it looks strange to one, right after the other ) and you'll still get a weird output:
Darned if you do/don't. Those two lines should have the same string. They don't. No amount of encoding/decoding is going to help.
The trick, other than tweaking HTML::Template's source to include file filter layer thingamabobs, is to decode the contents of the file it opens up.
How to do that.
Trolling through the HTML::Template mailing list archives leads to the idea of using a HTML::Template filter that matches everything, that then does our decoding:
This sort of lines up all the data to be UTF-8 encoded and aware and all that stuff that the unicodefaqthingy perldoc tells you to do.
But, oh, it gets better.
DON'T use that filter trick thing if you're using a scalarref, or a properly encoded file handle! You'll get a nice error, like this:
HTML::Template->new() : fatal error occured during filter call: Cannot decode string with wide characters at /System/Library/Perl/5.10.0/darwin-thread-multi-2level/Encode.pm line 162.
at /Library/Perl/5.10.0/HTML/Template.pm line 1697
HTML::Template::_init_template('HTML::Template=HASH(0x1008aafb8)') called at /Library/Perl/5.10.0/HTML/Template.pm line 1238
HTML::Template::_init('HTML::Template=HASH(0x1008aafb8)') called at /Library/Perl/5.10.0/HTML/Template.pm line 1124
Brilliant.
So I don't know what the best advice is to give. If you're passing the template as a scalarref, DON'T use that filter, unless you want to, perhaps encode your template beforehand (which makes little sense?)
If it's a filename, use that filter trick perhaps (or edit the sourcecode of HTML::Template).
#!/usr/bin/perl -w
use strict;
use Encode;
use HTML::Template;
my $template = HTML::Template->new(
filehandle => *DATA,
);
print Encode::encode('UTF-8', $template->output);
__DATA__
¡™£¢∞§¶•ªº
prints, ¡™£¢∞§¶•ªº (or something like that!)
In the example above, this makes sense, since we're printing on an open filehandle (even if it's only to our magical, DATA) that we didn't put a file layer filter thingy to. That's easy to fix:
prints, ¡™£¢∞§¶•ªº, yay!
#!/usr/bin/perl -w
use strict;
use Encode;
binmode DATA, ':encoding(UTF-8)';
use HTML::Template;
my $template = HTML::Template->new(
filehandle => *DATA,
);
print Encode::encode('UTF-8', $template->output);
__DATA__
¡™£¢∞§¶•ªº
This also works if we want to just pass a reference to a scalar to HTML::Template:
prints, ¡™£¢∞§¶•ªº, yay!
#!/usr/bin/perl -w
use strict;
use Encode;
my $content = "\x{a1}\x{2122}\x{a3}\x{a2}\x{221e}\x{a7}\x{b6}\x{2022}\x{aa}\x{ba}";
use HTML::Template;
my $template = HTML::Template->new(
scalarref => \$content,
);
print Encode::encode('UTF-8', $template->output);
This doesn't work, if we want to just give it a name of a template file. This is really useful, since HTML::Template has a feature to allow you to search through a file structure (or at least an array of directories, looking for the file).
And this is where encoding madness begins.
Cause I know what you're thinking, just treat HTML::Template's output like information that's coming from outside your program (since, if you're using a template *file*, it kinda is).
So, all you need to do is decode (this is the WRONG WAY to solve the problem, but let's just make that mistake...) the return value of ->output, like this:
#!/usr/bin/perl -w
use strict;
use Encode;
my $content = "\x{a1}\x{2122}\x{a3}\x{a2}\x{221e}\x{a7}\x{b6}\x{2022}\x{aa}\x{ba}";
my $filename = 'utf8string.tmpl';
open my $fh, '>:encoding(UTF-8)', $filename or die $!;
print $fh $content;
close $fh;
use HTML::Template;
my $template = HTML::Template->new(
filename => $filename,
);
my $output = $template->output;
$output = Encode::decode('UTF-8', $output);
print Encode::encode('UTF-8', $output);
prints, ¡™£¢∞§¶•ªº. Yes.
But... what if you have a variable (it is a templating system) and the variable in the param() you pass has UTF-8 strings? MUAHAHA!
#!/usr/bin/perl -w
use strict;
use Encode;
my $content = "
<!-- tmpl_var one -->
\x{a1}\x{2122}\x{a3}\x{a2}\x{221e}\x{a7}\x{b6}\x{2022}\x{aa}\x{ba}
";
my $filename = 'utf8string.tmpl';
open my $fh, '>:encoding(UTF-8)', $filename or die $!;
print $fh $content;
close $fh;
use HTML::Template;
my $template = HTML::Template->new(
filename => $filename,
);
$template->param(
one => "\x{a1}\x{2122}\x{a3}\x{a2}\x{221e}\x{a7}\x{b6}\x{2022}\x{aa}\x{ba}",
);
my $output = $template->output;
$output = Encode::decode('UTF-8', $output);
print Encode::encode('UTF-8', $output);
Cannot decode string with wide characters at /System/Library/Perl/5.10.0/darwin-thread-multi-2level/Encode.pm line 162.
Bahahaha!
Take those decode/encode lines (I know it looks strange to one, right after the other ) and you'll still get a weird output:
¡™£¢∞§¶•ªº
¡™£¢∞§¶•ªº
Darned if you do/don't. Those two lines should have the same string. They don't. No amount of encoding/decoding is going to help.
The trick, other than tweaking HTML::Template's source to include file filter layer thingamabobs, is to decode the contents of the file it opens up.
How to do that.
Trolling through the HTML::Template mailing list archives leads to the idea of using a HTML::Template filter that matches everything, that then does our decoding:
#!/usr/bin/perl -w
use strict;
use Encode;
my $content = "
<!-- tmpl_var one -->
\x{a1}\x{2122}\x{a3}\x{a2}\x{221e}\x{a7}\x{b6}\x{2022}\x{aa}\x{ba}
";
my $filename = 'utf8string.tmpl';
open my $fh, '>:encoding(UTF-8)', $filename or die $!;
print $fh $content;
close $fh;
use HTML::Template;
my $template = HTML::Template->new(
filename => $filename,
filter => [
{ sub => \&decode_str, format => 'scalar' },
],
);
$template->param(
one => "\x{a1}\x{2122}\x{a3}\x{a2}\x{221e}\x{a7}\x{b6}\x{2022}\x{aa}\x{ba}",
);
my $output = $template->output;
print Encode::encode('UTF-8', $output);
sub decode_str {
my $ref = shift;
${$ref} = Encode::decode('UTF-8', ${$ref});
}
This sort of lines up all the data to be UTF-8 encoded and aware and all that stuff that the unicodefaqthingy perldoc tells you to do.
But, oh, it gets better.
DON'T use that filter trick thing if you're using a scalarref, or a properly encoded file handle! You'll get a nice error, like this:
HTML::Template->new() : fatal error occured during filter call: Cannot decode string with wide characters at /System/Library/Perl/5.10.0/darwin-thread-multi-2level/Encode.pm line 162.
at /Library/Perl/5.10.0/HTML/Template.pm line 1697
HTML::Template::_init_template('HTML::Template=HASH(0x1008aafb8)') called at /Library/Perl/5.10.0/HTML/Template.pm line 1238
HTML::Template::_init('HTML::Template=HASH(0x1008aafb8)') called at /Library/Perl/5.10.0/HTML/Template.pm line 1124
Brilliant.
So I don't know what the best advice is to give. If you're passing the template as a scalarref, DON'T use that filter, unless you want to, perhaps encode your template beforehand (which makes little sense?)
If it's a filename, use that filter trick perhaps (or edit the sourcecode of HTML::Template).
Subscribe to:
Posts (Atom)