Tuesday, February 23, 2010

Experimental Dada Mail w/unicode ¡Support! Released

(this is a repost from here, 'cause I'm pretty stoked on it)

This is the first step in the localization project, since we can't very well translate Dada Mail if Dada Mail can't use the translations available.

I have to let this project rest for a little bit (and collect my wits - it was a very difficult step!) but any and all feedback is welcome, if you'd like to give this a spin - bug reports/problems of any kind are very much appreciated.

This version of Dada Mail should basically be able to support any language that can in the unicode characters set and UTF-8 encoding. Which, should be, well, a lot of them. It doesn't (Dada Mail), but where it fails? I don't know - but it's a good time to test and see where it's wrong.

For simple Euro-centric stuff, like this:

Je peux manger du verre, ça ne me fait pas mal.

It should be fine. For something a little more wild:

أنا قادر على أكل الزجاج و هذا لا يؤلمني.

(which should be Arabic)

Well, I can only go on if something visually looks correct :) Even this email is sort of a test - I don't know if it's going to work, or not - so, fingers crossed! If it does - we're on a good track, since Dada Bridge taking a random email, having it go through the system that's mostly tested using a very specific way of creating emails and coming out readable on the other side is a great big step - not even talking about the online archive, rss/atom feeds, twitter thingie, etc, etc, etc.

Here's the download to the version I'm now running at the Dada Mail support site:



If you want to check it out via github, the branch is at:


To grab it with git, you have to do this:

git clone git://github.com/justingit/dada-mail.git
cd dada-mail
git fetch
git checkout --track -b your_local_branch_name origin/charset_work

Here's the explanation of all that:


There's a guide of what to expect with Dada Mail and unicode/UTF-8 you can read here:


Which I'll paste the contents of at the end of this message - but you may also want to compare it to the version of this doc for 4.0.2 STABLE:


(Long story short: 4.0.2 UTF-8/Unicode Support: "uhh...")

And, that's about it. This was a hard part of the project, since this is a 10+ y/o codebase - it very much pre-dates even unicode/UTF-8 support in Perl itself, so there's a reason, I guess, why the program was in such bad shape when it came to support it. Many,

many many bugs showed themselves, once this feature was asked for. I think a great majority of them have been solved.

Give it a spin if this interests you and if I can help out with anything, let me know,


Dada Mail can speak UTF-8 and almost expects that everything else around it does, too.

That means:

• It treats everything it handles as UTF-8
• Everything it returns is in UTF-8
How To Have a Pleasant Experience

If you're installing Dada Mail for the first time, there's nothing you'll need to do, but below are some great guidelines on how to keep your lists configured, so you continue to have a good experience.

If you're upgrading, make sure your configuration reflects the advice below.

It's heavily advised to keep everything in Dada Mail speaking UTF-8 without any real exceptions.

Config Variable: $HTML_CHARSET

By default, the config variable, $HTML_CHARSET is set to, UTF-8

Keep it that way, same case (UTF-8) - same everything.

Dada Mail is only tested with the charset set this way.

Advanced Sending Preferences

Default Character Set

Set this as, UTF-8 UTF-8

Default Plain Text/HTML Message Encoding

There's really only a few choices recommended for Dada Mail.

• 8bit
Should work.

• quoted-printable

If you have any trouble with 8bit, try quoted-printable. Because of the amount of time that Dada Mail creates, tweaks, formats and templates out email messages, the encoding can potentially get mucked up.

This potential mucking-up is mitigated when Dada Mail uses quoted-printable encoding internally. This should be the default for email messages.

Encode Message Headers

Have this option checked.

SQL Backends



Encoding for PostgreSQL databases is done when the database is created - make sure to create your database with a, UTF-8 encoding, like so:


Nothing you'll have to do.


Nothing you'll have to do.

DBM Files

DBM Files have no encoding support, but Dada Mail knows this and compensates.



The MySQL schemas are set to create tables with an encoding of, UTF-8


Nothing has changed.


Nothing has changed.


The current support SQL backends, mysql (MySQL), Pg (PostgreSQL) and SQLite all have different ways to somewhat, "enable" their UTF-8 support.


mysql_enable_utf8 => 1,
has been added to the $DBI_PARAMS hashref.

• PostgreSQL

pg_enable_utf8 => 1,
has been added to the $DBI_PARAMS hashref.

• SQLite

sqlite_unicode => 1
has been added to the $DBI_PARAMS hashref.

No explicit encoding/decoding is done in Dada Mail when saving/retrieving data. Hopefully, the drivers are UTF-8-aware enough.


The Plugins and Extensions that come with Dada Mail have not been as thoroughly tested as the main program. There's still warts.

Dada Bridge

Dada Bridge has a unique position needing to handle a lot of different stuff thown at it and deal with it gracefully. Dada Mail does, in fact, handle, any realistic character set/encoding you throw at it, but Dada Mail will convert messages it receives to its internal format, before it resends it out to your list.

This means the encoding of your choice (8bit or quoted-printable) and the charset of your choice (as long as your charset is, UTF-8)


You are potentially going to have problems.

Its possible that, since List Settings were never decoded/encoded correctly in past versions, they'll show up the program (once you've upgrade) incorrectly. The easiest thing to do is to edit the mistakes and resave the information. For most of the program, you're going to have to manually export the information and re-import it with the correct encoding, sadly. Dada Mail will probably fail gracefully with old information, but it's possible that you'll see squiggly charaters, instead of what you want to see. There's nothing in Dada Mail that will stop this from happening. If you experience it (from old information), we're not going to count it as a bug, but rather a known issue.


Please let us know via the Support Boards:


Or the developer mailing list:



See Also:

• The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)


• perlunitut - Perl Unicode Tutorial


• perlunifaq - Perl Unicode FAQ





List Information:


Developer Info:

No comments: