Luamail: a mail client built into luakit

Similarly to how back in 2009 there was no browser that works in a way I find sane, and I started solving that with uzbl, now I'm fed up with the lack of an email client that works in a way I find sane. Uzbl turned out to be a bit cumbersome for my taste, so I switched to the uzbl-inspired but more pragmatic luakit browser, which is much in the same vein, except that all configuration, extensions, event handling, programmatic input etc are done by interfacing with lua API's. Now I want to build the "luakit of email clients". Let me explain what's that all about...

Basically the story is pretty much the same as it was with uzbl. There are no mail clients which offer a high level of customization and interfacing possibilities. There are some mail clients aimed at "power users" and "lightweight mail clients" like mutt/alpine/nmh etc but those are also restricted in extensibility and often crippled in terms of features. Currently I'm using Claws-mail, which is the least sucky client I found, but it's also nowhere near what I want.

So, what do I want in a mail client?

  • fast and non-bloaty (of course my definition of bloat might differ from yours, for me it's a combination of run-time performance and source code elegancy)
  • minimal but efficient UI. keyboard controlled. modal interface optional but enabled by default
  • extensive customisation options
  • plain text configuration and data files, a binary format only where appropriate (like a search index or a database)
  • rendering HTML mails and rich content (images, embedded videos, flash, HTML5) properly. (I receive fancy newsletters from companies and rich content such as news messages from rss feeds by means of rss2email-xdg)
  • great link following, and for external files: integration with download software, proxies, etc). I.e. "what luakit has"
  • support Unicode and attachments (mime) properly
  • play nice with window managers (window title, urgency hint, ..), support many key bindings and use UI widgets that are not a pain (i.e. no ncurses, be an X client)
  • provide a unified interface to email sending/receiving/searching
  • keep my homedir clean. Use xdg basedir spec (i.e. cache mail and search indices in $XDG_CACHE_DIR)
  • key binds to show/hide quoted text, focus unquoted text when viewing mail by default.
  • play nice with mailing lists (track all mailing lists I'm on, allow me to unsubscribe with a shortcut)
  • filtering and mail (or displaying thereof) altering: for some specific mailing lists, prepend the subject with the name of the mailing list (i.e. lists that use the List-Id header and do not rewrite subjects)
  • high level of control over the actual message that gets sent out. I.e. allow me to change and modify all headers
  • when sending a message fails, keep it in a retry-buffer. give me an interface to that buffer (I.e. a list of all mails which could not be sent, why not, when they'll be retried, shortcuts to retry immediately, etc)
  • allow me to define sieve rules in my mail client configuration, and automatically update server side when needed.
  • allow me, as a user to change anything in a fine-grained way. Some examples:
    • Differentiate properly between received mails that are "definitely optional for me", and "aimed to me". I.e. mark all mails from a mailing list low-priority, unless a 'To'/'Cc'/'Bcc' field contains any of the addresses I use to send from or when the mail is a reply to a mail I sent earlier (check Message-ID and mail subject)
    • Use a tag-based interface towards gmail. Gmail uses tags instead of folders, which it cannot really expose properly over IMAP (it must "duplicate" the same message in multiple folders when speaking IMAP). I want to modify my client so that for gmail accounts, it also uses tags instead of folders
    • Use different key bindings dependent on account in use or any property such as sender of a mail message. Example: when deleting a message, delete straight away when it has a certain spamassasin score or is an rss2email message, move to Trash otherwise. When the message is from anyone I know (in my contact list), ask me to confirm and offer moving to an archive folder instead.
    • When I get a mail from nagios with a critical alert, show an inotify error message containing the right information (parse the nagios mail first), and set urgent flag. Do the same when my boss sent me a mail containing the phrase 'urgent', unless the 'To' field contains more than 20 addresses. Give me a key bind that sends a mail containing "I'm on it!" and switch to my browser workspace and open all my monitoring pages in my browser. Don't disturb me for any other e-mail
    • Use bottom posting by default, but when I'm replying to mailing list foo or person bar, use top-posting instead.
    • Display mails by default threaded by subject and date. But for my news feeds messages (which arrive on the same account!), group by sender. maybe even categorize senders so that I can quickly delete all humorous messages from various senders/feeds if I have no time or feel guilty about slacking off

Implementation

My first idea was to build a standalone mail client by reusing the luakit C-based host program and all bindings it exposes (webkitGtk widget with bindings, gtk bindings, lots of lua code for menu's, keyboard control, downloading external files, proxy code, etc. very useful stuff); which would mean we supply a set of different config files for the luakit host program, along with some more lua libraries and/or bindings to C software.
But instead we choose to neatly integrate the mail client in luakit itself, which makes it easier to develop and keep up with luakit (the diff is minimal instead of extensive), and it's probably more convenient for the user. So we just forked the luakit project and try to do our stuff in a separate branch, adding the mail client bits in between the existing browser bits. The good news is, so many needed things already exist, so building the client shouldn't be that much work. I've been researching our options (and still am), this is how the list of things we'll rely on looks like so far:
  • HTML/rich content: webkitGtk widget provided by luakit
  • key bindings, proxy, external files downloader, link follower, ...: luakit's lua code
  • IMAP library: luaimap
  • SMTP: msmtp (could not find any suitable lua SMTP library with proper TLS/SSL support)
  • maildir/mh/other local formats: TBD
  • message parsing: TBD. we might end up writing bindings to existing C libraries, as we couldn't find a good lua library yet, and it's hard to write one
  • search: TBD
Mason (luakit maintainer) liked the idea from the start; Gerry (luaimap maintainer) also liked the idea and joined the project right away :)

Why not uzbl-style?

Gerry asked: "why not structure the mail client like uzbl is (fast c core with shell and python around it)?"
The main lesson I learned from doing uzbl is: the very ideology-based approach of "one program for each thing" is nice in theory, but can become a hassle in practice. The means of IPC are limited: passing command line arguments, setting environment variables, or creating your own little protocol that needs to be parsed. Advanced constructs like loops and branches complicate things more, and when passing stuff through various processes like we did, it can be tricky to figure out how to quote your strings so that they are expanded at the right stage. It's also hard to jump between various pieces of code; the separation between processes start to feel like real barriers.
Luakit provides lua API's, which means all extensions should be primarily written in lua (which is a burden if you don't feel like learning a new language), you can still exec any other script or program if you want to from lua however, but the API's are very fine grained, and pretty much everything is exposed,so there's lots of things you can do, but in a way that feels more robust and reliable. And things work faster too.
I'm not that experienced yet with luakit to grasp all its downsides, but it feels more reliable. And maybe subconsciously, the fact that luakit feels as nice as uzbl without me needing to do anything (Mason did all the hard work of writing lua bindings and writing all lua code), whereas uzbl involved lots of hard work for me, maybe that twists my perception a bit. But not too much, I think.
The same upsides and downsides go for writing a mail client; so luakit-style seems more appropriate to me. Especially since so many libraries and components already exist, and just need to stitched together.

Show me the code already!!

We actually have a very rough, but working prototype. Ready for you to try out. See the luakit-dev post for details.

Add comment