This is an old revision of the document!
Table of Contents
local IMAP and webmail
Project outline
My IMAP folder at xs4all is getting a bit full (at 98 %, while only 1.2 GiB) and all the E-mail is sitting at their servers. I would like to have it a my local server. Also the roundcube implementation they use, reacts quite slowly sometimes. maybe that a local solution will work better.
The Plan:
- get local E-mail delivery going via fetchmail for more than 1 user.
- Install Dovecot imap server
- Install local roundcube server
- Let roundcube deliver somewhere else. (already done by configuring local E-mail delivery and smarthost)
- Enable https
- Transfer all old E-mail to new server and delete all old E-mail. (partly)
- Get inbound E-mail at local roundcube via fetchmail
- Enable encrypted TLS connection to Dovecot
- Configure Roundcube some more. (Context menu's, display correctly, omit host to connect to, Spam and blacklisting, etc. TODO)
- Issues and tidbits (TODO)
Local E-mail
Use the local exim4 install for Debian Linux. (Done)
Let it only listen to 127.0.0.1 (Done)
replace mail from www-data for this E-mail address to my own. (Done)
Add the following line to /etc/exim4/update-exim4.conf.conf
smpt_accept_queue_per_connection=0
Otherwise when a lot of mail comes in, it will stay stuck in Exim limbo.
Local Dovecot IMAP server
Installing dovecot on my local debian machine goes as follows:
See: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-postfix-e-mail-server-with-dovecot
apt-get istall dovecot-common dovecot-core dovecot-imapd
Standard settings don't allow dovecot to manage /var/mail/ and it goes bonkers because of that. It shows this with the following error in /var/log/syslog. Change the configuration of dovecot to use the mail group for accessing the mail spool.
Jun 10 14:38:35 s dovecot: imap(bernhard): Error: file_dotlock_create(/var/mail/bernhard) failed: Permission denied (euid=1000(bernhard) egid=1000(bernhard) missing +w perm: /var/mail, we're not in group 8(mail), dir owned by 0:8 mode=0775) (set mail_privileged_group=mail)
Fix it by changing the following line in file /etc/dovecot/conf.d/10-mail.conf
mail_privileged_group = mail
Dovecot authentification
(Update 2015-02-19)
Proper authentication for dovecot in combination with PAM also proved a challenge. By default, roundcube will access the imap server for every item it needs to download. So a mail item with 10's of pictures will do 10 request. If you're sorting out your mailbox, then roundcube will spam dovecot, which in turn will spam PAM for authentication on your system. And PAM will stop reacting within 5 minutes after a lot of access and roundcube freezes for at least 5 minutes, to get it's bearing.
This is shown by the following errors.
Dovecot/mail log:
Feb 19 19:56:55 manbak dovecot: imap-login: Disconnected (auth failed, 1 attempts in 8 secs): user=<jhaand>, method=PLAIN, rip=::1, lip=::1, secured, session=<WwZuf3UPlgAAAAAAAAAAAAAAAAAAAAAB> Feb 19 19:57:01 manbak dovecot: auth-worker(533): pam(jhaand,::1): pam_authenticate() failed: Authentication failure (password mismatch?) Feb 19 19:57:03 manbak dovecot: imap-login: Disconnected (auth failed, 1 attempts in 4 secs): user=<jhaand>, method=PLAIN, rip=::1, lip=::1, secured, session=<4wMrgHUPmQAAAAAAAAAAAAAAAAAAAAAB> Feb 19 19:58:01 manbak dovecot: auth-worker(533): pam(jhaand,::1): pam_authenticate() failed: Authentication failure (password mismatch?)
/var/log/auth.log
Feb 19 19:41:54 manbak auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=jhaand rhost=::1 user=jhaand
Solve these authentification problems by using the following configuration options.
/etc/dovecot/conf.d/10-auth.conf
# User database specifies where mails are located and what user/group IDs # own them. For single-UID configuration use "static" userdb. # # <doc/wiki/UserDatabase.txt> #!include auth-deny.conf.ext #!include auth-master.conf.ext !include auth-system.conf.ext
Then edit the auth-system.conf.ext file to the following:
# PAM authentication. Preferred nowadays by most systems. # PAM is typically used with either userdb passwd or userdb static. # REMEMBER: You'll need /etc/pam.d/dovecot file created for PAM # authentication to actually work. <doc/wiki/PasswordDatabase.PAM.txt> #auth_cache_key_size = 1024 auth_cache_size = 1024 passdb { driver = pam args = failure_show_msg=yes max_requests=100 cache_key=%u%s dovecot # [session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=<n>] # [cache_key=<key>] [<service name>] #args = dovecot }
Final Dovecot config
So the final configuration for dovecot looks like dovecot.conf_out.txt.
Local Roundcube server
Debian just decided to drop Roundcube from 'Jessie'. So I will have pull it from unstable/experimental myself. (Update 2015-10-23: Roundcube is back in Debian again.)
Manually pulled the roundcube packages from Debian Unstable. Went for the Sqlite edition. also got -plugins and -plugins-extra. Try to install them all using “dpkg -i ….” and see how it goes.
roundcube-sqlite3: depends on php5-sqlite (done) roundcube-core: depends on the following: dbconfig-common php-auth php-net-smtp php-net-socket php-mail-mime tinymce , succes: roundcube already selects sqlite3. roundcube-plugins: no problems roundcube-plugins-extra: depends on libjs-jquery-mousewheel php-net-sieve roundcube: no problem
And this is what we end up with.
breetai@mainbak:/mnt/sdb1/home/breetai/Download/roundcube$ dpkg -l roundcube\* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-==========================-==================-==================-========================================================= ii roundcube 0.9.5+dfsg1-4.2 all skinnable AJAX based webmail solution for IMAP servers - ii roundcube-core 0.9.5+dfsg1-4.2 all skinnable AJAX based webmail solution for IMAP servers un roundcube-mysql <none> <none> (no description available) un roundcube-pgsql <none> <none> (no description available) ii roundcube-plugins 0.9.5+dfsg1-4.2 all skinnable AJAX based webmail solution for IMAP servers - ii roundcube-plugins-extra 0.9.2-20130819 all skinnable AJAX based webmail solution - extra plugins un roundcube-sqlite <none> <none> (no description available) ii roundcube-sqlite3 0.9.5+dfsg1-4.2 all metapackage providing SQLite dependencies for RoundCube breetai@mainbak:/mnt/sdb1/home/breetai/Download/roundcube$
Configure apache to recognize the roundcube URL.(SRC/inspiration) Edit /etc/apache2/conf-available/roundcube and uncomment:
Alias /roundcube/program/js/tiny_mce/ /usr/share/tinymce/www/ Alias /roundcube /var/lib/roundcube
Then restart Apache
sudo service apache2 restart
Now enter roundcube by going to your own server: http://localhost/roundcube
For the server choose the local IMAP server or a remote IMAP server of your E-mail provider (gmail, ziggo, xs4all, etc.)
If you now log into localhost with a local user, you should see the inbox of that user.
Configure to use localhost as default
This is quite easy. Just edit /etc/roundcube/main.inc.php and change “default_host” to localhost.
Migrating old IMAP mail to Dovecot
In order to get my mail from xs4all IMAP server to the local dovecot, I will stick with the following plan.
- Backup mail from xs4all
- Put it into local dovecot for new user
- Eradicate mail from IMAP at xs4all
If any new mail comes in, in the mean time. Fetchmail will get it later. Also Dovecot stores all the mail as mbox. Thus, any text editor and copy paste action can also move the mail to the local server if needs be.
The first 2 steps are described at the Dovecot wiki. http://wiki2.dovecot.org/Migration/Dsync
Improved security via https
it's great you can now read your E-mail on your own server. However the mail now goes through an unsecured channel. Get a security certificate via CA-cert and make sure that the local apache server uses it. It's a bit tricky to get working, but there's enough material on the web to make it work. Also don't forget to renew the certificate before it expires.
Now make sure that roundcube uses only SSL. Modify /etc/roundcube/main.inc.php and change the following line:
// enforce connections over https // with this option enabled, all non-secure connections will be redirected. // set the port for the ssl connection as value of this option if it differs from the default 443 //$rcmail_config['force_https'] = false; $rcmail_config['force_https'] = true;
Go to the normal http address and check that it automatically forwards towards the secured https version.
Copy all mail
There exist several methods for transferring mail via IMAP4. After doing some research. I seemed that the best way for transferring mail remained the program ImapCopy. You can find the home-page here and download the program here. It seems that the github edition is a somewhat older version but it's allowed to use it according to the very simple licence.
Imapcopy has some very annoing dependencies for Perl. But fortunately they're all part of the standard debian repository. Install them via (src):
aptitude install libmail-imapclient-perl # Mail::IMAPClient aptitude install libterm-readkey-perl # Term::ReadKey aptitude install libio-socket-ssl-perl # IO::Socket::SSL aptitude install libdigest-hmac-perl # Digest::HMAC_MD5 Digest::HMAC_SHA1 aptitude install liburi-perl # URI::Escape aptitude install libfile-copy-recursive-perl # File::Copy::Recursive aptitude install libio-tee-perl # IO::Tee aptitude install libunicode-string-perl # Unicode::String
Now, for copying all the mail. Just enter the host1, user1, host2 and user2 on the command line for Imapcopy. Omit the passwords on the command line. (You don't want to have unencrypted passwords in your .bash_history files for the next 500 commands.) So it looks like this example:
imapcopy --host1 imap.xs4all.nl --user1 jhaand --host2 localhost --user2 jhaand
And them we wait until 1.15 GiB of mails are transported. Which took around and hour and produced the following statistics:
++++ Statistics Transfer started on : Sun Feb 8 15:48:47 2015 Transfer ended on : Sun Feb 8 16:39:02 2015 Transfer time : 3014.7 sec Messages transferred : 17672 Messages skipped : 0 Messages found duplicate on host1 : 0 Messages found duplicate on host2 : 0 Messages void (noheader) on host1 : 0 Messages void (noheader) on host2 : 0 Messages deleted on host1 : 0 Messages deleted on host2 : 0 Total bytes transferred : 1189272676 (1.108 GiB) Total bytes duplicate host1 : 0 (0.000 KiB) Total bytes duplicate host2 : 0 (0.000 KiB) Total bytes skipped : 0 (0.000 KiB) Total bytes error : 0 (0.000 KiB) Message rate : 5.9 messages/s Average bandwidth rate : 385.3 KiB/s Reconnections to host1 : 0 Reconnections to host2 : 0 Memory consumption : 141.3 MiB Biggest message : 13949149 bytes Initial difference host2 - host1 : -17670 messages, -1189271814 bytes (-1.108 GiB) Final difference host2 - host1 : 2 messages, 1408 bytes (1.375 KiB) Detected 0 errors Homepage: http://imapsync.lamiral.info/ Log file is LOG_imapsync/2015_02_08_15_48_47_jhaand.txt
Move the inbox on the local server to a separate folder. Because fetchmail will pull it in later. Because of complete failure of my 0 inbox policy of the last 2 weeks. It's completely full and I don't want to delete 160+ mails.
Configure fetchmail to deliver mail to local users
Currently I run fetchmail as a local user. Every time I open a shell, the login script checks if I'm already running fetchmail. If it isn't, then fetchmail is started. This needs to be reconfigured to a global fetchmail configuration that also delivers for the new user.
1 First disable the local E-mail for the local user. 2 Then configure the global /etc/fetchmailrc 3 Test run with fetchmail as root 4 Enable fetchmail service via /etch/default/fetchmail
Disable the local E-mail
The local user has the following code in the .bash_login to make sure that fetchmail runs after this user logs in. And polls more than once per minute. (57 seconds in this case) Just for the sake of demonstration I will show it here.
fetchm=`ps -U user | grep fetchmail ` if [ -z "$fetchm" ] ; then fetchmail -d 57 fi
Global fetchmailrc
Now lets create a new global /etc/fetchmailrc. Inspiration taken from here or you can also find it in /usr/share/doc/fetchmail/examples..
# /etc/fetchmailrc for system-wide daemon mode # This file must be chmod 0600, owner fetchmail set daemon 57 # Poll every 57 seconds (why waste a good tradition) set syslog # use syslog for logging set postmaster root set no bouncemail # avoid loss on 4xx errors # on the other hand, 5xx errors get # more dangerous... ########################################################################## # Hosts to pool ########################################################################## # Defaults =============================================================== # Set antispam to -1, since it is far safer to use that together with # no bouncemail defaults: timeout 300 antispam -1 batchlimit 100 poll pop.xs4all.nl protocol POP3: envelope X-XS4ALL-TO aka xs4all.nl user "some_user1" there with password "<password>" is "some_user1" here fetchall; user "some_user2" there with password "<password>" is "some_user2" here fetchall;
The with *envelope* is needed to prevent warnings from syslog. It identifies which header fetchmail should use to feed it to the local mail server.
The second value with *aka* proved more tricky. All the mail was delivered to the default user of localhost, because fetchmail doesn't know the mail address for the local user. It was delivered as jhaand@xs4all and fetchmail didn't know it should go to jhaand@localhost. This situation has worked for 15 years for the default user. Because it was a single user system up until now. I've seen the warning in the headers in the early 2000's but everything worked then, so it didn't need any follow-up.
With the new user it didn't work that great. Now all the 160+ mails were delivered to the default user. By using some advanced tools from my mail client Mutt, I was able to sort this out quicker than finding the correct setting to get everything going. It took a while but even that bug was squished.
Make sure that the daemon starts with the following setup for /etc/fetchmailrc:
sudo chmod 600 /etc/fetchmailrc sudo chown user:root /etc/fetchmailrc
Test run
The test run didn't pan out that well. It's better to just start the daemon in regular mode. But having acute logging open of /var/log/syslog and /var/log/mail.log via “tail -f” I observed what happened. Like for instance my mail being downloaded and then disappearing. Exim deemed the culprit in this case. It had a limit of only 10 new entries per delivery. Nice for a big site, but worthless for a small site like mine. So adapt the setting for Exim. Figuring this out is deemed out of scope for this article.
Enable fetchmail globally
Enable fetchmail by adapting the setting in /etc/default/fetchmail.
# Declare here if we want to start fetchmail. 'yes' or 'no' START_DAEMON=yes
And then start the fetchmail service “sudo service fetchmail start”
Check in the logging on what will happen and where the E-mail goes.
Delete all the remote mail
(TODO)
It should be possible to wipe all the mail folders using the IMAP protocol from a command line tool. Still need to figure out how.
Enabe encrypted remote IMAP
(TODO) The IMAP connection on localhost between Dovecot and Roundcube remains unencrypted. This should not be such a problem. The only ones who could intercept this would need root perissions. However, I would also like to receive mail using my Android phone. That's a complete untrusted connection and it needs encryption. Otherwise my username and password will get intercepted.
I think it's possible to encrypt the IMAP connection without too much hassle. But I actually don't want to expose the unencrypted interface to the outside world. Thus the lo interface can have the unencrypted interface and the eth0 interface shall only show the encrypted interface.
Since apache already has a certificate, recycle these for dovecot. Follow the instructions from Dovecot. http://wiki2.dovecot.org/SSL/DovecotConfiguration