Jhaand wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== local IMAP and webmail ======

===== Project outline =====


My IMAP folder at xs4all is getting a bit full (at 98 %, while only 1.2 GiB) and all the E-mail is sitting at their servers. I would like to have it a my local server. Also the roundcube implementation they use, reacts quite slowly sometimes. maybe that a local solution will work better. 

The Plan:
   * get local E-mail delivery going via fetchmail for more than 1 user.
   * Install Dovecot imap server 
   * Install local roundcube server
   * Let roundcube deliver somewhere else. (already done by configuring local E-mail delivery and smarthost)
   * Enable https 
   * Transfer all old E-mail to new server and delete all old E-mail. (partly)
   * Get inbound E-mail at local roundcube via fetchmail 
   * Enable encrypted TLS connection to Dovecot 
   * Configure Roundcube some more. (Context menu's, display correctly, omit host to connect to, Spam and blacklisting, etc. TODO)
   * Issues and tidbits (TODO)

===== Local E-mail =====
Use the local exim4 install for Debian Linux. (Done)\\
Let it only listen to 127.0.0.1 (Done)   \\
replace mail from www-data for this E-mail address to my own. (Done)  \\

Add the following line to /etc/exim4/update-exim4.conf.conf
<code>
smpt_accept_queue_per_connection=0
</code>
Otherwise when a lot of mail comes in, it will stay stuck in Exim limbo.

==== Local Dovecot IMAP server ====

Installing dovecot on my local debian machine goes as follows:

See: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-postfix-e-mail-server-with-dovecot

<code>
apt-get install dovecot-common dovecot-core dovecot-imapd
</code>
Standard settings don't allow dovecot to manage /var/mail/ and it goes bonkers because of that. It shows this with the following error in /var/log/syslog. Change the configuration of dovecot to use the mail group for accessing the mail spool.
<code>
Jun 10 14:38:35 s dovecot: imap(bernhard): Error: 
file_dotlock_create(/var/mail/bernhard) failed: Permission 
denied (euid=1000(bernhard) egid=1000(bernhard) missing +w 
perm: /var/mail, we're not in group 8(mail), dir owned by 
0:8 mode=0775) (set mail_privileged_group=mail)
</code>
Fix it by changing the following line in file /etc/dovecot/conf.d/10-mail.conf
<code>
mail_privileged_group = mail
</code>

=== Dovecot authentification ===
(Update 2015-02-19)\\
Proper authentication for dovecot in combination with PAM also proved a challenge. By default, roundcube will access the imap server for every item it needs to download. So a mail item with 10's of pictures will do 10 request. If you're sorting out your mailbox, then roundcube will spam dovecot, which in turn will spam PAM for authentication on your system. And PAM will stop reacting within 5 minutes after a lot of access and roundcube freezes for at least 5 minutes, to get it's bearing. 
This is shown by the following errors.

Dovecot/mail log:
<code>
Feb 19 19:56:55 manbak dovecot: imap-login: Disconnected (auth failed, 1 attempts in 8 secs): user=<jhaand>, method=PLAIN, rip=::1, lip=::1, secured, session=<WwZuf3UPlgAAAAAAAAAAAAAAAAAAAAAB>
Feb 19 19:57:01 manbak dovecot: auth-worker(533): pam(jhaand,::1): pam_authenticate() failed: Authentication failure (password mismatch?)
Feb 19 19:57:03 manbak dovecot: imap-login: Disconnected (auth failed, 1 attempts in 4 secs): user=<jhaand>, method=PLAIN, rip=::1, lip=::1, secured, session=<4wMrgHUPmQAAAAAAAAAAAAAAAAAAAAAB>
Feb 19 19:58:01 manbak dovecot: auth-worker(533): pam(jhaand,::1): pam_authenticate() failed: Authentication failure (password mismatch?)
</code>

/var/log/auth.log
<code>
Feb 19 19:41:54 manbak auth: pam_unix(dovecot:auth): authentication failure; logname= uid=0 euid=0 tty=dovecot ruser=jhaand rhost=::1  user=jhaand
</code>
Solve these authentification problems by using the following configuration options.\\
/etc/dovecot/conf.d/10-auth.conf
<code>
# User database specifies where mails are located and what user/group IDs
# own them. For single-UID configuration use "static" userdb.
#
# <doc/wiki/UserDatabase.txt>

#!include auth-deny.conf.ext
#!include auth-master.conf.ext

!include auth-system.conf.ext
</code>
Then edit the auth-system.conf.ext file to the following:
<code>
# PAM authentication. Preferred nowadays by most systems.
# PAM is typically used with either userdb passwd or userdb static.
# REMEMBER: You'll need /etc/pam.d/dovecot file created for PAM
# authentication to actually work. <doc/wiki/PasswordDatabase.PAM.txt>
#auth_cache_key_size = 1024
auth_cache_size = 1024

passdb {
  driver = pam
  args = failure_show_msg=yes max_requests=100 cache_key=%u%s dovecot
  # [session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=<n>]
  # [cache_key=<key>] [<service name>]
  #args = dovecot
}
</code>

=== Final Dovecot config ===

So the final configuration for dovecot looks like {{dovecot.conf_out.txt}}. 

==== Local Roundcube server ====
Debian just decided to drop Roundcube from 'Jessie'. So I will have pull it from unstable/experimental myself. :-( 
(**Update 2015-10-23**: Roundcube is back in Debian again.)

Manually pulled the roundcube packages from Debian Unstable. Went for the Sqlite edition. also got -plugins and -plugins-extra. Try to install them all using "dpkg -i ...." and see how it goes.

roundcube-sqlite3: depends on php5-sqlite (done)
roundcube-core: depends on the following: dbconfig-common php-auth php-net-smtp php-net-socket php-mail-mime tinymce , succes: roundcube already selects sqlite3. 
roundcube-plugins: no problems
roundcube-plugins-extra: depends on libjs-jquery-mousewheel php-net-sieve
roundcube: no problem

And this is what we end up with.
<code>
breetai@mainbak:/mnt/sdb1/home/breetai/Download/roundcube$ dpkg -l roundcube\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                       Version            Architecture       Description
+++-==========================-==================-==================-=========================================================
ii  roundcube                  0.9.5+dfsg1-4.2    all                skinnable AJAX based webmail solution for IMAP servers - 
ii  roundcube-core             0.9.5+dfsg1-4.2    all                skinnable AJAX based webmail solution for IMAP servers
un  roundcube-mysql            <none>             <none>             (no description available)
un  roundcube-pgsql            <none>             <none>             (no description available)
ii  roundcube-plugins          0.9.5+dfsg1-4.2    all                skinnable AJAX based webmail solution for IMAP servers - 
ii  roundcube-plugins-extra    0.9.2-20130819     all                skinnable AJAX based webmail solution - extra plugins
un  roundcube-sqlite           <none>             <none>             (no description available)
ii  roundcube-sqlite3          0.9.5+dfsg1-4.2    all                metapackage providing SQLite dependencies for RoundCube
breetai@mainbak:/mnt/sdb1/home/breetai/Download/roundcube$ 
</code>

Configure apache to recognize the roundcube URL.([[http://www.debiantutorials.com/how-to-install-roundcube-on-squeeze/|SRC/inspiration]]) Edit  /etc/apache2/conf-available/roundcube and uncomment:
<code>
Alias /roundcube/program/js/tiny_mce/ /usr/share/tinymce/www/
Alias /roundcube /var/lib/roundcube
</code>

Then restart Apache
<code>
sudo service apache2 restart
</code>

Now enter roundcube by going to your own server: %% http://localhost/roundcube %% \\
For the server choose the local IMAP server or a remote IMAP server of your E-mail provider (gmail, ziggo, xs4all, etc.)

If you now log into localhost with a local user, you should see the inbox of that user. 

=== Configure to use localhost as default ===
This is quite easy. Just edit /etc/roundcube/main.inc.php and change "default_host" to localhost.  

==== Migrating old IMAP mail to Dovecot ====

In order to get my mail from xs4all IMAP server to the local dovecot, I will stick with the following plan. 
   * Backup mail from xs4all
   * Put it into local dovecot for new user
   * Eradicate mail from IMAP at xs4all

If any new mail comes in, in the mean time. Fetchmail will get it later. Also Dovecot stores all the mail as mbox. Thus, any text editor and copy paste action can also move the mail to the local server if needs be. 

The first 2 steps are described at the Dovecot wiki. http://wiki2.dovecot.org/Migration/Dsync


==== Improved security via https ====

it's great you can now read your E-mail on your own server. However the mail now goes through an unsecured channel. Get a security certificate via [[http://cacert.org|CA-cert]] and make sure that the local apache server uses it. It's a bit tricky to get working, but there's enough material on the web to make it work. Also don't forget to renew the certificate before it expires.

Now make sure that roundcube uses only SSL. Modify /etc/roundcube/main.inc.php and change the following line:
<code>
// enforce connections over https
// with this option enabled, all non-secure connections will be redirected.
// set the port for the ssl connection as value of this option if it differs from the default 443
//$rcmail_config['force_https'] = false;
$rcmail_config['force_https'] = true;
</code>
Go to the normal http address and check that it automatically forwards towards the secured https version. 

==== Copy all mail ====
There exist several methods for transferring mail via IMAP4. After doing some research. I seemed that the best way for transferring mail remained the program ImapCopy. You can find the home-page [[http://imapsync.lamiral.info/|here]] and download the program [[https://github.com/imapsync/imapsync|here]]. It seems that the github edition is a somewhat older version but it's allowed to use it according to the [[http://imapsync.lamiral.info/NOLIMIT|very simple licence.]] 

Imapcopy has some very annoing dependencies for Perl. But fortunately they're all part of the standard debian repository. Install them via ([[https://github.com/imapsync/imapsync/blob/master/INSTALL|src]]):
<code>
aptitude install libmail-imapclient-perl # Mail::IMAPClient
aptitude install libterm-readkey-perl # Term::ReadKey
aptitude install libio-socket-ssl-perl # IO::Socket::SSL
aptitude install libdigest-hmac-perl # Digest::HMAC_MD5 Digest::HMAC_SHA1
aptitude install liburi-perl # URI::Escape
aptitude install libfile-copy-recursive-perl # File::Copy::Recursive
aptitude install libio-tee-perl # IO::Tee
aptitude install libunicode-string-perl # Unicode::String
</code>
Now, for copying all the mail. Just enter the host1, user1, host2 and user2 on the command line for Imapcopy. Omit the passwords on the command line. (You don't want to have unencrypted passwords in your .bash_history files for the next 500 commands.) 
So it looks like this example:
<code>
imapcopy --host1 imap.xs4all.nl --user1 jhaand --host2 localhost --user2 jhaand
</code>
And them we [[http://i0.kym-cdn.com/photos/images/original/000/159/334/9266932.jpg|wait]] until 1.15 GiB of mails are transported. Which took around an hour and produced the following statistics:

<code>
++++ Statistics
Transfer started on               : Sun Feb  8 15:48:47 2015
Transfer ended on                 : Sun Feb  8 16:39:02 2015
Transfer time                     : 3014.7 sec
Messages transferred              : 17672 
Messages skipped                  : 0
Messages found duplicate on host1 : 0
Messages found duplicate on host2 : 0
Messages void (noheader) on host1 : 0
Messages void (noheader) on host2 : 0
Messages deleted on host1         : 0
Messages deleted on host2         : 0
Total bytes transferred           : 1189272676 (1.108 GiB)
Total bytes duplicate host1       : 0 (0.000 KiB)
Total bytes duplicate host2       : 0 (0.000 KiB)
Total bytes skipped               : 0 (0.000 KiB)
Total bytes error                 : 0 (0.000 KiB)
Message rate                      : 5.9 messages/s
Average bandwidth rate            : 385.3 KiB/s
Reconnections to host1            : 0
Reconnections to host2            : 0
Memory consumption                : 141.3 MiB
Biggest message                   : 13949149 bytes
Initial difference host2 - host1  : -17670 messages, -1189271814 bytes (-1.108 GiB)
Final   difference host2 - host1  : 2 messages, 1408 bytes (1.375 KiB)
Detected 0 errors


Homepage: http://imapsync.lamiral.info/
Log file is LOG_imapsync/2015_02_08_15_48_47_jhaand.txt
</code>

Move the inbox on the local server to a separate folder. Because fetchmail will pull it in later. Because of complete failure of my 0 inbox policy of the last 2 weeks. It's completely full and I don't want to delete 160+ mails. 

==== Configure fetchmail to deliver mail to local users ====
Currently I run fetchmail as a local user. Every time I open a shell, the login script checks if I'm already running fetchmail. If it isn't, then fetchmail is started. This needs to be reconfigured to a global fetchmail configuration that also delivers for the new user. 
**Update 2015-10-23** Procmail makes much more sense to use. Because the local exim server doesn't understand mailing lists and bcc's. Which results in mail ending up in the root account. [[http://www.tuxradar.com/content/get-started-fetchmail-procmail-and-dovecot]]

   1 First disable the local E-mail for the local user. 
   2 Configure the global /etc/procmailrc
   3 Then configure the global /etc/fetchmailrc
   4 Enable fetchmail service via /etch/default/fetchmail 
  
=== Disable the local E-mail ===
The local user has the following code in the .bash_login to make sure that fetchmail runs after this user logs in. And polls more than once per minute. (57 seconds in this case) Just for the sake of demonstration I will show it here. 
<code>
fetchm=`ps -U user | grep fetchmail `

if [ -z "$fetchm" ] ; then
        fetchmail -d 57
fi
</code>

=== Global procmailrc ===
This should just remain a back to basics procmailrc. Deliver all the mail in the user's mailbox.
It looks like this:
<code>
MAILDIR=/var/spool/mail
DEFAULT=$MAILDIR/$LOGNAME
LOGFILE=/var/log/procmail
VERBOSE=on
</code>

Check that all the directories exist and the individual mail files have user and group owners as "user:mail" 

=== Global fetchmailrc ===
Now lets create a new global /etc/fetchmailrc. Inspiration taken from [[https://www.howtoforge.com/debian_etch_fetchmail|here]] or you can also find it in /usr/share/doc/fetchmail/examples.
<code>
# /etc/fetchmailrc for system-wide daemon mode
# This file must be chmod 0600, owner fetchmail

set daemon        57            # Poll every 57 seconds (why waste a good tradition) 
set syslog                      # use syslog for logging
set postmaster  root

set no bouncemail               # avoid loss on 4xx errors
                                # on the other hand, 5xx errors get
                                # more dangerous...

##########################################################################
# Hosts to pool
##########################################################################

# Defaults ===============================================================
# Set antispam to -1, since it is far safer to use that together with
# no bouncemail
defaults:
timeout 300
antispam -1
batchlimit 100

poll  pop.xs4all.nl protocol POP3: 
envelope  X-XS4ALL-TO
 user "some_user1" there with password "<password>" is "some_user1" here fetchall
mda '/usr/bin/procmail -d some_user1'

poll  pop.xs4all.nl protocol POP3: 
envelope  X-XS4ALL-TO
user "some_user2" there with password "<password>" is "some_user2" here fetchall 
mda "/usr/bin/procmail -d some_user2"
</code>
The with *envelope* is needed to prevent warnings from syslog. It identifies which header fetchmail should use to feed it to the local mail server. 

The *mda* command tells fetchmail to use procmail and tell procmail to deliver it to the local user. also don't use single quotes ( ' ) but double quotes on the command line. Otherwise strange stuff happens. 

Also addressing multiple users was quite difficult. A lot of tutorials use either the multiple users per single drop and don't use the mda. Or use a single user and then use mda with procmail. I tried using the "procmail -d %T" method across multiple lines but then it ust didn't work. So it's better to have 2 complete separate definitions and hardcode procmail to the correct user per session. Haven't had a problem since. 

Make sure that the daemon starts with the following setup for /etc/fetchmailrc:
<code>
sudo chmod 600 /etc/fetchmailrc
sudo chown user:root /etc/fetchmailrc
</code>

=== Test run ===
The test run didn't pan out that well. It's better to just start the daemon in regular mode. But having acute logging open of /var/log/syslog, /var/log/mail.log and /var/log/procmail via "tail -f" I observed what happened. Like for instance my mail being downloaded and then disappearing. Exim deemed the culprit in this case. It had a limit of only 10 new entries per delivery. Nice for a big site, but worthless for a small site like mine. So adapt the setting for Exim. Figuring this out is deemed out of scope for this article.

=== Enable fetchmail globally === 
Enable fetchmail by adapting the setting in /etc/default/fetchmail. 
<code>
# Declare here if we want to start fetchmail. 'yes' or 'no'
START_DAEMON=yes
</code>
And then start the fetchmail service "sudo service fetchmail start"\\
Check in the logging on what will happen and where the E-mail goes. 
==== Delete all the remote mail ==== 
(TODO)

It should be possible to wipe all the mail folders using the IMAP protocol from a command line tool. Still need to figure out how. 


===== Enabe encrypted remote IMAP ===== 
(TODO) The IMAP connection on localhost between Dovecot and Roundcube remains unencrypted. This should not be such a problem. The only ones who could intercept this would need root perissions. However, I would also like to receive mail using my Android phone. That's a complete untrusted connection and it needs encryption. Otherwise my username and password will get intercepted. 

I think it's possible to encrypt the IMAP connection without too much hassle. But I actually don't want to expose the unencrypted interface to the outside world. Thus the lo interface can have the unencrypted interface and the eth0 interface shall only show the encrypted interface. 

Since apache already has a certificate, recycle these for dovecot. Follow the instructions from Dovecot. http://wiki2.dovecot.org/SSL/DovecotConfiguration