Search

March 31, 2013

HeuristicMixedException with PostgreSQL

Just a quick post in case someone else ever encounters this issue. I got the following error when migrating an unchanged web application to CentOS 6.3 from an earlier version of CentOS:

2012-12-18 15:57:09,675 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (WorkManager(2)-7) [com.arjuna.ats.internal.jta.resources.arjunacore.preparefailed] [com.arjuna.ats.internal.jta.resources.arjunacore.preparefailed] XAResourceRecord.prepare - prepare failed with exception XAException.XAER_RMERR
2012-12-18 15:57:09,676 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (WorkManager(2)-7) [com.arjuna.ats.arjuna.coordinator.BasicAction_36] - BasicAction.End() - prepare phase of action-id -53e50a38:d833:50d0d48d:bf failed.
2012-12-18 15:57:09,676 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (WorkManager(2)-7) [com.arjuna.ats.arjuna.coordinator.BasicAction_38] - Action Aborting
2012-12-18 15:57:09,692 WARN  [com.arjuna.ats.jta.logging.loggerI18N] (WorkManager(2)-7) [com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror] [com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror] XAResourceRecord.rollback - xa error XAException.XAER_RMERR
2012-12-18 15:57:09,693 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (WorkManager(2)-7) [com.arjuna.ats.arjuna.coordinator.BasicAction_52] - Top-level abort of action -53e50a38:d833:50d0d48d:bf received heuristic decision: TwoPhaseOutcome.HEURISTIC_HAZARD
2012-12-18 15:57:09,703 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (WorkManager(2)-7) org.jboss.resource.adapter.jms.inflow.JmsServerSession@37515772 failed to commit/rollback
javax.transaction.HeuristicMixedException
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1422)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:135)
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
        at org.jboss.resource.adapter.jms.inflow.JmsServerSession$XATransactionDemarcationStrategy.end(JmsServerSession.java:657)
        at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:243)
        at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:213)
        at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Checking the PostgreSQL logs I found:

ERROR:  prepared transactions are disabled
HINT:  Set max_prepared_transactions to a nonzero value.
STATEMENT:  PREPARE TRANSACTION '131075_MS0tNTNlNTBhMzg6ZDgzMzo1MGQwZDQ4ZDo2ZQ==_LTUzZTUwYTM4OmQ4MzM6NTBkMGQ0OGQ6NzM='

The issue turns out to be that the new version of PostgreSQL does not have support for prepared statements turned on out of the box. To enable them, edit data/postgresql.conf as follows:

max_prepared_transactions = 10          # zero disables the feature

Don’t forget to restart PostgreSQL after changing the configuration.

March 24, 2013

Development Mail Server

If you have ever needed to develop a web application you have probably needed to send email from the application. Frequently in the applications I work on, we end up using the email address as the login, which means we need lots of email addresses for unit test and integration testing, etc. Using real email addresses for this is not very convenient, so we have used a separate development mail server. Ideally this mail server will accept mail from any development machine, but will not relay mail to any real addresses (so it won’t be an open relay).

Up until now we have used a Windows mail server for this purpose but we are in the process of converting much of our infrastructure to linux servers. So I needed to set something up on CentOS 6.3. After a little research and a few pointers from the fine denizens of the #centos IRC chat room on freenode.net, I settled on using postfix.

Postfix was already installed on my server, so all I needed to do was configure it to answer for a virtual mailbox on a fake domain. First we need a user and corresponding group to own the mailboxes, as well as a directory to hold them:

groupadd --system --gid 2525 vmail
useradd --system --uid 2525 \
        --comment "Virtual Mailbox Owner" \
        --shell /sbin/nologin \
        --home /var/mail/vhosts \
        --no-user-group -g vmail vmail
mkdir /var/mail/vhosts
mkdir /var/mail/vhosts/example.com
chown -R vmail:vmail /var/mail/vhosts

I am using vmail as the user and group and example.com for the fake domain.

Next we have some changes to the /etc/postfix/main.cf configuration file

# Allow postfix to listen on all IP addresses
inet_interfaces = all
#inet_interfaces = localhost

# Configure postfix to only accept mail for itself and never relay
smtpd_recipient_restrictions = reject_unauth_destination

# VIRTUAL DOMAINS
# See /usr/share/doc/postfix-2.6.6/README_FILES/VIRTUAL_README
virtual_mailbox_domains = example.com
virtual_mailbox_base = /var/mail/vhosts
virtual_mailbox_maps = hash:/etc/postfix/vmailbox
virtual_uid_maps = static:2525
virtual_gid_maps = static:2525

Next we need to set up users and passwords. Since we are dealing with a simple development server, we do not need anything too robust like an RDBMS. We’ll just use postfix’s ability to create and use Berkeley databases. Also, we want to create a bunch of generic accounts, so we’ll use a script:

cd /etc/postfix
# Create accounts userN@example.com for N from 1 to 20
for i in $( seq 1 20 )
do
    echo -e "user${i}@example.com\texample.com/user${i}/Maildir/" >>vmailbox
done
# Compile the database and reload the server
postmap /etc/postfix/vmailbox
postfix reload
# We need a restart because we changed the listen IP address
service postfix restart

Now we can test our configuration by using telnet to send some mail. Connect to postfix via telnet localhost smtpand send an email directly. Use the following example as a guide:

220 youserver.localdomain ESMTP Postfix
EHLO YOURSERVER
250-youserver.localdomain
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
MAIL FROM:user1@example.com
250 2.1.0 Ok
RCPT TO:user2@example.com
250 2.1.5 Ok
DATA
354 End data with <CR><LF>.<CR><LF>
Subject:Testing email

This is a test of postfix of our postfix configuration.
.
250 2.0.0 Ok: queued as 16CAE260A1D
QUIT
221 2.0.0 Bye
Connection closed by foreign host.

Be sure to check /var/log/maillog to ensure the mail is actually delivered. You may want to just open a terminal and tail -f /var/log/maillog for the rest of the way. You’ll also want to check that postfix created mail files for user2 under /var/mail/vhosts/example.com.

Before we get to the other side of the equation, reading email, we should configure our firewall to allow other machines to connect via SMTP and IMAP. First run iptables -L --line-numbers to determine the current state of your firewall and determine the number of the last rule. You’ll need to change the rule numbers below to that number and one more than that number. You’ll also want to change the --source argument to match your LAN.

iptables --insert INPUT 11 \
         --match state \
         --state NEW \
         --protocol tcp \
         --destination-port smtp \
         --source 10.0.0.0/24 \
         --jump ACCEPT
iptables --insert INPUT 12 \
         --match state \
         --state NEW \
         --protocol tcp \
         --destination-port imap \
         --source 10.0.0.0/24 \
         --jump ACCEPT
iptables -L --line-numbers

Test your firewall configuration by trying the telnet session from another machine. If everything is working as desired, save configuration via service iptables save, otherwise rollback with service iptables restart.

Now we need an IMAP server so that we can read email. I chose to use dovecot. I needed to install it using the following:

yum install dovecot
chkconfig --levels 2345 dovecot on

Next we configure dovecot to use a simple password file. Again, we don’t need anything fancy and all users will have the same unoriginal password, password. Modify /etc/dovecot/conf.d/10-auth.conf, changing lines as follows:

disable_plaintext_auth = no
#!include auth-system.conf.ext
!include auth-passwdfile.conf.ext

Tell dovecot where the mailboxes are by editing /etc/dovecot/conf.d/10-mail.conf:

mail_location = maildir:/var/mail/vhosts/%d/%n/Maildir

Next modify /etc/dovecot/conf.d/auth-passwdfile.conf.ext:

passdb {
  driver = passwd-file
  args = /etc/dovecot/%d-passdb
}

userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/mail/vmail/%d/%n
}

Create the password file /etc/dovecot/example.com-passdb using the script

cd /etc/dovecot
for u in $( cut -f1 /etc/postfix/vmailbox )
do
    echo -e "${u}:{PLAIN}password" >>example.com-passdb
done
#
# Now start dovecot; it shares the /var/log/maillog with postfix so continue
# to monitor that log file
service dovecot start

You should be able to connect to your server using an IMAP client like Thunderbird. (Recent versions of Thunderbird seem to want to try to discover the configuration itself, just abort that and configure it by hand.) Note you’ll need to use the full email address, user1@example.com, for the login.

Now simply configure your new web application to use your server as the SMTP server. As long as it only ever tries to send mail to example.com addresses, you should be set.