- Posted in: Admin-tips, Development
It’s been a few months since we‘ve started actively using ActiveMQ queue server in our project. For some time we had pretty weird problems with it and even started thinking about switching to something else or even writing our own queue server which would comply with our requirements. The most annoying problem was the following: some time after activemq restart everything worked really well and then activemq started lagging, queue started growing and all producer processes were stalling on push() operations. We rewrote our producers from Ruby to JRuby, then to Java and still – after some time everything was in a bad shape until we restarted the queue server.
So, long story short, after a lots of docs and source code reading we’ve found really interesting thing. There is a “feature” added in the recent ActiveMQ release (5.X) called “flow control“, which is used by ActiveMQ core to slow down too fast producers when consumers lag or just aren’t as fast as ActiveMQ wants them to be. This results in unpredictable producers stalls and all kinds of weird problems.
If you’re experiencing the same problems, you could try to add the following to your activemq.xml file to make it stop using flow control and let your producers spool data to the queue as fast as it is possible (that’s exactly what I’d expect from a queue server):
1 2 3 4 5 6 7 | <destinationPolicy> <policyMap> <policyEntries> <policyEntry queue=">" producerFlowControl="false" memoryLimit="64mb" /> </policyEntries> </policyMap> </destinationPolicy> |
When this is added, all your queues (if they take more than 64Mb of RAM) will start persisting their messages to the disk and freeing your RAM to consume more messages w/o any slowdowns.
Another useful feature we use in our producer is Async Sends which basically means that your producers won’t require any acknowledgments from ActiveMQ after it pushes your messages to a queue. This makes push() operations MUCH faster and your producers’ throughput will become tremendously higher. If you use Java as your producer language, you could use the following code to make your connection async:
1 2 3 4 5 6 | queueConnectionFactory = new ActiveMQConnectionFactory(queue_user, queue_password, queue_url); queueConnectionPool = new PooledConnectionFactory(queueConnectionFactory); queueConnectionPool.setMaxConnections(200); // This line is used to make your connection async ((ActiveMQConnectionFactory)queueConnectionFactory).setUseAsyncSend(true); |
After we’ve made the changes explained above, our queue server and producers became rock stable and blazing fast so there is no point to look for another solution because looks like I was right – ActiveMQ is an amazing piece of software – you just need to learn it before using.
This is it for today, if you have any questions, feel free to ask them in the comments and I’d be glad to answer.