Shawn on the net

Recently I started to run into issues with my load balancing solution.  I was running HAProxy on a GoGrid instance and it was working pretty well.  Eventually though as our traffic went up problems started to appear.  After completing as many networking optimizations as I could it was clear that I needed to find another solution.  I was debating between HAProxy on a dedicated server at ServePath, a self managed hardware load balancer or a managed load balancer from ServePath when another contender entered the ring.

The nice people at Zeus set me up with a trial of their ZXTM load balancing solution to try so I figured it was worth a shot.  Setup was pretty easy well ok very easy on a fresh GoGrid instance, I made some networking configuration optimizations then went about setting up my pools.  To ensure it wouldn’t just completely die with our traffic I set it up as a server under our existing HAProxy setup then just gradually ramped up the percentage of requests going to it.  Once it was almost at 100% I made preparations and swapped it out with our HAProxy system.

There was immediately a noticeable improvement in latency which if you’ve read my blog before will know is very important to me.  The interface was a pleasure to work with as well enabling me to easily monitor traffic levels and issues on our operations screen.  After setup it was pretty much left to its own while I continued investigating the other solutions.

Performance wise it went very well.  During the testing period we peaked at I believe around 4000 requests per second and commonly ran at over 2000 per second for hours at a time.  While we had some slowdowns during this time it wasn’t anything dramatic and probably had more to do with running it on a virtual server than a problem with Zeus itself.

ZXTM also offers a pretty cool ability to move some application logic forward into the load balancer.  Though it wasn’t suitable for my needs I could certainly think of a lot of uses in other situations.

In the end my month with ZXTM was certainly a good experience and I can strongly recommend their software.  As for my setup, while I ended up going back to HAProxy only this time running it on dedicated hardware and it is still going strong.

Zabbix is a network monitoring system which uses a central server and remote agents which report back to it. Today I setup my central server and after much document reading, and Google searching got it all configured and working with one agent. Going forward now that I’m familiar with the system the rest of the agents should be a lot easier to setup. Next task is to configure the screens.

During the server setup I ran into a few problems that you might have as well so read on and hopefully you’ll save yourself a lot of head scratching.

1. Zabbix doesn’t support SMTP-Auth. This is certainly a pain but thanks to this Forum Post it’s not that hard to work around. I started developing a patch to add in SMTP-Auth support and I might finish it off this week but I needed the alerts up and running right away so thats on hold.

2. Hosts, Items, Triggers, Actions oh my!
- Hosts are the systems you want to monitor, these need to have zabbix_agent/zabbix_agentd installed and configured for the server. When adding hosts select a template to get a bunch of default items/triggers out of the box, these will save you a lot of time.
- Items are things to watch IE. the amount of free space on a drive.
- Triggers are conditions of items that cause an alert. For example the amount of free space on partition /usr drops below 100MB
When you select a host template a lot of these will be setup so you’ll probably want to start off by just cloning/editing existing ones.

Thats really about it to get started I have personally just scratched the surface so I can’t go much further but this should be enough to get you up and running fast and give you some extra time to study the docs.

Grab Zabbix Today.

I’ve been working a lot with GoGrid lately as we needed some west coast servers and Amazon doesn’t offer US-west yet.  Maybe I’ve just been spoiled by Amazons offering but I’ve found GoGrid to be almost unusable for day to day work.  A few of the issues so far.

1. OS images – They only offer centos/RHEL for the linux side which I am not comfortable with plus I hate yum so I end up spending large amounts of time fighting the system to configure new servers

2. Load balancer – You get a free load balancer which is awesome! What they don’t tell you is that they are not configurable.  Add a new server to your cluster?  You need to delete the load balancer and recreate it which of course means downtime.  Like 10-15 minutes of downtime everytime you want to add a new server.  Thats just not acceptable.  Even support can’t change this for you when you call up.

3. No stored images – If you customize your servers quite heavily you will be heartbroken to find out that you cant store that image to use on new servers.  No instead on your next server you’ll have to do it all again so write it down.

4. no upgradeability – If you start with say 1 or 2gig servers and realize after a while that you need to upgrade them you can’t.  I spoke to support about this as well and was told to create a new server and delete the old one.  Thats fine if you can use the default image but I can’t so that means 1-2 hours wasted for each server I have to change plus see issue 2 I’ll have to have 10-15 minutes of downtime when I replace the load balancer.

I’ve been working a lot of cloud environments lately including 20 boxes on Amazon EC2, and 4 on GoGrid. I’m looking at Mosso this week.  Once done I’ll post a round up of them all.