Shawn on the net

After my first post about GoGrid and the lack of server images at the time I was kindly advised that real server admins use tools like puppet, cfengine and chef to handle this.  Now if I was a real server admin this would have been quite the blow to my ego, luckily however I’m not and just play one on the internet.  That said this advice did cause me to look into Puppet, Chef and cfengine and I liked what I saw.

I read through the project pages quickly and decided to start with Puppet for some reason.  So off I went setting everything up.  My first plan was to test with one “class” of server that I need to scale up and down often.  Creating my scripts was well somewhat of a challenge, a lot of a challenge actually.  I ended up buying a book on puppet which helped quite a bit and after a few days(Yes days!) I finally had everything working to load and update a server using puppet.  This wasn’t encouraging since I still didn’t feel like I really knew puppet well enough to go ahead and fly through the other server types.  In fact as sad as it sounds I was still pretty confused most of the time.

Puppet did run well though and interfaced nicely with our nagios monitoring so all was good but I was convinced I could do better.  Well not me exactly but rather the people that spend time building this great software and not better exactly but more suitable for my skills and mastery of point and click.

So along comes Chef.  Chef was EASY.  Really easy.  I felt like I flew through setting up my cookbooks and the interface made me very happy as I could show off not only my pointing and clicking skills to my colleagues but also my dragging and dropping capabilities.  Chef soon replaced puppet in our network and things were smooth sailing until the day.  I call it slow Thursday.

Slow Thursday started out normal enough.  Until about 3pm when suddenly the alerts started coming in “Server 1 slow response”, “Server 10 slow response”, etc.  All my servers under Chef were replying dreadfully slow or even not at all.  Now, not being a real admin I didn’t exactly put two and two together and get 4.  No, instead I spent until about 4 trying to figure it all out.  It was only when I decided to kill Chef on one of the servers(with the intention of restarting it) only to get an immediate speedup did I realize the culprit.  Chef was killing my servers.  I restarted the client on the server and it went slow again shortly after.  So I started looking at the server.  After much Googling and hmmm and thinking and restarting of all chef related servers I finally just restarted the whole server and started chef on it again.  Everything on the clients went back to normal.

That was weird I thought no big deal though until it happened again, and again.  Now Chef is still new software so I should and did expect some glitches and this was fine.  It just ment I couldn’t use it at the present time to manage my servers.  I still use it for the most important task the initial configuration.

Scorecard time:

Puppet

[CON] Hard for a newbie like me to configure and manage(I’ve heard good things from more experienced users though)

[PRO] Seemed solid while we ran it

[PRO] Easy to monitor with Nagios thanks to existing scripts

[PRO] Book exists!

Chef

[PRO] Cool interface

[PRO] Lots of easily found cookbooks

[PRO] Easy for a newbie to create cookbooks

[CON] Still has some stability issues

End result: Tie!

Seriously though they are both great pieces of fairly young software.  Many thanks to all the developers on both projects for taking the time to make my life a heck of a lot easier.  I look forward to them both maturing further over the years and getting to use them time and time again.

So quite a while ago I released a review of a couple different cloud providers.  It’s been a few months so I thought I’d give an updated picture.  Like last time I still use Amazon, GoGrid and Rackspace cloud.  Also like last time Rackspace cloud is still my least used though unlike last time Amazon is a very close second.

During my last post I believe I mentioned how important latency was to me.  Specifically latency to a number of services on the west coast in the bay area.  GoGrid still is the best of the providers in this regard and due to it I’ve now moved almost all of my services over to GoGrid.  There is some backups on Rackspace cloud and some applications that are just too painful to move on Amazon still but in general 95% of my servers are now with GoGrid.

I’ll start with Amazon.  They still really are the ones setting the bar in this space and they haven’t slowed down.  I get constant new feature updates from them.  I still use scalr instead of their own console though I have gone in there to play around a few times and it seems pretty nice.  My only real complaint with Amazon is their location and latency to the west coast that I need to be talking to.  The prices on their reserved servers are also nice but I would prefer GoGrids monthly prepay option here as well.  Almost forgot support.  A few months back something went wrong at amazon causing a number of EBS volumes to fail.  Just as I was about to go on vacation no less.  It happened to take down our primary DB which isn’t fun when you’re only stopping into the office to waste time before a flight.  Trying to get any information out of their support was like pulling teeth and their outage page was rarely updated with no set pattern or frequency.  It was also wrong for half of the outage.  They didn’t even admit there was a problem for the first while.  I had to go to twitter to see a lot of other posts about it before being convinced it wasn’t just my systems breaking.  Thats the only time I’ve had to use support with Amazon which is a good sign but the support I received deserves a D-.

Rackspace cloud we use as an offsite backup so it doesn’t get much use but the interface is awesome from what little I’ve used it with all the important information easily found.  Nothing much really to complain about here but alternatively I haven’t used them enough to give me much specifically to praise either.   I’ll leave this one to someone else.

That leaves only GoGrid who took a lot of harsh criticism in my last update.  I’m now using them almost exclusively and a lot.  They still have some problems but I’ve found a lot of good since then so in the interest of saving the best for last I’ll start with the bad.

The first problem for me is their lack of different server images still.  Still only CentOS and RHEL for the linux side.  The ability to have debian or ubuntu based distros has been promised for a long time but no apparent update given.

This leads directly to my second biggest problem with GoGrid is their schedules for releasing features are consistently sliding back.  I work in development so I understand schedules slide but you can compensate for this by adding time to what the developers say.  If they always say next month and always ends up being 3 months just say 3 months and save the hassle of having to explain to me why it’s 2 months late(instead of early if they do actually get it done in a month).  This happened with MyGSI which was pushed back repeatedly and then released with a number of missing features and it’s happened with the release of new server images.

My last gripe is the interface.  It looks cool but has a number of bugs in firefox that make it difficult to work with.  The lack of scrollbars on the main page has ment that multiple times while working on my laptop away from my desk I’ve been unable to see all my servers in the list.  This also effects my ip lists with me unable to see all my ip blocks if the billing side box is open.  They are ment to resize apparently but this has never worked for me.  Maybe a minimalistic interface option?

Lastly is forms.  I understand forms are sometimes needed for things.  But every time I want another block of ip addresses I end up filling out the ip request form 2-3 times for different support people.  Same with the SMTP unblocking form which was resolved when I was able to better explain in an email to support what I needed mail for.

Ok thats it for complaints now onto the good things I’ve found with go grid.  First is support.  They have awesome support.  I work in Australia so I mostly hit their night crew and they are all very helpful and quick too.  Their upstream support is also pretty quick to fix problems at these times as well.

Servers launch fine and fast and once you get past the annoyance of requesting new ip blocks all the time they do actually provision them near immediately.

MyGSI has it’s bugs still but I’ve tried it this week and despite some confusion in the documentation about how I needed to configure the cloud storage first it seemed to work well.

The API comes in handy for a lot of what we do now having so many servers going.

Lastly is the ability to link up to servepath dedicated servers using a private vland.  We haven’t fully utilized this but will be doing so in the near future for their load balancing solutions.

My end result for GoGrid is that it still has a ways to go but has come a long way and the terrific support makes up for a lot of the short comings I’ve found.

As for recommendations.  If you need a west coast setup or you want to utilize their hybrid hosting go with GoGrid.  Otherwise AWS is still the way to go.

Let me know if you have any questions about AWS or GoGrid as I’ve certainly had my fair share of experience on both now and I’ll try to update this again Q1 2010.

So I launched a slicehost slice today and finally got around to installing the copy of JIRA I bought during their $5 sale a short time ago.  Installation went smoothly and I have JIRA up and running with a MySQL backend.

It looks pretty good and I started setting up my first project and getting myself use to the environment when I ran into my first real problem.

I couldnt get it to work with github though nor could I find a way to do it on Google that I could actually understand.  Unfortunately I just don’t have time right now to struggle trying to get it to work and I’m really not keen on switching to svn so for now I’m going to have to shelf JIRA for a bit longer.  At least until someone can give me a solid method of getting to it work with git.

In exchange though I went ahead and started up my lighthouse account again.  I’ll play around with that for a bit now.

Wow it’s been a while.  As for my last race.  I completed it! However it felt too easy to abuse so I decided not to launch it.  I plan to start posting more frequently now.

OpenBounty.org will be a site where people can open up bounties on a variety of tasks.  Initially I’ll be targetting open source software development but I will leave it open for anything from software to community assistance.  Users create a bounty by pledging a set amount of money.  Other users can then contribute to that bounty with their own pledges.  Bounties will need to have an expiry date at which point an unclaimed funds would be returned to the users. 

There will be two types of bounties and the type of bounty set will determine the selection process.  In service bounties,  claimants will write a proposal which will go out to all those who have pledged.  Those individuals will then get to vote based on the percentage of the pledge they’ve committed on which proposal to accept.  Once the task is complete the users will then vote again to determine if the bounty task has been completed.  If it has then the funds are released to the claimant.  If not then it will go to a site arbitrator to try and resolve the problem.  This method of bounty would be useful for community projects like "Clean up the basketball courts".

The second type would be useful for projects like "Add feature X to product Y".  Users create bounties just as before but this time the first step is skipped and the vote only on the completed project.  First to complete it and pass a vote of pledged individuals collects the bounty.  If the pledged individuals reject a claim the claimant can choose to go through an arbitrator.

OpenBounty.com would handle and financiall processing through paypal, google checkout and amazon payments and possibly with direct credit card payments in the future.  It would also take a capped percentage from each bounty to cover costs.

For this project I am giving myself 1 week.  This could be tough as I am expecting to start a fulltime contract this week but I think even working part-time I should be able to get it going.

Site: OpenBounty.org

Deadline: Sunday August 10th as 1pm GMT+10

Goals:

1. All features outlined above except for payment processing

2. Paypal payment processing only

Welcome to Shawn on the net.  What follows is my bio and details about what I plan to do here.  It’s probably not of interest to anyone but I want to put it some where so here it is.

I’m Shawn O’Neill a software developer/code monkey from Toronto, Canada.  I started my career at CGEY in England in 2001 working as a C developer for a pretty big(17 million members at the time) customer loyalty system.  After doing that for a few years I left CGEY and started my own company.  By this time most of my personal development was done in PHP and I had started writing a social networking site for companies that would allow them to leverage employee relationships to assist in sales.  I was also doing consulting and the time and landed a pretty nice gig with Virgin Trains fixing up their station issue tracking system.

Eventually I decided to move back home to Toronto and so off I went.  I landed a job with Keane Canada and Morgan Stanley working on their ETL and distributed testing teams which was ok for learning but otherwise uninteresting as I was told I was being hired to program and I ended up doing server support 100% of the time.  But I did learn a few things about databases so I’m thankful for that.

Around the same time I was launching an online community for models, photographers, and makeup artists with someone else called YourNextX.com.  I did all the development on the first two versions after which it was taken over by another two companies.  That site caused a lot of stress in my life.  I never liked the deal to transfer ownership but was forced into it.  In the end the new companies ran the site into the ground, and one of them made some pretty shady deals on the side while trying to cut us out of the profits.  We eventually got control of the site back but by then it was too far gone and my partner and I couldn’t agree on a future for it so it was wound down.

After I left Keane/MS I started working for AzoogleAds(now EpicAds) which is one of the worlds largest affiliate networks.  The team was all around my age which was new for me (in the past almost all my co-workers were at least 15-20 years older than me) and I had a blast.  I was working in PHP and we always had lots of interesting things to do and great people to do them with.  I worked for Azoogle up to the start of 2008.

After Azoogle I went to work with Evolvenetworks which was developing a dating site called LoveIsSexy.com.  The tech director was my former team lead from Azoogle and a few other developers from there were joining me so it was an exciting time.  The work was fun and interesting and I got to spend a lot of time improving on the scaling and optimization skills I’d developed working for AzoogleAds.  My co-workers were a blast too and were all very talented.  3 months or so after starting my girlfriend and I decided we wanted to take some time to travel.  We settled on Australia and as summer began in Canada and as soon as she finished teachers college we were Australia bound.

Before I left I founded SiteThree Studios Inc with a good friend and business partner in Toronto.  We have big plans for a few projects that should hopefully be out by the end of 2008.

That pretty much sums it up.  As for this site, I’m just going to use it to post what I’m currently doing development wise, and projects I’m working on, books I’m reading etc.  It probably won’t be interesting to anyone but if it is let me know.

P.S. If you want to read about my time in Australia and traveling check out http://shawnsandtawns.com