Dark Pork

I realized that I haven’t posted anything about my thesis on this blog.  Since I am now able to spend most of the day working on it, I am able to make progress a lot quicker.

My topic in a Nutshell is power efficiency in Server Processors.  I am basically looking at both Intel and AMD server processors and comparing them with a performance per watt measurement.  I am using blade server technology from HP.  I am been reading up on the technicial specs of our blades and what the different companies are saying about the technology.  AMD is much more focused on the individual technologies that are implemented in each of the chips.  Intel is more concerned with the overall effectiveness of power efficiency, from the cooling systems to the processor.

A couple things that I have found from just reading.  It would seem from the literature that each company puts and the specs of the processor that AMD would have a better performance per watt.  One thing that currently is against AMD is the lack of a Quad-core Processor.  I know that AMD has released it but HP hasn’t put it into a Blade yet so I am unable to test it.  We will see as we get data which uses more power.

So the company I was working for laid me off on Tuesday. They are closing down the office here in Orem and moving the division to headquarters in LA. So I am out looking for a job again. So I guess I need to figure out what I really want to do when I grow up.

Eventually I would like to be in management. I think a team lead position would be a good place to start. I have done a lot with Nagios and Monitoring systems. I enjoying working with new technology and will solve a problems given to me. I will be posting my resume up next week. If you know of anything out there, let me know. I am not opposed to moving out of state, or even out of country.

So I have been working with the new version of Nagios for this Migration that we are doing at work.  I have been impressed with what has been happening with the Nagios development.  Now I haven’t done a whole lot with Nagios since 2.4 so some of the features that I mention may have been implemented before but I am going to try and check them from he what’s new section of the Docs.

One of the first features that I have noticed is that flap detection actually works.  Now I may have not just set it up right before, but I never got it working before.  This time around, I haven’t done anything special and I am getting flap notifications.  It is nice for when a box is bouncing up and down.

Using Hostgroups as host group members, is one feature I have really used.  Since we have duplicates or more of every type of server,  It is nice to have them all in one hostgroup.  And then instead of name each individual host for the more general host groups, I can just add the hostgroup for each computer type.  That saves a lot of work and makes it easy to add new hosts.

This is one that could have been there before but being able to add hosts to hostgroups in the host definition is great.  Then I can look at a host and tell it what groups it is part of rather than editing the group definition as well.

The Parallelization of service checks is a great addition to Nagios 3.  I know you could do this before with a setting in the config file, but it should have just worked out of the box that way.

As I am reading through the what’s new, I see you can use multiple templates for a host or service.  I might have to play with this one.

Performace gains:  I have noticed that Nagios is a lot more  responsive in the interface.  Also, when I schedule a check in the interface it gets done a lot quicker.

The predictive server dependency checks looks like it is a nice feature.  It will basically start checking dependencies the moment it gets a soft failure.  It will also check all the dependencies up the chain, so you get the alert quicker.

A lot of work has been done on Host Checks.  They now run in parallel which speeds up Nagios a lot.  They had the predictability mentioned above.  They can use active host checks much more effectively.  Lot of other little changes have been added too.

Timerperiods have gotten something they needed.  The ability to rotate around a set period of time rather than just days of the week.  This means you can actually put a pager rotation into timeperiods and schedule admins around it.  I haven’t set this up yet but it is one my list to get done after the migration calms down a bit.

Splunk got integrated into the nagios interface.  We don’t use splunk, but I am considering it.

There are a bunch of other features that were added to Nagios 3, but those were some that I thought important.  It is nice that it doesn’t require a big change for the configs in upgrading to the latest version.  There are a few things that were deprecated and a few things that were added, but nothing in terms of major functionality.

I am also working on being able to have a failover server.  I am going to follow the documentation for this.  I will let you know how it goes once I get it down.

Sorry for the delay in posting, I have been fighting some HP servers.  You might remember that we are working on a Data Center migration where I work.  We had purchased HP DL 140 G3 servers that come with Lights-Out 100i.  I have worked with iLO before and thought it was the same.  Boy was I wrong.  Lights-Out 100i is nothing like iLo.  It only has telnet and a web interface available to start.

We had the ip address put in the bios to manage the servers with the lo 100i interface.  We found out that the basic console redirection is done through the serial port.  This is not turned on by default.  We had to activate the trial version of the Advanced features to get a Virtual KVM to be able to get in the bios to turn this on.  The Virtual KVM needs Java 1.4 to work properly.  It won’t work with Java 6.  I have heard of some people having success with Java 5.  The console redirection stops right before the server tries to PXE boot so you can’t fix things if they need extra parameter or if you need to pass a kernel parameter on boot.

I would suggest that anyone who needs/uses the remote capabilities of servers to stay away from this interface.  It won’t suit your needs.  I am not looking forward to having to rely on this for the next few years until the servers get replaced.  We are definitely getting servers with the iLO interface from now on.

A few months ago at the Utah Open Source Conference, DirectPointe handed out a CD for their Central Pointe Server. This server was a compilation of a bunch of open source software that tied together with a web interface and designed to be a drop in solution. That was the way that I understood from what was said at the conference. I took the free CD and decided to look at it later. The situation came up a few months later. My grandmother was staying in the hospital fighting cancer. The hospital was block outbound port 25. I am sure this was preventing any of the computers inside the hospital from being part of a botnet, but it also prevented my grandmother from sending emails. Since I am the tech guy of the family, it was up to me to save the day. Since I had a machine that had a public IP that just got freed up from what it was doing before, I thought it would be a great project to use this software that DirectPointe had given.

So I installed the CPS. The installation went fine with no problems. Once the system was installed, I was present with a semi gui prompt. It was X or atleast and was a common xdm type login prompt. I logged in as root and found that all I could do was to change the networking information. I was confused. I had to get the manual that was included with the CD to see what was going on. I learned that the system is designed to be all controlled from the Web interface. So I got on my desktop and launched a browser to the server’s web interface. It is a nice web interface. It isn’t very confusing and has lots of options for you to work with. I was impressed with all the features that were available. It has Postfix, Apache, Spamassassin, Snort (IPS), SnortSam (IPS), a firewall interface, printing, file shares, mysql setup, etc. Basically anything a standard business would need from a server, it is included.

I started by trying to setup port forwarding to redirect a high level port to port 25 as this seemed like the easiest possibility. I went in to the firewall section and set up the port forward. Easy I thought. I then tried to telnet to the high level port in order to send an email and test the functionality. At this point let me tell you that there are two modes that you can can choose when installing the server. There is the standalone mode (which is what I chose) and there is the gateway mode. Well my telnet test didn’t connect. So I tried to setup the port forwarding using the advanced section of the firewall. Again, It didn’t work. After a couple of tries, I couldn’t navigate around the interface anymore. It would just timeout. I got back on my console to the box and after a little while got on a command prompt and found that snort had locked me out since I was using telnet to go to a non-standard port. It thought I was a hacker and locked me out.

Once I got back into the system, I decided that maybe I should register the machine and see if there was an update or a bug with why the port forwarding wasn’t working. I signed up for an account on DirectPointe’s website. I found that I couldn’t add a computer because I didn’t have a subscription. Odd I thought. I could register the system to my account for the same reason. I also couldn’t find a place for updates. Around this time, I felt I had spent enough time messing around and I needed the system to be up and working. I installed Ubuntu and set postfix to listen on the high level port.

At this point, I thought I was done with the Central Pointe Server. However, I get an email a few days later from DirectPointe. They want to see how I thought of the system. The offered to take me out to lunch. Free lunch, who can say “no”. So I went. I was really surprised how they were. They listened to my whole story. Then they asked how it could have been better. I was floored. I was expecting them to come and tell me all the things I did wrong and how it wasn’t their software’s fault. I was impressed. Needless to say, the port forwarding didn’t work because I was in standalone mode rather than gateway mode and port forwarding isn’t enabled in standalone mode. That make sense but it wasn’t written anywhere. They told me that this was a new product and that is why I couldn’t find a subscription for it and why I had troubles with registering the system. I came away from the lunch impressed with the company.

Overall, I liked this product. It has a lot of capabilities. It is one of the best packages I have seen with everything up and running. If I was a business owner without a full-time IT staff, I would definitely be looking into one of these servers. However, as a Geek, I didn’t have the same mentality that this product did. That being said, I think this system has some great potential, and with a few tweaks could even satisfy the geeks out there. Now we just need to wait for the ability to download the iso right from their website and keep it updated.

After looking over the traffic that is coming to my site, I realized that my problem that I mentioned with the volume on the Blackberry Pearl is a common one.  I have seen lots of searches for that phrase that has brought people to my site.  I thought I might mention how I got my volume up on my ringer.  First you need to go to the Speaker Icon to change the ringer profile.  Then scroll all the way down to the bottom.  There is a divider and then one option called “Advanced..”.  Inside of that option, you can change your profiles as well.  If you highlight the one that you use and click the menu button, you can go in and edit the profile.  Then you have a choice of a bunch of different type of alerts.  The most common is the “phone” section.  This is the calls coming into you.  The SMS text and MMS are both common ones to change as well, since that is a text messages and a media message (a text message with a picture or sound).  If you click on the menu button and edit the option, you can change the volume as well as the number of vibrations that will happen on the chosen event.  You have 5 options for the volume: Mute, Low, Medium, High, and Escalating.  I usually set it to High.  I haven’t tried the Escalating, but since I use 2 vibrations, I am not sure it would be loud enough before the call was sent to voice-mail.

One final note that I wish was included is the repeat notification is only able to flash the LED.  I would like to be able to have it beep at me until  I checked somethings, especially since I am on-call a week a month and I need to respond to text messages that come in.  If I miss the first alert it might be awhile before I get another one.  Maybe I need to send some of these ideas to RIM.

I still am waiting for the update to fix the memory leak on the 4.2 software.  I have to pull the battery out occasionally to get it back to a usable state.

I found this company through a couple of different programs that I was looking at out. This is there complete Nagios/Cacti/Monitoring solution. It is much more than Nagios. Nagios maybe a portion of the backend, but they have integrated a number of opensource projects into this compilation. I was really taken by this project initially. It had everything I needed. It had the monitoring stuff and performance graphing. I was excited. I installed it and went through the basic configuration and just found it lacking. Maybe I just didn’t have the time to play around with it as much as I wanted. I think though I was put off by the philosophy of the program. They have a Pay program that is built on the opensource program with a lot more features in it. This is not the way we should be doing open source software. I understand and support selling support and services in addition to open source software, but I am put off when a company tries to take an opensource version of a product and limit it because they have a version that they sell.

For example, GW has added this concept of templates that go beyond a nagios template. The template sets up everything about the server, from monitoring to graphing and the like. Then you create the server with a name template and ip address. Done. Easy to add multiple hosts. And hosts can have multiple templates. However, they include 3. Thats it. One is a basic SSH template and I forget the other two. Templates are hard to create as well. I looked through creating one and got confused. Why not include more templates? Because there are tons in the paid version.

Groundwork Monitor uses a lot of Ajax for its web interface. It makes it smooth and enjoyable to use. Setting up hosts was a little complicated, since you weren’t setting up the host for Nagios, but also for Ganglia and other tools that are packed into this compilation.

Overall I liked this package. I don’t think that I will be using it in production, however. I am not sure how much I will follow the development as well. It will be interesting to see where this project ends up in a few years.
One of the other projects they Groundwork owns is called Fruity. Fruity was a configuration tool for Nagios. The author of the tool as recently left and is creating a fork since groundwork is sitting on the tool and not updating it. I am looking forward to see what this fork brings. Fruity looked like a great configuration tool.

OpsView is a Nagios wrapper found here: http://opsview.org/ . It is an impressive add-on for a business wanting a basic monitoring solution. There is a Web gui that allows you to configure Nagios with out touch a config file. They have added a few cool features like categories and the ability to have slave servers with hot failover. This means you can keep one central server that has on the configs. All you need to do is modify this one server and it will push out configs, scripts and what out to all the other servers that are setup in the interface. The interface is simple and easy to use. I was impressed as I started to use this.

Once I started to try and configure things is where I started to see problems or rather things that didn’t have enough polish. One of the biggest thing for me was that the web interface was slow. It took forever (10-15 seconds) to save a changes to a host or a service check. I had 40-50 servers that needed modification from the import of the current config. I got though a handful before I started editing the database directly. I could handle this problem because once you have all the machines setup you don’t need to do a whole lot of changing.

There were a few confusing points to the whole thing. Opsview uses host categories and service groups to figure out the contacts for each alert. I couldn’t find a good way to see how these were setup. It was confusing. It seemed like all the users that are associated with a server category got notified based of what the service was told to notify. Very confusing.

They don’t have support for host templates. We use these a lot in my environment at work. We have a bunch of servers that do the same thing. There are times when I need to make a change to all of them. This is where the Host template comes into play. I can just make the change to the host template and it gets changed on all the servers using that template.

A couple other things were issues for me. The Slave servers had to be setup from the master. You setup ssh keys and push out a script. You can’t install the software and say “you are a slave to X” and tell X that the server is a slave and have it work. In addition, I had to spend a bunch of time trying to get the packaging to work with CentOS 5. I eventually got it and reported what was needed back to the developers. Just frustrating. One last thing that is an annoyance, is that all the nagios scripts have to begin with “check_”. A lot of the custom scripts that we use don’t use that convention, since it makes more sense to call them something else.

One nifty feature that OpsView added to Nagios was the exceptions. You can specify a check with a given parameters and change it for a given host or a given time period. Very nifty feature. Not one that I currently need, but I can see the benefit for it.

Overall, I though this was a great package. I was impressed with the overall layout. I am sure that if you bought the package with the support, it would work a lot better for you. Since I am fairly knowledgeable with Nagios already, it had a few shortcomings. One of the best positives for this program is that the developers are fairly active in the Nagios community. I have seen them respond to people on the user list and submit patches back into the main Nagios line. This is how projects like this should operate.

I have been looking at add-ons to Nagios. At work we have a decent number of servers to monitor. I wanted to see if there were some products out there that could help reduce the management of our Nagios installation. There are three main additions that we wanted to add to our current setup.

The first is Failover between Nagios servers. This is for redundancy. We don’t want each system to be running checks but we want to keep them in sync as much as it is possible. I have read through the documentation and seen how Nagios has said how to do it. There was also a paper out by Ricardo Martins that uses perfParse to increase the sync between the two servers. After looking over the different solutions, I am still not sure how we want to do it. That is why I am also looking at alternatives.

The second benefit we would like to see is an easy administration of the config files. Since we have three different environments that we operate in, it becomes messy to maintain them all. I am trying to make it easier so that we don’t have to go and hack on the files all the time. This is becoming less of a possibility. There are a few solutions that I am still looking at. Ultimately we would like to use our HostDB to dump out Nagios config files.

The third benefit would be to set up a dedicated on-call rotation that would alert with out any manual interaction. Unfortunately, I haven’t found anything that would do this for us. It might have to be a hack to get this to work. We would need to setup escalations as well so if the on-call person didn’t respond in a given time, the rest of the systems team would be notified.

So with those in mind, I am in process of looking at different Nagios add-ons to improve our Nagios environment. I am going to try and review the ones that I test. Those should be my next few posts.

Well, I got a new Blackberry Pearl on a couple of recommendations. I have had some mixed feelings on it. I finally got it to sync, using schedule world and OTA (over the air) syncing that I found in Ubuntu’s forum. I tried to use the open source project barry to sync it directly but I couldn’t get it to recognize the device. I thought it was charging but I don’t think it really was. It is a shame. The project has some far. It might be that I am running on 64-bit Gutsy, but I don’t think so. They are getting there. I am excited to see the development of it.

So then you ask how do I like it. Well it is different. I have done a lot of other things to make it easier. The syncing is a big thing. That also meant that I had to update all my contacts to include phone numbers. That was good since I didn’t have a lot of contacts in my computer. Now I do. I need to add some more, but it is nice knowing that there information is there. It also crossed off another thing on the list. Speaking of lists, since the syncing doesn’t do todo lists I haven’t done that yet. I am still getting used to the keyboard layout and input more that a basic text. It works great. I am also getting used to having an extra column of keys on both sides of the numbers. I was dialing a number today and failed 2 or 3 time because I pushed a letter key and not the number key.

I am disappoint in the volume of the ring. It doesn’t seem very loud. I wish it were loader. Maybe I need to tweak a few things. Plus when the phone is on the home screen and you push the volume up, it should turn up the ringer. It just sits there. One other annoyance that I have is that if you have the keyboard locked, and a phone call comes in. After the phone call, the keyboard is unlocked. Something I wish I could switch. and maybe I can. I haven’t found all the shortcuts yet. I am sure there is one for locking the keyboard, but I am still going through the menu to get it.
I installed mini Opera on it and have been very impressed how it handles. It is cool to see these webpages really small. My years of being on broadband really hit home though. It is tough to wait on the download speed. I guess it would be better if I had tabs or multiple browser windows, but only one page at a time. The good thing is no popups.

I love that alot of the features are just buttons in the menu. Like the keyboard lock. It doens’t ask for confirmation. One added thing that would be nice is to pop up the text message so you could read the subject if the keypad was locked. I haven’t set up any extra email yet. I did download and use the gmail app. It is nice. Not many other apps have worked. I think it is because it is the new version of BBos. I think it is 4.2.

A feature I really like is the voice dial. No more going through menu’s. I push the button and say “call jessica mobile” and it calls. Sweet.

Well that is it for now.