Opsview Software

Mocking an SNMP agent and developing SNMP checks

We regularly get customer requests to write snmp checks for their devices but a problem we have is we don't necessarily get access to those devices - in the past we have used output from an snmpwalk command and written the check scripts as best we can, and then shipped the new check to the customer for testing.

This doesn't sit very well with me as I don't think this gives a good customer experience.  Why should the customer do the testing when they have paid us to write the code?  How much time and effort has the customer got to put in to then get all debug data back to us to improve the check?

Well, there is a better way, by using SNMP::Persist

The snmpd agent on most unix type systems can be extended such that a script can be used to respond to all snmp requests for a particular oid tree.  This can be done by creating a script such as this:

#!/usr/bin/perl
use strict;
use warnings;
use SNMP::Persist qw(&define_oid &start_persister &define_subtree);
define_oid(".1.3.6.1.4.1.123.123"); # base of OID tree
start_persister();
while (1) {
 # define all sub OIDS within the base to respond to
 define_subtree(
 { '1.1' => [ 'STRING', 'string value' ],
 '1.2' => [ 'INTEGER', 9 ],
 '1.3' => [ 'Counter32', 342.3 ],
 }
 );
 sleep 300;
}

and then add an entry to your snmpd.conf such as

pass_persist .1.3.6.1.4.1.123.123 /path/to/perl/script

After restarting your snmpd agent you should now be able to query the oids

$ snmpwalk -v2c -c public localhost .1.3.6.1.4.1.123.123

While the above isn't in a fit state to use for TAP tests, it shouldn't take too much imagination 
to extend it by using 'pass' instead of 'pass_persist' in NET::SNMP in a small server started 
when running the project tests. 

Faster Nagios CGIs

The great thing about open source software is the unexpected contributions you may suddenly receive.

Jonathan Kamens of Advent Software sent a patch into the nagios-devel mailing list about a speed up to the status CGIs. He identified, using gprof, that status.cgi was taking the most time in the sorting routine of downtimes and comments.

His patch was simple and effective - instead of trying to sort the order of the objects every time, just do it once at the end.

We've been looking at the patch and produced some statistics of the old behaviour versus the new behaviour. The times can be found from running the t/660status-downtimes-comments.t perl script within the Nagios code.

Comments and downtimes	Nagios 3.2.0	With patch
10	0	0
100	0	0
1000	1	0
10000	14	1
100000	1680 (= 28mins - and it didn't finish!)	53

For 100000 comments, the status.cgi was taking 28 minutes just reading the current status data - and in the test it didn't actually finish! With Jonathan's patch, this reduced down to 53 seconds.

There's probably still some work to do as an increase of 10 from 10000 to 100000 increases the execution time by a factor of 53, but this is a good start!

There seem to be some work in looking at alternative ways of getting status data, including NDOutils database backend, or mklivestatus to get the information directly from the Nagios daemon.

For Opsview, we're betting on the database backend, because there are other advantages (arbitrarily complex search criteria, historical information, separation of work from Nagios instance), although we like to provide the old style Nagios CGIs as well, so we'll be adding this patch to Opsview. Funnily enough, the Nagios users get the patch earlier than the Opsview users do! But that's all part of working with open source software and improving the tools for everyone.

Nagios scheduling bug

This Monday morning, we got lots of calls from our users where Opsview slave systems running Nagios were raising freshness alerts because checks weren't being run within their specified period.

Suspiciously, this was also the weekend where clocks went back one hour due to daylight savings changes.

Hosts and services were meant to be scheduled in 5 minute intervals on our internal system, were marked to be run on 26th October at 23:00 instead of 25th October.

So there seems to be a problem within Nagios where the rescheduling has been adversely affected by the clock changes. We expect this will affect any countries still using Daylight Saving Time when they revert over the next few weeks.

This affects all active checks on all Nagios systems, not just a distributed environment.

We're going to investigate this more deeply within Nagios because this definitely worked fine over the last few years. From the Nagios mailing lists, it seems to be in Nagios 3.2.0, but not Nagios 3.0.6. However, it has affected Opsview systems that run Nagios 3.0.6 and I think it is due to this patch which we brought forward from Nagios 3.2.0.

The workaround at the moment is to recheck all your hosts and services again. In Opsview 3.3.2, you can use the Mass recheck functionality.

Alternatively, you can use this script which we've written. Download it, put it in /usr/local/nagios/bin and run it. It will submit a SCHEDULE_HOST_CHECK and SCHEDULE_HOST_SVC_CHECKS for all hosts in your system, using the objects.cache file to get all the host names. There is a random 5 minute difference applied, so that not all the checks will run at the same time.

You may need to adapt the script if you use different paths, but it will work on all Opsview systems.

In a distributed environment, you only need to run this on the Opsview master server and the requests will be sent to slaves automatically.

Update: We've been testing this and have found the following results:

• If you use a timezone such as Europe/London, then the bug was triggered between 2009-10-25 23:00:00 and 2009-10-26 00:00:00
• If you use a timezone such as America/New_York, then the bug will be triggered between 2009-11-01 23:00:00 and 2009-10-02 00:00:00
• If you use UTC, then the bug will not be triggered

Basically, it occurs on the day when the time for the nagios daemon goes back an hour.

We've managed to recreate the bug in a libtap test, so we're working on a fix to Nagios.

Update: We've applied a patch to core Nagios for this fix. Using our tests, we've found that the bug happens only during the 23:00 to 00:00 on the day when clocks go back in time.

Since it is an automated test, we ran the test checking every half an hour, a million times and ... it started to fail ... in 2038. This is a known Year 2038 problem. I think there will be changes to Nagios before then :)

Opsview and the Open Core Model

Opsview ® is the award winning Open Source application and infrastructure monitoring software from Opsera. It provides comprehensive system management capabilities used to monitor the health of today’s most complex data centres. 

Opsera have recently adopted an Open Core business model for their Opsview software. This article explains why we decided to take this direction and the benefits to Opsview customers. 

Traditional Open Source business models 

The traditional Open Source business model is to give the software away for free and sell services such as support, training and consulting. This approach can support a small team of developers but relies on good community involvement and donation of code but does not offer the resources to provide major enhancements to the software. 

Feature sponsorship is part of the answer to this, but it is rarely possible to secure funding from end users for the entire development effort, especially as the software matures. 

Alternative Open Source business models have emerged in order to address these issues. Some are considered more ‘open source’ than others. 

Sell hardware: Companies such as Sourcefire and Vyatta sell their software packaged in the form of a server. Their customers are happy to pay for the software when delivered in a tightly integrated and fully supported form. Since hardware is a physical asset it is often easier to justify these fees than services and support fees alone. 

Dual licensing: There are several variants of the dual-licensing model but essentially the software is released simultaneously under both Open Source and proprietary licenses. MySQL sells their database software to customers who don’t want to follow the GPL license.  

Other companies such as SugarCRM offer their premium level product under a proprietary license, while releasing the basic version under an Open Source license. The cynical view is that the Open Source ‘lite’ version exists for marketing purposes only. 

The ‘Open Core’ model 

The Open Core model shares similarities to the dual license model but it also addresses some of the shortcomings. The Open Core model also allows a reasonable balance between the commercial aims of the software developers and the needs of the user community. 

The Open Core Licence model is based on providing a free core version of the software containing fully featured and comprehensive functionality and then extending this with specific functionality in paid for modules. Usually the software ‘core’ contains frameworks and other common components used by the modules. Over time the functionality provided by modules may be integrated into the core product or the module is made available for free. 

The Opsview Approach 

Our direction was inspired by the ‘freemium’ model adopted by many Web 2.0 companies. Their strategy is to build a large user base around a free product then offer premium priced services to those users who desire the extra value, functionality and risk management these services provide. 

We offer a common feature set between each software edition and differentiate between Community and Enterprise versions via the software release process and service offerings. Opsview Enterprise contains mature features, has a predictable release schedule, a clear road map and is well suited to large, business critical monitoring requirements. Opsview Community is aimed at startups, developers and anyone wanting to evaluate our software or try the latest features. 

The functionality offered by the Enterprise version is also extended through Modules. These have the following characteristics: 

• Modules are free standing software with their own feature set and documentation 
• Modules provide functionality aimed at large systems and enterprise customers 
• Modules provide integration with other systems or software 

By adopting this model for Opsview we are able to serve our enterprise customers by delivering a software edition targeted at their need for maturity, predictability and accountability; and our community users by continuing to develop and release a free, fully featured product.  

Conclusion 

The Open Core model provides Opsera with a sustainable business model and solid platform for the continued development of our products in a manner that benefits both our enterprise customers and user community. 

Attribution: This article was first published on the Opsera corporate website and a PDF version may be downloaded from here: http://www.opsera.com/downloads/article_open_core_model.pdf

Opsview's relationship with the Nagios project

We have a long history of working with Nagios. Most of our technical staff has used Nagios for many years – both here at Opsera and at previous employers. Our Opsview product is based on Nagios because we believe it is the best open source monitoring engine available.

The recent public fork and ensuing debate has raised potential concerns about the health of the Nagios project. To help out, we have provided more direct support to the Nagios project by committing changes from our code base into the central Nagios code repository. We see this is a mutually beneficial arrangement as we get our changes incorporated into future versions while all Nagios users get the benefit of bug fixes and enhancements. We will of course watch with interest to see how the Nagios code fork fairs over the coming months.

Opsera are committed to work with all open source software for the benefit of our customers and the community and we will continue to work in developing Nagios. As Ton Voon, our Opsview architect, and Duncan Ferguson our Release Maintainer are now officially part of the Nagios development team, we hope to positively influence the future direction of Nagios', to make both Nagios and Opsview a better product for everyone.

Opsview: The Next Chapter

Back in October 2008, Opsera acquired Altinity on the strength of the Opsview product and our customer and user base. It was a great marriage as Opsera were providing IT consultancy and wanted to expand into products, of which Opsview is now added with Ops Mail Manager. And there are some really smart people working at Opsera, which is always a good thing!

Since then, we've been really busy integrating Altinity's IT systems and procedures with the rest of Opsera, taking the best practices from both companies. And with Opsera being a larger organisation, we've had more resources to handle the large number of new customers and work requests.

So it gives us great pleasure to announce the availability of Opsview 3. We've been working really hard over the last few months to get this release out.

The number one question we had towards the middle of last year was: "When will you use Nagios 3"? This was quite funny because we've been involved in getting lots of our changes into Nagios 3, as we've worked with the community on enhancing the core software. But there have been some other great changes in Nagios 3, so it's good to get back to the edge of the Nagios development cycle.

(As an aside, we used to have 39 patches that we made to Nagios 2.10, but against Nagios 3.0.6 we only have 26 patches - our aim is to have as little as possible. We've also seen one problem during our testing which we've contributed back and has been included in the main code. We've got some more patches to Nagios which are good for general use, which we'll blog about soon.)

So, Opsview 3 is here now, based on Nagios 3 and lots of other goodies. And it is all released under GPL.

You can see more about Opsview and our unique take on status views and configuration in our new Quick Start guide.

Enhancing NRPE for large output

NRPE is great for getting plugin information from a remote host. We wanted to use it to get passive data regarding events, such as syslog entries that SEC had highlighted. This meant we needed two things: multi-line support and larger amounts of output.

Multi-line is already in NRPE 2.12 - this was added by Matthias Flacke last year. However, the limit for data is 1K.

We wanted to be able to bump that figure up to 16K. There's a common.h variable which is called MAX_PACKETBUFFER_LENGTH which is set to 1024. We found we could increase this value and then more data was returned. But there were two problems with it:

• it broke backwards compatibility


• it increased the size of each packetk

The 2nd had an impact on the network. Instead of 1K packets being sent between client and server, we now got 16K packets sent, even if the data contained was small.

The first was worst: it meant you needed to update the client (check_nrpe) with the server (nrpe) at the same time, otherwise you'd get lots of NRPE errors in Nagios with only one change.

So we've designed a compatible way: we've added a new packet type called RESPONSE_PACKET_WITH_MORE.

The idea is that check_nrpe will see if the packet returned is of the type RESPONSE_PACKET_WITH_MORE. If so, it will read subsequent packets and append that to the existing data, until it gets a RESPONSE_PACKET. So to read 16K worth of data, check_nrpe reads 16 x 1K packets. Of course, only updated nrpe daemons will send this, so this remains fully backwards compatible with existing nrpe daemons.

The patch is here. We've also cleanup up some of the graceful_close calls.

Now the process to update your NRPE agents would be:

1. update the central check_nrpe, then


2. update your agents at your leisure

And you won't get any alerts during this period!

Note: during testing, we found that the limit for returned data from some linux kernels was 4K, even though nrpe was coded with 16K as the limit. This is due to kernel limitations in using pipe() for the interprocess communication.

NDOutils on Solaris 10

Michael Prochaska was having trouble with compiling NDOutils on Solaris 10. Since we have an interest in getting Opsview working on Solaris (the upcoming 2.12 release will add Solaris 10 as a supported platform), we offered to help. So this is the result of his company, Bacher Systems, sponsoring our work.

Monitoring Cisco Netflow Data

Netflow is a great feature of Cisco IOS that allows you a view into the traffic that flows over your Cisco network devices, what that traffic is, where it came from and where it is going.

We wanted to make good use of this information and so we started looking for a way for Opsview to monitor it.

With a little configuration of IOS and some open source magic we achieved just that. Now our Opsview servers are keeping tabs on the data moving across our Cisco devices.

So true to our open source way of life we published our setup as part of the Opsview documentation.

NSCA's aggregate writing

In our continual task to try and speed up Opsview, we found a bug in NSCA's handling of aggregate writes when run in --single mode.

The specific failure scenario is this:

1. NSCA and Nagios are told to start up
   
2. A send_nsca request is received by NSCA before Nagios has created the nagios.cmd command pipe
   
3. NSCA tries to write to open the command file, but sees it is not there
   
4. NSCA opens the alternate dump file instead
   

Now when Nagios does create the nagios.cmd file, NSCA uses that ... unless aggregate mode is on and daemon mode is --single. In this case, it continues to use the alternate dump file, thus Nagios doesn't see the results from the slaves.

Here's the patch, which we've also added into our source for Opsview.

As we are very keen on good testing, we've managed to recreate the failing behaviour in a test script. You also need a test configuration file and a patch to the test framework. If you run this test, it will show the error and then after the patch is applied, the test should pass.