It all starts with a bang
Well, obviously long time before I actually knew about anything like Linux or Nagios. Shortcut – First appearance 1999, trademark issues with the name ‘netsaint’ in 2002, renamed to Nagios which is some acronym for whatever (google it). Versions 1.x and 2.x, simplicity with a plugin api capable for everyone, fetched attraction by many users. We can read that on Google, history, done.
Update 2016-01-15: It is still going on. Now with the Nagios::Plugin Perl CPAN module and legal threats. More to read over here.
Update 2014-01-24 09:29 CET:It seems that the drama continues before it ended, so if you came here reading about “Michael Friedrich”, “Netways” and trademarks, as Ethan G. outlined in his comment on a rhel bug, here’s some additional vita (the rest can be found on XING and LinkedIn). 30, Austrian, 2009-2012 Admin/Developer at the University of Vienna, 2012-2013 Senior Consultant at Netways, 2013-present Application Developer at Netways. Lead Icinga Core Developer since May 2009.
If you’re wondering why last names are abbreviated – I prefer not to expose the possibly offended characters too much. It’s become a psycho drama.
Once you’ve read all through, you can happily start at the very beginning here to read again. Look for the 5th ace and let me know. I’ll happily grant you a beer or two, if you find it.
It’s a drama loop now.
Chapter I: How to (not) deal with problems & fork you
I figure that at some point, when companies have had put Nagios into their monitoring stack, selling consulting, workshops, whilst the original author just wrote code, it got problematic. Especially when people are demanding too much, and provide nothing in return. I do have that feeling sometimes with Icinga users as well, but that makes us human people to tell everyone about it, figure out the input both sides may provide, and reject a feature request, if it does not work out. Still, to remark – listen to the community, and answer.
Start your own business, company, etc is completely legit. The problem arising – community requires love & feedback. If there is none, you really should consider giving up your one-man-show and open up for more developers. While Ethan G. claims that Icinga is “my fork”, it apparently is not. We generally do not own projects. We instead appreciate being part of it. I joined the team one month after they’ve forked from Nagios, looking for a way to integrate Oracle into the *DOUtils backend. The dead nagios-devel mailinglists had made it clear already – a fresh fork with motivated people would allow me to participate and bring my patches upstream.
Basically that fork just happened when everyone’s local fork (Nagios with patches) was growing too big and unmaintainable. It’s still reasonable to do so, also because Ethan G. also suggested forking to users in the past being annoyed by the not-very-reponsive leadership ten years ago.
Chapter II: How to (not) deal with problems & be a drama queen
In Europe, especially Germany, it’s quite common that trademark trolls register trademarks for existing open source projects, and forcing them to rename their projects, and lose their corporate identity and name within the community but also the search engine’s index. I do believe it when Julian says (and told me in person) that he decided to register (and buy) the Nagios trademark in Germany to prevent that from happening. And it all worked out quite well.
- NagVis was born in 2004, designed by German developers.
- In 2006, PNP4Nagios was created. The most famous Perfdata Graphing Addon based on RRDTool. Hosted in Germany.
- The Monitoring Portal was created in late 2003, operated by independent Germans. This is were the community meets. Statistic from 23.1.2014: “12,226 Members – 29,644 Threads – 197,306 Posts (average 51.04 Posts per day)”
- Monitoringexchange (former Nagiosexchange) operated by Netways offers 2000+ plugins and addons
Trademarks are not free. They need to be paid. So Ethan should’ve been grateful not to pay those fees in Germany afterall, and getting supported by companies caring about the community, letting them work on their spare time projects, and not meaning any harm.
Instead, the bitching about trademarks started. First off, it was about nagiosexchange.org (now monitoringexchange.org), then related to the famous Nagios conference (now Open Source Monitoring Conference) and other brands which were demanded to be handed over to Ethan himself. It ended all with misleading information, a smear campain on twitter.
Announcements by Nagios Enterprises: I II
At that time Icinga was already forked, and gained attraction by the existing Nagios community. Furthermore, Icinga is supported by Netways with server hosting and development manpower. Which obviously played a role in this fight, but it’s not that clear.
The outcome for the Nagios community in the end was
- People started to know Icinga
- Nagios Business Process addon lost its domain, and it took a while to let search engines adopt bpaddon.monitoringexchange.org
- the Nagios Conference was called Open Source Monitoring Conference and attracts more participants looking at Nagios & forks, OpenNSM, Cacti, Zabbix, etc
- Confusion between exchange.nagios.org and monitoringexchange.org
- Bookmark update fun
In the end, we also had the Nagios drama mailinglists announced.
Chapter III: How to (not) deal with problems & act like a child
At some point, Jean Gabes proposed his rewrite of Nagios in Python to the Nagios Developer’s list. While it was to be expected that Nagios developers wouldn’t just switch from C to Python, the reactions after making Shinken a “Nagios compatible rewrite” were a bit strange and childish. Especially in terms of approaching other people. Or attacking even.
Only quoting a passage here:
blah, blah, blah,
You heard about the drama on the mailing list but yet you are still busy creating more. If you truly believed that flaming is nonsense then why do you initiate it? It is very humble of you to request the flaming to stop after you have had your inaccurate dramatic statements posted. I suppose you thought your statements would not be challenged because we usually do not respond to ridiculous rantings of a seemingly angry person who jumps to inaccurate conclusions and makes rash assumptions.
Your reason for forking lacks substance. Nagios is an engine. A very powerful engine that has gained in popularity because Ethan Galstad has been a very successful programmer and gatekeeper of the project. If you are not happy with the progress of the project, so what! You are just one person . There are millions of Nagios users that are very happy with the Nagios engine. We do not feel the need to cater to one or a few angry people that are jealous of Ethans success. Your lack of respect for Ethan is appalling, knowing that Ethan spent 10 years of his life developing Nagios and cultivating the community.
You “forkers” try to justify your actions by saying it is for the good of the community. The rest of us know the real reason is to satisfy your own egos (really twisted). You must feel unethical about what you are doing or you would not have tried to blame us for your actions. IMO, you forked because you and Gerhard lack the skills and creativity to start your own project. You would rather “steal” someone else’s work. Further evidence of your lack of creativity is posted on your own website where even your tag line is stolen from Nagios.
It is time to stop the negative, inaccurate, flaming drama directed at Ethan and Nagios. Do your own hard work and stop attacking Ethan and the work that he has done for the last 10 years. Please share my response with your partner in crime, Gerhard Loser.
That doesn’t sound very constructive, and contains plently of personal attacks. Gerhard’s last name actually is spelled “Laußer”.
Chapter IV: How to (not) deal with problems & compare yourself
At some point, your customers ask you about your competitors. Or community members even. How do you compare to XY? We had that with Icinga a lot, and therefore created our version of it. While it may make Icinga look good, we’ve learnt one thing: The previous version compared Icinga with Nagios Core and Nagios XI (the commercial entity). While Icinga is free in every single line of code, Nagios XI isn’t. We decided to skip that part, and removed that column from the Icinga comparison chart.
After a while, we came across this interesting OpenNMS article talking about Nagios FUD. And we figured, that Nagios Enterprises had created a comparison chart for Nagios vs Icinga themselves. But actually that one lacks of any substance, comparing Icinga Core with Nagios XI (the commercial one), but only flagging it Nagios. At that time it was unclear that “Nagios” means “Nagios Commercial Version”, while “Nagios Core” follows the free open-core model.
Either way, the pdf also contains false information about IP violation done by Icinga and Netways. Both never have violated any intellectual property, but probably that’s the only arguments to tell managers and buyers that Icinga is a risk, while apparently it’s just pure hate against the fork, spreading FUD themselves all over the internet. How_Nagios_Compares_To_Icinga of that pdf, copyright both pdf and the quote below owned by NE.
Icinga is a young project designed as incompatible Nagios fork by German company Netways GmbH. Icinga suffers from high developer turnover and poses legal risks to organizations that deploy it.
Both Icinga and Netways GmbH have a history of intentionally violating international intellectual property laws. You may be putting yourself and your organization at risk if you deploy or implement Icinga in your organization.
Knowingly using or working with products or companies that violate intellectual property laws and treaties can have significant legal and financial repercussions.
After reading this bullshit, and not liking the icinga.org Matrix either, the developer in me decided to invest several days to create a comparison from a developer’s point of view. Since many developers don’t necessarily mean bad project quality. And quite frankly, being the release manager since 0.8.3 4,5 years ago, it’s still annoying to see that they advertise their lies publicly.
Btw – there are other comparisons with Centreon, Groundwork, Opsview and Zabbix.
Chapter V: How to (not) deal with problems & censor the internet
Wikipedia is an open-minded community, enforcing free speech in their articles. The english article about Nagios contains references to its forks. Some people obviously influenced or working for Nagios Enterprises (whois said so) have had a different opinion on that, and tried to censor that Nagios article by removing all unwanted strings and urls. Or, they’ve added subjective meanings into it, like “Nagios XI is the best monitoring tool in the world”. That did not end well for them after several attempts of fraud on that wikipedia article. It got even worse – Wikipedia banned Nagios Enterprises being sockpuppets.
And – trademarks join the drama queens the first time. The French Nagios community was formed by Olivier J. at former nagios-fr.org also talking about Icinga being a Nagios Fork. It seems that Ethan G. and Mary S. (wife) didn’t like that much, and attacked Olivier in a way, well, no words for that. But go figure, transferring the domain and changing the community portal to monitoring-fr.org was the only wise choice. And it’s still a good community portal for everyone, not only Nagios. And you may not censor their articles, but read them – Chrome offers auto-translate anyways.
Chapter VI: How to (not) deal with problems & treat community members like shit
In early 2012 there was an interview done by freesoftwaremagazine.com with Ethan G. and Michael L. – both sides added their arguments, comments, etc. While the article was great, I felt that it required some more additional content (some of it is found in this article as well now).
A short time after that article, Michael L., the former owner of nagios-portal.org got a letter from ICANN that Nagios Enterprises has claimed the trademark on the domain, and that the “Uniform Domain Name Dispute Resolution Policy” now applies. .org TLD disputes are directly handled by ICANN, while cTLD disputes for example for .de are handled by the local domain registry “DENIC” (and not even that, but this is handled under civil law in Germany). That ICANN procedure is pretty annoying and for non-native English speaking people a mess without lawyers.
At that time, it was already known thanks to the “Julian incident” that trademarks are now handled differently by Nagios. So the domain “monitoring-portal.org” had been prepared long time ago, the primary record was changed, and Google did get a hit. The final announcement left hope that the community would go one like before. And we weren’t mistaken – it’s even more international after all those years.
Sending a patch to Nagios to tracker.nagios.org in the name of the Icinga Development Team resulted in “Changelog clarifications” by Ethan G., and after claiming the copyright on that patch, on a ban on their tracker. Meanwhile on the nagios-* mailinglists the string “*@univie.ac.at” was banned (my previous employer) – i know that because I couldn’t write to nagios-plugdevel either. Holger unblocked me when we were debugging the root cause.
Chapter VII: How to (not) deal with problems & work with a zombie community
Have a Nagios core developer maintain the patch queue & redesign the Nagios Core 4 – awesome. Kicking that developer out of the dev team just because he didn’t test the cgis properly – priceless. So yeah, 2013 was when it all went shits with Nagios. Fun fact: Andreas chose the Open Source Monitoring Conference (the one which had to be renamed due to trademark claims by NE) to literally hijack the “Nagios future” presentation, telling the audience that he got removed, and forked Nagios 4 into Naemon. Well, he actually wrote 99% of it, so no big deal.
So it really feels like that the (former) Nagios community now meets at conferences, presenting new features and having lots of beers with community members. And it doesn’t matter if it’s called Nagios, Icinga, Shinken, Naemon or anything else. At OSMC 2013, we had a great time with Zabbix & OpenNMS developers too, it felt like one big happy family. Not celebrating that Nagios got forked again – only having a great time together, and leaving each other with visions & ideas.
So it’s not about Nagios anymore, and the community changes as well. nagios.org is bloated with do-it-yourself, student, pro and commercial editions so people won’t easily find the download urls anymore. Everything is about business and selling a product. And the difference between Nagios and Nagios XI is not clear – rather close the webpage and ask for alternatives these days.
Content in that picture is copyright by NE.
The final chapter
Many people still use Nagios. They are annoyed by bugs, or broken releases. Missing features like real distributed monitoring, or stable apis are bad, but there are other tools on the market solving those. You may still make Nagios write to graphite somehow. But at some point, #monitoringsucks will apply.
In any way, installing a Nagios/Icinga/Shinken/Naemon/Centreon/Opsview Core will require plugins. Small nifty executables, be it compiled source binaries, or just perl/python/etc scripts returning some output and an exit code. If not, your first look onto the webinterface won’t be green. Green means “Everything is OK, don’t panic”. As a matter of fact, every single admin gets nervous if there’s something not green. Especially after the first install.
That being said – without the enormous effort of the Monitoring Plugins Development Team (former Nagios Plugins Development Team) all those core check engines would be nothing. Not even a single customer would buy your support without plugins making this work out of the box. We (in terms of the Icinga team) include the Fedora EPEL repositories in our Vagrant demo boxes for Icinga 2 pulling the nagios-plugins package.
Last week, wednesday 15.1.2014 at 22:38 to be exact, the following dialog happened on IRC (#nagios-devel irc.freenode.net)
22:24:40 < dnsmichi> is that the iframe forwarder thing? 22:27:37 < emias> dnsmichi: http://22.214.171.124/ 22:28:11 < emias> dnsmichi: And if you query that web server with "Host: www.nagios-plugins.org", you'll get a modified version of our home page. 22:28:25 < dnsmichi> modified? 22:28:34 < emias> dnsmichi: I'm pretty sure yhey're taking over the domain. 22:28:42 < emias> "they're" 22:29:11 < dnsmichi> i guess. they are trying everything to hide projects talking about nagios forks. 22:29:57 < emias> dnsmichi: Yes. Ethan had sent an email a while back, seems ge wasn't happy with my response. 22:30:00 < emias> s/ge/he/ 22:31:59 < emias> Soooo! Are we totally sure? Then I guess we should put up monitoring-plugins.org and post to the list as soon as possible. 22:32:35 < dnsmichi> No. But 2014 is a good year to change things 22:32:41 < emias> 22:32:54 * emias wanted to go to bed early today ... 22:33:02 < dnsmichi> Like leaving sourceforge 22:33:47 < emias> Well that's done. 22:34:25 < emias> As we moved everything to our own server (+ GitHub), switching to monitoring-plugins.org would be very easy now technically. 22:35:46 < emias> http://xxxxxx/archive/np-new.html 22:36:00 < emias> That's the content I currently get from their server. 22:36:33 < emias> I think I'm sure enough. 22:38:10 < dnsmichi> ehm 22:38:24 < dnsmichi> i just figured 22:38:35 < dnsmichi> that webserver actually serves nagios-plugins.org 22:38:49 < dnsmichi> i miss icinga, neamon, shinken on the main website 22:39:11 < emias> dnsmichi: That's what I'm talking about. 22:39:40 < dnsmichi> yeah, but not only via telnet Host trickery, but live in my browser 22:40:10 < dnsmichi> can you edit the main page and look if your changes are applied? 22:40:37 < emias> heh 22:41:36 < dnsmichi> i think you have been hijacked.
So basically, at that time, the websites were identical, after the Nagios Plugins Development Team was required to rename their project. I may be biased and have reacted emotional in that situation, but my german speaking brain told me, that copying a website, and changing the dns records would mean: Put your own (censored) content into the user’s sight. He won’t notice. That’s just like getting a mail asking for your bank account details, showing you a website which _looks_ identical, but isn’t technically. So if nagios-plugins.org would allow to change your passwords, they would simply have fetched and stored your passwords being able to use that against you.
On Sunday, 19.1.2014 it looked like this
In that specific region, the problem is even more specific: nagios-plugins.org is offering a software download. A software which is installed on thousands of servers. By serving an identical software package, everyone will trust it and just upgrade/install it, as the documentation of many projects say. The user won’t recognize any changes. What if they decide to add a phone home functionality to their software? You’ll never know, because you are trusting the name. You are trusting a vendor, which had been replaced. That obviously means “compromising” or “hijacking” a website.
The legal part of copying a copyrighted content just came up later in the drama (being totally illegal to act like that). But still, NE at least recognized that, even if they didn’t put it into a public statement. nagios-plugins.org now looks a bit different than the original content now served by monitoring-plugins.org. Even though, their actions are still illegal. But if no-one stops them, they will continue like before.
Either way, there was an announcement the day it happened on the website, and later to the mailinglists, where a discussion started. Also some insights on the previously happened Naemon fork (look through the archive, if interested).
On thursday (16.1.2014) the discussion on IRC was still ongoing. At that time, I’ve taken the liberty to step forward, and inform the package maintainers of ‘nagios-plugins’ while the now kicked and renamed team started to continue their renaming process. Jan W. is also the Debian packager, so I just informed RHEL, SUSE, FreeBSD, OpenBSD, Gentoo and ArchLinux. These distributions are well aware of Icinga, and I do have some contacts over there. Or the other way around – me being a developer changing stuff wants to inform packagers making their lives easier. I’ve been maintaining the Icinga RPMs for 4 years now, so I know what I’m talking about. Upstream fucks up, you’re screwed with mass reworking your packages.
In any attempt made, it was about users and their changed upstream source infecting their chain of trust.
Somewhere between thursday and friday (17.1.2014) the story hit reddit (it wasn’t me). There were plenty of comments in 2 threads (linux and sysadmin). While ‘scumbag’ isn’t a nice description, reddit still allows you to name it like so. And apparently, given the fact that NE did not tell anyone about their website defacement, ‘scumbag’ the is very least one could say (not that I would, just saying, eh?).
Interestingly enough, that story was to be read on Slashdot too. And then a bit more widespread on twitter as well.
In parallel to all the news, and also overwhelming feedback & appreciation of the monitoring plugins (and also Icinga which normally benefits by any Nagios drama), there was the discussion on-going on the bugs I’ve opened for the packagers. While I was and am not essentially interested in helping NE to gain their ‘nagios-core-plugins’ package, it’s still vital to know that the Redhat bug unleashed a competitive discussion how to proceed here. While NE employee Andy B. always insisted that the Monitoring Plugins project is a fork of the original (and fixed the git repo with a single commit after my remark of trust – that same passage is also quoted in that interesting blog article referencing my community concerns), the general outcome was that no-one is able to use/update the ‘nagios-plugins’ package in EPEL. Which is a good deal imho so far.
The thing with a newly formed Nagios Plugins team is – they don’t know the code very well, even if claimed otherwise. Not even Eric S. who tends to be the only part-time core/ndo/nrpe/nsca developer of NE, may handle all the stuff. From my personal observation following their git history closely he broke more stuff than Andreas ever did (getting paid for breaking Nagios? Haha, Just kidding.).
On monday, 20.1.2014 Ethan G. joins the drama for even more drama. NE has announced the new Nagios Plugins team, while attacking Holger W. using his full name in their company blog. What a shame – no more words to add here.
In any attempt, nagios-plugins.org now looks different (Wikipedia tells about the controversy too). They’ve also changed their opinion on mailinglists (recently shut down to prevent Andreas E. announcing his Nagios fork, as rumors like Andreas E. do tell) and have now auto-subscribed all users already subscribed to monitoring-plugins list to the nagios-plugins list. Copying the subscribers database and using that personal data against the unwanted project? Well, nice try. Users are confused anyways, and will happily unsubscribe from the two mailinglists looking for alternatives to follow the news or look for support.
Anyways. Too much drama here involved. FreeBSD, OpenBSD, SUSE, Debian have reacted. Some of them will just move from ‘nagios-plugins’ to ‘monitoring-plugins’ and oprhan the old package. For RHEL, Sam K. is working on a new package (I’ve only done some additional package foo in order to help resolve all the remaining showstoppers) becoming the new ‘monitoring-plugins’ maintainer in EPEL 7. For EPEL5/6 there may not be an obvious solution, but we’ll make sure to get something working for our communities.
Even if a Nagios user installs packages from packages.icinga.org being happy about fixed problems. Is that a crime? No it isn’t. But NE thinks it is. But it doesn’t matter in the end. We (the forks) are the bad guys, while NE and Ethan G. are the ones who never started anything, and are being harmed by us. Tbh I consider this a personal attack, but hey – that’s a RHEL bug to fix problems. Not to generate more drama for the mama, eh?
An now for the closing end, some more dramatics dramas. Well, first off an interesting blog entry by Matt Simmons, which showed that my English is actually bad in terms of vocabulary. Using the term “compromised” actually had put the RHEL bug up to hacker & security news feeds. Well, public attention for what price. Still NE is probably using that argument against me, but who cares anyways. That really was a language accident or incident, call it whatever you like 😉 Oh, and I’ve made sure that the internet doesn’t forget even if this story goes offline.
There are probably some more addons or plugins around which are compatible with Icinga, or mention that on their websites. They’ll probably get an email soon-ish and being ordered for censorship serving the greater power of the NE dictatorship. And then they’ll lose their motivation. Or step down, because renaming all the stuff causes that much work, moving the domain, re-establishing a trust platform. Or they are forked by NE into their enterprise stack, like happened with Nagiosql (now Core Config Manager), NagTrap (now NSTI) or Teeny Nagios (now Nagios Mobile). But you as addon developer get all the support questions, that’s for sure.
Nagios Enterprises has shown in many ways how to kill their own community being grown around Nagios. Conferences have started popping up where people meet not to only talk about Nagios (the Nagios World Conference Italy renamed itself to Open Source Systems Management Confernce while the Open Source Monitoring Conference attracts >250 open source enthusiasts every year here in Nuremberg). Community platforms which have resisted against their censorship and renamed themselves (the real community lives at monitoring-portal.org). Addons which are looking for alternative hosting platforms. Developers who just do not like them for their contribution agreements (and other licenses which explicitely forbid you to fork). Everyone chooses their platforms where they’re free. Like we are on #icinga on freenode. Or social media where censorship just doesn’t work. Not even fork developers will send them patches anymore being either banned or ignored.
In any way, the ordinary user will not create any plugin or addon in the future. There are too many ways to fail, and if you’re open-minded you’ll get a letter from a lawyer forbidding you to talk about competitors with your Nagios-exclusive tool. A zombie community – and then it will be interesting to see how the open core business model of Nagios will be able to survive, if the community stops contributing. Or if the community starts over in Icinga/Shinken/Naemon, or any other cool devop tool such as Sensu.
But, that’s another drama for the future. In order to be prepared, get this t-shirt. Then meet with all Nagios forkers, create cool code & laugh with community members, and enjoy your beers.
And leave the drama to those having abandoned their old friends.