78% of companies “run on open source”

Black Duck Software recently released the results of their annual Future of Open Source survey. On the surface, it looks pretty good. As the title of this post says, 78% of companies “run on open source”. Open source usage has doubled in business IT environments since 2010. Two-thirds consider open source offerings before their proprietary counterparts.

Not only are companies using open source software, they’re contributing, too. Some 64% of companies participate, with nearly 90% expecting to increase their contributions in the next few years. Half of the companies say more than 50% of their engineers are working on open source. Many companies see open source participation as a recruiting tool as well.

But when you dig a little deeper, there are some issues, too. A majority of companies view open source software as having higher quality and security, but most don’t monitor the code for vulnerabilities. Companies lack formalized policies both for consumption and contribution. A lot of the terms are pretty vague, too. “Participation” in open source can take on a variety of meanings, some of which are basically token involvement for PR purposes.

What I found most interesting, though, was the projects listed as “most valuable”: OpenStack and Docker. I may be biased by my day job, but I see that as a sign of the rise of *aaS. Despite the growth that cloud services have already seen, there’s a lot more market out there to be tapped.

Another interesting item was the increase in venture capital investment, both in gross and per-deal measures. Hopefully, this reduces the issues faced by projects such as OpenSSL and PGP, where a lack of funding puts much of the Internet’s secure communication at risk.

Finally, my initial reaction to the headline was “the other 22% do and don’t know it.” As it turns out, I wasn’t that far off. Black Duck reported that 3% of respondents do not use open source software at all. (Where’s the remaining 19%?) I actually wonder if that’s true. It seems like you’d have to try pretty hard to avoid any of it. This will become increasingly true as time goes on, when even historically hostile companies like Microsoft being open sourcing some of their products.

How I shot myself in the foot with pylint

I mentioned this in passing in a recent post, but I thought I deserved to make fun of myself more fully here. One of the things I’ve tried to do as I work on code I’ve inherited from predecessors is to clean it up a bit. I’m not a computer science expert, so by “clean up”, I mostly mean style issues as opposed to improving the data structures or anything truly useful like that.

Most of the development I do is on Python code that gets compiled into Windows executables and run as an actuarial workflow. I discovered early on in the process that if I’m working on code that runs toward the end of the workflow, having to wait 20 minutes just to find out that I made some dumb syntax or variable name error is really annoying. I got in the habit of running pylint before I compiled to help catch at least some of the more obvious problems.

Over time, I decided to start taking action on some of the pylint output. Recently, I declared war on variables named “str”. Since str() is a Python function, pylint rightly complained about it. Since the method that used “str” did string replacement, I opted for the still-not-great-but-at-least-not-terrible “string”. I replaced all of the places “str” appeared as a variable and went about my business.

As I was testing some other changes, I noticed that some of my path replacement was failing (though I didn’t know that’s where it was at first). So I shoved a whole bunch of logger calls into the “prepare” script to see where exactly it was failing. Finally, I found it. Then I shoved more into the module where the failure happened. I had to work down through several method calls before I finally found it.

There was still one instance of “str” there, but now Python thought it was the str() builtin and got really confused. In hindsight, it should have been totally obvious that I had inflicted this pain on myself, but several days had passed and I had forgotten that I had messed around in that function. I should have consulted the revision history sooner.

Building my website with blatter

I recently came across a project called “blatter”. It’s a Python script that uses jinja2’s template engine to build static websites. This is exactly the sort of thing I’d been looking for. I don’t do anything too fancy with FunnelFiasco.com, but every once in a while I want to make a change across all (or at least most) pages. For example, I recently updated the default content license from CC BY-NC-SA 3.0 United States to CC BY-NC-SA 4.0 International. It’s a relatively minor change, but changing it everywhere is a real pain.

Sure, I could switch to a real CMS (heck, I already have WordPress installed!) or re-do the site in PHP, but that sounded too much like effort. I like my static pages that are artisinally hand-crafted slapped together in vi, but I also like being able to make lazy changes. And I really like page-to-page consistency. With blatter, I can create a few small templates and suddenly changes can be made across the whole site in just a few seconds.

Blatter smoothly merges static and templated content. The only downside is that because it seems to touch all files every time it builds (blats), pushing the new content to my website becomes a larger task. That’s not a huge concern because of the relatively small size of the content, but it’s something that seems fixable. So pretty much all of the site has been blatterized now. For the most part, you shouldn’t really notice any changes.

Communicating weather safety information

Weather is complicated and hyper-local. The general public often lacks a basic understanding of weather evolution and people are generally bad at risk assessment. These facts combined make it really hard to provide general safety advice. It’s made even harder by the fact that if you give bad advice, you may be responsible for injury or death.

What to do when you’re in a car and a tornado is coming is perhaps the epitome of this issue. The National Weather Service office in Kansas City recently posted a scenario to its Facebook page. I saw some dismay expressed about how many people said they’d keep driving in that scenario. But here’s the kicker, I think that’s (conditionally) the right answer.

In the scenario you’re smack in the middle of a six mile stretch of interstate highway that’s expected to be impacted by a tornado in 15 minutes and you’re at an exit. The overpass is clearly the wrong answer. A very good answer would be to go to one of the gas stations or restaurants in the picture and seek shelter there. A car is about the worst place to be in a tornado, so why did I say “keep driving” is the right answer?

Let’s assume you’re traveling at 60 miles per hour. In three minutes, you’ve reached the edge of the warned area. The tornado won’t reach that area for another 12 minutes. Of course, there’s likely some error in the projection, but even if the forward motion is twice what was stated, you still have a cushion of over three minutes. If, in addition, the danger area is twice as large as stated, you still have 30 seconds. That’s cutting it too close, but we’re being really conservative here.

Now let’s look at all of the underlying assumptions that I made. First, I assume that you can safely travel at normal speed the necessary distance. This means no traffic, accidents, construction zones, or debris from earlier storms. In some places, you’d probably have sufficient visibility to make that determination, but certainly not in all places, and not in the picture shown. Second, I assume that you are just passing through. If you’re 10 minutes from home, it might be tempting to try to get there, but that eats into a lot of your safety buffer. Third, I assume you’re traveling south or that the main part of the supercell (another assumption) does not contain heavy rain or large hail that would slow you down or cause damage/injury on its own.

What would I do in that situation? It would depend on my familiarity with the area, my awareness of the storm type and evolution, and (most importantly), my ability to process it all quickly enough.

What should you do in that situation? See above. The best default answer is to seek shelter in one of the buildings off the exit, but that’s not always the best answer.

Book Review: The Open Organization

Full disclosure: I own a small number of shares in Red Hat.

Three years after Red Hat become the first open source company to reach a billion dollars in annual revenue, CEO Jim Whitehurst published a tell-all book about his company. The Open Organization barely mentions the technology involved in Red Hat’s success, although Whitehurst holds a bachelor’s degree in computer science. The Open Organization, as the title suggests, is about the organizational culture of Red Hat that enables its success.

Whitehurst describes his time at Red Hat as a learning experience that made him a better leader. Previously, he had been the successful Chief Operating Officer at Delta Airlines, guiding that company through bankruptcy and revival in the wake of the September 11 terror attacks. The organizational structure of Delta is described as being “top down”, typical of most large companies.

Such a structure arises from an promotes risk aversion and central control. Red Hat prefers a bottom-up approach where employees are given a wide latitude to make decisions. The role of the CEO becomes motivator and context-setter, while accountability is handled by social pressure.

However, the bottom-up approach cannot be truly described as a democracy, a point that Whitehurst emphasizes repeatedly. Red Hat follows a “the best idea wins, no matter where it comes from” policy, but Whitehurst makes it clear that ideas have to be solicited, too. Employees have different preferences about communication, and they need different ways to provide their ideas.

In describing Red Hat’s culture across seven chapters, Whitehurst doesn’t prescribe the specifics to every other organization. In chapter 7, he acknowledges that Red Hat is still a work in progress. Nonetheless, the broader principles are applicable. Whitehurst cites examples from other companies across a variety of industries to demonstrate that it’s not only software companies that can follow Red Hat’s example.

The Open Organization is a well-written book that turns out to be an easy read. Unlike many management books, it focuses on practical effects instead of theory and provides numerous examples. The content is well laid out, establishing the “why” before moving on to the “what” and finally the “how”.

My main complaint is that Whitehurst does not address the potential criticisms of Red Hat’s method. The blunt and argumentative (although generally collegial) nature will not be appealing to everyone. Furthermore, the way the company aggressively defends its culture (a phenomenon described in several places) prevents whimsical change but it also could discourage appropriate changes from the outside.

Nevertheless, The Open Organization is an excellent book for leaders at any level of an organization. I strongly recommend it as a guide to opening up your own organization. Picking and chose what works for you.

The Open Organization is scheduled to be released on June 2. It is published by Harvard Business Review Press.

The strangest bug

Okay, this is probably not the strangest bug that ever existed, but it’s certainly one of the weirdest I’ve ever personally come across. A few weeks ago, a vulnerability in OS X was announced that affected all versions but was only fixed in Yosemite. That was enough to finally get me to upgrade from Mavericks on my work laptop. I discovered post-upgrade that the version of VMWare Fusion I had been running does not work on Yosemite. Since VMWare didn’t offer a free upgrade path, I decided not to spend the company’s money and switched to VirtualBox instead (see sidebar 1).

Fast forward to the beginning of last week when I started working on the next version of my company’s Risk Analysis Pipeline product. One of the executables is a small script that polls CycleServer to count the number of jobs left in a particular submission and blocks the workflow until the count reaches 0. It’s been pretty reliable since I first wrote it a year ago, and hasn’t seen any substantial changes.

Indeed, it saw no changes at all when I picked up development again last week, but I started seeing some unusual behavior. The script would poll successfully six times and then fail every time afterward. After adding some better logging, I saw that it was failing with HTTP 401, which didn’t make sense because it sent the credentials every time (see sidebar 2). I checked the git log to confirm that the file hadn’t changed. I spent some time fruitlessly searching for the error. I threw various strands of spaghetti at the wall. All to no avail.

I knew it had to work generally, because it’s the sort of thing that would be very noticeable to our customers. Particularly the part where this sort of failure would mean the workflow never completed. I wondered if something changed when I switched from VMWare Fusion to VirtualBox. After all, I did change the networking setup a bit when I did that, but I would expect the failure to be consistent in that case. (Well, to always fail, not to work six times before failing.)

So I tried the patch release I had published a few days before. It worked fine, which ruled out my local test server being broken. Then I checked out the git tag of that patch release and recompiled. The rebuild failed in the same way. This was very perplexing, since I had released the patch version after the OS X upgrade and resulting VM infrastructure changes.

Out of ideas, one of my colleagues suggested reinstalling Python. I re-ran the Python installer and built again. Suddenly, it worked. I’m at a loss to explain why. Maybe there was something different enough about the virtualized network devices that caused py2exe to get confused when it built. Maybe there’s some sort of counter in urrlib2 that implements the plannedObsolescence() method. Whatever it was, I decided I don’t really care. I’m just glad it works again.

Sidebar 1

The conversion process was pretty simple. For reasons that I no longer remember, I had my VMWare disk images in 2 GB slices, so I had to combine them first. VirtualBox supports vmdk images, though, so it was quick to get the new VMs up and running. My CentOS VM worked with no effort. My Windows 7 VM was less happy. I ended up having to reinstall the OS in order for it to boot in anything other than recovery mode. It’s possible that I failed to correctly install something at that time, but the timeline doesn’t support that. In any case, I’m always impressed by the way my virtual and physical Linux machines seem to handle arbitrary hardware changes with no problem.

Sidebar 2

I also learned something about the way the HTTP interactions worked. I’ve never had much reason to pay attention before, but it turns out that the call to the rest API is first met with a 401, then it sends the authentication and gets a 200. This probably comes as no surprise to anyone who has dealt with HTTP authentication, but it was a lesson for me. Never stop learning.

Sidebar 3

I didn’t mention this in the text above, so if you made it this far, I applaud your dedication to reading the whole post. The first half of my time spent on this problem was spent ruling out a self-inflicted wound. I had already spent a fair amount of time tracking down a bug I introduced trying to de-lint one of the modules. More on that in a later (and hopefully shorter) post.

Kids these days

A while ago, I heard about a website called Class 120. The service is designed to allow parents, coaches, and the like to see when college students are going to class. It uses the student’s smartphone location and class schedule to determine if the student attended the class or not. When I first heard of this, I rolled my eyes. I suspect many others have a similar reaction.

But the more I think about it, the less objectionable it seems. Going off to college is often the first time a teenager has real independence. It’s unreasonable to expect them all to do well with no experience. A little bit of passive supervision might be just what it takes to turn a $20,000 waste of a year into something more immediately useful. Sure, it could be a crutch, but sometimes crutches are necessary.

Speaking for myself, something like this might have been really useful in 2001. My first year as an undergraduate was pretty lousy academically, in part because I missed a lot of class. I learned my lesson eventually, but at the cost of effectively a wasted year. People have various ways to motivate themselves, some intrinsic, some extrinsic. Who am I to judge?

NWS TV studios?

The Washington Post’s excellent Capital Weather Gang blog had a post earlier this week about a new addition to the Birmingham, AL National Weather Service office. It seems that WFO Birmingham has added a high-quality setup for video recording. This has some private sector broadcasters worried that the NWS will be horning in on their business.

Let’s be honest: the NWS is hardly a digital juggernaut. Even if they wanted to, I don’t see the NWS doing live streams during severe weather events any time soon. But as times change, so too must the function of the NWS. The old model where the NWS generated products, but the media did most of the dissemination to the public is not suited to the 21st century (despite the fact that many NWS products are still in a 20th century paradigm). The public can easily connect directly with the NWS, and the nobody is served by hamstringing the NWS due to tradition.

The NWS has a mission to protect life and property. In a multimedia world, higher-quality weather briefings, interviews, and educational material advances that mission. If a broadcast meteorologist finds this disconcerting, they should seriously evaluate how much value they’re adding for their audience.

File a bug!

The ability for users to submit patches is often touted by advocates of open source software. “Patches welcome!” is a common refrain on mailing lists. Even if someone lacks the ability to diagnose and fix an issue, they can file a bug in the project’s system. Indeed, there’s an unwritten social expectation that users file a bug.

Sadly, these are often just empty words. “Patches welcome” can be a seemingly-polite way of saying “your problem is not important to me. Go solve it yourself and I’ll accept your solution.” And telling a user to go file a bug can be equally dismissive, especially if the bug-filing process is unpleasant.

I certainly don’t mean to imply that all or even most open source developers use these phrases as a polite cover for their disinterest. There are jerks, of course, but I suspect most developers genuinely want bug reports and patches. Last week, a very busy developer replied to a mailing list post with a terse “file a bug.” Now, I happen to know this particular developer and I know he’s keenly interested in fixing bugs. On this day, he was swamped and didn’t have time to get to the issue right away. Suspecting from the original email that the user who reported the bug wasn’t deeply technical, I took the liberty of reproducing the issue and filing a bug with the details.

Would the person who originally reported the issue have filed the bug done so if I hadn’t? We’ll never know, but I do know that he didn’t by the time I did. After creating a Red Hat Bugzilla account, selecting the right options, and filling out the bug report, he’d have to hope the developer meant it and that he bug would get fixed. As anyone who has been around a while can attest, just because a bug is filed, that doesn’t guarantee it will be fixed in the next few years.

One particular bug that I filed sticks in my memory. It was undeniably a bug, I attached a patch to fix it, and yet the bug remained untouched through several Fedora releases. Granted, it was a relatively minor issue, but if I weren’t involved with the project, I’d have been pretty put off by this. Of course, no one owes me a fix, but as Chris Siebenmann noted, if there’s a social obligation to file a bug, there’s also a social obligation to deal with the bug.

This morning,I asked on Twitter if anyone had what they’d call a “positive” bug reporting experience (as opposed to simply a not-negative experience). I was pleasantly surprised when several people chimed in to say they had. Small projects were generally the ones that drew the good responses (Jordan Sissel was mentioned by multiple people).

So what makes for a positive experience? Rapid acknowledgement and resolution were mentioned several times. Those who replied mentioned courteous and helpful interactions (e.g. asking constructive questions, filing an appropriate bug when the original bug was mis-filed). @phrawzty summed it up well: “Most importantly, however: good intentions were assumed. I never felt treated like a dangerous outsider.”

This is an area wide open for study (and I may end up doing just that), but in the meantime projects should consider how they present themselves. Large projects with more resources should make an active effort to devote some time to bug handling as a community outreach effort (I suspect large projects have worse bug experiences, in part due to the fact that nobody owns the process). When you say “patches welcome” or “file a bug”, consider what you’re really saying and if people will take it the way you mean it.

(Thanks to those who provided feedback: @cpj1, Matt SimmonsMatthew General, Patrick Cable@phrawzty, Rick Waldron@SysAdm DreddThomas Ballinger, @Twirrim, and anyone else I failed to mention.)

Using systemd tmpfiles

As part of my regular duties at my day job, I provide some community support on the HTCondor users’ mailing list. At the beginning of the week, someone came to the list with Fedora RPM troubles. I was able to track it down to the fact that the RPM didn’t correctly create some temporary directories.

But why not? It’s been a while since I did any RPM building, but at first glance, the spec file looked fine. The systemd tmpfiles configuration looked good, too. Then someone else found the cause of the problem: according to the Fedora packaging guidelines, a package should write the config file andcreate the directory. It turns out that tmpfiles.d is only processed at boot time (or when manually invoked), so a package that needs a file/directory to exist needs to create it at install time (or require a restart).

I was going to file an RFE, but I found a related one marked as CLOSED->WONTFIX. I understand the reasoning here, so I won’t reopen it to argue. It’s just a surprising behavior if it’s not something you regularly deal with.