No Matter Where You Go...

[midiamin]

eng: I have another change to push to production.
me: This fixes the problem you were working on earlier?
eng: Yeah.
me: You checked it into the current release branch?
eng: Yeah.
me: And you tested it?
eng: Yeah.
me: Why didn't this problem show up when you tested the earlier change?
eng: Oh, because I pushed the change into the

Comments

[iphy]

A good rant. And leads to:

I think people doing [jobs] are cursed to have the same kinds of arguments over and over throughout their careers.

And, really, I like your statement about trust and I think it applies out a lot more generally as well. There are a lot of job positions for which companies do not just hire 'random people off the street'. They hire people who are supposed to have certain skills & knowledge and know what they're doing. Yet, for an awful lot of them, people who do not have that skillset and knowledge seem to believe that it is "just " and anyone could do it -- that they know enough about it that they can decide on their own (again, without the subject matter skills and knowledge) to change it

[midiamin]

And, really, I like your statement about trust and I think it
applies out a lot more generally as well.

Indeed. Six months ago, this rant would've been all about IT. Some of
the circumstances would be different, but the sentiment would be the
same. It's quite possible my mechanics have the same gripe about me,
and so on.

Yet, for an awful lot of them, people who do not have that skillset
and knowledge seem to believe that it is "just " and
anyone could do it -- that they know enough about it that they can decide
on their own (again, without the subject matter skills and knowledge) to
change it.

That's the one that I think taarna, fruitylips,
wambold and I run into the most when we're doing release
or release-like things. The other big problem I run into is people
who have done the job before but aren't doing it now. In my

[midiamin]

...especially when the thing they're requesting seems like a very simple thing to implement. Often it is, but there are other more important issues blocking it...

Something I forgot here is a philosophy that's been in the back of my head for the last several years but finally found its way into words yesterday: It's not a question of what I can implement. It's a question of what I can afford to support

I have any number of quick solutions to problems that exist in my domain, but if implementing a quick fix to make a small number of people happy and supporting that quick fix means I don't have time to implement the thing that will make a larger number of people happy, I'm not going to do it. It's that support cost (most usually paid in the currency of time) that many people seem oblivious to.

Release engineering is hard. Let's go shopping!

[felinecanine]

Every process is slow and cumbersome when it doesn't let you do what you want to do Right Now.

The concept of "acceptable risk" from security seems relevant here. There's generally an inverse relationship between the speed and thoroughness of the process. The latter is directly related to the quality of the product, and that's where the acceptable risk comes in. Your process has to be at least through enough to reach the desired level of risk.

I think this is one part where the release engineer needs a fair amount of experience to get things right. That person has to understand how the process affects overall risk. Another part which I've so far neglected to mention is the efficiency of the process (i.e. thoroughness vs. speed).

On another loosely related topic, I find it interesting to compare release practices between hardware (eg. ASIC) engineering and software engineering

Too often, release engineering is imposed

[wambold]

The best situation a release engineer can be in is when the developers have risen up and screamed, "we need a release engineer". Sadly, release engineering is often imposed by management.

My current employer didn't see the need for release engineers for a long time. Sarbanes-Oxley was one of the major reasons why the company has release engineers now.

While the developers I'm working with are nice enough, they know they can build software, load it onto QA and production and what not, but they aren't allowed to. They are forced to do these things through me and this is annoying, because they have to wait for me. In many ways, I'd rather be arguing about good process than being a monkey because of "separation of duties".

It's hard to develop trust when the whole process is imposed by the outside.

Re: Too often, release engineering is imposed

[midiamin]

Oh, yeah, that's a good point. I guess the places where I've done release had already mostly bought into the importance of the role. But that does tie back to the "Release engineering is hard because..." statement. If we can't write a book on the subject, maybe we can at least hack together a faux chick tract to explain its importance.

[dustgrl]

There does though need to be balance. What I have found in the world of early startups is that opportunity cost is a cruel mistress. Everyday I am asked to balance what can scale, be supported, be reasonably certain of solving most cases of the problem versus the need to show something to investors, potential clients, etc that could get us money and useful business deals. Add to this that the interest of a partner or investor is incredibly fickle. They are interested now or never