Software engineer goals: fairness and career development

Introduction

As you might have surmised from the previous post, I care a lot about getting the right goals in place for software developers.
There are several reasons for that, and I'll unfold most of them as this post finds its way to conclusion. The first and the main principle is already revealed in the title though. It is fairness.

Goals are not just for people whom I manage. They are also for me: they make sure that everyone is assessed fairly based on their results, and not by personal bias or most recent memories. They also draw a fair, uniform, target that determines rewards and career progression based on performance rather than visibility, luck and various other subjective factors.

Software engineers: what do they do?

Ok, time to get off the soap box. Let's talk trivialities, and start by asking a semi-rhetorical question. What do software engineers get paid for? Well, they get paid for coding (or, if we hop back on the soap box for a second, they draw their paycheck for constructing software).

So, if we try and be as simplistic as possible, the more software is written in a mythical man-month with the right quality, the more rewarded should the employee be. Yes, I know, writing software is not flipping burgers, and we are way too simplistic here. That will be fixed before this article is over, but for now please hold the indignation: we all have to start somewhere. And, truth be told, there is strong evidence that there is a 10x difference in productivity factor among developers. If one guy creates ten times more value than his/her neighbour, shouldn't there be some kind of reward? So, let's develop the topic of

Productivity

Assuming we answer the previous question with an emphatic 'yes', we have to understand how productivity is to be measured. Now, this is a much taller hurdle. In fact, there is no acceptable answer - many have struggled to define various productivity metrics, and many have failed.
Just counting lines of code produced (LOC metrics) is too easy. LOC differs based on language, complexity of the task, and encourages bad copy/paste practices, verbosity etc. As an extreme example, imagine a hard defect that is resolved by a single line change after a couple of days of surgeon-like troubleshooting, and compare it to a copy/pasted test harness code. There are more cases such as these two where counting LOC fails, and fails hard.
There are slightly better productivity metrics: story points, functional points, and requirements implemented. Unfortunately, better does not mean good. Story points are subjective estimates, and it's not uncommon for one developer to price a task at 5 SPs, and for another 13 SPs. Functional points and requirements are often too bulky, and don't work well when multiple developers collaborate on the same requirement. As always, we need to be wary of side effects, where engineers motivated by speed with which they implement new requirements, and thus, eschew maintenance work or unit tests.

To avoid all this nastiness, I've arrived to fluffy comparative goals, which go along these lines:

Complete tasks at a pace consistent with engineers at level X

Yes, I know, very fluffy, not smart (upper- and lower-case), and prone to misinterpretation. However, it does provide a talking point if someone does not pull their own weight, or vice versa, if someone is indeed very productive. Future goals in this article will be better though - promise. For now, let's say that we want to talk about productivity, but not necessarily measure it with fine instruments. With that in mind, we can turn to greener pastures, and talk about

Quality

This is an easier one. Defects can be measured, and the more defects created the lower is the quality (yes, captain obvious is yet again at work). Obviously, one way to avoid defects is to avoid writing software, but hopefully the productivity goal, even in its vague form, puts a boundary around that.

So how are we going to measure good versus bad? One approach is to do a similar cop out as with the productivity goal, and say something like this:

Do not create more defects than expected from engineers at level X

This is better than nothing, but I feel we can do better. Sometimes, one bad defect that caused an exec-level aggravation is big enough to register, while five minor defects not visible to naked human eye don't. For this reason, we can take it one level forward:

Do not create any showstopper defects, not more than A major and B minor defects as expected from engineers at level X

This is a bit better, but in some situations we can take it further. Catching defects at code review vs. QA vs. customer differs by orders of magnitude in organizational costs. In other words, we want to encourage strong review and QA practices to create better software at lower costs. The main point though is that it is not just the manager's job to facilitate that. The developer should be the main stakeholder in catching defects as early as possible. They should be motivated to have a thorough code review, and be happy when their peer finds an issue in their work. They should take active interest in the QA plans, and appreciate testers who find issues.
For many people, this is self-evident, but definitely not for everyone. Assuming the point of goals is to encourage particular behaviour, we can go for this:

Do not create any showstopper defects, not more than A major defects found in code review, B found in QA, and C found by customers as expected from engineers at level X. Also, do not create more than D minor defects.

Maintainability

Before we move off this topic altogether, let me just touch upon an important and slightly controversial point. Defect is not just a case where the product does not behave as per specification. It is also a case where the code is so brittle that a defect is liable to appear once someone else touches it. Poor encapsulation, 100+ LOC functions, lack of OOD, absence of test coverage are an issue in themselves, even if the code (usually by accident) functions the right way. Hence, I tend to treat those as defects during code reviews. In case you disagree, think of a builder who uses hollow bricks here and there in your new house. Sure, the house does not have any defects today in the strict sense of the word. Is this house well built though, and wouldn't you like to reflect your opinion in the builder's assessment?

Tracking

Before we race too far though, let's talk about the evil twin of elaborate goals: performance tracking. If we work off a goal which enumerates a number of defects found during code review, minor/major and so on, we need to somehow measure these numbers over the performance period.
The more goals and numbers are there, the more there is to measure, and there's unlikely to be a friendly PA or accountant that would take care of the business. No, it would have to be the goals' creator, i.e. you, the manager.

Hence, there are two valid options:

1. Intentionally dumb down the goals, and rely more on your subjective opinion and working relationship with the employee when it comes to reviewing.
2. Work with approximate numbers as much as possible, but still have SMART definitions.

There's no right or wrong here. Usually, I go for the second option, and do lightweight tracking every week or so by registering any "abnormal" results, i.e. whenever a complex work is done to great quality or vice versa. In the vast majority of the cases, this suffices to provide a fair review. Rarely, when I and the reviewee have different perception of the work done, I might have to dig deeper and get more accurate numbers on demand for a task or two. Just make sure that your code review system, SCM and quality system are there to back you up.

Proficiency

So far, I've concentrated on the bread and butter of software engineering: i.e. creating software. However, this is not always the entire story. Software professionals also contribute by solving problems, reviewing code of others, responding to technical queries, designing solutions and so on. The higher an engineer is on the technical career ladder the less they code and the more they deal with aforementioned activities. This, in turn, means that we need more goals as coders become developers and architects.

However, measuring proficiency again bumps into lack of metrics. (For example, counting the number of technical emails answered is emphatically not the way to go.

One goal I tend to use goes along these lines:

Be the technical point of contact for components X, Y, and Z

Sometimes this can be extended into collective responsibility where an engineer is accountable in some degree for all defects in X, Y and Z. This usually happens further along the career curve, and has to come with informal and formal authority.

Adjusting

I've spent quite a bit of your and my time talking about what goals should be. However, now I'd like to pop that balloon and again say something controversial. The important part of goals is not what they are, but that they are a living and breathing mechanism.

If the manager reviews the goals every couple of months, checks whether they fairly assess their team's work, and adjusts them as necessary, then he/she would do a reasonable job, even if the starting point was off the mark. And vice versa, preparing the perfect goals, putting them in the fridge for 12 months, and unfreezing them come review time in many cases will yield a slightly spoiled result.

Of course, there are always developers who are happy where they are and do something very similar year after year, but they are usually not the focus of elaborate goal construction. It's those that develop their skillset and have aspirations that are the focus of this article, and for them goals must be constantly revised: same as their career.

Project-specific goals

One dilemma that came back to me year after year revolved around project-specific goals. For example:

Complete refactoring for the management front-end by Q3, release it to customers by Q4, with fewer than X major regressions

There's something very tempting about these goals: they cascade directly business goals to individuals. At the end of the day, I care less about amount of code produced, or the amount of entries in the ticketing system. What I care about is whether customers got the feature on time, and whether they are happy with it, and such goals allow expressing just that. 
However, they might not be less tempting for the individual, as sometimes not all factors are in their control (e.g. the business does not prioritize this feature enough, or there's no right QA skillset present, and defects escape). This is why I oscillated between adding and omitting those; at this point, I'm using them sparingly with a number of caveats to ensure that they are achievable and fair.

Summary

As usual, this post is less about providing answers, and more about sharing thoughts. I went through a fair amount of trial and error when managing engineers, and that process led me in a particular direction. There's no single size that fits all, and my last advice for today will be to use this post as food for thought rather than a template.

Software engineer goals: when SMART is not smart

This time around I'd like to talk about creating goals for software engineers.
First things first, let's get the well-known body of work behind us. You probably know about SMART goals. If you do not, the link to Wikipedia is in the previous sentence. One way or another, by the time we went past this paragraph, we're all on the same page with regards to specific, measurable, achievable, relevant, and time-bound (or finite, if smarf were a good acronym) goals.

However, once we leave this familiar territory behind us, we get into the big lawless desert that separates the SMART ideal from real, flesh and blood, people. As managers, we intuitively know what we'd like to get from our team, and we know what SMART means. What we don't always know is how to relate one to another.
In this mini-article I'll try dispelling the notion that goals can be generated in an algorithmic way, by ticking specific boxes.

Let's kick things off with a few counter-examples:

SMART goals that aren't smart

Counter Example #1

Resolve at least 10 bugs in the next 3 months

This goal is specific, measurable, achievable, relevant, and time-bound. It still has one big problem that should banish it into neverland.

Here's the thing: bugs are not born equal. A typo is a bug, and a subtle kernel packet routing problem is a bug. Expertise and time required for dealing with these two issues are incomparable. By setting such a goal, we encourage the reviewee to seek out the easier bugs: after all, they have to hit a number. And, by extension, we nudge their career into technical comfort zone, and inhibit progress.

Counter Example #2

Raise 2 patents in the next year

This goal also ticks all the 5 boxes, though one can argue that the "A" (achievable) is a bit shaky. However, just to make our discussion simpler, let's assume that the reviewee is working on projects that have potential for patents. There's still a big chasm however, one which is happily supplied by the legal system. It takes years to fully approve a patent, and until those have elapsed, it's hard to tell whether the patent brought benefit to the business. Internally, there are few people who can serve as the quality gate before the patent goes out to the bureau. So, we might get 2 patents, but those may waste our internal resources, employee's time or both - it's hard to control and assess their value.

Counter Example #3

Find 15 defects

I've seen this one a number of times, and it is one of the worse sins to be stamped on a QA engineer's goals' spreadsheet. Firstly, it creates antagonism between developers and testers: if a developer does their job well, then there are fewer defects to find. Hence, maybe the tester now is less interested in having strong quality before getting their hands on the new feature. Also, as per the first counter-example, bugs are not born equal. Often we'll value a single non-trivial, high impact defect detection over 10 typo findings.

Thinking about side effects

In each of these three counter-examples, we had a particular outcome in mind, dressed it in SMART clothing, but the result was full of side-effects that subverted our original aim far and wide. The intents are hard to argue with: we want developers to fix defects at a reasonable pace, we want them to be innovative, and we want our testers to release quality software. The problem is not with them, but with how they were expressed. The obvious question now comes: how should they be expressed in a smart (lowercase) way?

Revised examples

Counter Example #1

Resolve at least 10 bugs in the next 3 months

What was the original goal here? To be blunt, we want the developer to be productive, and not sit on a single work item for weeks while his teammates are sweating through the backlog. However, as described above, we solved that problem by creating several others.
The sad truth is that the number of defects resolved do not codify productivity. What does codify productivity is the total value or complexity of tasks completed. If we operate in an Agile-like environment, we could express this via percentage of tasks completed versus committed (which can be more than 100%):

Complete at least 90% of committed tasks each sprint

Of course, the word sprint can be replaced with month, or any other recurring time unit. Note that the goal does not prevent people from under-committing. The good news is that we are neither lawyers nor sportsmen. We don't have to legislate everything, and we can prevent under-commitment when planning our work, and ensuring that everyone gets their fair share.

Counter Example #2

Raise 2 patents in the next year

Here, we were obviously after innovation. There's a senior engineer on the payroll, and we'd like him/her to generate sound ideas, and not just implement what they're told. However, we well know that innovation does not come via patents, and the value of those patents can be hardly measured come review time. Perhaps it's time to detach ourselves from numbers and go for something like this:

Proactively suggest design improvements and/or feature enhancements

Now this goal is not SMART, as it is not measurable. However, given the right description, it sets clear expectations, and it provides enough to talk about. We probably should say that suggestions accepted by the team, including product management, count more, and that we're going after quality rather than number. Ideas, among other concepts, are hard to measure and count, and it's more honest and productive to accept that rather than retrofit measurable goals for the sake of measurable goals.

Counter Example #3

Find 15 defects

The intent here is simple enough: we don't want to get defects out. However, setting a goal of finding X defects is akin to setting a goal of catching Y criminals. For sure, the bugs/criminals will be found, but they are likely to be the wrong ones, while the right ones will escape.
And, this is exactly what matters: how many criminals bugs escape, not how many are detected. Also, bugs are not born equal, so a fair goal would go along these lines:

Have no more than 3 critical, 5 high, 7 medium and 9 trivial defects escape under area of your responsibility

Summary

This mini-article had examples, counter-examples, numbers and absence of numbers, but little in a way of recipes and templates. That is fully intentional: you know your team better than I do, so I can't tell what goals you should set. This is exactly why I used the word  crafting in the title, and not generating: people are unique, and so are their goals. SMART is not a rule or an axiom, but a guideline, and guidelines always have exceptions.

Should managers code?

Intro

Let's start with a question: is there a point in the career ladder where we stop contributing individually, and only coordinate (or facilitate) others?



My view is that this point is like the proverbial asymptotic curve; it can get as close as we like to zero, but will never quite reach it. Or, coming back to plain terms - everyone should and can contribute; it's only a question of how much and what for. The line between a manager and an individual contributor is not the same as a geographical border, which draws a distinct line between "us" and "them". Rather, it is a long blur, and throughout that blur we should care about what we bring to the team. As careers traverse these shades and their accompanying titles, personal contributions follow along that curve getting close to a zero, but - the main point - there are still good reasons to code/test/review. They become indirect: a manager rolls up his/her sleeve not to add manpower, but for other reasons, which I'll go into very shortly.

Matching personal experience and ability to influence

Let's go back (or forward) in time to that moment where you (will have) got the desired promotion letter. Presumably, you reached that moment by doing your job well, and gaining the recognition of your peers and managers, and again, presumably, you have strong experience with the technology and ability to influence. (You might have met managers who answer none of the conditions in this sentence, but they are neither the topic nor the audience for this article). With all of that, it makes little sense to go and abandon all of that useful experience on the next day.
So, there's a dilemma: on one hand, you want to and can code, on another, you're no longer paid to do that. Or, is it actually true? Let's start with:

Leadership by example

Most respected, though not necessarily the longest-lived, generals lead their troops and take part in battles. IT managers benefit from a quieter lifestyle; apart from an occasional coffee overdose, or a minor heart attack due to a misdirected e-mail, they are usually safe from workplace hazards. However, there is one parallel: both soldiers and software engineers respect leaders who from time to time can stand shoulder to shoulder with them and demonstrate that they can pull their own weight.

To start with, it breaks some of that barrier between "us", the workers in the trenches, and "them", the generals who command from above. There's more of that important team feeling, especially if the manager helps personally during crunch time. On the other hand, personal experience makes it easier to understand software engineers' grievances and requests for refactoring, switching toolsets and so on.
In my experience, there were a number of instances where personally seeing and maintaining particularly bad legacy code gave me the right mindset and conviction to push upper management for the right investment in refactoring. There was also a case or two where using development toolset helped lobbying for easier processes, and resisting introduction of cumbersome workflows that always seem fine to a manager, but not to someone who has to endure them every day.

The good news is that it's enough to do a few tasks every month to get all the benefit from the above. People remember the general that led them to a single battle; they don't necessarily expect him to do it on each and every one. Solving 3-4 small bugs every month shouldn't take more than a couple of work days; i.e. not more than ten percent of office time. This small investment can buy respect of your reports, as well as the right mindset and awareness to help them with their day-to-day job. However, it can also help you personally:

Experience and career continuity

Let's become egoistical for a moment, and think of our livelihood, i.e. ensuring that paycheck continues arriving month after month. Those that have been in the industry for long enough, know that nothing in this world is forever. Companies' fortunes turn around, bubbles pop, and financial crises appear every decade or so. Of course, the more indispensable you are, the less likely you are to be affected by the cataclysms and storms that rage outside, but at the end of the day the mantra is the same: nothing is forever, and nobody is untouchable. We must always think of our skillset and consider how we'll fare in the greater beyond should the worst happen.

Now, here's the thing: managers have it tough out there. The ratio of individual contributors to managers vacancies is 1-to-10 (or thereabouts), and software developers/testers simply have more openings to apply for. Moreover, a large proportion of employers look nowadays for managers with strong technical chops. In other words, a strategy of shying away from hands-on work is a dangerous one. In today's climate, it can leave one unemployed for a while should a disaster strike.
Of course, "hands-on" and coding is not one of the same. There are great leaders out there who never code, but yet understand and speak technology every day. However, if you're in middle management, you can count on being asked coding questions during interviews (thanks Google), and those 3-4 defects a month can make a big difference in keeping your toolset sharpened.
Just in case you haven't been fully convinced, let me throw another argument:

Fair reviews and career development

Providing team members with fair reviews and gently nudging their careers in the right direction is one of main responsibilities that we, managers, have. What are developers doing most of their time? Right, they code, design and review software. So, when the time comes, the bulk of their assessment should be based on how well they've done just that.

Unfortunately, "should" is often a word removed from reality; usually we review best only what we observe, and managers that distance themselves from coding can't observe. Instead, they review using peer feedback coupled with their personal interactions with the team, which tend to be e-mails, meeting participation, and general visibility. These are all fine, but remind slightly of an art critic that forms her opinion by not seeing the art in question, but reading second-hand opinions.
Hence, those personal coding tasks, the occasional code review, if chosen judiciously, can play an important role in observing and providing fair performance viewpoint. And, before I depart onward, let's not forget about the art of giving praise. It's one thing to praise using generic phrases: "hey, you've delivered feature X one week ahead of time, attaboy", and another to do so while being very specific: "you've delivered feature X week ahead of time by reusing module Y, which didn't occur to any of us, and then you had another nice shortcut with introducing an adaptor layer on top of existing interface framework". This difference can mean a lot to a developer who genuinely and rightfully takes pride in their work.

Avoiding time-critical tasks

It's not all plain sailing though. Though it's not too common, managers can err on the other side, and do too much hands-on work. The one rule to remember is: your role is to help others perform. Coding is often easier, and more gratifying than answering a long e-mail, preparing a budget, or crafting a personal review. The unfortunate reality is that these "dull" activities always must take priority; they are what we signed up for when the promotion came through. An addendum to that unfortunate reality is unpredictability; we can never tell when a critical e-mail will drop in the mailbox, or when we have a crisis to resolve.
Continuing the same train of thought: coding and context switches are not friends. Everyone in the team gets interrupted, but a developer usually has the mandate to park questions for an hour or two while they are putting finishing touches on a task. A manager cannot play this card; his/her official job description is to facilitate others, not to code. Hence, when a manager looks for tasks to take over for all the aforementioned reasons, these should not be time-critical. Ideally, if they are put on ice for a day or two, nobody will raise an alarm, and team's performance won't get affected.

Margin of error and personal responsibility

So far, I've been lumping coding and code reviews together, as both are technical, personal contribution, tasks. However, reviews are special in a way, so I'll tease them apart for the course of a couple of paragraphs.
While I was going on above about coordination, e-mails, budgets etc. etc, the truth is that management is first and foremost about responsibility; for R&D managers, responsibility over software their teams produce. Here, there is a whole spectrum of options of ensuring quality: from trusting the team all the way to personally reviewing each and every output.

As always in life, extremes have pronounced weaknesses. Reviewing each semicolon typed by the team does not scale. Trusting the team is fine up to a point where a particularly bad decision misfires. It's not hard to find core parts of the product where mistakes simply cost too much: think something along the lines of Facebook's personal wall, JavaScript inside google.com, packet management inside a web proxy etc. I'm sure if you give it a bit of thought, you can find similar business-sensitive zones in your product(s).
Personally watching over these areas has both the effect of adding another pair of eyeballs, and technical awareness which would allow you to react quicker if something does eventually go wrong.

Summary

Hopefully, this article has put an argument for managers not to divorce themselves from source code. Or, if you're not a manager, it may explain why your boss keeps looking over your should from time to time, and makes an appearance in your SCM tool, and why it's not necessarily a bad thing.

Time Management

Last time around I've spoken about taming meeting invites, and in passing mentioned that this technique is one of the cornerstones upon which effective time management resides.

Time management is a bit like playing a musical instrument: many people try it, few persevere with it, and even fewer get it right.
However, this is yet another case where mastery pays off. As a superficial example, let's take two imaginary technical leads: each gets, say, 20 daily e-mails/instant messages/mailing posts with requests for help. Due to efficient schedule organisation, one manages to respond to each and every one in the same day; the other responds on average within a couple of days.

What happens is a positive feedback loop. People, seeing that the first guy provides more immediate help, start flocking to him. More importantly, he gains more visibility, and more popularity points. And, eventually, when the time comes to move up the ladder, he will have the edge over the other one, even if the latter is as good on pure professional merit.

Here's a bit of visualisation:

Of course, it's a little bit idealistic, and no deity has cancelled office politics, or pure element of luck, but it's undeniable that strong time management skills are an asset for both individual and their company.
Since you're hopefully convinced by now that the topic is not negligible, we can take a look at a few helpful practices.

Time as currency

When we buy items (soft drinks, cars, houses, curiously shaped lava lamps), we always attach intrinsic value: i.e. how much we're willing to spend on them. However, dollars/pounds/euros/yuans (insert your monetary unit of choice) is not the only currency we hold. In today's hectic IT environment, another important and finite asset we've got is time.

Let me stress and underline the word finite here. Spend an hour on an obscure design question, and that will be an hour you cannot spend on helping others with their challenges, brushing on new technologies, or spending with your family or hobby. 

Now, with material objects, we usually know how much we're willing to spend. I'm happy to invest thousand pounds in a high-end guitar, and would not depart from more than a pound on a bicycle (since I don't ride them). For another person, it will be completely the other way around: both approaches are individual and perfectly valid.

Importantly, this analogy still holds true if we replace pounds (or your currency of choice) with minutes, hours and days. 

Defining a set of responsibilities for a new micro-service is something that as an architect I'd be willing to several hours for. Vice versa, as a manager, I'd be unwilling to examine an interface in microscopic detail, and will restrict my time there to 30 minutes at most; the goal will be just to ensure that I understand it at a high level, and can tie it to business goals.

Same task - different intrinsic values.

Just to avoid displaying managers as superficial - there are plenty of opposite examples. An engineer does not have to spend more than 15 minutes drafting a daily update e-mail. Manager can and should pay more than that as their role demands accurate and unequivocal messaging.

All in all, there are little mental price tags attached to all that we do; we just normally don't notice them, Being oblivious to those little tags can easily cause to over- or under-spend; it is as if we were pulling random wads of cash from the wallet when paying over the counter in a supermarket. Understanding how many minutes we're willing to pay on X, and staying true to that is a cornerstone technique for efficient time management.

Context switches

Despite what philosophers and humanists may say, people do resemble machines in more ways than one, and a prime example of that are context switches. You're probably familiar with the topic, but let me mention it anyway - context switches occur whenever we move from one unfinished task to another. Both humans and machines do not perform any useful work when they swap contexts; at worst, if we constantly switch back and forth between tasks rather than completing each one in turn, we're thrashing in a state of unproductive semi-panic. 

Of course, it's not hard to arrive to the solution: stop thrashing and do your work in a strict order. Or, as one Russian writer pointed out - "if you want to be happy, be so". 

The problem is that things don't always go our way, and we cannot always complete all our work sequentially. If a developer starts writing a module that would take 8 hours to complete, they can't should not ignore the world around them for an entire day, and not deal with important code reviews, cries for help etc.

The trick is quick prioritisation. When a new request comes in, I tend to take a quick priority call among the following response times:

• Drop everything and answer immediately. E.g. a widespread outage, an onsite sales engineer needing help, or a immediate call for action from my boss.
• 30 minutes. For example, someone else in the company is stuck with their (non-critical) task, and could be unblocked with my help.
• 2-4 hours. Typical examples are: a general request for information on a non-urgent customer case, or a code review.
• Next working day. Planned document review, request for a weekly report etc.

The idea is that unless the environment is extremely chaotic, and most incoming transmissions do not fall in the first bucket, I can at least manage my interruptions, and reach a certain point where it's easier to switch to something else.

But, things can sure fall in the second and third bucket a lot, and quick and dirty prioritisation does not bring full salvation.

Forecasting interruptions

If your role does not include a lot of coordination, i.e. your title does not have the magic words Manager, Lead or Architect in it, then you can safely stop reading here. In many ways, you can count your blessings: people and priorities don't pull in different directions, and you get to create something new without being constantly tapped on the shoulder. So, if you're still with me (either by the virtue of your role, or due to sheer curiosity), let's say that you get a lot - and I mean, a lot - of interruptions that demand response within the same day or sooner.

Here, the challenge of being able to get on with the day-to-day job is still pretty much there. Yes, we just mitigated it a bit by doing a quick pre-filter, but mitigating does not mean solving. It is still very easy to get bogged down in a constant reactive cycle, where you just respond to requests rather than  generate new value.

Yet, there is one little factor though that gives IT professionals a little edge towards solving this, and distinguishes software professionals and, say, football goalkeepers. The latter do not get any pattern in how often balls come their way. It might be once every 5 minutes, a cluster of ten balls towards the end of the game, or never. With software professionals, there's always a pattern. Normally, emails and questions come mostly during the first hour of the working day, towards the end of the day, and whenever any time zones we work with kick in.
If we zoom out for a moment, we can also observe a seasonal flow: end of fiscal year tends to be way more active than mid-July.

Understanding the organisation and its industry brings in more insight into when it's safer to plunge into long, focused, endeavours, and when it's better to deal with superficial tasks.
With all of that, I tend to pre-plan time slots which are open for involved, uninterruptible engagements; for example, it might be 10-12 AM, and 2-3 PM on a typical mid-week, mid-fiscal quarter day. Both slots avoid the time needed to answer morning and last night's questions, and also late afternoon when other time zones become active.

All that builds into day pre-planning. Before the day starts, I tend to know what meetings are scheduled, when I need to tend the goalposts (that goalkeeper analogy keeps on working!), and when I can switch into uninterruptible tasks, such as long-term planning or architecture design, without great risk of forsaking something important.

Summary

Altogether, there's no magic bullet. Yes, there's plenty of books about 7, 10, or 1024 habits of highly effective people, and no, you or me won't become highly efficient overnight by reading them, or indeed, this blog post. 
A large part of being efficient stems from good knowledge of your company, industry, people around you, and your own strength and weaknesses, including ability to organise the workday, as well as concentrate and switch from one topic to another. Moreover, time management employed by a CEO might not be right for a developer, or a mid-level manager.

As with the other posts, my goal was not to proclaim a world cure, but to share the little tips and tricks that worked for me, with the hope that they might help a few people in their careers and work/life balance.

P.S. Interestingly enough, it took over a month to write a blog on time management. That's life: it just loves irony.

Taming meeting invites

Introduction

As I was doing the detour into various programming topics, something was nagging me a bit in the background, something that was left unsaid in the previous series on Scrum.
After a short while, I realised that many of the recommendations in those series were underpinned by the invisible hand of time management, and this is the one meta-subject that haven't got the attention it deserves yet - at least on these pages.

For me, effective time management is a preliminary condition to success. Technical leaders have their attention drawn in many directions in many different ways, and many times within each day and each hour. Moreover, the more they lead the higher the frequency. At some point, we face a non-exclusive choice:

1. Work long hours
2. Ignore some of the detractors pleas for help
3. Actively decide how much time each particular subject merits

For years, I've been actively employing (3), though of course (1) is also unavoidable when the situation demands it. In a separate post, I'll bring up a few examples and guidelines on how one can quickly prioritise and allocate the right time slots, but first let's circle back: how all of this related to the subject of this post, namely "meetings"?

I can answer thus: meetings is whenever someone else decides for you how long you should spend on X. Hence, time management and meetings are deeply intertwined.

Now, before there is another step further, let me add a big disclaimer: I'm not saying that meetings are bad, and all of us should permanently sit in our rooms/cubicles/open spaces and lonely type at the keyboard. 
The art consists of organising efficient and effective meetings on stuff that matters and do not make them longer than they should be. After all, if you spent an hour in a meeting that helped neither human nor beast, you'll have either to ignore an hour-worth of something else, something that matters, or go home one hour later.

Anyhow, let's be a bit more positive: here's the list of

Three meeting types that work

The brainstorm

Few attendees: definitely less than 10, but ideally less than 5. Everyone participates, and new decisions get formulated.

Examples include a production incident war room, or interactive technical design.

The review

Also fewer than 10 attendees, with one main presenter and a preliminary material distribution. There are all kinds of reviews: deployment, design, planning, project etc. You can also add an internal product demo in the same bucket.

The update

This meetings tend to have a big number of attendees: anything from 5 up to 50000. Their purpose is to convey information, and not kick off discussions, or take new decisions.

Company-wide quarterly update, Scrum stand-up, product training, architecture overview all belong to this group.

So far, so good, so what?

Ok, so this all beyond obvious, so why even bother listing it? As with many other phenomena, the trick is not realising what works, but what doesn't. Our enemy in this particular post are inefficient meetings, and those exist in the big galactic void between those three constellations. In not so many words, they happen whenever someone does not have a clear picture of what type of meeting they are going to call in.

The organiser might go for a brainstorm with twenty people, where only two speak, and eighteen wish for earth to open underneath their feet and swallow them whole. They might call in a review where nobody knows in advance what this is about and end up spending eons discussing mundane topics. They might go as far as call in an update with 50 people who don't care about what they are going to be updated on.

Taking out a page from a diagnostician's book, let's see how we recognize an inefficient meeting once it's upon us, and how we prevent those from occurring in the first place.

How to recognize an inefficient meeting?

This section seems a bit redundant - criteria such as "am I bored out of my mind?" and "am I answering my e-mail while it's going along?" come to mind. But they are not precise: if you and the other guy are engaged and are chatting away, while the rest of the room is doing stress testing for Facebook, the meeting is still ineffective.

My criterion has been surveying the room from time to time: if 20%+ of the attendees are mentally elsewhere, then the meeting isn't going great. You can tweak the number up to derive more adjectives: for example, with 80%+ it can be safely called "waste of time and money".

This occasional visual inspection is not a meaningless exercise. It is important lesson and feedback, since some meetings are not inherently bad; it is just that the speaker needs to do a better job of engaging. One example I covered at length in the past was the status update meeting.

However some meetings cannot be made whole even with the most eloquent presenter; and it's possible to detect those in advance.

How to recognize an inefficient meeting before it starts?

Here, the meeting types come to the fore again:

The brainstorm

The most common fallacy in my book is lack previous material or preliminary discussion. 

I had the pleasure of receiving generic invites in the past; for example, "Design widget A", or "How do we make our customers happy?". 
The cause is noble, but then what helps everyone to be productive? What if one guy might have been thinking about widget A all of his wakeful hours, and everyone else would not consider widget A until you served it with watercress on a plate?

This means that rather than having a brainstorm, we are going to have an update, while we won't really know whether there are other alternatives to design the widget, and/or whether we've got the right one.

Organiser's job is making sure that all the invitees:

• Care about widget A
• Have something to say about widget A
• Know what others think about widget A

Brainstorming is much closer to playing chess than playing poker. The more we know which pieces/cards everyone else holds, the better we can choose what topics we spend time discussing, and the better we can build on each other's ideas.

In not so many words, brainstorms should not happen unless people were encouraged to think, contemplate and feed back offline first, so that we know exactly which points we are going to solve. 

Note: The one exception to that rule are war-room discussions, where there's no time for niceties, but these should be rare in a healthy organisation.

Of course, there are many other reasons why these meetings might not go well - more vocal people dominating the proceedings, political agenda etc. These are already covered to a great extent elsewhere, and are more about running rather than organising a meeting.

The review

Here, as with the brainstorm, more often than not people come to the meeting from different starting points.
For example, let's say that you're reporting on your team's progress with creating widget A. Two managers/architects in the room collaborated on said widget, two more know just what it's supposed to do, and two more got tacked on to the meeting and are vaguely aware that said widget exists.

Do you cater for the last group, and explain in minute details what this widget is for and how it came to be? Then 5.5 guys will be wasting their time (the 0.5 is you).
Do you cater for the first group and do technical update only on what happened in the last 2 weeks? Then only three people will be actually interested and get to ask meaningful questions.
Even with the best of intent, it's hard to make a great meeting out of this.

Another typical fallacy is people being undecided between review and brainstorm. I.e. they do a presentation, ask open questions, and let others have a discussion. That doesn't tend to succeed for two reasons:

1. Too many people in the meeting: brainstorms should have a small group, while reviews spread themselves out.
2. Lack of preparation: as per above, brainstorms need more thinking done in advance. 

In short, this kind of meeting works when all attendees know the subject, care about it to a reasonable degree, and already have a few questions in mind. The best way of getting there is pre-distributing the slides, answering basic questions offline if need be, and allowing attendants to be ready with the hard questions for the meeting itself.

The update

Updates have the most attendees, and tend to live or die by the skills of their presenter. Thus, they are a bit of a special case - it's harder to tell from a meeting invite how they are going to pan out. The only realistic warning sign are the words "discuss", "decide" or their siblings - they imply that the organiser is going for an open forum in a meeting least suited for that purpose.

What to do when an inefficient meeting lands in your calendar?

Rejecting is easy: Outlook has a convenient button for that very purpose. However, this is neither safe nor very helpful - especially if done without an accompanying note. 
Explaining why this meeting might not work is less fraught with political peril, though also requires caution.

For example, it is legitimate to ask for a crisp agenda definition in a brainstorm, but demanding slides in advance for a review might be construed as unnecessary pressure. Here, we are both feet within the minefield of office politics, and it has to be tread very carefully.

The important point though is that it's you who is going to go home extra three hours later, and it's you whose priorities were forced by the meeting invite. If you firmly believe that your presence there will not provide any value, then it's best to state that and bail out using the best political tactics at your disposal - or help the organiser do a better job.
Obviously, you'll also be spawning meetings yourself; here it's best to keep to the saying: "Do unto others as you would have them do unto you".

Regex engine and parallel string matching

The last post on Cython ended up with a surprising revelation on regexes - it turned out that Python matching of patterns OR-ed within a single regex was orders of magnitude faster than its regex.h cousin.

However, there were a few why-s, if-s and but-s left:

• What is so different between the two engines? I alluded to backtracking specifics unearthed in a StackOverflow post, but that explanation wasn't truly satisfactory, as there was no other public source that referred to the same disparity.
• Is this unique to regex.h? What about boost::regex or std::regex?
• Does the size of the matched string set matter? Remember, I've fixed the set of needles to be searched in the haystack, but could it be that one engine performs better on smaller/larger sets?

These are exactly the questions I'd like to answer today, and here's how we are going to proceed:

1. Prepare three extensions that implement pattern matching: one for boost::regex, one for C++11 regex, and one for regex.h.
2. Run a set of tests for 2, 20, 200 and 2000 patterns which go over the same set of inputs.
3. Compare, deduce and write a blog post.

All of these shall be unveiled shortly, with an important correction that had to be performed between steps b) and c). However, I'm jumping ahead: let's step back and look at step a. As usual, I'll share the sources, but for compile/build/embed specifics will refer to previous posts.

Here's the boost::regex one:

#include <boost/python.hpp>
#include <iostream>
#include <set>
#include <boost/regex.hpp>

using namespace std;

boost::python::list matchPatternsImpl(const string &content, const string &pattern_regex)
   {
   set<string> tempSet;
   boost::regex regex(pattern_regex);
   boost::match_results<std::string::const_iterator> what;
   boost::match_flag_type flags = boost::match_default;

   auto searchIt = content.cbegin();
   while (boost::regex_search(searchIt, content.cend(), what, regex, flags))
      {
      tempSet.emplace(what[1].first, what[1].second);
      searchIt = what[1].second;
      }
  
   boost::python::list ret;
   for (auto item : tempSet)
      ret.append(std::move(item));
   return ret;
   }

BOOST_PYTHON_MODULE(contentMatchPattern)
{
    using namespace boost::python;
    def("matchPatternsImpl", matchPatternsImpl);
}

Now, it all looks as same old, but in fact, there are a couple of novelties. First, note the usage of Boost::Python. Rather than doing complicated marshalling of C++ data structures to Python and back via Python.h, we use a nice shortcut on lines 28-32. Who said that writing C++ extensions for Python was hard?

To follow up on that, we're not returning a primitive type here, but rather a Python list. This is where boost::python::list does its little magic by defining a C++ type that can be automatically coerced to Pythonic structures.
On lines 22-25 we take our C++ set and create such a list while using C++11 move semantics to avoid unnecessary copy constructors. Another minor landmark is the usage of emplace which avoids additional copies and constructs a string directly in the container. As always, my advice is to read and experiment with those if you're serious about C++11.

However, let's get back to business. We have the Boost example sorted, so let's move on to std::regex. Here, there is a bit of an anti-climax, as it is almost a carbon copy of the code above, which is entirely unsurprising - std::regex was modelled after Boost. To get the code. just replace boost:: with std:: in the right places.

All we have remaining is regex.h, which is a throwback to my previous post. The main difference though is that we're back to conventional weapons; Cython for sure looked unorthodox, but we want to do a like-for-like comparison, so this will be a pure C++ wrapper.

#include <boost/python.hpp>
#include <iostream>
#include <set>
#include <regex.h>

using namespace std;

boost::python::list matchPatternsImpl(const string &content, const string &pattern_regex)
   {
   set<string> tempSet;
   regex_t regex_obj;
   regcomp(&regex_obj, pattern_regex.c_str(), REG_EXTENDED);
   
   size_t current_str_pos(0);
   regmatch_t regmatch_obj[1];
   int regex_res = regexec(&regex_obj, content.c_str(), 1, regmatch_obj, 0);

   while (regex_res == 0)
      {
      tempSet.emplace(content.begin() + current_str_pos + regmatch_obj[0].rm_so, content.begin() + current_str_pos + regmatch_obj[0].rm_eo);
      current_str_pos += regmatch_obj[0].rm_eo;
      regex_res = regexec(&regex_obj, content.c_str() + current_str_pos, 1, regmatch_obj, 0);
      }
  
   boost::python::list ret;
   for (auto item : tempSet)
      ret.append(std::move(item));
   return ret;
   }
   
BOOST_PYTHON_MODULE(contentMatchPattern)
{
    using namespace boost::python;
    def("matchPatternsImpl", matchPatternsImpl);
}

The Boost::Python wrapper is one and the same, and the main difference is with the regcomp, and regexec.

Right, we have step a) complete, so it's time to run and compare.

Searched string set size	Python	regex.h	boost::regex	std::regex
2	100	140	151	120
20	110	400	440	410
200	130	2310	2372	2355
2000	248	31270	31263	31278

Note: All measurements are in milliseconds, and the platform is Python 2.7.9, Cygwin/Windows, 4-core 2.10 Ghz CPU

Erm - what's going on here? It's reasonable that Python would perform similarly to the Boost regex engine, but having two orders of magnitude is entirely unexpected and counter-intuitive. This points to a defect at the user's end (i.e. my code) rather than a radical difference between the libraries. After a bit of head scratching, the problem was found - this line:

boost::match_flag_type flags = boost::match_default;

needed to be replaced with this:

boost::match_flag_type flags = boost::match_default | boost::match_any;

We have to find any match, and not necessarily scan for all of them; this is taken care of by the outer loop.
After changing the one line, we get a saner set of results:

Searched string set size	Python	boost::regex
2	100	90
20	110	100
200	130	130
2000	248	270

How much of a difference does one line make! Python terminates the regex search as soon as it finds a match, while the other libraries press on. This is why regex.h was non-performant: it simply does not have such an option (std::regex does, with very similar metrics to boost).

Here are my takeaways from this exercise:

• Boost::Python is a very convenient way of exposing C++ APIs. Unless you do highly customized parameters management or do not have access to Boost, it should be preferred to manual Python_ function calls.
• Whenever you hit a non-performant library, blame yourself first.
• Whenever doing parallel string matching via regexes, always look for match_any parameter, or variant thereof.
• Avoid regex.h, unless the powers to be force development in pure C.