Use Continuous Integration to Avoid IntegrationHeadaches

I've heard several stories about problems with integratingthe work across multi-site teams. Even with a lot of care put intodefining interfaces, integration problems still rear their ugly head.Often this is because it's very hard to properly specify thesemantics of an interface, so even if you have all the signaturesright, you still get tripped up on assumptions about what theimplementation will actually do.

The earliest XP practice to really take hold at Thoughtworkswas Continuous Integration, and we've become so accustomed to it thatwe were determined to use it with our offshore development. So fromthe beginning we've put everyone on a single code base, withCruiseControl running to both build and run tests. This way everyoneis kept close to the mainline, whatever their location happens tobe.

Everyone has been delighted with how well this works. Itsmain benefit is that problems that plague other groups withintegration just don't happen to us. The continuous integration andtest process flushes out many integration problems very quickly, as aresult they can be fixed before they become hard to find.

CruiseControl's web page allows all sites to see what's goingon abroad. First thing in the morning you can check the CruiseControlweb page to see what changes have been made by the other sites. Itprovides an easy way to catch up on what's going on at the other endof the world.

This does require good build discipline, where developersstrive hard not to break the build, and fix it immediately if itbecomes broken. It's generally accepted practice that if you commitchanges to the mainline, you should not go home until you havereceived the email message from CruiseControl that says that yourchanges resulted in a successful build. A late night bad build is muchmore serious when the remote office is running off the same build.

Although world-wide Continuous Integration is resoundinglypopular, we have run into some problems. Communication pipes aren't aswide and reliable as we'd like, so many source control operations canget awkward from a remote site. In general we keep the build serversin the same site as the majority of developers, but remote sites canfind it takes an annoyingly long time to get a fresh update from themainline. The longer the communication lines are, the more they areprone to anything from glitches to lines being down for a while. Having the repository accessible 24 hours makes it annoying to take itdown to do backups. All of these issues would mitigated by a clusteredcode repository, but we haven't experimented with anything like thatyet.

We have found that some source-code control systems areparticularly prone to long and painful check-outs -particularly if they are not configured well. It's worthspending time on finding ways to speed up check-out times, andindeed changing source code control systems if you're stuckwith one that doesn't handle remote working very well.

(Interestingly people assume that these communication problemsare specifically a problem with a remote site like India - butwe've found problems often occur with the infrastructure in thewest too. Continuous Integration requires good connectivity,often better connectivity than people are used to.)

Even with the best connectivity, however, there can beproblems if a system requires services that are locked insidean onshore firewall. You'll need to build goodtest doublesfor these services so the system can be effectively tested inthe development environment.

Have Each Site Send Ambassadors to the Other Sites

As I've said above, agile methods stress the importance offace to face human interaction. Even if everyone cannot be co-located,moving some people about clearly helps a lot. From the beginning wedecided to ensure that at all times there was someone from the US teampresent in India to facilitate the communication. Such an ambassadoralready knows the US based people and thus adds his personal contactsto help everyone communicate.

We've now expanded this to several levels. We found it usefulto send a US developer and a US analyst to India to communicate onboth technical and requirements levels. It's also valuable to sendsomeone from India to the US team. The plane fares soon repaythemselves in improved communication.

One of the benefits of a business-oriented ambassador on theoffshore team is that it helps provide business context to theoffshore team. Building software off just a list of requirementsmisses out a lot business context - developers are told what to dowithout being told why it's important. The why often makes a bigdifference in doing the what properly.

An important part of the ambassador's job is to communicategossip. On any project there's a lot of informal communication. Whilemuch of this isn't important, some of it is - and the trouble is thatyou can't tell which is which. So part of an ambassadors job is tocommunicate lots of tidbits which don't seem important enough for moreformal communication channels.

We usually rotate the ambassadors every few months (and insome cases every few weeks), since ifan ambassador spends too long abroad they lose contact with home. Thismakes it easier for the ambassadors, who don't want to be away for toolong. It also allows more people to get to know the remote team byspending time as an ambassador. In choosing ambassadors it's veryimportant to pay attention to individual needs and preferences. Somepeople don't want to spend several months thousands of miles away fromhome, so they shouldn't be ambassadors.

We've also found that it's important for project managers tospend some time as ambassadors. Much of a project manager's job is tohelp resolve conflicts and flush out problems before they becomeserious. Experience working on both side of the telephone line isreally important for them to do that effectively.

Ambassadors are an important part of trust building andgelling across the wire. As a result it's essential to sendout ambassadors as early as possible in the project. If aproject runs for a while without ambassadors in place,miscommunications and bad feelings will develop that take alot of work to fix. Ambassadors reduce this risk, but need tobe there early on before problems start setting in.

Use Contact Visits to build trust

Ambassadors are semi-permanent people who spend severalmonths in the 'other' location. This is vital, but not enough. Thereare also needs to be further contacts that involve a wider range ofpeople. These visits help to create and maintain the relationshipswhich need to be in place for remote communication to workeffectively.

You can think of two kinds of contact visits: seeding visitsoccur early the in the project and are intended to create therelationships, while maintaining visits help keep the relationshipgoing.

Seeding visits should be planned for early in the project andshould be pretty substantial in length - two weeks is the minimum.It's important for seeding visits to be working trips, since the wholepoint is to get people used to working with each other, so they shouldbe arranged around some joint task.

• Send some onshore customers and project managers to anoffshore site to create an initial release plan.
• Have the offshore analysts come onshore to take part inearly requirements gathering sessions.
• Have some onshore developers visit the offshore team forthe offshore team's first iteration working with an existing codebase.
• Have the offshore team visit the onshore location -particularly if it's on a client site.

Whatever the reason, remember that the primary purpose of thevisit isn't to do the task, but to the build the workingrelationships. It's a common mistake with these things to cram so manytasks into the visit that there's little time for the vital humancommunication. So keep the work pace relaxed, and don't be afraid toschedule in something informal where the hosts can show the visitorsaround and get some informal contacts going. Dinners and sightseeingcan often be the most useful activity that the visitors do with thehosts.

It is important to get seeding visits in as soon as youcan. Without decent personal relationships it is likely thatyou will run into problems due to miscommunication and lack oftrust. These problems will take a lot of energy to repair andcan easily cause lasting damage to the project. So seed earlyto head these problems off.

A special variant of the seeding visit is important if youhave developers on multiple sites from the beginning. In thiscase it's important to get the developers, or at least thesenior developers, together to build the first fewiterations. It's in these iterations that the crucialarchitectural decisions will get made, it's important to haveproximity during this period. Without this you can get a splitwhere different teams make different decisions, or one teammakes a decision that the other team doesn't understand.

Once the seeding is done, then less intense maintenancevisits should be used to keep the contact going. These can be shorterbut should be frequent, a one-week visit every couple of months is theminimum for these. Again they should be focused on necessary tasks,one good task to do with these is a retrospective. Don't be afraid touse longer and more frequent visits if you sense communicationproblems between the teams or if a significant issue comes up. We'venoticed that it's particularly useful to have a longer contact visitif significant changes occur to the system design.

Gifts, particularly those that help spread culture, arealways worth bringing. We particularly like people to bringgifts of snack food or sweets that are specialties of theremote location.

Don't Underestimate the Culture Change

One of the hardest parts of introducing agile methods into anorganization is the cultural change it causes. Indeed we've found thatthis is the major reason why organizations have problems with adoptingagile methods. Many companies operate with a command and control modelwhich assumes that seniors make decisions and lower level people carrythem out. To make agile methods work you need much more autonomy and decisionmaking by the doers.

We find this to be a big problem in western companies, butthe problem is amplified in Asia since Asian cultures reinforcedeference to superiors. (A training course I saw from amajor Indian contracting company defined management as “the science ofcontrol”.) In this environment people are often discouraged fromasking questions, talking about problems, warning about unfeasibledeadlines, or proposing alternatives toperceived instructions from superiors.

Western teams need to be wary of this tendency and shouldpush back when they sense an eastern team is passivelyagreeing. Beware that polite acceptance is often a sign of animportant issue not getting discussed. In addition westernteams need be wary of sounding authoritative - which reinforcesthis kind of situation.

The bad news for this is that getting teams to be morepro-active is an uphill battle, and one that inevitably takes a lot oftime. You can never assume that problems will be raised, even whenthey are spotted. Getting people used to a distributed control styleof management takes longer than you think.

But there is good news. Once people realize they have thefreedom, and the responsibility, of making decisions - they seem toreally relish it. Several of our Indian team told me how their friendsin other companies cannot believe how much autonomy they are given.This autonomy is a great motivator, allowing people to be both moreproductive and able to grow into greater responsibility. Offshore teammembers gain the trust the understanding to make decisions instead ofwaiting for the onshore team, which lead to a lot of delays. For me one ofthe most interesting things we will discover is what the longer termeffects are of this cultural impact, both in Asia and in the West.

Seeding visits play an important role here. People are muchmore likely to raise issues if they have a good personalrelationship.

Even talking about these cultural issues can causeproblems. Some (western) industry analysts who saw a draft ofthis article said that this section was patronizing andoffensive. One of our developers in Bangalore said I'm being fartoo mild. Another commented that it's an issue, but questionedas to whether it was worse that it is in many westerncompanies. But there seems to be some consensus that there arecultural forces in Asia that reinforce command and control, andthat this is changing.

(This is a particularly sharp issue for Thoughtworks. We havea pronounced anti-authority attitude that's striking even in the US.We decided from the beginning that we would retain that same culturein India. I'm glad to say that we certainly seem to besucceeding.)

Use wikis to contain common information

We've played around with various ways to hold commoninformation, our favorite so far is a wiki. Wikis work well becausethey are simple to use, can be worked with any browser, and are simpleto set up.

Any common information can be put there,story cards, designguidelines, build instructions, notes on progress - anything that needs to be writtendown for reference by the team. We've found it's very useful to usethe change notification capability that many wikis have, so that pagechanges trigger notifications through email or an RSS feed.

Wikis are by nature unstructured, and this lack of structureis part of the benefit. The team can usually evolve their ownstructure on the wiki as the project grows. This does mean thatsomeone needs to act as a gardener to make sure it doesn't getovergrown.

Use Test Scripts to Help Understand the Requirements

With greater distance, you need to put more ceremony intocommunicating requirements. We've been able to do that while stillsticking to many of the techniques that we use in single-sitedevelopment.

Increasingly I've found that more mature XP teams useacceptance tests as ways of communicating requirements. Such teams gettest scripts written out before the start of an iteration to helpclarify the requirements and give the development team a concretetarget to aim at. One style that's worked well is for a US basedcustomer to write a short narrative (a couple of pages) to flesh out afeature (story in XP lingo). An Indian based analyst/tester thencreates test scripts for this story. This can be done either forautomated or manual testing, although we very much prefer automatedtests. As the scripts are developed the US and Indian analystscoordinate by email and IM as well as regular (2-3 times a week)conference calls to review the test scripts.

We've found that this has very much helped both the Indiananalyst and the US customer really understand the requirements.Writing out the tests forces the Indian analyst to really understandwhat's needed and to ask questions of the US customer as questionsturn up. The developers find it easier to ask questions of the Indiananalyst rather than dig through the test scripts, so having an Indiananalyst/tester is still important. Search engines are good, but humansare often easier to work with.

The biggest problem we found with this technique is toengage the client staff in doing it. As a result on themajority of projects we haven't been able to do it - but whenwe have we've found the approach valuable.

Use Regular Builds to Get Feedback on Functionality

When people consider the requirements gathering in agilemethods, they often fail to see the importance of the feedback loop.Often the requirements process is seen as analysts providingrequirements, which developers go off and implement. At some laterpoint somebody checks to see if the developers have implementedwhat they were asked for. On an agile project, the close proximitybetween customer and developer allows the customer to monitor progressmuch more frequently, which allows them to spot misunderstandings morequickly. Furthermore a partially developed system can also educatethe customer, for often there's a difference between what's asked forand what's needed - and usually that's not apparent until there's someworking software.

Having regular integrated builds allows a US customer to pulldown last night's work and try it out. While this isn't quite asimmediate as co-location, it still allows the customer to correct anymisunderstandings quickly; as well as allowing them to refine theirown understanding of the requirements.

To make this work, it's critical to sort out the environmentissues so that you properly duplicate the environment on bothsides of the ocean. There's nothing worse than onshore peoplepulling down a build, finding problems, and the offshore peoplebeing unable to duplicate the problem due to environmentconfiguration issues. Make sure the environment is sorted outearly, and ensure someone is around to fix any environmentproblems if they appear.

Make sure people look at what's built regularly, even if it'sonly partial functionality. The quicker someone looks at apiece of work, the quicker they can spot anymiscommunication. Often people like to wait until something isfinished before showing it to others. In these situations,however, it isn't worth the wait.

Scrum has long advocated that the development team does ademo to the customers at the end of every iteration. We'veadopted this practice pretty widely now - we call it ashowcase. With remote teams we like to do a remote showcase,where the offshore developers show the new features in thesoftware with the help of remote desktop software. Getting theremote team to do this is another example of taking everyopportunity to build links between offshore and onshore, andbetween developers and customers.

Use Regular Short Status Meetings

Agile methods promote regular short status meetings for theentire team (Scrums in Scrum, stand up meetings in XP). Carrying thisover to remote groups is important, so they can coordinate with otherteams. Time zones are often the biggest problem here, particularlybetween the US and India where any time is awkward for somebody. Inour earlier projects we found that twice a week stand-ups seemed to work welland provide enough coordination.

In our more recent projects we've made a greater use of thewiki, and this has reduced the need for cross-shorestand-ups. Now we do stand-ups within a shore's team, but notbetween the different shores. We do however do daily cross-shoremeetings, but these don't involve the entire team - just thosewho focus on the cross shore collaboration. With small teams,however, we do do the cross-shore stand-ups.

We've found that it's a good habit to start conferencecalls with chit chat on local news. Recent odd bits of localcolor - politics, sport, weather - helps each side get a senseof the broader life context on the other side of the wire. (Itstrikes me that this is probably obvious to anyone other thanus nerds - when people are heavily task-oriented it's easy tolose sight of the broader context in which we live.)

With time zone problems it's important for both sides to giveand take in picking the time for calls. One of our clients would onlydo the calls during their business day, which forced the Indian folksto come in at very awkward hours. Pushing all the onus on one side toaccommodate isn't helpful for smooth collaboration.

This lack of knowledge, or consideration, runs into otherareas as well. One project scheduled a major release duringthe Indian holiday ofDiwali, which is the equivalent ofasking an American team to working overThanksgiving. Fortunately we were able to convince them tochange the date. Even in less serious cases remember thatholidays are rarely shared, so you'll often get times when oneteam is in holiday mode while another team is in a regularwork week.

Use Short Iterations

In general agile methods use shorter iterations than a lot ofother iterative approaches. At Thoughtworks almost all projects useiterations of one or two weeks in length. A few of the more experienced Indian developers haveworked at places which use two to three month iterations and theyreport that the shorter iterations are much better.

A couple of years ago our distributed projects tended toprefer two week iterations since they found it was difficult to useshorter iterations, but this has changed. Now we've learned how to doone week iterations comfortably on a distributed project.

Use an Iteration Planning Meeting that's Tailored for RemoteSites

On most of our projects we've found that a planning meetingat the beginning of each iteration that involves the whole team reallyhelps to get everyone coordinated on the next chunk of work. I'venoticed that most of our projects have come up with their ownvariation of the Iteration Planning Meeting (IPM) to suit the localcircumstances. (This kind of self-adaptation is an important part ofagile processes.)

A remote team adds its own set of constraints, particularlywhen you have awkward time zone issues. However despite the pain ofawkward meeting times, we still find the IPM to be extremelyuseful.

Before the IPM the US customer sends narratives for eachscheduled feature (story) which we like to turn into test scriptsbefore the IPM. During this period any questions are handled by email.Just before the IPM the development team breaks the features down intofiner grained tasks. These task breakdowns are shared with the US forfeedback.

All of this pre-work shortens the phone call which nowconcentrates on any issues that come up from the task breakdown. Wefind the calls usually last around a half to a couple of hours. It isimportant to keep the actual phone meeting short, as these kinds ofremote meetings are particularly arduous.

When Moving a Code Base, Bug Fixing Makes a Good Start

Two of our projects involved taking a large (hundreds ofthousands of lines of code) code base and moving substantialdevelopment of the code base to the Bangalore lab. In both of theseprojects the Indian team began with a few iterations of bug fixingbefore they started adding new functionality.

Starting with bug fixes allowed the Indian team to becomefamiliar with the code base, before they did substantial work on it,since bug fixes involve more code reading than changing. Although thisworked well, there is some concern that more experienced people mayconsider it to be a stigma to be doing only bug fixes. While somepeople may perceive this as a problem I believe that working on bugfixes or very localized feature changes is one of the best ways to getfamiliar with a large new code base.

The nature of communication of bug fixing can also make itan easier activity to work with offshore. Onshore teams can spendtheir day mapping out details of bugs, which can becommunicated to the offshore team and worked on during theonshore night. The onshore team can then review the fixes thenext day. I must stress that this is not more efficient thanhaving an on-site team fix the bugs, due to the communicationdifficulty, but it can be a less complicated way of workingwith an offshore team.

Separate teams by functionality not activity

Much of the traditional thinking on the onshore/offshoreboundaries is based on the activity that people do. So analysis anddesign is done onshore, construction done offshore, and acceptancetesting is done onshore. This obviously fits well with the waterfallmodel.

We've found that contrary to this, matters improve when wemake the offshore team handle as many activities as possible. So weprefer to see them do as much analysis and design as possible, subjectto the limitations that the requirements are coming from onshore. Whenwe do split an effort with onshore and offshore development teams, wedo this along functionality grounds rather than activities. We breakthe system into broad modules and let the offshore team tackle some ofthese modules. However unlike most groups that do this, we don't makea big effort to design and freeze the interfaces between thesemodules: continuous integration and weak code ownership allow themodule interfaces to evolve as development goes on.

An important part of this is to grow the analyst part of theoffshore team. The more someone local to the developers understands ofthe business, the more the development team can developefficiently. Instead of having to wait overnight to answer a question,developers can get answers right away - which removes blocks to progress.All this means that you have to focus on growing the business knowledgeof the offshore analyst. This takes time, but the local knowledge is avital counterpart to the business knowledge onshore.

A corollary to this is to not divide teams by horizontally(having one team do presentation and another do database). Ingeneral Iprefer a functional staff organization - and remote teams exacerbate the problems ofdividing teams by layers.

The most important thing to remember here is the dominantpower ofConway's Law - the structure of the system will mirrorthe structure of the team that built it. That means that it'simportant to separate distributed teams by modules that are asloosely couples as possible.

Expect to need more documents.

Agile methods downplay documentation from the observationthat a large part of documentation effort is wasted. Documentation,however, becomes more important with offshore development since theface to face communication is reduced. This work is, in a way, a wastesince it wouldn't be needed if the whole team was co-located. Butgiven the offshore model, you need to do them - so they are part ofthe price of doing things offshore. That's a price in the time forpeople to write them, the added time because it's harder for people tounderstand many things from a document, and also a price infrustration when people are using them.

As well as documents, you also have more need for more activecollaboration tools: wikis, issue tracking tools and the like. On thewhole it seems that it's often better to favor tools that impose lessstructure, that way the team can fit them better into how they want towork (one of the reasons that wikis can work so well.)

I always advise teams to check their documents into aversion control system so people can easily get the most up todate material. This is particularly important when you aredoing remote work.

Whether it's documents or anything else, remember that otherpeople's templates won't work for you, and you won't come up with theright scheme at the beginning. Make sure there's plenty ofcommunication about the form of the documents and how well they areworking. Expect to evolve the structure of documents as you learn whatworks best for your team.

There are two keys tosuccessful documentation on agile projects. The first is finding thepoint of “just enough” documentation. This is difficult to determineand will vary by project. Fortunately, the iterative nature of agiledevelopment allows you to experiment until you get it right. Thesecond key to successful agile documentation is to not get attached toit or have unrealistic hopes of keeping it updated. Documentation mustbe created to serve a specific purpose, and after it has served thatpurpose you'll all probably have more important things to do than keepupdating the documents. It may seem counter-intuitive, but it's oftenbetter to produce fresh documentation the next time some is clearlyrequired. A side benefit of starting over each time you need todocument part of your project is that it's great incentive to keepyour documentation efficient!

--[Simons]

Get multiple communication modes working early

Different communication tools work for different kinds ofproblems. At the minimum make sure you have a wiki, instantmessaging, and good telephone connections.

Instant messaging is good for quick questions and answers,but a particular strength of IM is that it tells you when peopleare available at their desks. You do have to get into the habitkeeping your IM status fresh, but that information is alwaysuseful. We see a surge of communication during overlap hours,which is particularly valuable when those overlap hours are short.

Many companies these days blockinstant messaging believing it to be a distraction. It alwayshurts us because it stops ThoughtWorkers asking questions offolks on other projects. It particularly hurts offshoreprojects as it removes informal one-to-on contacts.

If it's more than a quick few messages then it's best toswitch to phones. Make sure that it's easy to just pick up aphone. People shouldn't be deterred by fears of the cost ofphone calls, a phone call will usually save money onmisunderstands.

We've found a lot of value in videopresentations. Short lectures about the background of theproject that can be recorded and sent over to the remoteteam. These are often easier to prepare than a document,easier to sit through (if they're not too long) and cruciallythey also help the personal contact, since it's easier to geta broader picture of someone from a video than from adocument. They aren't so good for details, but work better fora broad picture.

Email can often be a mixed blessing. In particular we'vefound that it's good to discourage person-to-person email in favor ofbroadcast newsgroups or mailing lists. It's too easy for a piece ofinformation not to go to someone who needs it, or be unable to findit. By posting messages and requests in a newsgroup, everyone can seethe messages and it's easy to search. People find it easy to skip overthreads that they aren't interested in.

Getting multi-cast rather than one-to-one communication canalso be done with real time tools. A couple of teams havereported good effects from usingCampfire. (IRC could be usedin a similar way, although I haven't heard it mentioned yet.)

Perhaps the trickiest thing to sort out is how tocommunicate the big picture - the vision of the project. Mostcommunication, and discussion about communication,concentrates on the day-to-day details. It's important to getthese right, but there's a danger that in focusing on theday-to-day nobody pays attention to the overall vision.

This can hurt because lots of people make habitual smalldecisions based on their perception of the big vision. Thesesmall decision add up, so if there's no communication of thebig picture problems can sneak up on you.

This issue is particularly important forcommunicating the business context of a project. Often remotecommunication focuses too much on tactical details - what needto be built this week. But many technical decisions need abroader strategic context - so it's important for the remoteteam to have a broader picture of the direction the project,and the business, wants to take.

This kind of communication is often lacking in on-siteprojects, particularly when there is a lot of organizationalbarriers between business and technology. Remote developmentteams exacerbate this problem - just as distributeddevelopment exacerbates most communication difficulties.