Organizing Data Warehousing Teams
(Originally published by TDWI) By Maureen Clarry & Kelly Gilmore
(This article is available for download, as a PDF document.)
Organizing a data warehousing team can be tricky business, even for relatively simple projects. Technical and managerial challenges abound, and the frenetic pace of corporate life can make weeks seem like days. As with any IT project, the key is to judiciously employ a sound methodology. This article outlines one such step-by-step process for building and organizing DW teams, including several best practices we’ve developed over the years.
Before launching into the organizing process, there are three fundamental concepts that should be appreciated. First, flexibility is crucial. Circumstances in the business world constantly change, and your team must be malleable enough to accommodate such change. Second, any data warehousing initiative should be managed on two levels: program and project. The program is the overall vision of the warehouse: where it’s going, whom it serves, the value it provides. Projects are the incremental deliverables that increase functionality and deliver value to the business.
The third concept is that of the warehouse lifecycle. Data warehousing luminary Hugh Watson defines three stages in the lifecycle of a warehouse: initiation, growth and maturity. How you go about organizing your team will change depending upon where your warehouse is in the Watson lifecycle. During initiation, for example, you may need more ETL skills than during maturity; in the growth stage, you may want more involvement from the business community. So it pays to understand where you are in the overall process.
Step 1 – Define necessary skills
One of the most common mistakes made in building DW teams is starting the process by creating an organizational chart. Boxes are drawn to represent team members, and titles are then assigned. Roles and responsibilities become secondary to abstract titles and hierarchies. This is a recipe for trouble, because it doesn’t take into account the specific skills and responsibilities necessary to complete the project successfully. A better way to begin the process is to create a comprehensive litany of those required skills and responsibilities; worry about the org chart later.
Step 2 – Inventory the team
It’s impossible to intuitively know the abilities of everyone on a DW team. Such information must be painstakingly inventoried. The skills tallied should be broken down by program and project, with three subsets: business, functional and technical. Business skills are the most abstract, such as understanding industry trends (program), or knowing how to anticipate what users will want (project). Functional skills include developing an information architecture (program), or data modeling (project). Technical skills relate to the use of specific hardware and software. It’s our experience that a four-level ranking hierarchy works well for indicating expertise: novice, intermediate, advanced and expert.
Step 3 – Consider career goals
One way to build flexibility into your team is by actively considering the career goals of your team members. When you create the skills inventory, take into account not only which skills each person currently possesses, but also which ones they’d like to learn or improve. This serves the dual purposes of creating an environment in which employees can grow and learn, while giving your team sufficient depth and breadth in skills to achieve the flexibility so often necessary for timely success.
Step 4 – Conduct a skills gap analysis
The next step is to create a matrix showing the gaps between what you have and what you need. Wherever such gaps occur, you’ll need to do one of three things: train someone, hire a staff person or bring on a consultant. This is where the time-consuming inventory work begins to come in handy, because you’ll know which skills can be taught by members on your team, who wants to learn them; and which skills will need to be brought in via new employees or contractors.
Step 5 – Define responsibilities
At last, you can start handing out duties. The Executive Leadership Group (ELG) defines responsibility as a task that has been assigned, coupled with a commitment by an employee to complete that task. Clarity in this part of the process is obviously crucial. As always, it’s best to communicate tasks both in person and on paper. Documentation is necessary to ensure that everyone knows who’s responsible for what.
One effective tool for defining authorities is the RASI chart. This simple spreadsheet lists each task in rows, with the columns being used to show who’s Responsible for each task, who Approves the task, who Supports the task, and who is Informed about it. One pitfall to avoid is having more than one person responsible for the same task. Many people can support or be informed; only one can be responsible.
Step 6 – Create accountability
Accountability exists when responsibilities are clear and there are associated consequences, according to the management experts from ELG. It is imperative therefore that all team members understand the consequences of their actions. And remember that consequences don’t need to be negative: people who perform well should know they’ll be rewarded; just as those who under-perform should know the impact of their inaction.
Step 7 – Define authorities
Often, DW team members are “matrixed” to the DW project, leaving the project manager without authority to choose team members, remove team members or attach consequences. Accountability breaks down unless the project manager is given these authorities or negotiates with the functional managers to have input to influence them. Define the authorities of the project managers and the functional managers before personality conflicts arise.
Step 8 – Determine the structure
Now it’s time to build that org chart. You have a solid handle on which skills are necessary, and who will do what. Codify the structure as best you can in whatever form of organizational diagram is appropriate. But remember the importance of flexibility: just because the chart may work well at first, doesn’t mean it should remain static forever. Circumstances will change, and you should be prepared to modify your team’s structure accordingly.
Step 9 – Hire to fill the gaps
As with the other steps in this methodology, hiring to fill gaps should be done methodically. Start by defining the hiring process. Prioritize your requirements and be sure to obtain consensus on job descriptions. Take time to articulate relevant interview questions in advance – don’t play the interviews by ear. Make sure to interpret answers in a consistent fashion as well.
Step 10 – Communicate the structure
The importance of making sure everyone understands the structure can’t be overestimated. To ensure effective communication, use whatever tools are effective with your team: RASI charts, organizational diagrams, and/or job descriptions. In the absence of effective communication, success will be scarce.
Conclusion
Of course, each situation, each project, each team is unique. There is no cookie-cutter approach that works for all situations. The above methodology will, however, put you on the right track. But be wary of complacency: when circumstances change, you must be prepared to make some changes of your own. Versatility will undoubtedly come in handy, and when the time comes, you’ll be glad to have detailed documentation on the skill sets and long-term goals of your team members.