Monday, December 22, 2008

Introduction to SCRUM - Part 1

I wrote this post originally on and thought I would post it here as well because not everyone knows about that site.

I started to write a response to a recent blog post about the effect that a slowing economy will have on the adoption of Agile project management methodologies when I realized from the comments on that site that there may be some folks here that are not yet familiar with Agile/SCRUM. I'm going to speak specifically about SCRUM since it's currently the most popular "flavor" of Agile. If you're already familiar with SCRUM, then this is just another SCRUM 101...if you're new to the idea, then I recommend that you read this before continuing...

If you've made it here, then you should understand what a departure these four tenets are, for most organizations, from the way they've always managed solving technical problems. As we begin discussing SCRUM, I'll be painting the picture from the standpoint of a business that is using a single small team doing internal, LOB application development. There are minor variations of this content for those producing commercial products, and for instances where you might be a cog in a much larger machine. I hope to cover these scenarios in depth in future posts. That said, you need to understand that SCRUM lays out a few, simple roles:

  • A "Product Owner" [someone from "the business"] owns ultimate responsibility for the behaviors and functionality of the system as well as the schedule of what features will be delivered and when.
  • A "ScrumMaster" is responsible for removing any barriers - organizational, technical, inter-personal, etc. that are/might be preventing the team from delivering on it's commitments.
  • A "Team" consists of everyone involved in delivery...that may include architects, developers, designers, business analysts, DBA's, QA pros, and perhaps marketing, training developers, technical writers, etc. Popular guidance suggests that a team should consist of 5-9, although it can be larger or smaller, as always, you should really have a valid reason for doing it.

and a few ground rules:

  • All meetings are Time-Boxed.
  • Development is organized into fixed duration, completely encapusulated, development windows [typically 30 days] called a Sprint.
  • The Product Owner is responsible for collecting, capturing, prioritizing, and communicating the system requirements [called User Stories]
  • Each Sprint consists of four major meeetings...
    • Sprint Planning Meeting #1 is used by the team to review the Product Backlog and discuss the desired outcome of the Sprint.
    • Sprint Planning Meeting #2 is used by the team, having used the interim between the previous meeting to estimate the size of each item on the Product Backlog, to choose how many of the top items in the list can be completely delivered in the next 30 days. This subset of the Product Backlog is called the Sprint Backlog.
    • The "Daily Standup" is a daily, 15-minute meeting used by the team to report progress since the last meeting, report what each person will be working on before the next meeting, and finally to indicate what, if any, impediments there are to their progress.
    • The Retrospective meeting happens at the end of the Sprint and is intended as an opportunity for the Team, ScrumMaster, and Product Owner to candidly discuss what went right and what went wrong during the Sprint, and then to discuss and assign specific actions in order to adapt the process for the next Sprint.
  • The "Team" is self-organizing and is responsible, as a unit, for understanding and delivering the product envisioned by the Product Owner.

So, now you should have a general feel for who's involved in the effort and what each person's responsibilities are.

Now, the first thing you need to do to understand SCRUM is to set aside your feelings and understanding of organizational structures, reporting structures, and anything else that smells of command and control. SCRUM requires a new organizational philosophy that typically manifests itself as a shift from the scenario where management controls production to one where the workers are empowered to innovate and the role of management is to alleviate anything that slows or prevents production. Naturally, worker bees are usually OK with this because they are positively affected and empowered by this shift, manager bees struggle with it because it takes their "power" away and expects them step into a role of "Servant-Leader". If your ogranization [specifically management] can't adapt to this shift and you still try to "make" SCRUM work, you're either going to have severe managerial attrition, failed projects across the board, or best case will be marginal successes with burned-out developers who feel caught in the middle/terrorized by the PM. You should also understand that SCRUM mandates very little, in fact, the SCRUM methodology is extremely lightweight and does not provide specific proscription for how you might implement it in your organization. The result of this has been mis-application of the principles and philosophies behind SCRUM, but more on that in a later blog post.

The second big mindshift you need to understand is the Agile approach to requirements. Contrary to the "waterfall" and "modified waterfall" approach to development, the collection of detailed requirements in SCRUM is deferred until the last moment. In a "traditional" or "waterfall" development process, the requirements are collected in detail at the beginning of the process, and then the development team goes off into a closet and works to deliver according to the Statement of Work. In SCRUM, a User Story is a component of work and represents a reminder for the developer to have a detailed requirements collection conversation with the Product Owner regarding the specific piece of functionality. I'm sure that right now you're saying "this can't work", "my developers are too lazy to do that", and/or "what about architecture", and a thousand other questions. I'll address these in future blog posts as well [this things is already approaching book status]. The benefit and net result of this approach to requirements is that the "customer" [Product Owner] can change the "scope" of the project [by manipulating the Product Backlog] in the midst of the project. In truth, and practice, this type of "change" is actually encouraged as long as the change to the Product Backlog is not made to the Sprint Backlog [although this can also be accepted as long as the Product Owner chooses an item of comparable size that can be moved from the Sprint Backlog to the Product Backlog and that the item being removed is not already in progress]. Because SCRUM welcomes scope changes, the product that is ultimately delivered should be an exact match to the customer's business requirement.

The first thing to do when implementing SCRUM is to evaluate your organization's willingness, capacity, and commitment to making the change. Second, make the same assessment about your willingness to be an organizational change agent because whoever adopts SCRUM will have to be prepared to make a firm and consistent stand that you CAN, with some effort, produce a ballpark estimate of time and cost to deliver a project and you CANNOT produce a detailed cost and schedule estimate, Gantt Chart, Project Plan, and the plethora of other documentation that a traditional waterfall approach mandates. This is a particularly difficult concept to endorse when you're accustomed to managers, customers, executives, etc. that demand to know the "Not To Exceed" costs and schedule up front. This "cannot" stems from the fact that, with a new team at least, you cannot estimate velocity [again, that'll be in the next post] and as such you cannot determine how many Sprints it will take to work through the complete Product Backlog.

SCRUM and Agile are not silver bullets, and adoption of them requires a serious and significant commitment from the organization to embrace change. It's not for everyone, but for those that can embrace change, do believe that it's better to deliver smaller functional bites of an application than to deliver a monolithic application, and are willing to turn their organizational structure on it's ear....then you're on the right path...please stay tuned for more.

Friday, October 3, 2008

It can't be this hard...

That was my lament as I worked through my first SQL Server Integration Services [SSIS]utility...something that was so simple in SQL Server 2000's Data Transmformation Services [DTS] has been pretty frustrating to reproduce using the SSIS tools integrated with Visual Studio 2005. We use Visual Studio 2008 Team Foundation Server with eScrum, so needless to say I found it pretty disappointing that SSIS still only works with VS2005...which means that the SSIS Package, the DB Scripts, etc. cannot be put under Source Control.

First, let me describe my development environment...I'm running SQL Server 2005 locally on a Windows Server 2008 x64 machine. On top of that, we're a Microsoft shop that is learning to use Scrum in our environment and, along those lines, we're starting to use VS 2008 with Team Foundation Server and TDD. I have an existing database that was very poorly designed and I'm working on refactoring the design to get rid of the major smells. One of the things I'm working on is taking some generic lists [like countries, states, months, weekdays, etc.] and putting them into tables. In SQL Server 2000, this would be a breeze using, bang, we're working with SQL 2005 and the Business Intelligence Studio...VS 2005 with a bow...and life isn't so easy.

My first task is to take a list of countries [swiped from the US State Department website and -->view source --> copy/paste --> some Excel voodoo --> and tada, a delimited list of countries...the database design is configured NOT to allow NULL values in any of the fields and default [empty strings] are specified...the Dept. of State list has a number of NULL values for countries without a "long name" my first challenge is that I need to interrogate each row and replace any Null values with an empty string [because, although there are defaults defined in the database, and although I know that SQL Server will allow me to specify "No Value" when inserting a record, it will not allow me to explicitly insert a NULL into a column that expressly forbids it]. After getting an Excel Connection Manager, and an OleDB Connection Manager [a SQL Server connection didn't work because of my 64bit environment...more can be found HERE and HERE] on the Data Flow canvas and configured to point to my Excel Spreadsheet [Named Range], drag a "Script Component" Transformation onto the canvas, drag the green arrow to the script component and configure it...most notable, the Script menu has a button to "Design Script" this button and viola, a VB.NET class opens...

' Microsoft SQL Server Integration Services user script component
' This is your new script component in Microsoft Visual Basic .NET
' ScriptMain is the entrypoint class for script components
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper
Public Class ScriptMain
Inherits UserComponent
Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)

If Row.ShortName_IsNull Then
Row.ScriptShortName = ""
Row.ScriptShortName = Row.ShortName
End If
If Row.LongName_IsNull Then
Row.ScriptLongName = ""
Row.ScriptLongName = Row.LongName
End If
If Row.Abbreviation_IsNull Then
Row.ScriptAbbreviation = ""
Row.ScriptAbbreviation = Row.Abbreviation
End If

End Sub
End Class

This is somewhat familiar as it is similar to the ActiveX Script task type in DTS. Next, I need to convert the Unicode strings that come out of Excel to SQL VarChar data in order to insert it into my table. This part was seamlessly handled by DTS, now I need to explicitly convert it or else I get an error indicating that Unicode Strings cannot be converted to VarChar. What a next I drag a "Data Conversion" Transformation onto the canvas and map the input columns to the output columns...finally I connect the Ouput of the conversion activity to the OleDb connection...after a day of messing around with this, I finally got it to perform the simplest task...something that would have literally taken 5 minutes in DTS.

Monday, July 14, 2008

Decisions, Decisions, Decisions

Alright, more off-topic comment...

So, I have a friend who is in a difficult situation that is not so different . Which is it, save the kid and risk everything....or save everything and risk the kid? What if the "risk" is the result of a prodigal son's long-running chain of poor decisions? What if...what if....what if...?

Thursday, July 3, 2008

Hello World

Well, hello...

Just what the world needs is another blog...another "expert" telling a vast audience how tiny and ignorant they my friend Alies says..."right"

The purpose for this blog is to provide a means of tracking and documenting some of the problems and solutions that we have found. So, here goes...