At the Epicenter of Big Data Research


I am wiped out as I am on the plane home from my most recent experience at MIT.  This week the topic was Big Data.  Big Data is a big topic around the water cooler these days, so I thought it would be important to learn more about it.  The title of the course was Big Data: Making Complex Things Simpler.  This was a 2 day Executive Education course designed to brief managers and executives on this exploding field.

First of all, if you are interested in this topic I highly recommend making the investment of time and money to attend a future offering.  Professors Erik Brynjolfsson and Sandy Pentland, leading researchers in Big Data present a well-crafted curriculum that connects a great deal of their research around how Big Data now provides the technology framework to do, very quickly, what researchers have done for years – create hypotheses, design experiments and analyze results.  Because of Big Data technologies, organizations can become more data-driven in their operations and/or product development.  Key issues including data privacy and data ownership are discussed as well, but this landscape is changing very rapidly, so it was challenging to go into too much depth.

If you are looking to better understand Big Data technologies, this is not the course to take.  However, if you are looking to spend a few days better understanding the ramifications of Big Data and how they impact organizations, I highly recommend making the investment.  The participants in the class contributed greatly to the discourse, which I appreciated as well. Plus, it was a great place to network and find out what is happening in other industries related to Big Data.

Blog Post Review – Agile Requirements: Not an Oxymoron

I was reading an interesting blog post about requirements within Agile processes.  It’s a topic I have been discussing with my team as we struggle to find the appropriate balance of process and agility.

The big “a ha” for me was the concept of “done” requirements.  The author has written about this in a book, so I will be digging in further to further suggest process changes.  We use a “done” criteria in our user stories and it seems there are parallels to this concept.  This would give the developers more insight into the state of the requirement which would allow them to commit to implementing the feature.

It also implies that there is a need to have dedicated business analysis resources on Agile teams.  We do have this role on the team, but it has not yet morphed into the dynamic role it needs to be.  We’ll keep working on it.  A few years ago an Agile Business Analyst was indeed an oxymoron in the Agile community, I think, because we didn’t understand that requirements (in the form of user stories) are still needed.

Thanks to Ellen Gottesdiener, the author of the post.  Very enlightening, indeed.  I feel like what I have been saying to my team for the last 3 years was correct.

Software Quality Assurance and Social Media — Is There A Connection?

A few months ago I posed the question to a vendor of mine — what do social media technologies have to do with software quality assurance?  Does it even matter?  I still think they are or should be connected but I don’t know exactly how.  So, I am creating this post to hopefully generate some discussion and more ideas.

Here are some ideas I had:

1.  Use microblogging technology (e.g. Twitter) internally within the dev team to quickly communicate what is happening real-time.

2.  Could a test plan be a Wiki page rather than a document?  I suspect this wouldn’t work in a regulated environment, but maybe not?

3.  Use microblogging to ask quick questions to anyone on the team — might reduce email clutter?

4.  Analyze social media traffic to determine how the customers are using the product being tested.  Could this influence the direction of the test plan?

5.  Microblogging with linking to point to specific issues in the team system (e.g. TFS) needing attention – maybe a defect, or a user story?

6.  Other indirect uses: development team morale, searching microblogs for internal trends (create some retrospective ideas?), etc.

What connections do you see?  Can’t wait to uncover some ideas!

Traceability in Agile: How Do We Do That?

The good news is that the product my team is developing is being adopted broadly within the organization.  The bad news is that in the future in order for my software to be used in products subject to regulated enviroments we will not pass scrutiny.  Seeing this problem out there on the horizon has inspired me to challenge my team to take the next step of the maturity journey and evolve our Agile/Scrum process.  I am not an expert on the specifics of achieving compliance, but it seems to me it is pretty simple.  Basically it entails documenting what is to be built, build what you say you will build, then show evidence you tested that you built what you intended.  Also, involved is change management over the whole lifecycle.  So if that’s the general process, I keep peeling the onion back and finding traceability at the root.

I brought this to the team, and we’re having some trouble getting traction on trying to improve our traceability story.  The lack of action makes me wonder if I am asking for the wrong thing.  How do the rest of you Agile-ites deal with traceability?  Specifically, I am looking for the ability to look at the potential impact of change by knowing all stories impacted should we change a particular story.  In the RUP, we had a neat little report called the traceability matrix.  It was easy to see impact of changes.

Please feel free to share your tips to getting to a solid traceability story.  Without it, I am likely shut out of some important business opportunities to “sell” my application into scenarios where regulation is required.  Looking forward to getting some ideas.

Partial Resources: It’s Always The “Other Project”

In a perfect world, we’d like all of our Scrum team members to be fully dedicated and committed to delivery on a project.  The reality is that scrum masters and product owners have to deal with less than perfect conditions under which they have to deliver high-quality software.  It’s always the “other project” that prevents our resources from executing on the project we are managing.  So, what do we do?  I think there’s a simple strategy that can help everyone involved.

I recommend deliberately and consistently scheduling project available time.  Let’s say you are 25% allocated to another project.  In a 40 hour (yeah, right!) work week, which equates to 10 hours of available time to work on that project.  That could be done 2 hours per day, or maybe one full day plus a couple of hours on another day.  What that means is that during those scheduled times, you are unavailable to do anything but work on the other project – no meetings or anything – like you are not even there.  Use your online calendar or team project management system to clearly lay out the schedule for the whole team to see.

It's always the "other project"...

Consistently following a schedule will help you time-box this effort, help with consistent planning and execution, plus it will help colleagues respect these boundaries such that everything can get done.  It will take a little discipline to respect these boundaries, but I believe it is what is necessary to mitigate the risk of partial resources being on the project.

Proximity: How Much Does It Really Matter?

We are facing a great problem at work — growth.  With that, comes the whole notion of space planning to house new people joining the team.  And adding space and things like cubes, phones and so on do has cost.  As I enter into this space planning mode, I can’t help but wonder if it would be worth the effort to look at the whole team and assess their proximity to one another while I am doing the space planning.

So, the big question is that if I am going to make a facilities investment, what would be the improvement in productivity for our Agile software engineering teams if a vast majority of them have improved proximity to one another.  Right now, most people will travel to meetings on the same floor of the building.  I’m thinking they miss a lot of interesting and inciteful “drive-by meetings” (as I like to call them) because they lack close proximity.  So, how much is it worth?  Is it worth the effort to care about it?

Doing Offshore Development: My Key Success Factors

A little over two years ago, I took on the endeavor of offshoring development of my software project.  This was something new to me, from an execution point of view.  As I embarked on the process of engaging an offshore team, I got a lot of advice from colleagues.  Some of it was even solicited by me (rim shot).  To be sure, much has been written and said about this topic, but here are my key learnings that hopefully will help you as you start down the offshore path to software development.

Cost Savings Should Not Be Expected Early

Many firms go into this with an expectation that development costs will go down between 50 and 65% right away after deciding to do offshore.  This is an unrealistic expectation that needs to be managed.  My research revealed that it takes 1-2 years to fully integrate and optimize an offshore team into the local team.  In the interim, there are many hurdles to overcome such as language, cultural differences, understanding individual contributor technical expertise, shared understanding of the software process used and the like.  Overall, expect productivity to go down and costs to actually rise in the first 6-9 months (I think team size is the big variable here – Mythical Man Month rules apply) while the team figures out how to work together effectively.

Recognize That Optimizing Offshore Development Is a Journey

In my experience, integrating offshore is an exercise in maturing overall team performance.  It’s really no different than working with a team that sits in your midst.  Mistakes will be made, miscommunications happen and you get some downright bad results early on.  It may seem like nothing is happening because you can’t see people working on your software project.  In my experience, I had to manage the local offshore team’s expectations more than anything.  They had to adjust to working with colleagues who they perceived were not as strong technically.  My team early on overcompensated for what they saw to be as poor results.  This type of behavior actually proved to be counter-productive to the development of an overall working relationship and software engineering process.

My recommendation:  keep the process as simple and clear as possible and build upon successes.  At some point, you as the leader will need to inspire another evolution of process as the team matures.  There will always be room for improvement.  Don’t try to do it all at once.  We try to infuse internal process improvements into each development iteration to drive the most urgent improvement opportunities.

Agile Process Is The Way

In an offshore development model, I have come to embrace Agile as the only way to develop quality software.  We use the Scrum brand of Agile and I found that with two-week sprints I was able to frequently provide feedback and check in on progress.  I observed how the team interacted with each other, which people seemed to consistently step forward in sprint reviews and planning sessions as well as probing deeper on process improvements.  When there are gaps, I could provide advice and direction on what needed to change.  I also used these as opportunities to form opinions on which team members were top contributors and which were at the bottom.  In partnership with my offshore partner, we were able to replace the lower performers quickly and increase team effectiveness. 

HD Video Teleconferencing Equipment

The best technology investment I made was to outfit our team scrum room with high-definition video teleconferencing capabilities.  I vividly recall the day we enabled the system — it was so cool to see the people we are working with live and in person.  It didn’t take more than a few days for the programmers on both ends to get used to using the equipment — the daily scrums were much more compelling in HD video!  It seemed like the table became one, longer table that everyone sat around.  The names became people that we could talk to and laugh with even though we are a world apart geographically.  Picking up on non-verbal cues really helped.

Let’s face it, building software with a geographically dispursed team is really hard.  I don’t think you can get around it.  But, I hope these few ideas I have based on my lessons learned can help you create conditions on which to build successful software projects.  It is a journey indeed.