Elsewhat

Last-in/First-Out Queue for android asynchronous tasks that require background processing and updates on UI when completed

dparnas — Mon, 23 Jan 2012 20:47:41 +0000

During the development of the “Photo Voyages of Trey Ratcliff” app, I ran into a problem with file IO read operations. Bascially, the read of a large photo (300KB-900KB) would take from 300 ms to 2000 ms based on file system load. Because of this, I needed to do the read in the background so that it wouldn’t affect the UI’s responsiveness.

That was trivial to do, but I then ran into problems when the user was swiping through lots of photos in a short periode of time. The root cause was that the read operations were not able to finish before the next on came, and soon I could have 15-20 ongoing file reads. This would also cause memory problems.

In order to get around this, I’ve implemented a generic last-in/first-out queue for asynchronous tasks that require background processing and updates on UI when completed. All my file reads are queued through this and it has had a big effect on performance. Especially, since it is using WeakReferences to make sure the object can be garbage collected and checks if it is valid before the photo is read.

My code consist of three classes:

AsyncQueuableObject: The interface to implement for each task
AsyncReadQueue : The last-in/first-out queue that takes AsyncQueuableObjects
QueueablePhotoObject: Implementation of the interface for my use

AsyncQueuableObject

package com.elsewhat.slideshow.api;


public interface AsyncQueueableObject {

	/**

	 *

	 * Perform the operation in a background thread

	 * @return

	 */

	public void performOperation();

/** * Handle the result in the UI thread * * @param result */ public void handleOperationResult(); }

AsyncReadQueue

package com.elsewhat.slideshow.api; import java.util.ArrayList; import java.util.EmptyStackException; import java.util.Iterator; import java.util.Stack;


import android.content.Context;

import android.os.AsyncTask;

import android.util.Log;
/**

 * Class which implements a Last-In/First-Out queue for operations that require background processing and an update to the UI thread

 *

 * My usage is to read large photos from the file system

 *

 * @author dagfinn.parnas

 */

public class AsyncReadQueue {

	protected static final String LOG_PREFIX="Slideshow PhotoIOQueue";

	//if the tasks can be processed in parallel (for example http), increase this to 2 or more

	protected int numberOfThreads=1;

	//that workers

	protected ArrayList  readerTasks;
	Context context;

	//stack is synchronized

	Stack queuedObjects;

	AsyncQueueListener listener;
	/**

	 * Constructor

	 * @param context

	 * @param listener The listener which will be notified when a task has completed

	 */

	public AsyncReadQueue(Context context,AsyncQueueListener listener) {

		super();

		this.context = context;

		this.listener=listener;
		//lets make sure the List is synchronized

		queuedObjects= new Stack();

	}
	/**

	 * Listener which is notified when the read is finished is complete or retrieved error

	 */

	public interface AsyncQueueListener {

		/*These methods should be synchronized when implemented*/

		void onAsyncReadComplete (AsyncQueueableObject queueableObject);

	}
	/**

	 * Add an object to the queue.

	 * Will trigger the worker tasks to start if they are not running

	 *

	 * @param queueObject

	 */

	public void add(AsyncQueueableObject queueObject){

		queuedObjects.push(queueObject);
		if(!hasRunningTasks()){

			if(queuedObjects.size()>1){

				Log.d(LOG_PREFIX, "Added new queued object, queue size="+queuedObjects.size()+". New tasks triggered"+ queueObject );

			}
			processQueue();

		}else {

			Log.d(LOG_PREFIX, "Added new queued object, queue size="+queuedObjects.size()+". Processed by already running tasks "+ queueObject );

		}
	}
	protected void processQueue(){

		//if this is called, we assume there hasRunningTasks()==false has been called first

		//Log.i(LOG_PREFIX, "Starting new reader tasks to process queue" );

		readerTasks= new ArrayList(numberOfThreads);

		for (int i = 0; i < numberOfThreads; i++) {

			ReaderTask readerTask = new ReaderTask(i);

			readerTasks.add(readerTask);

			readerTask.execute();

		}

	}
	protected boolean hasRunningTasks(){

		if(readerTasks==null || readerTasks.size()==0){

			return false;

		}

		boolean hasActiveTask= false;

		for (Iterator iterator = readerTasks.iterator(); iterator.hasNext();) {

			ReaderTask readerTask = iterator.next();

			if(readerTask.isFinished()==false){

				hasActiveTask=true;

			}

		}

		if(hasActiveTask){

			return true;

		}else {

			return false;

		}

	}
	/**

	 * Stops the ongoing tasks. Any queued objects are removed.

	 * To start it again, a new call to add method must be made

	 *

	 */

	public void stop(){

		for (Iterator iterator = readerTasks.iterator(); iterator.hasNext();) {

			ReaderTask readerTask = iterator.next();

			if(readerTask.isFinished()==false){

				Log.i(LOG_PREFIX, "Stopping ongoing async reader task");

				readerTask.cancel(true);

			}

		}

		queuedObjects.clear();
	}
    /**

     * AsyncTask which represent the worker threads

     *

     */

    public class ReaderTask extends AsyncTask {

    	boolean hasError=false;
		Throwable throwable;

    	String userErrorMsg;

    	int threadId;

    	boolean isFinished=false;

    	T result;
    	public ReaderTask(int threadId){

    		this.threadId=threadId;

    		isFinished=false;

    	}
		@Override

		protected Void doInBackground(Void... arg0) {

			//loop which continues until the queue/stack is empty

			while(!queuedObjects.isEmpty()){

				if(isCancelled()){

					Log.d(LOG_PREFIX,"Async task was cancelled");

					return null;

				}

				AsyncQueueableObject asyncObject=null;

				try {

					asyncObject= queuedObjects.pop();

				}catch (EmptyStackException e) {

					//can happen if we have more than one running ReaderTask

					return null;

				}

				if(asyncObject!=null){

					//process the async operation

					//this is the time consuming task

					asyncObject.performOperation();
					//publish progress will cause the handleOperationResult to be run on the UI thread

					publishProgress(asyncObject);

				}
			}

			isFinished=true;

			return null;

		}
		/**

		 * Called when one of the queue objects have performmed the operation

		 *

		 * @see android.os.AsyncTask#onProgressUpdate(Progress[])

		 */

		@Override

		protected void onProgressUpdate(AsyncQueueableObject... asyncObjects) {

			if(asyncObjects.length==1){

				AsyncQueueableObject asyncObject = asyncObjects[0];

				asyncObject.handleOperationResult();

				listener.onAsyncReadComplete(asyncObject);

			}else {

				Log.w(LOG_PREFIX, "Unexpected number of DownloadableObject in onProgressUpdate:"+asyncObjects.length );

			}

		}
		protected boolean isFinished(){

			return isFinished;

		}
		@Override

		protected void onPostExecute(Void result) {

			//isFinished=true;

			//Log.i(LOG_PREFIX, "Async reader task " + threadId + " completed" );
			//handle a case where a new queued object has been added at the same time we are finished this task

			if(queuedObjects!=null && queuedObjects.size()>0 && hasRunningTasks()==false){

				//Log.w(LOG_PREFIX, "Queue not empty at end of run. Restarting async reader tasks"  );

				processQueue();

			}

		}

/* (non-Javadoc) * @see android.os.AsyncTask#onCancelled() */ @Override protected void onCancelled() { Log.i(LOG_PREFIX, "Download task stopped"); super.onCancelled(); } } }

QueueablePhotoObject

package com.elsewhat.slideshow.api;


import java.io.File;

import java.io.IOException;

import java.lang.ref.WeakReference;
import com.stuckincustoms.slideshow.premium.R;
import android.graphics.drawable.Drawable;

import android.util.Log;

import android.view.View;

import android.widget.ImageView;
/**

 * Queueable object that cause a heavy file IO read load (300-2000 ms)

 *

 * Will use WeakReferences in order to make sure objects are still needed when it is ready for processing

 *

 * @author dagfinn.parnas

 *

 */

public class QueueablePhotoObject implements AsyncQueueableObject {

	protected SlideshowPhoto slideshowPhoto;

	protected File rootFolder;

	protected WeakReference weakRefslideshowView;

	protected ImageView myImageView;

	protected boolean wasGarbageCollected = false;

	protected boolean wasOutOfMemory = false;

	protected Drawable result;

	protected String tagOnCompletion=null;

	protected int maxWidth;

	protected int maxHeight;
	/**

	 * Constructor The View is stored as a WeakReference

	 *

	 * @param slideshowPhoto

	 * @param imageView

	 */

	public QueueablePhotoObject(SlideshowPhoto slideshowPhoto,

			View slideshowView, File rootFolder, String tagOnCompletion, int maxWidth, int maxHeight) {

		this.slideshowPhoto = slideshowPhoto;

		this.rootFolder = rootFolder;

		this.weakRefslideshowView = new WeakReference(slideshowView);

		this.tagOnCompletion=tagOnCompletion;

		this.maxWidth=maxWidth;

		this.maxHeight=maxHeight;

	}
	/**

	 * Perform the operation of reading the photo from file in the background

	 *

	 */

	@Override

	public void performOperation() {

		View slideshowView = weakRefslideshowView.get();

		if (slideshowView == null) {

			wasGarbageCollected = true;

			return;

		} else {

			try {

				result= slideshowPhoto.getLargePhotoDrawable(rootFolder,maxWidth,maxHeight);

				return;

			} catch (OutOfMemoryError e) {

				Log.i("QueueablePhotoObject",

				"Out of memory while getting drawable");

				wasOutOfMemory = true;

				return;

			} catch (IOException e2) {

				Log.i("QueueablePhotoObject",

				"IOException file reading photo "+ slideshowPhoto,e2);

				wasOutOfMemory = true;

			}

		}

	}	
	/**

	 * Handle operation results will be run on the UI thread

	 * and will be responsible for setting the read drawable to the ImageView drawable

	 *

	 */

	@Override

	public void handleOperationResult() {

		View slideshowView = weakRefslideshowView.get();
		if(wasOutOfMemory){

			return;

		}else if(wasGarbageCollected){

			return;

		}else if (slideshowView == null) {

			Log

					.d("QueueablePhotoObject",

							"Drawable loaded, but imageview has been garbage collected since read started");

			return;

		} else {

			ImageView imageView = (ImageView)slideshowView.findViewById(R.id.slideshow_photo);

			imageView.setImageDrawable(result);

			if(tagOnCompletion!=null){

				imageView.setTag(tagOnCompletion);

			}

			imageView.requestLayout();

			return;

		}

	}

public String toString(){ if(slideshowPhoto!=null){ return "QueueablePhotoObject:"+slideshowPhoto.getTitle(); }else { return super.toString(); } } }

RedditTV for android

dparnas — Tue, 19 Apr 2011 14:00:24 +0000

I’ve just released the RedditTV for android app. It makes YouTube videos popular on reddit.com easily available on your android device.

Got a Media Center with a big screen?
Now you can even send the video from your phone and directly to your Media Center! Just long-click any video you want to send.

Currently supports:
-Google TV
(requires Google TV Remote or Logitech Harmony app)
-XBMC
(requires configuration of connection parameters in settings)

Please upvote the related reddit post and provide feedback on existing or new functionality.

Most likely, the code will be open-sourced in the future.

Reddit TV – Videos popular on Reddit – Optimized for Google TV

dparnas — Sat, 05 Feb 2011 16:27:08 +0000

I’ve just released what would be best characterized as a mashup between youtube and reddit.

It is available at http://reddittv.elsewhat.com/ or through the chrome web store .

In essence it provides you with a youtube player which uses the social site reddit.com as the source for the playlist.
It uses the Youtube API and the Reddit API and is all done through client-side javascript coding.

Thanks to Salman Arshad which provided an example of a video player using the Youtube API. This became the base for my implementation.

Social Charity Hub: A novel new way for charities to use Social Media formed at #Innowe10

dparnas — Fri, 19 Nov 2010 14:06:17 +0000

One of the most interesting parts of Innovation Weekend is that you are working on actual business cases and challenges that matter for non-profit organization. In some cases the business case becomes just as, or more, important than the actual technical implementation. We believe that the scenario we worked with during #Innowe10 in Las Vegas is one of these.

The key challenge all non-profit organizations face is how they can reach potential donors and volunteers. Once they reach them they are generally very good at presenting their story and have a reasonably high rate of conversion. Today, traditional channels such as e-mail, postal letters, sms are used for reaching the potential donors and volunteers. However, we believed that proper use of social media could drastically change this and that was the business case we set out to solve in our team at Innovation weekend.

Social media strategy for any company is not a trivial task and it is very difficult to get right. For a non-profit organization the challenge is to get the attention of all the users of facebook, twitter, linkedin, MySpace etc. Collecting followers is generally considered the key metric for how well you are doing. However, non-profit organizations are not managing to gain the same follower numbers as celebrities and corporations. This is illustrated in the table below showing number of followers (people who like) from facebook today 20.10.2010.

Facebook followers
Non-profit	Celebrity	Corporation
Doctors without Borders: 343 515	Michael Jackson: 22 190 941	Starbucks: 15 862 452
American Red Cross: 211 776	Lady GaGa: 21 031 946	Coca-Cola: 14 766 636
Arts Umbrella Vancouver: 537	Britney Spears: 5 290 852	SAP: 41 309

Our key insight was that in order for a non-profit to use social media in an effective way, they need to piggyback ride on the networks of celebrities and corporations. That’s how we make sure our message of charity reach as many people as possible. (people spreading it in their own personal network can of course be done as well, but we believe it has less potential )To motivate corporations and celebrities to spread our message of charity we quickly realized that we had to stimulate friendly competition and apply free market principles. We do this through “The Challenge” and the “Challenge Update” mechanisms

“The Challenge”

The Challenge” is posted to one or more social media channels by our Social Charity Hub solution. “The Challenge” is posted to the page of corporations and celebrities that we want to challenge to raise money for our charity campaign. (new corporations and celebrities can of course be added as the campaign proceeds). In the example above Lady GaGa has received a challenge which contains the following:

A description of the charity
A challenge to raise as much money as possible
A competitor/archrival which will be their primary opponent (here Britney Spears
Link to our donation page with parameters identifying that the donation is from Lady GaGa’s network

Once the challenge is posted on the wall of the celebrity or corporation, it is visible to their followers. But what we really want is for the challenged celebrity or corporation to define their own strategy for how to raise donations. They might do a pledge saying that for every dollar donated, they will also donate the same. Or they might send an e-mail with the link to all their employees and encourage them to participate. This is where the free market principles kicks in and since this is combined with the competition, people are sure to become very creative in how to raise money. The corporations and celebrities also get positive publicity because of this campaign.

“Challenge Update”

Once all the challenges have been made by our Social Charity Hub, the solution as a whole will constantly monitor the transactions made and use analytics to see how the competition is proceeding. Below we’ve illustrated the high-level overview picture of the general flow.

The analytics solution consists of a dashboard solution allowing the competitors and the campaign management to viewing the current status of the charity. However, the Social Charity Hub will also retrieve aggregated data from here, which it will use as a basis for deciding if new updates should be published through social media channels. These new message should update everyone on how the competition is proceeding and raise the competitive instincts in all the people involved. An example of such an update is shown below, where it shows that Lady GaGa is actually trailing Britney Spears by 12%

Challenge completed

When the campaign has reached its target set by the campaign manager, the Social Charity Hub will also publish a detailed thank you note to each celebrity and corporation involved through social media. This will include a link to a dashboard built with SAP Crystal Dashboard(Xcelsius) that allows anyone to analyze all the donations and the different competitions defined.

SAP Technology used

A lot of exciting SAP technologies were used for this scenario, but the details and experiences really warrant a separate blog. We weren’t able to connect all the different systems within the limited time of Innovation weekend, but perhaps you’ll see parts of this solution on SAPCodeExchange in the future. The technologies were as follows:

StreamWork with Gravity collaborative process modeling plugin
Used for working with the business case and defining the process for rolling out a new campaign and updating social media
NW Business Process Management 7.3
Used for modeling the overall process governing the Social Charity Hub
Visual Composer 7.3
Used as user interface for BPM screens
WebDynpro for ABAP
Used for the donations page
ABAP
Used for storing the donation transaction data
SAP Crystal Dashboard (Xcelsius)
Used for analyzing the transaction

The team

The team dynamics were amazing to see in action. It is very interesting to see how engaged people are, even though they’ve just met. The team consisted of:

Dagfinn Parnas Bouvet ASA (Norway) – SAP Mentor
Paul Aschmann Faist Chemtec INC. (South Africa)
Carlos Pereira Cisco Systems (Brazil)
Sergio Oliveira FH Consulting (Brazil)
Srini Marada Commercial Metals Company (USA)
Ivan Mirisola SAP (Brazil)
Marlo Simon Complex (Brazil)

Finally, we would like to thank Craig Cmehil, Marilyn Prat, Kai van der Loo and all the other SAP and non-SAP who helped organize and participated at Innovation weekend at SAPTeched in Las Vegas 2010.

Interview on Lean, Agile and Future of the Enterprise II at #SAPTechEd2010

dparnas — Fri, 19 Nov 2010 14:02:59 +0000

Following up on the success of the 2009 TechEd interview on “How Lean, Agile and Scrum will shape your company of the future” …

Vijay Vijayasankar (manager at IBM running very large international SAP implementations), Mark Finnern (SAP Chief Community Evangelist) and myself did a follow up on Lean, Agile and Future of the Enterprise II.

It is available from http://bit.ly/ca7QQE

Smart dad tip#1: Keep your nightvision with a Pirate eyepatch (Yarr)

dparnas — Tue, 27 Jul 2010 20:06:55 +0000

Having become a dad for the first time recently, I’ve realized that there are a whole new set of parent related challenges.

In addition to all the standard ones, I caught myself in an unforseen situation.
My son Morris usually goes to bed at 22:15, but doesn’t like to sleep before 23:15. Therefore, I usually lie next to him and get connected with the outside world through my smartphone.
However, as the bedroom gradually gets darker, I found that the smartphone ruined my nightvision. It was therefore almost impossible to see what Morris was up to.

Being inspired by the Mythbuster pirate special in which the myth ”Pirates wore eyepatches to preserve night vision in one eye” was deemed plausible. I immedately came to the following conclusion:

This calls for an experiment!

An Ode To Lexus Smart Tech

dparnas — Fri, 04 Jun 2010 05:30:31 +0000

Lexus IS 220d champagne

Originally uploaded by dagfinn77

Today I am trading in my Lexus IS 220d which I’ve had for three years. I’ve enjoyed this car to a high extent and part of this is to do with its smart use of technology.

So what are the Smart Tech on Lexus? Here is my top ten list:

Smart Entry and Start system (aka Keyless Go)
No more need to find the keys in order open the door and start the car. Amazingly practical and I’m looking into the options of having the same kind of tech in my house
Tilting side-mirrors when reversing
When reversing, the side-mirrors tilt slightly in order for you to see any obstacles more easily
Touch screen navigation
The touchscreen navigation of the Lexus is superb. For example when you want to enter an addresse, it filters out the possibilities in your current country as you type and when there are only a few locations left it displays a user friendly list. Only complaint I have that it locks down when the car is moving for safety reasons eventhough you might have a passenger who use it safely
Rear-view camera
I was laughing about this feature when I bought the car used, but actually it is very practical. Allows you to see if any kids toys or other obstacles are in the way
(and you could probably make a cool rap video from the camera)
Welcome home lights
As part of the Smart Entry system it displays a small light under the side mirrors when you approach the car. Just a way of saying welcome home
Electronic sun-shade
There is an electronic sun-shade in the rear window that can be opened and closed by the click of a button
Seating preferences store
For each front seat there are three preferences that can be stored for the electronic seats. An improvement to the system would be to automatically connect the right preferences with a smart key
Surround sound system
The Mark Levinson 14 speaker and 300 watt sound system is amazing and is very tailor made to the car experience. They even made it so the internal mirror moves in beat with the bass-line if you play high enough :) I actually prefer to listen to some types of music in my car, eventhough I have decent enough stereo systems at home
VDIM stability system
Sometimes I’ve been driving the car on ice, and even though it is a rear-wheel drive car it is amazingly stable in tough conditions. The Vechile Dynamics Integrated Management system brings together all the related stability technologies. You need to experience first hand how it corrects a small slide through individual wheel braking
Adaptive front-light system
The adaptive front-light system has a swivelling projector which moves as you turn into a corner. An improvement would be to include the welcome home lights in the mirrors when going at slow speed (as some of VW models does)
Bluetooth phone integration
Nothing unique about Bluetooth phone integration, but it is very practical. I’m using this in conjunction with my Bluetooth on Motion Android app, which automatically turns on Bluetooth on my smartphone when it detects movement of a certain speed over time

That’s the end of my top 10 11 list. Going to miss the car.

Lazy developers hate agile and scrum

dparnas — Thu, 20 May 2010 06:35:52 +0000

During the #sapphirenow key note, CO-CEO of SAP Jim Hagemann Snabe said that he saw no drawbacks of introducing scrum to part of SAP product development and the challenge now was to scale it to the entire company.

I tweeted this, but received immediately a response from an analyst that ”Developers hate scrum”

I thought this was a somewhat strange statement, as in my experience 98% of developers love scrum and the rest fear it because of the transparency.

I then got two more interesting replies from twitter, one from @webtonull stating the rest fear it because it is experienced as surveilance (meaning your not really doing it as intended). The other was from @cvitan who came up with the catch-line “lazy developers hate scrum”.

Myself and Twan van den Broek, who has experience from running scrum in SAP projects (see his presentation on the topic), figured out we’d do a short video to share with the world and the SAP Community. This video is shown below

VMWare image for Shapado (open-source stackoverflow.com clone)

dparnas — Wed, 21 Apr 2010 13:31:57 +0000

Having been intrigued with the effectiveness of http://stackoverflow.com , I’ve been interested in the possibility of using such a solution internally within the enterprise for other purposes than IT.

Since Stackoverflow.com currently is not open-sourced, I looked at the alternatives and the most promising to me was Shapado . It is an open-source solution, but also provides a hosted option.

Therefore, I’ve created an virtual image which can be used in order to test out the solution locally. You can test it by:

Downloading the image from http://dl.dropbox.com/u/4379928/Shapado/turnkey-rails-shapado.zip and unzip it
(if you want to start using dropbox, you can use this link in order to give me some sorely needed extra space)
Download the VMWare player from http://downloads.vmware.com/d/info/desktop_downloads/vmware_player/3_0
Start the VMWare image through the .vmx file and log in with user:root password:
(afterall it is a test image, not a production one)
Get the ip-adresse from the VMWare image and on the client (not in the image) modify your host file (C:\Windows\System32\drivers\etc) so that it contains
localhost.lan group1.localhost.lan group2.localhost.lan
From the VMWare image start the MongoDB through the script /opt/shapado/startdb.sh
From the VMWare image Shapado application by going to directory /opt/shapado and running ./script/server
Access shapado through http://localhost.lan:3000

The image was created with basis in the excellent Turnkey linux VMWare images by roughly completing the following steps:

Get the Rails turnkey linux image from http://www.turnkeylinux.org/rails and unzip it
Start image from 1. in the VMWare player
Upgrade to ruby 1.8.7 by following the excellent instructions at http://www.vanutsteen.nl/2008/06/29/installing-ruby-187-and-guessnet-on-hardy/ (if not you get an error for the count method when posting a question)
Install MongoDB according to the instructions at http://shapado.com/questions/how-to-install-shapado-on-amazon-ec2
Run apt-get install devise -v 1.0.5 and retrieve the specification file ref. http://shapado.com/questions/unpacked-gem-devise-1-0-5-in-vendor-gems-has-no-specification-file
Follow the instructions form installing Shapado at http://gitorious.org/shapado/shapado/blobs/master/README (note you will have to use a workaround for the rake gems:install step , see 6. above)

Comment to “Why Agile didn’t work for me”

dparnas — Tue, 20 Apr 2010 10:23:30 +0000

Fellow SAP Mentor Vijay (@vijayasankarv) recently posted a blog entry where he states the reason why Agile didn’t work for him: http://andvijaysays.wordpress.com/2010/04/16/south-beach-diet-didnt-work-for-me-and-neither-did-agile-development/

It is a good read, though I don’t agree with the conclusions.

I’ve posted a long comment on the blog which is reproduced below. It will be interesting to see the announced response of the http://enterprisegeeks.com

Thanks for the post Vijay,

Whilst I might not agree with your conclusion, I think you’ve brought up a lot of important and critical points that both customers and contractors need to be aware of when going down the Agile path.

You don’t provide much information on how you’ve tried Agile and in a future blog post it would be interesting to get some details. Hopefully this discussion will not turn into a witch-hunt;I’ve seen too many examples of people giving valid criticism of Agile being meet with the response “well then you didn’t do agile right”.

I’ll try to give my views and experience on the key points you bring up. Since Agile is such an umbrella term, I will respond with basis that Scrum as methodology/framework with no premature adjustments of the rules of the game. Afterall, the rules of Scrum are there for a reason and it’s generally not recommended to modify them before you are a mature Agile organisation (which will take years and years).

Would you pay an Agile contractor to build your deck?

I need a deck built, and I don’t know how to build one. So I hire some one else to do that. And this dude tells me he can do it one of two ways. 1. I can discuss with him on how a basic deck has to be built, and he can give me an estimate. Or 2. I can give him a rough idea, and he will start building the deck, and every day or two – he and I can get together to see how it is going and what changes I want, and I can pay him for work that he has done every day. Of course in the second option, he cannot tell me how much the deck will cost me or how long it will take him to build me one – but I can see progress every day. I don’t know about you – but I know what option I will go for. Same thing with my clients – if the work and schedule are not predictable – it is hard for them to just pay as I go.

First of all, contract negotiation for any IT project is difficult and if possible I would recommend the customer to run the project themself and staff it with a combination of own personell and contractors with relevant experience. This will require significant effort from the customer in the area of IT project portfolio management and prioritization, but such a move will also be benificial in that it will move the company away from the evils of budgeting (ref. Beyond Budgeting http://www.bbrt.org/).

For many companies the above alternative will not yet be an option and they would like to buy a tailor-made solution which fits their current needs. Traditional projects will estimate the cost either based on previous experience, preferred as it minimizes risk, or by partioning the task at hand into seperate pieces that can be estimated. Scrum doesn’t say that you cannot do a similar estimation in order to make a contract with a customer. However, it states that you must expect these requirements to change during the duration of the project. Therefore, it places more value in responding to change than to fulfill a contract defined prematurely.It also states that the requirements in general should be in the form of user stories which focus on the business value, thereby improving the communication with the guy how pays the bill(typically in the form: as a , I want because ).

Customers in general want to have a quoted priced with an RFP and Agile projects are no exception. The price model is usually target price as this allows the risk to be shared with potential benefits for both customer and contractors. The major difference is that the estimation is done at a slightly higher-level and should contain requirements that can be directly into a product backlog for when the project starts. Planning poker estimation is also very useful in order to do relative estimates (can of course also be used in waterfall projects). In Norway we have a Scrum version of a recommended IT contract agreement (PS2000) http://bit.ly/albLgg (link unfortunately only in Norwegian).

To conclude, Agile doesn’t equal no estimation and need to face the realities of the business world.

Global and Agile are like Oil and Water – They don’t mix.

I forgot the last time I had all the IT and Business guys and gals working in the same location. Most often – we have people working on a project from all over the globe. It is seldom possible to get teams from Japan, India, Germany and US to be on a conference call. And even if you do – without a written document explaining the problem and/or solution – it is hard to get anything done.

From my experience and what others have quoted on empirically studies; having a co-located team is always the most effective. Anything else is sub-optimal (ie. waste) and requires mitigating actions.

If you have a Scrum product with several teams, I believe that it should be an absolute requirement that the team is co-located or at least in a similar time zone. This is essential in order to build the self-organising team, gain collective ownership and share competence. However, different teams within the same product doesn’t necessarily need to be located in the same time zone as long as they have some means of coordinating modifications that affect other teams.

Again, you have the same problem with more traditional project and Scrum is merely bringing the waste up to the surface where you can perform mitigating actions. A mitigating action might be to provide more detailed specification on the interfaces between teams and there is nothing from stopping you to do this whilst still being Agile.

There are only so many super stars in this world

In any given team, the norm is to have a few super stars, several average performers and a few below average ones. This has a direct effect on pulling off delivery in an agile fashion. Not every one can go away with minimal instructions and come back with the right solution, and right questions to ask for next day.

It can be argued that Agile needs less programmers, and hence you can keep just the super stars and let go of every one else. This argument works only if the scope is small. The day still has only 24 hours – and even super stars cannot work 24X7 all the time to get everything done.

You say:” Not every one can go away with minimal instructions and come back with the right solution, and right questions to ask for next day.” However, a well-comprised and self-organizing team consisting of both juniors and maybe a super star will combined be significantly better suited than a single super start. Agile is a team effort!

It is very interesting to discuss how to best use your super stars. Scrum believes that team work is the best way of harnessing the value of your project members. By sharing a collective responsibility, knowledge is shared and you are on the way of producing high-performing teams (much of Scrum research is focused in this field where certain teams are 50 times more efficient than others). This is a pardigm change and requires a lot of focus, which is one of the main reasons you have a dedicated scrum master (who is not at all a project manager) who constantly challenges the team to reduce waste and improve. It is not unheard of for the Scrum Master to ask the team who is the most critical person on your team, and instruct them that for the next sprint he will not be allowed to code, only mentor the others. Some super stars who are used to be locked in a cage and then produce amazing results will have particular problems in doing this transformation.

Most projects do not have dedicated business users.

The whole idea of Agile is to get something back to business users faster than in a waterfall model, and keep them informed of progress frequently. This is very good – except it won’t happen in most projects. Most companies find it hard to dedicate full-time business people to a project. More often, business users have to do project work on the side. So – even if the tech team wants constant face time , the chance of that working out is low.

A few product companies have tried out Agile successfully. However, in the couple of cases where I got a chance to talk to people from the team – it seems, the customer was almost never present in the scrum meetings. Instead, the product manager assumed the role of being the customer voice. If that is the case, I would have to wonder aloud “so what is new?”. Product manager is not the one who has to live with the product after it is out – it is the customer.

How do you expect projects to succeed when the people how know the business and are going to use the solution afterwards aren’t involved sufficiently? We how work in IT need to become more professional and part of that is making clear the premisses and expectations we require in order to implement a successful project. User involvement is one of, if not the, key success criteria of IT projects.

In my experience, the minimum requirements in Scrum project should be that the business provides a product owner which is responsible for the prioritization of user requirements for each sprint based on the business value they will have. This will not be full time job. Also, send the person to a product owner scrum course.

If the customer is not present in Scrum meetings, you have been too pragmatic and most likely this will affect the project negatively. Instead quote the absolute rules of Scrum in this area and stand your ground that you cannot be Agile if the customer doesn’t do his part. It will then be visible to the organisation and steering group that the project has a high risk of failure due to lack of involvement of the customer.

What works for you – people over process? or process over people?

This is what it boils down to – Agile manifest claims the superiority of people over process. And traditional waterfall puts process over people. For me – as a manager of big teams, with a deadline and budget that seldom cuts me any slack – I would trust a good process to compensate for the human errors most of the time. I think it is a high risk for me to trust that every one in the team will perform to the same high standards. Having a disciplined process helps me get the best out my team, despite not every one being a super star.

You say:”I would trust a good process to compensate for the human errors most of the time.”

I say that I would rather be able to show business value to the customer for each sprint and avoid a big-bang GoLive.

Would you build a mission critical solution using Agile?

I don’t know – but I keep wondering if NASA would use Agile for designing systems for their next space mission? I somehow can’t see any application that is very important and deals with life and death, or dealing with large amounts of money – like air traffic control or stock market transactions – to use Agile.

I hope every one knows about 3C. This was the big Chrysler project that was the poster child for XP (Extreme programming). Well, guess what – that didn’t quite work out. Check this out http://en.wikipedia.org/wiki/Chrysler_Comprehensive_Compensation_System or just google and you will find several interesting takes on it.

Building a car and building a software solution have similarities in design. However, there are significant differences too. Just as it is impossible to build a car without knowing what exactly needs to be built – we cannot build a good software solution without knowing what the heck we are building. Otherwise, even if you follow Toyota manufacturing Process to build a Camry – you could end up with a Chevy Malibu.

A very good question, that I don’t really have the background to answer. I would recommend in general that customers do not choose mission critical solution as their first Agile projects as there is a lot of lessons to be made.

The C3 reference is interesting and new to me. It does however reference only Extreme Programming (XP), which in my view primarily consist of good developer principles and not a framework such as Scrum for securing progress in your projects

Is Agile really good for long-term stability of a solution?

Most software is built by first putting a framework in place, and then building on top of it. It is the rough equivalent of putting a good foundation for your house. If you know that you will have a house with 2 floors – you will probably put a certain amount of concrete in your foundation. Now that you have finished the house – and for some reason, you now want one more floor – would you put one more floor without also doing some additional foundation work? And if you build by constantly messing with your foundation – I am definitely not going to buy that house or rent it for living. Same thing with software – the way sprints happen from what I have seen, I doubt if it is possible to put a solid foundation in place.

Solution architecture is hard to get right in any project, and perhaps even more so Agile projects.

Scrum’s approach is a focus on refactoring as you gain more information. It is common to run one or more architecture sprints at the start of the project, where the focus is analysing the right architecture and implementing a user story which requires functionality in all layers.

When evaluating of different tools from a vendor, prototyping is an invaluable method and should be encouraged instead of an endless list of requirements that all vendors say yes to anyhow.

It is fun, but is that good enough for customer to pay?

One thing I really like about Agile is that every one involved in it usually has more fun than in a waterfall project. This improves team morale and all the good stuff – temporarily. When you cannot get user involvement, or if a blame game starts – where there is no documentation to go back and clarify what every one agreed to, this fun does not always last. And fun for the development team, while important in a project, is not the sole reason why some one pays for a solution.

Waterfall is not such a terrible thing to do, as its opponents make it to be. And it is a big exaggeration to say the team does a very long design up front, and that user gets to see things only at the end. That is not how most projects run. Waterfall can also have users involved more frequently. Also, you can build in plenty of feedback options and test driven development in a waterfall project. Also there are very strong visualization tools that can give the users a taste of the solution very early in the process. For example – iRise is a great tool to use for that. Do try it out, and your whole perspective will change. Also, once you have a good change control process established, changing requirements can be handled very effectively.

Agree that fun for the development team is not high on the list of priority. Hopefully, the sense of ownership the individuals experience and the self-organizing team will mean that the blame-game is avoided.

Again thanks for the dicussions and looking forward to the reply from the Enterprisegeeks :)