Monitoring Hadoop MapReduce Applications

April 29, 2013

While users have access to many tools that assist in performing large scale data analysis tasks, understanding the performance characteristics of their parallel computations, such as MapReduce jobs, remains difficult.  Step #1 is to create a test suite that you can reliably run after every change.

http://www.myhowto.org/java/hadoop/2013/03/06/monitoring-performance-of-your-hadoop-mapreduce-application/


Faster & Secure Web

April 15, 2013

you should definitely try disconnect.me   It makes your web browsing experience faster and I was amazed at the number of companies tracking my browser usage (Now blocked with disconnect.me)


Lessons learnt in IT Offshore Management for Startups

April 12, 2013

No matter how brilliant your new software  may be, it’s doomed if you can’t figure out how to make it efficiently, consistently and economically.

Starts up engage offshore teams due to lower costs & ease of getting teams engaged with tools like e-lance.  However, Creating brilliant and functional software is more difficult than ever due to a lack of offshore team  management skills . My experience at managing offshore IT organizations has taught me some important lessons about software design and production that, if heeded by software startups, provide an opportunity to bring innovative products to market without suffering setbacks – or even failure – from preventable mistakes.

1 – Get Inside the Software Shop

I’ve met too many people in this game who spend several weeks researching software vendors, in some cases even identify top talent, hand over project details  and get on the phone once a week for a status report.  This leads to enormous problems at launch time – Badly designed product and a mismatch between expectation & software delivered.

Speak on a daily basis.  Speak directly to the individual developers, testers, Business Analysts and not just the project managers.  You’ll be amazed at what you learn about the software development process and your software development partner.  Seemingly small pieces of information from these conversations can later help you refine product design or even clue you in to larger issues with the partner’s management.

 

2 – Build Prototypes Closer to home

You often get what you pay for in this realm, so it isn’t where you want to pinch pennies. Use the prototyping phase to refine, refine and refine some more. That way, when it comes time to spend money on software development,   you only have to do it once.

The added benefit of prototyping close to home is that you can iterate faster.  Expedited turnaround times accelerate overall development cycles, and in turn, reduce development costs. Rapid development also gets your product to market faster than the competition!

 

3 – The job does not end after launch

Once you launch (congrats!), resist the temptation to sit back and watch it all happen. To the contrary, this is when real development begins.  Ensure you have left a good budget for further development.    I have seen far too many software startups spend money pre-launch, leaving no allotment for development activities once they have received feedback from real users.  This is where prototyping also helps.  Continue to meet with offshore partner on daily basis.

 

 


Should I use traditional HPC or Hadoop?

April 12, 2013

Excellent blog by Guident – http://tinyurl.com/bne6t8k  

comparing a  Hadoop versus Traditional High Performance Computing.  A specific use case of reading large log files is compared and Hadoop is the winner in terms of performance.

What happens if we have access to traditional HPC hardware?  Should we use Hadoop on HPC?   Check out excellent article by S.Krishnan on this – http://tinyurl.com/cwbzvof .   Results are not conclusive, but an interesting read.

Bottom line, it appears to be dependent on the use case?  Has anyone done a more detailed comparison?