How to make estimate (in software development)

One of the tricky question for a software engineer is to estimate when his code will be ready. It is a tricky question because ‘ready’ is generally not well defined. Ready can mean a lot of different things, depending to whom you ask. The developer might assume having his code working and tested, the program manager most certainly means available for ABTesting, while the leadership team is looking for the time when the change are available for all users. I have a mind list of the milestones that needs to be hit for any new development work, and I’m using it whenever I need to provide estimate or when I’m asking for estimate. It also helps to clarify what are the expectations on the qualities of the work that needs to be accomplished.

  • Analysis: about understanding the requirements and how the system will have to change to support them
  • Design: about closing the technical design (overlapping all involved team), enabling to have a clear end to end picture of the change
  • Coding: actually writing the code change
  • Unit testing: UTs can come before the coding if you live in a test driven development (TDD) world, this is about making sure the functions of your code are working properly in the obvious cases
  • Reviewing: having peer reviewing the code change
  • Iterating on Coding / Reviewing: depending on the developer experience, the criticality of the code changed, and the pickiness of the reviewers, the coding and the reviewing can be based on multiple iterations
  • Testing: validating the code is correctly behaving, even in non-obvious cases, validating overlap with multiple pieces works as intended too.
  • Experimentation / ABTesting: validating that your changes are actually bringing value to the customers and that your key performance indicators (KPIs) are, at least, still looking good (ideally they should improve).
  • Deployment: make the new functionalities available to all your users

Given the above list, when I’m talking estimate with my gang, I can get suspicious if someone tells me that he will be ready with his change in a couple of days!

Introduction to Machine Learning, from data acquisition to a production service

In this post I want to share some notions of machine learning and a how-to get started with Microsoft Azure ML studio.
Machine learning is a quite complex topic, especially if you want to understand the theory and algorithms that are behind it. If you want to go deeper in understanding ML I would encourage you to register for the free online course by Andrew Ng on coursera.org. I have done it, it is really great but be ready to invest enough time and brain waves
J.

Thankfully, ML studio offers a simple way to do machine learning without the need of understanding any of the algorithms as long as you follow a workflow. My simplistic workflow is split in 4 steps: data acquisition, data preparation, training and tuning, web-service deployment. Let me go through all these steps.

Machine learning

Data acquisition

Machine learning needs two things to work, data (lots of it) and models. When acquiring the data, be sure to have enough features (aspect of data that can help for a prediction, like the surface of the house to predict its price) populated to train correctly your learning model. In general, the more data you have the better so make to come with enough rows!

Data is generally stored as a CSV file (comma separated value file created with excel) or any other supported type of dataset. Currently the supported max size is 1.95 GB (which is enough for anything not “big-data”).

Handling the data: data preparation


Once you have the data available from Azure ML, it’s time to prepare it. There are multiple transformation operations supported: filtering, manipulation, sampling, and scaling and reducing. Full documentation for data transformation. To start, you would want to scrubber missing values, edit the columns, and then split your dataset for training and validation. This will be fed to your training model along with the learner selected.

 

Training and tuning

The first thing you need to figure out first is what type of analysis you need to run (see full documentation). It can either be a classification analysis (spam vs non-spam), a clustering analysis (automatic classification), or a regression analysis (for prediction and forecasting). Once you have this figured out, what you need to do is compare the results of multiple models to know which one is the most efficient for your dataset. The logic here is to:

  1. Select a model with default value
  2. Train the model
  3. Score the model
  4. Evaluate the model (to figure out which model is the most efficient)
  5. Sweep the model (to figure out the best configuration for your model)
  6. Evaluate the model
  7. Save the trained model (to be used in production)

When comparing the evaluation of models, you will want to validate that the main values are going up (like accuracy, precision and recall, AUC, F1 score).

Deployment

Assuming above steps are completed and you have found satisfactory values for your model evaluation, you can clean-up your workflow to have only one model, with the optimum configuration. Publish you score model as an input and define your output. Run and publish your web service. From there you will have access to the API help page containing sample code for C#, R, and Python and a test URL where you can manually set some values.

In your web service configuration tab, you can enable it as ‘ready for production’ and start using it in real production project!

Resources

Machine learning for beginners.
If you want to go deeper in understanding ML, Andrew Ng have a fantastic class on coursera.org.
All the videos tutorial that you need to get started with Azure ML.
Predictive solution walk-through.

Embracing change

We all experienced change in our life. Whether small, like moving to a new home, or larger, like changing job or moving to a new country, it always comes with a fair amount of stress. I believe that the stress associated with change is coming from all the uncertainty that change triggers (I’m not a psychologist, at all.). I have had a fair share of change in my life, moving across multiple countries, changing jobs, becoming a husband and a dad, and more recently going through re-organization and layoffs at my workplace. Looking back, and simplifying a bit, there is really 3 phases of change (psychologists would argue there are multiple stages: precontemplation, contemplation, determination, action, maintenance). In the first phase you come to be aware of upcoming change, the second phase is the proper transition period, and the third phase is when the change is completed. Below are some tactics I have been using to have a smooth transition.

Upcoming change

It’s crucial to acknowledge when change is coming your way. At this point you might not know where you are heading to but you should not ignore the change. Being aware of the change have a couple of positive aspects, you can stay in motion with the change and actively help influencing the direction of the change. Not knowing where the journey will end is stressful, no question about this but it is important to have a north star to ‘survive’ the cycle. Don’t fall into a panic mode (people around you might be freaking out, don’t let this impact you – help them see the benefits of the change or its temporality). Understanding the root of the change will be extremely useful during the next phases.

Transition

Change is ongoing, embrace it! At this point, there is probably a lot of chaos, a lot of uncertainty, and a lot of frustration. As with most things, we can turn this time to have a couple of positive outcome. Get involved in the transition, help make it successful, stir it in the right direction. Getting knowledge on the change will help you adjust to it. Stay aware that you are in a transition phase, level down your anxiety and be patient.

Post change

At this point, you should have adjusted to the change. You can start focusing on building up on the new foundation. Have a look back and be proud of the journey you just had, another one is coming soon…

Career superpowers – book summary

I loved this book, career superpowers: succeeding on purpose by James Whittaker. It is refreshing to strategically think about your career and to feel in control. The author breaks down the career superpowers in 8 attributes: ambition, passion, specialization, storytelling, imitation, derivation, creativity, and leadership.

Ambition

Ambition sets the bar for you career, you increase your chance of climbing high by aiming high. It means aligning your skills and work ethic to achieve what you want from life. To fulfill your ambition, you will have to identify stupid rules and refuse to allow them to have power over you (loyalty to your company is one stupid rule).
Out delivering your hard working colleagues is more likely to burn you out rather than help you reach your long term goal. Working hard is a necessary condition of success but it is not sufficient in itself. You need to work smart, do not invest time in impressing your manager(s), and spend time to better your situation and yourself. Have a larger view, take the pulse of the industry rather than the pulse of your company. You have to learn to stand out.
Your career has to be handled as a project, keep track of it daily think where is it headed. What’s in your way, how is your progress? Are you ever going to get where you want to be on your current course?

Passion

If you are not passionate about your job then you won’t have the additional energy, enthusiasm and work ethic that comes with passion. You should share your passion broadly (blog, papers, manifestos) and advertise the thing that turns you on. Also, being intensely interested in the subject you are presenting makes the material more interesting to your audience.

Specialize

A specialty is the ultimate career aide, choose one that matters (high value), that is visible, and has implication across the industry and not just your company. Pick a specialty that is well within your skill set and you can master completely (being good only is not enough). Don’t stretch to be the dumbest guy at the level above you, instead be the smartest at the level below.
Once you got a specialty you love, sell it! Have a good elevator pitch (write it, practice it, and promote it) for all your specialty (don’t be a one trick pony, be ready to change), it should explains what you do and why it is valuable in one simple sentence.
You will enjoy the dividend of you specialty during the rest of your career. You need to learn, master, and own your specialty.

Storytelling

Directly influence how people think and get them to see your ideas and values in full color.

Imitation

The people around you have a great power over who you become and how high you reach. Choose them well. If you need to learn something from someone, make sure they are really good at it.

Derivation

It is not about making the invention, it is about being the one who makes the invention meaningful. You should be aware of your industry and how you are using product developed by your industry. You should be able to name at least 10 industry-insider websites that publish news about your industry. Looking at the world and wanting to fix it is part of the derivative superpower.

Creativity

The more you know, the more creative you can become. Find out when and where you can be the most creative, build your day to enable creativity.

Leadership

All previous attributes, make you successful. Being successful creates gravity around you, this is call being a leader. You need to know yourself to be a leader, knowing where you want to go and where you want to be in the future. Work on product that is important to the company, no career is served by working on a product that the company doesn’t really care about. Leader can build a team where everybody can recite the product elevator pitch, each understand why the product is important, what and who the competition is, and how the product fits in the company’s lineup.

 

Above summary is really short and might not make sense to someone who haven’t read the full book. I definitely recommend this one for anybody who wants to drive their career.

 

Facebook SDK 4.0.0 for PHP: A working sample to manage sessions

 

Once you have a working sample of Facebook SDK 4.0.0 for PHP, you will notice upon refreshing the page an error:
Fatal error: Uncaught exception ‘Facebook\FacebookAuthorizationException’ with message ‘This authorization code has expired.’

Well, this is quite annoying as it breaks the user navigation on your site. To get around this issue, record the FacebookSession token to the user’s session and use it next time the page load.

<?php 
session_start();

require_once( 'Facebook/FacebookSession.php' );
require_once( 'Facebook/FacebookRedirectLoginHelper.php' );
require_once( 'Facebook/FacebookRequest.php' );
require_once( 'Facebook/FacebookResponse.php' );
require_once( 'Facebook/FacebookSDKException.php' );
require_once( 'Facebook/FacebookRequestException.php' );
require_once( 'Facebook/FacebookAuthorizationException.php' );
require_once( 'Facebook/GraphObject.php' );
require_once( 'Facebook/GraphSessionInfo.php' );

use Facebook\FacebookSession;
use Facebook\FacebookRedirectLoginHelper;
use Facebook\FacebookRequest;
use Facebook\FacebookResponse;
use Facebook\FacebookSDKException;
use Facebook\FacebookRequestException;
use Facebook\FacebookAuthorizationException;
use Facebook\GraphObject;
use Facebook\GraphSessionInfo;

$appid = ''; // your AppID
$secret = ''; // your secret

// Initialize app with app id (APPID) and secret (SECRET)
FacebookSession::setDefaultApplication($appid ,$secret);

// login helper with redirect_uri
$helper = new FacebookRedirectLoginHelper( 'http://www.metah.ch/' );

try 
{
  // In case it comes from a redirect login helper
  $session = $helper->getSessionFromRedirect();
} 
catch( FacebookRequestException $ex ) 
{
  // When Facebook returns an error
  echo $ex;
} 
catch( Exception $ex ) 
{
  // When validation fails or other local issues
  echo $ex;
}

// see if we have a session in $_Session[]
if( isset($_SESSION['token']))
{
	// We have a token, is it valid? 
	$session = new FacebookSession($_SESSION['token']);	
	try
	{
		$session->Validate($appid ,$secret);
	}
	catch( FacebookAuthorizationException $ex)
	{
		// Session is not valid any more, get a new one.
		$session ='';
	}
}

// see if we have a session
if ( isset( $session ) ) 
{   
	// set the PHP Session 'token' to the current session token
	$_SESSION['token'] = $session->getToken();
	// SessionInfo 
	$info = $session->getSessionInfo();	
	// getAppId
	echo "Appid: " . $info->getAppId() . "<br />"; 
	// session expire data
	$expireDate = $info->getExpiresAt()->format('Y-m-d H:i:s');
	echo 'Session expire time: ' . $expireDate . "<br />"; 
	// session token
	echo 'Session Token: ' . $session->getToken() . "<br />"; 
} 
else 
{
  // show login url
  echo '<a href="' . $helper->getLoginUrl() . '">Login</a>';
}
?>

The session expire time (getExpiresAt()) can be used to monitor how long the token can be used and potentially request a new token.