Author Archives: Ahmet

About Ahmet

Husband and father of two, Swiss relocated in Seattle, WA. Program Manager in Bing Ads. Views and opinions expressed are mine.

Azure Machine Learning Deployment at Scale Using ARM and AMLPS.

Introduction

In this post, I will demonstrate how to do a simple but useful scenario when managing Azure Machine Learning Workspaces and Experiments, copying all the experiments under one workspace, deploy a new workspace using ARM (Azure Resource Manager) in another region, and then copy the experiment under the newly deployed workspace.

Preparation

You will need to install a couple of things:

  • Azure Resource Manager PowerShell modules
  • Azure Service Management PowerShell modules
  • Azure Machine Learning PowerShell modules (via github).
  • # Install the Azure Resource Manager modules from the PowerShell Gallery
    Install-Module AzureRM -Scope CurrentUser
    
    # Install the Azure Service Management modules from the PowerShell Gallery
    Install-Module Azure -Scope CurrentUser
    

    Installing AMLPS, is not yet as easy but it is not too hard :).

  • Download the latest zip file from https://github.com/hning86/azuremlps/releases (as of today, the beta is 0.2.8), unzip to a folder (let’s say c:\amlps) – this is a manual step.
  • Navigate to c:\amlps
  • Unblock the file
  • Load the module
  • #Unblock the downloaded dll file so Windows can trust it.
    Unblock-File .\AzureMLPS.dll
    
    #import the PowerShell module into current session
    Import-Module .\AzureMLPS.dll
    

    Of course, you will also need to have an Azure Machine Learning account.

    Configure

    Now that all the modules are installed and imported, let’s configure our session.
    First making sure we authenticate to Azure RM and SM and can list our workspaces.

    # Authenticate (enter your credentials in the pop-up window)
    Login-AzureRmAccount
    
    # List all workspaces
    Get-AzureRmResource |? { $_.ResourceType -Like "*MachineLearning/workspaces*"} 
    

    At this point, you should see all your workspaces listed.
    We are now ready to configure AMLPS, for this you will need to retrieve the workspace ID, and the authorization token (detailed step by step for AMLPS configuration). We will update config.json (in c:\amlps) for simplicity. From the above list of workspaces, select the one you want to duplicate. In my example, the workspace name is “workspaceus”.

    # Select workspace with the name “workspaceus”
    $wsp = Get-AzureRmResource |? { $_.Name -Like "workspaceus"}
    
    # Get the workspaceId
    $wid = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Properties.workspaceId
    
    # Get the primary token
    $wpt = (Invoke-AzureRmResourceAction -ResourceId $wsp.ResourceId -Action listworkspacekeys -Force).primaryToken
    
    # Get the location of the workspace
    $wil = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Location
    
    # Create the JSON config file
    (New-Object psobject | Add-Member -PassThru NoteProperty Location $wil | Add-Member -PassThru NoteProperty WorkspaceId $wid | Add-Member -PassThru NoteProperty AuthorizationToken $wpt) | ConvertTo-Json > config.json
    

    Deploy

    Now that we have installed and configured all the tools, we can start our example. We will copy a workspace and its experiment located in “South Central US” and copy it to “West Europe”. Below is an illustration representing the different steps in our journey.
    machinelearningworkspace

    Get workspace information

    We will export all experiment graph as a JSON file so we can import them back on the new workspace.

    # Create folder for export
    New-Item -Name "Export" -ItemType "directory" -Force
    
    # Export all experiments in the workspace
    Get-AmlExperiment |% {$i=0}{Export-AmlExperimentGraph -ExperimentId $_.ExperimentId -OutputFile "c:\AzureMLPS\export\exp$i.json"; $i++}
    

    Deploy new workspace in another location

    To deploy the new workspace, you can refer to this more detailed article to get a sample ARM template to deploy a new machine learning workspace.

    # Create a new resource group in West Europe.
    $rg = New-AzureRmResourceGroup -Name "uniquenamerequired723" -Location "West Europe"
    
    # Deploy a Resource Group, TemplateFile is the location of the JSON template.
    $rgd = New-AzureRmResourceGroupDeployment -Name "demo" -TemplateFile "mlworkspace.json" -ResourceGroupName $rg.ResourceGroupName
    

    Copy the experiment into the new workspace

    First, we need to update the configuration for AMLPS to the new location.

    # Select workspace just created
    $wsp = Get-AzureRmResource |? { $_.ResourceGroupName -Like $rgd.ResourceGroupName -AND $_.ResourceType -Like "Microsoft.MachineLearning/Workspaces"}
    
    # Get the workspaceId
    $wid = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Properties.workspaceId
    
    # Get the primary token
    $wpt = (Invoke-AzureRmResourceAction -ResourceId $wsp.ResourceId -Action listworkspacekeys -Force).primaryToken
    
    # Get the location of the workspace
    $wil = (Get-AzureRmResource -Name $wsp.Name -ResourceGroupName $wsp.ResourceGroupName -ResourceType $wsp.ResourceType -ApiVersion 2016-04-01).Location
    
    # Create the JSON config file
    (New-Object psobject | Add-Member -PassThru NoteProperty Location $wil | Add-Member -PassThru NoteProperty WorkspaceId $wid | Add-Member -PassThru NoteProperty AuthorizationToken $wpt) | ConvertTo-Json > config.json
    

    Now we are able to import the experiments we copied from the previous workspace.

    Get-ChildItem export\* -Include *.json |% {Import-AmlExperimentGraph -InputFile $_ }
    

    Test that all is working correctly

    The simplest thing to do at this point is to list all the experiments under the workspace.

    Get-AmlExperiment
    

    The result of this command should be all the experiment listed under your newly deployed workspace.

    If you have no idea where to start with you Machine Learning experiment, you can have a look at a tutorial I wrote a while ago about getting started with Azure Machine Learning. You should also check the Cortana Intelligence Gallery where plenty of experiments are available for free.

     
    3 Kudos
    Don't
    move!

    The lean startup – book notes

    The lean startup by Eric Ries is a very interesting book about creating a successful business.

    It’s really a great read even if you are not working in a startup. Actually most large company moved away from previous development processes that didn’t adapt well to the new technologies. In today world, we ship software daily, sometimes multiple times per day. We even have multiple versions of the software being shipped in parallel (AB testing) to find out which version resonate the most with our users. This is, of course, very different than shipping a software on a CD every n year.

    Notes

    Below are some quotes from the book with some comments.

    The fundamental goal of entrepreneurship is to engage in organization building under conditions of extreme uncertainty, its most vital function is learning. […] Learn what customers really want, discover whether we are on a path that will lead to growing a sustainable business.

    Behind the word “learning” hides a lot of complexity, it’s about demonstrating empirically that we are going in the right direction. This also means that we should avoid vanity metrics (e.g. instead of tracking the number of visitor on your product page, track the number of visitor actually taking crucial actions within your product).

    Our main concerns in the early days dealt with the following questions: What should we build and for whom? What market could we enter and dominate? How could we build durable value that would not be subject to erosion by competition.

    Too often we don’t ask ourselves these questions, we are too focus in building a great product (from the engineering perspective) that we lost focus on the important point. The what, whom, how much are questions you should always have in mind.

    The effort that is not absolutely necessary for learning what customers want can be eliminated. I call this validated learning because it is always demonstrated by positive improvements in the startup’s core metrics.

    The idea is to build the minimum viable product that will confirm whether your product is going in the right direction, as measure by your core metrics. In large company, there is a tax that comes with being part of it. Some of the minimum things that needs to be build are not always relevant to our learning but are present to reinforce the company core values (e.g. security and privacy can to some extend be punted to later by startup, while group from a large company cannot.)

    The value hypothesis tests whether a product or service really delivers value to customers once they are using it. […] Growth hypothesis tests how new customers will discover a product or service.

    This should be part of your core metrics, are we delivering value? How does our product growth happen? They are the most important leaps of faith questions any startup face.

    Answer four questions before investing engineering resources:

    1. Do consumers recognize that they have the problem you are trying to solve?
    2. If there was a solution, would they buy it?
    3. Would they buy it from us?
    4. Can we build a solution for that problem?

    For larger company, there is an additional layer of complexity, there is often a sales organization interacting directly with the customers. Typically, product teams were not directly engaged with the customers. In my current group at Microsoft (Azure), the program managers are directly interviewing with the customers, engaging with them to understand their challenges and to establish great relationship with them.

    Build a process of identifying risk and assumptions before building anything and then testing those assumptions experimentally

    Use prototypes and interviews, it’s cheaper than building a product!

    Build experiment, identify the elements of the plan that are assumptions rather than facts, and figure out way to test them. Using this insights, we could build a minimum viable product.

    These reiterate the previous point, the goal is to reduce the cost of building a product that will be liked and used by customers.

    The MVP is that version of the product that enables a full turn of the Build-Measure-Learn loop with a minimum amount of effort and the least amount of development time. […] MVP is designed not just to answer product design or technical questions. Its goal is to test fundamentals business hypothesis. […] A video can be a great form of MVP, a demonstration of how the technology, targeted at a community of technology early adopters.

    To perform well within this cycle, your infrastructure needs to support fast iteration. You can’t really build, measure, and learn if it takes you one month to ship bits to production. Moving away from the traditional infrastructure is probably one of the biggest investment a company need to do in order to move to a leaner development cycle. Some fundamentals required:

    1. Support for fast deployment to production
    2. Support for Test in Production and experimentation (AB testing)
    3. Channel to gather feedback from users and measure their interest.

      Traditional approaches such as interaction design or design thinking are enormously helpful.


    Give the concierge treatment to your early adopters, learn more and more about what it takes to make the product great. […] Measured according to traditional criteria, this is a terrible system, entirely nonscalable and a complete waste of time. But as a results of the learnings, the development efforts involve less waste than typical.

    That’s the whole premise, invest time in learning from your users and make sure you build what they need. Then invest in building what delivers the greatest value for them first. Do things that doesn’t scale.

    Most modern business and engineering philosophies focus on producing high-quality experiences for customers as a primary principle. […] These discussions of quality presuppose that the company already knows what attributes of the product the customers will perceive as worthwhile.

    If your MVP feels rough to your customers, use it as an opportunity to learn about what they care about. Always ask yourself whether the customer care about design in the same way you do.

    The truth is that most managers in most companies are already overwhelmed with good ideas. Their challenge lies in prioritization and execution, and it is those challenges that give a startup a hope of surviving.

    A good reading about this is disruptive innovation. If you are part of a larger company, the prioritization effort should come from the learning you made from your users. When working with a lot of talented engineers it is sometime more exciting to solve big technical challenges rather than building the set of features your users need.

    A startup’s job is to 1) rigorously measure where it is right now, confronting the hard truths that assessment reveals, and then 2) devise experiments to learn how to move the real numbers closer to the ideal reflected in the business plan.

    This is true, not only for startup, but for all companies, not matter where they are in their lifecycle.

    The rate of growth depends primarily on three things: the profitability of each customer, the cost of acquiring new customers, and the repeat purchase rate of existing customers.

    It’s interesting to focus on rate of growth without being obsess by other cost factor (cost to serve), as the other cost will likely have an economy of scale.

    The three learning milestones are: 1) use a minimum viable product to establish real data on where the company is right now, 2) tune the engine (of growth) from the baseline toward the ideal, 3) pivot or persevere.

    Step #3 is rarer for larger company, I can’t remember a lot of example where large company pivoted. It seems the outcome of#3 is more fail or persevere.

    Funnel metrics: behavior that are critical to your engine of growth (e.g. customer registration, download of application, trial, repeated usage, purchase).

    User funnel analysis is so helpful to discover how you are doing and to help identify friction points that deserve your attention. When building new features to your product, always make sure that you can measure easily all the steps leading to using the feature and how the feature is used.

    Cohort analysis: instead of looking at cumulative totals or gross numbers such as total revenue or total number of customers, one looks at the performance of each group of customers that comes into contact with the product independently (i.e. customer who joined each month).

    Cohort analysis are useful to track the progress of metrics following some changes as well, once you ship new features, you can measure their effects on the cohorts carrying the change.

    Metrics should honor the three A’s: Actionable (clear cause and effect), accessible (everyone can understand and access them), auditable (ensure the data is credible).

    As soon as we formulate a hypothesis that we want to test, the product development team should be engineered to design and run this experiment as quickly as possible, using the smallest back size that will get the job done. Remember that although we write the feedback loop as build-measure-learn because the activities happen in that order, our planning really works in the reverse order: we figure out what we need to learn and then work backwards to see what product will work as an experiment to get that learning.

    In theory that resonate really well to me, in practice however it sounds harder to pull. The development team is often assigned to work on multiple things and have different priorities. This assume that it’s pulling out a team of developer from their ‘normal’ duty is good. I’ve never seen this work successfully in larger and older group.

    Technically, more than one engine of growth can operate in a business at a time […] successful startups usually focus one just one engine of growth, specializing in everything that is required to make it work.

    Engine of growth: 1) sticky engine (relies on repeated usage), 2) viral engine (network effect), 3) paid engine (to acquire customer).

    Andon cord: “Stop production so that production never has to stop”. […] You cannot trade quality for time. Defects cause a lot of rework, low morale, and customer complaints, all of which slow progress and eat away at valuable resources.

    Balancing speed and quality is complex, having a system that force in the quality might have a higher cost at first but in the long run it will provide multiple benefits.

    Use the Five Whys: this help go to the root of every seemingly technical problem.

    But don’t use it to blame people, use it to learn and prevent similar problem to ever happen again. 5 Whys article on Wikipedia.

    Conclusion

    This book will stay on my shelf, what I like the most about it is:

    • Forefront the learning you will have from any development, make sure you can measure your success and failure. Build an MVP.
    • Stay in sync with you customers, learn what they need and how they work
    • The end goal is to be successful, this will most probably mean failure (learning opportunities) which will require for perseverance or for pivots.
    • Avoid vanity metrics, focus on real metrics
     
    0 Kudos
    Don't
    move!

    What makes a team effective?

    I often wonder what makes some team more effective than other. Why I felt more empowered in some team compared to some others. The simplistic reasons that pops to mind are about the quality of the leadership and the joy experimented at work. The People Operation team at Google shared interesting conclusions from a study they made. It’s seems the most important factors to make a team effective are:

    1. Psychological safety: Can we take risks on this team without feeling insecure or embarrassed?
    2. Dependability: Can we count on each other to do high quality work on time?
    3. Structure & clarity: Are goals, roles, and execution plans on our team clear?
    4. Meaning of work: Are we working on something that is personally important for each of us?
    5. Impact of work: Do we fundamentally believe that the work we’re doing matters?

    Definitely, some of these points should really be pushed by the leadership of the team / group. Other points can actually be enforced by the team itself, by acknowledging their importance.

    Meaning, impact, dependability, and the psychological safety are values that can (and should) be driven by the team itself.

    So, next time you are looking for a new team, try to assert how they are doing in these five categories. Also, if your current team doesn’t appear to do well on any of these categories, see what you can do to improve it.

     
    0 Kudos
    Don't
    move!

    Team of Teams: New Rules of Engagement for a Complex World – book notes

    In the midst Microsoft’s transformation, in how we work within and between teams and groups, I’ve found this excellent book (suggested by one of our executive), Team of Teams: New Rules of Engagement for a Complex World by General Stanley McChrystal. This book relates how the best teams in the U.S. Army were not able to work together as one unit because of the organizational, hierarchical, and decision process that were in place. The general goes in great details in illustrating the history behind the previous military organization and how evolving was a necessity to survive in a complex world.

    Notes

    Below are some notes I marked from the book, with some comments.

    We were stronger, more efficient, more robust. But AQI was agile, and resilient. In complex environments, resilience often spells success, while even the most brilliantly engineered fixed solutions are often insufficient or counterproductive.

    This is critical to understand and accept. Complex environments are really hard to predict, agility (quick iteration in setting up and pursuing goals, ability to deal with volatility) and resiliency (being able to reconfigure following context changes) are two skills that are essentials to any organizations.

    Peter Drucker has a catchy statement: “Efficiency is doing things right; effectiveness is doing the right thing.” If you have enough foresight to know with certainty what the “right thing” is in advance, then efficiency is an apt proxy for effectiveness.

    In this quote from Peter Drucker, is hidden a pearl. You should build capabilities to your organization to have improved foresight, being efficient on the wrong thing is pointless. Efficiency should not be seen as a managerial holy grail as it negatively impacts both flexibility and resilience.

    Team members tacking complex environments must all grasp the teams’ situation and overarching purpose. Only if each of them understands the goals of a mission and the strategic context in which it fits can the team members evaluate the risks on the fly and know how to behave in relation to their team mate. Individual SEALs have to monitor the entirety of their operation just as soccer players have to keep track of the entire field. They must be collectively responsible for the team’s success and understand everything that responsibility entails.

    If you ever work in a situation where you are not clear about the strategy and the goals, how can you make good decisions? Sounds too obvious, unfortunately, too often we can interact with people that only focus on their specialized area instead of seeking to understand their role in the uber-mission and uber-strategy. Putting this in place requires a lot of work: share all information with everybody (and don’t silo information per role / team / grade), make sure that everybody can share input on the tactics used to pursue the strategy. Keep clarifying what is the strategy, what we are going after, what the goals are.

    We needed to enable a team operating in an interdependent environment to understand the butterfly-effect ramifications of their work and make them aware of the others teams with whom they would have to cooperate in order to achieve strategic – not just tactical – success.

    This build on top of the previous comment, to make sure that everyone understands each teams’ prerogative, focus, and specialization.

    “System engineering” or “system management” is an approach built on the foundation of “systems thinking”. This approach, contrary to reductionism, believes that one cannot understand a part of the system without having at least a rudimentary understanding of the whole.

    Although some people prefer to knows only what is relevant for them, they need to understand that each decision they take will impact other teams. It’s only by making sure that enough people with the appropriate knowledge should make a decision together. Again, the uber-point is to share information as much as possible between individuals and teams.

    Effective leadership in the new environment as more akin to gardening than chess. The move-by-move control that seems natural to military operations proved less effective than nurturing the organization – it’s structure, processes, and culture – to enable the subordinate components to function with “smart autonomy”. […] the efforts of every part of the team were tightly linked to a common concept for the fight, but it allowed those forces to be enabled with a constant flow of “shared consciousness” from across the force, and it freed them to execute actions in pursuit of the overall strategy as best they saw fit.

    The metaphor comparing leadership to either playing chess or gardening is working solidly. When playing chess, the leader decides of every move, the team execute. When gardening, you empower your team to be efficient. Leadership is not about taking the decisions anymore but enabling everyone to have enough information to make the right decision as quickly as possible. The leader should still be active, “Eyes-on, Hands-off”.

    Conclusion

    This is one of the best organizational book I’ve read in a long time. The ideas being presented are not revolutionary but setting them up would requires some revolution in most organization. Sharing all information constantly with all teams, empowering people to make decision, removing the friction of silos and hierarchical abstraction, and being transparent with our goals. That’s the future of any successful organization dealing with complex environments.

     

     
    0 Kudos
    Don't
    move!

    An Astronaut’s guide to life on earth – book notes

    This book from Chris Hadfield was really fun to read, the first half of the book is about lessons he learned, as the title suggest, while the other part of the book is more a biographical stories about his time in preparing and staying in space.

    Notes

    Below are some quotes, that I have generalized for the none astronaut reading my blog, I hope to remember from the book:

    Success is feeling good about the work you do throughout the long, unheralded journey that may or may not wind up at the launch pad. Training […] got to be an end in itself, […] secret is to enjoy it.

    There is one thing I can control: my attitude (orientation) during the journey. Too many variables are out of my control.

    Learn to push past fear by gradually building up, the progressive sense of confidence will make you more skilled.

    Response positively to criticism, learn from it. Draw your attention to your own missteps and miscalculations. Even hyper-competitive people needs to be able to openly talk about their screw-ups.

    Good leadership means leading the way, not hectoring other people to do things your way. Think about the group to get the whole picture.

    Know the boldface (actions critical to survival) and recognize when to use it.

    Conclusion

    What I really liked from this book is the motivation that Hadfield shows in his book. Even if the odds were against him to become astronaut, he always kept ready and worked hard for it. His unique capacity to self-motivate and keep focus on the journey and not on the destination was enlightening. I definitely recommend this book J.

     
    0 Kudos
    Don't
    move!

    Mindset: the new psychology of success – book notes

    I recently finished reading “Mindset: The New Psychology of Success” from C. Dweck. I picked it up because it was part of the recommended books list from the C.E.O of the company where I work. There was some great learnings that I will definitely apply, mostly with my children and with myself from time to time J. This book had a lot of redundancy to validate the point of the author, tackling the point from multiple angles. In short, she claims that people can have two mindsets. The first one is a fixed mindset believing that we are born with a fixed set of capacities, that intelligence is static, and this leads us to desire to look smart. While the growth mindset is convinced that hard work and planning will increase our aptitudes and that intelligence can be developed, and this leads us to desire to learn.

    Comparing the two mindsets, fixed vs growth

    Fixed Growth
    Challenges Avoid challenges (might prove that I’m not smart) Embrace challenges (can learn more and grow)
    Obstacles Get defensive or give up easily Persist in the face of setbacks
    Effort See effort as fruitless or worse, not supposed to need it, and rob you of all excuses See effort as the path to mastery
    Criticism Ignore useful negative feedback Learn from criticism
    Success of others Feel threatened by the success of others Find lessons and inspiration in the success of others
    Result They may plateau early and achieve less than their full potential They reach ever-higher levels of achievement

    Of course, if you recognize yourself in the fixed mindset, I would really encourage you to read the book! If you do recognize you with the growth mindset, it might not be as valuable for you, although there are still somethings to learn and grow from this book =).

    Using the growth mindset

    Every day, ask yourself “What are the opportunities for learning and growth today? For myself? For the people around me?” As you think of opportunities, form a clear plan and ask “When, where, and how will I embark on my plan”? Expect setback and re-iterate on your plan accordingly. Don’t forget to maintain your growth: “What do I have to do to maintain and continue the growth?”

    When you think about your hero, role model, consider the idea that they used better strategies, taught themselves more, practices harder, and worked their way through obstacles. You can do that too.

    People and teams embracing the growth mindset are more likely to state their honest opinions and openly express their disagreement, everyone is part of the learning process.

    Teaching the growth mindset to your child(ren)

    With my wife, we decided to ask ourselves and our children about our learning for the day, we do it every day at dinner time. The goal is to teach them that we appreciate the effort they put in learning new things and help them growth. More importantly the goal is to not teach them the growth mindset. Below are some sentences that you can use with your children to develop the growth mindset. Instead of saying, “awesome job, you are so intelligent!”, try something like this:

    “I like that you took on that challenging project for your science class. It will take a lot of work – doing the research, designing the apparatus, buying the parts, and building it. Boy, you are going to learn a lot of great things.”

    “I know school used to be easy for your and you used to feel like the smart kid all the time. But the truth is that you weren’t using your brain to the fullest. I’m really excited about how you’re stretching yourself now and working to learn hard things.”

    “That homework was so long and involved. I really admire the way you concentrated and finished it.”

    “That picture has so many beautiful colors. Tell me about them.”

    “The passion you put into that piano piece gives me a real feeling of joy. How do you feel when you play it?”

    When your child doesn’t deserve praise but encouragement, be honest.

    “We all have learning curves. It may take more time for you to catch on this and be comfortable with this material, but if you keep at it like this you will.”

    “Everyone learns in a different way. Let’s keep trying to find the way that works for you.”

        “It really makes me upset when you don’t do a full job. When do you think you can complete this?”

    “Is there something you didn’t understand in the assignment? Would you like me to go over it with you?”

    “I feel sad when I see you missing a chance to learn. Can you think of a way to do this that would help you learn more?”

    “This look like a really boring assignment. You have my sympathy. Can you think of a way to do this that would help you learn more?”

    Conclusion

    Personally this book resonated well with me, as a child, it would have helped me to get some coaching on the growth mindset. As a parent, I hope to be able to cultivate a passion for learning and for growing.

     
    0 Kudos
    Don't
    move!

    Zero to ONE – book summary

    “Zero to ONE: Notes on Start Ups, or How to Build the Future” is a great book by Peter Thiel (co-founder of PayPal and Palantir) about how to build the next Microsoft or Google. He has a pretty unique perspective on market and competition that are worth reading and knowing about. The book could be summarized as: to be successful, get a proprietary technology and find a small niche market, then expend from this.

    Below are some notes I took while reading the book, they are meant for me, but sharing this here just in case it convinces anyone to read this book.

    The challenge of the future

    His point in this chapter is that despite the fact that computers and communications have really changed since the last midcentury, the remaining things around us still looks old. The challenge for the future is how to create new technologies that will make the 21st century more prosperous than the 20th.

    Party like it’s 1999

    Entrepreneurs learned four big lessons from the dot-com crash:

    1. Make incremental advances
      Small incremental steps are the only safe path forward.
    2. Stay lean and flexible
      You should not know what your business will do; planning is arrogant and inflexible. Trying things out and iterating is the way to go. Entrepreneurship is about experimenting.
    3. Improve on the competition
      The only way to know you have a real business is to start with an already exciting customer, so you should build your company on recognizable products already offered.
    4. Focus on product, not sales
      The only sustainable growth in technology is viral growth.

    He then spins this around a bit proposing four principles:

    1. It is better to risk boldness than triviality.
    2. A bad plan is better than no plan.
    3. Competitive markets destroy profits.
    4. Sales matters just as much as product.

    All happy companies are different

    He basically differentiates monopolist companies and non-monopolist companies:

    • Non-monopolists exaggerate their distinction by defining their market as the intersection of various smaller markets.
    • Monopolists, by contrast, disguise their monopoly by framing their market as the union of several large markets (helping to pretend for competition).

    Last movers advantage

    Four characteristics of a monopoly:

    1. Proprietary technology
      Must be at least ten times better than its closest substitute in some important dimension to lead to a real monopoly advantage.
    2. Network effects
      Make a product more useful as more people use it. They must start with small markets, the initial markets are so small that they often don’t even appear to be a business opportunities at all.
    3. Economies of scale
      A monopoly get stronger as it gets bigger: the fixed cost of creating a product (engineering, management, office space) can be spread out over ever greater quantities of sales.
    4. Branding
      A company has a monopoly on its own brand by definition, so creating a strong brand is a powerful way to claim a monopoly, however no technology company can be built on branding alone.

    The way to build a monopoly is very deliberate and must be done carefully:

    1. Start small and monopolize
      Every start-up should start with a very small market, always err on the side of stating too small, allowing the startup to be a position as a monopoly for this market. The perfect target for a startup is a group of particular people concentrated together and served by few or no competition.
    2. Scaling up
      Once a specific niche is dominated, scale to adjacent markets.
    3. Don’t disrupt
      Originally, disruption was about using a new technology to introduce low-end product at low prices, improve the products over time, and eventually overtake even the premium products offered by incumbent companies. However, this limit your vision to focus on the other company product, to create something really new, the act of creation is far more important than the old industries that might not like what you create. Avoiding competition is the safest road.
    4. The last will be the first
      Be the one making the last great development in a specific market and enjoy years of monopoly profits. Start by dominating a small niche and scale up from there, toward your ambitious long-term vision.

    Follow the money

    We don’t live in a normal world; we live under a power law (“Pareto principle” – 20% of the company will out power the 80%). You should focus relentlessly on something you are good at doing, but before that you must think hard about whether it will be valuable in the future. In a power law world, you can’t afford not to think hard about where your action will fall in the curve.

    Secrets

    Four social trends have conspired to prevent us to look for more secrets:

    1. Incrementalism
      From an early age we are taught that the right way to do thinks is to proceed one very small step at a time.
    2. Risk aversion
      People are scared of being wrong.
    3. Complacency
      Why search for a new secrets if you can comfortably collect rent on everything that was already discovered.
    4. Flatness
      If it were possible to discover something new, wouldn’t someone from the faceless global talent pool of smarter and more creative people have found it already?

    Foundations

    Thiel’s law: “a startup messed up at its foundation cannot be fixed”. Founders should know each other and work well together. This count as much as their technical ability and complementary skills.

    Clarify this 3 concepts:

    • Ownership: who legally owns a company’s equity?
    • Possession: who actually runs the company on a day-to-day basis?
    • Control: who formally governs the company’s affair?

    The mechanics of mafia

    Startups should make their early staff as personally similar as possible. Startups have limited resources and small teams. It is easier to survive when everyone shares an understanding of the world. Every employee in the startup should be responsible and evaluated for doing only one thing, this helps reduce conflicts.

    If you build it, will they come?

    Engineer are biased toward building cool stuff rather than selling it. Making customer come is harder than it looks (respect marketing). If you have invented something new but haven’t invented an effective way to sell it, you have a bad business (no matter how good the product).

    Two metrics are important:

    1. Customer lifetime value (how much earn from a customer during the life of your relationship)
    2. Customer acquisition cost (amount spend to acquire the customer)

    Of course, CLV must be > than CAC.

    Man and machine

    Big data is actually dumb data, computer don’t know how to compare patterns from different source or how to interpret complex behaviors. Actionable insights can only come from a human analyst.

    Seeing green

    Why did the green industry failed, because they didn’t answer the 7 questions every business must answer:

    1. Engineering question: “Can you create a breakthrough technology instead of incremental improvements”?
    2. Timing question: “Is now the right time to start your particular business”?
    3. Monopoly question: “Are you starting with a big share of a small market”?
    4. People question: “Do you have the right team”?
    5. Distribution question: “Do you have a way to not just create but deliver your product”?
    6. Durability question: “Will your market be defensible 10 and 20 years into the future”?
    7. Secret question: “Have you identified a unique opportunity that others don’t see”?
     
    0 Kudos
    Don't
    move!

    Dealing with PowerShell Execution Policy

    I’m generally a huge fan of security features, as long as they don’t go in my way. Today, the PS execution policy actually came in my way, so a quick way to remove it is here:

    Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Scope CurrentUser

    Use it at your own risk, this will remove the execution policy security for your current user. You can find all the needed information from TechNet.

     
    0 Kudos
    Don't
    move!

    Last day at Bing Ads, Monday will be my first day at Bing Web Data Platform

    In September 2012, I joined the Bing Ads team, working on the delivery engine, where I had a blast! Working on an online system, with such a massive scale (Bing is now serving 20% of all US search queries) and with so high expectation (latency, availability, …) really made the journey a thrill. I learned a lot, I learned to be a PM (I was a engineer in test before), I learned how to build systems that scale, and I learned to be obsess with metrics (in Ads revenue is a big deal obviously).

    So why do I change if things are so exiting?

    Challenge yourself!

    I love the feeling of being new to a team, having to learn and having to understand a lot of things. When the day to day life becomes dull, it’s generally a signal for me that I should be looking around for opportunities.

    But if I stayed …

    I spoke with a bunch of folks (in Seattle area) that are staying in their current position, hoping to get a promotion soon, or too afraid of being challenged. I don’t believe it’s the right approach, keep challenging yourself, you can only become better this way!

    If we worked together during these past years, thank you, I had a blast!

     

     
    1 Kudos
    Don't
    move!

    Prioritizing features and requests

    As an engineer we often get approached by a PM, or other teams members, for features, update in requirements, or any other changes. Some change will sounds really exciting, either because they are intellectually challenging or because their outcome looks obviously great.

    However, before getting too excited there a few questions we can ask that will help us stay grounded and keep priorities clear and in order. This is especially important for PMs but I believe that all engineers, disregarding their role title, should always understand why we are doing things and how it will translate in increased consumer satisfaction.

    • Who is the consumer for the feature?
    • What are the e2e use-cases?
    • Why do we need it?
    • Why is it important?
    • What are the data proving the need and the importance?
    • Will it impact other team?

     

    The beauty of these questions is that they help us and the requestors having clarity on the feature.
    Knowing the consumer, help to refine the requirements with the right stakeholder.
    Knowing the e2e use-cases, help understand and scope the feature(s).
    Understanding the ‘why’ is powerful to set the priority and to understand how it impact our consumers.
    Proving the needs and its importance by data is generally the best way to prevent throw away work and to increase the return on investment.
    Finally, identifying the impact on other team, especially external one, will help aligning the plan for dependencies.

    This methodology is a shorter way to define specifications and will help in taking decisions.

     
    2 Kudos
    Don't
    move!

    The Happiness Advantage

    Short before the holiday break, I watched a video titled “happiness as a competitive advantage” where Shawn Achor talk about happiness and how it can help to be more successful at work. I obviously like being happy and I have a pretty competitive mindset, so I really enjoyed the talk. Something really struck me, he said something along this line: “people are trying to be successful as a mean of being happy, instead, studies shows that happy people tends to be more successful”. At that time, I was focusing on the next step on my career, thinking how better everything would be once I would be there. This made me thinking about the fact that I should focus a bit more about how to sustain and increase my happiness level!

    So, I rented the book at the library and below write-up is what I learned and what I will try to keep practicing.

     

    Principle #1: The happiness advantage

    Happiness is the center around which success orbit and gives us a competitive edge. Practicing some techniques (find something to look forward to, commit conscious act of kindness, infuse positivity into your surroundings, exercise, spend money on experiences (not on stuff), exercise a signature strength) can help our brains be more positive, which makes people more intelligent, more creative, more healthy and more successful.

    Principle #2: The fulcrum and the lever

    Focus on all the reasons you will succeed, rather than fail. A specific and concerted focus on your strengths during a difficult task produce the best results.

    Principle #3: The Tetris effect

    Train your brain to focus on the positive, which makes your brain more skilled to noticing and focusing on possibilities for personal and professional growth. A simple exercise to practice daily: write down three good things that happened that day.

    Principle #4: Falling up

    With every setback (mistake, obstacle, failure, disappointment, suffering, …) comes some opportunity for growth that we can teach ourselves to see and take advantage of. As said Tal Ben-Shahar, “things do not necessarily happen for the best, but some people are able to make the best out of things that happen”. Most successful people see adversity not as a stumbling block but as a stepping stone to greatness.

    When people perceive an event in their life, our brain generate a “counterfact”, an alternate scenario used to help evaluate and make sense of what really happens. However, the counterfact is hypothetical, we should force ourselves to pick a positive one.

    One practical way of seeing the path from adversity to opportunity is to do the ABCD model of interpretation:

    1. Adversity – the event we can’t change
    2. Belief – our reaction to the even (why we think it happened and what we think it means for the future)
    3. Consequence – if we see some solutions and the temporality of the belief, it can be positive, otherwise we might become helpless
    4. Disputation – only if we have negative belief and consequence, tell ourselves that our belief is not a fact, dispute (challenge) it: what are the evidence of the belief, are they airtight?

    Principle #5: Zorro circle

    When people believe that the power lies within their control, they have higher achievement. Knowing that our actions determine our fates in life can only spur us to work harder.

    Zorro circle, is a technique to scope on a controllable piece of a problem, fixing it, being proud of it, and creating a new zorro circle (in algorithmics we call this divide and conquer).

    Identifying what part of the problem we have control over and what we don’t have control over also help reduce the stress. A simple exercise is to have a 2 columns table (what we control and what we don’t control) and to classify any stresses, daily challenges, and goals within.

    Principle #6: The 20 second rule

    Habits forms because our brain changes in response to frequent practice.

    By adding barrier (batteries out of the remote control, at 20 sec away) to activities we want to stop doing, we are likely to be more successful. Same for activities we want to do more, making sure we remove as much barrier as possible (within 20 seconds reach), decrease the ‘activation energy’ required to start it.

    Willpower is a limited resource, that get weakened every time we use it (activation, choice).

     

    Principle #7: Social investment

    Social connections (family, friends, colleagues) have a correlation of 0.7 with happiness. It is actually the biggest factor influencing happiness.

     
    0 Kudos
    Don't
    move!

    Keys for influencing without authority

    At the time I was a soldier in the Swiss army (nearly every men in Switzerland have to do its time for a few months), my lieutenant asked me to organize the team so we would set up the trenches for the night. He told me that he wanted to give me the taste of commanding so that it would motivate me to make my grades – which did not happen.

    Typically, in the army, authority is used to make things happen. However, I really hated any forms of authority at that time (still a bit the case actually when I think about it) so I did not felt comfortable giving orders to my peers. I decided to build a common effort, we had a couple of trenches to build and organize the bivouacs, we had open discussions about the best way to position our trenches, where to place our shelters and so on and so forth. I was quite surprised by the fact that everybody seemed eager to help and was active in the discussion, I was really afraid that nobody would be involved. This was the first time in my life I realized that leading without authority was possible and potentially more effective than with authority! I believe my peers were happy to help me succeed for multiple reasons:

    1. We shared the same goals (being able to rest comfortably as fast as possible)
    2. Everybody understood why we were working after the goals (to a certain extend, as we could have gone to the hotel instead …)
    3. I had always behaved respectfully with all of them and tried to be helpful whenever possible

    More than ten years later, I’m still using some of the strategies I developed during that time to influence people I’m working with. Also after nearly a decade of software engineering, I have learnt to deal with different types of people. Believe me or not but some engineer are pretty hard to deal with :).

    Goals, data, information

    This might sound obvious but before trying to influence anyone, we have some homework to do. Understanding the goals (ours and theirs) is a prerequisite to have fruitful conversation. We also need to be able to support our arguments with real data. Data acquisition should be the first step of most project anyway. Data can also be information that can be acquired by discussing with the right people, then we need to validate our data, and get the reality of the current data. With all of these, we should be able to paint with a fair accuracy the big picture and understand what the stake of the different people involved are. People will tend to listen to influencing signal of people who are known to have accurate data and understanding.

    Attitude, character

    How we behave with people counts a lot, for me respect is a prerequisite for any form of collaboration.. Trust is hard to earn, easy to lose. We should be genuine with ourselves and be passionate, with good communication style, have grit, and ready to collaborate (make a win-win situation, don’t push your agenda). Having a clear vision of what we we want to fix is also quintessential to motivate people to join our projects.

    Network

    Being connected with people around you help influencing. If you need to influence someone and you have no direct relationship with him, try to find people in his network. They can put you in contact and can potentially influence early.

    Conclusion (TL; DR)

    As a PM (product | program manager), we don’t get any granted authority. Using data, a positive attitude, and a strong network will help making progress on our projects.

     

     
    1 Kudos
    Don't
    move!

    How to lie with statistics

    Just before the thanksgiving break, I attended to a book club event (internal to Bing) and Scott Berkun advised to read the book “How to lie with statistics”. I decided to use the thanksgiving break time to read it.

    Written by Darrell Huff in 1954, the book was extremely refreshing (the tone used in the book is illustrative of a mindset that we seem to have lost). Understanding the science behind statistics can helps us in multiple ways. First, it is easier to identify statistical bias (conscious or unconscious), secondly it helps asking the right questions when presented with graphs or numbers, lastly it helps presenting and comprehending correctly hypothesis done day to day. It is a super easy read and I strongly recommend it to any people interested in refreshing their statistical science (not math) knowledge.

    The sample with the built-in bias

    Every sample have a built-in bias, the people that gets the survey, the people that answers the survey, the emotional impact of the interviewers on the interviewee, the way the questions are asked. So when looking at sampled data, ask yourself what is the sample representing?

    The well-chosen average

    Different types of average needs to be correctly understood:

    1. Mean: summing up all values and then dividing by the number of entries
    2. Median: the middle value from all entries
    3. Mode: most frequently met value from all entries

    The little figures that are not there

    Using small sample groups will help with ‘injecting’ luck in the experiment. Repeat your experiment as many time as necessary and only keep the results that you like (tossing a penny ten times might be far away from 50%, whereas a thousand times it will be really close to 50%).

    Test of significance: reporting how likely it is that a test figure represents a real result rather that something produced by chance. It is generally expressed as a probability. For most purpose, nothing poorer than 5% of significance is good enough (meaning 95% chance that the result is real).

    It is also necessary to understand the deviation from the average that is given to understand how it might varies.

    The gee-whiz graph

    Representation can be misleading, changing the proportion between the ordinate and the abscissa can have a huge effect on interpretation (small 10% growth compared to a huge 10% growth). Changing the amount of data displayed can also impact the interpretation (cut the graph to only where the line is and suddenly this looks like a 50% increase).

    The one dimensional picture

    Although representing number by pictures (for example, money bag representing the amount of money), the perceived ratio of the picture will not be mono-dimensional. Instead people will perceive 3 dimensions. So doubling the size of a money bag, actually double its width and its length as well as its volume.

    The semiattached figure

    “If you can’t prove what you want to prove, demonstrate something else and pretend that they are the same thing”. There are often many ways of expressing any figure, 1% return on sales, 15% return on investment, $10M profit, increase in profit of 40% (compared to year – 10), a decrease of profit of 60% (compared to year -1).

    Post hoc rides again

    If B follows A, then A has to cause B, however B might be causing A, or have no correlation whatsoever. There are different type of correlation, produced by chance (cannot be easily reproduced), covariation (relationship exist but is unclear what variable is the cause and which the effect), or neither of the variable has any effect at all on the other, yet there is a real correlation (poor grades among cigarettes smoker). The cause and effect nature of the relationship is only a matter of speculation.

    How to statisticulate

    Statisculate: misinforming people by the use of statistical material.

    • Using average: mean, medians, mode.
    • Using decimals number paint a false accuracy (instead of a poor approximation).
    • Shifting base with percentage can distort reality (to offset a pay cut of 50%, you must get a raise of 100%).
    • Adding percentages together when they are not related.
    • Confusion between percentage and percentage points is another confusion (from 3% to 6% is a 3% points increase or a 100% increase).
    • Percentile – a rank between each one hundred

    How to talk back to a statistic

    5 questions to proof the statistic:

    1. who says so?
      1. look for conscious bias (misstatement or ambiguous statement; selection of favorable data; shift in units of measurement; using one year for a comparison and sliding to another more favorable; improper measure, mean vs median vs mode)
      2. look for unconscious bias
      3. When an “OK name” (a trustable one, like doctor a reputed university) make sure the conclusion came from the author, not just the data.
    2. How do you know?
    1. look for the sample size, how many people answered from the pool
    2. watch out for reported correlation, is there enough cases for significance? Is the standard error and probable error shared?
      1. What’s missing?
      2. Did somebody changed the subject?
      3. Does it make sense?
     
    4 Kudos
    Don't
    move!

    How to make estimate (in software development)

    One of the tricky question for a software engineer is to estimate when his code will be ready. It is a tricky question because ‘ready’ is generally not well defined. Ready can mean a lot of different things, depending to whom you ask. The developer might assume having his code working and tested, the program manager most certainly means available for ABTesting, while the leadership team is looking for the time when the change are available for all users. I have a mind list of the milestones that needs to be hit for any new development work, and I’m using it whenever I need to provide estimate or when I’m asking for estimate. It also helps to clarify what are the expectations on the qualities of the work that needs to be accomplished.

    • Analysis: about understanding the requirements and how the system will have to change to support them
    • Design: about closing the technical design (overlapping all involved team), enabling to have a clear end to end picture of the change
    • Coding: actually writing the code change
    • Unit testing: UTs can come before the coding if you live in a test driven development (TDD) world, this is about making sure the functions of your code are working properly in the obvious cases
    • Reviewing: having peer reviewing the code change
    • Iterating on Coding / Reviewing: depending on the developer experience, the criticality of the code changed, and the pickiness of the reviewers, the coding and the reviewing can be based on multiple iterations
    • Testing: validating the code is correctly behaving, even in non-obvious cases, validating overlap with multiple pieces works as intended too.
    • Experimentation / ABTesting: validating that your changes are actually bringing value to the customers and that your key performance indicators (KPIs) are, at least, still looking good (ideally they should improve).
    • Deployment: make the new functionalities available to all your users

    Given the above list, when I’m talking estimate with my gang, I can get suspicious if someone tells me that he will be ready with his change in a couple of days!

     
    0 Kudos
    Don't
    move!

    Introduction to Machine Learning, from data acquisition to a production service

    In this post I want to share some notions of machine learning and a how-to get started with Microsoft Azure ML studio.
    Machine learning is a quite complex topic, especially if you want to understand the theory and algorithms that are behind it. If you want to go deeper in understanding ML I would encourage you to register for the free online course by Andrew Ng on coursera.org. I have done it, it is really great but be ready to invest enough time and brain waves
    J.

    Thankfully, ML studio offers a simple way to do machine learning without the need of understanding any of the algorithms as long as you follow a workflow. My simplistic workflow is split in 4 steps: data acquisition, data preparation, training and tuning, web-service deployment. Let me go through all these steps.

    Machine learning

    Data acquisition

    Machine learning needs two things to work, data (lots of it) and models. When acquiring the data, be sure to have enough features (aspect of data that can help for a prediction, like the surface of the house to predict its price) populated to train correctly your learning model. In general, the more data you have the better so make to come with enough rows!

    Data is generally stored as a CSV file (comma separated value file created with excel) or any other supported type of dataset. Currently the supported max size is 1.95 GB (which is enough for anything not “big-data”).

    Handling the data: data preparation


    Once you have the data available from Azure ML, it’s time to prepare it. There are multiple transformation operations supported: filtering, manipulation, sampling, and scaling and reducing. Full documentation for data transformation. To start, you would want to scrubber missing values, edit the columns, and then split your dataset for training and validation. This will be fed to your training model along with the learner selected.

     

    Training and tuning

    The first thing you need to figure out first is what type of analysis you need to run (see full documentation). It can either be a classification analysis (spam vs non-spam), a clustering analysis (automatic classification), or a regression analysis (for prediction and forecasting). Once you have this figured out, what you need to do is compare the results of multiple models to know which one is the most efficient for your dataset. The logic here is to:

    1. Select a model with default value
    2. Train the model
    3. Score the model
    4. Evaluate the model (to figure out which model is the most efficient)
    5. Sweep the model (to figure out the best configuration for your model)
    6. Evaluate the model
    7. Save the trained model (to be used in production)

    Some useful notions for evaluation each mode (from here):

  • MAE (Mean Absolute Error): The average absolute difference between the predicted value and the actual value
  • RMSE (Root Mean Squared Error): The square root of the average of squared error of prediction made on the dataset
  • RAE (Relative Absolute Error): The average of absolute errors relative to the absolute difference between actual values and the average of all actual values
  • RSE (Relative Squared Error): The average of squared errors relative to the squared difference between the actual values and the average of all actual values.
  • CoD (Coefficient of Determination): Also known as the R squared value, this is a statistical metric indicating how well a model fits the data.
  • For each error, a smaller value indicate a closer match. For CoD, the closer it is from 1.0 the better the prediction.

    When comparing the evaluation of models, you will want to validate that the main values are going up (like accuracy (proportion of true results to total cases), precision (true over all positive results) and recall (fraction of all correct results returned by the model), AUC (Area Under Curve – provide a single number that let compare different model), F1 score (measure of accuracy balancing precision and recall)). See Metrics used for model evaluation for more details.

    Deployment

    Assuming above steps are completed and you have found satisfactory values for your model evaluation, you can clean-up your workflow to have only one model, with the optimum configuration. Publish you score model as an input and define your output. Run and publish your web service. From there you will have access to the API help page containing sample code for C#, R, and Python and a test URL where you can manually set some values.

    In your web service configuration tab, you can enable it as ‘ready for production’ and start using it in real production project!

    Resources

    Machine learning for beginners.
    If you want to go deeper in understanding ML, Andrew Ng have a fantastic class on coursera.org.
    All the videos tutorial that you need to get started with Azure ML.
    Predictive solution walk-through.

     
    0 Kudos
    Don't
    move!