Monthly Archives: November 2014

How to lie with statistics

Just before the thanksgiving break, I attended to a book club event (internal to Bing) and Scott Berkun advised to read the book “How to lie with statistics”. I decided to use the thanksgiving break time to read it.

Written by Darrell Huff in 1954, the book was extremely refreshing (the tone used in the book is illustrative of a mindset that we seem to have lost). Understanding the science behind statistics can helps us in multiple ways. First, it is easier to identify statistical bias (conscious or unconscious), secondly it helps asking the right questions when presented with graphs or numbers, lastly it helps presenting and comprehending correctly hypothesis done day to day. It is a super easy read and I strongly recommend it to any people interested in refreshing their statistical science (not math) knowledge.

The sample with the built-in bias

Every sample have a built-in bias, the people that gets the survey, the people that answers the survey, the emotional impact of the interviewers on the interviewee, the way the questions are asked. So when looking at sampled data, ask yourself what is the sample representing?

The well-chosen average

Different types of average needs to be correctly understood:

  1. Mean: summing up all values and then dividing by the number of entries
  2. Median: the middle value from all entries
  3. Mode: most frequently met value from all entries

The little figures that are not there

Using small sample groups will help with ‘injecting’ luck in the experiment. Repeat your experiment as many time as necessary and only keep the results that you like (tossing a penny ten times might be far away from 50%, whereas a thousand times it will be really close to 50%).

Test of significance: reporting how likely it is that a test figure represents a real result rather that something produced by chance. It is generally expressed as a probability. For most purpose, nothing poorer than 5% of significance is good enough (meaning 95% chance that the result is real).

It is also necessary to understand the deviation from the average that is given to understand how it might varies.

The gee-whiz graph

Representation can be misleading, changing the proportion between the ordinate and the abscissa can have a huge effect on interpretation (small 10% growth compared to a huge 10% growth). Changing the amount of data displayed can also impact the interpretation (cut the graph to only where the line is and suddenly this looks like a 50% increase).

The one dimensional picture

Although representing number by pictures (for example, money bag representing the amount of money), the perceived ratio of the picture will not be mono-dimensional. Instead people will perceive 3 dimensions. So doubling the size of a money bag, actually double its width and its length as well as its volume.

The semiattached figure

“If you can’t prove what you want to prove, demonstrate something else and pretend that they are the same thing”. There are often many ways of expressing any figure, 1% return on sales, 15% return on investment, $10M profit, increase in profit of 40% (compared to year – 10), a decrease of profit of 60% (compared to year -1).

Post hoc rides again

If B follows A, then A has to cause B, however B might be causing A, or have no correlation whatsoever. There are different type of correlation, produced by chance (cannot be easily reproduced), covariation (relationship exist but is unclear what variable is the cause and which the effect), or neither of the variable has any effect at all on the other, yet there is a real correlation (poor grades among cigarettes smoker). The cause and effect nature of the relationship is only a matter of speculation.

How to statisticulate

Statisculate: misinforming people by the use of statistical material.

  • Using average: mean, medians, mode.
  • Using decimals number paint a false accuracy (instead of a poor approximation).
  • Shifting base with percentage can distort reality (to offset a pay cut of 50%, you must get a raise of 100%).
  • Adding percentages together when they are not related.
  • Confusion between percentage and percentage points is another confusion (from 3% to 6% is a 3% points increase or a 100% increase).
  • Percentile – a rank between each one hundred

How to talk back to a statistic

5 questions to proof the statistic:

  1. who says so?
    1. look for conscious bias (misstatement or ambiguous statement; selection of favorable data; shift in units of measurement; using one year for a comparison and sliding to another more favorable; improper measure, mean vs median vs mode)
    2. look for unconscious bias
    3. When an “OK name” (a trustable one, like doctor a reputed university) make sure the conclusion came from the author, not just the data.
  2. How do you know?
  1. look for the sample size, how many people answered from the pool
  2. watch out for reported correlation, is there enough cases for significance? Is the standard error and probable error shared?
    1. What’s missing?
    2. Did somebody changed the subject?
    3. Does it make sense?
 
4 Kudos
Don't
move!

How to make estimate (in software development)

One of the tricky question for a software engineer is to estimate when his code will be ready. It is a tricky question because ‘ready’ is generally not well defined. Ready can mean a lot of different things, depending to whom you ask. The developer might assume having his code working and tested, the program manager most certainly means available for ABTesting, while the leadership team is looking for the time when the change are available for all users. I have a mind list of the milestones that needs to be hit for any new development work, and I’m using it whenever I need to provide estimate or when I’m asking for estimate. It also helps to clarify what are the expectations on the qualities of the work that needs to be accomplished.

  • Analysis: about understanding the requirements and how the system will have to change to support them
  • Design: about closing the technical design (overlapping all involved team), enabling to have a clear end to end picture of the change
  • Coding: actually writing the code change
  • Unit testing: UTs can come before the coding if you live in a test driven development (TDD) world, this is about making sure the functions of your code are working properly in the obvious cases
  • Reviewing: having peer reviewing the code change
  • Iterating on Coding / Reviewing: depending on the developer experience, the criticality of the code changed, and the pickiness of the reviewers, the coding and the reviewing can be based on multiple iterations
  • Testing: validating the code is correctly behaving, even in non-obvious cases, validating overlap with multiple pieces works as intended too.
  • Experimentation / ABTesting: validating that your changes are actually bringing value to the customers and that your key performance indicators (KPIs) are, at least, still looking good (ideally they should improve).
  • Deployment: make the new functionalities available to all your users

Given the above list, when I’m talking estimate with my gang, I can get suspicious if someone tells me that he will be ready with his change in a couple of days!

 
0 Kudos
Don't
move!