Let me say I am a huge fan of Monte Carlo Simulations. For those of you who are not familiar with Monte Carlo Simulations – Monte Carlo Simulations use random numbers and a model to simulate an outcome or event but the real strength in Monte Carlo Simulations is that you repeat the simulation hundreds if not thousands of times. After running the simulation, the results as a whole give you some great insight in to possible outcomes you can expect. The results are often more robust and accurate than other methods like regressions.

For this tutorial, I am going to replicate the core functionality of this report from Lumenaut. The model itself has a couple of fixed costs: Labor, Price per Widget and Rent. The model also has two variable items: Variable Cost per Widget and Number of Widgets Sold. So just like in real life if I want to figure out how much profit I am going to make I need to subtract my total costs from my revenue.

But, where we see the strength of a Monte Carlo Simulation is that it says let’s take a look at this model again but, maybe I can buy my widgets a little cheaper and maybe I can sell a couple more. In addition, every month is a little different some you sell a couple more some a couple of less.

To simulate this we need to use some random numbers but a normal random number generator will not work because that will give us a uniform distribution. Some random number for a Monte Carlo Simulation needs to have very specific distributions: normal, log, triangular, gamma, or something else to better simulate real life. The React.NET library gives us a very easy simple way to simulate all of these very easily.

So for Lumenaut’s model we need two normal distribution random number generators and the profit model outlined in the excel table and following graph. Please note that the Cost Per Widget is 5 with a standard deviation of 0.5 and the Number of Widgets Sold is 2,000 with a standard deviation of 200.

Here is the code I use.

using System.Collections.Generic;
using React.Distribution;

namespace MonteCarloSimulation.Models
{
    /// <summary>
    /// Monte Carlo Simulation
    /// </summary>
    public class MonteCarloModel
    {
        private const double FixedCost = 170000;
        private const double FixedSellingPrice = 100;
        private readonly double _costPerWidget;
        private readonly double _costPerWidgetSd;
        private readonly double _numOfSimulations;
        private readonly double _numOfWidgetSd;
        private readonly double _numOfWidgets;
        public int NumberOfSimulations = 10000;
        public List<double> RESULTS = new List<double>();

        /// <summary>
        /// Initializes a new instance of the <see cref="MonteCarloModel"/> class.
        /// </summary>
        /// <param name="costPerWidget">The cost per widget.</param>
        /// <param name="costPerWidgetSd">The cost per widget sd.</param>
        /// <param name="numOfWidgets">The num of widgets.</param>
        /// <param name="numOfWidgetSd">The num of widget sd.</param>
        /// <param name="numOfSimulations">The num of simulations.</param>
        public MonteCarloModel(double costPerWidget,
                               double costPerWidgetSd,
                               double numOfWidgets,
                               double numOfWidgetSd,
                               double numOfSimulations)
        {
            _costPerWidget = costPerWidget;
            _costPerWidgetSd = costPerWidgetSd;
            _numOfWidgets = numOfWidgets;
            _numOfWidgetSd = numOfWidgetSd;
            _numOfSimulations = numOfSimulations;
            Run();
        }

        /// <summary>
        /// Runs the Monte Carlo Simulation
        /// </summary>
        private void Run()
        {
            // Set up our Normal distributions with the mean, and the Standard Deviation
            var costPerWidgetDist = new Normal(_costPerWidget, _costPerWidgetSd);
            var numberOfWidgetDist = new Normal(_numOfWidgets, _numOfWidgetSd);

            for (int i = 0; i < _numOfSimulations; i++)
            {
                // Get the next ranom number for our model
                double costPerWidget = costPerWidgetDist.NextDouble();
                double numberOfWidgetsSold = numberOfWidgetDist.NextDouble();

                // Calculate the revenue
                double revenue = numberOfWidgetsSold*FixedSellingPrice;

                // Calculate the costs
                double cost = (costPerWidget*numberOfWidgetsSold + FixedCost);

                // Add result to our results list
                RESULTS.Add(revenue - cost);
            }
        }
    }
}

Here is a little display I put together and as you can see, the results are very similar; they will not be exactly the same as there is a random component to all of this.

Download the code!

Note:
There are a number of things to note here:

1. This is nowhere near production code (read as good code) – it is simply for you to look at and get you going down the road to Monte Carlo Simulations. I think I spent all of 30 minutes putting it together so don’t expect anything nice and neat. :-)
2. I had to remove the Dundas graphing library, as I cannot distribute it per my license. But, it should work just fine with the Microsoft graphing library.
3. Enjoy!

As an extension of yesterdays post here is another fairly cool chart you can do in R. For this little sample we are using the same data as before but for my sql query I have to do a crosstab query. So lets take a look at the code:

# includes
library(RODBC)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT
Country AS 'Country',
Year AS 'Year',
[96741] AS 'GDP growth (annual %)--WDI-2009',
[96841] AS 'GDP per capita (constant 2000 US$)--WDI-2009',
[99941] AS 'Population growth (annual %)--WDI-2009',
[100041] AS 'Population, total--WDI-2009'
FROM
(
SELECT DISTINCT CountryID, Country, Year, IndicatorID, IndValue
FROM [Time Series Data]
WHERE (
((IndicatorID) = 96741) OR
((IndicatorID) = 96841) OR
((IndicatorID) = 99941) OR
((IndicatorID) = 100041))
AND
(((CountryID) = 4118) OR
((CountryID) = 4125) OR
((CountryID) = 4129) OR
((CountryID) = 4134) OR
((CountryID) = 4141) OR
((CountryID) = 4145) OR
((CountryID) = 4164) OR
((CountryID) = 4186) OR
((CountryID) = 4213) OR
((CountryID) = 4327) OR
((CountryID) = 4219) OR
((CountryID) = 4221) OR
((CountryID) = 4227) OR
((CountryID) = 4230) OR
((CountryID) = 4243) OR
((CountryID) = 4326) OR
((CountryID) = 4268) OR
((CountryID) = 4272) OR
((CountryID) = 4273) OR
((CountryID) = 4325) OR
((CountryID) = 4300) OR
((CountryID) = 4308) OR
((CountryID) = 4309) OR
((CountryID) = 4311) OR
((CountryID) = 4316))
AND
(NOT (Year IS NULL)) AND (Year >= 1960) AND
(Year <= 2007))
ps
PIVOT (
MAX(IndValue)
FOR IndicatorID IN ([96741], [96841], [99941], [100041] ) )
AS
pvt
order by Country, Year")

#close connection
odbcClose(channel)

#Plot charts
plot(myData[3:6], col="orange", main="Select Indicators for Europe and Central Asia")

Here is the result

After getting Joseph Adler’s book “Baseball Hacks” I’ve been wanting to get in to R. R is simply an amazing open source statistics/graphing application. For this example we are going pull data from a MSSQL database and make a histogram lattice of a couple of countries.

First, I pulled the data from HealthSystems2020 time series database and imported the data in to MSSQL. I did some minor touch ups to the database giving the indicator an id etc. The second thing you need to do is create an ODBC connection for your database here is a fairly good tutorial. In this example I called my ODBC DSN “HealthDB”. Also make sure you adjust you sql query so that they are pulling the correct names/values.

Finally, here is the code:

# includes
library(RODBC)
library(lattice)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT Country, IndValue
FROM         [YOURTABLE]
WHERE     (id = 96841) AND (
(Country = 'Afghanistan') OR
(Country = 'Bangladesh') OR
(Country = 'Bhutan') OR
(Country = 'India') OR
(Country = 'Maldives') OR
(Country = 'Nepal') OR
(Country = 'Pakistan') OR
(Country = 'China') OR
(Country = 'Indonesia') OR
(Country = 'Sri Lanka'))")

#close connection
odbcClose(channel)

#create histogram
histogram(~ myData[,3] | myData[,1], type="count", col="red", main = "GDP per capita (constant 2000 US$)", xlab="Country")

And here is the result:

In Toby Segaran’s book “Programming Collective Intelligence” one additional methods used “to determine the similarity between people’s interests is to use the Pearson’s correlation coefficient. In statistics Pearson’s correlation coefficient is often symbolized as simply r. I also covered Toby’s Euclidean Distance Score here.

Is how r is calculated.

And here is some sloppy source code to get you going:

using System;
using System.Linq;

namespace PearsonTest
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var myP = new Correlation();

            var lisaRose = new double[] {0, 2, 4, 6, 8, 10, 12};
            var jackMatthews = new[] {2.1, 5, 9, 12.6, 17.3, 21, 24.7};

            double score = myP.PearsonCorrelation(lisaRose, jackMatthews);

            Console.WriteLine(score);
            Console.ReadLine();

            // The answer is 0.99887956534852
        }
    }

    internal class Correlation
    {
        public double PearsonCorrelation(double[] x, double[] y)
        {
            double result;
            double xMean = 0;
            double yMean = 0;
            double xDenom = 0;
            double yDenom = 0;
            double denominator;
            double numerator = 0;
            double n;

            // Make sure arrays are same size and greater than 1
            if ((x.Count() == y.Count()) && (x.Count() >= 1))
            {
                n = x.Count();
            }
            else
            {
                result = 0;
                return result;
            }

            // Find Means
            for (int i = 0; i <= n - 1; i++)
            {
                xMean += x[i];
                yMean += y[i];
            }
            xMean = xMean/n;
            yMean = yMean/n;

            // Caluculate numerator and denominator
            for (int i = 0; i <= n - 1; i++)
            {
                //Caluculate numerator
                double numX = x[i] - xMean;
                double numY = y[i] - yMean;
                numerator += numX*numY;

                // Caluculate denominator parts
                xDenom += Math.Pow(numX, 2);
                yDenom += Math.Pow(numY, 2);
            }

            // Caluculate denominator
            denominator = Math.Sqrt(xDenom*yDenom);

            // Check for division by zero
            if (denominator == 0)
            {
                result = 0;
            }
            else
            {
                result = numerator/denominator;
            }

            return result;
        }
    }
}

I came across this on Our Signal today, it’s a free e-book: The Elements of Statistical Learning: Data Mining, Inference, and Prediction written by Trevor Hastie, Robert Tibshirani and Jerome Friedman. The book and accompanying site is located here.

I know this is a bit late but it give some pretty good insight in to Nate Silver of fivethirtyeight.com and how he analyzes polls.

Source: [bloggingheads.tv]

I guess it was only a matter of time before I started to take a look at the NYSE, in part because I use a lot of the tools analysts use in looking at stocks but, I use them when looking at the environment or health systems.

Anyway, it is oddly difficult to find a complete listing of the stocks on the exchange – so I compiled a list of all the stocks (not including NYSE ARCA, NYSE EURONEXT, NYSE ALTERNEXT).

Download the excel file. [Source]

Some of the things I use or find interesting when looking at indicators:

Moving Average Convergence/Divergence (MACD)
Relative Strength Index (RSI)
Williams %R
Bollinger Bands

I am currently reading Toby Segaran’s book “Programming Collective Intelligence” and one of the first topics it covers is how do you determine of similar two people are.

One approach is to use the Euclidean Distance Score. Arun Vijayan C has an excellent power point presentation “Finding more people like you” – on this topic:

I then wrote up some quick code in C# that uses the values in Arun’s presentation:

// Euclidean Distance Score

using System;
using System.Collections.Generic;

namespace ConsoleApplication1
{
    internal class Program
    {
        private static void Main()
        {
            // Load People and Values
            var myP = new List
                          {
                              new People("John", 1.5, 4),
                              new People("Ravi", 4.5, 1.5),
                              new People("Kiran", 1, 3.5),
                              new People("Deepti", 3, 5)
                          };

            // Print header
            Console.WriteLine("People And Scores");
            Console.WriteLine("###################");
            Console.WriteLine();

            // Loop through people and values
            foreach (People people in myP)
            {
                Console.WriteLine(people.Name + "\t" + people.xScore + "\t" + people.yScore);
            }

            // Print Distance And Value Headers
            Console.WriteLine();
            Console.WriteLine("Distance Comparison");
            Console.WriteLine("###################");
            Console.WriteLine();

            // Loop through people and scores
            int myCount = 1;
            for (int i = 0; i < myP.Count; i++)
            {
                for (int j = myCount; j < myP.Count; j++)
                {
                    // Euclidean Distance Score
                    // Sqrt( (x1-x2)^2 + (y1+y2)^2)
                    Console.WriteLine(myP[i].Name + "\t" + myP[j].Name + ":\t" +
                                      Math.Sqrt(Math.Pow(myP[i].xScore - myP[j].xScore, 2) +
                                                Math.Pow(myP[i].yScore - myP[j].yScore, 2)).ToString("0.##"));
                }

                // Skip to the next guy
                myCount++;
            }

            // Print Closer
            Console.WriteLine();
            Console.WriteLine("Press enter to continue...");
            Console.ReadLine();
        }
    }

    internal class People
    {
        public string Name;
        public double xScore;
        public double yScore;

        public People(string _Name, double _xScore, double _yScore)
        {
            Name = _Name;
            xScore = _xScore;
            yScore = _yScore;
        }
    }
}

I love watching the polls. So I thought I would post some of my favorite links to site that I think do a pretty good job at covering them.

Pollster

Real Clear Politics

Five Thirty Eight

Intrade

Benford’s Law roughly states that values of measurements in nature often have a logarithmic distribution. Which produces a counter-intuitive result that when applied to a wide variety of natural data sets, including stock prices and population numbers, etc. these things tend to follow that distribution.

In this video it give a rough explanation of how Benford’s Law can be used in Fraud Detection.