eric.ness.net » Statistics

Scatterplots Using R and MSSQL

Eric — Fri, 27 Nov 2009 15:00:57 +0000

As an extension of yesterdays post here is another fairly cool chart you can do in R. For this little sample we are using the same data as before but for my sql query I have to do a crosstab query. So lets take a look at the code:

# includes
library(RODBC)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT
Country AS 'Country',
Year AS 'Year',
[96741] AS 'GDP growth (annual %)--WDI-2009',
[96841] AS 'GDP per capita (constant 2000 US$)--WDI-2009',
[99941] AS 'Population growth (annual %)--WDI-2009',
[100041] AS 'Population, total--WDI-2009'
FROM
(
SELECT DISTINCT CountryID, Country, Year, IndicatorID, IndValue
FROM [Time Series Data]
WHERE (
((IndicatorID) = 96741) OR
((IndicatorID) = 96841) OR
((IndicatorID) = 99941) OR
((IndicatorID) = 100041))
AND
(((CountryID) = 4118) OR
((CountryID) = 4125) OR
((CountryID) = 4129) OR
((CountryID) = 4134) OR
((CountryID) = 4141) OR
((CountryID) = 4145) OR
((CountryID) = 4164) OR
((CountryID) = 4186) OR
((CountryID) = 4213) OR
((CountryID) = 4327) OR
((CountryID) = 4219) OR
((CountryID) = 4221) OR
((CountryID) = 4227) OR
((CountryID) = 4230) OR
((CountryID) = 4243) OR
((CountryID) = 4326) OR
((CountryID) = 4268) OR
((CountryID) = 4272) OR
((CountryID) = 4273) OR
((CountryID) = 4325) OR
((CountryID) = 4300) OR
((CountryID) = 4308) OR
((CountryID) = 4309) OR
((CountryID) = 4311) OR
((CountryID) = 4316))
AND
(NOT (Year IS NULL)) AND (Year >= 1960) AND
(Year <= 2007))
ps
PIVOT (
MAX(IndValue)
FOR IndicatorID IN ([96741], [96841], [99941], [100041] ) )
AS
pvt
order by Country, Year")

#close connection
odbcClose(channel)

#Plot charts
plot(myData[3:6], col="orange", main="Select Indicators for Europe and Central Asia")

Here is the result

Histogram Lattices Using R and MSSQL

Eric — Thu, 26 Nov 2009 19:22:41 +0000

After getting Joseph Adler’s book “Baseball Hacks” I’ve been wanting to get in to R. R is simply an amazing open source statistics/graphing application. For this example we are going pull data from a MSSQL database and make a histogram lattice of a couple of countries.

First, I pulled the data from HealthSystems2020 time series database and imported the data in to MSSQL. I did some minor touch ups to the database giving the indicator an id etc. The second thing you need to do is create an ODBC connection for your database here is a fairly good tutorial. In this example I called my ODBC DSN “HealthDB”. Also make sure you adjust you sql query so that they are pulling the correct names/values.

Finally, here is the code:


# includes
library(RODBC)
library(lattice)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT Country, IndValue
FROM         [YOURTABLE]
WHERE     (id = 96841) AND (
(Country = 'Afghanistan') OR
(Country = 'Bangladesh') OR
(Country = 'Bhutan') OR
(Country = 'India') OR
(Country = 'Maldives') OR
(Country = 'Nepal') OR
(Country = 'Pakistan') OR
(Country = 'China') OR
(Country = 'Indonesia') OR
(Country = 'Sri Lanka'))")

#close connection
odbcClose(channel)

#create histogram
histogram(~ myData[,3] | myData[,1], type="count", col="red", main = "GDP per capita (constant 2000 US$)", xlab="Country")

And here is the result:

Pearson’s Correlation Coefficient

Eric — Sun, 25 Oct 2009 19:33:25 +0000

In Toby Segaran’s book “Programming Collective Intelligence” one additional methods used “to determine the similarity between people’s interests is to use the Pearson’s correlation coefficient. In statistics Pearson’s correlation coefficient is often symbolized as simply r. I also covered Toby’s Euclidean Distance Score here.

Is how r is calculated.

And here is some sloppy source code to get you going:

using System;
using System.Linq;

namespace PearsonTest
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var myP = new Correlation();

            var lisaRose = new double[] {0, 2, 4, 6, 8, 10, 12};
            var jackMatthews = new[] {2.1, 5, 9, 12.6, 17.3, 21, 24.7};

            double score = myP.PearsonCorrelation(lisaRose, jackMatthews);

            Console.WriteLine(score);
            Console.ReadLine();

            // The answer is 0.99887956534852
        }
    }

    internal class Correlation
    {
        public double PearsonCorrelation(double[] x, double[] y)
        {
            double result;
            double xMean = 0;
            double yMean = 0;
            double xDenom = 0;
            double yDenom = 0;
            double denominator;
            double numerator = 0;
            double n;

            // Make sure arrays are same size and greater than 1
            if ((x.Count() == y.Count()) && (x.Count() >= 1))
            {
                n = x.Count();
            }
            else
            {
                result = 0;
                return result;
            }

            // Find Means
            for (int i = 0; i <= n - 1; i++)
            {
                xMean += x[i];
                yMean += y[i];
            }
            xMean = xMean/n;
            yMean = yMean/n;

            // Caluculate numerator and denominator
            for (int i = 0; i <= n - 1; i++)
            {
                //Caluculate numerator
                double numX = x[i] - xMean;
                double numY = y[i] - yMean;
                numerator += numX*numY;

                // Caluculate denominator parts
                xDenom += Math.Pow(numX, 2);
                yDenom += Math.Pow(numY, 2);
            }

            // Caluculate denominator
            denominator = Math.Sqrt(xDenom*yDenom);

            // Check for division by zero
            if (denominator == 0)
            {
                result = 0;
            }
            else
            {
                result = numerator/denominator;
            }

            return result;
        }
    }
}

The Elements of Statistical Learning: Data Mining, Inference, and Prediction. [Free Book]

Eric — Thu, 15 Oct 2009 17:41:21 +0000

I came across this on Our Signal today, it’s a free e-book: The Elements of Statistical Learning: Data Mining, Inference, and Prediction written by Trevor Hastie, Robert Tibshirani and Jerome Friedman. The book and accompanying site is located here.

Nate Silver & Conn Carroll Video.

Eric — Tue, 16 Jun 2009 17:59:42 +0000

I know this is a bit late but it give some pretty good insight in to Nate Silver of fivethirtyeight.com and how he analyzes polls.

Source: [bloggingheads.tv]

New York Stock Exchange Listing

Eric — Sat, 28 Mar 2009 16:29:22 +0000

I guess it was only a matter of time before I started to take a look at the NYSE, in part because I use a lot of the tools analysts use in looking at stocks but, I use them when looking at the environment or health systems.

Anyway, it is oddly difficult to find a complete listing of the stocks on the exchange – so I compiled a list of all the stocks (not including NYSE ARCA, NYSE EURONEXT, NYSE ALTERNEXT).

Download the excel file. [Source]

Some of the things I use or find interesting when looking at indicators:

Moving Average Convergence/Divergence (MACD)
Relative Strength Index (RSI)
Williams %R
Bollinger Bands

Euclidean Distance Score

Eric — Fri, 31 Oct 2008 02:14:19 +0000

I am currently reading Toby Segaran’s book “Programming Collective Intelligence” and one of the first topics it covers is how do you determine of similar two people are.

One approach is to use the Euclidean Distance Score. Arun Vijayan C has an excellent power point presentation “Finding more people like you” – on this topic:

Finding more people like you

View SlideShare presentation or Upload your own. (tags: socialnetwork compare)

I then wrote up some quick code in C# that uses the values in Arun’s presentation:

// Euclidean Distance Score

using System;
using System.Collections.Generic;

namespace ConsoleApplication1
{
    internal class Program
    {
        private static void Main()
        {
            // Load People and Values
            var myP = new List
                          {
                              new People("John", 1.5, 4),
                              new People("Ravi", 4.5, 1.5),
                              new People("Kiran", 1, 3.5),
                              new People("Deepti", 3, 5)
                          };

            // Print header
            Console.WriteLine("People And Scores");
            Console.WriteLine("###################");
            Console.WriteLine();

            // Loop through people and values
            foreach (People people in myP)
            {
                Console.WriteLine(people.Name + "\t" + people.xScore + "\t" + people.yScore);
            }

            // Print Distance And Value Headers
            Console.WriteLine();
            Console.WriteLine("Distance Comparison");
            Console.WriteLine("###################");
            Console.WriteLine();

            // Loop through people and scores
            int myCount = 1;
            for (int i = 0; i < myP.Count; i++)
            {
                for (int j = myCount; j < myP.Count; j++)
                {
                    // Euclidean Distance Score
                    // Sqrt( (x1-x2)^2 + (y1+y2)^2)
                    Console.WriteLine(myP[i].Name + "\t" + myP[j].Name + ":\t" +
                                      Math.Sqrt(Math.Pow(myP[i].xScore - myP[j].xScore, 2) +
                                                Math.Pow(myP[i].yScore - myP[j].yScore, 2)).ToString("0.##"));
                }

                // Skip to the next guy
                myCount++;
            }

            // Print Closer
            Console.WriteLine();
            Console.WriteLine("Press enter to continue...");
            Console.ReadLine();
        }
    }

    internal class People
    {
        public string Name;
        public double xScore;
        public double yScore;

        public People(string _Name, double _xScore, double _yScore)
        {
            Name = _Name;
            xScore = _xScore;
            yScore = _yScore;
        }
    }
}

Presidential Election Poll Stats

Eric — Wed, 10 Sep 2008 18:01:33 +0000

I am love watching the polls. So I thought I would post some of my favorite links to site that I think do a pretty good job at covering them.

Benford’s Law

Eric — Thu, 04 Sep 2008 16:58:22 +0000

Benford’s Law roughly states that values of measurements in nature often have a logarithmic distribution. Which produces a counter-intuitive result that when applied to a wide variety of natural data sets, including stock prices and population numbers, etc. these things tend to follow that distribution.

In this video it give a rough explanation of how Benford’s Law can be used in Fraud Detection.

Forecasting Dates

Eric — Mon, 13 Aug 2007 02:39:50 +0000

So this is a little thing that I have been working on – nothing too interesting but, I like it. Essentially, if you have a dataset with time on the x axis and a projected goal this function will tell you the exact date your data projection will intersect with your goal.