eric.ness.net » Visualization

UUorld Map Visualization

Eric — Mon, 01 Feb 2010 20:50:28 +0000

UUorld has quickly become one of my new favorite toys as of late. What is it? As the site explains it “provides an immersive mapping environment, high-quality data, and critical analysis tools.”

Here is just a simple result of what the map looks like:

Here are some of the highlights:

Supports time series data
Fairly extensive database to pull data from online (apparently of 10,000 different datasets)
Create you own datasets via csv files
Export to video
Export to KML file for use in google maps/earth.
Has the following border sets: Country, US States/Counties, US Zip Codes

There is one caveat that I feel I must add to not give the impression that all is rosy – I bought the application over a week ago and it took several e-mails for them to reply to me and finally get my full download.

Scatterplots Using R and MSSQL

Eric — Fri, 27 Nov 2009 15:00:57 +0000

As an extension of yesterdays post here is another fairly cool chart you can do in R. For this little sample we are using the same data as before but for my sql query I have to do a crosstab query. So lets take a look at the code:

# includes
library(RODBC)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT
Country AS 'Country',
Year AS 'Year',
[96741] AS 'GDP growth (annual %)--WDI-2009',
[96841] AS 'GDP per capita (constant 2000 US$)--WDI-2009',
[99941] AS 'Population growth (annual %)--WDI-2009',
[100041] AS 'Population, total--WDI-2009'
FROM
(
SELECT DISTINCT CountryID, Country, Year, IndicatorID, IndValue
FROM [Time Series Data]
WHERE (
((IndicatorID) = 96741) OR
((IndicatorID) = 96841) OR
((IndicatorID) = 99941) OR
((IndicatorID) = 100041))
AND
(((CountryID) = 4118) OR
((CountryID) = 4125) OR
((CountryID) = 4129) OR
((CountryID) = 4134) OR
((CountryID) = 4141) OR
((CountryID) = 4145) OR
((CountryID) = 4164) OR
((CountryID) = 4186) OR
((CountryID) = 4213) OR
((CountryID) = 4327) OR
((CountryID) = 4219) OR
((CountryID) = 4221) OR
((CountryID) = 4227) OR
((CountryID) = 4230) OR
((CountryID) = 4243) OR
((CountryID) = 4326) OR
((CountryID) = 4268) OR
((CountryID) = 4272) OR
((CountryID) = 4273) OR
((CountryID) = 4325) OR
((CountryID) = 4300) OR
((CountryID) = 4308) OR
((CountryID) = 4309) OR
((CountryID) = 4311) OR
((CountryID) = 4316))
AND
(NOT (Year IS NULL)) AND (Year >= 1960) AND
(Year <= 2007))
ps
PIVOT (
MAX(IndValue)
FOR IndicatorID IN ([96741], [96841], [99941], [100041] ) )
AS
pvt
order by Country, Year")

#close connection
odbcClose(channel)

#Plot charts
plot(myData[3:6], col="orange", main="Select Indicators for Europe and Central Asia")

Here is the result

Histogram Lattices Using R and MSSQL

Eric — Thu, 26 Nov 2009 19:22:41 +0000

After getting Joseph Adler’s book “Baseball Hacks” I’ve been wanting to get in to R. R is simply an amazing open source statistics/graphing application. For this example we are going pull data from a MSSQL database and make a histogram lattice of a couple of countries.

First, I pulled the data from HealthSystems2020 time series database and imported the data in to MSSQL. I did some minor touch ups to the database giving the indicator an id etc. The second thing you need to do is create an ODBC connection for your database here is a fairly good tutorial. In this example I called my ODBC DSN “HealthDB”. Also make sure you adjust you sql query so that they are pulling the correct names/values.

Finally, here is the code:


# includes
library(RODBC)
library(lattice)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT Country, IndValue
FROM         [YOURTABLE]
WHERE     (id = 96841) AND (
(Country = 'Afghanistan') OR
(Country = 'Bangladesh') OR
(Country = 'Bhutan') OR
(Country = 'India') OR
(Country = 'Maldives') OR
(Country = 'Nepal') OR
(Country = 'Pakistan') OR
(Country = 'China') OR
(Country = 'Indonesia') OR
(Country = 'Sri Lanka'))")

#close connection
odbcClose(channel)

#create histogram
histogram(~ myData[,3] | myData[,1], type="count", col="red", main = "GDP per capita (constant 2000 US$)", xlab="Country")

And here is the result:

K-Means Document Clustering

Eric — Fri, 06 Nov 2009 17:35:48 +0000

Using our previous example as a basis to move to the next step let’s take a look at clustering using the K-Means clustering algorithm to group the documents in to their appropriate categories.

In the paper â€œIndexing by Latent Semantic Analysisâ€ (Deerwester et al.) they have an example of 9 titles of different papers grouped in to two categories â€œhuman computer interactionâ€ & â€œgraphs & treesâ€. So far, we’ve used Singular Value Decomposition (SVD) and Latent Semantic Indexing (LSI) to better understand the relationship of words and documents. In the last blog post we then took the results in LSI to plot words and documents on a two dimensional Cartesian plane.

All of this is pretty interesting stuff in and of itself however, the next step really is to see which documents belong in each group. One way to do this is by using K-Means clustering.

Simply speaking k-means clustering is an algorithm to classify or to group your objects based on attributes/features into K number of group. K is positive integer number. The grouping is done by minimizing the sum of squares of distances between data and the corresponding cluster centroid. Thus the purpose of K-mean clustering is to classify the data. [Kardi Teknomo]

A big chunk of the code is built off of the same project we are working on. I am using Aresh Saharkhiz K-Means implementation in the project with some minor changes/refactoring done by me.

Let take a look at the code!

This first part is the display (an ASP.NET app.)

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="LSITest._Default" %>
<%@ Register Assembly="DundasWebChart" Namespace="Dundas.Charting.WebControl" TagPrefix="DCWC" %>




    LSI Test

This is the code behind for the ASP.NET page. Because we are only dealing with two known categories K-Means is plotting out the two categories and if you wanted to do more you would definitely have to re-write the ColorCodeDocuments function.

using System;
using System.Data;
using System.Drawing;
using System.Web.UI;
using Dundas.Charting.WebControl;

namespace LSITest
{
    public partial class _Default : Page
    {
        /// 
        /// Handles the Load event of the Page control.
        /// 
        /// The source of the event.
        /// The  instance containing the event data.
        protected void Page_Load(object sender, EventArgs e)
        {
            // Perform LSI
            var mylsi = new lsi();
            mylsi.LSITest();
            double[,] myDocs = mylsi.MyDocs;

            // Plot Documents and the k-means
            const string distanceType = "manhattan";
            PlotDocuments(myDocs, mylsi.MyDocRowCount);
            PlotKMeansPoints(myDocs, 2, distanceType);
            ColorCodeDocuments(distanceType);

            // If you want to plot the words just un-comment the next two lines
            //double[,] myWords = mylsi.MyWords;
            //PlotWords(myDocs, mylsi.MyWordsRowCount);

            // comment this line out to show words in legend
            Chart1.Series["Series2"].ShowInLegend = false;
        }

        /// 
        /// Plots the words.
        /// 
        /// My words.
        /// My words row count.
        private void PlotWords(double[,] myWords, int myWordsRowCount)
        {
            for (int i = 0; i < myWordsRowCount; i++)
            {
                Chart1.Series["Series2"].Points.AddXY(myWords[i, 0], myWords[i, 1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series2"].LegendText = "Words";
            Chart1.Series["Series2"].Color = Color.Gray;
            Chart1.Series["Series2"].MarkerStyle = MarkerStyle.Circle;
            Chart1.Series["Series2"].MarkerSize = 6;
        }

        /// 
        /// Plots the documents.
        /// 
        /// My docs.
        /// My doc row count.
        private void PlotDocuments(double[,] myDocs, int myDocRowCount)
        {
            // Load documents
            for (int i = 0; i < myDocRowCount; i++)
            {
                Chart1.Series["Series1"].Points.AddXY(myDocs[i, 0], myDocs[i, 1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series1"].LegendText = "Documents";
            Chart1.Series["Series1"].Color = Color.Red;
            Chart1.Series["Series1"].MarkerStyle = MarkerStyle.Diamond;
            Chart1.Series["Series1"].MarkerSize = 12;
        }

        /// 
        /// Plots the K means points.
        /// 
        /// The items.
        /// The k.
        /// 
        private void PlotKMeansPoints(double[,] items, int k, string distanceType)
        {
            ClusterCollection clusters = kmeans.ClusterDataSet(k, items, distanceType);

            for (int i = 0; i < clusters.Count; i++)
            {
                Chart1.Series["Series3"].Points.AddXY(clusters[i].ClusterMean[0], clusters[i].ClusterMean[1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series3"].LegendText = "Cluster";
            Chart1.Series["Series3"].Color = Color.Gold;
            Chart1.Series["Series3"].MarkerStyle = MarkerStyle.Star6;
            Chart1.Series["Series3"].MarkerSize = 18;
        }

        /// 
        /// Colors the code documents.
        /// 
        /// Type of the distance.
        private void ColorCodeDocuments(string distanceType)
        {
            var myDist = new similarity();

            // Extract data
            DataSet myDocs = Chart1.DataManipulator.ExportSeriesValues("Series1");
            DataSet myKMeansPoints = Chart1.DataManipulator.ExportSeriesValues("Series3");

            // Document counter
            int count = 0;

            // Get co-ordinates for k-means points
            double firstKMeansX = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[0]["X"]);
            double firstKMeansY = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[0]["Y"]);
            double secondKMeansX = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[1]["X"]);
            double secondKMeansY = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[1]["Y"]);

            foreach (DataRow docRow in myDocs.Tables[0].Rows)
            {
                // get co-ordinates for current doc
                double currentDocX = Convert.ToDouble(docRow["X"]);
                double currentDocY = Convert.ToDouble(docRow["Y"]);

                // load in to arrays
                double[] firstX = {currentDocX, currentDocY};
                double[] firstY = {firstKMeansX, firstKMeansY};
                double[] secondX = {currentDocX, currentDocY};
                double[] secondY = {secondKMeansX, secondKMeansY};

                // find the distance
                double firstDist = myDist.FindDistance(firstX, firstY, distanceType);
                double secondDist = myDist.FindDistance(secondX, secondY, distanceType);

                // Color accordingly
                Chart1.Series["Series1"].Points[count].Color = firstDist < secondDist ? Color.Blue : Color.Gray;
                count++;
            }
        }
    }
}

This is the K-Means class written by Aresh Saharkhiz with my changes

/// Most of this code was written by Aresh Saharkhiz
/// Re-organized by me
/// See Code Project: http://www.codeproject.com/KB/recipes/K-Mean_Clustering.aspx
using System;
using System.Collections;
using System.Data;
using System.Diagnostics;

namespace LSITest
{
    public class kmeans
    {
        /// 
        /// Calculates The Mean Of A Cluster OR The Cluster Center
        /// 
        /// 
        /// A two-dimensional array containing a dataset of numeric values
        /// 
        /// 
        /// Returns an Array Defining A Data Point Representing The Cluster Mean or Centroid
        /// 
        public static double[] ClusterMean(double[,] cluster)
        {
            int rowCount = cluster.GetUpperBound(0) + 1;
            int fieldCount = cluster.GetUpperBound(1) + 1;
            var dataSum = new double[1,fieldCount];
            var centroid = new double[fieldCount];

            for (int j = 0; j < fieldCount; j++)
            {
                for (int i = 0; i < rowCount; i++)
                {
                    dataSum[0, j] = dataSum[0, j] + cluster[i, j];
                }

                centroid[j] = (dataSum[0, j]/rowCount);
            }

            return centroid;
        }

        /// 
        /// Seperates a dataset into clusters or groups with similar characteristics
        /// 
        /// The number of clusters or groups to form
        /// An array containing data that will be clustered
        /// 
        /// A collection of clusters of data
        public static ClusterCollection ClusterDataSet(int clusterCount, double[,] data, string type)
        {
            int rowCount = data.GetUpperBound(0) + 1;
            int fieldCount = data.GetUpperBound(1) + 1;
            int stableClustersCount = 0;
            double[] dataPoint;
            var random = new Random();
            Cluster cluster;
            var clusters = new ClusterCollection();
            var clusterNumbers = new ArrayList(clusterCount);
            var myDist = new similarity();

            while (clusterNumbers.Count < clusterCount)
            {
                int clusterNumber = random.Next(0, rowCount - 1);

                if (!clusterNumbers.Contains(clusterNumber))
                {
                    cluster = new Cluster();
                    clusterNumbers.Add(clusterNumber);
                    dataPoint = new double[fieldCount];

                    for (int field = 0; field < fieldCount; field++)
                    {
                        dataPoint.SetValue((data[clusterNumber, field]), field);
                    }

                    cluster.Add(dataPoint);
                    clusters.Add(cluster);
                }
            }

            while (stableClustersCount != clusters.Count)
            {
                stableClustersCount = 0;
                ClusterCollection newClusters = ClusterDataSet(clusters, data, type);

                for (int clusterIndex = 0; clusterIndex < clusters.Count; clusterIndex++)
                {
                    if ((myDist.FindDistance(newClusters[clusterIndex].ClusterMean, clusters[clusterIndex].ClusterMean, type)) == 0)
                    {
                        stableClustersCount++;
                    }
                }

                clusters = newClusters;
            }

            return clusters;
        }

        /// 
        /// Seperates a dataset into clusters or groups with similar characteristics
        /// 
        /// A collection of data clusters
        /// An array containing data to b eclustered
        /// 
        /// A collection of clusters of data
        public static ClusterCollection ClusterDataSet(ClusterCollection clusters, double[,] data, string type)
        {
            double[] dataPoint;
            double firstClusterDistance = 0.0;
            int rowCount = data.GetUpperBound(0) + 1;
            int fieldCount = data.GetUpperBound(1) + 1;
            int position = 0;
            var myDist = new similarity();

            // create a new collection of clusters
            var newClusters = new ClusterCollection();

            for (int count = 0; count < clusters.Count; count++)
            {
                var newCluster = new Cluster();
                newClusters.Add(newCluster);
            }

            if (clusters.Count <= 0)
            {
                throw new SystemException("Cluster Count Cannot Be Zero!");
            }

            for (int row = 0; row < rowCount; row++)
            {
                dataPoint = new double[fieldCount];

                for (int field = 0; field < fieldCount; field++)
                {
                    dataPoint.SetValue((data[row, field]), field);
                }

                for (int cluster = 0; cluster < clusters.Count; cluster++)
                {
                    double[] clusterMean = clusters[cluster].ClusterMean;

                    if (cluster == 0)
                    {
                        firstClusterDistance = myDist.FindDistance(dataPoint, clusterMean, type);
                        position = cluster;
                    }
                    else
                    {
                        double secondClusterDistance = myDist.FindDistance(dataPoint, clusterMean, type);

                        if (firstClusterDistance > secondClusterDistance)
                        {
                            firstClusterDistance = secondClusterDistance;
                            position = cluster;
                        }
                    }
                }

                newClusters[position].Add(dataPoint);
            }

            return newClusters;
        }

        /// 
        /// Converts the data table to array.
        /// 
        /// The table.
        /// 
        public static double[,] ConvertDataTableToArray(DataTable table)
        {
            int rowCount = table.Rows.Count;
            int fieldCount = table.Columns.Count;

            var dataPoints = new double[rowCount,fieldCount];

            for (int rowPosition = 0; rowPosition < rowCount; rowPosition++)
            {
                DataRow row = table.Rows[rowPosition];

                for (int fieldPosition = 0; fieldPosition < fieldCount; fieldPosition++)
                {
                    double fieldValue;
                    try
                    {
                        fieldValue = double.Parse(row[fieldPosition].ToString());
                    }
                    catch (Exception ex)
                    {
                        Debug.WriteLine(ex.ToString());
                        throw new InvalidCastException("Invalid row at " + rowPosition + " and field " + fieldPosition,
                                                       ex);
                    }

                    dataPoints[rowPosition, fieldPosition] = fieldValue;
                }
            }

            return dataPoints;
        }
    }

    /// 
    /// A class containing a group of data with similar characteristics (cluster)
    /// 
    [Serializable]
    public class Cluster : CollectionBase
    {
        private double[] _clusterMean;
        private double[] _clusterSum;

        /// 
        /// The sum of all the data in the cluster
        /// 
        public double[] ClusterSum
        {
            get { return _clusterSum; }
        }

        /// 
        /// The mean of all the data in the cluster
        /// 
        public double[] ClusterMean
        {
            get
            {
                for (int count = 0; count < this[0].Length; count++)
                {
                    _clusterMean[count] = (_clusterSum[count]/List.Count);
                }

                return _clusterMean;
            }
        }

        /// 
        /// Returns the one dimensional array data located at the index
        /// 
        public virtual double[] this[int index]
        {
            get
            {
                //return the Neuron at IList[index]
                return (double[]) List[index];
            }
        }

        /// 
        /// Adds a single dimension array data to the cluster
        /// 
        /// A 1-dimensional array containing data that will be added to the cluster
        public virtual void Add(double[] data)
        {
            List.Add(data);

            if (List.Count == 1)
            {
                _clusterSum = new double[data.Length];

                _clusterMean = new double[data.Length];
            }

            for (int count = 0; count < data.Length; count++)
            {
                _clusterSum[count] = _clusterSum[count] + data[count];
            }
        }
    }

    /// 
    /// A collection of Cluster objects or Clusters
    /// 
    [Serializable]
    public class ClusterCollection : CollectionBase
    {
        /// 
        /// Returns the Cluster at this index
        /// 
        public virtual Cluster this[int index]
        {
            get
            {
                //return the Neuron at IList[index]
                return (Cluster) List[index];
            }
        }

        /// 
        /// Adds a Cluster to the collection of Clusters
        /// 
        /// A Cluster to be added to the collection of clusters
        public virtual void Add(Cluster cluster)
        {
            List.Add(cluster);
        }
    }
}

Here is the similarity class than can calculate Euclidean, Manhattan, Chebyshev, Minkowski distances

/// Most of this code was written by Aresh Saharkhiz
/// Re-organized by me
/// See Code Project: http://www.codeproject.com/KB/recipes/Quantitative_Distances.aspx
using System;

namespace LSITest
{
    public class similarity
    {
        /// 
        /// Finds the distance.
        /// 
        /// The x.
        /// The y.
        /// The type.
        /// 
        /// 
        public double FindDistance(double[] x, double[] y, string distanceType)
        {
            double distance;

            switch (distanceType.ToLower())
            {
                case "euclidean":
                    distance = EuclideanDistance(x, y);
                    break;
                case "manhattan":
                    distance = ManhattanDistance(x, y);
                    break;
                case "minkowski":
                    distance = MinkowskiDistance(x, y, 1);
                    break;
                case "chebyshev":
                    distance = ChebyshevDistance(x, y);
                    break;
                default:
                    distance = 0.0;
                    break;
            }

            return distance;
        }

        /// 
        /// Finds the Euclideans distance.
        /// 
        /// The x.
        /// The y.
        /// 
        public double EuclideanDistance(double[] x, double[] y)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }

            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum += Math.Pow(Math.Abs(x[i] - y[i]), 2);
            }

            double distance = Math.Sqrt(sum);
            return distance;
        }

        /// 
        /// Finds Manhattan distance.
        /// 
        /// The x.
        /// The y.
        /// 
        public double ManhattanDistance(double[] x, double[] y)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }

            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum += Math.Abs(x[i] - y[i]);
            }

            double distance = sum;
            return distance;
        }

        /// 
        /// Finds Chebyshevs distance.
        /// 
        /// The x.
        /// The y.
        /// 
        public static double ChebyshevDistance(double[] x, double[] y)
        {
            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }
            int count = x.Length;
            var newData = new double[count];

            for (int i = 0; i < count; i++)
            {
                newData[i] = Math.Abs(x[i] - y[i]);
            }
            double max = double.MinValue;

            foreach (double num in newData)
            {
                if (num > max)
                {
                    max = num;
                }
            }
            return max;
        }

        /// 
        /// Finds Minkowskis distance.
        /// 
        /// The x.
        /// The y.
        /// The order.
        /// 
        public double MinkowskiDistance(double[] x, double[] y, double order)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }
            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum = sum + Math.Pow(Math.Abs(x[i] - y[i]), order);
            }

            double distance = Math.Pow(sum, (1 / order));
            return distance;
        }
    }
}

And finally the same LSI class used in the previous examples.

using System;
using SmartMathLibrary;

namespace LSITest
{
    public class lsi
    {
        // this returns the formated html results
        public int MyDocColumnCount;
        public int MyDocRowCount;
        public double[,] MyDocs;
        public double[,] MyWords;
        public int MyWordsColumnCount;
        public int MyWordsRowCount;
        public string ToPrint;

        /// 
        /// LISs the test.
        /// 
        public void LSITest()
        {
            //Create Matrix
            var testArray = new double[,]
                                {
                                    {1, 0, 0, 1, 0, 0, 0, 0, 0},
                                    {1, 0, 1, 0, 0, 0, 0, 0, 0},
                                    {1, 1, 0, 0, 0, 0, 0, 0, 0},
                                    {0, 1, 1, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 1, 2, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 0, 1, 1, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 0, 0, 0, 0, 1},
                                    {0, 0, 0, 0, 0, 1, 1, 1, 0},
                                    {0, 0, 0, 0, 0, 0, 1, 1, 1},
                                    {0, 0, 0, 0, 0, 0, 0, 1, 1}
                                };

            // Load array in to Matrix
            var a = new Matrix(testArray);

            // print original matrix
            PrintMatrix(a);

            // preform Latent Semantic Indexing
            GetDocumentWordPlots(a);
        }

        /// 
        /// Prints the matrix.
        /// 
        /// My matrix.
        private void PrintMatrix(IMatrix myMatrix)
        {
            ToPrint += "

";

            for (int i = 0; i < myMatrix.Rows; i++)
            {
                for (int j = 0; j < myMatrix.Columns; j++)
                {
                    ToPrint += String.Format("{0:0.##}", myMatrix.MatrixData[i, j]) + "\t";
                }
                ToPrint += "
";
            }
        }

        /// 
        /// Gets the document word plots.
        /// 
        /// My matrix.
        private void GetDocumentWordPlots(Matrix myMatrix)
        {
            // Run single value decomposition
            var svd = new SingularValueDecomposition(myMatrix);
            svd.ExecuteDecomposition();

            // Put components into individual matrices
            Matrix wordVector = svd.U.Copy();
            Matrix sigma = svd.S.ToMatrix();
            Matrix documentVector = svd.V.Copy();

            // get value of k
            // you can also manually set the value of k
            var k = (int) Math.Floor(Math.Sqrt(myMatrix.Columns));

            // reduce the vectors
            Matrix reducedWordVector = CopyMatrix(wordVector, wordVector.Rows, k - 1);
            Matrix reducedSigma = CreateSigmaMatrix(sigma, k - 1, k - 1);
            Matrix reducedDocumentVector = CopyMatrix(documentVector, documentVector.Rows, k - 1);

            // Recalculate the matrix
            Matrix docs = reducedDocumentVector*reducedSigma;
            Matrix words = reducedWordVector*reducedSigma;

            // Fill doc plot locations
            MyDocs = new double[docs.Rows,docs.Columns];
            for (int i = 0; i < docs.Rows; i++)
            {
                for (int j = 0; j < docs.Columns; j++)
                {
                    MyDocs[i, j] = docs.MatrixData[i, j];
                }
            }

            // Fill word plot locations
            MyWords = new double[words.Rows,words.Columns];
            for (int i = 0; i < words.Rows; i++)
            {
                for (int j = 0; j < words.Columns; j++)
                {
                    MyWords[i, j] = words.MatrixData[i, j];
                }
            }

            // Set counts for charts
            MyDocRowCount = docs.Rows;
            MyWordsRowCount = words.Rows;

            PrintMatrix(docs);
            PrintMatrix(words);
        }

        /// 
        /// Creates the sigma matrix.
        /// 
        /// The matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CreateSigmaMatrix(IMatrix matrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < columnEnd; i++)
            {
                copyMatrix.MatrixData[i, i] = matrix.MatrixData[i, 0];
            }

            return copyMatrix;
        }

        /// 
        /// Copies the matrix.
        /// 
        /// My matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CopyMatrix(IMatrix myMatrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < rowEnd; i++)
            {
                for (int j = 0; j < columnEnd; j++)
                {
                    copyMatrix.MatrixData[i, j] = myMatrix.MatrixData[i, j];
                }
            }

            return copyMatrix;
        }
    }
}

And what do the results look like?

As you can see the K-Means clustering algorithm correctly grouped the documents in the appropriate categories.

Recommended reading and thanks goes to Aresh Saharkhiz for sharing his implementation of K-Means Clustering.

Plotting Documents & Words: Using Latent Semantic Indexing

Eric — Fri, 06 Nov 2009 00:32:22 +0000

In the last blog post we looked over a couple of great papers talking about using Singular Value Decomposition (SVD) to do Latent Semantic Indexing (LSI) using the SmartMathLibrary. Now that we have the results we should plot them to get a sense of where these words and documents lay on a two dimensional Cartesian plane.

Jennifer Flynnâ€™s presentation “Latent Semantic Indexing Using SVD and Riemannian SVD” actually goes on to tell us how to do this. Essentially the process is the same as before however, k must equal 2. We ended up with k = 2 in our previous example however, in larger examples k will more than likely be a different number. Regardless, here we want to end up with a matrix with two columns giving us our (x,y) – if you wanted to plot these items in a three dimensional space k=3 and if you find an awesome way to plot where k=5 e-mail me.Â The formulas we use are as follows after we have performed SVD.

Documents = U*âˆ‘

Words = V*âˆ‘

The resulting matrices give us our (x,y) co-ordinates that we can then plot. I have been using the Dundas charting library for over two years now but the library is expensive so you should go and get the free library here since Microsoft acquired them and is a free download. And again for simplicities sake, this project is just a simple ASP.NET application.

The LSI Class:

Please note that this is almost exactly the same as in the previous blog however, here at the end of the GetDocumentWordPlots function we use the formulas mention above to load the co-ordinates of the words and documents in to a double array that we will ultimately pass to the chart.

using System;
using SmartMathLibrary;

namespace LSITest
{
    public class lsi
    {
        // this returns the formated html results
        public int MyDocColumnCount;
        public int MyDocRowCount;
        public double[,] MyDocs;
        public double[,] MyWords;
        public int MyWordsColumnCount;
        public int MyWordsRowCount;
        public string ToPrint;

        /// 
        /// LISs the test.
        /// 
        public void LSITest()
        {
            //Create Matrix
            var testArray = new double[,]
                                {
                                    {1, 0, 0, 1, 0, 0, 0, 0, 0},
                                    {1, 0, 1, 0, 0, 0, 0, 0, 0},
                                    {1, 1, 0, 0, 0, 0, 0, 0, 0},
                                    {0, 1, 1, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 1, 2, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 0, 1, 1, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 0, 0, 0, 0, 1},
                                    {0, 0, 0, 0, 0, 1, 1, 1, 0},
                                    {0, 0, 0, 0, 0, 0, 1, 1, 1},
                                    {0, 0, 0, 0, 0, 0, 0, 1, 1}
                                };

            // Load array in to Matrix
            var a = new Matrix(testArray);

            // print original matrix
            PrintMatrix(a);

            // preform Latent Semantic Indexing
            GetDocumentWordPlots(a);
        }

        /// 
        /// Prints the matrix.
        /// 
        /// My matrix.
        private void PrintMatrix(IMatrix myMatrix)
        {
            ToPrint += "

";

            for (int i = 0; i < myMatrix.Rows; i++)
            {
                for (int j = 0; j < myMatrix.Columns; j++)
                {
                    ToPrint += String.Format("{0:0.##}", myMatrix.MatrixData[i, j]) + "\t";
                }
                ToPrint += "
";
            }
        }

        /// 
        /// Gets the document word plots.
        /// 
        /// My matrix.
        private void GetDocumentWordPlots(Matrix myMatrix)
        {
            // Run single value decomposition
            var svd = new SingularValueDecomposition(myMatrix);
            svd.ExecuteDecomposition();

            // Put components into individual matrices
            Matrix wordVector = svd.U.Copy();
            Matrix sigma = svd.S.ToMatrix();
            Matrix documentVector = svd.V.Copy();

            // get value of k
            var k = 2;

            // reduce the vectors
            Matrix reducedWordVector = CopyMatrix(wordVector, wordVector.Rows, k - 1);
            Matrix reducedSigma = CreateSigmaMatrix(sigma, k - 1, k - 1);
            Matrix reducedDocumentVector = CopyMatrix(documentVector, documentVector.Rows, k - 1);

            // Recalculate the matrix
            Matrix docs = reducedDocumentVector*reducedSigma;
            Matrix words = reducedWordVector*reducedSigma;

            // Fill doc plot locations
            MyDocs = new double[docs.Rows,docs.Columns];
            for (int i = 0; i < docs.Rows; i++)
            {
                for (int j = 0; j < docs.Columns; j++)
                {
                    MyDocs[i, j] = docs.MatrixData[i, j];
                }
            }

            // Fill word plot locations
            MyWords = new double[words.Rows,words.Columns];
            for (int i = 0; i < words.Rows; i++)
            {
                for (int j = 0; j < words.Columns; j++)
                {
                    MyWords[i, j] = words.MatrixData[i, j];
                }
            }

            // Set counts for charts
            MyDocRowCount = docs.Rows;
            MyWordsRowCount = words.Rows;

            PrintMatrix(docs);
            PrintMatrix(words);
        }

        /// 
        /// Creates the sigma matrix.
        /// 
        /// The matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CreateSigmaMatrix(IMatrix matrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < columnEnd; i++)
            {
                copyMatrix.MatrixData[i, i] = matrix.MatrixData[i, 0];
            }

            return copyMatrix;
        }

        /// 
        /// Copies the matrix.
        /// 
        /// My matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CopyMatrix(IMatrix myMatrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < rowEnd; i++)
            {
                for (int j = 0; j < columnEnd; j++)
                {
                    copyMatrix.MatrixData[i, j] = myMatrix.MatrixData[i, j];
                }
            }

            return copyMatrix;
        }
    }
}

Here is the code behind for the web page that displays the chart and the matrices. Essentially we iterate through the double array pulled from the LSI class and load them in to a chart series.

using System;
using System.Drawing;
using System.Web.UI;
using Dundas.Charting.WebControl;

namespace LSITest
{
    public partial class _Default : Page
    {
        /// 
        /// Handles the Load event of the Page control.
        /// 
        /// The source of the event.
        /// The  instance containing the event data.
        protected void Page_Load(object sender, EventArgs e)
        {
            var mylsi = new lsi();
            mylsi.LSITest();
            Label1.Text = mylsi.ToPrint;
            double[,] myDocs = mylsi.MyDocs;
            double[,] myWords = mylsi.MyWords;

            // Load documents
            for (int i = 0; i < mylsi.MyDocRowCount; i++)
            {
                Chart1.Series["Series1"].Points.AddXY(myDocs[i, 0], myDocs[i, 1]);
            }

            // Load words
            for (int i = 0; i < mylsi.MyWordsRowCount; i++)
            {
                Chart1.Series["Series2"].Points.AddXY(myWords[i, 0], myWords[i, 1]);
            }

            // Set title
            Chart1.Series["Series1"].LegendText = "Documents";
            Chart1.Series["Series2"].LegendText = "Words";

            // Set point colors and shapes
            Chart1.Series["Series1"].Color = Color.Red;
            Chart1.Series["Series1"].MarkerStyle = MarkerStyle.Diamond;
            Chart1.Series["Series1"].MarkerSize = 12;
            Chart1.Series["Series2"].Color = Color.Gray;
            Chart1.Series["Series2"].MarkerStyle = MarkerStyle.Circle;
            Chart1.Series["Series2"].MarkerSize = 6;
        }
    }
}

And finally here is the ASP.NET web page.

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="LSITest._Default" %>
<%@ Register Assembly="DundasWebChart" Namespace="Dundas.Charting.WebControl" TagPrefix="DCWC" %>




    LSI Test

So lets see the result!

Now obviously you could/should probably write this a different way but it gets you to where you need to be.

I would also recommend you read in Flynn’s presentation on how to compare two words/documents by using the dot product of two row vectors. Or one could also use the Euclidean Distance Score. And if you are also interested I would recommend Sujit Pal’s blog post “IR Math in Java : Cluster Visualization” for additional reading.

Backtesting With Heat Maps

Eric — Mon, 25 May 2009 15:56:24 +0000

I have found that placing back test data in to heat maps and animating it over time is a great way to visualize the data. You get some cool results with some crazy patterns.

Map Resources

Eric — Tue, 10 Jun 2008 05:03:43 +0000

I have been a bit of research on a couple of map resources I thought I would put together.

Arc2Earth

Integrating ArcGIS and Google Earth for Crime Analysis

thematicmapping.org

Convert your GPS data for use in Google Earth

Add Your Own Custom Map

Automatic Tile Cutter

Some experiments with Google Maps

Google Maps API Tutorial

Ubigraph

Eric — Thu, 22 May 2008 14:20:18 +0000

UbiGraph is a tool for visualizing dynamic graphs. It works with Python, Ruby, PHP, Java, C, C++, C#, Haskell, and OCaml. For More info check it out here.

Timeline .NET

Eric — Thu, 27 Mar 2008 01:49:22 +0000

Rob Chartier just released an API wrapper for .NET of the MIT Simile Timeline API and this thing is pretty cool.

[read full article]

Flash Force Directed Graph

Eric — Wed, 22 Aug 2007 00:01:52 +0000

I came across this nice little flash force directed graph today. It was put together by Creative Synthesis.

The thing that makes this nice actually is (a) it works and (b) the xml file is very intuitive.

Doesn’t get much simpler.