eric.ness.net » Programming

Cryptanalysis Using n-Gram Probabilities

Eric — Sat, 01 May 2010 09:35:31 +0000

One of my favorite programmers is Peter Norvig who is currently Director of Research at Google. This summer I picked up a book called Beautiful Data in which Norvig contributed a chapter called “Natural Language Corpus Data” in which he outlined a number of very cool things you can do with n-grams in the google corpus. It covers some of the things you’d imagine that it would cover: spelling correction, word segmentation, etc. The one item covered that I had never considered was in the area of cryptanalysis.

The cool thing is that Google will give you their corpus to download. The only problem is that it contains “1,024,908,267,229 words of running text” and is 24 GB compressed in size. This is a bit impractical to run on your dev box. Enter Microsoft – the Microsoft Web N-gram Service just went Beta and is now available to Professors and Students so I immediately signed up and I have to say that it pretty cool!

So I wanted to try out the new service using one of Norvig’s examples in his book – specifically using n-gram probabilities and character shifting. This is a very simple example and fairly basic type of encryption where the if the user types an ‘a’ it gets shifted to ‘n’ or whatever. So you simply run through all 26 possibilities and use the individual words combined probabilities to determine the answer to the encoded message.

This project has a Service Refrence connected to Microsoft’s n-Gram Service. The service requires an n-gram model and a user id which you get when you sign up (see their quickstart tutorial). So let’s take a look at some code:

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using MicrosoftNGramTest.NGramService;

namespace MicrosoftNGramTest.classes
{
    internal class Shift
    {
        #region Variables

        private readonly string _alphabet = "abcdefghijklmnopqrstuvwxyz";
        private readonly string _ngramModel = ConfigurationManager.AppSettings.Get("ngramModel");
        private readonly string _userToken = ConfigurationManager.AppSettings.Get("userToken");

        #endregion

        #region Run The Test

        /// 
        /// Runs the test
        /// 
        public void Test()
        {
            // Print title
            Console.WriteLine("Character Shift Cryptanalysis");
            Console.WriteLine("#############################");

            // Local Variables
            const string phrase = "Yvfgra, qb lbh jnag gb xabj n frperg?";
            string[] words = phrase.ToLower().Split(' ');
            var newPhrase = new string[26];
            var client = new LookupServiceClient();
            var result = new Dictionary();

            try
            {
                // Loop the word variations
                foreach (string s in words)
                {
                    char[] currentWord = s.ToCharArray();

                    foreach (char c in currentWord)
                    {
                        for (int i = 0; i < 26; i++)
                        {
                            newPhrase[i] += CharShift(c, i);
                        }
                    }

                    for (int i = 0; i < newPhrase.Count(); i++)
                    {
                        newPhrase[i] += " ";
                    }
                }

                // Print phrases with probabilities
                foreach (string s in newPhrase)
                {
                    string[] newWords = s.Split(' ');
                    double prob = 0;
                    foreach (string word in newWords)
                    {
                        prob += client.GetProbability(_userToken, _ngramModel, word);
                    }
                    Console.WriteLine(s + " " + Convert.ToInt32(prob));
                    result.Add(s, Convert.ToInt32(prob));
                }

                // Print answer
                Console.WriteLine();
                Console.WriteLine("The answer is:");
                KeyValuePair q = (from t in result
                                               orderby t.Value descending
                                               select t).FirstOrDefault();
                Console.WriteLine(q.Key + " " + q.Value);
            }
            finally
            {
                client.Close();
            }
        }

        #endregion

        #region Shifting

        /// 
        /// Gets the alphabet array.
        /// 
        /// 
        private char[] GetAlphabetArray()
        {
            return _alphabet.ToCharArray();
        }

        /// 
        /// Gets the current char array position.
        /// 
        /// The c.
        /// 
        private int GetCurrentCharArrayPosition(char c)
        {
            int position = 0;
            int count = 0;

            foreach (char letter in GetAlphabetArray())
            {
                if (letter == c)
                {
                    position = count;
                }
                count++;
            }
            return position;
        }

        /// 
        /// Shifts the character.
        /// 
        /// The c.
        /// The increase.
        /// 
        private char CharShift(char c, int increase)
        {
            const int numOfLetters = 26;
            char[] alphabet = GetAlphabetArray();
            int currentArrayPosition = GetCurrentCharArrayPosition(c);
            char letter = c;

            if (IsCharInArray(c))
            {
                if ((currentArrayPosition + increase) < numOfLetters)
                {
                    letter = alphabet[currentArrayPosition + increase];
                }
                else
                {
                    int newPosition = (currentArrayPosition + increase) - numOfLetters;
                    letter = alphabet[newPosition];
                }
            }
            return letter;
        }

        /// 
        /// Determines whether the char is in the array.
        /// 
        /// The c.
        /// 
        /// 	true if [is char in array] [the specified c]; otherwise, false.
        /// 
        private bool IsCharInArray(char c)
        {
            bool isCharInArray = false;
            IEnumerable q = (from t in GetAlphabetArray()
                                   where t == c
                                   select t);
            if (q.Count() > 0)
            {
                isCharInArray = true;
            }
            return isCharInArray;
        }

        #endregion
    }
}

And here is the result!

Apriori Algorithm

Eric — Tue, 02 Mar 2010 00:43:31 +0000

I’ve been meaning to get in to the Data Mining SDK at code plex for a while as it has a couple of good items in it. The one item I was really interested in was the apriori algorithm.

Wikipedia describes Apriori:

In computer science and data mining, Apriori is a classic algorithm for learning association rules. Apriori is designed to operate on databases containing transactions (for example, collections of items bought by customers, or details of a website frequentation). Other algorithms are designed for finding association rules in data having no transactions (Winepi and Minepi), or having no timestamps (DNA sequencing).

The classic example is if you own a store and someone buys milk what is the probability that he will also buy bread and eggs or if voters in one state voted for one issue what is the chance he voted for something else. The applications for this approach are pretty much limitless.

The code in the SDK is pretty good with a couple of exceptions: there is little documentation and it only supports XML files and OleDb data connections. I have reworked it so it will also connect to a MSSQL database.

For this test application I created a simple C# Console Application and imported the “APriori” project in to the solution. In the APriori project you will to add these two bits of code to classes to the APriori project:

Add this method to DataAccessLayer.cs

	public Data GetTransactionsData(string rdbmsConnectionString, string dataSource)
        {
            myDatabase = new Data();
            string query = "SELECT * FROM " + dataSource;
            var myConn = new SqlConnection(rdbmsConnectionString);
            var myDBAdapter = new SqlDataAdapter(query, myConn);

            myConn.Open();
            try
            {
                myDBAdapter.Fill(myDatabase, "TransactionTable");
            }
            finally
            {
                myConn.Close();
            }
            return myDatabase;
        }

Add this method to DataMining.cs

public Data MarketBasedAnalysis(double supportCount, double minimumConfidence, string connectionString, string dataSource)
        {

            Database database = new Database();
            ItemsetCandidate Item = new ItemsetCandidate();

            this.AP = new APriori.Apriori();
            this.AP.ProgressMonitorEvent += new ProgressMonitorEventHandler(this.OnProgressMonitoringCompletedEvent);
            this.dataBase = database.GetTransactionsData(connectionString, dataSource);
            database.Transactions = this.dataBase;
            this.transactionsCount = this.dataBase.TransactionTable.Count;

            supportCount = ((supportCount / 100) * this.transactionsCount);

            minimumConfidence = (minimumConfidence / 100);

            string support = "SupportCount >= " + supportCount + " AND Level > 1";

            string sort = "SupportCount, Level";
            ItemsetCandidate uniqueItems = AP.CreateOneItemsets(database);
            AP.AprioriGenerator(uniqueItems, database, Convert.ToInt32(supportCount));
            ItemsetArrayList[] keys = database.GetItemset(support, sort);
            string msg = "Creating Frequent Subsets for Items";
            ProgressMonitorEventArgs e = new ProgressMonitorEventArgs(1, 100, 95, "DataMining.MarketBasedAnalysis(3)", msg);
            this.OnProgressMonitorEvent(e);

            for (int counter = 0; counter < keys.Length; counter++)
            {
                AP.CreateItemsetSubsets(0, keys[counter], null, database);
            }

            msg = "Completed C#.NET Data Mining Market Based Analysis";
            e = new ProgressMonitorEventArgs(1, 100, 100, "DataMining.MarketBasedAnalysis(3)", msg);
            this.OnProgressMonitorEvent(e);

            //Set the public properties of the class
            this.minimumSupportCount = supportCount;
            this.minimumConfidence = minimumConfidence;
            this.connectionString = connectionString;
            this.dataSource = dataSource;
            this.dataSourceCommand = dataSourceCommand;

            //return the database of transactions
            return this.dataBase;

        }

Here is my class in my console application

using System;
using System.Data;
using VISUAL_BASIC_DATA_MINING_NET;
using VISUAL_BASIC_DATA_MINING_NET.CustomEvents;

namespace APr2.classes
{
    internal class testrun
    {
        private Data _dataAnalysis;
        public event ProgressMonitorEventHandler ProgressMonitorEvent;

        /// 
        /// Runs the Apriori.
        /// 
        public void RunApriori()
        {
            // Create Data Mining Object
            var myDM = new DataMining();

            // Register Event
            myDM.ProgressMonitorEvent += OnProgressMonitorEvent;

            // Connect To Data Base & Process Items
            _dataAnalysis = myDM.MarketBasedAnalysis(2,             // Support Count
                                                     2,             // Minimum Confidence
                                                     @"Data Source=(local);Initial Catalog=Apriori;Integrated Security=True;", // Connection String
                                                     "Example");    // Table in db

            // Copy to Data View
            var dataView = new ViewData();
            _dataAnalysis.Tables.Add(dataView.CreateViewRulesTable(2, _dataAnalysis).Copy());
            _dataAnalysis.Tables.Add(dataView.CreateViewSubsetTable(_dataAnalysis).Copy());

            // Spacer Line
            Console.WriteLine();

            // Print Items
            foreach (DataRow row in dataView.ViewDataSet.Tables[1].Rows)
            {
                double per = Convert.ToDouble(row.ItemArray[2].ToString().Substring(0, (row.ItemArray[2].ToString().Length -1)));
                Console.WriteLine(row.ItemArray[0] + "\t" + row.ItemArray[1] + "\t" + String.Format("{0:###.##%}", (per/100)));
            }
        }

        /// 
        /// Called when [progress monitor event].
        /// 
        /// The sender.
        /// The  instance containing the event data.
        public void OnProgressMonitorEvent(object sender, ProgressMonitorEventArgs e)
        {
            // Prints Event Messages
            Console.Write("\r" + e.EventMessage);
        }
    }
}

Your MSSQL Code will be this

GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[Example](
	[TransactionID] [int] IDENTITY(1,1) NOT NULL,
	[Transactions] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
 CONSTRAINT [PK_Example] PRIMARY KEY CLUSTERED
(
	[TransactionID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

And these records:

1	Books, CD, Video
2	CD, Games
3	CD, DVD
4	Books, CD, Games
5	Books, DVD
6	CD, DVD
7	Books, DVD
8	Books, CD, DVD, Video
9	Books, CD, DVD
10	Books, Games
11	Games, Lasers

Run the RunApriori() method in my class and it will yield you the correct results. Have fun.

Benford’s Law and Trailing Digit Tests

Eric — Wed, 11 Nov 2009 14:23:50 +0000

As of late I’ve been coming across Benford’s Law all over the place and so I thought I would revisit the topic. Beford’s Law essentially states “that in lists of numbers from many (but not all) real-life sources of data, the leading digit is distributed in a specific, non-uniform way”. More specifically you should expect the leading digit ’1′ to appear about 30.1% fo the time, ’2′ about 17.6% and so on [see table]. One classic example for the use of Benford’s Law is in fraud detection.

One of the reasons I wanted to revisit this topic is that after having a conversation with a friend about this very topic, I came across a series of articles written by Nate Silver regarding the polling firm Strategic Vision. Strategic Vision released a poll focusing on Oklahoma students and it motivated Silver to ask a series of questions about the firm.

Now I don’t want to really focus in on all the Strategic Vision stuff but, I do want to talk about a method Silver used to detect anomalies in their polling. As Silver correctly suggests, polling would not be a good candidate to using Benford’s Law (i.e. take the last Presidential race where for the most part the two candidates were going back and forth in the 40~50% range – Benford’s wouldn’t work). However, there is another method that might give you a little insight as Silver explains:

For each question, I recorded the trailing digit for each candidate or line item. For instance, if Strategic Vision had Barack Obama beating John McCain 48-43 in a particular state, I’d record a tally in the 8 column and another in the 3 column. Or if they had voters opposing a particular policy 50-45, I’d record a tally in the 0 column (for 50) and another in the 5 column (for 45).

And what Silver says essentially is, that if you look at the last digit you should have roughly a uniform distribution. Put it another way, that if I have roughly 200 4′s I would expect roughly 200 8′s too. Silver also says that when using the trailing digit method that in some cases, you might find deviations from this distribution might be due to rounding error’s or a specific mathematical method. The trailing digit test is clearly not a sure fire way to detect fraud or anything, but just another useful tool to see if the data passes the smell test.

Silver’s insight prompted me to write some code and play around with these two methods just to see what comes up. Because I am fairly familiar with the the United Nation’s World Health Database, I thought I would run some tests using these methods. Here are the results:

Gross Domestic Product (GDP)

As you can see, GDP generally follows Benford’s Law and the trailing digit test. Something to note on the trailing digit results – the average for each number (bin) is 23.7 so the number of 1′s and the number of 5′s are roughly equidistant from the average, even though it seems a little odd finding a GDP with a value ending in 1 is almost twice as likely as finding a value ending in 5.

Life expectancy at birth (years)

Now here is an example where Benford’s Law will not work. The reason is because the range of life spans for all countries is from 40-83 years of age, so we are going to have to focus in on the trailing digit test.

You will notice that there is something weird here with the trailing digits. First, the average value is 19.3 and we have 0′s and 2′s appearing almost 3 times more than 7′s. One thing that might account for this disparity is the fact that we have a little over 20 countries that fall in the range of 80-83, which would tilt the values of 0,1,2,3 a little higher than normal. And the number of countries in the 40′s is also a little sparse. So what I did was remove these from the set and re-ran the test. Here are the following results.

You will notice that after cleaning up the data a 3 is still almost 3 times as likely to appear as a 7 in what should be some fairly naturally occurring numbers. The average for this set is 14.8.

Now I think it would be inappropriate for me to draw any hard conclusions about the Life Expectancy data other than to say that some form of rounding has likely occurred due to the fact that the data are all whole numbers.

That said though, you can see how these two simple/practical tests can assist in determining whether there has been some human manipulation of the data. Here is the code that I used:

using System;
using System.Linq;

namespace BenfordsLaw
{
    /// 
    /// Benfords Law Class
    /// 
    public class Benfords
    {
        /// 
        /// Adds the data.
        /// 
        /// The data.
        /// 
        public double[] CalculateBenfordsDistribution(double[] data)
        {
            if (data.Count() == 0)
            {
                throw new ArgumentException("Error: There are no items in your data array.");
            }

            //Create benford bins to hold counts
            var benfordsContainer = new double[9];

            // Loop through array
            foreach (double number in data)
            {
                // Get absolute value of number
                double currentNumber = Math.Abs(number);

                // for items smaller than 1 * multiply it so you
                // can find the first number
                if ((currentNumber < 1) && (currentNumber > 0))
                {
                    double num = (currentNumber*10000);
                    while (num >= 10)
                        num /= 10;
                    PackageNumberInBenfordBin(benfordsContainer, num);
                }
                else
                {
                    double num = currentNumber;
                    while (num >= 10)
                        num /= 10;
                    PackageNumberInBenfordBin(benfordsContainer, num);
                }
            }

            return benfordsContainer;
        }

        /// 
        /// Trailings the digit check.
        /// 
        /// The data.
        /// 
        public double[] TrailingDigitCheck(double[] data)
        {
            if (data.Count() == 0)
            {
                throw new ArgumentException("Error: There are no items in your data array.");
            }

            //Create benford bins to hold counts
            var trailingContainer = new double[10];

            // Loop through array
            foreach (double number in data)
            {
                // Get absolute value of number
                double currentNumber = Math.Abs(number);
                string numTemp = currentNumber.ToString();
                string numTemp2 = numTemp.Substring(numTemp.Length - 1);
                PackageNumberInTrailingBin(trailingContainer, Convert.ToDouble(numTemp2));
            }

            return trailingContainer;
        }

        /// 
        /// Packages the number in benford bin.
        /// 
        /// My container.
        /// The num.
        private static void PackageNumberInBenfordBin(double[] myContainer, double num)
        {
            // Update container totals
            int myNum = Convert.ToInt32(Math.Floor(num));
            myContainer[myNum-1] += 1;
        }

        /// 
        /// Packages the number in trailing bin.
        /// 
        /// My container.
        /// The num.
        private static void PackageNumberInTrailingBin(double[] myContainer, double num)
        {
            // Update container totals
            int myNum = Convert.ToInt32(Math.Floor(num));
            myContainer[myNum] += 1;
        }
    }
}

K-Means Document Clustering

Eric — Fri, 06 Nov 2009 17:35:48 +0000

Using our previous example as a basis to move to the next step let’s take a look at clustering using the K-Means clustering algorithm to group the documents in to their appropriate categories.

In the paper â€œIndexing by Latent Semantic Analysisâ€ (Deerwester et al.) they have an example of 9 titles of different papers grouped in to two categories â€œhuman computer interactionâ€ & â€œgraphs & treesâ€. So far, we’ve used Singular Value Decomposition (SVD) and Latent Semantic Indexing (LSI) to better understand the relationship of words and documents. In the last blog post we then took the results in LSI to plot words and documents on a two dimensional Cartesian plane.

All of this is pretty interesting stuff in and of itself however, the next step really is to see which documents belong in each group. One way to do this is by using K-Means clustering.

Simply speaking k-means clustering is an algorithm to classify or to group your objects based on attributes/features into K number of group. K is positive integer number. The grouping is done by minimizing the sum of squares of distances between data and the corresponding cluster centroid. Thus the purpose of K-mean clustering is to classify the data. [Kardi Teknomo]

A big chunk of the code is built off of the same project we are working on. I am using Aresh Saharkhiz K-Means implementation in the project with some minor changes/refactoring done by me.

Let take a look at the code!

This first part is the display (an ASP.NET app.)

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="LSITest._Default" %>
<%@ Register Assembly="DundasWebChart" Namespace="Dundas.Charting.WebControl" TagPrefix="DCWC" %>




    LSI Test

This is the code behind for the ASP.NET page. Because we are only dealing with two known categories K-Means is plotting out the two categories and if you wanted to do more you would definitely have to re-write the ColorCodeDocuments function.

using System;
using System.Data;
using System.Drawing;
using System.Web.UI;
using Dundas.Charting.WebControl;

namespace LSITest
{
    public partial class _Default : Page
    {
        /// 
        /// Handles the Load event of the Page control.
        /// 
        /// The source of the event.
        /// The  instance containing the event data.
        protected void Page_Load(object sender, EventArgs e)
        {
            // Perform LSI
            var mylsi = new lsi();
            mylsi.LSITest();
            double[,] myDocs = mylsi.MyDocs;

            // Plot Documents and the k-means
            const string distanceType = "manhattan";
            PlotDocuments(myDocs, mylsi.MyDocRowCount);
            PlotKMeansPoints(myDocs, 2, distanceType);
            ColorCodeDocuments(distanceType);

            // If you want to plot the words just un-comment the next two lines
            //double[,] myWords = mylsi.MyWords;
            //PlotWords(myDocs, mylsi.MyWordsRowCount);

            // comment this line out to show words in legend
            Chart1.Series["Series2"].ShowInLegend = false;
        }

        /// 
        /// Plots the words.
        /// 
        /// My words.
        /// My words row count.
        private void PlotWords(double[,] myWords, int myWordsRowCount)
        {
            for (int i = 0; i < myWordsRowCount; i++)
            {
                Chart1.Series["Series2"].Points.AddXY(myWords[i, 0], myWords[i, 1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series2"].LegendText = "Words";
            Chart1.Series["Series2"].Color = Color.Gray;
            Chart1.Series["Series2"].MarkerStyle = MarkerStyle.Circle;
            Chart1.Series["Series2"].MarkerSize = 6;
        }

        /// 
        /// Plots the documents.
        /// 
        /// My docs.
        /// My doc row count.
        private void PlotDocuments(double[,] myDocs, int myDocRowCount)
        {
            // Load documents
            for (int i = 0; i < myDocRowCount; i++)
            {
                Chart1.Series["Series1"].Points.AddXY(myDocs[i, 0], myDocs[i, 1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series1"].LegendText = "Documents";
            Chart1.Series["Series1"].Color = Color.Red;
            Chart1.Series["Series1"].MarkerStyle = MarkerStyle.Diamond;
            Chart1.Series["Series1"].MarkerSize = 12;
        }

        /// 
        /// Plots the K means points.
        /// 
        /// The items.
        /// The k.
        /// 
        private void PlotKMeansPoints(double[,] items, int k, string distanceType)
        {
            ClusterCollection clusters = kmeans.ClusterDataSet(k, items, distanceType);

            for (int i = 0; i < clusters.Count; i++)
            {
                Chart1.Series["Series3"].Points.AddXY(clusters[i].ClusterMean[0], clusters[i].ClusterMean[1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series3"].LegendText = "Cluster";
            Chart1.Series["Series3"].Color = Color.Gold;
            Chart1.Series["Series3"].MarkerStyle = MarkerStyle.Star6;
            Chart1.Series["Series3"].MarkerSize = 18;
        }

        /// 
        /// Colors the code documents.
        /// 
        /// Type of the distance.
        private void ColorCodeDocuments(string distanceType)
        {
            var myDist = new similarity();

            // Extract data
            DataSet myDocs = Chart1.DataManipulator.ExportSeriesValues("Series1");
            DataSet myKMeansPoints = Chart1.DataManipulator.ExportSeriesValues("Series3");

            // Document counter
            int count = 0;

            // Get co-ordinates for k-means points
            double firstKMeansX = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[0]["X"]);
            double firstKMeansY = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[0]["Y"]);
            double secondKMeansX = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[1]["X"]);
            double secondKMeansY = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[1]["Y"]);

            foreach (DataRow docRow in myDocs.Tables[0].Rows)
            {
                // get co-ordinates for current doc
                double currentDocX = Convert.ToDouble(docRow["X"]);
                double currentDocY = Convert.ToDouble(docRow["Y"]);

                // load in to arrays
                double[] firstX = {currentDocX, currentDocY};
                double[] firstY = {firstKMeansX, firstKMeansY};
                double[] secondX = {currentDocX, currentDocY};
                double[] secondY = {secondKMeansX, secondKMeansY};

                // find the distance
                double firstDist = myDist.FindDistance(firstX, firstY, distanceType);
                double secondDist = myDist.FindDistance(secondX, secondY, distanceType);

                // Color accordingly
                Chart1.Series["Series1"].Points[count].Color = firstDist < secondDist ? Color.Blue : Color.Gray;
                count++;
            }
        }
    }
}

This is the K-Means class written by Aresh Saharkhiz with my changes

/// Most of this code was written by Aresh Saharkhiz
/// Re-organized by me
/// See Code Project: http://www.codeproject.com/KB/recipes/K-Mean_Clustering.aspx
using System;
using System.Collections;
using System.Data;
using System.Diagnostics;

namespace LSITest
{
    public class kmeans
    {
        /// 
        /// Calculates The Mean Of A Cluster OR The Cluster Center
        /// 
        /// 
        /// A two-dimensional array containing a dataset of numeric values
        /// 
        /// 
        /// Returns an Array Defining A Data Point Representing The Cluster Mean or Centroid
        /// 
        public static double[] ClusterMean(double[,] cluster)
        {
            int rowCount = cluster.GetUpperBound(0) + 1;
            int fieldCount = cluster.GetUpperBound(1) + 1;
            var dataSum = new double[1,fieldCount];
            var centroid = new double[fieldCount];

            for (int j = 0; j < fieldCount; j++)
            {
                for (int i = 0; i < rowCount; i++)
                {
                    dataSum[0, j] = dataSum[0, j] + cluster[i, j];
                }

                centroid[j] = (dataSum[0, j]/rowCount);
            }

            return centroid;
        }

        /// 
        /// Seperates a dataset into clusters or groups with similar characteristics
        /// 
        /// The number of clusters or groups to form
        /// An array containing data that will be clustered
        /// 
        /// A collection of clusters of data
        public static ClusterCollection ClusterDataSet(int clusterCount, double[,] data, string type)
        {
            int rowCount = data.GetUpperBound(0) + 1;
            int fieldCount = data.GetUpperBound(1) + 1;
            int stableClustersCount = 0;
            double[] dataPoint;
            var random = new Random();
            Cluster cluster;
            var clusters = new ClusterCollection();
            var clusterNumbers = new ArrayList(clusterCount);
            var myDist = new similarity();

            while (clusterNumbers.Count < clusterCount)
            {
                int clusterNumber = random.Next(0, rowCount - 1);

                if (!clusterNumbers.Contains(clusterNumber))
                {
                    cluster = new Cluster();
                    clusterNumbers.Add(clusterNumber);
                    dataPoint = new double[fieldCount];

                    for (int field = 0; field < fieldCount; field++)
                    {
                        dataPoint.SetValue((data[clusterNumber, field]), field);
                    }

                    cluster.Add(dataPoint);
                    clusters.Add(cluster);
                }
            }

            while (stableClustersCount != clusters.Count)
            {
                stableClustersCount = 0;
                ClusterCollection newClusters = ClusterDataSet(clusters, data, type);

                for (int clusterIndex = 0; clusterIndex < clusters.Count; clusterIndex++)
                {
                    if ((myDist.FindDistance(newClusters[clusterIndex].ClusterMean, clusters[clusterIndex].ClusterMean, type)) == 0)
                    {
                        stableClustersCount++;
                    }
                }

                clusters = newClusters;
            }

            return clusters;
        }

        /// 
        /// Seperates a dataset into clusters or groups with similar characteristics
        /// 
        /// A collection of data clusters
        /// An array containing data to b eclustered
        /// 
        /// A collection of clusters of data
        public static ClusterCollection ClusterDataSet(ClusterCollection clusters, double[,] data, string type)
        {
            double[] dataPoint;
            double firstClusterDistance = 0.0;
            int rowCount = data.GetUpperBound(0) + 1;
            int fieldCount = data.GetUpperBound(1) + 1;
            int position = 0;
            var myDist = new similarity();

            // create a new collection of clusters
            var newClusters = new ClusterCollection();

            for (int count = 0; count < clusters.Count; count++)
            {
                var newCluster = new Cluster();
                newClusters.Add(newCluster);
            }

            if (clusters.Count <= 0)
            {
                throw new SystemException("Cluster Count Cannot Be Zero!");
            }

            for (int row = 0; row < rowCount; row++)
            {
                dataPoint = new double[fieldCount];

                for (int field = 0; field < fieldCount; field++)
                {
                    dataPoint.SetValue((data[row, field]), field);
                }

                for (int cluster = 0; cluster < clusters.Count; cluster++)
                {
                    double[] clusterMean = clusters[cluster].ClusterMean;

                    if (cluster == 0)
                    {
                        firstClusterDistance = myDist.FindDistance(dataPoint, clusterMean, type);
                        position = cluster;
                    }
                    else
                    {
                        double secondClusterDistance = myDist.FindDistance(dataPoint, clusterMean, type);

                        if (firstClusterDistance > secondClusterDistance)
                        {
                            firstClusterDistance = secondClusterDistance;
                            position = cluster;
                        }
                    }
                }

                newClusters[position].Add(dataPoint);
            }

            return newClusters;
        }

        /// 
        /// Converts the data table to array.
        /// 
        /// The table.
        /// 
        public static double[,] ConvertDataTableToArray(DataTable table)
        {
            int rowCount = table.Rows.Count;
            int fieldCount = table.Columns.Count;

            var dataPoints = new double[rowCount,fieldCount];

            for (int rowPosition = 0; rowPosition < rowCount; rowPosition++)
            {
                DataRow row = table.Rows[rowPosition];

                for (int fieldPosition = 0; fieldPosition < fieldCount; fieldPosition++)
                {
                    double fieldValue;
                    try
                    {
                        fieldValue = double.Parse(row[fieldPosition].ToString());
                    }
                    catch (Exception ex)
                    {
                        Debug.WriteLine(ex.ToString());
                        throw new InvalidCastException("Invalid row at " + rowPosition + " and field " + fieldPosition,
                                                       ex);
                    }

                    dataPoints[rowPosition, fieldPosition] = fieldValue;
                }
            }

            return dataPoints;
        }
    }

    /// 
    /// A class containing a group of data with similar characteristics (cluster)
    /// 
    [Serializable]
    public class Cluster : CollectionBase
    {
        private double[] _clusterMean;
        private double[] _clusterSum;

        /// 
        /// The sum of all the data in the cluster
        /// 
        public double[] ClusterSum
        {
            get { return _clusterSum; }
        }

        /// 
        /// The mean of all the data in the cluster
        /// 
        public double[] ClusterMean
        {
            get
            {
                for (int count = 0; count < this[0].Length; count++)
                {
                    _clusterMean[count] = (_clusterSum[count]/List.Count);
                }

                return _clusterMean;
            }
        }

        /// 
        /// Returns the one dimensional array data located at the index
        /// 
        public virtual double[] this[int index]
        {
            get
            {
                //return the Neuron at IList[index]
                return (double[]) List[index];
            }
        }

        /// 
        /// Adds a single dimension array data to the cluster
        /// 
        /// A 1-dimensional array containing data that will be added to the cluster
        public virtual void Add(double[] data)
        {
            List.Add(data);

            if (List.Count == 1)
            {
                _clusterSum = new double[data.Length];

                _clusterMean = new double[data.Length];
            }

            for (int count = 0; count < data.Length; count++)
            {
                _clusterSum[count] = _clusterSum[count] + data[count];
            }
        }
    }

    /// 
    /// A collection of Cluster objects or Clusters
    /// 
    [Serializable]
    public class ClusterCollection : CollectionBase
    {
        /// 
        /// Returns the Cluster at this index
        /// 
        public virtual Cluster this[int index]
        {
            get
            {
                //return the Neuron at IList[index]
                return (Cluster) List[index];
            }
        }

        /// 
        /// Adds a Cluster to the collection of Clusters
        /// 
        /// A Cluster to be added to the collection of clusters
        public virtual void Add(Cluster cluster)
        {
            List.Add(cluster);
        }
    }
}

Here is the similarity class than can calculate Euclidean, Manhattan, Chebyshev, Minkowski distances

/// Most of this code was written by Aresh Saharkhiz
/// Re-organized by me
/// See Code Project: http://www.codeproject.com/KB/recipes/Quantitative_Distances.aspx
using System;

namespace LSITest
{
    public class similarity
    {
        /// 
        /// Finds the distance.
        /// 
        /// The x.
        /// The y.
        /// The type.
        /// 
        /// 
        public double FindDistance(double[] x, double[] y, string distanceType)
        {
            double distance;

            switch (distanceType.ToLower())
            {
                case "euclidean":
                    distance = EuclideanDistance(x, y);
                    break;
                case "manhattan":
                    distance = ManhattanDistance(x, y);
                    break;
                case "minkowski":
                    distance = MinkowskiDistance(x, y, 1);
                    break;
                case "chebyshev":
                    distance = ChebyshevDistance(x, y);
                    break;
                default:
                    distance = 0.0;
                    break;
            }

            return distance;
        }

        /// 
        /// Finds the Euclideans distance.
        /// 
        /// The x.
        /// The y.
        /// 
        public double EuclideanDistance(double[] x, double[] y)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }

            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum += Math.Pow(Math.Abs(x[i] - y[i]), 2);
            }

            double distance = Math.Sqrt(sum);
            return distance;
        }

        /// 
        /// Finds Manhattan distance.
        /// 
        /// The x.
        /// The y.
        /// 
        public double ManhattanDistance(double[] x, double[] y)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }

            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum += Math.Abs(x[i] - y[i]);
            }

            double distance = sum;
            return distance;
        }

        /// 
        /// Finds Chebyshevs distance.
        /// 
        /// The x.
        /// The y.
        /// 
        public static double ChebyshevDistance(double[] x, double[] y)
        {
            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }
            int count = x.Length;
            var newData = new double[count];

            for (int i = 0; i < count; i++)
            {
                newData[i] = Math.Abs(x[i] - y[i]);
            }
            double max = double.MinValue;

            foreach (double num in newData)
            {
                if (num > max)
                {
                    max = num;
                }
            }
            return max;
        }

        /// 
        /// Finds Minkowskis distance.
        /// 
        /// The x.
        /// The y.
        /// The order.
        /// 
        public double MinkowskiDistance(double[] x, double[] y, double order)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }
            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum = sum + Math.Pow(Math.Abs(x[i] - y[i]), order);
            }

            double distance = Math.Pow(sum, (1 / order));
            return distance;
        }
    }
}

And finally the same LSI class used in the previous examples.

using System;
using SmartMathLibrary;

namespace LSITest
{
    public class lsi
    {
        // this returns the formated html results
        public int MyDocColumnCount;
        public int MyDocRowCount;
        public double[,] MyDocs;
        public double[,] MyWords;
        public int MyWordsColumnCount;
        public int MyWordsRowCount;
        public string ToPrint;

        /// 
        /// LISs the test.
        /// 
        public void LSITest()
        {
            //Create Matrix
            var testArray = new double[,]
                                {
                                    {1, 0, 0, 1, 0, 0, 0, 0, 0},
                                    {1, 0, 1, 0, 0, 0, 0, 0, 0},
                                    {1, 1, 0, 0, 0, 0, 0, 0, 0},
                                    {0, 1, 1, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 1, 2, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 0, 1, 1, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 0, 0, 0, 0, 1},
                                    {0, 0, 0, 0, 0, 1, 1, 1, 0},
                                    {0, 0, 0, 0, 0, 0, 1, 1, 1},
                                    {0, 0, 0, 0, 0, 0, 0, 1, 1}
                                };

            // Load array in to Matrix
            var a = new Matrix(testArray);

            // print original matrix
            PrintMatrix(a);

            // preform Latent Semantic Indexing
            GetDocumentWordPlots(a);
        }

        /// 
        /// Prints the matrix.
        /// 
        /// My matrix.
        private void PrintMatrix(IMatrix myMatrix)
        {
            ToPrint += "

";

            for (int i = 0; i < myMatrix.Rows; i++)
            {
                for (int j = 0; j < myMatrix.Columns; j++)
                {
                    ToPrint += String.Format("{0:0.##}", myMatrix.MatrixData[i, j]) + "\t";
                }
                ToPrint += "
";
            }
        }

        /// 
        /// Gets the document word plots.
        /// 
        /// My matrix.
        private void GetDocumentWordPlots(Matrix myMatrix)
        {
            // Run single value decomposition
            var svd = new SingularValueDecomposition(myMatrix);
            svd.ExecuteDecomposition();

            // Put components into individual matrices
            Matrix wordVector = svd.U.Copy();
            Matrix sigma = svd.S.ToMatrix();
            Matrix documentVector = svd.V.Copy();

            // get value of k
            // you can also manually set the value of k
            var k = (int) Math.Floor(Math.Sqrt(myMatrix.Columns));

            // reduce the vectors
            Matrix reducedWordVector = CopyMatrix(wordVector, wordVector.Rows, k - 1);
            Matrix reducedSigma = CreateSigmaMatrix(sigma, k - 1, k - 1);
            Matrix reducedDocumentVector = CopyMatrix(documentVector, documentVector.Rows, k - 1);

            // Recalculate the matrix
            Matrix docs = reducedDocumentVector*reducedSigma;
            Matrix words = reducedWordVector*reducedSigma;

            // Fill doc plot locations
            MyDocs = new double[docs.Rows,docs.Columns];
            for (int i = 0; i < docs.Rows; i++)
            {
                for (int j = 0; j < docs.Columns; j++)
                {
                    MyDocs[i, j] = docs.MatrixData[i, j];
                }
            }

            // Fill word plot locations
            MyWords = new double[words.Rows,words.Columns];
            for (int i = 0; i < words.Rows; i++)
            {
                for (int j = 0; j < words.Columns; j++)
                {
                    MyWords[i, j] = words.MatrixData[i, j];
                }
            }

            // Set counts for charts
            MyDocRowCount = docs.Rows;
            MyWordsRowCount = words.Rows;

            PrintMatrix(docs);
            PrintMatrix(words);
        }

        /// 
        /// Creates the sigma matrix.
        /// 
        /// The matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CreateSigmaMatrix(IMatrix matrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < columnEnd; i++)
            {
                copyMatrix.MatrixData[i, i] = matrix.MatrixData[i, 0];
            }

            return copyMatrix;
        }

        /// 
        /// Copies the matrix.
        /// 
        /// My matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CopyMatrix(IMatrix myMatrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < rowEnd; i++)
            {
                for (int j = 0; j < columnEnd; j++)
                {
                    copyMatrix.MatrixData[i, j] = myMatrix.MatrixData[i, j];
                }
            }

            return copyMatrix;
        }
    }
}

And what do the results look like?

As you can see the K-Means clustering algorithm correctly grouped the documents in the appropriate categories.

Recommended reading and thanks goes to Aresh Saharkhiz for sharing his implementation of K-Means Clustering.

Plotting Documents & Words: Using Latent Semantic Indexing

Eric — Fri, 06 Nov 2009 00:32:22 +0000

In the last blog post we looked over a couple of great papers talking about using Singular Value Decomposition (SVD) to do Latent Semantic Indexing (LSI) using the SmartMathLibrary. Now that we have the results we should plot them to get a sense of where these words and documents lay on a two dimensional Cartesian plane.

Jennifer Flynnâ€™s presentation “Latent Semantic Indexing Using SVD and Riemannian SVD” actually goes on to tell us how to do this. Essentially the process is the same as before however, k must equal 2. We ended up with k = 2 in our previous example however, in larger examples k will more than likely be a different number. Regardless, here we want to end up with a matrix with two columns giving us our (x,y) – if you wanted to plot these items in a three dimensional space k=3 and if you find an awesome way to plot where k=5 e-mail me.Â The formulas we use are as follows after we have performed SVD.

Documents = U*âˆ‘

Words = V*âˆ‘

The resulting matrices give us our (x,y) co-ordinates that we can then plot. I have been using the Dundas charting library for over two years now but the library is expensive so you should go and get the free library here since Microsoft acquired them and is a free download. And again for simplicities sake, this project is just a simple ASP.NET application.

The LSI Class:

Please note that this is almost exactly the same as in the previous blog however, here at the end of the GetDocumentWordPlots function we use the formulas mention above to load the co-ordinates of the words and documents in to a double array that we will ultimately pass to the chart.

using System;
using SmartMathLibrary;

namespace LSITest
{
    public class lsi
    {
        // this returns the formated html results
        public int MyDocColumnCount;
        public int MyDocRowCount;
        public double[,] MyDocs;
        public double[,] MyWords;
        public int MyWordsColumnCount;
        public int MyWordsRowCount;
        public string ToPrint;

        /// 
        /// LISs the test.
        /// 
        public void LSITest()
        {
            //Create Matrix
            var testArray = new double[,]
                                {
                                    {1, 0, 0, 1, 0, 0, 0, 0, 0},
                                    {1, 0, 1, 0, 0, 0, 0, 0, 0},
                                    {1, 1, 0, 0, 0, 0, 0, 0, 0},
                                    {0, 1, 1, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 1, 2, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 0, 1, 1, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 0, 0, 0, 0, 1},
                                    {0, 0, 0, 0, 0, 1, 1, 1, 0},
                                    {0, 0, 0, 0, 0, 0, 1, 1, 1},
                                    {0, 0, 0, 0, 0, 0, 0, 1, 1}
                                };

            // Load array in to Matrix
            var a = new Matrix(testArray);

            // print original matrix
            PrintMatrix(a);

            // preform Latent Semantic Indexing
            GetDocumentWordPlots(a);
        }

        /// 
        /// Prints the matrix.
        /// 
        /// My matrix.
        private void PrintMatrix(IMatrix myMatrix)
        {
            ToPrint += "

";

            for (int i = 0; i < myMatrix.Rows; i++)
            {
                for (int j = 0; j < myMatrix.Columns; j++)
                {
                    ToPrint += String.Format("{0:0.##}", myMatrix.MatrixData[i, j]) + "\t";
                }
                ToPrint += "
";
            }
        }

        /// 
        /// Gets the document word plots.
        /// 
        /// My matrix.
        private void GetDocumentWordPlots(Matrix myMatrix)
        {
            // Run single value decomposition
            var svd = new SingularValueDecomposition(myMatrix);
            svd.ExecuteDecomposition();

            // Put components into individual matrices
            Matrix wordVector = svd.U.Copy();
            Matrix sigma = svd.S.ToMatrix();
            Matrix documentVector = svd.V.Copy();

            // get value of k
            var k = 2;

            // reduce the vectors
            Matrix reducedWordVector = CopyMatrix(wordVector, wordVector.Rows, k - 1);
            Matrix reducedSigma = CreateSigmaMatrix(sigma, k - 1, k - 1);
            Matrix reducedDocumentVector = CopyMatrix(documentVector, documentVector.Rows, k - 1);

            // Recalculate the matrix
            Matrix docs = reducedDocumentVector*reducedSigma;
            Matrix words = reducedWordVector*reducedSigma;

            // Fill doc plot locations
            MyDocs = new double[docs.Rows,docs.Columns];
            for (int i = 0; i < docs.Rows; i++)
            {
                for (int j = 0; j < docs.Columns; j++)
                {
                    MyDocs[i, j] = docs.MatrixData[i, j];
                }
            }

            // Fill word plot locations
            MyWords = new double[words.Rows,words.Columns];
            for (int i = 0; i < words.Rows; i++)
            {
                for (int j = 0; j < words.Columns; j++)
                {
                    MyWords[i, j] = words.MatrixData[i, j];
                }
            }

            // Set counts for charts
            MyDocRowCount = docs.Rows;
            MyWordsRowCount = words.Rows;

            PrintMatrix(docs);
            PrintMatrix(words);
        }

        /// 
        /// Creates the sigma matrix.
        /// 
        /// The matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CreateSigmaMatrix(IMatrix matrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < columnEnd; i++)
            {
                copyMatrix.MatrixData[i, i] = matrix.MatrixData[i, 0];
            }

            return copyMatrix;
        }

        /// 
        /// Copies the matrix.
        /// 
        /// My matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CopyMatrix(IMatrix myMatrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < rowEnd; i++)
            {
                for (int j = 0; j < columnEnd; j++)
                {
                    copyMatrix.MatrixData[i, j] = myMatrix.MatrixData[i, j];
                }
            }

            return copyMatrix;
        }
    }
}

Here is the code behind for the web page that displays the chart and the matrices. Essentially we iterate through the double array pulled from the LSI class and load them in to a chart series.

using System;
using System.Drawing;
using System.Web.UI;
using Dundas.Charting.WebControl;

namespace LSITest
{
    public partial class _Default : Page
    {
        /// 
        /// Handles the Load event of the Page control.
        /// 
        /// The source of the event.
        /// The  instance containing the event data.
        protected void Page_Load(object sender, EventArgs e)
        {
            var mylsi = new lsi();
            mylsi.LSITest();
            Label1.Text = mylsi.ToPrint;
            double[,] myDocs = mylsi.MyDocs;
            double[,] myWords = mylsi.MyWords;

            // Load documents
            for (int i = 0; i < mylsi.MyDocRowCount; i++)
            {
                Chart1.Series["Series1"].Points.AddXY(myDocs[i, 0], myDocs[i, 1]);
            }

            // Load words
            for (int i = 0; i < mylsi.MyWordsRowCount; i++)
            {
                Chart1.Series["Series2"].Points.AddXY(myWords[i, 0], myWords[i, 1]);
            }

            // Set title
            Chart1.Series["Series1"].LegendText = "Documents";
            Chart1.Series["Series2"].LegendText = "Words";

            // Set point colors and shapes
            Chart1.Series["Series1"].Color = Color.Red;
            Chart1.Series["Series1"].MarkerStyle = MarkerStyle.Diamond;
            Chart1.Series["Series1"].MarkerSize = 12;
            Chart1.Series["Series2"].Color = Color.Gray;
            Chart1.Series["Series2"].MarkerStyle = MarkerStyle.Circle;
            Chart1.Series["Series2"].MarkerSize = 6;
        }
    }
}

And finally here is the ASP.NET web page.

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="LSITest._Default" %>
<%@ Register Assembly="DundasWebChart" Namespace="Dundas.Charting.WebControl" TagPrefix="DCWC" %>




    LSI Test

So lets see the result!

Now obviously you could/should probably write this a different way but it gets you to where you need to be.

I would also recommend you read in Flynn’s presentation on how to compare two words/documents by using the dot product of two row vectors. Or one could also use the Euclidean Distance Score. And if you are also interested I would recommend Sujit Pal’s blog post “IR Math in Java : Cluster Visualization” for additional reading.

Latent Semantic Indexing

Eric — Sun, 01 Nov 2009 15:48:26 +0000

Latent Semantic Indexing (LSI) is commonly described as a “indexing and retrieval method that uses a mathematical technique called Singular Value Decomposition (SVD) to identify patterns in the relationships between the terms and concepts contained in an unstructured collection of text.”. To be a bit more clear Sujit Pal has one of the best descriptions of what LSI is and how it occures:

Latent Semantic Indexing attempts to uncover latent relationships among documents based on word co-occurence. So if document A contains (w1,w2) and document B contains (w2,w3), we can conclude that there is something common between documents A and B. LSI does this by decomposing the input raw term frequency matrix (A, see below) into three different matrices (U, S and V) using Singular Value Decomposition (SVD). Once that is done, the three vectors are “reduced” and the original vector rebuilt from the reduced vectors. Because of the reduction, noisy relationships are suppressed and relations become very clearly visible.

So how is this done?

To start with let’s use the example in “Indexing by Latent Semantic Analysis” (Deerwester et al.) because you see this example repeated in a number of places on the web. The example in the paper says let’s take a look at 9 titles of papers that fall in to two categories “human computer interaction” & “graphs & trees”.

In this example the matrix is comprised of the word counts in the different document. The next step is take this matrix and break it down in to it’s different parts using SVD. The result looks like this:

After you have preformed SVD on the original matrix you then reduce the individual vectors. What the reduction of the vectors does is get rid of some of the “noise” – exposing the relationship between words and documents.

One question that arises is how much do you want to reduce the vectors (often called k)? There seems to be no hard and fast rule to this as different papers have different approaches/results with different values. In Sujit Pal’s post he uses the square root of the number of columns of the original matrix (m) which is then rounded down minus 1 which is I think a good method to use. It also happens to be the value that is used in the Deerwester paper k=2. The following picture shows what this looks like (please see Jennifer Flynn’s presentation Latent Semantic Indexing Using SVD and Riemannian SVD for a more elaborate example):

After the vectors have been reduced all that is required to do is take the vectors and multiply them back together again and that is it. See the result:

[Update: as one of the readers (Jorge) noted v is not exactly correct it should be V.Transpose. Please check out the “Indexing by Latent Semantic Analysis” (Deerwester et al.) paper starting on page 26 for the correct values of the matrices http://lsa.colorado.edu/papers/JASIS.lsi.90.pdf I will try to update this here shortly]

So lets take a look at the code – it follows the example outlined in the Deerwester paper and please keep in mind that is just a little class i put together in a asp.net test app that shows a html formatted matrix of the original (m) and the LSI result:

using System;
using SmartMathLibrary;

namespace LSITest
{
    public class lsi
    {
        // this returns the formated html results
        public string ToPrint;

        /// 
        /// LISs the test.
        /// 
        public void LSITest()
        {
            //Create Matrix
            var testArray = new double[,]
                                {
                                    {1, 0, 0, 1, 0, 0, 0, 0, 0},
                                    {1, 0, 1, 0, 0, 0, 0, 0, 0},
                                    {1, 1, 0, 0, 0, 0, 0, 0, 0},
                                    {0, 1, 1, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 1, 2, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 0, 1, 1, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 0, 0, 0, 0, 1},
                                    {0, 0, 0, 0, 0, 1, 1, 1, 0},
                                    {0, 0, 0, 0, 0, 0, 1, 1, 1},
                                    {0, 0, 0, 0, 0, 0, 0, 1, 1}
                                };

            // Load array in to Matrix
            var a = new Matrix(testArray);

            // print original matrix
            PrintMatrix(a);

            // preform Latent Semantic Indexing
            Transform(a);
        }

        /// 
        /// Prints the matrix.
        /// 
        /// My matrix.
        private void PrintMatrix(IMatrix myMatrix)
        {
            ToPrint += "

";

            for (int i = 0; i < myMatrix.Rows; i++)
            {
                for (int j = 0; j < myMatrix.Columns; j++)
                {
                    ToPrint += String.Format("{0:0.##}", myMatrix.MatrixData[i, j]) + "\t";
                }
                ToPrint += "
";
            }
        }

        /// 
        /// Transforms the specified my matrix.
        /// 
        /// My matrix.
        private void Transform(Matrix myMatrix)
        {
            // Run single value decomposition
            var svd = new SingularValueDecomposition(myMatrix);
            svd.ExecuteDecomposition();

            // Put components into individual matrices
            Matrix wordVector = svd.U.Copy();
            Matrix sigma = svd.S.ToMatrix();
            Matrix documentVector = svd.V.Copy();

            // get value of k
            // you can also manually set the value of k
            var k = (int) Math.Floor(Math.Sqrt(myMatrix.Columns));

            // reduce the vectors
            Matrix reducedWordVector = CopyMatrix(wordVector, wordVector.Rows, k - 1);
            Matrix reducedSigma = CreateSigmaMatrix(sigma, k - 1, k - 1);
            Matrix reducedDocumentVector = CopyMatrix(documentVector, documentVector.Rows, k - 1);

            // re-compute matrix
            Matrix a = reducedWordVector*reducedSigma*reducedDocumentVector.Transpose();

            // print result
            PrintMatrix(a);
        }

        /// 
        /// Creates the sigma matrix.
        /// 
        /// The matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CreateSigmaMatrix(IMatrix matrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < columnEnd; i++)
            {
                copyMatrix.MatrixData[i, i] = matrix.MatrixData[i, 0];
            }

            return copyMatrix;
        }

        /// 
        /// Copies the matrix.
        /// 
        /// My matrix.
        /// The row end.
        /// The column end.
        /// 
        private static Matrix CopyMatrix(IMatrix myMatrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < rowEnd; i++)
            {
                for (int j = 0; j < columnEnd; j++)
                {
                    copyMatrix.MatrixData[i, j] = myMatrix.MatrixData[i, j];
                }
            }

            return copyMatrix;
        }
    }
}

With much thanks to Sujit Pal, Jennifer Flynn and Deerwester for their excellent explanations.

Recommended Reading

IR Math with Java : TF, IDF and LSI – Sujit Pal

Indexing by Latent Semantic Analysis – Deerwester et al

Latent Semantic Indexing Using SVD and Riemannian SVD – Jennifer Flynn

Singular Value Decomposition

Eric — Mon, 26 Oct 2009 16:00:06 +0000

Singular Value Decomposition is something I’ve been wanting to wrap my head around for a while now that I am getting really into Machine Learning. Unfortunately, a lot of the material out there is often hard to understand and believe it or not there are few libraries that are available in .NET.

So what is singular value decomposition (SVD)? Probabably, the best description I’ve run across is:

Singular Value Decomposition is a way of factoring matrices into a series of linear approximations that expose the underlying structure of the matrix. SVD is extraordinarily useful and has many applications such as data analysis, signal processing, pattern recognition, image compression, weather prediction, and Latent Semantic Analysis. [iMetaSearch]

SVD formula is:

M=Uâˆ‘V*

M is simply a m-by-n matrix, U form a set of orthonormal “output” basis vector directions for M, Î£ are the singular values, which can be thought of as scalar “gain controls” by which each corresponding input is multiplied to give a corresponding output and V* form a set of orthonormal “input” or “analysing” basis vector directions for M. The best walk through I’ve come across is over at iMetaSearch here.

A lack of good .NET Libraries.

I tried out four different libraries: SmartMathLibrary, LatoolNet, ALGLIB and DotNetMatrix. Out of these four I could only get two of them completely working and I ultimately came to the conclusion that SmartMathLibrary was the best for doing SVD.

The Code

Here is the code to replicate this tutorial over at MIT.

using System;
using SmartMathLibrary;

namespace MatrixTest2
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            SVDTest();
            Console.ReadLine();
        }

        private static void SVDTest()
        {
            // Create/load array
            var holeDifficulty = new double[,]
                                     {
                                         {2, 1, 0, 0},
                                         {4, 3, 0, 0}
                                     };

            // Load in to Matrix
            var a = new Matrix(holeDifficulty);

            // Singular Value Decomposition
            var SVD = new SingularValueDecomposition(a);
            SVD.ExecuteDecomposition();

            // Get the general vector
            GeneralVector s = SVD.S;

            // Display results
            Console.WriteLine(a.Transpose().ToString());
            Console.WriteLine();
            Console.WriteLine(s.ToString());
            Console.WriteLine();
            Console.WriteLine(SVD.U.ToString());
            Console.WriteLine();
            Console.WriteLine(SVD.V.ToString());
        }
    }
}

Additional Resources

iMetaSearch

IR Math with Java : TF, IDF and LSI

SVD Tutorial

Pearson’s Correlation Coefficient

Eric — Sun, 25 Oct 2009 19:33:25 +0000

In Toby Segaran’s book “Programming Collective Intelligence” one additional methods used “to determine the similarity between people’s interests is to use the Pearson’s correlation coefficient. In statistics Pearson’s correlation coefficient is often symbolized as simply r. I also covered Toby’s Euclidean Distance Score here.

Is how r is calculated.

And here is some sloppy source code to get you going:

using System;
using System.Linq;

namespace PearsonTest
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            var myP = new Correlation();

            var lisaRose = new double[] {0, 2, 4, 6, 8, 10, 12};
            var jackMatthews = new[] {2.1, 5, 9, 12.6, 17.3, 21, 24.7};

            double score = myP.PearsonCorrelation(lisaRose, jackMatthews);

            Console.WriteLine(score);
            Console.ReadLine();

            // The answer is 0.99887956534852
        }
    }

    internal class Correlation
    {
        public double PearsonCorrelation(double[] x, double[] y)
        {
            double result;
            double xMean = 0;
            double yMean = 0;
            double xDenom = 0;
            double yDenom = 0;
            double denominator;
            double numerator = 0;
            double n;

            // Make sure arrays are same size and greater than 1
            if ((x.Count() == y.Count()) && (x.Count() >= 1))
            {
                n = x.Count();
            }
            else
            {
                result = 0;
                return result;
            }

            // Find Means
            for (int i = 0; i <= n - 1; i++)
            {
                xMean += x[i];
                yMean += y[i];
            }
            xMean = xMean/n;
            yMean = yMean/n;

            // Caluculate numerator and denominator
            for (int i = 0; i <= n - 1; i++)
            {
                //Caluculate numerator
                double numX = x[i] - xMean;
                double numY = y[i] - yMean;
                numerator += numX*numY;

                // Caluculate denominator parts
                xDenom += Math.Pow(numX, 2);
                yDenom += Math.Pow(numY, 2);
            }

            // Caluculate denominator
            denominator = Math.Sqrt(xDenom*yDenom);

            // Check for division by zero
            if (denominator == 0)
            {
                result = 0;
            }
            else
            {
                result = numerator/denominator;
            }

            return result;
        }
    }
}

SQL Server Migration Assistant for Access Walk Through.

Eric — Fri, 16 Oct 2009 10:07:47 +0000

I am currently traveling and do not have MS SQL Server Workgroup edition loaded up on my laptop and needed to import some data from an Microsoft Access database. In doing a quick search I cam across this great utility called SQL Server Migration Assistant for Access which you can download for free here. Does everything I need and is pretty easy to use – but, I took some screen shots to show you how it’s done.

My toolbox: A little list of programming tools.

Eric — Sat, 06 Jun 2009 17:05:53 +0000

List of tools I use:

Add-In’s

Resharper: Is one of those add-ins that once you start using it would be hard to give up â€“ as it provides solutions to programming errors, helps out with refactoring, unit testing, code formatting and clean up. [link]
VisualSVN: You need a plug-in that connects your solution to some form of version control. Iâ€™ve always been a fan of Subversion [free] and VisualSVN helps me connect to it. AnkhSVN is also a great free alternative. I would also recommend that you check out Visual SVN Server as it allows you configure and manage Subversion for your whole team and it is free. [link]
GhostDoc: This free extension automatically generates your documentation comments for your code. A must. [link]
Clone Detective: This add-in tells you where you are repeating code. Very helpful but sometimes does not play well with others add-ins. [link]

Controls

Dundas: Best graphing framework on .NET period. Only negative is that it is not cheap but you can accomplish most stuff with ASP.NET Charting Controls. [link]
Virtual Earth: Fun control to use Virtual Earth. [link]
Telerik RadControls: Great controls and is a must but like Dundas is not cheap. [link]
ASP.Net Ajax Framework: A must if you are still doing web forms page. [link]
ASP.Net Ajax Toolkit: The bells and whistles for the framework. [link]