1. Online: I kind of knew that I would probably need a program that is at least partly online. This is mainly due to the fact that my work often requires me to travel often for up to two weeks at a time.
2. Program Type: I am in kind of a weird place academically. I graduated with a Bachelors of Science in Information Technology even though I considered myself to be in the Computer Science program. I actually took all the required classes for the Comp Sci degree with the exception of all the math. This is now a bit of a detriment because I could have used about two more math classes that most masters programs require. It really leaves me at looking at Software Engineering or Business Intelligence DB programs.
3. Computer Science Programs: Interestingly enough there are many aspects to Computer Science that I am not interested in: namely many programs have you focus in a particular area (i.e. graphics). The only focus area that really interests me is Machine Learning and often this gets put under AI – which often is not really the same.
4. DC – Area: The DC area is kind of a strange place to look for schools. There are only a couple of programs that are actually convenient for me to travel to: George Washington, American, Georgetown, Howard and the University of the District of Columbia. But, truth be told I am actually not really interested in any of these schools for various reasons.

Wikipedia describes Apriori:

In computer science and data mining, Apriori is a classic algorithm for learning association rules. Apriori is designed to operate on databases containing transactions (for example, collections of items bought by customers, or details of a website frequentation). Other algorithms are designed for finding association rules in data having no transactions (Winepi and Minepi), or having no timestamps (DNA sequencing).

The classic example is if you own a store and someone buys milk what is the probability that he will also buy bread and eggs or if voters in one state voted for one issue what is the chance he voted for something else. The applications for this approach are pretty much limitless.

The code in the SDK is pretty good with a couple of exceptions: there is little documentation and it only supports XML files and OleDb data connections. I have reworked it so it will also connect to a MSSQL database.

For this test application I created a simple C# Console Application and imported the “APriori” project in to the solution. In the APriori project you will to add these two bits of code to classes to the APriori project:

Add this method to DataAccessLayer.cs

	public Data GetTransactionsData(string rdbmsConnectionString, string dataSource)
        {
            myDatabase = new Data();
            string query = "SELECT * FROM " + dataSource;
            var myConn = new SqlConnection(rdbmsConnectionString);
            var myDBAdapter = new SqlDataAdapter(query, myConn);

            myConn.Open();
            try
            {
                myDBAdapter.Fill(myDatabase, "TransactionTable");
            }
            finally
            {
                myConn.Close();
            }
            return myDatabase;
        }

Add this method to DataMining.cs

public Data MarketBasedAnalysis(double supportCount, double minimumConfidence, string connectionString, string dataSource)
        {

            Database database = new Database();
            ItemsetCandidate Item = new ItemsetCandidate();

            this.AP = new APriori.Apriori();
            this.AP.ProgressMonitorEvent += new ProgressMonitorEventHandler(this.OnProgressMonitoringCompletedEvent);
            this.dataBase = database.GetTransactionsData(connectionString, dataSource);
            database.Transactions = this.dataBase;
            this.transactionsCount = this.dataBase.TransactionTable.Count;

            supportCount = ((supportCount / 100) * this.transactionsCount);

            minimumConfidence = (minimumConfidence / 100);

            string support = "SupportCount >= " + supportCount + " AND Level > 1";

            string sort = "SupportCount, Level";
            ItemsetCandidate uniqueItems = AP.CreateOneItemsets(database);
            AP.AprioriGenerator(uniqueItems, database, Convert.ToInt32(supportCount));
            ItemsetArrayList[] keys = database.GetItemset(support, sort);
            string msg = "Creating Frequent Subsets for Items";
            ProgressMonitorEventArgs e = new ProgressMonitorEventArgs(1, 100, 95, "DataMining.MarketBasedAnalysis(3)", msg);
            this.OnProgressMonitorEvent(e);

            for (int counter = 0; counter < keys.Length; counter++)
            {
                AP.CreateItemsetSubsets(0, keys[counter], null, database);
            }

            msg = "Completed C#.NET Data Mining Market Based Analysis";
            e = new ProgressMonitorEventArgs(1, 100, 100, "DataMining.MarketBasedAnalysis(3)", msg);
            this.OnProgressMonitorEvent(e);

            //Set the public properties of the class
            this.minimumSupportCount = supportCount;
            this.minimumConfidence = minimumConfidence;
            this.connectionString = connectionString;
            this.dataSource = dataSource;
            this.dataSourceCommand = dataSourceCommand;

            //return the database of transactions
            return this.dataBase;

        }

Here is my class in my console application

using System;
using System.Data;
using VISUAL_BASIC_DATA_MINING_NET;
using VISUAL_BASIC_DATA_MINING_NET.CustomEvents;

namespace APr2.classes
{
    internal class testrun
    {
        private Data _dataAnalysis;
        public event ProgressMonitorEventHandler ProgressMonitorEvent;

        /// <summary>
        /// Runs the Apriori.
        /// </summary>
        public void RunApriori()
        {
            // Create Data Mining Object
            var myDM = new DataMining();

            // Register Event
            myDM.ProgressMonitorEvent += OnProgressMonitorEvent;

            // Connect To Data Base & Process Items
            _dataAnalysis = myDM.MarketBasedAnalysis(2,             // Support Count
                                                     2,             // Minimum Confidence
                                                     @"Data Source=(local);Initial Catalog=Apriori;Integrated Security=True;", // Connection String
                                                     "Example");    // Table in db

            // Copy to Data View
            var dataView = new ViewData();
            _dataAnalysis.Tables.Add(dataView.CreateViewRulesTable(2, _dataAnalysis).Copy());
            _dataAnalysis.Tables.Add(dataView.CreateViewSubsetTable(_dataAnalysis).Copy());

            // Spacer Line
            Console.WriteLine();

            // Print Items
            foreach (DataRow row in dataView.ViewDataSet.Tables[1].Rows)
            {
                double per = Convert.ToDouble(row.ItemArray[2].ToString().Substring(0, (row.ItemArray[2].ToString().Length -1)));
                Console.WriteLine(row.ItemArray[0] + "\t" + row.ItemArray[1] + "\t" + String.Format("{0:###.##%}", (per/100)));
            }
        }

        /// <summary>
        /// Called when [progress monitor event].
        /// </summary>
        /// <param name="sender">The sender.</param>
        /// <param name="e">The <see cref="VISUAL_BASIC_DATA_MINING_NET.CustomEvents.ProgressMonitorEventArgs"/> instance containing the event data.</param>
        public void OnProgressMonitorEvent(object sender, ProgressMonitorEventArgs e)
        {
            // Prints Event Messages
            Console.Write("\r" + e.EventMessage);
        }
    }
}

Your MSSQL Code will be this

GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[Example](
	[TransactionID] [int] IDENTITY(1,1) NOT NULL,
	[Transactions] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
 CONSTRAINT [PK_Example] PRIMARY KEY CLUSTERED
(
	[TransactionID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

And these records:

1	Books, CD, Video
2	CD, Games
3	CD, DVD
4	Books, CD, Games
5	Books, DVD
6	CD, DVD
7	Books, DVD
8	Books, CD, DVD, Video
9	Books, CD, DVD
10	Books, Games
11	Games, Lasers

Run the RunApriori() method in my class and it will yield you the correct results. Have fun.

Here is just a simple result of what the map looks like:

Here are some of the highlights:

1. Supports time series data
2. Fairly extensive database to pull data from online (apparently of 10,000 different datasets)
3. Create you own datasets via csv files
4. Export to video
5. Export to KML file for use in google maps/earth.
6. Has the following border sets: Country, US States/Counties, US Zip Codes

There is one caveat that I feel I must add to not give the impression that all is rosy – I bought the application over a week ago and it took several e-mails for them to reply to me and finally get my full download.

# includes
library(RODBC)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT
Country AS 'Country',
Year AS 'Year',
[96741] AS 'GDP growth (annual %)--WDI-2009',
[96841] AS 'GDP per capita (constant 2000 US$)--WDI-2009',
[99941] AS 'Population growth (annual %)--WDI-2009',
[100041] AS 'Population, total--WDI-2009'
FROM
(
SELECT DISTINCT CountryID, Country, Year, IndicatorID, IndValue
FROM [Time Series Data]
WHERE (
((IndicatorID) = 96741) OR
((IndicatorID) = 96841) OR
((IndicatorID) = 99941) OR
((IndicatorID) = 100041))
AND
(((CountryID) = 4118) OR
((CountryID) = 4125) OR
((CountryID) = 4129) OR
((CountryID) = 4134) OR
((CountryID) = 4141) OR
((CountryID) = 4145) OR
((CountryID) = 4164) OR
((CountryID) = 4186) OR
((CountryID) = 4213) OR
((CountryID) = 4327) OR
((CountryID) = 4219) OR
((CountryID) = 4221) OR
((CountryID) = 4227) OR
((CountryID) = 4230) OR
((CountryID) = 4243) OR
((CountryID) = 4326) OR
((CountryID) = 4268) OR
((CountryID) = 4272) OR
((CountryID) = 4273) OR
((CountryID) = 4325) OR
((CountryID) = 4300) OR
((CountryID) = 4308) OR
((CountryID) = 4309) OR
((CountryID) = 4311) OR
((CountryID) = 4316))
AND
(NOT (Year IS NULL)) AND (Year >= 1960) AND
(Year <= 2007))
ps
PIVOT (
MAX(IndValue)
FOR IndicatorID IN ([96741], [96841], [99941], [100041] ) )
AS
pvt
order by Country, Year")

#close connection
odbcClose(channel)

#Plot charts
plot(myData[3:6], col="orange", main="Select Indicators for Europe and Central Asia")

Here is the result

First, I pulled the data from HealthSystems2020 time series database and imported the data in to MSSQL. I did some minor touch ups to the database giving the indicator an id etc. The second thing you need to do is create an ODBC connection for your database here is a fairly good tutorial. In this example I called my ODBC DSN “HealthDB”. Also make sure you adjust you sql query so that they are pulling the correct names/values.

Finally, here is the code:

# includes
library(RODBC)
library(lattice)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT Country, IndValue
FROM         [YOURTABLE]
WHERE     (id = 96841) AND (
(Country = 'Afghanistan') OR
(Country = 'Bangladesh') OR
(Country = 'Bhutan') OR
(Country = 'India') OR
(Country = 'Maldives') OR
(Country = 'Nepal') OR
(Country = 'Pakistan') OR
(Country = 'China') OR
(Country = 'Indonesia') OR
(Country = 'Sri Lanka'))")

#close connection
odbcClose(channel)

#create histogram
histogram(~ myData[,3] | myData[,1], type="count", col="red", main = "GDP per capita (constant 2000 US$)", xlab="Country")

And here is the result:

One of the reasons I wanted to revisit this topic is that after having a conversation with a friend about this very topic, I came across a series of articles written by Nate Silver regarding the polling firm Strategic Vision. Strategic Vision released a poll focusing on Oklahoma students and it motivated Silver to ask a series of questions about the firm.

Now I don’t want to really focus in on all the Strategic Vision stuff but, I do want to talk about a method Silver used to detect anomalies in their polling. As Silver correctly suggests, polling would not be a good candidate to using Benford’s Law (i.e. take the last Presidential race where for the most part the two candidates were going back and forth in the 40~50% range – Benford’s wouldn’t work). However, there is another method that might give you a little insight as Silver explains:

For each question, I recorded the trailing digit for each candidate or line item. For instance, if Strategic Vision had Barack Obama beating John McCain 48-43 in a particular state, I’d record a tally in the 8 column and another in the 3 column. Or if they had voters opposing a particular policy 50-45, I’d record a tally in the 0 column (for 50) and another in the 5 column (for 45).

And what Silver says essentially is, that if you look at the last digit you should have roughly a uniform distribution. Put it another way, that if I have roughly 200 4’s I would expect roughly 200 8’s too. Silver also says that when using the trailing digit method that in some cases, you might find deviations from this distribution might be due to rounding error’s or a specific mathematical method. The trailing digit test is clearly not a sure fire way to detect fraud or anything, but just another useful tool to see if the data passes the smell test.

Silver’s insight prompted me to write some code and play around with these two methods just to see what comes up. Because I am fairly familiar with the the United Nation’s World Health Database, I thought I would run some tests using these methods. Here are the results:

Gross Domestic Product (GDP)

As you can see, GDP generally follows Benford’s Law and the trailing digit test. Something to note on the trailing digit results – the average for each number (bin) is 23.7 so the number of 1’s and the number of 5’s are roughly equidistant from the average, even though it seems a little odd finding a GDP with a value ending in 1 is almost twice as likely as finding a value ending in 5.

Life expectancy at birth (years)

Now here is an example where Benford’s Law will not work. The reason is because the range of life spans for all countries is from 40-83 years of age, so we are going to have to focus in on the trailing digit test.

You will notice that there is something weird here with the trailing digits. First, the average value is 19.3 and we have 0’s and 2’s appearing almost 3 times more than 7’s. One thing that might account for this disparity is the fact that we have a little over 20 countries that fall in the range of 80-83, which would tilt the values of 0,1,2,3 a little higher than normal. And the number of countries in the 40’s is also a little sparse. So what I did was remove these from the set and re-ran the test. Here are the following results.

You will notice that after cleaning up the data a 3 is still almost 3 times as likely to appear as a 7 in what should be some fairly naturally occurring numbers. The average for this set is 14.8.

Now I think it would be inappropriate for me to draw any hard conclusions about the Life Expectancy data other than to say that some form of rounding has likely occurred due to the fact that the data are all whole numbers.

That said though, you can see how these two simple/practical tests can assist in determining whether there has been some human manipulation of the data. Here is the code that I used:

using System;
using System.Linq;

namespace BenfordsLaw
{
    /// <summary>
    /// Benfords Law Class
    /// </summary>
    public class Benfords
    {
        /// <summary>
        /// Adds the data.
        /// </summary>
        /// <param name="data">The data.</param>
        /// <returns></returns>
        public double[] CalculateBenfordsDistribution(double[] data)
        {
            if (data.Count() == 0)
            {
                throw new ArgumentException("Error: There are no items in your data array.");
            }

            //Create benford bins to hold counts
            var benfordsContainer = new double[9];

            // Loop through array
            foreach (double number in data)
            {
                // Get absolute value of number
                double currentNumber = Math.Abs(number);

                // for items smaller than 1 * multiply it so you
                // can find the first number
                if ((currentNumber < 1) && (currentNumber > 0))
                {
                    double num = (currentNumber*10000);
                    while (num >= 10)
                        num /= 10;
                    PackageNumberInBenfordBin(benfordsContainer, num);
                }
                else
                {
                    double num = currentNumber;
                    while (num >= 10)
                        num /= 10;
                    PackageNumberInBenfordBin(benfordsContainer, num);
                }
            }

            return benfordsContainer;
        }

        /// <summary>
        /// Trailings the digit check.
        /// </summary>
        /// <param name="data">The data.</param>
        /// <returns></returns>
        public double[] TrailingDigitCheck(double[] data)
        {
            if (data.Count() == 0)
            {
                throw new ArgumentException("Error: There are no items in your data array.");
            }

            //Create benford bins to hold counts
            var trailingContainer = new double[10];

            // Loop through array
            foreach (double number in data)
            {
                // Get absolute value of number
                double currentNumber = Math.Abs(number);
                string numTemp = currentNumber.ToString();
                string numTemp2 = numTemp.Substring(numTemp.Length - 1);
                PackageNumberInTrailingBin(trailingContainer, Convert.ToDouble(numTemp2));
            }

            return trailingContainer;
        }

        /// <summary>
        /// Packages the number in benford bin.
        /// </summary>
        /// <param name="myContainer">My container.</param>
        /// <param name="num">The num.</param>
        private static void PackageNumberInBenfordBin(double[] myContainer, double num)
        {
            // Update container totals
            int myNum = Convert.ToInt32(Math.Floor(num));
            myContainer[myNum-1] += 1;
        }

        /// <summary>
        /// Packages the number in trailing bin.
        /// </summary>
        /// <param name="myContainer">My container.</param>
        /// <param name="num">The num.</param>
        private static void PackageNumberInTrailingBin(double[] myContainer, double num)
        {
            // Update container totals
            int myNum = Convert.ToInt32(Math.Floor(num));
            myContainer[myNum] += 1;
        }
    }
}

In the paper “Indexing by Latent Semantic Analysis” (Deerwester et al.) they have an example of 9 titles of different papers grouped in to two categories “human computer interaction” & “graphs & trees”. So far, we’ve used Singular Value Decomposition (SVD) and Latent Semantic Indexing (LSI) to better understand the relationship of words and documents. In the last blog post we then took the results in LSI to plot words and documents on a two dimensional Cartesian plane.

All of this is pretty interesting stuff in and of itself however, the next step really is to see which documents belong in each group. One way to do this is by using K-Means clustering.

Simply speaking k-means clustering is an algorithm to classify or to group your objects based on attributes/features into K number of group. K is positive integer number. The grouping is done by minimizing the sum of squares of distances between data and the corresponding cluster centroid. Thus the purpose of K-mean clustering is to classify the data. [Kardi Teknomo]

A big chunk of the code is built off of the same project we are working on. I am using Aresh Saharkhiz K-Means implementation in the project with some minor changes/refactoring done by me.

Let take a look at the code!

This first part is the display (an ASP.NET app.)

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="LSITest._Default" %>
<%@ Register Assembly="DundasWebChart" Namespace="Dundas.Charting.WebControl" TagPrefix="DCWC" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>LSI Test</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <DCWC:Chart ID="Chart1" runat="server" Height="400px" Width="400px"
            ImageType="Jpeg">
            <Legends>
                <DCWC:Legend Name="Default" Alignment="Center" Docking="Bottom"></DCWC:Legend>
            </Legends>
            <Titles>
                <DCWC:Title Name="Title1">
                </DCWC:Title>
            </Titles>
            <Series>
                <DCWC:Series Name="Series1" ChartType="Point" MarkerBorderColor="64, 64, 64"
                    ShadowOffset="1">
                </DCWC:Series>
                <DCWC:Series Name="Series2" ChartType="Point" MarkerBorderColor="64, 64, 64"
                    ShadowOffset="1">
                </DCWC:Series>
                <DCWC:Series Name="Series3" ChartType="Point" MarkerBorderColor="64, 64, 64"
                    ShadowOffset="1">
                </DCWC:Series>
            </Series>
            <ChartAreas>
                <DCWC:ChartArea Name="Series2">
                    <axisy interval="0.5" maximum="2" minimum="-1">
                        <majorgrid linecolor="Gray" linestyle="Dash" />
                    </axisy>
                    <axisx interval="0.5" maximum="2.5" minimum="-0.5">
                        <majorgrid linecolor="Gray" linestyle="Dash" />
                    </axisx>
                </DCWC:ChartArea>
            </ChartAreas>
        </DCWC:Chart>
    </div>
    </form>
</body>
</html>

This is the code behind for the ASP.NET page. Because we are only dealing with two known categories K-Means is plotting out the two categories and if you wanted to do more you would definitely have to re-write the ColorCodeDocuments function.

using System;
using System.Data;
using System.Drawing;
using System.Web.UI;
using Dundas.Charting.WebControl;

namespace LSITest
{
    public partial class _Default : Page
    {
        /// <summary>
        /// Handles the Load event of the Page control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.EventArgs"/> instance containing the event data.</param>
        protected void Page_Load(object sender, EventArgs e)
        {
            // Perform LSI
            var mylsi = new lsi();
            mylsi.LSITest();
            double[,] myDocs = mylsi.MyDocs;

            // Plot Documents and the k-means
            const string distanceType = "manhattan";
            PlotDocuments(myDocs, mylsi.MyDocRowCount);
            PlotKMeansPoints(myDocs, 2, distanceType);
            ColorCodeDocuments(distanceType);

            // If you want to plot the words just un-comment the next two lines
            //double[,] myWords = mylsi.MyWords;
            //PlotWords(myDocs, mylsi.MyWordsRowCount);

            // comment this line out to show words in legend
            Chart1.Series["Series2"].ShowInLegend = false;
        }

        /// <summary>
        /// Plots the words.
        /// </summary>
        /// <param name="myWords">My words.</param>
        /// <param name="myWordsRowCount">My words row count.</param>
        private void PlotWords(double[,] myWords, int myWordsRowCount)
        {
            for (int i = 0; i < myWordsRowCount; i++)
            {
                Chart1.Series["Series2"].Points.AddXY(myWords[i, 0], myWords[i, 1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series2"].LegendText = "Words";
            Chart1.Series["Series2"].Color = Color.Gray;
            Chart1.Series["Series2"].MarkerStyle = MarkerStyle.Circle;
            Chart1.Series["Series2"].MarkerSize = 6;
        }

        /// <summary>
        /// Plots the documents.
        /// </summary>
        /// <param name="myDocs">My docs.</param>
        /// <param name="myDocRowCount">My doc row count.</param>
        private void PlotDocuments(double[,] myDocs, int myDocRowCount)
        {
            // Load documents
            for (int i = 0; i < myDocRowCount; i++)
            {
                Chart1.Series["Series1"].Points.AddXY(myDocs[i, 0], myDocs[i, 1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series1"].LegendText = "Documents";
            Chart1.Series["Series1"].Color = Color.Red;
            Chart1.Series["Series1"].MarkerStyle = MarkerStyle.Diamond;
            Chart1.Series["Series1"].MarkerSize = 12;
        }

        /// <summary>
        /// Plots the K means points.
        /// </summary>
        /// <param name="items">The items.</param>
        /// <param name="k">The k.</param>
        /// <param name="distanceType"></param>
        private void PlotKMeansPoints(double[,] items, int k, string distanceType)
        {
            ClusterCollection clusters = kmeans.ClusterDataSet(k, items, distanceType);

            for (int i = 0; i < clusters.Count; i++)
            {
                Chart1.Series["Series3"].Points.AddXY(clusters[i].ClusterMean[0], clusters[i].ClusterMean[1]);
            }

            // Set point colors and shapes
            Chart1.Series["Series3"].LegendText = "Cluster";
            Chart1.Series["Series3"].Color = Color.Gold;
            Chart1.Series["Series3"].MarkerStyle = MarkerStyle.Star6;
            Chart1.Series["Series3"].MarkerSize = 18;
        }

        /// <summary>
        /// Colors the code documents.
        /// </summary>
        /// <param name="distanceType">Type of the distance.</param>
        private void ColorCodeDocuments(string distanceType)
        {
            var myDist = new similarity();

            // Extract data
            DataSet myDocs = Chart1.DataManipulator.ExportSeriesValues("Series1");
            DataSet myKMeansPoints = Chart1.DataManipulator.ExportSeriesValues("Series3");

            // Document counter
            int count = 0;

            // Get co-ordinates for k-means points
            double firstKMeansX = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[0]["X"]);
            double firstKMeansY = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[0]["Y"]);
            double secondKMeansX = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[1]["X"]);
            double secondKMeansY = Convert.ToDouble(myKMeansPoints.Tables[0].Rows[1]["Y"]);

            foreach (DataRow docRow in myDocs.Tables[0].Rows)
            {
                // get co-ordinates for current doc
                double currentDocX = Convert.ToDouble(docRow["X"]);
                double currentDocY = Convert.ToDouble(docRow["Y"]);

                // load in to arrays
                double[] firstX = {currentDocX, currentDocY};
                double[] firstY = {firstKMeansX, firstKMeansY};
                double[] secondX = {currentDocX, currentDocY};
                double[] secondY = {secondKMeansX, secondKMeansY};

                // find the distance
                double firstDist = myDist.FindDistance(firstX, firstY, distanceType);
                double secondDist = myDist.FindDistance(secondX, secondY, distanceType);

                // Color accordingly
                Chart1.Series["Series1"].Points[count].Color = firstDist < secondDist ? Color.Blue : Color.Gray;
                count++;
            }
        }
    }
}

This is the K-Means class written by Aresh Saharkhiz with my changes

/// Most of this code was written by Aresh Saharkhiz
/// Re-organized by me
/// See Code Project: http://www.codeproject.com/KB/recipes/K-Mean_Clustering.aspx
using System;
using System.Collections;
using System.Data;
using System.Diagnostics;

namespace LSITest
{
    public class kmeans
    {
        /// <summary>
        /// Calculates The Mean Of A Cluster OR The Cluster Center
        /// </summary>
        /// <param name="cluster">
        /// A two-dimensional array containing a dataset of numeric values
        /// </param>
        /// <returns>
        /// Returns an Array Defining A Data Point Representing The Cluster Mean or Centroid
        /// </returns>
        public static double[] ClusterMean(double[,] cluster)
        {
            int rowCount = cluster.GetUpperBound(0) + 1;
            int fieldCount = cluster.GetUpperBound(1) + 1;
            var dataSum = new double[1,fieldCount];
            var centroid = new double[fieldCount];

            for (int j = 0; j < fieldCount; j++)
            {
                for (int i = 0; i < rowCount; i++)
                {
                    dataSum[0, j] = dataSum[0, j] + cluster[i, j];
                }

                centroid[j] = (dataSum[0, j]/rowCount);
            }

            return centroid;
        }

        /// <summary>
        /// Seperates a dataset into clusters or groups with similar characteristics
        /// </summary>
        /// <param name="clusterCount">The number of clusters or groups to form</param>
        /// <param name="data">An array containing data that will be clustered</param>
        /// <param name="type"></param>
        /// <returns>A collection of clusters of data</returns>
        public static ClusterCollection ClusterDataSet(int clusterCount, double[,] data, string type)
        {
            int rowCount = data.GetUpperBound(0) + 1;
            int fieldCount = data.GetUpperBound(1) + 1;
            int stableClustersCount = 0;
            double[] dataPoint;
            var random = new Random();
            Cluster cluster;
            var clusters = new ClusterCollection();
            var clusterNumbers = new ArrayList(clusterCount);
            var myDist = new similarity();

            while (clusterNumbers.Count < clusterCount)
            {
                int clusterNumber = random.Next(0, rowCount - 1);

                if (!clusterNumbers.Contains(clusterNumber))
                {
                    cluster = new Cluster();
                    clusterNumbers.Add(clusterNumber);
                    dataPoint = new double[fieldCount];

                    for (int field = 0; field < fieldCount; field++)
                    {
                        dataPoint.SetValue((data[clusterNumber, field]), field);
                    }

                    cluster.Add(dataPoint);
                    clusters.Add(cluster);
                }
            }

            while (stableClustersCount != clusters.Count)
            {
                stableClustersCount = 0;
                ClusterCollection newClusters = ClusterDataSet(clusters, data, type);

                for (int clusterIndex = 0; clusterIndex < clusters.Count; clusterIndex++)
                {
                    if ((myDist.FindDistance(newClusters[clusterIndex].ClusterMean, clusters[clusterIndex].ClusterMean, type)) == 0)
                    {
                        stableClustersCount++;
                    }
                }

                clusters = newClusters;
            }

            return clusters;
        }

        /// <summary>
        /// Seperates a dataset into clusters or groups with similar characteristics
        /// </summary>
        /// <param name="clusters">A collection of data clusters</param>
        /// <param name="data">An array containing data to b eclustered</param>
        /// <param name="type"></param>
        /// <returns>A collection of clusters of data</returns>
        public static ClusterCollection ClusterDataSet(ClusterCollection clusters, double[,] data, string type)
        {
            double[] dataPoint;
            double firstClusterDistance = 0.0;
            int rowCount = data.GetUpperBound(0) + 1;
            int fieldCount = data.GetUpperBound(1) + 1;
            int position = 0;
            var myDist = new similarity();

            // create a new collection of clusters
            var newClusters = new ClusterCollection();

            for (int count = 0; count < clusters.Count; count++)
            {
                var newCluster = new Cluster();
                newClusters.Add(newCluster);
            }

            if (clusters.Count <= 0)
            {
                throw new SystemException("Cluster Count Cannot Be Zero!");
            }

            for (int row = 0; row < rowCount; row++)
            {
                dataPoint = new double[fieldCount];

                for (int field = 0; field < fieldCount; field++)
                {
                    dataPoint.SetValue((data[row, field]), field);
                }

                for (int cluster = 0; cluster < clusters.Count; cluster++)
                {
                    double[] clusterMean = clusters[cluster].ClusterMean;

                    if (cluster == 0)
                    {
                        firstClusterDistance = myDist.FindDistance(dataPoint, clusterMean, type);
                        position = cluster;
                    }
                    else
                    {
                        double secondClusterDistance = myDist.FindDistance(dataPoint, clusterMean, type);

                        if (firstClusterDistance > secondClusterDistance)
                        {
                            firstClusterDistance = secondClusterDistance;
                            position = cluster;
                        }
                    }
                }

                newClusters[position].Add(dataPoint);
            }

            return newClusters;
        }

        /// <summary>
        /// Converts the data table to array.
        /// </summary>
        /// <param name="table">The table.</param>
        /// <returns></returns>
        public static double[,] ConvertDataTableToArray(DataTable table)
        {
            int rowCount = table.Rows.Count;
            int fieldCount = table.Columns.Count;

            var dataPoints = new double[rowCount,fieldCount];

            for (int rowPosition = 0; rowPosition < rowCount; rowPosition++)
            {
                DataRow row = table.Rows[rowPosition];

                for (int fieldPosition = 0; fieldPosition < fieldCount; fieldPosition++)
                {
                    double fieldValue;
                    try
                    {
                        fieldValue = double.Parse(row[fieldPosition].ToString());
                    }
                    catch (Exception ex)
                    {
                        Debug.WriteLine(ex.ToString());
                        throw new InvalidCastException("Invalid row at " + rowPosition + " and field " + fieldPosition,
                                                       ex);
                    }

                    dataPoints[rowPosition, fieldPosition] = fieldValue;
                }
            }

            return dataPoints;
        }
    }

    /// <summary>
    /// A class containing a group of data with similar characteristics (cluster)
    /// </summary>
    [Serializable]
    public class Cluster : CollectionBase
    {
        private double[] _clusterMean;
        private double[] _clusterSum;

        /// <summary>
        /// The sum of all the data in the cluster
        /// </summary>
        public double[] ClusterSum
        {
            get { return _clusterSum; }
        }

        /// <summary>
        /// The mean of all the data in the cluster
        /// </summary>
        public double[] ClusterMean
        {
            get
            {
                for (int count = 0; count < this[0].Length; count++)
                {
                    _clusterMean[count] = (_clusterSum[count]/List.Count);
                }

                return _clusterMean;
            }
        }

        /// <summary>
        /// Returns the one dimensional array data located at the index
        /// </summary>
        public virtual double[] this[int index]
        {
            get
            {
                //return the Neuron at IList[index]
                return (double[]) List[index];
            }
        }

        /// <summary>
        /// Adds a single dimension array data to the cluster
        /// </summary>
        /// <param name="data">A 1-dimensional array containing data that will be added to the cluster</param>
        public virtual void Add(double[] data)
        {
            List.Add(data);

            if (List.Count == 1)
            {
                _clusterSum = new double[data.Length];

                _clusterMean = new double[data.Length];
            }

            for (int count = 0; count < data.Length; count++)
            {
                _clusterSum[count] = _clusterSum[count] + data[count];
            }
        }
    }

    /// <summary>
    /// A collection of Cluster objects or Clusters
    /// </summary>
    [Serializable]
    public class ClusterCollection : CollectionBase
    {
        /// <summary>
        /// Returns the Cluster at this index
        /// </summary>
        public virtual Cluster this[int index]
        {
            get
            {
                //return the Neuron at IList[index]
                return (Cluster) List[index];
            }
        }

        /// <summary>
        /// Adds a Cluster to the collection of Clusters
        /// </summary>
        /// <param name="cluster">A Cluster to be added to the collection of clusters</param>
        public virtual void Add(Cluster cluster)
        {
            List.Add(cluster);
        }
    }
}

Here is the similarity class than can calculate Euclidean, Manhattan, Chebyshev, Minkowski distances

/// Most of this code was written by Aresh Saharkhiz
/// Re-organized by me
/// See Code Project: http://www.codeproject.com/KB/recipes/Quantitative_Distances.aspx
using System;

namespace LSITest
{
    public class similarity
    {
        /// <summary>
        /// Finds the distance.
        /// </summary>
        /// <param name="x">The x.</param>
        /// <param name="y">The y.</param>
        /// <param name="type">The type.</param>
        /// <param name="distanceType"></param>
        /// <returns></returns>
        public double FindDistance(double[] x, double[] y, string distanceType)
        {
            double distance;

            switch (distanceType.ToLower())
            {
                case "euclidean":
                    distance = EuclideanDistance(x, y);
                    break;
                case "manhattan":
                    distance = ManhattanDistance(x, y);
                    break;
                case "minkowski":
                    distance = MinkowskiDistance(x, y, 1);
                    break;
                case "chebyshev":
                    distance = ChebyshevDistance(x, y);
                    break;
                default:
                    distance = 0.0;
                    break;
            }

            return distance;
        }

        /// <summary>
        /// Finds the Euclideans distance.
        /// </summary>
        /// <param name="x">The x.</param>
        /// <param name="y">The y.</param>
        /// <returns></returns>
        public double EuclideanDistance(double[] x, double[] y)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }

            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum += Math.Pow(Math.Abs(x[i] - y[i]), 2);
            }

            double distance = Math.Sqrt(sum);
            return distance;
        }

        /// <summary>
        /// Finds Manhattan distance.
        /// </summary>
        /// <param name="x">The x.</param>
        /// <param name="y">The y.</param>
        /// <returns></returns>
        public double ManhattanDistance(double[] x, double[] y)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }

            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum += Math.Abs(x[i] - y[i]);
            }

            double distance = sum;
            return distance;
        }

        /// <summary>
        /// Finds Chebyshevs distance.
        /// </summary>
        /// <param name="x">The x.</param>
        /// <param name="y">The y.</param>
        /// <returns></returns>
        public static double ChebyshevDistance(double[] x, double[] y)
        {
            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }
            int count = x.Length;
            var newData = new double[count];

            for (int i = 0; i < count; i++)
            {
                newData[i] = Math.Abs(x[i] - y[i]);
            }
            double max = double.MinValue;

            foreach (double num in newData)
            {
                if (num > max)
                {
                    max = num;
                }
            }
            return max;
        }

        /// <summary>
        /// Finds Minkowskis distance.
        /// </summary>
        /// <param name="x">The x.</param>
        /// <param name="y">The y.</param>
        /// <param name="order">The order.</param>
        /// <returns></returns>
        public double MinkowskiDistance(double[] x, double[] y, double order)
        {
            double sum = 0.0;

            if (x.GetUpperBound(0) != y.GetUpperBound(0))
            {
                throw new ArgumentException("the number of elements in x must match the number of elements in y");
            }
            int count = x.Length;

            for (int i = 0; i < count; i++)
            {
                sum = sum + Math.Pow(Math.Abs(x[i] - y[i]), order);
            }

            double distance = Math.Pow(sum, (1 / order));
            return distance;
        }
    }
}

And finally the same LSI class used in the previous examples.

using System;
using SmartMathLibrary;

namespace LSITest
{
    public class lsi
    {
        // this returns the formated html results
        public int MyDocColumnCount;
        public int MyDocRowCount;
        public double[,] MyDocs;
        public double[,] MyWords;
        public int MyWordsColumnCount;
        public int MyWordsRowCount;
        public string ToPrint;

        /// <summary>
        /// LISs the test.
        /// </summary>
        public void LSITest()
        {
            //Create Matrix
            var testArray = new double[,]
                                {
                                    {1, 0, 0, 1, 0, 0, 0, 0, 0},
                                    {1, 0, 1, 0, 0, 0, 0, 0, 0},
                                    {1, 1, 0, 0, 0, 0, 0, 0, 0},
                                    {0, 1, 1, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 1, 2, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 0, 1, 1, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 0, 0, 0, 0, 1},
                                    {0, 0, 0, 0, 0, 1, 1, 1, 0},
                                    {0, 0, 0, 0, 0, 0, 1, 1, 1},
                                    {0, 0, 0, 0, 0, 0, 0, 1, 1}
                                };

            // Load array in to Matrix
            var a = new Matrix(testArray);

            // print original matrix
            PrintMatrix(a);

            // preform Latent Semantic Indexing
            GetDocumentWordPlots(a);
        }

        /// <summary>
        /// Prints the matrix.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        private void PrintMatrix(IMatrix myMatrix)
        {
            ToPrint += "<br /><br />";

            for (int i = 0; i < myMatrix.Rows; i++)
            {
                for (int j = 0; j < myMatrix.Columns; j++)
                {
                    ToPrint += String.Format("{0:0.##}", myMatrix.MatrixData[i, j]) + "\t";
                }
                ToPrint += "<br />";
            }
        }

        /// <summary>
        /// Gets the document word plots.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        private void GetDocumentWordPlots(Matrix myMatrix)
        {
            // Run single value decomposition
            var svd = new SingularValueDecomposition(myMatrix);
            svd.ExecuteDecomposition();

            // Put components into individual matrices
            Matrix wordVector = svd.U.Copy();
            Matrix sigma = svd.S.ToMatrix();
            Matrix documentVector = svd.V.Copy();

            // get value of k
            // you can also manually set the value of k
            var k = (int) Math.Floor(Math.Sqrt(myMatrix.Columns));

            // reduce the vectors
            Matrix reducedWordVector = CopyMatrix(wordVector, wordVector.Rows, k - 1);
            Matrix reducedSigma = CreateSigmaMatrix(sigma, k - 1, k - 1);
            Matrix reducedDocumentVector = CopyMatrix(documentVector, documentVector.Rows, k - 1);

            // Recalculate the matrix
            Matrix docs = reducedDocumentVector*reducedSigma;
            Matrix words = reducedWordVector*reducedSigma;

            // Fill doc plot locations
            MyDocs = new double[docs.Rows,docs.Columns];
            for (int i = 0; i < docs.Rows; i++)
            {
                for (int j = 0; j < docs.Columns; j++)
                {
                    MyDocs[i, j] = docs.MatrixData[i, j];
                }
            }

            // Fill word plot locations
            MyWords = new double[words.Rows,words.Columns];
            for (int i = 0; i < words.Rows; i++)
            {
                for (int j = 0; j < words.Columns; j++)
                {
                    MyWords[i, j] = words.MatrixData[i, j];
                }
            }

            // Set counts for charts
            MyDocRowCount = docs.Rows;
            MyWordsRowCount = words.Rows;

            PrintMatrix(docs);
            PrintMatrix(words);
        }

        /// <summary>
        /// Creates the sigma matrix.
        /// </summary>
        /// <param name="matrix">The matrix.</param>
        /// <param name="rowEnd">The row end.</param>
        /// <param name="columnEnd">The column end.</param>
        /// <returns></returns>
        private static Matrix CreateSigmaMatrix(IMatrix matrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < columnEnd; i++)
            {
                copyMatrix.MatrixData[i, i] = matrix.MatrixData[i, 0];
            }

            return copyMatrix;
        }

        /// <summary>
        /// Copies the matrix.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        /// <param name="rowEnd">The row end.</param>
        /// <param name="columnEnd">The column end.</param>
        /// <returns></returns>
        private static Matrix CopyMatrix(IMatrix myMatrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < rowEnd; i++)
            {
                for (int j = 0; j < columnEnd; j++)
                {
                    copyMatrix.MatrixData[i, j] = myMatrix.MatrixData[i, j];
                }
            }

            return copyMatrix;
        }
    }
}

And what do the results look like?

As you can see the K-Means clustering algorithm correctly grouped the documents in the appropriate categories.

Recommended reading and thanks goes to Aresh Saharkhiz for sharing his implementation of K-Means Clustering.

Jennifer Flynn’s presentation “Latent Semantic Indexing Using SVD and Riemannian SVD” actually goes on to tell us how to do this. Essentially the process is the same as before however, k must equal 2. We ended up with k = 2 in our previous example however, in larger examples k will more than likely be a different number. Regardless, here we want to end up with a matrix with two columns giving us our (x,y) – if you wanted to plot these items in a three dimensional space k=3 and if you find an awesome way to plot where k=5 e-mail me. The formulas we use are as follows after we have performed SVD.

Documents = U*∑

Words = V*∑

The resulting matrices give us our (x,y) co-ordinates that we can then plot. I have been using the Dundas charting library for over two years now but the library is expensive so you should go and get the free library here since Microsoft acquired them and is a free download. And again for simplicities sake, this project is just a simple ASP.NET application.

The LSI Class:

Please note that this is almost exactly the same as in the previous blog however, here at the end of the GetDocumentWordPlots function we use the formulas mention above to load the co-ordinates of the words and documents in to a double array that we will ultimately pass to the chart.

using System;
using SmartMathLibrary;

namespace LSITest
{
    public class lsi
    {
        // this returns the formated html results
        public int MyDocColumnCount;
        public int MyDocRowCount;
        public double[,] MyDocs;
        public double[,] MyWords;
        public int MyWordsColumnCount;
        public int MyWordsRowCount;
        public string ToPrint;

        /// <summary>
        /// LISs the test.
        /// </summary>
        public void LSITest()
        {
            //Create Matrix
            var testArray = new double[,]
                                {
                                    {1, 0, 0, 1, 0, 0, 0, 0, 0},
                                    {1, 0, 1, 0, 0, 0, 0, 0, 0},
                                    {1, 1, 0, 0, 0, 0, 0, 0, 0},
                                    {0, 1, 1, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 1, 2, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 0, 1, 1, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 0, 0, 0, 0, 1},
                                    {0, 0, 0, 0, 0, 1, 1, 1, 0},
                                    {0, 0, 0, 0, 0, 0, 1, 1, 1},
                                    {0, 0, 0, 0, 0, 0, 0, 1, 1}
                                };

            // Load array in to Matrix
            var a = new Matrix(testArray);

            // print original matrix
            PrintMatrix(a);

            // preform Latent Semantic Indexing
            GetDocumentWordPlots(a);
        }

        /// <summary>
        /// Prints the matrix.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        private void PrintMatrix(IMatrix myMatrix)
        {
            ToPrint += "<br /><br />";

            for (int i = 0; i < myMatrix.Rows; i++)
            {
                for (int j = 0; j < myMatrix.Columns; j++)
                {
                    ToPrint += String.Format("{0:0.##}", myMatrix.MatrixData[i, j]) + "\t";
                }
                ToPrint += "<br />";
            }
        }

        /// <summary>
        /// Gets the document word plots.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        private void GetDocumentWordPlots(Matrix myMatrix)
        {
            // Run single value decomposition
            var svd = new SingularValueDecomposition(myMatrix);
            svd.ExecuteDecomposition();

            // Put components into individual matrices
            Matrix wordVector = svd.U.Copy();
            Matrix sigma = svd.S.ToMatrix();
            Matrix documentVector = svd.V.Copy();

            // get value of k
            var k = 2;

            // reduce the vectors
            Matrix reducedWordVector = CopyMatrix(wordVector, wordVector.Rows, k - 1);
            Matrix reducedSigma = CreateSigmaMatrix(sigma, k - 1, k - 1);
            Matrix reducedDocumentVector = CopyMatrix(documentVector, documentVector.Rows, k - 1);

            // Recalculate the matrix
            Matrix docs = reducedDocumentVector*reducedSigma;
            Matrix words = reducedWordVector*reducedSigma;

            // Fill doc plot locations
            MyDocs = new double[docs.Rows,docs.Columns];
            for (int i = 0; i < docs.Rows; i++)
            {
                for (int j = 0; j < docs.Columns; j++)
                {
                    MyDocs[i, j] = docs.MatrixData[i, j];
                }
            }

            // Fill word plot locations
            MyWords = new double[words.Rows,words.Columns];
            for (int i = 0; i < words.Rows; i++)
            {
                for (int j = 0; j < words.Columns; j++)
                {
                    MyWords[i, j] = words.MatrixData[i, j];
                }
            }

            // Set counts for charts
            MyDocRowCount = docs.Rows;
            MyWordsRowCount = words.Rows;

            PrintMatrix(docs);
            PrintMatrix(words);
        }

        /// <summary>
        /// Creates the sigma matrix.
        /// </summary>
        /// <param name="matrix">The matrix.</param>
        /// <param name="rowEnd">The row end.</param>
        /// <param name="columnEnd">The column end.</param>
        /// <returns></returns>
        private static Matrix CreateSigmaMatrix(IMatrix matrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < columnEnd; i++)
            {
                copyMatrix.MatrixData[i, i] = matrix.MatrixData[i, 0];
            }

            return copyMatrix;
        }

        /// <summary>
        /// Copies the matrix.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        /// <param name="rowEnd">The row end.</param>
        /// <param name="columnEnd">The column end.</param>
        /// <returns></returns>
        private static Matrix CopyMatrix(IMatrix myMatrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < rowEnd; i++)
            {
                for (int j = 0; j < columnEnd; j++)
                {
                    copyMatrix.MatrixData[i, j] = myMatrix.MatrixData[i, j];
                }
            }

            return copyMatrix;
        }
    }
}

Here is the code behind for the web page that displays the chart and the matrices. Essentially we iterate through the double array pulled from the LSI class and load them in to a chart series.

using System;
using System.Drawing;
using System.Web.UI;
using Dundas.Charting.WebControl;

namespace LSITest
{
    public partial class _Default : Page
    {
        /// <summary>
        /// Handles the Load event of the Page control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="System.EventArgs"/> instance containing the event data.</param>
        protected void Page_Load(object sender, EventArgs e)
        {
            var mylsi = new lsi();
            mylsi.LSITest();
            Label1.Text = mylsi.ToPrint;
            double[,] myDocs = mylsi.MyDocs;
            double[,] myWords = mylsi.MyWords;

            // Load documents
            for (int i = 0; i < mylsi.MyDocRowCount; i++)
            {
                Chart1.Series["Series1"].Points.AddXY(myDocs[i, 0], myDocs[i, 1]);
            }

            // Load words
            for (int i = 0; i < mylsi.MyWordsRowCount; i++)
            {
                Chart1.Series["Series2"].Points.AddXY(myWords[i, 0], myWords[i, 1]);
            }

            // Set title
            Chart1.Series["Series1"].LegendText = "Documents";
            Chart1.Series["Series2"].LegendText = "Words";

            // Set point colors and shapes
            Chart1.Series["Series1"].Color = Color.Red;
            Chart1.Series["Series1"].MarkerStyle = MarkerStyle.Diamond;
            Chart1.Series["Series1"].MarkerSize = 12;
            Chart1.Series["Series2"].Color = Color.Gray;
            Chart1.Series["Series2"].MarkerStyle = MarkerStyle.Circle;
            Chart1.Series["Series2"].MarkerSize = 6;
        }
    }
}

And finally here is the ASP.NET web page.

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Default.aspx.cs" Inherits="LSITest._Default" %>
<%@ Register Assembly="DundasWebChart" Namespace="Dundas.Charting.WebControl" TagPrefix="DCWC" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>LSI Test</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <DCWC:Chart ID="Chart1" runat="server" Height="400px" Width="400px"
            ImageType="Jpeg">
            <Legends>
                <DCWC:Legend Name="Default" Alignment="Center" Docking="Bottom"></DCWC:Legend>
            </Legends>
            <Titles>
                <DCWC:Title Name="Title1">
                </DCWC:Title>
            </Titles>
            <Series>
                <DCWC:Series Name="Series1" ChartType="Point" MarkerBorderColor="64, 64, 64"
                    ShadowOffset="1">
                </DCWC:Series>
                <DCWC:Series Name="Series2" ChartType="Point" MarkerBorderColor="64, 64, 64"
                    ShadowOffset="1">
                </DCWC:Series>
            </Series>
            <ChartAreas>
                <DCWC:ChartArea Name="Series2">
                    <axisy interval="0.5" maximum="2" minimum="-1">
                        <majorgrid linecolor="Gray" linestyle="Dash" />
                    </axisy>
                    <axisx interval="0.5" maximum="2.5" minimum="-0.5">
                        <majorgrid linecolor="Gray" linestyle="Dash" />
                    </axisx>
                </DCWC:ChartArea>
            </ChartAreas>
        </DCWC:Chart>
        <br />
        <asp:Label ID="Label1" runat="server" Text=""></asp:Label>
    </div>
    </form>
</body>
</html>

So lets see the result!

Now obviously you could/should probably write this a different way but it gets you to where you need to be.

I would also recommend you read in Flynn’s presentation on how to compare two words/documents by using the dot product of two row vectors. Or one could also use the Euclidean Distance Score. And if you are also interested I would recommend Sujit Pal’s blog post “IR Math in Java : Cluster Visualization” for additional reading.

Latent Semantic Indexing attempts to uncover latent relationships among documents based on word co-occurence. So if document A contains (w1,w2) and document B contains (w2,w3), we can conclude that there is something common between documents A and B. LSI does this by decomposing the input raw term frequency matrix (A, see below) into three different matrices (U, S and V) using Singular Value Decomposition (SVD). Once that is done, the three vectors are “reduced” and the original vector rebuilt from the reduced vectors. Because of the reduction, noisy relationships are suppressed and relations become very clearly visible.

So how is this done?

To start with let’s use the example in “Indexing by Latent Semantic Analysis” (Deerwester et al.) because you see this example repeated in a number of places on the web. The example in the paper says let’s take a look at 9 titles of papers that fall in to two categories “human computer interaction” & “graphs & trees”.

In this example the matrix is comprised of the word counts in the different document. The next step is take this matrix and break it down in to it’s different parts using SVD. The result looks like this:

After you have preformed SVD on the original matrix you then reduce the individual vectors. What the reduction of the vectors does is get rid of some of the “noise” – exposing the relationship between words and documents.

One question that arises is how much do you want to reduce the vectors (often called k)? There seems to be no hard and fast rule to this as different papers have different approaches/results with different values. In Sujit Pal’s post he uses the square root of the number of columns of the original matrix (m) which is then rounded down minus 1 which is I think a good method to use. It also happens to be the value that is used in the Deerwester paper k=2. The following picture shows what this looks like (please see Jennifer Flynn’s presentation Latent Semantic Indexing Using SVD and Riemannian SVD for a more elaborate example):

After the vectors have been reduced all that is required to do is take the vectors and multiply them back together again and that is it. See the result:

[Update: as one of the readers (Jorge) noted v is not exactly correct it should be V.Transpose. Please check out the “Indexing by Latent Semantic Analysis” (Deerwester et al.) paper starting on page 26 for the correct values of the matrices http://lsa.colorado.edu/papers/JASIS.lsi.90.pdf I will try to update this here shortly]

So lets take a look at the code – it follows the example outlined in the Deerwester paper and please keep in mind that is just a little class i put together in a asp.net test app that shows a html formatted matrix of the original (m) and the LSI result:

using System;
using SmartMathLibrary;

namespace LSITest
{
    public class lsi
    {
        // this returns the formated html results
        public string ToPrint;

        /// <summary>
        /// LISs the test.
        /// </summary>
        public void LSITest()
        {
            //Create Matrix
            var testArray = new double[,]
                                {
                                    {1, 0, 0, 1, 0, 0, 0, 0, 0},
                                    {1, 0, 1, 0, 0, 0, 0, 0, 0},
                                    {1, 1, 0, 0, 0, 0, 0, 0, 0},
                                    {0, 1, 1, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 1, 2, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 1, 0, 0, 0, 0},
                                    {0, 0, 1, 1, 0, 0, 0, 0, 0},
                                    {0, 1, 0, 0, 0, 0, 0, 0, 1},
                                    {0, 0, 0, 0, 0, 1, 1, 1, 0},
                                    {0, 0, 0, 0, 0, 0, 1, 1, 1},
                                    {0, 0, 0, 0, 0, 0, 0, 1, 1}
                                };

            // Load array in to Matrix
            var a = new Matrix(testArray);

            // print original matrix
            PrintMatrix(a);

            // preform Latent Semantic Indexing
            Transform(a);
        }

        /// <summary>
        /// Prints the matrix.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        private void PrintMatrix(IMatrix myMatrix)
        {
            ToPrint += "<br /><br />";

            for (int i = 0; i < myMatrix.Rows; i++)
            {
                for (int j = 0; j < myMatrix.Columns; j++)
                {
                    ToPrint += String.Format("{0:0.##}", myMatrix.MatrixData[i, j]) + "\t";
                }
                ToPrint += "<br />";
            }
        }

        /// <summary>
        /// Transforms the specified my matrix.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        private void Transform(Matrix myMatrix)
        {
            // Run single value decomposition
            var svd = new SingularValueDecomposition(myMatrix);
            svd.ExecuteDecomposition();

            // Put components into individual matrices
            Matrix wordVector = svd.U.Copy();
            Matrix sigma = svd.S.ToMatrix();
            Matrix documentVector = svd.V.Copy();

            // get value of k
            // you can also manually set the value of k
            var k = (int) Math.Floor(Math.Sqrt(myMatrix.Columns));

            // reduce the vectors
            Matrix reducedWordVector = CopyMatrix(wordVector, wordVector.Rows, k - 1);
            Matrix reducedSigma = CreateSigmaMatrix(sigma, k - 1, k - 1);
            Matrix reducedDocumentVector = CopyMatrix(documentVector, documentVector.Rows, k - 1);

            // re-compute matrix
            Matrix a = reducedWordVector*reducedSigma*reducedDocumentVector.Transpose();

            // print result
            PrintMatrix(a);
        }

        /// <summary>
        /// Creates the sigma matrix.
        /// </summary>
        /// <param name="matrix">The matrix.</param>
        /// <param name="rowEnd">The row end.</param>
        /// <param name="columnEnd">The column end.</param>
        /// <returns></returns>
        private static Matrix CreateSigmaMatrix(IMatrix matrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < columnEnd; i++)
            {
                copyMatrix.MatrixData[i, i] = matrix.MatrixData[i, 0];
            }

            return copyMatrix;
        }

        /// <summary>
        /// Copies the matrix.
        /// </summary>
        /// <param name="myMatrix">My matrix.</param>
        /// <param name="rowEnd">The row end.</param>
        /// <param name="columnEnd">The column end.</param>
        /// <returns></returns>
        private static Matrix CopyMatrix(IMatrix myMatrix, int rowEnd, int columnEnd)
        {
            var copyMatrix = new Matrix(rowEnd, columnEnd);

            for (int i = 0; i < rowEnd; i++)
            {
                for (int j = 0; j < columnEnd; j++)
                {
                    copyMatrix.MatrixData[i, j] = myMatrix.MatrixData[i, j];
                }
            }

            return copyMatrix;
        }
    }
}

With much thanks to Sujit Pal, Jennifer Flynn and Deerwester for their excellent explanations.

Recommended Reading

IR Math with Java : TF, IDF and LSI – Sujit Pal

Indexing by Latent Semantic Analysis – Deerwester et al

Latent Semantic Indexing Using SVD and Riemannian SVD – Jennifer Flynn

Check out his site Reuben Margolin