Friday, March 8th, 2013 at 2:01 pm.

Data mining or Knowledge Discovery in Databases is the  process of discovering patterns in large data sets with artificial intelligence, machine learning, statistics, and database systems.

The overall goal of a data mining process is to extract information from a data set and transform it into an understandable structure for further use.

Here is a simple but fascinating example of how data mining helped dissipate wrong assumptions and conclusions about girls, and take action with tremendous social impact.

For long time, the high rate of dropout of girls in schools in developing countries were explained with sociological and cultural hypothesis: girls are not encouraged by indigenous societies, parents treat girls differently, girls are pushed to get married earlier or loaded with much more work than boys. Some  others using economic theories, speculated that girls education is not seen by those societies as a good investment.

Then, in the years 90s, came a group of young data miners who plugged into several schools records on absenteeism, and slowly discovered that girls were missing schools for few days every month, with stunning regularity and predictability.  A little bit more analysis reveals that girls were missing schools mostly during their menstruation period, and because there were no safe way for them to feel clean and comfortable to come to school during that period.

Consequence, millions of girls living in developing countries like Uganda skip up to 20% of the school year simply because they cannot afford to buy mainstream sanitary products when they menstruate. This deliberate absenteeism has enormous consequences on girls education and academic potential. — Afripads.com

In western countries and in Asia, companies and governments are using data mining to make great discoveries. We can do the same in Africa. There are numerous free tools to do so. I have collected the best of them here for you. Try it, start slowly but persist with patience. It could yield amazing and transformational results like Afripads is now helping African girls stay at school. (You can also download the MIT Open course materials on Data Mining here )

1. RapidMiner

RapidMiner is unquestionably the world-leading open-source system for data mining. It is available as a stand-alone application for data analysis and as a data mining engine for the integration into own products. Thousands of applications of RapidMiner in more than 40 countries give their users a competitive edge.

2. RapidAnalytics

Built around RapidMiner as a powerful engine for analytical ETL, data analysis, and predictive reporting, the new business analytics server RapidAnalytics is the key product for all business critical data analysis tasks and a milestone for business analytics.