E-Learning-Inclusivo (Mashup)

A lot has been written about the ways that big data has changed scientific enquiry, but as supercomputers increase in power and the tools to use them become less obtuse, whole new academic disciplines are beginning to feel the benefits of crunching data.

Believe it or not, some people even think we can forecast the future with big data. Predicting world-changing events is a possibility, some claim, if you treat society and history like a big data problem. It's how big data analyst Kalev Leetaru found where Osama bin Laden had been hiding, in a way.

Graph-tool is an efficient Python module for manipulation and statistical analysis ofgraphs (a.k.a. networks). Contrary to most other python modules with similar functionality, the core data structures and algorithms are implemented in C++, making extensive use of template metaprogramming, based heavily on the Boost Graph Library. This confers it a level of performance which is comparable (both in memory usage and computation time) to that of a pure C/C++ library.

An extensive array of features is included, such as support for arbitrary vertex, edge or graph properties, efficient "on the fly" filtering of vertices and edges, powerful graph I/O using the GraphML, GML and dot file formats, graph pickling, graph statistics (degree/property histogram, vertex correlations, average shortest distance, etc.), centrality measures, standard topological algorithms (isomorphism, minimum spanning tree, connected components, dominator tree, maximum flow, etc.), generation of random graphs with arbitrary degrees and correlations, detection of modules and communities via statistical inference ,,,,,, 

What else can we predict? In theory, any event that is not random, provided we have enough data to model the context. Examples include passenger load in public transports, availability of parking spots, traffic jams, waste production, energy consumption and revenues of a shop in a specific street. These all share a common underlying principle: use context rather than history to predict behavior.

In themselves, each of these predictions could lead to amazing new products and services. The real power though comes from integrating everything together and modeling an entire city and its interactions with people. For instance, if you can predict where people will need to go tomorrow, then you can create optimal bus routes, minimizing time to destination and walking distance, taking into account predicted traffic, weather and garbage collection schedules. In this ideal system, all services would be optimal and available to citizens at anytime. We call this new way of designing cities "Algorithmic Urbanism".

Our smart local moving (SLM) algorithm is an algorithm for community detection (or clustering) in large networks. The SLM algorithm maximizes a so-called modularity function. The algorithm has been successfully applied to networks with tens of millions of nodes and hundreds of millions of edges. The details of the algorithm are documented in a paper (preprint available here).

The SLM algorithm has been implemented in the Modularity Optimizer, a simple command-line computer program written in Java. The Modularity Optimizer can be freely downloaded. The program can be run on any system that supports Java version 1.6 or higher. In addition to the SLM algorithm, the Modularity Optimizer also provides an implementation of the well-known Louvain algorithm for large-scale community detection developed by Vincent Blondel and colleagues. An extension of the Louvain algorithm with a multilevel refinement procedure, as proposed by Randolf Rotta and Andreas Noack, is implemented as well. All algorithms implemented in the Modularity Optimizer support the use of a resolution parameter to determine the granularity level at which communities are detected.

SLM algoritm. Very nice move in this complex approach of collective intelligence.