A glimpse of code

Having a lot of time pushes me - I hope - to be organised. So in an attempt to sort my data - mostly pieces of code on Python - in order to make it available when I need it I made a new github repository where I store iPython notebook with a few explanations.

Everything is here random_notebooks and maybe it will be of some people interests, or not.

News from the space

I have been working on a new exhibition, hopefully in Berlin. For now I haven't found a place neither what I will show. But I have been experimenting, working on my images. Below is an example of the kind of visuals I want to produce, you have to keep in mind those images almost 1m by 1m big.

I'm not sure if all the images are interesting, but I like the contrast between the first on the top and the last one in the bottom.

I'm not sure neither about the color I used to frame each little planet version in that illustration. But if you are interested in knowing it's fuchsia.

A small recap to our HR friends

I can do Python, I can do data analysis (stats, modeling, machine learning, analytic and so on), I can do data visualization, I can learn fast, I can create, I can communicate, I can translate (or report information to different departments), I can write, I can connect people, I can invest myself a lot, I can team play. Basically I can do a lot and I want to do a lot. I also run pretty fast.

What I can't do is to come with miracle solutions to every problems before knowing the problems... Or it's pure luck and I should really start playing lottery then.

And I'm looking for a job.

About not being an expert as a data scientist and other tech stuffs

Last evening I did attend a joined Meetup from the Python User Berlin (PUB) and the Zalando Tech Event hosted by Zalando and offering talks on Natural Langage Processing (NLP). Both talks went well and gave two views on the topic: one on the state of the art of the tools for NLP using Python and a second more applied.

The discussions I add after while enjoying a club mate - la boisson des champions - were equally interesting. First of all I started discussing with a expert of NLP trying to explain why I joined this event and what was my link with NLP. In my very recent job experience at EyeEm I just touched the surface of NLP preparing data for Machine Learning (ML) using nltk together with WordNet, ImageNet. Actually I didn't do much of text analysis but batching word definition. In that experiment the text analysis will have come after this step and that's where semantic is jumping into the discussion. Because working with the word dictionary is one side of the problem: you have one word with its definition and often - at least with scientists or engineers - you are in the inverse configuration which is you having words when actually you want to extract a definition, an idea, an information... And I let you google automatic image tagging, deep learning.

After exchanging ideas and experiences about NLP I did continue seeping the offered mate with one Zalando employee. I was curious - as usual - to understand what it means to be a data scientist here. Because if the definition is very general - a data scientist works with data, we are not expert - it's interesting to see how many fields we - I'm one of those people - cover in our daily work. Using the same language - e.g. Python - we can go from signal processing, computer vision, image retrieval, NLP, how to deal with Databases - a year ago I wrote on the topic Databases and natural Langage graphs en stock - how to present your results to non expert by doing nice visualization and many more... So if we are not expert we need to be pretty fast I acquiring skills from various fields and/or use the appropriate tools.

Struggle for social graph and datavizzz

The holly Grail of the day
Build an interactive data visualization of my own networks where I could jump from one network to the other and navigate in time.  On the paper it sounds easy: use your own network data (facebook (FB), linkedin (LI), twitter, instagram, EyeEm...) to exercise yourself on social graph. In other words use tools from your beloved statistic toolbox (Matlab, Python, R...).

The why
Why, why and why using your own data? First reason and obvious to me, you know the data - or at least part of it - and it should be bit easier to navigate through them. About the first why bother to do that? Once again it's simple and the answer is curiosity. The more people use a buzz word in all conversations the less they understand what it means and I don't like to not understand.

Social graphs are interesting because they illustrate part of our multiple identities - this of course if you decided to look at your own network instead of looking at the interaction between people forming a group which is also interesting (data journalism loves to dissect political social network to find out who are the leaders). We don't know the same people/don't play the same character depending of the network as they describe different interactions (e.g. FB vs LI).

The reverse engineer path
The path I did follow wasn't probably the most efficient but I'm getting better every day. Plotting a social graph isn't the most difficult task. Using gephi you can relatively fast generate beautiful graphs. In parallel I took in statistic and social network analysis to refresh parts of my brain on the topic.

The prototype
As inmaps isn't available any more I ended up on another automatic solution called socilab.con that requires you to log with your linkedin account. It's nicely made, you get a graph and several score values that describe your network and which role you play in it. Sadly it is limited to 500 contacts, so if your contact list is much bigger the analysis is incomplete. But this website allows you to download this version of your contact list. And actually what you are downloading is the formatted data from your LI account under the form of an adjacency matrix. I had to clean a bit the data using Python and Pandas which make any manipulation of csv file a real pleasure.

The adjacency matrix
This matrix - if I understood correctly - should be square where both columns and rows have the same names: your contact name list. Depending of the cell value 0 or 1 you know if your contact know each other or not, the matrix isn't symmetric. It's a particular case of data, because if you look at a FB group of people liking peanut butter toast for diner they may not know each other but they are all connected by their irrational attraction to fatty cream and low safe consideration.

Where the trouble starts
It starts right when you want to access your data... Building by hand this matrix is doable but is a really silly task. And both LI and FB do make the task easy neither. You will need to play with their API (I haven't checked for twitter, instagram and more yet) to access your account and download/build your matrix.

Cyborg time

About two years ago I started with a friend - who happen to be a color scientist as well - a series of research experiment having for goal to study projector displays and wondering if we could evaluate some of theirs properties - such as response curve and color gamut - without proper measuring device - meaning without expensive measuring devices.

Said like that I admit it doesn't sound very sexy. But that were the fun come in, it is not because it doesn't sound über cool that the approach can't be fun. Inspired by the work of other researchers where instead of using measuring devices experiments were conducted using a panel of human observers. A series of images were presented to the observers. Based on their decision, to say if pairs of color patches were identical in brightness/intensity - one made of a single gray continuous value and the second made using halftone technique - the famous response curve could be described. There is so much things you can do with the response curve but I will not pursue in that direction today.

Our idea is simple and we are not the first to have had this idea: to replace the observer by a camera, a webcam. Three simple reasons to do so: for having been several time one of the observer I know it is super annoying to be part of such experiment, mimicking an eye with a camera is a nice challenge and webcam are cheap. And not to mention that opencv is there to avoid you to loose to much time in accessing the video stream of your webcam and programming in Python is pleasant.

To be short, we are re-doing a pair single color patch comparison. One patch is a continous patch and the second halftoned.  The project is called devForWebCam, it's programmed in Python and located here. The code should help to setup an experiment in order to gather data.

Pano for the people 2

I "closed" this project by adding some final examples. More precisely examples where something can go wrong and showing what it means in images.

One important thing to be able to create a beautiful little planet is to have made first a full spherical panorama picture. This panorama has to be presented as a rectangular image, an equirectangular image of ration 2:1, this means the virtual camera that would have taken this image has an equirectangular lens of field of view 360°. That is the requirement for my scripts and function that generate template file to control hugin. But you are free to distort any rectangle images with different ratios.

All the codes and guidelines here panoToLittlePlanet.

What does it mean to generate a little planet from the south hemisphere of full spherical panorama? The answer below:

full spherical image with only the south hemisphere visible.

the resulting little planet.

Enter the vivarium

Il n'a pas baissé les bras, il a retroussé ses manches, ouvert son porte monnaie et acheté un vieux thinkpad pour faire comme les grands (vieux thinkpad = vieille volvo, le design est limité, c'est carré et ca roule tout le temps, partout).

La première angoisse a été "et merde le wifi marche pas" jusqu'à ce qu'on comprenne qu'il fallait activer un interupteur discrètement placé sur le côté de l'ordinateur. On s'était bien sûr rué sur google avant.

De plus, mes petits bouts de python continue de fonctionner!!! Uno miraculo, j'ai même réussi à forker mon projet sur github, j'ai des frissons, je me fais peur.

Et puis c'est plus simple que sur mac, opencv marche de suite, joie.

Sinon j'ai installé avec l'aide d'un collègue la dernière version de Ubuntu (actuellement la 12.10 je crois) et ca ressemble plus à ce que j'avais connu en 2007. C'est quoi ces liens vers amazon dans le menu? On va vite virer les trucs non indispensable et continuer à pythoner.

Tiens si me je mettais au Python?...

... me dis-je tout guilleret et plein de bonne intention la semaine dernière, et si je mettais à pythoner for real? Ceci afin de maitriser un peu d'autres outils, de lâcher un peu de leste sur Matlab, de faire un pas vers la grande famille des développeurs du dimanche, de tâter d'un langage que je trouve à priori joli à programmer et agréable à lire tout en étant relativement efficace.

C'est pas gagné est la version polie de reste calme et ne jette pas tes ordis par la fenêtre la garantie est passée. Sur la papier c'est faisable,  on - car nous sommes deux - a déjà un truc qui tourne sur Matlab, on sait ce qu'on veut faire et par petit bout tout ce qu'on veut faire à priori fonctionne. En gros on a un flux vidéo à récupérer, à traiter, des images à créer, la webcam à calibrer, des bouts d'images à selectionner, à extraire, des différences de couleur et/ou d'intensité à calculer... rien d'insurmontable.

Niveau outils qu'avons nous? Et bien python 2.7, numpy, scipy et matplotlib pour faire "à la Matlab" et surtout openCV pour récuper facilement un flux vidéo. Sur mon ordi du boulot avec win tout va bien, même Eclipse et Pydev tournent. Par contre sur mon Mac c'est la merde et je ne comprend rien à l'installation d'openCV et surtout de faire admettre à mon python que oui openCV est là. Le pire c'est qu'il apparait dans ma liste de package, mais à chaque import cv ca fout tout en l'air.

Bref après quelques jours de recherche de tutoriels c'est encore confus. Il y a plusieurs façon de gérer des images, de les lire, les afficher... ce qui est bien, il n'y a jamais qu'une solution. Là où je ne suis pas doué est que je n'arrive pas à trouver d'information convenable et me casse les dents à essayer de traduire du Matlab (probablement une erreur de ma part). La gestion des images à N canaux pour N > 1 n'est normalement pas trop compliqué avec ndimage, mais c'est oublié mon handicape informatique et une certaine poisse digitale. Ce qui me fait penser que j'ai pas encore essayer le jouer avec des images GIS et python.

Donc je commence à y voir plus clair même si la simple création d'image à valeur constante (e.g. une image complètement rouge) m'a bien pourri la journée.