Fun with signal processing and ... matplotlib

Well, I remember overhearing a conversation one day in a  laboratory once 10 years ago.

Electronic engineers were proud to have helped their team to higher CDD resolution with a trick they published:
they would use a median filter to filter out the photon arriving from the CDD from the thermal noise. They were even very proud of seeing a very signal far smaller than the noise.

I thought to myself. Such a scam, this is to stupid it cannot work.

So I decided to play with matplotlib to test it.

First we generate white noise, then given a probability of 1/n a photon is seen, and its amplitude is 1/A of the signal.

Then, I used #python-fr to have advices on the best way to do a moving average (which is the usual filter used for filtering out signals), then I tried the median filter.

A median filter is just as silly as taking the median in a moving window on a time serie.

It cant work, can't it?

Well, it does.

Is it not hard  to code, but beware your eyes will cry from PEP8 insanity and mapltotlib style mixed with pythonic style (call it art):
(https://gist.github.com/3794506)



Moral of the story? 

 

Well, I should have to publish the code that shows why 40 for median and 75 for movering average are the optimal settings, and how dazzling fast matplotlib is.

But the real point is python + matplotlib are amazing. And that sometimes simple ideas give amazing results.

Have fun!

The startup checklist for developer's interviews

As a foreword, I'd say when candidating for a job interview for a startup be more aware of the risk you are taking:
- cash flow is not yet existing so you take the risk of not being paid or being fired;
- if the product is innovative by nature ... you are not sure you will make it work.

The tarpit of the startups

 

I have had not very much luck with my job interviews these last weeks, so I resorted to startup. Well, startup interviews are quite different than normal job interviews, a developer must also interview the startup to assess the risk he takes.

Job description

Most startups wants a developer. When you hear them, everything is done, they just need more manpower with polyvalent sets of skills. The truth is the more impressive a job description is, the more problems you will have. That is the reason why I usually get pretty inquisitive and make the founders spill the beans about what are the remaining problems. Usually, only some little details are missing; they have made a prototype that works and only  need some industrialization.

Evil lies in the details

Never ever believe a startuper saying it is 95% done you'll only have to work on the remaining 5%.

As funny as it seems, the little details are often big details. Since I have NDAs to respect, I cannot say how laughable the 95% claim often is.

Let's sum up by saying the 5 remaining percent of the job are just 5 huge percent. Like computing a NP complex problem in a polynomial time. Or having software with real time constraint distributed amongst more than one computer and OS that are no real time OS with constraint from less than .1s using UDP protocols tunneled in TCP on any networks.

Just think of any problems for which -seasoned developers- know we have no the solutions and stuff it with a lot of other problems in the remaining 5%.

I dare say that solving a high bandwidth low latency multi host real time robust distributed signal processing with no R&D team, no budget in less than 9 months is a 1% detail on a startup scale.


Evil also lies in deadlines and wages


I have no problems trying to be smarter than the average. I have a problem in trying to solve a problem that more than one brilliant team bit their teeth on. Not that I am modest. Modesty is not my first value. I am pretty arrogant. But, when there is no state of the art for a given problem, given no hard data to analyse I pretty much chicken out given a deadline, because as far as I am concerned either my knowledge is outdated (which means I will fail) or I am right on the fact that a problem is a tricky one that requires Research and Development or is an absolute un-achievable problem of computer science.

When, you decide to be a developer for a team that has a deadline, no funding and no ideas what they face and that their are entering the realm of research you are scared.

The nature of research is you don't know how to solve a problem, and as long as you have not studied with hard enough data and theory you have a hard time
-  knowing if it is possible to deliver;
- and when!

While researching you may discover that your company is doomed!

So, adding a stressful situation to low wages (usually we are talking of 40k$/year) it is quite  unfair: you feel like atlas having to wear on your shoulder the weight of the world and having a peanut as an incentive.

Why not be rock'n roll and take a risk? 

 

Would you bet on you alone against some of  the most brilliant teams of the world of computer industry (MS, apple, google, facebook...) given that the odd are against you and if you win, you will be strip off everything?

Well,  you could. Given that you have:

- other incentives;
- trust in your team.

E  pluribus unum



Usually, companies looking for the developer doing the remaining 5% already has at least 1 CTO and 1 expert. If you say that the 5% belongs to the realm of research and not engineering, you are already facing a strong disapprobation from two members of the team, especially when you say that research on the opposite of engineering is far harder to deliver in time. So you are not even hired that you are challenging at least one of the most prominent figure of a startup. Well, believe my experience these are no good omens.

The other problem that it enlightens is that the company that wants to hire you have the belief it is highly versed in bleeding edge computer science, but it seems to ignore all of the field they pretend to dominate. There are 2 problems:
- they cannot assess the risk;
- they cannot assess the benefit;
- they cannot assess the cost.

Do you really want to work or contract for a company that has no money and given some good sense as no idea how to make some?

Well you could...

The last hit ... 

Imagine that you have a solution and can save them.

Do you know you will earn nothing from your hard work?

Well it is called IP law. The result of the intellectual work from any contractant or worker from a company belongs to the company. So imagine you find a new solution to a problem (innovation), not only would you earn nothing from it, but would you reuse it would you be sued for counterfeiting your own fucking innovation.

So to sum up, working as a developer of a (small french) startup is often like taking all the risks and hoping no benefits.

What is the problem with startups? 



Some say the problem is only French. In France more than once had I this discussion with executive and strangely artists that the ability to produce something is less important  than the ability to imagine things.

Let me tell you  my truth:
- everyone has ideas;
- few can deliver them in a given deadline.
 

So if you want startup ideas I throw a bunch of them:
- itinerant coffee vendors with a light stand that would switch from subway exits  to subway exits according to frequentation;
- auto-adaptive real time delay line for signal transmission;
- power computing efficiency consulting companies;
- printing server that would change the font to save money on printing (a liter of ink cost more than a kilo of gold);
 - highly efficient dating sites (based on my experience of automated bots on website I made);
- agency for computer talent recruitment;
- women oriented soft pornographic content. 

The problems with ideas is that you need to deliver them. And to deliver them you need a capital in order to sustain you between the time you have your idea and you deliver it. If you have been ruined once as I, you don't have any capital. You just have a know how you can negotiate.

My point is ideas are bullshits if you can't deliver them. Thus here is my factual checklist to check if you can work for a startup:

Don't get mad get even


  1. how many percents of problems remains to be solved before launching the product?
  2. what are the few remaining percent made of?
  3. if the product is already launched what are the recurring problems?
  4. when is the deadline for sustainability? 
  5. are all the technical problems understood?
  6. is there a test bench for testing all the parameters of the solution?
  7. do they understand the complexity of the problems they try to solve?
  8. do they know the state of the art? (did they even fucking read the manual or googled their problem?)
  9. what is their budget for R&D? operations? Did they at least figured both problems were quite not the same? 
  10.  do they have a QA for answering and solving their customers problem? 
  11. can they reproduce their customer's problem (it is a rephrasing of do they have a lab for experimentation?)
  12. what are their growth prediction? in terms of customers (scalability) and business model (income)
  13. do they document their code/APIs? have a bugtracker? have an SCM?
  14. do they have hard data they got from their prototype?
  15. do they understand the cost of computer engineering (effort) vs hardware (peanuts) and of infrastructure (recurring costs)?
  16. do they have a realistic charging model?
It is stupid, but you can't do R&D based on the assumptions of  a self declared technical guru or expert. You need to test, experiment, try, reproduce and understand. The absence of a lab or hard data in a supposedly innovative startup is a NO GO. The absence of knowledge of the revelant concepts they deal with (asynchronous programming, real time, distributed programming, b(i|u)g data, data processing) is a clue they are clueless. And if they are clueless, they won't know how much your knowledge worths.

 So my piece of advice is:
- if the wages are indecently excessive take the money and shut up, and prepare a backup plan because you'll need it;
- else run like hell.

You don't know how fast I am actually running!

EDIT: thx bruno 

When Free Software was a nest of black swans

Listening to this song while reading might help understand the post:


I guess you wouldn't believe how free software was in the beginning: a nest of freaks.

When I went to my first Libre Software Meeting in 2000 I met a lot of interesting persons. And now that I know them, retrospectively I knew only freaks:
- people suffering of chronic seizure;
- of Sclerosis;
- of type II diabete;
- transgenders, bi sexual, gays and lesbians;
- people suffering from psychosis (chronic depression, bipolar disorder, schizophrenia);
- Asperger's syndroma;
- ADHD;
- handicapped people;
- social outcast for miscellaneous reasons;
- and... no women.

With an over-representation in regard to the actual freak percentage in society.

And this is just what I know from talking to some people in private, and amongst them are well known members of FLOSS community.

Have you ever attended  a speech where the stuttering speecher  would be not only accepted but would be granted a warm welcome?  That is the Free Software I embraced. A radically non discriminative community (© Benjamin Mako Hill).

Is this an exception? It has taken me years to discover the truth, because they were not wearing their differences as an excuse for not respecting the rules, and that might have been the secret of Free Software success: being a nest of freaks that want to be treated as everyone: only what you achieve define people, not appearance, nor social status, nor difference.

And now, as time has passed by, I think that not only this was wonderful and an example for society as a whole, but also the reason of the rise, and maybe a possible reason for the fall to come of the free software community.


Status hierarchy vs Competence hierarchy

 

I will now err on the very risky path of  giving my interpretation: I think that the 80's has been the stage of a nameless counter reaction to the 70's, and that Free Software might have been a nameless revolution that took place in the 90's. No words were spoken, still outcasts gathered.

Is there any cultural trace of what I say?

Teenage movies in the 1980's were often about how nerds being bullied that would fight back. Musical bands such as N.E.R.D.S or No doubt, or weezer or 311 would speak about it.

France were I was grown up is not U.S.A. we had our own history based on the elites and a very strong social reproduction based on social casts.

In order to have a computer degree that was socially recognized it was better to integrate selective schools. Given the actual data, I believe that 1968 revolution was a revolution in words and a counter revolution in disguise (social diversity maximum is  almost exactly in 1968).

What kind of hobby was cool for the freaks? Computer stuff: whatever your social status was, only your skills mattered and social skills were useless. I don't know if it was a conscious claim or a logical evolution, still computer hobbyists were counting numerous nerds that could shine through their skills.

Don't think computer scene was open to everyone, computers at this time were really expensive toys, so it was mainly structured around well born social outcasts from (white) educated families.



Cargo cult programming vs Digital craftsmanship

Conformism against diversity



I hereby begin one of the controversial part of my thesis. We did better at coding because we were not elite engineers and engineering was not about learning how to code, but how to behave and talk. (*Of course my statement is a bit excessive.)

In (my) university we were not as well equipped as engineering schools. We had cheap (PC) or obsolete stuff (Old unices and stations). At this time I remember that embracing linux, C bash, Perl and PHP was scolded. Real pros would use: Java, Design Patterns, UML and merise, Corba, VB, C++ ... Networking future in France was about Token Ring, ATM, minitel, Internet was a joke designed by (what a joke) PhD. We were hobbyists tinkering at most and betting on the wrong technologies.

Real engineers with a future learnt all the right theory of programming, theory of compilators, they could spoke the words of IT (commonly known as BS or buzzwords),  while we  were due to code practical problems, such as plotting our own results taken from our lab equipment. I was graduating in physics, maybe it helped. We had no time to learn any hype coding we were due to deliver with short deadlines.

We were scolded, despised, mocked but we all delivered in our fields of competences. Some became famous, some lost. Free Software was a much more selective world than any selective school in France: it was based on what you really do, not how nice you were, or how you could handle concepts or the hype you raised, or how web 2.0 coolish your blog was. And shipping stuff was the entry token. Code is not the only thing you can ship: documentation, web magazine, coordination, packaging, socialisation, normalisation are important parts of free software.

 Doing is an aristocracy that needs no excuse

The cool stuff when feeling you suffer an injustice is too fight back the only way you opponents understand: we, the freaks of a conformist society, have beaten the pulp out of the arrogant people with nice suits and soft words by delivering stuffs. FLOSS success was neither a technological or a political common fight, it was the sum of individual fighting separately for acknowledgement.

So when I see articles about: «why you should not hire the black swan» I wonder if the fall of FLOSS is not coming? Being different but delivering quality products is at my opinion a corner stone of Free Software Community.


Mega Troll: why women have so much problems in free software?



In a pretty misogyn chapter of «the social contract» Rousseau stated that women used their supposed weakness as a strength. After stating that the strongest tend to bully the weakest, he then stated that women claims of weakness  should not be taken for granted since they tends to side with the strongest.

I would like to point the fact that if you accept my assumption that free software pioneers were pretty much standing out of the conformity, and were rejected when trying to pick up chicks then women rejection as a trauma can make sense. Oddly enough, this community was made of men.

You could figure how strange people are cock blocked for  being uncool, or not hype enough, or poorly dressed or just plain different and have a mental picture of how some of the freaks may have been hurt. You could also read fantasy to have a glimpse in the mind of women: K. Kurtz, A Mc Muster Bujold, U. K. Le Guin... all heroes in feminine stories are good, because they are the good deeds of good parents. Parenthood is the key to heroism 9 out of 10 times.You don't date monsters unfit for being the father of your children.

Most heroes in male written stories are however background-less: only their action makes them interesting. (Okay, I admit that with my vision of fantasy, I'd be relieved we discover that LOTR was influenced by a women).

If and only if my assumptions are right, I say deep wounds take time to heal so there will be no immediate acceptance of women. And, to test my theory I have made up a way to test it: given feminine guinea pigs accepting the following premises:

- women should consider that masculine gender is in fact a neutral gender on internet and should hide any hints pointing towards their sex;
- women should consider they deserve nothing for being a women neither good nor bad, and they should embrace the idea that what you do defines what you are, not who you are defines what you can (that is radical non discrimination);
- on internet no one knows I am a women;
- people that were bullied in their youth tends to develop a strong personality and thus are assertive/pushy, so should women if they want to blend in if they really think they really suffered their difference.  

If nice guinea pigs tries my experimental behavior compared to a set of unbriefed women; then I dare predict they will be accepted in free software communities. And, I also can predict that all hell will break loose when people will uncover the women's true identity in real life (either strong attraction or repulsion). And for a few, given enough respect earned by their actions, they might even be accepted as the person they are.


The conclusion in music too:

The limitation of frameworks.

In my young days when I began coding, the synonym for code reusability was libraries in procedural languages and classes in object oriented languages, then it became modules or software packages, and afterwards it became frameworks.

Trends change, the sense of these words may differ but the goal stays the same: doing more in less code you share with the others.

I could do a very abstract presentation of the limitations of frameworks, but let's talk of code as if it is literature.


What is a language?



I do learn computer languages like I learn foreign languages, I like to focus on typical expressions that make the language interesting, I don't try to translate one word for one word. These are the idiomatic expressions. Some language put a stress on manipulating vectors of data like a single variable, others on being terse, others on letting you express the way you want.

A language in itself to be efficient has to be embraced as a whole, as a frame set. And the reward for accepting the language are new ideas, and new ways to express yourself. For me a programmer likes a language more than the others for a question of aesthetic. That's why language wars are so heated and opinionated.


What is a program?



A program is a story you tell in a language and some stories sound better in some languages. For instance erlang or node.js seem better suited to tell asynchronous stories. R seems better suited to tell a story about how a series of data can be interpreted as information....

A program is an essay; your customer set a stage with characters,  one or more places, some events situation he wants you to build a story that leads to the desired conclusion. In the middle you are alone with a sort of bible given by the customers with all that is relevant to set the stage, and no one cares how you do it, but you have to tell a consistent story driving the characters in the desired stage.


What is a pattern?


A (design) pattern is like a cliché in a story. When you see a hero saving a nice lady in distress you expect them to kiss in the end. Well, as a coder I mainly think this way: given an initial conformation of data and a desired outcome, I sometimes have a feeling of déjà vu and  will lazily tell my story in an obvious way (that may not be obvious to a duchess for it is sometimes crude). This is not code since words may vary. We all have a style or constraints given by the customer that may alter the possibility to use a crude cut'n paste. So patterns are not about actual code it is about a global direction.

It is like when you want to make a gag in a movie, you can rely on quiproquo, rolling gags, misdirections... These are mechanisms you can rely on, you still have to describe the situation and write the dialogue, you cannot just adapt a sketch of another movie.

Wrapping up and stating what a framework is



I see programming as writing an essay given constraints given by an editor. The editor wants us to write in a genre, with given elements, with a desired conclusion and a deadline. We could write anything in the world if we had enough time, but since our schedules are tight we have to be terse, fast, and reuse patterns and code.

Frameworks are mainly story generators. 

Do you see these TV shows  with stereotyped characters, stereotyped situations, and stereotyped development according to a «bible»? Do you remember Starsky & Hutsch, J.A.G. (and all Belissario production), New York Special Unit ...
These TV shows are to literature what frameworks are to the computer languages: a restriction of what you can do with your imagination. They were enjoyable shows amongst these ones (Wild Wild West for instance) and it is not a bad thing. Of course, some scenarists made TV shows evolve by not telling the same story other and other during the whole show and still they stayed consistent: I still have fond memories of Twin Peaks, or babylon 5.

I don't say frameworks are bad: we need them, because of the deadlines.

I say they limit our options to already existing stories. As a coder, I do think that writing a story that already exists is a waste of time. Adapt an existing story (buy a software) if you want something close to an existing story. When you make variation to an existing show you take the risk to alter the conceptual integrity of the story. In programming I call this «disengaging a framework». This as far as I am concerned is tough. I remember having to add a custom authentication mechanism to turbogears for a flash application and it took me two weeks to have it right.

My experience is the more constraints a framework imposes, the less idiomatic (in regard to the original language it is coded in) a framework is, the harder to disengage it will be.


Why disengaging a framework matters?



I remember why I decided I would never code in python one day.

It was in 2004 I was directing a topic at Free Software Meeting and since the organisation was eating its own dogfood, the website was in a CMS framework (Plone?) in python. I pride myself in understanding languages quickly because they all share common roots (the more languages you know the easier it is to learn new ones), and I had a problem: speeches were not sorted in alphabetical order but regarding the DB id. I thought it would be easy to find where the list was sorted to introduce the right ordering functions. In 6 hours I was unable to find how it was working, because it was not a language, it was a frame set very hard to grok. And frameworks are not a language they are yet more powerful, but so less expressive.

But when I tell you the customers are coming with strict situations, characters and definitive ideas of what they wanted : I lied

"Programmers don't burn out on hard work, they burn out on change-with-the-wind directives and not 'shipping'."
Mark Berry 
Customers are the biggest liars, they always want something slightly different than the initial specification, and different from what the framework you have chosen proposes. Thus you will have to disengage sooner or later.



Can we improve the situation on the code side?


Well, the situation has improved already since a long time ago.

What I described so far as frameworks are jumbo web frameworks like Ruby on Rails, symfony, Django, turbogears, Plone, ASP.NET ...

Thanks to developers such as Blake Mizerany (who inspired Armin Roacher I guess for flask) and numerous other developers we have already have lightweight frameworks: ramaze (ruby), play (java) ,flask (python), sinatra (python), dancer (perl) .... frameworks are not bad, they are fantastic tools to code less, but they freeze you in a mindset while our added value lies in our agility. Thus, the main quality of a modern framework (in my opinion) is its flexibility.

Furthermore, these last years I have met developers that have almost always coded with one framework. I notice some of them hardly knew the finesse of the language they code in (and I like to call them monkey coders). But for the others I wonder what they will become when their framework will become obsolescent? Will they still be able to code in python (ruby, PHP, Perl or whatever)? Maybe Zed Shaw was wrong, Ruby on Rails was not a ghetto, maybe all jumbo frameworks are ghettos.

Like the dealers in «the wire» speaking their slang for too long ... staying in their culture for too long and hardly able to leave their former life of dealers or junkies.  


PS thanks to bruno  for the corections
«the wire» is an amazing show
If I was a rubyist I would code web sites with ramaze
If I was a perl coder I would use dancer
And if I was a coder I would not use PHP because it is neither consistent nor terse.