Agile : the smell of the death of software economy

How many times did I have to complexify permissions in a backend or a controller to do stuff that are stupid?

Like: letting managers tamper with the workflows, let commercial change data that were meant for invoicing, having to delegate specific authority to executant in a specific context that were not supposed to do their managers tasks, using floats instead of decimal....

And you know what ? Insanity in the organization reflects itself in specifications and thus in bugs. But with agile, I don't have the means to point out the problem is in the real world.

Giving to much power to a non authorized user in a complex context requires code that can bypass an initial separation of concerns. And these little shunt multiplies and gives a potential for a lot of problems :
  1. embezzlement;
  2. local exploit;
  3. inconsistency
We got rid of specifications in the name of the death of the rigidity of waterfall.

But, specifications were there to protect the users and make sure what was delivered was even trying to do the job correctly in respect to practices and a lot of regulations.

Since software guarantee neither the conformity of results nor subscribe to any obligation of means, software industry is right now the far west for the one who have money to create arbitrary rents on shovels.

First era of computer industry we have been making quite successful at making a virtual automata taking documents and transform, store them, and make stats and propagate them to other automates.

This virtual clerk can be used to monitor people (like a punch clock), record a human activity that can through repetition be learned a derived 'best way to' the same tasks. It just can work tireless. And we cannot see what they are doing.

We have thus created stealth virtual clerks exchanging invisible information and being part of our real world and affecting it.

We have been selling it to everyone, so now, even governments citizens and every one have  integrated the idea any tasks can be efficiently managed through opaque information passing and we can guarantee privacy.

Hence we have created an economy that can go faster and use more human resources inefficiently with diminishing returns for diminishing costs without being auditable. We created an incentive for global inefficiency.

Our economy is in a phase where in all our software we map the true complexity of any activities with deterministic process. Since computers are good at it, we created complex software for simple tasks (like leaving notes on chats instead of papers).

Our Virtual Clerk lack the most important thing human have : the capacity to adapt or recognize they could have failed their reason to be. They are virtually psychopath. And we actually forgot they were there.

They do more efficiently then humans task humans wouldn't do. They could execute an innate on the death row without a flinch with the most painless method encoded for the benefits of the dead to be. Resulting in a so called progress of humanity.

So with the disappearance of specifications large part of dark "works" are being done in the secrecy of the code and of the protocols. Like robots for manipulating markets.

Undetectable nor provable criminal activities, since the criminals are virtual and the information exchanged over secured layers. And no intention of wronging the society can be proven since the code does not tell the intended effects the author was envisioning. Elaborated invisible puppets, masking by multiplicity the true master and their intentions.

And we have a pseudo scientific revolution that makes us revert to the victorian era.

The belief in determinism. The one of the birth, and of a certain science of organization; the virtual positivism from Auguste Comte. The place of the engineer at the highest march of the society.

What is fun with my CTOs and colleague the most convinced they know everything about science, the less they understand probability. The keystone of modern science.

They even don't understand that no cause can travel faster than the speed light. Or that there is a limit to the precision we can have on measurements. Or the notion of irreversibility. Having more money seems to be helping having diplomas but may not help work hard to understand your academics.

Hence, I introduce bugs over bugs, or made software that I knew would fail deterministicly in some cases that were bound to happen. But, we had deadlines: c'est pour garder ton travail, tu comprends?

Bugs could be avoided. It is just we coders have been stripped of more and more responsibilities through the use of new management techniques using smartly softwares (ticketing, code versioning, kanban organizer....). Making it look like a fatality. Just open a ticket, that will go in the backlog and will die of boredom until we find a coder creatively putting more code at working around it (still not fixing the problem). We violate business rules of accountability? It is a non topic in a scrum meeting.

It is funny how the more our industry progress the more we have taken part in ripping other profession from the heart of their job (being able to take decisions based on information), the more it hurts to understand how it works when it happens to you.  And sometimes we forgot software are just puppets. It is not actually a software that decides who change the states of tickets.

Also, computers lack of context, they suck at making decision in unknown circumstances when it is the most important time. But they enforce Role Playing Games of a virtual hierarchy that reverts the authority and responsibilities. We believe the fallacy of the interface that we don't have the right to change the priority of a bug and that we are not in charge. They have the intelligence of an automate often piloted by a human.

Put a robot in surveillance of a critical place when all citizens are sleeping and imagine that it does not see through its captors a threat (a lot of smoke coming fast at the horizon for instance). Maybe that could be very unfortunate.

Software are good at doing in same conditions, they sux to adapt by themselves they sux at understanding how they change their environment and therefore when they fail, they fail badly and rigidly.

The problem in automation with coupled loops of feedback (basically interconnected distributed software) is we already know this sux.  We have numerous example of how it already failed in ventilation, gaz transport, electricy blackout ...

We generated uncontroled feedback loops. And the more we couple states, the hardest it is to detect, predict, fix or prevent them and maintain this.

Our modern problem of information based technology is that the automated controllers are opaque in their design and functioning. They cannot perceive, nor doubt themselves and they can be tampered. Hence, we put our lives under the tyranny of mindless automates copying efficiently eventually incorrect practices who can be tampered. They cannot see the problem in doing a task a million task per second for something that should be done once in a while. And thanks to the secrecy of our protocol and IP laws, no one can audit them.  They do not see what is wrong in letting someone change the data in the middle.

We submit ourselves to stupid rules and beliefs enforced by our software. We believe that if an option is not in an interface, it does not exists. Software makes us stupid.

It is not normal that a huge corpus of knowledge from electronics and physics are systematically ignored predicting that we are heading for a massive unpredictable collapse (snowball effect from coupled amplifying feedback loops). Like when S3 fell sometimes, or github, or whatever cloud component. And our software affect the real world, from the grids to our tax payment and paycheck.

True engineering practices are barred from our work in the name of benefits. But who benefits from the failure of our systems?

Ourselves and since we decided to remove the need for formal and technical specifications we also removed any audit trail proving that we knew : there is no crime without evidences.

Letting an industry shape the economy with practices diverging from the norm of auditing (by removing any contractual obligations of either means or results) cannot converge towards better software.

An industry as a whole not seeing the problem cannot regulate itself. It is very dangerous that we let an industry substract itself from any common rules of accountability. Financially, fiscally commercially, legally ...

We are being psychopaths citizens coding an ecosystem of unstable psychopathic virtual clerks pupetted by code or operators acting in secrecy telling the whole world everything is fine: trust me I am an engineer. The fallacy of the title.

But real engineers DO make every efforts possible to keep track of what they were trying to build, for which reason they did what they did in regards of costs of failure and their model so they can anticipate and control their creations. True engineers can explain stuff simply. True engineers have authority on blocking buggy code to go to production. True engineers take decisions in arbitration with customer and its hierarchy. They don't only follow their hierarchy. They have legal liabilities to respect complex regulations that entrust them special authority that no boss can tamper. They speak the business language of customers and make sure their interest is ensured.

Quality is not writing unit or functional tests or nice looking interface or code. It is much more checking that what was made actually fit what was ordered and conform to the common rules and practices in regulation and also the terms of the contract. Engineers make sure the costing is sound, and would try to make sure the construction can be audited for the purpose of diagnostics, corrections, conformity to the norms and arbitration. They live normally by the contract. And write test if and only if it is cost efficient and/or useful in the process of operations.

Best engineers take a lot of care making sure their product don't have adverse effect on human, like aeronautical engineers.

Most coders are no engineers, because engineering is adverse to our business models of screwing up people with complex Terms of Use and delivering product that don't conform to their description with obfuscated complex billing models. How many so called engineers can tell the costs and price of their software and will be called on trial to certify the conformity of their work? None. If you never had the authority to plan and the legal liability over your production you are no engineers.

This new generation of coders are no engineers nor artists, nor craftsman. They are just high skilled underpaid clerks that avoid conflicts to keep a good position in a dysfunctional economy that they help to reinforce. The new IT industry create a large scale tool that can and will be used for fraud because we make it possible by lack of concern for corrections.

Waterfall maybe dead not for its costs but because it was efficient.

Staging phase used to be the moment where customers could refuse to pay for non conformities.

Design phase was the moment were engineers could make sure that they had enough resources to build what was planned in a correct way.

Nowadays, budget are thrown at the task of developing products with no clear cost analysis in regard to the need of the customers. We behave as if we have become natural monopolies or administration just caring about self sustaining ourselves selfishly and that we do not have any reasons to justify the cost of our own activities. Do we live on free money? 

 It is not our software that are crappy, it is the work organization that makes them crappy by incentivizing the wrong behaviors.

It is time government remember they regulate for the greater good of the people. And the tyranny should be from the humans on the tools for the benefits of the people and not solely for the benefits of a minority that gives itself a way to escape regulations and obligations sneakily.

Computers will by nature never result in transparency. Even less when using privacy. We have to put human beings back in control of this sector of the economy if we want to avoid a dark economy based on obfuscation by incompetence and complexity to rise.

Our future of IT will logically results in close to criminal behaviors. Like tax evasion. manipulation of markets, distortions of concurrence, appropriation of the creation of others, the non respect of legal obligations, distortions of market by asymmetrical informations (linked in, indeed, ...), opaque market.

It is really time for the bubble to explode. We need a lot of the wrong doers to leave this sector, and the market could take care of it by squeezing their liquidities.

Before regulating, it would be nice if the government stopped fueling the bubble with cheap money on the market. Because this situation is a direct consequence of the behaviors of central bank since 2007. The lever effects are still huge, favoring business that are giving a huge advantage on the one with capital (even as workers) over the one with practical knowledge. It is like a conflict of interest. The one benefiting of conservatism being given an unfair advantage and benefits over the control of innovation.

Modern software industry is actually just pure money laundering from the quantitative easing. It is basically a lot of your tax money given to your boss without any solid needs of justifications of any results over investment. 

We are de facto creating the economical condition for the emergence of a kakocracy (kakos = bad, cratein = to rule) capturing at the detriment of the global economy an unfair share of value on the creation of wealth in world based on biasing systematically any competitions.  (Market, tax)

It cannot end well. Pericles called this ploutocracy and said that whatever the system is (democracy, monarchy, tyrany) ploutocracy is the ennemy to any system, bringing systematic corruption.

Because letting the interest of a few direct the world without control will always favor the rogues that will emerge one day. We still fail the same, but with the amplification of potential incompetency backed by authority. Who watches the watchmen? Code doing tasks should be regulated and leave a trail of responsibility for their behaviors in case a code does something that results either in accidents or illegal activities. 

Also money is the less reliable way to measure any virtue. Virtue comes from what we do, not who we were lucky to be. It comes from struggling because it is by failing we truly learn. Not in success. Successful unlucky individuals may rise to the top with true skills, giving the idea it is a promotion process. But the amount of works for the benefits of "objective" talent given the skew of the initial luck of being born wealthy is globally advantaging the wrong person at the good places with a result worse than random picking globally.
Yes the best of us will always have success in our system. But, at the cost of increasing antagonizing biases between remuneration and actual risks being taken.

We give incentive for building a system that fail, and Agile is one of the symptom. We build because we plan to have the money long enough and we don't care about the costs and that's all. It really smells of the symptom of an economy of grass hoper surviving from short term activitie in a global context of instability where we need ants (Esope/La Fontaine).

I doubt that in times of famine releasing huge population of glutton grass hopers on the crops is a good idea.

We need a regulation to make software auditable so people can trust our domain of activity the same way people trust planes. The pionneer era has to stop and we should prepare for a serious process of conforming to industry and legal standards in terms of design and control of the process and results based on real studies. We have to give evidence we made our best. We have to be prudent with the investment that were entrusted in our hands, because every dollars invested comes originally from the money of workers, small investors or retired persons. Spending carelessly the money of investors is a crime.

And code operating in regulated area should be given a status of a legal proxy with the exact same rules as the initial human regulated activities whose actions can have consequences and for which someone have to be held financially and legally responsible. There should be a legal matriculation of software guaranteeing their conformity in their functioning and any legal rules and that we can know at any moments who operated it and who gave the order to design it this way in a cheap way.
If we made a self driving car, I would call for every single self driving car to pass the same driving license as for human and put a contact device able to access in urgency the support team and its legal responsible physically and immediately. And if a car fail, I would like to have the certainty we can have ways to audit what happens and determine legal liabilities, fast. the legal consequences of the action of any software should be clearly established in any circumstances and their operators should always be reachable when they are operated.

If this is to costly to achieve, we should not get rid of waterfall and seriousness, but of the actors in the industry that cannot survive with basic sane legal and business obligations.

Pronetarians and peaceful revolution.

I have been reading Why Should Software Architects Write Code? and I was like hey it was our job before.

Like I noticed a pattern for more of our added value work requiring an intuition of our job to have been taken away from coders: QA, product management, higher stages of design. On the other hand we have been as coders given more and more time consuming side tasks like administrative and management tasks (such as agile), coordination, specifications that are normally better efficiently given to near orthogonal sets of skills while the expectations in terms of productivity have been held the same in an increased workload.  Generating the -proven to responsible for burnout- crunches.

On the other hand the juicy part of our job have been deported to better paid more business friendly units with less and less know how of practical software. A know how necessary in the estimation of prices and costs (time is money). Thus efficiently making business.

Project management, business analyst, product managers, Human Resources, even boss lead the work of skilled coders are not themselves coders taking decision impacting the economy without the experimental knowledge of cost and feasibility. Experimental first hand knowledge seems to be the only true knowledge that betters software cost estimation hence pricing and costing, not by much. But by a tad. Hence should lead to a tad more cost efficient business decisions.

Having people steal our jobs without skills sure hurt the whole economy. The very few businesses that succeed in IT are those whose executive have been amazing geeks themselves. And the most successful one are professional organization with all the function of the company fairly coordinate together in terms of authority (benefits of position) vs responsibility (inherents risk of the realisation).
There  is a simple name for the total non overlap of authority and responsibility : it is called  total stress.

We are facing an incredible time of stress of the interests of those whose codes and those who craft the market.

But in the first place it hurts even more to deny the true nature of creation that is both hard to deliver and which benefits of the creation are given as an incentive to ignorant of the craftsmanship who can just need to capture more coders (designers, redactors, journalists, event organizer) in their factory generating rents of IP in exchange for flexible jobs and squeeze them like lemons without regard for neither the economical nor human costs to make more money by having enough money in the first place. You can even generate the first batch of capital by spending your free time to craft your own capital. Giving anyway for this an unfair advantage to the people having time and money in the first place to have hobbies. IT market is rigged on the demand side.

Our market is rigged by Intellectual Property on the offer side. And how it favours birth and antagonizes the know how. On both the offer and on the demand level. And it makes a very counter productive systems. The regulations creates this unstable equilibrium by favouring the capital over the craftsmanship in the worst possible way possible that cannot result in anything than a counter-adaptation of the business to the market, then contradicting the very sole purpose of economy which is to efficiently handles the limited physical resources on earth. Our Computer based economy is just dystopic, giving incentive to actually use resources far less efficiently compared to sooner in our progressive history. Like a first example of dys-ruption. A negative industrial revolution that fuels its local success on being inefficient at global level. Like a bug in the design at the macro economical level.
Even among coders: they are the ones who get their legitimity by experience others by expensive diploma even though Computer Science diploma are still not correlated with any clear proof of indeed producing better (or worst) coders. Another distortion of the market leading to a double problem of over confidence of the under skilled (Doner Kruger effect?).

Regulations and perverted macro economical effects are affecting locally our work organization and it is proving to be having an adverse effect. And (dys)efficient software are amplifying this in structure adopting their work organization to our software build on wrong assumptions of the value of human in the creation of business value.

Giving the benefits and the authority of the works not to those who know their jobs (praxis) but to those who to know to make benefits out of it that have totally contradictory interests (doxein) is creating a bug in our actual world with non null consequences.

The word pronetarians can be used in this context.

And this does not apply to coders only, it apply hence for the heart of the whole "service" economy. 

And that is the ugly secret behind our modern crisis. Giving the reins to create working innovative business models to those who have interest in conservatism and lack of disruption. So the interest our bosses are conflicting our interests. And using our work to make it more efficient in the process to destroy our past progress. Plus, we are incentivized at building a system efficiently helping our bosses to get rid of us for their benefits. 

It is time we face the truth: this is a clear effect of the intellectual property economy. But also of probably more interconnected political system that like our software counterparts is having leverage effects able in worst case to amplify good and bad directions very fast.

The same reason that makes works inefficient or of low quality exists in the organization in the first place generates software that will reflect it. Also dys-organising the economy by changing the way people are accounted for their creation of value.

Software and coders model the real world.

And if IP is the root cause, since it is a global context, thus it affects all software.And also chaotic properties of our system are affecting our economy (HFT) the other way round.

We can be smart here, and also think that software affecting the world we can change the world by thinking politically the implication in refusing to code.

So now, I hereby prefer to resign being a coder in these economical conditions.

As long as this situation goes on I quit coding, because I had enough good realization to live happily ever after remembering  some smalls successes while waiting for the improvement.

Stepping down and not taking part in a poorly designed system, be it at the sacrifice of the only stuff I know how to do in life, is my share of trying to improve the system.

Sometimes, you can vote with your feet. In mass it would make the software economy unable to sustain itself with an high level of stress and attrition. And that could change a lot of things. Not only people have power in not buying, but they also have power in not offering their workforce.

The peaceful revolution we can make is to leave the market and make this economy die slowly by lack of offer and also of demand. I myself disfavor amazon and apps stores and find cheaper low tech substitutes to software functions.

I quit this market to object our actual way of coding software, and to stop taking part in this game of slaughter on both customers revenue, coders revenue, and some profession we take part in enslaving.

I did not wake up any morning in my life thinking I was the bad guy. But sometimes I go to sleep thinking it.
I accepted and made my juniors accept knowingly a job where we would be in-equitably stripped from the value we were creating transforming fixed wages into incredible levers called patents/IP rights resulting in disproportionate revenues of capital that are rents obtained preferentially by birth rights. And we did this by transforming and modeling others job or traditional knowledge in an ultra competitive way.  Competition efficiency at lower level of works to support birth biased practices efficiently resulting in de facto monopolies. Monopolies are bad. Concentration of power and money results in historical high amplitude fatal issues.

And I think I am plain stupid for having believed in being able to improve the world by building good software or using open source/free software. I did the opposite.

Hell is paved with good intentions. And our economy is suffering from our lack of questioning not our intentions at being good at heart but refusing to face the objective daily truth of the evolution of our society towards a dystopia the better we code.

The more I am good at my work, the more I hurt someone else.

This is I think what the Milgram experiment is all about.

You know, the pain was not true (Milgram never let people die). But sometimes it is okay to reason with empathy. Even if emotions are clouding are reasons. They can be a good compass in life. I suffer for having written piece of code of big brother like software making telemarketers closely monitored in a precise inhuman way. I was imagining their life when I was taking part at improving the monitors' view while degrading the UI of less paid user's side by more kludge making their work harder. Our way of building software is resulting in making the less skilled worker's life horrible.

We lie to ourselves saying the world will end if we stop coding or someone else will take our job and nothing will change except we would be miserable given the economy. But the more we invest with our skills in this economy the more we enforce this situation.

Times will be tough. Today, I courageously plan to never work again in the actual software industry. I will also selectively change for a more reasonable use of computers, software and incite people to be careful themselves.

Rethinking to work in my field is peaceful and if it inspires 1.1 other persons, I will have done my job at trying.

I wish the market to slowly change so we don't destroy the good stuff we made. A revolution made by the coders peacefully refusing to code in these conditions could be smart to avoid other less peaceful one.

And I wish myself to find a job where I have objective way to check I improve people's life, everyone's life.  Not only the one from my boss, coworkers, or providers but also from all the people, even the beggars.

I am scared as shit for I am taking huge risks on a feeling.

Wish me luck, for like Odysseus I set once again my sail away having erred too much in the arms of Nausicaa. Still unsure I will defeats the Gods and the hazards of my new journey to come for something claimed to be a  delusion. To live happily ever after, making people happy around me.

Coders are junkies, IT dealers making tax payers pay their shot

I am gonna tell you how us coders are stealing the money of the tax payers for our own unhygienic benefits.  

Every morning when I wake up, since I am born, I am bored.

Probably the curse of a gifted. I am gifted at being a loser. I am also smart. I cheated often for my academics. Not learning, but guessing what my teachers wanted us to say in opposition to what they claimed to teach us.

I learned to give purposefully the right answer, because studying was even more boring than boredom itself and I could skip classes.  Playing baby-foot and drinking beers trying to hook up with girls worthed more excitement than rigged exams when you see what to answer without learning.

So, I would tend to prepare exams last minutes to be able to aggregate the corpus of pattern faster.

However time flowing, my workhours at schools went to more than 35h/week + at least 5 hours homeworks and sport activities (between 2 and 8 hours per week)

This has been my  standard for 10 years. I would skip classes to not get hammered down by the workload, had to do presenteism at university to not be negatively noted by some teachers whereas my work in calm libraries was far more productive, and would have to burn-in before exams. With the usual "post-exam" depression as we called it. My first experience with burn-out.

But still, I got my exams, and would wake up bored.

So I chose to be a developer. It was an heaven for the bored persons. Especially free software. A place where you were free to organize yourselves as long as you produced.

So I developed my skills at producing efficiently by myself. How to write code, think of small details like config files, handling SIGHUP for reload, using syslogs, documentating, having bug trackers, code versioning, packaging .... Dealing with creativity being unpredictable and learning to not work to much. I learned very soon that fatigue/burn-out really harm substantially the quality of my own code. I also noticed the curse of the burn-in. Truth is when you are motivated you can piss awesome software that works amazingly well in small amount of time. But this enthusiasm has a cost: your code is harder to understand afterwards. Hyper focus gives you a cursed power: the ability to store in your immediate memory way more data than usually.  I would say, the burn in results in doing amazing things in awful ways. And you feel overfilled with power. It is awesome. It is the kind of code that I call important Kleenex code. I favour it for prototyping. Like a small model of what the software could be. And then, I prefer to reread it later, and wonder how to make it maintainable, if possible. And since, these are my projects, I accepted from time to time to fail. I decided that unmaintainable code was useless, and used prototype issued from burn in to evaluate if any readable/maintainable software could be written when I was in "normal state". Sometimes I noticed the complexity was inherent to the design and the complexity was unmanageable. Like a FSM with arbitrary States that can be modified on the fly without limitations. Resulting in more than linear growth not scaling up... diminishing returns.

I learned how to use burn-in with moderation.

While I was working in companies.... that were like schools, I noticed a disturbing pattern.

While we were well paid on the paper, (a lot for 35h/week) our boss were expecting us to work more per week than the contractual length. And where pushing us very often to do death march/crunches.

Basically crunches are burn-in inductor: you put stress on people, you make them work longer hours by constraint, you give them interesting problems, you excite them by inspirational BS technics, free cofee, free bier, promiscuity in a crowded noisy place, and BOOOM. Burn-in automagically kick. If and only if, you have recovered from previous burnout. Rule of thumb:
- length of burnout = length of burnin
- amplitude of burnout = amplitude of burnin,
- amplitude of burnout are cumulative.

With aggravating factors. Such as commuting time in crowded place that does not help resting. Having to adapt your schedule to regular shops closing hours... having to sacrifice physical activities that becomes a burden. Being blamed for the effect resulting from the work conditions...

It is true that in these conditions we produced more software. But crappy one, at the cost of the mental health of coders. Triple penalty for caring developers, releasing shits and feeling like one afterwards and being logically blamed for it.  Thus being thrown away systematically at burnout time.

And when in severe burnout, you are unable to be any use to society. Production time of a worker that cannot be taxed and result in healthcare costs. It is systematic loss on the economy or a nation.

Every one lose at this game except the IT industry.

Basically tax payers are paying for the systematic use of burnin.  The drug coders love, the drug companies deal, and the collectivity & the users pays for the damages it generates.

And that is when I wondered if this problem was new.

And by studying papers on the effect of work hours on productivity I noticed this effect had been noticed in 1853 in UK, and explained in 1917 by the UK army during WWI.

It is the reasoning behind the maximum 43 hours per week and resting plus the max of 48 hours weekly workhours.

It is even a rule emitted by the OIT that most leading countries in software development have signed.

What should inspire people is that heavy workload are a losing situation globally.

In 1917 UK was searching for sabotagers. Ammunition produced in the rear where randomly exploding killing tommies destroying expensive canon and lacking of killing ennemies.

In fact it took them 3 years to link these QA problem with good willing workers that wanted to over-work to help the nation produce more. The problem is human get fatigued if they never rest.

They noticed the true optimal was below 43, but considered that with a tad of diminishing return we could accept this a standard optimum. They also noticed repeated stress was cumulative over time and that a definitively broken worker was producing far less than a "good enough" worker.

Even before the end of the war USA adopted this measure.

Burnout is not a new problem. It has been identified centuries ago and we already have scientific papers, laws to protect the tax payers, the workers and the companies from it.

We just do not enforce these laws. What would be needed to solve this issue of burnout is first to rename it fatigue. And then to wonder why and how we came to forgot about something that trivial in a world of increasingly accessible information.

Burnin feels great. That is the real truth behind the beauty of coding.

IT industry working culture is toxic by design and coders are junkies loving this shit.

I used to be a junkie, but wish me luck for I have chosen to take the matter in my own hand by never again accepting to let a company harm myself again.

The social cumulative Ponzi scheme behind our society

Imagine a world that does not exist. An utopia. No a board game. Let's gamify life and make you able to build either a dystopia or an utopia.

People work to prove things like they are the best and get a reward according to rules, and the competition is fair through market/rules, hence the best issues happens to the society.

Market can be regulated (socialism) or regulated (liberalism). (no typo here, a market requires perfect symmetry of information for the buyers whatever the market, hence a regulation either by normalization (liberalism: IEEE, IETF, ISO, CE) or by institutions (socialism: FDA, SEC, FCC)) and often both. Social liberalism is the norm almost everywhere nowadays. So we have a kind of regulation based model, hence we can gamify it.

A real market works better if people learn to learn from feedbacks. Which at my experience happens after you fail your first business. Natural first time winner in business get blinded easily. Failing is very important to learn. Like heroin amazing first shot seems to be the curse of every former heroin addict : the kiss of death.  The problem with life, is you have only one. So a game could be cool if it taught you about the effect of your actions without having to actually do it. Just like a flight simulator but for life.

Failing costs money. You need to invest either in education, or culture, or experimentation, resources, or luck  to be even authorized to fail. You need money to live, and there is no unlimited credits. (kof kof, except if you borrow money that doesn't exist from the future and enslave your kids to pay your debts, (kof, kof))

If we were to model what makes a successful business man, we would have to randomly pickup a factor for the remuneration of success apart from the intrinsic quality to ship an ideal product which is fucking luck. Something randomly picked but deterministic. The inner talent. The random capacity at being good at something. And also a random context that deterministicly biases the expression of the talent. The conjunction of talent and opportunity modeled by 2 main equally random variables that are not coupled initially.

You see, here in Montréal in café concert, the music is awesome from garage band. In France shitty music makes it to the top. I mean, they work there instruments. They are more often in bars to play music than to brag about playing music.

Why aren't "better" musicians not making it to the top? Because they don't have access to the market the same way... I would say randomly. Why are the french musician so shitty? Maybe because they lack of competition based on objective skills but on random property (being born there).

How would you explain that market can reach local dysfunctional optimum with objective measure of better.

You thus have to make a model based on partly predictable success based on skills and a random part based on all the dimensions we cannot control. Basically success in business is determined by action influencing the outcome of a random dice.  Just like a stupid dice roll at D&D. Basically I claim real life can be modeled as a MMORPG based on random characteristics.

So let's try to play this game with dices and a board.

Every player on the market would be given sets of random properties : age, culture, sex, geography, family history, INT, STR.... and a proficiency skill (a generator of production) that would express the intrinsic objective genius of the person. And its contextual luck bias. Talent could be expressed as the speed at which it can generate capital per "fixed turn". Like a gifted musician can produce a good song ten times faster than another one, and in dysfunctional culture (so it has to be updated according to the context) one hundred times faster. And a good musician may be a good sound engineer, and probably better than a randomly deaf person.

The context is the place, the time, the age of the players, and the biases. It changes all the values of genius deterministicly by influencing your probability of sucess per dice trhows. But intrinsic genius growth would tend to be kind of chaotic according to the initial conditions. It is your capacity to generate HP.

Then, there would have to be Hit Points or ... Capital. At 0, you don't play anymore and you are condemned to watch the game until the end, helplessly unless the other players decide otherwise, by transferring HP. Every transfers has at least a fixed cost.

The game would begin by choosing an hidden card taken randomly called "life goals" taken from a pile randomly specifically tailored dream according to your context. The most common being I want to live a life that favours me. It is called competition. Others would be, I wake up being mere theresa ... Cooperation. And Others would be I just want to play because I am there, and it could be fun. Or I don't care about the game, just leave me alone, games are BORING. In fact it would be not a card. But the reality.

You could with enough point in the game actually buy the card to leave the board game you had been forced into.

The problem in this game, is that people cannot chose and they are forced into it because of the 0 limit also called poverty. They can be sided. And it is not a game, it is a social experiment on the submission to the authority of the rule. Like Milgram and the SF prison experiments, but on our submission to our own culture. 

Actually nothing prevents the hostage of flipping the board and say fuck you. Nothing prevents the over player to stop the pain of the game and go take a pizza.

But, if I was to make this game I would make it painfully realtime slow. Like you need 7min to cover any year after the starting point. And it requires hours before you can roll the dices triggered by decisions taken years before.

I would also enable people favoured by luck to be able to rig the game to make it funnier. You could even invest in tweaking luck in your favour.

Like education could grow your revealed genius (which expresses the efficiency of your genius according to a context). You could buy extra turns like going in schools with influential future business man to higher the probability of rolling dices in the section finding an investor. You would level up faster by buying DLC bonus.

You see, this kind of late game advantage based on amplification of small increase of stats is exactly like leagues of legend. Ask any players, it is part of the game to choose wisely and to use this to reinforce your natural skills. And cumulative effects are awesome. Even if there are based on random events (like you being better at farming then your opponent, or your kill ratio).

You could also influence on global variables on the side of the board that would be random stuff. Like sex discrimination, fashion, the positive or negative impact of culture on the access to market according to your context. Like a clever LoL player.

Now, imagine you have a second sets of guinea pig spawning randomly in the middle of the party to relieve the players. We will call them kids.

They can share with less friction in HP. They cab transfer HP with members of their family. By either solidarity or theft. People could decide to throw dice for either transfering or stealing money. 

Family is anything the random game would find the most probable according to your context. Even a carpenter could be the father of a son that is not his. Still actions are always taken by random dices on predetermined success table varying with your initial random throw. But now you can influence it within game negotiations.

But, the wealthiest characters at some point of the game have more probability to change the game thanks to the accumulation of advantage according to time.

So you see, I have played enough monopoly and diplomacy game to know what this game would do without even DLC.

It would make every one throw at each others throats very fast.

Okay what are the irritating effects in this gameplay?

There obviously a social pressure for playing games. And some people sure have a lot of fun.

But with less capital in game initially, you know you will play less in real life. IT IS BORING.

The reason to be bored? Random luck at initial dice throw to chose your characteristics determine your ability to play.  And the newest players are guaranteed to have less luck than the first generation. People enter the games with substantially less luck to eventually play the games but are  forced to stay at a table event as painfully boring as a family dinner where everybody else have fun but you without having the right to open your mouth.

And at the beginning of the game it is fun. You can make a self taught kids from a shanty town wins the game against a king. The fun in games is about going against the destiny. It is funnier to win when you came in unfavorable position. And that what are all entrepreneurship story telling about. Winning against the odd. Proving by good choices you can influence the outcome in your favour. It is a positive self boosting ego game of competition. And competition is sure fun.

But,  this game will cumulatively force more bored players to be inactive at the side of the tables while the rules get increasingly in their clear disadvantage.  The more the game will evolve, the less the capacity of incoming late players will be driven by nothing else than luck, thanks to the mechanism of frictionless propagation  of the HP between families.

One of the winning strategy is of course when you are still playing and in capacity of influencing luck (genius with poor initial characteritic or the opposite) to make gaming alliance based on extending the notions of family and trading your genius for luck, or luck for having a genius.

Hence creating factions based on common characteristics : gender, age, nation of birth, of location, religion, random background point... And you create a coupling with luck and genius. Late player game will have their initial luck being overly amplified and your genius factor will be useless. The only thing that will matter will be the diplomacy game you are forced out because of lack of HP.

And at one point, like in diplomacy you do know that to make the new family emerge, you have to prepare privacy and the possibility to make alliances that might include theft of your own "initial" family to inherit characteristic from a new blended family. Yeah, betrayal is being more and more incentived and exacerbated over time. They are game incentives to betray more in frequency and amplitude over time.

You could imagine a whole generation of post war conflicting families (the liberal and socialist) deciding to base the payment of retirement at the detriment of their own kids. And then their kids by weired propagation of consequences having to betray them later (in a its you or its me fashion).

You could imagine a lot things. Those who played diplomacy knows how this kind of games destroys friendship ans solidarity in real life.

So the more the games will evolve, the less fun there will be for a vast majority. But for the one still in competition the game would become more intense and fun. And betrayal will become more common. And some people will grind their teeth preparing to betray in order to play a little. Higher betrayal for smaller gains.

There will be players forced to stay hours without even given the opportunity to play a turn.  Just because of a random initial choice. And attentive observers will notice that the game will be changing the players rationality in real life.

So you see, sometimes, the problem is not about the players, nor about the game itself.

It is about getting lost in the game and not thinking about the effect not on the personae incarnated but on the human forced to play this game.

 The more this competitive game evolve, the funnier it is and more intense... for less and less players.

I think time is the accident of the accident. And that some people have less time in their life because they are prevented to play the game of life. Because they are at the side of the table seeing people having fun and being prevented to run at the outside whatever kind of other games they may wish to play. It is not the game that is bad. It is forcing in pretending to love this game. Life is the result of our meaningful actions. By not letting people play the game of life, people are denied life. Randomly.
And I think an actual games based on our actual world would prove any hardcore players that it is not gaming that sux, it is the rules we make and how we are loosing the fun of a tad of competition.

It is not how we play nicely a shitty game that can bend this game, it is is in thinking we are forced to play this game in the first place. The game can be either influenced from the inside and it can be fun, but also, you should also be given the right to not play a game. And we should remember we get modified either positively (3D orientation) or negatively (???) by games. They can effect us.

However we could avoid the systematical cumulative boringness of the game by removing the amplification of randomness over time. Pure lottery is not a fun game to play when you have no tickets a glass of water and are forced to see people having super prices, wines and meal while you are forced to stay hearing boring discussions about how the game is fun by tweaking on small diminishing part of talent.

Basically, private property and inheritance should be capped at an acceptable level of randomness. Because a too deterministic game would be boring too.

I think we could teach a lot by making a board game for all kids in the world that would help them understand a good model of their life to come. GURPS and the creator of diplomacy are my two objective choices with Gary Gigax for this task. Just to be sure to have unicorns and troll eventually and some dragons.

Not only to depress them a tad, but also to give them hope in understanding how that they can rig the rules to increase the maximum fun for every players in and out of the games. I think that teaching kids a game that result in improving the game, teaching them how randomness and potential can influence future could be a good idea. Teaching them about the boredom of arbitrary world based on purely deterministic rules does not make funny games nor the equally boring world of pure randomness. There has to be a balance in the rules. But a constant for all kids' heart : it is funnier sharing the fun. It is a kind of a way modeling in physics, we just imagine something and validate it could work by reproducing in experiments and checking with the real world. Game should help in testing life's choice.

I say to everyone that people should accept the intrinsic unfairness of the randomness of the world as a given. We should value it. We should just maybe think of controlling it a way that makes it funnier for every players. We should accept for the sake of fun a part of symbolic limited randomness in socially randomly fixed competition where everyone could shine based on their random "talents". Like a nice free kermess for kids or a mario party where you could always choose not to play. 

We could make a better society if we remembered the fun we had playing games as kids, and what it means having fun together. 

Without regard for genius or randomness. A game where even the unluckiest would have fun.

But nowadays, the game board is increasingly boring for most of the players except a few  one chosen by luck reducing the numbers of players more every turn.

Traité de noire cuisine

Dans la bataille des chef le héros fait parfois des plats qui ont l'air horrible et évoque des tentacules perverses violentant ceux qui les goûtent.  Oui! J'aime comment les animés japonais donnent une approche saine et réaliste des choses.

C'est depuis devenu un de mes défis personnel de tenter de trouver la plus simple expression de l'horreur en cuisine qui débouche sur un résultat édifiant.

Et j'y suis arrivé. Un plat que mes frères et moi faisions étant étudiant: le bœuf à la citronnelle.

Une bonne quantité de viande que tu fais rissoler avec des graines de sésames, de l'ail et un peu d'échalote avec de la sauce de poisson fermentée, une bonne cuillérée de pate de crevette fermentée, et de la citronnelle en grosse quantité, ainsi qu'un peu de sauce de soja fermentée.
De l'autre dans ton rice cooker avec du riz parfumé au jasmin tu rajoute du lait de coco.
Un terre mer avec des trucs aussi horribles et tous plus pourris les uns que les autres, mes amis c'est tout ce qui peut horrifier une personne saine.

La pâte de crevette embaume tout l'appartement comme une rouquemoutte préado qui se néglige sous la jupe dès que tu ouvres le couvercle. La sauce de poisson n'est pas non plus sans remugles putrides. Et la coco un peu obsédante. La viande au final à une apparence grise noire pire que la soupe au vomi servie dans un mc do. On dirait de la chaire de zombi avec des insectes dedans.

Mais telle la lumière dans un puit profond emplis de sensations négative, le goût perce et vous emporte loin de tout ça dans un goût chaleureux avec toute la largeur du spectre gustatif. Les goûts se complètent et sont étrangement doux et chaleureux. Le syndrome de Stockholm appliqué à la cuisine.

Je crois que la cuisine moderne qui fait primer l'apparence ne comprends rien à la cuisine. Je décrète la quête du plat le meilleur qui envoie le plus de mauvais signaux  mais que les gens aimeraient comme l'objectif de la noire cuisine.

Il y a un vrai plaisir à surprendre les gens en cuisine.

Et je crois que cette recette simple mérite d'en être l'étalon.

Of mastery vs expertise and of the boxing geek

Loved this

But I learnt boxing not martial art.

And I think he is all wrong to trying to change people by antagonizing them.

The too long too read is what we call experts are idiots in apprenticeship like young monks in the story of the Tao. And as a good monk he blames it on the person not having the enlightenment of the religion of Art or programming vs the Technique/expertise of programming.

The martial art bull crap that made me stop karate-do, anything do in fact including viet vo dao.

I was a nerd as a kid. And I was bullied. So I did the logical stuff. Learning to fight.
But martial art were not fighting. To ridiculuously codified.

Do you have nice salute and respect from your opponents in the street? Do you have intimidations, tricks?


There was ALWAYS with the tenant of any martial art of trying to prove their superiority with an attack situation where they could finish you in a death grip with the finishing falcon punch of their kind.

Then, I learnt boxing. French boxing. Derived from the street fighting.

And then they did not won a single time their testosteron concept in which they would assert an alpha male dominancy by tricking you in accepting false condition of fight.

My only trick, was the one that works. Feinting to test your opponent and protect yourself as long as you have not judged your opponent. Be fierce, defensive, and always hiding your true intention so that you can surprise an eventually stronger opponent.

Yep. Just double your fastest direct at longest distance you are able to connect, and when you can connect hide the true intention of connecting an heavily balanced direct to the tip of the jaw that would ensure fast lucky victory which is a good idea when you don't know your opponent.  Always aim at defeating better opponents by any mean possible. A good ultimate boxer was the ferocious young myke tyson.

And then he became confident.

I have survived a lot of true fighting situations. The basics of surviving is trusting no rules.

Most aggression situation I have been facing were always due to a confidence in their numerical superiority.

On a ring in 1v1 conditions, french guard is risible. That is why competition uses the english guard.

In the street, taking a good habit at not losing information of your peripheral vision is an hell of a good idea.

When people are distracting you by speaking loud and seeming impressive, you should really take a stand as soon as you begin to see they are trying anyway to make a situation of superiority that did not existed.

In the cité speaking loud is a way to call for reinforcement to generate the situation of superiority. The same way speaking with the loud word of authority calls for the forces of order elsewhere.

Most experienced fighter know that until the fight is made, the outcome of a fight are really violent and uncertain. And fighters that want to become legends by staying alive the longest reduce the uncertainties.

Either by artificially rigging the competition to a less uncertain outcome in their favour, or by avoiding situations in their defavor.

Me being a nerd, not very athletic, but surprisingly good on a  boxing ring, I learnt to run like hell as soon as I was seeing opponents creeping in the back.

Thanks to the expertise of my maître d'arme.

Thanks to boxing I have learned to run away like a coward. I learned there was no pride in going headstrong to a defeat that was planned. I learned also that one of my relations "from the street" I appreciated a lot died of being proud ... "on a ring" in Las Vegas.

 It is not the violence of the streets that killed him but the one of the ring.

I don't like boxing for its guidance in life. And that's where I hate the bulsshit crap of martial art pretending to be something it is not.

Boxing is about staying a living nobody rather than become a dead legend. 

For the ungifted like me, there is no art in staying alive. There are tricks, and learning tricks.

Learning to punch vast, move fast, look fast, stay alert, and be scared like shit.

Sometimes I was scared to put my gloves one. Scared. Like hell. But repeating the same moves, learning to spare, physicial conditions all tricked me into confidence.

And I learned that fighting requires good teaching from masters. But teaching are just tricks in making you have confidence... Because the ultimate trick was that in a fight you had to look like nothing that was in a book to be able to fight efficiently. Else you can be anticipated.

The biggest trick about boxing is learning by the book, and then understanding that the book was a very good joke about throwing away the book at the end. And that fighting is about fighting against the rings and not the fighters.

What about the joy of sparring with friends. Would you go with a friend to the point of probably finishing with one of you in the coma ?

Well, if you love fighting you may like to fight ... friendly. And be able to say stop when the opponent made a point.

We do not need to wait until one on the kendo practioner and the canne d'arme practitioner lost an eye to see who is the funniest. Needless to say, moving in straight line is not very efficient. You can stop at first blood, or when wood makes a loud knocks on the bones :)

I don't know if I would survive out of a nice garden and in a friendly situation a fight. I hate violence. Neither having to endure it, nor giving it appeal to me. But I love boxing when it is friendly, because it is a place where we make confrontation codified, non lethal and made on anything but merit.

That's the joke of efficient street fighting. You may dislike violence, but you begin to enjoy the fight. Because you have a ring that makes it fun ... between practitioners ... without a public.

The violent part of the ring is not what happens on the ring. I do love to watch some match. But most monetary incentive in sport sometimes make it look like a boucherie where people are more interested on the blood on the screen than the good health of the athletes. They don't seem to have an empathy with boxers on the ring wishing to show their best boxes, but with the violence. Violence is not in boxing, but in the eye of the managers that are setting the ring.

I think the mastery versus expertise difference is indeed existing in our jobs too.

Like leveling up with experience points versus Dowloadable Playing Content (you buy your superiority). And like how we are valuing more DLC behaviours versus experience built behaviours.

There is an aristocracy of design that is brutal. But is not violent. The same way boxing result in brutal results but the violence comes from the context.

Some people are confusing the brutality of community management based on results with violence. Like Codes Of Conducts flourishing from social experts coming for their share on the back of the coders.

Did social science never had been curious about how free / source open software community is by a lot of standard one of the most diverse in the world?

Mono cultural on computers, but shared with hundreds of countries, type of government, political opinion, religious opinion, "races" and sexual orientations.  Except for free software, NO QUESTIONS ASKED. Radical non discrimination in the realm of craft. And CoC is about asking questions all the time. That are non relevant to producing.

As if having an obvious common shared bias could erase the other biases. And as far as I am concerned, it works. Not always at my liking though. But it works better than corporate programming.

We saw the raise of the UFC/MMA federation called IT corporation and startup is making the ring violent. Like :

Street fight has no gloves, let's remove the gloves ... and the shoes, and the capacity to run like hell ... 
Okay, unless you are not a homeless in very special condition, in France you have shoes. You even have clothes... that you could even use as weapons....

The MMC/UFC makes fight more realistic by pushing the vulnerability... No serious... In the street, in real boxing, not any serious fighter will go for increasing his vulnerability compared to the real situation.

You act accordingly to the situation and you adapt and not fighting is totally okay, even if you have something precious to protect, you run away.

Expertise is about making developers more vulnerable to the external context.

It is about selecting the best athletes, pushing them on a ring, having a rigged organization and generating reasons of conflicts that do not exists in the first place to have people throw at each others throats.

The no poaching agreement makes developer having to have an artificial loyalty to companies to keep their job. Companies with their paradigm of coding/framework and BS crap they exclusively share or adhere to. In weird extreme polarizations.

The holy grails of IA, machine learning, Agile & scrums, Go/Rust, RDBMS vs nosql .... whatever you want you name it.

But these fights are just a ring that had been set. I do think that IA is overprimising and will not likely to be deliver because the craftman problem is about just building more robusts architecture. One that know it can fail even when its inventors said the opposite.

All these tools are like combat technique. They are nice and must be learned by the book.

But the problem we have now is not a problem of knowing new fighting techniques in a specific context, but much more focus on winning the fights really at hands here in real conditions.

More complexity and dogmatism thrown away at solving inherently complex architecture will not result in more robust system.

Coding and fighting are alike. There is only one proof of being a good one, it is by first surviving and there are no diplomas nor easy way to prove it other than dealing with it. Good coders might not think the way you want, but that may be the reason why they are still efficient coders.

And sometimes there are days the technique we are spreading are inherently the wrong one.

Where the tyrany of the expertise make everyone blind about the problems at hand.

Politics does not mix well with code. And an organization big enough will put its politic in its code.

An organization are for me buildings made of politic. From the vision of a designer emerge an organizational unit that is consistent in a corporate culture that becomes good at probem solving.

This is made through sharing an informal order that ignore itself: culture.

Like the belief that we can solve an initially screwed idea by throwing more stupidity at the initial stupidity?

Simple question, is it worth caring about timezones?

It is a fight already lost. The TZ format try to solve the situation to the bad initial conceptual way to solve the need for a universal time by throwing even more confidence in persevering in the absurd decision of agreeing to disagree. Nice.

It is not the physical nature of time that makes us unable to synchronize. It is not the computers. It is not the coder. It is a fucking XML file...

That is the result of very well intentioned person updating the changes in the time zones based on publication of offices reporting that will, may have changed the definition of time locally.

Well the first problem is since people have been confusing the books and reality there is a nice story about this in a book.

It is called the Babel tower.

It is not the unification of the data format that will improve the world. Or more technicity. Today the problem are probably in the handling of an overly connected multicultural world.

I think the bug in our coding might come from our cultural bias in what we are supposed to be versus what we really are.

A conway law, code is reflecting the way the organization are evolving.

And the more we coders are successful the more the powerful and economy likes us. We are tools too. And if our corporate professional culture is shifting towards expertise (LDC) vs mastery (XP), it is not only because we eat our own dogfood, but it is also because it is part of our culture to enforce it. We are trying to be the ideal user we plague our design with. We are trying to win hard on ring of the idealism and the belief in engineering/rock star/frameworks ... while the real world is made of human failures.

We focus very hard on imposing a view of the righteousness of engineering/our techniques while our product decrease in measurable quality. Conformity to the real condition of use.

Good coders are not overconfident. They doubt because they are scared to lose. Because good reflex comes from healthy fear of being crushed.

But they move. They are confident. Not only because they trained. But most importantly because they survived. And sometimes they failed. And they also learn to stand up again. To improve and gain confidence, but not too much and remember how much defeat is painful.

More prudent, yet aggressive... Displaying confidence in your skill, is darn stupid for someone who survived real fights. Over confidence is what put you in trouble. The first user from which a design has to protect itself is the designer. So good designer should accept they can fail and doubt themselves.

My dear expert dealer of bullshit crap, you maybe the best technicians I ever saw out there. You are impressive.... I admit these techniques that need a lot of time to master are impressive... if ever you are in a tank that was ready to blast a poor bee passing by and call it an efficient technique against a poor bee passing by.

 But you are as impressive as romans senator babbling about the techniques to be the most efficient surviving gladiator in the Arena, not seeing the gladiators wiped every months. And as Spartacus have been proving, the best way to not lose a rigged fight is to rebel against rigged fights when you don't want to lose.

I loved coding. It was funny fights, some of which I won. But now, I am throwing away the gloves.

Search engines, please add temperature to your settings

The web is big. It is a network which value is to be connected. And we need an index to access it, else information are lost. It is called search engine. And I think they function to well.

Okay some engines are plain unusable. I am talking about the good ones.

What is a document's meaning?

It is at first approximation a vector in a non orthogonal multi dimensional base constituted by the invariant form of words with their occurrences.

This vector points to a direction. For describing this direction we use "key words".

You can visualize it as a transformation of a whole text into "the smallest canonical non reducible key words" that are idempotent to a bigger text. Like a mapping to the space of sets of words to a sub space of set of words. Key words forming a new base to express the meaning of thousands of words in a synthetic way. 

A documents contents can be easily expressed in a base of key words that are "strong meaning full words". You can thus reduce language and ideas without to much loss of meaning.

These vectors can be measured and you can make normalization, cross products, scalar product.

A scalar product is a projection of a vector on a vector and it results in telling you how much time vector A is compared to vector B. Hence, you can after normalization sort easily and compare text that are similar to the key world ideal texts. You can also "compress with loss" a text in a smaller base  made of key words. This what being a base is. A reduction to the smaller set of dimensions that are orthogonal. A reduction of the degrees of liberty. Geometrically, it makes sense.

Given the fact we have very fine tools in Euclidean geometry, with 2500 year old knowledge it is a very convenient way to represents text.

There are some caveats of course.

At the opposite of school geometry, the base is not complete... language are not all constructed the same ... there are more than one form, ambiguities ... This is what NLP deals with. And it is freaking harder than doing geometry. But I am focusing on geometry right now. I consider NLP as an accidental problem not an essential one on this topic.

The meaning of a word can change according to the context meaning that "diagonalisation" requires to sometimes degenerate a dimension (word meaning) in more than one according to the other words.

Words have a little uncertainty in their meaning. And a small step for an algorithm is not a step for a staircase.

So... how do you actually make the magic of compressing a 10k word document into 1 or more keywords?

The way it is done is by taking human beings that are very good at tagging text and let them define the keywords for corpuses of text. And learn. I guess machine learning automate this process. You can by using enough tagger "diminish the bias" of the human taggers with statistical treatment used in everyday experimental activities. It works.

You can  make a statistical analysis to then determine according to the input what is the separate probabilities for one or n orthogonal dimensions (made of a linear combination of words or a single one from the input text) to appear when a given keyword is given. Xhi² is a great tool for this. You measure positive, negative contributions and you also for each dimension considered check it is not random. For instance "the", "that", "a", "an" tends to not be correlated to any keywords so you can filter them out as not being part of any basis of any keywords. You diminish the degrees of liberty without loosing meaning.

You deduce from this or other methodologies from a learning corpus ways to guess keywords from frequential analysis. (text to keyword)

Of course, you can use meta data to change the occurence (tittle can be considered more heavy than words in the chapters/section/paragraph)

Then, you can just do cosinus similarities from the sets of "ideal documents" triggered by the keywords matching score using distances. Distances following the imperative properties of being defined normed and positive. So you can actually choose other norms than L2 (classical euclidean norm).

It gives you a relation of order thus a ranking.

Until now, I am fine with this.

I guess machine learning comes into handy for over industrializing this.

However one thing bugs me. As much as the difference between precise and exact.

Feedback loops... with amplification.

S = - k . ln(O)

What makes information is having the less numerous  more relevant choices being shown to you over the whole corpus of information. The "first page" accuracy.

Basically search engine relevance tends to minimize absolutely the informational entropy. Which seems a good goal.

If a kid ask for a recipe of a cake and fall on porn this is not cool.

However, because of using the "network/social/link/domain" context, we introduce a feedback loop based on how much "other people" rate the keywords validity. And without being a wizard, I guess mathematicians already guessed that collecting data on the "personnal" context of a user help increase the relevance based on what you expect and what your social context tends to find relevant.  And it is cool too.

If I need to do text processing, it might point me to the "state of the art" if my neighborhood are professional. In a professional context it standardizes the education. Leading you to stuff like stackoverflow where discussion happens and letting you avoid a lot of pit traps. To be honest I don't know if search engines go as far as using sociogram as an input. But, that would increase the relevance of keywords in a given social context. 

And, for instance if I ask a keyword for something ambiguous on which I am biased, it is better for increasing my liking of the results to show me what I like initially.

Just like when you go on youtube ask for "joe dassin" and when you are a metalhead (like me) youtube will show you on the next suggestions a lot of joe dassin and metal. Not rap, or traditional music, or whatever. Just metal. (it is much more like a moving average with a weight  decreasing over time to be honest, but still this is accidental not essential).

It is indeed what I like and I am often pleased with it, and I do indeed make some nice discoveries.

It also reinforce my biases with time. I go on youtube sometimes to be surprised, to discover stuff.

And I feel cornered into a caricature of my own self.

And I fear that most of us of get  reinforced in our own biases. But these are just feelings and theories and vague intuition. Nothing tangible.

I guess with time and enough data about people's query over time we could measure if my hypothesis are real or not. We could measure the evolution of the musical choices and diversity of "patterns" in playlist according to the age of persons over time. (melodies, arrangements, artists, ....) and we could influence people's culture.

Clustering of opinions reinforced by social networks.

This one is simple. Some people don't want to change. Some people don't want to hear the earth is flat, other it is patatoidal, and some spheric.

Me, I love Sir Terry Pratchett's Disc world and Erasthothènes and watch NASA pictures of the earth. So I am okay with all the possible shapes of the earth. Even the ring shaped earth from Niven's SF.

However it is not everybody's case, and  some people with biases prefer to concentrate in clusters of reading/writing that are mutually enforcing belief....  like some conspiracy theorists.

For instance we all fear propaganda from terrorism on the internet. But how does it happens you never randomly fell on one of these sites, and oppositely how can this person never get in touch with your culture? You know they exist, but you never had the occasion to speak with them and magic of humanity happening sometimes help them turn into better persons. You could also fell for their idea to be honest. So should we be scared?  Are some people irreversibly bad?

I am a great fan of Periclès. He used to say "polemic is life".

The world of progress (as opposed to immobility) comes from ideas not words.

Words are imperfect media for ideas, because ideas are grey, intangible, a moving target ...

And for this moving target to progress, it requires dialog/exchanges that are not always comfortable. Yes basically I say polemists (called trolls nowadays) are a necessary evil of all progressive regime. Do we need progress? Tell me : is the world in a trajectory you like? Is global warming cool? Are wars cool? Is terrorism cool? Is the increase of pollution, poverty cool?
Well, I don't benefit any of these, so my own personal contextual selfish answer is no. I want society to progress. Can I do it alone? No. So I have to able to be in touch with other people and dialog.

Making people see what they want above all at my own personal opinion (I share with myself) goes against the acceptation of diversity of points of views and dialog.

Search algorithm will get all the more precise that the feedback loop will  reinforce the contextual meaning of them.

But exact is not precise. pi = 3.14159 is precise. pi = 4 [+- 2] is exact in Euclidean geometry. pi = 4 is both exact and precise in Taxicab geometry.   

A potential solution to this unproven problem ?

In multi agent simulation based on physical statistics they used to model people's rationality in accordance to Fermi Dirac or Maxwell Boltzman distribution of energy. As if "economical agent" were rational but for modeling the uncertainty/irrationality of human behaviour they would add a factor temperature. Something saying : well there is a clear advantage for agent X to behave this way, BUT you never know. This temperature factor could vary more or less. A parameter you could set in accordance to real world observations. Basically you'd replace a fully deterministic algorithm by one tainted with some randomization. The "amount" of randomization being related to its physical equivalence of temperature.

In some model magnetic model could be used to model the influence of the neighborhood. Sometimes positively (better use the same software as industry is demanding) sometimes negatively (I don't want to wear the same shirt as my neighbor).  

What I loved was a simulation on the behaviour of the fish market in Marseilles.

They had a simulation that basically validate an experimental strategy used by buyer that was to be loyal to ONE buyer (because you get discounts for instance), but sometimes explore the competition in case the remaining competition either increase its competivity or your buyer decrease.

Trust you are right, but check.

These simulation were not the true world. Sometimes they were matching experimental evidences though. And making the market converge to less instability in prices, less fish thrown away.

However some stuffs were perturbing. In a simulation without temperature the agents can evolve in non interacting clusters or in constant noise.  Both cases would lead to an unstable market with a global loss of utility/income for everyone. Lose - lose situations.

Another stuff was perturbing if you made the "influence" parameter recomputed every turn according to the distance and effectiveness of the influence, the more a cluster was polarized and strong, the more it would make itself harder, and could become irreversible and the compensation in temperature to fix this states would go higher. Making the problem non reversible.

Slowly ghettoing people in their own behaviour.

When I look at my social networks, they all seem clusterized this way .. in a sort of progression of radicalization of opinions.

I do feel a disturbance in the increased use of algorithm that works to well to show me what I want.

Actually my little brain could be wrong. Who am I to question the smartest engineers in the world when I a kind of small imperfect person on the Internet?

Maybe you are perfect. I am not. I am human, I do err, I do make mistakes, and I like to believe in my capacity of correcting myslef. For this, I need to be exposed to "noise".

Please dear search engines, give me back my capacity to lower my biases and give me a setting for loosing the "precision" on your result. I want a stupid button ranging from "I don't want to see noise 'cause I am focused on technical problem solving it the one best way" to "I am in the mood for exploring the world and question myself and see totally crazy surprising stuffs".

I would gladly accept to be "polarized" in the one best way of thinking if first I believed in a non ambiguous proven immutable truth, and also if it did not resulted in increasing the violence of the exchanges. Something about this could trigger instability and violent moves at my opinion.

And I dare say it could be measured. By applying measures of the entropy on searches over time. But economically I fear there are indirect incentives for polarizing people's opinion when you are both judge of what is relevant and benefit from directing people in comfortable clusters that generates revenues. Why do not we want to see it? Because, we all enjoy to live a peaceful life without conflicts. Sometimes like a frog in water slowly heating and so numbed by comfort we forget to jump out of the water when temperature gets critical. 

But, I am not like every frogs I also want to see what I do not want so that I can apply my own critical judgement to my own self and improve.

I could use more than one search engine you think like on a fish market. But actually we know one engine has the biggest overall relevance that also influences the direction of the other ones. So maybe the "temperature" factor is fubar for this case and we may rely on the big elephant in the room to wake up.

I just wonder if by avoiding small conflicts now for comfort we are not building up a bigger more violent one later.