Why make a makefile? Can reproducible build ever be achieved again?

When I code I like to pride myself in doing code that works everywhere. If unit testing is a good way to go, the fundation of testing a tool chain does work the same everywhere is checking artefacts are the same.

I recently made a code for a sociogram and added at the top of my « make » (a bash script that does all assembling in a suppositly deterministic way) a claim that by using the same input you would have the same output.

Let's see if I lied by taking a snapshot of both the last frame of videos built on 2 different computers :

If the graph is the same, the topology is not. For something about building geometrical shape this may some questions.

First : why can't computer academics build graphviz, but applied physicist do ?

Graphviz is breaking an unspoken standard of academic computer programming : it is based on a probabilistic simulation with a lot of random in it. Basically you first layout the nodes randomly, and randomly you swap to nodes, count the numbers of edges crossing and keep if less edges crossed than before. The kind of dumb algorithm that works but INVOLVES RANDOMNESS.

Well, computer scientific DO reproducible build ! Don't they ? And RANDMONESS is non reproducible, isn't it ?

Hard Scientists (which exclude the litterature lovers called computer scientists) use PRNG when working, so we can reproduce our builds ?

Seems better :D But not perfect once we set the SEED of the random generator in the graphviz output (the only obvioous source of randomness in the code I control).

Let's wonder what kind of non deterministic element I introduced in my build ?

multi-processing/threading is non déterministic

Enough experience in coding will make you smell were danger is. But, I love danger so I knowingly wanted to be a GOOD hard scientist and make my CPU BURN to 100%. It is a stuff I learned to embrace in physics lab. So, I parallized my code (see bash exemple on how to to it with a snippet :

NB_CORE=$(getconf _NPROCESSORS_ONLN)
function pwait() {
    local nb_core=${1:-$NB_CORE}
    while [ $(jobs -p | wc -l) -ge $nb_core ]; do
        sleep 1
    done
}
for i in *dot; do
    $DOT -Tjpg "$i" > $( basename "$i" .dot).jpg &
    ppids+=( "$!" )
    echo -n .
    pwait ;
done

It does the same thing as python multiprocessing async apply : fork process in the background and wait for all the processes to finish before going on. And, it is clear by exploration of both videos that I miss frames, well, ffmpeg (involved in the process is quite explicit :

[swscaler @ 0x5580a8337840] [swscaler @ 0x5580a8375fc0] deprecated pixel format used, make sure you did set range correctly
[swscaler @ 0x5580a8337840] [swscaler @ 0x5580a8b9dd00] deprecated pixel format used, make sure you did set range correctly

So I put a sleep 10, and it helped ... but not enough, because well modern computing on linux is chaotic :

• signal delivery is unreliable : signals may be missed or can be « acausal », a core can say : I finished while the kernel is still waiting for a file to be written
• versions of software may differ even on 2 debian based distros (one is debian testing (hence outdated), the other is linuxmint current (less outdated)

So ... I did my possible to have the same result given the same parameters by fixing all I controled including seeds of PRNG to have the same results and suffice to say, that at the end of the day, EVEN FOR A SIMPLE SCRIPT deterministic « same results are impossible ».

It is fucking neither the same topology, NOR chronology. For a dynamic picture of a topology it kind of sucks terribly and should not be considered « good enough ».

It is good enough for a « side project », but not for a production ready project.

Famous last words

When I code in python, I have a freeBSD station, because BSDs are more boring than linuxes. However I can't play my favourite games with wine (Need for speed, Quake III, Urban Terror, various pinballs), hence the reason when I code for fun it's on my linux computers. (check my script to build a tailored freebsd qemu image on linux) but I dare say modern coding is not the « boring » activity I grew up with, hence my manic way of trying to make the most I can to ensure « maximum reproducibility in my power ». My power is about rationality, I give up when it comes to the entfshification of linux distributions that clearly went down the path of not caring, after all, you just need to spin a docker alpine to make stuff reproducible, don't you ?

I'm a single man army that want to code, not maintain a kubernetes cluster just for the sake of creating a « snowflake » of reproducibility that negates the purpose of coding. When I was in 1982 I could give my basic source code on a floppy and I was sure that another C64 would yield the same result given the same input. Nowadays, this is a wishful thinking.

The state of modern computing (especially on linux) is BAD. I even think of reconverting to COBOL on OS/360 to gain back a minimum of sanity back regarding my expectations.

I strongly advice devs to have put more effort on their assembling code (docker build, makefile, your own tool) than their « muscle code » (unit test included), because it's not in the code you control you might find the most surprising factor of loss of control, but in the lack of trust you should reasonably have from your hardware, your OSes and distributions. And it's a shift of normality, a new NORMAL, not an absolute NORMAL state.

Annexe

Here is the command line involved

SHOW=1 EDGE_SCALE=1 MIN_MAIL=20 \
  PERS_EDGE_SCALE=.2 BY_DAYS=50 SHOW=1 \
  THRESHOLD_ILOT=1 DOT="dot"  ./make very_clean movie

the « make » involved. And the main script. It DOES NOT EVEN MAKE 500 lines of code. It is small by any standards, even a « BASIC » program of the 1980s. I AM PISSED ! (╯°Д°)╯彡┻━┻

Addendum : and that's how I discovered the too silent fan stopped working silently without telling a thing in the log or the sensors complaining about heat on my laptop. Even hardware are becoming shitty. Addendum 2 : after some tinkering I discovered I nearly burnt my CPU thanks to debian removing fan control/cpufreqd ... AGAIN (╯°Д°)╯彡┻━┻ Now I must under clock this computer to make it compute anything because. Fuck debian !