To explain this paradox one main cause have been proposed since 1968: IT projects are late and tend to blow off there expectations in investment and operational costs more than in any other sectors.
The pathetic excuse of Fred Brooks that all the IT have using for failing is: we are late because creation is by nature an unpredictable thus, we cannot plan neither know how much investments are required, nor how much the operational costs will be. That is the reason why IT economy is not based on obligation of results (all corporate software decline liability) but obligation of means. Joel Splosky, and a lot of developer I respect recommend (as I do) to read this book. I do recommend to have a critical reading though.
I think it is true, and that it is much more an effect of belief and self predicting realisation than a fatality. In this essay I will try to tackle so much in depth this idea that I wish to hear its ankle crack.
Costing and pricing are impossible, still IT companies are doped like soviets athletes with investments (thanks to Dragi, Yellen, YC, and all investors). And still IT projects are late, under delivering and considered a cost sink.
Well actually, not all activities that are creative fail to deliver on time.
As a former groom in a theater I know for sure that a lot of creative are able to deliver on time astounding new creations:
dance shows, circus shows, plastic art exhibits, movies, concerts... with an expected quality (except for plastic arts (lol)).
You may think, artists are not tied by the ridiculous requirements of economics and technics and this is not a competitive market.
Well, it happens I have been experimentally covering for La croix when I was a teenager the Avignon festival (I am the last person to have interviewed Heiner Müller), my brother has been managing a hip hop dance company (Trafic de style) and my mother programming artists in a national scene. And I am born french (appeal to the cliché for this one) but still am familiar with other stuff than France :)
So I can tell as an insider that the inability of software teams to deliver on time and in the expected envelop of costs something said to be by nature creative does not hold a ground. I also happen to have studied engineering of organisation in CNAM Paris. I could also tackle the problem with an engineering point of view, but it would be less fun.
Delivering in time and in budget is totally possible... unless you build a Babel tower.
The misconception about creations being inherently unpredictable in costs and delivery
I have been interviewing Richard Owens at WIPO in Geneva a long time ago. Software industrie is not protected by patents but by author's right because it is said that the value is not in what the software does, but HOW it does it. The international protection based on Bern/Genveva conventions values the original expression of the author. As a result legally speaking software industry has to be considered the same as any creative industry. Software is indistinguishable from a piece of art.
What you may not now about art companies is that the myth of it being heavily subsidized is false for most companies : for one subsidized show in Avignon there are hundreds of non subsidized show (the off festival). Still a show is not I do whatever I want on stage and everyone can do it. It is an awful lot of work
Even small companies have to format the delivery of their production in a formal way.
Requirements in terms of space, volume, weight, capacity to move the piece of hardware, if ever they bring their light/sound engineers/equipment, the time, the required size of the stage, the cost of the show, the dates, housing, transport .... respect of security norms (like use of pyrotechnics), planning insurances, the audience (you don't show adult contents to kids) ... any shows requires artists to provide a huge list of requirements.
It also requires to deal with intellectual properties. Authors have to be registered as well as interprets, and all the external assets (music, text) have to be declared and paid.
You may think it requires less coordination. Well no. Carpenters, decorators, plasticians, actors, realisation, production engineers, technicians have to be coordinated and if you use external resources from a theater (light/sound engineers/technicians) they have to have their directions formalized. So companies may also have to deliver procedures for complex tasks in a formalized way. Because they are outsourcing part of their functions.
So showing on time for an artist is as complex as delivering on time for software engineers. It requires as much "costs of coordination", it involves as much "unpredictability" in the planning, still they show on time.
They do it.
Benefits, still are unpredictable. You never know if you will succeed. But every artists knows that not showing on time is gonna ruin any possibility to earn any benefits, else they are ruined.
They lose trust of the industry (tourneur, and theater), no money comes in... once you begin to lose trust, you begin to lose revenues.
Some says size matters. But, delivering on time and delivering their "technical notes" is the common ground of all companies. Size really does not matters. Stomp, Cirque du Soleil, the circus touring in the country side, the semi pro companies, street artists all do follow the rules of having both "packaged" their delivery according to the standards AND on time, AND reasonably in the budget.
They build to budget.
It looks like from the eyes of a computer developer like a permanent Quality Assurance process.
First the metteur en scène (or the company as a whole) comes with an idea and decide to divide it into smaller pieces. And according to the complexity of the play/budget they plan their date of show, and implicitly devise milestones according to their former experiences of delivering on time.
They setup a proof of concept at the very beginning, and try and repeat their idea. Every days like maniac. Dancers are the most impressive.
At no point a company have no materials.
Since day one, people will have something to show. Actors can memorize their texts and repeat it to time it. Dancers can have small gimmick they show. Decorators/plasticians will show a prototype...
As soon as needed the other corps de métiers are mixed together.
People are repeating, training all the time. As soon as the budget for the pure creation part is exhausted, they repeat. Over, and over again. From the begining to the end. The show is permanently played, checked, modified, replayed. From day 1 to the end it never ends. When you make a creation, you have to submit to the entity founding you a business plan. Else you will have no founding, from no one. And for having seen the quantity of papers that are demanded in order to prove an artistic project is viable it has nothing to envy to formal companies. I would say it is very rigorous.
At a given time varying according to the complexity of the show, a filage begins before the première.
My fuzzy memories for a long time ago says that it is for a classical show it is something like weeks or months before the first presentation.
During a filage, maniacally people play the show over and over again until it is perfect. During the filage, the show is put in its chronological order/final form and more and more external eyes are introduced to get a feedback, and if time is missing the réalisateur might cut or rearrange the show. It looks like integration.
You may think that the creation is carved into marble after the first the show. It might be true for big shows (musicals, huge companies like stomp), but most of the time, during the tournée the company will improve the show to maybe introduce stuff that could not be shown because it was not ready for day one, or cut stuff that for which the audience is not receptive.
People may think that shits never happens to artists. It is true that the rotation of the artists in a company is way lower than in the IT sector, and it helps a lot. However, legs get broken, stuff get stolen... And sometimes you need to replace complex piece of equipment and know how or artists on the fly.
A new artist or technician even though every company works differently (autocratic with narcissists tendencies ruling their show as business men, almost hippy companies sharing everything, enthusiasts, punk/squatt grown blobs) always integrate fast.
Because everyone share a same concern : the show must go on.
And what about the money?
Mainly from the idea of the show to the first show we are running into the KPEX part of a project.
An artistic show begins like a cost sink.
But, after the first show, according to the success the company will adapt its operational costs to the gauge and the time of the show and the frequency of the show that it can aim regarding the success it has.
Gauge can range from n shows of one spectator every 5 minutes (street performance/event) to 90 persons, 700, 1400 show in a theater, or 100 000 in a one time event when it is Découflé or Royal Deluxe making a défilé.
Artistic companies have been far more serious at following their costs and expenses at my experience than any manager I have seen in IT industry.
Plus, they adapted their cost/revenue packaging to the uncertainty of their job: once a show as been seen, it will have a tendency to not be seen again. So shows have a 0 to climax then climax to long trail kind of economical curve. The show changes constantly in terms of prices, costs and expenses according to the curve of success of the show.
Circus are not having the same expectations of revenues than a one time event subsidized by either a sponsor or a state. Artistic business is as spread in its pattern as IT industry.
So how is IT failing with a stronger requirements on diplomas or project managements?
There can be one an only one answer.
Something is rotten in the state of DenmarkNo I am joking. But, yes I do think it is systemic to our field of work. Something is definitively rotten in the IT industry at a fundamental level.
I am thinking on it, and plan to write an essay in the future on this topic. But I may have some leads.
By being scientific now that we know what is common (creativity and expected processes of respecting business and technical requirements) we can now think of what differs. I can also add something surprising: since there has been a financial bubble on education the education does not differs. Yes, I have met artists with MBA, Ivy leagues diplomas, no diplomas ... There is no difference in education.
Maybe it is the the structural forms as defined by Mintzberg : IT companies tends to adopt bureaucratic/specialized division of the work whereas artistic activities tends to chose functional ad-hocraty ?
Maybe it is because the artistic market is more competitive than the IT market, since there are fewer entry barriers in terms of diplomas, or legal requirements, and the market is less opaque?
Wrong incentives given to recruit workers ?
Maybe even though formal division of work is very standardized in art (artists are NO sound/light engineers, régisseur, manager and vice versa), all these functions are sharing a common lingua franca avoiding a Babel tower effect and share a clear incentive/vision of what has to be built and the constraints.
But still one thing strikes me: artists have a stronger ethic, and I would suggest that artistic incentive based on reputation rather than money helps recruiting people with a Beruf is better than stock options.
Still ESA was able to build the Philae Rosetta mission. NASA was able to make software function on mars... I don't think this problem is inherent to programming in itself, or the culture of a country but rather to so called profesionnal sector.
What differs? What is the smallest set of parameters explaining why we suck at delivering on time?
When armies are defeated, I tend to blame the strategies, not the soldiers nor the guns.
What is wrong? How do clowns and other saltimbanques coming from squatts outperform engineers coming from Ivy leagues ?