Making tests before installation with setuptools

I dream that packages don't install if the tests are failing. I made it at least.

My solution is gory but practical.

in setup.py I added:

def test():
    """let's script the command line python -munittest discover"""
    loader= unittest.TestLoader()
    suite=loader.discover(".", "test_.*.py")
    runner=unittest.TextTestRunner()
    result=runner.run(suite)
    if  not result.wasSuccessful():
        raise Exception( "Test Failed: Aborting install")
    print("#### Test passed")

if "install" in sys.argv or "bdist_egg" in sys.argv or "sdist" in sys.argv:
    test()

And since practicality beats purity...

Still wondering if it is a bad idea

Okay, I test my packages before deploying, okay, there is tox. But no test can make as much variations as what users have as an environment. Even though I don't find it nice, at least it has the authoritative psycho-rigid behaviour I want.

Just like in Perl, if it does not pass the tests, it should not be installed.

I still wish I could:

• make an autoreport tool (calling a REST server) to know how reliable are my packages, and which OSes/python versions have problems;
• have a tool that let user interract with ticketting system;
• bypass the tests with a --force flag, and call the test suite in a unified way once the package is installed.

 Still dreaming.

OOP Sux

I have seen posts on hackernews on the topic that Object Oriented Programming (OOP) sux. These are stratospheric considerations. OOP does not sux on a theoretical point of view it sux in real life.

Guh! Don't I like python? Yes I do. But what I hate is not really OOP. It is monkey developers that care about OOP and not about coding.

Coding is like kicking a problem in the nuts. Some problems get kicked in the nuts by OOP, some don't. People that don't understand deserve a slow and painful death by enucleation with a plastic spoon.  

To make myself clear:

• I am gonna make a short theoretical statement about what is wrong with some devs;
• a short practical code review of a code that pisses me off.

OOP sux as much as developers that don't know how to code. 

It is indeed true that I am old school. The data structure is more important for me than the workflow or algorithms. And I do hate OOP for mixing both when it's done poorly.

A good object is for me data with multiple views.

And there is a signature for poor OOP:

• code that should be better written as a script;
• code that has strong coupling;

End of theory. Let's practice.

My code sux!

Sometimes I am too lazy to write code, so all permissions granted I reuse other's code. Pypi-stat (a package of mine) and I suffered my laziness.

Rule 1: all code that could be written as a function should be written as such

If you watch https://github.com/jul/pypi-stat/blob/master/pypi_get_stat.py#L206 you'll notice that the borrowed code is called this way:

PyPIDownloadAggregator(pkg.strip()).stats()

When you see this, don't think: this a script disguised as an object.

By the way, what is the problem to solve? Given a package name, I want all the stats given by an RPC call to pypi in the form of a dict.

Basically I want to :

• fetch the stats;
• transform them;

Fairly easy?

__init__ -the constructor- is stragthforward, it is initialisation. So you expect stats to be easy too.

Well, cry with me lads and wenches.
https://github.com/jul/pypi-stat/blob/master/pypi_get_stat.py#L147

def stats(self):
        """Prints a nicely formatted list of statistics about the package"""
        self.downloads # explicitly call, so we have first/last upload data
        ...

Wut?!

Rule 2: when you need to call methods or properties in a defined order so that it works, you have a coupling problem

Why do I ask a property I don't use at your opinion?

Because, the property is processing hidden states and coupling are on their way.

   

    @property
    def downloads(self, force=False):
        """Calculate the total number of downloads for the package"""

        if len(self._downloads) == 0 or force:
            for release in self.releases:
                urls = self.proxy.release_urls(self.package_name, release)
                self._downloads[release] = 0
                for url in urls:
                    # upload times
                    uptime = datetime.strptime(url['upload_time'].value, "%Y%m%dT%H:%M:%S")
                    if self.first_upload is None or uptime < self.first_upload:
                        self.first_upload = uptime
                        self.first_upload_rel = release

                    if self.last_upload is None or uptime > self.last_upload:
                        self.last_upload = uptime
                        self.last_upload_rel = release

                    self._downloads[release] += url['downloads']

        return self._downloads

downloads is not a property, it is a method doing processing and setting internal properties on the way. So is releases. This stuff that by conventions are stateless (properties) do modify the state of the freaking object. Thank you man! When reading this kind of code where inert stuffs (properties) is active I feel like a rabbit trapped in a dense mine field.


What the heck?

So at one point, I decided I was not reading or improving this «easy» code. I just put a rant in the comments about how much I hate stupid OOP developers.

This very straight forward code has therefore a coupling (you can't call stats without accessing the downloads property because it silently mixes up code and processing, with some states logics).

That's what make script coders like I hate OOP.
The workflow is not sequential (it jumps from part of the code forward and backward), and data are not processed sequentially. It is not code, it is Snakes and Ladders.

That is the reason I hooked my save method on stats: this code was a maze and I knew the entry point was the constructor and the end point was the stats method. I am weak, I know.

Conclusion

OOP is not bad if you fear Alzeihmer and want to train your brain to most astonishing logics.

With poor coders object oriented programming is like taking acids with the mad hatter. Pretty intellectually stimulating, but not very efficient.

Still I think that whatever the paradigm used, this coder would have done code «looking» academically nice, but definitively wrong. Coding is not about following what is written in a book about OOP or functional or imperative programming.

It is about making a complex problem look simple, not the opposite.

EDIT: Very good Pycon talk on the topic

Packaging in python from a former Perl dev point of view

It's easy

I use github, readthedocs post commit hooks,  http://guide.python-distribute.org/ and it all works fine: I am delighted to be honest.

As far as I am concerned packaging in python is freaking easy, a fun and rewarding experience. So here is my point: this is not a rant since I love packaging in python.

However, to be honest there are quite a few things that trouble me:

• I don't really know what I am doing since I mostly follow cookbooks;
• I am lacking some of the CPAN features;
• and I do have (the maybe wrong feeling) that the packaging culture amongst python community is not as strong as in Perl (this also applies to my production).

from CPAN import wisdom

Most of Perl's quality module are not coming from PEP they are coming from cultural habits. I dare cut and paste some of tutorial for perl

• Write the documentation for a module first, before writing any code. Discuss the module with other people first, before writing any code. Plan the module first, before writing any code.
• It's easy to come up with a solution to a problem. It takes planning to come up with a good solution. Remember: the documentation, not the code, defines what a module does.
• Every module should have a purpose. There's a proliferation of modules with names like "perlutils.pm", "rcs_utils.pm", and "utilUtils.pm" that have no obvious purpose, and it's difficult to know what each does. This leads to confusion and duplication of code.

Well, since we are plagued on pypi with the infamous «nested list printer» and ports of PHP file_get_contents, I guess this wisdom is not yet totally in python.

I think python tutorial focus too much on the technical part (how to build a package) and not enough on the QA part (how to make a useful and maintainable package).

There is also something in CPAN I love, and I don't follow because I lost myself in geeking with sphinx, it is the straightforward documentation in one page following this plan:

• Name 
• Version
• Synopsis (short code snippet that works)
• Description (in full english with no code)
• Methods (with code snippets if useful)
• Notes (extra informations needed)
• See also (similar packages)
• Limitations
• Bugs
• Author
• Licence

It is a kind of very informative plan.

These are what the culture provides. Plans are not imposed to the packager, it just converged as being efficient.

from CPAN import tools

Now, CPAN has also great tools we miss in python: for instance prior to installation there are the automated tests and eventually automated reports.

I do always test a package before pushing it, but, I'd rather force tests that prevent installation if they fail on the user side. I tried ditribute test suites feature, but I fumbled.

You know what, I miss this feature, and the deployment matrix.

I miss to see if maintainer is active by being able to watch its ticketing queue. The ticketing system is in CPAN.

I also miss the direct link to the source. Or the dependencies chart.

I also like how they handle the «Missing In Action» of packagers and how they can decide to hand over a package's maintenance to another maintainer.

This have very few chances to happen in a close future. However, I can see how we can all improve our package.

from packager import good_will

Good news is we don't need code to solve most of these problems.

Documentation

In the README I provide with my packages I (try to) include:

• a link to the sources
• a link to the full documentation (on readthedocs and package.python.org)
• a link to the ticketing of github
• a synopsis
• requirements (in case my dependencies get weirdly not computed)
• a changelog

I will try from now on to be terser in my documentation and follow the previously mentioned Perl plan.

I noticed repoze.lru changed its former nice README for a useless one. I am sad.

Testing

 
Always test before pushing. It may seem obvious, but I have noticed some maintainers don't. Build a sdist, make a clean virtualenv with nothing to install your package before uploading it too. It is nice.

Versioning

Follow http://www.python.org/dev/peps/pep-0386/

Always tag your source code in your repository with the adequate pypi version.

Don't ask GvR what you can achieve by yourself

First I understand nothing of the mess between distribute, distribute2, setuptools, so I gave up dreaming of hacking my way to a solution through brute coding. And I guess you don't make a donkey that is not thirsty drink water.

My workaround is as a packager I can by showing the example and hope people will follow me improve python packaging with my own lever: practice and culture.

In the ecosystem I am not only a producer I am also a consumer of packages. So I think that as consumers of packages YOU can also improve the packaging ecosystem by checking that a package follows most of these rules before installing it:

• is the README on pypi including 
• a link to the source, 
• the ticketing system, 
• a synopsis, 
• a changelog, 
• a link to the the full documentation (1 point per present info);
• is the full documentation following the Perl canonical Plan (in regard to the complexity of the package don't be too picky (5 points if the doc is relevant))
• does the source code contains a test suite? (5 points)
• can I reach the maintainer IRL (2 points)
• are there outstanding issues in the ticketing (2 points if all issues are opened for less than 1 month)
• is there an auto reporting tool in the package (that triggers the test suite and submit it to a ticketing system) (5 points (I won't have them)). 
• versioning the PEP way (3 points)

In my case I won't use a package having less than 20 points on my own scale. If we all do that we have a chance that packaging improves. I guess Perl has made mistakes so I really don't advocate following their steps blindly, I advocate that we do also slowly build our own strong culture of Quality Assurance the python way.

Oh, and since I am proud of finalizing this «Book» (perl dev) snippet, here is a polyglot in Perl and python to do i++ and ${A}++

q = 0 or """ #=;$A=41;sub A { ~-$A+2}; A() && q' """
A=lambda A: -~A #';
print A(41) # python + perl = <3

I can walk the land of illusions

Have you ever wondered if you were crazy? Sometimes sanity lies in assuming you are. I have what some dumbass call an Asperger Syndrom.

I don't have a syndrom, I can travel back and forth from the land of symbols. I can tell stories equally consistent that may be true of false. When walking this land, I am totally engulfed in another reality.

Let me tell you a story that may be true, or false. I don't know myself where the truth is. And as far as I concerned I don't tell the truth, I tell you a fictional story.

I do recognize what is said about Asperger is true: I can't decypher facial expressions, we are crippled when it comes to feel empathy. This part of the brain seems to be cut from its natural final destination. Brains rewire. Human adapt. We have a part of the brain specialized in symmetry/pattern/singularity recognition that is left unused in our early ages.

We use this part of the brain for other things we get attracted to: flows, mechanisms, music, drawing. When this part of the brain activate, we have the same feelings as if we saw expressions. We crave to see forms in symbols that are the evocation of positive feelings. Regularities, symmetries, pattern of symbols are triggering in our brains their human counterparts.

As young people, we are avid of knowledge, symbols, all field that attract us. A world of symbols slowly mature. We have mutiple maps slowly building in our imagination.

We also need rituals that are yet immutable but varying to bring us our fix of input. We constantly saturate our brain with informations, avidly looking for patterns. Sometimes given certain conditions (sleep deprivation, some emotional pattern), we can fully snap in the world of symbols.

All our circuitry is temporally rewired for analysis: verbal circuits, facial recognition, emotions, pain, sensations all get rewired. And then, we can travel as if we were in the symbols themselves. We can travel our imagination as if it was the true world.

Causality, facts, informations all get intuitively accessible through all day life experience. We walk the map. We have the very feeling to know the truth behind the reality. However, the path is made of orbital with bifurcations each one leading to a contradicting truth. That's why I call this world the world of illusions. I can loose myself there, that's why I repeat so often the mantra of the map and territory. I can distinguish both world. Both are equally false. And I travel on the fine line of sanity.

My rewired and saturated circuits are anarchically generating signals in my brains. It triggers as a side effect emotion, pleasure and pain in a way I can't control. It is intoxicating. It feels like a trip under strong hallucinogens.

Not to loose myself, I need in the real world a passeur. Someone that helps me take the safe way, that helps me deliver my visions so that I don't overload. My expressions are being clearly so influenced by the intoxication that I need someone to decypher all I can say that don't make sense.

I also need him or her so that I can trust her not to make me take the dark paths. Some experiences are best avoided. Who wants to know the actual feeling of death for instance?

The bound is the one of talking. The language is halfway through imagination and reality. It is a journey. I feel like being a weird computer semi directing the calculation operated by someone in communion.

I know it is false, it is only a story. And like any story it ends.

When it ends, the saturation of my brains is followed by an intense relaxation. I can feel the world, the emotions, the wind on my skin as if it was the first time. I sometimes have the feeling to be able to distinguish the emotions of people by watching their face. I can cry and laugh. By coming back, I have a short time walking your world. I am normal, for once. Until, a new crisis will come back. In between, I will be a freak that will be lost in between. Experiencing glimmers of both worlds. With no joys or emotions. I will be a zombie, but nobody will see me as I am when I walk the forest of the symbols.

When it happens, I feel both like a shaman, and a plain fool. And I wonder, if it is crazy, or true. I don't know. I only know these moment of inspirations are frightening the normal people. That they fear us, or fear (for good reasons) we may definitely loose our sanity.

The only thing I know is, that it may be a delirium or a self induced hallucination, but is fucking cool.

I am not like you, I may be crazy, I may see truths that are untold, but I don't care: I can travel in places you will never see. I can walk the path of my imagination without any drugs.