sweet testing:

ain't nobody got time for that

my name is yeukhon

we are software engineers

Building software components is like building a LEGO city.

we want a pluggable module

But we usually end up like this

what people (us) actually do

but why?

5704.010s > 1 hour and 35 min

worse than that?

Conclusion?

Punch the person who broke the test.

facepalm

Long tests

But tests are essential!

you need some assurance, even if they are terrible tests.

Solution?

Write different kind of tests for different purpose

you need tests mainly for development

Must be QUICK.

We normally call them unit-tests

then there is the rest of the tests

integration, and system tests

They can take minutes to run.

useful for:

So why are good tests so rare?

epic fail. can we fix it?

yes. if we are willing to be a newbie again

---

types of tests

definitions

u.nit-testing | noun

formal

tests each module [or function or method] in isolation to increase your confidence that it meets its requirements.

---

informal

touches only one function or method and replace all dependencies with impersonators

u.nit-testing | example

def foo(parm1, parm2):
    # do something

def bar(parm1, parm2, parm3):
    # do something

def foobar(parm1, parm2, parm3):
    ....
    result = foo(parm1, parm2)
    next_result = bar(parm1, parm2, parm3)
    return next_result

if the subject is foobar, then foo and bar are dependencies.

in·te·gra·tion | noun

formal

individual software modules are combined and tested as a group

informal

a clone of unit-testing, except, not all dependencies have to be replaced (most of them do not)

sys·tem testing | noun

formal

testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements

---

informal

simulate real usage from some entry point of the program

Ambiguity

Every CS people know natural language has ambiguity.

If you start writing integration and system test, you will find overlap!

But this is just the beginning ....

Testing is personal and cultural

Different developer from different organization uses different words

system test ?= functional test ?= end-to-end test ?= acceptance test

Definition flame war

Even developers on Testing In Python (TIP) mailing list will argue over these definitions.

But we have to be on the same page!

Google is the rescuer

small, medium, large

Use it! It will save you hours from arguing.

Define unit tests?

RTFC

Read the fame chart

I just don't have a better word for the letter "F" ....

Moving right along

Testing techniques

Real live code

user.py

from gcs2 import GCS, GCSError

def user_exists(param1=None):
    # do something

def _add_user_to_aurum(parm1, parm2, parm3, parm4):
    # do something

def register_user(username, password, masteru, masterp):
    if user_exists(username=username):
        raise GCSError

    gcs = GCS(masteru, masterp)
    user_id = gcs.register(username, password)

    result = _add_user_to_aurum(user_id, username, password, DBSession)
    return result

problems

two common solutions

  1. dependencies injection
  2. monkeypatching

dependency injection

pass in the dependency as a parameter to the constructor or the function.

instead of writing

from models import DBSession, User
def _add_user_to_aurum(user_id, username, password):
    # do something
    DBSession.query(....)

you write

def _add_user_to_aurum(user_id, username, password, dbsession):
    # do something
    dbsession.query(....)       # could be any "dbsession"

How does that help?

your dbsession parameter can be different for different situation

DI in testing

You can create a fake dbsession object.

class FakeDBSession(object):

    def query(self, table):
        return "I love food"

class TestAddUserToAurum(unittest.TestCase):
    def test_dummy(self):
        result = _add_user_to_aurum("id123", "myusername", "mypassword", FakeDBSession())
        self.assertEqual(result, "I love food")

Can we automate injection?

Use Mock!

Mock Basic

[sudo] pip install mock

import mock
class TestAddUserToAurum(unittest.TestCase):
    def test_dummy(self):

        fake_dbsession = Mock()
        fake_dbsession.return_value.query.return_value = "I love food"

        result = _add_user_to_aurum("id123", "myusername", "mypassword", fake_dbsession)
        self.assertEqual(result, "I love food")

Better mocking?

use Mock(spec=Object)!

import mock
from project.models import DBSession

class TestAddUserToAurum(unittest.TestCase):
    def test_dummy(self):

        fake_dbsession = Mock(spec=DBSession)
        fake_dbsession.query.return_value = "I love food"

        result = _add_user_to_aurum("id123", "myusername", "mypassword", fake_dbsession)
        self.assertEqual(result, "I love food")

Use spec=ObjectName so Mock can create a fake object with all the attributes and methods the actual object would have.

DI is not applicable to register_user

For register_user, injection is not really that helpful. It would be awkaward if we do

def register_user(username, password, masteru, master,\
    user_exists, GCS, _add_user_to_aurum):

dependency injection: pros vs cons

Okay. Try monkeypatching

Idea:

modify a piece of Python code (could be the entire module, or just a function) at runtime.

But how to monkeypatch?

Use Mock!

Mock: simple example

from mock import Mock, patch

def foo(foo_var):
    return foo_var

def bar(bar_var):
    return bar_var

def foo_plus_bar(fv, bv):
    return foo(fv) + bar(bv)

@patch('__main__.foo')
def main(mock_foo):
    mock_foo.return_value = 0
    result = foo_plus_bar(20, 40)
    assert result == 40

if __name__ == '__main__':
    main()

Mock: simple example

from mock import Mock, patch

def foo(foo_var):
    return foo_var

def bar(bar_var):
    return bar_var

def foo_plus_bar(fv, bv):
    return foo(fv) + bar(bv)

@patch('__main__.foo')
def main(mock_foo):
    mock_foo.return_value = 0
    result = foo_plus_bar(20, 40)
    assert result == 40

if __name__ == '__main__':
    main()

patch performs a monkeypatch. It impersonates the real foo function by always returning zero.

@patch decorator gotcha 1

When you use @patch decorator, you have to pass in a mock argument into the function being decorated.

Instead of writing

@patch('__main__.foo')
def main():
   # assert things

you have to write

@patch('__main__.foo')
def main(mock_foo):
   # assert things

@patch decorator gotcha 2

You patch the object/function/module based on the way you used in the code you want to test.

# this is file A, named moduleA.py
def foo(...):
    pass

# this is file B, named moduleB.py
import A

def fun_i_want_to_test():
    return A.foo(...)

then when you patch, you write

@patch('A.foo')

@patch decorator gotcha 2 (cont)

whereas

# this is file A, named moduleA.py
def foo(...):
    pass

# this is file B, named moduleB.py
from A import foo

def fun_i_want_to_test():
    return foo(...)

then when you patch, you write

@patch('B.foo')

monkeypatch the real code

class TestRegisterUserFunc(UnittestBaseCase):

    @patch('aurum.user.user_exists', autospec=True)
    @patch('aurum.user.GCS', autospec=True)
    @patch('aurum.user._add_user_to_aurum', autospec=True)
    def test_register_user01_successful(self, adduser, gcs, userexist):
        expected = {'user_id': self.uid, 'shared_key': self.shared_key}
        self.userexist.return_value = False
        self.adduser.return_value = expected
        self.assertEqual(register_user(\**self.data), expected)

conclusion

You patch the module/function depending on the namespace. Just patch the name you use to call in the function!

What else?

Use setUp* and tearDown*!

unittest module

subclass unittest.TestCase!

DRY again! You don't want to write the same setUp* and tearDown* twenty times if you have twenty functions share the same depenendices!

keep subclassing....

I am serious!

UnittestBaseCase

class UnittestBaseCase(unittest.TestCase):
    """ Base class for unittesting. """

    __test__ = False
    @classmethod
    def setUpClass(cls):

        cls.scoped = patch('aurum.models.scoped_session', autospec=True)
        cls.scoped.start()

        cls.session = patch('aurum.models.sessionmaker', autospec=True)
        cls.session.start()

        cls.zte = patch('aurum.models.ZopeTransactionExtension', autospec=True)
        cls.zte.start()
        cls.requests = FakeRequest
        .......

End results?

No one is writing 100 different unittest testclass for the same module.

Uniform test classes are better.

Fixtures?

Say no to Satan

I am looking at you, Django O_o

Problem with fixtures

No fixture. How do we run integration or functional test?

Use model factories

Ruby does this right: factory_girl

Python has one too, but most of features are Django exclusive: factory_boy

class User(Base):
    __tablename__ = 'users'
    id = Column(Integer, primary_key=True)
    name = Column(Text, unique=True)

 def __init__(self, name)
     self.name = name

# now use factory_boy
import factory
class UserFactory(factory.Factory):
    FACTORY_FOR = User
    name = "pre-defined-name"

user = UserFactory.build()  # in-memory database entity, no DB connection at all

Tests should be self-contained

Never assume ordering. Tests have to be independent of each other.

however, sequence is needed for large tests

Write the sequence down in a single monolithic test case.

Never use SQLite

LOLCAT is angry when people run SQLite as development database.

Why do they do?

Because it's disposable.

True, but are you running SQLite on production?

NO! Then use PostgreSQL or MySQL!

Your tests may pass on SQLite, but can fail on your real DBMS. However, the truth is usually the reverse.

SQLite compatbility to PSQL and MYSQL low.

Use rollback

Use a good ORM (object-relational manager) such as SQLAlchemy.

def tearDown(self):
    self.dbsession.rollback()

@classmethod
def tearDownClass(cls):
    Base.metadata.drop_all(cls.engine)

Next, integration

I meant medium test

in·te·gra·tion test | noun

one level above pure unit test, but does not require all dependencies to be removed!

Real live code

our code

def register_user(username, password, masteru, masterp):
    if user_exists(username=username):
        raise GCSError

    gcs = GCS(masteru, masterp)
    user_id = gcs.register(username, password)

    result = _add_user_to_aurum(user_id, username, password, DBSession)
    return result

GCS is a class that performs HTTP requests. We need to fake this dependency!

class TestRegisterUserIntegration(HeavyTestBaseCase):

def setUp(cls):

self.gcs_patcher = patch('aurum.user.GCS', autospec=True) self.gcs = cls.gcs_patcher.start() .....

def test_register_user_successful_return_useid_and_shared_key(self):

result = register_user('1', self.password, self.masteru, self.masterp) self.assertEqual(result.keys(), ['user_id', 'shared_key']) self.assertEqual(result['user_id'], self.uid)

Any interaction that is not local to your project, you may have to mock it out.

Finally, large test

dat chart...

What to test?

Testing aid tools

  1. virtualenv
  2. nosetests
  3. nosetests --with-progressive
  4. tox
  5. py.test
  6. Jenkins

virtualenv

sudo pip install virtualenv
virtualenv /path/to/virtualenv --use-distribute
source /path/to/virtualenv

nosetests

nosetests --with-progressive

see nose-progressive.

when it fail .. image:: http://i.imgur.com/qwLZSni.png

tox

a testing configuration for automated testing, basically. See tox <http://tox.readthedocs.org/en/latest/>_.

# content of: tox.ini , put in same dir as setup.py
[tox]
envlist = py26,py27
[testenv]
deps=pytest       # install pytest in the venvs
commands=nosetests  # or 'py.test' or ...

py.test

Alternative to nosetests. Many people consider it better than nose.

I plan to use it at some point.

Jenkins

Continous integration stystem written in Java. This is for nightly build.

This is must for deployment and releasing engineering.

Finally, how do we apply this in team work?

I wish I had more time on this, but do STDDD.

STDDD == Scaffolding-Test-Documentation-Drive-Development

def register_user(username, password, masteru, masterp):
    pass
SpaceForward
Left, Down, Page DownNext slide
Right, Up, Page UpPrevious slide
POpen presenter console
HToggle this help