Skip to content

Waldur Testing Guide

Test Writing Best Practices

1. Understand Actual System Behavior

  • Always verify actual behavior before writing tests - Don't assume how the system should work
  • Test what the system actually does, not what you think it should do
  • Example: Basic permission queries don't automatically filter expired roles

2. Use Existing Fixtures and Factories

  • Always use established fixtures - Don't invent your own role names
  • Use CustomerRole.SUPPORT not CustomerRole.MANAGER (which doesn't exist)
  • Use fixtures.ProjectFixture() for consistent test setup with proper relationships
  • Use factories.UserFactory() for creating test users with proper defaults

3. Error Handling Reality Check

  • Test for actual exceptions, not ideal ones
  • If the system raises AttributeError for missing attributes, test for AttributeError
  • Only test for PermissionDenied when the system actually catches and converts errors

4. Mock Objects for Complex Testing

  • Use Mock objects effectively for nested permission paths
  • Create realistic mock structures: mock_resource.project.customer = self.customer
  • Test permission factory with multiple source paths: ["direct_customer", "project.customer"]
  • Mock objects help test complex scenarios without database overhead

5. Time-Based Testing Patterns

  • Understand explicit vs implicit time checking
  • Basic has_permission() doesn't check expiration times automatically
  • Test boundary conditions: exact expiration time, microseconds past expiration
  • Create roles with timezone.now() ± timedelta() for realistic time testing

6. Test Base Class Selection

Choose the right test base class for each test:

  • Default: test.APITestCase — uses transaction rollback, much faster
  • test.APITransactionTestCase is a last resort. It truncates every table between tests. Only two situations actually need it:
  • Threading or multi-process database access (select_for_update under real concurrency)
  • transaction.on_commit() callbacks must fire and wrapping the triggering call is impractical

Three reasons that look like they need it, but do not:

Looks like it needs TransactionTestCase What to do instead
transaction.on_commit() must fire Wrap the triggering call in self.captureOnCommitCallbacks(execute=True)
Deliberate IntegrityError Raise it inside transaction.atomic() — the savepoint keeps the wrapping transaction intact
responses.start() in setUp Register addCleanup(responses.reset) before addCleanup(responses.stop); leaked mocks come from a missing reset(), not from the base class 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# GOOD: Default to APITestCase
class MyTest(test.APITestCase):
    def test_something(self):
        ...

# GOOD: on_commit under APITestCase — capture the callbacks explicitly
class OrderProcessingTest(test.APITestCase):
    def test_order_triggers_task(self):
        with self.captureOnCommitCallbacks(execute=True):
            self.client.post(self.url, payload)
        mock_task.delay.assert_called_once()

# GOOD: a deliberate IntegrityError, contained by a savepoint
class UniqueConstraintTest(test.APITestCase):
    def test_duplicate_is_rejected(self):
        with self.assertRaises(IntegrityError), transaction.atomic():
            Model.objects.create(**duplicate)

A CI lint job (scripts/analyze_transaction_test_cases.py --ci --baseline 0) enforces this: any APITransactionTestCase class the analyzer cannot see a reason for fails the pipeline. When the reason lives in production code the analyzer cannot see — typically an on_commit() in a signal handler the test drives through the API — state it in a comment on the class:

1
2
3
4
# APITransactionTestCase required: the order handler dispatches the task
# from transaction.on_commit
class OrderNotificationTest(test.APITransactionTestCase):
    ...

7. Performance Testing Considerations

  • Include query optimization tests where appropriate
  • Use override_settings(DEBUG=True) to count database queries
  • Test with multiple users/roles to ensure performance doesn't degrade

8. System Role Protection

  • Test that system roles work correctly even when modified
  • System roles like CustomerRole.OWNER should maintain functionality
  • Test that role modifications don't break core functionality
  • Verify that predefined roles have expected permissions

9. Edge Case Testing

  • Test None values, missing attributes, and circular references
  • Handle AttributeError when accessing missing nested attributes
  • Test with inactive users, deleted roles, removed permissions
  • Verify behavior with complex nested object hierarchies

10. HTTP Mocking Patterns

Preferred: @responses.activate per method — fully isolated, no cleanup needed:

1
2
3
4
5
6
class MyTest(test.APITestCase):
    @responses.activate
    def test_external_call(self):
        responses.add(responses.GET, "https://api.example.com/data", json={"ok": True})
        result = my_function()
        self.assertEqual(result, {"ok": True})

Class-wide mocking with responses.start() — requires APITransactionTestCase:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
class ExternalAPITest(test.APITransactionTestCase):
    """responses.start() in setUp leaks state across TestCase classes."""

    def setUp(self):
        super().setUp()
        responses.start()
        responses.add(responses.GET, "https://api.example.com/data", json={"ok": True})

    def tearDown(self):
        responses.stop()
        responses.reset()
        super().tearDown()

Using responses.start() in setUp with APITestCase causes leaked mock state across test classes because TestCase doesn't fully reset process-level state between classes.

11. Multiple Inheritance Pitfall

When combining APITransactionTestCase with a mixin that extends APITestCase, Python's MRO can silently break TransactionTestCase behavior:

1
2
3
4
5
6
7
8
9
# BAD: MRO puts TestCase._fixture_teardown first
class MyTest(test.APITransactionTestCase, SomeTestMixin):
    ...  # SomeTestMixin extends APITestCase — TransactionTestCase teardown is skipped

# GOOD: Ensure all parents use TransactionTestCase, or use standalone setup
class MyTest(test.APITransactionTestCase):
    def setUp(self):
        super().setUp()
        # Set up mocks directly instead of inheriting from a TestCase mixin

The declaration is already misleading — the class reads as a TransactionTestCase while running with TestCase semantics — and it turns into a hard error the moment the first base is migrated: a base class may not precede its own subclass, so class MyTest(test.APITestCase, SomeTestMixin) cannot be linearised and the whole module fails to import. Drop the redundant base rather than rewriting it:

1
2
3
# GOOD: SomeTestMixin already supplies APITestCase
class MyTest(SomeTestMixin):
    ...

analyze_transaction_test_cases.py fails CI on an unlinearisable base list and warns about a mixed one.

12. OpenStack Backend Test Patterns

When writing standalone backend tests that don't inherit from BaseBackendTestCase:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
class StandaloneBackendTest(test.APITransactionTestCase):
    def setUp(self):
        super().setUp()
        self.fixture = openstack_fixtures.OpenStackFixture()
        # Mock all 5 OpenStack clients
        self.mock_admin = mock.patch("waldur_openstack.openstack_base.backend.AdminSession").start()
        self.mock_session = mock.patch("waldur_openstack.openstack_base.backend.SessionManager").start()
        self.mock_nova = mock.patch("waldur_openstack.openstack_base.backend.NovaClient").start()
        self.mock_neutron = mock.patch("waldur_openstack.openstack_base.backend.NeutronClient").start()
        self.mock_cinder = mock.patch("waldur_openstack.openstack_base.backend.CinderClient").start()

    def tearDown(self):
        mock.patch.stopall()
        super().tearDown()

Test Guidelines

  • Test behavior, not implementation
  • One assertion per test when possible
  • Clear test names describing scenario
  • Use existing test utilities/helpers
  • Tests should be deterministic

Debugging Complex Systems

When fixing performance or accuracy issues:

  1. Isolate the problem:
  2. Run individual failing tests to understand specific issues
  3. Use pytest -v -s for verbose output with print statements

  4. Check if multiple tests fail for the same underlying reason

  5. Understand test expectations:

  6. Read test comments carefully - they often explain intended behavior
  7. Check if tests expect specific error types

  8. Look for conflicting expectations between test suites

  9. Fix systematically:

  10. Fix one root cause at a time
  11. After each fix, run full test suite to check for regressions

  12. Update related tests for consistency when changing behavior

  13. API changes require test updates:

  14. When changing function signatures or default parameters, expect test failures
  15. Update tests for consistency rather than reverting functional improvements
  16. Document parameter behavior changes clearly