Skip to content

Waldur Testing Guide

Test Writing Best Practices

1. Understand Actual System Behavior

  • Always verify actual behavior before writing tests - Don't assume how the system should work
  • Test what the system actually does, not what you think it should do
  • Example: Basic permission queries don't automatically filter expired roles

2. Use Existing Fixtures and Factories

  • Always use established fixtures - Don't invent your own role names
  • Use CustomerRole.SUPPORT not CustomerRole.MANAGER (which doesn't exist)
  • Use fixtures.ProjectFixture() for consistent test setup with proper relationships
  • Use factories.UserFactory() for creating test users with proper defaults

3. Error Handling Reality Check

  • Test for actual exceptions, not ideal ones
  • If the system raises AttributeError for missing attributes, test for AttributeError
  • Only test for PermissionDenied when the system actually catches and converts errors

4. Mock Objects for Complex Testing

  • Use Mock objects effectively for nested permission paths
  • Create realistic mock structures: mock_resource.project.customer = self.customer
  • Test permission factory with multiple source paths: ["direct_customer", "project.customer"]
  • Mock objects help test complex scenarios without database overhead

5. Time-Based Testing Patterns

  • Understand explicit vs implicit time checking
  • Basic has_permission() doesn't check expiration times automatically
  • Test boundary conditions: exact expiration time, microseconds past expiration
  • Create roles with timezone.now() ± timedelta() for realistic time testing

6. Test Base Class Selection

Choose the right test base class for each test:

  • Default: test.APITestCase — uses transaction rollback, much faster
  • Use test.APITransactionTestCase only when:
  • transaction.on_commit() callbacks must fire (e.g., Celery task dispatch)
  • IntegrityError is deliberately triggered (breaks TestCase's wrapping transaction)
  • Threading or multi-process database access is needed
  • responses.start() in setUp for class-wide HTTP mocking (leaks across TestCase classes)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# GOOD: Default to APITestCase
class MyTest(test.APITestCase):
    def test_something(self):
        ...

# GOOD: Use APITransactionTestCase when on_commit is needed
class OrderProcessingTest(test.APITransactionTestCase):
    def test_order_triggers_task(self):
        # on_commit callback fires Celery task
        ...

A CI lint job (scripts/analyze_transaction_test_cases.py --ci --baseline N) enforces this — adding new unjustified APITransactionTestCase classes will fail the pipeline. The baseline is lowered as classes are migrated.

7. Performance Testing Considerations

  • Include query optimization tests where appropriate
  • Use override_settings(DEBUG=True) to count database queries
  • Test with multiple users/roles to ensure performance doesn't degrade

8. System Role Protection

  • Test that system roles work correctly even when modified
  • System roles like CustomerRole.OWNER should maintain functionality
  • Test that role modifications don't break core functionality
  • Verify that predefined roles have expected permissions

9. Edge Case Testing

  • Test None values, missing attributes, and circular references
  • Handle AttributeError when accessing missing nested attributes
  • Test with inactive users, deleted roles, removed permissions
  • Verify behavior with complex nested object hierarchies

10. HTTP Mocking Patterns

Preferred: @responses.activate per method — fully isolated, no cleanup needed:

1
2
3
4
5
6
class MyTest(test.APITestCase):
    @responses.activate
    def test_external_call(self):
        responses.add(responses.GET, "https://api.example.com/data", json={"ok": True})
        result = my_function()
        self.assertEqual(result, {"ok": True})

Class-wide mocking with responses.start() — requires APITransactionTestCase:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
class ExternalAPITest(test.APITransactionTestCase):
    """responses.start() in setUp leaks state across TestCase classes."""

    def setUp(self):
        super().setUp()
        responses.start()
        responses.add(responses.GET, "https://api.example.com/data", json={"ok": True})

    def tearDown(self):
        responses.stop()
        responses.reset()
        super().tearDown()

Using responses.start() in setUp with APITestCase causes leaked mock state across test classes because TestCase doesn't fully reset process-level state between classes.

11. Multiple Inheritance Pitfall

When combining APITransactionTestCase with a mixin that extends APITestCase, Python's MRO can silently break TransactionTestCase behavior:

1
2
3
4
5
6
7
8
9
# BAD: MRO puts TestCase._fixture_teardown first
class MyTest(test.APITransactionTestCase, SomeTestMixin):
    ...  # SomeTestMixin extends APITestCase — TransactionTestCase teardown is skipped

# GOOD: Ensure all parents use TransactionTestCase, or use standalone setup
class MyTest(test.APITransactionTestCase):
    def setUp(self):
        super().setUp()
        # Set up mocks directly instead of inheriting from a TestCase mixin

12. OpenStack Backend Test Patterns

When writing standalone backend tests that don't inherit from BaseBackendTestCase:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
class StandaloneBackendTest(test.APITransactionTestCase):
    def setUp(self):
        super().setUp()
        self.fixture = openstack_fixtures.OpenStackFixture()
        # Mock all 5 OpenStack clients
        self.mock_admin = mock.patch("waldur_openstack.openstack_base.backend.AdminSession").start()
        self.mock_session = mock.patch("waldur_openstack.openstack_base.backend.SessionManager").start()
        self.mock_nova = mock.patch("waldur_openstack.openstack_base.backend.NovaClient").start()
        self.mock_neutron = mock.patch("waldur_openstack.openstack_base.backend.NeutronClient").start()
        self.mock_cinder = mock.patch("waldur_openstack.openstack_base.backend.CinderClient").start()

    def tearDown(self):
        mock.patch.stopall()
        super().tearDown()

Test Guidelines

  • Test behavior, not implementation
  • One assertion per test when possible
  • Clear test names describing scenario
  • Use existing test utilities/helpers
  • Tests should be deterministic

Debugging Complex Systems

When fixing performance or accuracy issues:

  1. Isolate the problem:
  2. Run individual failing tests to understand specific issues
  3. Use pytest -v -s for verbose output with print statements

  4. Check if multiple tests fail for the same underlying reason

  5. Understand test expectations:

  6. Read test comments carefully - they often explain intended behavior
  7. Check if tests expect specific error types

  8. Look for conflicting expectations between test suites

  9. Fix systematically:

  10. Fix one root cause at a time
  11. After each fix, run full test suite to check for regressions

  12. Update related tests for consistency when changing behavior

  13. API changes require test updates:

  14. When changing function signatures or default parameters, expect test failures
  15. Update tests for consistency rather than reverting functional improvements
  16. Document parameter behavior changes clearly