Movatterモバイル変換


[0]ホーム

URL:


Skip to content
DEV Community
Log in Create account

DEV Community

Danilo Desole
Danilo Desole

Posted on • Edited on

     

Boto3 and Python unittest.mock

I start this post by saying I'm not a professional software developer, I work mainly in IT Operations, although I write especially for IAC and small lambdas functions.

When developing a Lambda function most of the time I need to interact with AWS Services via the famousboto3 library;boto3 is a powerful library developed and maintained by AWS which provides a communication framework to interact with native AWS Cloud Services.

Every time I struggle to mock the library. I even considered using themoto stubber but, I'm not happy with what is provided. I aim to

  1. Create a mock which can stub the original call
  2. I want the mock to expose all the methods to check which parameters have been used during the calls, how many times it has been called, etc... all the stuff included within theMock class

In this example, we will mockboto3 while it creates a client forRDS.

Consider the following code

importboto3classMyRDSManager:def__init__(self)->None:self._rds_client=boto3.client("rds")defdelete_db_cluster_snapshot(self)->None:self._rds_client.delete_db_cluster_snapshot(DBClusterSnapshotIdentifier="ABC")defget_snapshots(self)->list[dict]:snapshots=[]paginator=self._rds_client.get_paginator("describe_db_cluster_snapshots")pages=paginator.paginate(DBClusterIdentifier="MyCluster")forpageinpages:page_snapshots=page.get("DBClusterSnapshots")forsnapshotinpage_snapshots:snapshots.append(snapshot)returnsnapshots
Enter fullscreen modeExit fullscreen mode

To test the above class I developed the following tests

importpytestfromunittest.mockimportMock,patchfromdatetimeimportdatetimefrommainimportMyRDSManager@pytest.fixture(scope="function")defprepare_mock():withpatch("main.boto3.client")asmock_boto_client:# the first thing to do is to set the return_value attribute as itself# this will return the mock itself when the code runs `boto3.client("rds")`mock_boto_client.return_value=mock_boto_clientdefmock_paginate_describe_db_cluster_snapshots(*args,**kwargs):snapshot_type=kwargs.get("SnapshotType")# get all the parameter passed to the callreturn[{"DBClusterSnapshots":[{"DBClusterSnapshotIdentifier":"ABC","SnapshotCreationTime":"2024-01-01","SnapshotType":"manual",}]}]mock_boto_client.get_paginator=Mock()mock_paginator=Mock(return_value=None)mock_paginator.paginate=Mock(return_value=None)mock_paginator.paginate.side_effect=mock_paginate_describe_db_cluster_snapshotsmock_boto_client.get_paginator.return_value=mock_paginatormock_boto_client.delete_db_cluster_snapshot=Mock(return_value=None)my_rds=MyRDSManager()yieldmy_rds,mock_boto_clientdeftest_one(prepare_mock):mock_my_rds,mock_boto_client=prepare_mockmock_my_rds.delete_db_cluster_snapshot()mock_boto_client.delete_db_cluster_snapshot.assert_called_once_with(DBClusterSnapshotIdentifier="ABC")result=mock_my_rds.get_snapshots()assertresult==[{"DBClusterSnapshotIdentifier":"ABC","SnapshotCreationTime":"2024-01-01","SnapshotType":"manual",}]
Enter fullscreen modeExit fullscreen mode

The first thing to notice is that we need to set themock_boto_client.return_value tomock_boto_client, which is itself, and this will return the mockinstance you are configuring when inMyRDSManager the code runsself._rds_client = boto3.client("rds"). If you don't set this MagicMock will return the default value, therefore anew MagicMock, and not what you are configuring!

Another important point to note is that we patchboto3 within the specific module being tested, rather than thegeneralboto3 library. In other words, you should patchmain.boto3.client, wheremain refers to the module you have written and are currently testing.

Next, configure the Mock as needed by adding methods and attributes. It's worth noting that setting aside_effect allows the mock to invoke the function specified in theside_effect, passing along all the arguments that the code supplies to the mock.

With this setup, you should be able to fulfil the initial requirements and therefore stub the original behaviour and use all the Mock-provided features.

Hope this will help you all!

Any feedback is appreciated.

Cheers

Post published also on Virtuability's websitehere.

Top comments(3)

Subscribe
pic
Create template

Templates let you quickly answer FAQs or store snippets for re-use.

Dismiss
CollapseExpand
 
mauricebrg profile image
Maurice Borgmeier
A little bit of everything
  • Location
    Varna, Bulgaria
  • Work
    Freelance IT Consultant / Technical Trainer
  • Joined

Nice post :)

I'm curious howmoto didn't work for you - are the API calls not supported (yet)?

CollapseExpand
 
panilo profile image
Danilo Desole
I'm Danilo Desole, made in Italy and living in Ireland; I'm an IT Engineer specialising in Cloud technologies, focusing on AWS Cloud. I also handle WordPress websites.
  • Location
    Dublin and Milan
  • Joined

Hi Maurice! Thank you :)moto didn't work for me for 2 reasons

  1. I didn't want to configure themoto virtual AWS environment
  2. It doesn't give you the possibility to leverage Mock-related features likeboto_mocked_function.assert_called_once_with(...) and so on...

Said that though I agreemoto might fit most of the developers :)

CollapseExpand
 
mauricebrg profile image
Maurice Borgmeier
A little bit of everything
  • Location
    Varna, Bulgaria
  • Work
    Freelance IT Consultant / Technical Trainer
  • Joined

I see - yeah,moto is more suitable if you want to test the behavior on a slightly higher level, i.e. the assertion fordelete_db_cluster_snapshot would be checking if the given snapshot has been removed from thelist_db_cluster_snapshot. The other benefit ofmoto is that you get all the parameter validation the boto3 does when you use an API call.

I frequently use moto for integration tests where I want to verify the interaction of multiple components.

Are you sure you want to hide this comment? It will become hidden in your post, but will still be visible via the comment'spermalink.

For further actions, you may consider blocking this person and/orreporting abuse

I'm Danilo Desole, made in Italy and living in Ireland; I'm an IT Engineer specialising in Cloud technologies, focusing on AWS Cloud. I also handle WordPress websites.
  • Location
    Dublin and Milan
  • Joined

More fromDanilo Desole

DEV Community

We're a place where coders share, stay up-to-date and grow their careers.

Log in Create account

[8]ページ先頭

©2009-2025 Movatter.jp