Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

test: Cover scrapy.extensions.memusage.MemoryUsage#7004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Draft
prakashsellathurai wants to merge9 commits intoscrapy:master
base:master
Choose a base branch
Loading
fromprakashsellathurai:master

Conversation

@prakashsellathurai
Copy link

@prakashsellathuraiprakashsellathurai commentedAug 11, 2025
edited
Loading

fixes#7002

@prakashsellathuraiprakashsellathurai marked this pull request as ready for reviewAugust 11, 2025 12:01
@wRAR
Copy link
Member

This is a good start but I would remove all tests that check internals of the extension and add tests that run a crawl and check that it finishes with proper log messages and finish reason.

prakashsellathurai reacted with thumbs up emoji

@prakashsellathuraiprakashsellathurai marked this pull request as draftAugust 11, 2025 12:19
Disables correctly when turned off or when the resource module is missingLogs and updates stats when hitting warning or limit thresholdsSends email alerts for both warnings and hard limitsHandles normal usage without false alarms and tracks peak usageCleans up all running tasks when the spider closesbasic  CrawlerRunner and validates settings validation
@codecov
Copy link

codecovbot commentedAug 11, 2025
edited
Loading

❌ 8 Tests Failed:

Tests completedFailedPassedSkipped
368683678204
View the top 3 failed test(s) by shortest run time
tests\test_extension_memusage.py::test_extension_memusage::TestMemoryUsage::test_extension_cleanup_on_spider_close
Stack Traces | 0.032s run time
self = <tests.test_extension_memusage.TestMemoryUsage object at 0x00000191E3D9C8F0>    async def test_extension_cleanup_on_spider_close(self):        """Test that MemoryUsage extension properly cleans up when spider closes."""        settings = {            "MEMUSAGE_ENABLED": True,            "MEMUSAGE_WARNING_MB": 100,            "MEMUSAGE_CHECK_INTERVAL_SECONDS": 0.5,        }            crawler = get_crawler(TestSpider, settings)            spider = crawler._create_spider("memory_test")            # Get reference to extension before crawl        extension = None        for ext in crawler.extensions.middlewares:            if isinstance(ext, MemoryUsage):                extension = ext                break    >       assert extension is not None, "MemoryUsage extension should be loaded"E       AssertionError: MemoryUsage extension should be loadedE       assert None is not Nonetests\test_extension_memusage.py:282: AssertionError
tests\test_utils_log.py::test_utils_log::TestLogCounterHandler::test_accepted_level
Stack Traces | 0.061s run time
self = <tests.test_utils_log.TestLogCounterHandler object at 0x7ff9b3986350>crawler = <scrapy.crawler.Crawler object at 0x7ff98a1aca10>logger = <Logger test (NOTSET)>    def test_accepted_level(self, crawler: Crawler, logger: logging.Logger) -> None:        logger.error("test log msg")        assert crawler.stats>       assert crawler.stats.get_value("log_count/ERROR") == 1E       AssertionError: assert 4 == 1E        +  where 4 = get_value('log_count/ERROR')E        +    where get_value = <scrapy.statscollectors.MemoryStatsCollector object at 0x7ff98a1ea330>.get_valueE        +      where <scrapy.statscollectors.MemoryStatsCollector object at 0x7ff98a1ea330> = <scrapy.crawler.Crawler object at 0x7ff98a1aca10>.statstests/test_utils_log.py:104: AssertionError
tests\test_spider_start.py::test_spider_start::TestMain::test_start_deprecated_super
Stack Traces | 0.103s run time
self = <tests.test_spider_start.TestMain object at 0x7ff9b44d9b50>    @deferred_f_from_coro_f    async def test_start_deprecated_super(self):        class TestSpider(Spider):            name = "test"                async def start(self):                for item_or_request in super().start_requests():                    yield item_or_request            with pytest.warns(            ScrapyDeprecationWarning, match=r"use Spider\.start\(\) instead"        ) as messages:            await self._test_spider(TestSpider, [])>       assert messages[0].filename.endswith("test_spider_start.py")E       AssertionError: assert FalseE        +  where False = <built-in method endswith of str object at 0x7ff9b709cb10>('test_spider_start.py')E        +    where <built-in method endswith of str object at 0x7ff9b709cb10> = '.../scrapy/utils/reactor.py'.endswithE        +      where '.../scrapy/utils/reactor.py' = <warnings.WarningMessage object at 0x7ff98ae203b0>.filenametests/test_spider_start.py:146: AssertionError

To view more test analytics, go to theTest Analytics Dashboard
📋 Got 3 mins?Take this short survey to help us improve Test Analytics.

@prakashsellathuraiprakashsellathurai marked this pull request as ready for reviewAugust 11, 2025 19:03
…h @deferred_f_from_coro_f, used use maybe_deferred_to_future()
@prakashsellathuraiprakashsellathurai marked this pull request as draftAugust 13, 2025 18:58
@prakashsellathuraiprakashsellathurai marked this pull request as ready for reviewAugust 13, 2025 18:58

@mock.patch.object(MemoryUsage, "get_virtual_size")
def test_email_notification_on_limit(self, mock_get_size):
"""Test that email notification is sent when limit is exceeded."""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Can you explain how does this test that?

assert size > 0

def test_warning_only_triggered_once(self):
"""Test that warning is only triggered once even if threshold is exceeded multiple times."""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Can you explain how does this test that?

Comment on lines 246 to 253
# Initially not warned
assert not extension.warned

# Simulate warning triggered
extension.warned = True

# Should remain warned
assert extension.warned
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Can you explain what is tested here?

@wRAR
Copy link
Member

Please make sure the tests work locally before publishing the changes.

@wRARwRAR marked this pull request as draftAugust 13, 2025 19:56
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@wRARwRARwRAR left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Coverscrapy.extensions.memusage.MemoryUsage with tests

2 participants

@prakashsellathurai@wRAR

[8]ページ先頭

©2009-2025 Movatter.jp