Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

WiP: Fix looping Cloudflare challenge, Resolves #1036#1163

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Draft
ilike2burnthing wants to merge2 commits intoFlareSolverr:master
base:master
Choose a base branch
Loading
fromilike2burnthing:looping

Conversation

ilike2burnthing
Copy link
Contributor

Thanks to@juanfrilla for#1036 (comment).

Unfortunately, currently this only works on Windows, and the looping challenges return if using proxies or VPNs.

juanfrilla, Hunam6, m33ts4k0z, SickProdigy, thedutchruben, depado, Foxy-NC, Gauvino, Net005, pacobarbe31, and 10 more reacted with heart emoji
@garfield69
Copy link
Contributor

garfield69 commentedApr 21, 2024
edited
Loading

FWIW
My win10 is on chrome 124, and i don't use VPN or proxy.
I've tested this PR (as a source based python run), and it solves for trupornolabs, riperam, marinetracker, devil-torrents, 52BT, which were indexers that were giving me issues previoulsy.
Also tested against most of the other cloudflare protected indexers that were previously working for me, and they continue to work with this PR.
Some indexers however continue to fail, leporno still returns the invalid cookies error, and ext-torrents which now fails on ext.to but works for the other 2 alternate domains.

But after each solve there remains a chrome subtask that starts to spin up to 15% CPU and I have to manually kill them off.
Should I test using this PR win10 exe?
[edit] Oh wait, there isn't one.

@juanfrilla
Copy link

juanfrilla commentedApr 21, 2024
edited
Loading

Another thing that I've noticed is that in the user-agent headless replacement:

self.execute_cdp_cmd("Network.setUserAgentOverride",                    {"userAgent":self.execute_script("return navigator.userAgent"                        ).replace("Headless","")                    },                )

I don't know why but If I hardcode the user-agent using the exact that my computer has like this:

user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36"options.add_argument(f"--user-agent={user_agent}")

it bypasses cloudflare, but if i put this to make it automatically like you have it on line 533 fromsrc/undetected_chromedriver/__init__.py it does not work.

So an alternative could be to setup a driver only to get the user agent:

defget_user_agent(driver):returndriver.execute_script("return navigator.userAgent;").replace("Headless","")

And then pass the user-agent to the definitive driver

PD: I only can tell you what I've discovered to see if we can go through the solution cuz I'm having troubles to get the project installed/set up 😅

@m33ts4k0z
Copy link

m33ts4k0z commentedApr 21, 2024
edited
Loading

Hello. I just wanted to let you know that this didnt bypass the challenge on arab-torrents.net, showing the same internal server error. I am not using VPN, proxies and I dont have a datacenter IP. Let me know if you need any info to troubleshoot this further.

I didnt actually use this branch, it worked fine after I switched to it. Thanks

@ilike2burnthing
Copy link
ContributorAuthor

ilike2burnthing commentedApr 21, 2024
edited
Loading

@garfield69 yea this seems to be an issue with Chrome v124. You can revert to v123 in the mean time if it's easier -#1161

Alternatively, build your own binaries, which will use Chromium v123:

pip install -r requirements.txtpip install pyinstallercd srcpython build_package.py

@ilike2burnthing
Copy link
ContributorAuthor

@m33ts4k0z were you doing this on Windows?

@m33ts4k0z
Copy link

@m33ts4k0z were you doing this on Windows?

Yes on a Windows 11 VM on Unraid but it did work in the end. I updated my first post here with the cause.

ilike2burnthing reacted with thumbs up emoji

@garfield69
Copy link
Contributor

Alternatively, build your own binaries, which will use Chromium v123:

python src/build_package.py

Oh cool, did not know I could build on windows.
Built successfully and tested. Much better, no left over chrome tasks chewing CPU anymore :-)

ilike2burnthing reacted with thumbs up emoji

@ilike2burnthing
Copy link
ContributorAuthor

ilike2burnthing commentedApr 22, 2024
edited
Loading

@juanfrilla sorry for the delay in replying, been busy and only got to a few quick ones on my phone.

I'll have a look at the UA idea when I next get a chance, thanks.

Assuming you're following the run from source instructions, what issue are you having?https://github.com/FlareSolverr/FlareSolverr#from-source-code

@juanfrilla
Copy link

@ilike2burnthing my main problem is that i cannot install Xvfb on MacOS

@ilike2burnthing
Copy link
ContributorAuthor

Tried XQuartz?

@juanfrilla
Copy link

juanfrilla commentedApr 22, 2024
edited
Loading

Tried XQuartz?

yessir now the project is set up, let's see what I can fix

@21hsmw
Copy link
Contributor

What exactly is left to do on this to get it merge? I tried to guess with the comments here and some different issues but I can't get the current status of this. It seems to be stale for quite some time, so what's needed?

@ilike2burnthing
Copy link
ContributorAuthor

Unfortunately, currently this only works on Windows, and the looping challenges return if using proxies or VPNs.

I'll have a look at the UA idea when I next get a chance, thanks.

@21hsmw
Copy link
Contributor

Unfortunately, currently this only works on Windows, and the looping challenges return if using proxies or VPNs.

I'll have a look at the UA idea when I next get a chance, thanks.

Well, I made my own implementation of this "new tab" idea and I was able to make it work with every website I could (ext.to, www3.yggtorrent.cool, dodi-repacks.site, hd-torrents.me/login.php, nhentai.net) on my Linux system using a VPN / socks5 proxy and also with my container image on my own remote Linux server, which was blocked by cloudflare too.
Unfortunately I can't test on Windows, so if someone can test that and report back please do.

Public image with my edits:21hsmw/flaresolverr:fixlooping
Code here:21hsmw@da6cc9d

Diudid, boubiro, depado, vrozaksen, CyberPoison, bertho-zero, pacobarbe31, Tereus12, Seros, PonyWarrior, and 5 more reacted with thumbs up emojiBreakerZero reacted with thumbs down emojiboubiro, vrozaksen, CyberPoison, pacobarbe31, notnotdee, devIlluSioN, and JohnnyPh reacted with hooray emojiTayZ3r, younesehb, CyberPoison, pilar6195, Diudid, boubiro, depado, vrozaksen, AnonTester, gauthier-th, and 6 more reacted with heart emojiTayZ3r, younesehb, CyberPoison, pilar6195, Diudid, boubiro, depado, vrozaksen, pacobarbe31, Koenic, and 2 more reacted with rocket emoji

@ilike2burnthing
Copy link
ContributorAuthor

That's working 95% of the time on Windows for me, even with a proxy, but failing 95% of the time on Docker. Usual error:

Error: Error solving the challenge. 'NoneType' object has no attribute 'startswith'

Seems it's related toget_correct_window and trying to getdriver.current_url. Adding some extra logging shows that the URL is returned asNone. Adding some additional sleeps then shows the correct URL, but I'm still getting challenge loops or crashed.

@21hsmw
Copy link
Contributor

That's working 95% of the time on Windows for me, even with a proxy, but failing 95% of the time on Docker.

When you say it fails on Docker, is it still on Windows or Linux?

I got this error on Linux while doing my implementation, but have not been able to replicate it since. For the looping challenges, it seems to be a timing issue. Playing with the timer values can make it work in some cases, but it's not easy to know what works for everyone since it seems to take network latency into account. For example, if I use a proxy close to my location, it works 100% of the time with the sites I listed earlier, but if I use a proxy very far from me, it works 50% of the time.
Can you try to increase the timers to something like 6, 8 or more?

@ilike2burnthing
Copy link
ContributorAuthor

Linux.

I'll play around with timings again (I did a bunch yesterday), see if I can get something that works both on my Docker and Windows.

@21hsmw
Copy link
Contributor

Linux.

Strange then. I'm able to solve the challenges of all sites I try on my Debian and Fedora systems with different VPNs/Proxies with and without Docker involved.
Can you share an example of one of your tests with debug enabled?

Here's an example with dodi-repacks.site using the docker image I shared previously:
https://pastebin.com/nBramRXq

@aevrard

This comment was marked as off-topic.

@zenderzender
Copy link

Unfortunately, currently this only works on Windows, and the looping challenges return if using proxies or VPNs.

I'll have a look at the UA idea when I next get a chance, thanks.

Well, I made my own implementation of this "new tab" idea and I was able to make it work with every website I could (ext.to, www3.yggtorrent.cool, dodi-repacks.site, hd-torrents.me/login.php, nhentai.net) on my Linux system using a VPN / socks5 proxy and also with my container image on my own remote Linux server, which was blocked by cloudflare too. Unfortunately I can't test on Windows, so if someone can test that and report back please do.

Public image with my edits:21hsmw/flaresolverr:fixlooping Code here:21hsmw@da6cc9d

Thanks for your workaround@21hsmw
Here is a temporary image for anybody needing arm build :)
zender/flaresolverr-fixed:arm

Working withLANG=fr-FR

@aevrard the solution you provide will kill the killswitch if you're using something like gluetun...

@LoicBrison
Copy link

Thanks@21hsmw !
Works for YGG with YGGCookie and YGGtorrent; LANG=en_US

depado and butts136 reacted with thumbs up emoji

@vrozaksen
Copy link

vrozaksen commentedMay 3, 2024
edited
Loading

21hsmw/flaresolverr:fixlooping

Worked for me on whatbox.ca

services:flaresolverr:image:21hsmw/flaresolverr:fixloopingenvironment:           -LOG_LEVEL=${LOG_LEVEL:-info}           -LOG_HTML=${LOG_HTML:-false}           -CAPTCHA_SOLVER=${CAPTCHA_SOLVER:-none}           -TZ=UTC           -PORT=25000           -HOST=127.0.0.1network_mode:hostpull_policy:alwaysrestart:unless-stopped
depado and AnonTester reacted with thumbs up emoji

@juanfrilla
Copy link

juanfrilla commentedMay 3, 2024
edited
Loading

Unfortunately, currently this only works on Windows, and the looping challenges return if using proxies or VPNs.

I'll have a look at the UA idea when I next get a chance, thanks.

Well, I made my own implementation of this "new tab" idea and I was able to make it work with every website I could (ext.to, www3.yggtorrent.cool, dodi-repacks.site, hd-torrents.me/login.php, nhentai.net) on my Linux system using a VPN / socks5 proxy and also with my container image on my own remote Linux server, which was blocked by cloudflare too. Unfortunately I can't test on Windows, so if someone can test that and report back please do.

Public image with my edits:21hsmw/flaresolverr:fixlooping Code here:21hsmw@da6cc9d

replacing the image of the dockerfile for this:
python:3.11-slim-bullseye works perfectly locally on MacOS M2 with and without proxies (tested for my website"https://www.icj-cij.org/sites/default/files/case-related/187/187-20231215-ord-01-00-en.pdf", I can get the cf_clearance cookie

I tested as well on a centOS server with the previous image (python:3.11-slim-bookworm) and it doesnt work

@tifo71

This comment was marked as duplicate.

@S-C-H
Copy link

I believe@21hsmw you are aware of this one - although reading through the comments on this discussion, I did see some attempts were made to fix sessions. I can create a session but when I try to destroy it, it errors and then the task queue grows (doesn't complete).

2025-02-08 21:24:52 ERROR    Task exception was never retrievedfuture: <Task finished name='Task-2978' coro=<Connection.aclose() done, defined at /app/nodriver/core/connection.py:298> exception=RuntimeError("Task <Task pending name='Task-2978' coro=<Connection.aclose() running at /app/nodriver/core/connection.py:306>> got Future <Future pending cb=[shield.<locals>._outer_done_callback() at /usr/local/lib/python3.11/asyncio/tasks.py:908]> attached to a different loop")>Traceback (most recent call last):  File "/app/nodriver/core/connection.py", line 306, in aclose    await self.websocket.close()  File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/connection.py", line 617, in close    async with self.send_context():  File "/usr/local/lib/python3.11/contextlib.py", line 217, in __aexit__    await anext(self.gen)  File "/usr/local/lib/python3.11/site-packages/websockets/asyncio/connection.py", line 916, in send_context    await asyncio.shield(self.connection_lost_waiter)RuntimeError: Task <Task pending name='Task-2978' coro=<Connection.aclose() running at /app/nodriver/core/connection.py:306>> got Future <Future pending cb=[shield.<locals>._outer_done_callback() at /usr/local/lib/python3.11/asyncio/tasks.py:908]> attached to a different loop2025-02-08 21:24:52 WARNING  Loop <_UnixSelectorEventLoop running=False closed=True debug=False> that handles pid 1183 is closed2025-02-08 21:26:31 WARNING  Task queue depth is 12025-02-11 07:40:46 WARNING  Task queue depth is 2

@ilike2burnthing
Copy link
ContributorAuthor

@21hsmw looks like the Docker version is being detected, whereas running from source on Windows is working fine. Basically the challenge is found, checkbox is clicked, but challenge just reloads. I tried an apt-get update and upgrade, and restarting the container, but no luck.

BadFlo, sodanexus, S-C-H, and xavier8706 reacted with thumbs up emojisodanexus reacted with eyes emoji

@laur89
Copy link

@garfield69 yea this seems to be an issue with Chrome v124. You can revert to v123 in the mean time if it's easier -#1161

Alternatively, build your own binaries, which will use Chromium v123:

pip install -r requirements.txtpip install pyinstallercd srcpython build_package.py

Want to compare dockerized vs native performance; running locally however gives me:

$ python -u src/flaresolverr.py Traceback (most recent call last):  File"/tmp/FlareSolverr/src/flaresolverr.py", line 7,in<module>    from bottle import run, response, Bottle, request, ServerAdapter  File"/home/laur89/.pyenv/versions/flaresolverr/lib/python3.13/site-packages/bottle.py", line 38,in<module>    import base64, cgi, email.utils, functools, hmac, itertools, mimetypes,\            os, re, subprocess, sys, tempfile, threading, time, warnings, hashlibModuleNotFoundError: No module named'cgi'

Any pointers?

@Mylloon
Copy link

either downgrade python to 3.11 or install legacy-cgi with pip, i guess

nrg101 and laur89 reacted with thumbs up emoji

@nrg101
Copy link

"It was removed in Python 3.13 after being deprecated in Python 3.11.
....
The last version of Python that provided the cgi module was Python 3.12"

https://docs.python.org/3/library/cgi.html

laur89 reacted with thumbs up emoji

@tifo71

This comment was marked as spam.

@ilike2burnthing
Copy link
ContributorAuthor

@tifo71 you have been warned plenty, and been around for long enough to see others warned as well. The issue has already been reported, your comment was unnecessary. Stop posting what is essentially just spam. If you do so again, you'll be blocked. No, do not reply to this comment.

jfmartinr, romquenin, BadFlo, and Gauvino reacted with thumbs up emoji

@tifo71

This comment was marked as spam.

@cedricdu78
Copy link

cedricdu78 commentedFeb 22, 2025
edited
Loading

Flaresolverr full work for me on VM ubuntu with Google chrome (no docker)
Maybe install google chrome in your docker image and I think work :)

@tifo71
Copy link

Flaresolverr full work for me on VM ubuntu with Google chrome (no docker) Maybe install google chrome in your docker image and I think work :)

thanks, finally i put alexfozor's image and it works mais 21hsmw will probably do an update so that it works with its version

@Jojont54
Copy link

Flaresolverr full work for me on VM ubuntu with Google chrome (no docker) Maybe install google chrome in your docker image and I think work :)

thanks, finally i put alexfozor's image and it works mais 21hsmw will probably do an update so that it works with its version

Hello, on my side alexfozor's image doesn't works I guess it's due to the fact I have an intel processor on my unraid station meaning I need 386 version or something like that.

@nlaille

This comment was marked as duplicate.

@tifo71

This comment was marked as duplicate.

@xavier8706

This comment was marked as duplicate.

@tifo71
Copy link

I can confirm that alexfozor/flaresolverr:pr-1300-experimental is working (the v2 does not work) as a workaround for me

Yes, there are currently 2 versions that work. so Alexfozor's image: alexfozor/flaresolverr:pr-1300-experimental

and another called Byparr: ghcr.io/thephaseless/byparr:latest

the current image of@21hsmw does not work. Waiting for an update

gotson, Swish24, rdiulian212, and 42aroger reacted with heart emoji

@21hsmw
Copy link
Contributor

My branch has been updated after the recent captcha looping issue.
If you see any change in behavior, please report it to me here.

  • Update Chromium to version 133
  • Update nodriver to version 0.40
  • Fix captcha looping issue

Image:21hsmw/flaresolverr:nodriver

Code:https://github.com/21hsmw/FlareSolverr

RalphORama and tp0 reacted with thumbs up emojilaur89, xavier8706, depado, SadE54, navels, sebion59, actuallyevan, romquenin, hyzor, and RalphORama reacted with hooray emojiSadE54, actuallyevan, romquenin, Sxtanna, and RalphORama reacted with rocket emoji

@mokhunv-paloalto
Copy link

My branch has been updated after the recent captcha looping issue. If you see any change in behavior, please report it to me here.

  • Update Chromium to version 133
  • Update nodriver to version 0.40
  • Fix captcha looping issue

Image:21hsmw/flaresolverr:nodriver

Code:https://github.com/21hsmw/FlareSolverr

Hi, does your solution work with non-interactive cloudflare challenges? Even with your latest changes, I get "Challenge not detected!" when trying to access a page

@ilike2burnthing
Copy link
ContributorAuthor

You'll need to give an example URL for anyone to be able to help you.

@sebion59

This comment was marked as spam.

@mokhunv-paloalto
Copy link

mokhunv-paloalto commentedMar 6, 2025
edited
Loading

You'll need to give an example URL for anyone to be able to help you.

Yea, sorry, here is aURL.
It has a non-interactive challenge, and when trying to bypass I get that challenge is not found.
@21hsmw is your solution supposed to work with those kinds of turnstiles?

@ilike2burnthingilike2burnthing mentioned this pull requestMar 7, 2025
4 tasks
@ilike2burnthing
Copy link
ContributorAuthor

FlareSolverr doesn't currently work with this type of integrated check, just whole page challenges.

mokhunv-paloalto reacted with thumbs up emoji

@xlyralycanx

This comment was marked as spam.

@ilike2burnthing
Copy link
ContributorAuthor

@21hsmw updating Chrome to v135 on Windows is causing problems:

  • launches but then almost immediately returns the errorError solving the challenge. cannot unpack non-iterable NoneType object
  • reverting tov134 works
  • working on Docker (didn't test updating Chromium though)

@thebetauser
Copy link

@21hsmw updating Chrome to v135 on Windows is causing problems:

  • launches but then almost immediately returns the errorError solving the challenge. cannot unpack non-iterable NoneType object
  • reverting tov134 works
  • working on Docker (didn't test updating Chromium though)

Would it be possible to set the chrome version to download/run via ENV variables? This is more convenient for docker users who pull from the repo instead of rebuild images.

@21hsmw Any chance that you can release docker images for arm64 arch?

@ilike2burnthing
Copy link
ContributorAuthor

v3.3.22 - Windows and Linux binaries available, as is an updatedflaresolverr/flaresolverr Docker image.

If it's working, only react to this comment, DO NOT leave your own comment.

If it's not working, see this Discussion -#1501

jcolfej, lomanski, qpeten, and dragonmasta37070 reacted with thumbs up emojijcolfej, navels, and lomanski reacted with hooray emoji

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
help wantedExtra attention is neededneeds investigation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

69 participants
@ilike2burnthing@garfield69@juanfrilla@m33ts4k0z@21hsmw@aevrard@zenderzender@LoicBrison@vrozaksen@itanivalkyrie@Maiikoo@Hyperz@0rsa@tenettow@researchersec@Larryadd@xinghui233@Investigamer@danutzzzzz@Gamegenie13@today2004@zxsleebu@TonyLiooo@DilpreetSingh332Codez@PonyWarrior@japnimedev@nanmuye@simonc56@SadE54@Noaaam@frederik-uni@xcybermanx@alanstrok@tifo71@bonswouar@flipper203@Gauvino@lolomin@BridgeSenseDev@x-tropic86@ukharley@HellxGodLike@Kamalaja@nonopag@romquenin@nathang21@ggogel@sodanexus@bibiwan@Amcro@Bembix23@OpenPrunus@stevegoossens@OscarPoels@danielw97@Fuyucch1@gfediere@S-C-H@laur89@Mylloon@nrg101@cedricdu78@Jojont54@nlaille@xavier8706@mokhunv-paloalto@sebion59@xlyralycanx@thebetauser

[8]ページ先頭

©2009-2025 Movatter.jp