Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

[Lock] AddMysqlStore#45982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
rtek wants to merge11 commits intosymfony:7.4
base:7.4
Choose a base branch
Loading
fromrtek:6.1
Open

[Lock] AddMysqlStore#45982

rtek wants to merge11 commits intosymfony:7.4fromrtek:6.1

Conversation

rtek
Copy link
Contributor

@rtekrtek commentedApr 9, 2022
edited
Loading

QA
Branch?6.1 6.2
Bug fix?no
New feature?yes
Deprecations?no
TicketsFix#45251
LicenseMIT
Doc PRTODO

AddMysqlStore based on MySQLGET_LOCK() functionality

Based on#25578

Welcoming feedback before adding docs.

@GromNaNGromNaN self-requested a reviewApril 10, 2022 06:20
Copy link
Member

@GromNaNGromNaN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks for taking over this feature. It's too late for 6.1, but will be a good addition for 6.2.

Since the first PR,@jderusse created PostgreSqlStore, which is very close to this MysqlStore.

Also, we split stores for PDO Vs DBAL connection. Once the PDO version is complete, you can create theDoctrineDbalMysqlStore

$name = (string) $key;
$name = \strlen($name) > 64 ? hash('sha256', $name) : $name;

$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))');
Copy link
Member

@GromNaNGromNaNApr 10, 2022
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

PostgreSqlStore uses the InMemoryStore to avoid duplicate lock on the same connection.

returnself::$storeRegistry[$namespace] ??self::$storeRegistry[$namespace] =newInMemoryStore();

Copy link
ContributorAuthor

@rtekrtekApr 11, 2022
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Can you elaborate on this requirement?

Im concerned that using theInMemoryStore can cause unexpected behavior in situations where either:

  1. A combined store is used with with Mysql and Postgres
  2. A the same key is passed to twoLockFactory::createLockFromKey, one using Mysql and the other using Postgres

I moved the state into the store since the tests assert thatexists() is bound to theKey instance, whilesave() is bound to theKey::$resource. Does this work?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

  1. They should not share the same instance ofInMemoryStore

$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn();
}

return $this->connectionId.'_'.spl_object_id($key);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

spl_object_id is not reliable.

The above code should work

$key = new Key('foo');$store->save($key);$key = clone($key);$this->assertTrue($store->exists($key));

private function getLockId(Key $key): string
{
if (!isset($this->connectionId)) {
$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This line triggersPDOException: SQLSTATE[HY000] [2002] No such file or directory on my local machine

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Replaced the MYSQL_HOST with MYSQL_DSN envar so you can test using a socket dsn. I think that's what your error is related to.


public function exists(Key $key): bool
{
return self::$locksAcquired[$this->getLockId($key)] ?? false;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

That's wrong, the implementation must assert that the lock is still present in the backend. (Ie the database restart, the network is down, ...)

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This wont be possible since GET_LOCK is bound to the connection so its always released when the connection is terminated.

Copy link
Member

@jderussejderusseApr 11, 2022
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Well..getLockId is based onconnection_id() which is (for MySQL 8 at least) an auto-increment that is reset when server starts.

This means, that each time the server restarts, the application might retrieve theconnection_id previously affected to another process. Which makes the key used inself::$locksAcquired untrustable.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I think this should do it.

I cant add this test case to the since it requires rebooting mysql but I did run it manually to confirm proper handling even if the connection id is the same

class ManualMysqlStore extends MysqlStoreTest{    public function testMysqlClosedConnection()    {        // manually restart mysql to reset conn id counter        $pdo = $this->getPdo();        $storeA = new MysqlStore($pdo);        $key = new Key('foo');        $storeA->save($key);        $connIdA = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn();        while (true) {            try {                $this->assertTrue($storeA->exists($key));                sleep(1);                // manually restart mysql to reset conn id counter            } catch (\PDOException $e) {                // mysql has gone away                break;            }        }        $pdo = $this->getPdo();        $storeB = new MysqlStore($pdo);        $connIdB = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn();        $this->assertFalse($storeB->exists($key));        $this->assertSame($connIdA, $connIdB);    }}

$storeB = $this->getStore();

try {
$storeB->save(new Key('foo'));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The test is incorrect. You are testing both:

  • 2 instances of Key
  • 2 instances of store

You don't know why thestoreB->save throws an exception.

For the record the below code is OK:

$resource = uniqid(__METHOD__, true);$key1 = new Key($resource);$key2 = new Key($resource);$storeA->save($key1);$storeb->save($key1);

Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.

makes sense I see where you're going now

$name = (string) $key;
$name = \strlen($name) > 64 ? hash('sha256', $name) : $name;

$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

  1. They should not share the same instance ofInMemoryStore

Copy link
Contributor

@kaznovackaznovac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

great work@rtek, if you don't mind i have added a few remarks

return false;
}

$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), 1, 0)');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

this prepares the statement each time the method is called, consider moving it to after the connection is established and capturing statement in an object property

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

IF statement is technically not needed, return the result of comparison

Suggested change
$stmt =$this->getConnection()->prepare('SELECTIF(IS_USED_LOCK(:name) = CONNECTION_ID(), 1, 0)');
$stmt =$this->getConnection()->prepare('SELECT IS_USED_LOCK(:name) = CONNECTION_ID()');

you would get the1 when the lock is held, andnull when it's not (safer to assume not1 value)

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

agree

Comment on lines 101 to 102
$stmt->bindValue(':name', self::getLockName($key), \PDO::PARAM_STR);
$stmt->execute();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

consider usingexecute with an argument for binding (theKey name has been already 'forced' to a string byletLockName) - it's easier to read without semantic drawbacks

Suggested change
$stmt->bindValue(':name',self::getLockName($key), \PDO::PARAM_STR);
$stmt->execute();
$stmt->execute([':name' =>self::getLockName($key)]);

Copy link
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

agree - did the others too

}

$name = self::getLockName($key);
$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

some mysql peculates should be documented (and maybe this store should prevent instantiation on unsupported version or scenario):

https://dev.mysql.com/doc/refman/5.7/en/locking-functions.html

  •   SELECT GET_LOCK('lock1',10);  SELECT GET_LOCK('lock2',10);  SELECT RELEASE_LOCK('lock2');  SELECT RELEASE_LOCK('lock1');

    In MySQL 5.7 or later, the second [GET_LOCK()] (https://dev.mysql.com/doc/refman/5.7/en/locking-functions.html#function_get-lock) acquires a second lock and bothRELEASE_LOCK() calls return 1 (success). Before MySQL 5.7, the secondGET_LOCK() releases the first lock ('lock1') and the secondRELEASE_LOCK() returns NULL (failure) because there is no 'lock1' to release.

  • MySQL 5.7 and later enforces a maximum length on lock names of 64 characters. Previously, no limit was enforced.

  • This function is [All lock functions are] unsafe for statement-based replication. A warning is logged if you use this function whenbinlog_format is set to STATEMENT.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

please investigate the isolation of this lock acquire.

i'm not sure theIF statement is atomic nor serialized, so i've posted the question on sfhttps://stackoverflow.com/questions/71957583/is-mysql-if-statement-atomic

Copy link
ContributorAuthor

@rtekrtekApr 22, 2022
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

MySQL 5.7 and later enforces a maximum length on lock names of 64 characters.

Handled bygetLockName

...unsafe for statement-based replication

After some digging, I don't believe this is relevant to the use case.

  1. The locks are "locked on a server-wide basis." so i dont one would expect a replica to respect locks on the master.
  2. TheSTATEMENT based replication warning is applicable if these various functions are used within DML (e.g.insert into foo values(is_used_lock('bar')) so a replica would get a different value when replicating that statement (since the lock situation on the replica is not the same as the master).

please investigate the isolation of this lock acquire.

These functions do not interact with transactions per "Locks obtained with GET_LOCK() are not released when transactions commit or roll back."

Heres a quick demo:

$pdoA = ...$pdoB = ...$dump = function (\PDO $pdo, string $sql) {    printf("%s : %s\n",         $sql,        var_export($pdo->query($sql)->fetchColumn(), true)    );};$dump($pdoA, "select get_lock('foo',0)");$dump($pdoB, "select get_lock('foo',0)");$dump($pdoA, "select release_lock('foo')");$pdoA->beginTransaction();$dump($pdoA, "select get_lock('foo',0)");$dump($pdoB, "select get_lock('foo',0)");
select get_lock('foo',0) : 1select get_lock('foo',0) : 0select release_lock('foo') : 1select get_lock('foo',0) : 1select get_lock('foo',0) : 0

kaznovac reacted with heart emojikaznovac reacted with rocket emoji
* @author rtek
* @author Jérôme TAMARELLE <jerome@tamarelle.net>
*/
class MysqlStore implements PersistingStoreInterface
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

maybe rename it toMysqlLocksStore to indicate Mysql user locks are used (not to be confused with data storage in Pdo store

@fabpotfabpot modified the milestones:6.1,6.2Apr 22, 2022
@nicolas-grekasnicolas-grekas removed this from the6.2 milestoneNov 5, 2022
@nicolas-grekasnicolas-grekas added this to the6.3 milestoneNov 5, 2022
@GromNaN
Copy link
Member

Hello@rtek do you want to continue this PR? It is fine if you want to work on in a few weeks. Let me know if you want to hand it over.

@rtek
Copy link
ContributorAuthor

Yes - Id like to wrap this up. Should I close and re-open to get a fresh review going?

@GromNaN
Copy link
Member

Yes - Id like to wrap this up. Should I close and re-open to get a fresh review going?

You can rebase and update this PR.

@nicolas-grekasnicolas-grekas modified the milestones:6.3,6.4May 23, 2023
@nicolas-grekasnicolas-grekas modified the milestones:6.4,7.1Nov 15, 2023
@xabbuhxabbuh modified the milestones:7.1,7.2May 15, 2024
@fabpotfabpot modified the milestones:7.2,7.3Nov 20, 2024
}

if (-1 === $result) {
throw new LockConflictedException('Lock already acquired by this connection.');
Copy link

@javarrjavarrDec 18, 2024
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hi, thanks for your work!, I have one feedback:
I think that is necessary to throw different type of exception if user already has LOCK, i sow the implementation in vendor/symfony/lock/Lock.php:69, is better to ignore it if the user/connection already has the lock, or i am wrong?

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what yourexists check tries to avoid also)

// mysql limits lock name length to 64 chars
$name = (string) $key;

return \strlen($name) > 64 ? hash('xxh128', $name) : $name;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

hashing only longer names will mean that there is 2 resources that share the same MySQL lock name

kaznovac reacted with thumbs up emoji
}

if (-1 === $result) {
throw new LockConflictedException('Lock already acquired by this connection.');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what yourexists check tries to avoid also)

@fabpotfabpot modified the milestones:7.3,7.4May 26, 2025
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@kaznovackaznovackaznovac left review comments

@stofstofstof left review comments

@javarrjavarrjavarr left review comments

@GromNaNGromNaNGromNaN requested changes

@jderussejderusseAwaiting requested review from jderussejderusse is a code owner

Assignees
No one assigned
Projects
None yet
Milestone
7.4
Development

Successfully merging this pull request may close these issues.

[Lock] MySQL non-expiring locking usingGET_LOCK
10 participants
@rtek@GromNaN@kaznovac@stof@jderusse@javarr@fabpot@nicolas-grekas@xabbuh@carsonbot

[8]ページ先頭

©2009-2025 Movatter.jp