Uh oh!
There was an error while loading.Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork9.6k
[Lock] AddMysqlStore
#45982
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:7.4
Are you sure you want to change the base?
[Lock] AddMysqlStore
#45982
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Thanks for taking over this feature. It's too late for 6.1, but will be a good addition for 6.2.
Since the first PR,@jderusse created PostgreSqlStore, which is very close to this MysqlStore.
Also, we split stores for PDO Vs DBAL connection. Once the PDO version is complete, you can create theDoctrineDbalMysqlStore
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
$name = (string) $key; | ||
$name = \strlen($name) > 64 ? hash('sha256', $name) : $name; | ||
$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
PostgreSqlStore
uses the InMemoryStore to avoid duplicate lock on the same connection.
returnself::$storeRegistry[$namespace] ??self::$storeRegistry[$namespace] =newInMemoryStore(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Can you elaborate on this requirement?
Im concerned that using theInMemoryStore
can cause unexpected behavior in situations where either:
- A combined store is used with with Mysql and Postgres
- A the same key is passed to two
LockFactory::createLockFromKey
, one using Mysql and the other using Postgres
I moved the state into the store since the tests assert thatexists()
is bound to theKey
instance, whilesave()
is bound to theKey::$resource
. Does this work?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
- They should not share the same instance of
InMemoryStore
$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn(); | ||
} | ||
return $this->connectionId.'_'.spl_object_id($key); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
spl_object_id
is not reliable.
The above code should work
$key = new Key('foo');$store->save($key);$key = clone($key);$this->assertTrue($store->exists($key));
private function getLockId(Key $key): string | ||
{ | ||
if (!isset($this->connectionId)) { | ||
$this->connectionId = $this->getConnection()->query('SELECT CONNECTION_ID()')->fetchColumn(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This line triggersPDOException: SQLSTATE[HY000] [2002] No such file or directory
on my local machine
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Replaced the MYSQL_HOST with MYSQL_DSN envar so you can test using a socket dsn. I think that's what your error is related to.
public function exists(Key $key): bool | ||
{ | ||
return self::$locksAcquired[$this->getLockId($key)] ?? false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
That's wrong, the implementation must assert that the lock is still present in the backend. (Ie the database restart, the network is down, ...)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
This wont be possible since GET_LOCK is bound to the connection so its always released when the connection is terminated.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Well..getLockId
is based onconnection_id()
which is (for MySQL 8 at least) an auto-increment that is reset when server starts.
This means, that each time the server restarts, the application might retrieve theconnection_id
previously affected to another process. Which makes the key used inself::$locksAcquired
untrustable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
I think this should do it.
I cant add this test case to the since it requires rebooting mysql but I did run it manually to confirm proper handling even if the connection id is the same
class ManualMysqlStore extends MysqlStoreTest{ public function testMysqlClosedConnection() { // manually restart mysql to reset conn id counter $pdo = $this->getPdo(); $storeA = new MysqlStore($pdo); $key = new Key('foo'); $storeA->save($key); $connIdA = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn(); while (true) { try { $this->assertTrue($storeA->exists($key)); sleep(1); // manually restart mysql to reset conn id counter } catch (\PDOException $e) { // mysql has gone away break; } } $pdo = $this->getPdo(); $storeB = new MysqlStore($pdo); $connIdB = $pdo->query('SELECT CONNECTION_ID()')->fetchColumn(); $this->assertFalse($storeB->exists($key)); $this->assertSame($connIdA, $connIdB); }}
$storeB = $this->getStore(); | ||
try { | ||
$storeB->save(new Key('foo')); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
The test is incorrect. You are testing both:
- 2 instances of Key
- 2 instances of store
You don't know why thestoreB->save
throws an exception.
For the record the below code is OK:
$resource = uniqid(__METHOD__, true);$key1 = new Key($resource);$key2 = new Key($resource);$storeA->save($key1);$storeb->save($key1);
Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Because stores are stateless, and you are allowed to use 2 instances of store to manipulate the same lock.
makes sense I see where you're going now
$name = (string) $key; | ||
$name = \strlen($name) > 64 ? hash('sha256', $name) : $name; | ||
$stmt = $this->conn->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
- They should not share the same instance of
InMemoryStore
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
great work@rtek, if you don't mind i have added a few remarks
return false; | ||
} | ||
$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), 1, 0)'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
this prepares the statement each time the method is called, consider moving it to after the connection is established and capturing statement in an object property
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
IF statement is technically not needed, return the result of comparison
$stmt =$this->getConnection()->prepare('SELECTIF(IS_USED_LOCK(:name) = CONNECTION_ID(), 1, 0)'); | |
$stmt =$this->getConnection()->prepare('SELECT IS_USED_LOCK(:name) = CONNECTION_ID()'); |
you would get the1
when the lock is held, andnull
when it's not (safer to assume not1
value)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
agree
$stmt->bindValue(':name', self::getLockName($key), \PDO::PARAM_STR); | ||
$stmt->execute(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
consider usingexecute
with an argument for binding (theKey
name has been already 'forced' to a string byletLockName
) - it's easier to read without semantic drawbacks
$stmt->bindValue(':name',self::getLockName($key), \PDO::PARAM_STR); | |
$stmt->execute(); | |
$stmt->execute([':name' =>self::getLockName($key)]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
agree - did the others too
} | ||
$name = self::getLockName($key); | ||
$stmt = $this->getConnection()->prepare('SELECT IF(IS_USED_LOCK(:name) = CONNECTION_ID(), -1, GET_LOCK(:name, 0))'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
some mysql peculates should be documented (and maybe this store should prevent instantiation on unsupported version or scenario):
https://dev.mysql.com/doc/refman/5.7/en/locking-functions.html
SELECT GET_LOCK('lock1',10); SELECT GET_LOCK('lock2',10); SELECT RELEASE_LOCK('lock2'); SELECT RELEASE_LOCK('lock1');
In MySQL 5.7 or later, the second [GET_LOCK()] (https://dev.mysql.com/doc/refman/5.7/en/locking-functions.html#function_get-lock) acquires a second lock and bothRELEASE_LOCK() calls return 1 (success). Before MySQL 5.7, the secondGET_LOCK() releases the first lock ('lock1') and the secondRELEASE_LOCK() returns NULL (failure) because there is no 'lock1' to release.
MySQL 5.7 and later enforces a maximum length on lock names of 64 characters. Previously, no limit was enforced.
This function is[All lock functions are] unsafe for statement-based replication. A warning is logged if you use this function whenbinlog_format is set to STATEMENT.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
please investigate the isolation of this lock acquire.
i'm not sure theIF
statement is atomic nor serialized, so i've posted the question on sfhttps://stackoverflow.com/questions/71957583/is-mysql-if-statement-atomic
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
MySQL 5.7 and later enforces a maximum length on lock names of 64 characters.
Handled bygetLockName
...unsafe for statement-based replication
After some digging, I don't believe this is relevant to the use case.
- The locks are "locked on a server-wide basis." so i dont one would expect a replica to respect locks on the master.
- The
STATEMENT
based replication warning is applicable if these various functions are used within DML (e.g.insert into foo values(is_used_lock('bar')
) so a replica would get a different value when replicating that statement (since the lock situation on the replica is not the same as the master).
please investigate the isolation of this lock acquire.
These functions do not interact with transactions per "Locks obtained with GET_LOCK() are not released when transactions commit or roll back."
Heres a quick demo:
$pdoA = ...$pdoB = ...$dump = function (\PDO $pdo, string $sql) { printf("%s : %s\n", $sql, var_export($pdo->query($sql)->fetchColumn(), true) );};$dump($pdoA, "select get_lock('foo',0)");$dump($pdoB, "select get_lock('foo',0)");$dump($pdoA, "select release_lock('foo')");$pdoA->beginTransaction();$dump($pdoA, "select get_lock('foo',0)");$dump($pdoB, "select get_lock('foo',0)");
select get_lock('foo',0) : 1select get_lock('foo',0) : 0select release_lock('foo') : 1select get_lock('foo',0) : 1select get_lock('foo',0) : 0
* @author rtek | ||
* @author Jérôme TAMARELLE <jerome@tamarelle.net> | ||
*/ | ||
class MysqlStore implements PersistingStoreInterface |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
maybe rename it toMysqlLocksStore
to indicate Mysql user locks are used (not to be confused with data storage in Pdo store
Hello@rtek do you want to continue this PR? It is fine if you want to work on in a few weeks. Let me know if you want to hand it over. |
Yes - Id like to wrap this up. Should I close and re-open to get a fresh review going? |
You can rebase and update this PR. |
} | ||
if (-1 === $result) { | ||
throw new LockConflictedException('Lock already acquired by this connection.'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what yourexists
check tries to avoid also)
// mysql limits lock name length to 64 chars | ||
$name = (string) $key; | ||
return \strlen($name) > 64 ? hash('xxh128', $name) : $name; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
hashing only longer names will mean that there is 2 resources that share the same MySQL lock name
} | ||
if (-1 === $result) { | ||
throw new LockConflictedException('Lock already acquired by this connection.'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
a Lock already acquired by this connection should not throw a LockConflictedException (btw, this is what yourexists
check tries to avoid also)
Uh oh!
There was an error while loading.Please reload this page.
6.16.2Add
MysqlStore
based on MySQLGET_LOCK()
functionalityBased on#25578
Welcoming feedback before adding docs.