Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

[Lock] Add MysqlStore that use GET_LOCK#25578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Closed
GromNaN wants to merge10 commits intosymfony:masterfromGromNaN:get_lock

Conversation

GromNaN
Copy link
Member

@GromNaNGromNaN commentedDec 22, 2017
edited
Loading

QA
Branch?master
Bug fix?no
New feature?yes
BC breaks?no
Deprecations?no
Tests pass?yes
Fixed tickets#25400
LicenseMIT
Doc PRWIP
  • Key is hashed with sha256 to ensure it stays between 1 and 64 characters.

MySQL 5.7.5 and later enforces a maximum length on lock names of 64 characters. Previously, no limit was enforced.

  • Create a new PDO connection for each lock, to avoid multiple locks on the same name in the same session. Using a shared PDO connection lead to edge cases when using multiple lock simultaneously.

It is even possible for a given session to acquire multiple locks for the same name. Other sessions cannot acquire a lock with that name until the acquiring session releases all its locks for the name.

willemstuursma, hacfi, and Aterniad reacted with thumbs up emoji
@nicolas-grekas
Copy link
Member

thanks for this PR
fabbot has some valid points,
and travis hangs on the Lock test suite for now :)

use Symfony\Component\Lock\Key;
use Symfony\Component\Lock\StoreInterface;

class MysqlStore implements StoreInterface
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

should be final

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Why not, but why ?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

We keep having this conversation again and again :)
We don't do final by default in Symfony, it would reduce the value of the code base with no real benefit for maintenance.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

On the contrary, it increases the value of the code base, because the developer can clearly see which are extension points and which are not. It will also force the developer to use composition and service decorators, because they can't get around with ugly hacks such as changing the service class.

It will actually make maintenance a lot easier because it's no longer an extension point 🤔

We don't do final by default in Symfony

Maybe we should ;)

unkind reacted with thumbs up emoji
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

We don't do final by default in Symfony

Wait, but it is not by default, it's by@iltar's request. :) I don't see valid use case to inheritMysqlStore as well. What's the point to support it? 🤔

@GromNaNGromNaNforce-pushed theget_lock branch 2 times, most recently fromf934b95 tod0ea57bCompareDecember 23, 2017 14:20
* Key is hashed with sha256 to ensure it stays between 1 and 64 characters* Create a new PDO connection for each lock, to avoid multiple locks on the same name in the same session* Makes wait timeout configurable
@GromNaN
Copy link
MemberAuthor

GromNaN commentedDec 23, 2017
edited
Loading

It is still work in progress.

  • Integrate with framework bundle configuration
  • Load MySQL service in travis
  • Update docs

@GromNaN
Copy link
MemberAuthor

GromNaN commentedDec 27, 2017
edited
Loading

In the FrameworkBundle configuration, stores are configured with a single string "dsn". The connection is initialized from this string. But we need more than a dsn to initialize a connection to Mysql (seeDoctrine\DBAL\Connection,\PDO of\MySQLi contructors).

  1. What about allowing a service name ?
  2. What about creating a new dsn syntax for db ?
framework:lock:enabled:trueresources:default:                -flock                -semaphore                -memcached://user:pass@localhost?weight=33                -redis://example.com:1234# New for MysqlStore                -"@=service(doctrine.dbal.default_connection)"                -mysql:host=localhost;port=13306;dbname=testdb;user=bruce;password=mypass# Upcoming, the PDO Postgres driver already accept this syntax:                -pgsql:host=localhost;port=5432;dbname=testdb;user=bruce;password=mypass

/cc@jderusse

@jderusse
Copy link
Member

Issue with storing key in database is: the lock has a dependency to the connection:

  • what if connection is down (think long process in console)
  • how to not auto release the lock and keep it between 2 http calls?

Copy link
Member

@jderussejderusse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

IMO we can use dsn without extra service configuration. All required information should be available in dsn.

BTW I would love to use such syntaxmysql://root:password@localhost/my_database

}

/**
* {@inheritdoc}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Do we still keep inheritDoc comments?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Yes

GromNaN reacted with thumbs up emoji
// no timeout for impatient
$timeout = $blocking ? $this->waitTimeout : 0;

$connection = new \PDO($this->dsn, $this->username, $this->password, $this->options);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Here you open a new connection for each lock (is it necessary?). It could trigger a max connections limit. Moreover you don't close theme.

Copy link
MemberAuthor

@GromNaNGromNaNDec 27, 2017
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The connection is closed automatically when thePDO instance is destructed. That's done by removing thePDOStatement from the key state.

I chose to open a new connection for each lock to be sure that we don't get the same lock multiple times on the same connection.

In MySQL 5.7.5,GET_LOCK() was reimplemented using the metadata locking (MDL) subsystem and its capabilities were extended. Multiple simultaneous locks can be acquired and GET_LOCK() does not release any existing locks.It is even possible for a given session to acquire multiple locks for the same name. Other sessions cannot acquire a lock with that name until the acquiring session releases all its locks for the name.

But if we allow the injection of a connection, we need to keep this state in-memory in the MysqlStore instance.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Indeed by sharing PDO you have to to avoid concurrency inside the same process. This side effect should be warned in the documentation IMO

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I've found this workaround to avoid multiple connections:

SELECT IF(IS_USED_LOCK(:key) ISNULL, GET_LOCK(:key, :timeout),0)

Commit is coming.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

be sure that we don't get the same lock multiple times on the same connection.

Is this really what we want? Looks opinionated to me. flock() doesn't blocks when acquiring twice a lock on a file, I would expect the same here, so that I have more choices in the end (my responsibility to create several connections if I need intra process locking)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@nicolas-grekas there is a test for that (https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Lock/Tests/Store/AbstractStoreTest.php#L80). Tools like amphp (ReactPHP) needs locks inside the same process.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

But same key (remember, a key contains the resource locked + the lock state) can be reused and locked several times inside the same process.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Ok indeed

$this->username = $options['db_username'] ?? '';
$this->password = $options['db_password'] ?? '';
$this->options = $options['db_connection_options'] ?? array();
$this->waitTimeout = $options['wait_timeout'] ?? -1;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

In others stores blocking mode is waiting forever. Such option is not needed IMO

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The "wait timeout" feature was not available in flock and semaphore, isn't it the reason why it was not implemented? It is a useful alternative toRetryTillSaveStore when the lock needs to be acquired in a limited time with a clean error handling.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It wasn't implemeted in the store because I didn't see end to end use case, and such feature can be handled by a decorator and bring the feature to every store.

@nicolas-grekas
Copy link
Member

My 2cts: it's fine if drivers have their specificities, and creating one connection per lock is too opinionated and reduces a lot the value of the implementation.

@GromNaN
Copy link
MemberAuthor

GromNaN commentedDec 28, 2017
edited
Loading

PR updated trying to get the best from all the great reviews.

Here is the implemented behavior (updated 2018-01-19) :

  • non-blocking mode
    • IF already locked by otherTHEN conflict
    • IF already locked by same connectionTHEN conflict
    • IF not lockedTHEN lock withGET_LOCK
  • blocking mode
    • IF already locked by otherTHEN wait withGET_LOCK
    • IF already locked by same connectionTHEN conflict
    • IF not lockedTHEN lock withGET_LOCK

what if connection is down (think long process in console)

Releasing the lock when the connection is lost is a feature of this specific driver. Especially useful when the task that needs this lock use the database.

how to not auto release the lock and keep it between 2 http calls?

It seems impossible. I don't have this use case.

creating one connection per lock is too opinionated and reduces a lot the value of the implementation.

Modified to inject the connection in the constructor: an instance ofPDO orDoctrine\DBAL\Connection.

@GromNaNGromNaNforce-pushed theget_lock branch 2 times, most recently from3816d61 to02558a5CompareDecember 28, 2017 16:23
@jderusse
Copy link
Member

jderusse commentedDec 28, 2017
edited
Loading

The fact that the lock is auto released when the connection to the DB is down is not a feature. Some workflow don't need connection to the DB at 100% (using external web services, sending emails, the application may be able to renew/reconnect the connection to finish the job).

Moreover the user should be able to know that it didn't have the lock anymore (which is not the case with the current implement) and should be addressed in that PR IMHO.
What's about using the putOff method to refresh the connection to MySQL (with a ping for instance) and better increase the connection TTL?

@GromNaN
Copy link
MemberAuthor

Good idea@jderusse, I've modified theputOffExpiration method to extend thewait_timeout to the given TTL.

@willemstuursma
Copy link

Note that you cannot nest locks in MySQL in MySQL < 5.7.5. This limitation is pretty important and should be documented.

Before 5.7.5, only a single simultaneous lock can be acquired and GET_LOCK() releases any existing lock.

public function putOffExpiration(Key $key, $ttl)
{
// the GET_LOCK locks forever, until the session terminates.
$stmt = $this->connection->exec('SET SESSION wait_timeout=GREATEST(@@wait_timeout, :ttl)');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

hmm if someone uses the passed$connection elsewhere (maybe in a long running cli process) then this could lead to surprises as it can be closed earlier than expected?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This SQL statement can only increase thewait_timeout (takes the max between the current value and the minimum expected value).

If the connection is closed intentionally, then the lock is automatically released. It is a limitation/feature of this locking mecanism.

$success = $stmt->fetchColumn();

if ($blocking && '-1' === $success) {
throw new LockAcquiringException('Lock already acquired with the same MySQL connection.');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Throwing that exception in blocking mode is weird IMO, the purpose of the the blocking is to wait until the lock can be acquired.

Use case: With ReactPHP and a Session Storage that used this MySQL store and 2 concurrent HTTP request (think ajax). The second HTTP request shouldn't fail just because it performs a "session_start" which trigger the lock

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

By default, the wait timeout is infinite (-1). This exception will be thrown in a specific scenario that is not

I've updated to use this "wait timeout" feature in non-blocking mode and throw the sameLockConflictedException if the lock is token by an other or the by the same connection.

}

// store the release statement in the state
$releaseStmt = $this->connection->prepare('DO RELEASE_LOCK(:key)');
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

now connection is injected in this class, do we still need to store the statement in the key?

Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

No, I've updated the code to prepare the statement in thedelete method.

@@ -50,6 +50,7 @@ public function testBlockingLocks()
// Wait the start of the child
pcntl_sigwaitinfo(array(SIGHUP), $info);

$store = $this->getStore();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

good catch

GromNaN reacted with hooray emoji
Copy link
MemberAuthor

@GromNaNGromNaN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

I've updated the class to use the wait timeout in non-blocking mode. It allows the method$lock->acquire($blocking=false) to wait for a limited time. By default, this time is 0.

public function putOffExpiration(Key $key, $ttl)
{
// the GET_LOCK locks forever, until the session terminates.
$stmt = $this->connection->exec('SET SESSION wait_timeout=GREATEST(@@wait_timeout, :ttl)');
Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This SQL statement can only increase thewait_timeout (takes the max between the current value and the minimum expected value).

If the connection is closed intentionally, then the lock is automatically released. It is a limitation/feature of this locking mecanism.

$success = $stmt->fetchColumn();

if ($blocking && '-1' === $success) {
throw new LockAcquiringException('Lock already acquired with the same MySQL connection.');
Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

By default, the wait timeout is infinite (-1). This exception will be thrown in a specific scenario that is not

I've updated to use this "wait timeout" feature in non-blocking mode and throw the sameLockConflictedException if the lock is token by an other or the by the same connection.

}

// store the release statement in the state
$releaseStmt = $this->connection->prepare('DO RELEASE_LOCK(:key)');
Copy link
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

No, I've updated the code to prepare the statement in thedelete method.

@GromNaN
Copy link
MemberAuthor

GromNaN commentedJan 19, 2018
edited
Loading

@willemstuursmathe following query ensure a consistent behavior with older mysql versions as it prevent nested lock. It returns-1 if the lock has already been acquired by the same connection :SELECT IF(IS_USED_LOCK(:key) = CONNECTION_ID(), -1, GET_LOCK(:key, :timeout))

Edit: Tested on MySQL 5.6, it doesn't support acquiring multiple locks on the same session, even with different keys. This needs to be documented or detected cleanly.

willemstuursma reacted with thumbs up emoji


$storedKey = $key->getState(__CLASS__);

$stmt = $this->connection->prepare('SELECT IF(IS_USED_LOCK(:key) = CONNECTION_ID(), 1, 0)');
Copy link
MemberAuthor

@GromNaNGromNaNJan 19, 2018
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@jderusse This query insure that: 1. the connection is still alive & 2. the lock is still assigned to the current connection.

@willemstuursma
Copy link

@GromNaN

Tested on MySQL 5.6, it doesn't support acquiring multiple locks on the same session, even with different keys. This needs to be documented or detected cleanly.

It's very nasty, there should be an exception if it tries to get a lock if the session already has one, else the other lock will (silently) be released.


/**
* @param \PDO|Connection $connection
* @param int $waitTimeout Time in seconds to wait for a lock to be released, for non-blocking lock.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

To be acquired, and not to be released right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

It will be released after the timeout so that sounds right to me. You could also say "The time in seconds the lock is acquired for".

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Regardingthe documentaiton the parameter timeout, define the maximum amount of time allowed to acquire the lock. Once you get it, you can keep it as long as you want.

T0: ask for acquiring a new lock (wait for other process to release the lock for instance)
T1: lock is acquired
T2: lock is released

Here we are talking about the time between T1 and T0.

$stmt->execute();

// 1: Lock successful
// 0: Already locked by another session
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Null, process killed by admin, or machine out of memory

@nicolas-grekasnicolas-grekas modified the milestones:4.1,nextApr 20, 2018
@jderusse
Copy link
Member

There is a big issue with the current implementation of the store. The entiere mechanism rely on the connection between the application and the database. If for some reason the connection is closed (or the database restart, or whatever), the "lock" is released for the database, but the application is not aware of that and continue to work.

I'd rather use the same pattern as#27346 by creating a dedicated table, and use atomic queries like

INSERT INTO lock_table (field, token, expired_at)  VALUES (:key, :token, :date)ON DUPLICATE KEY UPDATE  expired_at = IF(token = VALUES(token), VALUES(expired_at), expired_at);// then check itSELECT COUNT(*) FROM lock_table WHERE field=:key AND token := tokenCOMMIT;

This pattern is IMO safer, and could be used by most of database and version
A garbage collection to purge old locks could be run randomly

GromNaN reacted with thumbs up emoji

@jderussejderusse mentioned this pull requestMay 31, 2018
@GromNaN
Copy link
MemberAuthor

I don't have time to rework this PR. I close it to let anyone interested works on this feature.

@GromNaNGromNaN closed thisJun 5, 2018
fabpot added a commit that referenced this pull requestSep 4, 2018
This PR was merged into the 4.2-dev branch.Discussion----------[LOCK] Add a PdoStore| Q             | A| ------------- | ---| Branch?       | master| Bug fix?      | no| New feature?  | yes| BC breaks?    | no| Deprecations? | no| Tests pass?   | yes| Fixed tickets |#25400| License       | MIT| Doc PR        |symfony/symfony-docs#9875This is an alternative to#25578Commits-------46fe1b0 Add a PdoStore in lock
@nicolas-grekasnicolas-grekas modified the milestones:next,4.2Nov 1, 2018
@rtekrtek mentioned this pull requestApr 9, 2022
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@nicolas-grekasnicolas-grekasnicolas-grekas left review comments

@hacfihacfihacfi left review comments

@jderussejderussejderusse left review comments

@dmaicherdmaicherdmaicher left review comments

@unkindunkindunkind left review comments

@linaorilinaorilinaori left review comments

Assignees
No one assigned
Projects
None yet
Milestone
4.2
Development

Successfully merging this pull request may close these issues.

10 participants
@GromNaN@nicolas-grekas@jderusse@willemstuursma@hacfi@dmaicher@unkind@linaori@xabbuh@carsonbot

[8]ページ先頭

©2009-2025 Movatter.jp