swooleを試してみた

More Related Content

1.バッチ処理を早くしたいswooleを試してみた2019/05/24Arcana Meetup #50勝見　幸弘
2.swooleとは• phpのエクステンション。
- composerでinstallするやつじゃなくてmakeとかするやつ• PHPerKaigi2019でいくつか取り上げられてました。• phpで非同期処理ができたりします。• 試したい！
3.swooleのサイトhttps://www.swoole.com/
4.swooleの導入方法• https://github.com/swoole/swoole-srcgit clone https://github.com/swoole/swoole-src.gitcd swoole-srcphpize./configuremake && make installの後にextensionファイルをphp.iniに追加する• 導入自体は簡単。ただ、xdebugが入ってると警告が出たのでxdebugは外す必要がありそうです。
5.httpで試すのは大変そうなので、コマンドで試します
6.試すコマンド
7.こんな感じのテーブルレコードにCREATE TABLE `users` (`id` int(11) NOT NULL AUTO_INCREMENT,`mail` varchar(255) NOT NULL,`pass` varchar(255) NOT NULL,`update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,PRIMARY KEY (`id`)
8.こんな処理private function passHash(){Log::info(‘passHash’.’_start’);while(true){$datas = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->limit(100)->get();if ($datas->isEmpty()) {break;}foreach($datas as $data){$datas = DB::table('users')->where('id', $data->id)->update(['pass' => Hash::make($data->pass)]);}}Log::info('passHash'.'_end');}
9.ざっくり説明• とあるテーブルのとあるカラムのデータをphpでHash::makeってやった後のデータで更新する処理。
10.private function passHash(){Log::info(‘passHash’.’_start’);while(true){$datas = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->limit(100)->get();if ($datas->isEmpty()) {break;}foreach($datas as $data){$datas = DB::table('users')->where('id', $data->id)->update(['pass' => Hash::make($data->pass)]);}}Log::info('passHash'.'_end');}こんな処理とあるテーブルのデータを取得
11.private function passHash(){Log::info(‘passHash’.’_start’);while(true){$datas = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->limit(100)->get();if ($datas->isEmpty()) {break;}foreach($datas as $data){$datas = DB::table('users')->where('id', $data->id)->update(['pass' => Hash::make($data->pass)]);}}Log::info('passHash'.'_end');}44GT44KT44Gq5Yem55CGHash::makeでデータ更新
12.cHJpdmF0ZSBmdW5jdGlvbiBwYXNzSGFzaCgpew==XExvZzo6aW5mbygncGFzc0hhc2gnLidfc3RhcnQnKTs=d2hpbGUodHJ1ZSl7JGRhdGFzID0gXERCOjp0YWJsZSgndXNlcnMnKS0+d2hlcmUoXERCOjpyYXcoJ2xlbmd0aChwYXNzKScpLCAnPCcsIDUwKS0+bGltaXQoMTAwKS0+Z2V0KCk7aWYgKCRkYXRhcy0+aXNFbXB0eSgpKSB7YnJlYWs7fQ==Zm9yZWFjaCgkZGF0YXMgYXMgJGRhdGEpew==JGRhdGFzID0gXERCOjp0YWJsZSgndXNlcnMnKS0+d2hlcmUoJ2lkJywgJGRhdGEtPmlkKS0+dXBkYXRlKFsncGFzcycgPT4gXEhhc2g6Om1ha2UoJGRhdGEtPnBhc3MpXSk7fQ==fQ==XExvZzo6aW5mbygncGFzc0hhc2gnLidfZW5kJyk7fQ==44GT44KT44Gq5Yem55CG
13.このソースはフィクションですこの処理のコマンドは件数によってかなり時間がかかりそうなので、これがどれだけ早くなるか試してみました。
14.前哨戦
15.データ準備（通常）private function insertUser($count){Log::info('insertUser'.'_start');for($i = 0;$i < $count;$i++){for($j = 0;$j < $count;$j++){$sql = "insert into users (mail, pass) VALUES ('$i mail@mail.com', '$i abcdefghij');";DB::statement($sql);}}Log::info('insertUser'.'_end');}• $count=100で1万件作成。• ローカル環境で実施（mac + docker）。
16.データ準備（通常）• 大体15秒程度。mysql> SELECT * FROM `users` ORDER BY `update_time` DESC LIMIT 1;+-------+------------------+---------------+---------------------+| id | mail | pass | update_time |+-------+------------------+---------------+---------------------+| 10000 | 99 mail@mail.com | 99 abcdefghij | 2019-05-19 11:17:57 |+-------+------------------+---------------+---------------------+1 row in set (0.01 sec)mysql> SELECT * FROM `users` ORDER BY `update_time` ASC LIMIT 1;+----+-----------------+--------------+---------------------+| id | mail | pass | update_time |+----+-----------------+--------------+---------------------+| 1 | 0 mail@mail.com | 0 abcdefghij | 2019-05-19 11:17:42 |+----+-----------------+--------------+---------------------+1 row in set (0.00 sec)
17.データ準備（swoole）private function insertUserSwoole($count){Log::info('insertUser'.'_start');for($i = 0;$i < $count;$i++){go(function () use ($i, $count) {$mysql = $this->getMysql();for($j = 0;$j < $count;$j++){$statement = $mysql->prepare("insert into users (mail, pass) VALUES (‘$i $j mail@mail.com', ‘$i $jabcdefghij');");$result = $statement->execute();}});}Log::info('insertUser'.'_end');}• さっきのと同じ条件で実行。• go()で囲まれているとこが非同期に実行される（んだと思う）。private function getMysql(){$mysql = new SwooleCoroutineMySQL();$mysql->connect(['host' => env('DB_HOST'),'port' => 3306,'user' => env('DB_USERNAME'),'password' => env('DB_PASSWORD'),'database' => env('DB_DATABASE'),'timeout' => 4, // 設定しといた方がよい]);return $mysql;}
18.• 大体4秒程度。mysql> SELECT * FROM `users` ORDER BY `update_time` DESC LIMIT 1;+------+---------------------+------------------+---------------------+| id | mail | pass | update_time |+------+---------------------+------------------+---------------------+| 9329 | 44 94 mail@mail.com | 44 94 abcdefghij | 2019-05-19 11:29:37 |+------+---------------------+------------------+---------------------+1 row in set (0.01 sec)mysql> SELECT * FROM `users` ORDER BY `update_time` ASC LIMIT 1;+----+-------------------+----------------+---------------------+| id | mail | pass | update_time |+----+-------------------+----------------+---------------------+| 1 | 3 0 mail@mail.com | 3 0 abcdefghij | 2019-05-19 11:29:33 |+----+-------------------+----------------+---------------------+1 row in set (0.01 sec)データ準備（swoole）
19.15秒→4秒
20.おおすごい
21.では本番
22.ハッシュ（通常）private function passHash(){Log::info('passHash'.'_start');while(true){$datas = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->limit(100)->get();if ($datas->isEmpty()) {break;}foreach($datas as $data){$datas = DB::table('users')->where('id', $data->id)->update(['pass' => Hash::make($data->pass)]);}}Log::info('passHash'.'_end');}• さっき作成した1万件で実行。
23.• 大体9分17秒程度。mysql> SELECT * FROM `users` ORDER BY `update_time` DESC LIMIT 1;+-------+--------------------+--------------------------------------------------------------+---------------------+| id | mail | pass | update_time |+-------+--------------------+--------------------------------------------------------------+---------------------+| 10000 | 0 99 mail@mail.com | $2y$10$oFmEp9d6qtK5WrvUtQ4PRevs8BtFK5CcLEF04qD4Vl0TdQaAY2yMa | 2019-05-19 12:14:41 |+-------+--------------------+--------------------------------------------------------------+---------------------+1 row in set (0.01 sec)mysql> SELECT * FROM `users` ORDER BY `update_time` ASC LIMIT 1;+----+--------------------+--------------------------------------------------------------+---------------------+| id | mail | pass | update_time |+----+--------------------+--------------------------------------------------------------+---------------------+| 1 | 68 0 mail@mail.com | $2y$10$JdBPfgEMn2kAJm2sotn9NObS7hHzuoZwfamx0VrrtQHuhBlxMu5rm | 2019-05-19 12:05:24 |+----+--------------------+--------------------------------------------------------------+---------------------+1 row in set (0.01 sec)ハッシュ（通常）
24.ハッシュ（swoole）private function passHashSwoole($count){Log::info('passHashSwoole'.'_start');$coefficient = 2;$parent = $count / $coefficient;$child = $count * $coefficient;for($i = 0;$i < $parent;$i++){$sql = DB::table('users')->limit($child);if ($i > 0) {$sql->offset($i * $child);}$datas = $sql->get();go(function () use ($datas) {$mysql = $this->getMysql();foreach($datas as $data){$pass = Hash::make($data->pass);$id = $data->id;$data = $mysql->query("update users set pass = '$pass' where id = $id;");}});Log::info('passHashSwoole_child'.'_end');}Log::info('passHashSwoole'.'_end');}• 色々あって何とか辿りついたソース。
25.ハッシュ（swoole）mysql> SELECT * FROM `users` ORDER BY `update_time` DESC LIMIT 1;+------+---------------------+--------------------------------------------------------------+---------------------+| id | mail | pass | update_time |+------+---------------------+--------------------------------------------------------------+---------------------+| 8794 | 64 97 mail@mail.com | $2y$10$dJEFSl9CikhSv.E4ICumMO3uOCqbVvK2E0uTzZWON8tPAejrIm5wq | 2019-05-19 14:54:31 |+------+---------------------+--------------------------------------------------------------+---------------------+1 row in set (0.01 sec)mysql> SELECT * FROM `users` ORDER BY `update_time` ASC LIMIT 1;+----+-------------------+--------------------------------------------------------------+---------------------+| id | mail | pass | update_time |+----+-------------------+--------------------------------------------------------------+---------------------+| 1 | 5 0 mail@mail.com | $2y$10$PFUJN/mKBE3uVNtX/GpE6Oaz5I.VjEFkRxWvh2wiFStQySkdSQJcy | 2019-05-19 14:45:58 |+----+-------------------+--------------------------------------------------------------+---------------------+1 row in set (0.01 sec)
26.(　ﾟдﾟ) ・・・(つд⊂)ｺﾞｼｺﾞｼ
27.• 大体8分33秒程度。mysql> SELECT * FROM `users` ORDER BY `update_time` DESC LIMIT 1;+------+---------------------+--------------------------------------------------------------+---------------------+| id | mail | pass | update_time |+------+---------------------+--------------------------------------------------------------+---------------------+| 8794 | 64 97 mail@mail.com | $2y$10$dJEFSl9CikhSv.E4ICumMO3uOCqbVvK2E0uTzZWON8tPAejrIm5wq | 2019-05-19 14:54:31 |+------+---------------------+--------------------------------------------------------------+---------------------+1 row in set (0.01 sec)mysql> SELECT * FROM `users` ORDER BY `update_time` ASC LIMIT 1;+----+-------------------+--------------------------------------------------------------+---------------------+| id | mail | pass | update_time |+----+-------------------+--------------------------------------------------------------+---------------------+| 1 | 5 0 mail@mail.com | $2y$10$PFUJN/mKBE3uVNtX/GpE6Oaz5I.VjEFkRxWvh2wiFStQySkdSQJcy | 2019-05-19 14:45:58 |+----+-------------------+--------------------------------------------------------------+---------------------+1 row in set (0.01 sec)ハッシュ（swoole）
28.(；ﾟдﾟ) ・・・(つд⊂)ｺﾞｼｺﾞｼｺﾞｼ
29.9分17秒→8分33秒
30.(ﾟ´Д｀ﾟ)ﾟ・:*:・゜’★
31.全然早くない
32.この動く処理にたどり着くまでに既に結構大変だったのに。。。
33.GCPの良いインスタンスで試したりとかもほぼ無駄だった。
34.いろいろ調べた結果
35.• Cpuを使い切れてない様子。• ローカルでも2つCpuが使えれば、せめて倍くらいの速度は出せるはず。top - 21:41:33 up 9:04, 0 users, load average: 0.79, 0.40, 0.22Tasks: 15 total, 2 running, 13 sleeping, 0 stopped, 0 zombieCpu0 : 1.0%us, 2.3%sy, 0.0%ni, 96.3%id, 0.3%wa, 0.0%hi, 0.0%si, 0.0%stCpu1 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st実行サーバ
36.Swoole ProcessManager
37.プロセスを自分で管理できるやつです。詳しく知らないので説明はできません。これを試してみました。
38.ハッシュ（複数プロセス）private function passHashSwooleProcess(){Log::info('passHashSwooleProcess'.'_start');$workers = [];$worker_num = 2;$data_count = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->count();for($i = 0; $i < $worker_num; $i++){$sql = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->limit($data_count / 2);if ($i > 0) {$sql->offset($i * $data_count / 2);}$userss = $sql->get();$process = new SwooleProcess(function($process) use ($userss){go(function () use ($process, $userss) {$mysql = $this->getMysql();foreach($userss as $users){$pass = Hash::make($users->pass);$id = $users->id;$data = $mysql->query("update users set pass = '$pass' where id = $id;");}Log::info('passHashSwooleProcess_child'.'_end');$process->exit(0);});});$pid = $process->start();$workers[$pid] = $process;}Log::info('passHashSwooleProcess'.'_end');}
39.private function passHashSwooleProcess(){Log::info('passHashSwooleProcess'.'_start');$workers = [];$worker_num = 2;$data_count = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->count();for($i = 0; $i < $worker_num; $i++){$sql = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->limit($data_count / 2);if ($i > 0) {$sql->offset($i * $data_count / 2);}$userss = $sql->get();$process = new SwooleProcess(function($process) use ($userss){go(function () use ($process, $userss) {$mysql = $this->getMysql();foreach($userss as $users){$pass = Hash::make($users->pass);$id = $users->id;$data = $mysql->query("update users set pass = '$pass' where id = $id;");}Log::info('passHashSwooleProcess_child'.'_end');$process->exit(0);});});$pid = $process->start();$workers[$pid] = $process;}Log::info('passHashSwooleProcess'.'_end');}ハッシュ（複数プロセス）プロセスを作って
40.ハッシュ（複数プロセス）private function passHashSwooleProcess(){Log::info('passHashSwooleProcess'.'_start');$workers = [];$worker_num = 2;$data_count = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->count();for($i = 0; $i < $worker_num; $i++){$sql = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->limit($data_count / 2);if ($i > 0) {$sql->offset($i * $data_count / 2);}$userss = $sql->get();$process = new SwooleProcess(function($process) use ($userss){go(function () use ($process, $userss) {$mysql = $this->getMysql();foreach($userss as $users){$pass = Hash::make($users->pass);$id = $users->id;$data = $mysql->query("update users set pass = '$pass' where id = $id;");}Log::info('passHashSwooleProcess_child'.'_end');$process->exit(0);});});$pid = $process->start();$workers[$pid] = $process;}Log::info('passHashSwooleProcess'.'_end');}プロセス動かす。
41.private function passHashSwooleProcess(){Log::info('passHashSwooleProcess'.'_start');$workers = [];$worker_num = 2;$data_count = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->count();for($i = 0; $i < $worker_num; $i++){$sql = DB::table('users')->where(DB::raw('length(pass)'), '<', 50)->limit($data_count / 2);if ($i > 0) {$sql->offset($i * $data_count / 2);}$userss = $sql->get();$process = new SwooleProcess(function($process) use ($userss){go(function () use ($process, $userss) {$mysql = $this->getMysql();foreach($userss as $users){$pass = Hash::make($users->pass);$id = $users->id;$data = $mysql->query("update users set pass = '$pass' where id = $id;");}Log::info('passHashSwooleProcess_child'.'_end');$process->exit(0);});});$pid = $process->start();$workers[$pid] = $process;}Log::info('passHashSwooleProcess'.'_end');}ハッシュ（複数プロセス）終わったら殺す。
42.実行サーバでTOP
43.いけてそう！！！！top - 21:34:34 up 1 day, 14:42, 0 users, load average: 0.54, 0.51, 0.58Tasks: 5 total, 3 running, 2 sleeping, 0 stopped, 0 zombieCpu0 : 94.0%us, 3.0%sy, 0.0%ni, 2.3%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%stCpu1 : 93.0%us, 4.3%sy, 0.0%ni, 0.0%id, 2.0%wa, 0.0%hi, 0.7%si, 0.0%st実行サーバでTOP
44.計測
45.• 大体4分39秒程度。mysql> SELECT * FROM `users` ORDER BY `update_time` DESC LIMIT 1;+-------+---------------------+--------------------------------------------------------------+---------------------+| id | mail | pass | update_time |+-------+---------------------+--------------------------------------------------------------+---------------------+| 10000 | 91 99 mail@mail.com | $2y$10$60bx7lqjuNUys/tHUC3pOOkooM6AyN2/KabeP3flsmGWXfAn7HoRG | 2019-05-19 15:23:22 |+-------+---------------------+--------------------------------------------------------------+---------------------+1 row in set (0.01 sec)mysql> SELECT * FROM `users` ORDER BY `update_time` ASC LIMIT 1;+----+-------------------+--------------------------------------------------------------+---------------------+| id | mail | pass | update_time |+----+-------------------+--------------------------------------------------------------+---------------------+| 1 | 0 0 mail@mail.com | $2y$10$h6ziNoQHFSTsMbe8qOKJ8O/FDfM80vmy21/9nt7Q6H76aV9lcC1VW | 2019-05-19 15:18:43 |+----+-------------------+--------------------------------------------------------------+---------------------+1 row in set (0.01 sec)ハッシュ（swoole）
46.• 非同期とかプロセスとか結構大変だった。
- 気軽には勧められないかも。• フレームワークを使えばhttpでも比較的簡単に使える。
- らしい。試してない。来世で試します。• 今回試した奴は、コマンド2回叩けば同じことできるんじゃないかと思う。処理工夫してサーバ複数でやる方が、非同期とかプロセスとかより簡単。swooleの感想