- Notifications
You must be signed in to change notification settings - Fork54
Description
I've been trying to debug a memory issue and I think I've been able to pin it down to that it has something to do withecto_sqlite3 but not withexqlite.
I've posted my journey here:https://elixirforum.com/t/measuring-memory-consumption-how-to-figure-out-what-triggers-jumps-in-memory/73139/9?u=tcoopman
Let me reproduce some context here. This is the script that I use:
Mix.install([{:ecto_sql,"~> 3.13.2"},{:ecto_sqlite3,"~> 0.22"},{:exqlite,"~> 0.33.1"}])Application.put_env(:myapp,Repo,database:"./dev.db")defmoduleRepodouseEcto.Repo,otp_app::myapp,adapter:Ecto.Adapters.SQLite3enddefmoduleMaindo@sql~s""" SELECT s0."id", e1."id", e1."type", e1."data", e1."inserted_at", s0."stream_id", s0."stream_version" FROM "stream_events" AS s0 INNER JOIN "events" AS e1 ON s0."event_id" = e1."id" WHERE s0."stream_id" = '$all' AND s0."stream_version" >= 0 ORDER BY s0."id" DESC LIMIT 1"""defsqlitedoIO.inspect"before"print_memory()save_allocations("./before_sqlite"){:ok,_}=Repo.start_link([])_s1=Ecto.Adapters.SQL.query!(Repo,@sql,nil)IO.inspect"after"print_memory()save_allocations("./after_sqlite")enddefexqlitedoIO.inspect"before"print_memory()save_allocations("./before_exqlite"){:ok,conn}=Exqlite.Sqlite3.open("./dev.db"){:ok,statement}=Exqlite.Sqlite3.prepare(conn,@sql){:row,_results}=Exqlite.Sqlite3.step(conn,statement)IO.inspect"after"print_memory()save_allocations("./after_exqlite")enddefpprint_memory()doIO.inspect"memory:#{:erlang.memory(:total)/1_000_000}"enddefpsave_allocations(name)do{:ok,x}=:instrument.allocationsFile.write!(name,inspect(x,limit::infinity,pretty:true))endend# Main.sqlite()Main.exqlite()
When runningMain.sqlite the before and after memory jumps hugely (more than 50MB in this script, but when running in my phoenix app it's more like 300MB):
The difference I see in allocations is this:
What's weird is that the exact query matters.
- Changing
DESCtoASCand the issue is reduced a lot: the memory still jumps but a lot less: from 59 to 95 instead from 59 to 130. - Changing
$allto any otherstream_idand the issue is completely gone.
But that doesn't clarify why this issue does not represent itself when running directly withexqlite. If I useexqlite directly I don't see any of these issues?
The database is about 280MB so not small, but not huge. I could probably share it privately if that helps you.