- Notifications
You must be signed in to change notification settings - Fork18.4k
Closed
Labels
Milestone
Description
My longevity test crashes after handling a few thousand requests. By turning offPreciseGC, the crashes stopped.I have run my application under different linux kernels- 3.5.0-26-generic- 3.5.0-28-genericI have used different versions of go1.1:- 1.1rc1- some older randomly picked version- tipI didn't notice this issue before because I ran my app with ab -n 2000 -c 16http://localhost:8001/...However, once I increased it to let it run for hours (e.g. set n to 2000000), it startscrashing after handling a few thousand requests e.g. after 4957, 16688, 1721, 7820,8406, 13981 requests. I've looked through my code over the last 2 days to see if there is something I'm doing,and have seen nothing to suggest my code may be the culprit. As expected, I makejudicious use of sync.Mutex, sync.Cond and channels in my codebase, and I don't copysync.Mutex or sync.Cond anywhere.However, when I switched turned off PreciseGC (by setting IgnorePreciseGC=1 in mgc0.cand rebuilding), the crashes stopped, and my load test continued forever (till my HDD /partition filled up - I just posted a message on the golang-devs about that). I use ulimit to prevent an issue in my code from locking up my whole machine (ithappened many times during development). So I run my app using sh run.sh, where run.shcontains: ulimit -v 2000000 && ulimit -c unlimited && GOTRACEBACK=crash GOMAXPROCS=8 ./myappWhen I use gdb to look at the crash, I get that the crash typically happens around thesame point, but that point typically changes each time I reboot my machine. The points Ihave found are:---------------------------------------(gdb) bt#0 runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1 0x000000000041aa1f in runtime.crash ()#2 0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3 0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4 0x0000000000412cc7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5 0x000000000041b2eb in runtime.copy (to=void, fm=void, width=void, ret=void) at /opt/go-tip/src/pkg/runtime/slice.c:257#6 0x000000000059cdcc in bufio.(*Reader).Read (b=0xc20021f720, p=..., n=1, err=...) at /opt/go-tip/src/pkg/bufio/bufio.go:156#7 0x000000000043f327 in io.ReadAtLeast (r=..., buf=..., min=1, n=0, err=...) at/opt/go-tip/src/pkg/io/io.go:284#8 0x000000000043f47f in io.ReadFull (r=..., buf=..., n=4305628, err=...) at/opt/go-tip/src/pkg/io/io.go:302#9 0x000000000050fbcd in ugorji.net/ndb.(*client).readD (c=0xc200219870, numToRead=1,expecting=0 '\000', bs=..., cat=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:141#10 0x000000000050ff6d in ugorji.net/ndb.(*client).readTag (c=0xc200219870, v=0 '\000') at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:156#11 0x0000000000511b52 in ugorji.net/ndb.(*Db).SvrQuery (l=0xc2001b0240, ctxId=...,pkey=0xc2005e6a10, kind=..., opts=0x0, filters=..., qs=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:421#12 0x00000000005222cf in ugorji.net/ndb.func·001 (k3=..., cursor3=..., err3=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/low_level_driver.go:195#13 0x00000000004232e0 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#14 0x0000000000000000 in ?? ()(gdb) q------------------------------------(gdb) bt#0 runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1 0x000000000041aa1f in runtime.crash ()#2 0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3 0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4 0x0000000000412cc7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5 0x0000000000423c75 in runtime.memmove (to=void, fr=void, n=void) at /opt/go-tip/src/pkg/runtime/memmove_amd64.s:48#6 0x000000000041b2dc in runtime.copy (to=void, fm=void, width=void, ret=void) at /opt/go-tip/src/pkg/runtime/slice.c:259#7 0x000000000059ce3c in bufio.(*Reader).Read (b=0xc2003044e0, p=..., n=2, err=...) at /opt/go-tip/src/pkg/bufio/bufio.go:156#8 0x000000000043f357 in io.ReadAtLeast (r=..., buf=..., min=2, n=0, err=...) at/opt/go-tip/src/pkg/io/io.go:284#9 0x000000000043f4af in io.ReadFull (r=..., buf=..., n=833226801153, err=...) at/opt/go-tip/src/pkg/io/io.go:302#10 0x000000000050fbfd in ugorji.net/ndb.(*client).readD (c=0xc20035bd20, numToRead=2,expecting=0 '\000', bs=..., cat=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:141#11 0x000000000051009f in ugorji.net/ndb.(*client).readNumNoTag (c=0xc20035bd20, v=0) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:184#12 0x00000000005106d0 in ugorji.net/ndb.(*client).readSliceNoTag (c=0xc20035bd20, v=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:226#13 0x000000000051216b in ugorji.net/ndb.(*Db).SvrQuery (l=0xc2001ae240, ctxId=...,pkey=0xc2006e2c10, kind=..., opts=0x0, filters=..., qs=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:439#14 0x00000000005222ff in ugorji.net/ndb.func·001 (k3=..., cursor3=..., err3=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/low_level_driver.go:195#15 0x0000000000423310 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#16 0x0000000000000000 in ?? ()------------------------------------(gdb) bt#0 runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1 0x000000000041aa1f in runtime.crash ()#2 0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3 0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4 0x0000000000412cc7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5 0x00000000007e2881 in string.* ()#6 0x000000000059c8cc in bufio.(*Reader).fill (b=0xc2002d7e40) at/opt/go-tip/src/pkg/bufio/bufio.go:79#7 0x000000000059ccd0 in bufio.(*Reader).Read (b=0xc2002d7e40, p=..., n=1, err=...) at /opt/go-tip/src/pkg/bufio/bufio.go:147#8 0x000000000043f327 in io.ReadAtLeast (r=..., buf=..., min=1, n=0, err=...) at/opt/go-tip/src/pkg/io/io.go:284#9 0x000000000043f47f in io.ReadFull (r=..., buf=..., n=4305628, err=...) at/opt/go-tip/src/pkg/io/io.go:302#10 0x000000000050fbcd in ugorji.net/ndb.(*client).readD (c=0xc200254640, numToRead=1,expecting=0 '\000', bs=..., cat=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:141#11 0x000000000050ff6d in ugorji.net/ndb.(*client).readTag (c=0xc200254640, v=0 '\000') at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:156#12 0x0000000000511b52 in ugorji.net/ndb.(*Db).SvrQuery (l=0xc2001af240, ctxId=...,pkey=0xc20055b830, kind=..., opts=0x0, filters=..., qs=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:421#13 0x00000000005222cf in ugorji.net/ndb.func·001 (k3=..., cursor3=..., err3=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/low_level_driver.go:195#14 0x00000000004232e0 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#15 0x0000000000000000 in ?? ()(gdb) -------------------------------------(gdb) bt#0 runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1 0x000000000041aa1f in runtime.crash ()#2 0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3 0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4 0x0000000000412cc7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5 0x00000000004232e0 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#6 0x00007f704c166100 in ?? ()#7 0x00007f704c166fb8 in ?? ()#8 0x00007f704c1660f8 in ?? ()#9 0x000000000040b4c7 in runtime.MCentral_AllocList (c=void, n=void, pfirst=void) at /opt/go-tip/src/pkg/runtime/mcentral.c:62#10 0x00007f705e3a7000 in ?? ()#11 0x00007f705e3f7000 in ?? ()#12 0x00007f704c1db000 in ?? ()#13 0x00007f704c1e3000 in ?? ()#14 0x00007f704c192000 in ?? ()#15 0x00007f704c173000 in ?? ()#16 0x00007f704c1c9000 in ?? ()#17 0x00007f704c0b6000 in ?? ()#18 0x00007f704c0d2000 in ?? ()#19 0x0000000000000006 in ?? ()#20 0x0000000000000280 in ?? ()#21 0x000000000041ae99 in runtime.appendstr (t=void, x=void, y=void, ret=void) at /opt/go-tip/src/pkg/runtime/slice.c:147#22 0x2010000000000076 in ?? ()#23 0x0000000000000001 in ?? ()#24 0x0000000000000003 in ?? ()#25 0x0000000000410503 in runtime.markallocated (v=void, n=void, noptr=void) at /opt/go-tip/src/pkg/runtime/mgc0.c:2213#26 0x000000000041e103 in runtime.mallocgc (size=void, flag=void, dogc=void, zeroed=void) at /opt/go-tip/src/pkg/runtime/malloc.goc:88#27 0x000000000041e103 in runtime.mallocgc (size=void, flag=void, dogc=void, zeroed=void) at /opt/go-tip/src/pkg/runtime/malloc.goc:88#28 0x0000000000421d5b in gostringsize (ret=void, l=void) at/opt/go-tip/src/pkg/runtime/string.goc:48#29 0x0000000000422411 in runtime.slicebytetostring (b=void, s=void) at/opt/go-tip/src/pkg/runtime/string.goc:274#30 0x000000000046533e in strconv.quoteWith (s=640, quote=43 '+', ASCIIonly=false,~anon3=...) at /opt/go-tip/src/pkg/strconv/quote.go:82#31 0x00000000004660c9 in strconv.Quote (s=..., ~anon1=...) at/opt/go-tip/src/pkg/strconv/quote.go:91#32 0x000000000042a6ca in fmt.(*fmt).fmt_q (f=0x2010000000017902, s=...) at/opt/go-tip/src/pkg/fmt/format.go:325---Type <return> to continue, or q <return> to quit---#33 0x000000c2004de468 in ?? ()#34 0x2010000000017902 in ?? ()#35 0x000000000000000a in ?? ()#36 0x0000000000000000 in ?? ()(gdb) -----------------------------(gdb) bt#0 runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1 0x000000000041aa1f in runtime.crash ()#2 0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3 0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4 0x0000000000412ca7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5 0x000000000040a8e7 in runtime.efacethash (e1=void, ret=void) at/opt/go-tip/src/pkg/runtime/iface.c:670#6 0x000000000042f042 in fmt.(*pp).printField (p=0xc200130d00, field=..., verb=118,plus=false, goSyntax=false, depth=0, wasString=false) at /opt/go-tip/src/pkg/fmt/print.go:765#7 0x00000000004348c6 in fmt.(*pp).doPrintf (p=..., format=..., a=...) at/opt/go-tip/src/pkg/fmt/print.go:1111#8 0x000000000042bfa5 in fmt.Sprintf (format=..., a=..., ~anon2=...) at/opt/go-tip/src/pkg/fmt/print.go:229#9 0x0000000000534d4a in ugorji.net/logging.logR (calldepth=2 '\002', ctx=...,level=1000, message=..., params=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/logging/logging.go:264#10 0x0000000000535053 in ugorji.net/logging.Trace (ctx=..., message=..., params=...,~anon3=...) at /home/ugorji/depot/repo/src/ugorji.net/logging/logging.go:284#11 0x0000000000511b9a in ugorji.net/ndb.(*Db).SvrQuery (l=0xc2001b0240, ctxId=...,pkey=0xc200968630, kind=..., opts=0x0, filters=..., qs=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:422#12 0x000000000052235f in ugorji.net/ndb.func·001 (k3=..., cursor3=..., err3=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/low_level_driver.go:195#13 0x00000000004232e0 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#14 0x0000000000000000 in ?? ()------------------------------------Which compiler are you using (5g, 6g, 8g, gccgo)?6gWhich operating system are you using?Linux 3.5.0-28-generic #42-Ubuntu SMP Fri Mar 8 23:18:20 UTC 2013 x86_64 x86_64 x86_64GNU/LinuxWhich version are you using? (run 'go version')go version devel +30c566874b83 Wed May 08 16:06:25 2013 -0700 linux/amd64Please provide any additional information below.