Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings
/goPublic

runtime: PreciseGC causing app to crash during longevity tests #5443

Closed
Milestone
@ugorji

Description

@ugorji
My longevity test crashes after handling a few thousand requests. By turning offPreciseGC, the crashes stopped.I have run my application under different linux kernels- 3.5.0-26-generic- 3.5.0-28-genericI have used different versions of go1.1:- 1.1rc1- some older randomly picked version- tipI didn't notice this issue before because I ran my app with ab -n 2000 -c 16http://localhost:8001/...However, once I increased it to let it run for hours (e.g. set n to 2000000), it startscrashing after handling a few thousand requests e.g. after 4957, 16688, 1721, 7820,8406, 13981 requests. I've looked through my code over the last 2 days to see if there is something I'm doing,and have seen nothing to suggest my code may be the culprit. As expected, I makejudicious use of sync.Mutex, sync.Cond and channels in my codebase, and I don't copysync.Mutex or sync.Cond anywhere.However, when I switched turned off PreciseGC (by setting IgnorePreciseGC=1 in mgc0.cand rebuilding), the crashes stopped, and my load test continued forever (till my HDD /partition filled up - I just posted a message on the golang-devs about that). I use ulimit to prevent an issue in my code from locking up my whole machine (ithappened many times during development). So I run my app using sh run.sh, where run.shcontains:  ulimit -v 2000000 && ulimit -c unlimited && GOTRACEBACK=crash GOMAXPROCS=8 ./myappWhen I use gdb to look at the crash, I get that the crash typically happens around thesame point, but that point typically changes each time I reboot my machine. The points Ihave found are:---------------------------------------(gdb) bt#0  runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1  0x000000000041aa1f in runtime.crash ()#2  0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3  0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4  0x0000000000412cc7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5  0x000000000041b2eb in runtime.copy (to=void, fm=void, width=void, ret=void)    at /opt/go-tip/src/pkg/runtime/slice.c:257#6  0x000000000059cdcc in bufio.(*Reader).Read (b=0xc20021f720, p=..., n=1, err=...)    at /opt/go-tip/src/pkg/bufio/bufio.go:156#7  0x000000000043f327 in io.ReadAtLeast (r=..., buf=..., min=1, n=0, err=...) at/opt/go-tip/src/pkg/io/io.go:284#8  0x000000000043f47f in io.ReadFull (r=..., buf=..., n=4305628, err=...) at/opt/go-tip/src/pkg/io/io.go:302#9  0x000000000050fbcd in ugorji.net/ndb.(*client).readD (c=0xc200219870, numToRead=1,expecting=0 '\000', bs=...,     cat=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:141#10 0x000000000050ff6d in ugorji.net/ndb.(*client).readTag (c=0xc200219870, v=0 '\000')    at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:156#11 0x0000000000511b52 in ugorji.net/ndb.(*Db).SvrQuery (l=0xc2001b0240, ctxId=...,pkey=0xc2005e6a10, kind=...,     opts=0x0, filters=..., qs=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:421#12 0x00000000005222cf in ugorji.net/ndb.func·001 (k3=..., cursor3=..., err3=...)    at /home/ugorji/depot/repo/src/ugorji.net/ndb/low_level_driver.go:195#13 0x00000000004232e0 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#14 0x0000000000000000 in ?? ()(gdb) q------------------------------------(gdb) bt#0  runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1  0x000000000041aa1f in runtime.crash ()#2  0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3  0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4  0x0000000000412cc7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5  0x0000000000423c75 in runtime.memmove (to=void, fr=void, n=void)    at /opt/go-tip/src/pkg/runtime/memmove_amd64.s:48#6  0x000000000041b2dc in runtime.copy (to=void, fm=void, width=void, ret=void)    at /opt/go-tip/src/pkg/runtime/slice.c:259#7  0x000000000059ce3c in bufio.(*Reader).Read (b=0xc2003044e0, p=..., n=2, err=...)    at /opt/go-tip/src/pkg/bufio/bufio.go:156#8  0x000000000043f357 in io.ReadAtLeast (r=..., buf=..., min=2, n=0, err=...) at/opt/go-tip/src/pkg/io/io.go:284#9  0x000000000043f4af in io.ReadFull (r=..., buf=..., n=833226801153, err=...) at/opt/go-tip/src/pkg/io/io.go:302#10 0x000000000050fbfd in ugorji.net/ndb.(*client).readD (c=0xc20035bd20, numToRead=2,expecting=0 '\000', bs=...,     cat=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:141#11 0x000000000051009f in ugorji.net/ndb.(*client).readNumNoTag (c=0xc20035bd20, v=0)    at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:184#12 0x00000000005106d0 in ugorji.net/ndb.(*client).readSliceNoTag (c=0xc20035bd20, v=...)    at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:226#13 0x000000000051216b in ugorji.net/ndb.(*Db).SvrQuery (l=0xc2001ae240, ctxId=...,pkey=0xc2006e2c10, kind=...,     opts=0x0, filters=..., qs=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:439#14 0x00000000005222ff in ugorji.net/ndb.func·001 (k3=..., cursor3=..., err3=...)    at /home/ugorji/depot/repo/src/ugorji.net/ndb/low_level_driver.go:195#15 0x0000000000423310 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#16 0x0000000000000000 in ?? ()------------------------------------(gdb) bt#0  runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1  0x000000000041aa1f in runtime.crash ()#2  0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3  0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4  0x0000000000412cc7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5  0x00000000007e2881 in string.* ()#6  0x000000000059c8cc in bufio.(*Reader).fill (b=0xc2002d7e40) at/opt/go-tip/src/pkg/bufio/bufio.go:79#7  0x000000000059ccd0 in bufio.(*Reader).Read (b=0xc2002d7e40, p=..., n=1, err=...)    at /opt/go-tip/src/pkg/bufio/bufio.go:147#8  0x000000000043f327 in io.ReadAtLeast (r=..., buf=..., min=1, n=0, err=...) at/opt/go-tip/src/pkg/io/io.go:284#9  0x000000000043f47f in io.ReadFull (r=..., buf=..., n=4305628, err=...) at/opt/go-tip/src/pkg/io/io.go:302#10 0x000000000050fbcd in ugorji.net/ndb.(*client).readD (c=0xc200254640, numToRead=1,expecting=0 '\000', bs=...,     cat=...) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:141#11 0x000000000050ff6d in ugorji.net/ndb.(*client).readTag (c=0xc200254640, v=0 '\000')    at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:156#12 0x0000000000511b52 in ugorji.net/ndb.(*Db).SvrQuery (l=0xc2001af240, ctxId=...,pkey=0xc20055b830, kind=...,     opts=0x0, filters=..., qs=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:421#13 0x00000000005222cf in ugorji.net/ndb.func·001 (k3=..., cursor3=..., err3=...)    at /home/ugorji/depot/repo/src/ugorji.net/ndb/low_level_driver.go:195#14 0x00000000004232e0 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#15 0x0000000000000000 in ?? ()(gdb) -------------------------------------(gdb) bt#0  runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1  0x000000000041aa1f in runtime.crash ()#2  0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3  0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4  0x0000000000412cc7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5  0x00000000004232e0 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#6  0x00007f704c166100 in ?? ()#7  0x00007f704c166fb8 in ?? ()#8  0x00007f704c1660f8 in ?? ()#9  0x000000000040b4c7 in runtime.MCentral_AllocList (c=void, n=void, pfirst=void)    at /opt/go-tip/src/pkg/runtime/mcentral.c:62#10 0x00007f705e3a7000 in ?? ()#11 0x00007f705e3f7000 in ?? ()#12 0x00007f704c1db000 in ?? ()#13 0x00007f704c1e3000 in ?? ()#14 0x00007f704c192000 in ?? ()#15 0x00007f704c173000 in ?? ()#16 0x00007f704c1c9000 in ?? ()#17 0x00007f704c0b6000 in ?? ()#18 0x00007f704c0d2000 in ?? ()#19 0x0000000000000006 in ?? ()#20 0x0000000000000280 in ?? ()#21 0x000000000041ae99 in runtime.appendstr (t=void, x=void, y=void, ret=void)    at /opt/go-tip/src/pkg/runtime/slice.c:147#22 0x2010000000000076 in ?? ()#23 0x0000000000000001 in ?? ()#24 0x0000000000000003 in ?? ()#25 0x0000000000410503 in runtime.markallocated (v=void, n=void, noptr=void)    at /opt/go-tip/src/pkg/runtime/mgc0.c:2213#26 0x000000000041e103 in runtime.mallocgc (size=void, flag=void, dogc=void, zeroed=void)    at /opt/go-tip/src/pkg/runtime/malloc.goc:88#27 0x000000000041e103 in runtime.mallocgc (size=void, flag=void, dogc=void, zeroed=void)    at /opt/go-tip/src/pkg/runtime/malloc.goc:88#28 0x0000000000421d5b in gostringsize (ret=void, l=void) at/opt/go-tip/src/pkg/runtime/string.goc:48#29 0x0000000000422411 in runtime.slicebytetostring (b=void, s=void) at/opt/go-tip/src/pkg/runtime/string.goc:274#30 0x000000000046533e in strconv.quoteWith (s=640, quote=43 '+', ASCIIonly=false,~anon3=...)    at /opt/go-tip/src/pkg/strconv/quote.go:82#31 0x00000000004660c9 in strconv.Quote (s=..., ~anon1=...) at/opt/go-tip/src/pkg/strconv/quote.go:91#32 0x000000000042a6ca in fmt.(*fmt).fmt_q (f=0x2010000000017902, s=...) at/opt/go-tip/src/pkg/fmt/format.go:325---Type <return> to continue, or q <return> to quit---#33 0x000000c2004de468 in ?? ()#34 0x2010000000017902 in ?? ()#35 0x000000000000000a in ?? ()#36 0x0000000000000000 in ?? ()(gdb) -----------------------------(gdb) bt#0  runtime.raise (sig=void) at /opt/go-tip/src/pkg/runtime/sys_linux_amd64.s:85#1  0x000000000041aa1f in runtime.crash ()#2  0x0000000000413bd7 in runtime.dopanic (unused=void) at/opt/go-tip/src/pkg/runtime/panic.c:434#3  0x0000000000413d17 in runtime.throw (s=void) at/opt/go-tip/src/pkg/runtime/panic.c:472#4  0x0000000000412ca7 in runtime.sigpanic () at/opt/go-tip/src/pkg/runtime/os_linux.c:239#5  0x000000000040a8e7 in runtime.efacethash (e1=void, ret=void) at/opt/go-tip/src/pkg/runtime/iface.c:670#6  0x000000000042f042 in fmt.(*pp).printField (p=0xc200130d00, field=..., verb=118,plus=false, goSyntax=false,     depth=0, wasString=false) at /opt/go-tip/src/pkg/fmt/print.go:765#7  0x00000000004348c6 in fmt.(*pp).doPrintf (p=..., format=..., a=...) at/opt/go-tip/src/pkg/fmt/print.go:1111#8  0x000000000042bfa5 in fmt.Sprintf (format=..., a=..., ~anon2=...) at/opt/go-tip/src/pkg/fmt/print.go:229#9  0x0000000000534d4a in ugorji.net/logging.logR (calldepth=2 '\002', ctx=...,level=1000, message=...,     params=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/logging/logging.go:264#10 0x0000000000535053 in ugorji.net/logging.Trace (ctx=..., message=..., params=...,~anon3=...)    at /home/ugorji/depot/repo/src/ugorji.net/logging/logging.go:284#11 0x0000000000511b9a in ugorji.net/ndb.(*Db).SvrQuery (l=0xc2001b0240, ctxId=...,pkey=0xc200968630, kind=...,     opts=0x0, filters=..., qs=..., err=void) at /home/ugorji/depot/repo/src/ugorji.net/ndb/client.go:422#12 0x000000000052235f in ugorji.net/ndb.func·001 (k3=..., cursor3=..., err3=...)    at /home/ugorji/depot/repo/src/ugorji.net/ndb/low_level_driver.go:195#13 0x00000000004232e0 in ?? () at /opt/go-tip/src/pkg/runtime/asm_amd64.s:278#14 0x0000000000000000 in ?? ()------------------------------------Which compiler are you using (5g, 6g, 8g, gccgo)?6gWhich operating system are you using?Linux 3.5.0-28-generic #42-Ubuntu SMP Fri Mar 8 23:18:20 UTC 2013 x86_64 x86_64 x86_64GNU/LinuxWhich version are you using?  (run 'go version')go version devel +30c566874b83 Wed May 08 16:06:25 2013 -0700 linux/amd64Please provide any additional information below.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions


      [8]ページ先頭

      ©2009-2025 Movatter.jp