This repository was archived by the owner on Nov 23, 2017. It is now read-only.

python/asyncioPublic archive

NotificationsYou must be signed in to change notification settings
Fork185
Star1.1k

Use bytearray for buffer in _UnixWritePipeTransport#385

Closed

methane wants to merge1 commit intopython:masterfrommethane:unix-events-bytearray-buffer

Closed

Use bytearray for buffer in _UnixWritePipeTransport#385

methane wants to merge1 commit intopython:masterfrommethane:unix-events-bytearray-buffer

Conversation

Copy link

Member

methane commentedJul 22, 2016•
edited
Loading

fixes#384

This pull request is a control experiment of#339.

Use bytearray for buffer in _UnixWritePipeTransport

6a2d2c1

fixespython#384

Copy link

MemberAuthor

methane commentedJul 22, 2016•
edited
Loading

benchmark:draintest.py

master:

DEBUG:asyncio:Using selector: KqueueSelector1e+07 0.87678503990173342e+07 8.5216450691223144e+07 34.822274923324585^C

socketpair:writev (#339)

DEBUG:asyncio:Using selector: KqueueSelector1e+07 0.0283958911895751952e+07 0.0491249561309814454e+07 0.093784093856811521e+08 0.244533061981201172e+08 0.50157094001770024e+08 0.9511210918426514

methane:unix-events-bytearray-buffer

DEBUG:asyncio:Using selector: KqueueSelector1e+07 0.03060889244079592e+07 0.0574760437011718754e+07 0.110268115997314451e+08 0.278968811035156252e+08 0.56292700767517094e+08 1.111664056777954

methane mentioned this pull request

Jul 22, 2016

Fix #338 os.writev() instead of os.write(b.join()) and sendmsg() instead of send()#339

Closed

Copy link

MemberAuthor

methane commentedJul 22, 2016•
edited
Loading

Another benchmarkdraintest2.py

#339 :

$ time python3 draintest2.pyDEBUG:asyncio:Using selector: KqueueSelector1e+07 0.100913047790527342e+07 0.255423069000244144e+07 0.50108098983764651e+08 1.25108885765075682e+08 2.46446180343627934e+08 4.908963203430176real    0m9.668suser    0m6.515ssys 0m3.689s

this pull request:

$ time python3 draintest2.pyDEBUG:asyncio:Using selector: KqueueSelector1e+07 0.049723863601684572e+07 0.109540939331054694e+07 0.209958076477050781e+08 0.52494502067565922e+08 1.08761906623840334e+08 2.0913729667663574real    0m4.258suser    0m3.024ssys 0m1.657s

Copy link

MemberAuthor

methane commentedJul 22, 2016•
edited
Loading

More combination of chunksize.draintest3.py

#339 :

DEBUG:asyncio:Using selector: KqueueSelector10000 * 1000 0.128736972808837920000 * 1000 0.253782033920288140000 * 1000 0.49232983589172363100000 * 1000 1.2934849262237549200000 * 1000 2.859189033508301400000 * 1000 4.8905310630798341000 * 10000 0.059910058975219732000 * 10000 0.16155982017517094000 * 10000 0.371472120285034210000 * 10000 1.001580953598022520000 * 10000 2.02912998199462940000 * 10000 4.136893033981323100 * 100000 0.02856612205505371200 * 100000 0.0627448558807373400 * 100000 0.124001026153564451000 * 100000 0.498528003692626952000 * 100000 1.51949882507324224000 * 100000 3.5665440559387207real    0m23.664suser    0m14.943ssys 0m10.145s

this pull request:

DEBUG:asyncio:Using selector: KqueueSelector10000 * 1000 0.0597200393676757820000 * 1000 0.1016159057617187540000 * 1000 0.22113490104675293100000 * 1000 0.5041630268096924200000 * 1000 1.0610871315002441400000 * 1000 2.18842101097106931000 * 10000 0.036201953887939452000 * 10000 0.075136899948120124000 * 10000 0.155641078948974610000 * 10000 0.3964748382568359420000 * 10000 0.740575075149536140000 * 10000 1.5679919719696045100 * 100000 0.05654788017272949200 * 100000 0.10537409782409668400 * 100000 0.187467098236083981000 * 100000 0.430628061294555662000 * 100000 0.96805596351623544000 * 100000 1.6960680484771729real    0m10.811suser    0m6.724ssys 0m5.137s

So, when chunk is small (~10KB),bytearray is faster always.

When chunk is large (100KB),deque [bytes] is faster. Butbytearray overtakes when.write() called more than 1000 times.

Copy link

MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

use.extend() seems it is faster

Copy link

MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is bad knowhow: operator is faster than method, thanks to slot.

In [2]:defpluseq():   ...:buf=bytearray()   ...:for_inrange(1000):   ...:buf+=b'foo'   ...:In [3]:defextend():   ...:buf=bytearray()   ...:for_inrange(1000):   ...:buf.extend(b'foo')   ...:In [4]:%timeitpluseq()10000loops,bestof3:80.2µsperloopIn [5]:%timeitextend()10000loops,bestof3:135µsperloop

Copy link

MemberAuthor

methaneJul 22, 2016•
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

And more important: bytearray.extend() accepts sequence of integers [0, 255).

In [6]: buf = bytearray()In [7]: buf += [1,2,3]---------------------------------------------------------------------------TypeError                                 Traceback (most recent call last)<ipython-input-7-d99af3f0cf4a> in <module>()----> 1 buf += [1,2,3]TypeError: can't concat list to bytearrayIn [8]: buf.extend([1,2,3])In [9]: bufOut[9]: bytearray(b'\x01\x02\x03')

Copy link

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

In [6]: buf = bytearray()
In [7]: buf += [1,2,3]

This should not be allowed. Supporting this would make life for other loop implementations harder with no obvious benefits.

Copy link

MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

@1st1 Yes, it raises TypeError, as you can see above.

Copy link

socketpair commentedJul 22, 2016

@methane

Could you confirm draintest3.py bench?

what does that mean ?

Copy link

MemberAuthor

@socketpair I meant could you confirm draintest3.py measures right thing, and benchmark in you environment.
FYI, original draintest.py measures only.drain(). It doesn't include.write(data) time.
So it was not good for benchmarking bytearray vs deque[bytes].