Movatterモバイル変換

PRESENTATION TITLE GOES HERESwift Object Storage:Adding Erasure CodesPaul Luse, Sr. Staff Engineer – Intel CorporationKevin Greenan, Staff Software Engineer – BoxSep 2014

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.SNIA Legal NoticeThe material contained in this tutorial is copyrighted by the SNIA unlessotherwise noted.Member companies and individual members may use this material inpresentations and literature under the following conditions:Any slide or slides used must be reproduced in their entirety without modificationThe SNIA must be acknowledged as the source of any material used in the body ofany document containing material from these presentations.This presentation is a project of the SNIA Education Committee.Neither the author nor the presenter is an attorney and nothing in thispresentation is intended to be, or should be construed as legal advice or anopinion of counsel. If you need legal advice or a legal opinion pleasecontact your attorney.The information presented herein represents the author's personal opinionand current understanding of the relevant issues involved. The author, thepresenter, and the SNIA do not assume any responsibility or liability fordamages arising out of any reliance on or use of this information.NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.2

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.AbstractSwift Object Storage: Adding Erasure CodesThis session will provide insight into this extremely successfulcommunity effort of adding an Erasure Code capability to theOpenStack Swift Object Storage System by walking theaudience through the design and development experiencethrough the eyes of the developers from key contributors. Anoverview of Swift Architecture and basic Erasure Codes will befollowed by design/implementation details.3

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Agenda• Swift– A Community Project– Swift Overview– Storage Policies• Erasure Codes– History– Variations– Matrix encode/decode– PyECLib & liberasurecode• Erasure Code Implementation for Swift– Design considerations– Architecture overview

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Swift:A Community Project• Vibrant community,• top contributing companies for Juno include: SwiftStack*,Intel, Redhat*, IBM*, HP*, Rackspace*, Box*• The path to EC…• Core OpenStack* Service• One of the original 2 projects• 100% Python• ~ 35K LOC• > 2x that in unit, functional, error injection codeJunoFall ‘14IceHouseSpring ‘14HavanaFall ‘13KiloSpring ‘15Early ECDiscussionPoliciesDevelopmentPoliciesReleased EC Ready!EC LibraryDevelopmentSwift ECDevelopment

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Swift OverviewcontainerobjectObjects are organizedwith containers• Uses container model for grouping objectswith like characteristics– Objects are identified by their paths and haveuser-defined metadata associated with them• Accessed via RESTful interface– GET, PUT, DELETE• Built upon standard hardware and highlyscalable– Cost effective, efficient

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.What Swift is Not• Block Storage System– Does not provide block-level storage serviceNot a “One Size Fits All” Storage Solution• Distributed File System– Does not provide POSIX file system API support• Relational Database– Does not support ACID semantics• NoSQL Data Store– Not built on the Key-Value/Document/Column-Familymodel

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.wsgi serveraccount controllermiddlewareswift proxywsgi applicationcontainer controllerobject controllerhelper functionswsgi serverreplicatormiddlewareswift objectwsgi applicationauditorexpirerupdaterreplicatorswift accountwsgi applicationauditorreaperhelper functionsreplicatorswift containerwsgi applicationauditorsyncupdaterProxyNodesStorageNodesSwift Software Architecture

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Swift 2.0:Why Storage Policies?containerobjectYou get N-replication inevery container• There were no core capabilities to expose ormake use of differentiated hardware within thecluster• If several nodes of a cluster have newer/fastercharacteristics, they can’t be fully realized (theadministrator/users are at the mercy of the dispersionalgorithm alone for data placement).• There’s was no extensibility for additionaldurability schemes– Use of erasure codes (EC)– Mixed use of schemes (some nodes do 2x, some do3x, some do EC)• Prior to, durability scheme applies to entire cluster– Can do replication of 2x, 3x, etc., however the entirecluster must use that setting

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.• The ring is a static data structure maintained external tothe cluster (tools provided)Ringpartitions• Each partition maps to a list of devices via two arrayelements within the ring structure• Devices are assigned to partitions with several policies(regions, zones, etc.) and constraints to assure faulttolerance and load balancingIdx Copy 1 Copy 2 Copy 30 11 21 43… … … …10 34 1 0… … … …Idx Device0 Node 3, device 11 Node 12, device 2… ….34 Node 1, device 4… …• An object name maps to a partition via MD5 hashMD5*(URL) = indexNote: Swift uses a modified MD5 consistent hashing ringreplica2part2dev_iddevsThe Swift Ring

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.TripleReplicationn locations,object fragments• Introduction of multiple object rings– Swift supports multiple rings already, but only one forobject – the others are for account and container DB.• Introduction of container tag: X-Storage-Policy– New immutable container metadata– Policy change accomplished via data movement– Each container is associated with a potentially different ringReducedReplicationErasureCodes3 Different Policies:3 Different Rings3 locations,same object2 locations,same objectWhat are Policies?

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Capacity TierAccess TierPutting it AllTogetherZone 1 Zone 2 Zone 3 Zone 4 Zone 5ClientsRESTful API, Similar to S3DownloadCopy 1Copy 2Copy 3UploadObj A• Handle incoming requests• Handle failures, ganged responses• Scalable shared nothing architecture• Consistent hashing ring distributionScalable for concurrency and/or capacity independently• Actual object storage• Variable replication count• Data integrity services• Scale-out capacityAuthServiceObj A

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Agenda• Swift– Swift Overview– Storage Policies• Erasure Codes– Background– Example encode/decode using Reed-Solomon– Minimizing reconstruction cost– PyECLib & liberasurecode• Erasure Code Implementation for Swift– Design considerations– Architecture overview

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.History of Erasure CodingSource: Tutorial on Erasure Coding for Storage Applications, Part 1. James Plank FAST ‘131960’s• Coding Theory• Reed Solomon, Berlekamp–Massey algorithm1990’s• Storage• RAID-6: EVENODD, RDP, X-Code• Graph Theory• LDPC Codes (Tornado, Raptor, LT)2000’s• Coding Theory• Network / Regenerating Codes2010’s• Storage• Non-MDS codes for cloud and recovery

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Terminology• Split a file into k chunks and encode into n chunks, where n-k=m• Systematic vs. Non-systematic– Systematic: encoded output contains input symbols– Non-systematic: encoded output does not contain input symbols• Code word– A set of data and parity related via a set of parity equations– Systematic: f(data) = code word = (data, parity)• Layout– Flat horizontal: each coded symbol is mapped to one device– Array codes have multiple symbols per device: horizontal and vertical• MDS vs. non-MDS– MDS: any k chunks can be used to recover the original file– Non-MDS: k chunks may not be sufficient to recover the fileTraditionally, storage systems use systematic, MDS codes

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.• Reed-Solomon Codes• Fountain Codes• RAID-6 EVENODD• RAID-6 X-Code• Generalized XOR• Pyramid Codes• Local Repairable Codes (LRC)• Partial MDS Codes (PMDS)• Simple Regenerating Codesand the list goes on…Variations

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Reed Solomon SystematicHorizontal Code LayoutExample RS(8,5)d2,0d2,1d2,2d2,3Datad3,0d3,1d3,2d3,3d4,0d4,1d4,2d4,3d0,0d0,1d0,2d0,3d1,0d1,1d1,2d1,3p0,0p0,1p0,2p0,3p1,0p1,1p1,2p1,3p2,0p2,1p2,2p2,3Parity• Can Tolerate (n-k) Failures• Overhead of just (n/k)total disks = n = 8parity disks = m = 3data disks = k = 5Code word

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Reed Solomon SystematicGenerator Matrix1 α01 α02 α03 α041 α11 α12 α13 α141 α21 α22 α23 α241 α31 α32 α33 α341 α41 α42 α43 α441 α51 α52 α53 α541 α61 α62 α63 α641 α71 α72 α73 α741 0 0 0 00 1 0 0 00 0 1 0 00 0 0 1 00 0 0 0 11 1 1 1 1g0 g1 g2 g3 g4g5 g6 g7 g8 g9f(x) = c0 +c1x1 + c2x2 + ... + ck-1xk-1 Coefficients arethe dataf(α0) = y0f(α1) = y1…f(αn-1) = yn-1Reed-Solomon is encoded byoversampling a polynomialElementaryOpsResulthas samerankf(α0)

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Reed Solomon SystematicMatrix Encoding ProcessAll operations are done in a Galois fieldAny (k x k) sub-matrix is invertibleCode word is the vector-matrix productof the generator matrix and source datapi = d0 + gid1 + gi2d2 + gi3d3 + … + gik-1dk-1d0d1d2d3d4=d0d1d2d3d4p1p2p3Generator Matrix data code word1 0 0 0 00 1 0 0 00 0 1 0 00 0 0 1 00 0 0 0 11 1 1 1 1g0 g1 g2 g3 g4g5 g6 g7 g8 g9

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Step 1: Eliminate all but k available rows in the generator matrixGenerator Matrix × data = parityd0d1d2d3d4=d0d1d2d3d4p1p2p3XXXThesedisksjustdiedReed Solomon SystematicMatrix Decoding Process1 0 0 0 00 1 0 0 00 0 1 0 00 0 0 1 00 0 0 0 11 1 1 1 1g0 g1 g2 g3 g4g5 g6 g7 g8 g9

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Step 2:Invert the resulting matrix0 1 0 0 00 0 1 0 00 0 0 0 11 1 1 1 1g0 g1 g2 g3 g4Reed Solomon SystematicMatrix Decoding ProcessInvertG’

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Step 3: “Solve” by multiplying k element vector of availablesymbols by corresponding rows of G’=G’ G G’DataRemaining= G’d0d1d2d3d4d1d2d4p1p2d0 = g’00d1 + g’01d2 + g’02d4 + g’03p1 + g’04p2d3 = g’30d1 + g’31d2 + g’32d4 + g’33p1 + g’34p2Here’s the missing data!Reed Solomon SystematicMatrix Decoding Process

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Minimizing Reconstruction Cost• Reed-Solomon requires k available elements to reconstruct any missing element• This has given rise to many codes that minimize repair costs– Regenerating codes, locally repairable codes, flat-XOR codes, etc.– Trade space efficiency for more efficient reconstruction• Replication repair-optimal, RS is space-optimal, these are somewhere in the middle• Simple XOR-only example with k = 6, m = 4:P0 = D0 + D1 + D3P1 = D1 + D2 + D5P2 = D0 + D2 + D4P3 = D3 + D4 + D5××D0 = P0 + D1 + D3D0 = P2 + D2 + D4Only requires 3 devices to reconstruct one failed device

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.PyECLib†Relative tests conducted with home grown test code using ISA-L libra• Goal: provide a pluggable, easy-to-use EC library for Python• Swift is the main use-case• Originally had all logic in PyECLib, but have offloaded “smarts” to liberasurecode– Separation of concerns: one converts between Python and C, and the other does erasure coding– API of PyECLib is same as liberasurecodeencodeAPICorePython-C Interfaceliberasurecodedecode reconstruct required_fragments check_metadata segement_info

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.liberasurecode†Relative tests conducted with home grown test code using ISA-L libra• Goal: Separate EC-specific logic from language-specific translation• Embedded metadata: original file size, checksum, version info, etc.• Provides ability to plug-in and use new erasure code schemes/libraries– In addition to XOR codes, we currently provide Jerasure and ISA-Lbackend plug-in layerCoreBackend APIAPIbuilt-in utilsBackend-A Backend-B …External LibrarieslibA libB …XOR CodesChecksumsHelpersPre/Post-Processingencode decode reconstruct required_fragments check_metadata segement_info

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Agenda• Swift– Swift Overview– Storage Policies• Erasure Codes– History– Variations– Matrix encode/decode– PyECLib & liberasurecode• Erasure Code Implementation for Swift– Design considerations– Architecture overview

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Design Considerations• GET/PUT Erasure Code encode/decode done at proxy server– Aligned with current Swift architecture to focus hardware demanding services in the accesstier– Enable in-line Erasure Code directed by client as well as off-line Erasure Code directed bysideband application / management tier• Build Upon Storage Policies– New container metadata will identify whether objects within it are erasure coded• Keep it simple and leverage current architecture– Multiple new storage node services required to assure Erasure Code chunk integrity as wellas Erasure Code stripe integrity; modeled after replica services– Storage nodes participate in Erasure Code encode/decode for reconstruction analogous toreplication services synchronizing objects

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Swift With EC Architecture High Levelwsgi serverexisting modulesmiddlewareswift proxywsgi applicationwsgi servermiddlewareswift objectwsgi applicationswift accountwsgi applicationswift containerwsgi applicationProxyNodesStorageNodescontroller modificationsPyECLibPlug in 1 Plug in 2existing modules existing modulesexisting modules EC AuditorEC ReconstructorPyECLibPlug in 1 Plug in 2metadata changes

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Capacity Tier (Storage)Access Tier (Concurrency)Swift With Erasure CodeZone 1 Zone 2 Zone 3 Zone 4 Zone 5ClientsRESTful API, Similar to S3DownloadFrag 1Frag 2Frag 3Frag 4Frag NDecoderUploadEncoderObj A Obj A• Applications control policy• EC can be inline or offlineErasure Code Technology Lowering TCO for Swift• Supports multiple policies• EC flexibility via plug-inAuthService

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.For More Information…• Trello discussion board:https://trello.com/b/LlvIFIQs/swift-erasure-codes• Launchpad blueprints:https://blueprints.launchpad.net/swift• Swift Code (see feature/EC branch):https://code.launchpad.net/swift• PyECLib:https://bitbucket.org/kmgreen2/pyeclib• Liberasurecode:https://bitbucket.org/tsg-/liberasurecode

Insert tutorial title in footer© 2013 Storage Networking Industry Association. All Rights Reserved.Attribution & Feedback31Please send any questions or comments regarding this SNIATutorial to tracktutorials@snia.orgThe SNIA Education Committee thanks the followingindividuals for their contributions to this Tutorial.Authorship HistoryName/Date of Original Author here:Paul Luse, Kevin Greenan. 8/2014Updates:NoneAdditional ContributorsNone

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Block File ObjectBlock, File & Object33Specific location ondisks / memoryTracksSectorsSpecific folder infixed logical orderFile pathFile nameDateFlexiblecontainer sizeData and MetadataUnique ID

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Object StoreScalabilityFlat namespaceNo volume semanticsNo Locking/AttributesContains metadataDurabilityReplication orErasure codeManageabilityREST APILow overheadConsistencyEventually consistent

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.Swift Use Casesserver1:/exp1 server2:/exp1DISTRIBUTED VOLUMEBRICK BRICKSwift ClusterStorageNodeStorageNodeWeb/Mobile• Images/Audio/Video• Middleware for dynamicresizing• Ex: WikipediaIAAS-Public or private-Multi-tenant storage-Multiple available gatewaysVM Images-OpenStack* Integration-Managed by GlanceCloudBackup-Large unstructured dataProxy NodesObjectsAccessTierCapacityTierLoad BalancerAuth

Swift Object Storage: Adding Erasure Codes© 2014 Storage Networking Industry Association. All Rights Reserved.The CAP Theorem: Pick 2Consistency(same view)Availability(data access)Partition Tolerance(node access)EnforcedConsistencyEventualConsistencySwift chooses Availability and Partition Tolerance over Consistency

Movatterモバイル変換

Change Language

SNIA : Swift Object Storage adding EC (Erasure Code)

Embed presentation

Recommended

More Related Content

What's hot

Viewers also liked

Similar to SNIA : Swift Object Storage adding EC (Erasure Code)

More from Odinot Stanislas

Recently uploaded

SNIA : Swift Object Storage adding EC (Erasure Code)