CN116225323A - A data storage method and device - Google Patents
A data storage method and deviceDownload PDFInfo
- Publication number
- CN116225323A CN116225323ACN202310034670.4ACN202310034670ACN116225323ACN 116225323 ACN116225323 ACN 116225323ACN 202310034670 ACN202310034670 ACN 202310034670ACN 116225323 ACN116225323 ACN 116225323A
- Authority
- CN
- China
- Prior art keywords
- data
- generation algorithm
- data storage
- vehicle
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Biophysics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本申请实施例涉及智能驾驶数据存储技术，尤指一种数据存储方法和装置。The embodiment of the present application relates to intelligent driving data storage technology, especially a data storage method and device.
背景技术Background technique
随着汽车智能驾驶技术的持续进步，以及国家在智能驾驶领域相关政策的不断出台和完善，汽车智能化程度越来越高，所产生的数据量也越来越大。10台智能驾驶路采车每天大概新增300TB左右非结构化(图片、音频、视频、对象)数据；在量产车方面，每天大概新增80TB智能驾驶数据(未筛选前180TB左右，由170万辆车产生)。现在主要采用块存储、文件存储和对象存储等静态存储模式，每天需要380TB左右磁盘容量才能满足数据存储需求，每月需要存储资源1.1PB。随着智能驾驶场景逐步增加，路采规模和智驾数据范围急速扩大，存储需求越来越大，呈几何上升趋势。按照每年新增10台路采车，汽车销量150万辆来估算，到2025年每月需要存储近3PB，一年需要投入4000万资金成本用于采购存储资源，且在不断上升，数据存储资源成本占用比例太高。With the continuous advancement of automobile intelligent driving technology and the continuous introduction and improvement of relevant national policies in the field of intelligent driving, the degree of intelligence of automobiles is getting higher and higher, and the amount of data generated is also increasing. 10 intelligent driving road mining vehicles add about 300TB of unstructured (picture, audio, video, object) data every day; in terms of mass-produced vehicles, about 80TB of intelligent driving data are added every day (about 180TB before screening, from 170TB million vehicles generated). At present, static storage modes such as block storage, file storage, and object storage are mainly used, and about 380TB of disk capacity is required every day to meet data storage requirements, and 1.1PB of storage resources are required every month. With the gradual increase of intelligent driving scenarios, the scale of road mining and the scope of intelligent driving data are rapidly expanding, and the demand for storage is increasing, showing a geometric upward trend. Based on an estimate of 10 new road vehicles and 1.5 million car sales each year, by 2025, nearly 3PB will be stored per month, and 40 million capital costs will be invested in the purchase of storage resources a year, and it will continue to rise. Data storage resources The cost ratio is too high.
目前在业界只能通过压缩技术减少数据容量，节约存储资源。但是数据压缩的空间非常有限，不同数据类型和文件类型的压缩率均不一样，特别在音视频数据文件压缩方面压缩率非常有限。Currently in the industry, only compression technology can be used to reduce data capacity and save storage resources. However, the space for data compression is very limited, and the compression rates of different data types and file types are different, especially in the compression of audio and video data files.
发明内容Contents of the invention
本申请实施例提供了一种数据存储方法和装置，能够大幅度减少车辆数据对存储空间的占用，节省存储空间。Embodiments of the present application provide a data storage method and device, which can greatly reduce the storage space occupied by vehicle data and save storage space.
本申请实施例提供了一种数据存储方法，所述方法可以包括：An embodiment of the present application provides a data storage method, the method may include:
获取待存储的车辆数据；Obtain vehicle data to be stored;
获取采用预设的生成算法生成所述车辆数据时所需的生成算法文本数据；Acquiring the generation algorithm text data required when the vehicle data is generated using a preset generation algorithm;
将所述生成算法文本数据存储入预设的人工智能AI数据存储模型中。Store the generated algorithmic text data into the preset artificial intelligence AI data storage model.
在本申请的示例性实施例中，在将所述生成算法文本数据存储入预设的人工智能AI数据存储模型中之前，所述方法还可以包括：In an exemplary embodiment of the present application, before storing the generating algorithm text data into a preset artificial intelligence AI data storage model, the method may further include:
对所述生成算法文本数据进行压缩。Compress the text data of the generating algorithm.
在本申请的示例性实施例中，所述方法还可以包括：In an exemplary embodiment of the present application, the method may further include:
预先获取多种车辆数据的生成算法文本数据，作为训练数据；Acquire the text data of the generation algorithm of various vehicle data in advance as the training data;
采用所述训练数据对预先创建的用于存储数据的神经网络模型进行训练，获取AI数据存储模型。The training data is used to train the pre-created neural network model for storing data to obtain the AI data storage model.
在本申请的示例性实施例中，在将所述生成算法文本数据存储入预设的人工智能AI数据存储模型中以后，所述方法还可以包括：In an exemplary embodiment of the present application, after storing the generated algorithm text data into a preset artificial intelligence AI data storage model, the method may further include:
在需要查询所述车辆数据时，由预设的数据生成器根据查询请求、所述生成算法以及所述AI数据存储模型中存储的相应的生成算法文本数据生成所要查询的车辆数据。When the vehicle data needs to be queried, the preset data generator generates the vehicle data to be queried according to the query request, the generation algorithm and the corresponding generation algorithm text data stored in the AI data storage model.
在本申请的示例性实施例中，所述由预设的数据生成器根据查询请求、所述生成算法以及所述AI数据存储模型中存储的相应的生成算法文本数据生成所要查询的车辆数据，可以包括：In an exemplary embodiment of the present application, the preset data generator generates the vehicle data to be queried according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model, Can include:
由所述数据生成器根据所述查询请求发送查询指令到所述AI数据存储模型，并获取所述AI数据存储模型根据所述查询指令查询到的生成算法文本数据；The data generator sends a query instruction to the AI data storage model according to the query request, and obtains the generation algorithm text data queried by the AI data storage model according to the query instruction;
由所述数据生成器对查询到的所述生成算法文本数据进行解压，并根据解压后的生成算法文本数据，执行所述生成算法，生成所要查询的车辆数据，并将生成的所述车辆数据返回给查询所述辆数据的外部数据服务。The data generator decompresses the queried generation algorithm text data, executes the generation algorithm according to the decompressed generation algorithm text data, generates the vehicle data to be queried, and converts the generated vehicle data Returned to the external data service querying the vehicle data.
在本申请的示例性实施例中，所述生成算法可以包括：深度卷积对抗生成网络Dcgan算法和/或掩膜自编码器MAE算法。In an exemplary embodiment of the present application, the generation algorithm may include: a deep convolutional adversarial generation network Dcgan algorithm and/or a masked autoencoder MAE algorithm.
在本申请的示例性实施例中，在由预设的数据生成器根据查询请求、所述生成算法以及所述AI数据存储模型中存储的相应的生成算法文本数据生成所要查询的车辆数据以后，所述方法还可以包括：In an exemplary embodiment of the present application, after the preset data generator generates the vehicle data to be queried according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model, The method may also include:
通过预设的判别器对生成的所述车辆数据进行判别和矫正。The generated vehicle data is discriminated and corrected by a preset discriminator.
在本申请的示例性实施例中，所述判别器采用Dcgan算法体系中的样本数据模型。In an exemplary embodiment of the present application, the discriminator uses a sample data model in the Dcgan algorithm system.
在本申请的示例性实施例中，所述获取待存储的车辆数据，可以包括：In an exemplary embodiment of the present application, the acquiring the vehicle data to be stored may include:
根据预定的数据上报频次，通过数据快递或网络传输将车端采集的车辆数据传输到预设的云端数据缓冲区。According to the scheduled data reporting frequency, the vehicle data collected by the vehicle end is transmitted to the preset cloud data buffer through data express or network transmission.
本申请实施例还提供了一种数据存储装置，可以包括处理器和计算机可读存储介质，所述计算机可读存储介质中存储有指令，当所述指令被所述处理器执行时，实现所述的数据存储方法。The embodiment of the present application also provides a data storage device, which may include a processor and a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and when the instructions are executed by the processor, the The data storage method described above.
与相关技术相比，本申请实施例可以包括：获取待存储的车辆数据；获取采用预设的生成算法生成所述车辆数据时所需的生成算法文本数据；将所述生成算法文本数据存储入预设的人工智能AI数据存储模型中。通过该实施例方案，大幅度减少了车辆数据对存储空间的占用，节省了存储空间。Compared with related technologies, the embodiment of the present application may include: acquiring the vehicle data to be stored; acquiring the generation algorithm text data required when the vehicle data is generated using a preset generation algorithm; storing the generation algorithm text data in In the preset artificial intelligence AI data storage model. Through the solution of this embodiment, the occupation of the storage space by the vehicle data is greatly reduced, and the storage space is saved.
本申请的其它特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请而了解。本申请的其他优点可通过在说明书以及附图中所描述的方案来实现和获得。Additional features and advantages of the application will be set forth in the description which follows, and, in part, will be obvious from the description, or may be learned by practice of the application. Other advantages of the present application can be realized and obtained through the schemes described in the specification and drawings.
附图说明Description of drawings
附图用来提供对本申请技术方案的理解，并且构成说明书的一部分，与本申请的实施例一起用于解释本申请的技术方案，并不构成对本申请技术方案的限制。The accompanying drawings are used to provide an understanding of the technical solution of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solution of the present application, and do not constitute a limitation to the technical solution of the present application.
图1为本申请实施例的数据存储方法流程图；Fig. 1 is the flow chart of the data storage method of the embodiment of the present application;
图2为本申请实施例的一种数据存储方法结构示意图；FIG. 2 is a schematic structural diagram of a data storage method according to an embodiment of the present application;
图3为本申请实施例的存储后数据查询方案示意图；FIG. 3 is a schematic diagram of a stored data query solution according to an embodiment of the present application;
图4为本申请实施例的数据存储装置组成框图。FIG. 4 is a block diagram of a data storage device according to an embodiment of the present application.
具体实施方式Detailed ways
本申请描述了多个实施例，但是该描述是示例性的，而不是限制性的，并且对于本领域的普通技术人员来说显而易见的是，在本申请所描述的实施例包含的范围内可以有更多的实施例和实现方案。尽管在附图中示出了许多可能的特征组合，并在具体实施方式中进行了讨论，但是所公开的特征的许多其它组合方式也是可能的。除非特意加以限制的情况以外，任何实施例的任何特征或元件可以与任何其它实施例中的任何其他特征或元件结合使用，或可以替代任何其它实施例中的任何其他特征或元件。The application describes a number of embodiments, but the description is illustrative rather than restrictive, and it will be obvious to those of ordinary skill in the art that within the scope of the embodiments described in the application, There are many more embodiments and implementations. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are possible. Except where expressly limited, any feature or element of any embodiment may be used in combination with, or substituted for, any other feature or element of any other embodiment.
本申请包括并设想了与本领域普通技术人员已知的特征和元件的组合。本申请已经公开的实施例、特征和元件也可以与任何常规特征或元件组合，以形成由权利要求限定的独特的发明方案。任何实施例的任何特征或元件也可以与来自其它发明方案的特征或元件组合，以形成另一个由权利要求限定的独特的发明方案。因此，应当理解，在本申请中示出和/或讨论的任何特征可以单独地或以任何适当的组合来实现。因此，除了根据所附权利要求及其等同替换所做的限制以外，实施例不受其它限制。此外，可以在所附权利要求的保护范围内进行各种修改和改变。This application includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The disclosed embodiments, features and elements of this application can also be combined with any conventional features or elements to form unique inventive solutions as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive solutions to form another unique inventive solution as defined by the claims. It is therefore to be understood that any of the features shown and/or discussed in this application can be implemented alone or in any suitable combination. Accordingly, the embodiments are not to be limited except in accordance with the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.
此外，在描述具有代表性的实施例时，说明书可能已经将方法和/或过程呈现为特定的步骤序列。然而，在该方法或过程不依赖于本文所述步骤的特定顺序的程度上，该方法或过程不应限于所述的特定顺序的步骤。如本领域普通技术人员将理解的，其它的步骤顺序也是可能的。因此，说明书中阐述的步骤的特定顺序不应被解释为对权利要求的限制。此外，针对该方法和/或过程的权利要求不应限于按照所写顺序执行它们的步骤，本领域技术人员可以容易地理解，这些顺序可以变化，并且仍然保持在本申请实施例的精神和范围内。Furthermore, in describing representative embodiments, the specification may have presented a method and/or process as a particular sequence of steps. However, to the extent the method or process is not dependent on the specific order of steps described herein, the method or process should not be limited to the specific order of steps described. Other sequences of steps are also possible, as will be appreciated by those of ordinary skill in the art. Therefore, the specific order of the steps set forth in the specification should not be construed as limitations on the claims. In addition, claims for the method and/or process should not be limited to performing their steps in the order written, those skilled in the art can easily understand that these orders can be changed and still remain within the spirit and scope of the embodiments of the present application Inside.
本申请实施例提供了一种数据存储方法，如图1、图2所示，所述方法可以包括步骤S101-S103：The embodiment of the present application provides a data storage method, as shown in Figure 1 and Figure 2, the method may include steps S101-S103:
S101、获取待存储的车辆数据；S101. Obtain vehicle data to be stored;
S102、获取采用预设的生成算法生成所述车辆数据时所需的生成算法文本数据；S102. Obtain the generation algorithm text data required when the vehicle data is generated using a preset generation algorithm;
S103、将所述生成算法文本数据存储入预设的人工智能AI数据存储模型中。S103. Store the generated algorithm text data into a preset artificial intelligence AI data storage model.
在本申请的示例性实施例中，由于智能驾驶研发和量产场景规模快速增加，智能驾驶数据存储资源呈几何上升，存储成本投入越来越高。从技术角度，如何提高智能驾驶存储资源使用价值，减少智能驾驶存储资源需求量，同时又能满足智能驾驶数据存储需求，是一个迫切需要研究解决的课题。In the exemplary embodiment of this application, due to the rapid increase in the scale of intelligent driving research and development and mass production scenarios, intelligent driving data storage resources are geometrically increasing, and storage costs are getting higher and higher. From a technical point of view, how to improve the use value of intelligent driving storage resources, reduce the demand for intelligent driving storage resources, and at the same time meet the demand for intelligent driving data storage is an urgent research topic.
在本申请的示例性实施例中，从人工智能算法的角度减少了数据对存储空间的占用量，存储原理与传统存储方式完全不同。In the exemplary embodiment of the present application, from the perspective of artificial intelligence algorithms, the amount of storage space occupied by data is reduced, and the storage principle is completely different from traditional storage methods.
在本申请的示例性实施例中，通过人工智能算法获取待存储的车辆数据的生成算法文本数据，并将该生成算法文本数据存储到人工智能存储模型中，从而由存储生成算法文本数据代替传统的存储原始的车辆数据，由于存储生成算法文本数据占用的存储空间远远小于原始的车辆数据占用的存储空间要小，从而实现了降低存储空间占用的目的。In the exemplary embodiment of the present application, the generation algorithm text data of the vehicle data to be stored is acquired through an artificial intelligence algorithm, and the generation algorithm text data is stored in the artificial intelligence storage model, thereby replacing the traditional The original vehicle data is stored, because the storage space occupied by the text data of the storage generation algorithm is much smaller than the storage space occupied by the original vehicle data, thereby achieving the purpose of reducing the storage space occupation.
在本申请的示例性实施例中，该原始的待存储的车辆数据可以包括但不限于：视频数据、图片数据、语音数据、雷达数据等各种形式的数据。In an exemplary embodiment of the present application, the original vehicle data to be stored may include, but not limited to, various forms of data such as video data, picture data, voice data, and radar data.
在本申请的示例性实施例中，例如，当待存储的车辆数据为一条视频数据时，直接对该原始的视频数据进行存储将会占用很大的空间，此时可以采用预设的生成算法生成该视频数据，并获取生成该视频数据过程中产生的生成算法文本数据，作为保存对象，并不直接存储原始的视频数据。该视频数据对应的生成算法文本数据所占用的存储空间要远小于视频数据本身占用的存储空间，因此，通过该方案大大减少了存储空间的占用。In an exemplary embodiment of the present application, for example, when the vehicle data to be stored is a piece of video data, directly storing the original video data will take up a lot of space, and a preset generation algorithm can be used at this time The video data is generated, and the generation algorithm text data generated in the process of generating the video data is obtained as a storage object, and the original video data is not directly stored. The storage space occupied by the generation algorithm text data corresponding to the video data is much smaller than the storage space occupied by the video data itself, therefore, the storage space occupation is greatly reduced through this solution.
在本申请的示例性实施例中，存储空间的节省率不依赖或针对特定的文件类型，不管是普通文本数据，还是视频数据、语音数据，均可生成相应的生成算法文本数据。In the exemplary embodiment of the present application, the saving rate of the storage space does not depend on or is specific to a specific file type, regardless of whether it is ordinary text data, video data, or voice data, corresponding generation algorithm text data can be generated.
在本申请的示例性实施例中，生成算法文本数据可以是指根据预设的生成算法在生成车辆数据(例如一条视频，一张图片)过程中所需要的全部或部分参数、变量等数据；每个车辆数据(例如一条视频，一张图片)可以唯一对应一组生成算法文本数据；该生成算法文本数据可以包含一个或一组数据表达式。In an exemplary embodiment of the present application, generating algorithm text data may refer to all or part of parameters, variables and other data required in the process of generating vehicle data (such as a video, a picture) according to a preset generating algorithm; Each vehicle data (such as a video, a picture) can uniquely correspond to a set of generation algorithm text data; the generation algorithm text data can contain one or a set of data expressions.
在本申请的示例性实施例中，通过改变存储数据形态(将原生的车辆数据转化为该车来那个数据对应的生成算法文本数据)，保存的不是原始的车辆数据，而是保存能够重新生成该车辆数据的生成算法文本数据，减少了数据对存储空间的占用，提高了存储空间的使用价值。In the exemplary embodiment of the present application, by changing the storage data form (converting the original vehicle data into the text data corresponding to the generation algorithm of the vehicle data), what is saved is not the original vehicle data, but the data that can be regenerated The vehicle data generation algorithm text data reduces the storage space occupied by the data and improves the use value of the storage space.
在本申请的示例性实施例中，例如，一张10MB的图片数据，其生成算法大概在1.2MB，在加上数据压缩，可以极大减少存储资源占用。又例如，假设每天新增380TB的智能驾驶数据，如果用传统的直接存储原始车辆数据存储模式进行保存，需要使用380TB的存储空间资源才能完成这些数据的保存。如果使用本申请实施例方案进行存储：380TB的数据不变，但是只需要46TB(空间节省比率≈0.87)存储空间即可满足数据保存需求；而且现在用于存储1天数据的存储资源，通过使用本申请的存储方法，可存储9天的数据。In the exemplary embodiment of the present application, for example, a 10MB picture data, its generation algorithm is about 1.2MB, and data compression is added, which can greatly reduce the storage resource occupation. For another example, assume that 380TB of intelligent driving data is newly added every day. If the traditional direct storage of original vehicle data storage mode is used for storage, 380TB of storage space resources are required to complete the storage of these data. If the solution of the embodiment of this application is used for storage: 380TB of data remains unchanged, but only 46TB (space saving ratio ≈ 0.87) of storage space is required to meet the data storage requirements; and now the storage resources used to store 1 day of data, by using The storage method of this application can store 9 days of data.
在本申请的示例性实施例中，所述生成算法可以包括但不限于：深度卷积对抗生成网络Dcgan算法和/或掩膜自编码器MAE算法。In an exemplary embodiment of the present application, the generation algorithm may include, but is not limited to: a deep convolutional adversarial generation network Dcgan algorithm and/or a masked autoencoder MAE algorithm.
在本申请的示例性实施例中，所述获取待存储的车辆数据，可以包括：In an exemplary embodiment of the present application, the acquiring the vehicle data to be stored may include:
根据预定的数据上报频次，通过数据快递或网络传输将车端采集的车辆数据传输到预设的云端数据缓冲区。According to the scheduled data reporting frequency, the vehicle data collected by the vehicle end is transmitted to the preset cloud data buffer through data express or network transmission.
在本申请的示例性实施例中，车端通过摄像头、传感器、雷达等设备采集并临时存储智能驾驶相关的车辆数据到安全介质(例如，存入硬盘)，可以根据预定的数据上报频次，通过数据快递(人工运输存储介质)或安全专网(网络传输)将这些车辆数据传输到云端数据缓冲区。In an exemplary embodiment of the present application, the vehicle end collects and temporarily stores intelligent driving-related vehicle data to a secure medium (for example, stored in a hard disk) through cameras, sensors, radars, and other equipment. According to the predetermined frequency of data reporting, through Data express (manual transport storage medium) or secure private network (network transmission) transmits these vehicle data to the cloud data buffer.
在本申请的示例性实施例中，预设的存储转换器可以根据预设的生成算法和云端数据缓冲区中的车辆数据获取与该车辆数据对应的生成算法文本数据，并将该生成算法文本数据存储到AI(人工智能)数据存储模型中，完成对生成算法文本数据的保存，相当于完成了对相应的原始的车辆数据的保存，此时云端数据缓冲区中的原生数据可以被清除。本申请实施例方案中保存的是用于重新生成原生的车辆数据的生成算法文本数据，不是原生的车辆数据。In an exemplary embodiment of the present application, the preset storage converter can obtain the generation algorithm text data corresponding to the vehicle data according to the preset generation algorithm and the vehicle data in the cloud data buffer, and store the generation algorithm text The data is stored in the AI (artificial intelligence) data storage model, and the preservation of the generated algorithm text data is completed, which is equivalent to the preservation of the corresponding original vehicle data. At this time, the original data in the cloud data buffer can be cleared. What is saved in the solution of the embodiment of the present application is the generation algorithm text data used to regenerate the original vehicle data, not the original vehicle data.
在本申请的示例性实施例中，在将所述生成算法文本数据存储入预设的人工智能AI数据存储模型中之前，所述方法还可以包括：In an exemplary embodiment of the present application, before storing the generating algorithm text data into a preset artificial intelligence AI data storage model, the method may further include:
对所述生成算法文本数据进行压缩。Compress the text data of the generating algorithm.
在本申请的示例性实施例中，通过对生成算法文本数据的压缩可以进一步减小生成算法文本数据占用的空间。In an exemplary embodiment of the present application, the space occupied by the text data of the generation algorithm can be further reduced by compressing the text data of the generation algorithm.
在本申请的示例性实施例中，可以根据智能驾驶数据的完整性要求，选择有损压缩或无损压缩。In an exemplary embodiment of the present application, lossy compression or lossless compression can be selected according to the integrity requirements of intelligent driving data.
在本申请的示例性实施例中，压缩比取决于选择的压缩算法，如果使用有损压缩，压缩率较高，但是数据会失真；如果使用无损压缩，压缩率较低，但是数据还原度较高，可根据不同数据的完整性要求选择相应的压缩算法。In the exemplary embodiment of the present application, the compression ratio depends on the selected compression algorithm. If lossy compression is used, the compression ratio is higher, but the data will be distorted; if lossless compression is used, the compression ratio is lower, but the degree of data restoration is higher. High, the corresponding compression algorithm can be selected according to the integrity requirements of different data.
在本申请的示例性实施例中，所述方法还可以包括：In an exemplary embodiment of the present application, the method may further include:
预先获取多种车辆数据的生成算法文本数据，作为训练数据；Acquire the text data of the generation algorithm of various vehicle data in advance as the training data;
采用所述训练数据对预先创建的用于存储数据的神经网络模型进行训练，获取AI数据存储模型。The training data is used to train the pre-created neural network model for storing data to obtain the AI data storage model.
在本申请的示例性实施例中，在实施本申请实施例方案之前，可以首先通过模型训练的方式获取该AI数据存储模型。In the exemplary embodiment of the present application, before implementing the solution of the embodiment of the present application, the AI data storage model may first be obtained through model training.
在本申请的示例性实施例中，可以预先创建神经网络模型，该神经网络模型用于存储数据，并基于该神经网络模型，收集大量训练数据(例如，收集车端获取的视频数据、图片数据、雷达数据、传感器数据等，并通过预设的生成算法生成这些视频数据、图片数据、雷达数据、传感器数据等，分别获取这些数据相对应的生成算法文本数据，将这些生成算法文本数据作为训练数据)，采用收集的训练数据对该神经网络模型进行训练，以使得该神经网络模型仅对于车辆数据对应的生成算法文本数据进行保存，并对于保存的每条生成算法文本数据设置相应的标识(例如、时间、编号、地址等)以便于后续进行数据查询；在对该神经网络模型进行多次训练后，便可以获取上述的AI数据存储模型，用于对车辆数据对应的生成算法文本数据进行识别和存储。In an exemplary embodiment of the present application, a neural network model can be created in advance, and the neural network model is used to store data, and based on the neural network model, a large amount of training data (for example, collecting video data and picture data obtained by the vehicle end) , radar data, sensor data, etc., and generate these video data, picture data, radar data, sensor data, etc. through the preset generation algorithm, respectively obtain the corresponding generation algorithm text data of these data, and use these generation algorithm text data as training data), using the collected training data to train the neural network model, so that the neural network model only saves the corresponding generation algorithm text data of the vehicle data, and sets a corresponding identification for each saved generation algorithm text data ( For example, time, number, address, etc.) for subsequent data query; after multiple trainings on the neural network model, the above-mentioned AI data storage model can be obtained, which is used to generate algorithm text data corresponding to vehicle data. identification and storage.
在本申请的示例性实施例中，在进行实时数据存储过程中，获取待存储的车辆数据所对应的生成算法文本数据以后，便可以将该生成算法文本数据输入训练好的AI数据存储模型，由该AI数据存储模型对该生成算法文本数据进行识别和存储，如果识别出当前数据不是生成算法文本数据，或者不是车辆数据对应的生成算法文本数据可以进行报错，或者直接忽略，不进行存储。In the exemplary embodiment of the present application, during the real-time data storage process, after obtaining the generation algorithm text data corresponding to the vehicle data to be stored, the generation algorithm text data can be input into the trained AI data storage model, The generated algorithm text data is identified and stored by the AI data storage model. If it is recognized that the current data is not the generated algorithm text data, or the generated algorithm text data corresponding to the vehicle data, an error can be reported, or it can be ignored directly and not stored.
在本申请的示例性实施例中，生成算法文本数据的存储占用量远远小于原生的车辆数据，可以极大减小存储资源占用量，很大程度上提升了存储资源的使用价值。另外，AI数据存储模型中的算法数据可通过数据治理模型进行管理，例如，可以包括但不限于数据清洗、数据脱敏、数据日常管理等。In the exemplary embodiment of the present application, the storage usage of the generated algorithm text data is much smaller than that of the original vehicle data, which can greatly reduce the usage of storage resources and greatly increase the use value of storage resources. In addition, the algorithm data in the AI data storage model can be managed through the data governance model, for example, it can include but not limited to data cleaning, data desensitization, daily data management, etc.
在本申请的示例性实施例中，如图3所示，在将所述生成算法文本数据存储入预设的人工智能AI数据存储模型中以后，所述方法还可以包括：In an exemplary embodiment of the present application, as shown in FIG. 3, after storing the generated algorithm text data into a preset artificial intelligence AI data storage model, the method may also include:
在需要查询所述车辆数据时，由预设的数据生成器根据查询请求、所述生成算法以及所述AI数据存储模型中存储的相应的生成算法文本数据生成所要查询的车辆数据。When the vehicle data needs to be queried, the preset data generator generates the vehicle data to be queried according to the query request, the generation algorithm and the corresponding generation algorithm text data stored in the AI data storage model.
在本申请的示例性实施例中，所述由预设的数据生成器根据查询请求、所述生成算法以及所述AI数据存储模型中存储的相应的生成算法文本数据生成所要查询的车辆数据，可以包括：In an exemplary embodiment of the present application, the preset data generator generates the vehicle data to be queried according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model, Can include:
由所述数据生成器根据所述查询请求发送查询指令到所述AI数据存储模型，并获取所述AI数据存储模型根据所述查询指令查询到的生成算法文本数据；The data generator sends a query instruction to the AI data storage model according to the query request, and obtains the generation algorithm text data queried by the AI data storage model according to the query instruction;
由所述数据生成器对查询到的所述生成算法文本数据进行解压，并根据解压后的生成算法文本数据，执行所述生成算法，生成所要查询的车辆数据，并将生成的所述车辆数据返回给查询所述辆数据的外部数据服务。The data generator decompresses the queried generation algorithm text data, executes the generation algorithm according to the decompressed generation algorithm text data, generates the vehicle data to be queried, and converts the generated vehicle data Returned to the external data service querying the vehicle data.
在本申请的示例性实施例中，根据查询指令，可以通过执行Dcgan算法或MAE算法(或其它同类算法)来生成与原生的车辆数据等值的数据，并可以根据算法参数对生成的数据进行验证。In an exemplary embodiment of the present application, according to the query instruction, data equivalent to the original vehicle data can be generated by executing the Dcgan algorithm or MAE algorithm (or other similar algorithms), and the generated data can be processed according to the algorithm parameters. verify.
在本申请的示例性实施例中，在由预设的数据生成器根据查询请求、所述生成算法以及所述AI数据存储模型中存储的相应的生成算法文本数据生成所要查询的车辆数据以后，所述方法还可以包括：In an exemplary embodiment of the present application, after the preset data generator generates the vehicle data to be queried according to the query request, the generation algorithm, and the corresponding generation algorithm text data stored in the AI data storage model, The method may also include:
通过预设的判别器对生成的所述车辆数据进行判别和矫正。The generated vehicle data is discriminated and corrected by a preset discriminator.
在本申请的示例性实施例中，可以通过判别器再次对生成的数据进行判别矫正，进一步提升与原生的车辆数据的一致性。In the exemplary embodiment of the present application, the generated data can be discriminated and corrected again by the discriminator, so as to further improve the consistency with the original vehicle data.
在本申请的示例性实施例中，所述判别器可以采用Dcgan算法体系中的样本数据模型，用于判别生成的数据与原生的车辆数据的一致性。In an exemplary embodiment of the present application, the discriminator may use a sample data model in the Dcgan algorithm system to determine the consistency between the generated data and the original vehicle data.
在本申请的示例性实施例中，至少包括以下优势：In an exemplary embodiment of the present application, at least the following advantages are included:
1、对存储空间的节约率非常高，不管是10MB的数据，还是100MB的数据，其生成算法文本数据的大小均在1MB左右，存储空间节约比例远远大于传统的存储方式，特别在智能驾驶这类数据量极大的非结构数据场景中。1. The saving rate of storage space is very high. Whether it is 10MB data or 100MB data, the size of the generated algorithm text data is about 1MB, and the storage space saving ratio is far greater than the traditional storage method, especially in intelligent driving This type of unstructured data scenario with a huge amount of data.
2、存储空间节省率不依赖或针对特定的文件类型，不管是普通文本数据，还是视频数据、图片数据、语音数据、雷达数据等，均可生成相应的生成算法文本数据。2. The storage space saving rate does not depend on or is specific to a specific file type. No matter it is ordinary text data, video data, picture data, voice data, radar data, etc., the corresponding generation algorithm text data can be generated.
本申请实施例还提供了一种数据存储装置1，如图4所示，可以包括处理器11和计算机可读存储介质12，所述计算机可读存储介质12中存储有指令，当所述指令被所述处理器11执行时，实现所述的数据存储方法。The embodiment of the present application also provides a data storage device 1, as shown in FIG. When executed by the processor 11, the data storage method is realized.
在本申请的示例性实施例中，前述的数据存储方法中的任意实施例均适用于该装置实施例中，在此不再一一赘述。In the exemplary embodiment of the present application, any embodiment of the aforementioned data storage method is applicable to the device embodiment, and will not be repeated here.
本领域普通技术人员可以理解，上文中所公开方法中的全部或某些步骤、系统、装置中的功能模块/单元可以被实施为软件、固件、硬件及其适当的组合。在硬件实施方式中，在以上描述中提及的功能模块/单元之间的划分不一定对应于物理组件的划分；例如，一个物理组件可以具有多个功能，或者一个功能或步骤可以由若干物理组件合作执行。某些组件或所有组件可以被实施为由处理器，如数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。这样的软件可以分布在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, the functional modules/units in the system, and the device can be implemented as software, firmware, hardware, and an appropriate combination thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical components. Components cooperate to execute. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310034670.4ACN116225323A (en) | 2023-01-10 | 2023-01-10 | A data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310034670.4ACN116225323A (en) | 2023-01-10 | 2023-01-10 | A data storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116225323Atrue CN116225323A (en) | 2023-06-06 |
Family
ID=86577869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310034670.4APendingCN116225323A (en) | 2023-01-10 | 2023-01-10 | A data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116225323A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004280156A (en)* | 2003-03-12 | 2004-10-07 | Mazda Motor Corp | Image processing method, simulation device, program, and recording medium |
| CN111340901A (en)* | 2020-02-19 | 2020-06-26 | 国网浙江省电力有限公司 | A Generative Adversarial Network-Based Compression Method for Transmission Network Pictures in Complex Environments |
| US20210048949A1 (en)* | 2019-08-12 | 2021-02-18 | Micron Technology, Inc. | Communications between processors and storage devices in automotive predictive maintenance implemented via artificial neural networks |
| CN115098597A (en)* | 2022-05-27 | 2022-09-23 | 武汉光庭信息技术股份有限公司 | GIS data generating track method, system, electronic equipment and storage medium |
- 2023
- 2023-01-10CNCN202310034670.4Apatent/CN116225323A/enactivePending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004280156A (en)* | 2003-03-12 | 2004-10-07 | Mazda Motor Corp | Image processing method, simulation device, program, and recording medium |
| US20210048949A1 (en)* | 2019-08-12 | 2021-02-18 | Micron Technology, Inc. | Communications between processors and storage devices in automotive predictive maintenance implemented via artificial neural networks |
| CN111340901A (en)* | 2020-02-19 | 2020-06-26 | 国网浙江省电力有限公司 | A Generative Adversarial Network-Based Compression Method for Transmission Network Pictures in Complex Environments |
| CN115098597A (en)* | 2022-05-27 | 2022-09-23 | 武汉光庭信息技术股份有限公司 | GIS data generating track method, system, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Johanson et al. | Big automotive data: Leveraging large volumes of data for knowledge-driven product development | |
| CN114528231B (en) | A method, device, electronic device and storage medium for dynamic data storage | |
| CN111639072A (en) | Data storage method and system in Internet of vehicles scene and readable storage medium | |
| CN105721883B (en) | Video sharing method and system based on information of tracing to the source in a kind of cloud storage system | |
| CN107409152A (en) | Method and apparatus for compressing data received over a network | |
| CN112306026A (en) | Equipment production line scheduling method based on smart park and cloud computing server | |
| CN103455555A (en) | Recommendation method and device based on mobile terminal similarity | |
| CN113391973B (en) | Internet of things cloud container log collection method and device | |
| CN113162960A (en) | Data processing method, device, equipment and medium | |
| CN111694766B (en) | A CAN data collection method and device | |
| CN115576973B (en) | Service deployment method, device, computer equipment and readable storage medium | |
| WO2020147700A1 (en) | Event subscription notification method, network side device, application entity, internet of things system and storage medium | |
| CN116562399A (en) | Model training method and device with end Bian Yun cooperated | |
| CN111211993B (en) | Incremental persistence method, device and storage medium for stream computation | |
| CN119105711A (en) | Method, system, electronic device and storage medium for storing multi-dimensional data of a fleet | |
| CN116225323A (en) | A data storage method and device | |
| CN117708212A (en) | Metadata acquisition method and device and electronic equipment | |
| WO2021092784A1 (en) | Data writing, consistency checking and reading method and apparatus | |
| CN108108183B (en) | Data management system and method based on model embedded software development | |
| CN115391355B (en) | Data processing method, device, equipment and storage medium | |
| CN111651438A (en) | MapDB-based structured data deduplication method, device, equipment and medium | |
| CN114640944B (en) | Vehicle-road cooperation multi-domain multi-source resource management method based on information availability | |
| CN104717254B (en) | The method and cloud gateway of file access | |
| CN103929404B (en) | Method for analyzing HTTP chunked code data | |
| CN116166709B (en) | Time length correction method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |