Disclosure of Invention
Aiming at the defects of the prior art, the application provides a distributed-based cross-network-segment file transmission method, which aims to solve the problems.
A transmission method of a cross-network section file based on distribution comprises the following steps:
The method comprises the steps that S1, file transmission service is arranged in a local area network, so that the service can be communicated with all node networks in the local area network, and the file transmission service network communication with other nodes is required to be ensured;
step S2, before the file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through file transmission service;
Step S3, when the file is required to be transmitted from one node to another node, determining a transmission path according to whether the local area network contains a target node or not;
s4, after the transmission path is acquired, encrypting the file fragments according to the path information, and sending the file fragments to a next transmission node;
s5, the receiving node decrypts the data packet according to the source node information and continues to transmit according to the route information until reaching the destination;
step S6, after receiving all the fragment data, the receiving party combines the fragment data into a complete file and returns confirmation information of successful receiving to the sending party;
and S7, if the sender does not receive the receiving confirmation for a long time, retransmitting the unacknowledged fragmented data packet.
By adopting the technical scheme, file transmission service is arranged in a local area network, the service can be communicated with all node networks in the local area network, meanwhile, the communication with file transmission service networks of other nodes is ensured, before file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through the file transmission service, complex network configuration required by cross-network segment file transmission is simplified, operation and maintenance cost is reduced, the safety and the trust degree of a system are enhanced, when the file is required to be transmitted from one node to another node, an optimal transmission path is determined according to whether a target node is contained in the local area network, then the file is subjected to fragmentation processing and encryption, and then is sent to the next transmission node, and a receiving node decrypts a data packet according to source node information and continues to transmit according to route information until a destination is reached. And after receiving all the fragmented data, the receiver combines the fragmented data into a complete file, returns acknowledgement information of successful reception to the sender, and resends unacknowledged fragmented data packets if the acknowledgement is not received for a long time. The process not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of a system, supports large file transmission, simplifies error processing, optimizes resource utilization, and provides a solid foundation for realizing efficient, safe and easily-managed large-scale data transmission.
Optionally, the file transmission service includes a file transmission module, an encryption module, a message queue module and a transmission node management module, where the file transmission module is configured to perform fragmentation processing on a file and write the file into the message queue module, the encryption module performs encryption processing on content of the file transmission, the message queue module is configured to serve as a hub for content transmission of the fragmented file, and the transmission node management module is configured to store a mutually authenticated node list, and obtain a transmission path before file transmission.
By adopting the technical scheme, the file transmission module is responsible for carrying out slicing processing on the file and writing the file into the message queue module, so that the transmission efficiency is improved, the memory occupation is reduced, the file is transmitted while being read, the encryption component carries out encryption processing on the file content, the confidentiality and the integrity of the data in the transmission process are ensured, unauthorized access and data tampering are prevented, the message queue module is used as a hub for slicing file content transmission, effectively manages and schedules the data slicing to be transmitted, optimizes the system throughput, supports the breakpoint continuous transmission function, improves the reliability and the stability of the system, the transmission node management module stores mutually authenticated node lists, acquires the optimal path before file transmission, simplifies the complex network configuration required by cross-network-section file transmission, and reduces the operation and maintenance cost.
Optionally, the file transfer module supports registering Minio, FTP, SFTP types of file data sources.
Optionally, the specific steps for encrypting the file in step S4 are as follows:
s4.1, registering each other between transmission nodes, and storing a public key and a private key of the other party and an access mode of a message queue module;
S4.2, when the file transmission module needs to send the file, encrypting the file fragments by using the public key of the receiver;
s4.3, writing the encrypted file fragments into a message queue module and sending the file fragments to a next transmission node;
And S4.4, the receiving node reads the data packet from the message queue module and decrypts the data packet by using the corresponding private key according to the source node information.
By adopting the technical scheme, the security of the file content is enhanced, the fact that even if data is intercepted, an unauthorized third party cannot read the file content is ensured, meanwhile, a verification means of data integrity is provided, potential data damage or tampering can be detected, in addition, the mechanism realizes security authentication among nodes, reduces the risk of unauthorized access, simplifies key management, keeps the encryption state all the time on the whole transmission link until reaching a final destination, thereby ensuring the end-to-end security, the encryption method also supports a breakpoint continuous transmission function, and can continuously and safely transmit the file based on the existing key without renegotiating the key even after the network is interrupted.
Optionally, the access manner of the message queue module in step S4.1 includes IP, port, user name and password.
The size of the fragments is defined according to the transmitted file size, specifically, the file with the file size smaller than 10MB is set as a small file, the fragment size is set as 512KB-1MB, the file with the file size of 10MB-1GB is set as a medium file, the fragment size is set as 4MB-16MB, the file with the file size larger than 1MB is set as a large file, and the fragment size is set as 16MB-32MB.
By adopting the technical scheme, the method can optimize the network bandwidth utilization, improve the transmission efficiency and enhance the fault tolerance, small fragments can rapidly finish transmission for small files, and large fragments reduce the total fragments and speed up the whole transmission speed for large files, and in addition, when transmission errors or network interruption occur, only specific fragments which are not successfully transmitted are required to be retransmitted instead of the whole files, so that the recovery process is accelerated. The strategy also simplifies the file transfer management flow, facilitates configuration and maintenance, and improves user experience, particularly in applications where large amounts of data are handled or time sensitive. In summary, adjusting the size of the fragments based on the file size not only improves the efficiency and reliability of file transmission, but also enhances the flexibility and user satisfaction of the system.
Optionally, when file data is read from the file data source, the specified file is read by a client provided by the file data source, converted into a file stream, and converted into a byte array when read in slices.
By adopting the technical scheme, the efficiency and reliability of file transmission are obviously improved, the method supports transmission while reading, reduces memory occupation, enables a system to process files with any size without causing memory overflow, and simultaneously enhances the fault tolerance and recovery efficiency of the system, because only specific fragments which are not successfully transmitted but the whole file are required to be retransmitted even when network interruption or error occurs. In addition, the network bandwidth utilization is optimized by dynamically adjusting the sizes of the fragments, the problem of congestion or packet loss caused by large data packets is reduced, each fragment is treated as an independent data unit, an error detection and correction mechanism is simplified, the problem of single fragment is solved, finally, clients (such as Minio SDK, FTPCLIENT of Apache Commons Net and the like) provided by file data sources are used for accessing different types of storage services, support of various file protocols is ensured, and the flexibility and compatibility of the scheme are improved.
Optionally, the content transmitted to the message queue module by the file transmission module is a byte stream, and the byte stream includes a route length section, a route content section, a source node section, a fragment sequence number section, a file length section, a file content section and a file hash value section.
By adopting the technical scheme, when the file transmission assembly reads the data packet, the length of the route content is acquired through the route length section, the route content section is read according to the length, then the source node section is read, the source node number is acquired, then the fragment sequence number section is read, the index of the file fragment is acquired, then the file length section is read, the file content section is read according to the file length, finally the file hash value section is read, and whether the file is complete after the file is merged is judged.
Optionally, the file transmission module monitors the data in the message queue module, and once a new file fragment is detected, the file transmission module immediately consumes and parses the file and transmits the file fragment to the next node.
By adopting the technical scheme, the file transmission component monitors the data in the message queue module, and immediately consumes, analyzes and transmits the data to the next node when detecting the new file fragments, and the mechanism remarkably improves the efficiency and the reliability of file transmission. By processing newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, and the speed of the whole file transmission process can be accelerated. Meanwhile, the instant processing mode is beneficial to quickly finding and responding to possible transmission errors or failures, and supports a breakpoint continuous transmission function, namely, after the network is interrupted, the transmission can be continued from the last successfully transmitted fragment without retransmitting the whole file, so that the success rate of transmitting the large file across network segments is improved. In addition, dividing the file into multiple small blocks for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies error handling logic so that most of the data is guaranteed to be properly transmitted even in the case of poor network conditions. Asynchronous operation is supported based on the design of the message queue module, the file transmission service and other services are allowed to work in parallel, and the concurrent processing capacity of the system is improved.
In summary, the present application includes at least one of the following beneficial technical effects:
1. The file transmission service is arranged in a local area network, the service can be communicated with all node networks in the local area network, meanwhile, the file transmission service network communication with other nodes is ensured, before file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through the file transmission service, complex network configuration required by cross-network segment file transmission is simplified, operation and maintenance cost is reduced, safety and trust degree of a system are enhanced, when the file is required to be transmitted from one node to another node, an optimal transmission path is determined according to whether a target node is contained in the local area network, then the file is subjected to fragmentation processing and encryption, and then is sent to the next transmission node, and a receiving node decrypts a data packet according to source node information and continues transmission according to route information until a destination is reached. And after receiving all the fragmented data, the receiver combines the fragmented data into a complete file, returns acknowledgement information of successful reception to the sender, and resends unacknowledged fragmented data packets if the acknowledgement is not received for a long time. The process not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of a system, supports large file transmission, simplifies error processing, optimizes resource utilization, and provides a solid foundation for realizing efficient, safe and easily-managed large-scale data transmission.
2. The process enhances the security of the file content, and ensures that even if the data is intercepted, an unauthorized third party cannot read the file content; in addition, the mechanism realizes the security authentication among nodes, reduces the risk of unauthorized access, simplifies key management, keeps the encryption state all the time on the whole transmission link until reaching the final destination, thereby ensuring the end-to-end security, and the encryption method also supports the breakpoint continuous transmission function, can continuously and safely transmit the file based on the existing key without renegotiating the key even after the network is interrupted.
3. The method can optimize network bandwidth utilization, improve transmission efficiency and enhance fault tolerance, small fragments can rapidly finish transmission for small files, and large fragments reduce total fragments and speed up overall transmission speed for large files, and in addition, when transmission errors or network interruption occur, only specific fragments which are not successfully transmitted but not the whole files are retransmitted, so that recovery process is accelerated. The strategy also simplifies the file transfer management flow, facilitates configuration and maintenance, and improves user experience, particularly in applications where large amounts of data are handled or time sensitive. In summary, adjusting the size of the fragments based on the file size not only improves the efficiency and reliability of file transmission, but also enhances the flexibility and user satisfaction of the system.
4. The method supports transmission while reading, reduces memory occupation, enables a system to process files with any size without causing memory overflow, and enhances fault tolerance and recovery efficiency of the system because only specific fragments which are not successfully delivered but the whole file are required to be retransmitted even when network interruption or error occurs. In addition, the network bandwidth utilization is optimized by dynamically adjusting the sizes of the fragments, the problem of congestion or packet loss caused by large data packets is reduced, each fragment is treated as an independent data unit, an error detection and correction mechanism is simplified, the problem of single fragment is solved, finally, clients (such as Minio SDK, FTPCLIENT of Apache Commons Net and the like) provided by file data sources are used for accessing different types of storage services, support of various file protocols is ensured, and the flexibility and compatibility of the scheme are improved.
5. The file transmission component monitors the data in the message queue module, and immediately consumes, analyzes and transmits the data to the next node when detecting the new file fragments, and the mechanism remarkably improves the efficiency and the reliability of file transmission. By processing newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, and the speed of the whole file transmission process can be accelerated. Meanwhile, the instant processing mode is beneficial to quickly finding and responding to possible transmission errors or failures, and supports a breakpoint continuous transmission function, namely, after the network is interrupted, the transmission can be continued from the last successfully transmitted fragment without retransmitting the whole file, so that the success rate of transmitting the large file across network segments is improved. In addition, the file is divided into a plurality of small blocks to process, so that the use of network bandwidth and other computing resources is optimized, error processing logic is simplified, most of data can be guaranteed to be correctly transmitted even under the condition of poor network conditions, asynchronous operation is supported based on the design of a message queue module, file transmission service and other services are allowed to work in parallel, and the concurrent processing capacity of a system is improved.
Detailed Description
In order that the application may be readily understood, a more particular description thereof will be rendered by reference to specific embodiments that are illustrated in the appended drawings. It will be understood that when an element is referred to as being "fixed" to another element, it can be directly on the other element or one or more intervening elements may be present therebetween. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or one or more intervening elements may be present therebetween. The terms "vertical," "horizontal," "left," "right," and the like are used herein for illustrative purposes only.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and/or" as used in this specification includes any and all combinations of one or more of the associated listed items.
The embodiment of the application discloses a transmission method of a cross-network-segment file based on distribution. Referring to fig. 1 and 2, the method comprises the steps of S1, arranging file transmission service in a local area network, ensuring that the service can communicate with all node networks in the local area network and needs to ensure that the service communicates with file transmission service networks of other nodes, S2, registering transmission nodes among the local area networks through the file transmission service before file transmission authentication configuration is carried out, S3, determining a transmission path according to whether a target node is contained in the local area network when the file needs to be transmitted from one node to the other node, S4, after the transmission path is acquired, encrypting the file fragments according to path information and sending the file fragments to the next transmission node, S5, decrypting a data packet according to source node information and continuing to transmit the data packet according to route information until reaching a destination, S6, combining the receiving party into a complete file and returning acknowledgement information of successful reception to a sender, and S7, if the sender does not receive acknowledgement for a long time, retransmitting the unacknowledged fragment data packet. The file transmission service is arranged in a local area network, the service can be communicated with all node networks in the local area network, meanwhile, the file transmission service network communication with other nodes is ensured, before file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through the file transmission service, complex network configuration required by cross-network segment file transmission is simplified, operation and maintenance cost is reduced, safety and trust degree of a system are enhanced, when the file is required to be transmitted from one node to another node, an optimal transmission path is determined according to whether a target node is contained in the local area network, then the file is subjected to fragmentation processing and encryption, and then is sent to the next transmission node, and a receiving node decrypts a data packet according to source node information and continues transmission according to route information until a destination is reached. And after receiving all the fragmented data, the receiver combines the fragmented data into a complete file, returns acknowledgement information of successful reception to the sender, and resends unacknowledged fragmented data packets if the acknowledgement is not received for a long time. The process not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of a system, supports large file transmission, simplifies error processing, optimizes resource utilization, and provides a solid foundation for realizing efficient, safe and easily-managed large-scale data transmission.
Referring to fig. 1 and 2, the file transmission service includes a file transmission module, an encryption module, a message queue module, and a transmission node management module, where the file transmission module is configured to perform a slicing process on a registrable file data source and write the processed file data source into the message queue module, and the file transmission module supports registering Minio, FTP, SFTP types of file data sources. The encryption component encrypts the file transmission content, the message queue module is used as a hub for transmitting the fragmented file content, and the transmission node management module is used for storing a mutually authenticated node list and acquiring a transmission path before file transmission. The file transmission module is responsible for carrying out slicing processing on the file and writing the file into the message queue module, which not only improves the transmission efficiency, but also reduces the memory occupation, supports the transmission while reading, the encryption component carries out encryption processing on the file content, ensures the confidentiality and the integrity of the data in the transmission process, prevents unauthorized access and data tampering, the message queue module is used as a hub for slicing file content transmission, effectively manages and schedules the data slicing to be transmitted, optimizes the system throughput, supports the breakpoint continuous transmission function, improves the reliability and the stability of the system, and the transmission node management module stores mutually authenticated node lists, acquires the optimal path before file transmission, simplifies the complex network configuration required by cross-network file transmission, and reduces the operation and maintenance cost.
Referring to fig. 1 and 2, the file transfer module listens to the data in the message queue module, and once a new file fragment is detected, immediately consumes, parses, and transfers to the next node. The file transmission component monitors the data in the message queue module, and immediately consumes, analyzes and transmits the data to the next node when detecting the new file fragments, and the mechanism remarkably improves the efficiency and the reliability of file transmission. By processing newly arrived data in real time, the waiting time of the data in the message queue module can be reduced, and the speed of the whole file transmission process can be accelerated. Meanwhile, the instant processing mode is beneficial to quickly finding and responding to possible transmission errors or failures, and supports a breakpoint continuous transmission function, namely, after the network is interrupted, the transmission can be continued from the last successfully transmitted fragment without retransmitting the whole file, so that the success rate of transmitting the large file across network segments is improved. In addition, dividing the file into multiple small blocks for processing not only optimizes the use of network bandwidth and other computing resources, but also simplifies error handling logic so that most of the data is guaranteed to be properly transmitted even in the case of poor network conditions. Asynchronous operation is supported based on the design of the message queue module, the file transmission service and other services are allowed to work in parallel, and the concurrent processing capacity of the system is improved.
Referring to fig. 1 and 2, the specific steps of encrypting the file in step S4 include the steps of registering each other between the transmission nodes, storing the public key and the private key of the other party and the access mode of the message queue module, the step S4.2 of encrypting the file fragments by using the public key of the receiving party when the file transmission module needs to transmit the file, the step S4.3 of writing the encrypted file fragments into the message queue module and transmitting the encrypted file fragments to the next transmission node, and the step S4.4 of reading the data packet from the message queue module by the receiving node and decrypting the data packet by using the corresponding private key according to the source node information. Before the file transmission authentication configuration is carried out, transmission node registration is needed to be carried out among all local area networks through a file transmission module.
Referring to fig. 1 and 2, a file transfer component a needs to initiate an authentication request to a file transfer component B, the file transfer component a generates a public key a and a private key a, the file transfer component B generates a public key B and a private key B, the file transfer component a stores the private key a and the public key B, the file transfer component B stores the public key a and the private key B, and stores connection information of a transmitting node of an opposite party and an access mode (IP, port, user name, password) of a message queue module. The document delivery component B and the document delivery component C, the document delivery component B and the document delivery component D, and the document delivery component C and the document delivery component D also need to do this authentication. The public and private key pairs are generated through an RSA algorithm, encryption is carried out through the public key in the subsequent data transmission process, and the receiver decrypts through the private key. The process enhances the security of the file content, and ensures that even if the data is intercepted, an unauthorized third party cannot read the file content; in addition, the mechanism realizes the security authentication among nodes, reduces the risk of unauthorized access, simplifies key management, keeps the encryption state all the time on the whole transmission link until reaching the final destination, thereby ensuring the end-to-end security, and further supports the breakpoint continuous transmission function, and can continuously and safely transmit the file based on the existing key without renegotiating the key even after the network is interrupted.
Referring to fig. 1 and fig. 2, when a file on a node a needs to be transmitted to a node F, the file transmission service first confirms whether the local area network includes the node F, if so, the return transmission path is [ node a- > -file transmission component a- > -node F ], if the local area network does not include the node F, a request is sent to a node device that is only registered (in this example, a file transmission component B is registered, an acquisition path request is sent to the file transmission component), and so on, the file transmission component B returns to the path 1 [ file transmission component B- > -file transmission component C- > -node F ] and the path 2 [ file transmission component B- > -file transmission component C- > -node F ], and according to the length of the transmission path, the finally selected transmission path is [ node a- > -file transmission component B- > -file transmission component C- > -node F ].
Referring to fig. 1,2 and 3, after a transmission path is acquired, the file transmission component a reads the file on the node a in slices, the sizes of the slices are defined according to the size of the transmitted file, the file with the file size smaller than 10MB is set as a small file, the size of the slices is set as 512KB-1MB, the file with the file size of 10MB-1GB is set as a medium file, the size of the slices is set as 4MB-16MB, the file with the file size larger than 1GB is set as a large file, and the size of the slices is set as 16MB-32MB. When file data is read from a file data source, a specified file is read, converted to a file stream, and converted to a byte array at the time of fragment reading, by a client (e.g., FTPCLIENT of Minio SDK, apache common net) provided by the file data source. The method can optimize network bandwidth utilization, improve transmission efficiency and enhance fault tolerance, small fragments can rapidly finish transmission for small files, and large fragments reduce total fragments and speed up overall transmission speed for large files, and in addition, when transmission errors or network interruption occur, only specific fragments which are not successfully transmitted but not the whole files are retransmitted, so that recovery process is accelerated. The strategy also simplifies the file transfer management flow, facilitates configuration and maintenance, and improves user experience, particularly in applications where large amounts of data are handled or time sensitive. The method supports transmission while reading, reduces memory occupation, enables a system to process files with any size without causing memory overflow, and enhances fault tolerance and recovery efficiency of the system because only specific fragments which are not successfully delivered but the whole file are required to be retransmitted even when network interruption or error occurs. In addition, the network bandwidth utilization is optimized by dynamically adjusting the sizes of the fragments, the problem of congestion or packet loss caused by large data packets is reduced, each fragment is treated as an independent data unit, an error detection and correction mechanism is simplified, the problem of single fragment is solved, finally, clients (such as Minio SDK, FTPCLIENT of Apache Commons Net and the like) provided by file data sources are used for accessing different types of storage services, support of various file protocols is ensured, and the flexibility and compatibility of the scheme are improved.
Referring to fig. 1, 2 and 3, the contents transmitted to the message queue module by the file transmission module are byte streams, and the byte streams include a route length section, a route content section, a source node section, a fragment sequence number section, a file length section, a file content section and a file hash value section. When the file transmission component reads the data packet, the length of the route content is obtained through the route length section, the route content section is read according to the length, then the source node section is read, the source node number is obtained, then the fragment sequence number section is read, the index of the file fragment is obtained, then the file length section is read, the file content section is read according to the file length, finally the file hash value section is read, and whether the file is complete after the file is merged is judged. For example, when the file transmission component reads a data packet, the first 8 bits of the byte stream are acquired, the length of the routing content is acquired, the routing content is read according to the length, then the data of 32 bytes after reading is used for indicating the source node number, the next 8 bits are used for indicating the index of the file fragment, the next 8 bits are used for indicating the file length, the file content is acquired according to the file length, and the last 256 bits are used for indicating the hash value of the file, and the hash value is used for judging whether the file is complete after the file is merged.
The implementation principle of the distributed cross-network file transmission method is that file transmission service is arranged in a local area network, the service can be communicated with all node networks in the local area network, meanwhile, the file transmission service network communication with other nodes is guaranteed, before file transmission authentication configuration is carried out, transmission node registration is carried out among all local area networks through the file transmission service, complex network configuration required by cross-network file transmission is simplified, operation and maintenance cost is reduced, safety and trust degree of a system are enhanced, when a file is required to be transmitted from one node to another node, an optimal transmission path is determined according to whether a target node is contained in the local area network, then the file is subjected to fragmentation processing and encryption, and then the file is sent to the next transmission node, and a receiving node decrypts a data packet according to source node information and continues transmission according to route information until a destination is reached. And after receiving all the fragmented data, the receiver combines the fragmented data into a complete file, returns acknowledgement information of successful reception to the sender, and resends unacknowledged fragmented data packets if the acknowledgement is not received for a long time. The process not only improves the efficiency and the safety of file transmission, but also enhances the reliability and the stability of a system, supports large file transmission, simplifies error processing, optimizes resource utilization, and provides a solid foundation for realizing efficient, safe and easily-managed large-scale data transmission.
The above embodiments are not intended to limit the scope of the application, so that the equivalent changes of the structure, shape and principle of the application are covered by the scope of the application.