This article is a tutorial on transferring a large file using Java Nio. It will take shape via two examples demonstrating a simple local file transfer from one location on hard disk to another and then via sockets from one remote location to another remote location.

1. Introduction

This tutorial will make use of theFileChannel abstraction for both remote and local copy. Augmenting the remote copy process will be a simple set of abstractions (ServerSocketChannel &SocketChannel) that facilitate the transfer of bytes over the wire. Finally we wrap things up with an asynchronous implementation of large file transfer. The tutorial will be driven by unit tests that can run from command line using maven or from within your IDE.

2. Technologies used

The example code in this article was built and run using:

• Java 1.8.101 (1.8.x will do fine)
• Maven 3.3.9 (3.3.x will do fine)
• Spring source tool suite 4.6.3 (Any Java IDE would work)
• Ubuntu 16.04 (Windows, Mac or Linux will do fine)

2. FileChannel

AFileChannel is a type ofChannel used for writing, reading, mapping and manipulating aFile. In addition to the familiarChannel (read, write and close) operations, thisChannel has a few specific operations:

• Has the concept of an absolute position in theFile which does not affect theChannels current position.
• Parts or regions of aFile can be mapped directly into memory and work from memory, very useful when dealing with large files.
• Writes can be forced to the underlying storage device, ensuring write persistence.
• Bytes can be transferred from oneReadableByteChannel /WritableByteChannel instance to another ReadableByteChannel /WritableByteChannel, whichFileChannel implements. This yields tremendous IO performance advantages that some Operating systems are optimized for.
• A part or region of aFile may be locked by a process to guard against access by other processes.

FileChannels are thread safe. Only one IO operation that involves theFileChannels position can be in flight at any given point in time, blocking others. The view or snapshot of aFile via aFileChannel is consistent with other views of the sameFile within the same process. However, the same cannot be said for other processes. A file channel can be created in the following ways:

• …FileChannel.open(...)
• … FileInputStream(...).getChannel()
• …FileOutputStream(...).getChannel()
• …RandomAccessFile(...).getChannel()

Using one of the stream interfaces to obtain aFileChannel will yield aChannel that allows either read, write or append privileges and this is directly attributed to the type of Stream (FileInputStream orFileOutputStream) that was used to get theChannel. Append mode is a configuration artifact of aFileOutputStream constructor.

4. Background

The sample program for this example will demonstrate the following:

1. Local transfer of a file (same machine)
2. Remote transfer of a file (potentially remote different processes, although in the unit tests we spin up different threads for client and server)
3. Remote transfer of a file asynchronously

Particularly with large files the advantages of asynchronous non blocking handling of file transfer cannot be stressed enough. Large files tying up connection handling threads soon starve a server of resources to handle additional requests possibly for more large file transfers.

5. Program

The code sample can be split into local and remote domains and within remote we further specialize an asynchronous implementation of file transfer, at least on the receipt side which is arguably the more interesting part.

5.1. Local copy

FileCopy

final class FileCopy    private FileCop() {        throw new IllegalStateException(Constants.INSTANTIATION_NOT_ALLOWED);    }    public static void copy(final String src, final String target) throws IOException {        if (StringUtils.isEmpty(src) || StringUtils.isEmpty(target)) {            throw new IllegalArgumentException("src and target required");        }        final String fileName = getFileName(src);        try (FileChannel from = (FileChannel.open(Paths.get(src), StandardOpenOption.READ));                FileChannel to = (FileChannel.open(Paths.get(target + "/" + fileName), StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE))) {            transfer(from, to, 0l, from.size());        }    }    private static String getFileName(final String src) {        assert StringUtils.isNotEmpty(src);        final File file = new File(src);        if (file.isFile()) {            return file.getName();        } else {            throw new RuntimeException("src is not a valid file");        }    }    private static void transfer(final FileChannel from, final FileChannel to, long position, long size) throws IOException {        assert !Objects.isNull(from) && !Objects.isNull(to);        while (position < size) {            position += from.transferTo(position, Constants.TRANSFER_MAX_SIZE, to);        }    }}

• line 14: weopen thefrom Channel with theStandardOpenOption.READ meaning that thisChannel will only be read from. The path is provided.
• line 15: theto Channel is opened with the intention to write and create, the path is provided.
• line 31-37: the twoChannels are provided (from & to) along with theposition (initially where to start reading from) and thesize indicating the amount of bytes to transfer in total. A loop is started where attempts are made to transfer up toConstants.TRANSFER_MAX_SIZE  in bytes from thefrom Channel to theto Channel. After each iteration the amount of bytes transferred is added to theposition which then advances the cursor for the next transfer attempt.

5.2. Remote copy

FileReader

final class FileReader {    private final FileChannel channel;    private final FileSender sender;    FileReader(final FileSender sender, final String path) throws IOException {        if (Objects.isNull(sender) || StringUtils.isEmpty(path)) {            throw new IllegalArgumentException("sender and path required");        }        this.sender = sender;        this.channel = FileChannel.open(Paths.get(path), StandardOpenOption.READ);    }    void read() throws IOException {        try {            transfer();        } finally {            close();        }    }    void close() throws IOException {        this.sender.close();        this.channel.close();    }    private void transfer() throws IOException {        this.sender.transfer(this.channel, 0l, this.channel.size());    }}

• line 12: theFileChannel is opened with the intent to readStandardOpenOption.READ, thepath is provided to the File.
• line 15-21: we ensure we transfer the contents of theFileChannel entirely and the close theChannel.
• line 23-26: we close thesender resources and then close theFileChannel
• line 29: we calltransfer(...) on thesender to transfer all the bytes from theFileChannel

FileSender

final class FileSender {    private final InetSocketAddress hostAddress;    private SocketChannel client;    FileSender(final int port) throws IOException {        this.hostAddress = new InetSocketAddress(port);        this.client = SocketChannel.open(this.hostAddress);    }    void transfer(final FileChannel channel, long position, long size) throws IOException {        assert !Objects.isNull(channel);        while (position < size) {            position += channel.transferTo(position, Constants.TRANSFER_MAX_SIZE, this.client);        }    }        SocketChannel getChannel() {        return this.client;    }    void close() throws IOException {        this.client.close();    }}

line 11-17: we provide theFileChannel,position andsize of the bytes to transfer from the givenchannel. A loop is started where attempts are made to transfer up toConstants.TRANSFER_MAX_SIZE  in bytes from the providedChannel to theSocketChannelclient. After each iteration the amount of bytes transferred is added to theposition which then advances the cursor for the next transfer attempt.

FileReceiver

final class FileReceiver {    private final int port;    private final FileWriter fileWriter;    private final long size;    FileReceiver(final int port, final FileWriter fileWriter, final long size) {        this.port = port;        this.fileWriter = fileWriter;        this.size = size;    }    void receive() throws IOException {        SocketChannel channel = null;        try (final ServerSocketChannel serverSocketChannel = ServerSocketChannel.open()) {            init(serverSocketChannel);            channel = serverSocketChannel.accept();            doTransfer(channel);        } finally {            if (!Objects.isNull(channel)) {                channel.close();            }            this.fileWriter.close();        }    }    private void doTransfer(final SocketChannel channel) throws IOException {        assert !Objects.isNull(channel);        this.fileWriter.transfer(channel, this.size);    }    private void init(final ServerSocketChannel serverSocketChannel) throws IOException {        assert !Objects.isNull(serverSocketChannel);        serverSocketChannel.bind(new InetSocketAddress(this.port));    }}

TheFileReceiver is a mini server that listens for incoming connections on thelocalhost and upon connection, accepts it and initiates a transfer of bytes from the acceptedChannel via theFileWriter abstraction to the encapsulatedFileChannel within theFileWriter. TheFileReceiver is only responsible for receiving the bytes via socket and then delegates transferring them to theFileWriter.

FileWriter

final class FileWriter {    private final FileChannel channel;    FileWriter(final String path) throws IOException {        if (StringUtils.isEmpty(path)) {            throw new IllegalArgumentException("path required");        }        this.channel = FileChannel.open(Paths.get(path), StandardOpenOption.WRITE, StandardOpenOption.CREATE_NEW);    }    void transfer(final SocketChannel channel, final long bytes) throws IOException {        assert !Objects.isNull(channel);        long position = 0l;        while (position < bytes) {            position += this.channel.transferFrom(channel, position, Constants.TRANSFER_MAX_SIZE);        }    }        int write(final ByteBuffer buffer, long position) throws IOException {        assert !Objects.isNull(buffer);        int bytesWritten = 0;                while(buffer.hasRemaining()) {            bytesWritten += this.channel.write(buffer, position + bytesWritten);                    }                return bytesWritten;    }    void close() throws IOException {        this.channel.close();    }}

TheFileWriter is simply charged with transferring the bytes from aSocketChannel to it’s encapsulatedFileChannel. As before, the transfer process is a loop which attempts to transfer up toConstants.TRANSFER_MAX_SIZE bytes with each iteration.

5.2.1. Asynchronous large file transfer

The following code snippets demonstrate transferring a large file from one remote location to another via an asynchronous receiverFileReceiverAsync.

OnComplete

Want to be a Java NIO Master ?

Subscribe to our newsletter and download the JDBCUltimateGuideright now!

In order to help you master Java NIO Library, we have compiled a kick-ass guide with all the major Java NIO features and use cases! Besides studying them online you may download the eBook in PDF format!

Download NOW!

Thank you!

We will contact you soon.

@FunctionalInterfacepublic interface OnComplete {    void onComplete(FileWriterProxy fileWriter);}

TheOnComplete interface represents a callback abstraction that we pass to ourFileReceiverAsync implementation with the purposes of executing this once a file has been successfully and thoroughly transferred. We pass aFileWriterProxy to theonComplete(...) and this can server as context when executing said method.

FileWriterProxy

final class FileWriterProxy {    private final FileWriter fileWriter;    private final AtomicLong position;    private final long size;    private final String fileName;    FileWriterProxy(final String path, final FileMetaData metaData) throws IOException {        assert !Objects.isNull(metaData) && StringUtils.isNotEmpty(path);        this.fileWriter = new FileWriter(path + "/" + metaData.getFileName());        this.position = new AtomicLong(0l);        this.size = metaData.getSize();        this.fileName = metaData.getFileName();    }        String getFileName() {        return this.fileName;    }    FileWriter getFileWriter() {        return this.fileWriter;    }        AtomicLong getPosition() {        return this.position;    }        boolean done() {        return this.position.get() == this.size;    }}

TheFileWriterProxy represents a proxy abstraction that wraps aFileWriter and encapsulatesFileMetaData. All of this is needed when determining what to name the file, where to write the file and what the file size is so that we know when the file transfer is complete. During transfer negotiation this meta information is compiled via a custom protocol we implement before actual file transfer takes place.

FileReceiverAsync

final class FileReceiverAsync {    private final AsynchronousServerSocketChannel server;    private final AsynchronousChannelGroup group;    private final String path;    private final OnComplete onFileComplete;    FileReceiverAsync(final int port, final int poolSize, final String path, final OnComplete onFileComplete) {        assert !Objects.isNull(path);        this.path = path;        this.onFileComplete = onFileComplete;                try {            this.group = AsynchronousChannelGroup.withThreadPool(Executors.newFixedThreadPool(poolSize));            this.server = AsynchronousServerSocketChannel.open(this.group).bind(new InetSocketAddress(port));        } catch (IOException e) {            throw new IllegalStateException("unable to start FileReceiver", e);        }    }    void start() {        accept();    }    void stop(long wait) {                try {                        this.group.shutdown();                        this.group.awaitTermination(wait, TimeUnit.MILLISECONDS);        } catch (InterruptedException e) {            throw new RuntimeException("unable to stop FileReceiver", e);        }    }    private void read(final AsynchronousSocketChannel channel, final FileWriterProxy proxy) {        assert !Objects.isNull(channel) && !Objects.isNull(proxy);        final ByteBuffer buffer = ByteBuffer.allocate(Constants.BUFFER_SIZE);        channel.read(buffer, proxy, new CompletionHandler<Integer, FileWriterProxy>() {            @Override            public void completed(final Integer result, final FileWriterProxy attachment) {                if (result >= 0) {                    if (result > 0) {                        writeToFile(channel, buffer, attachment);                    }                                        buffer.clear();                    channel.read(buffer, attachment, this);                } else if (result < 0 || attachment.done()) {                    onComplete(attachment);                    close(channel, attachment);                }            }            @Override            public void failed(final Throwable exc, final FileWriterProxy attachment) {                throw new RuntimeException("unable to read data", exc);            }        });    }        private void onComplete(final FileWriterProxy proxy) {        assert !Objects.isNull(proxy);                this.onFileComplete.onComplete(proxy);    }    private void meta(final AsynchronousSocketChannel channel) {        assert !Objects.isNull(channel);        final ByteBuffer buffer = ByteBuffer.allocate(Constants.BUFFER_SIZE);        channel.read(buffer, new StringBuffer(), new CompletionHandler<Integer, StringBuffer>() {            @Override            public void completed(final Integer result, final StringBuffer attachment) {                if (result < 0) {                    close(channel, null);                } else {                                        if (result > 0) {                        attachment.append(new String(buffer.array()).trim());                    }                    if (attachment.toString().contains(Constants.END_MESSAGE_MARKER)) {                        final FileMetaData metaData = FileMetaData.from(attachment.toString());                        FileWriterProxy fileWriterProxy;                        try {                            fileWriterProxy = new FileWriterProxy(FileReceiverAsync.this.path, metaData);                            confirm(channel, fileWriterProxy);                        } catch (IOException e) {                            close(channel, null);                            throw new RuntimeException("unable to create file writer proxy", e);                        }                    } else {                        buffer.clear();                        channel.read(buffer, attachment, this);                    }                }            }            @Override            public void failed(final Throwable exc, final StringBuffer attachment) {                close(channel, null);                throw new RuntimeException("unable to read meta data", exc);            }        });    }    private void confirm(final AsynchronousSocketChannel channel, final FileWriterProxy proxy) {        assert !Objects.isNull(channel) && !Objects.isNull(proxy);        final ByteBuffer buffer = ByteBuffer.wrap(Constants.CONFIRMATION.getBytes());        channel.write(buffer, null, new CompletionHandler<Integer, Void>() {            @Override            public void completed(final Integer result, final Void attachment) {                while (buffer.hasRemaining()) {                    channel.write(buffer, null, this);                }                read(channel, proxy);            }            @Override            public void failed(final Throwable exc, final Void attachment) {                close(channel, null);                throw new RuntimeException("unable to confirm", exc);            }        });    }    private void accept() {        this.server.accept(null, new CompletionHandler() {            public void completed(final AsynchronousSocketChannel channel, final Void attachment) {                // Delegate off to another thread for the next connection.                accept();                // Delegate off to another thread to handle this connection.                meta(channel);            }            public void failed(final Throwable exc, final Void attachment) {                throw new RuntimeException("unable to accept new connection", exc);            }        });    }    private void writeToFile(final AsynchronousSocketChannel channel, final ByteBuffer buffer, final FileWriterProxy proxy) {        assert !Objects.isNull(buffer) && !Objects.isNull(proxy) && !Objects.isNull(channel);        try {            buffer.flip();            final long bytesWritten = proxy.getFileWriter().write(buffer, proxy.getPosition().get());            proxy.getPosition().addAndGet(bytesWritten);        } catch (IOException e) {            close(channel, proxy);            throw new RuntimeException("unable to write bytes to file", e);        }    }    private void close(final AsynchronousSocketChannel channel, final FileWriterProxy proxy) {        assert !Objects.isNull(channel);        try {            if (!Objects.isNull(proxy)) {                proxy.getFileWriter().close();            }            channel.close();        } catch (IOException e) {            throw new RuntimeException("unable to close channel and FileWriter", e);        }    }

TheFileReceiverAsync abstraction builds upon the idiomatic use ofAsynchronousChannels demonstrated in thistutorial.

6. Running the program

The program can be run from within the IDE, using the normal JUnit Runner or from the command line using maven. Ensure that the test resources (large source files and target directories exist).
Running tests from command line

mvn clean install

You can edit these in theAbstractTest andFileCopyAsyncTest classes. Fair warning theFileCopyAsyncTest can run for a while as it is designed to copy two large files asynchronously, and the test case waits on a CountDownLatch without a max wait time specified.

I ran the tests using the “spring-tool-suite-3.8.1.RELEASE-e4.6-linux-gtk-x86_64.tar.gz” file downloaded from theSpringSource website. This file is approximately 483mb large and below are my test elapsed times. (using a very old laptop).

Test elapsed time

Running com.javacodegeeks.nio.large_file_transfer.remote.FileCopyTestTests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.459 sec - in com.javacodegeeks.nio.large_file_transfer.remote.FileCopyTestRunning com.javacodegeeks.nio.large_file_transfer.remote.FileCopyAsyncTestTests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.423 sec - in com.javacodegeeks.nio.large_file_transfer.remote.FileCopyAsyncTestRunning com.javacodegeeks.nio.large_file_transfer.local.FileCopyTestTests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.562 sec - in com.javacodegeeks.nio.large_file_transfer.local.FileCopyTest

7. Summary

In this tutorial, we demonstrated how to transfer a large file from one point to another. This was showcased via a local copy and a remote transfer via sockets. We went one step further and demonstrated transferring a large file from one remote location to another via an asynchronous receiving node.

8. Download the source code

This was a Java NIO Large File Transfer tutorial

Do you want to know how to develop your skillset to become aJava Rockstar?

Subscribe to our newsletter to start Rockingright now!

To get you started we give you our best selling eBooks forFREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to theTerms andPrivacy Policy

Sign up

Thank you!

We will contact you soon.