- Notifications
You must be signed in to change notification settings - Fork11
jcustovic/data-exchange-client
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
The projectData Exchange Client allows you to connect to SFTP or FTP servers and configure upload and download pullers.
Current stable version:1.5.2
## How to use
The setup is quite easy and it requires you to only edit application.yaml configuration file.
To directly start the app run:
mvn spring-boot:runTo produce executable jar and run:
mvn clean installjava -jar target/data-exchange-client*.jarThe configuration is split up in 2 parts:
- sftps - define your sftp servers
- ftp - define your ftp servers
After that each and every server can have a list of upload and download pollers.
Example configuration:
app:sftps: -name:localhostSFTPhost:localhostport:8888username:charispassword:pass# private-key: classpath:keys/user@ssh-cluster_id-dsa# private-key-passphrase: changeitupload-pollers: -name:testSftpUploadPollerinput-folder:D:\Temp\sftp-outbound-test\inputprocessed-folder:D:\Temp\sftp-outbound-test\processedremote-output-folder:/sftp-outbound-testregex-filter:.+\.xml$ -input-folder:D:\Temp\sftp-outbound-test\input\test2processed-folder:D:\Temp\sftp-outbound-test\processed\test2remote-folder:/sftp-outbound-test/test2regex-filter:ftps: -name:localhostFTP# arbitrary name, should be uniquehost:localhostport:12345username:testpassword:testupload-pollers: -name:testFtpUploadPoller# arbitrary name, should be uniqueinput-folder:D:\Temp\ftp-outbound-test\input# the directory from where to upload filesprocessed-folder:D:\Temp\ftp-outbound-test\processedremote-output-folder:/ftp-outbound-testregex-filter:.+\.xml$download-pollers: -name:testFtpDownloadPollerdownload-folder:D:\Temp\ftp-outbound-test\downloadingoutput-folder:D:\Temp\ftp-outbound-test\queueoutput-file-name-expression:"'bla_' + payload.lastModified() + payload.name"# Defaults to null which means original filenameremote-input-folder:/ftp-input-testregex-filter:.+\.txt$delete-remote-file:false# default is truepoll-interval-milliseconds:30000# 30s, defaults to 10s#poll-cron: 0 50 * * * ? # Run every 50th min every hour#modified-date-after-minutes: 5
app:sftps: -name:# arbitrary name, should be uniquehost:# sftp hostport:# sftp portusername:# usernamepassword:# passwordprivate-key:# if ssh key enabled login here you put the path (classpath:, file:, http:)private-key-passphrase:# passphare for the key file if enabledupload-pollers: -# list of upload pollers...download-pollers: -# list of upload pollers...
app:ftps: -name:# arbitrary name, should be uniquehost:# ftp hostport:# ftp portusername:# usernamepassword:# passwordftp-parser-date-format:# Optional: in case of legacy systems that do not respect the pattern for date format, e.g. 'dd/MM/yy HH:mm:ss'upload-pollers: -# list of upload pollers...download-pollers: -# list of upload pollers...
app:sftps:# or ftp -upload-pollers: -name:# arbitrary name, should be uniqueinput-folder:# the directory from where to upload filesprocessed-folder:# successfully uploaded files will be moved hereremote-output-folder:# remote directory where to upload filesregex-filter:# regex for files in input folder
app:sftps:# or ftp -download-pollers: -name:# arbitrary name, should be uniquedownload-folder:# the directory where the files will be placed temporary until they finish downloadingoutput-folder:# the directory where the files will be placed when fully downloadedoutput-file-name-expression:# Optional: rename expression (SpEL) in case the files are not named uniquelyremote-input-folder:# Remote directory where to listen for files to be downloadedregex-filter:# Remote file filterdelete-remote-file:# default is truepoll-interval-milliseconds:# defaults to 10spoll-cron:# Optional: in case cron is needed for polling instead of fixed intervalsemaphore-file-suffix:# Optional: it will wait for semaphore file before starting to download the file eg. '.sem'modified-date-after-minutes:5# Optional: This option will pick only pick up files that have last modified timestamp later than given time
app:sftps:# or ftp -download-pollers: -...s3-configuration:input-folder:${java.io.tmpdir}/sftpClient/queuebucketName:aws-region:aws-account:aws-accessKey:aws-secretKey:aws-role:server-side-encryption:true
Push throughput speeds for every file to ElasticSearch. Internally we are using RestHighLevelClient so it needs to be configured via Spring Boot.
spring:elasticsearch:rest:uris:localhost# Needed for RestHighLevelClientapp:es:index_pattern:"'data-exchange-client-' + T(java.time.LocalDate).now().format(T(java.time.format.DateTimeFormatter).ofPattern('YYYY-MM'))"
This library comes with a support for reading remote configuration from Consul or AWS Parameter Store. By default, this feature is disabled,but by writing your own bootstrap.yml file in your project, you can make use of it. It is mainly for storing connection credentials (i.e. hostname, port,username, password) in one common place.
Here is one example configuration of your bootstrap.yml file, which enables Consul support.Don't forget to specify-Dspring.profiles.active in order for the correct profile to be loaded.
spring:cloud:consul:enabled:trueport:...scheme:httpsconfig:enabled:trueprefix:...format:...data-key:...---spring:profiles:devcloud:consul:host:...config:acl-token:...
This library uses Spring Cloud AWS to integrate with AWS Parameter Store.It is configured usingaws.paramstore.* properties described in Spring Cloud AWS documentation.Its functionality is extended by allowing you to configureAWSStaticCredentialsProvider orSTSAssumeRoleSessionCredentialsProviderusingaws.credentials.* properties described below.
Here is one example configuration of your bootstrap.yml file, which enables AWS Parameter Store support.
spring:cloud:consul:enabled:falseaws:credentials:use-static-provider:trueaccess-key:...secret-key:...region:...# specify below properties if you need to assume STS rolests-role-arn:...role-session-name:data-exchange-clientparamstore:enabled:trueprefix:/configname:data-exchange-client
Using the above configuration, a parameter stored in AWS Parameter Store under the key/config/data-exchange-client/connections/test/passwordwill be available in Spring asconnections.test.password.
In order to let the pollers know that they will take the configuration from remote store, you need to do the following changes. Following is an example of a download pollerbut same works for upload poller:
ftps: - name: localhostFTP # arbitrary name, should be unique remoteConfigName: <config_name_from_remote_configuration_store> download-pollers: - name: testFtpDownloadPoller file-type: txt download-folder: D:\Temp\ftp-outbound-test\downloading output-folder: D:\Temp\ftp-outbound-test\queue output-file-name-expression: "'bla_' + payload.lastModified() + payload.name" # Defaults to null which means original filename regex-filter: .+\.txt$ delete-remote-file: false # default is true poll-interval-milliseconds: 30000 # 30s, defaults to 10sSo with Consul the following have been replaced.remote-config-name --> user, password, host, portfile-type --> remote-input-folder or remote-output-folder
As a result all the remote configuration can be stored in one place and shared between different applications.
When the app is uploading files it will suffix the original file being uploaded with.uploading and when the file is uploaded fully it will rename it back to the original file. So it is important to no start processing the file on the other end that has a suffix.uploading.
The upload and download pollers rememberin-memory thefilename andcreated timestamp of all the files polled and will not repeat them (it will upload / download files only once). Keep in mind if yourestart the app the fileswill be picked up again.
The application exposes HTTP monitoring endpoint to monitor which connections are UP or DOWN and info about last file for every poller.
The default url is:http://localhost:8080/monitoring/connection-health-check
In addition it is possible to write your own checks in Spring SpEL for advanced usage in for e.g. Sensu.
Context in this case is the connection with pollers [ConnectionStatus.java] (src/main/java/com/dataexchange/client/domain/model/ConnectionStatus.java)
http://localhost:8080/monitoring/connection-health-check/{connectionName}?expression=YOUR_SPELAnd context here is only the specific poller [PollerStatus.java] (src/main/java/com/dataexchange/client/domain/model/PollerStatus.java).
http://localhost:8080/monitoring/connection-health-check/{connectionName}/{pollerName}?expression=YOUR_SPELExample:ifhttp://localhost:8080/monitoring/connection-health-check/ return:
{ "exampleSftp": { "status": "UP", "lastCheck": "2019-05-21T08:14:05.899", "downSince": null, "lastError": null, "pollers": { "demoDownloadPoller": { "direction": "DOWNLOAD", "lastTransfer": "2019-05-21T08:01:06.263", "lastFilename": "myFile-013615-2019-05-21_07-59-43.xml" } } }}we can use the endpointhttp://localhost:8080/monitoring/connection-health-check/exampleSftp?expression=status.toString()%20==%20%27UP%27 (NOTE: expression is URL encoded) to evaluate connection status.
status.toString() == 'UP'and the response will be 200 OK with the body:
trueor we can use the endpointhttp://localhost:8080/monitoring/connection-health-check/exampleSftp/demoDownloadPoller?expression=lastTransfer.compareTo(T(java.time.LocalDateTime).now().minusHours(2))%20>%200 to evaluate if we received a file on the specific poller in the last hour.
lastTransfer.compareTo(T(java.time.LocalDateTime).now().minusHours(1)) > 0About
Data exchange client - SFTP/FTP
Topics
Resources
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Contributors3
Uh oh!
There was an error while loading.Please reload this page.