File Operations in Camel (ftp,file components)
In this tutorial, we are going to learn how to do file operations with Apache Camel. We will use two camel components for this purpose – File and Ftp. Generally, File component is used for local file input/output operations and Ftp for remote file operations as an ftp client for Ftp servers.
Example Scenario:
Case 1 : File component
We have an input folder which has a sub-folder and files with different extensions. We want to copy all the text files from it and its sub folders to another folder “output” and rename them and change their contents.
Case 2: Ftp component
We want to copy a file from our local to a remote ftp server with few configurations for ftp connection.
Project artifacts:
1. Technologies used –
a. Eclipse IDE for Java EE developer 4.5.0.20150621-1200 (Mars)
b. Preinstalled Maven with Eclipse Mars
c. Camel 2.16.0
d. Java 1.8
2. Eclipse project folder structure:
3. Jars Used
If you are not using Maven, here is the list of jars you would need for this project:
camel-core-2.16.0.jar
slf4j-api-1.6.6.jar
jaxb-core-2.2.11.jar
jaxb-impl-2.2.11.jar
camel-spring-2.16.0.jar
spring-context-4.1.6.RELEASE.jar
spring-beans-4.1.6.RELEASE.jar
spring-core-4.1.6.RELEASE.jar
commons-logging-1.2.jar
spring-expression-4.1.6.RELEASE.jar
spring-aop-4.1.6.RELEASE.jar
aopalliance-1.0.jar
spring-tx-4.1.6.RELEASE.jar
commons-net-2.0.jar
camel-ftp-2.16.0.jar
jsch-0.1.53.jar
4. Maven Dependencies
file:pom.xml . Paste the below content to pom.xml. It has camel and spring dependencies. It can pass two arguments – “java” for running camel in java DSL route and “xml” for running as Spring routes.
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.javanbeyond</groupId> <artifactId>CamelFileOperations</artifactId> <version>0.0.1-SNAPSHOT</version> <dependencies> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-core</artifactId> <version>2.16.0</version> </dependency> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-spring</artifactId> <version>2.16.0</version> </dependency> <dependency> <groupId>commons-net</groupId> <artifactId>commons-net</artifactId> <version>2.0</version> </dependency> <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-ftp</artifactId> <version>2.16.0</version> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>exec-maven-plugin</artifactId> <version>1.1.1</version> <executions> <execution> <phase>test</phase> <goals> <goal>java</goal> </goals> <configuration> <mainClass>com.javanbeyond.MainClass</mainClass> <arguments> <!--<argument>java</argument> --> <argument>xml</argument> </arguments> </configuration> </execution> </executions> </plugin> </plugins> </build> </project>
5. Main Class
file: MainClass.java . Main class is our starting point and is called by maven. It can run Java DSL or Spring DSL based on argument passed. Also, we have to comment the line based on the component we don’t want to run.
package com.javanbeyond; import org.apache.camel.CamelContext; import org.springframework.context.support.ClassPathXmlApplicationContext; public class MainClass { public static void main(String[] args) throws Exception { if (args.length < 1) { System.err.println("Please pass java or xml as argument"); System.exit(1); } // Pass route id to execute the particular route if (args[0].equalsIgnoreCase("java")) { // executeJavaDSL("executeFileComponent"); executeJavaDSL("executeFtpComponent"); } else if (args[0].equalsIgnoreCase("xml")) { // executeSpringDSL("executeFileComponent"); executeSpringDSL("executeFtpComponent"); } else { System.out.println("Argument not supported."); System.exit(1); } // Wait 10 seconds for camel to finish file operation Thread.sleep(10000); } @SuppressWarnings("resource") public static void executeSpringDSL(String routeId) throws Exception { new ClassPathXmlApplicationContext("camel-context.xml") .getBean("camel", CamelContext.class).startRoute(routeId); } public static void executeJavaDSL(String routeId) throws Exception { new CamelJavaDSL().getCamelContext().startRoute(routeId); } }
6a. Java DSL class
file: CamelJavaDSL.java . The class where Camel routes are defined for different components. In our case, file and ftp.
Case 1: “executeFileComponent“
I have used camel file expressions to rename the output files and camel simple expressions to change the content of each file.
Case 2: “executeFtpComponent”
I have used filegenie server as a remote ftp server. You need to register in their site http://www.filegenie.com/ before you can use it for ftp connections. At the time of writing, storage less than 20 MB can be used freely. Use your userid and password provided by them through email.
package com.javanbeyond; import org.apache.camel.CamelContext; import org.apache.camel.builder.RouteBuilder; import org.apache.camel.impl.DefaultCamelContext; import org.apache.camel.util.jndi.JndiContext; import org.apache.commons.net.ftp.FTPClientConfig; public class CamelJavaDSL extends RouteBuilder { private static CamelContext context; public CamelJavaDSL() throws Exception { JndiContext jndiContext = new JndiContext(); // Will be used in ftp component jndiContext.bind("ftpConfig", getFtpConfig()); context = new DefaultCamelContext(jndiContext); context.addRoutes(this); context.start(); } @Override public void configure() throws Exception { // noop - don't delete file in the source. // recursive - look into subfolders. // include - pick the files whose names matches the regex. // autoCreate - create folders in output if not available. // fileName - rename the output files based on the pattern mentioned by // file expression. // keepLastModified - preserves the Last Modified date of source files // in the destination folder. // setBody method replaces the message body with the content provided. from("file:input?noop=true&recursive=true&include=.*\\.txt").autoStartup(false).routeId("executeFileComponent") .setBody(simple("${body} Javanbeyond")) .to("file:output?autoCreate=true&fileName=${file:name.noext}_cp.${file:name.ext}&keepLastModified=true"); from("file:input/sub-input?noop=true").autoStartup(false).routeId("executeFtpComponent") // We can configure additional options on the ftpClient // from the URI directly by using the "ftpClient." prefix // Additionally, FTPClientConfig class can be referred // by its name in Jndi Context binding .to("ftp://ftp.filegenie.com?username=javanbeyond&password=XXXXXX&ftpClient.dataTimeout=30000&ftpClientConfig=#ftpConfig"); } public CamelContext getCamelContext() { return context; } public FTPClientConfig getFtpConfig() { // Configuration for ftp component FTPClientConfig ftpConfig = new FTPClientConfig(); ftpConfig.setServerLanguageCode("en"); return ftpConfig; } }
6b. Camel Spring XML DSL
file:camel-context.xml. Spring XML equivalent to Java DSL .
<?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd"> <camelContext id="camel" xmlns="http://camel.apache.org/schema/spring"> <!-- noop - keeps file in the source recursive - looks into subfolders include - picks the files whose names matches the regex autoCreate - create folders in output if the available fileName - rename the output files based on the pattern mentioned by file expression keepLastModified - preserves the Last Modified date of source files in the destination folder. setBody method replaces the message body with the content provided --> <route id="executeFileComponent" autoStartup="false"> <from uri="file:input?noop=true&recursive=true&include=.*\.txt" /> <setBody> <!-- Using simple expression to append string in message body --> <simple>${body} Javanbeyond</simple> </setBody> <to uri="file:output?autoCreate=true&fileName=${file:name.noext}_cp.${file:name.ext}&keepLastModified=true" /> </route> <!-- We can configure additional options on the ftpClient from the URI directly by using the "ftpClient." prefix. Additionally, FTPClientConfig class can be referred by its id in Spring context --> <route id="executeFtpComponent" autoStartup="false"> <from uri="file:input/sub-folder?noop=true" /> <to uri="ftp://ftp.filegenie.com?username=javanbeyond&password=XXXXXX&ftpClient.dataTimeout=30000&ftpClientConfig=#ftpConfig" /> </route> </camelContext> <bean id="ftpConfig" class="org.apache.commons.net.ftp.FTPClientConfig"> <property name="serverLanguageCode" value="en" /> </bean> </beans>
Example Execution:
Case 1: File component
- Make changes to Main class by calling either executeJavaDSL or executeSpringDSL with “executeFileComponent” as argument and commenting other methods.
- Run the example by running maven clean and install and passing either “xml” or “java” to the Main Class.
- Wait for the program to end. Refresh the project.
- An output folder containing text files should get created with file names changed and “javanbeyond” added to the content.
Case 2: Ftp component
- Make changes to Main class by calling either executeJavaDSL or executeSpringDSL with “executeFtpComponent” as argument and commenting other methods.
- If needed, configure ftp server name and username and password in ftp URIs.
- Run the example by running maven clean and install and passing either “xml” or “java” to the Main Class.
- You should see the input1.txt file inside ftp server. Using Filezilla, following is the view:
