2 Leverage Enterprise Integration Patterns with Apache Camel and Twitter Bruno Borges Oracle Product Manager for Latin America Java EE, GlassFish, WebLogic, Coherence
3 Who am I? Java developer since 2000 Speaker at Conferences – JustJava, JavaOne Brazil, The Developers’ Conference, ApacheCon Evangelized Apache Camel and Apache Wicket in Brazil Joined Oracle on July 2012 – Product Manager for Java EE, GlassFish and WebLogic – Latin America Married, lives in Sao Paulo, has a Golden Retriever, hiker and gamer Bruno Borges
6 Randy Stafford Oracle “If you are involved with the operation or development of an enterprise application, there will doubtless come a time when you will need to integrate your application with another using the emerging preferred approach of messaging.”
7 Enterprise Integration Patterns Why do we need patterns for integration? Why is it so hard? Asynchronous messages Where and when to use them? Cloud Computing depends on it Integration, integration, integration…
9 What is Apache Camel? Use implementations of Enterprise Integration Patterns Design routes for Enterprise Integration Patterns Use out-of-the-box or develop components and endpoints Process asynchronous and synchronous messages Connect distinct and independent systems Receive, transform and deliver data A Java framework that enables the developer to:
10 Quick overview of Apache Camel Components Endpoints Routes Exchanges and Messages Consumers Producers Processors The core of Camel is about
14 Example using Spring Because you love XML <camelContext xmlns="http://camel.apache.org/schema/spring"> <route> <from uri="file:///var/usr/inbox/"/> <choice> <when> <xpath>$foo = 'widget'</xpath> <to uri="seda:widget"/> </when> <when> <xpath>$foo = 'gadget'</xpath> <to uri="seda:gadget"/> </when> <otherwise> <to uri="seda:lixo"/> </otherwise> </choice> </route> </camelContext>
16 Apache Camel DSLs for Java, Spring XML and Scala Standard URIs for Endpoints Message routing based on Predicates and Expressions Lots of components – JMS, HTTP, MINA, JDBC, FTP, WebServices, EJB, JPA, Hibernate, IRC, JCR, AS/400, LDAP, Mail, Nagios, POP, Printers, Quartz, Restlet, RMI, RSS, Scalate, XMPP… Integrates with CDI – CamelContext can be started with EJB3’s @Startup/@Singleton Implementation of Enterprise Integration Patterns
18 Social Media 29% of consumers post negative comments on social networks 49% of 16-24 y’o have posted negative comment following bad CX 71% that had complaints on social networks were ignored Of 29% who did get responses, 51% had positive reaction after 17% of those who had a response, wrote a positive comment 13% deleted their negative post The Rise of Social Media as a Communications Channels Source: 2011 Customer Experience Index Report “The Era of Impatience” http://www.oracle.com/us/products/applications/uk-cei-report-1641675.pdf
19 Social Data Twitter for instance – Twitter, on average, receives 3,9k tweets per second – Peaks during main events can reach 20k tweets per second – This can more than 300 million tweets per day – One tweet may have 140 characters 560 bytes (Twitter uses UTF-8) – Per day, more than 150 GB of raw text tweets Not counting indexing and storage overhead Let’s not forget about Facebook, Google+, LinkedIn and many others Reading and processing all that data
21 Filtering and Processing Social Data Read status updates Filter based on general keywords Identify the issue based on their content Route to who will best process that tweet Reply if possible Store for future conversations with original sender – Store retweets that mention users who thought the same thing Customize as needed It looks like a lot with Enterprise Integration Patterns, isn’t?
25 Camel Twitter CAMEL-1520 – https://issues.apache.org/jira/browse/CAMEL-1520 Presented at ApacheCon NA’09 during the BarCamp Discussions with Camel committers at ApacheCon led me to build a broader component, that could support other social networks Challenge: same URIs and features for different Social Networks (“social data providers”). Starting with Facebook, Twitter, LinkedIn, Foursquare, and (the already dead) Google Buzz Proposed as a Camel Component to the ASF in 2009
26 Camel Social How would it look like? Started in 2010 Goal – The Camel Social component objective was to be able to poll social data from several networks in a uniform way to be processed through a route URI format – social://[social provider] [/social path]?parameters Features – Post, Read, Search
27 Camel Social Examples Sending a tweet "direct:foo" to "social://twitter/status" Reading a Facebook Timeline "social://facebook/timeline/brunocborges" ==> { to("log:facebookWall") } Searching for events on Facebook "direct:doSearch" to "social://facebook/events" to "log:events"
28 Camel Social Problem – It was very hard to design an standard API for different Social Nets Died in 2010 What about Spring Social? – Didn’t exist back then – Now they have an abstract API for Social Providers and for OAuth – One extra API per Social Network – It “could” bring the component back to life An unsuccessful attempt http://code.google.com/p/camel-social
29 Camel Twitter May 2011 – project rebooted on GitHub – Focus only on Twitter http://github.com/brunoborges/camel-twitter Kudos to Brett Meyer, who finished the job Bilgin Ibryam added support for the Twitter Streaming API Thank you guys! I owe you two a beer :-) A rebirth, thanks to GitHub @brettemeyer
30 Camel Twitter Available since Apache Camel 2.10 Features – Send a tweet (update status) – Send and read Direct Messages (DMs) – Search using REST (polling) or Streaming – Sample streaming from public tweets – Timeline reads for home, mentions, retweets, specific user Uses the well-know Twitter4J library (Apache licensed) Final version
31 Sending a tweet "jms:queue:tweetingQueue1" to "twitter://timeline/user" Reading a Timeline "twitter://timeline/home?type=polling&delay=5" ==> { to("log:homeTweets") } Searching for keywords "direct:doSearch" to "twitter://search?keywords=JavaOne“ to "log:homeTweets" Examples Using the Scala DSL
33 Camel Twitter Demo Connects to the Search Streaming channel based on keywords Filter for tweets with media only Check Coherence Cache and filter duplicates Put image/tweet URL as key/value on Coherence Cache Transform tweets into small POJOs Transform POJOs into JSON Strings Sends to users connected to the WebSocket channel * based on the Camelympics project developed by Bilgin Ibryam https://github.com/bibryam/camelympics How it works
34 About Camel Coherence Component A fork of the Camel Cache component – Instead of EHCache, now uses Oracle Coherence Same features as the core component – Operations: add, check, get, remove, removeAll – URI: coherence://cacheName Coherence is LRU by default (‘local-scheme’) – Great for the demo! Stores the recent tweets without blowing the Heap – If more storage is needed, deploy new Coherence Cache instance The #1 Distributed Caching technology
37 The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.