Share
System.out.println("USE A LOGGER!!!");
No matter if you are developing a small library, a complex framework or an application for end users - at some point you will come to the point that you want to log information. Here the Java community offers a whole wealth of solutions: Besides java.util.Logging there are libraries like commons-logging, Log4J or Logback. Where are the differences and unique selling points of the frameworks? In addition, actually everyone says that only the use of slf4J is "best practice". But why should I use an abstraction when developing a concrete application? Or does slf4j perhaps offer completely different advantages? And as if that weren't enough questions, with Java 9 a new logger, System.Logger, was quietly integrated into the OpenJDK.
This talk gives an understandable guide through the logging jungle and shows a few practices that help to get the logging of libraries and applications under control. If you're on the verge of going back to System.out out of frustration, this session is for you.
Hendrik Ebbers
Transcript
87 slides
-
This is a very very very long gag @hendrikEbbers Hendrik
• Karakun Co-Founder • Founder of JUG Dortmund • JSR EG member • JavaOne Rockstar, Java Champion • AdoptOpenJDK TSC member -
This is a very very very long gag @hendrikEbbers Stephan
• Karakun Co-Founder • SoCraTes CH community member • OpenWebStart committer • Conference speaker -
This is a very very very long gag @hendrikEbbers Content
• Logging 101 • Java Logging & Libraries • Integration & Architecture • Centralized Logging • Outlook - Future of Logging -
This is a very very very long gag @hendrikEbbers Content
Go to www.menti.com and use the code 4490 2536 -
This is a very very very long gag @hendrikEbbers Definition
of 'log' • a thick piece of wood cut from a tree • a written record of things that happen, especially an official record of a journey on a ship or in a plane • a logarithm * from the Macmillan Dictionary -
This is a very very very long gag @hendrikEbbers Why
should you log? Log = Events which happened in the past • retrace what has happened • understand why an event occurred • gather information about how the software is used • learn from the past -
This is a very very very long gag @hendrikEbbers What
should you log? • important events • input from outside the software / module • unexpected behavior -
This is a very very very long gag @hendrikEbbers What
should you NOT log? • too many details • events in a loop • sensitive data 08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1' 08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A' 08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A3' 08:34:23 INFO User 'Max' with birthday '01/01/1970' mutates password field with new value '1A3.' -
This is a very very very long gag @hendrikEbbers How
long should you keep a log? • forever • until you are certain you will never ever need the log any more Since you can never be certain: KEEP IT FOREVER -
This is a very very very long gag @hendrikEbbers •
performance • code cluttering LOG.info("We start the transaction"); manager.beginTransaction(); LOG.info("DB query will be executed"); LOG.debug("DB query: select * from users"); long start = now(); users = manager.query("select * from users"); LOG.info("DB query executed"); LOG.debug("DB query executed in " + (now() - start) + " ms"); LOG.info("Found " + users.size() + " entities"); manager.endTransaction(); LOG.info("Transaction done"); Has logging any downsides? -
This is a very very very long gag @hendrikEbbers Logging
Concept • What's your motivation ? • who is reading / monitoring the log ? • what do you want to extract from the logs ? • what do you not want to extract ? • how is logging different in production/test/dev ? -
This is a very very very long gag @hendrikEbbers System.out
System.out.print("Who needs a logger ??") -
This is a very very very long gag @hendrikEbbers Data
from Application • Message • Exception (optional) • Marker (optional) -
This is a very very very long gag @hendrikEbbers Data
from Logger • Logger Name • Log Level (Info, Warning, Debug ...) • Timestamp • Thread • Additional Context -
This is a very very very long gag @hendrikEbbers Data
from Logger • Calling Class • Calling Method • Calling File • Calling Line Number • Calling Stack -
This is a very very very long gag @hendrikEbbers Additional
Context • Context of the Logger • Properties/Attributes of the Logger • Context from the Application (MDC) In general this is based on the current thread -
This is a very very very long gag @hendrikEbbers Appender
- Filter • Filter messages and only append the relevant to the log • By log level • By marker • By any other data provided by the logger -
This is a very very very long gag @hendrikEbbers Appender
- Layout • Layout the message to fit the storage • Plain String / HTML • JSON / XML • GELF (Graylog Extended Log Format) • Optionally group messages to bulks -
This is a very very very long gag @hendrikEbbers Appender
• Send the message to the storage • Console (stdout/stderr) • File (plain text, html, json, xml, etc) • DB • Remote -
This is a very very very long gag @hendrikEbbers Logging
Frameworks 2012 2013 2014 2016 2017 2019 2020 2015 2018 Log4j Logback Log4j2 Commons Logging SLF4J Java 8 Java 9 Java 11 TinyLog -
This is a very very very long gag @hendrikEbbers Logging
Frameworks 2012 2013 2014 2016 2017 2019 2020 2015 2018 Log4j Logback Log4j2 TinyLog Commons Logging SLF4J Java 8 Java 9 Java 11 -
This is a very very very long gag @hendrikEbbers Performance
- Create Messages • Guard logging code • Delay string concatenation • Delay arguments calculation LOG.info("Hello " + username + "!"); if (LOG.isInfo()) {LOG.info("Hello " + getUserNameFromServer() + "!");} LOG.info("Hello {}!", userName); LOG.info("Hello {}!", () -> getUserNameFromServer()); -
This is a very very very long gag @hendrikEbbers Performance
- Enhance Message • Avoid expensive meta data • Delay calculation of expensive meta data 08:34:23 INFO [com.sample.MyClass] com.sample.MyClass.java:319 Hello World! LOgger name Source Information -
This is a very very very long gag @hendrikEbbers Performance
- Enhance Message • Avoid expensive meta data • Delay calculation of expensive meta data 08:34:23 INFO [com.sample.MyClass] com.sample.MyClass.java:319 Hello World! LOgger name Source Information Do we really need this? -
This is a very very very long gag @hendrikEbbers Performance
- Persist Message • Asynchronous write to the storage • Bulk write to storage -
This is a very very very long gag @hendrikEbbers Application
Architecture • Let's have a look at some samples • As always we start with 'Hello World' -
This is a very very very long gag @hendrikEbbers Application
Architecture HelloLogger.java Just a simple java class -
This is a very very very long gag @hendrikEbbers Application
Architecture HelloLogger.java java.util.logging We use the 'Standard' Java logging -
This is a very very very long gag @hendrikEbbers Application
Architecture package com.karakun; import java.util.logging.Logger; public class HelloLogging { private static final Logger LOG = Logger.getLogger("com.karakun.HelloLogging"); public static void main(final String[] args) { LOG.info("Hello World"); } } -
This is a very very very long gag @hendrikEbbers Application
Architecture HelloLogger.java java.util.logging app-log.txt Level: INFO logging.properties We can configure the logger by using a properties file -
This is a very very very long gag @hendrikEbbers Application
Architecture • Well that worked quite good • Let's have a look at a more complex application -
This is a very very very long gag @hendrikEbbers Application
Architecture My Application database-access security-lib Our application has some dependencies -
This is a very very very long gag @hendrikEbbers Application
Architecture My Application database-access security-lib jul Logging app-log.txt Level: INFO logging.properties We can still configure the logger by using a properties file -
This is a very very very long gag @hendrikEbbers Application
Architecture My Application database-access Log4j security-lib jul Logging app-log.txt Level: INFO logging.properties The Dependency uses another logger to create log messages -
This is a very very very long gag @hendrikEbbers Application
Architecture My Application database-access Log4j security-lib TinyLog jul Logging app-log.txt Level: INFO logging.properties ... and here we have another logger -
This is a very very very long gag @hendrikEbbers Application
Architecture My Application database-access Log4j security-lib TinyLog jul Logging app-log.txt Level: INFO logging.properties -
This is a very very very long gag @hendrikEbbers Application
Architecture My Application database-access Log4j security-lib TinyLog jul Logging ??????? Level: ? app-log.txt Level: INFO ??????? Level: ? logging.properties Where can we find that Logging? -
This is a very very very long gag @hendrikEbbers Application
Architecture My Application database-access Log4j security-lib TinyLog jul Logging db-log.txt Level: INFO app-log.txt Level: INFO sec-log.txt Level: INFO logging.properties log4j.properties tinylog.properties -
This is a very very very long gag @hendrikEbbers Application
Architecture My Application logging.properties log4j.properties tinylog.properties log4j2.properties db-log.txt app-log.txt sec-log.txt -
This is a very very very long gag @hendrikEbbers Application
Architecture • Providing different configs for all possible logging libs can't be the solution • Let's get back to the table and sketch a good solution -
This is a very very very long gag @hendrikEbbers A
Perfect LOG 08:34:23 INFO Application starts 08:34:24 DEBUG Connection to DB 'my-db' 08:34:26 DEBUG Database connection created 08:34:28 INFO Application started 08:35:12 INFO User 'Hendrik' tries to log in to the system 08:35:13 WARN Password for user 'Hendrik' is incorrect 08:35:46 INFO User 'Hendrik' tries to log in to the system 08:35:47 INFO User 'Hendrik' logged in 08:36:22 DEBUG User 'Hendrik' opens search dialog 08:36:34 INFO User 'Hendrik' starts search 08:36:34 DEBUG User 'Hendrik' starts search with name='*java*' 08:36:34 DEBUG Executing query on db 08:36:47 DEBUG ResulSet contains 7 entries 08:36:47 DEBUG User 'Hendrik' - Search result count: 7 08:36:48 INFO User 'Hendrik' successfully executed search 08:36:52 INFO User 'Hendrik' logged out -
This is a very very very long gag 08:34:23 INFO
Application starts 08:34:24 DEBUG Connection to DB 'my-db' 08:34:26 DEBUG Database connection created 08:34:28 INFO Application started 08:35:12 INFO User 'Hendrik' tries to log in to the system 08:35:13 WARN Password for user 'Hendrik' is incorrect 08:35:46 INFO User 'Hendrik' tries to log in to the system 08:35:47 INFO User 'Hendrik' logged in 08:36:22 DEBUG User 'Hendrik' opens search dialog 08:36:34 INFO User 'Hendrik' starts search 08:36:34 DEBUG User 'Hendrik' starts search with name='*java*' 08:36:34 DEBUG Executing query on db 08:36:47 DEBUG ResulSet contains 7 entries 08:36:47 DEBUG User 'Hendrik' - Search result count: 7 08:36:48 INFO User 'Hendrik' successfully executed search 08:36:52 INFO User 'Hendrik' logged out @hendrikEbbers A Perfect LOG My Application jul Logging app-log.txt Level: ALL -
This is a very very very long gag 08:34:23 INFO
Application starts 08:34:24 DEBUG Connection to DB 'my-db' 08:34:26 DEBUG Database connection created 08:34:28 INFO Application started 08:35:12 INFO User 'Hendrik' tries to log in to the system 08:35:13 WARN Password for user 'Hendrik' is incorrect 08:35:46 INFO User 'Hendrik' tries to log in to the system 08:35:47 INFO User 'Hendrik' logged in 08:36:22 DEBUG User 'Hendrik' opens search dialog 08:36:34 INFO User 'Hendrik' starts search 08:36:34 DEBUG User 'Hendrik' starts search with name='*java*' 08:36:34 DEBUG Executing query on db 08:36:47 DEBUG ResulSet contains 7 entries 08:36:47 DEBUG User 'Hendrik' - Search result count: 7 08:36:48 INFO User 'Hendrik' successfully executed search 08:36:52 INFO User 'Hendrik' logged out @hendrikEbbers A Perfect LOG My Application Log4j Logging db-log.txt Level: ALL DB Lib -
This is a very very very long gag @hendrikEbbers A
Perfect LOG 08:34:23 INFO Application starts 08:34:24 DEBUG Connection to DB 'my-db' 08:34:26 DEBUG Database connection created 08:34:28 INFO Application started 08:35:12 INFO User 'Hendrik' tries to log in to the system 08:35:13 WARN Password for user 'Hendrik' is incorrect 08:35:46 INFO User 'Hendrik' tries to log in to the system 08:35:47 INFO User 'Hendrik' logged in 08:36:22 DEBUG User 'Hendrik' opens search dialog 08:36:34 INFO User 'Hendrik' starts search 08:36:34 DEBUG User 'Hendrik' starts search with name='*java*' 08:36:34 DEBUG Executing query on db 08:36:47 DEBUG ResulSet contains 7 entries 08:36:47 DEBUG User 'Hendrik' - Search result count: 7 08:36:48 INFO User 'Hendrik' successfully executed search 08:36:52 INFO User 'Hendrik' logged out How can we achieve this combination in one file? -
This is a very very very long gag @hendrikEbbers A
stupid idea • Maybe we can just configure all logging libs to write in the same file... -
This is a very very very long gag @hendrikEbbers A
stupid idea • Maybe we can just configure all logging libs to write in the same file... Bild: Freepik.com Concurrency issues File Locks Multiple Messages in one line Really a bad idea... Don't do this at home -
This is a very very very long gag @hendrikEbbers Logging
issues • We have several loggers on the classpath • We want to have one central configuration • We want to have one central output • We do not want to miss logging messages from dependencies -
This is a very very very long gag @hendrikEbbers Let's
sketch a solution My Application & Dependencies Logging Framework A Logging Framework B Logging Framework C Central Appender/Config Single Storage collect all messages -
This is a very very very long gag @hendrikEbbers SLF4J
as Logging Facade • The SLF4J projects provide then needed functionalities • Logging facade • Logging bridges / adapters • Logging bindings Common API Forward messages to API/Facade Implementation of API/Facade -
This is a very very very long gag @hendrikEbbers Logging
redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar -
This is a very very very long gag @hendrikEbbers Logging
redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar Facade / API -
This is a very very very long gag @hendrikEbbers Logging
redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar Adapters -
This is a very very very long gag @hendrikEbbers Logging
redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar log4j-over-slf4j.jar replaces log4j.jar -
This is a very very very long gag @hendrikEbbers Logging
redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar log4j-over-slf4j.jar replaces log4j.jar log4j-over-slf4j.jar contains a custom org.apache.log4j.Logger class -
This is a very very very long gag @hendrikEbbers Logging
redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar Implementation of API & Binding to Logging Framework -
This is a very very very long gag @hendrikEbbers Logging
redirection My Application & Dependencies Commons logging API SLF4J API slf4j-api.jar Log4j API jcl-over-slf4j.jar log4j-over-slf4j.jar Underlying logging framework logback-classic.jar No configuration! logger.properties Only configure The underlying Logger -
This is a very very very long gag @hendrikEbbers Solutions
• There is no 'simple' solution that just fits for all scenarios and setups • Projects like SLF4J provides all that is needed to configure even complex use cases • Knowledge of the libs is needed -
This is a very very very long gag @hendrikEbbers Logging
Facades 2012 2013 2014 2016 2017 2019 2020 2015 2018 Log4j Logback Log4j2 TinyLog Commons Logging SLF4J Java 8 Java 9 Java 11 -
This is a very very very long gag @hendrikEbbers Java
System.Logger • Part of OpenJDK (since 9) • See JEP 264 (https://openjdk.java.net/jeps/264) • Minimal Logging API / Facade • Used internally in the OpenJDK • Default adapter is java.util.logging -
This is a very very very long gag @hendrikEbbers Java
System.Logger public class MainApp { private static System.Logger LOGGER = System.getLogger("MainApp"); public static void main(String[] args) { LOGGER.log(Level.ERROR, "error test"); LOGGER.log(Level.INFO, () -> "info test"); } } Remember our performance discussion? -
This is a very very very long gag @hendrikEbbers Non-Goals
of System.Logger • It is not a goal to define a general-purpose interface for logging. The service interface contains only the minimal set of methods that the JDK needs for its own usage. • It is not a goal to support programmatic logging configuration (setting levels, destination files, etc...) via the API. -
This is a very very very long gag @hendrikEbbers Best
Practice for System.Logger • Do not use it as your application logger • Configure a bridge within the logger facade you are using (by using SPI) • Use it directly in small libraries to remove dependencies (no slf4j-api is needed anymore) -
This is a very very very long gag @hendrikEbbers Application
landscapes today • We now have a real good solutions to create and store the logging for one application • In a real world scenario this is not enough! • Let's take this a step further... -
This is a very very very long gag @hendrikEbbers Modern
usecases • Will I have 1 log file for each node if my application is clustered? • Can I get my application log synchronised with the log of the database or the docker container? • How can I search in my logs and display all logs for 1 specific request? -
This is a very very very long gag @hendrikEbbers Modern
usecases LOG LOG LOG LOG LOG LOG LOG LOG -
This is a very very very long gag @hendrikEbbers Let's
sketch a solution • We need to find a way to centralise our logging • All log messages must be pushed / pulled to one central place • Based on the amount of logging this needs a special solution • The solutions needs good search functionality -
This is a very very very long gag @hendrikEbbers Centralized
Log Management Logger Logentries json-file System Journal Syslog fluentd Windows Event Store Splunk GELF Amazon CloudWatch Google Cloud UDP / TCP TCP / HTTPS TCP / HTTPS TCP / HTTPS TCP / (HTTP & HTTPS) TCP UDP / TCP / TLS HTTP / HTTPS -
This is a very very very long gag @hendrikEbbers Graylog
/ GELF example Applications GELF Docker Containers GELF UDP / TCP UDP / TCP DB / System / Services GELF UDP / TCP JSON Format -
This is a very very very long gag @hendrikEbbers Tracing
• Tracing libraries provide important metadata in a distributed environment • Metadata is added to HTTP header • Follow a request over several services Use MDC & Metadata to see all log messages of this flow -
This is a very very very long gag @hendrikEbbers Analyze
Logs using AI • Use 'log intelligence' to get better feedback • Detect issues automatically • Detect anomalies before they create issues • Understand and react to users and their behavior -
• Use a Logging Facade • Use Centralized Logging (even
of you have only 1 log file) • Use MDC and Markers