Recently I tried out a new cookie recipe and was unsure how long to keep the cookies in the oven. I had to watch quite closely that I’m not burning them. That reminded me about log monitoring, a crucial part of security that sometimes gets overlooked.
Why do you need to monitor logs?
Too often, I see that applications write logs, but nobody rarely takes a look at them. This kind of behavior is problematic for several reasons:
- Performance and availability decrease. You may not notice early signs of faults that may cause you downtime later. Error messages stating that the disk or memory is nearly full and other performance-related things are calls for action.
- Data breach attempts go unnoticed. Loads of failed logins? Successful admin logins and weird times or from unusual places? A sudden increase in input validation errors? These could be signs of a data breach, so it’s good to check what’s going on.
- Insider attacks go unnoticed. The attack can also come from within, using legitimate privileges. Escalating privileges and creating new user accounts or wiping logs are a few examples mentioned in Graylog’s article that details what information to collect to correlate suspicious activity that might indicate an insider attack.
- Your system is faulty. If your server is returning loads of error messages, it could simply indicate that you have broken links on your website or a fault in the system. Errors and hiccups impact your user experience negatively, so you should fix them.
Log monitoring is so vital that Insufficient Logging & Monitoring was added to the list of most typical application vulnerabilities, OWASP Top 10 of 2017. According to OWASP, attackers rely on the likeliness that the lack of monitoring will help them perform their attacks without getting detected.
What should you write into logs?
To be able to detect attacks or faults, you need to log the correct things. So what should you write into logs?
OWASP’s Logging Cheat Sheet provides excellent insight into this topic with its comprehensive list of log sources to monitor and what kind of events to log. Some of the essential things to write into logs include:
- Successful and unsuccessful authentication events (logins and logouts)
- Authorization failures (a user attempts to access something they don’t have permission for)
- Input and output validation failures
- Syntax and runtime errors
- Starting, stopping, and shutting down services or systems
- Filesystem errors
- Virus detection
- Configuration changes
- Use of administrative privileges and the action taken
- Changing permissions of users or creating new user accounts
- Data imports and exports
- File uploads
- Granting or withdrawing consent, opt-ins, and opt-outs
- Severity or type of the event (error, warning, fatal, info, etc.)
The rule of thumb is to log all high-risk functionality, like accessing personal data, making payments, or administrating a system. For example, due to regulatory reasons, it’s often essential that no user or administrator can later deny that they didn’t do something.
What things must you NOT write into logs?
It might sound tempting to write pretty much everything to logs so it would be later available for review. However, to protect user privacy and system security, you need to omit, mask or scramble certain things.
Again, the OWASP Logging Cheat Sheet is a good reference for checking what should not be written in logs. Items to skip or mask include, for example:
- Sensitive personal data
- Directly personally identifiable data (such as government-issued IDs, addresses, etc.)
- Payment card folder data
- Passwords and authentication keys
- Session identifiers (you can hash them if you need to track sessions)
- Encryption keys
- Database connection strings
- Other confidential or sensitive data (message contents, trade secrets, etc.)
- Information that is illegal to collect or the user has opted out of
The proper log format is the key
After you’ve got the “what happened” part of the log contents right, it’s time to consider the rest of the required information and the formatting. According to the OWASP Logging Cheat Sheet, you should also record when the event happened, where it happened, and who did it to help you understand the situation better.
“When” is, of course, the date and time. What is not so obvious is that all your servers should use NTP to synchronize clocks, be in the same timezone, and log the timezone information (or log in UTC) so it’s evident when the event happened.
“Where” can a line in code, the name of the module, application, service, server or workstation, an IP address, an URL, or any relevant identifier.
“Who” can be a username, device name, IP address, or identification number. Note that you must consider the rules of what not to log.
Logging in all parts of the application should follow the same format, so it’s easier to parse the logs in a log management system, search the events and create alerts.
The art of alerting
The average time to detect a data breach in 2020 was 228 days, as stated in the Cost of a Data Breach Report 2020 by IBM. Such a long detection time means that an occasional glimpse at logs is not enough. In fact, you should be monitoring logs like a cake or cookies in the oven.
To catch security-critical events early, you should create alerts and send them to a monitoring team. According to Sematext’s Complete Guide to Metrics, Monitoring & Alerting, the purpose of log alerts is typically to collect essential information for a human who needs to start investigating the issue. In these cases, you should include the alert severity. Sometimes, you can also trigger automated actions.
Your log management system may have ready-made rules that create alerts. You can also create custom alerts based on your searches, for example.
Alerting rules require maintenance. If your rules are outdated or very strict, they may not catch all security-critical events. On the other hand, if your alert criteria are very lax and you get tons of false alerts, you may get dull with alert fatigue and miss the important events.
Chocolate chip raising cookies to watch over
I was yearning for chocolate chip cookies with raisins, so I combined a chocolate chip cookie recipe and a raisin cookie recipe (both recipes are unfortunately only in Finnish but Google translation looks readable). The baking times for the two recipes were slightly different, so I had to keep an eye on the oven.
To bake chocolate chip raisin cookies, you’ll need:
- 150 g pastry margarine
- 1 dl sugar
- 1 dl brown sugar
- 1 egg
- 3,5 dl wheat flour
- 1 tsp vanilla sugar
- 1 tsp baking powder
- 0,5 dl raisins
- 80 g chocolate
Beat the butter and sugar into a foam. Add the egg and mix well. Mix the flour, baking powder, and vanilla sugar and add in batches to the dough mix. Cut the chocolate into small pieces, and mix it into the dough together with the raisins.
Preheat the oven to 200 ºC (or 180 ºC if you are using a convection oven). Cover a baking tray with paper. Use two teaspoons to make tiny heaps onto the baking tray. Leave plenty of space in between because the cookies will spread. I used two baking trays and made 21 cookies.
Bake the cookies for 9-11 minutes; the exact time depends on your oven and how large cookies you have. The cookies can remain a bit soft because they will harden a bit as they cool. I kept them in the oven for 11 minutes and kept a close watch so that the cookies wouldn’t start getting too dark on the edges.
Maybe checking those log alerts is much more fun when you have a plate of these crunchy chocolate chip and raisin cookies on your desk?
Bake Security In 