The SecureByDefault Brief #011: Who's Been Knocking on My Server? How to Read Linux Logs Without Panicking

Proton Mail offers what others won’t:

We don’t scan your emails. We don’t sell your data. And we don’t make you dig through settings to find basic security. Proton is built for people who want control, not compromise.

Simple, secure, and free.

Explore Proton’s benefits

Last time I said we'd point grep at the logs of a real server. Here it is: my server, one week of SSH logs.

Nobody has ever been told this server exists. It has no website anyone visits and no one links to it. And yet, in the seven days before I looked, it recorded 53 failed password logins.

That's not a crisis. It's what the internet's front door looks like. Let's learn to read it.

Here's how the lesson started for me. I ran what every tutorial says:

Nothing came back. Not an error, just silence.

The records were there. They were in the systemd journal, not that file. On my server, auth.log existed but was tiny, because the service that writes it had only just started. Linux can keep logs in more than one place, and "no output" can simply mean "wrong place."

On most modern systems, start with the journal:

-u means "this service." On Debian and Ubuntu the SSH service is ssh. On many other distributions it's sshd. (On a system that keeps /var/log/auth.log, it's still worth a look. Just don't assume it's the only place.)

Here's a real line from my server, with the address partly hidden:

Left to right:

Once you can read one line, you can read ten thousand. You just need to ask the right questions.

A journal can be huge, and a command that reads all of it can take a long time. So limit the time window and save it to a file once:

(To read the journal you'll need sudo, or to be in the adm or systemd-journal group. Without access, a normal user can see -- No entries --, which looks like an empty log but is really a permissions problem. Same lesson as before: no output doesn't always mean nothing happened.)

1. How many failures?

Mine: 53.

2. Who's trying?

Mine: seven different addresses. Five of them made 9 to 12 attempts each. Those five account for 51 of the 53.

3. Which usernames did they try?

Mine: root was tried 23 times, so more than 4 in 10. After that came names like random, guest, test, and support.

That sort | uniq -c | sort -rn pattern is the most useful thing in this issue. It means "count the repeats, biggest first." You'll use it on every log you ever read.

The pattern looks automated. In one burst, an address tried eight different usernames in about thirteen seconds (developer, git, root, vyos, steam, and a few more). Nobody types that fast. I'd describe it as something scanning the internet and guessing common account names.

What it does not say:

The question that actually matters is whether anything succeeded:

On mine, the answer was a single line: me, logging in to run these very commands. If you ever see a login you don't recognize, that's the moment to stop reading and start responding.

Here's the part I didn't expect. Two settings decide how much of this noise has a chance of working:

sshd -T prints the settings your SSH server is actually using. On my server it printed:

That means anyone can try a password against the root account, which is exactly what those 23 root attempts were doing. Nothing got in, but I shouldn't be relying on luck. (OpenSSH's own upstream default for PermitRootLogin is prohibit-password, which blocks password logins for root. But when I spun up a brand-new Ubuntu instance to double-check, it printed the same two lines. A fresh server is not automatically a hardened one.)

So reading your logs does two jobs. It shows you what's knocking, and it sends you to check what the door allows. Key-based logins and no direct root password login are the usual direction. I'll walk through changing them safely in the next issue. Until you know how to test first, don't edit SSH settings on a server you can't afford to lock yourself out of.

On any Linux machine that uses systemd, no server required:

That shows error-level messages since the last boot. Pick one line and read it the way we read the SSH line: when, which program, what happened. If you do have a server, run the three questions above on the last 7 days.

You can learn all of this for free on a virtual machine on your own computer. But a real server on the internet teaches you things a VM won't, like how quickly the knocking starts.

When I want a VPS to learn on, or a production server, my preference is Akamai Cloud (formerly Linode) over AWS. It's simpler to get started with, the pricing has been easier for me to follow, the support has been good in my experience, and the documentation is excellent. It teaches you what you need to know, including a guide on setting up and securing a new server, which is exactly where we're headed next issue.

AWS is a strong platform and worth learning too, especially if you're aiming at cloud roles. This is just where I'd start a beginner.

Akamai Cloud | Akamai Cloud documentation

Put it after any command that prints one item per line, and you get a ranked list of what shows up most. It works on IP addresses, usernames, error messages, and file names. If one idea from this issue sticks, make it this.

Learning Linux from scratch?

The free Linux Starter Kit includes the commands, a practice plan, log examples, and a beginner server checklist.

Next up: the first things to lock down on a new server, before the knocking gets any further.

Talk soon,

Ron
SecureByDefault