SSH breach and attempt to use server

**hairygunther** · 06-27-2009, 03:07 PM

I found a cracker breaching my server, netstat showed he/she was on SSH, and other readouts showed they were using perl to send mail via my server. I kicked them off and locked them out.

But how in crap did they breach ssh? I mean is ordinary brute-forcing of an ssh login possible? After a few failed tries wouldn't that machine be locked out?

Does anyone know what the likelihood is that they got in via my password? they were logged in as root on ssh. Lord knows why they were stupid enough to start their evil work before locking me out? I suppose they thought I wouldn't see them, so they thought they could do loads of it and get away with it by not arousing my suspicion.

Blighters. Anyone know the technicalities surrounding this issue? Please pour out as much technical info and jargon as you want - I'm keen to know how the muthas got into my machine.

**tripy** · 03:45 PM

I personally uses denyhosts. http://denyhosts.sourceforge.net/
I have configured it as that after 2 failed tries from the same ip, this ip is totally blacklisted from the server.
Even an http connection will be refused.
It's a daemon that analyzes the logs live, and add the ip's in hosts.deny.
It can even send stats and a list of blocked ip to a central server, and you can download those lists too.
So, the peoples trying to brute force one place can be blocked even before they attempt to attack you.

Another thing you can do, is to disable the password auth in ssh, and use only certificates.
You run "ssh-keygen -t dsa" from the client station, and give a password to your key.
This will create your ssh key. It's composed of 2 files: the private key, and the public key.

Always keep your private key secret.
Your public key (~/.ssh/id_dsa.pub) can be uploaded on the server you want to log in.
Simply copy it's content into the keystore of your user on the server(cat id_dsa.pub>>~/.ssh/authorized_keys).

Open a new terminal, and try to login, you should be authenticated by the public/private key duo, and no password should be asked.

This allows you to deactivate the password login from ssh ("PasswordAuthentication no" in /etc/ssh/sshd_config).

**hairygunther** · 04:31 PM

i have pm'd this post to you, so that the cracker can't check my present operational status, on the off chance that it's a personal attack rather than some generic factory productionline korea/india/mid-east/usa based almost entirely robotic attack, in which i'm just another number on a list of countless targets being tried and tested round the clock

**hairygunther** · 07-04-2009, 06:11 AM

I have only managed to contain the problem now, last night I managed it - the hacker returned from different IPs, in Korea and China, and was managing to access my server via SSH whether or not I was logged in, no matter how many times I changed the password, etc

I have not really understood how they really got in yet, and if you have any ideas about how to investigate that, please let me know. I do know that the hacker got fed up of me booting him off so he switched off my ssh daemon himself, to prevent me getting back in - however being a fan of Dr Who, hours before the hacker did this to me I thought of a few measures to take in case of such an attack, and thank cow I did, because otherwise my server (and its profits) would have to go into a few weeks highly undesirable downtime.

But where in hell do I look? If the hacker was willing to switch off sshd completely, how were they getting in? The point is that they were unafraid to disable ssh, it helped them get me out the way while they did their work - so how did they get in?

I have been reading the linux o'reilly book all yesterday and I'll be off to read loads more of it today, so given enough hours, days, weeks, I'm sure I'll have all the answers myself, but even so, unless your surname is Beeblebrox, two heads (or more) is better than one.

Any more ideas?

**tripy** · 07-04-2009, 08:29 AM

Try to update all your system.
They could use a bug in a program that allows them to execute arbitrary code due to a stack overflow.

**hairygunther** · 09:28 PM

update - i had some trouble before, but it worked, and as far as i can see it didn't need any further changes other than what you instructed - strange about that delay to begin with... any idea why? it took 10 to 15 mins before i could use it.

so i have not set pubkeyauthentication to yes after all.

update2 - i now know why the delay happened - the primary failsafe didn't go off properly, somewhere a file had improper permissions from the last time i used the failsafe (after it's used once it shuts itself off)

original version of this post...

Quote:

Originally Posted by tripy

I personally uses denyhosts. http://denyhosts.sourceforge.net/
I have configured it as that after 2 failed tries from the same ip, this ip is totally blacklisted from the server.
Even an http connection will be refused.
It's a daemon that analyzes the logs live, and add the ip's in hosts.deny.
It can even send stats and a list of blocked ip to a central server, and you can download those lists too.
So, the peoples trying to brute force one place can be blocked even before they attempt to attack you.

Another thing you can do, is to disable the password auth in ssh, and use only certificates.
You run "ssh-keygen -t dsa" from the client station, and give a password to your key.
This will create your ssh key. It's composed of 2 files: the private key, and the public key.

Always keep your private key secret.
Your public key (~/.ssh/id_dsa.pub) can be uploaded on the server you want to log in.
Simply copy it's content into the keystore of your user on the server(cat id_dsa.pub>>~/.ssh/authorized_keys).

Open a new terminal, and try to login, you should be authenticated by the public/private key duo, and no password should be asked.

This allows you to deactivate the password login from ssh ("PasswordAuthentication no" in /etc/ssh/sshd_config).

What about "known hosts"? Any need to update that somehow?

Also, first time I tried your instructions I was locked out. Further inspection of the issue suggests that maybe there is more to change in the sshd_config than just that passwordauthentication doohicky. Luckily I'd put a timelord-failsafe on the server so I can get back in and try again without someone having to go and physically reboot the machine!

By the look of it I may also need to do this, once I get back inside my server!

Quote:

PubkeyAuthentication yes

It worked fine, no password, when I did it before restarting the sshd having changed the passwordauthentication as instructed.

But after that it just froze up when I tried to log in. It asked for my password as normal at my end, for my new wotsit, but then it just sat there. i.e. my passphrase.

Is it supposed to take a long time? I assumed not, after I'd waited a few minutes, so I'm currently trying to get back in, but I'm not entirely able to at the moment. One of my failsafe mechanisms isn't 100% functioning, although it may be sorted out (in which case I'll be back inside in 4 minutes), otherwise there's another failsafe which will happen in about 3 hours anyway, so I suspect I won't have too much trouble getting back in.

Any idea why it just sits there after I put in my passphrase?

**hairygunther** · 08:16 AM

Remotely working on delicate parts of a commercial server is a scary business when you don't have much experience of highly delicate stuff!

But I'm getting the hang of it the more I do.

My home-made patrolling system caught numerous further ssh breaches in recent days even after switching to non-password authentication, ie via public/private key wotsit thingy.

They came from Panama, Chile, Indonesia, Korea, China and even, yes, even the USA (and the spammers filling my cgi forms with rubbish were from Russia, so far). Apart from the USA I've banned all the other offending/offensive countries from my server! If Google were running its crawlers from the uk, and if vast swathes of UK isps didn't end in .com's and risk being seen as American, I'd probably block out the USA too!

Naturally I then used some advice Tripy gave me and examined known security holes in openssh and decided to upgrade my openssh to a version without those known vulnerabilities. That was the scariest part - replacing your openssh remotely is something you hesitate over the first time! But I did it and it was successful.

So now I have to sit and wait another day and see if my patrolmen stop finding german parachuters wandering around my english coast!

Ah the Luftwaffe.

Just before I successfully updated openssh, I was "this close" to deciding i should just pay one of the close talented friends I have in the linux world to take over the technical side from me and I'll retire to sales, relaxing and romance. But the dream didn't last long. I rapidly succeeded in the technical task and realised, in the end, that this somewhat painful way of life is mine for good. Still, it is very rewarding. As you know I just love programming and all this stuff I do. True, sometimes it's so difficult and anxiety-ridden that you wanna give up and let some other poor chump do it. But the excitement of being a military tool, a precision guided human weapon, a hacker... is what gives you the motivation to keep on - one day I'll literally "know it all"... but until that day, 45 years from now, i guess I'll keep learning! And then I expect they'll replace Linux after that and I'll have to start from scratch.

**hairygunther** · 08:54 AM

Quote:

tcp 0 52 woohoo.co.u:ssh ::ffff:190.34.149.232:44395 ESTABLISHED
tcp 0 0 woohoo.co.u:ssh 190.34.149.232:34763 SYN_RECV
tcp 0 0 woohoo.co.u:ssh static-64-61-143-29.i:54917 TIME_WAIT
UPDATED TO (censored)
so ssh breaches may stop now
tcp 0 21 woohoo.co.u:ssh 66-76-108-157.laft.su:57182 LAST_ACK

that chap came in from Paris Texas, it seems (a quick whois from the commandline reveals)

anyway, Tripy, WCFB... do you know if that LAST_ACK means the mutha didn't get through thanks to the tougher openssh that's now in place?

re the ssh thingy after my new security went into place,
what does "LAST_ACK" really mean. Some preliminary investigation on my part suggests it's to do with my server telling the intruder that he can't come in... is that correct? I guess if I wait long enough I'll see if there end up being any ESTABLISHED ones
from some of my security logs

**hairygunther** · 09:30 AM

btw I added a user I wanted to add to the wheel, and I tried to add it to the sudoers, by visudo, but even though i can insert the username, and use :q! to quit and save, when i reopen it, the changes are not there, and the user does not have su priveleges yet... any instructions, tripy?

I also tried uncommenting the %wheel thingy - but no change i make to the sudoers wotsit sticks, and yes i am logged in as root!

update - that prob is solved..
right. got it, i needed to do :w before :q!
now i'm managing to save changes...

yes, it worked. my other user now has su priveleges. so i suppose i'm ready to get rid of root's ssh permissions! how would you recommend i do that? merely "PermitRootLogin no", perhaps?

(plus also the other vital thing i'm keen on knowing is what LAST_ACK really means)

cheers

**grass321** · 07-06-2009, 09:16 AM

your idea is very good ,you should put it into practice ,i think it will attract more visitors

**hairygunther** · 07-06-2009, 09:26 AM

I'll make a note that you want to buy a copy of my English Comprehension software (for training people on comprehending things they read in English) when it's released next decade.

**Lashtal** · 07-10-2009, 12:22 AM

Re: LAST_ACK

http://www.outpostfirewall.com/forum...read.php?t=187

---

Were you able to tell whether or not you received a confirmation of the connection's close?

from what i've read on Google, (which has lead me to several forum posts), LAST_ACK is a connection within itself that comes *prior* to receiving confirmation of a close between your server and the IP('s) in question.

it deals with you sending a message to the IP to close the connection, and that IP's response to acknowledge your request. From what I have also read, you may wish to confirm that a close was actually initiated, because the connection can hang on without a response (wasting resources).

---

someone with a similar question here, wondering how to close his connection from the LAST_ACK state: http://lists.apple.com/archives/macn.../msg00103.html

and people with some more insight into why and what this is at all (which should give some added insight about it): http://cobalt-users.1stserv.com/msg62120.html

this might give you some insight to see what CMD to run, to see if and how many LAST_ACK connections are presently open on your server (and what some people recommend): http://forums.theplanet.com/index.php?showtopic=3569

for good measure: http://itmission.org/Main/Netstat

any confirmation at all that has led you to believe LAST_ACK was closed?

**hairygunther** · 07-10-2009, 06:00 PM

The last_acks I spotted did not last very long - none of them was still there after 5 mins. The reason for them would probably be that now I have a nonvulnerable openSSH, people using the previous hack or informed of it are trying the connection and being told to take a hike.

Definitely no more open connections. And no further trouble on my server. All nicely under control, but now with the added advantage of aggressive inhouse monitoring methods.

It was on netstat that I found it in the first place, don't worry.

cheers

**hairygunther** · 07-12-2009, 05:20 AM

Quote:

Originally Posted by grass321

your idea is very good ,you should put it into practice ,i think it will attract more visitors

I've been given an"infraction" for calling grass321 a spammer!

Can anyone believe this?!!

How come grass321's posts aren't being removed? They are total meaningless spam? What's the plot here?

Anyway, if you don't want me to post here fair play - I guess talented I.T. experts like grass321 need their opinion spread to people like you asap rather than pointless unrelated garbled crap like mine!

**hairygunther** · 07-14-2009, 03:08 PM

My amateur security patrolling system has picked up many more attempted breaches, but I checked inside the secure files in the logs and discovered that actions taken to keep the muthas out have worked and these attempted breaches are being noted as false and not permitted.

Good thing I've built my patrolling system, it means I can see quite a lot of what people attempt to do, even though they fail.

One was in North Carolina - I don't know if these are the criminals who tried to hack me or whether they are just mugs whose server has been hacked and used as a platform to launch attacks, but this is the level of detail I have about the hackers...

Quote:

network:Org-Name:Lawrence Realty Co
network:Street-Address:1135 Kildaire Farm Rd
network:City:Cary
network:State:NC

There you go. Just goes to show. Strong security is just as easy to ensure as hacking is easy to do! All other attempts to hack me came from South America and the Far East. Many countries. Of course it's not possible just from the surface data for me to know whether they were launched through proxies or other "false bases" so to speak and originated elsewhere. I am happy to note that there have been NO ATTACKS FROM BRITISH MACHINES... perhaps this may turn out to be a civilized nation after all.

Anyway, the moral is - do add as many levels of "amateur" security as you can because clearly it can give you a lot of strength.

**hairygunther** · 07-14-2009, 03:39 PM

I checked their site - Lawrence Realty in NC has used "Weblink India" in delhi to do their webdesign. I strongly suspect that Lawrence Realty did not hack me but in fact have hackers walking in and out of their server space somehow.

Just goes to show - don't go for the glossy, even cheap web design. Use intelligent artful coders who know what they're doing (there's much more important stuff than looking nice and generic, like everything "our" society pats on the back).

**Lashtal** · 07-27-2009, 11:01 AM

Quote:

Originally Posted by hairygunther

I've been given an"infraction" for calling grass321 a spammer!

Can anyone believe this?!!

How come grass321's posts aren't being removed? They are total meaningless spam? What's the plot here?

Anyway, if you don't want me to post here fair play - I guess talented I.T. experts like grass321 need their opinion spread to people like you asap rather than pointless unrelated garbled crap like mine!

lol

I was wondering where you went.

Personally i'd like to see more people like you here than spammer, for sure. Just tell a mod, perhaps VanGogh (he's cool) about the situation and inform him. There's a great chance they did this on accident, or even half-accidentally- as this board does get A LOT of spam.

---

regarding the LAST_ACK situation, glad you got it under control and all connections closed.

Regarding what you suspect about that realty company, I would imagne you're on the money with your hypothesis. And if you achieve persistent attacks from their direction would you inform the people who pay the site's webmaster and update them of the situation? Just wondering what your protocal is on that.