Jump to content

Check us out:

Photo
* * * * * 1 votes

Bunker 1 lag issues (18/10/2013)

recent server lag october 18 2013

  • Please log in to reply
93 replies to this topic

#21
Red

Red

    On The Road To Success

  • Bunker Members
  • 208 posts
  • Gender:Male
  • Location:Massachusetts
  • Server:Bunker #10
Click to view battle stats

Yep unplayable for sure.



#22
Jecoliah

Jecoliah

    I Come Here Often

  • Bunker Admins
  • 3,297 posts
  • Gender:Male
  • Location:Texas
  • Server:Bunker #1
Click to view battle stats

Sudd3n is checking into it.

 

 

 

Jec



#23
neurosis

neurosis

    Will Become Famous Soon Enough

  • Bunker Members
  • 1,776 posts
  • Gender:Male
  • Location:Seattle, wa
  • Server:Bunker #1
Click to view battle stats

It was very bad again tonight.  I dont want to be the whiner (ala sneak) but people scattered to other servers due to lag.



#24
pumperjp

pumperjp

    Postin Ain't Easy

  • Bunker Members
  • 583 posts
  • Gender:Male
  • Location:Florida
  • Server:Unselected
Click to view battle stats

Went from 20+ to 6 in a matter of minuets. Unplayable!



#25
neurosis

neurosis

    Will Become Famous Soon Enough

  • Bunker Members
  • 1,776 posts
  • Gender:Male
  • Location:Seattle, wa
  • Server:Bunker #1
Click to view battle stats

A few of us stayed as long as we could.  I think that Andy, Winny and I were the last to leave. It was totally unplayable. Pumper is right. It cleared out very fast.

 

I hope someone is paying attention. This is going to kill the server fast.   ;(



#26
peyote

peyote
  • Gender:Male
  • Location:Basel
  • Server:Bunker #2
Click to view battle stats

Got now full access to server since a few hours ... il be on TS, if anyone wants to share a thought.. :)

 

Hopefully I can fix it.


  • pumperjp likes this

#27
neurosis

neurosis

    Will Become Famous Soon Enough

  • Bunker Members
  • 1,776 posts
  • Gender:Male
  • Location:Seattle, wa
  • Server:Bunker #1
Click to view battle stats

Let us know what you find Pey.  This has been clearing the server for a few weeks now.  Not sure how many weeks of this can go on before it affects the server population.

 

Thanks!



#28
ontheqt

ontheqt

    Postin Ain't Easy

  • Bunker Members
  • 584 posts
  • Xfire:shrikester
  • Gender:Male
  • Location:California
  • Server:Bunker #1
Click to view battle stats
Hope this gets fixed before I get back.

#29
Jecoliah

Jecoliah

    I Come Here Often

  • Bunker Admins
  • 3,297 posts
  • Gender:Male
  • Location:Texas
  • Server:Bunker #1
Click to view battle stats

Now that Peyote has Root access he removed the block and we now can see B1 on TrackBase Again!

And Now, when the lag issue occurs, he will have access and the tools to track it down and hopefully put and end to this!

 

 

 

Jec


  • leaDpoisAn likes this

#30
Mongo

Mongo

    Sticking Around

  • Bunker Admins
  • 1,063 posts
  • Gender:Male
  • Location:Earth
  • Server:Bunker #2
Click to view battle stats

At the B1 Party right now and people talk abot lag. I dont see mutch more then normal but if you see this Peyote it whould be great to check it out. They call for the Uber server Admin Peyote.



#31
peyote

peyote
  • Gender:Male
  • Location:Basel
  • Server:Bunker #2
Click to view battle stats

The server has been under a 'chargen' ddos attack.

Or maybe b1 was amplifying it - to attack someone else, dunno.

http://www.iss.net/s..._of_Service.htm

 

Unless we get a massive ddos attack - which just kills the network going to b1, it should not happen again!

 

Yep b1 is again under same attack, looks like it was luck yesterday that it stopped. Investigating -


Edited by peyote, 02 November 2013 - 05:47 AM.


#32
Bean

Bean
  • Tb:1889806
  • Gender:Male
  • Location:Upper Canada
  • Server:Bunker #1
Click to view battle stats

We appreciate your work on figuring this out, pey.



#33
peyote

peyote
  • Gender:Male
  • Location:Basel
  • Server:Bunker #2
Click to view battle stats

Lag has cleared up :D

New land in sight!


  • Sakura likes this

#34
Mongo

Mongo

    Sticking Around

  • Bunker Admins
  • 1,063 posts
  • Gender:Male
  • Location:Earth
  • Server:Bunker #2
Click to view battle stats

Danke



#35
Ace

Ace

    Becoming a Part of the Forum

  • Bunker Members
  • 307 posts
Click to view battle stats

Thanks Pey!



#36
ontheqt

ontheqt

    Postin Ain't Easy

  • Bunker Members
  • 584 posts
  • Xfire:shrikester
  • Gender:Male
  • Location:California
  • Server:Bunker #1
Click to view battle stats
Good job. Thanks Pey!

#37
kael

kael

    Becoming a Part of the Forum

  • Bunker Members
  • 270 posts
  • Gender:Male
  • Server:Bunker #1
Click to view battle stats

Good work Pey! How'd you mitigate?



#38
Forrest

Forrest

    Postin Ain't Easy

  • Bunker Members
  • 529 posts
  • Gender:Male
  • Location:Southern Jersey
  • Server:Bunker #10
Click to view battle stats

Well done! Peyote



#39
peyote

peyote
  • Gender:Male
  • Location:Basel
  • Server:Bunker #2
Click to view battle stats

Thanks for all the thanks :)

 

Let us know what you find Pey.  This has been clearing the server for a few weeks now.  Not sure how many weeks of this can go on before it affects the server population.

 

Thanks!

 

Good work Pey! How'd you mitigate?

 

I had first to learn a lot about this, I already had configured firewalls.. but that was more following a guide then exactly knowing what I am doing.

I already knew something is going on before I had gotten root, I mean it could had been a faulty network card, a network problem at interserver or something else.

This command spit out a very lot of IP's, and none of those where playing on b1. A short search revealed they are mostly from china and india.

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)

 

So once I had root, I tried same, but I had still no clue what is happening.  I googled a bit and found suggestions to use iftop.

Once I had started iftop, I could see that tons of IP's where coming in from port chargen; tbh I still wonder how to make iftop show the portnumber. Anyway chargen is a service running on port 19. Which is a new way to DDOS apparently.

But I don't know if B1 was the target, or if B1 was amplifying it to target someone else. Probably b1 was attacked, as I could not find a chargen service running on b1 - therefore b1 could not have amplified it  - I guess.

 

Now I had to block port 19 - and that is what is confusing me.  Because at that time I had already reconfigured the firewall. Originally it was let through everything, and block certain stuff. So port 19 was open. I changed that to block everything and only let ET, ssh, and ping through. So how the heck did the port 19 attack get through??

Well that is where I have really a lack of knowledge. I don't know if the attack went through or not. All I know is someday before I had around 60% packet loss, and could hardly do anything over ssh (everything took like 1-5mins to show up). Yesterday I had around 20-30%packet loss (and it took around 10s-1min). And once I modified the Firewall to first of all drop everything incoming from port 19, the ddos had stopped around 10-20 mins later. Before that rule did not exist, So I suppose the ddos packages went through the whole rule chain of iptables - and I think some of them even got through.

 

The thing is, once again I don't know, if the ddos just stopped. If Interserver made something, or if my firewall was working.

 

Now I still saw about an hour later an ip from russia still doing the ddos from port 19 - so I guess interserver did not block it. Oh and about IP's and their origin, this is all going over UDP - therefore IP's can be spoofed - the source of the attack could had been a single server anywhere you like - as far as I understand.

 

 

Some more technical infos:

I used a guide over here, http://wiki.centos.o...etwork/IPTables for the basics.

This was the first rule originally, and I suppose this is what let port 19 through - as they had already been getting through - I suppose this line might have decided to continue let em through - I dont' know yet.

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

 

Around 10-20 mins after executing this one, the ddos stopped, this -I INPUT 1 makes it get inserted as first rule in the INPUT table. So port 19 got really dropped no matter what or who. And also as first. I think that makes a difference, because over 100'ips had been connecting to b1. And the earlier the get dropped the better I guess.

iptables -I INPUT 1 --sport 19 -j DROP

 

Strange enough is, that iftop continues to show those coming in from port 19 (chargen). As far as I know they are indeed coming in knocking at b1's network card - but at least they got now dropped very fast.

 

Yer thats more or less all.


  • neurosis and *CapTn* like this

#40
G0rt

G0rt

    Galactic Policeman

  • S Moderator
  • 3,599 posts
  • Steam:G0rt2002
  • Gender:Male
  • Location:Diagonal from VY Canis Majoris
  • Server:Bunker #10
Click to view battle stats

interesting.  very interesting.  sounds like a variation on the get_status flood attack we had experienced a year or so back.  B1 just may have been an un-likely target but being udp packets (connection-less) seems more likely it WAS targeted from an earlier scan.  Nice little write up on chargen dos here.







Also tagged with one or more of these keywords: recent, server, lag, october, 18, 2013