Site Maintenance issues thread
- Thread starter neptronix
- Start date
We had some instability over the last 24 hours due to search engines bum rushing the server progressively more over the last couple days.
Basically... be careful what you ask for.. i invited search engines to the party a bit ago and our traffic is up about 80%.
The problem turned out to not be server overload but hitting a default in-memory session database size limit of 16mb.
That size has been cranked up to 128mb and i will monitor the situation over the coming weeks..
..we are thinking that xenforo is not properly maintaining this table and this is a bug in the software that seems to afflict very large forums like ours.
( PHPbb had it's own weird quirks just like this )
Sorry for the rocky road. This is our first summer on Xenforo and the last couple weeks have been a game of whack a mole with problems we've never seen before.
Basically... be careful what you ask for.. i invited search engines to the party a bit ago and our traffic is up about 80%.
The problem turned out to not be server overload but hitting a default in-memory session database size limit of 16mb.
That size has been cranked up to 128mb and i will monitor the situation over the coming weeks..
..we are thinking that xenforo is not properly maintaining this table and this is a bug in the software that seems to afflict very large forums like ours.
( PHPbb had it's own weird quirks just like this )
Sorry for the rocky road. This is our first summer on Xenforo and the last couple weeks have been a game of whack a mole with problems we've never seen before.
Good news..
Absolutely no errors in the logs or in the admin panel for a week. The database table xf_session_activity continues to grow but at a slower rate.. looks like the platform does not manage the size of this table like it should so we're tacking on a script to clean it once a week.
Absolutely no errors in the logs or in the admin panel for a week. The database table xf_session_activity continues to grow but at a slower rate.. looks like the platform does not manage the size of this table like it should so we're tacking on a script to clean it once a week.
Howdy.
We had some instability over the last 2h due to the new search system blowing out the CPU capacity.
I've done surgery on the server and we've bumped up the performance of search *a lot*
Should be stable at this point... fingers crossed
We had some instability over the last 2h due to the new search system blowing out the CPU capacity.
I've done surgery on the server and we've bumped up the performance of search *a lot*
Should be stable at this point... fingers crossed
Search has been unreliable!
On saturday, elasticsearch blew up and didn't recover. I tuned some things, and went from elasticsearch blowing up every 2 days to every 3.
It appears elasticsearch has a ram/cpu spike every X days that causes this, i've seen this on 2 other systems running elasticsearch.
Solution: elastic search gets rebooted at 1AM pacific standard time from now on, and will be down from 1:00am-1:01am
On saturday, elasticsearch blew up and didn't recover. I tuned some things, and went from elasticsearch blowing up every 2 days to every 3.
It appears elasticsearch has a ram/cpu spike every X days that causes this, i've seen this on 2 other systems running elasticsearch.
Solution: elastic search gets rebooted at 1AM pacific standard time from now on, and will be down from 1:00am-1:01am
I guess I've been lucky; I haven't seen it blow up since the first time.
Now if we could just add some features to the search function....
Now if we could just add some features to the search function....
FYI, search blew up on august 8th and had an outage for different reasons than before.
Well that was a nice month long stretch of elastic search reliability at least
I have what i hope is a successful mitigation in pace, but there are 3 different other things to try.
It will be a tune and test affair, not a slam dunk to fix this.
From what i can tell, the problem is elasticsearch updating a geolocator IP address database, which causes the memory to get exhausted ( sad because we have a 3-4gb overhead )
Expect search to have ~90% uptime until i have this issue licked.
Well that was a nice month long stretch of elastic search reliability at least
I have what i hope is a successful mitigation in pace, but there are 3 different other things to try.
It will be a tune and test affair, not a slam dunk to fix this.
From what i can tell, the problem is elasticsearch updating a geolocator IP address database, which causes the memory to get exhausted ( sad because we have a 3-4gb overhead )
Expect search to have ~90% uptime until i have this issue licked.
Last edited:
