Server issues - Aware Restart Needed

If you have questions or if you want to share your opinion about Aware IM post your message on this forum
Post Reply
eagles9999
Posts: 2253
Joined: Mon Jul 02, 2012 12:24 am
Location: Ulaanbaatar, Mongolia

Server issues - Aware Restart Needed

Post by eagles9999 »

I have a server that has been running quite happily for the last 4 years.

I have recently upgraded it to version 8.6 (latest build)

Since then, we are having issues with something freezing up and users not being able to log in.
The only way to fix is to restart Aware on the server.
It is happening between 2-4 times a day.

Even though it stops users logging in, I can see the Aware IM Log and the Tomcat log still running.

There is plenty of memory (6Gb for Aware and 6Gb for Tomcat)
There is plenty of hard disk space.
Hardly any load on the CPU or memory.

Tried running and a Service and from CP with the same issue.

I have no idea what the issue is. But, until I can pin it down, I need some way of either....

A) Being notified as soon as it happens so that I can RDP in and restart Aware

or

B) Some way on the server to recognise that Aware is not accepting logins and having Aware automatically restart.

Any ideas or suggestions would be hugely appreciated.
Cheers,
Mark
_________________
AwareIM 6.0, 8.4, 8.6 (build 2941) MariaDB, Windows 10, Ubuntu Linux. Theme: Default, Browser: Chrome, Edge
Upcloud
Image
UnionSystems
Posts: 187
Joined: Fri Jun 17, 2016 7:10 am
Location: Brisbane Australia
Contact:

Re: Server issues - Aware Restart Needed

Post by UnionSystems »

Hi Mark,

For issue (A) we use https://uptimerobot.com/ for monitoring public facing URLs. Primarily it checks on our clients websites but we do have it checking a public facing AwareIM server as well. It cheap and reliable.

Be interested in anyone’s suggestions for (B).
AWS Linux, Windows Server, AIM 8.4 & 8.6
Jaymer
Posts: 2080
Joined: Tue Jan 13, 2015 10:58 am
Location: Tampa, FL
Contact:

Re: Server issues - Aware Restart Needed

Post by Jaymer »

Yes dave, I do a similar thing. I don’t recall the service. I set up a REST endpoint and have some service out there that hits that endpoint and looks for data. He gets it every 5 to 10 minutes. It definitely will alert us. I’m not the one who set that up, but I could find more details, but it’s similar to what you posted.
Click Here to see a collection of my tips & hacks on this forum? Or search for "JaymerTip" in the search bar at the top.

Jaymer
Aware Programming & Consulting - Tampa FL
Jaymer
Posts: 2080
Joined: Tue Jan 13, 2015 10:58 am
Location: Tampa, FL
Contact:

Re: Server issues - Aware Restart Needed

Post by Jaymer »

Mark
Sorry, that is annoying.
One thing I have seen in my 8.6 tomcat logs is some Yahoo making a request that throws an error with a rest servlet.
In this particular server, I have nothing using rest and I’ve never set it up. I can’t tell from the log what data the offender is sending, it appears null/empty in the log.
Tim stamp shows that something did come in at the exact time of the error in the server/tomcat output.
But aware sees something in that request and then complains that the REST.props file was not found.

For me though, it’s not forcing a restart.
I think I posted a thread several months back that some guy posted something to me and it caused a tomcat issue, I’ll have to research that.
Version 8.5 or 8.6 moved us up to tomcat 9 and maybe some internal request is getting through that’s crashing tomcat.

I think it’s possible that if it is coming from a hacker, it’s something in the data stream that’s coming to your server doing this.
And that same data stream/exploit isn’t coming to others and that’s why we’re not seeing it across the board.
Click Here to see a collection of my tips & hacks on this forum? Or search for "JaymerTip" in the search bar at the top.

Jaymer
Aware Programming & Consulting - Tampa FL
Jaymer
Posts: 2080
Joined: Tue Jan 13, 2015 10:58 am
Location: Tampa, FL
Contact:

Re: Server issues - Aware Restart Needed

Post by Jaymer »

Click Here to see a collection of my tips & hacks on this forum? Or search for "JaymerTip" in the search bar at the top.

Jaymer
Aware Programming & Consulting - Tampa FL
rocketman
Posts: 1174
Joined: Fri Jan 02, 2009 11:22 pm
Location: Preston UK
Contact:

Re: Server issues - Aware Restart Needed

Post by rocketman »

Had an identical issue back in 7.9 and it looked for all the world like it was MySQL refusing the connection.

Tore my hair out for weeks, but when I went to 8.0 everything returned to normal. I know that’s not much help but it might just point to the fact that it might not be AIM or Tomcat or some corruption in Tomcat.

If memory serves, I think I upped the allowed connections in MySQL which mitigated the issue to days rather than 2-3 times a day, but I always had the feeling I’d just stuck a band aid on an abscess
Rocketman

V8.1 Developer Edition. Server 2012 Standard edition. MySql 5.5
Post Reply