1
0

Site was down, maybe attacked


 invite response                
2021 Oct 19, 10:45am   611 views  46 comments

by Patrick   ➕follow (55)   💰tip   ignore  

No response to pings, couldn't ssh in. Nothing.

I was able to reboot from the ISP's control panel and manually restart processes.

I think it was attacked. Looking into it.

Comments 1 - 40 of 46       Last »     Search these comments

1   Karloff   2021 Oct 19, 10:48am  

Be sure you keep frequent backups of the site which are directly in your possession. You can't trust anyone these days and I've seen many cases where people have lost everything they've uploaded when the provider decides to pull the plug on them for some contrived reason.
2   Patrick   2021 Oct 19, 10:49am  

Yes, I have backups directly in my possession. I will increase the backup frequency and look for alternate providers. Maybe it is indeed time to host it from home.
3   Ceffer   2021 Oct 19, 10:57am  

If you lose the 'Wit and Wisdom' of AF archives, can we ever forgive you?
4   Ceffer   2021 Oct 19, 10:57am  

Yeah, something is happening right now. The site isn't framing correctly.
5   Patrick   2021 Oct 19, 10:59am  

Tell me more. It looks OK to me.
6   Ceffer   2021 Oct 19, 11:02am  

It's OK now. Maybe just my computer. I think the internet is gumming up with crap. I have had to restart my computer about ten times today.
7   Patrick   2021 Oct 19, 11:04am  

Digital Ocean has zero phone support now. You literally cannot call them, you can only file a ticket. Not good service at all.

Most recent email I got from them said there would be maintenance on Oct 6th, but now it's the 19th, so that shouldn't have been it:

Start: 2021-10-06 07:00:00 (UTC)
End: 2021-10-06 15:00:00 (UTC)

Hello,

We have identified a potential issue with the physical machine(s) hosting your Droplet(s) listed below. During the above window, we will be performing migrations of all affected Droplets to new physical machines, to ensure no unexpected disruption to service.

In order to minimize downtime, we will attempt to perform live migrations in all possible cases. A live migration would result in no downtime, but minor performance decreases in disk I/O and a second or less of packet loss as the network is switched over to the new physical host.

In the event that we are not able to perform a live migration of a Droplet, we will perform an offline migration during which the Droplet will be powered off and migrated offline during the window.
8   MAGA   2021 Oct 19, 11:24am  

Maybe it was the NAR.
9   zzyzzx   2021 Oct 19, 11:43am  

MAGA says
Maybe it was the NAR.


Zillow but hurt over their recent flop.
10   Tenpoundbass   2021 Oct 19, 11:47am  

Patrick says
you can only file a ticket.


This is why I'm in no hurry to get back into Enterprise development. 9 months now just loafing at home, doing some support work here and there for my previous company.

In the end they were doing everything in a ticket based system. It was stupid and a complete waste of my time. We would be in a meeting, and someone would ask me a technical question, and I would satisfy their quest with the correct answer. But still a ticket would get created for it. People would email me a question or request, and I would just whip it out and do it on the spot, but a ticket ended up in the ticket system. More often than not the text in the request is useless figure out what was wrong. I was getting constantly nagged about clearing out my tickets. I just kept telling them, if you don't like seeing tickets in the system, then stop making frivolous tickets, and assigning every catchall ticket to me, that you don't understand the question or what transpired during the communication.

I refused to clean it out, I had my own ticket, I was developing applications, but still fielding questions and quick one off requests. I'm not going to go soak and marinate in a convoluted ticket system. That I waste huge amounts of time on tickets, only to find out, those were quick questions and answers, and were no longer an issue.

Nontechnical people Suck balls, they don't understand shit, but they demand to be in the center of the whole technical universe. I don't have the stomach for them, and will call their stupid asses out in meetings. Yeah I needed a break from the enterprise.
11   mell   2021 Oct 19, 11:56am  

Tenpoundbass says
Patrick says
you can only file a ticket.


This is why I'm in no hurry to get back into Enterprise development. 9 months now just loafing at home, doing some support work here and there for my previous company.

In the end they were doing everything in a ticket based system. It was stupid and a complete waste of my time. We would be in a meeting, and someone would ask me a technical question, and I would satisfy their quest with the correct answer. But still a ticket would get created for it. People would email me a question or request, and I would just whip it out and do it on the spot, but a ticket ended up in the ticket system. More often than not the text in the request is useless figure out what was wrong. I was getting constantly nagged about clearing out my tickets. I just kept telling them, if you don't like seeing tickets in the system, then stop making frivolous tickets, and assig...


Haha agreed, bogus tickets and bogus metrics based on those bogus tickets are a total waste of time
12   Patrick   2021 Oct 19, 11:58am  

I think it is probably time to really try to host the site from home.

The big issue is the cost of upload bandwidth.
13   richwicks   2021 Oct 19, 12:26pm  

Patrick says
I think it is probably time to really try to host the site from home.

The big issue is the cost of upload bandwidth.


Do you PAY for upload bandwidth?

I'm saying you should have it setup so you CAN run locally. I don't suggest it as a permanent solution.
14   Patrick   2021 Oct 19, 12:31pm  

Don't you pay for your ISP service? I just mean that $65/month is getting me only about 1.8 mbps upload bandwidth from home, and that's not enough to run the site.

I do want to run the site from home eventually. If I could get reliable and cheap upload bandwidth from home (meaning the download speed that you all get when using the site) then the only downside I see is being interrupted when moving. And I'm definitely thinking about moving out of CA.
15   richwicks   2021 Oct 19, 12:40pm  

Patrick says
Don't you pay for your ISP service? I just mean that $65/month is getting me only about 1.8 mbps upload bandwidth from home, and that's not enough to run the site.


I bet it is. In fact, I'd reduce the bandwidth to 1 mbs, and you should be OK. You can rate limit with a program called trickle.

All you need to do is open port 80 and 443, get some website name from www.dynu.com then use let's encrypt to setup a certificate. I think you already use let's encrypt so the only new thing is using www.dynu.com
16   PerfectlyFlawed   2021 Oct 19, 12:50pm  

Oh no - the illuminati control structure strikes again! They dont like people outside of their perview..
17   Automan Empire   2021 Oct 19, 12:52pm  

I'm getting ready to migrate my tiny business website to a discount host. For YEARS I let an outfit called Hibu run it at an uncompetitively high price, because I'd rather leave it alone than F with it. Then this pushy alcoholic-looking saleswoman called me recently telling me the "old website" is going to be taken down because they're migrating to a new backend platform. They were up to like $110/month and wanted to bump this to $125+/mo. I told her it was already uncompetitive and the only reason I hadn't switched is because I can ignore it. If I have to mess with it, even to orally approve an increase in the price with one word of effort, then it becomes worth switching.

She obliviously tried to hard sell past this, telling me how "they have me on 53 websites and all this SEO bullpucky and they're updating everything for ADA requirements because of text readers for the blind...

I cut in again, saying my site is organically at the top of google search results because I am tops in my niche, and this is with 4 static pages without a keystroke of deliberate SEO effort. She tried to push the ADA issue, saying that text readers have problems which I cut off again to say I RUN AN AUTO REPAIR SHOP WOMAN, HOW MANY VISUALLY IMPAIRED POTENTIAL CUSTOMERS DO YOU REALLY THINK I'M LOSING TO TEXT READER COMPATIBILITY ISSUES IN 2021 REALLY?!?

I'm getting ready to spend half a day setting it up on a discount shared server, where 2 months of Hibu's billing will pay for 3 YEARS of hosting my dick-simple website.
18   richwicks   2021 Oct 19, 12:55pm  

Automan Empire says
I'm getting ready to spend half a day setting it up on a discount shared server, where 2 months of Hibu's billing will pay for 3 YEARS of hosting my dick-simple website.

You're the first small business owner that I've ever run into that appears to favor the DNC.
19   Patrick   2021 Oct 19, 12:58pm  

Automan Empire says
I'm getting ready to spend half a day setting it up on a discount shared server, where 2 months of Hibu's billing will pay for 3 YEARS of hosting my dick-simple website.



I think you could host such a simple site for $5/month on https://www.linode.com/pricing/
20   Patrick   2021 Oct 19, 1:01pm  

richwicks says
I bet it is. In fact, I'd reduce the bandwidth to 1 mbs, and you should be OK. You can rate limit with a program called trickle.

All you need to do is open port 80 and 443, get some website name from www.dynu.com then use let's encrypt to setup a certificate. I think you already use let's encrypt so the only new thing is using www.dynu.com

OK, I'll do some experiments today. I have an always-on laptop I can use to host it.
21   richwicks   2021 Oct 19, 1:11pm  

Patrick says
OK, I'll do some experiments today. I have an always-on laptop I can use to host it.


If you have virtualbox installed on your machine, I suggest using that.
22   Tenpoundbass   2021 Oct 19, 1:24pm  

Patrick says
I think you could host such a simple site for $5/month on https://www.linode.com/pricing/


I'm extremely happy with Host Gator they got me off of Network Solutions, and migrated over everything perfectly and even my mailboxes.
Their tech support were Johnny on the spot and very helpful. For simple questions you might get an Indian in a call center somewhere. But when the metal hits the road and you need to talk to a technical person. You get people named Jeff or Skip in Texas and they really know their shit. None of this being put on hold or given a ticket number, while Rashika goes and relays your issue with the people that know their shit. They put you through directly to those resources.
23   Patrick   2021 Oct 19, 1:48pm  

richwicks says
Patrick says
OK, I'll do some experiments today. I have an always-on laptop I can use to host it.


If you have virtualbox installed on your machine, I suggest using that.


I've used it, but find it to be a pain, just another layer of cruft to deal with.

Ideally, I'd have an a laptop that just has the same linux I have now on my server so I could migrate without recompiling anything or re-installing any pacakages, just a mass copy.
24   richwicks   2021 Oct 19, 3:05pm  

Patrick says
richwicks says
Patrick says
OK, I'll do some experiments today. I have an always-on laptop I can use to host it.


If you have virtualbox installed on your machine, I suggest using that.


I've used it, but find it to be a pain, just another layer of cruft to deal with.

Ideally, I'd have an a laptop that just has the same linux I have now on my server so I could migrate without recompiling anything or re-installing any pacakages, just a mass copy.


The reason I suggest a virtual machine is that it's really easy to archive - just shut down the machine, and tar -czf archive.tgz [directory] - another thing I use it for is when I'm doing a new setup, I prefer to do it one a machine I don't worry about screwing up, before I move it to my real machine. For example, when I'm compiling libraries that have to be installed on my base system.

NOTE: Virtualbox MUST be installed not from your package manager but from this site:

https://www.virtualbox.org/

There's two parts, the virtual box program itself, and the VirtualBox 6.1.28 Oracle VM VirtualBox Extension Pack - they have to be downloaded at the same time. The Extension Pack allows you to resize the screen, use USB 2.0/3.0 - stuff you take for granted, if you don't install the extension pack, you have a machine, but you want to ssh into it, and you won't want to work directly on the display (I think it's like 800x640 of something in screen size).

Virtual machines are VERY USEFUL to segregate work and experiment with new versions of operating systems. I've run AROS on mine - which is a re-implementation of the Amiga OS (it sucks!), but I also have multiple windows and linux machines. With a VM, I don't have to worry about breaking something, if I'm doing something weird.
25   Patrick   2021 Oct 19, 3:21pm  

True, VMs have a place for moving images archived images around and segregating work spaces.

But they also add a lot of work and complexity: installing virtualbox, the extension pack, setting up networking to map it to the host machine, starting up and keeping the vm running etc.

I just found that for a single website, they felt like more of a pain in the ass than they were worth.

If Digital Ocean would just let me upload an run a virtualbox image, that would be compelling. But I think they do not.
26   NuttBoxer   2021 Oct 19, 3:39pm  

Yeah, containers seem to be the way to go now. All I run are internal services for the house though, so don't even bother with that. Just a headless server, and I'm good to go.
27   Patrick   2021 Oct 19, 3:42pm  

I tried containers too, like Docker, but they were even more painful than virtualbox imho.
28   richwicks   2021 Oct 19, 3:42pm  

Patrick says
True, VMs have a place for moving images archived images around and segregating work spaces.

But they also add a lot of work and complexity: installing virtualbox, the extension pack, setting up networking to map it to the host machine, starting up and keeping the vm running etc.

I just found that for a single website, they felt like more of a pain in the ass than they were worth.


Docker might be more appropriate. I'm not well versed in it though. A VM is a staging area for me. I have a bunch of them. All you have to do is screw up your main machine once, and suddenly you'll see the value of a VM.

Setting up VirtualBox isn't hard, and once you've made one VM, it's basically a 30 minute job, 25 of the minutes are waiting for it to install.

Patrick says
If Digital Ocean would just let me upload an run a virtualbox image, that would be compelling. But I think they do not


You're almost CERTAINLY running on some sort of VM, probably Docker. I should learn Docker.

https://www.youtube.com/watch?v=eGz9DS-aIeY

I've not done it myself, but it LOOKS pretty easy.
29   Patrick   2021 Oct 19, 3:47pm  

I wasted several months of my after-work time a couple of years ago just getting Docker going and trying to figure out how to use the same image on my laptop and my server.

I was not impressed.

The main problem is that hosting sites don't let you just drop images on their servers and run them. At least they didn't at the time.

It was less work in the end to simply start from a new, say, Debian distro, and install the things I needed.
30   porkchopXpress   2021 Oct 19, 4:15pm  

@Patrick - if you need to host it somewhere that costs more money, I bet a bunch of us would chip in annually. Your site is my home away from home.
31   Patrick   2021 Oct 19, 4:21pm  

Thanks @porkchopexpress that's a kind thought.

I think if I do this right it should not cost any more to host at home than I pay now.
32   Patrick   2021 Oct 19, 4:22pm  

HunterTits says
Patrick -- I no longer get emails notifying me if someone LIKES my posts/comments since the site was restarted. Do you have something you need to restart for that too?



Thanks for telling me @HunterTits. I don't see why that should have happened because I still get them.

Could they have been marked spam?
33   Patrick   2021 Oct 19, 4:24pm  

I don't see any bounces from mails sent to you, so that's not it either.
34   Automan Empire   2021 Oct 19, 4:30pm  

Now that he mentioned it I didn't get notifications for the replies ITT.

Can be useful to keep up with long running threads.
35   anonymous   2021 Oct 19, 4:35pm  

Comment from a test account. I'll like it and see if the test user gets the email.
36   Patrick   2021 Oct 19, 4:36pm  

Yes, when I liked that, my test user got an email.

Huh.

So the mechanism clearly works, mails are being sent and not bouncing.
37   Patrick   2021 Oct 19, 4:39pm  

Ah, most of a working day later, I get a reply to my ticket:

Thank you for contacting DigitalOcean and I am sorry that you were having issues accessing your website on your Droplet.
The Droplet is on a hypervisor that was undergoing emergency maintenance. We migrated your Droplet automatically in order to try and keep things operating normally. I understand this has a serious impact on your deployment and these situations are only done as an emergency measure and not intentional. We appreciate your patience around this issue and I am glad that you were able to return your Droplet back to working condition.
38   Automan Empire   2021 Oct 19, 4:58pm  

A couple just came through on my end so some module must be back to functionality.
39   richwicks   2021 Oct 19, 5:08pm  

Patrick says
Yes, when I liked that, my test user got an email.

Huh.


As a feature, why not just have a list of likes that ages out after, say, 7 days.

What I'd really like is a notification when somebody replies to me.

But, of course, your site.
40   Patrick   2021 Oct 19, 7:18pm  

Wow, that's weird.

It must have something to do with the reboot, but I don't understand what. Sorry about the flood.

Maybe a bunch of them were queued up but couldn't be sent for some reason until the reboot.

Comments 1 - 40 of 46       Last »     Search these comments

Please register to comment:

api   best comments   contact   latest images   memes   one year ago   random   suggestions