Hi Brian,

Is this a physical or cloud installation of Sage X3?

Is it a single server or multiple servers setup?

What is the expected RTO and budget for the RTO expectation?

Consider having duplicate of the production server but permanently offline.

There should be a way from cloud side through templating and in physical, just buyback current infrastructure but keep them all offline until needed or for manual sync-up weekly/monthly with IT team supervision.

Currently running everything in a single on-site VM. I have mirroring via Veeam to two separate hosts. I think I will split to 2 or 3 VMs for performance and so that I can have multiple techs working on different aspects.

The offline method is good. With VMs I can bring it up on any host but I have to be sure to redo the offline copy whenever we have any major updates. Might go to massive numbers of image backups so that I will have a lot of Recovery Points to choose from. My fear is that a hacker will silently install a bunch of back doors and then wait till all of my backup systems have cycled those back doors into every copy I have. Or there will be some database structure changes that make current data not usable with old images.

We have had pretty good uptime under my care for decades, so as for budget, I can get anything I can justify a need for.

Probably need to add in cloud based images with DRAAS. Pretty easy to drop big bucks doing that, and it might still be hackable if I am not careful.

Hi Brian Turner
Thank you posting. My name is Bruno GONZALEZ. I'm the technical leader of the Sage X3 center of excellence at Sage.

Can you please indicate:

- Your region

- your hosting details: on prem, MSP, cloud players like Azure, AWS or GCP

- Your realistic RTO/RPO

Usually, we work with our partners and customers to define what X3 architecture would best suit our customers needs based on cost of failure, time to recover, available technologies and skills, etc.

Let me give you an example. Let's say you are a customer running Sage X3 on a platform from an MSP with VMWare. You have 3VMs:
- X3 App and runtime,
- Syracuse/MongoDB/Elastic Search
- Sage X3 report server

The MSP will provide backup services taking form as a VEEAM VMs replication. If you have a failure, the vm is restored at the time of replication.

While there are many ways to address Disaster Recovery, please note that Sage X3 can also be Highly Available if need be (DR <> HA).

Usually, you don't cover your RTO/RPO needs by reinstalling all the tech stack manually by yourself or by calling partners or even Sage, but we rather try to find the best technological option with a balance in terms of costs and availability as it is quickest and safest than reinstalling things manually.

Please feel free to call your Sage X3 partner, indicate your worries and define your needs. Then either your partner can either open a ticket to Sage Support - mentioned my name if need be - or you can both reach out to me directly via private messages in Sage City.

I hope this will help you.
Kind regards?
Bruno

Sorry for the delayed response. I did not know you replied.

Yes, DR<>HA. My experience with HA is that it actually adds complexity to the point where overall reliability drops. I am only focusing on DR. You are confirming that re-installation is not a valid DR plan.

We are in the United States.

We have an on prem installation with VMware and Veeam. Everything is in one VM. I have dual primary hosts and a third backup host. I have plenty of resources to split into multiple VMs. The installation was done shortly before the last system requirements document was published where it clearly states that 2 or 3 VMs is preferred.

I asked this question in the community because I was hoping there would be more depth of knowledge than a single Sage Partner would have. I find that Sage often refers me back to my partner whenever I go looking for more expertise.

If re-installation is not an option, that makes me more vulnerable to a ransomware attack.

Hi Brian Turner,
Thank you for the details.
Sage being in an indirect model, we always refer to the partner as they are our customers primary point of contact along with Sage support. When they don't know or are seeking deeper expertise, then they reach out to their regional center of excellence which can eventually come back to me
Back on your topic: HA
X3 can achieve HA rather easily, but it is my point of vue as I deal with it every day. Please find below a Visio schema showing how it would look like:

I usually take 2 hours to explain how it works, but in a nutshell: you double down on each tech stack, with less resources, a bit like what you probably did with your VMWare cluster: You want a 70% max utilization of resources so that in case of failover or cluster node crash, you still remain at 100% utilization maximum to guarantee a good usage of the X3 solution. Indeed, like in VMWare, if you need to manage 160%, then nothing will work properly.
It's not so much the complexity that is an issue but rather the cost of having so many VMs - some of which being mutualized for cost efficiency, like Syracuse, MongoDB and Elastic Search.
Sage X3 can even specialize resources based on a a type of workload: SOAP Web Services, Batch tasks and interactive sessions. While this adds complexity and cost, it actually helps when troubleshooting.

Theoretically, you'll need to replicate this infrastructure on your remote DC, but we can think of various ways to prepare a remote site for a massive crash failover - VM replications, passive VMs (dormant) prepared in advance.

Reinstallation is technically possible, and scriptable but you need to consider reimporting mongodb dump, SQL Server database backups, X3 folder structure, based on the prerequisites that all these are rather recent and available and usable. Seems like a lot on a non-cloud environment. If you were in AWS, that would be different, as you have there the tools to auto provision and we have experience with it.

Regarding ransomware attacks, based on my experience (we see 1 at least every month):

- Most of the attacks come from inside of a company through phishing or things like that. It means you cannot trust accesses from your company, secure them properly, ideally namely. For example, every time I do a technical health check for a customer, I required a dedicated windows account, not a generic one used by everyone. That way the customer knows what I'm doing and the account can be deactivated as soon as I'm done.

- Make sure your X3 installation respects X3 security best practices with the proper access rights to the right domain/local windows accounts

- Don't open shares outside a given Windows user scope - I still see way too many "everyone - full control" pointing to X3 Folders structure or even database backup files because "it is easier".

- Keep recent copies (daily) of a MongoDB backup, SQL Server db backup, X3 folder structure backup in a different place: Cloud, different DC, storage account of any kind. The typical mistake if too everything on the same servers, these servers gets cryptolocked, and you're done for...

Sage X3 can do a folder export from X3 console using parallel data extraction and integration, which can also help as you'll have everything in a single Zip file.

- Test your backups. I cannot count how many times people wanted to use a backup they had which was unusable for any reasons. At least once or twice a year: spin a VM with SQL Server Dev edition, restore your backup and make sure you can access the database tables. It will also give an estimate of how long it took. Don't hesitate to think about restoration speed optimization techniques like using 2 or 3 striped disks in Windows Storage Spaces to maximize throughput, use 2 backup files to parallelize, etc.

There are a number of precautions to take when you take a database backup to make sure it can be used later. My advice: don't set a backup strategy, set a restore strategy.

- Do not keep test and dev on the same database as production. I see this every day, and while I understand the costs, it will bite you back hard as it will double, triple, or quadruple the amount of data and time to backup but foremost, to recover.

- Don't forget that X3 supports advanced SQL Server feature like table compression. When we do a Technical Health Check for customers, we give a list of tables to compress based on utilization to reduce disk size, IOs consumptions and SQL Server Cache utilization.

Don't hesitate to ask for a meeting with your partner and Sage if you want to investigate your options together. It will be with a consultant from US center of excellence, and it could be with me if you require so, at least on the first stages.