ConnectWise Automate security is something at the forefront of discussion for every MSP. Previously we’ve discussed security from the WAN angle, and how reverse proxy technology can be implemented to reduce the attack surface. However, internal LAN communications can also be an attack vector, and here at Automation Theory, we wanted to give proper attention to this often neglected aspect of Automate security.
Below we’ll review the risks and remediations for the different components of Automate, including the database, integrations, and the OS. Before we dive into Automate, let’s review the LAN and its entry points.
Breaking into the Automate network
Our below scenarios assume that an attacker has infiltrated the network segment Automate is running on, so preventing this scenario is obviously the thing to do. However, at some point, it’s inevitable that some threat will come into the MSP internal network (and that’s why most MSPs run AV/EDR/SIEM/etc. tools on workstations and servers). As MSPs grow and staff are added, the probability of human error goes up. It’s less likely for technical staff who are aware of the latest security threats, but most MSPs we’ve talked to agree that the hypothetical scenario of a sales/accounting/etc. staff member falling for a high-quality phishing attack could occur.
The critical factor will be what can be accessed from the primary LAN network of the MSP. Smaller firms might have the Automate server directly on the LAN, while larger MSPs will likely have the server in its own network segment. However, it’s important that ACLs are in place to ensure that the segments are indeed isolated, and otherwise that it’s impossible to move laterally from one segment to another using traditional lateral movement techniques or secondary vulnerabilities.
With this in mind, here at Automation Theory we universally suggest keeping Automate in a separate network completely isolated and inaccessible from your regular LAN (either workgroup or a separate Active Directory environment). However, the following attack vectors will exist inside any internal network segment, and hardening is still a necessity.
The MySQL attack vector
A recent study found that 56% of on-premise databases have at least one vulnerability, and here at Automation Theory, we’ve found that to be true (as most Automate stacks never see a MySQL patch after they are deployed). Anecdotally, we’ve found the average Automate MySQL database install age dates back to 2017 — and 468 vulnerabilities in MySQL have been released from 2018 to 2021. Needless to say, if an attacker was able to get onto the same network segment they would have their choice of attack method with an average deployment.
What can an attacker do with MySQL access? Part of that depends on the database configuration (misconfigured databases can be used to get system-level access on the hosting server!), but if an attacker gains access to the Automate database it’s essentially game-over for the MSP — a single SQL command is all it would take to deploy a malicious payload to all agents. Even read-only access could inflict damage if the computers and passwords table were exfiltrated, as those two tables contain enough data to construct automated attacks.
It’s also worth noting that a database doesn’t have MFA, aggressive lockout abilities, or much for security logging for authentication failures. This makes them more challenging to protect than traditional systems, but there are a few techniques that can aid in protecting them.
- First, restricting network access is by far the best way to protect a MySQL server. If possible, we suggest setting the network settings to listen on localhost only. This is suitable for single-server Automate stacks with no external integrations, but for split-servers or integrations that access the DB over the network (dashboards, reporting, etc.) we suggest using firewall rules to restrict access to a whitelist of devices.
- Otherwise, patching the database, using strong passwords for any integrations, and using least-privilege wherever possible are simple and easy ways to harden Automate’s MySQL database.
Finally, we’d like to make a note about the application architecture as it relates to MySQL and split servers: the MySQL traffic between the application and the database is plain text by default in MySQL 5.6 and earlier — so if an attacker can sniff the traffic they will eventually be able to glean credentials for further movement and exploitation. We advise partners with split-server environments to ACL and segment the network as tightly as possible to ensure that no unauthorized network sniffing can occur.
The integration attack vector
The integration ecosystem is of great benefit to Automate partners. However, it can also introduce attack surface. As an example of this, we’ll look at the popular dashboarding platform Grafana.
CVE-2018-15727 is a vulnerability that allows for an authentication bypass into Grafana via a forged cookie that only requires a valid username. Once connected to Grafana, a simple query will dump all the user password hashes — and from there they can be loaded into a variety of tools for cracking. If the Automate database contains any users with weak passwords the attackers will gain access to the database, and from there as described above, it’s a one-liner to deploy a malicious payload to all agents.
Prevention of these attacks boils down to the basics of least privilege, network isolation, strong credentials, and application patching. If the MySQL account for Grafana can only access the tables it needs (as we describe here) then it won’t be able to be used to get authentication data from the internal database. If the Grafana server can only be accessed by a whitelist of devices it becomes much harder to attack. If all the credentials in the database are strong then the attack will stall out in the hash cracking stage — and if Grafana is patched then the attack vector won’t exist.
The above preventative measures are applicable to all integrations — but greater care should be taken for anything that is internal and is trusted to access the Automate network. It’s common for internal tool maintenance to fall behind in the managed IT sector, and thus it’s very easy for the thoughts of “it’s working” and “we don’t want to risk downtime” and “it’ll be next quarter’s rock” and “we’ll do it when we have the time” to turn into persistent vulnerabilities.
Windows OS vulnerabilities
Most MSPs are using Automate to manage Windows devices, so Windows OS security is a very familiar topic. In the context of local network security, all the typical advice applies. However, it’s worth it for MSPs to spot-check a few items.
First, it’s extremely common to see Automate servers running without the UAC enabled — and it’s also common to see the Windows firewall disabled, or with firewall rules that are wide open (especially if the environment was built by ConnectWise). This is obviously low-hanging fruit; there is no reason to have the UAC disabled or the firewall turned off — and we universally suggest enabling and using all the security layers provided by Windows.
Beyond that, Server 2019 support should be coming this month (October 2021). Server 2019 includes new security features, such as Kernel Control Flow Guard and System Guard Runtime Monitor (and it has Windows Defender ATP baked into the OS). In light of the additional security (and the performance improvements to IIS), we suggest that MSPs plan on upgrading to Server 2019 once official support is announced. Server 2019 will also support MySQL 8 — and for those planning around software life cycles MySQL 5.7 support ends in October of 2023, so upgrading to Server 2019 before then is advisable.
Finally, hardening the Automate server at the OS layer is a task that is often missed. Earlier this year official recommendations were published — but here at Automation Theory, we didn’t find them to be very robust. They included a handful of the most basic hardening steps, and nothing more (apart from references to STIGs that would require someone spending a day editing the registry by hand). MSPs are needing to protect against highly skilled and well-funded attackers, and thus a more robust OS hardening is in order.
For any partners looking to improve Connectwise Automate security, we like to suggest the open-source project HardeningKitty. It is a PowerShell script that can apply CIS, DoD, Microsoft, and other security baseline standards to a machine. It can also run in audit-only mode, and create backups of the impacted registry keys. Please note that the aggressive templates can cause issues, and even the default template will disable RDP and create deny rules in the Windows Firewall for SMB — so good backups are highly suggested.
Connectwise Automate Security: A plan of attack for MSPs
It can be a big task for an MSP to do major upgrades to core business systems — especially when plates are already full with the day-to-day. With that, we’d like to suggest what we see as a typical upgrade path below:
- Install the 2021.10 Automate patch
- Upgrade to MySQL 8
- Upgrade to Server 2019
- Harden the OS
OS upgrades are also the perfect time to migrate or split servers (as needed). Here at Automation Theory, we can assist partners with server migrations. We also offer MySQL upgrades as a stand-alone service or as a part of our MySQL Maintenance Package subscription (which is designed to meet all the security and performance needs of a growing MSP).
We hope you’ve found this post helpful. If you want more detailed information on this topic, the recording of the companion webinar can be found below:
