JetBrains TeamCity critical vulnerability is being actively exploited to create hundreds of administrator accounts on more than 1400 servers.
JetBrains TeamCity is a continuous integration and continuous development (CI/CD) platform used to automate the building, testing and deployment of software projects. On Monday, JetBrains patched and disclosed a critical flaw, tracked as CVE-2024-27198, that allows an authentication bypass to gain administrative control over TeamCity servers.
LeakIX reported on Wednesday that 1,442 local instances of TeamCity showed “clear signs of fake user creation.” LeakIX is a search engine that indexes Internet connection services and provides information about vulnerable services and data leaks.
The TeamCity vulnerability was discovered by Rapid7, which said in a post on Monday that exploiting the bug would give an attacker “full control over all TeamCity projects, builds, agents and artifacts,” setting the stage for potential supply chain attacks.
“We started hearing from some of our customers who noticed that their servers were compromised shortly after Rapid7 released its full disclosure with examples of exploits, and later exploitation module,” JetBrains TeamCity Solutions engineer Daniel Gallo told SC Media. “The first was just a few hours after this information was published online.”
All managers of on-premises instances of TeamCity must upgrade to version 2023.11.4 to prevent exploitation of CVE-2024-27198. LeakIX stated that all users who used the vulnerable version should “assume compromise”.
TeamCity users should monitor the servers for unusual activity
Gallo gave SC Media a few recommendations for users who may have patched late or suspect a compromise. First, users should shut down the server to isolate it from external and internal networks, Gallo said.
The server’s “Audit” screen should be checked for actions such as “unknown user access tokens created or deleted, all unknown users created, login actions, etc.,” which can be filtered by the “User Actions” category, Gallo explained.
The attackers created between 3 and 300 new user accounts on the compromised servers, with a pattern of using eight alphanumeric characters as usernames, LeakIX told BleepingComputer.
Additional ways to check for suspicious activity include the “teamcity-activities” log file, where users check for unknown plugins, projects or build configurations, and the host operating system’s event logs, which should be checked for unknown events logged by external processes, he said. Gallo.
“Checks should be performed for the presence of any unrecognized processes running in the background on the host server, such as malware, crypto-spoofing and other unauthorized tools,” Gallo said.
If suspicious activity is detected, the server should remain isolated as credentials are rotated for all users.
“There will also be a different number of credentials for all external services that TeamCity and its host operating system had access to, such as repositories and connections (eg Slack, AWS, Docker registries, etc.), TeamCity database connection, and network drives,” Gallo noted. “If a server was compromised, all build agents connected to the server during that period should also be considered compromised.”
Stephen Fewer, principal security researcher at Rapid7, told SC Media that any server with signs of compromise should be reviewed by an incident response team to uncover the full extent of the breach.
“The best practice would be to rebuild that server, including the underlying operating system, so that the persistence that the attacker achieved would no longer prevail,” Fewer said.
LeakIX said a total of 1,711 vulnerable TeamCity servers had been discovered by Wednesday, down from the 2,000 it discovered on Monday, according to the X post.
The Shadowserver Foundation, a non-profit organization that monitors vulnerabilities and malicious activity on the Internet, said on X that it first discovered the CVE-2024-27198 exploit on March 4 at 22:00 UTC. Shadowserver’s vulnerability time series dashboard currently shows 1,182 vulnerable servers as of Wednesday.
Cybersecurity firm GreyNoise observed 33 malicious IPs targeting CVE-2024-27198 for exploitation, with a large spike in activity detected on Tuesday, March 5.
JetBrains, Rapid7 clash over discovery time
Both JetBrains and Rapid7 have released timelines of the events that led to the discovery of the critical authentication bypass flaw, along with a high severity path traversal flaw tracked as CVE-2024-27199.
While the timelines largely agree with respect to the dates of the contacts and what was generally discussed, they reveal an apparent conflict between the two sides’ disclosure philosophies.
JetBrains first acknowledged Rapid7’s report on February 19th, with Rapid7’s timeline stating that they were first approached on February 15th and then four days later.
On February 21, JetBrains reserved the CVEs for the flaws and suggested making them public a few days after the fix was released, while emailing users about the patch in the meantime. JetBrains said this approach complies with its Coordinated Disclosure Policy.
Rapid7 responded that this approach contradicted its own Vulnerability Disclosure Policy, as they considered the delay in public disclosure to be “quiet patching”. After further discussion about disclosure policies and Rapid7’s definition of silent patching, JetBrains announced on February 23 that it had decided not to coordinate disclosure with Rapid7.
On March 4, JetBrains released a fixed version of TeamCity with an initial release that did not disclose security flaws and did not notify Rapid7 prior to releasing the release. The company began emailing customers about the patch and published its blog post disclosing the security flaws about an hour after the initial release.
Minutes after this second blog post, according to JetBrains, Rapid7 contacted JetBrains saying they would publish their discovery within hours. Rapid7’s timeline suggests that they contacted JetBrains before publishing the second blog post saying they planned to release their version within 24 hours in line with their silent patching policy.
A Rapid7 blog post about the vulnerabilities was finally published about five and a half hours after the patch was released.
“The reason we ended up not agreeing to a coordinated disclosure with Rapid7 was their insistence on releasing the full details of the exploit at the same time the fix is available to our customers. This would immediately provide attackers with the necessary tools to exploit TeamCity servers without any opportunity for our users to apply mitigations,” Gallo told SC Media.
“We fully support timely disclosure of vulnerability details when a fix is released. However, we provide customers with only the necessary details to understand the scope and severity of the vulnerability (such as CVE details and attack vector description), allowing them to take appropriate action without revealing so much information that facilitates easy exploitation,” added Gallo.
Rapid7 declined to comment to SC Media in response to JetBrains’ criticism of the timing of the release.
However, the company’s previous blog post on silent patching, which it cites in its timeline, states that while delaying public disclosure of details about vulnerabilities prevents a “random attacker” or “script kid” from exploiting them, “you also exclude a large population of IT defenders.”
“Most importantly, you’re excluding the most important audience for patching: ordinary IT administrators and managers who need to sort the incoming patch stream based on some risk and severity criteria and make a call to stall and schedule updates based on those criteria,” the 2022 post said. , authored by then research director Tod Beardsley.