According to F5’s The State of the State of Application Exploits in Security Incidents report, web application attacks were the leading incident pattern among data breaches for six of the last eight years. Meanwhile, the number of new vulnerabilities discovered each year continues to rise, making it difficult for security teams to effectively keep up, especially in the modern software development lifecycle, which prioritizes continuous development and release. Security has traditionally been viewed as an impediment, slowing down this process. New approaches to application vulnerability management, however, take advantage of the CI/CD pipeline to discover vulnerabilities earlier and reduce the number of exploitable vulnerabilities in deployed applications.
Here are some best practices for improving your application vulnerability management.
Incorporate security early in the development process
In our post on vulnerability best practices, we discussed the concept of shifting security left, meaning incorporating security earlier in the application development process. This involves continuous vulnerability scanning in the CI/CD pipeline, but it also means that security teams should be involved in the planning phases, a process known as threat modeling. Threat modeling provides security professionals the opportunity to sit down with developers to identify areas of the application that will deal with sensitive information so that these areas of the infrastructure can be given extra security attention. Identifying entry points and building in mitigation strategies from the start is more efficient than trying to add on security features after an application has been built.
Scan your containers
The COVID-19 pandemic accelerated use of the cloud as businesses quickly adapted to accommodate remote work and the increased need for fast digital transformation of services. According to CloudHealth’s report, How 2020 Changed the Way We Use the Cloud, container usage was 38.7% higher in September 2020 than in January 2020. Containers, units of software with all of the code and dependencies needed to run the application, regardless of computing environment, have allowed companies the workload portability necessary for rapid development across hybrid environments.
While containers provide numerous benefits for application development, they also come with significant risk. Container images are frequently pulled from public repositories, which introduces the opportunity for image squatting, a practice similar to typosquatting, in which a bad actor uploads malicious images with names similar to commonly used container images. It is absolutely essential that, as a first line of defense, you put in place policies for acceptable sources of images to reduce the likelihood of incorporating a deliberately harmful image. When selecting a third-party base image, consider the trustworthiness of the image’s source and host, whether or not the source code for the components of the image is available, and the default installed software of the image. Having non-essential software provides additional tools for the attacker if they are able to gain access to the container and offers more opportunity for susceptibility to zero-day vulnerabilities down the line. These policies should also consider the update frequency of the images being used. Using outdated images is one of the many ways that containers end up with known vulnerabilities.
Even images from trusted sources, however, can be dangerous. A study by Prevasio found that 51% of the 4 million Docker Hub images analyzed had exploitable vulnerabilities. Therefore, it is best practice to incorporate image scanning into every stage of the CI/CD pipeline. You can then use admission controllers to ensure that container images with known fixable vulnerabilities above a certain severity will not be pushed to the production-accessible container registry until they are scanned and a security validation decision is made. You can also use a policy engine in combination with the admission controller to make the decision in the cluster, as opposed to in the image scanner, so that cluster information can also be used, increasing the context on which to make the decision.
Beware of tag mutability
You’ve integrated vulnerability scanning into your CI/CD pipeline and put in place admission controls to stop vulnerabilities from being deployed, so you’re safe, right? Well, there is still a chance that the image that ends up deployed in the cluster is not the one that passed scanning checks. Someone with direct access to the registry can mutate a tag after the image has been scanned. A registry is where the container images are stored, and these images are grouped into repositories that contain multiple tags. Tags are essentially pointers for specific container images (specifically their manifest file). An immutable tag would be one that would point to the same image forever, but in reality, most tags are mutable, meaning that they can be changed. If an attacker gets push access to your container registry, they can create a new malicious image and tag it with an existing approved tag, and the tag now points to this malicious image, pulling it instead of the approved image.
If it is dangerous, why do mutable tags exist? Mutable tags offer the benefit of being able to push updates without any changes on the consumer side. Mutable tags such as “latest” are also commonly used to point to the latest version of the image, and “dev” or “prod” may be used to track versions in different environments. Using mutable tags, however, can introduce problems outside of security as well. The image maintainer may introduce breaking changes between the time that the image is first pulled and when the application is deployed in the production cluster, crashing the application in production because the image being used is now different, but the tag being pulled remains the same.
Using immutable tags can protect against these issues, as long as your registry actually supports immutable tags, as some do not. Other additional security measures such as image signing can also be put in place to reduce the impact of tag mutability. Image signing creates a cryptographic key to create an encrypted signature with the image metadata. A public key counterpart can then be used to decrypt the image signature and make sure the contents have not been changed.
Use a mix of static and dynamic security testing tools
Throughout this post, we’ve been talking about integrating security into the software development lifecycle. One way that security testing can be incorporated earlier is through the use of static application security testing (SAST). SAST tools analyze source code to find security vulnerabilities. It is static because it does not require a running version of the application. This helps identify vulnerabilities earlier and ensure that they are not brought forward into builds that will then break. These tools can identify some of the most common application vulnerabilities from the OWASP Top Ten Vulnerabilities List, including SQL injection, cross-site scripting, and buffer overflows, and they provide real-time feedback for developers who can then fix these issues before the security team even needs to get involved.
When selecting a SAST tool, there are a few things to keep in mind. Your tool should be able to create a holistic view of your entire application code base by working with other types of tools, such as software composition analysis tools to analyze both proprietary and open-source code. Another important factor in selecting a tool is accuracy. One of the main complaints about SAST tools is the number of false positives and negatives they can generate from lack of context. Integration with other systems such as vulnerability databases and threat intelligence feeds can help remedy this, as can automation that will do additional analysis before inundating developers with alerts. How these tools integrate with other tools is also important when considering how it will work with your CI/CD pipeline. You’ll need to ensure that its output format is compatible with the databases you’re using.
SAST tools are great for catching vulnerabilities early, but as we’ve mentioned, they have the potential to generate false positives because of the lack of context, which is where they can work in conjunction with dynamic application security testing tools (DAST). Unlike static application testing, dynamic application testing takes place in real-time in production, simulating the real-world activity of the application. DAST operates similar to penetration testing in that it approaches security from the perspective of the hacker, attempting to uncover outcomes that should not be part of the normal user experience. This allows you to catch vulnerabilities only discoverable while the application is running and eliminate false positives from SAST testing that are null and void due to controls only visible when all pieces of the application are working together.
Monitor underlying infrastructure
Finally, don’t forget about the hosts that are running your containers. These are often overlooked because they’re thought of as being extremely secure, but if an attacker is able to compromise a container, they are going to look for ways to escape that container and exploit vulnerabilities in the host to compromise the entire cloud infrastructure. There are also some vulnerabilities that can be exploited without actually having access to the host, such as CVE-2021-20291, which is located in the container but can cause denial-of-service attacks.
Getting started
Not sure where to start with application security? RH-ISAC members have exclusive access to Member Exchange, our community discussion platform where retail and hospitality cybersecurity professionals collaborate and exchange knowledge. Check out a recent report of survey results on how RH-ISAC members design and implement application security awareness and training for IT employees. RH-ISAC members can also join RH-ISAC’s vulnerability management working group or our software security working group to participate in discussions and exchange of best practices. Not a member? Learn more about RH-ISAC membership.
