This short guide aims to aid companies by increasing the cost-effectiveness of penetration testing services. We'll start by finding common vulnerabilities, after which it's time to discuss the best approach for choosing pentest targets. Lastly we'll look at improving and streamlining deliverables.

Content headings

  1. Finding vulnerabilities yourself
  2. What to pentest
  3. Saving on deliverables
  4. Pentest preparation
  5. After a pentest

Notes, abbreviations and references can be found in the appendix.

Disclaimer

This is not a comprehensive security guide and its roots are mostly empirical.

1. Finding vulnerabilities yourself

Plucking the low-hanging vulnerability fruit before an assessment allows penetration testers to spend more time on advanced and creative hacks. The following low effort, high impact tricks can be performed by most technical personnel. A security partner might be needed to interpret the results.

Static Code Analysis in three lines

These tools are free for all purposes, and they don't send source code to external parties.

  • Find common vulnerable code patterns with Semgrep by running the following command. This will check over 2500 community-maintained patterns. Its powerful engine also makes it easy to write custom rules.
    • docker run --rm -v "${PWD}:/src" returntocorp/semgrep semgrep --config=auto
  • Find vulnerabilities for outdated packages with OWASP dependency scanner. Don't get frightened by the high CVSS ratings of CVEs: base the risks on the chance of an attacker reaching the vulnerable components. The client can be run with:
    • ./dependency-check.sh -s .
  • Find hardcoded secrets and keys with Trufflehog. In many enterprises it's easy to get read access to source code, but it's harder to obtain read access to key vaults. Don't let your source code be a key vault.
    • docker run --rm -it -v "$PWD:/pwd" trufflesecurity/trufflehog:latest github --repo https://github.com/trufflesecurity/test_keys

Easy network security

  • Nmap is available on Linux and it includes vulnerability scanning options. I recommend a general vulnerability scan and a scan to check for open shares. Browse discovered shares to see what's on them. The vulnerability scan can be bulky, so start with the assets that you plan to pentest.
    • nmap -sV --script vuln -Pn --open [ip range]
sudo nmap -sU -sS --script smb-enum-shares.nse -p U:137,T:139 [host]
  • Most cloud providers have built-in security audit tools, such as Amazon Inspector and Azure Security Center. Have someone actually check the output of these tools.

Ask around

As trivial as this may seem, I've been uncovering security vulnerabilities left and right by simply asking people about them. Try saving 20k by asking the following questions at the coffee machine:

  • What do you think is the best place for us to start improving security?
  • Are you aware of any security vulnerabilities we have at the moment?
  • What would you do to hack us?

Developers aren't given the time to work on security vulnerabilities, sysadmins know of security holes left by third parties and architects know where things are messiest. They either think everyone knows, or they 'mentioned it one time' and it was 'accepted by the business', i.e. it went straight over some manager's head.

⚠️ Employees are the best source of security information. Bravery of highlighting shortcomings should be praised and pro-actively encouraged. The worst security cultures are those where nobody dares to speak up.

Make time for the obvious

Here are some examples of miscellaneous plays that are worth the investment and require relatively little effort.

  • Validate assumptions on network boundaries. Cases like "It's not possible to reach that web interface from the guest network" can be checked by simply entering the IP in your browser (from the guest network).
  • Google default credentials for something with a login screen and try to login to said something.
  • Prevent subdomain hijacking by checking if DNS entries are up-to-date.
  • Most frameworks, applications and platforms have security guides these days. Take the time to check them out. Also check the OWASP cheat sheets and HackTricks for security improvements on used technology.

2. What to pentest

Nobody has the resources to constantly validate all security controls. Use attack- and threat modeling to find risky hotspots and attack paths. Perform sample assessments to see what can happen if an attacker can reach a certain place. This is called risk-based or scenario-based pentesting, and it's what most security experts, including myself, vouch for these days.

Make distinctions between systems that can be (partially) isolated from each other. We'll take the example of products and the organization's internal landscape. Products that customers use are partially isolated and applications in the internal landscape are part of a larger system.

Build products with security in. Embed security into the SDLC, and consider making a pentest mandatory for major releases. It's hard enough for a team to properly embed security in the development lifecycle of a single product. Adopting long-lasting security practices is not only a developer awareness problem, but it's also dependent on organizational factors. [1]

The internal landscape doesn't have a development pipeline like a product has. We can't simply copy SDLC security practices and expect it to work. If the internal landscape exists of hundreds of (ancient) applications, managed by tens of teams and third parties, it would be a waste of resources to try and pentest all of them for every release. Even if it's achievable it would neglect parts of the system that can't be secured in this way.

Treating applications as isolated components wastes resources, so we need to prioritize. Most security experts agree that risk-based and scenario-based assessments are the way to go. This article even claims that the risk-based approach is superior to using maturity models (in something that resembles a maturity model of risk approaches).

Choosing the best place to sample is a challenge. Decision-making models exist but don't offer qualitative insights that take the complexity of organizations into account. [2] If internal expertise is lacking, I advise spending a small portion of the pentest budget on getting advise on the logical next place to perform a security test.

Type of assessment

A pentest is used to check if security controls are missing; it's not a security control in itself. Its main function is to support secure development on all levels. Most assessments will have pentesters work together with personnel to efficiently obtain effective results. This crystal-box (or white-box) approach streamlines the intercourse between internal and external knowledge to deliver the best insights.

Red team attacks – and real attacks – can be thought of as tools for outsourcing the analysis of the most risky entry points and attack flows. I'd advise on doing red teaming if it's unclear where to start, or when security boundaries feel well established.

The semantics of assessment types can constrict inspiration for requesting security services. "Do I want [color] Teaming, a [color] pentest, or a [prefix] vulnerability assessment?". Don't get stuck on pre-packaged assessment names; focus on required insights and have them slap a name on it.

Useful targets to get started

The following list is meant as a source of inspiration for optimizing penetration testing services such that they cover large - or heavily condensed - risk areas in a relative short amount of time.

  • Ask a pentester spend a couple of days doing basic OSINT, scanning the external IP range and probing authentication portals. Nice additions are DNS hijacking and using the HaveIBeenPwned-API on the email addresses of employees. Some of these tasks can be done in parallel and they sample the external attack surface.
  • Perform a vulnerability assessment on applications that have huge authoritative potential, like an SSO or JWT flow. Here we sample the ability to manipulate the providers of authorization identifiers, since authorization is the fundament on which security is built.
  • Perform a pentest focused on things that manage a large part of your network, like Active Directory, Ansible or Kubernetes. Here we sample things that have so much power that a full compromise practically equals defeat.
  • Have a pentester spend a couple of hours assessing the security of a design. Most structural vulnerabilities can be prevented in this way.
  • Request advise on great places for scenario-based assessments.

3. Saving on deliverables

Leaving out bulk can make a report the enjoyable reading experience it should be. In addition, leaving out unnecessary tasks allows the pentester to find more vulnerabilities and to write better recommendations.

Intake

The intake is crucial for setting up a proper assessment. Check the following items for great results.

  • Include someone that can answer technical questions, and then explain why the pentest is being requested.
  • Point out the worst case scenarios as logical targets.
  • Don't waste money on finding known issues; communicate all known vulnerabilities so they can be skipped or simply noted.
  • Communicate the company's willingness to change, because the way of researching and reporting can be adapted to it. Is there willingness to refactoring all code, infrastructure and even the way of working based on the results, or is the goal to just find some quick wins?
  • End the intake with a list of requirements that will allow the pentester to use all functionality in scope.

Report details

Many reports include unnecessary information. Don't be afraid to say "If it saves you time, and of course if it saves us money, you don't have to include...":

  • A log of scans and checklists of vulnerabilities or attack vectors. These things are never accurate and keeping track disturbs the pentest process. Some audits like DigID have testing requirements. If the test is performed for compliance reasons then the requirements for that specific audit obviously need to be present.
  • A history of the company, a team description with pictures and personnel background stories, copyright disclaimers (the same copyright applies without the disclaimer), graphs of findings.
  • Static information, such as a pentesting methodology description or a description of how the findings are rated.

All that's needed for most assessments are the following items.

  • A management summary with an overall impression and a recommendation.
  • A scope description with targets, assessment type and goal, provided resources (accounts, code), dates and limitations.
  • Findings with a risk level, finding details (with reproduction steps), an impact description and recommendations.

CVSS rating

Companies tend to have an internal risk matrix that scales from 1-3 or 1-5. Ask the pentester if it saves time to just plot findings as risk = chance*impact on the same scale (low to high, informational to critical). The closer a pentester's scale is to the company's risk scale, the easier it will be to translate her estimation to internal business risks.

The CVSS might seem accurate because of its fine-grained rating system, but the rating hardly ever matches the way one would reason about the risk in an organizational context. Experts agree that its complexity creates misleading ratings for OT findings and the same can be said for IT.

Low risk noise

Most companies have a bug bounty statement where they've defined what they consider to be security noise. Ask the pentester to make quick notes of findings in this list instead of taking the effort to incorporate them into the report. A list of accepted vulnerabilities can also be handy for filtering noise.

Findings meeting

Don't skip the findings meeting if anything of interest was found. It's crucial for a clean knowledge transfer, since questions can be asked. In addition, the summary can be communicated to multiple people at once, which saves them from having to read it.

4. Pentest preparation

Be prepared to process the results

Spending 5-50k on a pentest is a shame if the pentest findings can't be tracked or processed. Utilize a suitable system to track the lifecycle of open, accepted and structural vulnerabilities. This can be as simple as a dedicated Kanban board.

Technical preparation for a pentest

  • Have the requirements from the intake ready and tested before the pentest starts. Make sure that the acceptance environment is working and that it's representative for production. Double check that all functionality in scope works.
  • Prevent tools like an IPS or a WAF from blocking attacks. Simple payloads to exploit a vulnerability might get blocked during the time-boxed pentest, but over time an attacker might find complex payloads that get through.
  • Have a developer or administrator on stand-by to answer technical questions that might arise during the assessment. Utilize secure communication channels that are superior to email if they are available.

    • ⚠️ Don't blindly trust pentesters, since one can't screen for bad intentions. This is why testing on acceptance environments should be preferred. The environment should be representative for production, but it should not contain sensitive data from it, credentials should differ, and no links to production may exist. Consider a pentested asset as potentially compromised.

    5. After a pentest

    Assess the quality of the deliverable

    An excellent pentester has the ability to create recommendations and to help reason about the risk of a finding. I've seen 'pentest companies' straight up dump (Nessus) scan reports into someone's mailbox without interpretation. Don't pay these companies as if they were offering professional services.

    "Excellent penetration testers will prioritize the discovered vulnerabilities based on the ease/likelihood of exploit, difficulty/cost to mitigate and impact to the business if exploited [...] beginner testers will provide no analysis at all, just a laundry list of vulnerabilities." [3]

    You have the power of providing feedback to make a deliverable work for you. How about having pentest companies directly deliver results to an API that converts the findings to backlog items? The sky is the limit here; we don't need to be stuck in the laundry list stone age.

    Don't lose track

    Accept or remove low risk noise if it stagnates the ability to mitigate - or clouds the vision of - serious security vulnerabilities. But before removing noise in bulk, consider asking a security engineer to check if these low-risk issues can't be combined for serious impact.*

    Take the psychological pressure on risk rating into account. There is incentive for researchers to exaggerate risks of findings and CVEs, as some perceive it to be the 'score' of their abilities. Utilize the findings meeting to ensure a clean knowledge transfer of a finding's intricacies, as it is required for an earnest translation to business risk.

    Track structural vulnerabilities separately. These vulnerabilities require large changes such as rewriting an application, changing the way of working or redesigning a network setup. Separating them allows for cleaner mitigation metrics.

    Extrapolate the samples

    Pentests are expensive time-boxed samples. Milk this knowledge to the fullest, as much can be extrapolated from a few simple findings.

    Start pulling threads towards root causes. Note that even root causes have root causes. It's turtles all the way up to management. But even there, it is not a given that the knowledge for solving challenges exists. An open and honest symbiotic knowledge relationship is required to create strong roots.

    Root cause analysis tends to drill down vertically, but vulnerabilities can also be approached horizontally. Scan for discovered vulnerable code patterns in all codebases, instead of just the application it was found in. The geometrics of addressing findings are less relevant than actions taken to resolve them.

    Conclusion

    Increase the cost-effectiveness of pentests by targeting risk-dense areas and filter easy vulnerabilities beforehand. Communication paves the way for a clear signal, and by properly processing the area's findings we won't get overwhelmed in the next round.

    In the end, it's all about managing complexity. People are re-inventing this wheel from application development to company management. We'd love to hear your inventions, so don't hesitate to share them with the rest of us.

    Keep trying to find pragmatic security solutions. It's not being simplified by bureaucrats overengineering laws, technocrats building on weak foundations and attackers taking advantage of these proliferating tech stacks. Nonetheless, we're making great headway, so don't lose hope, and make it work for you.

    Notes

    * While contributing to Gjoko Krstic's research on Building Management Systems, we chained five low- to medium-risk vulnerabilities to create an exploit to remotely root the Linear eMerge50 BMS.

    Abbreviations

    API (Application Programming Interface)
Code or network interface that programmers can easily plug their applications into.

CVE (Common Vulnerabilities and Exposures)
Public database of known vulnerabilities. Note that researchers have to register their vulnerability manually, so not all publicly known vulnerabilities have a CVE entry.

CVSS (Common Vulnerability Scoring System)
A scoring system for vulnerabilities that plots the severity of a vulnerability on a scale of 0 to 10.

DigID (Digital Identity)
Application from the Dutch government that citizens authenticate themselves with.

DNS (Domain Name System)
The system that handles domain name registration.

IPS (Intrusion Prevention System)
A system, commonly on the network level, that tries to block attacks.

JWT (JSON Web Token)
Mechanism that cryptographically signs JSON data to relay properties of an entity. JSON is an easily parsable text structure.

OSINT (Open Source Intelligence)
Public information that can be retrieved without requiring any hacks. The term commonly describes data collection on a target via public sources.

OT (Operational Technology)
Technology that has to do with controlling industrial systems, such as a factory.

OWASP (Open Worldwide Application Security Project)
An organization that tries to create a practical overview of secure development practices. It's mostly focused on web development.

SDLC (Systems/Software Development Life Cycle)
An acronym that can mean two annoyingly similar things. A Systems Development Life Cycle refers to the process description of an entire system's cycle, from the initial idea to its decommission, whereas the Software Development Life Cycle commonly describes the development process for components, from use case to the maintenance of the new functionality.

SSO (Single-Sign On)
Mechanism that allows users to automatically log in to applications after an initial sign-in.

WAF (Web Application Firewall)
Firewall for web applications that tries to block common attacks.

    References

    [1] Türpe, S., Kocksch, L., & Poller, A. (2016).
Penetration Tests a Turning Point in Security Practices? Organizational Challenges and Implications in a Software Development Team.
WSIW@SOUPS.

[2] Dor, D., & Elovici, Y. (2016).
A model of the information security investment decision-making process.
Comput. Secur., 63, 1-13.

[3] Geer, D.E., & Harthorne, J. (2002).
Penetration testing: a duet.
18th Annual Computer Security Applications Conference, 2002. Proceedings., 185-195.

    💡 I've never read a paper as brilliant as [3]. They foresee that the increase in (web) applications will erode network boundaries, recommend crystal-box pentesting and shift-left development practices and they show what the deliverable of an excellent pentest should look like, all 20 years ago. Its writing makes it a true creative work of art.

    The article was originally published on my LinkedIn, but I migrated all posts here.