One Low, Two Informational: Why Your Pentest Findings are so Boring

Application Pentests are costly, sometimes six-figures costly, and can be very time consuming for the hosting AppSec team. Even so, application pentests often yield very few meaningful findings, leaving potential security bugs in the wild for malicious actors to find and exploit. The goal of a pentest is often to find and remediate security issues before they become an even more expensive problem. But if the hosting company doesn't set pentesters up for success, the likelihood of a worthwhile pentest is abysmally low. While a well-done pentest could cost hundreds of thousands of dollars for an application with a highly complex attack surface, a crappy pentest could cost millions in ransom payouts & GDPR fines by giving the hosting company a false sense of assurance while adding no extra protection against security breaches. Avoiding common pitfalls in application pentest planning will yield better results and ensure broader coverage of the target application.

Outline

• Intro

  • Cost of an average application pentest: Maybe 10K, or maybe 100K - depends how big your app is & how much coverage you want
  • You may just be hosting a webapp pentest to check a compliance checkbox. That's fine, but if you're going to spend the money, make sure you're getting the most bang for your buck.
  • There are a number of potential mistakes that could make a pentest go sour and waste your company's time and money

• Mistake 1: Being unrealistic about time budgeting

  • How long do you think it takes to conduct a thorough webapp pentest? If your answer is "3 days", you're wrong.
  • First couple of days of a pentest, assume there will be access issues that the testers have to work through to even get started. Don't expect productivity until Day 3+
  • If you require testers to go through onboarding or to use your company's equipment, make sure to tack on an extra 2+ days to your pentest.
  • Time budgeting should account for how many APIs & pages the testers will need to touch, as well as how many different roles (admin, guest, regular user, etc.) your system has. Granular Role-Based access controls? That takes a long time to properly test.

• Mistake 2: Crappy scope

  • Giving a pentester a URL and telling them to go nuts is probably not going to yield the best results
  • What keeps your team up at night? What is your company's absolute worst-case scenario? What findings do you care about, and what findings would you accept as an OK risk?
  • If you have a bug bounty, or other way for external users to report security issues, leverage that data to identify areas pentesters should focus their attention
  • With complex apps, consider breaking the test down further into individual features, and test only a few features at a time to minimize context switching
  • Beware of scope creep - Be very clear what is and is not a part of the pentest, and don't pile more on in the middle of the engagement. 3rd party services & libraries are generally off the table unless you have that 3rd party's written permission.

• Mistake 3: Hiring the wrong company

  • Do your diligence: compare & contrast at least 3-5 pentest providers
  • Ask them about their area of expertise. Look at their blog posts. If you are scheduling a pentest of an iOS app, and you hire a company that specializes in cloud security, your results may not be what you expect.
  • Ask around - what is the company's reputation like among your peers?
  • Ask the company for sample reports and make sure those reports meet your expectations
  • Good reports should contain very detailed & clear remediation guidance. Super boiler plate-y language is a red flag

• Mistake 4: Time-wasting & poor communication

  • Do everything you can to ensure testers have access to everything they need on Day 1. If you have 3 pentesters working together, wasting 1 day could cost upwards of 6 grand.
  • Be clear with testers about your communication expectations. Do you want a status update weekly? Daily? When do you want to be notified of findings (especially high & critical risk findings)?
  • Involve the right technical experts from your dev team in the communication process - make sure they understand, agree with and know how to fix any findings that come up!
  • Set up a designated channel where testers can ask questions, and devs can answer. Make sure someone is paying close attention to that channel. Resolve blockers ASAP - again, wasted time could cost thousands of dollars.

• Mistake 5: Poor preparation

  • If you don't have enough documentation, testers will not have a solid understanding of your product
  • Provision the right kinds of accounts for your pentesters. Ideally, having a minimum of 2 different accounts at each role/access level will help with finding authZ and privesc issues. Don't expect much if you ask pentesters to hack on your product without any legitimate logins.
  • Black box testing - not the best! If you trust the company to conduct your pentest, trust them to also look through your code.
  • Do some internal threat modeling in preparation for the test - provide the results of your threat modeling exercises to the pentesters.
  • If your product/feature is half-baked and doesn't always work, don't schedule your pentest yet. Wait till you are close enough to code-complete that pentesters won't run into weird error conditions just trying to use the product.

• Mistake 6: No plan for remediation

  • Make sure the devs who will be fixing the vulnerabilities are on your readout call so they can ask questions
  • Make sure you have sufficient information from the pentesters to understand & fix issues. Confused? Speak up!
  • Are you re-testing? When are you re-testing?
  • Got any SLAs? If not, what are the expectations for remediation timelines