Here's a mechanic for creating suspense, for your players, not just for your characters.
Pass judgment on how good the PCs' diversion was, and pick a die with a proportionate number of sides; where a nice solid diversion might get a d10, a flimsy diversion might get a d4.
For as long as the diversion is in effect, roll the die each time the players take a significant action. If the die lands on the highest number on the die, the guards win and the PCs get caught. Be sure to tell the players what’s going on so they can enjoy the suspense!
Actions that would conceivably take up more game time (searching to find the file cabinet) can cause multiple GM rolls. This incentivizes players to plan ahead – suddenly casing the joint makes an actual mechanical difference
Every time an action could reasonably increase the PCs' chance of getting caught, count the target number down by one before you roll. For example: when the diversion begins, you pick a d10 and announce that the guards will appear on a roll of 10. But the PCs have to leave a dangling rope visible somewhere, and you announce that the guards will now appear on a roll of 9 or 10. Then they knock someone out and hide the body - you announce guards will now appear on an 8 - 10.
If a PC must pass through a wide open space on the map, you can temporarily reduce the target number. This gives you a handy mechanic for making un-stealthy actions more dangerous, without having to make up the guards' perception stats.
For failed stealth rolls, if you don't feel like making the guards come running every time a PC trips over something, rule that the failure lowers the target number as above.
With this mechanic, the likelihood of getting caught increases the longer the PCs stay in the area. But it’s still possible for the PCs to get caught on their very first action - and that would mean a loss of Potential Fun.
So the less fun the party has gotten to have snooping, the bigger the capture scene needs to be. In other words, if you ace your roll the first time they open a door, they don’t just get caught by the night watchman - they walk into a conference between the main bad guy and his henchmen and immediately end up in the dungeon (after seeing some important details of the Evil Plan, of course). If they get caught after they’ve found a few pieces of information, then have something slightly less bad (and probably less plot-escalating) happen.
If there’s one time your players are likely to call BS, it’s when NPCs they’ve never seen before come in to shut them down – with no suspenseful buildup. Using this mechanic solves both problems: it makes the suspense more immersive, and makes the “we got caught” scene a fun surprise for all of you – GM included.