When any of you guys make custom/new monsters for 5E stuff, what's your general philosophy?
Reskin, reskin, and reskin some more. There was a meme among various OSR YouTubers that was "just use bears" since it's a great stat block for almost any kind of monster.
That said, I heard some advice that you basically just need AC and a to-hit bonus. I never thought of this for DnD, which is odd because Savage Worlds has the concept of the d4 fodder, the d6 mook, and the d8 elite which is the same concept. For DnD, you might want to add some spell effects or vulnerabilities or something, but really anything like that should jump out at you.
Do you have a sense of what's a reasonable challenge for your PCs from just raw experience
Raw experience and a few near misses. I only had one TPK, but that was from a published adventure.
I think difficulty isn't really the point. You want to challenge the players, obviously, but I find adding a location specific modifier does that well enough. Difficult terrain, wild magic, sight blocks, environmental hazard, etc. The goal is to make it so they aren't using the same tactic all the time (most of the time is fine though). It doesn't take much either. eg. If the fighters strategy is running in a straight line to get into melee, it'll really fuck up his day to put the enemy up a tree, on the other side of a pit, or even on the other side of a bar. Got a guy who stands in the back throwing ranged attacks? Fog or heavy rain can obscure his vision. Even if it's just "you're attacks have disadvantage past 20ft" it might be enough for him to try something else.
One I had good luck with was weak monsters that come back to life some turns after they're killed. Another was a fight in a theater where the performers use illusion magic for the show. When the fight started there were walls of flame everywhere that messed up the PCs tactics until they realised it was all illusions. A memorable encounter was when the party had lots of magic items, I made an entire session take place in a wild magic zone.
Even if you roll in the open, you can still fudge things by not using powerful abilities you had planned, or lowering that AC a point a two provided there hasn't been a near miss.
You can also have a get-out-of-jail-free card in your back pocket. Most DMs call it railroading, but your players don't need to know.
In one game, I set up a rumour early on that drow slavers were seen. If there was a TPK they'd be saved by the drow, just to be taken into slavery and the next adventure would be a jail break. You can also have monsters flee just as they're about to deal the killing blow because of something big happening like a dragon roar or a bell tolling, like this scene.
You can't pull this all the time, but if it makes sense, this safety net allows you to not worry about balance since the end result is the same, the players just don't know it.
And worst case scenario, you can come clean and admit things are poorly balanced.