The Etrian Odyssey series from Atlus is probably my favorite series of all time, its on the DS and 3DS systems and it plays like oldschool SMT and Wizardry games. It's main gimmick is that you draw your own map on the bottom screen, but it also has the tightest difficulty balancing of any game I've ever played.
You make your guys at the start and you can carry 5 of them into the labyrinth, you're encouraged to have a team that covers eachother's weaknesses or emphasizes one particular strength of the team; if youre experiences enough with the game, you can also forego running a tank or healer. Later games also let you subclass for even more combinations.
Your TP (mana) basically acts as a meter for how far you can go before you need to head back to town and rest, similar to Persona and Metaphor's way of doing dungeons. Not being able to blast skills means the enemies can mop up pretty quickly, since most of your guys can die in 1-4 hits usually if you don't have any defense skills or healing.
Ailments actually matter a lot, usually each game has a few classes that apply ailments and others that benefit from enemies having ailments; infliction rates tend to be somewhat low, but once they're on, they do tremendous work (example is a blinding stab that has a 60% chance of infliction but reduces accuracy by 90%.) Etrian Odyssey also has a unique status called Binds, where inflicting a head/arm/leg bind will seal off skills that use that body part and reduce stats associated with the part, which can really help swing fights.
You need all the help you can get too, since boss fights are immaculately balanced to require competent team compositions, and even when you have one, they have enough HP and tricks to push your team to the limit.
Anyways you should definitely try one of the games and see how you like them, 4 tends to be the easiest for people to get into, 5 is my personal favorite, and 3 is the fan-favorite. Nexus somewhat relies on you having played the other games and calls back to them a lot, so I'd hold off on playing that one.