Livers are very vascular, too, which means they bleed readily and voluminously. If you've ever seen or handled a human liver, you'll know that they're very bloody. It can make them difficult to deal with in a laboratory setting: they're slippery, they don't always section well for the preparation of slides, and residual blood present in the tissue can interfere with testing. A sick liver can also be very friable, which means it tends to fall apart readily when touched or handled, even if you're being very careful.
To complicate matters even further, patients with severe liver disease often have fucked up coagulation and are at substantially increased risk of both thrombosis (blood clots) and hemorrhage. It's pretty common for liver transplant recipients to require a shitload of blood products, including packed red blood cells, fresh frozen plasma, cryoprecipitate (a blood product derived from plasma that contains concentrated clotting factors), and platelets during the procedure and for 24-48 hours afterwards. Too much bleeding is obviously bad for the patient, but it can also make surgery more difficult by making hands and instruments slippery and obscuring visualization of the operative site. Then, once you start replacing the lost blood volume, you have to start worrying about fluid balance and hemodynamics and shit lest you fuck up some other useful organ and don't have a spare on hand.
With the tendency for bleeding and all the vessels that need to be anastomosed, some of which are extremely tiny, I'd imagine that the liver is a pretty tricky organ to swap out, especially if it's a living donor transplant.
I did a cursory Google search and it seems like most transplant surgeons agree with the guy you asked - it's lungs and livers all the way down. I'd bet the pancreas is a bitch to transplant, too, but that's a less common one. In general, your pancreas very strongly prefers not to be messed with.
Tl,dr: There is a reason your local transplant surgeon has a swimming pool AND a racquetball court in the 10,000 square foot house that he's never at.
Edited to say that this post had interesting links but the site hiccuped while I was trying to submit the post and I lost it.