From my general understanding watchtowers, just watches the bitcoin lightning channels and makes sure no one is submitting a old payment channel transactions
Firstly, a correction. A watchtower doesn’t watch the Lightning channel per se. A Lightning channel is between two parties and although either party could choose to share more information on the channel to the watchtower there is no reason to. The only thing the watchtower is truly concerned about is what happens when your counterparty tries to close the channel. If your counterparty attempts a honest unilateral close with the correct latest state (ie correct balances in the channel) there is no issue and no need for a watchtower. However, if your counterparty attempts a dishonest unilateral close with the incorrect latest state (ie balances from a previous state in the channel rather than the latest) then you or your watchtower need to broadcast a justice transaction (we are assuming pre-eltoo here) in response to that dishonest unilateral close attempt. Hence the only thing the watchtower is concerned with is watching the blockchain, not the Lightning channel, for a channel close attempt from your counterparty.
(In the case of a honest unilateral close attempt the watchtower could inform you of what your channel counterparty is doing but the security model does not require it.)
What are some of the pitfalls and limitations of these watchtowers?
A watchtower is not required if you can stay online 24/7*, monitor the blockchain yourself and broadcast a justice transaction if you spot a dishonest unilateral channel close attempt from your counterparty. Hence if you don’t like watchtowers don’t use them. In this sense they are similar to an insurance policy. If nothing bad happens, your channel counterparty is honest and competent and you can maintain 24/7 uptime, then the insurance policy isn’t needed. But sometimes even when we don’t expect to we do fall back to relying on the payout of an insurance policy. If you employ the services of a watchtower you may end up to falling back to relying on them to spot the channel close attempt and broadcast the justice transaction.
*(As Sergei says in the comment technically the requirement is “not go offline for more than the channel’s agreed upon dispute window, which typically lasts for many hours or even a few days” but we’ll use 24/7 uptime for shorthand going forward.)
With regards to pitfalls and limitations, there are a few. It is not clear how to incentivize a competent watchtower to perform this service for you. Do you pay it a recurring fee out of band and leak some privacy to the watchtower? Does it receive a chunky fee from within a justice transaction if it is ever required to be broadcast? Then it is performing this service for you for free and only being compensated in what we expect to be an unlikely occurrence of a counterparty cheat attempt. To get round the incentives problem you may choose to run a watchtower on a different server to your Lightning node yourself and this will arguably be how most watchtowers are used to begin with.
Then there is holding the watchtower accountable. Suppose you are paying a watchtower a recurring fee out of band to perform this service for you. What happens if it fails to monitor the blockchain for you and (possibly) ends up failing to broadcast the justice transaction when necessary? You’ve been paying it for nothing. You could employ multiple watchtowers to mitigate this risk but that becomes more expensive and if they are all incompetent they might all fail to provide the service you have been paying for.
From a watchtower’s perspective they are required (pre-eltoo) to store a justice transaction for every previous state of the channel in case the channel counterparty attempts a dishonest unilateral close using that particular previous state. If the channel is particularly active that blows up storage requirements which ties into the previous discussion on incentives. There are also DoS concerns in that a watchtower might be asked to store data which they are told is relevant to the service they are providing but actually isn’t.
Finally there is going to be some kind of privacy leak to the watchtower. The watchtower needs to know which 2-of-2 multisig address on the blockchain it needs to watch and it needs to receive a new justice transaction for every new channel update. This new justice transaction could be encrypted, perhaps by data in the transaction broadcast by the channel counterparty that it only sees in the case of a channel close attempt. However the watchtower needs to be able to decrypt and broadcast it if required. Similarly to an insurance policy it is impossible to avoid some kind of privacy leak when you want to be able to fallback to a service provided by a third party.
There are a number of good presentations on this topic. The best ones are probably Conner Fromknecht at the Boltathon in April 2019 and Sergi Delgado at the Lightning Hack Day in May 2020.
