The torrent client used for file sharing on these connections was qbittorrent, and listened for incoming connections using a random TCP port assignment that changed each time the client was restarted. Outbound connections used something in the high range on the local side (e.g. TCP port 59999) while on the remote side the port would also be random. It was possible to establish a connection to remote hosts using these same port numbers using the same Xfinity IP, router, etc. and using telnet on the local end and, say, SSH assigned to listed to a custom port number on the remote end.
Qbittorrent has a few somewhat unique routing features. For example, the client has an "Anonymous" mode that strips user agent and Peer ID identifiers from qbittorrent connections. When anonymous mode is enabled, qbittorrent ceases to accept any incoming connections that are not received through a SOCKS5 or l2p proxy and will only establish outbound connections to trackers using some form of proxy. Its cool stuff, particularly when used with l2p, however many trackers fail to meet its connection criteria which can make downloading torrents using those trackers impossible. Anonymous mode was not enabled on these clients. Furthermore, l2p or onion routing was not part of the equation. Many of the torrent connections were encrypted, however.
Other routing settings varied based on location; so for example some locations used port forwarding and others did not. All hosts consistently showed a "green" connection status, no errors or failures of any kind were reported. Seeds and peers would be counted as available for each torrent, however no transfer could be initiated. From the client, it looked like people were seeding but that their available upload slots or throughput was exhausted. This went on for days.
Before I get to the fix, I should note something that was very strange that occurred immediately before torrents ceased downloading. I was online when this occurred, with the qbittorrent client open. Suddenly, the number of seeds and peers for every torrent listed in the client began increasing very fast - faster than I had ever seen, and much faster than made any kind of sense.
 |
| In reality these torrents have < 5 Peers each |
My first thought after seeing this behavior was that a torrent client developer had screwed up. Maybe a very popular tracker was broken. Remember, this behavior occurred on multiple computers on multiple locations with different settings, with different torrents and different trackers. Every torrent was impacted, and these are not popular torrents we are talking about. As mentioned at the beginning, these clients weren't downloading music or movies. Torrents that had less than 5 Peers increased to 400 Peers in under a minute.
I went into the settings and applied a quota of 0kbps on downloads and uploads; I also applied this quota to transport overhead. I wasn't sure what was happening to the torrent distribution network, but I didn't want my client to have anything to do with it. I then shut off the client and went to sleep. The next day, I rebooted and started the client back up. The seed and peer listings remained consistent (they in fact remain consistent to this day). I risked letting my client talk to the network again by resetting the quotas and restarting the client. All transfer had stopped up and down, regardless of quota. Over the next few days I identified the same issue at several other locations.
After a few days of continuous failure, I decided to visit a web page that claimed to test whether an ISP is filtering bittorrent traffic from your internet connection. This was the site I used:
http://www.bittorrenttest.com/bittorrent-test/
It was more of a lark than anything; it wasn't clear exactly what this website did; here is what the website claimed:
When you click "START BitTorrent TEST", a BitTorrent-like transfer between your machine and our server is created and we determine whether or not your ISP is limiting such traffic. The test is designed to detect whether your ISP is using any of the following techniques. Throttling all BitTorrent traffic, Throttling all traffic at well-known BitTorrent ports and Throttling BitTorrent traffic only at well-known BitTorrent ports.
Based on this I operated on the assumption the site just opened a bunch of connections using high-numbered and known BT ports. As I mentioned at the top of the article, I knew I could use these ports to connect to other services. I was interested to see if web-based calls to the same ports were also successful and if not, which ports reported failures.
What I didn't expect was for my bittorrent traffic to be immediately restored during the port scan performed by the website. But that is what happened. After nearly a week of complete P2P downtime, qbittorrent immediately resumed transferring files (both up & down) at normal speeds.
I'll be frank: I have no explanation for why this restored the service. My hunch is that Comcast implemented a P2P filtering doo-dad, and that establishing connections to bittorrent ports by way of a web service convinced that doo-dad of two things. First thing: that the qbittorrent connections were related to the connections established to the web service. And second thing: that since the initial connections were to a web service, these related connections were to a web service also. However, this explanation leaves out why the Peer list grew exponentially right before the initial service failure. Qbittorrent doesn't have very good logging functionality, and while the source code for the client is available via Github, I don't have much time to mess with it ATM.
Continuing on the theme of not having much time to mess with things: I just stumbled upon this fix. I don't know, for example, if the fix will remain consistent or if I will have to continue to repeat the portscan after so many days or so many days offline or so much time after no connection has been established on P2P-related ports. As always, I welcome feedback from readers - but this time in particular, given the weirdness of the situation.
I'd just like to reiterate that, although my hunch is Comcast is behind this, I do not have conclusive evidence that proves that. While I have worked with Comcast, I have never been involved with anything outside of their commercial and data center operations divisions, and that was some time ago. It's been even longer since I directly handled ISP routing - I'm talking dsl and frame relay. The point is, the current guts of Comcast's residential network infrastructure is a mystery to me. Theres plenty of good karma to be earned for any of you techs over there who would enlighten us plebs on the subject.
No comments:
Post a Comment