A few things to clear up here.
I did not realize how relatively underdeveloped the state of Battletech orbital defenses were. I thought since advanced stuff was used by the WOB in Jihad Final Reckoning that similar systems had existed before. I jumped to an erroneous conclusion about the state of BT Universe tech based upon my Jihad readings.
They existed during the Star League era in far greater numbers (and technological advancement) than they did during the Jihad. Final Reckoning offers up information on Casper drones and other space-based systems, as I recall. The Star League had these systems perfected and massively implemented on a significant number of core Terran Hegemony worlds. The naval battles seen during Final Reckoning and JHS: Terra pale in comparison to the battles seen during the Star League Civil War (specifically, look up the information on Terra and Epsilon Eridani off the top of my head). It's also possible that portions of these types of systems were implemented during the Star League era on some of the House capital worlds and Golden Worlds (I honestly can't recall). But during the Civil War, these systems were absolutely wrecked, so much so that I believe pieces survived on Terra only. These sort of defenses make systems a difficult nut to crack open, which means the conflict gets ugly, bloody, and nuclear very quickly. So, defense systems do exist. And there are even some more recent developments on that front (the Combine's Battlesats and Capital Missile equipped submarines, the Blakists Rattler, etc.). But again, they are very limited in production and implementation. With good reason honestly, as these systems seem to serve far more as a giant red target than massive deterrent. From what has been seen in BT's history, they tend to work...once. Some one either tests them and loses, or cracks them and wins. Either way, the cost is high. I can understand why more systems stick with run of the mill patrols, assault craft, Pocket WarShips, and fighter squadrons. If you keep making big sticks, so does your opponent (perhaps not in the real world, but BT and the real world parted ways a long time ago).
Naval sensors are a whole other beast in BT. I know that various folks have made very heavy technical breakdowns of how they tend to work in BT, and you can probably search for such threads on the official BT forums. To give a very simple reduction, space is BIG, REALLY BIG. Planet-based sensors picking out even jumping vessels, particularly if pirate points are used (or vessels jump in above or below the orbital plane) let alone fleets attempting any sort of communications blackout protocol...well, those sensors are going to struggle. Generally, drive plumes are the easiest way to detect things, but it is generally too late by then but do much more than prepare for contact in a few days. Same goes for the neutrino flares from fleets jumping in. Once they are in-system, you just better hope your defenses are up to snuff. And again, all of this is tempered by the fact that Battletech is about the ground battles. While some fleets do get stopped before landing, it is more than exception than the rule. Story needs and the community's interests tend to dictate that enemy forces reach the ground in some capacity, no matter how "unrealistic" it may seem. Rule of cool and all that.
And an interesting historical point about the time period from the Hegemony to the Star League. I'm afraid I'm not up on setting history that far back so I'll take your word for it.
If you're right that's an interesting notion: it may not be warships that are the problem.....merely large fleets of them.
From my readings in Dark Age stuff, there does seem to be anecdotal support for that statement.
That's always been my take on it. Others may obviously feel otherwise.
The only thing that sorta runs counter to all this is that I've seen alot of posts on the Battletech official forums with people asking about either warships or shipyards and post-Jihad era and ever time I can recall the answer from TPTB has been destroyed, mothballed, or the like. I'll have to go back over forum posts and double check my memory on this last point. I definately got that impression....but it may well have been me misunderstanding. I'll have to do further reading on that point to make sure I'm not way off base.
You're not off base. And it doesn't really run counter to the setting. There are a few factors in play here. You have the concern from outside the setting that continuing on with larger fleets is problematic, as was noted before. You also have a setting that has seen a level of destruction on par with the later stages of the 1st and 2nd Succession Wars, which are pretty much the most destructive periods in BT's history.
These two periods have quite literally hundreds of systems dying out entirely. It is unrestricted warfare at its most grotesque levels. The Jihad reached the same type of conflict (unrestricted warfare), though not the scale of those two earlier conflicts. This means that infrastructure is shattered. While it doesn't quite reach the levels of destruction seen on the Hegemony worlds after the Star League Civil War, it isn't far off in some cases. Just look at a handle full of these shipyards. Many of them were in key production planetary systems, and were among the first targets by the Blakists. Galax, Alarion, and so on. And it doesn't even end with the Jihad. The Bloody Millenial attacks do even more damage in this regard, doing lovely things like dropping giant rocks on Free World League aerospace production worlds, or the Regulans deciding to nuke everything until it glows. So, yes, the Star League Civil War and the Succession Wars were much worse. On the flip side, the Houses had much more production capability after the height of the Star League era (leading into the Succession Wars), so they could lose more. Production in the modern era is a drop in the bucket compared to that Golden Age. So when the FedSuns lose Galax and have massive damage to Kathil, we're talking about a near death blow to their naval production. The Blakists were crazy, but they weren't stupid (at least not all the time). They hit hard, early, and frequently. So, interstellar shipping has been massively disrupted. A new faction has been born around Terra. The Blakists drop back in a few years later for even more terror attacks. And these factions STILL haven't even really recovered from the losses of the Jihad at this point. So yeah, orbital facilities outside of getting merchant ships back in production take a backseat for quite some time.
You also have to add in the morale factor here. So you're a Successor State ruler, and you've lived through the Jihad. Your citizenry have seen the skies of a number of your worlds darkened by enemy forces. Some of these worlds (in some cases, even your own damn capitals!) were lost to the enemy for years. Your people have watched their fleets batted aside in nuclear fire. They've heard stories of cyborg monsters from a crazed religious order slaughtering people in the streets. And your might WarShips weren't enough to protect them. You are now faced with the prospect of rebuilding. Your armies are shattered, your navies are broken, and your production capabilities dropped so far in some cases that you were using units last produced during the Age of War several hundred years prior to the formation of the Star League (Retrotech). So, do you sink insane amounts of C-Bills into getting shipyards rebuilt that can produce at best, less than a dozen units a year? Or do you take that same money, rebuild your ground forces, your shattered civilian infrastructure, AND still have money left over to build some shipyards that can produce three or four times the number of smaller defensive units (Pocket WarShips)?
Now let's assume that you know the other states are going for the smaller defensive swarm fleets. Nobody is rebuilding WarShips with any real priority. Do you want to buck the trend? Will your two battlebarges do anything for you against your ancient enemy's dozen or so swarm fleets that can hit far more targets? I know which way I'd lean, but again, others may disagree. This was the situation they were facing at the end of the Jihad. Massive reconstruction on worlds that in many cases (like the capitals and some of the interior production worlds) had not seen even a raid win the last two hundred years. So, everybody sort of said "We'll get to the WarShips...eventually." Next thing you know, eventually becomes 40 or 50 years, and now it is even harder to get such a project off the ground. Research has been lost or forgotten, people don't see the need anymore, and armies are already on shoestring budgets due to the general downsizing of military forces.
This is what you're facing at the end of the Jihad and leading into the Dark Age. As I said earlier, WarShips are not gone forever. They're just taking a siesta over the corner. And that fits within the setting. It happened to some degree after the Reunification War. Again after the Star League Civil War (though in those two cases, on a much shorter/smaller scale). Again after the Succession Wars. Again after the FedCom Civil War. This pattern repeats again, and again. Honestly, the factions have actually come out of the Jihad better than they have in most of those other prior conflicts. At least this time around, they have the possibility of rebuilding their fleets at some point in time. During the Succession Wars, they even lost the ability to come close to constructing a single WarShip ANYWHERE.