I think you are right on the lottery only being for Brown Co. folks. It was started because of complaints much like yours. "Hey, why should I pay an extra .5% sales tax to pay for a stadium that hosts football games I will likely never get a chance to see in my lifetime?" As a compromise, they built in the "Lottery" seats and the average is supposed to be about 1 game every 5 years. But since not everyone registers, it gets higher. Only catch is that you can't pick your game and some folks don't want to go to pre-season games, cold games, etc.
Problem with buying a ticket at the stadium would be that it means the game isn't sold out and therefore wouldn't be on TV. But you can always find tix if you show up on game day (provided it isn't a big game). Often, folks sell them for face value around the stadium. If you can't find any in the parking lot, go across the street to "Scalper's corner" and you shouldn't have a problem if you are willing to pay. Especially after kickoff. Same as any other NFL city. Stadium is City of Green Bay property so no scalping ordinance applies. Across the street (any direction, the stadium is an "island") it is Ashwaubenon.
I'd rather see us sell out games and earn the revenue than have it the other way around.