Greetings!
The biggest problem is your system isn’t complete, and instead of solar coming last, if at all, you did that first. Solar s optional, more reliable charging methods are not.
A proper system charges the house battery(s) while driving FIRST, then by either a generator or shore power as a second choice, and solar only if you have money to burn. If you do solar, for a 100ah house battery you want at least 300-400 watts of solar panels, but even then you’ll need the other choices because solar can not be relied upon, especially in the winter when the days are short and frequently cloudy.
The odds are your inverter is drawing power constantly, even when not in use. There is a way to turn the inverter on and off so that it is only on when the fridge is running, but a smaller inverter would actually be more efficient. Most people choose simple ice chests or fuel powered fridges because electric/battery powered ones are just too big of energy hogs. The 3-way fridges offer the best of all worlds, battery powered while driving, and shore power or fuel powered when camped. 3-way fridges can be found used pretty cheap, but they do require outside venting. I just use an ice chest myself, simple, cheap, foolproof, and no added heat inside during the summer, plus I don’t believe in adding any extra holes in my van for any reason.
I gave up on solar and don’t use an inverter. Everything is 12v only. On the rare occasion that I need shore power, I can power up my generator and have all that I need. 99% of the time, just driving keeps my house battery charged, for the other 1% I use my generator with a battery charger that can also be used on shore power. The generator and battery charger combo also allow me to charge or jump start my starter battery if the need ever arises, but they get the most use charging the dead batteries of people who rely on solar. My cheapie, no name inverter generator cost $99, and the battery charger cost $29. The generator will run for up to 8 hours on a single gallon of gasoline.
There is also a good possibility that you have killed your house battery by abuse. Solar only is extremely hard on batteries. These days I get used, but nearly new, deep cycle batteries at junkyards dirt cheap, like under $20, and since I use proper charging methods, they’ll last 5-7 years.
Cheers!
"I can live like a king because I work like a dog." ~ An anonymous vandweller