You see, I feel the very same way.
I used a supplied "Granny" unit for about seven months, before I had a wall box fitted for charging my previous VW PHEV - This was back in around 2016.
I know owners have used there supplied "Granny" units on a regular basis, to charge their PHEV's & BEV's and that's absolutely fine !.
I used the provided unit for about six months I think ?.
Then I started to see the odd owners where witnessing failures of their units while charging PHEV's and a few BEV's.
Some where replaced under warranty, but a couple had to pay for replacements - not cheap !.
I think that your general usage case, may help you to decide if you should consider a wall box or not.
A "Granny" unit is likely to get a much easier life charging a PHEV of a low mileage user, than the owner of a full BEV - Purely because the size of the battery pack in a PHEV is much smaller than the pack in a BEV - and that's only one reason.
On my previous PHEV - It would take just under about four hours to fully charge the smaller HV battery.
Over that time frame, the top of the three pin 13 Amp plug WOULD get fairly hot.
Not MAD hot, but without doubt, hot.
How hot is hot, I hear you cry.
Well, you would not want to pitch the LIVE brass terminal between your finger and thumb after removing the plug straight from the socket outlet, when it had just finished charging.
It would burn your skin, let's put it that way !.
I know you would NOT do this, but I am trying to give you some idea of the level of heat generated here.
You are passing a constant 10Amps through the terminals of a fuse that is ratted at 13 Amps.
Therefore over a long time scale, heat will be created across that 13 Amp line fuse.
So, when subjecting a "Granny" unit to the same level of load for charging a BEV ( with a much larger battery ) heat IS going to build up without any doubt at all.
You need a GOOD quality socket outlet at the point where you will be charging the car from, that is ideally wired onto it's own dedicated circuit in the C.U.
Charging a mid sized EV battery ( 44+ kw's ) on a "Granny" unit is likely to take its toll on a "Granny" unit over time as well, purely given the length of time the unit is pulling that type of demand.
Something else that maybe worth considering here folks.
IF - Now your only form of transport is now, your lovely new EV .
Question :- Have you considered a "Back Up" plan, SHOULD your "Granny" unit suddenly just decide to fail ?.
If you are lucky enough to live close to either a public charger / rapid and it's working of course, then your covered, until you can get a replacement unit through the dealer network, which may take some time ??????.
So, speaking for myself, after pondering through the above items that I have mentioned above, back in 2016, I made the decision to invest in a wall box.
Not because I am a really high usage case ( 16,000 miles in two years ) but I wanted something that I considered was a more suitable / purpose built EV wall box that future proofed me IF I made the jump into a full BEV ( well we know how that ended - ZS EV of course in 2020 ).
I wanted a device that was built with the ability to provide a much higher given load rate, over a shorter time frame, one that could take full advantage of our cheaper "Off - Peak" rates and provide a charge in the early hours of the morning.
Waking up to a car that had been already cheaply charged, to a SOC level that we wanted to suit our needs, was brilliant.
Charging either a ZS EV or a MG5 ( or pretty much any EV from a low SOC to 100%) was not going to possible, in the cheap "Off - Peak" window on the provided "Granny".
I think "Octopus Energy" offer you about a four hour window, for using their cheaper rate ????.
Well, that's clearly not going to work from a given low SOC and wanting a full charge, from a "Granny" unit.
Four hours of "Off - Peak" is not going to get you any where CLOSE to a full charge.
You will then be forced into charging on your much more expensive standard daily rate tariff, or charge cheaply over two different nights !.
Thats not really practical in many cases.
Therefore, the supplied "Granny" for our PHEV was retired then was only there as the second "Back Up" charger ( should the wall box ever hit a problem ).
The well rested "Granny" unit, then came out when we went on short mini break away, when we knew the charging infrastructure was going to be poor at the location.
Not all places have a charger available, but almost ALL we be helpful in providing a three pin plug socket outlet !.
The ZS EV has been charged ( over the last two years ) via the wall box.
The MG provided "Granny" has made a few trips away and been put to good use, but on our return, it now lives on the garage wall.
I do like having SOME type of a back up plan in my back pocket.
It's just the way I am sorry ?

.
Like having a spare wheel / Type 2 public charging cable / a battery booster pack and a 10mm spanner in the boot, it's just the way I operate.
Not smug, just cautious

.
So in conclusion ( at last I hear you say ).
Would I recommend getting a wall box as a better, faster charging solution, that then offers you the back up of using the supplied "Granny" unit as spare - 100% YES YES YES !.
"Granny" units are NOT cheapest thing to replace IF they are not covered under warranty !.
You could be 50% into the price of a wall box, just to replace one.
Maybe THEN you will consider a wall box, I would hazard a bet.
Each to their own I know, but I would never be totally reliant on just a "Granny" unit for my charging needs, that's for sure.
Egg's in one basket, and peace of mind, I think they call it ?.
Or just plain CRAZY, you choose ?.