The only downsides - and what I see as one of the major stumbling blocks for self-driving vehicles - are suddenly changing and/or variable speed limits. Specifically school zones.
Around me, there are a couple places where the "regular" speed limit is 45 MPH, and 20 MPH in school zones. But it's only 20 MPH during certain times.
In some areas, it's "When children are present" - so the car would need to determine if children are present.
In some, it's "When lights are blinking" - at least that has an obvious signal that can be identified.
In some, it's "M-F, 7A-4P" or "School days, 7A-4P, or "School days, 7A-10A, 2P-4P" or variations. And some signs abbreviate it differently, have different active hours, etc. That means the car has to determine if it's valid or not. And "School days" is *SUPER* tough, since you may have some schools in session on some days and others not. (I live near where two different school districts boundaries are, and one starts 3 weeks before the other.)
And you don't want to get it wrong in EITHER direction. Obviously, you don't want your car going the faster speed when it should be going 20 MPH; but you don't want to go 20 when you should be going faster, either. And on mine, the automatic speed limit detection always sees the "20 school zone" and changes the on-screen to read 20.
Lastly, the BMW system still gets it wrong fairly often. I have one 20 MPH sign near me that it consistently reads as *70 MPH*. Yeah, that's not a mistake it should ever make. I've also had the reverse on the Interstate freeway - it reads 20 MPH when it should read 70! I wouldn't want my car slamming on the brakes on the interstate because it thinks the speed limit just got 50 MPH slower.
In theory, it sounds like a great idea - but it would require driver confirmation every time. Or a *MUCH* smarter system.