I have checked our code,
If the vendor address is empty then we showing the base address of the store using get_base_address() woocommerce function
and same for other fields.
If you change the flow then you have to custom code using below filter.
Using the above filter you can skip the setup wizard for vendor.
However coming to this line “he is taken to vendor store setup wizard and the weird part is: it loads admin address over there .”, please note as you are using the same browser for both vendor and admin, it is fetching the address from cache.