Without knowing your health background, it is difficult for me to answer that question. In most cases, doses of >800-1000 IU of Vit-D supplements are well tolerated, though whether they result in tangible health benefits remains unclear. If your doctor has valid health concerns behind his recommendation, he should be able to explain those clearly.

I’d like to direct you to a recent review of the issue “Vitamin D supplementation: upper limit for safety revisited?” (https://link.springer.com/article/10.1007/s40520-020-01678-x#Sec6) which possibly addresses your concerns, and which you could bring up in your next discussion with your doctor. If you have any question after reading the paper, feel free to ask me here.