Ron Baron, one of the largest shareholders in the company, said that his seat on the board of directors is insignificant and that he views Musk's 9.2% stake in the company as insignificant.

Baron told CNBC that it was a tiny investment. He has a company that is worth a trillion and is on the way to being worth $3 or $4 trillion.

Baron, the CEO of Baron Capital, said that his asset management group decided not to buy any of the company's stock when they found out that the CEO had joined the board. A person for Baron wouldn't comment. The company did not respond to a request for comment.

Musk disclosed his stake in the company. The shares of the social media company are trading around $47 per share, up over 20% from Friday's close.

There were concerns that Musk's involvement with the social network could have repercussions for the company.

He said that focusing on something that will never have an impact on anything takes away from looking at the big picture.

The Washington Post reported on Thursday that the SEC could investigate Musk after he reportedly filed his disclosure 11 days late.

Baron is not the only person with close ties to Musk who is upset with his decision to join the board of directors. Steve Westly, a former board member of the company, said on Tuesday that Musk would shake things up at the micro-blogging site.

Baron's full interview can be found on CNBC's website.