Whatever happened to the descendants from Mary and Joseph of biblical fame?

I have been engaged in genealogical research for a few years and got to wondering last night why there are no families that claim to have descended from these two. People proudfully boast of their heritage particularly from those who have someone with claim to fame, whether famous or infamous, in their tree and while I can understand not knowing that your ancestor was Seth, the one-eyed shoe maker from BFE, surely, the fact that one of your blood relations was the source of an entire new religion or at the very least a major prophet to the existing religion, would be important enough knowledge that would be passed down from generation to generation, if only something to be used to get dates in the bar.
We know that back in the day lineage was important to people and this would have been common knowledge among family members and a cursory view of the Bible shows that there are several “begats” indicating the ancestry of certain important biblical figures including Jesus but then when we get to Jesus and his family all available information ceases. I understand there was no use in continuing to follow that particular branch but what about the other members of his family? They would not suddenly cease to care where and who they were from.
Here is the genealogy of Jesus as reported by Matthew:
1. Abraham
2. Isaac
3. Jacob
4. Judah & Tamar
5. Perez
6. Hezron
7. Ram
8. Amminadab
9. Nahshon
10. Salmon & Rahab
11. Boaz & Ruth
12. Obed
13. Jesse
14. David & Bathsheba
15. Solomon & Naamah
16. Rehoboam
17. Abijam
18. Asa
19. Jehosaphat
20. Jehoram
21. Uzziah
22. Jotham
23. Ahaz
24. Hezekiah
25. Manasseh
26. Amon
27. Josiah
28. Jeconiah
29. Shealtiel
30. Zerubbabel
31. Abiud
32. Eliakim
33. Azor
34. Zadok
35. Achim 36. Eliud
37. Eleazar
38. Matthan
39. Jacob
40. Joseph & Mary
41. Jesus

Forty-one generations, at least eight if not nine hundred years of information so obviously knowing who your ancestors were was important and we are told that Mary and Joseph had several other kids; at least four other sons, named James, Joseph, Simon and Judas and a couple of daughters who, evidently, were not important enough to have names but one was probably Mary. So that is at least six familial lines of possible descendancy. Where are these folks?

Assuming Jesus was Marys’ first, and if it happened, it occurred some time around the year 0. That gives us a timeframe of about 2016 years. Assuming 18 years/generation that would give us 112 generations to work with and let us assume, for arguments sake, that two children survive to reach breeding age from each generation, that would give us 2,596,148,429,267,410,000,000,000,000,000,000 descendants of just one of the siblings. Obviously not all of them survived. If we assume that every other generation half of the descendants die off before breeding we still end up with 72,057,594,037,927,900 people and that is just from one of the brothers or sisters. Surely someone would remember.