Maybe the first question is what your budget is, both regarding money and time. For example, you could buy a pre-configured NAS from Synology or QNAP, which requires less technical skills but more money, or a home-made solution reusing used components (but fresh disks for reliability). Depending on your electricity costs, you may want to choose a low-power solution or something which you power off when not used. For storage, maybe a three-disk RAID5 is a good compromise. For backups, plain S3 cloud storage encrypted via restic is a good idea.
Yes, XMPP with proper TLS on the server side and Conversations or one of its forks (preferably fetched from F-Droid) using OMEMO encryption should be good enough. If you are brave or paranoid, give Tox a try: https://tox.chat/