How to restrict attachment download to a specific user?

I have a very specific use case where the site built for a lawyer and each of his clients can login to their own ‘specific page/portal’ (custom post type) without the ability to access wp-admin etc. (I created all the login/register/profile-editing pages in the front end). In this page/portal the lawyer will leave messages and files for the client to download, now theoretically speaking, one client can guess (or if has knowledge of another client’s files) other file names and download then thus creating an issue with privacy/security/confidential material etc.