What is your level and the other players level? You don't have to be on the rival list for a player to keep attacking you. If you are within XP range of each other, you could be showing up on the battle/fight list everyday. An Ambush can kill you depending on your health level when you're hit with one.
The 5 attacks the system is showing you is the ambushes you set, they are counted as attacks.
Per the help section under Rival Squads: 6. Successful ambushes count as an attack against your opponent.
On a personal note, I find it fun to blow through someones counters/ambushes they set all the time. If for no other reason, it's because I can. Especially when I know that once I get through all the counters/ambushes I'm going to beat them down anyway. There have been players that I've played this way with & they eventually figure out a way to up their defense & get stronger.
Then they come back for me.
I love the strategy. It's all part of the game.