"Honom" works but excludes all the female little fishes :(
You could either use "henom", a gender neutral mixture of "honom" and "henne", or use "den" (equivalent to "it", which is quite common to use for animals). The pronoun "hen" is controversial but it's in the Swedish Academy's dictionary.
Furthermore, I would probably change "låt" to "lämna" and use "i fred" instead of "ensam". For example, the English expression "Leave me alone!" is "Lämna mig i fred!" in Swedish (literal translation "Leave me in peace!"). "Ensam" has more depressing connotations, e.g: "He was all alone after his wife died" - "Han var alldeles ensam efter att hans fru dog".
Hope this helps!
Just learning.