While mounting observational evidence suggests that intermediate mass black holes (IMBHs) may be important in shaping the properties of dwarf galaxies both at high redshifts and in the local Universe, our theoretical understanding of how these IMBHs grow is largely incomplete. To address this, we perform high-resolution simulations of an isolated dwarf galaxy with a virial mass of 10^{10} Msun harbouring a 10^4Msun IMBH at its centre at a peak spatial resolution of < 0.01 pc. Within the fully multi-phase interstellar medium (ISM), we incorporate explicit sampling of stars from the initial mass function, photo-ionization, photoelectric heating, individual supernovae (SNe), as well as a Shakura-Sunyaev accretion disc model to track the evolution of BH mass and spin. We find that a nuclear star cluster (NSC) effectively captures the ISM gas and promotes formation of a circumnuclear disc (CND) on scales of < 7 pc. Simultaneously, gravitational torques from the NSC reduce CND angular momentum on (sub-)parsec scales, circularizing the gas onto the alpha-accretion disc and promoting sustained IMBH growth at ~ 0.01 of the Eddington rate. While in the innermost regions (< 0.5 pc), star formation is highly suppressed, the CND is susceptible to fragmentation, leading to the formation of massive, young stars. Interestingly, despite an in-situ SN rate of 0.3 Myr^{-1}, the dense CND persists, sustaining BH accretion and leading to its net spin-up. Our study demonstrates the complexity of IMBH accretion within a multi-phase ISM, and paves the way for next-generation studies where IMBH growth in a fully cosmological context can be captured.