There are many other costs associated with the act of borrowing other than interest costs. Example of such costs could be professional fees, arrangement fees, statutory fees etc.
While such costs may be considered capital expenditures, these costs are currently not tax deductible.
Recent budget annoucement has indicated a willingness to reconsider this area. Look out for more details from May 2007.