The scheme was considered by Southwark Council's planning committee at a virtual meeting held on Microsoft Teams on Monday night.
CIT and Union Car Parks owns the site which is bounded by St Thomas Street, Fenning Street, Vinegar Yard and Snowsfields.
Much of the site is now used as an open-air market and in recent years it was occupied by temporary offices during the redevelopment of London Bridge Station.
The scheme – designed by architects KPF – includes a 20-storey office block facing St Thomas Street, and a smaller pavilion at the junction of Snowsfields and St Thomas Street leading to an underground music venue with capacity for 200 people. A subterranean set of artists' studios is also proposed.
The office building would have space for around 2,000 staff, whilst the ground level would include a market hall to be managed by the people behind the existing Vinegar Yard pop-up market.
Planning officer Terence McLellan told the committee that the site had been identified in planning policy as suitable for a tall building.
Mr McLellan acknowledged that the new building would have a significantly adverse impact on daylight and sunlight to homes on Melior Street and Snowsfields, but said the effect was judged to be "acceptable on balance".
The council's design and conservation manager Michael Tsoukaris described the KPF scheme as an "exemplary design" that "reflects the industrial character of the area".
Mr Tsoukaris noted that the scheme involves the demolition of a two-storey warehouse on Fenning Street – which is part of the Bermondsey Street conservation area – which would be replaced by the 20-storey glass-clad lift and stairs core for the office block.
"We are satisfied that any harm is outweighed ... by the public benefits of the development," he said.
Toby O'Connor of the Old Bermondsey Village Neighbourhood Forum told councillors that the scheme represented "extreme overdevelopment" that would cause "significant harm" to the conservation area and set a "dangerous precedent" for future developments.
Mr O'Connor suggested that the long-term impact of the current pandemic on the demand for office space had not been considered: "There is a question of whether we really need these offices that have been designed in the previous era," he said.
Local resident and landscape architect Jessica Beattie warned that the height and massing of the building would create poor conditions at ground level in terms of the effect of wind.
"Vinegar Yard will be a front door to Southwark for the 90 million people who use London Bridge Station each year," said architect John Bushell of KPF.
He said that the proposed building would have "a brick and warehouse aesthetic that reflects the character of old Bermondsey".
Mr Bushell said that Guy's and St Thomas' NHS Foundation Trust had already expressed interest in taking office space in the building.
CIT's Ewen Puffett said that construction of the building would take around three years, and the company was hoping to make use of off-site prefabrication to speed up the process.
London Bridge & West Bermondsey councillor Humaira Ali addressed the committee to set out ward councillors' opposition to the scheme, including concerns about daylight, sunlight and lack of green space.
"This really doesn't cut the mustard," she said.
She highlighted the social housing block at 8-20 Snowsfields where she said residents would be faced by "a looming building which has a massive impact on their lives" and underlined the potential impact on pupils at Snowsfields Primary School.
Discussing the application before the vote, committee members were largely unsatisfied by the scheme.
Cllr Kath Whittam said she agreed with objections tabled by the Victorian Society warning that the Horseshoe Inn would be "completely swallowed up" by the new 20-storey block.
She said: "As a heritage asset [the pub] should be celebrated and shown off rather than covered up."
For Cllr Barrie Hargrove, the scheme's location in an designated 'opportunity area' and next to a transport hub were important. "My view is the benefits massively outweigh the harm," he said.
He added: "Conservation and history is only of use if it's of benefit to the future."
Cllr Adele Morris said that the upsides for the borough as a whole were "quite strong and quite obvious", but the impact on the locality was less positive: "The people living next door to it are least likely to feel the benefits of the scheme."
Cllr Margy Newens said she was unable to support the scheme which "had paid scant regard to the conservation area in which it seeks to implant itself" and was too dense.
Cllr Catherine Rose agreed that the scheme was too big and regretted the impact on the pub and the loss of the Fenning Street warehouse.
Sensing that the mood of the committee was moving towards refusing the application, chair Cllr Martin Seaton took the meeting into closed session to take advice from officers.
A motion to approve the scheme was voted down, with two councillors in favour and six against.
A second motion – that planning permission be refused on the grounds of "excessive height, scale and massing" and impact on the Bermondsey Street conservation area – was carried unanimously at the end of the five-hour meeting.
• Councillors had also been due to consider an application by Shard developer Sellar for the adjacent Vinegar Yard warehouse site and land between Snowsfields and Bermondsey Street, but a decision on that scheme was deferred to a future meeting.