Several gunmen stormed the Corinthia Hotel and opened fire in the reception area. A car bomb also exploded nearby.
Unconfirmed reports say some of the assailants have blown themselves up. The officials say the dead include one US and one French citizen.
The security forces say the stand-off has now been brought to an end.
The US State Department has confirmed the death of a US citizen, without giving any further details. The dead American is believed to have been a security contractor.
The death of the French and American citizens have been confirmed by their respective governments.
There are conflicting reports as to the total number of attackers.
A Twitter account linked to Islamic State said the militant group had carried out the attack.
There has been strong evidence to suggest an IS presence in the eastern city of Derna since October, with a group there publicly declaring allegiance to its leader Abu Bakr al-Baghdadi.
However the command structure is still a mystery to most foreign observers.
The BBC's Rana Jawad in Tripoli says that in the past month there has been a string of incidents in western Libya, including abductions and bombings, that have been claimed by IS social media accounts.
However our correspondent says that it has not been clearly established whether these groups are IS foot soldiers or people inspired by them.
Attack threat
A civilian who witnessed the attack told the BBC: "I suddenly heard shots and saw people running towards me, and we all escaped from the back [of the hotel] through the underground garage. The hotel did a lockdown after that."
Different sources at the scene said there were between three and five attackers - video footage released later on Tuesday showed the body of a man reported to be one of the militants.
A security source told the BBC that one gunman had been arrested. Four security guards are among the dead and several people are reported to be injured.
A number of foreign companies have makeshift offices in the hotel, our correspondent says, and housing the few foreigners who remain in the Libyan capital has always been known to be risky.
One hotel employee told the Associated Press news agency that the hotel was mostly empty at the time of the attack.
Meanwhile, a hotel security source told the BBC that the hotel had received a threat "a few days ago" warning managers "to empty the building".
'Revenge attack'
The Corinthia Hotel is used by foreign diplomats and government officials. The UN Support Mission in Libya (Unsmil) has hosted several workshops at the hotel.
Libyan Prime Minister Abdullah Al-Thinni condemned the attack and pledged that those responsible for it would be brought to justice.
Federica Mogherini, the EU's foreign affairs chief, also condemned the attack which took place as a second round of peace talks between Libya's warring factions ended in Geneva in what the UN described as a "positive atmosphere".
The Twitter account linked to IS said the group had carried out the attack in revenge for the death of Abu Anas al-Liby, a Libyan jihadist who was suspected of involvement in the bombings of two US embassies in East Africa in 1998.
Liby died in a US hospital on 2 January, days before he was due to stand trial.
Libya has been hit by instability since the overthrow of long-time ruler Col Muammar Gaddafi in October 2011.
Numerous militias govern their own patches of territory, with successive governments struggling to exercise control.
Source: BBC