A Technical Deep Dive on How HicEtNunc Was Recovered
HicEtNunc (HEN) is the #4 most active NFT platform. It is the most active app on the Tezos network with up to 85% of all Tezos contract calls on some days.
On November 12, @HicEtNunc2000 changed its profile description to discontinued. The founder, Rafael Lima, was unreachable by the community. Thousands of creators were wondering what would happen to their livelihood. Tens of thousands of collectors worried about their NFTs. The ecosystem was imperiled.
Our team had an emergency meeting, and started accounting for all the potential systemic risks and what we could do about it. Would HicEtNunc.xyz be taken down? Would the contract be pulled? What was the state of IPFS mirroring? Are people's NFTs safe? What about the images of the NFTs? What happens in people's trust in NFTs on Tezos?
Two hours later with direct community input, we launched HicEtNunc.art.
Fighting the fire
Minimizing uncertainty
Profiles on DNS.xyz are built Tezos-first, with showcases for HEN NFTs. A downtime would have made things significantly worse for our users. Our analysis was that if HEN went offline, it would: 1) create fear and uncertainty among the creators and collectors that rely on HEN for a living and for fun 2) harm the reputation of Tezos 3) drag prices down.
It looked like solutions were at minimum, days away. Twitter threads emerged wondering if the original creator would change his mind, if some of HEN's developers would bring the site back up, or if some of its competitors would run a mirror instance. Our priority was to minimize the damage from uncertainty.
Minimizing downtime
In order to get the site quickly back up on a mirror instance, it would have to be a simple clone of the original. This meant cloning the original HEN repository, and running a new instance.
We contacted Manitcor (@Manitcor), developer of TezTools, and explain our plans. He explained that he could help, as he already had an instance working in case of a rainy day at https://hen.teztools.io. This made the deployment of a new instance even easier: rather than having to deploy the code from the HEN repo from scratch, this would be a clone of his existing mirror site.
We started working on deploying the yet unnamed new instance. We also started working on deployment scripts for Heroku, so that we could easily deploy more instances easily if needed.
TezTools summarized these steps on Twitter:
The missing link(s)
We now had a new instance up and running. We needed to give it a name. At that point in time, only hen.teztools.io was running (as a backup), and no other instance. In addition, everyone's HicEtNunc.xyz links had died. This meant that thousands of artists had their links broken and began posting on Twitter about the impact on their livelihood, as well as what would happen to their content. The largest trend was to post "RIP HicEtNunc" updates and NFTs. Shokunin (@DNSCEO), our CEO, looked for Twitter Spaces to announce the efforts and talk with the community. The hope was to give people reassuring news in real-time, so they could spread updates to their timelines.
- We joined a couple of spaces and requested speaker access, but they were busy talking about the death of HEN and asking people to mint "RIP HEN" NFTs. The largest Space was the one hosted by Kevin Mehrabi (@KMehrabi), with over 250 attendees. We joined this one and he gave us a speaking slot. Some people thought this meant that he was involved in the efforts, or that we had a business relationship with him. We do not. He clarified this himself later:
On the Space, we gave real time updates of our collaboration with TezTools on the new mirror site. We needed a name that people would adopt quickly. It would need to be easy to remember, and the url paths for the NFTs and usernames would need to be compatible with HEN. Ideally, it would be a small change that any user could do by hand. We floated a number of options on the Space: should be it HicEtNunc? HEN? Maybe something else? Should it be .art, .co, something else? The people on the call were unanimous and liked the name HicEtNunc.art.
We now had a live mirror site at HicEtNunc.art. It was fully compatible with .xyz, most notably, utilizing the same contracts as before. All people had to do was to change three letters to see their NFTs. This was achieved within 2 hours of the site being taken down by surprise.
TezTools also shared these updates in tweets.
Twitter seemed pleased, and we hoped this reflects maintained confidence in NFTs overall:
Risks We Found with HEN
Throughout our time working with TezTools on the .art mirror site, we reviewed the entire infrastructure, website, and contract in order to find out what might need fixing. We looked for flaws, risks, or any items that could represent outstanding problems for the Tezos NFT community. To look for potential issues with the contract, our team started working with Codecrafting (@Codecrafting). He is one of the most reputable smart contract developers and the co-creator of the specification for FA2 NFTs on Tezos. We saw that even with a HEN mirror site up its NFTs was still in a precarious state:
- HEN IPFS data had a single point of failure and lost a number of NFTs
- Malicious site mirrors can set additional fees and destination wallets on top of the swap contract (v2)
- Unrevoked site permissions enable the swap contract to control a user's NFT
- The swap contract is unilaterally controlled by a single wallet held by Rafael Lima
- Contract could be made to force swaps to fail
- Royalties can be changed adversely and suddenly on creators and buyers
- No ability for contingency fixes
Decentralizing HEN's data to preserve everyone's NFTs
On one hand, HEN is more decentralized than most NFT marketplaces. Usually, when you go to an NFT platform, you'll need to login, enter your email, have a password. These sites use web2 service providers, making everything centralized. HEN is a static web app, which does everything from the browser. It syncs with your wallet to sign operations, and everything else happens on your client via APIs. Our team happens to be familiar with that unusual approach, because DNS is built with 100% decentralized profiles. Our architecture doesn't require signing messages to make every changes to the page, but the ideas are similar.
However, even though anyone can clone the entirety of HEN and run it off their machine, there are aspects that are very centralized. HEN only pins its data in one place: a single Infura instance. They are a great hosting platform, but the point of using IPFS is to make sure you'd use multiple nodes to ensure your data is decentralized. This doesn't appear to be the case for HEN NFTs. We saw this as a significant risk: if the Infura instance goes offline, or runs out of budget, the data may no longer be pinned. If nobody runs a node that has the same items pinned, the data is gone forever.
Simply being on IPFS doesn't make something decentralized
Using the data from Hicdex, TezTools was able to provide us with a list of every IPFS hash used by HEN. We used this data to look into whether or not some hashes had become unreachable. Our review of the data stored on Infura shows that 449 hashes are unreachable. Because of the way HEN pins data, an NFT can be represented by up to 4 hashes. This means that somewhere between 110-449 NFTs were lost by the use of Infura's hosting. Here is a list of all the missing hashes.
We have been informed of other efforts to mirror the entirety of HEN's data. The easiest way right now to ensure your HEN data is pinned elsewhere is to use NFTBiker's tool to pin all the items you have created or collected. You could also use Henpin to do the same if you're more technical.
We are making efforts to reach out to teams who might also be working on mirroring the data off of Infura. We have not received any replies in the 40 hours since we posted it. This is unsurprising: the hosting cost for all this data and traffic is approximately $10,000 / month.
While losing these NFTs is bad, it is only a fraction of HEN's 500,000+ NFTs. We have made it a priority to mirror the content to more nodes, in order to decentralize the content off of Infura.
How mirrored are HEN's NFTs?
To understand the scope of the problem, we had to analyze the distribution of the IPFS data from HEN across the IPFS network.
We took a sample of HEN's NFTs and looked at their CIDs. Our sample size was 21,813 CIDs. For these CIDs, we wanted to see how replicated they were across the IPFS network.
The IPFS network is hard to traverse, due to its decentralized nature. The best way to get a sense of if an NFT is available is to see which nodes announce that they are storing this CID in the DHT. It is by no means accurate. In our analysis, only 72% of HEN's NFTs hosted on the Infura instance showed in the DHT. This is likely because the remaining 28% haven't been queried in a long time. Still, traversing the DHT is the most reliable way to get directional results. If a file is not on the DHT, it simply doesn't exist to the network. It is only available if you query a specific node for it, and resolving it might take long.
We looked for nodes that were hosting our sample CIDs. Once we found these nodes, we looked at how many CIDs from the sample size said node was hosting. It would give us a sense of the coverage this node has of the sample CIDs.
What we found is that the overwhelming majority of NFTs from our sample were only on the Infura node. Only 5 other instances covered 10%+, and one at 9%. The rest is a long tail with a handful of NFTs. This means that even though the data is on IPFS, the vast majority of NFT data on HEN is centralized. Based on the host names, we assume these nodes are part of the NFT.storage service provided by Protocol Labs.
Ensuring IPFS decentralization
Our team has worked with IPFS since its inception. We contacted our friends at Pinata.cloud, the largest IPFS hosting service. On Friday evening, we kicked off a very large data migration task. It was probably one of the top data migrations on IPFS ever. The NFT data is being mirrored three times, on three different nodes. In addition, these are dedicated nodes that are independent from the rest of Pinata's infrastructure. This means that the rest of the IPFS traffic wouldn't affect the performance. The nodes also come with better caching.
The migration in numbers:
- Total migration time: ~36 hours
- 1.4 million unique CIDs were migrated
- Total content size: 4Tb of unique content (total deduplicated size)
* note: IPFS natively deduplicates content under the hood. If multiple objects share same child objects, then those child objects are only stored once
We were able to retrieve every single NFT item, besides 449. We haven't been able to find other mirrors that are providing this.
Examining the Contract and Uncovering Potential Risks
Looking for potential issues with the contract, our first priority working with Codecrafting was to identity whether or not it would be possible for the owner of the contract to cancel swaps, change user data, or other miscellaneous exploits. The HEN contract pays the swap fees to a single wallet controlled by the Rafael Lima, the creator of the site. That wallet also controls the contract. Originally, the swap fee was 2.5%. On November 13th, the fee was changed and settled on 1% after attempts to make it lower caused the contract to fail.
Our deep dive with Codecrafting found an exploit in the HEN site code and core contract vulnerabilities.
Unrevoked site permissions enable the contract to control a user's NFT
A flaw in the original HicEtNunc.xyz site code enables the contract to take control of NFTs a user receives back after initially sending it elsewhere. This is likely widespread in the site mirrors.
The site does not make the right smart contract calls for swaps. The UI shows that the smart contract has permission, and calls the smart contract to initialize the swap. However, it never revokes the permission to perform actions on the user's behalf. One risk is that if the user ever received that NFT again, the smart contract would be able to take full control of what the user can do with it. In such a scenario, the contract could theoretically put the item on sale without the user knowing.
Admin Has Exclusive Access on Key Contract Functions
We reviewed the raw Michelson contract on Better Call Dev. The functions the admin has sole exclusive access to are:
- Setting the fee
- Passing the admin ownership to another wallet or smart contract
We haven't made any changes to the contract. In the following section we further discuss the potential fallout from this exclusive access.
Current Contract Risks
Swaps can be unilaterally forced to fail
When a user tries to collect an item on HEN (eg: complete a swap), this makes a call to the swap contract to collect the item and agree on the swap that was proposed by the NFT's owner. This initiates a transfer to the manager (in this case, the owner's wallet) to collect the fee. If the contract manager happens to be a smart contract, that contract is executed. If that contract were to fail, the entire call to collect the swap would fail. This would essentially disable all means of completing a swap and buying NFTs with the swap contract.
The ownership of the contract could be pointed to a contract that would not perform anything. Historically, something like this happened when transitioning from the v1 to v2 swap contract: the v1 contract was made to fail, in order to get users to adopt the v2 contract.
Royalties can be unilateraly changed on creators and buyers
Swap fees can be suddenly changed on the community, up to 100%, by the single contract owner.
Site mirrors can set additional fees and destination wallets on top of contract
In addition to risks with changes that can be made by the owner of the contract, we have found another exploit in the contract due to the way it expects data from the site UI.
The HEN swap contract expects the royalty percentage and the original creator of the NFT to be passed by the site UI to the smart contract. Any malicious UI can pass a different percentage and destination wallet to the contract and completely bypass the spirit of the contract. The best solution for the short term is to use only trusted mirrors of HEN. This will require refactoring of the contract and further security audits to have a proper fix in place.
Centralized contract ownership prevents contingency fixes
Because the contract was controlled by one wallet, it is impossible to change anything in the contract if something happened to the owner. There is no way for a group of people to perform contingency fixes.
In addition, the fees also go to that single wallet, and makes it difficult to split them for various needs.
The correct fix for this is to change the ownership to a multisig wallet, and/or a DAO. This would ensure there is a way to share rewards and decision-making across multiple actors.
Conclusion
With the help of TezTools and Codecrafting, our team was able to fix a very large fire, in a record time. We were able to review the entire frontend, infrastructure, and smart contract during the course of three days. While it is true that anyone can simply clone the HEN repo and put it online, we had to ensure that the damage to the community was minimal, and that exploits and other risks were understood. The work needed to make HEN future proof is far from over. It will require refactoring, an infrastructure overhaul, and significant testing.However, the worst is behind us! 99% of the NFTs are preserved.
We have migrated the ownership of HicEtNunc.art to the TezTools team, who will be managing it on behalf of the Hen_community.
We take a lot of pride in the work our team was able to produce in the span of three days over the weekend, and look forward to continue helping the Tezos community as we focus on developing DNS!